Incident runbooks

Three incidents, each with what to do first, what the attacker or the fault can and cannot do, and how to recover. In every case: pause first, understand second, resume last, and write down what you did as you go.

Pausing

What each pause stops, who can set it, and who can lift it. Customers can always collect winnings and merge pairs on chain, whatever is paused; nothing below traps anyone's money.

Pause	Set by	Lifted by	Stops
CTFExchange pauseTrading()	The owner Safe	The Safe (unpauseTrading)	Every match on BNB Smart Chain. Redeeming and merging on CTF still work
JuaTonExchange Pause	The operator wallet or the owner multisig	The owner only	New orders and matches on TON. Cancels, expiries, redemption and fee claims still work
JuaVault pause()	The operator key or the Safe	The Safe only	Commission payouts out of the EVM fee vault
JuaTonVault Pause	The operator wallet or the owner	The owner only	Commission payouts out of the TON fee vault
A market's trading switch (portal)	Any staff member	Staff	New orders on that market, on every venue; resting orders can be cancelled
Trading across the event (portal)	Any staff member	Staff	Every outcome of an event at once
The Mini App (telegram.enabled off)	An admin, on the Telegram page	An admin	Telegram sign-in; the app says Jua is taking a break
The site banner (Settings)	An admin	An admin	Nothing; tells customers what is happening

An operator-side pause (the vaults, the TON exchange) is the fast one in an incident: the operator key can set it without gathering the Safe's signers. The exchange's own pause needs the Safe.

Stopping the API (docker compose stop api) stops matching on every venue too, since only Jua's operator sends matches, but it also stops the site. Prefer the switches above.

A stolen operator key

Assume it the moment the API host is suspect, a payout nobody asked for appears, a match you did not make shows on the operator address, or the signer alert fires.

What the key can and cannot do

On	Can	Cannot
CTFExchange	Settle any two signed orders whose prices cross, at their prices or better, including resting orders customers cancelled on the site (a signed order has no expiry on chain); charge a fee on each fill up to maxFeeRateBps, into Jua's fee receiver	Take anyone's money or shares outside a signed order; change the fee ceiling; add operators; unpause
JuaVault	Pay out up to maxPerPayout per payout and dailyLimit per 24-hour window (two windows' worth across a window's turn) to any address; pause	Sweep; raise the limits; unpause; pay an id twice
JuaTonExchange	Send matches within each order's own terms; close orders (their money goes back to their owners); open markets; pause	Fill at a worse price or for more; pay anyone but an order's owner or counterparty; unpause
JuaTonVault	Pay out within its limits to any address; pause	Sweep; raise the limits; unpause
The API host	Everything else in .env.api: the database, the service-role key, the RPC keys, the Telegram token, the webhook secrets

So customers' money is bounded by their own signatures; Jua's exposure is the vaults' daily limits and the fee ceiling.

Steps

1. Pause what the key can touch, from the owners. Safe: CTFExchange.pauseTrading() and JuaVault.pause() (the operator key can pause the vault too; do it from wherever is fastest). Owner multisig: Pause on JuaTonExchange and JuaTonVault.
2. Remove the key. Safe: CTFExchange.removeOperator(old), JuaVault.setOperator(0) (revokes without a replacement). Multisig: SetOperator on JuaTonExchange and JuaTonVault to a fresh wallet.
3. Treat the whole host as compromised. Rotate everything in .env.api (Keys and roles): the database password, the service-role key, the RPC and indexer keys, CRON_SECRET, the webhook secrets, the Telegram bot token (/revoke in BotFather, then Register the bot again), the Anthropic key. Rebuild the VM if the intrusion is on the host rather than in a leaked secret.
4. Find what it did. The operator address on BscScan: every transaction, against settlements and commission_payouts (a matchOrders or payout with no row is the attacker's). The fee vault's PaidOut events against commission_payouts. On TON: the operator wallet's transactions against ton_messages. Note every customer affected (a cancelled order filled, a wrong-recipient payout).
5. Make good. Fees charged above Jua's rate on attacker-sent matches went to Jua's fee receiver: refund them to the customers from the treasury. Payouts to the attacker's address from the vaults are Jua's loss, bounded by the limits.
6. New operator (Rotating keys), in a remote signer this time; settings-only deploy; addOperator(new) and setOperator(new); then unpause from the owners, exchange last. Watch the first matches confirm.
7. Tell customers what happened and what was done, in the site banner and the Telegram banner.

A chain RPC or indexer outage

Symptoms

• BNB Smart Chain: settlements rows stay submitted with attempts climbing, or pending with a node error in last_error; chain_markets rows with connection errors; the API's log full of [exchange] … errors naming no URL (they are redacted). Customers see trades pending for minutes, then "A trade could not be settled" as lost transactions are undone after ten minutes.
• TON: the two indexers must agree on every read, so one lagging or failing indexer halts the worker: ton_cursor stops advancing, ton_messages stay unapplied, the log says [ton-exchange] the pass failed: … or the stuck alert is recorded. Mini App orders sit "Sent. Waiting for the TON network…".

Steps

1. Confirm it is the provider, not the chain. Compare the chain's height on the explorer with what the RPC answers (curl -s -X POST $RPC -d '{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber","params":[]}'); check the provider's status page. For TON, call toncenter's and tonapi's /masterchain-info or the account's state and compare.
2. Replace the failing endpoint. Edit EXCHANGE_RPC_URL or EXCHANGE_RPC_URL_SECONDARY (or TON_TONCENTER_URL
   • TON_TONCENTER_API_KEY, TON_TONAPI_URL + TON_TONAPI_KEY) to another provider, from another company, and run a settings-only deploy (about a minute). Two providers, always: one lying endpoint must never decide money.
3. Watch recovery. The exchange worker rebroadcasts unmined matches and confirms by receipts; a match whose nonce was used elsewhere and that no RPC ever mined is undone after ten minutes, and the customers told. The TON worker resumes from its cursor; messages applied without an answer for ten minutes are sent again by themselves. Nothing needs resending by hand.
4. If a transaction is stuck at a nonce (the operator's pending queue blocked behind one the network will not mine): speed it up or replace it at that nonce from the operator key with a higher fee. The vault pays an id once and the exchange refuses a repeated fill, so a replacement that lands twice cannot double anything.
5. Tell customers if it lasts more than a few minutes: the site banner and the Telegram banner.

While the chain itself is down

Nothing settles, and nothing is lost: orders rest on Jua's book with their money in the customers' wallets (EVM) or under their orders in the exchange (TON). Consider switching trading off across the busiest events so prices do not move on fills that cannot settle. When the chain returns, the workers catch up by themselves.

After any incident

• Write the timeline, the cause and the fix into the team's incident log.
• Add the missing alert or check to Monitoring.
• If a key was involved, finish the rotation of every secret, not only the one that leaked.