Theme
Alerts and what to do with each
Jua tells staff when something needs a person. Alerts arrive in two places:
- The portal's Notifications page, for every admin, linking to the page the alert is about (the Crypto page by default). The unread count shows in the side nav and the header's bell.
- Out of band, when the operator has set
ALERT_WEBHOOK_URL: every alert is also posted to a chat channel or on-call relay as JSON with a severity, urgent first, one message a second, repeats folded into "+N more like it". An in-app alert is seen only while someone has the portal open, so real money should always have the webhook.
Severity: urgent (money may be going wrong: page someone now), warning (needs staff today), info (news). A title starting URGENT is urgent.
Platform alerts
| Alert | Meaning | What to do |
|---|---|---|
| URGENT: housekeeping has not run for N minutes | No housekeeping pass has finished for HOUSEKEEPING_STALE_MINUTES (10). Markets do not settle, settlements and TON orders wait | Tell operations at once. With the API's own timer: check the API's log for [housekeeping] errors. With HOUSEKEEPING_TIMER=off: check that the scheduler calls POST /api/cron/housekeeping every minute with a current CRON_SECRET. See Housekeeping |
| URGENT: the API cannot reach its database | The watchdog could not read the database for that long. Nothing is credited, paid or settled | Operations: the database (Supabase) or the connection string. Out of band only: the portal itself cannot read either |
| Housekeeping is running again (info) | A pass finished after the alert above | Nothing |
| Early resolution proposed | A staff member proposed a result before the market's closing time | Read the market's rules and the proposal. If it is premature, reopen trading before the window ends |
| Resolution disputed | A trader disputed a proposal | The Disputes page: Resolving and disputes |
| ID submitted for review | A customer uploaded an ID | Users → ID review: Users |
| One-time codes stopped: the hourly ceiling is reached | The built-in sign-in sent OTP_MAX_PER_HOUR codes in an hour: a flood | Look at the sign-in traffic (an attack, or a launch); raise the ceiling only if it is real |
| Some staff alerts were left out of this channel | The webhook queue was backed up; the least severe alerts were dropped from the channel, never from the portal | Read the portal's Notifications for what was left out, and find why the receiver was slow |
Trading from TON wallets
Recorded by the TON worker as durable alerts, each once, and listed among the Crypto page's events needing staff.
Delivery depends on the custody pass
These TON alerts are delivered to admins by the crypto housekeeping pass, which runs only where a custody provider is configured (CRYPTO_PROVIDER=vault). A deployment without one records them in the provider_events table (type alert) but does not deliver them: operations must watch that table or the TON worker's log until that is changed. See Monitoring and alerts.
| Alert | Meaning | What to do |
|---|---|---|
| TON trading is stuck on one of the exchange's transactions | The watcher cannot bring one exchange transaction into the books, and reads nothing after it: matches, closes and new orders wait | Operations: the alert names the transaction, its kind and the error. Usually an indexer disagreement or a decoding problem; see Incident runbooks |
| USDT on TON is held by the exchange for no one | Someone sent USDT to the exchange with too little TON for it to be sent back; the exchange keeps it as stranded | The alert gives the amount and the sender. The owner multisig returns it with PayStranded (never more than is counted). Tell the customer it is coming |
| Shares on TON are held by the exchange for no one | Someone sent shares to the exchange with too little TON for them to be sent back; the exchange keeps them as stranded | The alert gives the amount, the outcome, the market and the sender. The owner multisig returns them with PayStrandedShares (never more than is counted). Tell the customer they are coming |
| A message JuaTonExchange sent bounced | A payout or shares the exchange sent came back (out of gas, in practice); what it carried stayed with the exchange | Operations: check get config (bounces) on the exchange and the account it was for; return what is owed from the owner wallet |
| A referral commission payout on TON is not confirmed | A collection left the fee vault over an hour ago and was neither delivered nor returned (the referrer's USDT wallet may have refused it) | Check the referrer's wallet on a TON explorer. Delivered: settle the commission_payouts row by hand. Not delivered: ask operations to look at the vault's payout state |
| The TON operator wallet is short of TON | The operator wallet cannot pay for the messages waiting (it keeps a 1 TON reserve); matches, closes and refunds wait | Top up the operator wallet's TON (the alert gives the address). See Recurring tasks |
| The TON market maker is short of USDT or of its budget / …wallet is short of TON | Jua's own market maker on TON cannot quote | Top up its wallet's USDT or TON, or lower TON_MARKET_MAKER_MAX_USD; customers' money is never involved |
Custody alerts (the vault model)
Where a deployment still runs the on-chain vault (CRYPTO_PROVIDER=vault) for balances from before trading from wallets, its watcher raises alerts about deposits, payouts, RPCs and the operator key: URGENT: the vault paid out under an id that is not ours, The vault operator is low on gas, The vault watcher has stopped advancing, The RPCs disagree… and others. Each is explained, with what to do, in the repository's docs/crypto.md under When something goes wrong; the short version is pause withdrawals first (Crypto page), then investigate.
What customers are told
Customers are notified of their own events: fills, a trade the chain refused ("A trade could not be settled", nothing left their wallet), a limit order taken off the book ("Order cancelled: your wallet can no longer cover it"), trading paused, a proposed result, a dispute's decision, a cancelled market, a commission earned. Those are not staff alerts, but support should know them: Troubleshooting.