Monitoring and alerts

What tells you the platform is healthy, where alerts go, and the queries that show the state of the chain work.

The signals

Signal	Where	Means
Container health	docker compose ps on the server; the deploy's health gate	api answers /api/auth/config, web and admin serve /icon.svg. Proves the processes are up, not that pages render
The housekeeping heartbeat	HOUSEKEEPING_HEARTBEAT_URL (dead man's switch); job_heartbeats	A pass finished in the last minute
Staff alerts	The portal's Notifications; ALERT_WEBHOOK_URL	Something needs a person (the list)
The API's log	docker compose logs api	Per-worker lines; errors with secrets redacted
The chain	BscScan: the exchange, the CTF, the fee vault, the operator address. A TON explorer: the exchange, the vault, the operator wallet	What really happened
The database	The queries below	What the workers believe

The alert channel

Every staff alert is an in-app notification to every admin and, with ALERT_WEBHOOK_URL set, a POST of JSON:

{
  "id": "…", "source": "jua", "site": "https://juamarkets.com",
  "severity": "urgent", "title": "URGENT: housekeeping has not run for 10 minutes",
  "body": "…", "href": "https://admin.juamarkets.com/crypto", "at": "2026-10-30T09:00:00.000Z",
  "count": 1, "text": "[URGENT] URGENT: housekeeping has not run for 10 minutes\n…\nhttps://admin.juamarkets.com/crypto"
}

text is what a Slack- or Mattermost-style incoming webhook shows as is; x-jua-alert-severity carries the severity, and x-jua-signature: sha256=<hex> the HMAC of the body under ALERT_WEBHOOK_SECRET. Delivery: urgent first and never dropped; one message a second; a 429's Retry-After honoured; repeats of a waiting alert folded into one with a count; past 200 waiting, the least severe are left out and counted in a summary; a receiver failing a message backs off that alert alone. Route urgent alerts to something that pages a person.

TON alerts without a custody provider

The TON worker records its alerts durably in provider_events (type alert, provider ton-exchange), and the crypto housekeeping pass delivers them to admins. That pass runs only where CRYPTO_PROVIDER is set (the vault model). On a deployment without it, which is the current non-custodial model, these alerts are recorded but not delivered: until that is fixed in the code, watch the table:

select received_at, payload->>'title' as title, payload->>'body' as body
from provider_events
where type = 'alert' and payload->>'notifiedAt' is null
order by received_at desc;

Queries

Against the production database, read-only, from a checkout with DIRECT_URL (never from the server):

-- Matches not yet on chain, or not confirmed, older than ten minutes (should be empty)
select id, venue, status, attempts, last_error, created_at, submitted_at
from settlements
where status in ('pending', 'submitted') and created_at < now() - interval '10 minutes'
order by created_at;

-- Markets whose condition could not be prepared on CTF, or is taking long
select market_id, status, attempts, last_error, created_at
from chain_markets
where status = 'failed' or (status in ('pending', 'submitted') and created_at < now() - interval '10 minutes');

-- Results waiting, with why
select market_id, status, attempts, last_error
from chain_results
where status = 'pending' and last_error is not null;

-- Commission payouts that failed or are unconfirmed
select id, venue, status, amount_micro, wallet, attempts, last_error, created_at, submitted_at
from commission_payouts
where status = 'failed' or (status = 'submitted' and submitted_at < now() - interval '1 hour')
order by created_at desc;

-- The TON watcher's cursor
select * from ton_cursor;

-- Jua's messages to the TON exchange not applied yet
select id, wallet, status, last_error, created_at
from ton_messages
where status <> 'applied'
order by created_at desc
limit 20;

-- Housekeeping's last finished pass, and the workers' leases
select * from job_heartbeats;
select * from job_leases;

Column names follow the Drizzle schema (packages/db/src/schema.ts), camelCase there, snake_case in Postgres.

Balances to keep an eye on

Account	Threshold	Why
The EVM operator's BNB	Top up below 0.05 BNB; keep 0.1 or more	Every match, condition and commission payout is a transaction it pays for. There is no alert for the exchange operator's gas unless the vault model's watcher runs: watch it yourself
The TON operator wallet's TON	Several TON above the 1 TON reserve	Alerted when short, but only delivered with a custody provider (above)
The fee vaults' USDT	≥ what referrers can still collect	Sweeping
The TON market maker's USDT and TON	Its TON_MARKET_MAKER_MAX_USD budget	Alerted when short

A cheap way to watch the operator balances: a scheduled job (the dead man's switch service, a small script) that reads eth_getBalance for the EVM operator and the TON wallet's balance from toncenter, and alerts below the thresholds.

Server health

cd /opt/jua
docker compose --env-file .env.server --env-file .env.release ps        # api, web, admin healthy
curl -fsS http://127.0.0.1:5600/api/auth/config                        # the api on loopback
curl -sI https://juamarkets.com/ | head -1                              # HTTP/2 200
curl -sI https://admin.juamarkets.com/login | head -1
curl -sI https://docs.juamarkets.com/ | head -1
df -h /var/lib/docker                                                   # images need room (the deploy prunes old ones)

The health checks prove each process is up. A page that fails to render, or a relay that cannot reach the api, passes them: after a deploy, sign in to the site, open an event and see the order panel; sign in to the portal.

What is not monitored by the platform

• Supabase itself: use its dashboard's alerts, and turn on point-in-time recovery.
• The TLS certificates: certbot renews with --deploy-hook "systemctl reload nginx"; certbot renew --dry-run checks.
• The RPC providers' quotas and the indexers' plans: a provider that caps eth_getLogs or rate-limits silently shows up as settlements that stop confirming. Two providers each, always.
• Disk on the VM: images stream in at each deploy; the deploy prunes old tags and the build cache when tight.