Housekeeping: the timer and the cron

Everything scheduled in Jua runs from one housekeeping pass, started every minute. It is what expires orders, closes and settles markets, sends matches to the chains, pays commissions, translates listings and keeps Demo moving. If it stops, customers' trades stop settling, so a watchdog pages staff when it does.

What a pass does

In order, each step isolated so one failing never stops the next (apps/api/src/lib/housekeeping.ts):

1. Market housekeeping (runHousekeeping): expires good-till-date orders, closes markets whose closing time has passed (cancelling their resting orders), finalises undisputed proposals whose window has ended, opens the next edition of recurring events.
2. Retention (runRetention): once a day across every instance (the retention:daily claim in job_leases; skipped: true the rest of the day), deletes what nothing reads any more: sessions past their expiry (customers', staff's, the Mini App's), notifications read and older than 90 days, portfolio snapshots older than the two years the chart shows. Price points and the audit log are kept. The pass that did it logs [housekeeping] retention: deleted ….
3. The house market maker (runMarketMaker): play money only; re-quotes every open market.
4. Crypto housekeeping (runCryptoHousekeeping): the vault model's watcher, payout queue, cancel follow-ups, provider upkeep, the hourly reconciliation, and delivery of durable custody alerts. Runs only with CRYPTO_PROVIDER set; otherwise it returns at once.
5. The settlement worker (runSettlements): matches to CTF Exchange, confirmations, undoing refused ones, conditions, results, EVM commission payouts. Null while the exchange is off.
6. The TON worker (runTonExchange): the Mini App's exchange. Null while TON trading is off.
7. The heartbeat is recorded, and HOUSEKEEPING_HEARTBEAT_URL pinged.
8. Demo housekeeping: fills resting demo orders against the books, settles demo positions, advances simulated payments.
9. The Claude jobs, each started in the background and not awaited (a request can take minutes), each one at a time under its own lease, each a no-op without ANTHROPIC_API_KEY: listing translations; the listing scout, which runs when its last finished run (heartbeat listing-scout) is two hours old; and automatic resolution, which looks at a few closed events a pass (the AI). The cron route starts the same three, so an external scheduler translates and runs the AI too; their outcome is in the API's log ([translations], [listing-scout], [auto-resolution]), not in the route's answer.

At most two passes of one process overlap, so one slow pass never stops the next from settling markets, and slow passes never pile up.

Two ways to run it

The API's own timer (the default)

Every API process runs the pass on a one-minute timer, five seconds after it starts, and also nudges it from GET /api/events so a browsing customer keeps things moving in development. While TON trading is on, the same process also runs the TON worker every five seconds, so a Mini App order is booked and settled within moments. Across several API instances, each worker's database lease keeps it to one instance at a time.

Nothing to configure. This is how the compose deployment runs today.

An external scheduler (HOUSEKEEPING_TIMER=off)

For several API instances or none long-lived, set HOUSEKEEPING_TIMER=off on the API and call the cron route every minute from a scheduler. Then only the scheduler runs the jobs: a page view never does, so anonymous traffic cannot start chain work in every instance.

curl -fsS -X POST https://juamarkets.com/api/cron/housekeeping \
  -H "Authorization: Bearer $CRON_SECRET"

CRON_SECRET is required in production: an unset secret closes the route rather than opening it (finalising settles money). The route answers with what the pass did (closed, finalized, settlements, ton, demo…). Schedulers that work: Supabase pg_cron + pg_net (an example is at the end of packages/db/drizzle/0002_realtime_storage.sql), a systemd timer on the VM, or any cron.

TON needs more than a minute's cadence

With the API's timer off, nothing runs the TON worker between passes: Mini App orders are booked and settled only when the cron route is called. Call it more often than once a minute (every 5 to 10 seconds) while TON trading is on, or keep the API's own timer.

The heartbeat and the watchdog

• Every finished pass records a heartbeat (job_heartbeats, job housekeeping) and, when set, pings HOUSEKEEPING_HEARTBEAT_URL with a GET: point a dead man's switch at it (healthchecks.io, Cronitor, Better Stack), which raises the alarm when the pings stop, even with the whole machine down.
• Every API process checks the heartbeat every minute, whoever runs housekeeping. When no pass has finished for HOUSEKEEPING_STALE_MINUTES (10; 0 turns it off), one instance claims the alert and raises it, URGENT, in the portal and through ALERT_WEBHOOK_URL: "URGENT: housekeeping has not run for N minutes", with where the last pass finished and what to check. "Housekeeping is running again" follows when a pass finishes.
• When the database itself cannot be read for that long, nothing can be recorded or claimed, so "URGENT: the API cannot reach its database" goes straight to the webhook, once, until the database answers.

Test both before real money by stopping the scheduler (or the API) for ten minutes.

Leases

Workers that must run one at a time across instances take a row in job_leases (exchange:operator, ton-exchange:operator, the translation lease…) with an expiry. An instance that dies holding a lease is simply waited out: the exchange worker's lease lasts two minutes, after which another instance takes over. The leases are also how the watchdog's alert is claimed once per silence.

Reading the logs

cd /opt/jua && docker compose --env-file .env.server --env-file .env.release logs --since 10m api | grep -E '\[(housekeeping|exchange|ton-exchange|crypto|demo|translations|market-maker)\]'

A quiet pass logs nothing; a pass that did something logs one line per worker ("[exchange] sent 3, confirmed 2, failed 0 settlements"). Errors are [housekeeping] failed: …, [exchange] …, [ton-exchange] the pass failed: …, with URLs and keys redacted.