Live checks

An automated suite that checks the deployed site — the customer site (juamarkets.com), the staff portal (admin.juamarkets.com) and the docs site (docs.juamarkets.com) — for functionality, security and performance. It runs from GitHub Actions on demand, nightly, and after each successful production deploy, and it is read-only and production-safe by default. The code is in e2e/live/; .github/workflows/live-checks.yml runs it.

This is not the build-time gate (that is CI) and not the local end-to-end tests (bun run e2e:crypto, e2e:exchange, e2e:simulation, which stand up their own stack). It points at a running deployment and asks whether it actually works, is actually hardened, and is actually fast.

What it checks

Functional (read-only): every public page in all 11 languages (home, the markets list with a category and a country filter, an event page found through the API, leaderboard, learn, suggest, the legal pages, the login and referrals entry points, the Telegram Mini App shell at /tg/) — 200, the right lang/dir (Arabic right to left), translated text actually present (not the English fallback), no page or console error in a real browser, and every image, font and script loading; a crawl of the internal links finds no 4xx/5xx; robots.txt, sitemap.xml (well formed, its URLs resolve) and the Open Graph image route. The API answers the shapes the clients expect (/api/site, /api/auth/config, events list/detail/series/books, /api/exchange terms, leaderboard, and whether the OpenAPI document is public), with consistent numbers (prices in 1–99¢, a multi-outcome event's YES prices roughly summing to 100, non-negative volumes, uncrossed books) and the caching headers the code sets. The production configuration go-live needs is reported (the money mode as /api/site gives it, never assumed) and checked: demo login not offered with real money, no dev OTP display, no staff quick sign-in, wallet sign-in offered, WalletConnect configured, Telegram config present when the Mini App is on. The staff portal's login loads, every other portal page without a session redirects to /login (with next), and no staff data is in the HTML; the docs home, a user guide page and the walkthrough video (HEAD: content type and size) answer.

Security (non-destructive): TLS (a valid certificate per host, days to expiry, TLS 1.2+ only, HSTS, the http→https redirect, www behaviour); the headers on pages, the API and the portal (CSP with a per-request script nonce and no unsafe-inline scripts, frame protection, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, no Server/X-Powered-By version leak) and the cookies public responses set (Secure, HttpOnly, SameSite, the __Host- prefix); exposure (/.env, /.git/*, /package.json, node_modules, source maps of the client bundles, the cron route without its secret, the API reference in production, a 404/500 without a stack trace); authorisation (every /api/admin/* and /api/me/* route, each method, refused without credentials and with a forged or expired token; object routes refused a random or foreign id — never another account's data); input handling (SQL/template/path-traversal/XSS strings and unicode/RTL/emoji in search, slugs, query parameters and headers never 500 or reflect unescaped; malformed, oversized and wrong-content-type bodies are 4xx not 5xx); request forgery and CORS (a cross-origin cookie write refused, an arbitrary Origin not reflected with credentials, Host and X-Forwarded-Host spoofing not reaching a staff route or reshaping links); and one bounded probe per rate limiter (sign-in codes, wallet sign-in, password reset, the SSE stream) that a burst meets a 429 with Retry-After. The workflow also runs a passive OWASP ZAP baseline scan (spider + passive rules, no attacks); its known-accepted findings are in .zap/rules.tsv.

Performance: Lighthouse (mobile, throttled) on the home, markets, event and /tg/ pages, with budgets in e2e/live/budgets.json (LCP, TBT, CLS, total JS, score); the Telegram Mini App's caching (its content-hashed files under /tg/assets/ kept for a year, its page checked with the server on every open); the signed-in Mini App home, which Lighthouse cannot reach (it cannot sign in, so its /tg/ run sees only the screen for visitors outside Telegram): with LIVE_TELEGRAM_BOT_TOKEN and writes on, the home is opened as the test Telegram account on a phone with slow 4G and a 4× slower CPU, and its layout shifts are scored as CLS against the /tg/ budget, naming the elements that moved; API latency (p50/p95/p99 over a run of requests per endpoint) with budgets; and, within the load level, a small load generator over a weighted mix of public reads and pages (throughput, error rate, p50/p95/p99, the slowest endpoints), which aborts early if the error rate passes 2% or p95 doubles from its first-ten-seconds baseline.

The safety rules (enforced in the code)

• Read-only by default. No request creates, changes or deletes anything unless LIVE_WRITE=1. The write tests use only the test identities in the environment, never a real customer.
• Every request is identified with User-Agent: jua-live-check/1, so ops can filter it out of the logs. (ZAP identifies itself with its own agent.)
• Rate-limit probes stay within the limits (read from apps/api/src/lib/rate-limit.ts), one bounded burst per limiter that checks a 429 appears, from a dedicated test identity.
• Load stays within the level's budget and aborts the moment it degrades (see above). Heavy load is never the scheduled default and never runs under the production profile without LIVE_LOAD_CONFIRM=yes.
• Nothing is ever pointed at the live site from the sandbox. The suite is validated against a local production-like stack (bun run e2e:live:local).

The profile: preprod → production

LIVE_PROFILE decides how far the suite goes. It defaults to production (the safe profile) when unset.

• preprod (before go-live, with no real customers): write tests run with the test identities (LIVE_WRITE on by default), the heavy load level is allowed without the extra confirmation, and the workflow adds a ZAP active full scan (a separate job that runs only under preprod). With funded test wallets it also runs the "impossible scenarios" — the simulation's hacker probes and trader flows against the remote target (see below).
• production (at go-live, with real customers): read-only by default, the passive ZAP baseline only, and heavy load only with LIVE_LOAD_CONFIRM=yes.

Set the repository variable LIVE_PROFILE to preprod now; flip it to production at go-live. The dispatch input can override it for one run.

How to run it

Locally, against any deployment

LIVE_SITE_URL=https://juamarkets.com bun run e2e:live

Output goes to e2e/live/out/ (gitignored): a readable console log, results.json and summary.md (what a GitHub job summary shows). The process exits non-zero on any failure.

Configuration is all environment variables:

Variable	Default	Meaning
LIVE_SITE_URL	https://juamarkets.com	The customer site to check.
LIVE_ADMIN_URL	the portal of the default site, or localhost:3002 for a local site	The staff portal.
LIVE_DOCS_URL	the docs of the default site	The docs site (the docs checks SKIP when unset).
LIVE_PROFILE	production	preprod or production (above).
LIVE_SUITES	functional,security,performance	A comma list of the suites to run.
LIVE_LOAD	light	off, light (10 users × 60s) or heavy (ramp to 100 × 5 min).
LIVE_LOAD_CONFIRM	—	yes to allow heavy under the production profile.
LIVE_WRITE	off (prod) / on (preprod)	1 to run the write tests with the test identities.
LIVE_WRITE_ORDERS	off (prod) / on (preprod)	1 to also place and cancel a 1¢ order and a $1 market buy/sell.
LIVE_TEST_EVM_KEY, LIVE_TEST_EVM_KEY_2	—	Dedicated test wallet private keys (no meaningful funds).
LIVE_EVM_RPC_URL	—	An RPC for the exchange's chain, so the write tests can send approvals.
LIVE_ALLOW_MAINNET_WRITES	—	yes to allow write tests when /api/exchange reports a main-network chain id.
LIVE_STAFF_EMAIL/_PASSWORD/_TOTP_SECRET	—	A staff test account, for the staff flows (preprod).
LIVE_RESOLVER_EMAIL/_PASSWORD/_TOTP_SECRET	—	A resolver test account (checks it is refused admin-only routes).
LIVE_TELEGRAM_BOT_TOKEN, LIVE_TEST_TON_SEED, LIVE_TON_TONCENTER_URL	—	For the Telegram Mini App / TON flows (preprod). The bot's token alone signs the test Telegram account in for the signed-in Mini App home's layout-shift check.
LIVE_TEST_TELEGRAM_ID	910000099	The test Telegram account's id (launch data is signed for it).
LIVE_EVENT	—	A specific event slug to use, instead of the busiest active one.
LIVE_EXPECT_API_DOCS	—	1 if the API reference is meant to be public (otherwise it must be 404).

Validate against a local stack

bun run e2e:live:local

This brings up a production-like stack (the EVM and TON development chains, PostgreSQL, the production builds of the API, the site, the Mini App, and the staff portal, with NODE_ENV=production), funds test wallets on the dev chains, and runs the suite under the preprod profile with writes on. TLS/HSTS and the like report SKIP (the stack is http://localhost), never a false failure. This is how the suite is kept honest without reaching the live site.

In GitHub Actions

Actions → Live checks → Run workflow, with inputs for the suites, the load level, whether to write, and a profile override. It also runs nightly (functional + security + performance, light load, no writes) and after each successful Deploy. Site URLs come from the repository variables SITE_URL, ADMIN_URL and DOCS_URL; the profile from LIVE_PROFILE. The run uploads e2e/live/out/ as an artifact and writes the Markdown summary to the job summary. On a failed scheduled run it posts the failing checks and the run's link to ALERT_WEBHOOK_URL (signed with ALERT_WEBHOOK_SECRET), the way the server's own alerts go out.

The secrets the write and staff tests need (LIVE_TEST_EVM_KEY, LIVE_STAFF_*, the TON ones) are read only on the dispatch path, never on the schedule or after a deploy, so an automated run can never write.

The write and load switches, and their risks

• LIVE_WRITE=1 signs the test identities in and reads their own portfolio and referrals. On its own it changes nothing a customer would notice.
• LIVE_WRITE_ORDERS=1 (needs write) places a limit buy at 1¢ for the minimum size on one market and cancels it at once, then does a $1 market buy and sells it back. Against a real exchange this can, in the worst case, fill and cost a few cents (a resting 1¢ bid is normally never hit; a market buy fills at the book's price). Use a test wallet with no meaningful funds, and only under preprod. The suite refuses to sign orders when /api/exchange reports a main-network chain id unless LIVE_ALLOW_MAINNET_WRITES=yes.
• LIVE_LOAD=heavy ramps to 100 virtual users for five minutes. It is never the scheduled default, and under the production profile it needs LIVE_LOAD_CONFIRM=yes. Even then it aborts the moment the error rate passes 2% or p95 doubles, so it cannot keep hammering a site that is already struggling.

How to read a failure

The console log prints one line per check — PASS/FAIL/WARN/SKIP, the check named in plain English, the evidence (a status code, a header value, a timing) and, on a failure, a one-line fix: pointing at what to change. summary.md (the job summary) lists the failures first, each with its evidence and fix, then the warnings, then the numbers, then every check folded away. A FAIL is a defect or a misconfiguration (the run exits non-zero); a WARN is advice (a certificate due to expire, a recommended header missing); a SKIP says why a check did not apply (TLS on http://localhost, a write test with no test wallet). A performance budget that is wrong for the page's real cost is raised in e2e/live/budgets.json, not papered over in the check.

How to add a check

Each suite (e2e/live/suites/{functional,security,performance}.mjs) is a set of sections; a section calls report.expect(name, ok, { evidence, fix }) (or pass/fail/warn/skip/advise) for each thing it checks. The shared helpers are in e2e/live/lib/ — http.mjs (the identified, timed fetch), routes.mjs (the API's routes and access levels, read from the source), pool.mjs (a concurrency pool and percentiles), catalogue.mjs (what is listed, found once and shared), browser.mjs (Playwright), evm.mjs/staff.mjs/totp.mjs (the write and staff identities). The hacker probes and trader flows are reused from e2e/simulation/ rather than rewritten. To add a check: find the right section, call report.expect with a clear name and a fix, and give it a budget in budgets.json if it is a number. Then validate it with bun run e2e:live:local.