Skip to main content
What to do when something breaks. Written to be scannable under stress, not read casually. Each incident includes the symptom, the diagnosis, and the fix.

​
Settle latency spike

Symptom: x402_settle_duration_seconds p95 > 15s. Diagnose:
  1. x402_rpc_latency_seconds on your primary RPC. Elevated?
  2. x402_rpc_fallback_used_total incrementing?
  3. Stellar network status page.
Fix:
  • If primary RPC is slow, force failover by removing it from the list and restarting.
  • If the whole network is slow, there is no fix. Surface settlement_slow warnings to sellers and let them decide whether to keep serving 402s.
Symptom: Alert on x402_sponsor_balance_xlm < SPONSOR_MIN_BALANCE_XLM. Fix:
  1. Top up from treasury. Confirm settlement of the top-up before doing anything else.
  2. If balance dropped faster than expected, check x402_sponsor_fees_paid_xlm_total slope over the last hour. If the burn rate 2x’d, look for an abusive caller in x402_verify_total by source.

​
Verify success rate collapse

Symptom: x402_verify_total{code="ok"} / x402_verify_total drops below 90%. Diagnose:
  1. Which code is dominant? Group the counter by code.
  • signature_invalid spike → client SDK regression. Contact upstream integrators.
  • authorization_expired spike → clock skew on the facilitator host. ntpq -p.
  • insufficient_funds spike → buyer-side. Not your incident.
  • asset_not_supported spike → someone deployed a client pointed at wrong facilitator.
Fix: Depends on the code. Never disable verify to make the graph look better.

​
RPC provider down

Symptom: x402_rpc_up{url=<primary>}=0. Fix: Fallback should have taken over. Confirm x402_rpc_fallback_used_total incremented and x402_rpc_up{url=<fallback>}=1. If not:
  1. Add another fallback URL to config.
  2. Restart with new config.
  3. Only then debug the primary.

​
Bazaar index lag

Symptom: x402_bazaar_index_lag_seconds > 60s. Diagnose:
  • Storage backend health (Postgres CPU, WAL replication).
  • Index worker logs for errors.
Fix: Restart the index worker. If lag persists, scale the storage backend. The facilitator continues to verify and settle correctly — only discovery is degraded.

​
Key rotation

When: Scheduled (quarterly minimum), or on any suspected key compromise. Procedure:
  1. Generate new keypair.
  2. Fund the new sponsor account with a small amount (~5 XLM) as smoke test.
  3. Deploy config with new SPONSOR_SECRET_KEY. Confirm sponsorship works with a test transaction.
  4. Full-fund the new account from treasury.
  5. Drain the old sponsor account back to treasury, leaving SPONSOR_MIN_BALANCE_XLM as buffer for any in-flight tx.
  6. After 24 hours with zero traffic on the old account, close it (merge account operation).
Do not rotate mid-incident unless the key is the incident.

​
Contact

On-call escalation, upstream Stellar contacts, and RPC provider status pages should live in your internal wiki, linked from here. This page does not know your team.