> ## Documentation Index
> Fetch the complete documentation index at: https://docs.madra.finance/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbook

> Incident response for the situations that actually happen at 3am when running a StellarX402 facilitator.

What to do when something breaks. Written to be scannable under stress, not read casually. Each incident includes the symptom, the diagnosis, and the fix.

## Settle latency spike

**Symptom:** `x402_settle_duration_seconds` p95 > 15s.

**Diagnose:**

1. `x402_rpc_latency_seconds` on your primary RPC. Elevated?
2. `x402_rpc_fallback_used_total` incrementing?
3. Stellar network status page.

**Fix:**

* If primary RPC is slow, force failover by removing it from the list and restarting.
* If the whole network is slow, there is no fix. Surface `settlement_slow` warnings to sellers and let them decide whether to keep serving 402s.

## Sponsor account low balance

**Symptom:** Alert on `x402_sponsor_balance_xlm < SPONSOR_MIN_BALANCE_XLM`.

**Fix:**

1. Top up from treasury. Confirm settlement of the top-up before doing anything else.
2. If balance dropped faster than expected, check `x402_sponsor_fees_paid_xlm_total` slope over the last hour. If the burn rate 2x'd, look for an abusive caller in `x402_verify_total` by source.

## Verify success rate collapse

**Symptom:** `x402_verify_total{code="ok"}` / `x402_verify_total` drops below 90%.

**Diagnose:**

1. Which code is dominant? Group the counter by `code`.

* `signature_invalid` spike → client SDK regression. Contact upstream integrators.
* `authorization_expired` spike → clock skew on the facilitator host. `ntpq -p`.
* `insufficient_funds` spike → buyer-side. Not your incident.
* `asset_not_supported` spike → someone deployed a client pointed at wrong facilitator.

**Fix:** Depends on the code. Never disable verify to make the graph look better.

## RPC provider down

**Symptom:** `x402_rpc_up{url=<primary>}=0`.

**Fix:** Fallback should have taken over. Confirm `x402_rpc_fallback_used_total` incremented and `x402_rpc_up{url=<fallback>}=1`. If not:

1. Add another fallback URL to config.
2. Restart with new config.
3. Only then debug the primary.

## Bazaar index lag

**Symptom:** `x402_bazaar_index_lag_seconds` > 60s.

**Diagnose:**

* Storage backend health (Postgres CPU, WAL replication).
* Index worker logs for errors.

**Fix:** Restart the index worker. If lag persists, scale the storage backend. The facilitator continues to verify and settle correctly — only discovery is degraded.

## Key rotation

**When:** Scheduled (quarterly minimum), or on any suspected key compromise.

**Procedure:**

1. Generate new keypair.
2. Fund the new sponsor account with a small amount (\~5 XLM) as smoke test.
3. Deploy config with new `SPONSOR_SECRET_KEY`. Confirm sponsorship works with a test transaction.
4. Full-fund the new account from treasury.
5. Drain the old sponsor account back to treasury, leaving `SPONSOR_MIN_BALANCE_XLM` as buffer for any in-flight tx.
6. After 24 hours with zero traffic on the old account, close it (merge account operation).

**Do not** rotate mid-incident unless the key is the incident.

## Contact

On-call escalation, upstream Stellar contacts, and RPC provider status pages should live in your internal wiki, linked from here. This page does not know your team.