> ## Documentation Index
> Fetch the complete documentation index at: https://docs.madra.finance/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Metrics, tracing, and dashboards for a StellarX402 facilitator, and what every number on them means.

A facilitator has a small set of numbers worth watching. This page names them, says why each matters, and what to alert on.

## Metrics

All metrics are Prometheus-scrapable at `:$METRICS_PORT/metrics`.

### Request-level

| Metric                         | Type      | Meaning                                  |
| ------------------------------ | --------- | ---------------------------------------- |
| `x402_verify_total{code}`      | counter   | Verify requests, tagged by outcome code. |
| `x402_settle_total{code}`      | counter   | Settle requests, tagged by outcome code. |
| `x402_verify_duration_seconds` | histogram | Verify latency.                          |
| `x402_settle_duration_seconds` | histogram | Settle latency (includes on-chain wait). |

Alerting: settle\_duration p95 > 15s for 5 minutes is an incident. Verify p95 > 500ms is a warning.

### Sponsor account

| Metric                             | Type    | Meaning                        |
| ---------------------------------- | ------- | ------------------------------ |
| `x402_sponsor_balance_xlm`         | gauge   | Current sponsor balance.       |
| `x402_sponsor_fees_paid_xlm_total` | counter | Cumulative XLM burned on fees. |

Alerting: `x402_sponsor_balance_xlm < SPONSOR_MIN_BALANCE_XLM` is a page.

### RPC health

| Metric                              | Type      | Meaning                        |
| ----------------------------------- | --------- | ------------------------------ |
| `x402_rpc_up{url}`                  | gauge     | 1 if last probe succeeded.     |
| `x402_rpc_latency_seconds{url}`     | histogram | RPC round-trip.                |
| `x402_rpc_fallback_used_total{url}` | counter   | Times primary RPC failed over. |

### Bazaar

| Metric                          | Type  | Meaning                                 |
| ------------------------------- | ----- | --------------------------------------- |
| `x402_bazaar_listings`          | gauge | Total active listings.                  |
| `x402_bazaar_index_lag_seconds` | gauge | Time between first-seen and searchable. |
| `x402_bazaar_search_qps`        | gauge | Search queries per second.              |

## Tracing

OpenTelemetry OTLP export via `OTEL_EXPORTER_OTLP_ENDPOINT`. Spans emitted:

* `x402.verify` (parent for a verify request)
* `x402.settle` (parent for a settle request)
* `x402.rpc.submit` (Soroban submit)
* `x402.rpc.get_transaction` (Soroban polling)
* `x402.bazaar.index` (async catalog write)

Attach `x402.tx_hash` on settle spans as soon as it is known.

## Dashboards

A reference Grafana dashboard ships in `dashboards/facilitator.json`. It has four rows:

1. **Traffic** — verify/settle rate and error rate.
2. **Latency** — verify and settle latency percentiles.
3. **Sponsor** — balance and burn rate.
4. **Bazaar** — listing count, index lag, search QPS.

## Recommended SLOs

* Verify success rate > 99.5% (excluding client-side codes like `insufficient_funds`).
* Settle success rate > 99% given verify succeeded.
* Settle p95 latency \< 10s on mainnet.
* Bazaar index lag p95 \< 30s.