DocsSelf-hostingMetrics

Metrics

Kvasyr exposes Prometheus-format metrics at /metrics.

This page documents what is exported and how to secure and use it in production.

Access Control

/metrics is enabled by default but access is guarded unless you explicitly open it.

Relevant env vars:

METRICS_PUBLIC=false
METRICS_ALLOW_IPS=127.0.0.1/32
METRICS_BEARER_TOKEN=<strong-random-token>
METRICS_HIGH_CARDINALITY=false

Access rules:

  1. If METRICS_PUBLIC=true, everyone can read /metrics.
  2. Otherwise, access is allowed if source IP matches METRICS_ALLOW_IPS.
  3. Otherwise, access is allowed with Authorization: Bearer <METRICS_BEARER_TOKEN>.
  4. Otherwise, access is allowed with x-admin-password: <ADMIN_PASSWORD>.
  5. If none match, request returns 403.

Recommended for white-label/self-hosting:

  • keep METRICS_PUBLIC=false,
  • restrict by IP and/or bearer token,
  • avoid relying on x-admin-password for machine scraping.

Cardinality Mode

METRICS_HIGH_CARDINALITY controls whether user- and contract-level label breakdowns are emitted.

  • false (default): lower-cardinality aggregates, safer for long retention.
  • true: detailed labels (for example user_id, contract_address) useful for support/debug and chargeback analysis.

Enable high cardinality only if your Prometheus sizing and retention policy account for the extra series.

Metric Families

Build / Instance / Config

  • kvasyr_build_info{version} gauge (always 1)
  • kvasyr_instance_info{instance_id,hosted_mode} gauge (always 1)
  • kvasyr_indexer_poll_interval_ms gauge
  • kvasyr_chain_config_info{chain_id,chain_name} gauge (always 1)
  • kvasyr_chain_finality_depth{chain_id} gauge (depth fallback)
  • kvasyr_chain_finality_mode_info{chain_id,finality_mode} gauge (always 1)

Chain Progress

  • kvasyr_chain_head_block{chain_id} gauge
  • kvasyr_chain_indexed_block{chain_id} gauge
  • kvasyr_chain_finalized_block{chain_id} gauge
  • kvasyr_chain_head_lag{chain_id} gauge

Operational use:

  • monitor head_lag growth for indexing/finality delays,
  • compare chain gauges across chains to catch isolated regressions.

Events / Matches / Credits

  • kvasyr_events_total counter
  • kvasyr_events_by_chain_total{chain_id} counter
  • kvasyr_events_by_collection_total{chain_id,contract_address} counter (high-cardinality mode)
  • kvasyr_matches_total counter
  • kvasyr_matches_by_chain_total{chain_id} counter
  • kvasyr_matches_by_client_total{user_id} counter (high-cardinality mode)
  • kvasyr_matches_by_collection_total{chain_id,contract_address} counter (high-cardinality mode)
  • kvasyr_matches_by_client_collection_total{user_id,chain_id,contract_address} counter (high-cardinality mode)
  • kvasyr_stored_matches_total counter
  • kvasyr_credits_consumed_total{user_id} counter (user_id="all" when high-cardinality mode is off)
  • kvasyr_credits_exhausted_total{user_id} counter (user_id="all" when high-cardinality mode is off)

Webhook Pipeline

  • kvasyr_pending_deliveries gauge
  • kvasyr_webhook_queued_by_chain_total{chain_id} counter
  • kvasyr_webhook_queued_total{user_id,chain_id,contract_address} counter (high-cardinality mode)
  • kvasyr_webhook_attempts_by_chain_total{chain_id} counter
  • kvasyr_webhook_attempts_total{user_id,chain_id,contract_address} counter (high-cardinality mode)
  • kvasyr_webhook_sent_by_chain_total{chain_id,status} counter
  • kvasyr_webhook_sent_total{user_id,chain_id,contract_address,status} counter (high-cardinality mode)
  • kvasyr_webhook_latency_seconds{status} histogram

RPC Health / Latency

  • kvasyr_rpc_endpoint_configured{chain_id,endpoint} gauge (always 1)
  • kvasyr_rpc_endpoint_up{chain_id,endpoint} gauge (1 = recent success, 0 = recent error)
  • kvasyr_rpc_consecutive_failures{chain_id,endpoint} gauge
  • kvasyr_rpc_last_success_timestamp{chain_id,endpoint} gauge (unix seconds)
  • kvasyr_rpc_requests_total{chain_id,endpoint,status} counter
  • kvasyr_rpc_calls_total{chain_id,status} counter
  • kvasyr_rpc_attempts_per_call_bucket{chain_id,status,...} histogram
  • kvasyr_rpc_latency_seconds_bucket{chain_id,endpoint,...} histogram

Indexing Diagnostics

  • kvasyr_subscriptions_skipped_total{reason} counter
  • kvasyr_getlogs_split_total{chain_id,source} counter
  • kvasyr_getlogs_poison_blocks_total{chain_id,source} counter

These help identify malformed subscription filters, pathological ranges, and unstable RPC behavior.

Useful PromQL Queries

Webhook success rate (5m):

sum(rate(kvasyr_webhook_sent_by_chain_total{status="success"}[5m]))
/
sum(rate(kvasyr_webhook_sent_by_chain_total[5m]))

Pending delivery pressure:

kvasyr_pending_deliveries

RPC error ratio by chain (5m):

sum by (chain_id) (rate(kvasyr_rpc_requests_total{status="error"}[5m]))
/
sum by (chain_id) (rate(kvasyr_rpc_requests_total[5m]))

Chains with increasing finalized lag:

max by (chain_id) (kvasyr_chain_head_lag)

Subscriptions skipped by reason (15m):

sum by (reason) (rate(kvasyr_subscriptions_skipped_total[15m]))

Related

MonitoringPanel: Admin Users