Observe the guardrail

Goal: see whether the guardrail is alive and actually enforcing, follow one blocked call to the decision that refused it, and get both into the observability stack you already run.

The SDK serves no dashboard of its own. It emits primitives and ships importable artifacts; you own the backend.

1. Turn telemetry on

Four environment variables, no code change:

pip install -e 'backend[otel]'
export AGENTIC_AUTH_TELEMETRY_ENABLED=true
export AGENTIC_AUTH_TELEMETRY_KEY="$(openssl rand -hex 32)"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

AGENTIC_AUTH_TELEMETRY_KEY is required. It keys the identity pseudonyms in traces and logs and the session-derived trace ids. Leave it empty and the exporter refuses to start and logs once — the process keeps serving, because it will not export identities in the clear or behind an unkeyed digest that anyone holding the telemetry could reverse.

Each enforcement point configures its own, under AGENTIC_AUTH_SIDECAR_TELEMETRY_*. There is deliberately no fleet-wide switch.

With nothing configured at all, the local surfaces still work: GET /metrics, GET /__agentic/metrics, and JSON logs on stdout.

2. Scrape both surfaces

scrape_configs:
  - job_name: agentic-auth-control-plane
    static_configs:
      - targets: ['control-plane:8000']

  - job_name: agentic-auth-pep
    metrics_path: /__agentic/metrics
    static_configs:
      - targets: ['127.0.0.1:8081']

:::warning An enforcement point that is not scraped is not observable AgenticAuthPepSilent fires on absent(agentic_auth_pep_up). A sidecar bound to loopback and never scraped produces no series at all, so absence cannot be told from "never deployed". This is the one requirement the SDK meets by contract rather than in code. :::

3. Import the dashboards and alerts

# Grafana → Dashboards → Import → upload each file
observability/dashboards/guardrail-health.json
observability/dashboards/rollout-tuning.json
observability/dashboards/chain-integrity.json

# prometheus.yml
rule_files:
  - observability/alerts/guardrail.rules.yml

4. Check it is enforcing

curl -s localhost:8081/__agentic/status | jq

{
  "mode": "enforcing",
  "uptime_seconds": 10432,
  "upstreams": 3,
  "upstream_map_digest": "sha256:9f2c...",
  "pdp": { "reachable": true, "last_success_seconds_ago": 0.4 },
  "telemetry": { "enabled": true, "state": "enabled", "identity_disclosure": false },
  "version": "0.1.0"
}

mode is the answer to "is this thing actually enforcing". mode: "observing" decides and records but does not block — intended during a rollout, and a problem if nobody remembers.

pdp.reachable comes from the last real control-plane call, not a probe, so it cannot report healthy while real calls are failing.

5. Tell a working guardrail from a broken one

This is the distinction the layer exists for. From the agent's side, both are just a refused call:

sum by (cause) (rate(agentic_auth_pep_blocks_total[5m]))

cause	Meaning
policy	Policy said no. The guardrail worked.
pdp_unreachable, exchange_unreachable, mint_unreachable, audit_unavailable	The guardrail is fail-closing because it is broken. These page.
unauthorized, unresolvable_call, body_too_large, unbindable	The caller's or the operator's mistake. These do not page.

6. Follow one blocked call to its chain

With a caller that sends traceparent, every hop lands in one trace. Open the denying authz.decide span: it carries the outcome, the deny reason, the hop, the policy version, and authz.action_id.

Take that action id to the audit API, which is where identities live:

curl -H "Authorization: Bearer $AGENTIC_AUTH_AUDIT_READ_TOKEN" \
  "$CONTROL_PLANE/v1/audit/actions/$ACTION_ID/chain"

That returns the verified chain back to the consenting human.

:::info A trace id is never evidence A caller can claim any traceparent, so a trace id is a correlation handle only. The audit record, keyed by action_id, is the evidence — which is why every span carries one. :::

If the caller sends nothing, the decisions still form a single trace: the control plane and the enforcement point independently derive the same root from the session id, with nothing passed between them.

7. Before you enforce: see what would break

Run the enforcement points in observe mode with taint enforcement off, then watch rollout-tuning until the would-be blocks reach zero:

sum(rate(agentic_auth_pep_calls_total{outcome="observed_would_block"}[5m]))
sum by (reason) (increase(agentic_auth_denials_total[1h]))
sum by (propagation) (increase(agentic_auth_taint_marks_total[1h]))

Metric labels carry no identities, so "which reason" hands off to GET /v1/audit/decisions?outcome=deny&deny_reason=<reason> for "which calls".

What never appears in telemetry

No token, credential, request or response body, params_hash, binding, query string or stack trace — at any setting. Identities are pseudonymous unless you set AGENTIC_AUTH_TELEMETRY_IDENTITY_DISCLOSURE=true, and agentic_auth_telemetry_identity_disclosure reports which posture a deployment is in so you can alert on it.

See the telemetry surface reference for the full enumerated list.

If you run an OTLP-only pipeline

Metrics stay on Prometheus exposition so the published metric names keep working. Collect them with the Collector's prometheus receiver — the snippet is in observability/README.md, and docker compose --profile observability up -d runs a working local version of exactly that.