Observability layer

Four decisions in this layer look wrong at first glance and are not. Each one is written down here because the obvious "fix" would break something that is not obvious.

Why the enforcement point was the urgent part

The control plane already had a Prometheus surface and an append-only decision record. The sidecar — the component that actually blocks calls, fail-closes when the control plane is unreachable, and runs in observe mode — emitted nothing.

That is not a monitoring gap, it is a safety gap. A sidecar that has been silently forwarding every call looks exactly like a healthy one. Everything else this layer does improves debugging; this part is the one where the absence of telemetry is itself a risk.

The sharpest expression of it is agentic_auth_pep_blocks_total{cause=...}. From the agent's side, "policy refused this" and "the guardrail could not reach policy" are the same thing: a refused call. One means the system is working. The other means it is broken and failing closed. Nothing outside the enforcement point could tell them apart.

Why metrics stayed on Prometheus while traces and logs went to OpenTelemetry

Two telemetry stacks in one codebase is a smell, so it needs a reason.

The Phase 3 and Phase 4 metric names are a published contract. Integrators have written dashboards and alerts against them. OpenTelemetry's PrometheusMetricReader cannot preserve them:

• it appends otel_scope_name, otel_scope_version and otel_scope_schema_url labels to every series by default, plus a target_info metric — and adding a label keeps sum by (...) working while silently breaking every exact-match query;
• it registers a collector into a CollectorRegistry, so it becomes a second writer into the same exposition with a second naming scheme;
• it has no PROMETHEUS_MULTIPROC_DIR equivalent, so multi-worker counts would regress to whichever worker answered the scrape.

Traces and logs have no installed base, so they had a free choice, and OpenTelemetry is the only one that makes "configuration only, no backend lock-in" true.

An OTLP-only integrator collects metrics through the Collector's prometheus receiver. That is one line of Collector config and the documented path for exactly this case.

Why sampling needs a sampler and a processor

Refusals must survive the most aggressive sampling the layer permits. But Sampler.should_sample runs when a span starts, and the outcome is not known until it ends. A sampler alone decides too early.

So the work splits:

• At start, a custom sampler keeps every authorization span, overriding the caller's sampling decision. Without that override, an agent trace the integrator sampled out would take our deny span with it — and a deny nobody can see is the exact failure this layer exists to prevent.
• At end, a processor sheds a configurable fraction of allow spans, where the outcome attribute exists. Refusals never consult the ratio.

The cost is a partial trace: our span is exported, its parent is not. That is accepted. The span still carries authz.action_id and resolves to the verified chain on its own; losing the refusal would not be acceptable.

Collector-side tail sampling would also work, and was rejected: it moves a guarantee into infrastructure the SDK does not ship and cannot test, and the layer must not make a Collector mandatory.

Why a trace id is never evidence

Inbound trace context comes from an agent process the integrator may not control. It is untrusted input, and it gets three rules:

1. The propagator is tracecontext only, pinned in code rather than read from OTEL_PROPAGATORS. The SDK default includes baggage, which would extract arbitrary attacker-controlled key/value data on the way in and re-inject it into every outbound call — a smuggling channel through our own telemetry.
2. tracestate is discarded. Neither read nor propagated.
3. Nothing inbound reaches a decision. Not an outcome, not a deny reason, not an audit field, not a metric label.

What a caller can do is claim any trace id, and so place its spans inside someone else's trace. That is a correlation nuisance, not a security bypass, and it has one consequence worth stating plainly:

A trace id is a correlation handle. The audit record, keyed by action_id, is the evidence.

Which is why every span carries an action_id: an operator who distrusts a trace resolves it through the chain query and gets the verified answer from the append-only table.

Why one chain is one trace even with an uncooperative caller

The control plane and the enforcement point are separate processes with no shared context. When no caller supplies a traceparent, both derive the same trace id from a value they already hold — the session id — through a keyed HMAC:

trace_id = HMAC-SHA256(telemetry_key, "7:session:<session_id>")[:16]

No channel, no coordination, same trace.

The key matters: an unkeyed digest would make a trace id a reversible handle to a session id for anyone holding the telemetry.

:::note A planning decision that changed during implementation The design called for a custom OpenTelemetry IdGenerator. Its generate_trace_id(self) takes no arguments, so a generator cannot see the session id without process-global or contextvar state. Building a synthetic non-recording remote parent from the derived id is deterministic per request, needs no global state, and makes the trace id arrive from the parent context — so the SDK never generates one for these spans, and the is_trace_id_random caveat the design worried about does not arise at all. :::

Why identities are pseudonymous by default

backend/app/services/metrics.py has carried one rule since Phase 3: labels never carry identities, because that keeps audit identities behind the audit-reader check. Traces are where that rule gets tested, since most of the value of a trace comes from knowing which agent.

The resolution: pseudonymous by default, with one explicit per-deployment opt-in whose state is itself exported as a gauge. A keyed, truncated HMAC is stable within a deployment — so telemetry can be grouped by agent — and not reversible by anyone holding only the telemetry.

The triage cost is one extra step: take authz.action_id to the audit API. That is the deliberate price of keeping identities where they belong.

The leak path that was not obvious

The allowlist of permitted attribute keys catches a key someone added on purpose. It does not catch the keys the SDK adds on its own.

LoggingHandler._get_attributes copies every non-reserved LogRecord attribute into the exported record — which is how extra= becomes structured attributes for free, and also how a careless extra={"body": ...} would export a payload. And whenever exc_info is set, it adds exception.message (as str(value.args[0])) and exception.stacktrace, the full formatted traceback.

api/v1/authorize.py already called logger.exception() on three paths, and api/v1/token.py on one. A traceback through token verification can carry a token fragment in a frame-derived message. So those four sites lost their stack traces and record error.type instead.

That is a real reduction in local debuggability, bought deliberately, and it is recorded in specs/007-observability-layer/contracts/phase-3-4-deltas.md §6 — because a future reader will otherwise be tempted to put logger.exception back.

Fail-open, against an audit log that fails closed

These two postures are opposite on purpose.

The audit log is fail-closed: POST /v1/authorize refuses to return an allow for a decision it could not record. Evidence is mandatory.

Telemetry is fail-open: an unreachable, slow or rejecting collector changes no authorization outcome, and a telemetry setup failure leaves the process serving. A guardrail that refused to start because a collector was down would be an availability regression caused by an observability feature.

backend/tests/integration/test_telemetry_fail_open.py holds the second one to four injected faults; test_audit_before_dispatch.py has held the first since Phase 3.

Why the verifier library takes a Protocol

app/verify/ is importable with only pyjwt[crypto], and tests/unit/test_verify_imports.py enforces it. A resource server embeds it to guard its own tools; it should not acquire an OpenTelemetry dependency for the privilege.

There is a second reason. trace.set_tracer_provider() is process-global and warns-and-ignores on a second call. A library that installs telemetry in a host it does not own either hijacks the host's pipeline or silently emits nothing. Accepting a sink the host constructed is the only correct option — so TelemetrySink is a typing.Protocol, and the OTel-backed OtelSink lives in app/observability/ where the integrator opts into it.

What this layer is not

The PRD lists "not a full agent observability/tracing platform — we only emit primitives observability tools need" as a non-goal. So:

• no SDK-served console or dashboard page; the dashboards ship as importable JSON;
• no telemetry backend, storage or hosted service;
• no agent-level or model-level observability — prompts, completions, token usage, model cost. This layer observes authorization, not inference.

What it does ship is the enumerated surface, the artifacts, and a test that fails the build when a renamed metric would have left a dashboard quietly empty.