Agentic Telemetry Is Not an Audit Trail: Why We Built the Sovereign Trust Plane

AI agents now delegate work to each other, call tools, and touch APIs that move money and data. When someone asks, months later, what an agent was allowed to do and why, most organizations will reach for their traces and discover that a trace was never built to be evidence.

Today we are introducing the Sovereign Trust Plane in Traefik Hub. It is built to answer three questions about any agent action: who authorized it, under which policy it was permitted or refused, and whether the record has been altered since. It answers them from the one place that sees every hop.

Every vendor in the AI infrastructure market now sells a way to watch what agents do. Look closely, though, and most of it stops at the same place: the moment the call is allowed. The request was authenticated, the guardrail passed, the tool was permitted, and the trace was written. Everyone goes home.

Here is the problem. Six months later, nobody is asking whether the call was allowed. They are asking who allowed it, on what basis, and whether the record they are holding is the record that was written at the time. Those are different questions, and a trace cannot answer them.

I have spent ten years building the software that sits in the request path. With agents, that position turns out to be the only place governance can actually be finished.

An Agent Action Crosses Three Boundaries

Strip an agent down to what it does at runtime and you find three kinds of calls. It reasons with a model. It acts through a tool, increasingly over the Model Context Protocol. And its actions land on an API, yours or a partner's, where the money moves and the data changes.

Each of those boundaries can be governed on its own. Model gateways govern the first. MCP gateways govern the second. API gateways have governed the third for twenty years. What almost nobody does is govern all three with the same identity, the same policy and the same record, because in most estates they belong to three products owned by three teams.

That split has a specific failure mode I see constantly. Teams put a gateway between the agent and the MCP server, and feel covered. The MCP server then calls the backend API through a different gateway with different rules, or through no gateway at all. The first leg is governed. The second leg, where the effect actually happens, is somebody else's problem. The same thing happens on the model side, where each agent quietly holds its own provider key and nobody can say which agent used which model.

We built Traefik Hub so that the model gateway, the MCP gateway and the API gateway are one binary, one configuration model, and one middleware pipeline. That was not a packaging decision. It is the precondition for everything below, because a record is only complete if one system saw the whole action.

It also settles a question I get asked in every agentic conversation: should governance live inside the agent framework? No. An enterprise will run several frameworks, plus agents it bought and cannot instrument. The harness governs its own agents. The gateway governs all of them, at the boundary you still control.

Governance is Not Finished When the Call is Allowed

An audit asks three questions, in order. We built the Sovereign Trust Plane to answer each one on every call, and we named the capabilities (Delegate, Authorize, and Prove) after the questions.

Who Authorized This?

That is Delegate. When one agent hands work to another, it must not hand over its credentials. It exchanges them for a token scoped to that one task, for that one resource, for minutes rather than hours. Authority gets smaller at every hop, and the chain from a person to the first agent to the second stays legible in the token itself. The receiving agent holds something the sender could not have fabricated. This is standard OAuth token exchange. Specifically, it is the IETF Identity Assertion JWT Authorization Grant (ID-JAG), the draft behind Okta's Cross App Access, and what the newest revision of MCP calls enterprise-managed authorization. Traefik never mints authority. Your identity provider does. We carry it and narrow it.

Under What Policy?

That is Authorize. A credential is not a decision. A badge gets you into the building; it does not approve a fifty-thousand-euro purchase. So at the moment of the action, the gateway asks your policy engine whether this subject may take this action on this resource, and enforces the answer inline, for API traffic and for MCP tool calls alike. Your engine, your policy language. We speak the open AuthZEN standard to it, and we author no policy.

And when the answer is no, the no is not a bare 403 the agent retries against. It names the limit and the policy:

403 Forbidden
{
  "error":  "forbidden",
  "reason": "amount 2400.00 exceeds agent limit 500.00",
  "policy": "refund-cap-tier2"
}

An agent can act on that. It can reduce the amount, request approval, or escalate to a person. How much of that detail a caller is shown is yours to configure: a trusted internal agent can be told the policy, an external one can be told less. Either way, the refusal enters the record exactly like an approval would, which brings me to the third question.

How Do I Know You are Not Lying?

That is Prove, and it is the question everybody skips. Let's explore Traefik Hub's Prove capability in detail.

A Trace is Not Evidence

We emit OpenTelemetry for every model call, tool call and API call, and if you run agents in production you need it. But I want to be precise about what a trace is for, because the industry is quietly asking it to do a job it was never designed for.

A trace serves an engineer during an incident. It is optimized for query and for cost, which means it is sampled, it is filtered, and it expires on a retention schedule. Those are the right trade-offs for finding out why minute six was slow. They are disqualifying for evidence. Sampling one request in ten tells an auditor nothing about the nine. And a trace is written by the system under examination, which can rewrite it.

Think of a dashcam. It helps you work out what happened. It is not what the insurer settles on, and it overwrites itself every few days because it was never built to be evidence.

Evidence serves a different consumer, months later, and it needs different guarantees. It has to be complete, so every decision is in it, allow and deny alike. It has to carry the context that existed at the moment of the decision, because a decision cannot be explained later from a state that has since changed. And it has to be verifiable by someone who has no reason to take your word for it.

The mistake is to ask one pipeline to do both jobs. Our position is that the capture happens once, at the gateway, and then feeds as many consumers as the organization has. Incident telemetry goes to your observability stack and gets sampled. Security events go to the SOC's SIEM under its own retention. And a separate evidence sink gets every decision, unsampled, with integrity guarantees the others do not need. Same events, several sinks, different promises.

Signing Your Own Log Proves Nothing

Here is where most "immutable audit log" claims fall apart, and the reasoning is the product.

Suppose you hash-chain every decision so an edit in the middle breaks everything after it, put the chain in a Merkle tree so any entry can be proven present in a few dozen hashes, and sign a checkpoint of the tree with your key. Sound cryptography. It is exactly what we do. And on its own it proves nothing to anyone outside your organization.

Every engineer already knows why, because every engineer has used Git. A repository you control tells you nothing about what was force-pushed away. Drop the last five hundred entries, rebuild the tree, sign a fresh checkpoint: every proof still verifies. The log is perfectly self-consistent. It simply describes a history that never happened. Nobody edits the incriminating line; they stop the story before it gets there.

Bucket locks and append-only files have the same shape. They are a promise from the operator, enforced by the operator's own infrastructure. A good control. Not evidence.

The only way out is someone else's memory. A witness is an independent service that keeps exactly one thing per log: the last checkpoint it saw. Send it a new one and it verifies your signature, demands a proof that the new tree is an extension of the old one, and only then countersigns. Once it has seen your log at entry 184,203, it will never countersign 183,703. Monotonicity, enforced by a party you cannot compel. This is the construction behind Certificate Transparency, which has kept the public web's certificates honest for a decade, and every browser you use depends on it.

The part people get wrong is what the witness sees. Here is the entire payload that crosses your boundary:

POST /add-checkpoint
// the entire payload crossing your boundary

old 184203

logs.yourbank.internal
184203
Ilb9V0s7j...jJQL40wm0=

— logs.yourbank.internal tZ3TC14t...xwLDmQQ=
— witness.example.org x101AAAA...S1ATmkmk=

Notice there are no entries, payloads, or decisions. A bank's internal log can be witnessed by an outside party without revealing what is in it. Which means you choose the witness. Ours. Your own, run by a team that does not run the gateway. Or, in a regulated shop, the sharpest option of all: let your auditor be your witness. Their key, your network, and the party who will judge the evidence is the party attesting it.

What Actually Ships

I dislike launch posts that describe a design and call it a product, so here is what is in the early-access build this week, with general availability scheduled by September 30, 2026.

The transparency log ships. Every log line and every access-log line the gateway already writes, across API, model, and MCP traffic, is hashed and committed to an append-only Merkle log with signed checkpoints. It commits the hash, not the content. Your logs stay exactly where they are today, which is the point: this is a second sink for what the gateway already emits, not a new instrumentation project, and it adds no latency to the request path. The log lives on a plain filesystem or in object storage, on your infrastructure.

The verifier ships. It runs on your machine, reads the log from your storage, checks the checkpoint signature and any witness countersignatures, proves every entry's inclusion, and recomputes each committed hash from your original log files. If a single digit of a single line was changed, it tells you which one.

A Traefik-operated public witness ships with the launch, and you can run your own instead. Point the gateway at ours, run the implementation on infrastructure you control, or ask an independent team or your auditor to run it. Without a witness, the log detects tampering by anyone who does not hold the signing key. With one, it also holds the operator of the log to account, because nothing can be removed once a party you do not control has seen it. I want that distinction understood rather than glossed.

The record carries the decision, not just the request. Each entry names the decision the gateway made, the policy that produced it, and both the person and the agent that acted. Refusals are committed exactly like approvals, so the record shows what was prevented as well as what was allowed. The access-log line already carries the trace identifier, so any evidence entry links back to the full trace for the engineer who needs it.

Delegation and enforcement are verified, not asserted. We have run the full exchange end to end, RFC 8693 token exchange then RFC 7523 redemption, against Okta Cross App Access and against a self-hosted Janssen Auth Server, so the chain closes with no cloud identity service in the path. And we have run enforcement against OpenFGA and Cerbos through the AuthZEN middleware built into the gateway.

Sovereign by Construction

Sovereignty that depends on somebody else's cloud is not sovereignty. There is no SaaS control plane in the request path. The gateway is self-hosted with an offline-capable manager. The identity provider that grants authority is yours. The policy engine that decides is yours. The log is a directory on a filesystem you own, and the verifier runs on a laptop. The whole thing runs air-gapped, indefinitely, and the only thing that ever needs to cross the boundary is a hash.

It sits on the substrate we shipped in July: a FIPS 140-3 compliant build and the Distro Zero image with nothing left in it to harden. I wrote about that image as a security decision. It is also the qualifying criterion for the environments that need this most, where "can it run with no egress" is answered before anyone reads a feature list.

Where Our Job Ends

Any honest account of a control includes where it stops.

Traefik Hub governs the communication path, not the internals. It does not see the microservices behind your API, the agent's own logs, or the context a model was reasoning over. Where you cannot reach the processor, audit its inputs and outputs. The gateway can say exactly what happened at every boundary and on what basis. It cannot say why the agent wanted to. Behavior is observable. Cognition is not, and a model's emitted reasoning is just another model output, not a record of the computation. Vendors who blur that line are selling something they cannot deliver.

The log cannot prove an entry is true, only that it was not rewritten. Integrity is not accuracy.

The Record Cannot be Created Backwards

I have deliberately not built this argument on a regulatory deadline. Record-keeping obligations for high-risk AI under the EU AI Act now apply from December 2027, after one deferral, and what counts as high-risk is still being argued. The date has moved once and may move again. The requirement will not.

Here is the argument that survives any deadline. Every other control you can name is retrofittable. Add a policy engine next quarter and it governs next quarter's traffic. Tighten authorization next year and it works from the day you turn it on. The record is the one exception. Switch it on next year and you get next year. Everything before it stays blank, permanently, at any price.

Classification is settled later, and not only by you. If it resolves the other way in 2027, the gap in the record is the finding. That makes the record cheap insurance, and it only costs a configuration change today.

Questions to Ask Your Own Team

If you are the CISO: for the agent that moved money last quarter, can you produce who authorized it, under which policy, and prove the record is intact, without asking the platform team to reconstruct it? And who, outside the team that runs the platform, could confirm that the record was not truncated?

If you own risk and compliance: does your evidence come from the same sampled, expiring pipeline as your incident telemetry? Do your controls record refusals, so you can show what agents were prevented from doing, as well as what they did?

If you own the product and the agents themselves: when an agent makes a bad decision, can you reconstruct the context it had at that moment, or only what the system looks like now? And is the leg from your MCP server to the backend API governed by the same rules as the leg from the agent to the server, or by nobody?

Where to Go From Here

Most of the industry stops at the decision. We keep going to the record.

Read an engineering deep dive on why hash chains miss truncation, why Merkle trees buy two proofs a chain cannot, and what a witness changes. Then run the verifier against a log of your own, tamper with one entry, and watch it go red. And if you would rather have that conversation with us, we would like nothing more.

Don't trust us. Check the witness.

Frequently Asked Questions

What does an AI governance framework need to cover for agents?

Agent actions cross three boundaries: reasoning with a model, calling a tool over MCP, and acting on a backend API. A governance framework that only covers one of these leaves the other two unmonitored, so it has to authorize, enforce policy, and record evidence across all three from the same identity and the same policy source.

How does this relate to an MCP gateway?

An MCP gateway governs the tool-calling boundary, checking whether an agent may invoke a given tool. On its own, that only covers one leg of the journey. If the MCP server calls a backend API through a different gateway, or none, the action that actually moves money or data goes ungoverned even though the tool call was authorized.

Why isn't a trace good enough for audit purposes?

Traces are sampled, filtered, and built for debugging incidents, not for standing as evidence months later. Traefik Hub adds a separate, unsampled evidence sink alongside normal tracing, with integrity guarantees a trace was never designed to provide.

Why isn't a self-signed, tamper-evident log enough on its own?

Because whoever holds the signing key can still truncate history and re-sign a clean checkpoint, and every cryptographic proof will still verify against the shortened log. Internal consistency isn't proof to a party outside the organization.

What is Certificate Transparency, and why does it matter here?

Certificate Transparency is the public system that has kept web certificates honest for a decade by requiring certificate issuance to be logged where independent monitors can catch fraudulent entries. The Sovereign Trust Plane borrows its core mechanism, an independent witness that only ever countersigns forward, and applies it to agent decision records instead of certificates.

What is the Sovereign Trust Plane in Traefik Hub?

A capability that answers three questions about any agent action: who authorized it, under which policy, and whether the record has changed since. It captures this at the gateway, the single point that sees the model, tool, and API calls an agent makes.