Inference Anywhere. Governance Everywhere.

Most enterprises now run AI inference in more places than they did a year ago. A hosted API for the model a team is trying this week. A hyperscaler region with existing credits. A private platform for data that cannot leave the building. A GPU workstation under a developer's desk. A server on a factory floor.

None of that is a problem. It is the sign of a healthy AI program, and the right place for each workload will keep changing. The problem is what usually comes with it: a different way of governing AI in every one of those places. Inference anywhere is already true. Governance everywhere is not. This post is about closing that gap, and about why the place to close it is the boundary around the agents that trigger the inference.

Inference Is a Continuum, Not a Fixed Path

We have seen this before. Compute was supposed to move to the cloud or stay on-premises, and it did neither. In Flexera's 2026 State of the Cloud report, 73% of organizations run hybrid estates. Hybrid did not turn out to be a transition. It turned out to be the destination.

Inference is heading the same way. Hosted model APIs keep growing, and alongside them inference is spreading into private clouds, data centers and edge sites. Deloitte's Tech Trends 2026 describes three tiers: cloud for elasticity and burst, on-premises for steady production inference, and the edge for immediacy.

Those tiers are not a menu where you pick one, or a sequence every model must follow. They are points on a continuum, and each workload finds its own place on it. Some models stay on a hosted platform for good: new open-weight models are served there within days of release, sometimes on launch day, and safely. Fireworks AI does not store prompts or outputs for open models by default, and Baseten offers region-locked, dedicated deployments for data that has to stay in a jurisdiction. Some workloads earn a place on your own infrastructure, where the cost is predictable and the data stays in the building. Some tasks will always call for a frontier model. And many workloads combine places: local inference first with a burst to hosted capacity when demand spikes, a hosted model with a frontier model for the hardest requests, or a model tried hosted and later brought in-house.

There is no single right path, and you should not have to commit to one. The customers I talk to are choosing which workload goes where this quarter, and expecting to choose again.

That is what inference anywhere means: not a prescribed route, but the freedom to put each workload wherever suits it now, combine places, and change your mind. Governance everywhere makes that freedom safe, because the rules are already waiting at every point on the continuum.

Agents Decide Where Inference Is Triggered

The demand behind all of this increasingly comes from agents. On OpenRouter, agents first used more tokens than humans in February 2026, and by late September agentic API keys were using about seven times the tokens of human ones. And adoption has a long way to run: in Gartner's 2026 CIO survey, only 17% of organizations had deployed AI agents, yet more than 60% expected to within two years.

Agents also run anywhere: coding agents on developer laptops, vendor tasks in cloud sandboxes, managed agent runtimes on each of the three largest clouds, platform teams' own Kubernetes clusters, SaaS products and, increasingly, the edge. One enterprise can easily have all of these at once.

This changes the useful question. Inference is triggered from wherever an agent runs, and the model it calls may be somewhere else entirely. A coding agent on a laptop calling a hosted frontier model is the everyday case. So the map that matters for governance is not only where models are served. It is where requests start.

Sandboxes Contain Agents. They Do Not Govern Them.

Most of those agents run inside a sandbox, because nobody wants an autonomous process with a shell and a credential loose on a machine. The sandboxes come from many sources: NVIDIA's open-source OpenShell, the sandboxing built into Claude Code and OpenAI's Codex, Docker, platforms such as E2B, Daytona and Modal, each hyperscaler's code interpreters, and a Kubernetes agent-sandbox project. Underneath sit different isolation technologies, from Landlock and seccomp to microVMs and gVisor, each with its own way of controlling what leaves.

The good ones are genuinely good at containment. They limit what an agent can touch and which hosts it can reach, and the better ones keep real credentials out of the agent's hands by injecting them at a proxy. OpenShell goes further than most, with default-deny egress, per-endpoint rules and a log of every allow and deny decision.

But containment is only half the job. The agent still has to reach out: to models, to MCP tools, to internal APIs, to data. That traffic is where the enterprise questions live. Who is this agent acting for? Is it entitled to this model? Is a customer record about to leave in a prompt? How much has this team spent today? May this agent call this tool, with these arguments? And can we prove afterwards what happened? Among the sandboxes we reviewed, none documents that full set of controls on outbound traffic: identity per user and per agent, PII guardrails, token and cost limits, tool- and argument-level authorization, and evidence that ties each call to whoever asked for it. Where a vendor does offer the full set, it ships as a separate gateway that governs that vendor's own platform. That is not a criticism of sandboxes. It is a different job.

Fragmented by Default

Each hyperscaler offers its own guardrail service, with its own policy model, its own risk categories and its own billing unit. Some of these services can be called for models hosted elsewhere: Amazon's ApplyGuardrail API can assess any text, and Google's Model Armor works across clouds. But each team wires in each call itself, and a policy written for one service does not carry over to another. Run in three clouds and you maintain three definitions of what "sensitive" means.

Models served on your own hardware usually start with nothing at all. Ollama's local API does not require authentication. vLLM's security guidance tells operators to add authentication, rate limiting and logging at a proxy in front of it. Neither filters content by default. Add a different sandbox per agent framework, each with its own egress rules, and the places a policy must be written multiply again.

The cost of this shows up where you would expect. IBM's 2026 Cost of a Data Breach Report found that AI-related breaches rose to 21% of organizations, from 13% a year earlier, and 92% of those organizations lacked proper AI access controls. Gartner predicts that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance gaps found only after production incidents. Gartner's warning is against one blanket policy for every agent, and it is fair. The answer is not a different system in every place, but one place to enforce rules that can differ by agent, by team and by model. Underneath the statistics is a simpler problem: a CISO cannot define a policy once, measure it consistently, or prove anything after the fact. As we wrote when we looked at AI's cross-cutting concerns, they "become dangerous when they are implemented locally but expected to behave globally."

"Own Your Intelligence" Is the Right Idea, Usually Sold the Wrong Way

The usual vendor answer to fragmentation is consolidation on one stack, theirs, wrapped in language I actually like. Since 2024, "sovereign AI" has become standard vendor vocabulary, and this year "own your intelligence" joined it. The trouble is what it turns into. For a chip maker, sovereign AI is a revenue line. For a server maker, it is an AI factory you buy. For a hyperscaler, it is a sovereign region you commit to. Each pitch ends in a purchase order, and each covers only one of the places the customer will run.

Owning your intelligence should mean owning the decisions: where each workload runs, which rules apply to it, and the proof that those rules held. Analysts now call out "sovereign washing"; as theCUBE Research put it this year, sovereignty is "a property of the control plane and the jurisdiction." You do not own your intelligence if the rules belong to whichever place it happened to run. Owning the decisions does not require owning the rack. It requires owning where they are enforced.

Governance Everywhere Lives at the Boundary

That place is not another guardrail in each location. It is one governance plane in the path of every request that matters: a person reaching an agent, an agent reaching a model, an agent reaching a tool or an API. We have called this the Triple Gate: API, AI and MCP gates in one gateway platform, carrying the caller's identity across all three. Wherever the agent runs and wherever the model is served, the request crosses that plane, and the same rules apply.

Where it sits matters as much as what it does. It belongs outside the sandbox, at its boundary, configured from somewhere the agent cannot write. A prompt-injected agent can do a great deal inside its sandbox. What it cannot simply do is switch off governance that sits outside it. The sandbox contains the agent. The boundary governs what it reaches. In an earlier post on harness and platform engineering, we called this line the seam: the point where every model call, tool call and MCP request leaves the agent process. Governance sits on the far side of it.

This is an old idea, applied to a new kind of workload. James P. Anderson's 1972 report for the US Air Force described a reference monitor that must be tamper-proof, always invoked, and small enough to verify. NIST's zero trust architecture routes every access through a policy enforcement point, typically a gateway "in front of a resource that controls access." Google's 2025 paper on securing AI agents recommends policy engines "that operate outside the AI model's reasoning process." OWASP's 2026 Top 10 for LLM Applications tells builders to "implement authorization in logic rather than relying on an LLM to decide if an action is allowed or not." The 2026 guidance on agentic AI from CISA and its international partners warns against "granting broad or unrestricted access, especially to sensitive data or critical systems." The best sandboxes already follow the principle: Claude Code runs its sandbox proxy "outside the sandbox."

The two layers do different jobs, and you want both.

Connecting the two is less work than it sounds, because most sandboxes can already block everything except an allowlist. Allow one destination, the governance plane, and give the agent an endpoint there instead of a provider key. The agent carries a token that says who it is and who it is acting for; the provider credentials stay at the boundary. Everything the agent reaches, it reaches through the rules.

Put the governance there and it becomes portable. Identity, entitlements, guardrails, spending limits, tool policy and evidence are defined once, and guardrails can come from more than one vendor: as we showed in multi-vendor AI safety pipelines, guards backed by different models and services can run in parallel at one gateway, and they still apply when a request fails over to another provider. When an agent moves, or the model behind a stable name moves along the continuum in any direction, what changes is an endpoint, a credential and a model identifier. That is portable governance: the rules travel with the workload instead of belonging to the place it runs.

Requests out of bounds are blocked. Requests in bounds go through. And afterwards you can prove both. The evidence lands in the logs, metrics and traces you already collect. None of this makes an agent safe on its own: isolation inside the sandbox still matters, and the boundary itself has to be protected and configured well. What the boundary gives you is one place to set the rules, one place they are enforced, and one record that they held, no matter where the agent or the model happens to be.

The Boundary Has to Be Neutral

Hybrid cloud taught one more lesson. The layer that sets policy, identity and routing across an estate is where leverage collects. Kubernetes showed what happens when that layer is open and portable: leverage moved toward the people running the workloads, who could then choose where those workloads ran.

The governance plane is the control plane for agents and inference, and the same lesson applies. Every vendor that sells you something underneath it, a chip, a rack, a cloud, a model or a sandbox, has a reason to offer to govern it too, and its governance will cover its own platform. That is the fragmentation problem again, with a better logo. The boundary only works if it is neutral about every one of those choices, and it only stays yours if it meets a few conditions. It runs in your environment. Its policy lives in your Git as configuration you can read. Identity comes from your identity provider, authorization can come from your own policy engine, and evidence lands in your own observability stack, on open standards at each step. Any governance plane should face the question this post puts to sovereign AI: who holds it? Operators selling inference as a service hear it from their buyers as "Whose meter is this?"

That is the position Traefik has held for a decade. Traefik Proxy has been pulled more than 3.5 billion times from Docker Hub, and it is the default ingress in K3s and Canonical MicroK8s (Traefik Labs). In a great many clusters, Traefik is already at the boundary. Traefik Hub builds the governance plane on that foundation. It runs in your environment, on Kubernetes, Docker, Nomad or plain Linux hosts, configured as code. It carries delegated identity for agents through OAuth token exchange and ID-JAG, the IETF draft behind cross-app access, hands authorization to your own policy engine through AuthZEN, enforces MCP tool and argument policy, applies PII and safety guardrails, token limits and estimated cost limits on model calls, and emits OpenTelemetry GenAI telemetry, with an optional tamper-evident transparency log for evidence. It does not sell you the model, the GPU, the cloud or the sandbox. It governs all of them.

Start With One Agent

You don't need a program to begin. You need one agent.

1. Pick one agent that matters, and give the model it calls a stable name.
2. Put the boundary in front of it. The agent gets an endpoint and a token, the provider keys stay at the boundary, and its sandbox allows that one destination.
3. Write the rules once: who the agent acts for, which models and tools it may use, what must not leave in a prompt, and what it may spend.
4. Then move something. Send the model to another provider, bring it onto your own infrastructure, or run the agent somewhere new. Check that not one rule had to change, and that you can prove it from your own logs.

If the rules hold, you have portable governance for one agent. Everything after that is repetition.

Every enterprise will spend the next few years deciding, and deciding again, where inference runs. That choice should stay open. Who sets the rules should not. Settle it once, put the rules at the boundary, keep them in your own hands, and every later decision about models, GPUs, clouds and sandboxes becomes one you can reverse.

Inference anywhere is already true. Make governance everywhere true as well, starting with one agent.

Bring us your map. Book a session with our team and bring the places your agents run and your models are served. We'll show you the boundary in front of one of them, running in your environment and configured as code.