Governance Everywhere, in Practice: Traefik Hub on Spectro Cloud PaletteAI

In a recent blog article, Inference Anywhere. Governance Everywhere, we argued that the rules an AI call follows should not depend on where the model runs, and that the place to enforce them is the boundary every request crosses. This post shows that boundary running on a platform enterprises already use for inference: Spectro Cloud PaletteAI, where Traefik is the default gateway for Model as a Service.

We cover what PaletteAI brings and how a platform team turns on Traefik Hub with a profile bundle. Then we walk through the 6 checks on every model and tool call and show how each agent gets only the tools its identity allows. We finish with Day 2 on a pool that is already serving traffic. Every result in this post comes from compute pools we built from scratch on NVIDIA A100 GPUs, tested, and tore down, with PaletteAI 1.4.0 and Traefik Hub v3.21.7.

Start Where Inference Runs: PaletteAI and Inference Launchpad

PaletteAI runs models on compute pools, which are GPU-backed Kubernetes clusters or cluster partitions that PaletteAI provisions in the data center, in the cloud, in air-gapped sites, and at the edge. A platform team works in projects, so each team's models and rules stay apart. It builds pools from profile bundles, versioned blueprints that combine PaletteAI cluster profiles with PaletteAI workload profiles. Teams then use Model as a Service: they pick a model from Hugging Face or NVIDIA NIM, and PaletteAI deploys it to a pool with the right bundle.

PaletteAI Inference Launchpad is Spectro Cloud's solution for running inference on your own hardware. It ships with certified open-weight models and connectors for coding tools, Traefik as the default gateway, and can burst to frontier models when policy allows. Traefik already serves as its ingress, in front of the Inference Launchpad console and the OpenAI-compatible API.

So the map of where inference runs is already broad, and the calls to it come from everywhere: developers in their usual tools, AI applications, and agents calling tools on someone's behalf.

Close the Gap Between Every Caller and Every Model

The challenge across the industry is the layer between those callers and those models. Each place checks requests its own way. A public cloud applies its provider's rules, a data center applies network rules, and rules drift from one edge site to the next. A bare model server often starts without authentication: vLLM's security guidance tells operators to put authentication, rate limiting, and logging in a proxy in front of it. PaletteAI Model as a Service deploys every model behind the gateway by default, but each environment still configures it in its own way. Nobody can then apply one policy, or prove afterwards that it held.

That gap is what 7 questions expose, and every executive running AI at scale already asks them. Traefik Hub answers each one at the gateway, on every PaletteAI pool:

“Platform teams don’t want to approve and patch a separate gateway for every environment. With Traefik as the default in PaletteAI Model as a Service, the gateway ships with the platform. Customers who need identity-aware policy, data masking, and a signed audit trail add Traefik Hub without changing how they deploy.”

Saad Malik, CTO and Co-founder, Spectro Cloud

Deploy a Model, Governed From the First Call

Traefik is the default gateway for PaletteAI Model as a Service, and Traefik Hub adds its AI Gateway, MCP Gateway, and API management on top. We packaged Traefik Hub for PaletteAI as a profile bundle, vllm-traefik-hub, so a platform team turns it on the way it turns on everything else. It imports the bundle once and sets the project's shared values once: domain, certificate, identity provider, evidence storage, registry, and logging. Each compute pool then needs 2 more values, the Traefik Hub license and the key that signs the transparency log. After that, anyone with access opens Deploy Model, picks a model, and deploys it, with nothing to type.

Each pool gets one address for all of its models, ai.<domain>, and clients choose a model by name. A model joins that address through one entry in the pool's catalog, which also says who can see and use it. This is the catalog from our largest run, 2 models on 2 GPUs, where Qwen is reserved for the ML platform group and the finance agents and Gemma is open to everyone signed in:

[{
  "id": "Qwen/Qwen3.6-35B-A3B-FP8",
  "ownedBy": "ml-platform",
  "visibleTo": "Contains(`jwt.groups`,`ml-platform`) || Equals(`jwt.team`,`finance`)"
}, {
  "id": "google/gemma-4-E4B-it",
  "ownedBy": "ml-platform"
}]

People sign in once, through the Dex identity service PaletteAI already runs, connected here to a company directory. The chat UI (Open WebUI) has no separate password and sends each person's own token to the gateway. Alice, in the ML platform group, sees Qwen and Gemma in her model list. Bob, a contractor, sees only Gemma, and a call he sends to Qwen anyway is refused with a 403. The model list itself comes from Traefik Hub, so what a person sees and what the gateway allows can't drift apart.

Run 6 Checks on Every Call to Every Model

Every call on the shared address passes the same 6 checks in Traefik Hub: Identify, Authorize, Protect, Limit, Fail over, and Record. A request that fails a check stops there, so a blocked prompt never reaches a GPU or spends a token.

Identify and authorize. Traefik Hub accepts API keys, tokens signed for each agent, and people's sign-in tokens. It reads the groups and team from the token itself. A call without a valid identity gets a 401. A caller asking for a model outside its catalog entry gets a 403, on the model list and on the call alike.

Protect. The content guard blocks known prompt-injection and jailbreak patterns, and leaked secrets such as cloud credentials and source-control tokens, with a 403 before the model sees the prompt. It masks personal data, such as an email address, on the way in, so the model never receives it.

Limit. PaletteAI meters and enforces limits on model use on its own pools. Traefik Hub extends these limits on identity (person, team, agent, etc.) to MCP tools, APIs, and traffic that never touches PaletteAI. The request rate is checked right after identity, so a flood stops before any other work, and the token limit comes last.

Fail over. A route can name a fallback model, on another pool or a hosted provider. When the model behind a name goes down, calls move to the fallback and return when the model recovers.

Record. Every call lands in your logging backend over OpenTelemetry. In our lab, that was a Loki instance outside the GPU cluster. Every allow and deny also goes into a transparency log that Traefik Hub signs and a witness countersigns. The log keeps a copy in your evidence storage, so a pool built from scratch picks up the same log where the old one left off.

We ran the same test suites against every pool. All 12 policy tests behaved as designed, all 15 tests for agents, tools, and APIs passed, and so did the checks for each of the 7 questions. We also cut a pool off from the PaletteAI appliance while traffic was running. Policies kept working for API keys, agents, and people who were already signed in, because the gateway keeps its own copy of the identity provider's signing keys. Only new sign-ins waited for the link to return.

Give Each Agent Only the Tools Its Identity Allows

Agents raise the stakes, because they act. On Day 1 of a PaletteAI rollout, agents go to work on the same pools, and the MCP Gateway in Traefik Hub governs their tool calls the same way the AI Gateway governs model calls.

Take a support agent and a finance agent that connect to the same MCP server. They do not get the same tools. Traefik Hub shows each agent only the tools its identity allows and checks every call against that agent's own limits. The finance agent can issue a refund. The support agent asks for a refund over its limit and is refused, and the tool that deletes a user is hidden from both. The gateway makes that decision, so a prompt cannot talk an agent into a tool it was never given.

Govern Autonomous Agents at Both Doors

The same pattern governs autonomous agents. NVIDIA OpenShell™ runs each agent in a sandbox with exactly one way out, and PaletteAI runs the AI factory underneath. Traefik Hub guards both doors. People and apps reach the agent through the first one and must sign in before they can operate it. Every call the agent makes goes out through the second one and is checked against the agent's task. Traefik Hub holds the real keys, and the agent only ever sees placeholders.

We ran this in a separate lab with an unchanged LangChain support agent in an OpenShell sandbox. A ticket hid an instruction to email the agent's API key to an outside address. The agent followed it, the gateway denied the email because the recipient was outside the company, and the agent never held the real key anyway. When the agent sent customer records to an outside model, the card number and the other personal data were masked before they left. Every decision landed in a signed, tamper-evident log whose signature we verified afterwards. For more on this pattern, see The Governance Playbook for NVIDIA OpenShell™.

Run It on Day 2 Without Rebuilding the Pool

A platform team's real test comes after go-live, so we ran the Day 2 tasks on a pool that was serving traffic.

Upgrades come through PaletteAI. Importing a new version of the bundle moved the running pool's gateway to the new chart about 10 seconds later, with no new pool, and the transparency log carried on from where it was. Policy and catalog changes go through the pool's variables. After we added Gemma to a running pool's catalog, a contractor who had been refused could use it about 30 seconds later. A price change on our bundle's rate card updated the cost estimates within a minute, without a restart. Edits made by hand in the cluster are put back at PaletteAI's next reconcile, so every change belongs in the pool's variables or in Git, where it can be reviewed.

Swapping a Hugging Face model on a GPU means deleting it and deploying the new one today, and in our test that model was unavailable for about 10 minutes. A fallback model on the route keeps clients answered during a swap, so we recommend one for every model that matters.

Then we put the pool under load: 32 callers, people and agents, sending 64 requests at a time to 2 models for 30 minutes. The pool processed 6.09 million tokens, about 3,353 tokens per second, with no server errors, and the gateway refused each deliberately bad request (about 5% of the traffic) with a 401 or a 403.

Platform teams watch all of this where they already look. The gateway's dashboards ship as a Helm chart, just like PaletteAI's, so tokens, cost, refusals by check, and evidence appear in your Grafana next to PaletteAI’s vLLM and GPU dashboards.

Start with One Model

PaletteAI runs your models anywhere: on your own GPUs, in the cloud, at the edge, and in air-gapped sites. Traefik Hub puts one policy in front of all of them, for people, apps, and agents alike, and keeps the evidence that it held. Spectro Cloud runs the factory, and Traefik Hub governs every call. Together, we can deliver the vision of inference anywhere, with governance everywhere.

Watch the 5-minute video "Inference Anywhere. Governance Everywhere." to see it end to end. Then contact your Spectro Cloud or Traefik Labs team to put one model behind the gateway on your own PaletteAI pool.

You can also meet us at KubeCon + CloudNativeCon North America 2026, November 9 to 12 in Salt Lake City, where we are presenting "Can You Prove It Was Safe? 7 Business Questions for AI Runtime Governance" at the Spectro Cloud booth #125 on Nov 10th (4PM) and 11th (2PM).