Within the last ten years, deep learning has moved from a research area to wild industry adoption thanks to the recent generative AI gold rush. This has led to the arrival of Large Language Models (LLMs) into mission-critical infrastructure at an unprecedented pace. Organizations are rapidly discovering that different LLMs excel at different tasks – from OpenAI, Claude or Gemini for generic content creation and problem-solving, up to domain-specific models for highly specialized tasks. This multi-model approach has become essential for maximizing AI's business value.

However, such a gold rush has created a familiar pattern in tech history: rapid adoption leading to fragmented infrastructure and technical debt. Each AI vendor has developed their own proprietary APIs, authentication methods, and SDKs, creating a "Wild West" environment without any governance and where complexity is king. As organizations scale their AI initiatives, this uncontrolled growth threatens to undermine the very benefits these technologies promise.

The Enterprise AI Challenge Landscape

Now that GenAI is being integrated into production environments, the challenges of large-scale AI deployment is becoming more apparent. Let’s dive into the most critical ones.

The number one problem is integration complexity. Development teams struggle with managing multiple SDK implementations, each with its own quirks and requirements. This complexity slows down development and creates maintenance nightmares as teams juggle different API versions and implementation patterns. Additionally, integrations with those different stacks lead to being vendor-locked as there is no straightforward way to migrate from one model to another.

Once integrated with those SDKs, companies end up facing even more issues in production. Security being the most visible. API keys and credentials scattered across various systems create significant security risks. Without centralized control, organizations struggle to maintain secure access patterns and protect sensitive credentials from exposure. 

The lack of standardized governance leads to inconsistent policy enforcement across different AI services. Without uniform and centralized authentication, authorization or rate limiting rules, compliance issues and resource waste cannot be avoided.

Finally, this fragmentation also leads to major observability black holes, with no way to optimize costs, monitor performance, or detect issues effectively.

The benefits of GenAI do not come for free and barriers are very real. That’s why at Traefik Labs, we made the decision to tackle those problems, with our unique expertise in ingress and API management.

Enter Traefik AI Gateway 

Traefik Labs addresses these new challenges with Traefik AI Gateway, a powerful solution that unifies and streamlines LLM management at the API level. By transforming any AI endpoint into a secure, governable API, Traefik AI Gateway allows enterprises to connect to multiple LLMs through a unified AI API, simplifying integration and centralizing control.

Traefik AI Gateway offers effortless integration with various popular LLMs, eliminating the need for multiple SDKs and client integrations. It ensures centralized security and credential management, minimizing risk and facilitating consistent policy enforcement. It prevents vendor lock-in, enabling easy switching between LLM providers without changing client applications. It supports OpenTelemetry for comprehensive observability across LLM usage, empowering data-driven operations and performance optimizations. Furthermore, Traefik AI Gateway offers unified governance through centralized policy enforcement, ensuring compliance with industry regulations.

Traefik AI Gateway is natively integrated to Traefik Hub and lets organizations gain access to enterprise-grade API capabilities that bring LLMs connectivity to industry standards with advanced access controls, security at scale and premium integrations. It means that AI apps can now benefit from the most advanced APIs features like rate limiting by plan, enterprise access control with identity providers or API keys, deep troubleshooting thanks to traffic debugging, etc.

It’s extremely simple to deploy Traefik AI Gateway, here is a quick overview on how to set it up on Traefik Hub.

First, enable the AI gateway feature by upgrading your Traefik Hub deployment:

helm upgrade traefik -n traefik --wait \
  --reuse-values \
  --set "additionalArguments={--hub.experimental.aigateway}" \
     traefik/traefik-hub

Then, define & apply an AIService resource with any of the supported AI providers, for example OpenAI:

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-openai
  namespace: traefik
spec:
  openai:
    token: YOUR_OPENAI_TOKEN
    model: gpt-4o

Finally, attach the ai-openai AIService to an IngressRoute as a TraefikService:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: ai-test
  namespace: traefik
spec:
  routes:
    - kind: Rule
      match: Host(`ai.localhost`)
      services:
        - kind: TraefikService
          name: traefik-ai-openai@ai-gateway-service

As simple as that! Now you can request OpenAI on http://ai.localhost:

curl -d '{
    "messages": [
        {
            "role": "user",
            "content": "tell me a joke"
        }
    ]
}' http://ai.localhost

{
  "id": "chatcmpl-AYGYJuSVWlp6gRFVvO9ClTN36eRkQ",
  "object": "chat.completion",
  "created": 1732730143,
  "model": "gpt-4o-2024-08-06",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Why don't scientists trust atoms?\n\nBecause they make up everything!"
      },
      "finish_reason": "stop",
      "content_filter_results": {
        "hate": {
          "filtered": false
        },
        "self_harm": {
          "filtered": false
        },
        "sexual": {
          "filtered": false
        },
        "violence": {
          "filtered": false
        },
        "jailbreak": {
          "filtered": false,
          "detected": false
        },
        "profanity": {
          "filtered": false,
          "detected": false
        }
      }
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 12,
    "total_tokens": 23,
    "prompt_tokens_details": {
      "audio_tokens": 0,
      "cached_tokens": 0
    },
    "completion_tokens_details": {
      "audio_tokens": 0,
      "reasoning_tokens": 0
    }
  },
  "system_fingerprint": "fp_831e067d82"
}

Traefik AI Gateway implements the Open Telemetry GenAI semantic conventions which provides metrics (gen_ai.client.token.usage and gen_ai.client.operation.duration) designed to monitor generative AI client applications thanks to specific attributes like token type or model. AI apps present specific challenges compared to traditional software systems and Traefik AI Gateway provides unique LLM observability capabilities to ensure platform teams have the best situation awareness.

Traefik AI Gateway is a natural evolution of Traefik's established expertise in Ingress & API management. Built on top of one of the most widely deployed open-source ingress controllers & API Gateways, Traefik Hub brings its battle-tested experience in managing incoming traffic to the emerging challenge of egress AI traffic.

Shaping the Future of AI Gateways

As enterprises continue to scale their AI initiatives, the need for structured, governed access to LLM services becomes increasingly critical. Traefik AI Gateway provides immediate solutions to today's challenges while laying the groundwork for future developments. 

Traefik Labs is deeply invested in the Gateway API effort, the next evolution of Kubernetes Ingress, and is actively working on developing specifications for LLM routing within the Gateway API framework. This ongoing commitment to open standards ensures that Traefik AI Gateway will continue to evolve with enterprise needs.

Organizations looking to bring order to their AI infrastructure should consider Traefik AI Gateway as a strategic investment in their AI strategy. With its combination of production-proven expertise in the ingress & API management space, Traefik is uniquely positioned to help enterprises tame the Wild West of LLMs and build a foundation for sustainable AI growth.

Useful Links

• Traefik AI Gateway Website
• Our Community Forum