Traefik Proxy 3.0 — Scope, Beta Program, and the First Feature Drop

December 7, 2022

Great news for all our Traefik Proxy users, Traefik Proxy 3.0 is almost here!

Between the Hackaethon, the big number of community contributions, and the cool features we are sitting on, our third major version comes packed with a very long list of updates that we are eager to share with you — so eager that we just cannot wait until it is complete to share! There are a lot of features in 3.0 to test ahead of the GA, so we are releasing a set of betas to allow you to play with the new features ASAP.

Today, we are announcing the release of Traefik Proxy 3.0 Beta 1, bringing community-wanted features, like OpenTelemetry support and default EntryPoints, Hackaethon contributions, and support for new and exciting technologies.

Let’s explore our new major version, when you can expect which updates, and how you can get started with the released features in Beta 1.

Traefik Proxy 3.0 — milestone topics and upcoming betas

As is befitting of a new major release of Traefik Proxy, 3.0 will bring an abundance of big new features and major improvements. The 3.0 milestones begin with your most wanted features and continue with global improvements. For 3.0, as a whole, we are focusing on the following themes:

Long-outstanding community requests: In 3.0, we want to bring you a list of long-awaited community requests, including support for the future of observability, OpenTelemetry, default and dynamic EntryPoints, and TCP servers transport.
Paving the ground for all future 3.x releases: To prepare for all future 3.0 releases, inevitably, we need to make some breaking changes to the core of Traefik Proxy. We will keep these limited and be judicious in which breaks we make.
Performance improvements: One of our biggest priorities and major focus for 3.0 has been improving performance. With the upcoming major release, you should expect a 20% improvement in Traefik's HTTP performance.
Kubernetes user experience: Another major focus for this release is improving the user experience for all Kubernetes providers: improvements for CRD, Ingress and Gateway API, aligning with their constructs, and exploring new integrations.
Documentation overhaul: Traefik Proxy documentation has been a sore spot for many users. We want to let you know that we take your feedback seriously and we are putting a lot of effort into improving our documentation. We are working on language clarifications, more guides and use cases, making documentation more like a story to follow rather than only a reference document, and fixing glitches across the documentation.
Pushing features out of experimental: Last but not least, some features are heavily used and now proven, so we will be removing the experimental flag for HTTP/3 support and the plugin mechanism.

And while we designed Traefik Proxy 3.0 to make adoption easy — including keeping breaking changes to a minimum and being judicious with the architecture — we know that ensuring ample time for the community to explore and stress-test 3.0 is the right decision.

That’s why we will have multiple beta versions of Traefik Proxy 3.0, starting today with Beta 1. We want the community to have the chance to play with the features, stress-test, and pinpoint any feedback you wish to give early on so that we can incorporate changes quickly.

Traefik Proxy 3.0 Beta 1 and the first feature drop

And with no further ado, let’s jump into the goody bag of Traefik Proxy 3.0 Beta 1! The first beta of 3.0 introduces a long list of updates, primarily focused on providing the community with long-awaited features, as well as making any necessary core changes. We categorize 3.0 Beta 1 features as follows:

Community requests: OpenTelemetry support, default EntryPoints, and custom headers for the HTTP provider
Hackaethon contributions: Brotli compression and SPIFFE mTLS support
Internal contributions: gRPC-Web middleware, gRPC-Web metrics, and Tailscale certificate resolver
Breaking changes: new logging mechanism, consolidating middleware and router configurations, and reworking our internal load balancing

Let’s break it down.

OpenTelemetry support

Historically, there have been countless open source tools out there handling application telemetry as metrics and traces. Because observability is increasingly important for DevOps teams, the number of such tools grew exponentially, with each tool specializing in a specific area. This becomes problematic as teams would have to adopt numerous disconnected tools to fully cover their needs. OpenTelemetry is the initiative that brings everything under one roof!

With OpenTelemetry, DevOps teams don’t have to worry about configuring several kinds of instrumentation to collect data and get the coverage they need. The OpenTelemetry Collector can then be configured to receive, compute and forward telemetry information to your backend of choice. In short, OpenTelemetry is becoming the long-awaited single source of truth for all telemetry data, whether it is metrics, logs, or traces.

Traefik Proxy 3.0 now supports the OpenTelemetry Protocol (OTLP), which allows Traefik to send platform and language-agnostic telemetry data to any OpenTelemetry Collector, whether metrics or traces. After optional processing, the OpenTelemetry Collector can then send the telemetry data to any monitoring or observability-compliant solution you choose to use.

To enable it, just add the address of the collector and select what level of details you want to be included under metrics and/or tracing, and the new OpenTelemetry option. Here is an example with metrics:

# Static Configuration
metrics:
  openTelemetry:
    address: otel-collector:4318
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true

With Traefik now sending data to the OpenTelemetry Collector, you can update other tools to read from the Collector as well — for example, if you were previously running Prometheus, you can keep it as an exporter in the Collector while, at the same time, exposing the data in native OTLP format as well. This is useful when you're planning to migrate to OpenTelemetry gradually on a per-tool approach. Here is how the Collector config would look like in this scenario:

# Open Telemetry Collector Configuration
receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  prometheus:
    endpoint: ":8889"

service:
  telemetry:
    logs:
      level: error
    metrics:
      address: 0.0.0.0:8888
      level: basic
 
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

Note: It is not mandatory to set up a monitoring or observability tool to take advantage of OpenTelemetry with Traefik Proxy. Metrics can be directly fed from Traefik Proxy to the OpenTelemetry Collector and exported using OTLP.

Default EntryPoints

What EntryPoints are used by default when you do not specify an EntryPoint. With the old Traefik Proxy behavior, when an EntryPoint was not specified, the application was exposed to all available EntryPoints.

Following best practices, we are changing the behavior of Traefik Proxy and will now allow you to specify default EntryPoints in the static configuration, so when an EntryPoint is not specified, Traefik will automatically expose the application only to the predefined default EntryPoint, and no longer to all the available EntryPoints.

Here is an example:

# Static Configuration
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
    asDefault: true

This translates to using only the EntryPoint websecure to expose services that don’t explicitly set an EntryPoint.

SPIFFE mTLS

SPIFFE is a control plane that manages authentication and is a great way to authenticate certificates used for mTLS — when both the client and the server need a certificate to encrypt data. Thanks to our second-place Hackaethon winner, Julien Levesy, Traefik is now usable in a SPIFFE environment.

The contribution made by Julien — support for SPIFFE mTLS between Traefik Proxy and backend servers — makes it possible for Traefik Proxy to fetch certificates from SPIFFE before contacting a backend server. The configuration is done within the servers transport section of the Traefik static configuration.

# Static Configuration
serversTransport:
  spiffe:
    ids:
      - spiffe://trust-domain/id1

After defining the SPIFFE configuration parameters, it's just a matter of assigning it to your router services configuration, here’s an example using the File provider:

# Dynamic Configuration (File Provider)
http:
  services:
    whoami-svc:
      loadbalancer:
        serversTransport: spiffe

Brotli compression

We hear you! Support for the Brotli compression algorithm is a long-outstanding, community-wanted feature. That is why it is so special for us that this is a contribution from the amazing third-place Hackaethon winner, Greg Linton. Brotli further compresses (compared to gzip, for example) data sent over the wire, which is helpful in certain use cases — for example, when the backend sends an uncompressed response — provided that the client supports Brotli compression. Traefik Proxy 3.0 now leverages Brotli in the existing Compress middleware.

No changes are required on the Compress middleware declaration as the compression algorithm will be picked up automatically based on the negotiation between the client request and the backend.

gRPC-Web middleware and metrics

gRPC is Google’s reimplementation of a Remote Procedure Call (RPC) framework used for connecting devices, mobile applications, and browsers to backend services. It runs in all environments and offers high performance. gRPC-Web is a gRPC client made for the frontend application, so browser client applications can also connect to gRPC backends. And while gRPC has been around for some time now, gRPC-Web is a new and trendy technology but without massive adoption as browsers can’t work with protobuffs directly. We, however, can see gRPC-Web gaining attention fast in the near future, so we decided to bring it to Traefik Proxy and allow our users to benefit from this early adoption.

The gRPC-Web middleware for Traefik Proxy acts as the intermediary between a web application that uses HTTP and wants to communicate with gRPC backend, effectively translating HTTP to gRPC. To use the middleware, there is only a minimum configuration required.

# Dynamic Configuration (File Provider)
http:
  middlewares:
    mywebapp:
      grpcWeb:
        allowOrigins:
         - "*"

We are also adding support for gRPC-Web metrics. When Traefik Proxy exposes an application through the gRPC-Web middleware, it attaches a grpc or grpcweb to the ‘code’ label in all metrics collected. So, further along the line, when looking at your metrics on your monitoring or observability tool, you will be able to distinguish which data come from regular connections and which come from the gRPC-Web middleware.

Tailscale certificate resolver

Users of Tailscale, rejoice! Traefik Proxy now supports fetching TLS Certificates from tailnet. Front apps can now be accessible over tailnet.

Tailscale is a VPN service that allows you to create your own private networks from your home, using whatever device you want. It allows you to connect those devices using that personal network (tailnet), so you can access an application exposed in one of your devices, from other devices, and from any place around the word. Tailnet is secured with end-to-end encryption using WireGuard.

But the Tailscale team went beyond just providing a service to create a private network. Tailscale offers several cool features and tools that make the life of homelab developers, particularly, a lot easier. One of those features is TLS certificate management, where Tailscale provides you with a valid certificate for your internal Tailscale services. Behind the scenes, Tailscale will still get the certificate from Let’s Encrypt, so it is a valid certificate that can be used in a face-fronting domain exposed to the internet. The biggest benefit here is that Tailscale manages the certificate lifecycle for you, so no need to worry about renewing or exposing an endpoint to resolve TLS challenges between Let’s Encrypt and your proxy instance.

In 3.0 Beta 1, we are adding support for this functionality. This means that instead of using the typical Let’s Encrypt integration, you can now use the Tailscale certificate resolver that will connect to the Tailscale API, get the certificate from your private Tailscale network, and make it available to routers that you’re exposing with Traefik Proxy.

# Static Configuration
certificatesResolvers:
  myresolver:
    tailscale: {}

All you need to do now is reference the certificate resolver that was declared on the static configuration in your routers. Here is an example with the File provider:

# Dynamic Configuration (File Provider)
http:
  routers:
    whoami:
      entryPoints:
      - websecure
      service: whoami
      rule: Host(`mydomain`)
      tls:        
        certResolver: myresolver

This new addition to Traefik Proxy is particularly close to our hearts as it is a contribution made by two of our own engineers, during their off-work hours. Thank you, Kevin and Mathieu, for your everlasting dedication to improving Traefik Proxy!

Custom headers for the HTTP provider

The HTTP provider is a method that allows setting up Traefik Proxy without any configuration. The HTTP provider enables an endpoint in Traefik Proxy where you send the dynamic configuration payload. The provider then applies the router config sent in the payload to Traefik Proxy. With this release, we are making a small update to the existing HTTP provider. We are adding the headers option to the HTTP provider to define custom headers to be sent in the fetch request, which can help simplify configuration for complex use cases.

Here’s an example of the configuration:

# Static Configuration
providers:
  http:
    endpoint: "http://127.0.0.1:9000/api"
    headers:
      my-header: foobar

New logging mechanism

Part of our mission for the 3.0 release is to refine the inner working of Traefik Proxy to provide our users with a better experience overall, starting with improvements for our internal logging mechanism. This change aims to address a few issues:

Maintaining the fields’ order between logs
Improving readability for string outputs (minimizing escape characters)
Providing a more precise timestamp (milliseconds)
Improving readability for the common format (field positioning, colors, etc.)

For our new logging mechanism, we are switching from Logrus to zerolog. The main motivation behind this move was that Logrus entered maintenance mode. And while the project is being adequately maintained, there will be no new features. This, combined with the fact that the new generation of logging libraries provides a set of impressive features and better performance, made this change a no-brainer.

For now, this change only impacts Traefik logs and not access logs. The access logs will be using the old mechanism for Beta 1, but we plan to propagate the new mechanism to include the access logs for the Traefik Proxy 3.0 GA.

Old (common format):

New (common format):

Warning: We might change field names or log messages during the development of 3.0, so if you’re using a structured log format, such as JSON, and relying on specific field names or values, this could introduce a breaking change in your environment. Be sure to update your alerting rules or any other triggers that are based on log output.

Consolidating middleware and router configurations

In the previous Traefik Proxy versions, we were using different syntax for router rules and middleware configurations which has caused some confusion:

Regexps were defined using the gorilla/mux syntax, which arbitrarily enforces naming matching groups. This syntax added complexity, which was not even used by Traefik Proxy.
It was sometimes unclear which matcher used regexps (e.g., there were Host, and HostRegexp matchers, but not Path, and PathRegexp matchers).
Matchers that accept multiple values were ORing them, when it makes sense, (e.g., Host, Path, etc.), or ANDing them when it didn’t (e.g., Query, Headers, etc.).
Even though this approach is not wrong, it may cause confusion if a user expected those matchers to deal with their values similarly (i.e., ORing all of them).
Both the Query and Headers matchers expected a key and value as arguments.

In Traefik Proxy 3.0 Beta 1, we are now making the configuration syntaxes consistent between matchers:

Regular expression matchers now use Go's regexp syntax
Regular expression matchers are now explicitly named SomeMatcherRegexp.
All matchers now take a single value and should be explicitly combined using logical operators to mimic previous behavior
Removed the deprecated HostHeader matcher
Renamed Headers* matchers to Header*

These changes transform:

rule: Host(`example.com`, `example.org`) && Path(`/products/{id:[a-z]+}`) || Query(`mobile=true`)

into:

rule: (Host(`example.com`) || Host(`example.org`)) && PathRegexp(`/products/[a-z]+`) || Query(`mobile`, `true`)

Note: During the beta phase, we are exploring the possibility of adding automation mechanisms in the migration process of the dynamic configuration. However, since every provider has a different configuration scheme, the approach might also be different for each provider — it could range from an automated CLI that converts the File provider from old to new format, to a step-by-step guide on how to update labels on Docker. We’ll have more on this in the future.

Reworking our internal load balancing

With this release, we are making a change to how Traefik Proxy handles load balancing. Previously, we used regular round robin in some cases and weighted round robin (WRR) in others. Starting with 3.0 Beta 1, we use WRR by default to be consistent throughout all use cases. On top of that, services are now being reused between routers, giving us a slight increase in throughput performance while also improving the overall distribution of requests. These changes will impact users in a few ways — one slightly negative and several positive.

Breaking sticky sessions: Active sessions will be discarded and you will have to re-login to your application.
Service load balancing: Services are now reused by routers, so load balancing is shared, making the distribution more consistent overall.
Removing gorilla/mux regexp syntax: Naming matching groups in regular expressions is no longer required.
- Old syntax: /products/{id:[a-z]+}
- New syntax: /products/[a-z]+
We're also taking the opportunity to consolidate how you express rules throughout Traefik Proxy, for example:
- PathRegexp instead of Path
- Host(example.com) || Host(example.org) instead of Host(example.com, example.org)
Content-Type auto-detection is disabled by default: We reversed the behavior to explicitly disable the Golang content-type auto-detection by default — as this is the expected behavior from a proxy server — to not touch the content type header. To enable it again, you must add a middleware to the router configuration.
We measured an average of 2% increase in throughput performance on HTTP and HTTPS scenarios, mainly due to the reuse of services in the router load balancer.

Deprecations

Naturally, with so many changes and new features making their way into Traefik Proxy, we are also removing some already deprecated features. Namely, we removed:

the pilot option from the static configuration
the tracing.datadog.globaltag option
the namespace option of Consul, Consul Catalog, and Nomad providers
the tls.caOptional option from the ForwardAuth middleware, as well as from the HTTP, Consul, Etcd, Redis, ZooKeeper, Marathon, Consul Catalog, and Docker providers
sslRedirect, sslTemporaryRedirect, sslHost, sslForceHost, and featurePolicy options of the Headers middleware
the forceSlash option of the StripPrefix middleware
the preferServerCipherSuites option

For more information on the impact and how to update your configuration, see the migration guide.

More to come

Whew — it looks like a lot, but this is just the beginning! The second beta is expected in early 2023, and it targets another long list of features for you to test and play with, most notably, HTTP performance improvements by implementing FastHTTP over the base HTTP package and improved support for Kubernetes Defaults.

To get started with Traefik Proxy 3.0 Beta 1, check out the release notes and our documentation pages, and don’t forget to visit the Forum for all the latest community topics. You can also find the migration guide here. What’s more, as I mentioned earlier, for the Traefik Proxy 3.0 GA, we are looking into automation opportunities and bringing you a CLI migration tool, but more on that in future Beta releases.

Stay tuned!