How to use Traefik Proxy and HTTP/3 on AWS EKS

While working on Traefik Proxy and playing around with the implementation of HTTP/3, I came across an important limitation in Kubernetes.

Traefik Proxy shipped with experimental support for HTTP/3 in 2.5. It worked out of the box but required some special configuration to work in Kubernetes — since Traefik Proxy and HTTP/3 required TCP and UDP on the same port. Traefik Proxy 2.6 comes to save the day and make this process smoother.

Create a Secured Gateway to Your Applications with Traefik HubJoin us to learn how to secure and expose applications and services using a combination of a SaaS network control plane and a lightweight, open source agent.Register Today

Usually, in Kubernetes, an Ingress Controller is exposed through a LoadBalancer Service. Kubernetes ships multiple implementations of the service load balancer. This glue code calls out to various IaaS platforms (AWS, GCP, Azure, etc.)

Depending on the feature gates in your cluster, you are allowed (or not) to create a LoadBalancer Service with a mixed protocol. To enable mixed protocol on a LB Service, you can use the MixedProtocolLBService feature gate.

Today in AWS EKS, this feature gate is disabled by default. There is an open issue on the AWS Load Balancer Controller in order to create the listener for the TCP_UDP type on the provisioned LB Service with mixed protocols.

So, long story short, this requires you to find another solution in order to deploy Traefik Proxy and HTTP/3 with mixed protocol on a service. In this article, I’ll walk you through how to successfully switch to a NodePort Service to deploy a Network Load Balancer (NLB) in AWS and allow TCP and UDP on the same port.

Prerequisites

Before I jump in and get started, here’s what you’ll need to follow this tutorial.

Access to an AWS account with authorization to manage the following resources:
ec2:*
elasticloadbalancing:*
cloudwatch:*
autoscaling:*
iam:CreateServiceLinkedRol
eksctl command line to create and manage the EKS cluster
aws cli to be able to manage AWS resources

Note: I will be using AWS EKS for the purpose of this article, but the same principles apply to other Kubernetes distributions and cloud providers.

Getting started

Create the cluster

First things first — create a Kubernetes cluster with eksctl. It will take you approximately 20 minutes to have the cluster up and running.

export AWS_REGION=eu-north-1 # Define the AWS region
export CLUSTER_NAME=traefik-http3 # Cluster name
export KUBECONFIG=${HOME}/.kube/traefik-http3 # Define a new KUBECONFIG for the example

## Create the EKS cluster
eksctl create cluster \
 --name ${CLUSTER_NAME} \
 --region ${AWS_REGION} \
 --version 1.21

Make sure you have access to the created cluster:

kubectl get nodes

NAME                                            STATUS   ROLES    AGE    VERSION
ip-192-168-45-130.eu-north-1.compute.internal   Ready    <none>   102s   v1.21.5-eks-bc4871b
ip-192-168-87-177.eu-north-1.compute.internal   Ready    <none>   104s   v1.21.5-eks-bc4871b

Install Traefik Proxy

The next step is to install Traefik Ingress Controller into our EKS cluster to support HTTP/3.

This is a classic Traefik Proxy installation in Kubernetes. First, install the CRDs, and create some RBAC to allow Traefik Proxy to communicate with the Kubernetes API. Once you’re done, create a deployment.

The deployment stage is the most important. That’s when we need to configure Traefik proxy to support HTTP/3.

This feature is experimental on Traefik Proxy, so we have to add the --experimental.http3 argument in the static configuration.

Once it is activated, we have to enable HTTP/3 on the websecure entrypoint with the --entrypoints.websecure.http3 argument.

Install Traefik Proxy:

## Deploy CRDs
kubectl apply -f https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/0-crds.yaml
## Deploy RBAC
kubectl apply -f https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/1-rbac.yaml
## Deploy Traefik Proxy
kubectl apply -f https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/2-deployment.yaml

Check that Traefik Proxy running smoothly:

kubectl get pods

Exposing Traefik Proxy

Now that you have your Traefik proxy up and running, you need to find a way to expose it to the internet.

LoadBalancer Service

Let’s create a Kubernetes service of Type LoadBalancer and add annotations to ask for a NetworkLoadBalancer on AWS. HTTP/3 is running on QUIC, so you need to have a LoadBalancer listening to TCP and UDP on the same port.

## https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/services/3-service-lb.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb" # Ask for a Network Load Balancer on AWS
spec:
  type: LoadBalancer
  ports:
    - protocol: TCP
      name: web
      port: 80
    - protocol: TCP # We have to listen on TCP on port 443
      name: websecure-tcp
      port: 443
    - protocol: UDP # We have to listen on UDP on port 443 for HTTP/3
      name: websecure-udp
      port: 443
  selector:
    app: traefik

Next, deploy the LoadBalancer Service:

kubectl apply -f https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/services/3-service-lb.yaml

The Service "traefik" is invalid: spec.ports: Invalid value: []core.ServicePort{core.ServicePort{Name:"web", Protocol:"TCP", AppProtocol:(*string)(nil), Port:80, TargetPort:intstr.IntOrString{Type:0, IntVal:80, StrVal:""}, NodePort:0}, core.ServicePort{Name:"websecure-tcp", Protocol:"TCP", AppProtocol:(*string)(nil), Port:443, TargetPort:intstr.IntOrString{Type:0, IntVal:443, StrVal:""}, NodePort:0}, core.ServicePort{Name:"websecure-udp", Protocol:"UDP", AppProtocol:(*string)(nil), Port:443, TargetPort:intstr.IntOrString{Type:0, IntVal:443, StrVal:""}, NodePort:0}}: may not contain more than 1 protocol when type is 'LoadBalancer'

As you can see in the output, you are not authorized to create a LoadBalancer listing on both TCP and UDP simultaneously. This is due to a limitation of the AWS LoadBalancer Controller

NodePort Service

As a workaround to the limitation seen above, you have to create a NodePort Service and create the NLB manually.

## https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/services/3-service-node-port.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: NodePort
  ports:
    - protocol: TCP
      name: web
      targetPort: 80
      nodePort: 30442
      port: 80
    - protocol: TCP
      name: websecure-tcp
      targetPort: 443
      nodePort: 30443
      port: 443
    - protocol: UDP
      name: websecure-udp
      targetPort: 443
      nodePort: 30443
      port: 443
  selector:
    app: traefik

kubectl apply -f https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/services/3-service-node-port.yaml

Accessing the NodePort Service

You’ve just successfully created the NodePort. Now it’s time to create some AWS resources to access it.

The following commands will create a Network Load Balancer and configure it to forward the traffic to the NodePort you just created.

Retrieve information about the EKS cluster:

## Retrieve the VPC ID.
VPC_ID=$(aws eks describe-cluster --name ${CLUSTER_NAME} --query 'cluster.resourcesVpcConfig.vpcId' --output=text)

## Retrieve the node group ID.
NODE_GROUP_ID=$(aws eks list-nodegroups --cluster-name=${CLUSTER_NAME} --query 'nodegroups[0]' --output text)

## Retrieve instances subnet IDs.
INSTANCES_SUBNET_IDS=$(aws ec2 describe-instances --filters Name=network-interface.vpc-id,Values=$VPC_ID "Name=tag:eks:nodegroup-name,Values=$NODE_GROUP_ID" --query 'Reservations[*].Instances[*].SubnetId' --output text | tr '\n' ' ')

Create the NLB:

## Create the Network Load Balancer with subnets retrieve just before.
aws elbv2 create-load-balancer --name ${CLUSTER_NAME} \
  --type network \
  --subnets $(echo ${INSTANCES_SUBNET_IDS})

Create the target group for the NLB:

TG_NAME=${CLUSTER_NAME}-tg

## Create a TCP target group for web entrypoint on port 30442.
aws elbv2 create-target-group --name ${TG_NAME}-web --protocol TCP --port 30442 --vpc-id ${VPC_ID} \
  --health-check-protocol TCP \
  --health-check-port 30442 \
  --target-type instance

## Create a TCP_UDP target group for websecure entrypoint on port 30443.
aws elbv2 create-target-group --name ${TG_NAME}-websecure --protocol TCP_UDP --port 30443 --vpc-id ${VPC_ID} \
  --health-check-protocol TCP \
  --health-check-port 30443 \
  --target-type instance

## Retrieve instances IDs.
INSTANCES=$(kubectl get nodes -o json | jq -r ".items[].spec.providerID"  | cut -d'/' -f5)
IDS=$(for x in `echo ${INSTANCES}`; do echo Id=$x ; done | tr '\n' ' ')
echo $IDS

## Retrieve the target group Arn for the web entrypoint.
TG_ARN_WEB=$(aws elbv2 describe-target-groups --query 'TargetGroups[?TargetGroupName==`traefik-http3-tg-web`].TargetGroupArn' --output text)

## Register instances to the previous TargetGroup web.
aws elbv2 register-targets --target-group-arn ${TG_ARN_WEB} --targets $(echo ${IDS})

## Retrieve the target group Arn for the websecure entrypoint.
TG_ARN_WEBSECURE=$(aws elbv2 describe-target-groups --query 'TargetGroups[?TargetGroupName==`traefik-http3-tg-websecure`].TargetGroupArn' --output text)
## Register instances to the previous TargetGroup websecure.
aws elbv2 register-targets --target-group-arn ${TG_ARN_WEBSECURE} --targets $(echo ${IDS})

Create listeners:

## Retrieve the NLB Arn
LB_ARN=$(aws elbv2 describe-load-balancers --names ${CLUSTER_NAME} --query 'LoadBalancers[0].LoadBalancerArn' --output text)

## Create a TCP listener on port 80 that will forward to the TargetGroup traefik-http3-web
aws elbv2 create-listener --load-balancer-arn ${LB_ARN} \
  --protocol TCP --port 80 \
  --default-actions Type=forward,TargetGroupArn=${TG_ARN_WEB}

## Create a TCP_UDP listener on port 443 that will forward to the TargetGroup traefik-http3-websecure
aws elbv2 create-listener --load-balancer-arn ${LB_ARN} \
  --protocol TCP_UDP --port 443 \
  --default-actions Type=forward,TargetGroupArn=${TG_ARN_WEBSECURE}

Configure instances security groups:

# Retrieve the security groups for each instances.
SGs=$(for x in $(echo $INSTANCES); do aws ec2 describe-instances --filters Name=instance-id,Values=$x \
 --query 'Reservations[*].Instances[*].SecurityGroups[0].GroupId' --output text ; done | sort | uniq)

for x in $(echo $SGs); do
  echo SG=$x;
  aws ec2 authorize-security-group-ingress --group-id $x --protocol tcp --port 30442 --cidr 0.0.0.0/0;
  aws ec2 authorize-security-group-ingress --group-id $x --protocol tcp --port 30443 --cidr 0.0.0.0/0;
  aws ec2 authorize-security-group-ingress --group-id $x --protocol udp --port 30443 --cidr 0.0.0.0/0;
done

## Retrieve the NLB Arn and extract the ID.
NLB_NAME_ID=$(aws elbv2 describe-load-balancers --names ${CLUSTER_NAME} --query 'LoadBalancers[0].LoadBalancerArn' --output text | awk -F":loadbalancer/" '{print $2}')

## Retrieve the NLS DNS name.
NLB_DNS_NAME=$(aws elbv2 describe-load-balancers --names ${CLUSTER_NAME} --query 'LoadBalancers[0].DNSName' --output text)

And voila! Your application is now ready to be deployed.

Deploy your application

You can start by deploying the dashboard IngressRoute and following the steps below.

Deploy the dashboard IngressRoute:

kubectl apply -f https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/traefik/dashboard/4-ingressroute.yaml

Access the Traefik Proxy dashboard:

open https://${NLB_DNS_NAME}/dashboard/

Deploy a demo application:

kubectl apply -f https://raw.githubusercontent.com/traefik-tech-blog/traefik-http3/master/app/app.yaml

Access the application:

open https://${NLB_DNS_NAME}/

Check your application with http3check:

open https://http3check.net/?host=${NLB_DNS_NAME}

And last but not least, it’s time for some clean-up.

## Delete the Network Load Balancer.
aws elbv2 delete-load-balancer --load-balancer-arn ${LB_ARN}

## Delete the 2 target groups created.
aws elbv2 delete-target-group --target-group-arn ${TG_ARN_WEB}
aws elbv2 delete-target-group --target-group-arn ${TG_ARN_WEBSECURE}

## Delete the EKS cluster.
eksctl delete cluster --name ${CLUSTER_NAME} --region ${AWS_REGION}

Conclusion

This guide showed you one way to configure your Ingress Controller Traefik Proxy with the support of HTTP/3 on an EKS cluster. This process provides you with a manual solution to an important Kubernetes limitation — at least until the issue is resolved with the next Kubernetes release.

And as I mentioned earlier, the same principles apply to other distros and cloud providers.

You can find the source code used in this guide on GitHub.

If you want to learn more about the HTTP changes introduced in Traefik Proxy 2.6, check out Adrian Goins’ article and join us for our upcoming webinar, where we’ll explore all the new features of Traefik Proxy.