Secure Web Applications with Traefik Proxy, cert-manager, and Let’s Encrypt

secure web apps with traefik cert manager and lets encrypt

Managing TLS certificates has never been easier. Not that long ago, running secure websites was a tedious job for engineers as they had to deal with complex business processes and chores. Who does not remember the times when you had to make a purchase requisition, get in touch with your vendor with your Certificate Signing Request (CSR), watch out for an email to validate your domain, and eventually announce a maintenance window that you will have fun with deploying the certificate in production?

Phew — I’m certainly glad those days are gone!

Shaken by the revolutionary non-profit Certificate Authority, Let’s Encrypt, and its ACME protocol, the market gradually moved into fully-automated solutions that enabled developers to deliver secure websites at no costs with the least effort.

Since day one, Traefik Proxy provides a native Let’s Encrypt integration to automate the full lifecycle of certificates. Without the need to handle any third-party tooling, Traefik Proxy is the natural choice for automated certificate management.

While using a single instance of Traefik Proxy with Let's Encrypt works like a charm, however, running multiple instances can raise some issues. If your production environment requires you to use Let's Encrypt with high availability (HA) in Kubernetes, you always have the option of Traefik Enterprise, which includes distributed Let's Encrypt as a supported feature.

But if you want to stick with Traefik Proxy, you have nothing to fear!

With Kubernetes we got a powerful and extensible platform to solve a lot of complex scenarios. cert-manager is a powerful solution that helps us automate and manage almost everything around TLS certificates. It provides a set of Custom Resource Definitions (CRD) for various scenarios and integrates well with native Ingress or Gateway resources.

cert-manager stores and caches certificates and private keys in Kubernetes secrets, making them highly available for further consumption by ingress controllers (like Traefik Proxy) or applications.

Note: By default, cert-manager does not clean up secrets automatically, allowing it to re-attach to already issued certificates and avoid issuing new certificates. This becomes very handy in scenarios when you need to create and delete lots of resources and would not like to be rate limited.

cert-manager can interact with a variety of sources to issue certificates including Let’s Encrypt, HashiCorp Vault as well as private PKI. For unsupported cases like AWS Private Certificate Authority, Google Cloud Certificate Authority Service or Cloudflare Origin CA the External Issuer allows you to extend cert-manager capabilities..

But enough talk! Time to get down to business and dig into how you can use cert-manager to extend Traefik Proxy’s capabilities.

Prerequisites

To follow this tutorial, you’ll need the following:

A Kubernetes cluster >= v1.20
A public hosted DNS domain for Let’s Encrypt — for the purpose of this article I will use Cloudflare
A Kubernetes native ingress controller: Traefik Proxy 2.9, you can install the helm chart with this command:
```
helm install traefik traefik/traefik
```

cert-manager 1.10 which you can install with this command:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml

A service providing a web port. In this tutorial, I’ll be using whoami as an example:

apiVersion: v1
kind: Namespace
metadata:
 name: whoami
---
apiVersion: v1
kind: Service
metadata:
 name: whoami
 namespace: whoami
spec:
 ports:
   - name: web
     port: 80
     targetPort: web
 selector:
   app: whoami
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: whoami
 namespace: whoami
spec:
 selector:
   matchLabels:
     app: whoami
 template:
   metadata:
     labels:
       app: whoami
   spec:
     containers:
       - name: whoami
         image: traefik/whoami
         ports:
           - name: web
             containerPort: 80

Traefik Proxy with cert-manager and Let’s Encrypt

Let’s explore how we can secure a web application in combination with a Kubernetes ingress controller like Traefik Proxy and cert-manager. Let’s Encrypt provides multiple challenge types to validate control of a domain name. Depending on your requirements you may choose HTTP-01 when your service is public reachable or DNS-01 for private endpoints.

Please be aware of rate limits when using lets encrypt. To avoid unpleasant surprises it is recommended to use the Let’s Encrypt staging environment:

staging: https://acme-staging-v02.api.letsencrypt.org/directory`
production: https://acme-v02.api.letsencrypt.org/directory

HTTP challenge

For most common scenarios the HTTP-01 challenge is a convenient start to solve an ACME based validation. To make this scenario work, Traefik Proxy needs to be reachable from the internet on HTTP port 80, and the used DNS domain has to be configured to point to it.

When a new certificate needs to be issued (or renewed), cert-manager will create a temporary Ingress resource to route requests made by the ACME server to the specific matched host and ./well-known/acme-challenge/xxx path, so it can answer with the desired response.

Implementing the challenge

First you need to define a new cert-manager Issuer to represent a certificate issuing authority. This example uses the ACME-based Certificate Authority in conjunction with Let’s Encrypt.

Note: You need to change the server to production to retrieve a certificate that will be accepted by your browser.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: le-example-http
 namespace: whoami
spec:
 acme:
   email: [email protected]
   # We use the staging server here for testing to avoid hitting
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     # if not existing, it will register a new account and stores it
     name: example-issuer-account-key
   solvers:
     - http01:
         # The ingressClass used to create the necessary ingress routes
         ingress:
           class: traefik

Next, you’ll need a Kubernetes Ingress resource to define the domain for TLS we want to attach.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: whoami
 namespace: whoami
 annotations:
   cert-manager.io/issuer: "le-example-http"
spec:
 tls:
   - hosts:
       - whoami.example.com
     secretName: tls-whoami-ingress-http
 rules:
   - host: whoami.example.com
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: whoami
               port:
                 name: web

cert-manager automatically creates a new Certificate resource for the specified domain with the given secretName, provisions a CertificateRequest to request a signed certificate from one of the configured issuers, and stores the certificate and private key with the same name as the secret. The annotation cert-manager.io/issuer requires the name of the previously created Issuer and enables the resource to be managed by cert-manager.

Once the secret has been created, Traefik Proxy will fetch the certificate and private key and will serve it when the requested domain is called. Alternatively, you can also deploy a ClusterIssuer resource, which is accessible across all namespaces and referenced by the annotation cert-manager.io/cluster-issuer.

Note: cert-manager will not clean up certificates on its own, so they can be easily re-attached even if someone makes changes to the given Ingress object. If there is already an existing and valid certificate in place, it will be re-used.

DNS challenge

In some cases, you are not able to use the HTTP challenge (usually when your service is only internally available) and have to fall back to a DNS challenge. All you need to have in place is a registered DNS domain that can be resolved from the internet.

Unfortunately, cert-manager only supports a small range of DNS providers natively or dynamic DNS via RFC2136. Luckily there is the option to extend this with custom webhook solvers, so make sure to check out existing projects before implementing your own.

Implementing the challenge

The process looks almost the same as with the HTTP challenge. Instead of specifying the HTTP challenge, you need to set up the Issuer for using the DNS challenge. cert-manager will take care of creating the necessary validation records in the respected DNS zone.

apiVersion: v1
kind: Secret
metadata:
 name: cloudflare-api-token-secret
type: Opaque
stringData:
 api-token: <API Token>
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: le-example-dns
 namespace: whoami
spec:
 acme:
   email: [email protected]
   # We use the staging server here for testing to avoid hitting
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     # if not existing, it will register a new account and stores it
     name: example-issuer-account-key
   solvers:
     - dns01:
         cloudflare:
           apiTokenSecretRef:
             name: cloudflare-api-token-secret
             key: api-token

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: whoami
 namespace: whoami
 annotations:
   cert-manager.io/issuer: "le-example-dns"
spec:
 tls:
   - hosts:
       - whoami.example.com
     secretName: tls-whoami-ingress-dns
 rules:
   - host: whoami.example.com
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: whoami
               port:
                 name: web

Troubleshooting

All cert--manager resources provide handy status and event information. It helps you understand problems and verify everything is working as expected.

$ kubectl -n whoami get issuer -o wide
NAME              READY   STATUS
le-example-http   True    The ACME account was registered with the ACME server

$ kubectl -n whoami get certificateRequest -o wide
NAME                            APPROVED   DENIED   READY   ISSUER            STATUS
tls-whoami-ingress-http-fdw2x   True                True    le-example-http   Certificate fetched from issuer successfully

$ kubectl -n whoami get certificates
NAME                      READY   SECRET                    ISSUER            STATUS
tls-whoami-ingress-http   True    tls-whoami-ingress-http   le-example-http   Certificate is up to date and has not expired

$ kubectl -n whoami describe secret tls-whoami-ingress-http
Annotations:  cert-manager.io/alt-names: whoami.example.com
             cert-manager.io/certificate-name: tls-whoami-ingress-http
             cert-manager.io/common-name: whoami.example.com
             cert-manager.io/issuer-name: le-example-http

Type:  kubernetes.io/tls

Data
====
tls.crt:  2449 bytes
tls.key:  1679 bytes

Summary

This blog post just scratched the surface on the possibilities of cert-manager in conjunction with Let’s Encrypt. It already helps users to automate enrolling our application with publicly valid certificates for HTTPS while keeping it up to date.

Today, cert-manager is the almost perfect solution in Kubernetes for dealing with any kind of work with certificates. It is even possible to create your own simple private PKI without the need to deal with any CLI tools for automation.

I’d also recommend you explore more advanced features and use cases, like securing your pod-to-pod communication by leveraging the CSI driver for mTLS or the CSI SPIFFE driver.

Did you know that Traefik Proxy 3.0 Beta 1 added native support for SPIFFE? Check out the latest beta version of Traefik Proxy, play around with the new features and capabilities, and don’t forget to share your feedback!