Securing your Kubernetes environment against external traffic threats
Kubernetes delivers many quality-of-life improvements for software engineering teams out of the box by automating the deployment and lifecycle management of containerized applications. However, security is one area where developers must still be proactive, by identifying threat models and applying appropriate Kubernetes security best practices to address them.
Given that Kubernetes is often used to manage external-facing applications, the need for protecting applications from harmful external traffic is nearly universal. In this article we’ll review some of the general concerns that developers should keep in mind in this context, and then discuss relevant Kubernetes security approaches.
Top external traffic concerns
When it comes to security issues from external traffic, there are myriad concerns that range from unintentional but detrimental behavior to organized, malevolent attacks.
Unfair usage
Any application of even moderate complexity will exhibit significant variance in resource usage and execution time of requests. For example, certain operations might trigger backend batch jobs or invoke complicated queries to datastores. Accordingly, the overall load imposed by a specific external client will be a function of the request types it sends and their distribution over time (for example, bursty versus parallel requests). Because of this, it’s possible for a client to unintentionally use an unfair portion of provisioned resources and negatively impact other users.
Misconfigured clients
Particularly for API services, it’s common for incoming external traffic to have been generated by automated systems or client software whose behavior is driven by user-specific configuration settings. Settings that are misconfigured – whether maliciously or unintentionally – can lead to anomalous behavior, which can easily lead to undesirable load on the Kubernetes-managed application.
Malicious clients
An unfortunate reality for any internet-facing application is that it will inevitably face nefarious attempts to identify and/or exploit potential security vulnerabilities in the software and gain some form of unauthorized access. Such attacks include zero-day exploits and brute-force attempts to penetrate the system, among others. If successful, these attacks can have significant negative consequences beyond application downtime, including data breaches.
Denial-of-service (DDoS) and distributed denial-of-service (DDoS)
A malicious attacker may also intentionally try to limit or disrupt the resource availability for legitimate traffic, in what is known as a denial-of-service (DoS) attack. Such attacks often combine several of the previously mentioned elements. In their more sophisticated form, known as distributed denial-of-service (DDoS), they can also be orchestrated such that a distributed and possible dynamic set of clients works together to send malformed or excessive traffic simultaneously.
Mitigation strategies
Having reviewed the top concerns that harmful external traffic creates for applications, let’s look at some approaches to mitigating them. Specifically, by employing an Ingress controller for Kubernetes, multiple options become available to consider as part of a comprehensive security strategy.
IP allow/deny lists
One approach to integrating security best practices is to employ network-level mechanisms. Specifically, the IP addresses of external traffic can be used to make binary decisions on whether the traffic should be allowed at all. If applications have a stable set of known clients, a proactive allow list approach in which unknown IP addresses are ignored may be applicable. On the other hand, if valid traffic could come from anywhere, a reactive deny list can be maintained based upon the identification of malicious traffic sources.
Connection limits
Another network-level option is to limit the number of connections that an external client can create, which can help limit the impact of misconfigured clients as well as malicious users. One potential challenge, however, lies in determining an appropriate value to set for the connection limit. Many implementations require a global setting that impacts all sources of traffic. Particularly when imposing connection limits by source IP address, scenarios in which multiple clients may share an IP address – such as NAT, for example – should be considered.
Rate limiting
Moving up the networking stack, an Ingress controller can be used to enable rate limiting at the request level. These limits gate the number of requests in a given time window (for example, requests per second) and can be enforced in a variety of ways. Common options include setting rate limits based upon client IP address or a value in the HTTP header. Rate limits can also be configured so a maximum request rate is enforced at service hosts.
Limiting simultaneous requests
As mentioned earlier, request-processing times may vary based on a variety of factors. While rate limits can enforce the number of external requests submitted by a client in a given time frame, the number of concurrent requests being processed may vary significantly depending upon the relative processing time of each type of request. To address this dimension, external traffic can be managed by setting limits so that services don’t become overwhelmed by a large number of simultaneous, resource-intensive requests.
Audit logging
A question that often arises is how operations engineers can determine when the aforementioned mitigation mechanisms should be invoked, as well as their overall effectiveness. For example, it’s preferable to detect DDoS attacks in their earliest stages to minimize disruption to end users. How to spot them when they start? Access logs can be invaluable for this purpose. They are commonly incorporated as part of a broader security information and event management (SIEM) system. When employed with Kubernetes Ingress controllers like Traefik, these logs also make it possible to automate remediation strategies based upon detected external traffic patterns.
Stay vigilant
When considering how best to protect workloads from harmful external traffic, the most effective approach will likely require a combination of several of the mechanisms we’ve discussed. But which ones? The need for any one measure may not be immediately apparent, so its utility should be continuously reviewed with the help of data from audit logs. It’s important that applications teams consider the ability of their Kubernetes platform to provide a broad set of mechanisms that can be called upon as security requirements change and evolve.
Solutions like Traefik Enterprise align well with this mindset by providing a rich and robust set of security features for managing external traffic embedded in the Kubernetes Ingress controller. In our latest white paper, we cover the critical role that networking plays in modern application deployments with Kubernetes.
