5 tips for dealing with Day 2 Kubernetes operational challenges

Kubernetes is a wonderful but complex software that can present significant “Day Two” challenges when put into production.

Developers who are new to Kubernetes — and most are — face a large knowledge gap when they look to sustain and optimize Kubernetes clusters.

In this piece, I will share several ways to address problems as they arise.

Optimize your Kubernetes cluster for cost

As adoption of Kubernetes rises, the need for applications and engineers to access clusters is also growing. However, it is neither feasible nor cost-efficient to always use entire physical clusters to achieve this goal.

Virtual clusters are a great way to reduce costs. In a scenario of 100 developers, we calculated up to 78% savings by using open source virtual clusters.

Leveraging virtual clusters with open source software such as VirtualCluster or vcluster lets Kubernetes operators can run multiple virtual clusters within a single physical cluster, thereby increasing the tenancy of each. By utilizing computing resources via this more communal method, organizations can save on computing costs as opposed to operating entirely separate Kubernetes clusters.

Increase tenant isolation

Another great benefit of virtual clusters is that they are isolated from other users on the cluster. This gives each user their own workspace that looks and feels exactly like a physical Kubernetes cluster.

In addition, virtual clusters enable a stricter form of multitenancy compared to namespace-based multitenancy. One of the main concerns with namespace-based multitenancy is that it cannot contain cluster-scoped resources. Many applications must create, or at least access, cluster-scoped resources like nodes, cluster roles, persistent volumes and storage classes.

Virtual clusters also provide security benefits by increasing the isolation in multitenancy clusters via:

• Full control-plane isolation.
• Domain Name System (DNS) isolation.
• Resources created on a single namespace.

Organizations seeking a solution for multitenant applications that provide greater isolation for resources shared among their clusters should consider virtual clusters as an option. On top of saving costs and being simpler to deploy, they are also easier to manage than physical clusters.

Provide integrated development environments

When developing applications on Kubernetes, developers have to manually build and deploy applications via tools such as Docker Compose. This gives rise to productivity lag and hampers the overall experience that new developers face when getting started with Kubernetes.

One option to address this challenge is to leverage integrated development environments such as DevSpace, Skaffold or Tilt. With integrated development environments, engineers can use features such as:

• Hot reload running containers without rebuilding images or restarting them.
• Instant updates to the production environment.
• Easy to use dashboards in the browser.

These tools allow teams to define their development workflows as code, which allows for transparency, aids in troubleshooting problems and can help speed up developer onboarding. Development environments should be disposable and easy to reproduce, not handcrafted artistry.

Implement software security guardrails

According to the 2022 State of Cloud Security report, a major limiting factor with moving to the cloud is inefficient processes. Too often, cloud engineers are tasked with dealing with the fallout of hastily planned security processes. Misconfiguration incidents result in security issues that often require significant work from both security and operations teams. No wonder cybersecurity firm Cyble found over 900,000 Kubernetes clusters exposed to the Internet.

By leveraging policy engines, it’s possible to implement software security guardrails on your cloud-native Kubernetes infrastructure so that these processes are not only codified in your pipeline but in your processes as well. Some popular policy engines, such as Open Policy Agent (OPA), jsPolicy and Kyverno can be used to enforce Kubernetes rules.

A key part of using these tools is admission control. Platform engineers and security teams can define policies and then either reject or modify workloads that don’t conform automatically. Common uses include blocking the creation of privileged pods, which are a huge security risk, and ensuring that pods contain labels.

Know what’s in your containers

There has been a lot of attention on software supply chain security since the SolarWinds compromise was revealed in 2020. Knowing what is actually running in your containers is critical and can be even more of a challenge with open source software, where scrutiny of community contributions can vary greatly. Understanding what’s in your dependencies can be difficult, and if your developers are building containers that other teams or customers consume, they’ll want to ensure that it’s clear who built the containers and what is in them.

In the past, many teams relied on security scanners like Trivy to detect Common Vulnerabilities and Exposures (CVEs) in their containers. Scanning is an essential practice, but it has limitations, a big one being different security scanning tools can give you very different results.

That’s why I recommend using Sigstore. It is an open source project that significantly reduces the effort needed to sign, verify and monitor what you build. Sigstore’s components include Cosign, an app for signing and verifying Open Container Initiative (OCI) images, and Rekor, a tamper-resistant transparency log that can be queried for information about images before you use them. Sigstore has become very popular in the open source world, and the Kubernetes project now uses Cosign to sign its artifacts.

Wolfi is another new tool that helps teams build software images. It’s a Linux “un-distro” (it doesn’t contain a Linux kernel) that is free and can be used to build minimal containers with footprints much smaller than typical Docker images. In addition, Wolfi can generate Software Bills of Materials (SBOMs) for your builds, so users have clear knowledge of what is inside them.

Whether your developers are consuming open source dependencies or building artifacts used by your customers, these tools can help prevent security incidents that can sidetrack teams from working on new features for your products.

The future of Kubernetes operations

The best practices outlined here are a good start to resolving Day Two challenges. Kubernetes operations still have a long way to go before turnkey implementations can be enabled. As the ecosystem evolves, we will see more projects begin to establish new methods of deploying Kubernetes clusters more efficiently and adopt best practices.

We will see more implementations that give rise to a seamless developer experience and enable developers to do what they do best — create valuable products. The result will be accelerated adoption of Kubernetes and a richer cloud-native ecosystem.