⇠ Back to the blog
Cover for Cased Study: Working with `kubectl` in production

Cased Study: Working with `kubectl` in production

Christian Nguyen avatarWritten by Christian Nguyen

Overview

Kubernetes is an open source platform for managing containerized workloads and services. It's become ubiquitous and fundamental for production workflows as it allows you to easily scale, and deploy high-availability applications without taking them offline. kubectl is the Kubernetes CLI tool for communicating and running commands against a Kubernetes cluster control plane. kubectl is a very powerful tool for production. With this tool, you have the ability to create and delete resources right from your terminal. kubectl coupled with Cased's inline approvals, allows teams to increase engineering velocity in production, while staying safe and secure.

How to make kubectl production-ready

Setting up a Kubernetes production environment requires a lot of custom configuration that is different for every organization. However, there is one thing that they all need to consider: security and access management. You might think that this doesn't affect you; that your organization has all of your Kubernetes workflows automated. But software isn't perfect; your automation may break. And more times than not, you'll find yourself needing to access kubectl directly. You can't let everyone have complete kubectl access as that poses a huge risk for security and uptime. On the other hand, you don't want just one person to bear all the responsibilities for everything Kubernetes. You need a balance. You need role based access control, and approval workflows to make sure the right commands are being run at the right time. With Cased's groups and permissions - you can control who has access to specific prompts and containers. And with Cased's in-line approval workflows, organizations can whitelist or configure additional peer approval for anytime someone tries to run kubectl. All access and access requests will be logged in an audit trail for an additional layer of security and visibility.

Example workflow with Cased

Let's say you're a developer and you and your team just released a new container that needs to be updated with a new Docker image. Usually this task would be designated for the DevOps team, but for whatever reason, they're unavailable. Now, it's crucial this container doesn't go offline as users from around the world are using the application 24/7. The DevOps team unfortunately doesn't have a playbook that has all the commands you need to run, so you resort to Google and find the kubectl cheat sheet.

To update the container, you only need to run a couple of commands. You access the prompt you need to be in one-click via Cased's dashboard, and enter the reason you're accessing it. You get approved to access the host by a senior engineer.

Once you're in, you copy and paste a command from the kubectl cheat sheet and run it. Since you're not an admin, peer approval is required for this specific operation to run. A senior engineer on standby reviews the command, denies the operation, which exits you out of the host, and pings you over Slack. Turns out you copied and pasted the wrong command - a command that would've caused a service outage.

Just like that you could've taken down production by simply just copying and pasting the wrong code. You realize your mistake and eventually find the right command and run it.

The senior engineer carefully checks again, and approves the operation.

To make sure the change goes through you then run: kubectl rollout status deployments myapp.

This command doesn't require approval since the DevOps team already auto-approved this command in Cased. Eventually the status shows that it's complete and your DevOps work is done.

It's always important to have an extra set of eyes whenever working in production. With Cased's in-line approvals, organizations can work confidently in production.

If you're interested and want to learn more schedule a demo or visit our documentation.