Monday, May 15, 2017

Kubernetes Development Environment - Minimizing Configuration Drift, Maximizing Developer Speed

One of the most appealing aspects of  Container as a Service offerings like Kubernetes is that operational capability is mostly built in.  At a high level, applications and services are deployed to clusters with manifest files that contain configuration, required containers, and needed resources while the actual mechanics of the deployment are abstracted away from the end user. 

While PaaS offerings like Cloud Foundry offer the same level of operational automation, I believe Kubernetes has gained more traction than Cloud Foundry because it is possible to  move the stateful parts of an application into Kubernetes and run them as-is by mounting their containers to persistent volumes. I think that persistent disks are a work in progress in Cloud Foundry, but standard practice for persistence is to expose and bind to a service that provides persistence (e.g. database, file store)  via a CF cluster service broker.

Once a legacy app is containerized and ported into Kubernetes, it is possible to rapidly refactor from a monolith into purpose built microservices. Normally, without some kind of container orchestration, deployment complexity would increase dramatically. Kubernetes automates that deployment complexity away.

This is very exciting, especially to teams that are saddled with legacy codebases, faced with faster moving competitors, needing to deliver key features to capture or maintain market share. These teams know what they need to deliver, they just can’t do it fast enough. Velocity is key. Any technology that allows teams to ultimately deliver features faster allows them to then iterate on those features to gain and extend market fit. 

However, while porting a legacy codebase into Kubernetes is relatively straightforward, it is not possible to ‘lift and shift’ the engineering team skillset. So after the port, on day 1, engineering teams are still faced with a huge step function that prevents them from moving as quickly as they could. 

Making a long term successful migration to Kubernetes means ensuring that the engineering teams that own the applications and services that have been migrated can fulfill the promise of the system - by quickly refactoring and extending the legacy software from it's legacy state to a decomposed, flexible architecture that enables fast feature ideation.

To be able to make that leap,  developers who have been focused on a legacy codebase have to  ramp up on several potentially new concepts:  
  1. Microservice architectural patterns - so they can  start to refactor their monolithic legacy applications. 
  2. The Docker toolchain, so they can own the creation of containers for their applications and services.
  3. Key  concepts of Kubernetes, e.g. pods, deployments, services, labels, liveness checks,  etc, so they can understand how their application is deployed, patched, and scaled. 
  4. Operational aspects of Kubernetes, like logging,  attaching to containers, checking pod and service state, so they can debug and fix application issues once apps have been deployed. 
  5. Containerization in general - how containers are the unit of deployment and how that can be leveraged for tasks, tests, and ensuring that application configuration remains the same from their laptop to production.
What I don't address in that list: the very significant additional operational complexity of managing a Kubernetes cluster. Teams that need to run on premise - for instance teams that work for Financial Services companies - would need to come up to speed on that operational complexity as well.

Even in a managed Kubernetes instance, the complexity of multi service applications can pose problems to engineers used to coding to and debugging against a monolithic codebase.

One of the advantages of writing to a monolithic codebase is that the development environment allows for rapid iteration. Making changes to a monolithic codebase and then starting up the server on your local machine to validate those changes makes for a seamless, fast loop.  However, most local development environments don't resemble production environments at all, which leads to production problems that have configuration mis matches as a root cause.

The ideal development environment would allow for fast development, in an environment that parallels production as closely as possible, which enables rapid validation.




When an application or service is composed of several sub services that are in turn composed of multiple containers, setting up a good development environment gets much harder and that fast loop slows down, reducing the gains from container orchestration.  Prior to minikube, development was either done with a bunch of jury rigged containers (which meant massive configuration drift from the production environment) or with a remote kubernetes cluster, which preserved configuration at the cost of development speed because validation required deployment to that remote cluster.


Minikube runs Kubernetes on a single VM node, which removed the latency of deployment, but requires deployment to validate. This is a faster alternative to a remote cluster, but deployment to the cluster was still required:



This still didn't give me the speed I wanted. As a developer:
  1. I would rather spend more time in a 'local environment', e,g,  developing against live services while running a local version the service I'm writing. I would rather validate without having to deploy every single time. 
  2. I would always be developing a process that made calls to other services. I didnt want to have to 'fake' those services or drift my dev environment configuration from production. 
To get this behavior, I took advantage of the environment variables that Kubernetes creates per deployed application. If I was working on  microservice 1 that made calls to microservice 2, I would run microservice 2 in minikube, and export the environment variables that stored microservice 2's IP and port. Then I would set those variables to point to my Minikube instance for the IP, and the port that Minikube exposed microservice 2 on.


That setup let me run microservice 1 in my existing development environment and move through the code-debug-test loop as quickly as I used to with a monolith application.

I've posted the code on github. The readme shows how I set up for inter service development. As noted above, I ensured I was pointing at the docker runtime on the Minikube VM, I patched deployments instead of re-deploying, and (just like the original article above) I used makefiles to automate the commands I kept running:
  1. Building the Docker Image and deploying to Minikube
  2. enabling my service to run locally by setting environment variables
  3. pushing the image to Minikube by patching the deployment
  4. generating a service url once I've deployed to Kubernetes so that I can validate the deployed app.
I like this pattern, it allowed me to develop as fast as I normally do, while essentially keeping my development environment aligned to the production environment.



No comments:

Post a Comment