Part III — The infrastructure stack with AWS CDK

Here we continue our journey for setting up a development environment for Kubernetes microservices, targeting AWS as runtime environment.

In part I we introduced the ideas and motivations behind this example while in part II we presented the sample Spring Boot App and its Skaffold configuration (there is also a maven archetype for getting your started). In this post we will present our target CI/CD infrastructure and how we implemented it.

Disclaimer

I’m not going to enter in too many details about how Kubernetes, EKS, CodeBuild and other AWS services works: I assume the reader is quite knowledgeable about the subject. Bear also in mind that what I present here is just my use case which may (or may not!) be the starting point for a more complex delivery pipeline. And, of course, there may exist better ways to achieve the same results but, hey, we are all learning by sharing :)

Finally, when creating the required AWS resources you will incur in costs that I’m not responsible for, like the EKS control plane and EC2 instances.

The Infrastructure

The infrastructure required to support our use case is composed by the following parts:

1. the runtime environment (the Kubernetes cluster);
2. Git source and Docker image repositories for one microservice;
3. CI/CD pipeline that will take the source code, build it and then push the resulting image to the image repository so that the cluster will be able to pull it and update the environment.

For point 1 we are going to user AWS EKS, a managed Kubernetes service; for point 2 we are going to use AWS CodeCommit and Elastic Container Registry (ECR) respectively; for point 3 we are going to use AWS CodePipeline and CodeBuil and leverage a plethora of tools.

The sample service is going to be a Spring-based app, so 3rd parties (maven central repositories) will be involved when fetching dependencies (like the Spring framework itself), just like base images from the default Docker registry.

The sample infrastructure

The entire stack will be defined with AWS Cloud Development Toolkit (CDK), a tool that allows to define our infrastructure using Object-Oriented (OO) TypeScript code without having to deal with the YAML/JSON files that you usually use when doing infrastructure-as-code with AWS CloudFormation.

AWS CDK allows developers to define their infrastructure using existing abstractions and building new ones (what AWS calls the “construct programming model”) to give shape to their solutions. You can find the rationale and a lot more details in this keynote by Werner Vogels (Amazon.com CTO).

Ok, let me explain: I’ve a personal bias for highly verbose YAML and/or JSON configurations and, given my Java background, working with OO abstractions is more than welcome. Of course, this is not a replacement for knowing the AWS resources intimately but IMHO allows for code to be much more readable by humans. But if you don’t trust me, then you can’t go wrong with Vogels’ words! ;)

Enter the CDK

There three main constructs that compose the infrastructure are the Cluster Stack, The Internal Artifacts Stack, and the CI/CD Pipeline. Each of these stack will be deployed separately but there are some dependencies.

For example, CI/CD Stack depends on both Internal Artifacts and Cluster stacks since it must build from a Git Repository, push the images to the image repository, and the update the Cluster configuration accordingly.

For detailed build steps, consult the companion GitHub repository.

The Cluster Stack is a essentially a wrapper around the EKS Cluster stack but it also provides the Kubernetes namespace resource so that any pod will find its right place. The node group, defining the EC2 instances that will run our containers, is configured with defaults values (2x T3.medium instances) but it can easily re-configured without impacting the other stacks according to different needs.

Additional security configuration (e.g., security grops or additional VPC constraints for the cluster could also added here in the future) and also shared Kubernetes resources (e.g., additional namespaces, a service mesh or additional config maps).

The Internal Stack for our Hello World service keeps together the source code and the images for each microservice. Infact, there will be

• one Git repository (using AWS CodeCommit);
• one Docker image repository (using AWS ECR).

There is nothing fancy about this but I’ve decided that the CodeCommit repository will be preserved If I delete the stack (we don’t want to lose our source code, do we?). For the ECR repository I’ve opted for deleting it when the stack is detroyed: binaries can be re-built, source code … not that easy!

Of course, you can still delete everything from the Web Console.

The CI/CD Pipeline will deal with the build and deployment process, listening for code changes on on master branch and then updating the Kubernetes deployment for the Hello World microservice.

Wrapping Skaffold with CodePipeline

The CI/CD pipelines leverages

• Skaffold for orchestrating the build and resource deployment process;
• Jib for building the Docker image using Apache Maven;

The CodePipeline scheme has just two stages:

1. Code checkout (the Git repository is cloned);
2. Build & Deploy (the image is built and deployed.

Note that build and deploy are performed at the same time because I’m leveraging Skaffold and I find no need to separate the steps.

Everything is done by wrapping the right commands into a CodeBuild’s buildspec. I use the following build phases:

• install — the required binaries will be downloaded and used (e.g., the skaffold and kubectl executable from their public repositories);
• pre-build — Docker login and AWS assume role is executed so that CodeBuild service can have the right permission to deal with our resources.
• build — the actual skaffold build is performend assembling the correct command line to launch

Note: for additional security, you may want not to download additional software (e.g., executable from the Internet but, for example, put them into a S3 bucket and then use these. It’s out of scope for my case but I understand that this may be important for you! Additionally you may benefit from less bandwidth usage.

Source code

The source code for this infrastructure, like the hello-world microservice, is available on GitHub, where you can also find there more details about the implementation.

Follow the instructions in README.md if you want to the test the entire process: create your infrastructre, push your code and have the microservice being deployed at any change on Git repository’s master branch!

Conclusions and next steps

This concludes, for now, our journey with CI/CD in Kubernetes. There are several aspects that have been left out but may be interesting for future entries in this series. Just to name a few:

• CloudWatch integration (Container insights) — you want to monitor your performance and container logs, don’t you?
• Using a service mesh for implementing Ingress and routing (e.g., Istio or AWS AppMesh);
• Using cdk8s for managing Kubernetes resources instead of YAML files (but this may require to revisit the way we use Skaffold);
• Resource tagging, because if you want to administer your resources in AWS, having them well organized is a good practice!

In the meanwhile, see yah!