Running Jenkins Nodes on Kubernetes
A Component is Born
In this post, I'm going to demonstrate a new component called jenkins-node. jenkins-node is a new project that brings up a Jenkins slave on Kubernetes. It uses Helm to install into the cluster, connects to the Jenkins master and runs build executions in its workspace.
The Current Infrastructure
Before this project was conceived, we have several nodes running across the system:
- nodes 1-7 are running on EC2
- node 5 is a small box in the Boulder office running macOS builds for CLIs
These nodes are provisioned using an Ansible playbook.
The main reason we want to shift over to Kubernetes was for 3 reasons:
- it's expensive to run 6 nodes on EC2, mostly on standby
- we want to run more of our infrastructure on Kubernetes to battle-test it
- rolling out updates via ansible is a little harder than swapping the image release in Kubernetes
We thought it'd be a cool idea to deploy a few instances on Kubernetes and see how it goes.
How jenkins-node Works
Jenkins-node is simply a dockerized version of the ansible playbook. The component comprises of a Docker image used to run a worker node for CI jobs on https://ci.deis.io. This makes it easier for others to externally audit and update the dependencies on our jenkins build executors, while also allowing us to easily roll updates out to the CI server.
In order to connect to the master, /bin/start-node fetches the slave jar and connects using the given name and secret from the master. This part is done manually for now, however once things have settled and we feel more confident in this new component, we can consider using the Kubernetes plugin to provision nodes for us on-the-fly, greatly reducing cost for our build executors.
Installing jenkins-node on a Kubernetes cluster is dead simple. Create a new node on Jenkins that
can connect via the JNLP protocol, modify the chart's values file to include the node name and
secret, then run
helm install charts/jenkins-node --namespace=jenkins to deploy the chart.
There are a few caveats to the current approach which we eventually need to resolve.
For one, our end-to-end tests use a component called
e2e-runner to run the job.
This job is dockerized, but how it works is by mounting in the Docker host's workspace so it can
write logs to the Jenkins slave's filesystem. This simply won't do in a Dockerized world, since
we'd like to run multiple slaves on the same host. The fix here is to implement a new way to ship
logs from e2e-runner to the slave, or refactor e2e-runner entirely to run inside the slave.
The second caveat would be that the helm chart is not scalable past 1 replica. This is because all
instances share the same node name and node secret. For this reason, if we want to run multiple
build executors on the same cluster, we need to manually create
jenkins-node-n charts. This can
be resolved by moving towards the Kubernetes plugin for Jenkins.