Running Jenkins Nodes on Kubernetes

15 Nov 2016

A Component is Born

In this post, I'm going to demonstrate a new component called jenkins-node. jenkins-node is a new project that brings up a Jenkins slave on Kubernetes. It uses Helm to install into the cluster, connects to the Jenkins master and runs build executions in its workspace.

The Current Infrastructure

Before this project was conceived, we have several nodes running across the system:

  • nodes 1-7 are running on EC2
  • node 5 is a small box in the Boulder office running macOS builds for CLIs

These nodes are provisioned using an Ansible playbook.

The main reason we want to shift over to Kubernetes was for 3 reasons:

  1. it's expensive to run 6 nodes on EC2, mostly on standby
  2. we want to run more of our infrastructure on Kubernetes to battle-test it
  3. rolling out updates via ansible is a little harder than swapping the image release in Kubernetes

We thought it'd be a cool idea to deploy a few instances on Kubernetes and see how it goes.

How jenkins-node Works

Jenkins-node is simply a dockerized version of the ansible playbook. The component comprises of a Docker image used to run a worker node for CI jobs on https://ci.deis.io. This makes it easier for others to externally audit and update the dependencies on our jenkins build executors, while also allowing us to easily roll updates out to the CI server.

In order to connect to the master, /bin/start-node fetches the slave jar and connects using the given name and secret from the master. This part is done manually for now, however once things have settled and we feel more confident in this new component, we can consider using the Kubernetes plugin to provision nodes for us on-the-fly, greatly reducing cost for our build executors.

Installing jenkins-node on a Kubernetes cluster is dead simple. Create a new node on Jenkins that can connect via the JNLP protocol, modify the chart's values file to include the node name and secret, then run helm install charts/jenkins-node --namespace=jenkins to deploy the chart.

Some Caveats

There are a few caveats to the current approach which we eventually need to resolve.

For one, our end-to-end tests use a component called e2e-runner to run the job. This job is dockerized, but how it works is by mounting in the Docker host's workspace so it can write logs to the Jenkins slave's filesystem. This simply won't do in a Dockerized world, since we'd like to run multiple slaves on the same host. The fix here is to implement a new way to ship logs from e2e-runner to the slave, or refactor e2e-runner entirely to run inside the slave.

The second caveat would be that the helm chart is not scalable past 1 replica. This is because all instances share the same node name and node secret. For this reason, if we want to run multiple build executors on the same cluster, we need to manually create jenkins-node-n charts. This can be resolved by moving towards the Kubernetes plugin for Jenkins.

Any Questions?

If you have any more questions about jenkins-node, please feel free to raise an issue or feel free to ask on the #community channel on Slack.

Posted in Jenkins, Deis Workflow, Helm

triangle square circle

Did you enjoy this post?