The Workflow "Umbrella" Helm Chart
Last month's v2.9.0 release of Workflow included officially transitioning over to Kubernetes Helm. Here we look at how our CI system assembles the Workflow chart, which is itself an "umbrella" chart consisting of sub-charts for each of its core components.
What is an umbrella chart?
Just as applications of any complexity tend to consist of smaller, "building block" components, an umbrella chart is built up of multiple sub-charts brought together to form a larger whole.
The umbrella chart can then supply global configuration to these sub-charts by way of the top level
values.yaml file. The required version for each sub-chart is specified in the aptly-named
How CI assembles the Workflow chart
For Workflow, each component chart is defined and managed in its own GitHub repository. Packaging the Workflow chart gets even trickier when a different variant is desired depending on environment and particular CI pipeline. These environments come in the form of chart repos (AWS S3 buckets at time of writing), one each for pull requests, master builds, release candidates and official releases.
Component chart CI
As an example, let's look at the CI flow for pull requests against a component (sub-)chart (the flow is also very similar for merges to the master branch in each component, assuming the chart is affected):
The first job checks out the PR commit, sets the version to a patch-incremented, pre-release value such as
v1.2.4-<timestamp>-sha.<short_sha>(where the current official component release would be
v1.2.3) and then packages and uploads the chart to the component's
The next job packages the umbrella Workflow chart after setting said component's chart version to
v1.2.4-<timestamp>-sha.<short_sha>and chart repo to the
prvariant in the
requirements.yaml. (The rest of the component versions won't be specified; instead, chart repos for all will be the
helm dependency updatewill simply supply the latest chart in said repo.) The assembled chart is then uploaded to Workflow's
Lastly, a downstream E2E job is triggered, which will install the Workflow chart, using the exact version handed to it, onto a "leased" k8s cluster from a pool of ready-to-go GKE clusters we maintain. The Workflow cluster, assuming it stands up successfully, is then vetted against the current E2E test suite. The test results are archived and the job status is reported to the GitHub Pull Request.
Workflow chart CI
Another CI flow that may be informative is how we stage and release the Workflow chart:
The first job,
workflow-chart-stage, is kicked off with the intended chart version, which should match the semver release string that will be used to tag the repo. It will:
- Request the latest (git) release for all component repos (which align with their respective release chart versions)
- Set the versions appropriately in
requirements.yaml, make sure all chart repos are the production variants and run
helm dependency update
- Sign and package the workflow chart and upload to the production chart repo sans
index.yaml-- therefore, the signed release artifact is ready for official release if all testing gives the thumbs up; all that needs to be done is upload the updated
index.yamlto make it available
- After altering a few values in
values.yamlto point to their non-production counterparts, again package the workflow chart, this time uploading to the staging chart repo
E2E is run against the chart from the staging repo (and manual testing is executed)
Assuming the release candidate chart in staging is looking good, the last job is
workflow-chart-release, which simply does the aforementioned update and upload of the
index.yamlto the production chart repo, thereby allowing users to fetch and install the official release chart.
Lastly, a job is triggered to run
helm fetch --verifyon the workflow chart just released, to make sure there are no issues with verification of provenance.
Hopefully the examples presented above offer some ideas on how CI pipelines can support umbrella Helm chart packaging and releasing. All of the CI code mentioned above can be found in the deis/jenkins-jobs repo.
Have you also had to tackle similar with your Helm charts? If so, we'd love to hear about your approach. Holler at us on the #community channel on Slack.