Workflow Helm charts - Signed, Sealed, Delivered, They're Yours

11 Nov 2016

Workflow & Co., Now in Delicious Kubernetes Helm Flavor

You may have noticed Kubernetes Helm charts for Workflow and associated components being released, starting with Workflow v2.8.0. Though officially still experimental, we encourage users to try them out!

(As Helm nears its final 2.0.0 release, the Workflow team is currently putting the finishing touches on a migration strategy for upgrading clusters from Helm Classic to Helm. Stay tuned!)

Sign 'em if You Got 'em

So, we've authored these shiny, new Helm charts and will soon encourage users to transition to them for running Workflow -- but how do we guarantee the chart intended for installation on a Kubernetes cluster is the exact chart our CI system has stamped with approval?

Conveniently, Helm provides tools for establishing and verifying chart integrity. (For an in-depth overview, see the informative Provenance doc.) All release charts from the Deis Workflow team are now signed using this mechanism.

In practice, this comes down to one concise command:

helm package --sign --key 'Deis, Inc. (Helm chart signing key)' workflow

That was easy! Now, let's take a closer look our signing infrastructure...

CI Told You We Could Automate This

One way we could have approached release chart signing would be to have a designated release engineer/project maintainer sign with their own personal PGP key.

However, as we release updated charts for Workflow and all of its components on a regular basis, we were intent on avoiding this manual effort. We also wanted the key to represent an official Deis chart signing key, not tied to any individual engineer.

To that end, we created a dedicated Jenkins node to act as Deis' sole, designated chart signatory. In addition to restricted access as compared to our existing Jenkins nodes, this box is the only holder of the full signing keypair.

The current CI flow for delivering a signed Workflow chart, then, is as follows:

  1. The deis/workflow repo is tagged for release (say, v2.8.0)

  2. The tag event triggers a Jenkins job to package and push a candidate chart to staging, based on the checked-out v2.8.0 release tag

  3. The candidate chart is then vetted against the latest suite of E2E tests

  4. Assuming the E2E tests pass and additional manual testing concurs, a job is triggered to copy the candidate chart from staging to production

  5. The final job finds the signatory node tasked with fetching the production-hosted chart and signing it, uploading the resulting workflow-v2.8.0.tgz and workflow-v2.8.0.tgz.prov files to production

Our Chart Signing Key

pub   4096R/1D6A97D0 2016-11-03                    
Key fingerprint = 41AF 6B6A 9489 9B58 1EB6  9ED1 17E5 26B5 1D6A 97D0  
uid       [ultimate] Deis, Inc. (Helm chart signing key) <security@deis.com>  

The full public key can be found here, as well as the pgp.mit.edu keyserver and the official Deis Keybase account.

The key's fingerprint can be cross-checked against all of these sources before it is imported below.

Verifying a Signed Chart

Right, right -- but how does one actually verify our signed charts?

First, the public key must exist in a local keyring before said chart can be verified. (For the purposes of this demonstration, we will assume GPG is the PGP-compliant encryption software being used.)

To add the public key to the default ${HOME}/.gnupg/pubring.gpg keyring, any of the following commands will work:

$ # via our hosted location
$ curl https://deis.com/workflow/docs/security/1d6a97d0.txt | gpg --import

$ # via the pgp.mit.edu keyserver
$ gpg --keyserver pgp.mit.edu --recv-keys 1D6A97D0

$ # via Keybase with account...
$ keybase follow deis
$ keybase pgp pull

$ # via Keybase by curl
$ curl https://keybase.io/deis/key.asc | gpg --import

Signed Workflow charts can then be verified when fetched:

$ helm repo add deis https://charts.deis.com/workflow
"deis" has been added to your repositories

$ helm fetch --verify deis/workflow && echo
Verification: &{0xc820563e50 sha256:060d66fa95b6badad98b37572a887723ed49a153dd636dce0f2c4ff667022586 workflow-v2.8.0.tgz}

One can then inspect the fetched workflow-v2.8.0.tgz.prov provenance file.

If the chart was not signed, the command above would result in:

Error: Failed to fetch provenance "https://charts.deis.com/workflow/workflow-v2.8.0.tgz.prov"

Alternatively, the chart can also be verified at install time:

$ helm install --verify deis/workflow --namespace deis
Fetched deis/workflow to workflow-v2.8.0.tgz
NAME: olfactory-star
LAST DEPLOYED: Thu Nov 10 11:45:44 2016
NAMESPACE: deis
STATUS: DEPLOYED
...

Either way, one is assured of the origin and authenticity of any Workflow chart intended for installation on a Kubernetes cluster.

Posted in Deis Workflow, Helm

triangle square circle

Did you enjoy this post?