Deis Workflow 2.3 Release

3 Aug 2016

Another two weeks, another Workflow release. The 2.3 release brings with it some internal release changes, improved private registry support, tools to call for help, and faster deploys. You might say Papa's got a brand new bag.

Workflow now a Release of Releases

A major design goal for Deis Workflow during the Kubernetes re-platform is producing decoupled components that can be used and deployed on their own. The edge router used by Workflow can be deployed on any Kubernetes cluster, providing HTTP routing, SSL termination, SNI support.

To support the universe of independently usable projects we are now tagging and releasing each component. This provides stable, semver compliant components for folks deploying these components independent from Workflow.

A Workflow release is now a collection of the underlying component releases. This greatly simplifies the amount of work required for a given Workflow release! The most observable change will be that component versions may not match the version of Workflow. Components under active development will see multiple releases, while stable components will only be released when we need to bump versions or fix bugs.

Insecure registry no longer required!

Until Workflow 2.3 we required Kubernetes clusters to be provisioned with --insecure-registry. Starting with 2.3, that is no longer the case! We have included a registry proxy, deployed as a Kubernetes DaemonSet which allocates a cluster NodePort and transparently proxies traffic to the on-cluster registry.

The proxy will run on port 5555 by default, which can be customized during install.

So happy pushing and pulling. Workflow will now function right out of the box on a lot more Kubernetes clusters without having to re-provision or deal with certificate distribution.

Native ECR and GCR Registry Support

The first stable release of Deis Workflow introduced support for private Docker registries.

Private registry support has worked very well for DockerHub and registries, but IaaS native solutions like Google Container Registry (GCR) and Amazon EC2 Container Registry (ECR) have different authentication requirements. While Kubernetes has built support for fetching these registry credentials when pulling images, Workflow also needs to push images when building your application's Dockerfile.

Deis Workflow 2.3 now has a new component that will transparently refresh registry credentials, creatively named deis/registry-token-refresher. The token refresher monitors your Kubernetes cluster for creation of new apps and will update the appropriate imagePullSecret.

There are a few new registry configuration options in generate_params.toml:

# Set the location of Workflow's Registry
# Valid values are:
# - on-cluster: Run registry within the Kubernetes cluster
# - off-cluster: Use registry outside the Kubernetes cluster (example: dockerhub,,self-hosted)
# - ecr: Use Amazon's ECR
# - gcr: Use Google's GCR
registry_location = "on-cluster"

If you are using registries like DockerHub, or run your own registry that uses the Docker registry API, set registry_location = "off-cluster" and add connection details to the off_cluster_registry section:

hostname = ""        # e.g.,, leave blank if you are using dockerhub
organization = ""    # e.g. use "myuser" for
username = ""
password = ""

When Workflow is configured for any of these registry options, Helm Classic will not deploy an on-cluster managed repository. All images will be pushed and pulled from the configured registry.

Using AWS EC2 Container Registry (ECR)

If you are using ECR registry-token-refresher can either use static API credentials, or credentials fetched via an InstanceRole.

For example, if your ECR Repository URI was your params file should look like:

registry_location = "ecr"

# Your AWS access key. Leave it empty if you want to use IAM credentials.
accesskey = ""
# Your AWS secret key. Leave it empty if you want to use IAM credentials.
secretkey = ""
# Any S3 region
region = "us-west-2"
registryid = "somename"
hostname = ""

Using Google Container Registry (GCR)

For GCR, provide the JSON key for a service account that has access to the appropriate Google Cloud project:

key_json = ''''''
hostname = ""     # can be left blank, defaults to

Process Types Deploy in Parallel

For applications that have multiple processes types, Workflow 2.3 will now deploy all types in parallel. Previously, Workflow would deploy each process type sequentially. This change will significantly speed up deploys for applications with many process types.

Somebody call a doctor!

Sometimes, things don't always go as planned. Starting with Workflow 2.3 we have included a new tool to help assist troubleshooting. We call this Workflow Doctor. Workflow Doctor can be invoked by cluster operators, by running doctor in the Workflow Manager pod. The doctor command gathers debug information about the deis namespace, Workflow versions, and cluster events.

$ kubectl --namespace=deis get pod -l app=deis-workflow-manager
NAME                          READY     STATUS    RESTARTS   AGE
deis-workflow-manager-10s0g   1/1       Running   0          13
$ kubectl --namespace=deis exec deis-workflow-manager-10s0g doctor
Information sent to deis doctor is available at the following url

The URL returned by doctor can be sent to maintainers to help get to the bottom of the problem.

Upgrading to Workflow 2.3

Workflow 2.2 to 2.3 is the last upgrade that requires a brief period of downtime. We are now using Deployments for Workflow's router, controller and registry components. This means that application traffic should not be interrupted during Workflow upgrades starting with the 2.3 to 2.4 upgrade.

Do note that the Workflow API will still experience brief downtime while the on-cluster Postgres instance replays from WAL files. However, if you are using an off-cluster database, Workflow API will keep on humming.

Check out the upgrade instructions and enjoy Workflow 2.3.

Looking Ahead

So far we are happy with how Kubernetes Deployment support is working. We are planning on making Deployments the default in the next release. If you haven't had time to try deployments. Please check out the documentation and report any bugs.

Posted in Workflow, Release, Announcement

triangle square circle

Did you enjoy this post?