Production Deis Workflow on Google Container Engine, Part Two
This is part two of a two part series that walks you through a full production setup of Deis Workflow.
In part one, we set up off-cluster object storage, a Docker registry, and a Postgres platform database. We then installed Workflow on a Kubernetes cluster.
In this post, I will show you how to secure your cluster with SSL and get DNS set up for your Workflow domain. Finally, I will show you how to upgrade Workflow itself.
SSL for Workflow
Setting up SSL for Workflow ensures that all data passed between the Workflow controller and the CLI remains private.
To enable SSL for Workflow, you can add an SSL certificate to the Workflow router. You must provide either an SSL certificate that was registered with a third-party Certificate Authority (CA) or your own self-signed SSL certificate.
Note: The platform SSL certificate also functions as a default certificate for apps that are deployed via Workflow. If you would like to attach a specific certificate to an application and domain see the docs covering application SSL certificates.
Installing SSL on the Workflow Router
For this post, we are going to generate a self-signed SSL certificate.
In the following command, replace
example.com with the appropriate values:
After you run this command, you should two files in your working directory:
Now, create the Kubernetes
Once you've created the
deis-router-platform-cert secret, the Workflow router will notice and automatically update its own configuration:
Cloudflare SSL and DNS
Having a self-signed SSL certificate is great for securing Workflow, but let's improve things by putting Cloudflare in front of it. Cloudflare offers a number of security features, but the one we're most interested in here is an SSL certificate from a known CA. This means no pesky security warnings for your end users.
In this setup, traffic between Workflow and Cloudflare is secured by your self-signed certificate, and forward traffic to your end users is secured by Cloudflare.
Firstly, create a free Cloudflare account.
Once that's done, select Add Site from the top bar:
Fill your domain name and click Begin Scan:
Then, follow the instructions and update your domain DNS records.
Once your DNS is set up you must enable SSL support.
Select Crypto from the top bar and change Off to Full under the SSL section:
Setting SSL to Full allows you to use your self-signed SSL certificate. Once this is enabled, you can see that traffic is encrypted at every step:
Next visit Page Rules and select Create Page Rule:
Add your domain, creating a page rule to forward all traffic to HTTPS:
Select Save and Deploy and you are done for now.
Now you need to add your
deis-router IP address to your DNS records on Cloudflare. This will allow you to then access Workflow components by subdomains.
deis-router IP address:
The IP address you're looking for is listed under the
EXTERNAL-IP column. Here you can see it is
Go back to Cloudflare and select DNS from the top bar:
On the screen that comes up, add three records:
The records are:
deis-builder: used as remote Git endpoint
deis: the platform API for CLI use
grafana: to access Grafana remotely
Note: when doing this, you should change the default Grafana username and password for Grafana. See the docs for more info.
You can add other subdomains too if you want, e.g.
www. They do not even have to point to Workflow hosted apps.
Enable the HTTPS proxy (toggle the orange cloud symbol) for
grafana subdomains. This will force SSL. You do not need to do this for
deis-builder, as this service is only accessed via SSH.
If you are setting up subdomains for Workflow apps, you should configure forced HTTPS for them too. If you don't do this, end users will see an untrusted SSL certificate warning when trying to visit your app.
Once you've set up your subdomains, give it a few minutes for all the Cloudflare services to kick in before you proceed to the next section.
Set Up Workflow
Register With the Controller
To use Workflow, you must first register a user with the platform controller.
deis register with the controller URL to create a new admin user account.
Do that like so:
If the registration is successful you should see:
Next you should secure Workflow by only allowing admin users to create new users. If you don't do this, the system will be open and anybody will be able to create users.
You need to upload your SSH public key so you can use
git push to deploy applications to Workflow.
If you do not have an SSH key you must generate one. This includes Google Cloud Shell users, as an SSH key does not come pre-generated.
You can generate an SSH key like so:
Press Enter to accept the default location and filename. You can then press Enter two more times if you want to use an empty passphrase. It's more secure to use a passphrase, but this decision is entirely up to you.
Once you're done, change the file permissions on your new SSH public key:
Now, upload your SSH public key to Workflow:
Now that's done, you're ready to start deploying some apps!
I won't go over how to deploy apps to Workflow, because I've covered that before on the Deis blog. Check out this quick overview covering the different ways to deploy to Workflow, with some example videos. Or check out this more detailed three-part guide to both Workflow and Workflow app deployment.
Let's take a look at how to upgrade Workflow.
Deis Workflow releases may be upgraded in-place with a minimal downtime.
The upgrade process requires:
- Helm Classic, version 0.8.1 or newer
- Configured off-cluster storage and platform database (I showed you how to do this in part one)
- A copy of the Kubernetes builder secrets from your running cluster (which I will show you how to get)
First, export environment variables for the previous and latest Workflow versions. This will help reduce confusion later on.
Modify the above to suit your setup.
Now, fetch a copy of the new Workflow chart, like so:
This puts the chart into the
helmc workspace for further customization.
The first time Workflow is installed, Helm automatically generates Kubernetes secrets for the builder and database Workflow components. You need to rescue these from your current chart and re-apply them to your new chart.
Fetch the current database credentials:
Fetch the builder SSH keys:
You now have the important Kubernetes secrets saved on your local workstation. You will copy them into place in the next step.
Modify and Update Configuration
Before generating the manifests for the newest Workflow install, you should update the new
generate_params.toml file (which I cover in part one) to match the configuration used with the previous Workflow install. Specifically pay attention to the registry, object storage, and platform database sections.
Edit your previous
generate_params.toml file here:
Edit your new
generate_params.toml file here:
You should not just copy the old
generate_params.toml file across, because the new Workflow release may have added, removed, or changed some things.
Once you're done, generate templates for the new release like so:
Now you can copy your Kubernetes secrets into place.
Copy your database credentials like so:
Copy your builder SSH keys like so:
Everything is now set for you to apply the Workflow upgrade.
Apply the Workflow Upgrade
Helm Classic will remove all components from the previous release that are not marked as keepers. As of Workflow 2.3 and later, the controller, registry, and router are marked as keepers and will be left in-service.
Traffic to applications deployed through Workflow will continue to flow between the
If Workflow is not configured to use off-cluster Postgres, the Workflow API will experience a brief period of downtime while the database component recovers from backup.
Uninstall Workflow like so:
Install of latest Workflow release like so:
Wait a few minutes and then verify that all components have started and have passed their readiness checks:
Awesome job. Workflow has been upgraded to the latest version.
In this post we learnt how to:
- Secure Workflow with SSL and DNS
- Register an admin user with the platform
- Perform a platform upgrade
If all of this seems a bit involved, do not fret. Now you've done it the hard way, check out my deis-workflow-gke script which automates most of this for you.