Setting Up Your Own PaaS and CI/CD Pipeline
Docker and the ecosystem around it have done some great things for developers, but from an operational standpoint, it's mostly just the same old issues with a fresh coat of paint. Real change happens when we change our perspective from Infrastructure (as a Service) to Platform (as a Service), and when the ultimate deployment artifact is a running application instead of a virtual machine.
Even Kubernates still feels a lot like IaaS - just with containers instead of virtual machines. To be fair, there are already some platforms out there that shift the user experience towards the application (Cloud Foundry and Heroku come to mind), but many of them have a large operations burden, or are provided in a SaaS model only.
In the Docker ecosystem we are starting to see more of these types of platforms, the first of which was Dokku which started as a single machine Heroku replacement written in about 100 lines of Bash. Building on top of that work other, richer systems like Deis and Flynn have emerged, as well as custom solutions built in-house, like Yelp's PaaSta.
Actions speak louder than words, so I decided to document (and demonstrate) a platform built from the ground up (using Open Source projects) and then deploy an application to it via a Continuous Integration/Deployment (CI/CD) pipeline.
You could (and probably would) use a public cloud provider for some (or all) of this stack; however, I wanted to demonstrate that a system like this can be built and run internally, as not everybody is able to use the public cloud.
As I wrote this I discovered that while figuring out the right combination of tools to run was a fun process, the really interesting stuff was building the actual CI/CD pipeline to deploy and run the application itself. This means that while I'll briefly describe the underlying infrastructure, I will not be providing a detailed installation guide.
While an IaaS is not strictly necessary here (I could run Deis straight on bare metal), it makes sense to use something like OpenStack as it provides the ability to request services via API and use tooling like Terraform. I installed OpenStack across across a set of physical machines using Blue Box's Ursula.
Next the PaaS itself. I have familiarity with Deis already and I really like its (Heroku-esque) user experience. I deployed a three node Deis cluster on OpenStack using the Terraform instructions here.
Finally, there is a three-node Percona database cluster running on the CoreOS nodes, itself fronted by a load balancer, both of which use etcd for auto-discovery. Docker images are available for both the cluster and the load balancer.
The application I chose to demo is the Ghost blogging platform. I chose it because it's a fairly simple app with well-known backing service (MySQL). The source, including my
Dockerfile and customizations, can be found in the paulczar/ci-demo GitHub repository.
For development, I wanted to follow the GitHub Flow methodology as much as possible. My merge/deploy steps are a bit different, but the basic flow is the same. This allows me to use GitHub's notification system to trigger Jenkins jobs when Pull Requests are created or merged.
I used the Deis CLI to create two applications: ghost (e.g., http://ghost.ci-demo.paulcz.net) from the code in the
master branch, and stage-ghost (e.g., http://stage-ghost.ci-demo.paulcz.net) from the code in the
development branch. These are my
staging environments, respectively.
master branches are protected with GitHub settings that restrict changes from being pushed directly to the branch. Furthermore, any Pull Requests need to pass tests before they can be merged.
Deploying applications with Deis is quite easy and very similar to deploying applications to Heroku. As long as your git repo has a
Dockerfile (or supports being discovered by the cedar tooling), Deis will figure out what needs to be done to run your application.
Deploying an application with Deis is incredibly simple:
- First you use
deis createto create an application (on success the Deis CLI will add a remote git endpoint).
- Then you run
git push deis masterwhich pushes your code and triggers Deis to build and deploy your application.
After running the Jenkins Docker container I had to do a few things to prepare it:
docker exec -ti jenkins bashto enter the container and install the Deis CLI tool and run
deis loginwhich saves a session file so that I don't have to login on every job.
- Add the GitHub Pull Request Builder (GHPRB) plugin.
- Secure it with a password.
docker committo commit the state of the Jenkins container.
I also had to create the jobs to perform the actual work. The GHPRB plugin made this fairly simple and most of the actual jobs were variations of the same script:
Continuous Integration / Deployment
docker-compose is a great tool for quickly building development environments (combined with Docker Machine it can deploy locally, or to the cloud of your choice). I have placed a
docker-compose.yml file in the git repo to launch a
mysql container for the database, and a
I also included an
aliases file with some useful aliases for common tasks:
Running the development environment locally is as simple as cloning the repo and calling a few commands from the
aliases file. The following examples show how I added s3 support for storing images:
Docker Compose v1.5 allows variable substitution so I can pull AWS credentials from environment variables which means they don't need to be saved to git and each dev can use their own bucket etc. This is done by simply adding these lines to the docker-compose.yml file in the
I then added the appropriate environment variables to my shell and ran
up to spin up a local development environment of the application. Once it was running I was able to confirm that the plugin was working by uploading the following image to the s3 bucket via the Ghost image upload mechanism:
All new work is done in feature branches. Pull Requests are made to the
development branch of the git repo which Jenkins watches using the github pull request plugin (GHPR). The development process looks a little something like this:
Here I added the s3 module and edited the appropriate sections of the Ghost code. Following the GitHub flow I then created a Pull Request for this new feature.
Jenkins will be notified when a developer opens a new Pull Request against the development branch and will kick off tests. Jenkins will then create and deploy an ephemeral application in Deis named for the Pull Request ID (PR-11-ghost).
The ephemeral environment can be viewed at http://pr-xx-ghost.ci-demo.paulczar.net by anyone wishing to review the Pull Request. Subsequent updates to the PR will update the deployed application.
We can run some manual tests specific to the feature being developed (such as uploading photos) once the URL to the ephemeral application is live.
Jenkins will see that a Pull Request is merged into the development branch and will perform two jobs:
- Delete the
ephemeralenvironment for Pull Request as it is no longer needed.
- Create and deploy a new release of the contents of the
developmentbranch to the
stagingenvironment in Deis (http://stage-ghost.ci-demo.paulczar.net).
Originally when I started building this demo I had assumed that being able to perform actions on PR merges/closes would be simple, but I quickly discovered that none of the CI tools, that I could find, supported performing actions on PR close. Thankfully I was able to find a useful blog post that described how to set up a custom job with a webhook that could process the GitHub payload.
Promoting the build from
production is a two step process:
The user who wishes to promote it creates a pull request from the development branch to the master branch. Jenkins will see this and kick off some final tests.
Another user then has to merge that pull request which will fire off a Jenkins job to push the code to Deis which cuts a new release and deploys it to the
Coming from an operations background, I thought that figuring out how to build and run a PaaS from the metal up would be a really interesting learning exercise. It was! What I didn't expect to discover, however, was that actually running an application on that PaaS would be so compelling. Figuring out the development workflow and CI/CD pipeline was an eye-opener as well.
That said, the most interesting outcome of this exercise was increased empathy: the process of building and using this platform placed me directly in the shoes of the very developers I support. It further demonstrated that by changing the focus of the user experience to that person's core competency (the operator running the platform, and the developer using the platform) we allow the developer to "own" their application in production without them needing to worry about VMs, firewall rules, config management code, etc.
I also (re-)learned that while many of us default to cloud services such as AWS, Heroku, and Travis CI, there are solid alternatives that can be run in-house. I was also somewhat surprised at how powerful (and simple) Jenkins can be (even if it is still painful to automate).