16 Aug 2013 in Deis v1 PaaS

Command Line Bliss with Docopt

Starting to write the command-line client for the Deis open source PaaS, I was intimidated. I've written many command-line tools, most of them in Python, but the Deis CLI is more of a "swiss army knife" than I had attempted with argparse:

<code>$ deis register http://ec2-198-51-100-36.us-west-2.compute.amazonaws.com
$ deis create --flavor=ec2-eu-west-1
$ deis containers:scale web=4 worker=2
$ deis layers:create redis ec2-eu-west-1 --ssh_username=deisuser
$ deis help releases
Valid commands for releases:

releases:list        list a formation's release history
releases:info        print information about a specific release
releases:rollback    coming soon!

Use `deis help [command]` to learn more

Enter docopt

Docopt transforms conventional Usage text into a formal parser for command-line arguments. Simple as that. For many command- line tools, simply declaring a single Usage string and having docopt parse it can handle everything:

<code>"""Naval Fate.

  naval_fate.py ship new &lt;name&gt;...
  naval_fate.py ship &lt;name&gt; move &lt;x&gt; &lt;y&gt; [--speed=&lt;kn&gt;]
  naval_fate.py mine (set|remove) &lt;x&gt; &lt;y&gt; [--moored | --drifting]
  naval_fate.py (-h | --help)

  -h --help     Show this screen.
  --speed=&lt;kn&gt;  Speed in knots [default: 10].
  --moored      Moored (anchored) mine.
  --drifting    Drifting mine.

from docopt import docopt

if __name__ == '__main__':
    arguments = docopt(__doc__, version='Naval Fate 2.0')

This example Python program will now enforce the command line arguments given in the module's triple-quoted docstring. Arguments are returned as a simple dict, not a custom object. There is no further abstraction, just the declaration of intent.

Double down

Using docopt in this straightforward way covered nearly all our requirements for the Deis client, except for colon-separated "compound" commands that we wanted to phrase this way:

<code>$ deis flavors:delete ec2-eu-west-1
$ deis nodes:list

Here's the strategy @gabrtv came up with:

  • parse the command line using the module docstring
  • transform colons to underscores and expand shortcuts to full commands ("logout" is really "auth:logout," for example)
  • find a client method with the same name as the command
  • re-parse the command line using the method's docstring
  • call the method with the resulting arguments

Deis' approach here has several advantages:

  • Command syntax is the docstring for the relevant method, so there's no mismatch and no searching for definitions
  • Minimal coding required
  • With a bit of care, the same docstrings work well as readable documentation for a tool such as Sphinx

Docopt is a fresh approach to command-line parsing that immediately made us wonder why no one had thought of this before. We use Python, but there are implementations of docopt available for Ruby, JavaScript, and PHP.

Have a look at the single python file that implements the Deis client. I'm not sure how it could be more terse and self-documenting. You may find docopt a good fit for writing your next command-line tool.

6 Aug 2013 in Deis v1 PaaS

Deis 0.0.5 - Public Beta

Ready to deploy your own private PaaS based on Chef, Docker and Heroku Buildpacks? The Deis project is excited to announce our public beta!

What is Deis?

Deis is a Django/Celery API server, Python CLI and set of Chef cookbooks that combine to provide a Heroku-inspired application platform for public and private clouds. Your PaaS. Your Rules.

0.0.5 Summary

  • New http://deis.io website with integrated documentation
  • Sphinx docs including concepts, installation, usage, terms and full API reference
  • Bug fixes found during end-to-end testing
  • Reset of south migrations in preparation for public release
  • Published deis-cookbooks to Chef community site

What's Next?

Enhanced Docker Integration

Deis currently uses Docker as a LXC wrapper for running Buildpack slugs that are bind-mounted into default Buildstep images. We will be adding a new build process that creates and distributes images via Dockerfile builds and a private registry. We will also allow builds to reference existing Docker images by a fully qualified image path.

Log Aggregation

Deis will not be ready for production use until we have a log aggregation solution that provides end-users with a deis logs command. This is critical for troubleshooting app deployment issues. We also need a way to drain logs into tools like Splunk, Logstash, Graylog2, etc.

Admin Commands

Most frameworks require running one-off admin commands to setup databases, inspect settings and offer visibility into the Docker-based runtime environment. We will build a solution that offers a deis run bash capability.

SSL Support

Both the controller and formation proxies require SSL configuration. Right now it's just HTTP with cookies and passwords sent in the clear!

General Security Improvements

We've glossed over some pretty important security features in an effort to get Deis into developers hands sooner rather than later. For example, we need to implement iptables host-level firewalls, improve security group default rules, use Chef recipes to harden systems, etc. If you find any other security holes, please open a GitHub issue and tag it "Security".


Team Support

As of now, only a single user can control or push to a formation. We need to add simple sharing features, which we can then expand upon using finer-grained access controls.

Health Checks

We do not currently monitor nodes or container health -- though we have infrastructure in place to do it.

Cross-provider Formations

We are very close to offering cross-provider formations, where a single app deployment can span multiple zones, regions or even providers -- while being driven by a single git push or deis command.

Service Discovery

We need to make it as easy for ops folks to publish a set of reusable backing services (databases, queues, storage, etc) and allow developers to attach those services to formations. This will be done in a loosely coupled way, following Twelve Factor best practices.

How can you help?

Deis is still in its early stages -- we can use all the help we can get!

  • Star our GitHub repository
  • Help spread the word about @opendeis on Twitter
  • Explore contributing to the Deis project by joining the #deis channel on Freenode

You can learn about other ways to get involved on our website.

1 Aug 2013 in Deis v1 PaaS

Deis On Freenode Irc

Have a question? The #Deis channel on Freenode is where you'll find the Deis core developers. Come join the conversation.