etcd on CoreOS

3 Feb 2016

In my last post, we learnt about CoreOS installation on AWS EC2 instances. Specifically, we saw how to create a CoreOS cluster with three nodes and connect to nodes from terminal window.

One of the main building blocks of CoreOS is etcd, a distributed key-value store. When applications run on a cluster, accessing application config data in a consistent manner is a problem, since using the underlying system’s file system is not feasible.

If you want to build a failsafe cluster, application config needs to be available across the cluster. etcd solves this problem. Think of it as a file system available across all the cluster nodes. See my previous post that discusses etcd in more detail.

In this post, we’re going to look at how to use etc in real-life scenarios.

Get started

If you used CoreOS installation procedure as described in my previous post (or if your cloud-config file starts the etcd service) etcd should be available after bootup. To access the etcd service, you can use the etcdctl utility or curl.

We’ll see both of them one by one. First etcdctl, then curl.

After logging in to your EC2 instance, run:

etcdctl set /myKey Hello

It should look something like this:

etcd key output

If you want to use curl, run:

curl -L -XPUT -d value="Hello"

It should look like this:

curl output

This JSON response shows us we have a myKey key with a Hello value.

Now, any node in the cluster can query etcd for the key myKey and get the value Hello back.

Imagine running an application on the cluster now. Any settings, configuration, or temporary data your application needs is available across all nodes. So, it no longer matters which node your application is running on! This is an important requirement for shared-nothing, horizontally-scalable, fail-safe clusters.

To delete a key (and it's and related value) you can use the rm command:

etcdctl rm /myKey

Or, if you use curl, run:

curl -L -X DELETE


Things can get messy if there are several applications using the same etcd to store their configurations. Because etcd doesn't warn before overwriting data, there's a good chance two applications will try to use the same key name.

Fortunately, etcd provides directories as a way to manage key namespaces for different applications.

Let’s take a look at some directory commands.

Manage a directory

To create a directory, you can use the mkdir command:

etcdctl mkdir /myApplication

Once that's done, here's how you set a key using a directory:

etcdctl set /myApplication/myApp value1

Here's how you view the contents of a directory:

etcdctl ls /myApplication

And here's what that looks like:

etcd ls output

For curl users, create the directory and set a key by running:

curl -L -X PUT -d value="value1"

And you can read the directory contents by running:

curl -L

Which should look something like this:

curl output

Watch a directory

Sometimes you want to watch a key to see when it changes and trigger an action.

Let's take a look at how to do that.

To watch a directory, run:

etcdctl watch --recursive /myApplication

Or, if you use curl, run:

curl -L\&recursive=true

Whenever the value of a key changes in that directory, you’ll see it on your command prompt.

See this screenshot:

curl recursive output

The left window is tracking changes. We add a new key in the right window. The left window then notifies you of the change.

To trigger an command on a change, use the exec-watch command.

Try running:

etcdctl exec-watch --recursive /myApplication -- sh -c 'echo "\"$ETCD_WATCH_KEY\" key was updated to \"$ETCD_WATCH_VALUE\" value by \"$ETCD_WATCH_ACTION\" action"'

The -c switch provides the command to be executed.

The command in our example echoes a message to the terminal.

Here's what it looks like:

etcd exec-watch output

Specifically, you can see the output of the command on this line:

"/myApplication/myKey3" key was updated to "value3" value by "set" action

Note the $ETCD_WATCH_* variables used in the above commands. These are environment variables set by etcdctl.

Other commands

Let's take a look at some additional commands that make the coordination between clusters and applications easy.

Test and Set

Since etcd works as a centralized system with all the keys and values accessible uniformly, it can be used to create coordination services. For example: when you want an application to run after checking a dependency has been installed successfully.

The test and set functionality can be used in such cases.

Let's say you want to set myKey to B, but only if the previous value as A.

You'd run this command:

etcdctl set /myKey "B" --swap-with-value "A"

Or, if you're using curl, run:

curl -L -X PUT -d value=B


etcd also lets you set the Time to Live (TTL) for keys.

Once the TTL expires, the key is deleted and you’ll get an error code 100 (key not found) if you try to to look it up.

Let's say you want to set a TTL of 20 seconds.

You can run:

etcdctl set /myKey "value1" --ttl 20

Or, if you're using curl, run:

curl -L -X PUT -d value=bar


In this tutorial we took a look at etcd.

We covered:

  • How to set and get keys (from different nodes in a cluster)
  • How to manage directories
  • How to execute a command when the contents of a directory change
  • How to test and set values of a key
  • How to set TTLs

In my next post, we'll take a look at fleet in action.

Posted in etcd, CoreOS, Overview

triangle square circle

Did you enjoy this post?