Demonstrating the set-up of CoreOS and how to utilise Etcd along with Systemd and Fleet

CoreOS Essentials.md

Getting started

git clone git@github.com:coreos/coreos-vagrant.git
mv user-data.sample user-data
Change etcd2 to have the following content:

advertise-client-urls: http://$public_ipv4:2379,http://$public_ipv4:4001
initial-advertise-peer-urls: http://$private_ipv4:2380
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001
initial-cluster-token: core-01_etcd
initial-cluster: core-01=http://$private_ipv4:2380
initial-cluster-state: new

Services

Systemd: CoreOS’s init system
Etcd: distributed key-value store
Fleet: distributed init system (ties together systemd and etcd; NOT a container orchestrator)

Setup

# Access the CoreOS VM
vagrant up
vagrant ssh

# Check etcd and fleet are running
systemctl status etcd2
systemctl status fleet

# Check installed version of Docker
docker version

Etcd

# Verify Host/VM access to etcd
etcdctl set /foo bar
etcdctl get /foo
etcdctl rm  /foo
etcdctl set /foo-dir/foo-key "foo value"
etcdctl set /foo-dir/bar-key "bar value"
etcdctl ls  /foo-dir --recursive # works without --recursive and also with -recursive ?
etcdctl get /foo-dir/foo-key

HTTP API

Etcd has an HTTP API
The API always responds with JSON
e.g. {"action":"get","node":{"key":"/baz","value":"qux","modifiedIndex":884,"createdIndex":884}}
It is useful for client apps and host containers that don’t have etcdctl installed

# Verify HTTP API access to etcd
curl --location --request PUT http://127.0.0.1:2379/v2/keys/baz --data value="qux"
curl --location http://127.0.0.1:2379/v2/keys/baz
curl --location --request DELETE http://127.0.0.1:2379/v2/keys/baz

# Check docker0 interface IP
echo "$(ifconfig docker0 | awk '/\<inet\>/ { print $2 }'):2379"

# Run a container from the Alpine Linux distro and access etcd from within it
docker run -it alpine ash # that's not a typo "ash" is correct
apk update && apk add curl
curl --location http://<docker0 interface IP>:2379/v2/keys/

Watching

# Watch etcd for changes (requires two terminal shells)
etcdctl mkdir /foo-data
etcdctl watch /foo-data --recursive # unlike `ls` you MUST use --recursive flag (-recursive also works?)
etcdctl watch /foo-data --recursive --forever

From a second terminal shell…

vagrant ssh
etcdctl set /foo-data/can-you-see-me-now yes

TTLs

# Verify etcd's TTL feature
etcdctl set /my-ttl "I'm expiring in 30 seconds..." --ttl 30
etcdctl get /my-ttl

…30 seconds later…

etcdctl get /my-ttl

SystemD

Control of ‘unit’ files/services are on a per machine basis
So only use for simple tasks

# Verify systemd's unit file setup for managing container processes
sudo vi /etc/systemd/system/hello.service

Add following content to hello.service:

[Unit]
Description=HelloWorld
# this unit will only start after docker.service
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
# busybox image will be pulled from docker public registry
ExecStartPre=/usr/bin/docker pull busybox
# we use rm just in case the container with the name "busybox1" is left
ExecStartPre=-/usr/bin/docker rm busybox1
# we start docker container
ExecStart=/usr/bin/docker run --rm --name busybox1 busybox /bin/sh -c "while true; do echo Hello World; sleep 1; done"
# we stop docker container when systemctl stop is used
ExecStop=/usr/bin/docker stop busybox1

[Install]
WantedBy=multi-user.target

Enable the service:

This creates a symlink from an internal user directory to the main system directory

sudo systemctl enable /etc/systemd/system/hello.service

Start the service:

It can take a few seconds to complete as it has to pull a docker image from DockerHub

sudo systemctl start hello.service

Verify the service:

journalctl -f -u hello.service

Responds with tailed output:

-- Logs begin at Thu 2015-10-29 08:42:56 UTC. --
Oct 29 09:29:07 core-01 docker[2119]: Hello World
Oct 29 09:29:08 core-01 docker[2119]: Hello World
Oct 29 09:29:09 core-01 docker[2119]: Hello World
Oct 29 09:29:10 core-01 docker[2119]: Hello World
Oct 29 09:29:11 core-01 docker[2119]: Hello World
Oct 29 09:29:12 core-01 docker[2119]: Hello World
Oct 29 09:29:13 core-01 docker[2119]: Hello World

docker ps will also show the container is running

Check the status of the service:

sudo systemctl status hello.service

Stop the service:

sudo systemctl stop hello.service

Kill the service:

This doesn’t stop the container!
Check docker ps and you’ll see it still running
Also journalctl -f -u hello.service will show logs still coming from the container

sudo systemctl kill hello.service

Restart the service:

sudo systemctl daemon-reload # only needed if you've changed the service file
sudo systemctl restart hello.service

Disable the service:

sudo systemctl disable hello.service

Fleet

Fleet is a cluster manager that controls Systemd at the cluster level
It can still be used on a single machine

Fleet unit files are the same as Systemd unit files, but with additional properties that help them identify in the context of a cluster. The only other difference between them is that there is no [Install] in the fleet unit files, as this is replaced with [X-Fleet].

Let’s see what machines are available:

fleetctl list-machines

Which responds with:

MACHINE     IP            METADATA
46acb6e7... 172.17.8.101  -

Fleet unit files can be placed any where on the system and then ‘submitted’ to fleet.

Create the following fleet unit file twice (change the description and url to ping) and name the files ping.1.service and ping.2.service:

e.g. for the second version you could use
Description=Ping BBC
ExecStart=/usr/bin/ping bbc.co.uk

[Unit]
Description=Ping Google

[Service]
ExecStart=/usr/bin/ping google.com

[X-Fleet]

We keep X-Fleet empty because with one machine, specifying other settings will cause things to break (I’ve yet to figure out why?)

“Conflicts” prevents the unit being collocated with the other unit using glob-matching. Where “MachineMetadata” can be used to determine if an instance in the cluster is acceptable. For example:

# Allow a cluster instance to run this unit 
# Only if they have region set to one of these values
MachineMetadata=region=us-east-1
MachineMetadata=region=us-west-1

Or alternatively bundle multiple requirements into a single item:

MachineMetadata="region=us-east-1" "diskType=SSD"

Submit the fleet unit files:

fleetctl submit ping.1.service
fleetctl submit ping.2.service

Review list of unit files available:

fleetctl list-unit-files

Which responds with:

UNIT            HASH    DSTATE    STATE     TARGET
ping.1.service  4764f30 inactive  inactive  -
ping.2.service  683fbdc inactive  inactive  -

You can remove services from the fleet using destroy:

fleetctl destroy ping.1.service
fleetctl destroy ping.2.service

To start the fleet services:

fleetctl start ping.1.service
fleetctl start ping.2.service

Now listing the unit files again:

UNIT            HASH    DSTATE    STATE     TARGET
ping.1.service  4c2ca38 launched  launched  46acb6e7.../172.17.8.101
ping.2.service  86f264e launched  launched  46acb6e7.../172.17.8.101

You can also verify the services are running using journalctl:

journalctl -f -u ping.1.service
journalctl -f -u ping.2.service

Cloud Config vs User Data

When using Vagrant to set-up a local cluster, you’ll have a user-data file, which is really a cloud-config file in the cloud world.

When using Vagrant you’ll also have a config.rb which is used to construct the cloud-config file by manipulating the local user-data file.

Creating files on all clusters

write_files:
  - path: /home/core/test.txt
    permissions: 0644
    owner: core
    content: |
      Beep Boop

Then run the following commands:

vagrant provision && vagrant reload

Note: make sure it’s not nested inside the top-level coreos section! as it won’t work

Added metadata to all clusters

coreos:
  fleet:
    metadata: cluster=vagrant

Then SSH into the instance (e.g. vagrant ssh core-01) and then run:

fleetctl list-machines

Which responds with:

MACHINE		  IP		        METADATA
5fcf399b...	172.17.8.101	cluster=vagrant
987c97e7...	172.17.8.102	cluster=vagrant
c15aed24...	172.17.8.103	cluster=vagrant

Schedule a new fleet/service for the cluster

cd coreos-vagrant-localcluster && vagrant up
vagrant ssh core-03
vi hello-cluster.service (see below)
fleetctl start hello-cluster.service
journalctl -u hello-cluster.service -f

hello-cluster.service

[Unit]
[Service]
ExecStart=/usr/bin/bash -c "while true; do echo 'Hello Cluster'; sleep 1; done"

Response from `fleetctl` command

Unit hello-cluster.service inactive
Unit hello-cluster.service launched on 5fcf399b.../172.17.8.101

Demonstrating the set-up of CoreOS and how to utilise Etcd along with Systemd and Fleet

CoreOS Essentials.md

Getting started

Services

Setup

Etcd

HTTP API

Watching

TTLs

SystemD

Fleet

Cloud Config vs User Data

Creating files on all clusters

Added metadata to all clusters

Schedule a new fleet/service for the cluster

hello-cluster.service

Response from fleetctl command

Response from `fleetctl` command