Loading...
 

Consul

Introduction

Consul on Autopilot

Docker Compose

docker-compose.yml for Autopilot Consul on Triton
# Service definition for Consul cluster with a minimum of 3 nodes.
# Nodes will use Triton CNS for the service (passed in via the CONSUL
# env var) to find each other and bootstrap the cluster.

consul:
    image: autopilotpattern/consul:latest
    labels:
      - triton.cns.services=consul
    restart: always
    mem_limit: 128m
    ports:
        - 8500
    env_file:
      - _env
    command: >
      /usr/local/bin/containerpilot
      /bin/consul agent -server
        -bootstrap-expect 3
        -config-dir=/etc/consul
        -ui
docker-compose _env file for Joyent Public Cloud EU Amsterdam
# Consul bootstrap via Triton CNS
CONSUL=consul.svc.f045e9bf-056c-68fa-f340-f78dd0141896.eu-ams-1.cns.joyent.com
docker-compose _env file for Triton Private Cloud
# Consul bootstrap via Triton CNS
CONSUL=consul.svc.1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae.GB-Home.cns.gorillass.co.uk

Upgrading

registered bug for status=stopped filter not running in sdc-docker (joyent/sdc-docker: Issue #110). working process is:

  1. Firstly ensure /etc/containerpilot.json has a preStop for “consul leave” other wise step 6 is needed
  2. open the consul user interface and create a key/value (eg.: persistent:True)
  3. make changes (eg: to config file add/update labels to docker-compose.yml)
  4. scale from 3 to 7
  5. confirm changes (e.g. updated label from docker-compose.yml) on new instances
  6. connect to consul_1, added preStop, sent SIGHUP to container pilot
  7. repeat for consul containers 2-4
  8. stop AND remove containers 2-4 using “docker stop” (not docker-compose)
  9. verify data persistence
  10. verify scale by running docker-compose scale consul=3 (should result in “Desired container number already achieved”)
  11. verified config status by running docker-compose up -d (should result in “dockerlx_consul_5 is up-to-date, dockerlx_consul_6 is up-to-date, dockerlx_consul_7 is up-to-date”)

Consul Default

Docker Compose

docker-compose.yml file
version: '2'

services:
  consul:
    image: "consul"
    hostname: "consul1"
    command: "agent -server -bootstrap-expect 1 -ui -client 0.0.0.0"
    ports:
      - "8500:8500"
      - "8400:8400"
      - "8300:8300"
      - "8600:53/udp"
    environment:
      CONSUL_BIND_INTERFACE: "eth0"
#      CONSUL_CLIENT_INTERFACE: "eth0"


The environment variable CONSUL_BIND_INTERFACE is needed to as to be able to open up 8500 to remote connections for the UI interface. This also needs the argument -client 0.0.0.0 on the command to be able to run “command info” on the container. Otherwise it defaults to the loopback address (127.0.0.1) and results in an error.

Data Persistence



The image is using a separate Docker volumes for data storage:

Dockerfile
# Put Consul data on a separate volume to avoid filesystem performance issues
# with Docker image layers. Not necessary on Triton, but...
VOLUME ["/data"]

Across reboots


Data is preserved

Simple modification of docker-compose.yml


Simply adding description labels to the service definition in the docker-compose.yml file forces the containers to be recreated when running docker-compose up -d
Data is NOT preserved​ 

Leader Election



The raft protocol manages leader election between nodes and should manage this seamlessly as long as there is the right number of nodes to reach quorum. Having odd numbers of instances in a cluster prevents split-brain. However, under testing, it’s been found that scaling instances up and down has caused election failures due to remove old entries of node addresses from the raft DB. See “raft: Failed_to_make_RequestVote_RPC” section under troubleshooting.

Tidy-up

Reaping

When a node leaves, it specifies its intent to do so, and the cluster marks that node as having left. Unlike the failed case, all of the services provided by a node are immediately deregistered. If the agent was a server, replication to it will stop.

To prevent an accumulation of dead nodes (nodes in either failed or left states), Consul will automatically remove dead nodes out of the catalog. This process is called reaping. This is currently done on a configurable interval of 72 hours (changing the reap interval is not recommended due to its consequences during outage situations). Reaping is similar to leaving, causing all associated services to be deregistered.

Troubleshooting

raft: Failed to make RequestVote RPC

Summary

docker-compose logs -f
consul_2  |     2017/01/25 18:17:09 [ERR] agent: failed to sync remote state: No cluster leader
   ==> WARNING: Expect Mode enabled, expecting 3 servers
   ==> Starting Consul agent...
   ==> Starting Consul agent RPC...
   ==> Consul agent running!
         Version: 'v0.7.2'
         Node name: 'fcb746607bbe'
         Datacenter: 'dc1'
         Server: true (bootstrap: false)
         Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
         Cluster Addr: 192.168.78.110 (LAN: 8301, WAN: 8302)
         Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas:     
2017/01/25 17:47:38 [ERR] agent: failed to sync remote state: No cluster leader    
2017/01/25 17:47:40 [INFO] agent.rpc: Accepted client: 127.0.0.1:62221
2017/01/25 17:47:40     2017-01-25 17:47:40 containerpilot: No peers in raft    
2017/01/25 17:47:41 [INFO] agent.rpc: Accepted client: 127.0.0.1:40153    
2017/01/25 17:47:41 [INFO] agent: (LAN) joining: [consul.svc.1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae.GB-Home.cns.gorillass.co.uk]    
2017/01/25 17:47:41 [INFO] serf: EventMemberJoin: b7b6dbbdce24 192.168.78.103    
2017/01/25 17:47:41 [INFO] serf: EventMemberJoin: f20cb32a1daa 192.168.78.105    
2017/01/25 17:47:41 [INFO] consul: Adding LAN server b7b6dbbdce24 (Addr: tcp/192.168.78.103:8300) (DC: dc1)    
2017/01/25 17:47:41 [INFO] consul: Existing Raft peers reported by b7b6dbbdce24, disabling bootstrap mode    
2017/01/25 17:47:41 [INFO] consul: Adding LAN server f20cb32a1daa (Addr: tcp/192.168.78.105:8300) (DC: dc1)    
2017/01/25 17:47:41 [INFO] agent: (LAN) joined: 3 Err: 
2017/01/25 17:47:41 Successfully joined cluster by contacting 3 nodes....
consul_2  |     2017/01/25 18:17:10 [INFO] agent.rpc: Accepted client: 127.0.0.1:41280
consul_2  |     2017/01/25 18:17:10 [WARN] raft: Election timeout reached, restarting election
consul_2  |     2017/01/25 18:17:10 [INFO] raft: Node at 192.168.78.110:8300 [Candidate] entering Candidate state in term 235
consul_2  |     2017/01/25 18:17:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 192.168.78.103:8300 192.168.78.103:8300}: dial tcp 192.168.78.103:8300: i/o timeout
consul_2  |     2017/01/25 18:17:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 192.168.78.106:8300 192.168.78.106:8300}: dial tcp 192.168.78.106:8300: i/o timeout
consul_2  |     2017/01/25 18:17:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 192.168.78.105:8300 192.168.78.105:8300}: dial tcp 192.168.78.105:8300: i/o timeout


When using docker volumes (See line 50 of the Dockerfile), the raft DB seems to persist and keep a record of old raft members IP addresses. When performing docker-compose up -d after making changes. In this particular case docker-compose.yml was modified to build a new image rather than use the autopilotpatter/consul, prebuilt one.

docker-compose.yml
build: .
image: mesoform/consul:latest

Resolution


Moving the raft DB file out and rebooting forced the leader election to happen

$docker exec -it dockerlx_consul_2 sh
/ # mv /data/raft /data/raft.old
/ # reboot
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
         Version: 'v0.7.2'
         Node name: 'fcb746607bbe'
         Datacenter: 'dc1'
         Server: true (bootstrap: false)
         Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
         Cluster Addr: 192.168.78.110 (LAN: 8301, WAN: 8302)
         Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas:     
2017/01/25 18:18:27 [INFO] consul: Adding LAN server 09fed9f14022 (Addr: tcp/192.168.78.113:8300) (DC: dc1)    
2017/01/25 18:18:27 [INFO] consul: Adding LAN server 51f18dfe7e54 (Addr: tcp/192.168.78.116:8300) (DC: dc1)    
2017/01/25 18:18:27 [INFO] consul: Adding LAN server 3be18d40d970 (Addr: tcp/192.168.78.115:8300) (DC: dc1)    
2017/01/25 18:18:34 [ERR] agent: failed to sync remote state: No cluster leader    
2017/01/25 18:18:34 [WARN] raft: Heartbeat timeout from "" reached, starting election    
2017/01/25 18:18:34 [INFO] raft: Node at 192.168.78.110:8300 [Candidate] entering Candidate state in term 2    
2017/01/25 18:18:34 [INFO] raft: Election won. Tally: 3    
2017/01/25 18:18:34 [INFO] raft: Node at 192.168.78.110:8300 [Leader] entering Leader state    
2017/01/25 18:18:34 [INFO] raft: Added peer 192.168.78.111:8300, starting replication    
2017/01/25 18:18:34 [INFO] raft: Added peer 192.168.78.113:8300, starting replication    
2017/01/25 18:18:34 [INFO] raft: Added peer 192.168.78.116:8300, starting replication    
2017/01/25 18:18:34 [INFO] raft: Added peer 192.168.78.115:8300, starting replication    
2017/01/25 18:18:34 [INFO] consul: cluster leadership acquired    
2017/01/25 18:18:34 [INFO] consul: New leader elected: fcb746607bbe    
2017/01/25 18:18:34 [WARN] raft: AppendEntries to {Voter 192.168.78.113:8300 192.168.78.113:8300} rejected, sending older logs (next: 1)    
2017/01/25 18:18:34 [WARN] raft: AppendEntries to {Voter 192.168.78.115:8300 192.168.78.115:8300} rejected, sending older logs (next: 1)    
2017/01/25 18:18:34 [INFO] raft: pipelining replication to peer {Voter 192.168.78.115:8300 192.168.78.115:8300}    
2017/01/25 18:18:34 [INFO] raft: pipelining replication to peer {Voter 192.168.78.113:8300 192.168.78.113:8300}    
2017/01/25 18:18:34 [WARN] raft: AppendEntries to {Voter 192.168.78.111:8300 192.168.78.111:8300} rejected, sending older logs (next: 1)    
2017/01/25 18:18:34 [INFO] raft: pipelining replication to peer {Voter 192.168.78.111:8300 192.168.78.111:8300}    
2017/01/25 18:18:34 [WARN] raft: AppendEntries to {Voter 192.168.78.116:8300 192.168.78.116:8300} rejected, sending older logs (next: 1)    
2017/01/25 18:18:35 [INFO] raft: pipelining replication to peer {Voter 192.168.78.116:8300 192.168.78.116:8300}