Setup Highly Available ETCD Cluster on CentOS 7

The etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node.

1. Architecture Diagram#

2. System Requirements#

2.1. ETCD Nodes#

Component	Description
Number of VMs	3
CPU	2 Cores
Memory	4 GB
Disk Size	20 GB SSD
Operating System	CentOS 7 x64
File System	XFS
Privileges	ROOT access prefered

2.2. IP Allocation#

Component	Description
VM IPs	10.101.15.101 - 10.101.15.103

2.3. DNS Entries#

IP	Hostname	FQDN
10.101.15.101	etcd-1	etcd-1.cluster.local
10.101.15.102	etcd-2	etcd-2.cluster.local
10.101.15.103	etcd-3	etcd-3.cluster.local

3. Install and Configure ETCD HA Cluster#

3.1. Install prerequisites on ALL nodes#

3.1.1. Set server hostname.

# Example:
# sudo hostnamectl set-hostname etcd-1.cluster.local

sudo hostnamectl set-hostname <hostname>

3.1.2. Install prerequisites.

# Clean YUM repository cache
sudo yum clean all

# Update packages
sudo yum update -y

# Install prerequisites
sudo yum install -y vim curl ntp chrony net-tools yum-utils policycoreutils-python

3.1.3. Synchronize server time with default NTP servers. If you have your own NTP servers, please make sure to update the /etc/chrony.conf

# Set timezone to Asia/Colombo
sudo timedatectl set-timezone Asia/Colombo

# Enable NTP time synchronization
sudo timedatectl set-ntp true

3.1.4. Start and enable chronyd service.

# Start and enable chronyd service
sudo systemctl enable --now chronyd

# Verify if the service is started
sudo systemctl status chronyd

3.1.5. Display time synchronization status.

# Verify synchronisation state
sudo ntpstat

# Check Chrony Source Statistics
sudo chronyc sourcestats -v

3.1.6. Disable File Access Time Logging and enable Combat Fragmentation to enhance XFS file system performance. Add noatime,nodiratime,allocsize=64m to all XFS volumes under /etc/fstab.

# Edit /etc/fstab
sudo vim /etc/fstab

# Modify XFS volume entries as follows
# Example:
UUID="03c97344-9b3d-45e2-9140-cbbd57b6f085"  /  xfs  defaults,noatime,nodiratime,allocsize=64m  0 0

3.1.7. Tweaking the system for high concurrency and security.

cat <<"EOF" | sudo tee /etc/sysctl.d/00-sysctl.conf > /dev/null
#############################################################################################
# Tweak virtual memory
#############################################################################################

# Default: 30
# 0 - Never swap under any circumstances.
# 1 - Do not swap unless there is an out-of-memory (OOM) condition.
vm.swappiness = 30

# vm.dirty_background_ratio is used to adjust how the kernel handles dirty pages that must be flushed to disk.
# Default value is 10.
# The value is a percentage of the total amount of system memory, and setting this value to 5 is appropriate in many situations.
# This setting should not be set to zero.
vm.dirty_background_ratio = 5

# The total number of dirty pages that are allowed before the kernel forces synchronous operations to flush them to disk
# can also be increased by changing the value of vm.dirty_ratio, increasing it to above the default of 30 (also a percentage of total system memory)
# vm.dirty_ratio value in-between 60 and 80 is a reasonable number.
vm.dirty_ratio = 60

# vm.max_map_count will calculate the current number of memory-mapped files.
# The minimum value for mmap limit (vm.max_map_count) is the number of open files ulimit (cat /proc/sys/fs/file-max).
# map_count should be around 1 per 128 KB of system memory. Therefore, max_map_count will be 262144 on a 32 GB system.
# Reference: https://docs.confluent.io/current/kafka/deployment.html
# Default: 65530
vm.max_map_count = 2097152

#############################################################################################
# Tweak file descriptors
#############################################################################################

# Increases the size of file descriptors and inode cache and restricts core dumps.
fs.file-max = 2097152
fs.suid_dumpable = 0

#############################################################################################
# Tweak network settings
#############################################################################################

# Default amount of memory allocated for the send and receive buffers for each socket.
# This will significantly increase performance for large transfers.
net.core.wmem_default = 25165824
net.core.rmem_default = 25165824

# Maximum amount of memory allocated for the send and receive buffers for each socket.
# This will significantly increase performance for large transfers.
net.core.wmem_max = 25165824
net.core.rmem_max = 25165824

# In addition to the socket settings, the send and receive buffer sizes for
# TCP sockets must be set separately using the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem parameters.
# These are set using three space-separated integers that specify the minimum, default, and maximum sizes, respectively.
# The maximum size cannot be larger than the values specified for all sockets using net.core.wmem_max and net.core.rmem_max.
# A reasonable setting is a 4 KiB minimum, 64 KiB default, and 2 MiB maximum buffer.
net.ipv4.tcp_wmem = 20480 12582912 25165824
net.ipv4.tcp_rmem = 20480 12582912 25165824

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65536 25165824 262144
net.ipv4.udp_mem = 65536 25165824 262144

# Minimum amount of memory allocated for the send and receive buffers for each socket.
net.ipv4.udp_wmem_min = 16384
net.ipv4.udp_rmem_min = 16384

# Enabling TCP window scaling by setting net.ipv4.tcp_window_scaling to 1 will allow
# clients to transfer data more efficiently, and allow that data to be buffered on the broker side.
net.ipv4.tcp_window_scaling = 1

# Increasing the value of net.ipv4.tcp_max_syn_backlog above the default of 1024 will allow
# a greater number of simultaneous connections to be accepted.
net.ipv4.tcp_max_syn_backlog = 10240

# Increasing the value of net.core.netdev_max_backlog to greater than the default of 1000
# can assist with bursts of network traffic, specifically when using multigigabit network connection speeds,
# by allowing more packets to be queued for the kernel to process them.
net.core.netdev_max_backlog = 65536

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2

# Allowed local port range.
net.ipv4.ip_local_port_range = 2048 65535

# Protect Against TCP Time-Wait
# Default: net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# The maximum number of backlogged sockets.
# Default is 128.
net.core.somaxconn = 4096

# Turn on syncookies for SYN flood attack protection.
net.ipv4.tcp_syncookies = 1

# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Enable automatic window scaling.
# This will allow the TCP buffer to grow beyond its usual maximum of 64K if the latency justifies it.
net.ipv4.tcp_window_scaling = 1

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# Tells the kernel how many TCP sockets that are not attached to any
# user file handle to maintain. In case this number is exceeded,
# orphaned connections are immediately reset and a warning is printed.
# Default: net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_orphans = 65536

# Do not cache metrics on closing connections
net.ipv4.tcp_no_metrics_save = 1

# Enable timestamps as defined in RFC1323:
# Default: net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_timestamps = 1

# Enable select acknowledgments.
# Default: net.ipv4.tcp_sack = 1
net.ipv4.tcp_sack = 1

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks.
# net.ipv4.tcp_tw_recycle has been removed from Linux 4.12. Use net.ipv4.tcp_tw_reuse instead.
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_reuse = 1

# The accept_source_route option causes network interfaces to accept packets with the Strict Source Route (SSR) or Loose Source Routing (LSR) option set. 
# The following setting will drop packets with the SSR or LSR option set.
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP redirect acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

# Disables sending of all IPv4 ICMP redirected packets.
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Disable IP forwarding.
# IP forwarding is the ability for an operating system to accept incoming network packets on one interface, 
# recognize that it is not meant for the system itself, but that it should be passed on to another network, and then forwards it accordingly.
net.ipv4.ip_forward = 0

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

#############################################################################################
# Tweak kernel parameters
#############################################################################################

# Address Space Layout Randomization (ASLR) is a memory-protection process for operating systems that guards against buffer-overflow attacks.
# It helps to ensure that the memory addresses associated with running processes on systems are not predictable,
# thus flaws or vulnerabilities associated with these processes will be more difficult to exploit.
# Accepted values: 0 = Disabled, 1 = Conservative Randomization, 2 = Full Randomization
kernel.randomize_va_space = 2

# Allow for more PIDs (to reduce rollover problems)
kernel.pid_max = 65536
EOF

3.1.8. Reload all sysctl variables without rebooting the server.

sudo sysctl -p /etc/sysctl.d/00-sysctl.conf

3.1.9. Create Local DNS records.

cat <<"EOF" | sudo tee /etc/hosts > /dev/null
# localhost
127.0.0.1     localhost        localhost.localdomain

# When DNS records are updated in the DNS server, remove these entries.
10.101.15.101 etcd-1  etcd-1.cluster.local
10.101.15.102 etcd-2  etcd-2.cluster.local
10.101.15.103 etcd-3  etcd-3.cluster.local
EOF

3.1.10. The servers need to be restarted before continuing further.

sudo reboot

3.2. ETCD common configurations on ALL nodes#

3.2.1. Download and install the latest ETCD binary from GitHub.

important

You can download the latest ETCD binary from here.

# Set ETCD version
ETCD_VER=v3.4.14

# Download the latest tarball
curl -L -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz https://storage.googleapis.com/etcd/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz

# Extract the downloaded tarball
tar xfz /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp

# Move extracted binaries to /usr/local/bin
sudo mv /tmp/etcd-${ETCD_VER}-linux-amd64/etcd /usr/local/bin
sudo mv /tmp/etcd-${ETCD_VER}-linux-amd64/etcdctl /usr/local/bin

# Set permissions
sudo chown root:root /usr/local/bin/etcd
sudo chown root:root /usr/local/bin/etcdctl

# Remove temporary files
sudo rm -rf /tmp/etcd-${ETCD_VER}-linux-amd64*

3.2.2. Create ETCD service user.

# Create ETCD service user
sudo useradd --system --no-create-home etcd

3.2.3. Create ETCD directory structure and provide necessary permissions.

# Create ETCD directory structure
sudo mkdir -p /etc/etcd/tls /var/lib/etcd

# Provide necessary permissions
sudo chown -R etcd:etcd /etc/etcd /var/lib/etcd
sudo chmod 0700 /var/lib/etcd

# Restore SELinux context
sudo restorecon -RvF /etc/etcd /var/lib/etcd

3.2.4. Copy the correct SSL certificate, SSL key and the CA certificate files under /etc/etcd/tls

important

To generate a custom CA and CA signed SSL certificates, please follow this guide.
Other than the *.cluster.local domain, you must have 'localhost' as a Subject Alternative Name (SAN) since etcdctl uses it for internal API calls.

# Copy the correct CA certificate, TLS certificate and TLS key
/etc/etcd/tls/ca.pem
/etc/etcd/tls/tls.crt
/etc/etcd/tls/tls.key

# Set correct permissions
sudo chown -R etcd:etcd /etc/etcd/tls
sudo chmod 0600 /etc/etcd/tls/*

# Restore SELinux context
sudo restorecon -RvF /etc/etcd/tls

# If you are using a self-signed certificate, make sure to configure it as a trusted root certificate
sudo cp /etc/etcd/tls/ca.pem /etc/pki/ca-trust/source/anchors/ca.pem
sudo update-ca-trust

3.2.5. Open necessary firewall ports.

sudo firewall-cmd --permanent --add-port={2379,2380}/tcp
sudo firewall-cmd --reload

3.2.6. Create ETCD Systemd service.

# Create ETCD Systemd service
cat <<EOF | sudo tee /etc/systemd/system/etcd.service > /dev/null
[Unit]
Description="ETCD Service"
Documentation=https://github.com/etcd-io/etcd
After=network.target
 
[Service]
Type=notify
User=etcd
Group=etcd
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
NoNewPrivileges=yes
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=40000
 
[Install]
WantedBy=multi-user.target
EOF

3.2.7. Restore SELinux context.

# Restore SELinux context
sudo restorecon -Fv /etc/systemd/system/etcd.service

3.2.8. Reload systemd manager configurations.

# Reload systemd manager configurations
sudo systemctl daemon-reload

3.3. Configurations on ETCD-1 node.#

3.3.1. Add the configurations under /etc/etcd/etcd.conf

# ETCD-1 configurations
cat <<"EOF" | sudo tee /etc/etcd/etcd.conf > /dev/null
ETCD_NAME=ETCD-1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER="ETCD-1=https://etcd-1.cluster.local:2380,ETCD-2=https://etcd-2.cluster.local:2380,ETCD-3=https://etcd-3.cluster.local:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-1.cluster.local:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd-1.cluster.local:2379"
ETCD_TRUSTED_CA_FILE="/etc/etcd/tls/ca.pem"
ETCD_CERT_FILE="/etc/etcd/tls/tls.crt"
ETCD_KEY_FILE="/etc/etcd/tls/tls.key"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/tls/ca.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/tls/tls.key"
ETCD_PEER_CERT_FILE="/etc/etcd/tls/tls.crt"
EOF

3.3.2. Set correct permissions.

# Set permissions
sudo chown etcd:etcd /etc/etcd/etcd.conf
sudo chmod 0644 /etc/etcd/etcd.conf

# Restore SELinux context
sudo restorecon -RvF /etc/etcd

3.3.3. Start and enable etcd.service.

# Start and enable etcd.service
sudo systemctl enable --now etcd.service

3.3.4. If there are any errors, please check systemd logs.

# Check error messages in journald
sudo journalctl -f -b --no-pager -u etcd

3.4. Configurations on ETCD-2 node.#

3.4.1. Add the configurations under /etc/etcd/etcd.conf

# ETCD-2 configurations
cat <<"EOF" | sudo tee /etc/etcd/etcd.conf > /dev/null
ETCD_NAME=ETCD-2
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER="ETCD-1=https://etcd-1.cluster.local:2380,ETCD-2=https://etcd-2.cluster.local:2380,ETCD-3=https://etcd-3.cluster.local:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-2.cluster.local:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd-2.cluster.local:2379"
ETCD_TRUSTED_CA_FILE="/etc/etcd/tls/ca.pem"
ETCD_CERT_FILE="/etc/etcd/tls/tls.crt"
ETCD_KEY_FILE="/etc/etcd/tls/tls.key"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/tls/ca.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/tls/tls.key"
ETCD_PEER_CERT_FILE="/etc/etcd/tls/tls.crt"
EOF

3.4.2. Set correct permissions.

# Set permissions
sudo chown etcd:etcd /etc/etcd/etcd.conf
sudo chmod 0644 /etc/etcd/etcd.conf

# Restore SELinux context
sudo restorecon -RvF /etc/etcd

3.4.3. Start and enable etcd.service.

# Start and enable etcd.service
sudo systemctl enable --now etcd.service

3.4.4. If there are any errors, please check systemd logs.

# Check error messages in journald
sudo journalctl -f -b --no-pager -u etcd

3.5. Configurations on ETCD-3 node.#

3.5.1. Add the configurations under /etc/etcd/etcd.conf

# ETCD-3 configurations
cat <<"EOF" | sudo tee /etc/etcd/etcd.conf > /dev/null
ETCD_NAME=ETCD-3
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER="ETCD-1=https://etcd-1.cluster.local:2380,ETCD-2=https://etcd-2.cluster.local:2380,ETCD-3=https://etcd-3.cluster.local:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-3.cluster.local:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd-3.cluster.local:2379"
ETCD_TRUSTED_CA_FILE="/etc/etcd/tls/ca.pem"
ETCD_CERT_FILE="/etc/etcd/tls/tls.crt"
ETCD_KEY_FILE="/etc/etcd/tls/tls.key"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/tls/ca.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/tls/tls.key"
ETCD_PEER_CERT_FILE="/etc/etcd/tls/tls.crt"
EOF

3.5.2. Set correct permissions.

# Set permissions
sudo chown etcd:etcd /etc/etcd/etcd.conf
sudo chmod 0644 /etc/etcd/etcd.conf

# Restore SELinux context
sudo restorecon -RvF /etc/etcd

3.5.3. Start and enable etcd.service.

# Start and enable etcd.service
sudo systemctl enable --now etcd.service

3.5.4. If there are any errors, please check systemd logs.

# Check error messages in journald
sudo journalctl -f -b --no-pager -u etcd

4. Maintenance#

4.1. Verify ETCD Cluster Health#

4.1.1. Checks the healthiness of endpoints.

# Checks the healthiness of endpoints
ETCDCTL_API=3 etcdctl endpoint health \
 --endpoints="https://localhost:2379" \
 --cacert /etc/etcd/tls/ca.pem \
 --cert /etc/etcd/tls/tls.crt \
 --key /etc/etcd/tls/tls.key \
 --debug

4.1.2. Prints out the status of endpoints.

# Prints out the status of endpoints
ETCDCTL_API=3 etcdctl endpoint status \
 --endpoints="https://localhost:2379" \
 --cacert /etc/etcd/tls/ca.pem \
 --cert /etc/etcd/tls/tls.crt \
 --key /etc/etcd/tls/tls.key \
 --debug

4.2. Backup and Restore an ETCD Cluster#

4.2.1. Create an ETCD snapshot.

important

Recovering a cluster first needs a snapshot of the keyspace from an etcd member.
A snapshot may either be taken from a live member with the etcdctl snapshot save command or by copying the member/snap/db file from an etcd data directory.
For example, the following command snapshots the keyspace to the file snapshot.db

# Create a ETCD snapshot
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
 --endpoints="https://localhost:2379" \
 --cacert /etc/etcd/tls/ca.pem \
 --cert /etc/etcd/tls/tls.crt \
 --key /etc/etcd/tls/tls.key \
 --debug

4.2.2. Restore an ETCD Cluster.

important

To restore a cluster, all that is needed is a single snapshot db file. A cluster restore with etcdctl snapshot restore creates new etcd data directories; all members should restore using the same snapshot. Restoring overwrites some snapshot metadata (specifically, the member ID and cluster ID); the member loses its former identity. This metadata overwrite prevents the new member from inadvertently joining an existing cluster. Therefore in order to start a cluster from a snapshot, the restore must start a new logical cluster.
Snapshot integrity may be optionally verified at restore time. If the snapshot is taken with etcdctl snapshot save, it will have an integrity hash that is checked by etcdctl snapshot restore. If the snapshot is copied from the data directory, there is no integrity hash and it will only restore by using the configuration flag --skip-hash-check=true.
A restore initializes a new member of a new cluster, with a fresh cluster configuration using etcd's cluster configuration flags, but preserves the contents of the etcd keyspace. Continuing from the previous example, the following creates new etcd data directories under /var/lib/etcd for a three member cluster.
Before you execute the following commands, please make sure to BACKUP /var/lib/etcd directory.

4.2.2.1. Stop ETCD service and remove /var/lib/etcd directory on ALL nodes.

# Stop ETCD service
sudo systemctl stop etcd

# Remove /var/lib/etcd directory
sudo rm -rf /var/lib/etcd

4.2.2.2. Restore ETCD-1 node.

# Restore ETCD-1
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
 --name ETCD-1 \
 --data-dir /var/lib/etcd \
 --initial-cluster-token "etcd-cluster" \
 --initial-advertise-peer-urls https://etcd-1.cluster.local:2380 \
 --initial-cluster "ETCD-1=https://etcd-1.cluster.local:2380,ETCD-2=https://etcd-2.cluster.local:2380,ETCD-3=https://etcd-3.cluster.local:2380"

4.2.2.3. Restore ETCD-2 node.

# Restore ETCD-2
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
 --name ETCD-2 \
 --data-dir /var/lib/etcd \
 --initial-cluster-token "etcd-cluster" \
 --initial-advertise-peer-urls https://etcd-2.cluster.local:2380 \
 --initial-cluster "ETCD-1=https://etcd-1.cluster.local:2380,ETCD-2=https://etcd-2.cluster.local:2380,ETCD-3=https://etcd-3.cluster.local:2380"

4.2.2.4. Restore ETCD-3 node.

# Restore ETCD-3
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
 --name ETCD-3 \
 --data-dir /var/lib/etcd \
 --initial-cluster-token "etcd-cluster" \
 --initial-advertise-peer-urls https://etcd-3.cluster.local:2380 \
 --initial-cluster "ETCD-1=https://etcd-1.cluster.local:2380,ETCD-2=https://etcd-2.cluster.local:2380,ETCD-3=https://etcd-3.cluster.local:2380"

4.2.2.5. Set necessary permissions and SELinux context on ALL nodes.

# Set necessary permissions
sudo chown -R etcd:etcd /var/lib/etcd
sudo chmod 0700 /var/lib/etcd

# Restore SELinux context
sudo restorecon -RvF /var/lib/etcd

4.2.2.6. Start ETCD service on ALL nodes.

# Start ETCD service
sudo systemctl start etcd

# Verify if the ETCD service is running
sudo systemctl status etcd

4.2.2.7. If there are any errors, please check systemd logs.

# Check error messages in journald
sudo journalctl -f -b --no-pager -u etcd

1. Architecture Diagram#

2. System Requirements#

2.1. ETCD Nodes#

2.2. IP Allocation#

2.3. DNS Entries#

3. Install and Configure ETCD HA Cluster#

3.1. Install prerequisites on ALL nodes#

3.2. ETCD common configurations on ALL nodes#

important

important

3.3. Configurations on ETCD-1 node.#

3.4. Configurations on ETCD-2 node.#

3.5. Configurations on ETCD-3 node.#

4. Maintenance#

4.1. Verify ETCD Cluster Health#

4.2. Backup and Restore an ETCD Cluster#

important

important

5. References#