Multi-Master Kubernetes Cluster Setup with CRI-O and Ceph Block Storage on CentOS 8
Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.
#
1. Architecture Diagram#
2. System Requirements#
2.1. Nginx Load BalancerComponent | Description |
---|---|
Number of VMs | 2 |
CPU | 2 Cores |
Memory | 4 GB |
Disk Size | 20 GB SSD |
Storage Type | Thin Provision |
Operating System | CentOS 8 x64 |
File System | XFS |
Privileges | ROOT access prefered |
#
2.2. Master NodesComponent | Description |
---|---|
Number of VMs | 3 |
CPU | 2 Cores |
Memory | 8 GB |
Disk Size | 150 GB SSD |
Storage Type | Thin Provision |
Operating System | CentOS 8 x64 |
File System | XFS |
Privileges | ROOT access prefered |
#
2.3. Worker NodesComponent | Description |
---|---|
Number of VMs | 3 |
CPU | 4 Cores |
Memory | 16 GB |
Disk Size | 500 GB SSD |
Storage Type | Thin Provision |
Operating System | CentOS 8 x64 |
File System | XFS |
Privileges | ROOT access prefered |
#
2.4. IP AllocationComponent | Description |
---|---|
Load Balancer Virtual IP | 10.101.16.80 |
VM IPs | 10.101.16.100 - 10.101.16.108 |
MetalLB IP Pool | 10.101.16.200 - 10.101.16.250 |
#
2.5. DNS EntriesIP | Hostname | FQDN |
---|---|---|
10.101.16.80 | N/A | kubelb.cluster.local |
10.101.16.100 | kubelb01 | kubelb01.cluster.local |
10.101.16.101 | kubelb02 | kubelb02.cluster.local |
10.101.16.102 | kubemaster01 | kubemaster01.cluster.local |
10.101.16.103 | kubemaster02 | kubemaster02.cluster.local |
10.101.16.104 | kubemaster03 | kubemaster03.cluster.local |
10.101.16.105 | kubeworker01 | kubeworker01.cluster.local |
10.101.16.106 | kubeworker02 | kubeworker02.cluster.local |
10.101.16.107 | kubeworker03 | kubeworker03.cluster.local |
#
3. Configure Nginx Load Balancersimportant
Verify the MAC address and product_uuid are unique for every node. You can get the MAC address of the network interfaces using
ip link | grep link/ether
The product_uuid can be checked by using
cat /sys/class/dmi/id/product_uuid
3.1. Set server hostname.
3.2. Install prerequisites.
3.3. Synchronize server time with Google NTP server.
3.4. Start and enable chronyd service.
3.5. Display time synchronization status.
3.6. Permanently disable SELinux.
3.7. Disable IPv6 on network interface.
3.8. Execute the following commands to turn off all swap devices and files.
3.9. Disable File Access Time Logging and enable Combat Fragmentation to enhance XFS file system performance. Add noatime,nodiratime,allocsize=64m
to all XFS volumes under /etc/fstab
.
3.10. Tweaking the system for high concurrancy and security.
3.11. Reload all sysctl variables without rebooting the server.
3.12. Configure firewall for Nginx and Keepalived.
3.13. Create Local DNS records.
3.14. Configure keepalived failover on kubelb01 and kubelb02.
important
Don't forget to change auth_pass to something more secure.
Change interface ens192 to match your interface name.
Change virtual_ipaddress from 10.101.16.80 to a valid IP.
The priority specifies the order in which the assigned interface takes over in a failover; the higher the number, the higher the priority.
3.14.1. Please execute the following command on kubelb01 Server.
3.14.2. Please execute the following command on kubelb02 Server.
3.15. Start and enable keepalived service on both load balancer nodes.
3.16. To determine whether a server is acting as the master, you can use the following command to see whether the virtual address is active.
3.17. Configure nginx on both load balancer nodes.
3.18. Start and enable nginx service on both load balancer nodes.
3.19. The servers need to be restarted before continue further.
3.20. Verify the load balancer.
note
#
If the load balancers are working, you should get the following outputcurl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to https://kubelb.cluster.local:6443
#
4. Install and Configure Kubernetes#
4.1. Install prerequisites on BOTH Master and Worker nodesimportant
Verify the MAC address and product_uuid are unique for every node. You can get the MAC address of the network interfaces using
ip link | grep link/ether
The product_uuid can be checked by using
cat /sys/class/dmi/id/product_uuid
Verify the Linux Kernel version is greater than 4.5.0. It can be checked by using
uname -r
Docker, CentOS 8 and the XFS filesystem could be a trouble giving combination if you don't meet all the specifications of the overlay/overlay2 storage driver.
The overlay storage driver relies on a technology called "directory entry type" (d_type) and is used to describe information of a directory on the filesystem. Make sure you have a d_type enabled filesystem by running the
xfs_info / | grep ftype
command. The ftype value must be set to 1. If not do not continue further.
4.1.1. Set server hostname.
4.1.2. Install prerequisites.
4.1.3. Synchronize server time with Google NTP server.
4.1.4. Start and enable chronyd service.
4.1.5. Display time synchronization status.
4.1.6. Permanently disable SELinux.
4.1.7. Enable IP masquerade at the Linux firewall.
4.1.8. Disable IPv6 on network interface.
4.1.9. Execute the following commands to turn off all swap devices and files.
4.1.10. Enable auto-loading of required kernel modules.
4.1.11. Disable File Access Time Logging and enable Combat Fragmentation to enhance XFS file system performance. Add noatime,nodiratime,allocsize=64m
to all XFS volumes under /etc/fstab
.
4.1.12. Tweaking the system for high concurrancy and security.
4.1.13. Reload all sysctl variables without rebooting the server.
4.1.14. Create Local DNS records.
4.1.15. Configure NetworkManager before attempting to use Calico networking.
4.1.16. The servers need to be restarted before continue further.
4.1.17. Configure CRI-O Container Runtime Interface repositories.
important
Note: The CRI-O major and minor versions must match the Kubernetes major and minor versions. For more information, see the CRI-O compatibility matrix.
4.1.18. Install CRI-O package.
4.1.19. Start and enable CRI-O service.
4.1.20. Add Kubernetes repository.
4.1.21. Install kubeadm, kubelet and kubectl packages.
4.1.22. Configure runtime cgroups used by kubelet service.
4.1.23. Enable kubelet service.
4.1.24. Pull latest docker images used by kubeadm.
#
4.2. Configure MASTER nodes#
4.2.1. Prepare Master Nodes4.2.1.1. Open necessary firewall ports used by Kubernetes.
#
4.2.2. Configure the First Master Node (kubemaster01)4.2.2.1. Create the kubeadm config file.
important
Please make sure to change controlPlaneEndpoint value as appropriate
4.2.2.2. Initialize the first control plane.
#
You will get an output like this. Please make sure to record MASTER and WORKER join commands.Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join kubelb.cluster.local:6443 --token ti2ho7.t146llqa4sn8y229 \
--discovery-token-ca-cert-hash sha256:9e73a021b8b26c8a2fc04939729acc7670769f15469887162cdbae923df906f9 \
--control-plane --certificate-key d9d631a0aef1a5a474faa6787b54814040adf1012c6c1922e8fe096094547b65 \
--v=5Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join kubelb.cluster.local:6443 --token ti2ho7.t146llqa4sn8y229 \
--discovery-token-ca-cert-hash sha256:9e73a021b8b26c8a2fc04939729acc7670769f15469887162cdbae923df906f9 \
--v=5
4.2.2.3. To start using kubectl, you need to run the following command.
4.2.2.4. Install calico CNI-plugin.
4.2.2.5. Check NetworkReady status. It must be TRUE. If not, wait some time and check it again.
4.2.2.6. Watch the ods created in the kube-system namespace and make sure all are running.
4.2.2.7. Check master node status.
#
4.2.3. Configure other master nodes (kubemaster02 and kubemaster03).important
Make sure to join other master nodes ONE BY ONE when the kubemaster01 status becomes READY.
Before execute the kubectl join command, make sure to verify all pods are up and running using
kubectl get po,svc --all-namespaces
.Use
--v=5
argument with kubeadm join in order to get a verbose output.
4.2.3.1. Execute the control-plane join command recorded in step 4.2.2.2.
4.2.3.2. To start using kubectl, you need to run the following command.
4.2.3.3. Check master node status.
#
4.3. Configure WORKER nodesimportant
Make sure to join worker nodes ONE BY ONE when the MASTER nodes status becomes READY.
Before execute the kubectl join command on worker nodes, make sure to verify all pods are up and running on master nodes using
kubectl get po,svc --all-namespaces
.Use
--v=5
argument with kubeadm join in order to get a verbose output.
4.3.1. Open necessary firewall ports used by Kubernetes.
4.3.2. Execute the worker nodes join command recorded in step 4.2.2.2.
#
4.4. Configure MetalLB Load Balancerimportant
You MUST execute these commands on a MASTER node.
Make sure to follow these steps only when the both MASTER and WORKER nodes status becomes READY.
Make sure to execute
kubectl get po,svc --all-namespaces
on a master node and verify all pods are up and running.
4.4.1. Install MetalLB Load Balancer.
4.4.2. Create MetalLB ConfigMap.
4.4.3. Watch the Pods created in the metallb-system namespace and make sure all are running.
note
If you want to change the MetalLB IP Pool, please follow these steps.
Note the old IPs allocated to services.
Delete the old ConfigMap.
Apply the new ConfigMap
Delete the existing MetalLB pods.
New MetalLB pods will be created automatically. Please make sure the pods are running.
Inspect new IPs of services.
#
4.5. Configure Kubernetes Dashboardimportant
You MUST execute these commands on a MASTER node.
Make sure to follow these steps only when the both MASTER and WORKER nodes status becomes READY.
Make sure to execute
kubectl get po,svc --all-namespaces
on a master node and verify all pods are up and running.
4.5.1. Install Kubernetes Dashboard.
4.5.2. Create the Dashboard service account.
4.5.3. Bind the dashboard-admin service account to the cluster-admin role.
4.5.4. When we created the dashboard-admin service account, Kubernetes also created a secret for it. List secrets using the following command.
4.5.5. Get Dashboard Access Token.
4.5.6. Watch Pods and Service accounts under kubernetes-dashboard namespace.
4.5.7. Get logs of kubernetes-dashboard.
4.5.8. Create kubernetes-dashboard load balancer.
4.5.9. Get logs of kubernetes-dashboard.
4.5.10. Get kubernetes-dashboard External IP.
#
4.6. Configure ROOK-CEPH Distributed Block Storageimportant
You MUST execute these commands on a MASTER node.
Make sure to follow these steps only when the both MASTER and WORKER nodes status becomes READY.
Make sure to execute
kubectl get po,svc --all-namespaces
on a master node and verify all pods are up and running.
4.6.1. Download and extract latest rook binaries.
4.6.2. Deploy all the resources needed by the Rook Ceph operator.
4.6.3. Deploy the Ceph operator that will be in charge of the setup and of the orchestration of a Ceph cluster.
important
It takes about 10 minutes for the operator to be up and running.
Its status can be verified using
watch kubectl get pod -n rook-ceph
command.Once the operator is ready, it triggers the creation of a DaemonSet in charge of deploying a rook-discover agent on each worker node of the Kubernetes cluster.
Do NOT proceed further until both rook-ceph-operator and rook-discover pods are in RUNNING state.
4.6.4. Create Ceph cluster.
important
It takes about 15 minutes for the cluster to be up and running.
Verify the cluster status using
watch kubectl get pod -n rook-ceph
.Do NOT proceed further until all the Pods in the rook-ceph namespace are in RUNNING state.
You can read rook-ceph-operator logs using
kubectl logs --follow --namespace rook-ceph --tail=100 -l app=rook-ceph-operator
.
4.6.5. Create a ReplicaPool and a StorageClass to automate the creation of a Kubernetes PersistentVolume backed-up by Ceph block storage.
4.6.6. Install Rook Toolbox.
4.6.7. To verify that the cluster is in a healthy state, connect to the Rook Toolbox and run the ceph status command.
4.6.8. Create Ceph Dashboard load balancer.
4.6.9. Access Ceph Dashboard.
4.6.10. List Pods under rook-ceph namespace. It will take about 30 minutes to get it ready.
note
If you want to clean up the rook cluster, please follow ceph-teardown instructions here.
#
4.7. Deploy a Sample WordPress Blogimportant
You MUST execute these commands on a MASTER node.
Make sure to follow these steps only when the both MASTER and WORKER nodes status becomes READY.
Make sure to execute
kubectl get po,svc --all-namespaces
on a master node and verify all pods are up and running.
4.7.1. Deploy a sample WordPress application using rook persistent volume claim.
#
4.8. Clean up Kubernatescaution
- The following commands are used to RESET your nodes and WIPE OUT all components installed.
4.8.1. Remove Kubernetes Components from Nodes
4.8.2. Remove ROOK-CEPH data
important
- This MUST be run on ALL WORKER nodes
- You should perform this operation only after cleaning up Kubernates
#
5. References- Install and configure a multi-master Kubernetes cluster with kubeadm
- How to Deploy a HA Kubernetes Cluster with kubeadm on CentOS7
- Demystifying High Availability in Kubernetes Using Kubeadm
- Highly Available Control Plane with kubeadm
- Install and configure a multi-master Kubernetes cluster with kubeadm
- HA Cluster vs. Backup/Restore
- Kubernetes HA Cluster installation guide
- Creating Highly Available clusters with kubeadm
- Deploy Kubernetes on vSphere
- vSphere Cloud Provider Configuration
- Rook on Kubernetes
- Lab Guide - Kubernetes and Storage With the Vsphere Cloud Provider - Step by Step
- Use vSphere Storage as Kubernetes persistent volumes
- Dynamic Provisioning and StorageClass API
- ROOK - Teardown Cluster
- What You Need to Know About MetalLB
- MetalLB Layer 2 Configuration
- Bare-metal considerations
- Kubernetes Ingress 101: NodePort, Load Balancers, and Ingress Controllers
- Ingress Controllers
- Kubernetes Storage on vSphere 101 – Failure Scenarios
- Moving a Stateful App from VCP to CSI based Kubernetes cluster using Velero
- Verifying that DNS is working correctly within your Kubernetes platform
- Debugging DNS Resolution
- CRI-O
- Container Runtimes
- CRI-O as a replacement for Docker
- How to install Kubernetes cluster on CentOS 8