In this guide, we shall be focusing on the installation of Kubernetes Cluster on Rocky Linux 8 with Kubeadm & CRI-O container runtime engine. It is no doubt that Kubernetes continues to transform how we deploy and manage business applications, at scale. Irregardless of the model used to deploy applications, whether manual or through CI/CD pipelines, Kubernetes still remains the best choice to handle orchestration, management, and scaling your applications.
For those who do not know, Kubernetes has deprecated dockershim which means they no longer support docker as the container runtime. It is for this reason we’ll be using CRI-O container runtime. CRI-O is a lightweight alternative to using Docker as the runtime for kubernetes. Before you throw tantrums, it is important to note Cri-o works similar to Docker. The way Developer build and deploy application images is not affected.
What is CRI-O?
CRI-O is an implementation of the Kubernetes CRI (Container Runtime Interface) to enable using OCI (Open Container Initiative) compatible runtimes. CRI-O allows Kubernetes to use any OCI-compliant runtime as the container runtime for running pods.
CRI-O supports OCI container images and can pull from any container registry. It is a lightweight alternative to using Docker, Moby or rkt as the runtime for Kubernetes.
CRI-O architecture pulled from official project’s website:
1. Setup Pre-reqs
I’ll deploy a Kubernetes cluster on Rocky Linux 8 set of nodes as outlined below:
- 3 Control plane nodes (Masters)
- 4 Worker node machines (Data plane)
- Ansible version required 2.10+
But it is okay to setup a Kubernetes Cluster with 1 master node using the process demonstrated in this guide.
Virtual Machines and Specs
We can see all the running nodes in my lab environment using the virsh command since it is powered by Vanilla KVM:
[root@kvm-private-lab ~]# virsh list
Id Name State
--------------------------------------
4 k8s-bastion-server running
17 k8s-master-01-server running
22 k8s-master-02-server running
26 k8s-master-03-server running
31 k8s-worker-01-server running
36 k8s-worker-02-server running
38 k8s-worker-03-server running
41 k8s-worker-04-server runningThe first machine in the list will be our bastion / workstation machine. This is where we’ll perform the installation of Kubernetes Cluster on Rocky Linux 8 servers.
My list of servers with hostnames and IP addresses:
| Hostname | IP Address | Cluster Role |
| k8s-bastion.example.com | 192.168.200.9 | Bastion host |
| k8s-master-01.example.com | 192.168.200.10 | Master Node |
| k8s-master-02.example.com | 192.168.200.11 | Master Node |
| k8s-master-03.example.com | 192.168.200.12 | Worker Node |
| k8s-worker-01.example.com | 192.168.200.13 | Worker Node |
| k8s-worker-02.example.com | 192.168.200.14 | Worker Node |
| k8s-worker-03.example.com | 192.168.200.15 | Worker Node |
| k8s-worker-04.example.com | 192.168.200.16 | Worker Node |
Each machine in my cluster has the following hardware specifications:
# Memory
$ free -h
total used free shared buff/cache available
Mem: 7.5Gi 169Mi 7.0Gi 8.0Mi 285Mi 7.1Gi
# Disk space
$ df -hT /
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda4 xfs 39G 2.0G 37G 5% /
# CPU Cores
$ egrep ^processor /proc/cpuinfo | wc -l
2DNS Settings
Set A records for the hostnames in your DNS server and optionally on /etc/hosts file in each cluster node in case DNS resolution fails.
DNS bind records creation sample:
; Create entries for the master nodes
k8s-master-01 IN A 192.168.200.10
k8s-master-02 IN A 192.168.200.11
k8s-master-03 IN A 192.168.200.12
; Create entries for the worker nodes
k8s-worker-01 IN A 192.168.200.13
k8s-worker-02 IN A 192.168.200.14
k8s-worker-03 IN A 192.168.200.15
;
; The Kubernetes cluster ControlPlaneEndpoint point these to the IP of the masters
k8s-endpoint IN A 192.168.200.10
k8s-endpoint IN A 192.168.200.11
k8s-endpoint IN A 192.168.200.122. Prepare Bastion Server
Install basic tools required for the setup on the Bastion / Workstation system:
### Ubuntu / Debian ###
sudo apt update
sudo apt install git wget curl vim bash-completion
### CentOS / RHEL / Fedora / Rocky Linux ###
sudo yum -y install git wget curl vim bash-completionInstall Ansible configuration management
With Python 3:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --userWith Python 2:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py --userInstalling Ansible using pip
# Python 3
python3 -m pip install ansible --user
# Python 2
python -m pip install ansible --userCheck Ansible version after installation:
$ ansible --version
ansible [core 2.11.5]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.9.7 (default, Aug 30 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
jinja version = 3.0.1
libyaml = TrueUpdate /etc/hosts file in your Bastion machine:
$ sudo vim /etc/hosts
192.168.200.9 k8s-bastion.example.com k8s-bastion
192.168.200.10 k8s-master-01.example.com k8s-master-01
192.168.200.11 k8s-master-02.example.com k8s-master-02
192.168.200.12 k8s-master-03.example.com k8s-master-03
192.168.200.13 k8s-worker-01.example.com k8s-worker-01
192.168.200.14 k8s-worker-02.example.com k8s-worker-02
192.168.200.15 k8s-worker-03.example.com k8s-worker-03
192.168.200.16 k8s-worker-04.example.com k8s-worker-04Generate SSH keys:
$ ssh-keygen -t rsa -b 4096 -N ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:LwgX4oCWENqWAyW9oywAv9jTK+BEk4+XShgX0galBqE root@k8s-master-01.example.com
The key's randomart image is:
+---[RSA 4096]----+
|OOo |
|B**. |
|EBBo. . |
|===+ . . |
|=*+++ . S |
|*=++.o . . |
|=.o. .. . . |
| o. . . |
| . |
+----[SHA256]-----+Create SSH client configuration file
$ vim ~/.ssh/config
Host *
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
IdentitiesOnly yes
ConnectTimeout 0
ServerAliveInterval 30Copy SSH keys to all Kubernetes cluster nodes
# Master Nodes
for host in k8s-master-0{1..3}; do
ssh-copy-id root@$host
done
# Worker Nodes
for host in k8s-worker-0{1..4}; do
ssh-copy-id root@$host
done3. Set correct hostname on all nodes
Login to each node in the cluster and configure correct hostname:
# Examples
# Master Node 01
sudo hostnamectl set-hostname k8s-master-01.example.com
# Worker Node 01
sudo hostnamectl set-hostname k8s-worker-01.example.comLogout then back in to confirm the hostname is set correctly:
$ hostnamectl
Static hostname: k8s-master-01.example.com
Icon name: computer-vm
Chassis: vm
Machine ID: 7a7841970fc6fab913a02ca8ae57fe18
Boot ID: 4388978e190a4be69eb640b31e12a63e
Virtualization: kvm
Operating System: Rocky Linux 8.4 (Green Obsidian)
CPE OS Name: cpe:/o:rocky:rocky:8.4:GA
Kernel: Linux 4.18.0-305.19.1.el8_4.x86_64
Architecture: x86-64For a cloud instance using cloud-init check out the guide below:
4. Prepare Rocky Linux 8 servers
I wrote an Ansible role for doing the standard Kubernetes node preparation. The role contain the tasks to:
- Install standard packages required to manage nodes
- Setup standard system requirements – Disable Swap, Modify sysctl, Disable SELinux
- Install and configure a container runtime of your Choice – cri-o, Docker, Containerd
- Install the Kubernetes packages – kubelet, kubeadm and kubectl
- Configure Firewalld on Kubernetes Master and Worker nodes – open all ports required
Clone my Ansible role that we’ll use to setup Kubernetes Requirements before kubeadm init to your Bastion machine:
git clone https://github.com/jmutai/k8s-pre-bootstrap.gitSwitch to k8s-pre-bootstrap directory created from the clone process:
cd k8s-pre-bootstrapSet hosts inventory correctly with your Kubernetes nodes. Here is my inventory list:
$ vim hosts
[k8snodes]
k8s-master-01
k8s-master-02
k8s-master-03
k8s-worker-01
k8s-worker-02
k8s-worker-03
k8s-worker-04You should also update the variables in playbook file. The most important being:
- Kubernetes version: k8s_version
- Your timezone: timezone
- Kubernetes CNI to use: k8s_cni
- Container runtime: container_runtime
$ vim k8s-prep.yml
---
- name: Setup Proxy
hosts: k8snodes
remote_user: root
become: yes
become_method: sudo
#gather_facts: no
vars:
k8s_version: "1.21" # Kubernetes version to be installed
selinux_state: permissive # SELinux state to be set on k8s nodes
timezone: "Africa/Nairobi" # Timezone to set on all nodes
k8s_cni: calico # calico, flannel
container_runtime: cri-o # docker, cri-o, containerd
configure_firewalld: true # true / false
# Docker proxy support
setup_proxy: false # Set to true to configure proxy
proxy_server: "proxy.example.com:8080" # Proxy server address and port
docker_proxy_exclude: "localhost,127.0.0.1" # Addresses to exclude from proxy
roles:
- kubernetes-bootstrapCheck Playbook syntax:
$ ansible-playbook --syntax-check -i hosts k8s-prep.yml
playbook: k8s-prep.ymlIf your SSH private key has a passphrase then save it to prevent prompts at the time of executing playbook:
eval `ssh-agent -s` && ssh-addRun the playbook to prepare your nodes:
$ ansible-playbook -i hosts k8s-prep.ymlIf the servers are accessible execution should start immediately:
PLAY [Setup Proxy] ***********************************************************************************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************************************************************************
ok: [k8s-worker-02]
ok: [k8s-worker-01]
ok: [k8s-master-03]
ok: [k8s-master-01]
ok: [k8s-worker-03]
ok: [k8s-master-02]
ok: [k8s-worker-04]
TASK [kubernetes-bootstrap : Add the OS specific variables] ******************************************************************************************************************************************
ok: [k8s-master-01] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-master-02] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-master-03] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-01] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-02] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-03] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-04] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
TASK [kubernetes-bootstrap : Put SELinux in permissive mode] *****************************************************************************************************************************************
changed: [k8s-master-01]
changed: [k8s-worker-01]
changed: [k8s-master-03]
changed: [k8s-master-02]
changed: [k8s-worker-02]
changed: [k8s-worker-03]
changed: [k8s-worker-04]
TASK [kubernetes-bootstrap : Update system packages] *************************************************************************************************************************************************Here is a screenshot with playbook execution initiated:
After the setup confirm there are no errors in the output:
...output omitted...
PLAY RECAP *******************************************************************************************************************************************************************************************
k8s-master-01 : ok=28 changed=20 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0
k8s-master-02 : ok=28 changed=20 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0
k8s-master-03 : ok=28 changed=20 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0
k8s-worker-01 : ok=27 changed=19 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0
k8s-worker-02 : ok=27 changed=19 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0
k8s-worker-03 : ok=27 changed=19 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0
k8s-worker-04 : ok=27 changed=19 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0If you encounter any errors while running the playbook reach out using the comments section and we’ll be happy to help.
Login to one of the nodes and validate below settings:
- Configured
/etc/hostsfile contents:
[root@k8s-master-01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.10 k8s-master-01.example.com k8s-master-01
192.168.200.11 k8s-master-02.example.com k8s-master-02
192.168.200.12 k8s-master-03.example.com k8s-master-03
192.168.200.13 k8s-worker-01.example.com k8s-worker-01
192.168.200.14 k8s-worker-02.example.com k8s-worker-02
192.168.200.15 k8s-worker-03.example.com k8s-worker-03
192.168.200.16 k8s-worker-04.example.com k8s-worker-04- Status of cri-o service:
[root@k8s-master-01 ~]# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-09-24 18:06:53 EAT; 11min ago
Docs: https://github.com/cri-o/cri-o
Main PID: 13445 (crio)
Tasks: 10
Memory: 41.9M
CGroup: /system.slice/crio.service
└─13445 /usr/bin/crio
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.052576977+03:00" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.111352936+03:00" level=info msg="Conmon does support the --sync option"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.111623836+03:00" level=info msg="No seccomp profile specified, using the internal default"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.111638473+03:00" level=info msg="AppArmor is disabled by the system or at CRI-O build-time"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.117006450+03:00" level=info msg="Found CNI network crio (type=bridge) at /etc/cni/net.d/100-crio-bridge.co>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.120722070+03:00" level=info msg="Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.120752984+03:00" level=info msg="Updated default CNI network name to crio"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: W0924 18:06:53.126936 13445 hostport_manager.go:71] The binary conntrack is not installed, this can cause failures in network conn>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: W0924 18:06:53.130986 13445 hostport_manager.go:71] The binary conntrack is not installed, this can cause failures in network conn>
Sep 24 18:06:53 k8s-master-01.example.com systemd[1]: Started Container Runtime Interface for OCI (CRI-O).- Configured sysctl kernel parameters
[root@k8s-master-01 ~]# sysctl -p
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1- Firewalld opened ports:
[root@k8s-master-01 ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp1s0
sources:
services: cockpit dhcpv6-client ssh
ports: 22/tcp 80/tcp 443/tcp 6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 30000-32767/tcp 4789/udp 5473/tcp 179/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:5. Bootstrap Kubernetes Control Plane
We’ll use kubeadm init command to initialize a Kubernetes control-plane node. It will start by running a series of pre-flight checks to validate the system state before making changes
Below are the key options you should be aware of:
- –apiserver-advertise-address: The IP address the API Server will advertise it’s listening on. If not set the default network interface will be used.
- –apiserver-bind-port: Port for the API Server to bind to; default is 6443
- –control-plane-endpoint: Specify a stable IP address or DNS name for the control plane.