Home Containers Install Kubernetes Cluster on Rocky Linux 8 with Kubeadm & CRI-O

Install Kubernetes Cluster on Rocky Linux 8 with Kubeadm & CRI-O

November 13, 2021

7021

In this guide, we shall be focusing on the installation of Kubernetes Cluster on Rocky Linux 8 with Kubeadm & CRI-O container runtime engine. It is no doubt that Kubernetes continues to transform how we deploy and manage business applications, at scale. Irregardless of the model used to deploy applications, whether manual or through CI/CD pipelines, Kubernetes still remains the best choice to handle orchestration, management, and scaling your applications.

For those who do not know, Kubernetes has deprecated dockershim which means they no longer support docker as the container runtime. It is for this reason we’ll be using CRI-O container runtime. CRI-O is a lightweight alternative to using Docker as the runtime for kubernetes. Before you throw tantrums, it is important to note Cri-o works similar to Docker. The way Developer build and deploy application images is not affected.

What is CRI-O?

CRI-O is an implementation of the Kubernetes CRI (Container Runtime Interface) to enable using OCI (Open Container Initiative) compatible runtimes. CRI-O allows Kubernetes to use any OCI-compliant runtime as the container runtime for running pods.

CRI-O supports OCI container images and can pull from any container registry. It is a lightweight alternative to using Docker, Moby or rkt as the runtime for Kubernetes.

CRI-O architecture pulled from official project’s website:

My Lab environment setup

I’ll deploy a Kubernetes cluster on Rocky Linux 8 set of nodes as outlined below:

3 Control plane nodes (Masters)
4 Worker node machines (Data plane)
Ansible version required 2.10+

But it is okay to setup a Kubernetes Cluster with 1 master node using the process demonstrated in this guide.

Virtual Machines and Specs

We can see all the running nodes in my lab environment using the virsh command since it is powered by Vanilla KVM:

[[email protected] ~]# virsh list
 Id   Name                   State
--------------------------------------
 4    k8s-bastion-server     running
 17   k8s-master-01-server   running
 22   k8s-master-02-server   running
 26   k8s-master-03-server   running
 31   k8s-worker-01-server   running
 36   k8s-worker-02-server   running
 38   k8s-worker-03-server   running
 41   k8s-worker-04-server   running

The first machine in the list will be our bastion / workstation machine. This is where we’ll perform the installation of Kubernetes Cluster on Rocky Linux 8 servers.

My list of servers with hostnames and IP addresses:

Hostname	IP Address	Cluster Role
k8s-bastion.example.com	192.168.200.9	Bastion host

k8s-master-01.example.com	192.168.200.10	Master Node
k8s-master-02.example.com	192.168.200.11	Master Node
k8s-master-03.example.com	192.168.200.12	Worker Node

k8s-worker-01.example.com	192.168.200.13	Worker Node
k8s-worker-02.example.com	192.168.200.14	Worker Node
k8s-worker-03.example.com	192.168.200.15	Worker Node
k8s-worker-04.example.com	192.168.200.16	Worker Node

Kubernetes Cluster Machines

Each machine in my cluster has the following hardware specifications:

# Memory
$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.5Gi       169Mi       7.0Gi       8.0Mi       285Mi       7.1Gi

# Disk space
$ df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vda4      xfs    39G  2.0G   37G   5% /

# CPU Cores
$ egrep ^processor /proc/cpuinfo  | wc -l
2

DNS Settings

Set A records for the hostnames in your DNS server and optionally on /etc/hosts file in each cluster node in case DNS resolution fails.

DNS bind records creation sample:

; Create entries for the master nodes
k8s-master-01		IN	A	192.168.200.10
k8s-master-02		IN	A	192.168.200.11
k8s-master-03		IN	A	192.168.200.12

; Create entries for the worker nodes
k8s-worker-01		IN	A	192.168.200.13
k8s-worker-02		IN	A	192.168.200.14
k8s-worker-03		IN	A	192.168.200.15

;
; The Kubernetes cluster ControlPlaneEndpoint point these to the IP of the masters
k8s-endpoint	IN	A	192.168.200.10
k8s-endpoint	IN	A	192.168.200.11
k8s-endpoint	IN	A	192.168.200.12

Step 1: Prepare Bastion Server for Kubernetes installation

Install basic tools required for the setup on the Bastion / Workstation system:

### Ubuntu / Debian ###
sudo apt update
sudo apt install git wget curl vim bash-completion

### CentOS / RHEL / Fedora / Rocky Linux ###
sudo yum -y install git wget curl vim bash-completion

Install Ansible configuration management

With Python 3:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --user

With Python 2:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py --user

Installing Ansible using pip

# Python 3
python3 -m pip install ansible --user

# Python 2
python -m pip install ansible --user

Check Ansible version after installation:

$ ansible --version
ansible [core 2.11.5]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.7 (default, Aug 30 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
  jinja version = 3.0.1
  libyaml = True

Update /etc/hosts file in your Bastion machine:

$ sudo vim /etc/hosts
192.168.200.9  k8s-bastion.example.com k8s-bastion

192.168.200.10 k8s-master-01.example.com k8s-master-01
192.168.200.11 k8s-master-02.example.com k8s-master-02
192.168.200.12 k8s-master-03.example.com k8s-master-03

192.168.200.13 k8s-worker-01.example.com k8s-worker-01
192.168.200.14 k8s-worker-02.example.com k8s-worker-02
192.168.200.15 k8s-worker-03.example.com k8s-worker-03
192.168.200.16 k8s-worker-04.example.com k8s-worker-04

Generate SSH keys:

$ ssh-keygen -t rsa -b 4096 -N ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:LwgX4oCWENqWAyW9oywAv9jTK+BEk4+XShgX0galBqE [email protected]example.com
The key's randomart image is:
+---[RSA 4096]----+
|OOo              |
|B**.             |
|EBBo. .          |
|===+ . .         |
|=*+++ . S        |
|*=++.o . .       |
|=.o. .. . .      |
| o. .    .       |
|   .             |
+----[SHA256]-----+

Create SSH client configuration file

$ vim ~/.ssh/config
Host *
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no
    IdentitiesOnly yes
    ConnectTimeout 0
    ServerAliveInterval 30

Copy SSH keys to all Kubernetes cluster nodes

# Master Nodes
for host in k8s-master-0{1..3}; do
  ssh-copy-id [email protected]$host
done

# Worker Nodes
for host in k8s-worker-0{1..4}; do
  ssh-copy-id [email protected]$host
done

Step 2: Set correct hostname on all nodes

# Examples
# Master Node 01
sudo hostnamectl set-hostname k8s-master-01.example.com

# Worker Node 01
sudo hostnamectl set-hostname k8s-worker-01.example.com

Logout then back in to confirm the hostname is set correctly:

$ hostnamectl
   Static hostname: k8s-master-01.example.com
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 7a7841970fc6fab913a02ca8ae57fe18
           Boot ID: 4388978e190a4be69eb640b31e12a63e
    Virtualization: kvm
  Operating System: Rocky Linux 8.4 (Green Obsidian)
       CPE OS Name: cpe:/o:rocky:rocky:8.4:GA
            Kernel: Linux 4.18.0-305.19.1.el8_4.x86_64
      Architecture: x86-64

For a cloud instance using cloud-init check out the guide below:

Set Server Hostname in EC2|OpenStack|DigitalOcean|Azure Instance

Step 3: Prepare Rocky Linux 8 servers for Kubernetes (Pre-reqs setup)

I wrote an Ansible role for doing the standard Kubernetes node preparation. The role contain the tasks to:

Install standard packages required to manage nodes
Setup standard system requirements – Disable Swap, Modify sysctl, Disable SELinux
Install and configure a container runtime of your Choice – cri-o, Docker, Containerd
Install the Kubernetes packages – kubelet, kubeadm and kubectl
Configure Firewalld on Kubernetes Master and Worker nodes – open all ports required

Clone my Ansible role that we’ll use to setup Kubernetes Requirements before kubeadm init to your Bastion machine:

git clone https://github.com/jmutai/k8s-pre-bootstrap.git

Switch to k8s-pre-bootstrap directory created from the clone process:

cd k8s-pre-bootstrap

Set hosts inventory correctly with your Kubernetes nodes. Here is my inventory list:

$ vim hosts
[k8snodes]
k8s-master-01
k8s-master-02
k8s-master-03
k8s-worker-01
k8s-worker-02
k8s-worker-03
k8s-worker-04

You should also update the variables in playbook file. The most important being:

Kubernetes version: k8s_version
Your timezone: timezone
Kubernetes CNI to use: k8s_cni
Container runtime: container_runtime

$ vim  k8s-prep.yml
---
- name: Setup Proxy
  hosts: k8snodes
  remote_user: root
  become: yes
  become_method: sudo
  #gather_facts: no
  vars:
    k8s_version: "1.21"                                  # Kubernetes version to be installed
    selinux_state: permissive                            # SELinux state to be set on k8s nodes
    timezone: "Africa/Nairobi"                           # Timezone to set on all nodes
    k8s_cni: calico                                      # calico, flannel
    container_runtime: cri-o                             # docker, cri-o, containerd
    configure_firewalld: true                            # true / false
    # Docker proxy support
    setup_proxy: false                                   # Set to true to configure proxy
    proxy_server: "proxy.example.com:8080"               # Proxy server address and port
    docker_proxy_exclude: "localhost,127.0.0.1"          # Addresses to exclude from proxy
  roles:
    - kubernetes-bootstrap

Check Playbook syntax:

$ ansible-playbook  --syntax-check -i hosts k8s-prep.yml

playbook: k8s-prep.yml

If your SSH private key has a passphrase then save it to prevent prompts at the time of executing playbook:

eval `ssh-agent -s` && ssh-add

Run the playbook to prepare your nodes:

$ ansible-playbook -i hosts k8s-prep.yml

If the servers are accessible execution should start immediately:

PLAY [Setup Proxy] ***********************************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************************
ok: [k8s-worker-02]
ok: [k8s-worker-01]
ok: [k8s-master-03]
ok: [k8s-master-01]
ok: [k8s-worker-03]
ok: [k8s-master-02]
ok: [k8s-worker-04]

TASK [kubernetes-bootstrap : Add the OS specific variables] ******************************************************************************************************************************************
ok: [k8s-master-01] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-master-02] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-master-03] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-01] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-02] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-03] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-04] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)

TASK [kubernetes-bootstrap : Put SELinux in permissive mode] *****************************************************************************************************************************************
changed: [k8s-master-01]
changed: [k8s-worker-01]
changed: [k8s-master-03]
changed: [k8s-master-02]
changed: [k8s-worker-02]
changed: [k8s-worker-03]
changed: [k8s-worker-04]

TASK [kubernetes-bootstrap : Update system packages] *************************************************************************************************************************************************

Here is a screenshot with playbook execution initiated:

After the setup confirm there are no errors in the output:

...output omitted...
PLAY RECAP *******************************************************************************************************************************************************************************************
k8s-master-01              : ok=28   changed=20   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
k8s-master-02              : ok=28   changed=20   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
k8s-master-03              : ok=28   changed=20   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
k8s-worker-01              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
k8s-worker-02              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
k8s-worker-03              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
k8s-worker-04              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0

If you encounter any errors while running the playbook reach out using the comments section and we’ll be happy to help.

Configured /etc/hosts file contents:

[[email protected] ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.200.10 k8s-master-01.example.com k8s-master-01
192.168.200.11 k8s-master-02.example.com k8s-master-02
192.168.200.12 k8s-master-03.example.com k8s-master-03
192.168.200.13 k8s-worker-01.example.com k8s-worker-01
192.168.200.14 k8s-worker-02.example.com k8s-worker-02
192.168.200.15 k8s-worker-03.example.com k8s-worker-03
192.168.200.16 k8s-worker-04.example.com k8s-worker-04

Status of cri-o service:

[[email protected] ~]# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-24 18:06:53 EAT; 11min ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 13445 (crio)
    Tasks: 10
   Memory: 41.9M
   CGroup: /system.slice/crio.service
           └─13445 /usr/bin/crio

Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.052576977+03:00" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.111352936+03:00" level=info msg="Conmon does support the --sync option"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.111623836+03:00" level=info msg="No seccomp profile specified, using the internal default"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.111638473+03:00" level=info msg="AppArmor is disabled by the system or at CRI-O build-time"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.117006450+03:00" level=info msg="Found CNI network crio (type=bridge) at /etc/cni/net.d/100-crio-bridge.co>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.120722070+03:00" level=info msg="Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: time="2021-09-24 18:06:53.120752984+03:00" level=info msg="Updated default CNI network name to crio"
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: W0924 18:06:53.126936   13445 hostport_manager.go:71] The binary conntrack is not installed, this can cause failures in network conn>
Sep 24 18:06:53 k8s-master-01.example.com crio[13445]: W0924 18:06:53.130986   13445 hostport_manager.go:71] The binary conntrack is not installed, this can cause failures in network conn>
Sep 24 18:06:53 k8s-master-01.example.com systemd[1]: Started Container Runtime Interface for OCI (CRI-O).

Configured sysctl kernel parameters

[[email protected] ~]# sysctl -p
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

Firewalld opened ports:

[[email protected] ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0
  sources:
  services: cockpit dhcpv6-client ssh
  ports: 22/tcp 80/tcp 443/tcp 6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 30000-32767/tcp 4789/udp 5473/tcp 179/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Step 4: Bootstrap Kubernetes Control Plane (Single / Multi-node)

We’ll use kubeadm init command to initialize a Kubernetes control-plane node. It will start by running a series of pre-flight checks to validate the system state before making changes

Below are the key options you should be aware of:

–apiserver-advertise-address: The IP address the API Server will advertise it’s listening on. If not set the default network interface will be used.
–apiserver-bind-port: Port for the API Server to bind to; default is 6443
–control-plane-endpoint: Specify a stable IP address or DNS name for the control plane.
–cri-socket: Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.
–dry-run: Don’t apply any changes; just output what would be done
–image-repository: Choose a container registry to pull control plane images from; Default: “k8s.gcr.io“
–kubernetes-version: Choose a specific Kubernetes version for the control plane.
–pod-network-cidr: Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
–service-cidr: Use alternative range of IP address for service VIPs. Default: “10.96.0.0/12“

The following table lists container runtimes and their associated socket paths:

Runtime	Path to Unix domain socket
Docker	`/var/run/dockershim.sock`
containerd	`/run/containerd/containerd.sock`
CRI-O	`/var/run/crio/crio.sock`

Checking Kubernetes Version release notes

Release notes can be found by reading the Changelog that matches your Kubernetes version

Option 1: Bootstrapping single node Control Plane Kubernetes Cluster

If you have plans to upgrade a single control-plane kubeadm cluster to high availability you should specify the --control-plane-endpoint to set the shared endpoint for all control-plane nodes.

But if this is meant for test environment with single node control plane then you can ignore –control-plane-endpoint option.

[[email protected] ~]# ssh k8s-master-01
Warning: Permanently added 'k8s-master-01' (ED25519) to the list of known hosts.
Last login: Fri Sep 24 18:07:55 2021 from 192.168.200.9
[[email protected] ~]#

Then initialize the Control Plane:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16

Deploy Calico Pod network to the cluster

Use the commands below to deploy the Calico Pod network add-on:

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

Check pods status with the following command:

[[email protected] ~]# kubectl get pods -n calico-system
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-bdd5f97c5-6856t   1/1     Running   0          60s
calico-node-vnlkf                         1/1     Running   0          60s
calico-typha-5f857549c5-hkbwq             1/1     Running   0          60s

Option 2: Bootstrapping Multi-node Control Plane Kubernetes Cluster

The --control-plane-endpoint option can be used to set the shared endpoint for all control-plane nodes. This option allows both IP addresses and DNS names that can map to IP addresses

Example of A records in Bind DNS server

; Create entries for the master nodes
k8s-master-01		IN	A	192.168.200.10
k8s-master-02		IN	A	192.168.200.11
k8s-master-03		IN	A	192.168.200.12

;
; The Kubernetes cluster ControlPlaneEndpoint point these to the IP of the masters
k8s-endpoint	IN	A	192.168.200.10
k8s-endpoint	IN	A	192.168.200.11
k8s-endpoint	IN	A	192.168.200.12

Example of A records /etc/hosts file

$ sudo vim /etc/hosts

192.168.200.10 k8s-master-01.example.com k8s-master-01
192.168.200.11 k8s-master-02.example.com k8s-master-02
192.168.200.12 k8s-master-03.example.com k8s-master-03

##  Kubernetes cluster ControlPlaneEndpoint Entries ###
192.168.200.10 k8s-endpoint.example.com  k8s-endpoint
#192.168.200.11 k8s-endpoint.example.com  k8s-endpoint
#192.168.200.12 k8s-endpoint.example.com  k8s-endpoint

Using Load Balancer IP for ControlPlaneEndpoint

The most ideal approach for HA setups is mapping ControlPlane Endpoint to a Load balancer IP. The LB will then point to the Control Plane nodes with some form of health checks.

# Entry in Bind DNS Server
k8s-endpoint	IN	A	192.168.200.8

# Entry in /etc/hosts file
192.168.200.8 k8s-endpoint.example.com  k8s-endpoint

Bootstrap Multi-node Control Plane Kubernetes Cluster

[[email protected] ~]# ssh k8s-master-01
Warning: Permanently added 'k8s-master-01' (ED25519) to the list of known hosts.
Last login: Fri Sep 24 18:07:55 2021 from 192.168.200.9
[[email protected] ~]#

Update /etc/hosts file with this node IP address and a custom DNS name that maps to this IP:

[[email protected] ~]# vim /etc/hosts
192.168.200.10 k8s-endpoint.example.com  k8s-endpoint

To initialize the control-plane node run:

[[email protected] ~]# kubeadm init \
  --pod-network-cidr=192.168.0.0/16 \
  --control-plane-endpoint=k8s-endpoint.example.com \
  --cri-socket=/var/run/crio/crio.sock \
  --upload-certs

Where:

k8s-endpoint.example.com is a valid DNS name configured for ControlPlane Endpoint
/var/run/crio/crio.sock is Cri-o runtime socket file
192.168.0.0/16 is your Pod network to be used in Kubernetes
–upload-certs Flag used ti upload the certificates that should be shared across all the control-plane instances to the cluster

If successful you’ll get an output with contents similar to this:

...output omitted...
[mark-control-plane] Marking the node k8s-master-01.example.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: p11op9.eq9vr8gq9te195b9
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-endpoint.example.com:6443 --token 78oyk4.ds1hpo2vnwg3yykt \
	--discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca \
	--control-plane --certificate-key 999110f4a07d3c430d19ca0019242f392e160216f3b91f421da1a91f1a863bba

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-endpoint.example.com:6443 --token 78oyk4.ds1hpo2vnwg3yykt \
	--discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca

Configure Kubectl as shown in the output:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Test by checking active Nodes:

[[email protected] ~]# kubectl get nodes
NAME                                  STATUS   ROLES                  AGE    VERSION
k8s-master-01.example.com             Ready    control-plane,master   4m3s   v1.22.2

Deploy Calico Pod network to the cluster

Use the commands below to deploy the Calico Pod network add-on:

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

Check pods status with the following command:

[[email protected] ~]# kubectl get pods -n calico-system -w
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-bdd5f97c5-6856t   1/1     Running   0          60s
calico-node-vnlkf                         1/1     Running   0          60s
calico-typha-5f857549c5-hkbwq             1/1     Running   0          60s

Add other control plane nodes

Add Master Node 02

[[email protected] ~]# ssh k8s-master-02
Warning: Permanently added 'k8s-master-02' (ED25519) to the list of known hosts.
Last login: Sat Sep 25 01:49:15 2021 from 192.168.200.9
[[email protected] ~]#

Update /etc/hostsfile by setting the ControlPlaneEndpoint to first control node from where bootstrap process was initiated:

[[email protected] ~]# vim /etc/hosts
192.168.200.10 k8s-endpoint.example.com   k8s-endpoint
#192.168.200.11 k8s-endpoint.example.com  k8s-endpoint
#192.168.200.12 k8s-endpoint.example.com  k8s-endpoint

I’ll use the command printed after a successful initialization:

kubeadm join k8s-endpoint.example.com:6443 --token 78oyk4.ds1hpo2vnwg3yykt \
  --discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca \
  --control-plane --certificate-key 999110f4a07d3c430d19ca0019242f392e160216f3b91f421da1a91f1a863bba

What is CRI-O?

My Lab environment setup

Virtual Machines and Specs

DNS Settings

Step 1: Prepare Bastion Server for Kubernetes installation

Install Ansible configuration management

Step 2: Set correct hostname on all nodes

Step 3: Prepare Rocky Linux 8 servers for Kubernetes (Pre-reqs setup)

Step 4: Bootstrap Kubernetes Control Plane (Single / Multi-node)

Checking Kubernetes Version release notes

Option 1: Bootstrapping single node Control Plane Kubernetes Cluster

Deploy Calico Pod network to the cluster

Option 2: Bootstrapping Multi-node Control Plane Kubernetes Cluster

Using Load Balancer IP for ControlPlaneEndpoint

Bootstrap Multi-node Control Plane Kubernetes Cluster

Deploy Calico Pod network to the cluster

Add other control plane nodes

Add Master Node 02

Add Master Node 03