Kubernetes, abbreviated as K8s is an open-source tool used to orchestrate containerized workloads to run on a cluster of hosts. It is used to automate system deployments, scale, and manage containerized applications.

Normally, Kubernetes distributes workloads across the cluster and automates the container networking needs. It also allocates storage and persistent volumes and works continuously to maintain the desired state of container applications.

There are several tools one can use to set up a Kubernetes cluster. These tools include Minikube, Kubeadm, Kubernetes on AWS (Kube-AWS), Amazon EKS e.t.c. In this guide, we will walk through how to deploy HA Kubernetes Cluster on Rocky Linux 8 using RKE2.

What is RKE2?

RKE stands for Rancher Kubernetes Engine. RKE2 also known as the (RKE Government) is a combination of RKE1 and K3s. It inherits usability, ease-of-operations, and deployment model from K3s and close alignment with upstream Kubernetes from RKE1. Normally, RKE2 doesn’t rely on docker, it launches the control plane components as static pods that are managed by the kubelet.

The diagram below will help you understand the RKE2 cluster topology.

Deploy HA Kubernetes Cluster on Rocky Linux 8 using RKE2

RKE2 ships a number of open-source components that include:

  • K3s
    • Helm Controller
  • K8s
    • API Server
    • Controller Manager
    • Kubelet
    • SchedulerSet up Linux Nodes
    • Proxy
  • etcd
  • containerd/cri
  • runc
  • Helm
  • Metrics Server
  • NGINX Ingress Controller
  • CoreDNS
  • CNI: Canal (Calico & Flannel), Cilium or Calico

System Requirements

Use a system that meets the below requirements:

  • RAM: 4GB Minimum (we recommend at least 8GB)
  • CPU: 2 Minimum (we recommend at least 4CPU)
  • 3 Rocky Linux 8 Nodes
  • Zero or more agent nodes that are designated to run your apps and services
  • A load balancer to direct front-end traffic to the three nodes.
  • A DNS record to map a URL to the load balancer

Step 1 – Set up Rocky Linux 8 Nodes

For this guide, we will use 3 Rocky Linux nodes, a load balancer, and RKE2 agents(1 or more).

TASKHOSTNAMEIP ADDRESS
Server Node 1server1.computingforgeeks.com192.168.205.2
Server Node 2server2.computingforgeeks.com192.168.205.3
Server Node 3server3.computingforgeeks.com192.168.205.33
Load Balancerrke.computingforgeeks.com192.168.205.9
Agent Node1agent1.computingforgeeks.com192.168.205.43
Agent Node2agent2.computingforgeeks.com192.168.205.44

Set the hostnames as shown:

##On Node1
sudo hostnamectl set-hostname server1.computingforgeeks.com

##On Node2
sudo hostnamectl set-hostname server2.computingforgeeks.com

##On Node3
sudo hostnamectl set-hostname server3.computingforgeeks.com

##On Loadbalancer(Node4)
sudo hostnamectl set-hostname rke.computingforgeeks.com

##On Node5
sudo hostnamectl set-hostname agent1.computingforgeeks.com

##On Node6
sudo hostnamectl set-hostname agent2.computingforgeeks.com

Add the hostnames to /etc/hosts on each node

$ sudo vim /etc/hosts
192.168.205.2 server1.computingforgeeks.com
192.168.205.3 server2.computingforgeeks.com
192.168.205.33 server3.computingforgeeks.com
192.168.205.43 agent1.computingforgeeks.com
192.168.205.44 agent2.computingforgeeks.com
192.168.205.9 rke.computingforgeeks.com

Configure the firewall on all the nodes as shown:

sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo systemctl start nftables
sudo systemctl enable nftables

Step 2 – Configure the Fixed Registration Address

To achieve high availability, you are required to set up an odd number of server plane nodes(runs etcd, the Kubernetes API, and other control plane services). The other server nodes and agent nodes need a URL they can use to register against. This is either an IP or domain name of any of the control nodes. This is mainly done to maintain quorum so that the cluster can afford to lose connection with one of the nodes without impacting the functionality cluster.

This can be achieved using the following:

  • A layer 4 (TCP) load balancer
  • Round-robin DNS
  • Virtual or elastic IP addresses

In this guide, we will configure NGINX as a layer 4 (TCP) load balancer to forward the connection to one of the RKE nodes.

Install and configure Nginx on Node4

sudo yum install nginx

Create a config file:

sudo mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak
sudo vim /etc/nginx/nginx.conf

Create a new Nginx file with the below lines replacing where required:

user nginx;
worker_processes 4;
worker_rlimit_nofile 40000;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 8192;
}

stream {
upstream backend {
        least_conn;
        server <IP_NODE_1>:9345 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:9345 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:9345 max_fails=3 fail_timeout=5s;
   }

   # This server accepts all traffic to port 9345 and passes it to the upstream. 
   # Notice that the upstream name and the proxy_pass need to match.
   server {

      listen 9345;

          proxy_pass backend;
   }
    upstream rancher_api {
        least_conn;
        server <IP_NODE_1>:6443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:6443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:6443 max_fails=3 fail_timeout=5s;
    }
        server {
        listen     6443;
        proxy_pass rancher_api;
        }
    upstream rancher_http {
        least_conn;
        server