Kubernetes is a tool for orchestrating and managing containerized applications at scale on on-premise server or across hybrid cloud environments. Kubeadm is a tool provided with Kubernetes to help users install a production ready Kubernetes cluster with best practices enforcement. This tutorial will demonstrate how you can perform an installation Kubernetes Cluster on Ubuntu 20.04 with kubeadm. Kubeadm is a command line tool that provides kubeadm init and kubeadm join as best-practice “fast paths” for creating Kubernetes clusters.

There are two server types used in deployment of Kubernetes clusters:

  • Master: A Kubernetes Master is where control API calls for the pods, replications controllers, services, nodes and other components of a Kubernetes cluster are executed.
  • Node: A Node is a system that provides the run-time environments for the containers. A set of container pods can span multiple nodes.

The minimum requirements for the viable setup are:

  • Memory: 2 GiB or more of RAM per machine
  • CPUs: At least 2 CPUs on the control plane machine.
  • Internet connectivity for pulling containers required (Private registry can also be used)
  • Full network connectivity between machines in the cluster – This is private or public

My Lab setup contain three servers. One control plane machine and two nodes to be used for running containerized workloads. You can add more nodes to suit your desired use case and load, for example using three control plane nodes for HA.

Server TypeServer HostnameSpecs
Masterk8s-master01.computingforgeeks.com4GB Ram, 2vcpus
Workerk8s-worker01.computingforgeeks.com4GB Ram, 2vcpus
Workerk8s-worker02.computingforgeeks.com4GB Ram, 2vcpus

1. Install Kubernetes Servers

Provision the servers to be used in the deployment of Kubernetes on Ubuntu 20.04. The setup process will vary depending on the virtualization or cloud environment you’re using.

Once the servers are ready, update them.

sudo apt update && sudo apt -y full-upgrade
[ -f /var/run/reboot-required ] && sudo reboot -f

2. Install kubelet, kubeadm and kubectl

Once the servers are rebooted, add Kubernetes repository for Ubuntu 20.04 to all the servers.

Specify the version of Kubernetes to be installed.

VER=1.30

Then import GPG key and configure APT repository.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v${VER}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v${VER}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Then install required packages.

sudo apt update
sudo apt -y install vim git curl wget kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Confirm installation by checking the version of kubectl.

$ kubectl version --client && kubeadm version
Client Version: v1.29.6+k3s2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.3", GitCommit:"6fc0a69044f1ac4c13841ec4391224a2df241460", GitTreeState:"clean", BuildDate:"2024-07-16T23:53:15Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}

3. Disable Swap

Turn off swap.

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Now disable Linux swap space permanently in /etc/fstab. Search for a swap line and add # (hashtag) sign in front of the line.

$ sudo vim /etc/fstab
#/swap.img	none	swap	sw	0	0

Confirm setting is correct

sudo swapoff -a
sudo mount -a
free -h

Enable kernel modules and configure sysctl.

# Enable kernel modules
sudo modprobe overlay
sudo modprobe br_netfilter

# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload sysctl
sudo sysctl --system

4. Install Container runtime

To run containers in Pods, Kubernetes uses a container runtime. Supported container runtimes are:

  • Docker
  • CRI-O
  • Containerd

NOTE: You have to choose one runtime at a time.

Option 1: Install and Use Docker CE runtime:

# Add repo and Install packages
sudo apt update
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker-archive-keyring.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt install -y containerd.io docker-ce docker-ce-cli

# Create required directories
sudo mkdir -p /etc/systemd/system/docker.service.d

# Create daemon json config file
sudo tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# Start and enable Services
sudo systemctl daemon-reload 
sudo systemctl restart docker
sudo systemctl enable docker

For Docker Engine you need a shim interface. You can install Mirantis cri-dockerd as covered in the guide below.

Mirantis cri-dockerd CRI socket file path is /run/cri-dockerd.sock. This is what will be used when configuring Kubernetes cluster.

Option 2: Install and Use CRI-O:

Switch to root user

sudo su -

The run the commands below.

# Ensure you load modules
sudo modprobe overlay
sudo modprobe br_netfilter
# Set up required sysctl params
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# Reload sysctl
sudo sysctl --system
# Add Cri-o repo
CRIO_VERSION=v1.30
curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key|gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" |sudo tee /etc/apt/sources.list.d/cri-o.list
# Install CRI-O
sudo apt update
sudo apt install cri-o -y
# Update CRI-O CIDR subnet
sudo sed -i 's/10.85.0.0/172.24.0.0/g' /etc/cni/net.d/*
# Start and enable Service
sudo systemctl daemon-reload
sudo systemctl restart crio
sudo systemctl enable crio
sudo systemctl status crio

Option 3: Install and Use Containerd:

# Configure persistent loading of modules
sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

# Load at runtime
sudo modprobe overlay
sudo modprobe br_netfilter

# Ensure sysctl params are set
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload configs
sudo sysctl --system

# Install required packages
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

# Add Docker repo
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker-archive-keyring.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Install containerd
sudo apt update
sudo apt install -y containerd.io

# Configure containerd and start service
sudo mkdir -p /etc/containerd
sudo containerd config default|sudo tee /etc/containerd/config.toml

# restart containerd
sudo systemctl restart containerd
sudo systemctl enable containerd
systemctl status  containerd

To use the systemd cgroup driver, set plugins.cri.systemd_cgroup = true in /etc/containerd/config.toml. When using kubeadm, manually configure the cgroup driver for kubelet

5. Initialize master node

Login to the server to be used as master and make sure that the br_netfilter module is loaded:

$ lsmod | grep br_netfilter
br_netfilter           22256  0 
bridge                151336  2 br_netfilter,ebtable_broute

Enable kubelet service.

sudo systemctl enable kubelet

We now want to initialize the machine that will run the control plane components which includes etcd (the cluster database) and the API Server.

Pull container images:

$ sudo kubeadm config images pull
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.30.3
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.30.3
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.30.3
[config/images] Pulled registry.k8s.io/kube-proxy:v1.30.3
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.11.1
[config/images] Pulled registry.k8s.io/pause:3.9
[config/images] Pulled registry.k8s.io/etcd:3.5.12-0

If you have multiple CRI sockets, please use --cri-socket to select one:

# CRI-O
sudo kubeadm config images pull --cri-socket unix:///var/run/crio/crio.sock

# Containerd
sudo kubeadm config images pull --cri-socket unix:///run/containerd/containerd.sock

# Docker
sudo kubeadm config images pull --cri-socket unix:///run/cri-dockerd.sock 

These are the basic kubeadm init options that are used to bootstrap cluster.

--control-plane-endpoint :  set the shared endpoint for all control-plane nodes. Can be DNS/IP
--pod-network-cidr : Used to set a Pod network add-on CIDR
--cri-socket : Use if have more than one container runtime to set runtime socket path
--apiserver-advertise-address : Set advertise address for this particular control-plane node's API server

Bootstrap without shared endpoint

To bootstrap a cluster without using DNS endpoint, run:

### With CRI-O###
sudo sysctl -p
sudo kubeadm init \
  --pod-network-cidr=172.24.0.0/16 \
  --cri-socket unix:///var/run/crio/crio.sock

### With Containerd ###
sudo sysctl -p
sudo kubeadm init \
  --pod-network-cidr=172.24.0.0/16 \
  --cri-socket unix:///run/containerd/containerd.sock

### With Docker CE ###
sudo sysctl -p
sudo kubeadm init \
  --pod-network-cidr=172.24.0.0/16 \
  --cri-socket unix:///run/cri-dockerd.sock 

Bootstrap with shared endpoint (DNS name for control plane API)

Set cluster endpoint DNS name or add record to /etc/hosts file.

$ sudo vim /etc/hosts
172.29.20.5 k8s-cluster.computingforgeeks.com

Create cluster:

sudo sysctl -p
sudo kubeadm init \
  --pod-network-cidr=172.24.0.0/16 \
  --upload-certs \
  --control-plane-endpoint=k8s-cluster.computingforgeeks.com

Note: If 172.24.0.0/16 is already in use within your network you must select a different pod network CIDR, replacing 172.24.0.0/16 in the above command.

Container runtime sockets:

RuntimePath to Unix domain socket
Dockerunix:///run/cri-dockerd.sock
containerdunix:///run/containerd/containerd.sock
CRI-Ounix:///var/run/crio/crio.sock

You can optionally pass Socket file for runtime and advertise address depending on your setup.

# CRI-O
sudo kubeadm init \
  --pod-network-cidr=172.24.0.0/16 \
  --cri-socket unix:///var/run/crio/crio.sock \
  --upload-certs \
  --control-plane-endpoint=k8s-cluster.computingforgeeks.com

# Containerd
sudo kubeadm init \
  --pod-network-cidr=172.24.0.0/16 \
  --cri-socket unix:///run/containerd/containerd.sock \
  --upload-certs \
  --control-plane-endpoint=k8s-cluster.computingforgeeks.com

# Docker
# Must do https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/
sudo kubeadm init \
  --pod-network-cidr=172.24.0.0/16 \
  --cri-socket unix:///run/cri-dockerd.sock  \
  --upload-certs \
  --control-plane-endpoint=k8s-cluster.computingforgeeks.com

Here is the output of my initialization command.

....
[init] Using Kubernetes version: v1.30.3
[preflight] Running pre-flight checks
	[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0611 22:34:23.276374    4726 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0611 22:34:23.278380    4726 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.008181 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master01.computingforgeeks.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master01.computingforgeeks.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: zoy8cq.6v349sx9ass8dzyj
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join k8s-cluster.computingforgeeks.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
    --discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18 \
    --control-plane 

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-cluster.computingforgeeks.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
    --discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18

Configure kubectl using commands in the output:

mkdir -p $HOME/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Check cluster status:

$ kubectl cluster-info
Kubernetes master is running at https://k8s-cluster.computingforgeeks.com:6443
KubeDNS is running at https://k8s-cluster.computingforgeeks.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Additional Master nodes can be added using the command in installation output:

kubeadm join k8s-cluster.computingforgeeks.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
    --discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18 \
    --control-plane

6. Install network plugin on Master

In this guide we’ll use Calico. You can choose any other supported network plugins.

Download operator and custom resource files. See releases page for latest version.

VER=$(curl -s https://api.github.com/repos/projectcalico/calico/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
curl -O https://raw.githubusercontent.com/projectcalico/calico/v$VER/manifests/tigera-operator.yaml
curl -O https://raw.githubusercontent.com/projectcalico/calico/v$VER/manifests/custom-resources.yaml 

First, install the operator on your cluster.

$ kubectl create -f tigera-operator.yaml
namespace/tigera-operator created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created

If you wish to customize the Calico install, customize the downloaded custom-resources.yaml. For example we are updating CIDR.

sed -ie 's/192.168.0.0/172.24.0.0/g' custom-resources.yaml

Use the manifest locally then install Calico:

$ kubectl create -f custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created

Confirm that all of the pods are running:

$ kubectl get pods --all-namespaces -w
NAMESPACE         NAME                               READY   STATUS    RESTARTS   AGE
kube-system       coredns-5d78c9869d-5fmlg           1/1     Running   0          9m43s
kube-system       coredns-5d78c9869d-gtlwr           1/1     Running   0          9m43s
kube-system       etcd-focal                         1/1     Running   0          10m
kube-system       kube-apiserver-focal               1/1     Running   0          9m58s
kube-system       kube-controller-manager-focal      1/1     Running   0          9m59s
kube-system       kube-proxy-pgvnf                   1/1     Running   0          9m43s
kube-system       kube-scheduler-focal               1/1     Running   0          9m57s
tigera-operator   tigera-operator-549d4f9bdb-gfdcf   1/1     Running   0          91s

For Single node cluster allow Pods to run on master nodes:

kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl taint nodes --all  node-role.kubernetes.io/control-plane-

Confirm master node is ready:

# CRI-O
$ kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
ubuntu   Ready    control-plane,master   38s   v1.30.3   143.198.114.46   <none>        Ubuntu 20.04.3 LTS   5.4.0-88-generic   cri-o://1.30.3

# Containerd
$ kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
ubuntu   Ready    control-plane,master   15m   v1.30.3   143.198.114.46   <none>        Ubuntu 20.04.3 LTS   5.4.0-88-generic   containerd://1.7.19

# Docker
$ kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
k8s-master01   Ready    master   64m   v1.30.3   135.181.28.113   <none>        Ubuntu 20.04 LTS   5.4.0-37-generic   docker://27.1.1

7. Add worker nodes to the cluster

With the control plane ready you can add worker nodes to the cluster for running scheduled workloads.

If endpoint address is not in DNS, add record to /etc/hosts.

$ sudo vim /etc/hosts
172.29.20.5 k8s-cluster.computingforgeeks.com

The join command that was given is used to add a worker node to the cluster.

kubeadm join k8s-cluster.computingforgeeks.com:6443 \
  --token sr4l2l.2kvot0pfalh5o4ik \
  --discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18

Output:

[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.30" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run below command on the control-plane to see if the node joined the cluster.

$ kubectl get nodes
NAME                                 STATUS   ROLES    AGE   VERSION
k8s-master01.computingforgeeks.com   Ready    master   10m   v1.30.3
k8s-worker01.computingforgeeks.com   Ready    <none>   50s   v1.30.3
k8s-worker02.computingforgeeks.com   Ready    <none>   12s   v1.30.3

$ kubectl get nodes -o wide

If the join token is expired, refer to our guide on how to join worker nodes.

8. Deploy application on cluster

If you only have a single node cluster, check our guide on how to run container pods on master nodes:

We need to validate that our cluster is working by deploying an application.

kubectl apply -f https://k8s.io/examples/pods/commands.yaml

Check to see if pod started

$ kubectl get pods
NAME           READY   STATUS      RESTARTS   AGE
command-demo   0/1     Completed   0          16s

9. Install Kubernetes Dashboard (Optional)

Kubernetes dashboard can be used to deploy containerized applications to a Kubernetes cluster, troubleshoot your containerized application, and manage the cluster resources.

Refer to our guide for installation: How To Install Kubernetes Dashboard with NodePort

10. Install Metrics Server

Metrics Server is a cluster-wide aggregator of resource usage data. It collects metrics from the Summary API, exposed by Kubelet on each node. Use our guide below to deploy it:

11. Deploy Prometheus / Grafana Monitoring

Prometheus is a full fledged solution that enables you to access advanced metrics capabilities in a Kubernetes cluster. Grafana is used for analytics and interactive visualization of metrics that’s collected and stored in Prometheus database. We have a complete guide on how to setup complete monitoring stack on Kubernetes Cluster:

12. Persistent Storage Configuration ideas (Optional)

If you’re also looking for a Persistent storage solution for your Kubernetes, checkout:

13. Install Nginx Ingress Controller

If Nginx is your preferred Ingress controller for Kubernetes workloads, you can use our guide for the installation process:

14. Deploy MetalLB Load Balancer

Follow the guide below to install and configure MetalLB on Kubernetes:

More guides:

Similar Kubernetes deployment guides:

82 COMMENTS

  1. Several issues in these steps. Trying to follow with containerd setup – an example error is no step to create /var/lib/kubelet/config.yaml. Another, it enters into a root bash but doesn’t exit. And kubeadm config images pull leads to ‘unknown service runtime.v1alpha2.ImageService’ error.

  2. This not working:
    sudo kubeadm config images pull
    failed to pull image “k8s.gcr.io/kube-apiserver:v1.20.5″: output: time=”2021-03-25T12:37:30Z” level=fatal msg=”pulling image failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService”
    , error: exit status 1
    To see the stack trace of this error execute with –v=5 or higher

    • Hi Guido, Because the command install kubelet doesn’t determine the version, if you follow command of this guide, it will install the last version of kubelet, it doesn’t support on Ubuntu 20.4.
      Replace command install kubelet, kubeadm, kubectl as below:
      apt update && apt install -y vim git curl wget kubeadm=1.22.1-00 kubelet=1.22.1-00 kubectl=1.22.1-00

      Hope it’s helpful for you.

    • Hi Guido, Because the command install kubelet doesn’t determine the version, if you follow command of this guide, it will install the last version of kubelet, it doesn’t support on Ubuntu 20.4.
      Replace command install kubelet, kubeadm, kubectl as below:
      apt update && apt install -y vim git curl wget kubeadm=1.22.1-00 kubelet=1.22.1-00 kubectl=1.22.1-00

      Hope it is helpful for you.

  3. after reproducing the steps mentioned I found out that kubectl get nodes does not show the nodes
    and shows the error The connection to the server localhost:8080 was refused – did you specify the right host or port?
    Any help asap would be highly appreciated

    • Follow below commands and execute the “kubectl get nodes” using non-root user
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

  4. Some issues getting this to work. Here are the items I changed to get it going:

    First, there was an issue getting containerd to start. The issue was that the config was not saved properly. In the “Installing containerd” section above, there was a missing `>` in the following command (I really hope my pre and /pre tags work):

    containerd config default /etc/containerd/config.toml

    It needs to be:

    containerd config default > /etc/containerd/config.toml

    Then there was an issue with flannel but that resolved itself:
    https://github.com/flannel-io/flannel/issues/1482

    Then, once the control plane node was up, the worker nodes would not go to ready state. I resolved this by setting the containerd.default_runtime (Again, I really hope my pre and /pre tags work):

    [plugins.”io.containerd.grpc.v1.cri”.containerd.default_runtime]
    runtime_type = “io.containerd.runtime.v1.linux”

    Hopefully this helps someone else…

  5. Where does the IP of “cluster endpoint DNS name” come from? You’re using 172.29.20.5. My controlplane’s IP is 172.17.0.1. Should I use that?

  6. Hey Everyone,

    Great article, however I have a question where are the steps for installing kubenetes on the worker nodes do you follow the same procedure as the Controller Node? Were do you stop to get the worker node properly funtional?

    Sorry I am new to kubernetes
    Thanks,
    Michael

  7. Following your instructions completely using the CRIO install, no pods can access the internet. The master comes up fine and the nodes come up fine and I can do deployments and everything.

    pods just can not access the internet. I tried to setup jenkins and it was not able to access the internet to install the plugins

    Something is flawed in your instructions

    • Hi Chris.

      I noted the issue was with CRIO subnet defined in /etc/cni/net.d/100-crio-bridge.conf config file. It is different from Pod network defined when bootstrapping k8s cluster.

      You can retry with updated article. Basically change the subnet with command below:

      sudo sed -i ‘s/10.85.0.0/192.168.0.0/g’ /etc/cni/net.d/100-crio-bridge.conf

      Then restart crio service and bootstrap cluster. I hope this helps.

      • my home network is already running on 192.168.0 subnet. I thought you could not use a pod network the same as an already in use subnet?

      • sudo sed -i ‘s/10.85.0.0/192.168.0.0/g’ /etc/cni/net.d/100-crio-bridge.conf
        sed: can’t read /etc/cni/net.d/100-crio-bridge.conf: No such file or directory

        root@kube-master:~# more /etc/cni/net.d/100-crio-bridge.conf
        more: stat of /etc/cni/net.d/100-crio-bridge.conf failed: No such file or directory

  8. I tried all 3 container management types in this tutorial and none of the pods can access the internet. They can talk to each other but nothing in the outside world. I installed Jenkins in kubernetes and I need to install the plugins but can’t because if this issue. Please advise

  9. I am confused by setting the IP for cluster endpoint.
    The below is my network. What IP can i use for endpoint?

    4: docker0: mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:71:4d:87:43 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
    valid_lft forever preferred_lft forever

  10. Clear article!

    But when I run:
    sudo kubeadm config images pull –cri-socket /var/run/docker.sock

    Then I get this response:
    failed to pull image “k8s.gcr.io/kube-apiserver:v1.22.4″: output: time=”2021-12-02T20:00:52Z” level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
    , error: exit status 1
    To see the stack trace of this error execute with –v=5 or higher

    Does anyone know how to fix this?

    All the previous steps were executed succesfully.

      • Hi Josphat, Thank for writing this article. But I am facing similiar issue. Docker service is up for me but still not able to connect to this CRI-socket

        user “test” has been added to group docker and changes applied with “newGrp docker” command.

        K8s Version : 1.22
        test@l2030017515:~$ docker version
        Client: Docker Engine – Community
        Version: 19.03.15
        API version: 1.40
        Go version: go1.13.15
        Git commit: 99e3ed8919
        Built: Sat Jan 30 03:17:01 2021
        OS/Arch: linux/amd64
        Experimental: false

        Server: Docker Engine – Community
        Engine:
        Version: 19.03.15
        API version: 1.40 (minimum version 1.12)
        Go version: go1.13.15
        Git commit: 99e3ed8919
        Built: Sat Jan 30 03:15:30 2021
        OS/Arch: linux/amd64
        Experimental: false
        containerd:
        Version: 1.4.13
        GitCommit: 9cc61520f4cd876b86e77edfeb88fbcd536d1f9d
        runc:
        Version: 1.0.3
        GitCommit: v1.0.3-0-gf46b6ba
        docker-init:
        Version: 0.18.0
        GitCommit: fec3683
        test@l2030017515:~$

        test@l2030017515:~$ sudo systemctl status docker
        ● docker.service – Docker Application Container Engine
        Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
        Active: active (running) since Tue 2022-03-08 13:23:10 IST; 18s ago
        TriggeredBy: ● docker.socket
        Docs: https://docs.docker.com
        Main PID: 589981 (dockerd)
        Tasks: 14
        Memory: 39.1M
        CGroup: /system.slice/docker.service
        └─589981 /usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock

        Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.549821158+05:30″ level=warning msg=”Your kernel does not support cgroup rt runtime”
        Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.549833221+05:30″ level=warning msg=”Your kernel does not support cgroup blkio weight”
        Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.549844854+05:30″ level=warning msg=”Your kernel does not support cgroup blkio weight_device”
        Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.550067148+05:30″ level=info msg=”Loading containers: start.”
        Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.930384654+05:30″ level=info msg=”Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option>
        Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.065481959+05:30″ level=info msg=”Loading containers: done.”
        Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.431063768+05:30″ level=info msg=”Docker daemon” commit=99e3ed8919 graphdriver(s)=overlay2 version=19.03.15
        Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.431230178+05:30″ level=info msg=”Daemon has completed initialization”
        Mar 08 13:23:10 l2030017515 systemd[1]: Started Docker Application Container Engine.
        Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.602166953+05:30″ level=info msg=”API listen on /run/docker.sock”

        Error :

        test@l2030017515:~$ sudo kubeadm init –apiserver-advertise-address=$IP –pod-network-cidr=192.168.0.0/16 –cri-socket /run/docker.sock –upload-certs
        I0308 13:23:46.930188 590324 version.go:255] remote version is much newer: v1.23.4; falling back to: stable-1.22
        [init] Using Kubernetes version: v1.22.7
        [preflight] Running pre-flight checks
        error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time=”2022-03-08T13:23:52+05:30″ level=fatal msg=”connect: connect endpoint ‘unix:///run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
        , error: exit status 1
        [preflight] If you know what you are doing, you can make a check non-fatal with `–ignore-preflight-errors=…`
        To see the stack trace of this error execute with –v=5 or higher

        Would be great of you suggest a resolution for the same.

        br

      • sysadmin@sysadmin-virtual-machine:~$ systemctl status docker
        ● docker.service – Docker Application Container Engine
        Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
        Active: active (running) since Fri 2022-04-15 13:56:06 WIB; 22min ago
        TriggeredBy: ● docker.socket
        Docs: https://docs.docker.com
        Main PID: 8624 (dockerd)
        Tasks: 19
        Memory: 44.3M
        CGroup: /system.slice/docker.service
        └─8624 /usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock

        Apr 15 13:56:18 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:18.628776532+07:00″ level=info msg=”ignoring event” container=97a7af489a4131d6ccd4022ab9>
        Apr 15 13:56:21 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:21.884168954+07:00″ level=info msg=”ignoring event” container=e98e6eefe307e1161257876fbf>
        Apr 15 13:56:21 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:21.920915826+07:00″ level=info msg=”ignoring event” container=7b98046840f84ac9f6148e3670>
        Apr 15 13:56:25 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:25.571995048+07:00″ level=info msg=”ignoring event” container=b27cd9e8045fdb6145b4afd634>
        Apr 15 13:56:25 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:25.614309354+07:00″ level=info msg=”ignoring event” container=8a11505137a45fddf3df7c2cec>
        Apr 15 13:56:28 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:28.878086757+07:00″ level=info msg=”ignoring event” container=4ec09c5edf4939b4f567b82d0d>
        Apr 15 13:56:29 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:29.892410616+07:00″ level=info msg=”ignoring event” container=00207f98f2cb6276e58d9509bc>
        Apr 15 13:56:31 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:31.952429432+07:00″ level=info msg=”ignoring event” container=f218022d2f2c50fb3a63401f29>
        Apr 15 13:56:32 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:32.992268870+07:00″ level=info msg=”ignoring event” container=a29f524abb16b085cdbd4af591>
        Apr 15 13:56:35 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:35.043274360+07:00″ level=info msg=”ignoring event” container=8c3a0f063ddcfc5e4f2cb606a3>
        lines 1-21/21 (END)
        ^C
        sysadmin@sysadmin-virtual-machine:~$ sudo kubeadm config images pull –cri-socket /var/run/docker.sock
        [sudo] password for sysadmin:
        failed to pull image “k8s.gcr.io/kube-apiserver:v1.23.5″: output: time=”2022-04-15T14:19:22+07:00″ level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
        , error: exit status 1
        To see the stack trace of this error execute with –v=5 or higher
        sysadmin@sysadmin-virtual-machine:~$

        still same problem, docker status “running”

  11. The certificate created in the init command is invalid and flagged by Chrome/Safari when trying to progress through step 10 installing the dashboard.

  12. I ran into this problem. See console log, docker is installed and running:

    jacob@lazykub1:~$ sudo kubeadm config images pull –cri-socket /var/run/docker.sock
    failed to pull image “k8s.gcr.io/kube-apiserver:v1.23.1″: output: time=”2022-01-16T14:14:45Z” level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
    , error: exit status 1
    To see the stack trace of this error execute with –v=5 or higher
    jacob@lazykub1:~$ sudo systemctl status docker
    ● docker.service – Docker Application Container Engine
    Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Active: active (running) since Sun 2022-01-16 14:14:21 UTC; 3min 23s ago
    TriggeredBy: ● docker.socket
    Docs: https://docs.docker.com
    Main PID: 4187 (dockerd)
    Tasks: 8
    Memory: 33.7M
    CGroup: /system.slice/docker.service
    └─4187 /usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock

    Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947413897Z” level=warning msg=”Your kernel does not support CPU realtime scheduler”
    Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947421586Z” level=warning msg=”Your kernel does not support cgroup blkio weight”
    Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947428108Z” level=warning msg=”Your kernel does not support cgroup blkio weight_device”
    Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947609517Z” level=info msg=”Loading containers: start.”
    Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.077788214Z” level=info msg=”Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option –bip can be used to set a preferred IP address”
    Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.111888405Z” level=info msg=”Loading containers: done.”
    Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.133944179Z” level=info msg=”Docker daemon” commit=459d0df graphdriver(s)=overlay2 version=20.10.12
    Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.134263733Z” level=info msg=”Daemon has completed initialization”
    Jan 16 14:14:21 lazykub1 systemd[1]: Started Docker Application Container Engine.
    Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.154710559Z” level=info msg=”API listen on /run/docker.sock”
    jacob@lazykub1:~$

  13. swap statement is not working for me.
    “sudo sed -i ‘/ swap / s/^\(.*\)$/#\1/g’ /etc/fstab”
    It does not comment out the swap line

    # /etc/fstab: static file system information.
    #
    # Use ‘blkid’ to print the universally unique identifier for a
    # device; this may be used with UUID= as a more robust way to name devices
    # that works even if disks are added and removed. See fstab(5).
    #
    #
    # / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-LMZ6lL63Lq08X4WdpxYG1q2BXTyEcLs2cS2oIdfaWHUALXM3rHgG4j26DG8Hys2M / ext4 defaults 0 1
    # /boot was on /dev/sda2 during curtin installation
    /dev/disk/by-uuid/d13a56c4-40f8-4994-8b23-06b8f2785f5d /boot ext4 defaults 0 1
    /swap.img none swap sw 0 0

  14. Thank you for the tutorial!!!

    After some tries finally seems I got it working, now I’m looking how to integrate on GitLab.

    An quick hint for those with problems on coredns failing or dont get ready.

    Don’t change the range IP, post says its possible to customize and use seed for change after install, all time I have did that, pod never started, reading the file:

    https://docs.projectcalico.org/manifests/custom-resources.yaml

    There’s the note:

    # Note: The ipPools section cannot be modified post-install.

    And the default range is 192.168.0.0/16

  15. sudo sed -i ‘s/10.85.0.0/192.168.0.0/g’ /etc/cni/net.d/100-crio-bridge.conf
    sed: can’t read /etc/cni/net.d/100-crio-bridge.conf: No such file or directory

  16. Also seeing the following error which then stops anything further on working:

    sudo kubeadm config images pull –cri-socket /var/run/docker.sock
    failed to pull image “k8s.gcr.io/kube-apiserver:v1.23.3″: output: time=”2022-01-31T16:26:47Z” level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
    , error: exit status 1
    To see the stack trace of this error execute with –v=5 or higher

    any ideas?

  17. Hello
    I have an issue with this tutorial : My install is in 1.24.0. All works good but i have a problem with tokens in service account. Any token is generated when i create sa, and i have no token in the default sa. Can you help me ?

    Thx

  18. Hello, i have an issue with this guide.
    I have successfully installed my cluster with containerd and calico.
    Every works good except one thing :
    I haven’t any token generated for services account. When i type this command :
    kubectl get sa
    The default sa is present but with 0 secrets. The token section is set to
    It’s similar when i tried to create a new sa, no token.
    Can i have help please ?

    Thanks

  19. IF you need to specify a different pod cidr as “192.168.0.0/16” you need to edit the custom-resources.yaml for calico before applying.

  20. Hi,
    A great post, which helped me a lot.
    I have been trying to host k8 on AWS EC2 Instances ( an task )

    i have encountered two issues

    1# at worker node

    Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the ‘criSocket’ field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock
    To see the stack trace of this error execute with –v=5 or higher

    for this i have tried
    sudo kubeadm join {ip address} –token xgr4i8.kokoqjxo6yvs3d19 –discovery-token-ca-cert-hash sha256:d1b4e8ea7a87ae6b91198a2c0ac85663c48b2f2f9181d7075d13294336684379
    –cri-socket /run/cri-dockerd.sock

    but my command is being struck

    2#
    [preflight] Running pre-flight checks
    [WARNING SystemVerification]: missing optional cgroups: blkio

    with above message its been stuck, help me please

  21. I installed the K8s cluster (1 master, 2 worker) using this guide. I used docker as Container Runtime. However, when I create Pods, the Pods in different worker nodes take overlapping Ips. Pod from one node can’t talk to pod from another node.

  22. Thank you very much.
    If I’d add something to make this perfect is to add the details on how to generate the join token to add new workers.

  23. I followed this guide (docker runtime) with an additional step for Calico:

    kubectl apply -f https://projectcalico.docs.tigera.io/v3.23/manifests/calico.yaml

    An outstanding issue I have is even though I specified a pod network with kubeadm init, the pods are getting assigned addresses from the docker0 network (172.17.0.0/16) instead of the pod network.

    Any ideas how to resolve? I found a few similar issues on Stackoverflow but haven’t been able to resolve mine.

  24. I have problem about docker container cannot access external calls.
    I build a simple application which do api call and it says “resource temporarily unavailable”, after trying to exec into container and do apt-get update it says that cannot connect to deb.
    Any help about this ? i setup cluster as described above.

  25. Very nice article !
    I did manage to setup k8s but recently i started getting errors in my pods “resource temporarily unavailable” when doing external http call, anyone have some solution ?

    btw, i tried to exec into container and do “apt-get update” but it says that i cannot connect to deb.

  26. After installing the CRI-O runtime and using Calico as suggested, my calico-kube-controller is stuck in a Pending status.

    NAMESPACE NAME READY STATUS RESTARTS AGE
    calico-system calico-kube-controllers-657d56796-bvh25 0/1 Pending 0 30m
    calico-system calico-node-tdkjf 1/1 Running 0 30m
    calico-system calico-typha-89d87cb5c-fn887 1/1 Running 0 30m
    kube-system coredns-6d4b75cb6d-dbbmk 1/1 Running 0 31m
    kube-system coredns-6d4b75cb6d-twlkk 1/1 Running 0 31m
    kube-system etcd-res-u20-template 1/1 Running 0 32m
    kube-system kube-apiserver-res-u20-template 1/1 Running 0 32m
    kube-system kube-controller-manager-res-u20-template 1/1 Running 0 32m
    kube-system kube-proxy-cxmvt 1/1 Running 0 31m
    kube-system kube-scheduler-res-u20-template 1/1 Running 0 32m
    tigera-operator tigera-operator-6995cc5df5-4w9rt 1/1 Running 0 30m

    The only thing in the log that drew my attention was
    “E0808 23:12:25.258702 1 disruption.go:534] Error syncing PodDisruptionBudget calico-system/calico-typha, requeuing: Operation cannot be fulfilled on poddisruptionbudgets.policy “calico-typha”: the object has been modified; please apply your changes to the latest version and try again”

    Has anyone seen this issue? suggestions?

  27. After installing the CRI-O runtime and the Calico network plugin, the calico-kube-controllers stay in a `Pending` state.

    senften@res-k8s-master:~$ kubectl get pods –all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE
    calico-system calico-kube-controllers-657d56796-qzhxw 0/1 Pending 0 17h
    calico-system calico-node-62k9g 1/1 Running 0 17h
    calico-system calico-typha-8fdfcb55c-k6zcq 1/1 Running 0 17h
    kube-system coredns-6d4b75cb6d-bknbp 1/1 Running 0 17h
    kube-system coredns-6d4b75cb6d-jkfk7 1/1 Running 0 17h
    kube-system etcd-res-k8s-master 1/1 Running 0 17h
    kube-system kube-apiserver-res-k8s-master 1/1 Running 0 17h
    kube-system kube-controller-manager-res-k8s-master 1/1 Running 0 17h
    kube-system kube-proxy-jht5p 1/1 Running 0 17h
    kube-system kube-scheduler-res-k8s-master 1/1 Running 0 17h
    tigera-operator tigera-operator-6995cc5df5-j64s6 1/1 Running 0 17h

    I think it may be related to this bug, [calico-kube-controllers pending since not tolerate the “node-role.kubernetes.io/control-plane” · Issue #6087 · projectcalico/calico](https://github.com/projectcalico/calico/issues/6087), but, after reading through the issue, I am still uncertain how to proceed or even sure this is the issue I’ve run into.

    Any advice? Anyone else seeing this?
    Thanks

  28. While running this command
    sudo kubeadm config images pull –cri-socket unix:///run/cri-dockerd.sock
    i m getting this error
    failed to pull image “registry.k8s.io/kube-apiserver:v1.25.2″: output: time=”2022-10-13T11:52:55+05:30″ level=fatal msg=”unable to determine image API version: rpc error: code = Unavailable desc = connection error: desc = \”transport: Error while dialing dial unix /run/cri-dockerd.sock: connect: no such file or directory\””
    , error: exit status 1
    To see the stack trace of this error execute with –v=5 or higher

  29. i don’t know why i m not able to create file for docker at the given file i m following all steps
    Whenever i run this command sudo kubeadm config images pull –cri-socket unix:///run/cri-dockerd.sock
    i m getting this following error failed to pull image “registry.k8s.io/kube-apiserver:v1.25.2″: output: time=”2022-10-13T12:29:44+05:30″ level=fatal msg=”unable to determine image API version: rpc error: code = Unavailable desc = connection error: desc = \”transport: Error while dialing dial unix /run/cri-dockerd.sock: connect: no such file or directory\””
    , error: exit status 1
    To see the stack trace of this error execute with –v=5 or higher

    Any solution ??

  30. thanks for ur blogs its helps most of them and reilf more stress .
    Here im beginer to these .whats the use of docker . kubernates run upon or docker , or docker container runs inside the pod ? plzzz tell me im confused about all these help me out plzzz

  31. I followed your guide :

    Host OS: Ubuntu 22.04
    CNI and version: Flannel latest
    CRI and version: CRI-O latest

    kubeadm init –v=5 –pod-network-cidr=10.244.0.0/16 –upload-certs –control-plane-endpoint=k8s.mydomain.com

    The cluster initialized successfully but coredns pods cannot started

    kubectl version –short
    Client Version: v1.26.2
    Kustomize Version: v4.5.7
    Server Version: v1.26.2

    kubectl cluster-info
    Kubernetes control plane is running at https://k8s.mydomain.com:6443
    CoreDNS is running at https://k8s.mydomain.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

    kubectl get pod -n kube-system
    NAME READY STATUS RESTARTS AGE
    coredns-787d4945fb-g8hhj 0/1 CreateContainerError 0 7m1s
    coredns-787d4945fb-nkssp 0/1 CreateContainerError 0 6m29s
    etcd-k8s-52ts-master1 1/1 Running 1 12h
    kube-apiserver-k8s-52ts-master1 1/1 Running 1 12h
    kube-controller-manager-k8s-52ts-master1 1/1 Running 1 12h
    kube-proxy-m4mhc 1/1 Running 1 12h
    kube-proxy-rrdpb 1/1 Running 0 12h
    kube-scheduler-k8s-52ts-master1 1/1 Running 1 12h

    kubectl describe pod coredns-787d4945fb-g8hhj -n kube-system
    Events:
    Type Reason Age From Message
    —- —— —- —- ——-
    Normal Scheduled 65s default-scheduler Successfully assigned kube-system/coredns-787d4945fb-g8hhj to k8s-52ts-worker1
    Warning Failed 64s kubelet Error: container create failed: time=”2023-03-17T08:26:01+07:00″ level=warning msg=”unable to get oom kill count” error=”openat2 /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45fe767_098b_401f_8da9_1a1760d804f6.slice/crio-105d630363803602c6bfe9c1516b128192b95c519ff321ed1177fe0cacdc9b42.scope/memory.events: no such file or directory”
    time=”2023-03-17T08:26:01+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
    Warning Failed 63s kubelet Error: container create failed: time=”2023-03-17T08:26:02+07:00″ level=warning msg=”unable to get oom kill count” error=”openat2 /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45fe767_098b_401f_8da9_1a1760d804f6.slice/crio-2bbebc0f6e74a3c5a8f5e460e758e322a94ffe97964f34e357d831ce3fced345.scope/memory.events: no such file or directory”
    time=”2023-03-17T08:26:02+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
    Warning Failed 47s kubelet Error: container create failed: time=”2023-03-17T08:26:18+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
    Warning Failed 34s kubelet Error: container create failed: time=”2023-03-17T08:26:31+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
    Warning Failed 19s kubelet Error: container create failed: time=”2023-03-17T08:26:46+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
    Normal Pulled 9s (x6 over 64s) kubelet Container image “registry.k8s.io/coredns/coredns:v1.9.3” already present on machine

  32. Hi ,
    Thanks for this great tutorial, It’s really useful and helped me to bootstrap multiple clusters since 2022.
    I want to mention two points and hope that it will help someone.
    1 – For containerd, the old debian package 1.4 does not work with v1.25 and higher. the version that works it 1.6 . Here is one of the related errors :
    failed to pull image “registry.k8s.io/kube-apiserver:v1.25.10″: output: time=”2023-05-21T10:16:39Z” level=fatal msg=”validate service connection: CRI v1 image API is not implemented for endpoint \”unix:///var/run/containerd/containerd.sock\”: rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService”

    2 – The Urls on calico setup are not working anymore , the updated version could be found here :
    https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises

  33. Hi Josphat
    Thanks for this guide, very detail and helpful.
    Adding one more thing to the Calico installation part
    curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml -O
    as the resources.yaml seems using 192.168.0.0/16 as default cidr, but your cluster init command above is using –pod-network-cidr=172.24.0.0/16
    If one want to deploy Calico network plugin, he must manually update the cidr in this yaml file to have
    cidr: 172.24.0.0/16

LEAVE A REPLY

Please enter your comment!
Please enter your name here