Kubernetes is a tool for orchestrating and managing containerized applications at scale on on-premise server or across hybrid cloud environments. Kubeadm is a tool provided with Kubernetes to help users install a production ready Kubernetes cluster with best practices enforcement. This tutorial will demonstrate how you can perform an installation Kubernetes Cluster on Ubuntu 20.04 with kubeadm. Kubeadm is a command line tool that provides kubeadm init and kubeadm join as best-practice “fast paths” for creating Kubernetes clusters.
There are two server types used in deployment of Kubernetes clusters:
- Master: A Kubernetes Master is where control API calls for the pods, replications controllers, services, nodes and other components of a Kubernetes cluster are executed.
- Node: A Node is a system that provides the run-time environments for the containers. A set of container pods can span multiple nodes.
The minimum requirements for the viable setup are:
- Memory: 2 GiB or more of RAM per machine
- CPUs: At least 2 CPUs on the control plane machine.
- Internet connectivity for pulling containers required (Private registry can also be used)
- Full network connectivity between machines in the cluster – This is private or public
My Lab setup contain three servers. One control plane machine and two nodes to be used for running containerized workloads. You can add more nodes to suit your desired use case and load, for example using three control plane nodes for HA.
| Server Type | Server Hostname | Specs |
| Master | k8s-master01.computingforgeeks.com | 4GB Ram, 2vcpus |
| Worker | k8s-worker01.computingforgeeks.com | 4GB Ram, 2vcpus |
| Worker | k8s-worker02.computingforgeeks.com | 4GB Ram, 2vcpus |
1. Install Kubernetes Servers
Provision the servers to be used in the deployment of Kubernetes on Ubuntu 20.04. The setup process will vary depending on the virtualization or cloud environment you’re using.
Once the servers are ready, update them.
sudo apt update && sudo apt -y full-upgrade
[ -f /var/run/reboot-required ] && sudo reboot -f
2. Install kubelet, kubeadm and kubectl
Once the servers are rebooted, add Kubernetes repository for Ubuntu 20.04 to all the servers.
Specify the version of Kubernetes to be installed.
VER=1.30
Then import GPG key and configure APT repository.
curl -fsSL https://pkgs.k8s.io/core:/stable:/v${VER}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v${VER}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
Then install required packages.
sudo apt update
sudo apt -y install vim git curl wget kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
Confirm installation by checking the version of kubectl.
$ kubectl version --client && kubeadm version
Client Version: v1.29.6+k3s2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.3", GitCommit:"6fc0a69044f1ac4c13841ec4391224a2df241460", GitTreeState:"clean", BuildDate:"2024-07-16T23:53:15Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}
3. Disable Swap
Turn off swap.
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
Now disable Linux swap space permanently in /etc/fstab. Search for a swap line and add # (hashtag) sign in front of the line.
$ sudo vim /etc/fstab
#/swap.img none swap sw 0 0
Confirm setting is correct
sudo swapoff -a
sudo mount -a
free -h
Enable kernel modules and configure sysctl.
# Enable kernel modules
sudo modprobe overlay
sudo modprobe br_netfilter
# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# Reload sysctl
sudo sysctl --system
4. Install Container runtime
To run containers in Pods, Kubernetes uses a container runtime. Supported container runtimes are:
- Docker
- CRI-O
- Containerd
NOTE: You have to choose one runtime at a time.
Option 1: Install and Use Docker CE runtime:
# Add repo and Install packages
sudo apt update
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker-archive-keyring.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt install -y containerd.io docker-ce docker-ce-cli
# Create required directories
sudo mkdir -p /etc/systemd/system/docker.service.d
# Create daemon json config file
sudo tee /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# Start and enable Services
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker
For Docker Engine you need a shim interface. You can install Mirantis cri-dockerd as covered in the guide below.
Mirantis cri-dockerd CRI socket file path is /run/cri-dockerd.sock. This is what will be used when configuring Kubernetes cluster.
Option 2: Install and Use CRI-O:
Switch to root user
sudo su -
The run the commands below.
# Ensure you load modules
sudo modprobe overlay
sudo modprobe br_netfilter
# Set up required sysctl params
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# Reload sysctl
sudo sysctl --system
# Add Cri-o repo
CRIO_VERSION=v1.30
curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key|gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" |sudo tee /etc/apt/sources.list.d/cri-o.list
# Install CRI-O
sudo apt update
sudo apt install cri-o -y
# Update CRI-O CIDR subnet
sudo sed -i 's/10.85.0.0/172.24.0.0/g' /etc/cni/net.d/*
# Start and enable Service
sudo systemctl daemon-reload
sudo systemctl restart crio
sudo systemctl enable crio
sudo systemctl status crio
Option 3: Install and Use Containerd:
# Configure persistent loading of modules
sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
# Load at runtime
sudo modprobe overlay
sudo modprobe br_netfilter
# Ensure sysctl params are set
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# Reload configs
sudo sysctl --system
# Install required packages
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates
# Add Docker repo
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker-archive-keyring.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# Install containerd
sudo apt update
sudo apt install -y containerd.io
# Configure containerd and start service
sudo mkdir -p /etc/containerd
sudo containerd config default|sudo tee /etc/containerd/config.toml
# restart containerd
sudo systemctl restart containerd
sudo systemctl enable containerd
systemctl status containerd
To use the systemd cgroup driver, set plugins.cri.systemd_cgroup = true in /etc/containerd/config.toml. When using kubeadm, manually configure the cgroup driver for kubelet
5. Initialize master node
Login to the server to be used as master and make sure that the br_netfilter module is loaded:
$ lsmod | grep br_netfilter
br_netfilter 22256 0
bridge 151336 2 br_netfilter,ebtable_broute
Enable kubelet service.
sudo systemctl enable kubelet
We now want to initialize the machine that will run the control plane components which includes etcd (the cluster database) and the API Server.
Pull container images:
$ sudo kubeadm config images pull
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.30.3
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.30.3
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.30.3
[config/images] Pulled registry.k8s.io/kube-proxy:v1.30.3
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.11.1
[config/images] Pulled registry.k8s.io/pause:3.9
[config/images] Pulled registry.k8s.io/etcd:3.5.12-0
If you have multiple CRI sockets, please use --cri-socket to select one:
# CRI-O
sudo kubeadm config images pull --cri-socket unix:///var/run/crio/crio.sock
# Containerd
sudo kubeadm config images pull --cri-socket unix:///run/containerd/containerd.sock
# Docker
sudo kubeadm config images pull --cri-socket unix:///run/cri-dockerd.sock
These are the basic kubeadm init options that are used to bootstrap cluster.
--control-plane-endpoint : set the shared endpoint for all control-plane nodes. Can be DNS/IP --pod-network-cidr : Used to set a Pod network add-on CIDR --cri-socket : Use if have more than one container runtime to set runtime socket path --apiserver-advertise-address : Set advertise address for this particular control-plane node's API server
Bootstrap without shared endpoint
To bootstrap a cluster without using DNS endpoint, run:
### With CRI-O###
sudo sysctl -p
sudo kubeadm init \
--pod-network-cidr=172.24.0.0/16 \
--cri-socket unix:///var/run/crio/crio.sock
### With Containerd ###
sudo sysctl -p
sudo kubeadm init \
--pod-network-cidr=172.24.0.0/16 \
--cri-socket unix:///run/containerd/containerd.sock
### With Docker CE ###
sudo sysctl -p
sudo kubeadm init \
--pod-network-cidr=172.24.0.0/16 \
--cri-socket unix:///run/cri-dockerd.sock
Bootstrap with shared endpoint (DNS name for control plane API)
Set cluster endpoint DNS name or add record to /etc/hosts file.
$ sudo vim /etc/hosts
172.29.20.5 k8s-cluster.computingforgeeks.com
Create cluster:
sudo sysctl -p
sudo kubeadm init \
--pod-network-cidr=172.24.0.0/16 \
--upload-certs \
--control-plane-endpoint=k8s-cluster.computingforgeeks.com
Note: If 172.24.0.0/16 is already in use within your network you must select a different pod network CIDR, replacing 172.24.0.0/16 in the above command.
Container runtime sockets:
| Runtime | Path to Unix domain socket |
|---|---|
| Docker | unix:///run/cri-dockerd.sock |
| containerd | unix:///run/containerd/containerd.sock |
| CRI-O | unix:///var/run/crio/crio.sock |
You can optionally pass Socket file for runtime and advertise address depending on your setup.
# CRI-O
sudo kubeadm init \
--pod-network-cidr=172.24.0.0/16 \
--cri-socket unix:///var/run/crio/crio.sock \
--upload-certs \
--control-plane-endpoint=k8s-cluster.computingforgeeks.com
# Containerd
sudo kubeadm init \
--pod-network-cidr=172.24.0.0/16 \
--cri-socket unix:///run/containerd/containerd.sock \
--upload-certs \
--control-plane-endpoint=k8s-cluster.computingforgeeks.com
# Docker
# Must do https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/
sudo kubeadm init \
--pod-network-cidr=172.24.0.0/16 \
--cri-socket unix:///run/cri-dockerd.sock \
--upload-certs \
--control-plane-endpoint=k8s-cluster.computingforgeeks.com
Here is the output of my initialization command.
....
[init] Using Kubernetes version: v1.30.3
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0611 22:34:23.276374 4726 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0611 22:34:23.278380 4726 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.008181 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master01.computingforgeeks.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master01.computingforgeeks.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: zoy8cq.6v349sx9ass8dzyj
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join k8s-cluster.computingforgeeks.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
--discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-cluster.computingforgeeks.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
--discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18
Configure kubectl using commands in the output:
mkdir -p $HOME/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Check cluster status:
$ kubectl cluster-info
Kubernetes master is running at https://k8s-cluster.computingforgeeks.com:6443
KubeDNS is running at https://k8s-cluster.computingforgeeks.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Additional Master nodes can be added using the command in installation output:
kubeadm join k8s-cluster.computingforgeeks.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
--discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18 \
--control-plane
6. Install network plugin on Master
In this guide we’ll use Calico. You can choose any other supported network plugins.
Download operator and custom resource files. See releases page for latest version.
VER=$(curl -s https://api.github.com/repos/projectcalico/calico/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
curl -O https://raw.githubusercontent.com/projectcalico/calico/v$VER/manifests/tigera-operator.yaml
curl -O https://raw.githubusercontent.com/projectcalico/calico/v$VER/manifests/custom-resources.yaml
First, install the operator on your cluster.
$ kubectl create -f tigera-operator.yaml
namespace/tigera-operator created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created
If you wish to customize the Calico install, customize the downloaded custom-resources.yaml. For example we are updating CIDR.
sed -ie 's/192.168.0.0/172.24.0.0/g' custom-resources.yaml
Use the manifest locally then install Calico:
$ kubectl create -f custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
Confirm that all of the pods are running:
$ kubectl get pods --all-namespaces -w
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5d78c9869d-5fmlg 1/1 Running 0 9m43s
kube-system coredns-5d78c9869d-gtlwr 1/1 Running 0 9m43s
kube-system etcd-focal 1/1 Running 0 10m
kube-system kube-apiserver-focal 1/1 Running 0 9m58s
kube-system kube-controller-manager-focal 1/1 Running 0 9m59s
kube-system kube-proxy-pgvnf 1/1 Running 0 9m43s
kube-system kube-scheduler-focal 1/1 Running 0 9m57s
tigera-operator tigera-operator-549d4f9bdb-gfdcf 1/1 Running 0 91s
For Single node cluster allow Pods to run on master nodes:
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
Confirm master node is ready:
# CRI-O
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ubuntu Ready control-plane,master 38s v1.30.3 143.198.114.46 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic cri-o://1.30.3
# Containerd
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ubuntu Ready control-plane,master 15m v1.30.3 143.198.114.46 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.7.19
# Docker
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master01 Ready master 64m v1.30.3 135.181.28.113 <none> Ubuntu 20.04 LTS 5.4.0-37-generic docker://27.1.1
7. Add worker nodes to the cluster
With the control plane ready you can add worker nodes to the cluster for running scheduled workloads.
If endpoint address is not in DNS, add record to /etc/hosts.
$ sudo vim /etc/hosts
172.29.20.5 k8s-cluster.computingforgeeks.com
The join command that was given is used to add a worker node to the cluster.
kubeadm join k8s-cluster.computingforgeeks.com:6443 \
--token sr4l2l.2kvot0pfalh5o4ik \
--discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18
Output:
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.30" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run below command on the control-plane to see if the node joined the cluster.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01.computingforgeeks.com Ready master 10m v1.30.3
k8s-worker01.computingforgeeks.com Ready <none> 50s v1.30.3
k8s-worker02.computingforgeeks.com Ready <none> 12s v1.30.3
$ kubectl get nodes -o wide
If the join token is expired, refer to our guide on how to join worker nodes.
8. Deploy application on cluster
If you only have a single node cluster, check our guide on how to run container pods on master nodes:
We need to validate that our cluster is working by deploying an application.
kubectl apply -f https://k8s.io/examples/pods/commands.yaml
Check to see if pod started
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
command-demo 0/1 Completed 0 16s
9. Install Kubernetes Dashboard (Optional)
Kubernetes dashboard can be used to deploy containerized applications to a Kubernetes cluster, troubleshoot your containerized application, and manage the cluster resources.
Refer to our guide for installation: How To Install Kubernetes Dashboard with NodePort
10. Install Metrics Server
Metrics Server is a cluster-wide aggregator of resource usage data. It collects metrics from the Summary API, exposed by Kubelet on each node. Use our guide below to deploy it:
11. Deploy Prometheus / Grafana Monitoring
Prometheus is a full fledged solution that enables you to access advanced metrics capabilities in a Kubernetes cluster. Grafana is used for analytics and interactive visualization of metrics that’s collected and stored in Prometheus database. We have a complete guide on how to setup complete monitoring stack on Kubernetes Cluster:
12. Persistent Storage Configuration ideas (Optional)
If you’re also looking for a Persistent storage solution for your Kubernetes, checkout:
- How To Deploy Rook Ceph Storage on Kubernetes Cluster
- Ceph Persistent Storage for Kubernetes with Cephfs
- Persistent Storage for Kubernetes with Ceph RBD
- How To Configure Kubernetes Dynamic Volume Provisioning With Heketi & GlusterFS
- Deploy and Use OpenEBS Container Storage on Kubernetes
13. Install Nginx Ingress Controller
If Nginx is your preferred Ingress controller for Kubernetes workloads, you can use our guide for the installation process:
14. Deploy MetalLB Load Balancer
Follow the guide below to install and configure MetalLB on Kubernetes:
More guides:
Similar Kubernetes deployment guides:
- Install Kubernetes Cluster on CentOS 7 with kubeadm
- Install Production Kubernetes Cluster with Rancher RKE
- How To Deploy Lightweight Kubernetes Cluster in 5 minutes with K3s
- Deploy Production Ready Kubernetes Cluster with Ansible & Kubespray





































































Several issues in these steps. Trying to follow with containerd setup – an example error is no step to create /var/lib/kubelet/config.yaml. Another, it enters into a root bash but doesn’t exit. And kubeadm config images pull leads to ‘unknown service runtime.v1alpha2.ImageService’ error.
This not working:
sudo kubeadm config images pull
failed to pull image “k8s.gcr.io/kube-apiserver:v1.20.5″: output: time=”2021-03-25T12:37:30Z” level=fatal msg=”pulling image failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService”
, error: exit status 1
To see the stack trace of this error execute with –v=5 or higher
Hi Guido, Because the command install kubelet doesn’t determine the version, if you follow command of this guide, it will install the last version of kubelet, it doesn’t support on Ubuntu 20.4.
Replace command install kubelet, kubeadm, kubectl as below:
apt update && apt install -y vim git curl wget kubeadm=1.22.1-00 kubelet=1.22.1-00 kubectl=1.22.1-00
Hope it’s helpful for you.
Hi Guido, Because the command install kubelet doesn’t determine the version, if you follow command of this guide, it will install the last version of kubelet, it doesn’t support on Ubuntu 20.4.
Replace command install kubelet, kubeadm, kubectl as below:
apt update && apt install -y vim git curl wget kubeadm=1.22.1-00 kubelet=1.22.1-00 kubectl=1.22.1-00
Hope it is helpful for you.
Its working. thanks.
Awesome working example! Thanks.
Welcome!
after reproducing the steps mentioned I found out that kubectl get nodes does not show the nodes
and shows the error The connection to the server localhost:8080 was refused – did you specify the right host or port?
Any help asap would be highly appreciated
Follow below commands and execute the “kubectl get nodes” using non-root user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
To install Calico I had to:
curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml
Steps above did not work.
Any error when using steps in this guide?
yes
root@k-master:~# sudo apt install cri-o cri-o-runc
Reading package lists… Done
Building dependency tree
Reading state information… Done
E: Unable to locate package cri-o
kindly check
Did you follow steps under # Add Cri-o repo?
I followed using containerd and looks like @ihc have right. With his help i was able to run cluster
I just came here to say, Thank you ihc. Your comment saved me headche.
Some issues getting this to work. Here are the items I changed to get it going:
First, there was an issue getting containerd to start. The issue was that the config was not saved properly. In the “Installing containerd” section above, there was a missing `>` in the following command (I really hope my pre and /pre tags work):
containerd config default /etc/containerd/config.toml
It needs to be:
containerd config default > /etc/containerd/config.toml
Then there was an issue with flannel but that resolved itself:
https://github.com/flannel-io/flannel/issues/1482
Then, once the control plane node was up, the worker nodes would not go to ready state. I resolved this by setting the containerd.default_runtime (Again, I really hope my pre and /pre tags work):
[plugins.”io.containerd.grpc.v1.cri”.containerd.default_runtime]
runtime_type = “io.containerd.runtime.v1.linux”
Hopefully this helps someone else…
Thanks for the perfect working example and it worked for me.
Where does the IP of “cluster endpoint DNS name” come from? You’re using 172.29.20.5. My controlplane’s IP is 172.17.0.1. Should I use that?
Hey,
It will be an A record pointing to 172.17.0.1.
Hey Everyone,
Great article, however I have a question where are the steps for installing kubenetes on the worker nodes do you follow the same procedure as the Controller Node? Were do you stop to get the worker node properly funtional?
Sorry I am new to kubernetes
Thanks,
Michael
Hi,
It is captured in step 7.
Following your instructions completely using the CRIO install, no pods can access the internet. The master comes up fine and the nodes come up fine and I can do deployments and everything.
pods just can not access the internet. I tried to setup jenkins and it was not able to access the internet to install the plugins
Something is flawed in your instructions
Hi Chris.
I noted the issue was with CRIO subnet defined in /etc/cni/net.d/100-crio-bridge.conf config file. It is different from Pod network defined when bootstrapping k8s cluster.
You can retry with updated article. Basically change the subnet with command below:
sudo sed -i ‘s/10.85.0.0/192.168.0.0/g’ /etc/cni/net.d/100-crio-bridge.conf
Then restart crio service and bootstrap cluster. I hope this helps.
my home network is already running on 192.168.0 subnet. I thought you could not use a pod network the same as an already in use subnet?
sudo sed -i ‘s/10.85.0.0/192.168.0.0/g’ /etc/cni/net.d/100-crio-bridge.conf
sed: can’t read /etc/cni/net.d/100-crio-bridge.conf: No such file or directory
root@kube-master:~# more /etc/cni/net.d/100-crio-bridge.conf
more: stat of /etc/cni/net.d/100-crio-bridge.conf failed: No such file or directory
I tried all 3 container management types in this tutorial and none of the pods can access the internet. They can talk to each other but nothing in the outside world. I installed Jenkins in kubernetes and I need to install the plugins but can’t because if this issue. Please advise
See previous comment
Thanks for the reply.
I see the same issue when I build with docker or containerd as well
I’ve been having the same issue too. I’m using docker and none of my worker nodes can access the internet.
I am confused by setting the IP for cluster endpoint.
The below is my network. What IP can i use for endpoint?
4: docker0: mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:71:4d:87:43 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
It is mapped to primary interface IP address of your control plane node.
Thank you very much! the guide is very helpful and i managed to setup the K8 cluster.
Clear article!
But when I run:
sudo kubeadm config images pull –cri-socket /var/run/docker.sock
Then I get this response:
failed to pull image “k8s.gcr.io/kube-apiserver:v1.22.4″: output: time=”2021-12-02T20:00:52Z” level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
, error: exit status 1
To see the stack trace of this error execute with –v=5 or higher
Does anyone know how to fix this?
All the previous steps were executed succesfully.
Please check if Docker service is up – systemctl status docker
Hi Josphat, Thank for writing this article. But I am facing similiar issue. Docker service is up for me but still not able to connect to this CRI-socket
user “test” has been added to group docker and changes applied with “newGrp docker” command.
K8s Version : 1.22
test@l2030017515:~$ docker version
Client: Docker Engine – Community
Version: 19.03.15
API version: 1.40
Go version: go1.13.15
Git commit: 99e3ed8919
Built: Sat Jan 30 03:17:01 2021
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine – Community
Engine:
Version: 19.03.15
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 99e3ed8919
Built: Sat Jan 30 03:15:30 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.13
GitCommit: 9cc61520f4cd876b86e77edfeb88fbcd536d1f9d
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.18.0
GitCommit: fec3683
test@l2030017515:~$
test@l2030017515:~$ sudo systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-03-08 13:23:10 IST; 18s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 589981 (dockerd)
Tasks: 14
Memory: 39.1M
CGroup: /system.slice/docker.service
└─589981 /usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock
Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.549821158+05:30″ level=warning msg=”Your kernel does not support cgroup rt runtime”
Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.549833221+05:30″ level=warning msg=”Your kernel does not support cgroup blkio weight”
Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.549844854+05:30″ level=warning msg=”Your kernel does not support cgroup blkio weight_device”
Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.550067148+05:30″ level=info msg=”Loading containers: start.”
Mar 08 13:23:09 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:09.930384654+05:30″ level=info msg=”Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option>
Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.065481959+05:30″ level=info msg=”Loading containers: done.”
Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.431063768+05:30″ level=info msg=”Docker daemon” commit=99e3ed8919 graphdriver(s)=overlay2 version=19.03.15
Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.431230178+05:30″ level=info msg=”Daemon has completed initialization”
Mar 08 13:23:10 l2030017515 systemd[1]: Started Docker Application Container Engine.
Mar 08 13:23:10 l2030017515 dockerd[589981]: time=”2022-03-08T13:23:10.602166953+05:30″ level=info msg=”API listen on /run/docker.sock”
Error :
test@l2030017515:~$ sudo kubeadm init –apiserver-advertise-address=$IP –pod-network-cidr=192.168.0.0/16 –cri-socket /run/docker.sock –upload-certs
I0308 13:23:46.930188 590324 version.go:255] remote version is much newer: v1.23.4; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.7
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR CRI]: container runtime is not running: output: time=”2022-03-08T13:23:52+05:30″ level=fatal msg=”connect: connect endpoint ‘unix:///run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `–ignore-preflight-errors=…`
To see the stack trace of this error execute with –v=5 or higher
Would be great of you suggest a resolution for the same.
br
sysadmin@sysadmin-virtual-machine:~$ systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-04-15 13:56:06 WIB; 22min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 8624 (dockerd)
Tasks: 19
Memory: 44.3M
CGroup: /system.slice/docker.service
└─8624 /usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock
Apr 15 13:56:18 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:18.628776532+07:00″ level=info msg=”ignoring event” container=97a7af489a4131d6ccd4022ab9>
Apr 15 13:56:21 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:21.884168954+07:00″ level=info msg=”ignoring event” container=e98e6eefe307e1161257876fbf>
Apr 15 13:56:21 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:21.920915826+07:00″ level=info msg=”ignoring event” container=7b98046840f84ac9f6148e3670>
Apr 15 13:56:25 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:25.571995048+07:00″ level=info msg=”ignoring event” container=b27cd9e8045fdb6145b4afd634>
Apr 15 13:56:25 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:25.614309354+07:00″ level=info msg=”ignoring event” container=8a11505137a45fddf3df7c2cec>
Apr 15 13:56:28 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:28.878086757+07:00″ level=info msg=”ignoring event” container=4ec09c5edf4939b4f567b82d0d>
Apr 15 13:56:29 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:29.892410616+07:00″ level=info msg=”ignoring event” container=00207f98f2cb6276e58d9509bc>
Apr 15 13:56:31 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:31.952429432+07:00″ level=info msg=”ignoring event” container=f218022d2f2c50fb3a63401f29>
Apr 15 13:56:32 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:32.992268870+07:00″ level=info msg=”ignoring event” container=a29f524abb16b085cdbd4af591>
Apr 15 13:56:35 sysadmin-virtual-machine dockerd[8624]: time=”2022-04-15T13:56:35.043274360+07:00″ level=info msg=”ignoring event” container=8c3a0f063ddcfc5e4f2cb606a3>
lines 1-21/21 (END)
^C
sysadmin@sysadmin-virtual-machine:~$ sudo kubeadm config images pull –cri-socket /var/run/docker.sock
[sudo] password for sysadmin:
failed to pull image “k8s.gcr.io/kube-apiserver:v1.23.5″: output: time=”2022-04-15T14:19:22+07:00″ level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
, error: exit status 1
To see the stack trace of this error execute with –v=5 or higher
sysadmin@sysadmin-virtual-machine:~$
still same problem, docker status “running”
If you only installed Docker runtime, you don’t need to run this command again. You just need sudo kubeadm config images pull
The certificate created in the init command is invalid and flagged by Chrome/Safari when trying to progress through step 10 installing the dashboard.
Woooowwwwwwww !!! You have been enriching the POST, I congratulate you and very complete … !!!
Thanks for the positive vibe you’re dropping our way. We’re humbled.
I ran into this problem. See console log, docker is installed and running:
jacob@lazykub1:~$ sudo kubeadm config images pull –cri-socket /var/run/docker.sock
failed to pull image “k8s.gcr.io/kube-apiserver:v1.23.1″: output: time=”2022-01-16T14:14:45Z” level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
, error: exit status 1
To see the stack trace of this error execute with –v=5 or higher
jacob@lazykub1:~$ sudo systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-01-16 14:14:21 UTC; 3min 23s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 4187 (dockerd)
Tasks: 8
Memory: 33.7M
CGroup: /system.slice/docker.service
└─4187 /usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock
Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947413897Z” level=warning msg=”Your kernel does not support CPU realtime scheduler”
Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947421586Z” level=warning msg=”Your kernel does not support cgroup blkio weight”
Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947428108Z” level=warning msg=”Your kernel does not support cgroup blkio weight_device”
Jan 16 14:14:20 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:20.947609517Z” level=info msg=”Loading containers: start.”
Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.077788214Z” level=info msg=”Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option –bip can be used to set a preferred IP address”
Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.111888405Z” level=info msg=”Loading containers: done.”
Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.133944179Z” level=info msg=”Docker daemon” commit=459d0df graphdriver(s)=overlay2 version=20.10.12
Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.134263733Z” level=info msg=”Daemon has completed initialization”
Jan 16 14:14:21 lazykub1 systemd[1]: Started Docker Application Container Engine.
Jan 16 14:14:21 lazykub1 dockerd[4187]: time=”2022-01-16T14:14:21.154710559Z” level=info msg=”API listen on /run/docker.sock”
jacob@lazykub1:~$
are you manage to solve your problem? iam also facing same problem
I see your socket path is /run/containerd/containerd.sock
swap statement is not working for me.
“sudo sed -i ‘/ swap / s/^\(.*\)$/#\1/g’ /etc/fstab”
It does not comment out the swap line
# /etc/fstab: static file system information.
#
# Use ‘blkid’ to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-LMZ6lL63Lq08X4WdpxYG1q2BXTyEcLs2cS2oIdfaWHUALXM3rHgG4j26DG8Hys2M / ext4 defaults 0 1
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/d13a56c4-40f8-4994-8b23-06b8f2785f5d /boot ext4 defaults 0 1
/swap.img none swap sw 0 0
Hi,
What are the contents of your fstab file?
Thank you for the tutorial!!!
After some tries finally seems I got it working, now I’m looking how to integrate on GitLab.
An quick hint for those with problems on coredns failing or dont get ready.
Don’t change the range IP, post says its possible to customize and use seed for change after install, all time I have did that, pod never started, reading the file:
https://docs.projectcalico.org/manifests/custom-resources.yaml
There’s the note:
# Note: The ipPools section cannot be modified post-install.
And the default range is 192.168.0.0/16
sudo sed -i ‘s/10.85.0.0/192.168.0.0/g’ /etc/cni/net.d/100-crio-bridge.conf
sed: can’t read /etc/cni/net.d/100-crio-bridge.conf: No such file or directory
Also seeing the following error which then stops anything further on working:
sudo kubeadm config images pull –cri-socket /var/run/docker.sock
failed to pull image “k8s.gcr.io/kube-apiserver:v1.23.3″: output: time=”2022-01-31T16:26:47Z” level=fatal msg=”connect: connect endpoint ‘unix:///var/run/docker.sock’, make sure you are running as root and the endpoint has been started: context deadline exceeded”
, error: exit status 1
To see the stack trace of this error execute with –v=5 or higher
any ideas?
Are you using Docker container engine?. If so, is the service running?
We’ve updated the guide to use cri-dockerd shim. Also covered in a separate guide: https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/
Hello
I have an issue with this tutorial : My install is in 1.24.0. All works good but i have a problem with tokens in service account. Any token is generated when i create sa, and i have no token in the default sa. Can you help me ?
Thx
Hello, i have an issue with this guide.
I have successfully installed my cluster with containerd and calico.
Every works good except one thing :
I haven’t any token generated for services account. When i type this command :
kubectl get sa
The default sa is present but with 0 secrets. The token section is set to
It’s similar when i tried to create a new sa, no token.
Can i have help please ?
Thanks
IF you need to specify a different pod cidr as “192.168.0.0/16” you need to edit the custom-resources.yaml for calico before applying.
Hi,
A great post, which helped me a lot.
I have been trying to host k8 on AWS EC2 Instances ( an task )
i have encountered two issues
1# at worker node
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the ‘criSocket’ field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock
To see the stack trace of this error execute with –v=5 or higher
for this i have tried
sudo kubeadm join {ip address} –token xgr4i8.kokoqjxo6yvs3d19 –discovery-token-ca-cert-hash sha256:d1b4e8ea7a87ae6b91198a2c0ac85663c48b2f2f9181d7075d13294336684379
–cri-socket /run/cri-dockerd.sock
but my command is being struck
2#
[preflight] Running pre-flight checks
[WARNING SystemVerification]: missing optional cgroups: blkio
with above message its been stuck, help me please
I installed the K8s cluster (1 master, 2 worker) using this guide. I used docker as Container Runtime. However, when I create Pods, the Pods in different worker nodes take overlapping Ips. Pod from one node can’t talk to pod from another node.
Took me a few attempts but got there in the end! Thanks for taking the time to put together.
Welcome Phil
Thank you very much.
If I’d add something to make this perfect is to add the details on how to generate the join token to add new workers.
There is a link to a separate guide under “Join new Kubernetes Worker Node to an existing Cluster” on joining new node to an existing cluster.
I followed this guide (docker runtime) with an additional step for Calico:
kubectl apply -f https://projectcalico.docs.tigera.io/v3.23/manifests/calico.yaml
An outstanding issue I have is even though I specified a pod network with kubeadm init, the pods are getting assigned addresses from the docker0 network (172.17.0.0/16) instead of the pod network.
Any ideas how to resolve? I found a few similar issues on Stackoverflow but haven’t been able to resolve mine.
I have problem about docker container cannot access external calls.
I build a simple application which do api call and it says “resource temporarily unavailable”, after trying to exec into container and do apt-get update it says that cannot connect to deb.
Any help about this ? i setup cluster as described above.
Very nice article !
I did manage to setup k8s but recently i started getting errors in my pods “resource temporarily unavailable” when doing external http call, anyone have some solution ?
btw, i tried to exec into container and do “apt-get update” but it says that i cannot connect to deb.
After installing the CRI-O runtime and using Calico as suggested, my calico-kube-controller is stuck in a Pending status.
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-657d56796-bvh25 0/1 Pending 0 30m
calico-system calico-node-tdkjf 1/1 Running 0 30m
calico-system calico-typha-89d87cb5c-fn887 1/1 Running 0 30m
kube-system coredns-6d4b75cb6d-dbbmk 1/1 Running 0 31m
kube-system coredns-6d4b75cb6d-twlkk 1/1 Running 0 31m
kube-system etcd-res-u20-template 1/1 Running 0 32m
kube-system kube-apiserver-res-u20-template 1/1 Running 0 32m
kube-system kube-controller-manager-res-u20-template 1/1 Running 0 32m
kube-system kube-proxy-cxmvt 1/1 Running 0 31m
kube-system kube-scheduler-res-u20-template 1/1 Running 0 32m
tigera-operator tigera-operator-6995cc5df5-4w9rt 1/1 Running 0 30m
The only thing in the log that drew my attention was
“E0808 23:12:25.258702 1 disruption.go:534] Error syncing PodDisruptionBudget calico-system/calico-typha, requeuing: Operation cannot be fulfilled on poddisruptionbudgets.policy “calico-typha”: the object has been modified; please apply your changes to the latest version and try again”
Has anyone seen this issue? suggestions?
After installing the CRI-O runtime and the Calico network plugin, the calico-kube-controllers stay in a `Pending` state.
senften@res-k8s-master:~$ kubectl get pods –all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-657d56796-qzhxw 0/1 Pending 0 17h
calico-system calico-node-62k9g 1/1 Running 0 17h
calico-system calico-typha-8fdfcb55c-k6zcq 1/1 Running 0 17h
kube-system coredns-6d4b75cb6d-bknbp 1/1 Running 0 17h
kube-system coredns-6d4b75cb6d-jkfk7 1/1 Running 0 17h
kube-system etcd-res-k8s-master 1/1 Running 0 17h
kube-system kube-apiserver-res-k8s-master 1/1 Running 0 17h
kube-system kube-controller-manager-res-k8s-master 1/1 Running 0 17h
kube-system kube-proxy-jht5p 1/1 Running 0 17h
kube-system kube-scheduler-res-k8s-master 1/1 Running 0 17h
tigera-operator tigera-operator-6995cc5df5-j64s6 1/1 Running 0 17h
I think it may be related to this bug, [calico-kube-controllers pending since not tolerate the “node-role.kubernetes.io/control-plane” · Issue #6087 · projectcalico/calico](https://github.com/projectcalico/calico/issues/6087), but, after reading through the issue, I am still uncertain how to proceed or even sure this is the issue I’ve run into.
Any advice? Anyone else seeing this?
Thanks
For single node setup allow Pods to run on master nodes – https://computingforgeeks.com/how-to-schedule-pods-on-kubernetes-control-plane-node/
Thank you. Greatly appreciated.
I also found an answer by new-mikha in the comments of https://github.com/projectcalico/calico/issues/6087 so I downloaded https://docs.projectcalico.org/manifests/custom-resources.yaml referenced in step 6 and added
controlPlaneTolerations:
– key: node-role.kubernetes.io/control-plane
effect: NoSchedule
– key: node-role.kubernetes.io/master
effect: NoSchedule
as described in the comment and then, probably obviously, created the resources with this edited file.
How do you add additional master node into the cluster?
While running this command
sudo kubeadm config images pull –cri-socket unix:///run/cri-dockerd.sock
i m getting this error
failed to pull image “registry.k8s.io/kube-apiserver:v1.25.2″: output: time=”2022-10-13T11:52:55+05:30″ level=fatal msg=”unable to determine image API version: rpc error: code = Unavailable desc = connection error: desc = \”transport: Error while dialing dial unix /run/cri-dockerd.sock: connect: no such file or directory\””
, error: exit status 1
To see the stack trace of this error execute with –v=5 or higher
i don’t know why i m not able to create file for docker at the given file i m following all steps
Whenever i run this command sudo kubeadm config images pull –cri-socket unix:///run/cri-dockerd.sock
i m getting this following error failed to pull image “registry.k8s.io/kube-apiserver:v1.25.2″: output: time=”2022-10-13T12:29:44+05:30″ level=fatal msg=”unable to determine image API version: rpc error: code = Unavailable desc = connection error: desc = \”transport: Error while dialing dial unix /run/cri-dockerd.sock: connect: no such file or directory\””
, error: exit status 1
To see the stack trace of this error execute with –v=5 or higher
Any solution ??
thanks for ur blogs its helps most of them and reilf more stress .
Here im beginer to these .whats the use of docker . kubernates run upon or docker , or docker container runs inside the pod ? plzzz tell me im confused about all these help me out plzzz
See some resources that tries to explain the difference:
– https://www.dynatrace.com/news/blog/kubernetes-vs-docker/
– https://azure.microsoft.com/en-us/topic/kubernetes-vs-docker/
– https://www.ibm.com/cloud/blog/kubernetes-vs-docker
Phew, got my master node all setup finally, thanks for the workthrough. It’s briliant and requires some patience 🙂
Awesome… We are 😄 for you!
Thanks
Thanks for the comment and welcome
I followed your guide :
Host OS: Ubuntu 22.04
CNI and version: Flannel latest
CRI and version: CRI-O latest
kubeadm init –v=5 –pod-network-cidr=10.244.0.0/16 –upload-certs –control-plane-endpoint=k8s.mydomain.com
The cluster initialized successfully but coredns pods cannot started
kubectl version –short
Client Version: v1.26.2
Kustomize Version: v4.5.7
Server Version: v1.26.2
kubectl cluster-info
Kubernetes control plane is running at https://k8s.mydomain.com:6443
CoreDNS is running at https://k8s.mydomain.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-787d4945fb-g8hhj 0/1 CreateContainerError 0 7m1s
coredns-787d4945fb-nkssp 0/1 CreateContainerError 0 6m29s
etcd-k8s-52ts-master1 1/1 Running 1 12h
kube-apiserver-k8s-52ts-master1 1/1 Running 1 12h
kube-controller-manager-k8s-52ts-master1 1/1 Running 1 12h
kube-proxy-m4mhc 1/1 Running 1 12h
kube-proxy-rrdpb 1/1 Running 0 12h
kube-scheduler-k8s-52ts-master1 1/1 Running 1 12h
kubectl describe pod coredns-787d4945fb-g8hhj -n kube-system
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 65s default-scheduler Successfully assigned kube-system/coredns-787d4945fb-g8hhj to k8s-52ts-worker1
Warning Failed 64s kubelet Error: container create failed: time=”2023-03-17T08:26:01+07:00″ level=warning msg=”unable to get oom kill count” error=”openat2 /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45fe767_098b_401f_8da9_1a1760d804f6.slice/crio-105d630363803602c6bfe9c1516b128192b95c519ff321ed1177fe0cacdc9b42.scope/memory.events: no such file or directory”
time=”2023-03-17T08:26:01+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
Warning Failed 63s kubelet Error: container create failed: time=”2023-03-17T08:26:02+07:00″ level=warning msg=”unable to get oom kill count” error=”openat2 /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45fe767_098b_401f_8da9_1a1760d804f6.slice/crio-2bbebc0f6e74a3c5a8f5e460e758e322a94ffe97964f34e357d831ce3fced345.scope/memory.events: no such file or directory”
time=”2023-03-17T08:26:02+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
Warning Failed 47s kubelet Error: container create failed: time=”2023-03-17T08:26:18+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
Warning Failed 34s kubelet Error: container create failed: time=”2023-03-17T08:26:31+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
Warning Failed 19s kubelet Error: container create failed: time=”2023-03-17T08:26:46+07:00″ level=error msg=”container_linux.go:380: starting container process caused: exec: \”/coredns\”: stat /coredns: no such file or directory”
Normal Pulled 9s (x6 over 64s) kubelet Container image “registry.k8s.io/coredns/coredns:v1.9.3” already present on machine
Hi,
Did you install network plugin?
Hi, I reinstalled cluster with docker CRI docker and Mirantis cri-dockerd as shim , CNI Flannel and it works now.
Hi ,
Thanks for this great tutorial, It’s really useful and helped me to bootstrap multiple clusters since 2022.
I want to mention two points and hope that it will help someone.
1 – For containerd, the old debian package 1.4 does not work with v1.25 and higher. the version that works it 1.6 . Here is one of the related errors :
failed to pull image “registry.k8s.io/kube-apiserver:v1.25.10″: output: time=”2023-05-21T10:16:39Z” level=fatal msg=”validate service connection: CRI v1 image API is not implemented for endpoint \”unix:///var/run/containerd/containerd.sock\”: rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService”
2 – The Urls on calico setup are not working anymore , the updated version could be found here :
https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises
Hi Josphat
Thanks for this guide, very detail and helpful.
Adding one more thing to the Calico installation part
curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml -O
as the resources.yaml seems using 192.168.0.0/16 as default cidr, but your cluster init command above is using –pod-network-cidr=172.24.0.0/16
If one want to deploy Calico network plugin, he must manually update the cidr in this yaml file to have
cidr: 172.24.0.0/16
Thanks for this powerful comment we’ve updated the article it now reflects.