A Container is a standard way of packaging an application and all its dependencies for running across multiple or different environments seamlessly. Containers will contain the application code, system tools, run time, libraries, and all the settings required to run the app. Having multiple containers without a management platform will not give you the desired capabilities required for running microservices in an aggressive production environment. For this reason Kubernetes came into existence.

On the flip side, Kubernetes is a container management solution with the goal of automating deployment of containers, its scaling, descaling, load balancing among many other operations. The two major server components of Kubernetes is the Control Plane and Worker nodes. In this article we discuss in detail the process of deploying both control plane nodes and worker nodes using Flatcar Container Linux, on a KVM powered virtualized infrastructure.

Our Lab setup will be based on the following systems and hostnames.

Server RoleHostnameIP AddressOS
Control Plane / Etcdmaster01.k8s.cloudlabske.io192.168.1.10Flatcar Container Linux
Worker Nodenode01.k8s.cloudlabske.io192.168.1.13Flatcar Container Linux
Worker Nodenode02.k8s.cloudlabske.io192.168.1.14Flatcar Container Linux
Worker Nodenode03.k8s.cloudlabske.io192.168.1.15Flatcar Container Linux

For the purpose of this article, we’ve created a DNS zone k8s.cloudlabske.io in our DNS server (FreeIPA). We shall configure A records in the zone. If you have a different type of DNS server, configure it as per its official documentation.

Step 1 – Prepare KVM Infrastructure

KVM is an alimentation in the delivery of this article. If you’re doing this setup from scratch, refer to our articles for KVM installation and configurations.

After the installation check to confirm if virtualization extensions are enabled in BIOS for your CPU.

 egrep --color 'vmx|svm' /proc/cpuinfo

See if you see below lines in the output;

  • vmx – Intel VT-x, means virtualization support enabled in BIOS.
  • svm – AMD SVM, means virtualization enabled in BIOS.

Other commands that can be used to validate:

### Debian based systems ###
sudo kvm-ok

### RHEL Based systems ###
sudo virt-host-validate

Create Libvirt network

We’ll consider two options on network creation for this setup. One will be a local NAT network, and another being a bridge on an actual network interface.

Option 1: Using NAT network

When you install and start libvirtd service on a virtualization host, an initial virtual network configuration is created in network address translation (NAT) mode. By default, all VMs on the host are connected to the same libvirt virtual network, named default.

$ sudo  virsh  net-list
 Name         State    Autostart   Persistent
-----------------------------------------------
 default      active   yes         yes

In NAT mode network, the Virtual Machines on the network can connect to locations outside the host but are not visible to them. Outbound traffic is affected by the NAT rules, as well as the host system’s firewall. You can use the default network or create another NATed network on KVM.

Creating new NAT Libvirt network

Create a new files called k8s-network.xml

vim k8s-network.xml

Paste and edits given network configuration content. This uses 192.168.120.0/24 network.

  <network>
    <name>k8s-network</name>
    <forward mode='nat'>
      <nat>
        <port start='1024' end='65535'/>
      </nat>
    </forward>
    <bridge name='k8s' stp='on' delay='0' />
    <ip address='192.168.120.1' netmask='255.255.255.0'>
      <dhcp>
        <range start='192.168.120.2' end='192.168.120.254' />
      </dhcp>
    </ip>
  </network>

Create libvirt network and start it.

sudo virsh net-define k8s-network.xml
sudo  virsh net-start  k8s-network
sudo virsh net-autostart k8s-network

Confirm this was successful.

$ ip address show dev k8s
50: k8s: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:1f:e8:22 brd ff:ff:ff:ff:ff:ff
    inet 192.168.120.1/24 brd 192.168.120.255 scope global k8s
       valid_lft forever preferred_lft forever

$ sudo virsh net-list
 Name          State    Autostart   Persistent
------------------------------------------------
 default       active   yes         yes
 k8s-network   active   yes         yes

Option 2: Using Bridged network

For the Virtual Machines to appear on the same external network as the hypervisor, you must use bridged mode instead. We assume you already have a bridge created on your host system. Below are host configurations to be used for reference.

### NIC and Bridge configurations on RHEL based system ###
$ sudo vim /etc/sysconfig/network-scripts/ifcfg-enp89s0
TYPE=Ethernet
NAME=enp89s0
DEVICE=enp89s0
ONBOOT=yes
BRIDGE=bridge0

$ sudo vim /etc/sysconfig/network-scripts/ifcfg-bridge0
STP=no
TYPE=Bridge
NAME=bridge0
DEVICE=bridge0
ONBOOT=yes
AUTOCONNECT_SLAVES=yes
DEFROUTE=yes
IPADDR=172.20.30.7
PREFIX=24
GATEWAY=172.20.30.1
DNS1=172.20.30.252

With the bridge on host active we can define libvirt network that uses this bridge.

$ vim k8s-bridge.xml
<network>
  <name>k8s-bridge</name>
  <forward mode="bridge"/>
  <bridge name="bridge0"/>
</network>

Now we can define the network:

$ virsh net-define k8s-bridge.xml
Network k8s-bridge defined from k8s-bridge.xml

$ virsh net-start k8s-bridge
Network k8s-bridge started

$ virsh net-autostart k8s-bridge
Network k8s-bridge marked as autostarted

Checking network state.

$ sudo virsh net-list --all
 Name         State    Autostart   Persistent
-----------------------------------------------
 default      active   yes         yes
 k8s-bridge   active   yes         yes

$ sudo virsh net-info --network k8s-bridge
Name:           k8s-bridge
UUID:           6aa0e02b-fb37-4513-bc05-8e20004ae1e2
Active:         yes
Persistent:     yes
Autostart:      yes
Bridge:         bridge0

If interested in using Open vSwitch refer to guide below:

Step 2 – Create required DNS records for Kubernetes

Based on the network design in the table shared, we’ll create required DNS settings.

  • master01: 192.168.1.10
  • node01: 192.168.1.13
  • node02: 192.168.1.14
  • node03: 192.168.1.15

On my FreeIPA server, which is used as DNS server, the following commands were executed to add A records.

[root@ipa02 ~]# ipa dnsrecord-add k8s.cloudlabske.io --a-create-reverse --a-rec 192.168.1.10 master01
  Record name: master01
  A record: 192.168.1.10

[root@ipa02 ~]# ipa dnsrecord-add k8s.cloudlabske.io --a-create-reverse --a-rec 192.168.1.13 node01
  Record name: node01
  A record: 192.168.1.13

[root@ipa02 ~]# ipa dnsrecord-add k8s.cloudlabske.io --a-create-reverse --a-rec 192.168.1.14 node02
  Record name: node02
  A record: 192.168.1.14

[root@ipa02 ~]# ipa dnsrecord-add k8s.cloudlabske.io --a-create-reverse --a-rec 192.168.1.15 node03
  Record name: node03
  A record: 192.168.1.15

Once the records for all the nodes are added we can proceed to VMs deployment.

Step 3 – Create Flatcar Container Linux VMs

Flatcar Container Linux is designed from the ground up for running container workloads. I chose this operating system to power Kubernetes because of its immutable nature and automated atomic updates.

Terraform is our solution of choice used in provisioning Virtual Machines. Terraform is an open-source and cloud-agnostic IAC tool created by HashiCorp. This powerful provisioning tool is written in the Go language for SysAdmins and DevOps teams to automate different kinds of infrastructure tasks. In this guide we’ll use Terraform to provision Virtual Machine instances on KVM for running Kubernetes.

1. Install Terraform

We start by installing terraform on KVM node for ease of interaction with the hypervisor.

### Install Terraform on RHEL based systems ###
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

### Install Terraform on Fedora ###
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
sudo dnf -y install terraform

### Install Terraform on Ubuntu / Debian ###
sudo apt update
sudo apt install -y gnupg software-properties-common
# Import GPG key
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg

#Add repo
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
    https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
    sudo tee /etc/apt/sources.list.d/hashicorp.list

# Install terraform
sudo apt update
sudo apt install terraform

Confirm terraform is installed and working by checking its version.

terraform --version

2. Download Flatcar KVM base image

In this guide, all Virtual Machine’s files are stored in /var/lib/libvirt/images/ directory. This is not a mandatory requirement, so feel free to substitute that path if you have another storage pool configured.

Flatcar Container Linux is designed to be updated automatically with different schedules per channel. There are three main channels available:

  • Stable channel
  • Beta channel
  • Alpha channel

For production clusters the Stable channel should be used. We start by downloading the most recent stable disk image:

wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img.bz2{,.sig}
gpg --verify flatcar_production_qemu_image.img.bz2.sig
bunzip2 flatcar_production_qemu_image.img.bz2

Move image file to /var/lib/libvirt/images

sudo mv flatcar_production_qemu_image.img /var/lib/libvirt/images/flatcar_qemu_image.img

When you create an instance from the image it’s assigned 5GB disk, Let’s increase this default size by 30GB

sudo qemu-img resize /var/lib/libvirt/images/flatcar_qemu_image.img +30G

3. Create Terraform code for VM resources

Create directory that will contain all terraform code.

mkdir ~/flatcar_tf_libvirt
cd ~/flatcar_tf_libvirt

Create variables file with below content. Most don’t have default values as they’ll be set in a separate file.

$ vim variables.tf
variable "machines" {
  type        = list(string)
  description = "Machine names, corresponding to machine-NAME.yaml.tmpl files"
}

variable "cluster_name" {
  type        = string
  description = "Cluster name used as prefix for the machine names"
}

variable "cluster_domain" {
  type        = string
  description = "Cluster base domain name, e.g k8scluster.com"
}

variable "ssh_keys" {
  type        = list(string)
  description = "SSH public keys for user 'core'"
}

variable "base_image" {
  type        = string
  description = "Path to unpacked Flatcar Container Linux image flatcar_qemu_image.img (probably after a qemu-img resize IMG +20G)"
}

variable "virtual_memory" {
  type        = number
  default     = 2048
  description = "Virtual RAM in MB"
}

variable "virtual_cpus" {
  type        = number
  default     = 1
  description = "Number of virtual CPUs"
}

variable "storage_pool" {
  type        = string
  description = "Storage pool for VM images"
}

variable "network_name" {
  type        = string
  description = "Libvirt network used by Flatcar instances"
}

Create another terraform file that will handle actual resources provisioning.

$ vim kvm-machines.tf
terraform {
  required_version = ">= 0.13"
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
    }
    ct = {
      source  = "poseidon/ct"
    }
    template = {
      source  = "hashicorp/template"
    }
  }
}

# Define libvirt URI
provider "libvirt" {
  uri = "qemu:///system"
  #uri = "qemu+ssh://[email protected]/system"
}

#resource "libvirt_pool" "flatcar_storage_pool" {
#  name = "${var.cluster_name}-pool"
#  type = "dir"
#  path = "/data/libvirt/${var.cluster_name}-pool"
#}

resource "libvirt_volume" "base" {
  name   = "flatcar-base"
  source = var.base_image
  pool   = var.storage_pool
  format = "qcow2"
}

resource "libvirt_volume" "vm-disk" {
  for_each = toset(var.machines)
  # workaround: depend on libvirt_ignition.ignition[each.key], otherwise the VM will use the old disk when the user-data changes
  name           = "${var.cluster_name}-${each.key}-${md5(libvirt_ignition.ignition[each.key].id)}.qcow2"
  base_volume_id = libvirt_volume.base.id
  pool           = var.storage_pool
  format         = "qcow2"
}

resource "libvirt_ignition" "ignition" {
  for_each = toset(var.machines)
  name     = "${var.cluster_name}-${each.key}-ignition"
  pool     = var.storage_pool
  content  = data.ct_config.vm-ignitions[each.key].rendered
}

resource "libvirt_domain" "machine" {
  for_each = toset(var.machines)
  name     = "${var.cluster_name}-${each.key}"
  vcpu     = var.virtual_cpus
  memory   = var.virtual_memory

  fw_cfg_name     = "opt/org.flatcar-linux/config"
  coreos_ignition = libvirt_ignition.ignition[each.key].id

  disk {
    volume_id = libvirt_volume.vm-disk[each.key].id
  }

  graphics {
    listen_type = "address"
  }

# dynamic IP assignment on the bridge, NAT for Internet access
  network_interface {
    network_name   = var.network_name
    wait_for_lease = true
  }
}

data "ct_config" "vm-ignitions" {
  for_each = toset(var.machines)
  content  = data.template_file.vm-configs[each.key].rendered
}

data "template_file" "vm-configs" {
  for_each = toset(var.machines)
  template = file("${path.module}/machine-${each.key}.yaml.tmpl")

  vars = {
    ssh_keys = jsonencode(var.ssh_keys)
    name     = each.key
    host_name = "${each.key}.${var.cluster_name}.${var.cluster_domain}"
  }
}

Define configurations for outputting the IP addresses of created virtual machines.

$ vim outputs.tf
output "ip-addresses" {
  value = {
    for key in var.machines :
    "${var.cluster_name}-${key}" => libvirt_domain.machine[key].network_interface.0.addresses.*
  }
}

Define actual variables for this environment

Create a file called terraform.tfvars with the values for the variables defined in variables.tf