Home Containers Deploy and Use OpenEBS Container Storage on Kubernetes

Deploy and Use OpenEBS Container Storage on Kubernetes

August 24, 2023

10180

Managing storage and volumes on a Kubernetes cluster can be challenging for many engineers. Setting up Persistent volumes and dynamic allocation of the same can be made easy by the tool we are going to explore today, OpenEBS.

OpenEBS is a cloud native storage project originally created by MayaData that build on a Kubernetes cluster and allows Stateful applications to access Dynamic Local PVs and/or replicated PVs. OpenEBS runs on any Kubernetes platform and uses any Cloud storage solution including AWS s3, GKE and AKS.

OpenEBS adopts the Container Attached Storage (CAS) architecture in such a way that the volumes provisioned through OpenEBS are containerized.

OpenEBS utilizes disks attached to the worker nodes, external mount-points and local host paths on the nodes.

Features of OpenEBS

Containerized storage: OpenEBS volumes are always containerized as it used Container Attached Storage architecture.
Synchronous replication: OpenEBS can synchronously replicate data volumes when used wirh Cstor, Jiva or Mayastor for high availability of stateful applications.
Snapshots: Snapshots are created instantaneously when using Cstor. This makes it easy for data migration within the Kubernetes cluster.
Backup and restore: The backup and restore feature works with Kubernetes solutions such as Velero. You can backup data to object storage such as AWS s3 and GCP.
Prometheus Metrics: OpenEBS volumes are configured to generate granular data metrics e.g throughput, latency and IOPS. this can easily be shipped via prometheus data exporter and displayed on a Grafana dashboard for monitoring of the cluster health, disk failures and utilization.

OpenEBS Architecture

OpenEBS uses the CAS model. This means that each volume has a dedicated controller POD and a set of replica PODs.

OpenEBS has the following components:

Data plane components – cStor, Jiva and LocalPV

The data plane is reponsible for the actual IO path of the persistent volume. You can choose between the three storage engines discussed below depending on your workloads and preferences.

cStor – This is the preferred storage engine for OpenEBS as it offers enterprise-grade features such as snapshots, clones, thin provisioning, data consistency and scalability in capacity. This in turn allows Kubernetes stateful deployments to work with high availability. cStor is designed to have three replicas whereby data is written synchronously to the replicas hence allowing pods to retain data during terminating and rescheduling.
Jiva – Jiva runs exclusively in the user space with block storage capabilities such as synchronous replication. This option is ideal in situations where you have applications running on nodes that might not be able to add more block storage devices. This however is not ideal for mission-critical applications that require high performance storage capabilities.
LocalPV – This is the simplest storage engine of the three. A Local Persistent Volume is a directly-attached volume to a Kubernetes node. OpenEBS can make use of a locally attached disk or a path (mount-point) to provision persistent volumes to the k8s cluster. This is ideal in situations where you are running applications that do not require advanced storage capabilities such as snapshots and clones.

The table below highlights the features available on each storage engine discussed above.

deploy openebs container storage on kubernetes

Control plane components – volume exports, volume sidecars, API server and Provisioner

The Control plane is reponsible for volume operations such as provisioning volumes, making clones, exporting volume metrics and enforcing volume policies.

Node disk manager (NDM) – Used for discovery, monitoring and management of the media/disks attached on Kubernetes nodes.

Node Disk Manager is the tool used to manage persistent storage in Kubernetes for stateful applications. This brings flexibility in the management of the storage stack by unifying disks and creating pools and identifying them as kubernetes objects.

NDM discovers, provisions, manages and monitors the underlying disks for PV provisioners like OpenEBS and prometheus.

How to Setup OpenEBS on Kubernetes Cluster

This article will discuss how to setup OpenEBS on a kubernetes cluster. At the end of this article, we shall have covered the following:

Setup OpenEBS on Kubernetes
Provision Persistent Volumes on Kubernetes using OpenEBS
Provision Storage classes (SC) and Persistent Volume Claims (PVC) on Kubernetes.

Installing OpenEBS on Kubernetes

Before we can start our installation, we have to make sure that iSCSI client is installed and running on all the nodes. This is necessary for Jiva and cStor setups.

Verify that iscsid service is running on your nodes, otherwise, install it.

Ubuntu/Debian

sudo apt update && sudo apt install open-iscsi
sudo systemctl enable --now iscsid
systemctl status iscsid

RedHat/CentOS

yum install iscsi-initiator-utils -y
sudo systemctl enable --now iscsid
systemctl status iscsid

Method 1: Install OpenEBS on Kubernetes using Helm

We can deploy OpenEBS through Helm Charts. You have to check the version of Helm installed on your system.

$ helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.19.3"}

For Helm 2, Install the OpenEBS chart using the commands below:

helm repo add openebs https://openebs.github.io/charts
helm repo update
helm install --namespace openebs --name openebs openebs/openebs

Helm 3:

For Helm 3, we need to create the openebs namespace before we can deploy the chat:

$ kubectl create ns openebs
namespace/openebs created

Deploy openebs from helm chart.

helm repo add openebs https://openebs.github.io/charts
helm repo update
helm install --namespace openebs openebs openebs/openebs

Method 2: Install OpenEBS on Kubernetes through Kubectl

We can use Kubectl to install OpenEBS

Create openebs namespace

kubectl create ns openebs

Install OpenEBS:

kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

Verify Pods

After a successful installation, verify that the pods are up:

kubectl get pods -n openebs

Screenshot with command output showing running pods.

Verify Stoage Classes (SC)

Ensure that the default storage classes (SC) have been created.

$ kubectl get sc

You will see storage classes created.

root@bazenga:~# kubectl get sc
NAME                        PROVISIONER                                                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
openebs-device              openebs.io/local                                           Delete          WaitForFirstConsumer   false                  11m
openebs-hostpath            openebs.io/local                                           Delete          WaitForFirstConsumer   false                  11m
openebs-jiva-default        openebs.io/provisioner-iscsi                               Delete          Immediate              false                  11m
openebs-snapshot-promoter   volumesnapshot.external-storage.k8s.io/snapshot-promoter   Delete          Immediate              false                  11m

Verify Block Device CRs

OpenEBS NDM daemon-sets identifies the available block devices on the nodes and creates a CR for each. All the disks available on the nodes will be identified unless you had specified an exclusion in the vendor-filter and path-filter of the NDM config-map.

kubectl get blockdevice -n openebs

e.g

root@bazenga:~# kubectl get blockdevice 
NAME                                           NODENAME   SIZE          CLAIMSTATE   STATUS   AGE
blockdevice-4f749fd37d6bfc5613c9272b2eb75cc0   node02     10736352768   Unclaimed    Active   15m
blockdevice-59c0818b5f8b2e56028959d921221af2   node03     10736352768   Unclaimed    Active   15m
blockdevice-79b8a6c83ee34a7e4b55e8d23f14323d   node03     21473771008   Unclaimed    Active   15m

To verify which node a device CR belongs to, run the describe command.

kubectl describe blockdevice <blockdevice-cr> -n openebs

Example:

$ kubectl describe blockdevice blockdevice-4f749fd37d6bfc5613c9272b2eb75cc0 -n openebs
..........
Spec:
  Capacity:
    Logical Sector Size:   512
    Physical Sector Size:  512
    Storage:               10736352768
  Details:
    Compliance:            
    Device Type:           partition
    Drive Type:            SSD
    Firmware Revision:     
    Hardware Sector Size:  512
    Logical Block Size:    512
    Model:                 
    Physical Block Size:   512
    Serial:                
    Vendor:                
  Devlinks:
  Filesystem:
  Node Attributes:
    Node Name:  node02
  Partitioned:  No
  Path:         /dev/xvdb1
Status:
  Claim State:  Unclaimed
  State:        Active
Events:         <none>

Working with Storage Engines

As we had discussed before, OpenEBS provides three storage engines that one can choose to work with.

The engines are:

cStor
Jiva
Local PV

We shall discuss how to use the three storage engines.

Persistent volumes using cStor

For cStor, there are a number of operations needed to provision a persistent volume that utilizes this feature. These are:

Create cStor storage pools
Create cStor storage classes
Provision a cStor volume

We will go through the steps to achieve the above.

Step 1 – Create cStor Storage pool

The storage pool is created through specifying the block devices on the nodes. Use the steps below to create a cStor storage pool.

Get the details of the block devices attached to the k8s cluster

kubectl get blockdevice -n openebs -o jsonpath='{ range .items[*]} {.metadata.name}{"\n"}{end}'

Example:

root@bazenga:~# kubectl get blockdevice -o jsonpath='{ range .items[*]} {.metadata.name}{"\n"}{end}'
 blockdevice-4f749fd37d6bfc5613c9272b2eb75cc0
 blockdevice-59c0818b5f8b2e56028959d921221af2
 blockdevice-79b8a6c83ee34a7e4b55e8d23f14323d

Identify the unclaimed blockdevices: