DEV Community

Cover image for Ultimate EKS Baseline Cluster: Part 2 - Storage
Joaquin Menchaca
Joaquin Menchaca

Posted on

Ultimate EKS Baseline Cluster: Part 2 - Storage

#awscloud #kubernetes #devops #tutorial

This may come as a surprise to some, but the AWS managed Kubernetes service, EKS (Elastic Kubernetes Service), no longer comes with support for storage through Kubernetes persistent volumes, which is required for many applications, such as databases.

You will need to add a (CSI) driver to enable persistent volumes support.

This guide, Part 2 of Ultimate EKS Baseline Cluster series, covers doing just that by installing CSI) called AWS EBS CSI Driver.

0. Prerequisites

These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.

0.1 Knowledge: Systems

image of system concept icons

Basic concepts of systems, such as Linux and the shell (redirection, pipes, process substitution, command substitution, environment variables), as well as virtualization and containers are useful. The concept of a service (daemon) is important.

0.2 Knowledge: Kubernetes

image of Kubernetes concept icons

In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.

Exposure to other types of Kubernetes resource objects used in this guide are helpful: persistentvolumeclaims (pvc), storageclass (sc), pods, deployments, statefulsets (sts), configmaps, serviceaccount (sa) and networkpolicies.

0.3 Tools

image of tool icons

These are the tools used in this article series:

  • AWS CLI [aws] is a tool that interacts with AWS.
  • kubectl client [kubectl] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
  • eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
  • POSIX Shell [sh] such as bash[bash] or zsh [zsh] are used to run the commands.

These tools are highly recommended:

  • adsf [adsf] is a tool that installs versions of popular tools like kubectl.
  • jq [jq] is a tool to query and print JSON data
  • GNU Grep [grep] supports extracting string patterns using extended Regex and PCRE.

0.3 Setup Environment Variables

These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.

# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"

# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)

# ebs-csi-driver
export ROLE_NAME_ECSI="${EKS_CLUSTER_NAME}_EBS_CSI_DriverRole"
export ACCOUNT_ROLE_ARN_ECSI="arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME_ECSI"
POLICY_NAME_ECSI="AmazonEBSCSIDriverPolicy" # preinstalled by AWS
export POLICY_ARN_ECSI="arn:aws:iam::aws:policy/service-role/$POLICY_NAME_ECSI"
Enter fullscreen mode Exit fullscreen mode

0.4 AWS Setup

There's an assumption that AWS CLI have been setup and configured with a profile. This is required before creating or interacting with an EKS cluster.

You can test access to a configured profile with the following command:

export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity
Enter fullscreen mode Exit fullscreen mode

This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named kwisatzhaderach:

{
    "UserId": "AIDAXXXXXXXXXXXXXXXXX",
    "Account": "XXXXXXXXXXXX",
    "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/kwisatzhaderach"
}
Enter fullscreen mode Exit fullscreen mode

0.5 EKS Cluster

This article requires that an EKS cluster has been previously provisioned using eksctl tool.

I wrote a previous article that covered this how to provision EKS with two commands:

0.5.1 Existing EKS Cluster

If you have an existing EKS cluster, but need to configure KUBECONFIG for access, you can run this:

mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# update config pointed to by $KUBECONFIG 
aws eks update-kubeconfig \
  --name $CLUSTER \
  --region $REGION \
  --profile $PROFILE
Enter fullscreen mode Exit fullscreen mode

0.5.2 Create a new EKS Cluster

If you have not setup an EKS cluster, you can set it up with the following commands (~20 minutes process):

mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# provision EKS + add config to $KUBECONFIG 
eksctl create cluster \
  --version $EKS_VERSION \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --nodes 3

# setup OIDC provider for least privilege
eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve
Enter fullscreen mode Exit fullscreen mode

Details of this were covered in the previous article

0.6 Kubernetes Client Setup

If you use asdf to install kubectl, you can get the latest version with the following:

# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null
Enter fullscreen mode Exit fullscreen mode

This should show something like:

Client Version: v1.27.4
Kustomize Version: v5.0.1
Enter fullscreen mode Exit fullscreen mode

1. AWS EBS CSI driver

Title bar with icons for EBS and CSI

The current versions of EKS starting with 1.23 no longer come with a persistent volume support, so you have to install it on your own. The best method or at least the easiest way to install this, is using EKS add-ons facility. This will install the EBS CSI driver.

Installation of this component will require the following steps:

  1. Create IAM Role (e.g. EBS_CSI_DriverRole) and associate it to Kubernetes service account (i.e. ebs-csi-controller-sa).
  2. Deploy AWS EBS CSI driver using EKS add-ons facility, which also sets up the Kubernetes service account (i.e. ebs-csi-controller-sa) with an association back to the above IAM Role (e.g. EBS_CSI_DriverRole).
  3. Create storage class that uses new the EBS CSI driver

1.1 Setup IAM Role and K8S SA association

The following process will create an IAM Role with permissions to access AWS EBS API. The service account ebs-csi-controller-sa will be created later when installing the driver.

# AWS IAM role bound to a Kubernetes service account
eksctl create iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --attach-policy-arn $POLICY_ARN_ECSI \
  --role-only \
  --role-name $ROLE_NAME_ECSI \
  --approve
Enter fullscreen mode Exit fullscreen mode

This will create a IAM Role, which you can verify with:

aws iam get-role --role-name $ROLE_NAME_ECSI
Enter fullscreen mode Exit fullscreen mode

This should show something like this:

{
    "Role": {
        "Path": "/",
        "RoleName": "mycluster_EBS_CSI_DriverRole",
        "RoleId": "AROAZYKZFDW7YZGW3Q5S7",
        "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/mycluster_EBS_CSI_DriverRole",
        "CreateDate": "2023-07-07T21:19:32+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:aud": "sts.amazonaws.com",
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
                        }
                    }
                }
            ]
        },
        "Description": "",
        "MaxSessionDuration": 3600,
        "Tags": [
            {
                "Key": "alpha.eksctl.io/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "alpha.eksctl.io/iamserviceaccount-name",
                "Value": "kube-system/ebs-csi-controller-sa"
            },
            {
                "Key": "alpha.eksctl.io/eksctl-version",
                "Value": "0.141.0-dev+5c8318ed5.2023-05-12T11:33:48Z"
            }
        ],
        "RoleLastUsed": {}
    }
}
Enter fullscreen mode Exit fullscreen mode

1.2 Install AWS EBS CSI Drvier

This installation uses the EKS Addons feature to install the component.

# Install Addon
eksctl create addon \
  --name "aws-ebs-csi-driver" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --service-account-role-arn $ACCOUNT_ROLE_ARN_ECSI \
  --force

# Pause here until driver is active
ACTIVE=""; while [[ -z "$ACTIVE" ]]; do
  if eksctl get addon \