The Network File System (NFS) is a protocol that enables computers to access or share files over the network. It’s an easy solution to have a dedicated file storage where multiple users and heterogeneous client devices can retrieve data from centralized disk capacity. NFS protocol has been in active development since its first development by Sun Microsystems in 1984 and it offers the following set of features.
- NFS Over TCP/UDP support
- Support for Kerberos
- Support for WebNFS
- NFS Version 2, NFS Version 3, and NFS Version 4 Protocols
- Large File support with the added ability to manipulate files larger than 2 Gbytes.
- NFS Server Logging – a record of file operations that have been performed on its file systems are kept.
- Several extensions for NFS Mounting with the
automountdcommand. - Security Negotiation for the WebNFS Service – The client is able to negotiate a security mechanism with an NFS server.
- Network Lock Manager and NFS
It is worth mentioning that NFS works in a client/server model; the server stores shared data and manages access authorization of the clients. Once authenticated, the client systems are able to access the files just as if they existed on the local system.
The Need for Persistent Storage in Kubernetes
Within Kubernetes ecosystem, a PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. A PersistentVolumeClaim (PVC) is a request for storage by a user in the cluster. PVCs are requests for PV resources and also act as claim checks to the resource. Persistent Volumes may be provisioned either statically or dynamically(using StorageClass).
Containers were developed to be stateless and ephemeral just to speed up the application launch process. Whenever data needs to persist after the container dies, the stateless design is problematic. To ensure that data persists beyond the container’s lifecycle, the best practice is to separate data management from containers using Kubernetes Storage plugins. The storage plugins enables you to create, resize, delete, and mount persistent volumes in your containers.
Configure NFS as Kubernetes Persistent Volume Storage
We’ll begin with the configuration of NFS server then perform an integration into Kubernetes environment.
For Production grade persistent storage solution we recommend rook: How To Deploy Rook Ceph Storage on Kubernetes Cluster
1. Create Data directory for k8s on NFS Server
On my NFS server I have an LVM volume group with available storage space.
[root@kvm02 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
data 1 2 0 wz--n- 931.51g 151.51g
rl 1 3 0 wz--n- 475.35g 0I’ll create a logical volume just for Kubernetes containers data.
$ sudo lvcreate -n k8s_data -L 30G data
Logical volume "k8s_data" created.Next we create a filesystem on the logical volume.
$ sudo mkfs.xfs /dev/data/k8s_data
meta-data=/dev/data/k8s_data isize=512 agcount=4, agsize=1966080 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=7864320, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=3840, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.Create mount directory for the partition.
sudo mkdir -p /data/k8sEdit the /etc/fstab file to configure persistent mounting.
$ sudo vim /etc/fstab
#K8s data mount point
/dev/data/k8s_data /data/k8s xfs defaults 0 0Save the file and confirm it works.
sudo mount /data/k8s/Check using df command:
$ df -hT /data/k8s/
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/data-k8s_data xfs 30G 247M 30G 1% /data/k8s2. Install and Configure NFS Server
Install NFS Server packages on your system.
### Debian / Ubuntu ###
sudo apt update
sudo apt install nfs-kernel-server
### RHEL based systems ###
sudo yum -y install nfs-utilsSet your domain name in the file /etc/idmapd.conf
$ sudo vim /etc/idmapd.conf
Domain = yourdomain.comConfigure NFS exports
$ sudo vim /etc/exports
/data/k8s/ 192.168.1.0/24(rw,no_root_squash)Restart NFS server
### Debian / Ubuntu Linux systems ###
sudo systemctl restart nfs-server
### RHEL based Linux systems ###
sudo systemctl enable --now rpcbind nfs-server
sudo firewall-cmd --add-service=nfs --permanent
sudo firewall-cmd --reload3. Setup Kubernetes NFS Subdir External Provisioner
With the NFS server configured, we’ll configure NFS subdir external provisioner which is an automatic provisioner that uses existing and already configured NFS server to support dynamic provisioning of Kubernetes Persistent Volumes via Persistent Volume Claims. Persistent volumes are provisioned as ${namespace}-${pvcName}-${pvName}.
Deploy NFS Subdir External Provisioner in Kubernetes cluster
Make sure your NFS server is configured and accessible from your Kubernetes cluster. The minimum information required to connect to the NFS server is hostname/IP address and exported share path.
Install Helm on your system.
curl -L https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bashConfirm helm is installed and working.
$ helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}Also check if your kubectl command can talk to kubernetes cluster.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01 Ready control-plane 16d v1.24.6
node01 Ready <none> 16d v1.24.6
node02 Ready <none> 15d v1.24.6
node03 Ready <none> 15d v1.24.6The nfs-subdir-external-provisioner charts installs custom storage class into a Kubernetes cluster using the Helm package manager. It will also install