The standard way of exposing applications that are running on a set of Pods in a Kubernetes Cluster is by using service resource. Each Pod in Kubernetes has its own IP address, but a set of Pods can have a single DNS name. Kubernetes is able to load-balance traffic across the Pods without any modification in the application layer. A service, by default is assigned an IP address (sometimes called the “cluster IP“), which is used by the Service proxies. A Service is able to identify a set of Pods using label selectors.
What is an Ingress Controller?
Before you can answer this question, an understanding of Ingress object in Kubernetes is important. From the official Kubernetes documentation, an Ingress is defined like as:
An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.An Ingress in Kubernetes exposes HTTP and HTTPS routes from outside the cluster to services running within the cluster. All the traffic routing is controlled by rules defined on the Ingress resource. An Ingress may be configured to:
- Provide Services with externally-reachable URLs
- Load balance traffic coming into cluster services
- Terminate SSL / TLS traffic
- Provide name-based virtual hosting in Kubernetes
An Ingress controller is what fulfils the Ingress, usually with a load balancer. Below is an example on how an Ingress sends all the client traffic to a Service in Kubernetes Cluster:
For the standard HTTP and HTTPS traffic, an Ingress Controller will be configured to listen on ports 80 and 443. It should bind to an IP address from which the cluster will receive traffic from. A Wildcard DNS record for the domain used for Ingress routes will point to the IP address(s) that Ingress controller listens on.
It is fact that Kubernetes adopts a BYOS (Bring-Your-Own-Software) approach to most of its addons and it doesn’t provide a software that does Ingress functions out of the box. You can choose from the plenty of Ingress Controllers available. Kubedex does a good job as well on summarizing list of Ingresses available for Kubernetes.
With all the basics on Kubernetes Services and Ingress, we can now plunge into the actual installation of NGINX Ingress Controller Kubernetes.
Step 1: Deploy Nginx Ingress Controller in Kubernetes
We shall consider two major deployment options captured in the next sections.
Option 1: Install without Helm
With this method you’ll manually download and run deployment manifests using kubectl command line tool.
Step 1: Install git, curl and wget tools
Install git, curl and wget tools in your Bastion where kubectl is installed and configured:
# CentOS / RHEL / Fedora / Rocky
sudo yum -y install wget curl git
# Debian / Ubuntu
sudo apt update
sudo apt install wget curl gitStep 2: Apply Nginx Ingress Controller manifest
The deployment process varies depending on your Kubernetes setup. My Kubernetes will use Bare-metal Nginx Ingress deployment guide. For other Kubernetes clusters including managed clusters refer to below guides:
The Bare-metal method applies to any Kubernetes clusters deployed on bare-metal with generic Linux distribution(Such as CentOS, Ubuntu, Debian, Rocky Linux) e.t.c.
Download Nginx controller deployment for Baremetal:
controller_tag=$(curl -s https://api.github.com/repos/kubernetes/ingress-nginx/releases/latest | grep tag_name | cut -d '"' -f 4)
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/${controller_tag}/deploy/static/provider/baremetal/deploy.yamlRename deployment file:
mv deploy.yaml nginx-ingress-controller-deploy.yamlFeel free to check the file contents and modify where you see fit:
vim nginx-ingress-controller-deploy.yamlApply Nginx ingress controller manifest deployment file:
$ kubectl apply -f nginx-ingress-controller-deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch createdYou can update your current context to use nginx ingress namespace:
kubectl config set-context --current --namespace=ingress-nginxRun the following command to check if the ingress controller pods have started:
$ kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --watch
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-hpkzp 0/1 Completed 0 43s
ingress-nginx-admission-patch--1-qnjlj 0/1 Completed 1 43s
ingress-nginx-controller-644555766d-snvqf 1/1 Running 0 44sOnce the ingress controller pods are running, you can cancel the command typing Ctrl+C.
If you want to run multiple Nginx Ingress Pods, you can scale with the command below:
$ kubectl -n ingress-nginx scale deployment ingress-nginx-controller --replicas 2
deployment.apps/ingress-nginx-controller scaled
$ kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-lj278 0/1 Completed 0 18m
ingress-nginx-admission-patch-zsjkp 0/1 Completed 0 18m
ingress-nginx-controller-6dc865cd86-n474n 0/1 ContainerCreating 0 119s
ingress-nginx-controller-6dc865cd86-tmlgf 1/1 Running 0 18mOption 2: Install using Helm
If you opt in for the Helm installation method then follow the steps provided in this section.
Step 1: Install helm 3 in our workstation
Install helm 3 in your Workstation where Kubectl is installed and configured.
cd ~/
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.shThe installer script works for both Linux and macOS operating systems. Here is a successful installation output:
Downloading https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helmLet’s query for helm package version to validate working installation:
$ helm version
version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.18.9"}Step 2: Deploy Nginx Ingress Controller
Download latest stable release of Nginx Ingress Controller code:
controller_tag=$(curl -s https://api.github.com/repos/kubernetes/ingress-nginx/releases/latest | grep tag_name | cut -d '"' -f 4)
wget https://github.com/kubernetes/ingress-nginx/archive/refs/tags/${controller_tag}.tar.gzExtract the file downloaded:
tar xvf ${controller_tag}.tar.gzSwitch to the directory created:
cd ingress-nginx-${controller_tag}Change your working directory to charts folder:
cd charts/ingress-nginx/Create namespace
kubectl create namespace ingress-nginxNow deploy Nginx Ingress Controller using the following commands
helm install -n ingress-nginx ingress-nginx -f values.yaml .Sample deployment output
NAME: ingress-nginx
LAST DEPLOYED: Thu Nov 4 02:50:28 2021
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: example
namespace: foo
spec:
ingressClassName: example-class
rules:
- host: www.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: exampleService
port: 80
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tlsCheck status of all resources in ingress-nginx namespace:
kubectl get all -n ingress-nginxChecking runningPods in the namespace.
$ kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-6f5844d579-hwrqn 1/1 Running 0 23mTo check logs in the Pods use the commands:
$ kubectl -n ingress-nginx logs deploy/ingress-nginx-controller
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.0.4
Build: 9b78b6c197b48116243922170875af4aa752ee59
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.9
-------------------------------------------------------------------------------
W1104 00:06:59.684972 7 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1104 00:06:59.685080 7 main.go:221] "Creating API client" host="https://10.96.0.1:443"
I1104 00:06:59.694832 7 main.go:265] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.2" state="clean" commit="8b5a19147530eaac9476b0ab82980b4088bbc1b2" platform="linux/amd64"
I1104 00:06:59.937097 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1104 00:06:59.956498 7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1104 00:06:59.975510 7 nginx.go:253] "Starting NGINX Ingress controller"
I1104 00:07:00.000753 7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"5aea2f36-fdf2-4f5c-96ff-6a5cbb0b5b82", APIVersion:"v1", ResourceVersion:"13359975", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1104 00:07:01.177639 7 nginx.go:295] "Starting NGINX process"
I1104 00:07:01.177975 7 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I1104 00:07:01.178194 7 nginx.go:315] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1104 00:07:01.180652 7 controller.go:152] "Configuration changes detected, backend reload required"
I1104 00:07:01.197509 7 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader
I1104 00:07:01.197857 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-6f5844d579-hwrqn"
I1104 00:07:01.249690 7 controller.go:169] "Backend successfully reloaded"
I1104 00:07:01.249751 7 controller.go:180] "Initial sync, sleeping for 1 second"
I1104 00:07:01.249999 7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-6f5844d579-hwrqn", UID:"d6a2e95f-eaaa-4d6a-85e2-bcd25bf9b9a3", APIVersion:"v1", ResourceVersion:"13364867", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configurationTo follow logs as they stream run:
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller -fUpgrading Helm Release
I’ll set replica count of the controller Pods to 2:
$ vim values.yaml
controller:
replicaCount: 3We can confirm we currently have one Pod:
$ kubectl -n ingress-nginx get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-nginx-controller 1/1 1 1 43mUpgrade ingress-nginx release by running the following helm commands:
$ helm upgrade -n ingress-nginx ingress-nginx -f values.yaml .
Release "ingress-nginx" has been upgraded. Happy Helming!
NAME: ingress-nginx
LAST DEPLOYED: Thu Nov 4 03:35:41 2021
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 5
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.Check current number of pods after the upgrade:
$ kubectl -n ingress-nginx get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-nginx-controller 3/3 3 3 45mUninstalling the Chart
To remove nginx ingress controller and all its associated resources that we deployed by Helm execute below command in your terminal.
$ helm -n ingress-nginx uninstall ingress-nginx
release "ingress-nginx" uninstalledStep 2: Configure Nginx Ingress Controller on Kubernetes
With Ingress Controller installed, we need to configure external connectivity method. For this we have two major options.
Option 1: Using Load Balancer (Highly recommended)
Load balancer is used to expose an application running in Kubernetes cluster to the external network. It provides a single IP address to route incoming requests to Ingress controller application. In order to successfully create Kubernetes services of type LoadBalancer, you need to have the load balancer implementation inside / or outside Kubernetes.
When a service is deployed in cloud environment, Load Balancer will be available to your service by default. Ingress service should get the LB IP address automatically. But for Baremetal installations you’ll need to deploy Load Balancer implementation for Kubernetes, we recommend MetalLB, use guide in link below to install it.
1. Setting Nginx Ingress to use MetalLB
Check Nginx Ingress service.
$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.108.4.75 <none> 80:30084/TCP,443:30540/TCP 5m55s
ingress-nginx-controller-admission ClusterIP 10.105.200.185 <none> 443/TCP 5m55sIf your Kubernetes cluster is a “real” cluster that supports services of type LoadBalancer, it will have allocated an external IP address or FQDN to the ingress controller.
Use the following command to see that IP address or FQDN:
$ kubectl get service ingress-nginx-controller --namespace=ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.108.4.75 <none> 80:30084/TCP,443:30540/TCP 7m53sPatch ingress-nginx-controller service by setting service type to LoadBalancer.
kubectl -n ingress-nginx patch svc ingress-nginx-controller --type='json' -p '[{"op":"replace","path":"/spec/type","value":"LoadBalancer"}]'Confirm successful patch of the service.
service/ingress-nginx-controller patchedService is assigned an IP address automatically from Address Pool as configured in MetalLB.
$ kubectl get service ingress-nginx-controller --namespace=ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.108.4.75 192.168.1.30 80:30084/TCP,443:30540/TCP 10m2. Mapping DNS name for Nginx Ingresses to LB IP
We can create domain name, preferably wildcard for use when creating Ingress routes in Kubernetes. In our cluster, we have k8s.example.com as base domain. We’ll use a unique wildcard domain *.k8s.example.com for Ingress.
The mapping is *.k8s.example.com pointing to IP address 192.168.1.30 (Nginx Ingress LB IP).
Option 2: Using Specific Nodes to run Nginx Ingress Pods (NOT recommended)
This is not recommended implementation and exists to serve as reference documentation.
1. Label nodes that will run Ingress Controller Pods
The node selector is used when we have to deploy a pod or group of pods on a specific group of nodes that passed the criteria defined in the configuration file.
List nodes:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster01.example.com Ready control-plane,master 14h v1.23.5
k8smaster02.example.com Ready control-plane,master 14h v1.23.5
k8sworker01.example.com Ready <none> 13h v1.23.5
k8sworker02.example.com Ready <none> 13h v1.23.5
k8sworker03.example.com Ready <none> 13h v1.23.52. Edit ingress-nginx-controller service and set externalIPs
In a private Infrastructure Kubernetes deployment setup it is unlikely that you’ll have Load Balancer service support.
$ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.101.4.21 <none> 80:30248/TCP,443:30773/TCP 3m53sYou’ll notice the service of NodePort type. We’ll update the service by making Ingress bind to specific IP address by using External IPs.
Kubernetes supports assignment of external IP addresses to a Service spec.externalIPs field through the ExternalIP facility. This will expose an additional virtual IP address, assigned to Nginx Ingress Controller Service. This allows us to direct traffic to a local node for load balancing.
In my Kubernetes Cluster I have two control Plane nodes with below primary IP addresses:
- k8smaster01.example.com – 192.168.42.245
- k8smaster02.example.com – 192.168.42.246
IP addresses of the nodes can be checked by running the command:
kubectl get nodes -o wideI’ll create a file containing modification to add External IPs to the service.
$ vim external-ips.yaml
spec:
externalIPs:
- 192.168.42.245
- 192.168.42.246Let’s now apply the patch to the service.