How Kubernetes probes work

I’m going to show you, really show you, how probes work in Kubernetes. How they can make your application more resilient, and how they can help you prevent avoidable mistakes. Like restart loops that take hours to recover from, and dropping requests during rollouts.

Every interactive demo in this post uses webernetes, my partial port of Kubernetes to TypeScript. It contains more than 100,000 lines of ported Kubernetes Go code to run a simulated cluster right here in your browser. I verified the behaviour of these demos against k3s and managed to find a bug in Kubernetes! More on that later.

A pod without probes

I want to run a pod with a single container. Here’s its manifest, pod-a.yaml:

1apiVersion: "v1"2kind: "Pod"3metadata:⋯4  name: "pod-a"5spec:⋯6  containers:⋯7    - name: "app"⋯8      image: "my-app:latest"

This image, my-app:latest, spends a few seconds initialising before listening on port 8080. You will see this below when you click restart to send the container a signal, causing it to crash and get started back up by Kubernetes. You can pause or reset any demo at any time.

After the first crash, the container restarts straight away. After the second, Kubernetes imposes a CrashLoopBackOff on it before starting it again. By default this delay is 10 seconds, doubling with each crash up to a maximum wait of 5 minutes. I shortened it to 3 seconds for this demo.

In both cases, Kubernetes considers the container Ready as soon as it starts, even though we know it’s not. It’s still doing startup work and not listening on port 8080.

Next I’ll add pod-b, which sends a request to pod-a every 2 seconds. Throughout the post, you can think of pod-b as any source of client traffic: an ingress controller, a load balancer, inter-service requests, etc.

If you restart pod-a in the demo below while a request is on its way, that request will fail.

From the moment you restart the container until its startup work finishes, requests will fail, even though the container is considered Ready! This is not what I want. I need Kubernetes to know when pod-a is ready to receive traffic.

For this, Kubernetes gives us probes. Probes are periodic checks sent to containers to determine their health. They come in three flavours:

• Startup probes determine whether my application inside the container has started.
• Readiness probes determine whether my application is ready to receive traffic.
• Liveness probes determine whether my application needs to be restarted.

It sounds like startup probes are best suited to the problem I showed you in the demos above, so let’s start there.

Startup probes

Below, I’ve added a startup probe to pod-a.yaml:

1apiVersion: "v1"2kind: "Pod"3metadata:⋯4  name: "pod-a"5spec:⋯6  containers:⋯7    - name: "app"⋯8      image: "my-app:latest"9      startupProbe:⋯10        httpGet:⋯11          path: "/startup"12          port: 808013        periodSeconds: 114        failureThreshold: 5

It’s an httpGet probe that sends a GET /startup request to the pod on port 8080. Status codes 200-399 count as a success. This happens every periodSeconds seconds, and is allowed to fail failureThreshold consecutive times before Kubernetes kills the container. This gives my container ~5 seconds to complete its startup work.

Probes are sent by a process called the kubelet. Each node in the cluster has its own kubelet, and it’s the kubelet’s job to make sure the right pods are running and being probed for each node.

When you restart pod-a below, it now shows as NotReady. Kubernetes is now aware that pod-a hasn’t initialised yet. It only becomes Ready after the first startup probe succeeds.