Manage your Kubernetes Monitoring configuration

Kubernetes Monitoring gathers metrics, logs, and events, and calculates costs for your infrastructure. It also provides recording rules, alerting rules, and allowlists.

How Kubernetes Monitoring works

Kubernetes Monitoring uses the following to provide its data and visualizations.

cAdvisor

cAdvisor (one per Node) is present on each Node in your Cluster, and emits container resource usage metrics such as CPU usage, memory usage, and disk usage. Alloy collects these metrics and sends them to Grafana Cloud.

Cluster events

Kubernetes Cluster controllers emit information about events concerning the lifecycle of Pods, deployments, and Nodes within the Cluster. Alloy pulls these Cluster events using the Kubernetes API server and converts them into log lines, then sends them to Grafana Cloud logs.

Grafana Alloy

Grafana Alloy:

Collects all metrics, Cluster events, and Pod logs
Receives traces pushed from applications on Clusters
Sends the data to Grafana Cloud

kube-state-metrics

Kubernetes Monitoring uses kube-state-metrics service (one replica, by default) to enable you to see the links between Cluster, Node, Pod, and container. The kube-state-metrics service listens to Kubernetes API server events, and generates Prometheus metrics that document the state of your Cluster’s objects. Over a thousand different metrics provide the status, capacity, and health of individual containers, Pods, deployments, and other resources.

kube-state-metrics:

Generates metrics without modification
Listens to the Kubernetes API server (not present on each Node)
Provides metrics on the state of objects in your Cluster (Pods, Deployments, DaemonSets)

The Kubernetes Monitoring Cluster navigation feature requires the following metrics:

kube_namespace_status_phase
container_cpu_usage_seconds_total
kube_pod_status_phase
kube_pod_start_time
kube_pod_container_status_restarts_total
kube_pod_container_info
kube_pod_container_status_waiting_reason
kube_daemonset.+
kube_replicaset.+
kube_statefulset.+
kube_job.+
kube_node.+
kube_cluster.+
node_cpu_seconds_total
node_memory_MemAvailable_bytes
node_filesystem_size_bytes
node_namespace_pod_container
container_memory_working_set_bytes

kubelet

kubelet (one per Node):

Is the primary Node agent present on each Node in the Cluster
Emits metrics specific to the kubelet process like kubelet_running_pods and kubelet_running_container_count
Ensures containers are running
Provides metrics on Pods and their containers

Grafana Alloy collects these metrics and sends them to Grafana Cloud.

Kubernetes mixins

Kubernetes Monitoring is heavily indebted to the open source kubernetes-mixin project, from which the recording and alerting rules are derived. Grafana Labs continue to contribute bug fixes and new features upstream.

Node Exporter

The Prometheus exporter