Before exploring observability topics, start a small application to provide something to observe.

Run 3 instances of the random-logger container to start generating continuously random logging events.

kubectl create deployment random-logger --image=chentex/random-logger

Scale to 3 instances.

kubectl scale deployment/random-logger --replicas=3

The 3 pods will start shortly.

kubectl get pods

Thank you to Vicente Zepeda for providing this beautifully simple container. The source code is here.

General Inspection of a Cluster

When you first start interacting with a running cluster there are a few commands to help you get oriented with its health and state.

Inspect the cluster general state.

kubectl cluster-info

Inspect this Kubernetes cluster only Worker node.

kubectl describe node node01

For deeper details, you can generate a complete dump of the cluster state to a directory.

kubectl cluster-info dump --all-namespaces --output-directory=cluster-state --output=json

This creates a directory where each file is a report on all the nodes and namespaces.

tree cluster-state

There is a wealth of information you can mine.

• Show me all the container images names in the kube-system namespace. jq '.items[].status.containerStatuses[].image' cluster-state/kube-system/pods.json

• Show me when all the container images were started in the default namespace. jq '.items[].status.containerStatuses[] | [.image, .state[].startedAt]' cluster-state/default/pods.json

General Inspection for a Deployment

The running state of an application can be observed through a variety of kubectl describe commands across various resources.

Inspect the last deployment.

kubectl describe deployment random-logger

Specifically, the replica state.

kubectl describe deployments | grep "Replicas:"

Inspect the 3 pods.

kubectl get pods

kubectl describe pods

Events

Kubernetes also maintains a history of events.

kubectl get events

Scaling is a type of event. Scale down the Pod from 3 down to 2.

kubectl scale deployment/random-logger --replicas=2

Notice the last event will reflect the scaling request.

kubectl get events --sort-by=.metadata.creationTimestamp

These events are not to be confused with security audit logs which are also recorded.

Inspecting Containers

You can also typically get into a running container and inspect it as well. Get the name of the first Pod.

POD=$(kubectl get pod -o jsonpath="{.items[0].metadata.name}")

Inspect the script contents inside the container file system.

kubectl exec $POD -- cat entrypoint.sh

Or, shell into the container.

kubectl exec -it $POD -- /bin/sh

and come back out with the exit command.

There is a wealth of helpful Linux commands to give you information about the Linux containers. Here are just a few.

kubectl exec $POD -- uptime

kubectl exec $POD -- ps

kubectl exec $POD -- stat -f /

kubectl exec $POD --container random-logger -- lsof

kubectl exec $POD --container random-logger -- iostat

When the Pod has more than one container, the specific container name may be referenced.

kubectl exec $POD --container random-logger -- ls -a -l

Every Node in a Kubernetes cluster has a Kubelet process. Within each Kubelet process is a cAdvisor. The cAdvisor continuously gathers metrics about the state of the Kubernetes resources on each Node. This metrics information is vital to monitor to understand the state of the cluster. This wealth of information is available through the Resource Metrics API. Let's inspect the metrics.

Each node exposes statistics continuously updated by cAdvisor. For your cluster, get a list of the node names.

kubectl get nodes

For this small Kubernetes cluster on Katacoda, the two node names can be listed.

export NODE_0=$(kubectl get nodes -o=jsonpath="{.items[0].metadata.name}")

export NODE_1=$(kubectl get nodes -o=jsonpath="{.items[1].metadata.name}")

echo -e "The master node is $NODE_0 \nThe worker node is $NODE_1"

Open a proxy to the Kubernetes API port.

kubectl proxy > /dev/null &

Query the metrics for the master node.

curl localhost:8001/api/v1/nodes/$NODE_0/proxy/metrics

Query the metrics for the worker node.

curl localhost:8001/api/v1/nodes/$NODE_1/proxy/metrics

The Kubernetes API aggregates cluster-wide metrics at /metrics.

curl localhost:8001/metrics/ | jq

The de facto light monitoring application for Kubernetes is metrics-server. Metrics Server is a metrics aggregator. It discovers all nodes on the cluster and queries each node’s kubelet for CPU and memory usage. There is no long term metrics storage, it holds just the latest metrics. Typically, the server may be installed with a Helm chart.

Add the Bitnami chart repository for the Helm chart to be installed.

helm repo add bitnami https://charts.bitnami.com/bitnami

Install the chart.

helm install metrics-server bitnami/metrics-server \
  --version=4.2.2 \
  --namespace kube-system \
  --set apiService.create=true \
  --set extraArgs.kubelet-insecure-tls=true \
  --set extraArgs.kubelet-preferred-address-types=InternalIP

This will install the server in the kube-system namespace. It also add a new API endpoint named metrics.k8s.io. In a few moments you should be able to list metrics using the following command:

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq

If the metrics are not ready, this message will appear

Error from server (ServiceUnavailable): the server is currently unable to handle the request

Once the metrics are ready, a JSON dump of the metrics will appear. Additional metrics also appears in the top report.

kubectl top node

If the metrics are not ready you may get this message.

Error from server (ServiceUnavaliable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

or

error: metrics not available yet

However, once the metrics are available the normal message should look similar to this:

NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
controlplane   138m         6%     991Mi           52%
node01         79m          3%     575Mi           14%

Pod information can also be observed.

kubectl top pods --all-namespaces

Metrics information is also reflected in the dashboard. Launch the Kubernetes dashboard and in pages for each resource the same Top information appears in the UI. The also utilizes these vital metrics to make decisions to scale up and down Pod instances.

In the past, there was no Resource Metrics API and a service called Heapster, now deprecated, used to gather all the cAdvisor metrics and bit more manually. Around the 1.6 to 1.8 Kubernetes releases the Resource Metrics API was added. In concert, Heapster was removed and Metrics Server is now the de facto service that aggregates metrics from the Metrics API.

Metrics-server is a lighter version of Heapster. It gathers the latest metrics for reference and does not store historical data. For accumulation of trending metrics, the de facto Prometheus time-series database can optionally be added to a cluster.

The exposed Resource Metrics API is documented here.

Another metric gathering server is kube-state-metrics. It is used to provide metrics information for Prometheus. Once you need more metrics that are gathered over time, then typically Prometheus is added to the cluster.