Before exploring observability topics, start a small application to provide something to observe.

Run 3 instances of the random-logger container to start generating continuously random logging events.

kubectl create deployment random-logger --image=chentex/random-logger

Scale to 3 instances.

kubectl scale deployment/random-logger --replicas=3

The 3 pods will start shortly.

kubectl get pods

Thank you to Vicente Zepeda for providing this beautifully simple container. The source code is here.

The running state of an application can be observed through a variety of kubectl describe commands across various resources.

Inspect the whole cluster

kubectl cluster-info

or more verbose (and really too much)

kubectl cluster-info dump --all-namespaces

Inspect this Kubernetes cluster only Worker node.

kubectl describe node node01

Inspect the last deployment.

kubectl describe deployment random-logger

Specifically, the replica state.

kubectl describe deployments | grep "Replicas:"

Inspect the 3 pods.

kubectl get pods

kubectl describe pods

Events

Kubernetes also maintains a list of events.

kubectl get events

Scaling is a type of event. Scale down the pod from 3 down to 2.

kubectl scale deployment/random-logger --replicas=2

Notice the last event will reflect the scaling request.

kubectl get events --sort-by=.metadata.creationTimestamp

These events are not to be confused with security audit logs which are also recorded.

Inspecting Containers

You can also typically get into a running container and inspect it as well. Get the name of the first Pod.

POD=$(kubectl get pod -o jsonpath="{.items[0].metadata.name}")

Inspect the script contents inside the container file system.

kubectl exec $POD -- cat entrypoint.sh

Or, shell into the container.

kubectl exec -it $POD -- /bin/sh

and come back out with the exit command.

There is a wealth of helpful Linux commands to give you information about the Linux containers. Here are just a few.

kubectl exec $POD -- uptime

kubectl exec $POD -- ps

kubectl exec $POD -- stat -f /

kubectl exec $POD --container random-logger -- lsof

kubectl exec $POD --container random-logger -- iostat

When the Pod has more than one container, the specific container name may be referenced.

kubectl exec $POD --container random-logger -- ls -a -l

Every Node in a Kubernetes cluster has a Kubelet process. Within each Kubelet process is a cAdvisor. The cAdvisor continuously gathers metrics about the state of the Kubernetes resources on each Node. This metrics information is vital to monitor to understand the state of the cluster. This wealth of information is available through the Resource Metrics API. Let's inspect the metrics.

Each node exposes statistics continuously updated by cAdvisor. For your cluster, get a list of the node names.

kubectl get nodes

For this small Kubernetes cluster on Katacoda the two names are named master and node01.

export NODE=master

Open a proxy to the Kubernetes API port.

kubectl proxy &

Access the worker node statistics with this command to the Metrics API.

curl localhost:8001/api/v1/nodes/$NODE/proxy/stats/

curl localhost:8001/api/v1/nodes/$(kubectl get nodes -o=jsonpath="{.items[0].metadata.name}")/proxy/metrics

The Kubernetes API aggregates cluster-wide metrics at /metrics.

curl localhost:8001/metrics

The de factor light monitoring application for Kubernetes is metrics-server. Metrics Server is a metrics aggregator. It discovers all nodes on the cluster and queries each node’s kubelet for CPU and memory usage. There is no long term metrics storage, it holds just the latest metrics. Typically, the server may be installed with a Helm chart.

helm install stable/metrics-server \ --name metrics-server \ --namespace kube-system \ --set args[0]="--kubelet-preferred-address-types=InternalIP" \ --set args[1]="--kubelet-insecure-tls"

This will install the server in the kube-system namespace along with the last two parameters that allow it to work well in this ephemeral Katacoda cluster.

In a few minutes you should be able to list metrics using the following command:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

If the metrics are not ready, this message will appear

Error from server (ServiceUnavailable): the server is currently unable to handle the request

Once the metrics are ready, a JSON dump of the metrics will appear.

kubectl top node

If the metrics are not ready you may get this message.

Error from server (ServiceUnavaliable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

However, once the metrics are available the normal message should look like this:

NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master   125m         6%     1049Mi          55%
node01   84m          2%     922Mi           23%

Metrics information is also reflected in the dashboard. Launch the Kubernetes dashboard and in pages for each resource the same Top information appears in the UI. The Horizontal Pod Autoscalar also utilizes these vital metrics to make decisions to scale up and down Pod instances.

In the past, there was no Resource Metrics API and a service called Heapster, now deprecated, used to gather all the cAdvisor metrics and bit more manually. Around the 1.6 to 1.8 Kubernetes releases the Resource Metrics API was added. In concert, Heapster was removed and Metrics Server is now the de facto service that aggregates metrics from the Metrics API.

Metrics-server is a lighter version of Heapster. It gathers the latest metrics for reference and does not store historical data. For accumulation of trending metrics the de facto Prometheus time-series database can optionally be added to a cluster.

The exposed Resource Metrics API is documented here.

Another metric gathering server is kube-state-metrics. It is used to provide metrics information for PPrometheus. Once you need more metrics that are gathered over time, then typically Prometheus is added to the cluster.