Every node in a Kubernetes cluster has a Kubelet process. Within each Kubelet is a cAdvisor process. The cAdvisor is continuously gathering metrics about the state of the cluster. It's handy to extract information right out of cAdvisor as it's always available and provides a wealth of information.

First, open a proxy to the Kubernetes API port.

kubectl proxy &

Each node provide statistics that ar provided by cAdvisor. Access the node stats with this command.

curl localhost:8001/api/v1/nodes/$(kubectl get nodes -o=jsonpath="{.items[0].metadata.name}")/proxy/stats/

The Kubernetes API also gather the cAdvisor metrics at /metrics.

curl localhost:8001/metrics

The cAdvisor also serves up the metrics is a helpful HTML format. Without Katacoda your would point your browser at the cluster IP and port 4194 with this command.

open $(minikube ip):4194

However, on Katacoda a slight different route is provided. See the cAdvisor page here.

Before exploring more observabiltiy topics, start a small application to provide something to observe.

Run 3 instances of the random-logger container to start generating continuously random logging events.

kubectl run random-logger --image=chentex/random-logger --replicas=3

The 3 pods will start shortly.

kubectl get deployments,pods

Thank you to Vicente Zepeda for providing this beautifully simple container. The source code is here.

Inspect Resources

The cAdvisor data can also be observed through a variety of kubectl describe commands across various resources.

Inspect the main Minikube node.

kubectl describe node minikube

Inspect the last deployment.

kubectl describe deployment random-logger

Inspect the 3 pods.

kubectl get deployments,pods

kubectl describe pods

Much of this same data is found on the cAdvisor page.

Events

Kubernetes also maintains a list of events.

kubectl get events

Scaling is a type of event. Scale down the pod from 3 down to 2.

kubectl scale deployment/random-logger --replicas=2

Notice the last event will reflectthe scaling request.

kubectl get events

These events are not to be confused with security audit logs which are also recorded.

Inspecting Containers

You can also typically get into a running container and inspect it as well. Get the name of the first Pod.

POD=$(kubectl get pod -o jsonpath="{.items[0].metadata.name}")

Inspect the script contents inside the container file system.

kubectl exec $POD -- cat entrypoint.sh

Or, shell into the container.

kubectl exec -it $POD -- /bin/sh

and come back out with the exit command.

There is a wealth of helpful Linux commands to give you information about the Linux containers. Here are just a few.

kubectl exec $POD -- uptime

kubectl exec $POD -- ps

kubectl exec $POD -- stat -f /

kubectl exec $POD --container random-logger -- lsof

kubectl exec $POD --container random-logger -- iostat

When the Pod has more than one container, the specific container name may be referenced.

kubectl exec $POD --container random-logger -- ls -a -l

A common light monitoring application for Kubernetes was Heapster. However, it's in the process of being retired and the substitution is metrics-server.

Typically this would be installed with a Helm chart, but for Minikube there is a pretested and opinionated configuration available through Minikube's addon feature.

minikube addons list

Enable the metrics-server addon.

minikube addons enable metrics-server

Launch the Minikube dashboard and in a few moments, charts and line graphs will start appearing on the dashboard pages. These graphs are provided by the metrics-server.