Sysdig: Container Troubleshooting and Visibility by bencer

Steps

Sysdig: Container Troubleshooting and Visibility

Step 1 of 6

Step 1 - Setup Scenario

In this scenario, you will explore a two instances Wordpress with an HAproxy load-balancer and a MySQL database. All services will be deployed using Docker containers.

Setup scenario

To begin, execute the following commands to launch the scenario:

docker run --name mysql -v /data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD='password' -d mysql docker run --name wp1 -e VIRTUAL_HOST=wp --link mysql:mysql -d wordpress docker run --name wp2 -e VIRTUAL_HOST=wp --link mysql:mysql -d wordpress docker run --name proxy -p 80:80 -e DEFAULT_HOST=wp -v /var/run/docker.sock:/tmp/docker.sock:ro -d jwilder/nginx-proxy:alpine

Automatically, a background script will setup Wordpress and start making requests against the Wordpress load balancer so we can simulate some live connections and traffic.

Note: you can access yourself to this Wordpress installation here: https://[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com/

Step 2 - Launch Sysdig

Sysdig works by capturing system calls being made within the Linux Kernel. This provides Sysdig with unique visibility into how applications are operating even inside containers. Think of wireshark for your entire system, not only network activity, that understands the Docker containers and other orchestration tools that you use.

Sysdig can operate both on the real-time data from a system, or on previously captured data. This allows users to debug in real-time, but also capture data that can be analyzed later on, when a container doesn't exist anymore, or shared with other members of the team.

Install and run Sysdig

Sysdig can be installed as a package but also deployed as a container onto the Docker host:

docker run -it --rm --name=sysdig --privileged=true \ --volume=/var/run/docker.sock:/host/var/run/docker.sock \ --volume=/dev:/host/dev \ --volume=/proc:/host/proc:ro \ --volume=/boot:/host/boot:ro \ --volume=/lib/modules:/host/lib/modules:ro \ --volume=/usr:/host/usr:ro \ sysdig/sysdig

Once we run Sysdig, will capture and display information about all system calls and events on the system. This includes timestamp, CPU where it was executed, process name, thread ID, event type and all the relevant arguments. This is what can see when running sysdig inside of the Sysdig container.

Note: (finish it with CTRL+C).

All this information can be too much for an individual to process, that's why Sysdig offers filters and aggregation scripts known as chisels.

Step 3 - Sysdig filters and chisels

Sysdig has filters that allow you to distill specific system calls (information) in a similar fashion to tcpdump.

Example: list all system calls related to files under /etc

sysdig "fd.name contains /etc"

Sysdig chisels take all this information available through the system calls and aggregate it to display a summarized report.

Example: list all running containers (identifies executed system calls in different user namespaces and reports the container name for each namespace)

sysdig -c lscontainers

Understanding Sysdig filters

You can filter using information from different sources: system calls and events, file descriptors (not only files or sockets but higher level concepts like file names, directories or IP addresses), process name, uid, pid, latency and many more. You can get the full list running sysdig -l.

What makes Sysdig special is the ability to filter based on Docker container information using filters like like container.id, container.name, container.mounts. But can also understand Kubernetes resources like (namespaces, services, deployments or pods), Mesos tasks and Marathon apps or groups.

Understanding Sysdig chisels

Because all the system calls information would be too much for an individual to process, Sysdig has created chisels. A chisel enables users to take the stream of data captured by Sysdig and only display the relevant, aggregated data.

You have available chisels to report on system state, applications protocols like HTTP or memcached, CPU usage, errors, I/O activity, logging (file or syslog), network activity, performance bottlenecks or security.

To view all the chisels available, run the command sysdig -cl

Step 4 - Sysdig usage: container exploration

To begin with, let's explore what is currently happening, identify which containers are executing workload and processing the responses.

List CPU usage per container

sysdig -c topcontainers_cpu

Top CPU usage processes running

But we can drill deeper to look at the CPU usage of each process running inside the containers:

sysdig -pc -c topprocs_cpu

Note: The -pc flag makes sysdig tell us which container each connection belongs to.

List network traffic per container

sysdig -pc -c topcontainers_net

Top network traffic processes running

sysdig -pc -c topprocs_net

Network connections

sysdig -pc -c topconns

List IO usage per container

sysdig -c topcontainers_file

Top IO usage processes running

sysdig -pc -c topprocs_file

But how to use these to resolve real life performance and error troubleshooting issues? Let's use filters too!

Step 5 - Sysdig usage: troubleshooting real examples

One of the hardest things to do when troubleshooting Linux systems in general and container-based infrastructures in particular is observing the data that processes and/or containers are exchanging, not only at the network level but also at the file level. Because now we know what we have executing, we have an approximate idea where we want to dig deeper. Let's see a few examples:

Reconstruct the resolv.conf used by Apache

_cechofds is one of the most useful chisels, allows to trace the data sent on the filtered file descriptors, either network or file system activity:

sysdig -c echo_fds "fd.filename=resolv.conf and container.name=wp1"

Watch HTTP requests to one of the containers

sysdig -s 2048 -A -c echo_fds "fd.port=80 and container.name=wp1"

Note: -s 2048 incrases the buffer size to print all the information.

Find out the files being read and written by Apache

sysdig -c topfiles_bytes "fd.type=file and container.name=wp1 and proc.name=apache2"

List the clients connected to MySQL, by established connections

sysdig -c fdcount_by fd.cip "evt.type=accept and container.name=mysql and proc.name=mysql"

Traffic Between Two Containers

MYSQL=172.18.0.2 WP=172.18.0.3 sysdig -pc -A s 2048 -c echo_fds "fd.ip=$MYSQL and fd.ip=$WP"

Analyze MySQL and Apache activity

System calls run by Apache and MySQL processes, filtering out some noise calls we are not interested in:

sysdig -v "proc.name=apache2 and proc.name=mysqld and evt.type!=gettimeofday and evt.type!=switch and evt.type!=io_getevents and evt.type!=futex and evt.type!=clock_gettime and evt.type!=epoll_wait and evt.type!=getsockopt and evt.type!=wait4 and evt.type!=select and evt.type!=semop"

Note: -v makes output verbose, full content of buffers is printed on the screen.

Step 6 - csysdig

The previous steps have focused on using full output and chisels together with filters to gain insight into the system activity. It's also possible to use the Sysdig console interface called csysdig, a powerful but simple and intuitive curses UI for sysdig.

csysdig can be understood as a combination of popular tools like strace, tcpdump, lsof, etc with a UI similar to htop. Unlike htop, csysdig has multiple views available for different troubleshooting scenarios:

Containers
Container Errors
Kubernetes-based views
Mesos-based views
Network activity (by container, host, and process)
File Activities
Processes Activity / CPU / Memory / File
Threads
Log activity
User Activity
Traces
Spectrograms

Launch the UI by running the command csysdig. Driven by keyboard shortcuts, the views menu can be accessed using F2. Select the option using arrow keys and press Enter to select the required view.

On each view you will be able to see different columns explained as you browse between the views, for example on the 'Containers' view you can see CPU, process count, memory, file I/O or network by container name and id. You can always select any of the rows and you will start filtering by that container, you will then see the processes inside them.

To learn more about csysdig we recommend checking out csysdig explained visually.

cd <directory>	Change directory
ls	List directory
echo 'contents' > <file>	Write contents to a file
cat <file>	Output contents of file

i	In Command Mode	Change into Insert Mode, you can now insert and edit text in the file
esc	In Insert Mode	Change into Command Mode, you can now execute commands
:wq	In Command Mode	Save and Exit

Welcome!

bencer - Sysdig: Container Troubleshooting and Visibility

Congratulations!

You've completed the scenario!

Scenario Rating

Steps