Difficulty: Beginner
Estimated Time: 10 minutes

Goal:

After completing this scenario, users will be able to trace disk latency and IO utilization with eBPF.

Concepts included in this scenario:

  • Install bcc-tools
  • Run trivial eBPF tracing tools to get familiar with the concept
  • Using bpftool to verify use of eBPF
  • Tracing disk latency with eBPF
  • Tracing disk IO utilization with eBPF

Example Usecase:

To increase reliability of datacenter operations we want to monitor disk latency on the systems. Sudden spikes in disk latencies might indicate possible issues with the disks or storage systems. We also want to monitor what applications are utilizing most of the IO on these disks.

Kernel Tracing with eBPF [Technology Preview]

Step 1 of 6

Step 1

Install bcc-tools

yum install -y bcc-tools kernel-devel-4.18.0-80.el8

eBPF is a modern mechanism which allows instrumenting of kernel in secure restrictred environment without loading kernel modules. Technicaly eBPF is accessible via new syscall bpf(). The code of the eBPF program needs to be passed to the kernel via this syscall in the form of BPF assembly code. Fortunately bcc-tools package provide many useful pre-created tools created using eBPF.

Inspect the content of the bcc-tools package:

rpm -ql bcc-tools | grep /usr/share/bcc/tools/

All the pre-created tools are delivered under /usr/src/bcc/tools:

$ rpm -ql bcc-tools
...
/usr/share/bcc/tools/argdist
/usr/share/bcc/tools/bashreadline
/usr/share/bcc/tools/biolatency
/usr/share/bcc/tools/biosnoop
...