Creating Prometheus Dashboards with Grafana

Difficulty: beginner

Estimated Time: 20 minutes

As the number of parts of the system increases, it's essential to have to have a monitoring approach that is dynamic and scalable to match the system being built.

A popular open source solution is called Prometheus. Prometheus was built out of Soundcloud and now part of the Cloud Native Foundation.

Problem

What are my application and servers doing? How are they performing?

Solution

Collect metrics with Prometheus that can be used to produce Grafana dashboards

By using Prometheus we can collect system metrics via Node Exporters or instrument applications to provide metrics.

Your environment is currently being packaged as a Docker container and the download will begin shortly. To run the image locally, once Docker has been installed, use the commands

cat scrapbook_prometheus_creating-dashboards-with-grafana_container.tar | docker load

docker run -it /prometheus_creating-dashboards-with-grafana:20210301

Restart Scenario

Next Scenario

Step 1 of 5

Step 1 - Start Prometheus

The Prometheus server requires a configuration file that defines the endpoints to scrape along with how frequently the metrics should be accessed.

The first half of the configuration defines the intervals.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

The second half defines the servers and ports that Prometheus should scrape data from. In this example, we have defined two targets running on different ports.

scrape_configs:
  - job_name: 'prometheus'

    static_configs:
      - targets: ['127.0.0.1:9090', '127.0.0.1:9100']
        labels:
          group: 'prometheus'

9090 is Prometheus itself. Prometheus exposes information related to its internal metrics and performance and allows it to monitor itself.

9100 is the Node Exporter Prometheus process. This exposes information about the Node, such as disk space, memory and CPU usage.

More information on the default ports can be found at https://github.com/prometheus/prometheus/wiki/Default-port-allocations

Task

Copy the configuration to the editor to continue.

Step 2 - Start Node Exporter

Prometheus can run as a Docker Container with a UI available on port 9090. Prometheus uses the configuration to scrape the targets, collect and store the metrics before making them available via API that allows dashboards, graphing and alerting.

Task

The following command launches the container with the prometheus configuration. Any data created by prometheus will be stored on the host, in the directory /prometheus/data. When we update the container, the data will be persisted.

docker run -d --net=host \
    -v /root/prometheus.yml:/etc/prometheus/prometheus.yml \
    --name prometheus-server \
    prom/prometheus

Step 3 - View Prometheus Data

To collect metrics related to a node it's required to run a Prometheus Node Exporter. Prometheus has many exporters that are designed to output metrics for a particular system, such as Postgres or MySQL.

Task

Launch the Node Exporter container. By mounting the host /proc and /sys directory, the container has accessed to the necessary information to report on.

docker run -d \
  -v "/proc:/host/proc" \
  -v "/sys:/host/sys" \
  -v "/:/rootfs" \
  --net="host" \
  --name=prometheus \
  quay.io/prometheus/node-exporter:v0.13.0 \
    -collector.procfs /host/proc \
    -collector.sysfs /host/sys \
    -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

If you're interested in seeing the raw metrics, they can be viewed with curl localhost:9100/metrics

prometheus.yml

Terminal

Prometheus

cd <directory>	Change directory
ls	List directory
echo 'contents' > <file>	Write contents to a file
cat <file>	Output contents of file

i	In Command Mode	Change into Insert Mode, you can now insert and edit text in the file
esc	In Insert Mode	Change into Command Mode, you can now execute commands
:wq	In Command Mode	Save and Exit

Welcome!

Creating Prometheus Dashboards with Grafana

Problem

Solution

Congratulations!

You've completed the scenario!

Scenario Rating

Steps