Creating Prometheus Dashboards with Grafana

Step 1 - Start Prometheus

The Prometheus server requires a configuration file that defines the endpoints to scrape along with how frequently the metrics should be accessed.

The first half of the configuration defines the intervals.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

The second half defines the servers and ports that Prometheus should scrape data from. In this example, we have defined two targets running on different ports.

scrape_configs:
  - job_name: 'prometheus'

    static_configs:
      - targets: ['127.0.0.1:9090', '127.0.0.1:9100']
        labels:
          group: 'prometheus'

9090 is Prometheus itself. Prometheus exposes information related to its internal metrics and performance and allows it to monitor itself.

9100 is the Node Exporter Prometheus process. This exposes information about the Node, such as disk space, memory and CPU usage.

More information on the default ports can be found at https://github.com/prometheus/prometheus/wiki/Default-port-allocations

Task

Copy the configuration to the editor to continue.

Step 2 - Start Node Exporter

Prometheus can run as a Docker Container with a UI available on port 9090. Prometheus uses the configuration to scrape the targets, collect and store the metrics before making them available via API that allows dashboards, graphing and alerting.

Task

The following command launches the container with the prometheus configuration. Any data created by prometheus will be stored on the host, in the directory /prometheus/data. When we update the container, the data will be persisted.

docker run -d --net=host \
    -v /root/prometheus.yml:/etc/prometheus/prometheus.yml \
    --name prometheus-server \
    prom/prometheus

Step 3 - View Prometheus Data

To collect metrics related to a node it's required to run a Prometheus Node Exporter. Prometheus has many exporters that are designed to output metrics for a particular system, such as Postgres or MySQL.

Task

Launch the Node Exporter container. By mounting the host /proc and /sys directory, the container has accessed to the necessary information to report on.

docker run -d \
  -v "/proc:/host/proc" \
  -v "/sys:/host/sys" \
  -v "/:/rootfs" \
  --net="host" \
  --name=prometheus \
  quay.io/prometheus/node-exporter:v0.13.0 \
    -collector.procfs /host/proc \
    -collector.sysfs /host/sys \
    -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

If you're interested in seeing the raw metrics, they can be viewed with curl localhost:9100/metrics

Step 4 - Start Grafana

As Prometheus is scraping data, Grafana will allow us to create amazing dashboards to highlight the data.

Download the Grafana binaries from: cd ~ && curl -L -o ~/grafana-4.5.1.linux-x64.tar.gz https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.5.1.linux-x64.tar.gz && tar -zxvf ~/grafana-4.5.1.linux-x64.tar.gz

cd ~/grafana-4.5.1/bin/ && ./grafana-server

Note: This can also run as a Docker Container.

After starting Grafana, the dashboard will be available at http://[[HOST_SUBDOMAIN]]-3000-[[KATACODA_HOST]].environments.katacoda.com/

To login, the default username/password is admin/admin

Step 5 - Import Dashboard

Add a Data Source of our Prometheus server using the URL http://localhost:9090.

Make sure you set the Type to Prometheus

Go to https://[[HOST_SUBDOMAIN]]-3000-[[KATACODA_HOST]].environments.katacoda.com/dashboard/new?editview=import&orgId=1 and select "Import a dashboard"

Import the Dashboard with ID 22. This will graph data from the Node Exporter. https://grafana.com/dashboards/22

View your data.

cd <directory>	Change directory
ls	List directory
echo 'contents' > <file>	Write contents to a file
cat <file>	Output contents of file

i	In Command Mode	Change into Insert Mode, you can now insert and edit text in the file
esc	In Insert Mode	Change into Command Mode, you can now execute commands
:wq	In Command Mode	Save and Exit

Welcome!

Problem

Solution

Congratulations!

You've completed the scenario!

Scenario Rating

Steps