The Prometheus server requires a configuration file that defines the endpoints to scrape along with how frequently the metrics should be accessed.

The first half of the configuration defines the intervals.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

The second half defines the servers and ports that Prometheus should scrape data from. In this example, we have defined three targets running on different ports.

scrape_configs:
  - job_name: 'prometheus'

    static_configs:
      - targets: ['127.0.0.1:9090', '127.0.0.1:9100', '127.0.0.1:9323']
        labels:
          group: 'prometheus'

9090 is Prometheus itself. Prometheus exposes information related to its internal metrics and performance.

9100 is the Node Exporter Prometheus process. This exposes information about the Node, such as disk space, memory and CPU usage.

9323 is the Docker Metrics port just exposed. This reports information from the Docker Engine.

More information on the default ports can be found at https://github.com/prometheus/prometheus/wiki/Default-port-allocations

Copy the configuration to the editor to continue.

Prometheus runs as a Docker Container with a UI available on port 9090. Prometheus uses the configuration to scrape the targets, collect and store the metrics before making them available via API that allows dashboards, graphing and alerting.

Task

The following command launches the container with the Prometheus configuration. Any data created by Prometheus will be stored on the host, in the directory /prometheus/data. When we update the container, the data will be persisted.

docker run -d --net=host \
    -v /root/prometheus.yml:/etc/prometheus/prometheus.yml \
    --name prometheus-server \
    prom/prometheus

Once started, the dashboard. The next steps will explain the details and how to view the data.

The Docker Metrics endpoint will return information related to Docker. To collect metrics on the Docker Host it's required to run a Prometheus Node Exporter. Prometheus has many exporters that are designed to output metrics for a particular system, such as Postgres or MySQL.

Task

Launch the Node Exporter container. By mounting the host /proc and /sys directory, the container has accessed to the necessary information to report on.

docker run -d \
  -v "/proc:/host/proc" \
  -v "/sys:/host/sys" \
  -v "/:/rootfs" \
  --net="host" \
  --name=prometheus \
  quay.io/prometheus/node-exporter:v0.13.0 \
    -collector.procfs /host/proc \
    -collector.sysfs /host/sys \
    -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

If you're interested in seeing the raw metrics, they can be viewed with curl localhost:9100/metrics

With the containers launched, Prometheus will scrape and store the data based on the internals in the configuration.

Dashboards

The default Prometheus Dashboard reports internal information and provides a way to debug the metrics being collected. The dashboard can be viewed here

The dashboard will report the status of the scraping and the different targets at via the /targets page.

When running in production, you may wish to build a dashboard using Grafana, or a hosted Prometheus instance such as Weave Cortex.

Query Prometheus

To query the underlying metrics and create graphs, visit the graph page on the Dashboard at https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph

From here different metrics are queryable based on their name. This is done by adding a graph and entering a PromQL query or metrics label.

For example, querying for engine_daemon_container_actions_seconds_sum will show how many Docker actions are being performed. These actions are containers being started, deleted, created, committed, or changed.

The host metrics can be viewed too, for example, node_cpu will output the Docker Hosts CPU information.

Generate Metrics

Running additional containers will result in changes to the metrics produced, which are viewable via the graphs and queries.

docker run -d katacoda/docker-http-server:latest