Namespace for monitoring

We have created a namespace called pad-monitoring on OpenShift where we will deploy all our monitoring applications (Prometheus and Grafana).

• To make sure we are using our namespace run the following command:
  oc project pad-monitoring

• You can also go to the OpenShift console and login to the namespace there.

• The credentials to access the OpenShift console are:

  • Username: developer
  • Password: developer

Configure Prometheus for our application

In OpenShift, we use ConfigMaps to manage configurations for our applications (more info).

• We will use the following ConfigMap to set up our Prometheus instance.

apiVersion: template.openshift.io/v1          # Click on 'Copy to Editor' --->
kind: Template
objects:
  - apiVersion: v1          
    kind: ConfigMap
    metadata:
      name: prometheus-demo
      namespace: pad-monitoring
    data:     
      prometheus.yml: |     # Prometheus configuration starts below
        global:                   
          external_labels:
            monitor: prometheus
        scrape_configs:
          - job_name: 'prometheus'

            static_configs:
              - targets: ['localhost:9090'] # Configure Prometheus to scrape itself
                labels:
                  group: 'prometheus'

• Click on Copy to Editor for the above yaml block, to copy it to the editor. This will replace all the text in the editor with the above yaml text block

• Now we need to edit the ConfigMap for our Prometheus deployment, so that Prometheus knows to scrape our demo application for metrics. To do that, we need to add the following section:

              - targets: ['metrics-demo-app-metrics-demo.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com'] # Click on 'Copy to Editor'->
                labels:
                  group: 'pad'

In the above yaml block, we have defined a new targets list for our Prometheus to collect metrics from.

This targets list has the hostname for our demo application, i.e. metrics-demo-app-metrics-demo.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com

By default Prometheus collects metrics from the /metrics http endpoint. (More info on Prometheus Configuration)

Above, you can see that we have added a label group: 'pad'
So all the metrics from our demo application can be queried with a singe PromQL query, i.e. {group="pad"}

The ConfigMap file should be stored in a file named prometheus-configmap.yaml,
we will use it to update our Prometheus instance in the next step.

Deploy Prometheus with new configuration

• Once we have updated the configuration with our new target, we can go ahead and update this ConfigMap in our namespace:
  oc process -f ~/prometheus-configmap.yaml | oc apply -f -

• Update the Prometheus deployment using the following command:
  oc rollout latest dc/prometheus-demo -n pad-monitoring

After this, it might take a couple minutes for the Prometheus server to be ready to take requests.

The Prometheus Console should be available here

You can also use the OpenShift dashboard to check on the Prometheus deployment.

• The credentials to access the OpenShift console are:
  • Username: developer
  • Password: developer

Check if the configuration was updated successfully

Once the Prometheus server is ready to take requests,

• Visit the targets page for Prometheus, here.
  The targets page should show you a list of targets that Prometheus has been configured to collect metrics from.

It should look something like:

• If you cannot find the demo application, in the targets list,
  Try to fix your Prometheus configuration in the ConfigMap file prometheus-configmap.yaml and repeat the previous step.

Check application metrics

• Once Prometheus is deployed, you can click here to see all the metrics for the demo application. This should query the Prometheus instance for all the metrics with the label group="pad".

• To learn more about the Prometheus query language or PromQL you can visit this page.

Setting up Grafana to visualize metrics

Now, we will deploy Grafana and connect it to our Prometheus instance, so we are able to visualize metrics better.

Deploying Grafana

• We can directly deploy the official container image for Grafana using the following command:
  oc new-app grafana/grafana:6.6.1 -n pad-monitoring

  You should see some output like this:

  --> Creating resources ...
      imagestream.image.openshift.io "grafana" created
      deploymentconfig.apps.openshift.io "grafana" created
      service "grafana" created
  --> Success
      Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
       'oc expose svc/grafana'
      Run 'oc status' to view your app.
  

• Then we need to expose the service so that it is accessible outside of the cluster.
  oc expose svc/grafana -n pad-monitoring

• In about a minute, the Grafana console should be available at:
  https://grafana-pad-monitoring.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com

• You might see a warning like in the image below, this is because we don't generate valid ssl certificates for all our routes. You can click on Advanced... and then click on Accept the Risk and Continue to continue to the Grafana login page.

  

The default credentials for Grafana are:

• username: admin

• password: admin

  

• Next it will prompt you to change the default credentials,
  you can safely skip this for the purposes of this exercise.
  

• Next you should see the Grafana Welcome page.
  

Configuring Grafana to visualize metrics from Prometheus

• On the Grafana Welcome page, click on Add data source button to add Prometheus as a data source.

• You should see a list of Grafana compatible data source types. You can type in Prometheus to filter the results.
  

• When you click on the Prometheus Data Source, you should see the Data Source settings.
  

  • Fill the URL in the HTTP section with the Prometheus URL:
    http://prometheus-demo-route-pad-monitoring.[[HOST_SUBDOMAIN]]-80-[[KATACODA_HOST]].environments.katacoda.com

  • Then click on Save & Test, this should test the connection to the Prometheus server.

That's it, the connection between Grafana and Prometheus has been configured!

Visualizing Metrics using Grafana

Now, we will create a dashboard in Grafana and visualize a simple metric from the food store application.

We will visualize a metric called flask_http_request_total, it is the total number of requests served by the food store application.

Creating a Dashboard

• Let's start with creating a dashboard for our visualizations.

• In the left panel, click on Create and then select Dashboard
  

Creating a Panel

• You should see a New Panel like in the picture above. Click on Add Query.

  You can have multiple panels in a single dashboard, and each panel supports different types of visualizations (more info).
  But here we will just look at line charts as we are dealing with time-series data.

• Below the graphing area, you should have an empty text box for your PromQL queries.
  Enter your query flask_http_request_total

• In the upper right corner, you can change the graphing duration.
  Change the duration to Last 5 minutes.
  

  you can choose any other value as well just keep in mind that we have just set up the food store application monitoring, so only very recent metric values might be available.

• Once you can see some lines in your graphing area, click on the left arrow (<-) in the upper left corner.

• Now you should have a dashboard with a single panel, let's add another panel with a different metric.

• In the upper right corner of the dashboard page, click on Add Panel and repeat the previous steps only this time use a different query.
  rate(flask_http_request_total[2m]) this should give you the rate of requests served by the application over 2 minutes.

• Once you have set up the second panel, your dashboard should look something like this
  

If you want to see live changes to the dashboard, go to the food store application page and make some purchases.
Then click on the refresh button in the upper right corner.

That's it! you should have your own monitoring dashboard now.

You can do a lot more with dashboards like saving them, exporting as jsons, using variables, etc.
You can learn more about dashboards here