Difficulty: Advanced
Estimated Time: 10 minutes

Advanced: Querying with low tail-latency and low cost - Query caching with Thanos

Welcome to the tutorial where you learn how to introduce query caching using Query Frontend with Thanos.

In this tutorial you will learn:

  • Setup a simple observability stack using Prometheus
  • Deploy and configure Thanos Query Frontend in front of your Queriers so that we can cache query responses

We aim to achieve:

  • Low cost query execution
  • Low latency query execution
  • Si reduce cost of infrastructure

Feedback

Do you see any bug, typo in the tutorial or you have some feedback for us? Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io

Contributed by:

Summary

Congratulations! 🎉🎉🎉

You completed our Query caching tutorial with Thanos course! Goi ahead and try it on your own stack. To check out more advance patterns please have a look at kube-thanos!

Feedback

Do you see any bug, typo in the tutorial or you have some feedback for us? Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io

Advanced: Querying with low tail-latency and low cost - Query caching with Thanos

Step 1 of 2

Initial Prometheus and Thanos Setup

Simple observability stack with Prometheus and Thanos

NOTE: Click Copy To Editor for each config to propagate the configs to each file.

Let's imagine we have to deliver centralized metrics platform to multiple teams. For each team we will have a dedicated Prometheus. These could be in the same environment of in different environments (data centers, regions, clusters etc).

And then we will try to provide low cost, fast global overview. Let's see how we achieve that with Thanos.

Let's lay the ground work

Let's quickly deploy Prometheuses with sidecars and Querier.

Execute following commands to setup Prometheus:

Prepare "persistent volumes" for Prometheuses

mkdir -p prometheus_data

Deploy Prometheus

Let's deploy a couple of Prometheus instances and let them scrape themselves, so we can produce some metrics.

Prepare configuration

Click Copy To Editor for each config to propagate the configs to each file.

First, Prometheus server that scrapes itself:

global:
  scrape_interval: 5s
  external_labels:
    cluster: eu0
    replica: 0

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']
global:
  scrape_interval: 5s
  external_labels:
    cluster: eu1
    replica: 0

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9091']
global:
  scrape_interval: 5s
  external_labels:
    cluster: eu2
    replica: 0

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9092']

Deploy Prometheus

for i in $(seq 0 2); do
docker run -d --net=host --rm \
    -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \
    -v $(pwd)/prometheus_data:/prometheus"${i}" \
    -u root \
    --name prometheus"${i}" \
    quay.io/prometheus/prometheus:v2.22.2 \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/prometheus \
    --web.listen-address=:909"${i}" \
    --web.external-url=https://[[HOST_SUBDOMAIN]]-909"${i}"-[[KATACODA_HOST]].environments.katacoda.com \
    --web.enable-lifecycle \
    --web.enable-admin-api && echo "Prometheus ${i} started!"
done

Verify

Let's check if all of Prometheuses are running!

docker ps

Inject Thanos Sidecars

for i in $(seq 0 2); do
docker run -d --net=host --rm \
    -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \
    --name prometheus-sidecar"${i}" \
    -u root \
    quay.io/thanos/thanos:v0.17.0 \
    sidecar \
    --http-address=0.0.0.0:1909"${i}" \
    --grpc-address=0.0.0.0:1919"${i}" \
    --reloader.config-file=/etc/prometheus/prometheus.yml \
    --prometheus.url=http://127.0.0.1:909"${i}" && echo "Started Thanos Sidecar for Prometheus ${i}!"
done

Verify

Let's check if all of Thanos Sidecars are running!

docker ps

Prepare Thanos Global View

And now, let's deploy Thanos Querier to have a global overview on our services.

Deploy Querier

docker run -d --net=host --rm \
    --name querier \
    quay.io/thanos/thanos:v0.17.0 \
    query \
    --http-address 0.0.0.0:10912 \
    --grpc-address 0.0.0.0:10901 \
    --query.replica-label replica \
    --store 127.0.0.1:19190 \
    --store 127.0.0.1:19191 \
    --store 127.0.0.1:19192 && echo "Started Thanos Querier!"

Setup Verification

Once started you should be able to reach the Querier and Prometheus.

prometheus0.yml
prometheus1.yml
prometheus2.yml
frontend.yml
nginx.conf
Terminal
Prometheus
Thanos Query
Thanos Query Frontend