Since this application is running over Docker containers, in order to collect all monitoring data, the Datadog Agent runs as a container alongside the one of your application. The Datadog Agent container is configured via environment variables and mounting volumes on the underlying host:

agent:
    image: "datadog/agent:7.21.0"
    environment:
      - DD_API_KEY
      - DD_APM_ENABLED=true
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_PROCESS_AGENT_ENABLED=true
      - DD_DOCKER_LABELS_AS_TAGS={"my.custom.label.team":"team"}
      - DD_TAGS='env:ruby-shop'
    ports:
      - "8126:8126"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    labels:
      com.datadoghq.ad.logs: '[{"source": "datadog-agent", "service": "agent"}]'

By default, the Agent needs only your Datadog API key to start collecting metrics from your other containers, but by adding extra configurations, the Agent can also collect your application Traces, logs, and processes data:

Trace collection

To allow for Trace collection coming from other containers, the port 8126 is open on the Agent container.

Next, the DD_APM_ENABLED environment variable is set to true. Although enabled by default, setting this variable lets other people know you are using APM.

Learn more on the Datadog APM documentation

Log collection

The following configuration lines in your docker-compose.yml file at the root of the workshop directory allow to enable log collection for all running containers:

 agent:
   environment:
     # (...)
     - DD_LOGS_ENABLED=true
     - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
   volume:
     # (...)
     - /opt/datadog-agent/run:/opt/datadog-agent/run:rw

Refer to the Datadog Agent log collection documentation to learn more.

Collecting every type of data available is key to go beyond just collecting your logs. Each type of data represents a new facet or a new approach to solving a given issue. To bind the data together, Datadog uses three tags:

• The hostname tag (or the containerId if working with containers)
• The source tag
• The service tag

Hostname tag

The hostname (or containerId) tags are collected automatically by the Datadog Agent. To overwrite these tags, edit the main datadog.yaml configuration file or use an environment variable.

Source tag

The source tag enables log integration pipelines and enforces the Datadog naming conventions

Datadog has a wide range of log integrations. To enable the Log integration pipelines in Datadog, pass the source name as a value for the source attribute with a docker label. See the full list of supported log sources.

Learn more about Datadog attribute naming convention.

Service tag

The service tag binds metrics traces and logs.

It works automatically with known images, but if you have a custom application name, the service name may not match the services in APM, so you need to set it up with a custom label:

labels:
      com.datadoghq.ad.logs: '[{"source": "python", "service": "discounts-service"}]'

In the Datadog Container Ecosystem, Labels are key to leveraging the most out of Datadog Autodiscovery, if you look at your docker-compose.yml file at the root of the workshop directory, you can see that each container has associated labels:

version: '3'
services:
  agent:
    # (...)
    labels:
      com.datadoghq.ad.logs: '[{"source": "agent", "service": "agent"}]'
  discounts:
    # (...)
    labels:
      com.datadoghq.ad.logs: '[{"source": "python", "service": "discounts-service"}]'
      my.custom.label.team: "discount"
  frontend:
    # (...)
    labels:
      com.datadoghq.ad.logs: '[{"source": "ruby", "service": "store-frontend"}]'
      my.custom.label.team: "frontend"
  advertisements:
    # (...)
    labels:
      com.datadoghq.ad.logs: '[{"source": "python", "service": "ads-service"}]'
      my.custom.label.team: "advertisements"
  db:
    # (...)
    labels:
      com.datadoghq.ad.logs: '[{"source": "postgresql", "service": "postgres"}]'

Now that the data is collected, enable the corresponding integration in Datadog to pre-configure your platform:

1. Go in Integrations -> Integrations.
2. Enable the Datadog - Docker integration.
3. Enable the Datadog - SSH integration.
4. Refresh the integration page.

Out of the box, the Docker integrations installed a Dashboard:

• Docker Dashboard

Note that on any given dashboard, you can click on a displayed metric to switch to the corresponding logs:

Our application is running over Docker containers, as such Datadog has a Container centric page:

• Containers view

Where you can see all containers running with their associated:

• Processes
• Metrics
• Logs

When zooming on a given container, you can see the processes running inside it with its key metrics:

and the corresponding logs being emitted in real-time by this container:

When collecting traces and logs and after enabling their collection, you can now see your logs attached to your trace:

And your logs have a trace attached