Datadog Metrics

If you have a Grafana Cloud Advanced account with access to Enterprise plugins, you can forward metrics with Grafana Labs Datadog Metrics directly from your Datadog Agents. To learn more about the Grafana Cloud Enterprise stack, contact us.

To view a list of supported features, see Datadog metrics supported features.

The following configuration refers to <dd-cluster>.grafana.net in several places. The value of <dd-cluster> depends on your specific Grafana Cloud stack. To find out the value for your Grafana Cloud stack, go to the Prometheus metrics details in the desired stack, take the domain from the URL and replace prometheus- with dd-.

For example, if your Prometheus URL is https://prometheus-us-central1.grafana.net/api/prom, your Hosted Datadog Metrics API can be found on https://dd-us-central1.grafana.net/datadog.

Forwarding metrics

You can forward Datadgo metrics to Grafana Cloud from either the Datadog Agent or Datadog’s observability pipeline, Vector.

Start by obtaining a Grafana Cloud API key with the MetricsPublisher role as well as your Grafana Cloud username. Next, follow the instructions specific to the Datadog Agent or Vector.

Forward metrics using the Datadog Agent

If you want to forward metrics to Grafana Cloud using the Datadog Agent, follow the instructions that match your desired use case:

Forward metrics to both Datadog and Grafana Cloud
Only forward metrics to Grafana Cloud

Forward metrics to both Datadog and Grafana Cloud

You can set up your Datadog Agent to forward Datadog metrics simultaenously to Grafana Cloud and Datadog. We recommend this approach for current Datadog users who want to evaluate Grafana Cloud.

The instructions below ensure that the Datadog Agent handles metrics pushes to Grafana Cloud and Datadog in separate, isolated queues. This means that problems pushing to one will not affect the other.

The following use cases require different configurations:

The Datadog Agent runs as a service or on any scheduler that can use secrets, such as Kubernetes.
Export your credentials as the environment variables GRAFANA_CLOUD_USERNAME and GRAFANA_CLOUD_APIKEY.
After exporting your credentials, export the DD_ADDITIONAL_ENDPOINTS environment variable:
```
DD_ADDITIONAL_ENDPOINTS='{"https://$(GRAFANA_CLOUD_USERNAME):$(GRAFANA_CLOUD_APIKEY)@<dd-cluster>.grafana.net/datadog": ["grafana-labs"]}'
```

The Datadog Agent runs as a system service.

Export the environment variable in the daemon service configuration:

```bash
Environment=DD_ADDITIONAL_ENDPOINTS='{"https://<username>:<api-key>@<dd-cluster>.grafana.net/datadog": ["grafana-labs"]}'
```

You must use grafana-labs as the API key value for the additional Grafana Labs endpoint. Do not use your Datadog API key.

This is similar to setting the additional_endpoints key in datadog.yaml, however you can’t paste your API key directly into the YAML file because Datadog Agent will convert it to lowercase as part of the config parsing. Setting it through the DD_ADDITIONAL_ENDPOINTS environment variable does not go through that lowercase conversion.

Only forward metrics to Grafana Cloud

If you want to exclusively forward Datadog metrics to Grafana Cloud (rather than forwarding them simultaneously to Grafana Cloud and Datadog), you have two configuration options.

Use your Grafana Cloud credentials in the API key configuration section

# The host of the Datadog intake server to send Agent data to
dd_url: https://<dd-cluster>.grafana.net/datadog
api_key: <username>:<api-key>

Use basic authentication

# The host of the Datadog intake server to send Agent data to
dd_url: https://<username>:<api-key>@<dd-cluster>.grafana.net/datadog
api_key: grafana-labs

# Additional endpoints to forward metrics to other backends
additional_endpoints: {}

You must use grafana-labs as the value for the api_key field. Do not use your Datadog API key.

Forward metrics using Vector

To forward Datadog metrics to Grafana Cloud using Vector, you’ll need to add a new Datadog metrics sink to your Vector configuration.

Use the following sample configuration snippet below, replacing the <username>, <api-key> and <dd-cluster> fields. You must also update the inputs field to specify the source or transform ID(s) of the Datadog metrics you wish to forward.

[sinks.my_datadog_sink_id]
inputs = [ <list_of_source_or_transform_ids> ]
type = "datadog_metrics"
endpoint = "https://<dd-cluster>.grafana.net/datadog"
default_api_key = "<username>:<api-key>"
request.retry_attempts = 1 # See note below

The requirement to set request.retry_attempts to 1 is a workaround until the Grafana Cloud Datadog API supports sketch-type metrics. Today, when Vector sends a sketch metric, Grafana Cloud responds with a 404 error to indicate this metric type is not supported. By setting Vector’s retry_attempts to 1, we ensure that when this 404 error is received, Vector only retries the failed request once rather than retrying infinitely.

Grafana Datadog data source configuration

Configure a new Datadog data source in your Grafana instance.

In your Grafana instance, click Connections in the left-side menu.
Search for DataDog.
Install Datadog completing the following fields:
- Name: Datadog Grafana Cloud
- API URL: https://<dd-cluster>.grafana.net/datadog
- API Key: grafana-labs
- App Key: grafana-labs
- Enable Basic Auth: On
- Basic Auth User: <Username>
- Basic Auth Password: <Prometheus_API_Key>
  - Your API Key needs to have read permissions (e.g. the Viewer or Editor role)

To compare queries between your data source and the Datadog official API, create an additional data source using the same steps, but change following fields:

Name: Datadog
API URL: https://api.datadoghq.com
API Key: <Datadog_API_Key>
App Key: <Datadog_App_Key>
Enable Basic Auth: Off

Verify that metrics are available

When all of your Datadog Agents have been configured, verify that Grafana Cloud is properly receiving metrics.

In your Grafana instance, click Explore.
Select Datadog Grafana Cloud from the data source dropdown menu.
Use the panel to query any existing metric, for example datadog.Agent.running.
Click on Run query to validate that it returns the expected results.

Migrate Datadog dashboards to Grafana

You can use a publicly available Grafana tool to translate your Datadog dashboard files into Grafana dashboard files.

Prerequisites

Before you migrate your dashboards, you will need to download the Grafana migration binary and migration tool release.

Fetch the binary from the public URL
Releases are available under the domain dl.grafana.com.

$ agent_release_url=https://dl.grafana.com/tools/datadog-dash-translator/release
$ curl "$agent_release_url/latest_checksums.txt"
...
# choose the appropriate version and architecture from above
# for example datadog-dash-translator_0.1.0_Linux_x86_64.tar.gz
$ binary=datadog-dash-translator_0.1.0_Linux_x86_64.tar.gz
$ curl -O $binary "$agent_release_url/$binary"
$ tar -xvf $binary

Find a release
Access the list of releases using the CLI command, gsutil.

If you have Docker installed run the following commands:

$ BUCKET=gs://grafana-downloads/tools/datadog-dash-translator/release
$ docker run -it google/cloud-sdk:alpine gsutil ls $BUCKET

If gsutil is available locally run:

$ BUCKET=gs://grafana-downloads/tools/datadog-dash-translator/release
$ gsutil ls $BUCKET

Migrate a dashboard

Extract the binary and instructions.

$ tar -xvf datadog-dash-translator_0.1.0_Linux_x86_64.tar.gz
INSTRUCTIONS.md
datadog-dash-translator

In the Datadog UI, find the dashboard you want to migrate.
Click Export dashboard JSON.

Locate the JSON file and run the following command:

$ ./datadog-dash-translator translate -i example-dashboard.json
INFO[0000] Successful read of datadog file "example-dashboard.json"
INFO[0000] Successful datadog dashboard translate
INFO[0000] Successful grafana dashboard write to "example-dashboard.json.grafana.json"

Locate the new file. The grafana dashboard will be available in a new file, for example: example-dashboard.json.grafana.json.
In your Grafana instance, in Dashboards, click the New and select Import
Click Upload dashboard JSON file and select the new file.