Grafana Agent configuration for OpenTelemetry

The recommended way to send OpenTelemetry data to Grafana Cloud or your Grafana OSS stack is with the Grafana Agent Flow.

Prerequisites

This document assumes that you have already installed a Grafana Agent and that you have access to a Grafana OSS or a Grafana Cloud stack. Please follow the Grafana Agent installation instructions if needed. If you would like to work with Grafana Cloud but do not have an account yet, you can easily create one for free.

Configuration

An agent-config.river configuration file is needed for the Grafana Agent to run successfully in flow mode. You can download one that will configure an agent to start with both gRPC and HTTP servers. The gRPC and HTTP servers will be available on their default endpoints.

gRPC: http://localhost:4317
HTTP: http://localhost:4318

The configuration file requires several environmental variable to be set. They are described in the following table.

Environment Variable	Description	Example
GRAFANA_CLOUD_API_KEY	The API key generated above	eyJvSomeLongStringJ9fQ==
GRAFANA_CLOUD_PROMETHEUS_URL	"Remote Write Endpoint" when you click on "Details" in the "Prometheus" section on grafana.com	https://prometheus-prod-***.grafana.net/api/prom/push
GRAFANA_CLOUD_PROMETHEUS_USERNAME	"Username / Instance ID" when you click on "Details" in the "Prometheus" section on grafana.com	11111
GRAFANA_CLOUD_LOKI_URL	The "url" in "From a standalone host" when you click on "Details" in the "Loki" section on grafana.com (without the username and password part)	https://logs-prod-***.grafana.net/loki/api/v1/push
GRAFANA_CLOUD_LOKI_USERNAME	"User" when you click on "Details" in the "Loki" section on grafana.com	11112
GRAFANA_CLOUD_TEMPO_ENDPOINT	The "endpoint" in "Sending Data to Tempo" when you click on "Details" in the "Tempo" section on grafana.com	tempo-***.grafana.net:443
GRAFANA_CLOUD_TEMPO_USERNAME	"User" when you click on "Details" in the "Tempo" section on grafana.com	11113

Running the Agent

Download the above-mentioned agent-config.river file and set the necessary environment variables. Then start the Grafana Agent in flow mode with the following command. Remember to first update the path to your agent-config.river file, so it has the correct location.

_{Note the below command is not for agents running within a docker container.}

AGENT_MODE=flow; /bin/agent run <PATH_TO_FILE>/agent-config.river

Once the agent has started, the agent’s log should include output indicating that the gRPC and HTTP servers have started.

ts=2023-04... level=info component=otelcol.receiver.otlp.default msg="Starting GRPC server" endpoint=0.0.0.0:4317
ts=2023-04... level=info component=otelcol.receiver.otlp.default msg="Starting HTTP server" endpoint=0.0.0.0:4318

Optional Smoke Test

Click here for a sample HTTP POST request.

You can use the below curl statement to smoke test the grafana agent’s HTTP trace endpoint.

curl --location 'http://localhost:4318/v1/traces' \
--header 'Content-Type: application/json' \
--data '{
"id": "12345",
    "name": "postman-fake-trace",
    "start_time": 1.333058486989E9,
    "trace_id": "1-agent-test-999",
    "end_time": 1.333058487934E9
}

Agent Logging

Grafana Agent logging defaults to:

level = “info”
format = “logfmt”

Add the following block to agent-config.river if you wish to change the logging level or format.

logging {
  level = <LEVEL>
  format = <FORMAT>
}

Refer to the documentation for all logging options.

Feedback