---
title: "Instrument agents with OpenTelemetry | Grafana Cloud documentation"
description: "Send generative AI spans with OpenTelemetry and inspect prompts and responses in Agent Observability."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Instrument agents with OpenTelemetry

> Note
> 
> OpenTelemetry ingestion in Agent Observability is currently in [public preview](/docs/release-life-cycle/). Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

Send generative AI spans with OpenTelemetry to inspect prompts and responses in Agent Observability. You don’t need the Agent Observability SDK.

Generative AI instrumentation is code or a library that records your application’s AI calls as OpenTelemetry spans. Each span describes a call with attributes, for example, the model provider, model name, and token usage. It can also capture prompts and responses. For the setup in this guide, use instrumentation that follows the OpenTelemetry generative AI semantic conventions, with attributes that use the `gen_ai.*` prefix.

You can use your framework’s built-in instrumentation, an instrumentation library, or spans you create yourself. General OpenTelemetry instrumentation doesn’t necessarily capture AI calls or their generative AI attributes. This guide shows you how to send compatible spans to Agent Observability and includes a sample you can create yourself.

## Before you begin

- A Grafana Cloud stack with Agent Observability enabled. Refer to [Set up Agent Observability](/docs/grafana-cloud/observe-and-act/agent-observability/get-started/grafana-cloud/#enable-agent-observability).
- A Cloud Access Policy Token with `traces:write` permission.
- An application with OpenTelemetry generative AI instrumentation, or Python 3.10 or later to try the sample on this page.

Export only content your data policy permits. Remove secrets and sensitive data before export.

## Connect to Grafana Cloud

Configure OpenTelemetry Protocol (OTLP) export to send traces to your Grafana Cloud stack. Choose a setup path:

- [Recommended setup through Grafana Cloud integrations](/docs/grafana-cloud/send-data/otlp/send-data-otlp/#recommended-opentelemetry-setup-via-grafana-cloud-integrations): Use a connection tile for guided setup, including connection settings and collector configuration.
- [Manual setup](/docs/grafana-cloud/send-data/otlp/send-data-otlp/#manual-opentelemetry-setup-for-advanced-users): Get your OTLP endpoint, generate an authentication token, and configure the export environment variables yourself. The guide also explains endpoint paths and Python authentication header encoding.

Use the OTLP connection details for your stack. You don’t need the Agent Observability SDK endpoint for this setup. For the Python sample below, use the HTTP/protobuf exporter settings.

If you already export through Grafana Alloy or an OpenTelemetry Collector, keep that route and preserve the span and resource attributes described below.

## Configure your instrumentation

Use instrumentation that follows the [OpenTelemetry generative AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). Don’t opt in calls that the Agent Observability SDK already exports as generations, because you can create duplicate records.

1. Check that your spans include these attributes:
   
   Expand table
   
   | Attribute               | Value                                                            |
   |-------------------------|------------------------------------------------------------------|
   | `gen_ai.operation.name` | `chat`, `generate_content`, `text_completion`, or `invoke_agent` |
   | `gen_ai.provider.name`  | Your model provider, for example, `openai`                       |
   | `gen_ai.request.model`  | Your model name, for example, `gpt-4o`                           |
   
   `invoke_agent` spans also need a provider and model to appear as generations. Tool calls and other operations don’t create standalone generations.
2. Opt in your application’s generative AI spans with a resource attribute:
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   export OTEL_RESOURCE_ATTRIBUTES="agento11y.record=true"
   ```
   
   If *OTEL\_RESOURCE\_ATTRIBUTES* already has a value, append `,agento11y.record=true` instead of replacing it. You can also set `agento11y.record=true` on individual spans. Set `agento11y.record=false` on a span to exclude it.
3. Enable content capture in your instrumentation if you want to inspect prompts and responses. It must write `gen_ai.input.messages` and `gen_ai.output.messages` as span attributes in the generative AI message format shown in the sample below. Content sent only through OTLP logs doesn’t populate conversations.
4. Set `gen_ai.conversation.id` on spans and reuse it for calls in the same conversation. Without it, each trace forms a separate conversation. Set `gen_ai.agent.name` to identify your agent; otherwise, Agent Observability uses the resource’s `service.name`.

Restart your application with the export settings and run an agent request. Make sure your application or instrumentation library initializes an OpenTelemetry tracer provider and exporter; setting export environment variables alone doesn’t instrument your application. Spans dropped by application or collector sampling don’t appear in Agent Observability.

## Try a sample without a model provider

If you don’t have generative AI instrumentation yet, use this Python example to check your connection. It sends a synthetic prompt and response without calling a model provider. If you already send spans from your application, refer to [Verify data](#verify-data).

1. Install the OpenTelemetry packages:
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   python -m pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
   ```
2. Save the following code as `otel_example.py`:
   
   Python ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```python
   import json
   from uuid import uuid4
   
   from opentelemetry import trace
   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   
   provider = TracerProvider(resource=Resource.create({"service.name": "otel-example"}))
   provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
   trace.set_tracer_provider(provider)
   tracer = trace.get_tracer("otel-example")
   
   with tracer.start_as_current_span("chat gpt-4o", kind=trace.SpanKind.CLIENT) as span:
       span.set_attributes({
           "agento11y.record": True,
           "gen_ai.operation.name": "chat",
           "gen_ai.provider.name": "openai",
           "gen_ai.request.model": "gpt-4o",
           "gen_ai.conversation.id": str(uuid4()),
           "gen_ai.input.messages": json.dumps([
               {"role": "user", "parts": [{"type": "text", "content": "Say hello."}]}
           ]),
           "gen_ai.output.messages": json.dumps([
               {"role": "assistant", "parts": [{"type": "text", "content": "Hello!"}]}
           ]),
       })
   
   provider.shutdown()
   ```
   
   `agento11y.record=true` opts the span into Agent Observability. Reuse `gen_ai.conversation.id` across calls that belong to the same conversation.
3. Run the example:
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   python otel_example.py
   ```

## Verify data

Open **Observability** > **Agent Observability** > **Conversations** in Grafana Cloud. Select a time range that includes your request, filter by your agent name, and open the conversation to inspect its prompt and response. For the Python sample, filter by `otel-example`.

If no conversation appears, check your exporter for authentication or connection errors, then check the opt-in attribute, operation, provider, and model. If a conversation appears without content, check that content capture writes the message span attributes before export.

## Next steps

- [Browse conversations](/docs/grafana-cloud/observe-and-act/agent-observability/guides/conversations)
- [Review privacy and security](/docs/grafana-cloud/observe-and-act/agent-observability/privacy-and-security)