Documentation Grafana Cloud Observe and act Agent Observability Get started Instrument with OpenTelemetry

Instrument agents with OpenTelemetry

Send generative AI spans with OpenTelemetry to inspect prompts and responses in Agent Observability. You don’t need the Agent Observability SDK.

Generative AI instrumentation is code or a library that records your application’s AI calls as OpenTelemetry spans. Each span describes a call with attributes, for example, the model provider, model name, and token usage. It can also capture prompts and responses. For the setup in this guide, use instrumentation that follows the OpenTelemetry generative AI semantic conventions, with attributes that use the gen_ai.* prefix.

You can use your framework’s built-in instrumentation, an instrumentation library, or spans you create yourself. General OpenTelemetry instrumentation doesn’t necessarily capture AI calls or their generative AI attributes. This guide shows you how to send compatible spans to Agent Observability and includes a sample you can create yourself.

Before you begin

• A Grafana Cloud stack with Agent Observability enabled. Refer to Set up Agent Observability.
• A Cloud Access Policy Token with traces:write permission.
• An application with OpenTelemetry generative AI instrumentation, or Python 3.10 or later to try the sample on this page.

Export only content your data policy permits. Remove secrets and sensitive data before export.

Connect to Grafana Cloud

Configure OpenTelemetry Protocol (OTLP) export to send traces to your Grafana Cloud stack. Choose a setup path:

• Recommended setup through Grafana Cloud integrations: Use a connection tile for guided setup, including connection settings and collector configuration.
• Manual setup: Get your OTLP endpoint, generate an authentication token, and configure the export environment variables yourself. The guide also explains endpoint paths and Python authentication header encoding.

Use the OTLP connection details for your stack. You don’t need the Agent Observability SDK endpoint for this setup. For the Python sample below, use the HTTP/protobuf exporter settings.

If you already export through Grafana Alloy or an OpenTelemetry Collector, keep that route and preserve the span and resource attributes described below.

Configure your instrumentation

Use instrumentation that follows the OpenTelemetry generative AI semantic conventions. Don’t opt in calls that the Agent Observability SDK already exports as generations, because you can create duplicate records.

1. Check that your spans include these attributes:

   Expand table

   Attribute	Value
   gen_ai.operation.name	chat, generate_content, text_completion, or invoke_agent
   gen_ai.provider.name	Your model provider, for example, openai
   gen_ai.request.model	Your model name, for example, gpt-4o

   invoke_agent spans also need a provider and model to appear as generations. Tool calls and other operations don’t create standalone generations.

2. Opt in your application’s generative AI spans with a resource attribute:

   Bash Copy

   export OTEL_RESOURCE_ATTRIBUTES="agento11y.record=true"

   If OTEL_RESOURCE_ATTRIBUTES already has a value, append ,agento11y.record=true instead of replacing it. You can also set agento11y.record=true on individual spans. Set agento11y.record=false on a span to exclude it.

3. Enable content capture in your instrumentation if you want to inspect prompts and responses. It must write gen_ai.input.messages and gen_ai.output.messages as span attributes in the generative AI message format shown in the sample below. Content sent only through OTLP logs doesn’t populate conversations.

4. Set gen_ai.conversation.id on spans and reuse it for calls in the same conversation. Without it, each trace forms a separate conversation. Set gen_ai.agent.name to identify your agent; otherwise, Agent Observability uses the resource’s service.name.

Restart your application with the export settings and run an agent request. Make sure your application or instrumentation library initializes an OpenTelemetry tracer provider and exporter; setting export environment variables alone doesn’t instrument your application. Spans dropped by application or collector sampling don’t appear in Agent Observability.

Try a sample without a model provider

If you don’t have generative AI instrumentation yet, use this Python example to check your connection. It sends a synthetic prompt and response without calling a model provider. If you already send spans from your application, refer to Verify data.

1. Install the OpenTelemetry packages:

   Bash Copy

   python -m pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

2. Save the following code as otel_example.py:

   Python Copy

   import json
   from uuid import uuid4
   
   from opentelemetry import trace
   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   
   provider = TracerProvider(resource=Resource.create({"service.name": "otel-example"}))
   provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
   trace.set_tracer_provider(provider)
   tracer = trace.get_tracer("otel-example")
   
   with tracer.start_as_current_span("chat gpt-4o", kind=trace.SpanKind.CLIENT) as span:
       span.set_attributes({
           "agento11y.record": True,
           "gen_ai.operation.name": "chat",
           "gen_ai.provider.name": "openai",
           "gen_ai.request.model": "gpt-4o",
           "gen_ai.conversation.id": str(uuid4()),
           "gen_ai.input.messages": json.dumps([
               {"role": "user", "parts": [{"type": "text", "content": "Say hello."}]}
           ]),
           "gen_ai.output.messages": json.dumps([
               {"role": "assistant", "parts": [{"type": "text", "content": "Hello!"}]}
           ]),
       })
   
   provider.shutdown()

   agento11y.record=true opts the span into Agent Observability. Reuse gen_ai.conversation.id across calls that belong to the same conversation.

3. Run the example:

   Bash Copy

   python otel_example.py

Verify data

Open Observability > Agent Observability > Conversations in Grafana Cloud. Select a time range that includes your request, filter by your agent name, and open the conversation to inspect its prompt and response. For the Python sample, filter by otel-example.

If no conversation appears, check your exporter for authentication or connection errors, then check the opt-in attribute, operation, provider, and model. If a conversation appears without content, check that content capture writes the message span attributes before export.

Next steps

• Browse conversations
• Review privacy and security