Documentation Grafana documentation Troubleshooting Data sources diagnostic bundle

Generate a data source diagnostic bundle

When a panel returns an error, no data, or data that looks wrong, Grafana Labs Technical Support needs to know what your data source actually returned. In Grafana Cloud, you can grant support access to your stack so an engineer can investigate live. A self-managed instance isn’t reachable by support, so you collect the evidence yourself in a diagnostic bundle and send it in.

When you generate a diagnostic bundle, Grafana re-runs the panel’s queries with traffic capture active and packages the result as a single .tar.gz file containing the requests and responses exchanged with your data source, the data the data source plugin returned to Grafana, the panel and dashboard configuration, and metadata about the capture.

Before you begin

To generate a diagnostic bundle, you need the following:

• A self-managed Grafana instance. Diagnostic bundles aren’t available in Grafana Cloud.
• The Grafana server administrator role. The menu items described on this page don’t appear for other users.
• The grafana.onDemandDiagnostics feature toggle enabled.

Enable the feature toggle

The grafana.onDemandDiagnostics feature toggle is disabled by default. While it’s disabled, Grafana hides the diagnostics UI elements and the underlying endpoints return 404.

To enable it, add the following to your Grafana configuration file and restart Grafana:

ini Copy

[feature_toggles]
enable = grafana.onDemandDiagnostics

To set a feature toggle in other ways, such as environment variables, refer to the feature_toggles section of Configure Grafana and Manage feature toggles.

Generate a bundle for a single panel

1. Open the dashboard that contains the panel you want to diagnose.

2. Hover over any part of the panel to display the panel menu in the top right corner.

3. Click the menu and select More > Download diagnostics.

   Grafana opens a drawer describing what the bundle contains, including a May contain sensitive data warning.

4. Click Download diagnostics.

   Grafana re-runs the panel’s queries with capture active and downloads the bundle. The button reads Generating… while this happens, which can take a moment.

5. Review the bundle, then attach it to your support ticket.

Generate a bundle for an entire dashboard

Use a dashboard bundle when you don’t know which panel is at fault, or when the problem involves more than one panel.

1. Open the dashboard you want to diagnose.

2. In the dashboard toolbar, click Export > Download diagnostics.

3. Click Download diagnostics.

   Grafana captures panels one at a time and reports its progress, for example Capturing panel 3 of 12. Large dashboards can take a while to capture.

4. Review the bundle, then attach it to your support ticket.

Grafana captures each panel independently, so one failing panel doesn’t stop the rest of the bundle from being produced. Skipped and failed panels are recorded in manifest.json.

What the bundle contains

Extract the bundle to review it:

Bash Copy

tar -xzf <bundle>.tar.gz

A single-panel bundle contains the following files:

A dashboard bundle contains a panels/<id>-<title>/ directory holding those files for each panel, plus a manifest.json recording when the bundle was generated, how many panels ran, and the data sources, sizes, and any errors for each panel.

Data source support for capturing upstream traffic

traffic.har is usually the most valuable file in the bundle, but it isn’t available for every data source. When upstream traffic can’t be captured, Grafana still generates the rest of the bundle, and querydata.json still shows what the data source plugin returned.

Data sources that capture upstream traffic

The following data sources are built into Grafana and capture their upstream traffic with no extra configuration:

• Prometheus
• Graphite
• InfluxDB
• Azure Monitor

Data sources that don’t capture upstream traffic

• SQL and other database data sources, such as MySQL, PostgreSQL, Microsoft SQL Server, and MongoDB. These communicate over a database wire protocol instead of HTTP, so there’s no HTTP exchange to record.
• Amazon CloudWatch. Queries go through the AWS SDK, which doesn’t use the instrumented HTTP client that capture depends on.

Limitations

As an experimental feature, on-demand data source diagnostics currently has the following limitations:

• Grafana re-runs your queries. A bundle reflects a fresh execution at the moment you generate it, not the original failure, and it bypasses the query cache. An intermittent failure might not reproduce, so a healthy-looking bundle doesn’t prove the problem is resolved.

• Queries run as you. Because you generate the bundle as a server administrator, failures that depend on the affected user’s identity might not reproduce. Examples include per-user OAuth forwarding and row-level database permissions. If a specific user is affected, note that in your ticket.

• A bundle captures panel and dashboard queries only. Failures in the variable picker, in a data source’s Save & test button, in annotation queries, or during alert rule evaluation don’t appear.

• Server-side expressions and panel transformations are recorded in panel.json as configuration, not as the data that flows between them. So if querydata.json looks correct but the panel doesn’t, check that configuration.

• Large responses might be trimmed, and less commonly, generation fails outright. The following limits apply:

  • querydata.json is capped at 1 GiB for a single panel and 1.5 GiB across a whole-dashboard bundle. A response over that limit is replaced by a summary that records what was left out, and the rest of the bundle still generates. For a dashboard bundle, once the shared 1.5 GiB budget is used up, the remaining panels’ query data is skipped, and the reason is recorded in manifest.json.
  • traffic.har is limited to 64 MiB per request or response body and 256 MiB of retained traffic per panel. A body over the per-body limit is kept as a truncated prefix, and its entry reports bodySize: -1 with the actual size in content.size. When a panel’s traffic exceeds the per-panel limit, further entries keep their headers, sizes, and timings but omit the body text.

• Generating a bundle can fail outright, instead of trimming, when the entire request to Grafana exceeds 100 MiB. That request includes the queries, the panel and dashboard JSON, and, for a single panel, the frames your browser is holding. This is uncommon, because it requires an unusually large payload from the browser itself—which a large data source response alone doesn’t produce.

  If you hit one of these limits, or a bundle takes an unreasonably long time to generate, reproduce the problem on a minimal setup instead of the original dashboard:

  • Use a temporary panel or dashboard holding only the misbehaving panel.
  • Narrow the time range to the shortest window that still reproduces the problem. Fewer returned data points means a smaller bundle and a faster capture.

  If you generate the bundle from a reduced dashboard, note that in your ticket. A smaller, reproducible setup is also easier for Grafana Labs Technical Support to work with.

Related resources

• Send a panel to Grafana Labs support
• Generate a HAR capture to send to Grafana Labs support
• Troubleshoot data sources