Documentation Plugins Wavefront Troubleshooting

Troubleshoot Wavefront data source issues

This document provides solutions to common issues you might encounter when configuring or using the Wavefront data source. For configuration instructions, refer to Configure the Wavefront data source.

Authentication errors

These errors occur when credentials are missing, invalid, or don’t have the required permissions.

“Enter an API URL.” or “Enter a token.”

Symptoms:

• Save & test fails with a message like The configuration setup is incomplete.
• The detailed message is Enter an API URL. or Enter a token.

Solutions:

1. Open the data source configuration page.
2. Fill in the API URL field with your Wavefront tenant URL, for example https://myenvironment.wavefront.com.
3. Fill in the Token field with a valid Wavefront API token.
4. Click Save & test again.

“The credentials are incorrect. Check that the token is correct.”

Symptoms:

• Save & test fails with the status Unauthorized (401).
• Queries return 401 errors in the browser network tab or panel error messages.

Possible causes and solutions:

Connection errors

These errors occur when Grafana can’t reach the Wavefront API.

“Could not connect to Wavefront. This usually happens when the URL is incorrect.”

Symptoms:

• Save & test fails with the status Bad Request (400).
• The detailed message is Could not connect to Wavefront. This usually happens when the URL is incorrect.

Solutions:

1. Verify the API URL is the full tenant URL, including the https:// scheme and no trailing path. Common URL mistakes include:
   • Adding a trailing slash, for example https://myenvironment.wavefront.com/. Remove the slash.
   • Including an API path, for example https://myenvironment.wavefront.com/api/v2/. The data source appends API paths automatically.
   • Using http:// instead of https://.
2. Confirm the tenant is reachable from your Grafana instance. Try the URL in a browser or with curl.
3. If you’re using Grafana Cloud with a private Wavefront tenant, configure Private data source connect and enable Enable Secure Socks Proxy on the data source.
4. Check firewall rules allow outbound HTTPS (port 443) from the Grafana server to your Wavefront tenant.

Data source test times out

Symptoms:

• Save & test hangs or reports a timeout.
• Queries work intermittently.

Solutions:

1. Verify network connectivity from the Grafana server to the Wavefront tenant.
2. Increase the Request timeout in seconds setting in the data source configuration.
3. For private networks, ensure your VPN, peering, or Private data source connect path to Wavefront is healthy.

Query errors

These errors occur when running queries against the data source.

“No data” or empty results

Symptoms:

• The query runs without an error, but the panel shows No data.

Possible causes and solutions:

Query timeout

Symptoms:

• The query runs for a long time then fails with a timeout error.
• Error mentions context deadline exceeded or request timeout.

Solutions:

1. Narrow the time range to reduce the number of points returned.
2. Add filters to reduce the result set.
3. Add an aggregation function, for example avg or sum, to collapse high-cardinality series.
4. Increase Request timeout in seconds on the data source for heavier queries.

“Error: It seems the query have URL escaped characters.”

Symptoms:

• The Raw Query editor shows a red error about URL-escaped characters.
• The query includes sequences like %20, %3D, or %22.

Solutions:

1. Replace escaped sequences with literal characters. For example, replace %20 with a space and %3D with =.
2. Re-run the query. If the error persists, clear the Query Type selector and re-select the appropriate type.

PromQL query returns unexpected results

Symptoms:

• PromQL queries work on a Prometheus tenant but not on Wavefront, or return different shapes.

Solutions:

1. Confirm Query Type is set to PromQL or Hybrid in the Raw Query editor.
2. Review the Wavefront PromQL compatibility guidance for functions and selectors that Wavefront translates differently.
3. Try rewriting the query in WQL with Query Type set to WQL to compare output.

Template variable errors

These errors affect template variables backed by the Wavefront data source.

Variables return no values

Solutions:

1. Verify the data source connection with Save & test.
2. Confirm the Query Type matches the data you want. Use Metrics for metric names, Hosts / Sources for source names, and so on.
3. For Tag Values, make sure the Tag field contains a valid tag key.
4. Verify the token has permission to list the requested resources.

“Tag Values” returns all values or is empty

Symptoms:

• Tag Values returns every value in the tenant, or returns nothing.

Solutions:

1. Set a specific Query expression, for example ~sample.cpu.*, to scope the search.
2. Double-check the Tag field for typos. The field is case-sensitive.

Ad hoc filters show no keys or values

Symptoms:

• The ad hoc filter drop-down is empty.
• The filter shows keys, but no values for the selected key.

Solutions:

1. Confirm a helper variable named exactly metriclink exists on the dashboard. The data source uses this helper to decide which metric’s keys and values to list.
2. Make sure metriclink has a non-empty value. You can use a Custom variable with a list of metrics or a Query variable of type Metrics.
3. Refer to Ad hoc filters on the query editor page for full setup steps.

Legacy variable queries don’t return results

Symptoms:

• A dashboard imported from an older plugin version has variables with string queries like metrics: ts(...) or tagValues(env): ts(...).
• The variable returns no results or an invalid query error.

Solutions:

1. Open the variable for editing. The plugin migrates legacy queries automatically when the variable loads, so the editor shows the modern Query Type and Query fields.
2. Confirm the migrated Query Type matches what you expect. For example, sources: migrates to Hosts / Sources, and tagValues(env): migrates to Tag Values with Tag set to env.
3. If the migration didn’t pick the correct type, set the Query Type and Query manually.
4. Refer to Legacy query formats on the template variables page.

Annotation issues

These issues affect Wavefront events used as annotations on dashboards.

No annotations appear on the panel

Symptoms:

• An annotation query is configured, but no markers or regions appear on the panel.

Solutions:

1. Confirm Wavefront has events in the dashboard time range. Open the Wavefront UI’s Events browser to verify.
2. Check the annotation toggle on the dashboard’s annotation switch bar to confirm the annotation isn’t hidden.
3. Increase the Limit on the annotation query if the tenant has many events and yours might be after the cutoff.
4. Confirm the token has permission to read events.
5. Refer to Add an annotation for the full setup procedure.

Annotation tooltip shows the wrong text

Symptoms:

• Annotation markers appear, but the tooltip shows an unhelpful field, such as the event ID instead of the event name.

Solutions:

1. Open the annotation in dashboard settings.
2. In the Fields section, map response fields to the annotation Title, Text, and Tags.
3. For recommended mappings, refer to Map response fields to annotation properties.

Performance issues

These issues relate to slow queries, throttling, or API limits.

API throttling or rate-limit errors

Symptoms:

• Errors mentioning rate limits or HTTP 429 responses.
• Dashboard panels intermittently fail during peak refresh times.

Solutions:

1. Reduce the dashboard refresh frequency.
2. Increase the aggregation interval to reduce the number of points returned.
3. Enable query caching (available in Grafana Enterprise and Grafana Cloud).
4. Request a rate-limit increase from your Wavefront administrator or VMware support.

Alert rules time out or return No Data

Symptoms:

• Grafana-managed alert rules based on Wavefront queries fail with timeouts.
• Alert rules transition to No Data.

Solutions:

1. Simplify the alert query or add filters to limit the number of series.
2. Use moving-window functions, such as mavg or mmedian, to smooth high-cardinality results.
3. Add align() in WQL or a [duration] selector in PromQL so the query produces a regular time series for the Reduce expression.
4. Increase Request timeout in seconds on the data source so alert evaluations have more time to complete.
5. Avoid very long time ranges in alert queries; use the shortest range that still captures the signal.
6. For more guidance, refer to Alerting with the Wavefront data source.

Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to debug in the Grafana configuration file:

   ini Copy

   [log]
   level = debug

2. Restart Grafana.

3. Reproduce the issue and review logs in /var/log/grafana/grafana.log (or your configured log location). Look for log entries with logger=plugin.grafana-wavefront-datasource, which include backend errors from the data source as well as the URLs of HTTP requests sent to your Wavefront tenant.

4. To inspect the queries the plugin sends, also check the panel’s query inspector under Inspect > Query for the executedQueryString field.

5. Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried the solutions in this guide and still encounter issues:

1. Check the Grafana community forums for similar issues.
2. Consult the VMware Aria Operations for Applications documentation for tenant-specific guidance.
3. Raise a support ticket if you’re a Grafana Cloud Pro, Cloud Advanced, or Grafana Enterprise customer.
4. When reporting an issue, include:
   • Grafana version and plugin version (from Plugins and data > Plugins).
   • The exact error message (redact sensitive information).
   • Steps to reproduce.
   • Relevant configuration, with credentials redacted.