Documentation Plugins Infinity data source Troubleshooting

Troubleshoot Infinity data source issues

This guide helps you diagnose and resolve common issues with the Grafana Infinity data source. Issues are organized by category: connection errors, authentication errors, query and parsing errors, and performance issues.

Connection errors

These errors occur when the data source cannot connect to the target URL.

URL not allowed

Error message: “requested URL not allowed. To allow this URL, update the data source config in the Security tab, Allowed hosts section”

Symptoms: Queries fail with this error even when you believe the host is already in the Allowed hosts list. The error can also appear suddenly for a data source that previously worked.

Cause: Infinity validates the final request URL against the Allowed hosts list before sending it. The data source assembles and checks the URL in this order:

1. If a Base URL is set in the data source configuration, it’s prefixed to the query URL.
2. The combined URL is checked against the Allowed hosts list.
3. The query runs only when the URL matches an allowed entry.

An entry matches only when it’s a prefix of the final URL, including the scheme, and the host names match. Because the match includes the scheme and the full path prefix, small differences cause the URL to be rejected.

Possible causes and solutions:

To add or correct an allowed host:

1. Open the data source configuration in Grafana.
2. Navigate to the Security section.
3. Add the full host, including the scheme, to the Allowed hosts field, for example https://api.example.com.
4. Click Save & test.

Connection timeout

Error message: “context deadline exceeded” or “timeout awaiting response headers”

Cause: The target server is taking too long to respond.

Solution:

1. Verify the target URL is accessible from the Grafana server (not just your browser).
2. Check if the API has rate limiting that might be blocking requests.
3. Reduce the amount of data requested by adding filters or limiting time ranges.
4. Contact your API provider to check for service issues.

SSL/TLS certificate errors

Error message: “x509: certificate signed by unknown authority” or “certificate verify failed”

Cause: The target server’s SSL certificate isn’t trusted.

Solution:

1. If using a self-signed certificate, enable Skip TLS Verify in the data source configuration under TLS / SSL Auth Details.
2. For enterprise environments, add the CA certificate to TLS/SSL Root Certificate field.
3. Verify the certificate chain is complete on the target server.

Empty response received

Error message: “empty response body received for the URL”

Cause: The server returned a successful status but with no content.

Solution:

1. Verify the URL returns data when accessed directly (using curl or a browser).
2. Check if authentication is required but not configured.
3. Ensure the API endpoint is correct and accepts the HTTP method you’re using.
4. Check if the API requires specific headers that aren’t being sent.

HTTP method not allowed

Error message: “only GET and POST HTTP methods are allowed for this data source. To make use other methods, enable the ‘Allow dangerous HTTP methods’ in the data source configuration”

Cause: You’re trying to use PUT, PATCH, or DELETE methods which are disabled by default.

Solution:

1. Open the data source configuration.
2. In the Security section, enable Allow dangerous HTTP methods.
3. Click Save & test.

Grafana Cloud and network connectivity

These issues are specific to running the Infinity data source on Grafana Cloud or through a restricted network.

API blocks requests from Grafana Cloud

Symptoms: Queries work from a local Grafana instance but fail from Grafana Cloud, often with timeouts or connection errors. Your API firewall shows blocked requests from unknown IP addresses.

Cause: On Grafana Cloud, the Infinity data source sends requests from Grafana Cloud source IP addresses, not from your browser or your own network. If your API restricts inbound traffic, those addresses must be on your allowed list.

Solution:

1. Add the Hosted Grafana source IP addresses to the allowed list on your API. The current addresses are published in several formats:
   • JSON: https://grafana.com/api/hosted-grafana/source-ips
   • Text: https://grafana.com/api/hosted-grafana/source-ips.txt
   • DNS: src-ips.hosted-grafana.grafana.net
2. For the full reference, refer to List of source IP addresses to add to your allowlist.
3. If your API is on a private network that you can’t expose to the public internet, use private data source connect instead. Refer to the next entry.

Private data source connect works locally but fails from Grafana Cloud

Symptoms: A data source that uses private data source connect (PDC) works from a local agent but fails from Grafana Cloud. Firewall logs show no traffic, or the PDC agent reports DNS resolution failures.

Cause: The PDC agent can’t reach Grafana Cloud, or it can’t resolve the target host’s DNS name from within your private network.

Solution:

1. Confirm the PDC agent has outbound egress to the cluster-specific PDC endpoints. Refer to Configure private data source connect (PDC).
2. Verify the agent can resolve and reach the target host from its own network, for example with nslookup and curl run from the agent host.
3. In the data source, enter the internal hostname and port exactly as you would from inside your private network.
4. Confirm the data source is assigned to the correct PDC connection.
5. For how PDC works, refer to Private data source connect.

Private S3 bucket returns a 403 error

Symptoms: Requests to a private Amazon S3 bucket return a 403 error, even though the bucket name and credentials are correct.

Cause: Requests to S3 must be signed with AWS Signature Version 4 (SigV4). Without AWS authentication, Infinity doesn’t sign the request, so S3 rejects it. The default AWS service is monitoring (Amazon CloudWatch), not s3.

Solution:

1. Expand the Authentication section and select AWS.
2. Set Service to s3 and Region to the bucket’s region.
3. Enter your Access Key and Secret Key, and confirm the IAM identity has read access to the bucket.
4. Add the S3 endpoint to Allowed hosts, for example https://s3.<region>.amazonaws.com.
5. For a complete AWS example, refer to AWS API.

Authentication errors

These errors relate to authentication configuration issues.

Invalid or empty credentials

Error messages:

• “invalid or empty password detected”
• “invalid API key specified”
• “invalid or empty bearer token detected”

Cause: Required authentication credentials are missing or incorrectly configured.

Solution:

1. Open the data source configuration.
2. Verify all required credential fields are filled in.
3. For API keys, ensure the key is entered in the correct field (header value or query parameter).
4. Click Save & test to verify the credentials work.

Bearer token works in another client but fails in Infinity

Symptoms: A bearer token that succeeds in another client, such as Postman or curl, returns a 400 or 401 error in Infinity.

Cause: The token isn’t sent in the format the API expects, or the request is missing headers that the other client adds automatically. Infinity sends the token as Authorization: Bearer <token> and doesn’t replicate the default headers that other tools set.

Solutions:

1. If the API expects a prefix other than Bearer, such as Token or ApiKey, use API key authentication with the Key set to Authorization and the Value set to the full header value, for example Token <token>.
2. Compare the headers your other client sends with the headers Infinity sends, then add any missing headers in the query or data source configuration. Refer to API returns a 406 error for the default Accept header behavior.
3. Confirm the host is in the Allowed hosts list. Refer to URL not allowed.
4. Verify the token hasn’t expired.

API returns a 406 error

Symptoms: The API returns a 406 status, often only from Infinity and not from other clients. This is common with XML and SOAP endpoints.

Cause: Infinity sets a default Accept request header based on the query type:

Some servers reject these values and expect a specific media type, such as application/xml.

Solution:

1. In the query editor, expand the URL options and add a custom request header.
2. Set the Accept header to the media type your API expects, for example application/xml.
3. Run the query again. A custom Accept header overrides the default.

OAuth2 token errors

Error message: “error getting OAuth2 token” or authentication failures after initial success

Cause: OAuth2 configuration is incorrect or tokens have expired.

Solution:

1. Verify the Token URL is correct and accessible.
2. Check that Client ID and Client Secret are correct.
3. Ensure the required scopes are specified.
4. For Azure, verify the resource endpoint parameter is set correctly.
5. Check if the OAuth2 provider requires additional endpoint parameters.

For custom token configurations, refer to OAuth2 token customization.

OAuth2 client credentials fail with Microsoft Entra ID

Symptoms: OAuth2 client credentials authentication fails against Microsoft Entra ID (formerly Azure AD), often with a token error or a “URL not allowed” error.

Cause: The token URL, scopes, or allowed hosts are incorrect. With client credentials, Infinity makes two separate calls: one to the token endpoint and one to the data endpoint. Both hosts must be in the Allowed hosts list.

Solution:

1. Set the Token URL to your tenant endpoint, for example https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token.
2. Set Scopes to the resource scope. For client credentials, this is typically https://<resource>/.default.
3. Add both hosts to Allowed hosts: the token endpoint host, for example https://login.microsoftonline.com, and the data endpoint host, for example https://management.azure.com.
4. If your token endpoint requires custom headers, add them as token request headers. Multiple token request headers are supported in version 3.8.0 and later. Refer to OAuth2 token customization.

For a complete Azure example, refer to Azure API.

Forward OAuth fails for template variable queries

Symptoms: Panels authenticate correctly with Forward OAuth identity, but template variable queries that use the same data source fail with authentication errors.

Cause: Some plugin versions don’t apply the forwarded OAuth identity to variable queries.

Solution:

1. Upgrade to the latest version of the Infinity data source plugin, which addresses known issues with forwarded OAuth identity in variable queries.
2. If the problem persists, report it on the GitHub issues page with your Grafana and plugin versions, because a Grafana feature toggle might be required for your version.

AWS authentication failures

Error messages:

• “invalid/empty AWS access key”
• “invalid/empty AWS secret key”

Cause: AWS credentials are missing or incorrect.

Solution:

1. Verify the Access Key ID and Secret Access Key are correct.
2. Ensure the IAM user has the required permissions for the AWS service you’re querying.
3. Check that the AWS region is correctly specified.
4. Verify the service name matches the AWS service (for example, monitoring for CloudWatch).

401 or 403 errors

Error message: “unsuccessful HTTP response code” with status 401 or 403

Cause: Authentication failed or you don’t have permission to access the resource.

Solution:

1. Verify your credentials are correct.
2. Check if the API key or token has expired.
3. Ensure the authenticated user has permission to access the requested resource.
4. For OAuth2, verify the scopes include the required permissions.

Data source provisioning and naming

These issues relate to how the data source is provisioned and named.

Can’t save changes to a provisioned data source

Symptoms: You can open and test a provisioned Infinity data source, but Save & test doesn’t persist changes, and the configuration reverts when you reload the page.

Cause: Data sources created through Grafana provisioning are managed by configuration files, so they’re read-only in the UI. This is expected Grafana behavior, not an Infinity bug.

Solution:

1. Edit the data source’s provisioning file instead of the UI.
2. Reload provisioning or restart Grafana to apply the changes.
3. To manage the data source from the UI instead, remove it from provisioning and recreate it manually.
4. For details, refer to Provisioning Grafana.

Data source disappears shortly after creation

Symptoms: A data source is deleted automatically within minutes of creation, often after you name it something like grafanacloud-infinity.

Cause: On Grafana Cloud, names that begin with the grafanacloud- prefix are reserved for Grafana-managed data sources. An internal service removes user-created data sources that use a reserved name.

Solution:

1. Rename the data source to a name that doesn’t start with grafanacloud-, for example infinity or my-infinity.
2. Recreate the data source with the new name.

Query and parsing errors

These errors occur when processing the query or parsing the response.

Invalid root selector

Error message: “error converting json data to frame” or “invalid root selector”

Cause: The root selector path doesn’t match the API response structure.

Solution:

1. Use Format as > As Is to see the raw response structure.

2. Verify the root selector path matches your data. For example, if your JSON is:

   JSON Copy

   { "data": { "items": [...] } }

   The root selector should be data.items.

3. For arrays at the root level, leave the root selector empty or use $ for JSONata.

4. Use JSONata expressions for complex data extraction.

JSON parsing errors

Error message: “unable to parse response body as JSON”

Cause: The response isn’t valid JSON or isn’t in the expected format.

Solution:

1. Verify the API returns JSON (check the Content-Type header).
2. If the API returns XML, CSV, or another format, select the correct Type in the query editor.
3. Check for API errors that might return HTML error pages instead of JSON.
4. Use Format as > As Is to inspect the raw response.

CSV query returns no data

Symptoms: A CSV query silently returns no data, sometimes after a plugin upgrade, even though the file looks correct.

Cause: Trailing newlines or extra whitespace at the end of the CSV content can prevent the parser from reading rows. Recent plugin versions parse CSV more strictly.

Solution:

1. Remove trailing blank lines and whitespace from the end of the CSV content or file.
2. If you can’t change the source, switch the parser to Backend and use a JQ or JSONata expression to select the rows.
3. Use Format as > As Is to inspect the raw response and confirm where the extra content appears.

Raw output contains null before the data

Symptoms: The raw output shows null immediately followed by the response body, such as null{...}, and parsing fails.

Cause: The parser configuration doesn’t match the response, so an empty result is concatenated with the response body.

Solution:

1. Confirm the query Type matches the response format (JSON, CSV, XML, and so on).
2. Set the correct parser for your transformation and provide a root selector that points to the data.
3. Use Format as > As Is to inspect the raw response before applying a parser.

Time column not recognized or panel shows no data

Symptoms: A time series panel shows “No data” even though the query returns rows, or the time column isn’t plotted.

Cause: Time data isn’t being parsed correctly for time series visualization. When you don’t define columns in the Parsing options & Result fields section, Grafana can’t always auto-detect which column is the time field, so a time series panel has no time axis to plot.

Solution:

1. In the Parsing options & Result fields section, define the columns explicitly instead of relying on auto-detection.
2. Set the time column to the correct type: Time, Time (UNIX ms), or Time (UNIX s).
3. For custom time formats with the backend parser, specify the format in the column configuration.
4. Verify the time values are in a supported format (ISO 8601, Unix timestamp, or a recognized date string).
5. Refer to Time formats for supported formats.

UQL syntax errors

Cause: The UQL query has syntax errors or uses invalid commands.

Solution:

1. Check that pipe characters (|) separate each command.
2. Verify column names are quoted: "column_name".
3. Ensure string values use single quotes: 'value'.
4. Start with a parse command: parse-json, parse-csv, or parse-xml.
5. Refer to UQL parser for syntax reference.

Summarize returns an error for an empty result set

Symptoms: A JSONata count() or other summarize expression returns an error instead of 0 when the query matches no rows.

Cause: An aggregation over an empty result set has no input to operate on, so the expression fails instead of returning a default value.

Solution:

1. Add a filter or root selector that always returns an array, even when it’s empty.
2. Provide a fallback in the expression so that an empty input returns 0.
3. Confirm the root selector matches the response so the result set isn’t unexpectedly empty.

Parser options are limited in the query editor

Symptoms: The query editor doesn’t offer the parser you need. For example, a backend parser isn’t selectable for a given query.

Cause: The available parsers depend on the query type and context. Backend transformations require a backend parser.

Solution:

1. Select a query Type that supports the parser you need.
2. For transformations that must run on the backend, such as those used in alerting, recorded queries, or shared dashboards, select Backend parsing with JSONata or JQ.
3. Refer to Query editor for parser options.

Macro expansion errors

Error messages:

• “insufficient arguments to combineValues macro”
• “invalid customInterval macro”

Cause: Macro syntax is incorrect or missing required arguments.

Solution:

1. For $__combineValues(), ensure you have at least 4 arguments: prefix, suffix, separator, and at least one value.
2. For $__customInterval(), use valid duration syntax (for example, 1m, 1h, 1d) and ensure the argument count is odd.
3. Refer to Macros for correct syntax.

Reference data not found

Error message: “error getting reference data. Either empty or not defined”

Cause: The query references data that doesn’t exist in the data source configuration.

Solution:

1. Open the data source configuration.
2. Navigate to Reference data section.
3. Add or verify the reference data with the name your query expects.
4. Click Save & test.

Azure Blob Storage errors

These errors are specific to Azure Blob Storage queries.

Invalid Azure Blob configuration

Error messages:

• “invalid/empty azure blob account name”
• “invalid/empty azure blob key”
• “invalid azure blob service client. check storage account name and key”

Cause: Azure Blob Storage credentials are missing or incorrect.

Solution:

1. Verify the storage account name and key in the data source configuration.
2. Ensure the storage account exists and is accessible.
3. Check that the account key hasn’t been rotated.

Azure Blob 403 error

Error message: “http 403. check azure blob storage key”

Cause: Access denied to the Azure Blob Storage.

Solution:

1. Verify the storage account key is correct.
2. Check if the storage account has IP restrictions that block the Grafana server.
3. Ensure the container and blob permissions allow read access.

Performance issues

These issues relate to slow queries or resource usage.

Slow query execution

Cause: Queries return too much data or the API is slow.

Solution:

1. Reduce the dashboard time range to limit data volume.
2. Add filters in your query to reduce the data returned.
3. Use aggregations in UQL (summarize) instead of returning raw rows.
4. Enable caching if your API data doesn’t change frequently.
5. Consider using the backend parser for better performance with large datasets.

Large response handling

Cause: The API returns very large responses that are slow to process.

Solution:

1. Request only the fields you need by configuring specific columns.
2. Use pagination if the API supports it.
3. Apply server-side filters if the API supports query parameters.
4. Use limit in UQL to reduce rows: | limit 1000.

Alerting issues

These issues relate to using Infinity queries with Grafana Alerting.

Alert rule shows “no data” or doesn’t fire

Cause: The query uses a frontend parser that doesn’t support alerting.

Solution:

1. Alerting only works with backend parsers: JSONata or JQ.
2. Change your parser from Default, UQL, or GROQ to JSONata or JQ.
3. Verify the query returns numeric data that can be evaluated against alert thresholds.
4. Test the query in Explore to ensure it returns data.

Query works in panels but fails in alerts

Symptoms: A query returns data in Explore and dashboard panels, but the alert rule reports “no data” or doesn’t evaluate.

Cause: Alert evaluation needs a backend-parsed result with a recognized number field and, for time series, a correctly typed time field. A query that renders in a panel can still fail alert evaluation if the parser is frontend-only or the time column isn’t typed correctly. An existing alert rule can also retain an older query model after you change the query.

Solution:

1. Use a backend parser. Refer to Alert rule shows “no data” or doesn’t fire.
2. Set the time column to an explicit time type and confirm the value format. Refer to Time column not recognized or panel shows no data.
3. Make sure the query returns at least one numeric field for the alert condition to evaluate.
4. Reduce the result to the rows and columns the alert needs, so the condition has a clear series to evaluate.
5. If the rule still fails after the query works in Explore, recreate the alert rule so it picks up the current query configuration.

User macros not available in alerts

Cause: Alert queries run in a system context without user session information.

Solution:

User-related macros (${__user.login}, ${__user.email}, ${__user.name}) are not available in:

• Alert rules
• Recorded queries
• Public dashboards

If your query depends on user context, you’ll need to:

1. Remove user macros from alert queries.
2. Use static values or other macros that don’t require user context.
3. Consider using data source-level filters instead of user-based filters.

Template variable errors

These issues relate to using Infinity queries as template variable sources and to using variables inside queries.

Template variables return no data

Cause: Variable queries aren’t configured correctly.

Solution:

1. Test the query in the Explore view first to verify it returns data.
2. Ensure the query returns columns that can be used as variable options.
3. For custom text/value pairs, return columns named __text and __value.
4. Check that the data source connection is working.

Variable name collides with a GraphQL variable

Symptoms: A GraphQL query fails or behaves unexpectedly when a GraphQL variable name matches a dashboard variable name.

Cause: Grafana interpolates any $name token that matches a dashboard variable before the query runs. If your GraphQL query declares a variable with the same name, such as $user, Grafana replaces it instead of passing it to the GraphQL server.

Solution:

1. Rename the GraphQL variable to a name that doesn’t match any dashboard variable, for example $gqlUser instead of $user.
2. Alternatively, rename the dashboard variable to avoid the collision.

Multi-value variable passes only partial values

Symptoms: A downstream query receives only some of the values from a multi-value variable.

Cause: When a variable has Multi-value enabled but Include All disabled, only the currently selected values are passed. Hidden variables compound this because users can’t adjust the selection.

Solution:

1. Enable Include All on the variable to pass the complete set of values when needed.
2. Confirm the downstream query formats the values correctly. Refer to Template variables for multi-value formatting options such as ${var:csv} and ${var:pipe}.

Request fails with “URL too large” when many values are selected

Symptoms: When All or many values are selected in a multi-value variable, the query fails because the URL exceeds the length limit the API accepts.

Cause: Each selected value expands into the URL, so a high-cardinality variable can produce a URL that’s too long for the server to accept.

Solution:

1. Reduce the number of values by adding a more selective parent variable or grouping values into categories.
2. Send the values in the request body with a POST request instead of the URL, if the API supports it.
3. Apply server-side filtering so fewer values are needed in the request.

Dashboard freezes when a variable query returns a large payload

Symptoms: The dashboard becomes unresponsive when an Infinity query used as a variable source returns a large response, for example 100 KB or more.

Cause: Variable queries are processed in the browser, so a large payload can freeze the UI.

Solution:

1. Return only the fields needed for the variable, such as the __text and __value columns.
2. Use a backend parser with a root selector or filter to reduce the response size before it reaches the browser.
3. Add server-side filters or pagination to limit the number of rows returned.

Forward OAuth not applied to variable queries

Panel queries authenticate but variable queries fail with authentication errors. Refer to Forward OAuth fails for template variable queries in the authentication section.

Other common issues

The following issues don’t produce specific error messages but are commonly encountered.

Annotations not appearing

Cause: Annotation query configuration issues.

Solution:

1. Verify the query returns the required columns: time (or timeEnd), title, and text.
2. Ensure the time column contains valid timestamps within the dashboard time range.
3. Use time filter macros (${__timeFrom}, ${__timeTo}) to filter annotations by time range.
4. Refer to Annotations for configuration details.

Data appears incorrect or misaligned

Cause: Data type or formatting issues.

Solution:

1. Verify column types match your data (string, number, timestamp).
2. Check for null values that might affect calculations.
3. Use explicit column aliases to ensure consistent naming.
4. For numeric data stored as strings, use UQL’s tonumber() or toint() functions.

Requests originate from server, not browser

Cause: Unlike some frontend data sources, Infinity makes requests from the Grafana server.

Solution:

This is expected behavior. The Grafana server makes API requests on behalf of the user. This means:

1. The target API must be accessible from the Grafana server’s network.
2. Firewall rules should allow outbound connections from the Grafana server.
3. API rate limits apply to the Grafana server’s IP address.

Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the Grafana community forums for similar issues.
2. Review the GitHub issues for known bugs.
3. Enable debug logging in Grafana to capture detailed error information.
4. Check Grafana server logs for additional error details.

When reporting issues, include:

• Grafana version
• Infinity plugin version
• Error messages (redact sensitive information)
• Steps to reproduce
• Sample query configuration (redact sensitive data)
• Sample API response structure (redact sensitive data)