Plugins 〉X-Ray


Developer
Grafana


Sign up to receive occasional product news and updates:



Latest webinars
Data Source
grafana

X-Ray

  • Overview
  • Installation
  • Change log
  • Related content

X-Ray data source

X-Ray datasource plugin provides a support for AWS X-Ray. Add it as a data source, then you are ready to build dashboards or use Explore with X-Ray to look at traces, analytics, or insights.

Add the data source

  1. In the side menu under the Configuration link, click on Data Sources.
  2. Click the Add data source button.
  3. Select X-Ray in the Distributed tracing section.

Note: If you have issues getting this data source to work and Grafana is giving you undescriptive errors, then check your log file (try looking in /var/log/grafana/grafana.log).

NameDescription
NameThe data source name. This is how you refer to the data source in panels and queries.
DefaultDefault data source means that it will be pre-selected for new panels.
Default RegionUsed in query editor to set region. (can be changed on per query basis)
Auth ProviderSpecify the provider to get credentials.
Credentials profile nameSpecify the name of the profile to use (if you use ~/.aws/credentials file), leave blank for default.
Assume Role ArnSpecify the ARN of the role to assume.
External IDIf you are assuming a role in another account, that has been created with an external ID, specify the external ID here.

Authentication

In this section we will go through the different type of authentication you can use for X-Ray data source.

IAM Roles

Currently all access to X-Ray is done server side by the Grafana backend using the official AWS SDK. If your Grafana server is running on AWS you can use IAM Roles and authentication will be handled automatically.

See the AWS documentation on IAM Roles

Note: AWS Role Switching is not supported at the moment.

IAM Policies

Grafana needs permissions granted via IAM to be able to read X-Ray data and EC2 tags/instances/regions. You can attach these permissions to IAM roles and utilize Grafana's built-in support for assuming roles.

Here is a minimal policy example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "xray:BatchGetTraces",
        "xray:GetTraceSummaries",
        "xray:GetTraceGraph",
        "xray:GetGroups",
        "xray:GetTimeSeriesServiceStatistics",
        "xray:GetInsightSummaries",
        "xray:GetInsight",
        "xray:GetServiceGraph",
        "ec2:DescribeRegions"
      ],
      "Resource": "*"
    }
  ]
}

Example AWS credentials

If the Auth Provider is Credentials file, then Grafana tries to get credentials in the following order:

  • Hard-code credentials
  • Environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
  • Existing default config files
  • ~/.aws/credentials
  • IAM role for Amazon EC2

Refer to Configuring the AWS SDK for Go in the AWS documentation for more information.

AWS credentials file

Create a file at ~/.aws/credentials. That is the HOME path for user running grafana-server.

Note: If the credentials file in the right place, but it is not working, then try moving your .aws file to '/usr/share/grafana/'. Make sure your credentials file has at most 0644 permissions.

Example credential file:

[default]
aws_access_key_id = <your access key>
aws_secret_access_key = <your access key>
region = us-west-2

Query editor

The most important field in the editor is the query type. There are four query types:

  • Trace List (Traces in AWS)
  • Trace Statistics
  • Trace Analytics (Analytics in AWS)
  • Insights
  • Service map

x-ray-query-editor

Trace List

The Trace List type allows you to search for traces which will be shown in a table. Clicking on the trace id in the first column opens the trace on the right side. Notice the query field in the editor. You can write queries, filter expressions, or you can insert a single trace ID there that will be shown in a trace view. You can find more details about filter expressions in AWS X-Ray documentation.

x-ray-trace-list

Note: The Trace List will only show the first 1000 traces.

Trace Statistics

In Trace Statistics you can see a graph and a table showing information about error, fault, throttle, success, and total count. You can use the columns field in the query editor to only see specified columns.

x-ray-trace-statistics

Trace Analytics

In Trace Analytics you can visualize the following tables.

  • Root Cause
    • Response Time
      • Root Cause Service (Last service in path)
      • Path (multiple paths)
    • Error
      • Root Cause Service (Last service in path)
      • Path
      • Error Message
    • Fault
      • Root Cause Service (Last service in path)
      • Path
      • Error Message
  • End user Impact
  • URL
  • HTTP Status Code

Insights

In Insights you can see the summary table for Insights. Clicking the InsightId will take you to AWS console.

Service map

Service map in Grafana enables customers to view the map of their applications built using microservices architecture. Each node on the map represents a service such as an AWS Lambda function or API running on API Gateway or a DynamoDB table. With this map, customers can easily detect performance issues, or increase in error, fault or throttle rates in any of their services and dive deep into corresponding traces and root cause.

Service map

Service Map query type shows the same data as a service map inside X-ray console.

To display the service map:

  • Use the Node graph panel visualization in Grafana 7.4 plus.
  • Use Explore in Grafana 7.4 plus.
  • Show the data in a simple table. This is the only option if the Node graph panel is unavailable.

You can pan and zoom the view with buttons or you mouse. For details about the visualization, refer to Node graph panel.

Service map navigation

Similar to X-ray root nodes, nodes in the service map representing the client application are on the left side of the map.

Each service in the map is represented as a circle. Numbers on the inside shows average time per transaction and transactions per minute.

A colored circle around the service also matches colors and meaning from X-ray console representing percentage of requests.

  • green = success
  • red = fault
  • yellow = errors
  • purple = throttled responses

Click on the service or the edge to see a context menu with links additional links for quick navigation to other relevant information from X-ray. You can use the links to quickly navigate to a list of all error traces for a particular service and then continue to specific trace.

For more information about the Service map, refer to the official AWS X-ray documentation.

Alerting

Since X-Ray queries can return numeric data, alerts are supported. See the Alerting documentation for more on Grafana alerts.

Pricing

With AWS X-Ray, there are no upfront fees or commitments. You pay only for what you use, based on the number of traces recorded, retrieved, and scanned. The first 1,000,000 traces retrieved or scanned each month are free. Beyond the free tier, traces scanned cost $0.50 per 1 million traces scanned ($0.0000005 per trace). Please see the X-Ray pricing page for more details.

Configure the data source with provisioning

You can configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the provisioning docs page.

Here are some provisioning examples for this data source.

Using a credentials file

If you are using Credentials file authentication type, then you should use a credentials file with a config like this.

apiVersion: 1

datasources:

  • name: X-Ray type: grafana-x-ray-datasource jsonData: authType: credentials defaultRegion: eu-west-2

Using accessKey and secretKey

apiVersion: 1

datasources:

  • name: X-Ray type: grafana-x-ray-datasource jsonData: authType: keys defaultRegion: eu-west-2 secureJsonData: accessKey: ‘<your access key>’ secretKey: ‘<your secret key>’

Cross-Account Observability

The X-Ray plugin allows you to monitor traces across multiple AWS accounts within a region with a feature called Cross-Account Observability. Using cross-account observability, you can seamlessly search, visualize and analyze AWS traces without worrying about account boundaries.

Getting started

To enable cross-account observability, first enable the feature in AWS using the official CloudWatch docs, then add [two new API actions]({{< relref "../Metrics_and_Logs" >}}) to the IAM policy attached to the role/user running the plugin.

This feature is currently behind the cloudWatchCrossAccountQuerying feature toggle.

You can enable feature toggles through configuration file or environment variables. See configuration [docs]({{< relref "../setup-grafana/configure-grafana/#feature_toggles" >}}) for details. Grafana Cloud users can access this feature by opening a support ticket in the Cloud Portal.

Filtering Traces by Account Id

Once the feature is enabled, you will be able to both display traces across multiple accounts and filter those traces by account ID. When you select the Service Map query type in Grafana, an account dropdown displays and populates with the account IDs shown in the traces returned in the selected time frame.

You can also add account ID as part of a query filter expression in the Trace List query type.

Installing X-Ray on Grafana Cloud:

For more information, visit the docs on plugin installation.

Change Log

All notable changes to this project will be documented in this file.

2.4.0

  • Add spellcheck and fix misspellings by @sunker in https://github.com/grafana/x-ray-datasource/pull/158
  • Migrate to create-plugin by @iwysiu in https://github.com/grafana/x-ray-datasource/pull/148
  • Update code coverage workflow by @idastambuk in https://github.com/grafana/x-ray-datasource/pull/162
  • Update @grafana/aws-sdk by @kevinwcyu in https://github.com/grafana/x-ray-datasource/pull/164
  • Dependencies: Update @grafana dependencies to 9.3.2 by @idastambuk in https://github.com/grafana/x-ray-datasource/pull/167
  • Small Refactor: Do not fetch account id if not on service map page by @sarahzinger in https://github.com/grafana/x-ray-datasource/pull/169
  • Upgrade grafana-aws-sdk to v0.12.0 by @fridgepoet in https://github.com/grafana/x-ray-datasource/pull/171

Full Changelog: https://github.com/grafana/x-ray-datasource/compare/v2.2.0...v2.4.0****

2.3.0

  • Feature: Make it possible to filter on account id in https://github.com/grafana/x-ray-datasource/pull/157

2.2.0

  • Make properties of SummaryStatistics optional

2.1.2

  • Security: Upgrade Go in build process to 1.19.3

2.1.1

  • Security: Upgrade Go in build process to 1.19.2

2.1.0

  • Bump terser from 4.8.0 to 4.8.1 by @dependabot in https://github.com/grafana/x-ray-datasource/pull/130
  • Fix Trace List with variable by @iwysiu in https://github.com/grafana/x-ray-datasource/pull/132
  • Upgrade to grafana-aws-sdk v0.11.0 by @fridgepoet in https://github.com/grafana/x-ray-datasource/pull/139

Full Changelog: https://github.com/grafana/x-ray-datasource/compare/v2.0.1...v2.1.0

v.2.0.1

  • Bug fix for auth with keys: https://github.com/grafana/x-ray-datasource/pull/128
  • Code Coverage Changes
  • Code owners changes

v2.0.0

What's Changed

  • Stopping support for Grafana versions under 8.0.0 by @yaelleC in https://github.com/grafana/x-ray-datasource/pull/122

Bug fixes

  • Update minimum IAM policy by @iwysiu in https://github.com/grafana/x-ray-datasource/pull/115
  • Fix deprecated use of strings.Title by @fridgepoet in https://github.com/grafana/x-ray-datasource/pull/119

Other

  • Update grafana-aws-sdk by @andresmgot in https://github.com/grafana/x-ray-datasource/pull/121

v1.4.0

Bug fixes

  • Use non opt-in region when obtaining tokens for assumed roles
  • Fix wrong start and end times #89
  • Fix time durations after format change #88
  • Avoid error if the URL in tags cannot be parsed #84

Other

  • Modify the User-Agent for requests. Now it will follow this form: "aws-sdk-go/$aws-sdk-version ($go-version; $OS;) X-ray/$X-ray-version-$git-hash Grafana/$grafana-version"

v1.3.4

Bug fixes

  • Add readable names for stats in service map legend #82
  • Upgrade grafana-aws-sdk version fixing issue with assuming role #81

Other

  • Bump prismjs from 1.23.0 to 1.24.0 #80

v1.3.3

Other

  • Bump grafana/aws-sdk and Grafana dependency version #77

v1.3.2

Features / Enhancements

  • Use ConnectionConfig from grafana-aws-sdk #73
  • Bump prismjs from 1.21.0 to 1.23.0 #70

v1.3.1

Bug fixes

  • Remove usage of non backward compatible API preventing using the 1.3.0 version in Grafana before 7.4.0. #67

Other

  • Locks Grafana dependencies on latest stable release preventing compilation fails in CI. #68

v1.3.0

Features / Enhancements

  • Add Service Map query type that allows visualizing service map data in similar way to X-Ray console. #60

v1.2.0

Features / Enhancements

  • Updates the authentication settings in the plugin config page to include SDK default authentication mechanism and use our Grafana specific auth SDK for AWS. #59

v1.1.0

Features / Enhancements

  • Add regions selector to the query editor. #57

v1.0.1

  • Version bump needed for CI automated release

v1.0.0

  • Initial Release