Plugins 〉Amazon Athena
Amazon Athena
Grafana 10 breaking change: update Athena datasource plugin to >=2.9.3
Grafana 10.0.0 was shipped with the new React 18 upgrade. Changes in batching of state updates in React 18 cause a bug in the query editor in Athena versions <=2.9.2. If you’re using Grafana@>=10.0.0, please update your plugin to version 2.9.3 or higher in your Grafana instance management console.
Athena data source for Grafana
The Athena data source plugin allows you to query and visualize Athena data metrics from within Grafana.
This topic explains options, variables, querying, and other options specific to this data source. Refer to Add a data source for instructions on how to add a data source to Grafana.
Configure the data source in Grafana
To access data source settings:
- Hover your mouse over the Configuration (gear) icon.
- Click Data sources, and then click the AWS Athena data source.
Name | Description |
---|---|
Name | The data source name. This is how you refer to the data source in panels and queries. |
Default | Default data source means that it will be pre-selected for new panels. |
Auth Provider | Specify the provider to get credentials. |
Access Key ID | If Access & secret key is selected, specify the Access Key of the security credentials to use. |
Secret Access Key | If Access & secret key is selected, specify the Secret Key of the security credentials to use. |
Credentials Profile Name | Specify the name of the profile to use (if you use ~/.aws/credentials file), leave blank for default. |
Assume Role Arn (optional) | Specify the ARN of the role to assume. |
External ID (optional) | If you are assuming a role in another account, that has been created with an external ID, specify the external ID here. |
Endpoint (optional) | Optionally, specify a custom endpoint for the service. |
Default Region | Region in which the cluster is deployed. |
Catalog (datasource) | Athena catalog. The list of catalogs will be retrieved automatically. |
Database | Name of the database within the catalog. |
Workgroup | Workgroup to use. |
Output Location | AWS S3 bucket to store execution outputs. If not specified, the default query result location from the Workgroup configuration will be used. Please note that if Override client-side settings is enabled in the AWS console, Output Location will be ignored. |
Authentication
For authentication options and configuration details, see AWS authentication topic.
IAM policies
Grafana needs permissions granted via IAM to be able to read Athena metrics. You can attach these permissions to IAM roles and utilize Grafana's built-in support for assuming roles. Note that you will need to configure the required policy before adding the data source to Grafana.
Depending on the source of the data you'd query with Athena, you may need different permissions. AWS provides some predefined policies that you can check here.
This is an example of a minimal policy you can use to query Athena. It is based on the AmazonAthenaFullAccess
policy, without write permissions when possible, since Grafana should be used as read-only:
NOTE: Update the ARN of the S3 bucket if you are using a custom one.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AthenaQueryAccess",
"Effect": "Allow",
"Action": [
"athena:ListDatabases",
"athena:ListDataCatalogs",
"athena:ListWorkGroups",
"athena:GetDatabase",
"athena:GetDataCatalog",
"athena:GetQueryExecution",
"athena:GetQueryResults",
"athena:GetTableMetadata",
"athena:GetWorkGroup",
"athena:ListTableMetadata",
"athena:StartQueryExecution",
"athena:StopQueryExecution"
],
"Resource": ["*"]
},
{
"Sid": "GlueReadAccess",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions",
"glue:BatchGetPartition"
],
"Resource": ["*"]
},
{
"Sid": "AthenaS3Access",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:PutObject"
],
"Resource": ["arn:aws:s3:::aws-athena-query-results-*"]
},
{
"Sid": "AthenaExamplesS3Access",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": ["arn:aws:s3:::athena-examples*"]
}
]
}
Query Athena data
The provided query editor is a standard SQL query editor. Grafana includes some macros to help with writing more complex timeseries queries.
Macros
Macro | Description | Example | Output example |
---|---|---|---|
$__dateFilter(column) | $__dateFilter creates a conditional that filters the data (using column ) based on the date range of the panel. | $__dateFilter(my_date) | my_date BETWEEN date '2017-07-18' AND date '2017-07-18' |
$__parseTime(column,format) | $__parseTime cast a varchar as a timestamp with the given format. | $__parseTime(eventtime, 'yyyy-MM-dd''T''HH:mm:ss''Z') | parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss''Z') |
$__timeFilter(column,format) | $__timeFilter creates a conditional that filters the data (using column ) based on the time range of the panel. The second argument is used to optionally parse the column from a varchar to a timestamp with a specific format. Keep in mind that this macro uses Presto's Java Date Functions parse_datetime(string, format) when a custom format is passed as format argument. | 1. Without specifying a format: $__timeFilter(time) 2. Using the default format: $__timeFilter(time, 'yyyy-MM-dd HH:mm:ss') 3. With another custom format: $__timeFilter(time, 'yyyy-MM-dd''T''HH:mm:ss''+0000') | 1. Without specifying a format: time BETWEEN TIMESTAMP '2017-07-18 11:15:52' AND TIMESTAMP '2017-07-18 11:25:52' 2. Using the default format: TIMESTAMP time BETWEEN TIMESTAMP '2017-07-18T11:15:52Z' AND TIMESTAMP '2017-07-18T11:15:52Z' 3. With another custom format: parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss''+0000') BETWEEN TIMESTAMP '2017-07-18 11:15:52' AND TIMESTAMP '2017-07-18 11:25:52' |
$__timeFrom() | $__timeFrom outputs the current starting time of the range of the panel with quotes. | $__timeFrom() | TIMESTAMP '2017-07-18 11:15:52' |
$__rawTimeFrom() | $__rawTimeFrom outputs the current starting time of the range of the panel formatted as a string. An optional argument is used to specify the output format of the string using Joda's DateTime format. | 1. Without specifying a format: $__rawTimeFrom() 2. Using the default format: $__rawTimeFrom('yyyy-MM-dd HH:mm:ss') 3. With a custom format: $__rawTimeFrom('yyyy/MM/dd/HH) | 1. Without specifying a format: '2022-03-24 21:19:03' 2. Using the default format: '2022-03-24 21:19:03' 3. With another custom format: '2022/03/24/21' |
$__timeTo() | $__timeTo outputs the current ending time of the range of the panel with quotes. | $__timeTo() | TIMESTAMP '2017-07-18 11:15:52' |
$__rawTimeTo() | $__rawTimeTo outputs the current ending time of the range of the panel formatted as a string. An optional argument is used to specify the output format of the string using Joda's DateTime format. | 1. Without specifying a format: $__rawTimeTo() 2. Using the default format: $__rawTimeTo('yyyy-MM-dd HH:mm:ss') 3. With a custom format: $__rawTimeTo('yyyy/MM/dd/HH) | 1. Without specifying a format: '2022-03-24 21:19:03' 2. Using the default format: '2022-03-24 21:19:03' 3. With another custom format: '2022/03/24/21' |
$__timeGroup(column, '1m', format) | $__timeGroup groups timestamps so that there is only 1 point for every period on the graph. The third argument is used to optionally parse the column from a varchar to a timestamp with a specific format. | $__timeGroup(time,'5m','yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z') | FROM_UNIXTIME(FLOOR(TO_UNIXTIME(parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z'))/300)*300) |
$__unixEpochFilter(column) | $__unixEpochFilter achieves the same than $__timeFilter but when the time is a UNIX timestamp. | $__unixEpochFilter(time) | time BETWEEN 1637228322 AND 1637232700 |
$__unixEpochGroup(column, '1m') | $__unixEpochGroup achieves the same than $__timeGroup but when the time is a UNIX timestamp. | $__unixEpochGroup(time, '5m') | FROM_UNIXTIME(FLOOR(time/300)*300) |
$__table | $__table returns the table selected in the Table selector | $__table | my_table |
$__column | $__column returns the column selected in the Column selector (it requires a table) | $__column | col1 |
Table Visualization
Most queries in Athena will be best represented by a table visualization. Any query will display data in a table. Any query that returns results will display data in a table..
This example returns results for a table visualization:
SELECT {column_1}, {column_2} FROM {table};
Timeseries and Graph visualizations
Time series ad graph visualizations, you must:
- Select a column with a
date
ordatetime
type. Thedate
column must be in ascending order (usingORDER BY column ASC
). - Also select a numeric column.
Inspecting the query
Grafana supports macros that Athena does not, which means a query might not work when copied and pasted directly into Athena. To view the full interpolated query, which works directly in Athena, click the Query Inspector button. The full query is displayed under the Query tab.
Templates and variables
To add a new Athena query variable, refer to Add a query variable.
Any value queried from an Athena table can be used as a variable.
After creating a variable, you can use it in your Athena queries by using Variable syntax. For more information about variables, refer to Templates and variables.
Annotations
Annotations allow you to overlay rich event information on top of graphs. You can add annotations by clicking on panels or by adding annotation queries via the Dashboard menu / Annotations view.
Example query to automatically add annotations:
SELECT
time as time,
environment as tags,
humidity as text
FROM
tableName
WHERE
$__dateFilter(time) and humidity > 95
The following table represents the values of the columns taken into account to render annotations:
Name | Description |
---|---|
time | The name of the date/time field. Could be a column with a native SQL date/time data type or epoch value. |
timeend | Optional name of the end date/time field. Could be a column with a native SQL date/time data type or epoch value. (Grafana v6.6+) |
text | Event description field. |
tags | Optional field name to use for event tags as a comma separated string. |
Provision Athena data source
You can configure the Athena data source using configuration files with Grafana's provisioning system or using Grafana's Datasource JSON API . For more information, refer to the provisioning docs page.
Here are some provisioning examples.
Using AWS SDK (default)
apiVersion: 1
datasources:
- name: Athena
type: grafana-athena-datasource
jsonData:
authType: default
defaultRegion: eu-west-2
catalog: AwsDataCatalog
database: '<your athena database>'
workgroup: '<your athena workgroup>'
Using credentials' profile name (non-default)
apiVersion: 1
datasources:
- name: Athena
type: grafana-athena-datasource
jsonData:
authType: credentials
defaultRegion: eu-west-2
profile: secondary
catalog: AwsDataCatalog
database: ‘<your athena database>’
workgroup: ‘<your athena workgroup>’
Using accessKey
and secretKey
apiVersion: 1
datasources:
- name: Athena
type: grafana-athena-datasource
jsonData:
authType: keys
defaultRegion: eu-west-2
secureJsonData:
accessKey: ‘<your access key>’
secretKey: ‘<your secret key>’
catalog: AwsDataCatalog
database: ‘<your athena database>’
workgroup: ‘<your athena workgroup>’
Using AWS SDK Default and ARN of IAM Role to Assume
apiVersion: 1
datasources:
- name: Athena
type: grafana-athena-datasource
jsonData:
authType: default
assumeRoleArn: arn:aws:iam::123456789012:root
defaultRegion: eu-west-2
catalog: AwsDataCatalog
database: '<your athena database>'
workgroup: '<your athena workgroup>'
There are also some optional parameters to configure this datasource:
jsonData:
endpoint: https://'{service}.{region}.amazonaws.com'
externalId: '<your role external id>'
outputLocation: s3://'<your s3 bucket>'
Acknowledgment
The backend driver is based on the implementation of uber/athenadriver, which provides a fully-featured driver for AWS Athena.
Get the most out of the plugin
- Add Annotations.
- Configure and use Templates and variables.
- Add Transformations.
- Set up alerting; refer to Alerts overview.
Async Query Data Support
Async query data support enables an asynchronous query handling flow. With async query data support enabled, queries are handled over multiple requests (starting, checking its status, and fetching the results) instead of starting and resolving a query over a single request. This is useful for queries that can potentially run for a long time and timeout.
To enable async query data support, you need to set feature toggle athenaAsyncQueryDataSupport
to true
. See Configure feature toggles for details.
Async Query Caching
To enable query caching for async queries, you need to be on Grafana version 10.1 or above, and to set the feature toggles useCachingService
and awsAsyncQueryCaching
to true
. You'll also need to configure query caching for the specific Athena datasource.
Query Result Reuse
Query result reuse is a feature that allows Athena to reuse query results from previous queries. You can enable it per query by selecting the Enabled
checkbox under the Query result reuse
section in the query editor. Learn more in the AWS Athena documentation.
Note: Result reuse requires Athena to be on engine version 3. AWS provides instructions for Changing Athena engine versions.
Grafana Cloud Pro
- $25 / user / month and includes a free trial for new users
- Available with a Grafana Cloud Pro plan
- Access to 1 Enterprise plugin
- Fully managed service (not available to self-manage)
Grafana Cloud Advanced / Grafana Enterprise
- Available with a Grafana Cloud Advanced plan or Grafana Enterprise license
- Access to all Enterprise plugins
- Run fully managed or self-manage on your own infrastructure
Installing Amazon Athena on Grafana Cloud:
Installing plugins on a Grafana Cloud instance is a one-click install; same with updates. Cool, right?
Note that it could take up to 1 minute to see the plugin show up in your Grafana.
Installing plugins on a Grafana Cloud instance is a one-click install; same with updates. Cool, right?
Note that it could take up to 1 minute to see the plugin show up in your Grafana.
Installing plugins on a Grafana Cloud instance is a one-click install; same with updates. Cool, right?
Note that it could take up to 1 minute to see the plugin show up in your Grafana.
Installing plugins on a Grafana Cloud instance is a one-click install; same with updates. Cool, right?
Note that it could take up to 1 minute to see the plugin show up in your Grafana.
Installing plugins on a Grafana Cloud instance is a one-click install; same with updates. Cool, right?
Note that it could take up to 1 minute to see the plugin show up in your Grafana.
For more information, visit the docs on plugin installation.
Installing on a local Grafana:
For local instances, plugins are installed and updated via a simple CLI command. Plugins are not updated automatically, however you will be notified when updates are available right within your Grafana.
1. Install the Data Source
Use the grafana-cli tool to install Amazon Athena from the commandline:
grafana-cli plugins install
The plugin will be installed into your grafana plugins directory; the default is /var/lib/grafana/plugins. More information on the cli tool.
Alternatively, you can manually download the .zip file for your architecture below and unpack it into your grafana plugins directory.
Alternatively, you can manually download the .zip file and unpack it into your grafana plugins directory.
2. Configure the Data Source
Accessed from the Grafana main menu, newly installed data sources can be added immediately within the Data Sources section.
Next, click the Add data source button in the upper right. The data source will be available for selection in the Type select box.
To see a list of installed data sources, click the Plugins item in the main menu. Both core data sources and installed data sources will appear.
Changelog
2.11.1
- Update @grafana/aws-sdk to 0.1.2 to fix bug with temporary credentials
2.11.0
- Update grafana-aws-sdk to v0.19.1 to add
il-central-1
to opt-in region list
2.10.3
- Upgrade @grafana/async-query-data to reduce minimum query time https://github.com/grafana/athena-datasource/pull/265
2.10.2
- Upgrade @grafana/aws-sdk to v0.0.48 to use @grafana/runtime instead of grafanaBootData #267
- Remove unused updated field from structs #263
- Fix connection error when changing access and secret key #262
2.10.1
- Update dependencies and rebuild lockfile
2.10.0
- Upgrade @grafana/aws-sdk to v0.0.47 to support numeric values when applying template variables to SQL queries
2.9.6
- Fix: include partitions as columns https://github.com/grafana/athena-datasource/pull/244
- Fix: update go modules to fix column types being treated as strings https://github.com/grafana/athena-datasource/pull/252
2.9.5
- Fix: allow returning nil for missing values https://github.com/grafana/athena-datasource/pull/241
- Fix: handle expression queries correctly for async queries https://github.com/grafana/athena-datasource/pull/240
2.9.4
- Upgrade Readme.md re: Grafana 10 https://github.com/grafana/athena-datasource/pull/237
2.9.3
- Upgrade grafana/aws-sdk-react to 0.0.46 https://github.com/grafana/athena-datasource/pull/235
2.9.2
- Revert grafana-plugin-sdk-go version to 0.139.0 to fix https://github.com/grafana/athena-datasource/issues/233. Should be same behavior as behavior with no known issues.
- Update grafana-aws-sdk version to include new region to opt-in region list https://github.com/grafana/grafana-aws-sdk/pull/80
2.9.1
- Update async-query-data with a fix for errors in #232
2.9.0
- Update backend dependencies
2.8.0
- Add Query Result Reuse Support to Frontend by @kevinwcyu in #215
2.7.0
- Add header to Query Editor by @idastambuk in #217
- Add Query Result Reuse Support to Backend by @kevinwcyu in #214
2.6.0
- Hide Run query buttons in annotations in https://github.com/grafana/athena-datasource/pull/211
2.5.0
- Upgrade AWS go SDK to latest version by @annerajb in https://github.com/grafana/athena-datasource/pull/208
2.4.0
- Migrate to create-plugin by @iwysiu in https://github.com/grafana/athena-datasource/pull/190
- Update code coverage version to 0.1.13 by @idastambuk in https://github.com/grafana/athena-datasource/pull/197
- Re-enable annotations e2e test by @kevinwcyu in https://github.com/grafana/athena-datasource/pull/198
- A11y: Add a11y lint plugin by @idastambuk in https://github.com/grafana/athena-datasource/pull/199
- Upgrade grafana-aws-sdk to v0.12.0 by @fridgepoet in https://github.com/grafana/athena-datasource/pull/203
2.3.1
- Hide the stop button when async query data support is not enabled https://github.com/grafana/athena-datasource/pull/193
- Interpolate query properties with variables only if property is set https://github.com/grafana/athena-datasource/pull/194
2.3.0
- Add Async Query Data Support https://github.com/grafana/athena-datasource/pull/174
2.2.3
- Update @grafana dependencies to v8.5.10 https://github.com/grafana/athena-datasource/pull/191
2.2.2
- Security: Upgrade Go in build process to 1.19.3
2.2.1
- Security: Upgrade Go in build process to 1.19.2
2.2.0
- Upgrade to grafana-aws-sdk v0.11.0 by @fridgepoet in https://github.com/grafana/athena-datasource/pull/175
Full Changelog: https://github.com/grafana/athena-datasource/compare/v2.1.0...v2.2.0
2.1.0
- Add support for context aware autocompletion by @sunker in https://github.com/grafana/athena-datasource/pull/171
2.0.1
- Fix bug with caching invalid auth args (https://github.com/grafana/athena-datasource/issues/144)
- Code coverage and Codeowners updates
2.0.0
- Stopping support for Grafana versions under
8.0.0
. - Add template variables as options to Athena connection details (#151)
- Fix error message not forwarded when listing resources in config editor (#141)
1.0.5
- Fixes a bug that caused invalid tokens when assuming a role in private clouds (e.g.
gov
).
1.0.4
- Add macros to output the raw starting or ending time of the panel's range
1.0.3
- Fixes bugs for Endpoint and Assume Role settings
1.0.2
- Includes a curated dashboard for Amazon VPC Flow Logs.
- Fixes a bug in which modifications to the data source configuration were not being applied (#105).
- Add User-Agent to requests.
1.0.1
Fixes a bug that caused several instances of the plugin to accidentally connect to the same instance.
1.0.0
Initial release.
0.2.3
Add macros for Epoch & Unix timestamp data.
0.2.2
Fix queries with missing data
0.2.1
Improve tooltip text.
0.2.0
Allow to set data source output location.
0.1.2
Allow to cancel and stop requests. Bug fixing.
0.1.1
Added a curated dashboard
0.1.0
Initial release.