Build a data source plugin

Introduction

Grafana supports a wide range of data sources, including Prometheus, MySQL, and even Datadog. There’s a good chance you can already visualize metrics from the systems you have set up. In some cases, though, you already have an in-house metrics solution that you’d like to add to your Grafana dashboards. This tutorial teaches you to build a support for your data source.

In this tutorial, you’ll:

  • Build a data source to visualize a sine wave
  • Construct queries using the query editor
  • Configure your data source using the config editor

Prerequisites

  • Grafana 7.0
  • NodeJS 12.x
  • yarn

Set up your environment

Before you can get started building plugins, you need to set up your environment for plugin development.

To discover plugins, Grafana scans a plugin directory, the location of which depends on your operating system.

  1. Create a directory called grafana-plugins in your preferred workspace.

  2. Find the plugins property in the Grafana configuration file and set the plugins property to the path of your grafana-plugins directory. Refer to the Grafana configuration documentation for more information.

    [paths]
    plugins = "/path/to/grafana-plugins"
    
  3. Restart Grafana if it’s already running, to load the new configuration.

Alternative method: Docker

If you don’t want to install Grafana on your local machine, you can use Docker.

To set up Grafana for plugin development using Docker, run the following command:

docker run -d -p 3000:3000 -v "$(pwd)"/grafana-plugins:/var/lib/grafana/plugins --name=grafana grafana/grafana:7.0.0

Since Grafana only loads plugins on start-up, you need to restart the container whenever you add or remove a plugin.

docker restart grafana

Create a new plugin

Tooling for modern web development can be tricky to wrap your head around. While you certainly can write your own webpack configuration, for this guide, you’ll be using grafana-toolkit.

grafana-toolkit is a CLI application that simplifies Grafana plugin development, so that you can focus on code. The toolkit takes care of building and testing for you.

  1. In the plugin directory, create a plugin from template using the plugin:create command:

    npx @grafana/toolkit plugin:create my-plugin
    
  2. Change directory.

    cd my-plugin
    
  3. Download necessary dependencies:

    yarn install
    
  4. Build the plugin:

    yarn dev
    
  5. Restart the Grafana server for Grafana to discover your plugin.

  6. Open Grafana and go to Configuration -> Plugins. Make sure that your plugin is there.

By default, Grafana logs whenever it discovers a plugin:

INFO[01-01|12:00:00] Registering plugin       logger=plugins name=my-plugin

Anatomy of a plugin

Plugins come in different shapes and sizes. Before we dive deeper, let’s look at some of the properties that are shared by all of them.

Every plugin you create will require at least two files: plugin.json and module.ts.

plugin.json

When Grafana starts, it scans the plugin directory for any subdirectory that contains a plugin.json file. The plugin.json file contains information about your plugin, and tells Grafana about what capabilities and dependencies your plugin needs.

While certain plugin types can have specific configuration options, let’s look at the mandatory ones:

  • type tells Grafana what type of plugin to expect. Grafana supports three types of plugins: panel, datasource, and app.
  • name is what users will see in the list of plugins. If you’re creating a data source, this is typically the name of the database it connects to, such as Prometheus, PostgreSQL, or Stackdriver.
  • id uniquely identifies your plugin, and should start with your Grafana username, to avoid clashing with other plugins. Sign up for a Grafana account to claim your username.

To see all the available configuration settings for the plugin.json, refer to the plugin.json Schema.

module.ts

After discovering your plugin, Grafana loads the module.ts file, the entrypoint for your plugin. module.ts exposes the implementation of your plugin, which depends on the type of plugin you’re building.

Specifically, module.ts needs to expose an object that extends GrafanaPlugin, and can be any of the following:

Data source plugins

A data source in Grafana must extend the DataSourceApi interface, which requires you to defines two methods: query and testDatasource.

The query method

The query method is the heart of any data source plugin. It accepts a query from the user, retrieves the data from an external database, and returns the data in a format that Grafana recognizes.

async query(options: DataQueryRequest<MyQuery>): Promise<DataQueryResponse>

The options object contains the queries, or targets, that the user made, along with context information, like the current time interval. Use this information to query an external database.

The term target originates from Graphite, and the earlier days of Grafana when Graphite was the only supported data source. As Grafana gained support for more data sources, the term “target” became synonymous with any type of query.

Test your data source

testDatasource implements a health check for your data source. For example, Grafana calls this method whenever the user clicks the Save & Test button, after changing the connection settings.

async testDatasource()

Data frames

Nowadays there are countless of different databases, each with their own ways of querying data. To be able to support all the different data formats, Grafana consolidates the data into a unified data structure called data frames.

Let’s see how to create and return a data frame from the query method. In this step, you’ll change the code in the starter plugin to return a sine wave.

  1. In the current query method, remove the code inside the map function.

    The query method now look like this:

    async query(options: DataQueryRequest<MyQuery>): Promise<DataQueryResponse> {
      const { range } = options;
      const from = range!.from.valueOf();
      const to = range!.to.valueOf();
    
      const data = options.targets.map(target => {
        // Your code goes here.
      });
    
      return { data };
    }
    
  2. In the map function, use the lodash/defaults package to set default values for query properties that haven’t been set:

    const query = defaults(target, defaultQuery);
    
  3. Create a data frame with a time field and a number field:

    const frame = new MutableDataFrame({
      refId: query.refId,
      fields: [
        { name: 'time', type: FieldType.time },
        { name: 'value', type: FieldType.number },
      ],
    });
    

    refId needs to be set to tell Grafana which query that generated this date frame.

Next, we’ll add the actual values to the data frame. Don’t worry about the math used to calculate the values.

  1. Create a couple of helper variables:

    // duration of the time range, in milliseconds.
    const duration = to - from;
    
    // step determines how close in time (ms) the points will be to each other.
    const step = duration / 1000;
    
  2. Add the values to the data frame:

    for (let t = 0; t < duration; t += step) {
      frame.add({ time: from + t, value: Math.sin((2 * Math.PI * t) / duration) });
    }
    

    The frame.add() accepts an object where the keys corresponds to the name of each field in the data frame.

  3. Return the data frame:

    return frame;
    
  4. Rebuild the plugin and try it out.

Your data source is now sending data frames that Grafana can visualize. Next, we’ll look at how you can control the frequency of the sine wave by defining a query.

In this example, we’re generating timestamps from the current time range. This means that you’ll get the same graph no matter what time range you’re using. In practice, you’d instead use the timestamps returned by your database.

Define a query

Most data sources offer a way to query specific data. MySQL and PostgreSQL use SQL, while Prometheus has its own query language, called PromQL. No matter what query language your databases are using, Grafana lets you build support for it.

Add support for custom queries to your data source, by implementing a your own query editor, a React component that enables users to build their own queries, through a user-friendly graphical interface.

A query editor can be as simple as a text field where the user edits the raw query text, or it can provide a more user-friendly form with drop-down menus and switches, that later gets converted into the raw query text before it gets sent off to the database.

Define the query model

The first step in designing your query editor is to define its query model. The query model defines the user input to your data source.

We want to be able to control the frequency of the sine wave, so let’s add another property.

  1. Add a new number property called frequency to the query model:

    src/types.ts

    export interface MyQuery extends DataQuery {
      queryText?: string;
      constant: number;
      frequency: number;
    }
    
  2. Set a default value to the new frequency property:

    export const defaultQuery: Partial<MyQuery> = {
      constant: 6.5,
      frequency: 1.0,
    };
    

Bind the model to a form

Now that you’ve defined the query model you wish to support, the next step is to bind the model to a form. The FormField is a text field component from grafana/ui that lets you register a listener which will be invoked whenever the form field value changes.

  1. Add a new form field to the query editor to control the new frequency property.

    QueryEditor.tsx

    const { queryText, constant, frequency } = query;
    
    <FormField
      width={4}
      value={frequency}
      onChange={this.onFrequencyChange}
      label="Frequency"
      type="number"
    />
    
  2. Add a event listener for the new property.

    onFrequencyChange = (event: ChangeEvent<HTMLInputElement>) => {
      const { onChange, query, onRunQuery } = this.props;
      onChange({ ...query, frequency: parseFloat(event.target.value) });
      // executes the query
      onRunQuery();
    };
    

    The registered listener, onFrequencyChange, calls onChange to update the current query with the value from the form field.

    onRunQuery(); tells Grafana to run the query after each change. For fast queries, this is recommended to provide a more responsive experience.

Use the property

The new query model is now ready to use in our query method.

  1. In the query method, use the frequency property to adjust our equation.

    frame.add({ time: from + t, value: Math.sin((2 * Math.PI * query.frequency * t) / duration) });
    

Configure your data source

To access a specific data source, you often need to configure things like hostname, credentials, or authentication method. A config editor lets your users configure your data source plugin to fit their needs.

The config editor looks similar to the query editor, in that it defines a model and binds it to a form.

Since we’re not actually connecting to an external database in our sine wave example, we don’t really need many options. To show you how you can add an option however, we’re going to add the wave resolution as an option.

The resolution controls how close in time the data points are to each other. A higher resolution means more points closer together, at the cost of more data being processed.

Define the options model

  1. Add a new number property called resolution to the options model.

    types.ts

    export interface MyDataSourceOptions extends DataSourceJsonData {
      path?: string;
      resolution?: number;
    }
    

Bind the model to a form

Just like query editor, the form field in the config editor calls the registered listener whenever the value changes.

  1. Add a new form field to the query editor to control the new resolution option.

    ConfigEditor.tsx

    <div className="gf-form">
      <FormField
        label="Resolution"
        onChange={this.onResolutionChange}
        value={jsonData.resolution || ''}
        placeholder="Enter a number"
      />
    </div>
    
  2. Add a event listener for the new option.

    onResolutionChange = (event: ChangeEvent<HTMLInputElement>) => {
      const { onOptionsChange, options } = this.props;
      const jsonData = {
        ...options.jsonData,
        resolution: parseFloat(event.target.value),
      };
      onOptionsChange({ ...options, jsonData });
    };
    

    The onResolutionChange listener calls onOptionsChange to update the current options with the value from the form field.

Use the option

  1. Create a property called resolution to the DataSource class.

    export class DataSource extends DataSourceApi<MyQuery, MyDataSourceOptions> {
      resolution: number;
    
      constructor(instanceSettings: DataSourceInstanceSettings<MyDataSourceOptions>) {
        super(instanceSettings);
    
        this.resolution = instanceSettings.jsonData.resolution || 1000.0;
      }
    
      // ...
    
  2. In the query method, use the resolution property to calculate the step size.

    src/DataSource.ts

    const step = duration / this.resolution;
    

Get data from an external API

So far, you’ve generated the data returned by the data source. A more realistic use case would be to fetch data from an external API.

While you can use something like axios or the Fetch API to make requests, we recommend using the getBackendSrv function from the grafana/runtime package.

The main advantage of getBackendSrv is that it proxies requests through the Grafana server rather making the request from the browser. This is strongly recommended when making authenticated requests to an external API. For more information on authenticating external requests, refer to [Add authentication for data source plugins].

  1. Import getBackendSrv.

    src/DataSource.ts

    import { getBackendSrv } from "@grafana/runtime"
    
  2. Create a helper method doRequest and use the datasourceRequest method to make a request to your API. Replace https://api.example.com/metrics to point to your own API endpoint.

    async doRequest(query: MyQuery) {
      const result = await getBackendSrv().datasourceRequest({
        method: "GET",
        url: "https://api.example.com/metrics",
        params: query,
      })
    
      return result;
    }
    
  3. Make a request for each query. Promises.all waits for all requests to finish before returning the data.

    async query(options: DataQueryRequest<MyQuery>): Promise<DataQueryResponse> {
      const promises = options.targets.map((query) =>
        this.doRequest(query).then((response) => {
          const frame = new MutableDataFrame({
            refId: query.refId,
            fields: [
              { name: "Time", type: FieldType.time },
              { name: "Value", type: FieldType.number },
            ],
          });
    
          response.data.forEach((point: any) => {
            frame.appendRow([point.time, point.value]);
          });
    
          return frame;
        })
      );
    
      return Promise.all(promises).then((data) => ({ data }));
    }
    

Congratulations

Congratulations, you made it to the end of this tutorial!

You’ve built a complete data source plugin for Grafana that uses a query editor to control what data to visualize. You’ve added a data source option, commonly used to set connection options and more.

Learn more

Learn how you can improve your plugin even further, by reading our advanced guides: