Documentation Grafana Cloud Observe and act Connect to data sources Private data source connect (PDC) PDC over Tailscale

Query data sources on a Tailscale network

Private data source connect (PDC) can query a data source on a Tailscale network, called a tailnet, without a PDC agent and without opening that data source to the public internet. For the data sources you can query this way, refer to Supported data sources.

If the data source is not on a tailnet, deploy a PDC agent instead. Refer to Configure PDC.

Before you begin

You need the following:

• A Grafana Cloud stack. All plans including free tier are supported.
• A Tailscale account. All plans including free tier are supported.
• The tailnet address of the data source. That address is the MagicDNS machine name and port, or the Tailscale IPv4 address and port. For example, my-computer:9090.

• Optional, for an OAuth client secret: permission to create an OAuth client in the Tailscale admin console. Refer to OAuth clients for who can create one.
• Optional, for an auth key: permission to generate an auth key in the Tailscale admin console. Refer to Auth keys for who can create one.

Understand the connection

When you configure a data source for Tailscale, Grafana Cloud sends its queries through PDC. PDC adds an ephemeral node to your tailnet, or reuses a node that is already running. Tailscale then connects that node to the data source.

flowchart LR
  stack["Grafana Cloud stack"]
  pdc["PDC"]
  node["Ephemeral tailnet node"]
  ds["Data source on the tailnet"]
  stack -->|query| pdc
  pdc -->|"join or reuse"| node
  node -->|"MagicDNS or Tailscale IP"| ds

An ephemeral node is a machine Tailscale removes after it disconnects, so the machine does not stay in the Tailscale admin console.

The machine label becomes part of the ephemeral node name. Data sources that share a machine label share those nodes. The label does not need to be unique to a tailnet, so data sources on different tailnets can use the same label.

Limit access with tags

Tags on that node let you write ACLs and Grants that limit which endpoints the stack can access. Refer to the Tailscale documentation for tailnets, ephemeral nodes, tags, ACLs, and Grants.

Query more than one tailnet

You can query data sources on more than one tailnet. Credentials are set per data source, so create a separate data source for each tailnet and give each one the OAuth client secret or auth key for that tailnet. Grafana Cloud adds ephemeral nodes to each tailnet.

Limitations

PDC supports only backend data source plugins, including when using Tailscale. The agent throughput limits do not apply, because Tailscale does not use a PDC agent. The only other PDC limitation that applies over Tailscale is latency: the SOCKS5 protocol can add minor latency to queries, typically tens of milliseconds, which should not be readily noticeable.

Configure Tailscale access credentials

PDC accepts either a Tailscale OAuth client secret or a Tailscale auth key. We recommend the use of an OAuth client secret, because it does not expire. A Tailscale auth key expires after 90 days.

When PDC registers a node with an OAuth client secret, it advertises the tag:grafana-pdc tag. Your OAuth client must own that tag, and the tag must exist in your tailnet policy.

Create a Tailscale OAuth client

1. In the Tailscale admin console, open Trust credentials.
2. Select Credential, then OAuth.
3. Give the client the auth_keys and devices:core scopes and the tag:grafana-pdc tag.
4. Select Generate credential.
5. Copy the client secret before you close the page. Tailscale does not show the secret again.

PDC registers the node as ephemeral, so it does not stay in your tailnet. You do not set Ephemeral yourself.

Create a Tailscale auth key

A Tailscale auth key expires after 90 days. Before it expires, generate a new key and paste it into the data source.

1. In the Tailscale administrator console, open Keys.
2. Select Generate auth keys.
3. Select Reusable.
4. Select Ephemeral.
5. Copy the key before you close the page. Tailscale does not show the secret again.

Configure a data source

Add a data source and point it at the tailnet address. The Tailscale tab is under Private data source connect in the data source configuration.

1. Go to Connections > Add new connection and add a data source that supports PDC.
2. In the URL field, enter the tailnet address. For example, enter my-computer:3306 for MySQL on that machine and port. Some data sources need a protocol, for example http://my-computer:9090 for a local Prometheus server.
3. Under Private data source connect, select the Tailscale tab.
4. In Machine label, enter a short name for this tailnet. Use the same label for every data source on that tailnet.
5. Paste the OAuth client secret or auth key into the credential field.
6. Click Save & test.

After a short delay, Save & test succeeds and your tailnet shows a new machine named grafanacloud-<STACKID>-tailscale-<LABEL>. STACKID is your Grafana Cloud stack ID, and LABEL is the machine label. PDC shares connection state across its replicas, so you no longer get one machine per replica on your tailnet.

Troubleshoot

If you hit a problem connecting to a data source, check the following:

• The data source does not connect by machine name. Use the Tailscale IPv4 address and port instead of the MagicDNS machine name.
• Save & test shows an unexpected error in Chrome. The first few attempts can fail while the ephemeral node starts. Wait a few seconds and try again.
• Queries fail after the credential stops working. An invalid or expired credential fails with a generic error that does not name the cause. Regenerate the credential in Tailscale and paste the new secret into the data source.

For other connection errors, refer to Troubleshoot PDC.

Next steps

• Private data source connect (PDC)
• Configure PDC for a data source that is not on a tailnet