DocumentationGrafana CloudConnect to data sourcesPrivate data source connect (PDC)PDC scalability and security

PDC scalability and security

PDC is scalable and secure. The following sections go into more detail regarding both scalability and security.

Private data source connect (PDC) scalability

The PDC agent scales horizontally. In order to increase throughput, you can increase the number of replicas. Connections will be load balanced across the various replicas by the PDC service. The PDC service in Grafana Cloud is also a horizontally scalable distributed system. SSH connections from your network are spread across many instances to ensure high availability.

You do not need to set up a separate agent for each data source. A single agent connects a hosted Grafana instance to multiple data sources as long they are in the same network.

Using SSH allows multiplexing multiple SOCKS connections over a single SSH tunnel as long as the data sources all exist within the same network.

PDC allows for seamless failover. If you deploy three agents, each in two data centers, PDC will load balance across all six (3 x 2) replicas regardless of data center. If a connection is lost with one data center, the PDC service on Grafana Cloud will simply fail over to another replica.

Private data source connect (PDC) security

SSH tunnels are widely accepted by the security community to be secure. The existence of a tunnel does not increase the risk of the PDC proxy being compromised. You control the SSH tunnel initiated by the PDC agent. The tunnel can be shut down or killed in the event there is any suspicion of compromise.

Grafana Labs conducted an external penetration test and found no vulnerabilities. The full penetration test report can be found here.

Both Grafana Labs and you, the end user, are able to restrict communication within the tunnel. You can restrict the destinations reachable by Grafana Cloud over this tunnel by using the PermitRemoteOpen SSH option. This can be a combination of hosts and ports. The * wildcard character can be used for host or port to allow all hosts or all ports respectively.

For example, you can configure the PDC Agent to only allow traffic to 2 endpoints by passing the following command line argument:

Copy

-ssh-flag='-o PermitRemoteOpen=mysql.example.com:3306 prometheus.example.com:443'

By default, the agent generates a new SSH key-pair at startup. However, you can provide your own custom SSH key pair to the agent.

The Grafana PDC agent is open source, and the source code is located here. The agent is a small wrapper around OpenSSH, which ensures the correct SSH flags are set.

Grafana PDC supports mutual TLS certificates between the hosted Grafana instance and the PDC proxy. Certificates are managed internally.

Token storage and rotation

The PDC agent dynamically generates an SSH key pair used to encrypt the traffic between your private network and Grafana Cloud. At startup, the PDC agent uploads the public key to Grafana Cloud, which returns a short-lived (1 hour) SSH certificate for SSH authentication. Users can rotate the private key at any time by simply restarting the PDC agent process. How you restart the agent depends on your installation method.

Audit activity and the PDC agent

The PDC agent logs every connection attempt, whether it is successful, failed or denied. You can use these logs to audit traffic and ensure that no malicious or unwanted actors are trying to query your data.

The method you use to access and store the logs depends on where the agent is deployed, and your logging tool. See logs-config to configure logging.

The logs will be the standard debug logs from SSH, plus an introduction log when you’ve first connected to Grafana’s PDC Service.

When your PDC Agent successfully connects to Grafana Cloud, you will see the the following message in the logs:

Copy

This is Grafana Private Data Source Connect!

Successful logs when -log.level=debug is set contain lines like the following:

Copy

debug1: client_input_channel_open: ctype forwarded-tcpip rchan 1 win 2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 1234, originator ::1 port 61779
debug1: channel 1: new [::1]
debug1: confirm forwarded-tcpip
debug1: connect_next: host 34.205.150.168 ([34.205.150.168]:443) in progress, fd=10
debug1: channel 1: connected to 34.205.150.168 port 443
debug1: channel 1: free: ::1, nchannels 2

Failed connections may look like this:

Copy

debug1: client_input_channel_open: ctype forwarded-tcpip rchan 1 win 2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 1234, originator ::1 port 61917
debug1: channel 1: new [::1]
debug1: confirm forwarded-tcpip
debug1: rdynamic_connect_finish: requested forward not permitted
debug1: channel 1: free: ::1, nchannels 2

Changing the log verbosity

By default, pdc-agent does not print debug logs from OpenSSH. Change -log.level to debug to see OpenSSH debug messages.

Running with debug verbosity prints a lot of messages with SSH message numbers. The meanings of these message numbers are defined in RFC 4250.