PDC scalability and security
PDC is scalable and secure. The following sections go into more detail regarding both scalability and security.
Private data source connect (PDC) scalability
The PDC agent scales horizontally. In order to increase throughput, you can increase the number of replicas. Connections will be load balanced across the various replicas by the PDC service. The PDC service in Grafana Cloud is also a horizontally scalable distributed system. SSH connections from your network are spread across many instances to ensure high availability.
You do not need to set up a separate agent for each data source. A single agent connects a hosted Grafana instance to multiple data sources as long they are in the same network.
Using SSH allows multiplexing multiple SOCKS connections over a single SSH tunnel as long as the data sources all exist within the same network.
PDC allows for seamless failover. If you deploy three agents, each in two data centers, PDC will load balance across all six (3 x 2) replicas regardless of data center. If a connection is lost with one data center, the PDC service on Grafana Cloud will simply fail over to another replica.
Private data source connect (PDC) security
SSH tunnels are widely accepted by the security community to be secure. The existence of a tunnel does not increase the risk of the PDC proxy being compromised. You control the SSH tunnel initiated by the PDC agent. The tunnel can be shut down or killed in the event there is any suspicion of compromise.
Grafana Labs conducted an external penetration test and found no vulnerabilities. The full penetration test report can be found here.
Both Grafana Labs and you, the end user, are able to restrict communication within the tunnel. You can restrict the destinations reachable by Grafana Cloud over this tunnel by using the
PermitRemoteOpen SSH option. This can be a combination of hosts and ports. The
* wildcard character can be used for host or port to allow all hosts or all ports respectively.
For example, you can configure the PDC Agent to only allow traffic to 2 endpoints by passing the following command line argument:
-ssh-flag='-o PermitRemoteOpen=mysql.example.com:3306 prometheus.example.com:443'
By default, the agent generates a new SSH key-pair at startup. However, you can provide your own custom SSH key pair to the agent.
The Grafana PDC agent is open source, and the source code is located here. The agent is a small wrapper around OpenSSH, which ensures the correct SSH flags are set.
Grafana PDC supports mutual TLS certificates between the hosted Grafana instance and the PDC proxy. Certificates are managed internally.
Token storage and rotation
The PDC agent dynamically generates an SSH key pair used to encrypt the traffic between your private network and Grafana Cloud. At startup, the PDC agent uploads the public key to Grafana Cloud, which returns a short-lived (1 hour) SSH certificate for SSH authentication. Users can rotate the private key at any time by simply restarting the PDC agent process. How you restart the agent depends on your installation method.
Audit activity and the PDC agent
The PDC agent logs every connection attempt, whether it is successful, failed or denied. You can use these logs to audit traffic and ensure that no malicious or unwanted actors are trying to query your data.
The method you use to access and store the logs depends on where the agent is deployed, and your logging tool. See logs-config to configure logging.
The logs will be the standard debug logs from SSH, plus an introduction log when you’ve first connected to Grafana’s PDC Service.
When your PDC Agent successfully connects to Grafana Cloud, you will see the the following message in the logs:
This is Grafana Private Data Source Connect!
Successful logs will look like this:
debug1: client_input_channel_open: ctype forwarded-tcpip rchan 1 win 2097152 max 32768 debug1: client_request_forwarded_tcpip: listen localhost port 1234, originator ::1 port 61779 debug1: channel 1: new [::1] debug1: confirm forwarded-tcpip debug1: connect_next: host 220.127.116.11 ([18.104.22.168]:443) in progress, fd=10 debug1: channel 1: connected to 22.214.171.124 port 443 debug1: channel 1: free: ::1, nchannels 2
Failed connections may look like this:
debug1: client_input_channel_open: ctype forwarded-tcpip rchan 1 win 2097152 max 32768 debug1: client_request_forwarded_tcpip: listen localhost port 1234, originator ::1 port 61917 debug1: channel 1: new [::1] debug1: confirm forwarded-tcpip debug1: rdynamic_connect_finish: requested forward not permitted debug1: channel 1: free: ::1, nchannels 2
Note: There are different reasons that may be logged for the failed reason.
Changing the log verbosity
By default, the agent will run using a log verbosity of
2, meaning you will see
You can modify this through the flag
--log-level, which can be set as a value between
0 to disable debug logs, and
3 to enable all debug logs.