Private probes

Besides the public probes run by Grafana Labs, you can also install your own private probes. These are only accessible to you and only write data to your Grafana Cloud account. Private probes are instances of the open source Grafana Synthetic Monitoring Agent.

Make sure you have installed and configured Synthetic Monitoring before adding a private probe.

You can install multiple private probes. If you do, create a unique API key for each private probe.

Note: Installing from our APT repository on amd64-based Debian and Ubuntu systems is the only packaging option provided. The agent will need to be built from source on other systems.

Add a new probe in your Grafana instance

1. Navigate to Synthetic Monitoring > Probes.
2. Click New.
3. Enter a Probe Name.
4. Enter a Latitude and Longitude to see an accurate location on the map panel.
5. Enter a Region. Defaults are AMER, EMEA, and APAC, but you may define your own.
6. Optionally enter up to three custom labels to identify your probe.
7. Click Save.
8. Copy the Probe Authentication Token and save it for next step.

Install the agent on Ubuntu/Debian

You can install the agent using our official APT repository.

1. Add the required packages, and the repository GPG key:

   sudo apt install -y apt-transport-https
   sudo apt install -y software-properties-common wget
   wget -q -O - https://packages-sm.grafana.com/gpg.key | sudo apt-key add -
   

2. Add the repository for stable releases:

   echo "deb https://packages-sm.grafana.com/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
   

3. After adding the repository:

   sudo apt update
   sudo apt install synthetic-monitoring-agent
   

4. Configure the probe:

   1. Open /etc/synthetic-monitoring/synthetic-monitoring-agent.conf.
   2. Replace YOUR TOKEN HERE with the probe authentication token.
   3. Save the file.

5. Restart the agent:

   sudo service synthetic-monitoring-agent restart
   

The new probe appears on your Synthetic Monitoring > Probes page.

Deploy the agent using Docker

We publish Synthetic Monitoring Agent docker image on Docker Hub

You can use that and run agent as a docker container.

1. Pull image from Docker Hub.

   docker pull grafana/synthetic-monitoring-agent:latest
   

2. Export required config as environment variables.

   Replace YOUR TOKEN HERE with the probe authentication token.

   export API_TOKEN=YOUR_TOKEN_HERE
   export API_SERVER="synthetic-monitoring-grpc.grafana.net:443"
   

3. Start the agent.

   docker run grafana/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --verbose=true
   

The new probe appears on your Synthetic Monitoring > Probes page.

Deploy it using Kubernetes

Synthetic Monitoring Agent can be deployed to Kubernetes using Synthetic Monitoring Agent Docker image

We maintain a Kubernetes deployment example in agent repo.

We will be using these files to deploy Synthetic Monitoring Agent in a Kubernetes namespace.

1. Get deployment files.

   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/master/examples/kubernetes/namespace.yaml
   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/master/examples/kubernetes/secret.yaml
   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/master/examples/kubernetes/deployment.yaml
   

2. Create the synthetic-monitoring namespace.

   kubectl apply -f namespace.yaml
   

3. Create a secret with Probe Authentication Token.

   Replace YOUR_TOKEN_HERE in secret.yaml with your Probe Authentication Token.

   kubectl apply -f secret.yaml
   
   # verify secret
   kubectl get secrets -n synthetic-monitoring
   

4. Deploy the Synthetic Monitoring Agent.

   kubectl apply -f deployment.yaml
   

The new probe appears on your Synthetic Monitoring > Probes page.

Production Deployments

The agent process exposes Prometheus-style metrics on an HTTP server running on port 4050 (/metrics endpoint).

You can scrape and monitor your private probe in this way using either Prometheus or the Grafana Cloud Agent.

Next steps

You will need to reconfigure any existing checks to use your new probe, even if you selected all probes when initially creating the check.