Private probes
In addition to the public probes run by Grafana Labs, you can choose to install your own private probes. These are only accessible to you and only write data to your Grafana Cloud account. Private probes are instances of the open source Grafana Synthetic Monitoring Agent.
Make sure you have installed and configured Synthetic Monitoring before adding a private probe.
You can install multiple private probes. If you do, create a unique API key for each private probe.
Note: Installing from our APT repository on amd64-based Debian and Ubuntu systems is the only packaging option provided. The agent will need to be built from source on other systems.
Add a new probe in your Grafana instance
- Navigate to Monitoring > Synthetics > Probes.
- Click New.
- Enter a Probe Name.
- Enter a Latitude and Longitude to see an accurate location on the map panel.
- Enter a Region. Defaults are
AMER
,EMEA
, andAPAC
, but you may define your own. - Optionally enter up to three custom labels to identify your probe.
- Click Save.
- Copy the Probe Authentication Token and save it for next step.
Probe API Server URL
Based on the region of your stack, you need to use a different API server URL when setting up a private probe. If you are unsure about the region of your stack, go to your Grafana instance and visit the Config page under “Synthetic Monitoring” and search for “backend address”.
Region | API server URL | backend address |
---|---|---|
Amsterdam (Azure) | synthetic-monitoring-grpc-eu-west-3.grafana.net:443 | https://synthetic-monitoring-api-eu-west-3.grafana.net |
Australia | synthetic-monitoring-grpc-au-southeast.grafana.net:443 | https://synthetic-monitoring-api-au-southeast.grafana.net |
Brazil | synthetic-monitoring-grpc-sa-east-0.grafana.net:443 | https://synthetic-monitoring-api-sa-east-0.grafana.net |
Europe (AWS) | synthetic-monitoring-grpc-eu-west-2.grafana.net:443 | https://synthetic-monitoring-api-eu-west-2.grafana.net |
Europe (Azure) | synthetic-monitoring-grpc-eu-west-1.grafana.net:443 | https://synthetic-monitoring-api-eu-west-1.grafana.net |
Europe | synthetic-monitoring-grpc-eu-west.grafana.net:443 | https://synthetic-monitoring-api-eu-west.grafana.net |
India | synthetic-monitoring-grpc-ap-south-0.grafana.net:443 | https://synthetic-monitoring-api-ap-south-0.grafana.net |
Singapore | synthetic-monitoring-grpc-ap-southeast-0.grafana.net:443 | https://synthetic-monitoring-api-ap-southeast-0.grafana.net |
UK | synthetic-monitoring-grpc-gb-south.grafana.net:443 | https://synthetic-monitoring-api-gb-south.grafana.net |
United States (AWS) | synthetic-monitoring-grpc-us-east-0.grafana.net:443 | https://synthetic-monitoring-api-us-east-0.grafana.net |
United States (Azure) | synthetic-monitoring-grpc-us-central2.grafana.net:443 | https://synthetic-monitoring-api-us-central2.grafana.net |
United States | synthetic-monitoring-grpc.grafana.net:443 | https://synthetic-monitoring-api.grafana.net |
NOTE: Your user needs to have the Admin role in Grafana to access the Synthetic Monitoring config page
Install the agent on Ubuntu/Debian
You can install the agent using our official APT repository.
Add the required packages, and the repository GPG key:
sudo apt install -y wget gnupg sudo mkdir -p /etc/apt/keyrings/ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg
Add the repository for stable releases.
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
Run the following commands:
sudo apt update sudo apt install synthetic-monitoring-agent
Configure the probe.
- Open
/etc/synthetic-monitoring/synthetic-monitoring-agent.conf
. - Replace
<YOUR TOKEN HERE>
with the probe authentication token. - Repleace
API_SERVER
with Probe API Server URL based on your stack region. (defaults to US region) - Save the file.
- Open
Restart the agent:
sudo service synthetic-monitoring-agent restart
The new probe appears on your Synthetic Monitoring > Probes page.
Install the agent on distributions using yum
You can install the agent using the official YUM repository.
Add the repository and the GPG key to the configuration
sudo yum-config-manager --add-repo https://rpm.grafana.com wget https://rpm.grafana.com/gpg.key sudo rpm --import gpg.key
Install the agent
sudo yum install synthetic-monitoring-agent sudo systemctl status synthetic-monitoring-agent.service
This last command is going to tell you that the service is loaded but disabled. Before enabling it you need to add the probe in your Synthetic Monitoring configuration.
Configure the agent
Edit the file
/etc/synthetic-monitoring/synthetic-monitoring-agent.conf
and make the following changes:- Replace
<YOUR TOKEN HERE>
with the probe authentication token. - Repleace
API_SERVER
with API Server URL corresponding to your stack’s region (defaults to US region).
- Replace
Enable the service and start it
sudo systemctl enable synthetic-monitoring-agent.service sudo systemctl start synthetic-monitoring-agent.service
Deploy the agent using Docker
You can find the Synthetic Monitoring Agent docker image on Docker Hub
Pull image from Docker Hub.
docker pull grafana/synthetic-monitoring-agent:latest
Export required config as environment variables.
Replace
<YOUR TOKEN>
with the probe authentication token. Replace<PROBE_API_SERVER_URL>
with Probe API Server URL based on your stack region.export API_TOKEN=<YOUR_TOKEN> export API_SERVER="<PROBE_API_SERVER_URL>"
Start the agent.
docker run grafana/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --verbose=true
The new probe appears on your Synthetic Monitoring > Probes page.
Deployment with Kubernetes
Synthetic Monitoring Agent can be deployed to Kubernetes using Synthetic Monitoring Agent Docker image
We maintain a Kubernetes deployment example in agent repo.
We will be using these files to deploy Synthetic Monitoring Agent in a Kubernetes namespace.
Get deployment files.
wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/master/examples/kubernetes/namespace.yaml wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/master/examples/kubernetes/secret.yaml wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/master/examples/kubernetes/deployment.yaml
Create the
synthetic-monitoring
namespace.kubectl apply -f namespace.yaml
Create a secret with Probe Authentication Token.
Replace
<YOUR_TOKEN>
insecret.yaml
with your Probe Authentication Token. Replace<PROBE_API_SERVER_URL>
insecret.yaml
with Probe API Server URL based on your stack region.kubectl apply -f secret.yaml # verify secret kubectl get secrets -n synthetic-monitoring
Deploy the Synthetic Monitoring Agent.
kubectl apply -f deployment.yaml
The new probe appears on your Synthetic Monitoring > Probes page.
Troubleshooting Private Probes
If you are facing problems with your private probes, check the following:
- Check the probe version (use
-version
flag) and make sure you are running a supported version. - Make sure the probe has a valid probe access token.
- Make sure the probe is able to connect to the Synthetic Monitoring API server.
- Run the Synthetic Monitoring Agent in debug mode (use the
-debug
and-verbose
flags), and check for errors in the logs. - Check the Synthetic Monitoring Agent metrics. The default address for Synthetic Monitoring Agent Prometheus metrics is
localhost:4050/metrics
If you are still facing issues with Private Probes, ask for help in the community forum. See Account types and support available for Grafana Cloud users for more information.
Production Deployments
The Agent process exposes Prometheus-style metrics on an HTTP server running on port 4050 (/metrics endpoint).
You can scrape and monitor your private probe in this way using either Prometheus or the Grafana Agent.
Next steps
You must reconfigure any existing checks to use your new probe even if you selected all probes when initially creating the check.
Synthetic Monitoring Agent is open-source, and avaliable on GitHub.