Menu
Grafana Cloud

Private probes

Probes are blackbox agents responsible for carrying out the configured checks on specified targets, which are URLs or endpoints of web applications or services that you want to monitor. The probes simulate user interactions with these targets by sending requests and recording the responses.

Each probe sends metrics and logs to the Synthetic Monitoring backend, including such information as target availability and health and response latencies, which are then rendered in the Synthetic Monitoring dashboard.

In addition to the public probes run by Grafana Labs, you can choose to install your own private probes. These are only accessible to you and only write data to your Grafana Cloud account. Private probes are instances of the open source Grafana Synthetic Monitoring Agent.

Before you begin

  • User must have the Admin role in Grafana to access the Synthetic Monitoring config page.
  • Be sure to install and configure Synthetic Monitoring before adding a private probe.
  • Installing from our APT repository on amd64-based Debian and Ubuntu systems is the only packaging option provided. The agent will need to be built from source on other systems.
  • You can install multiple private probes. If you do, create a unique API key for each private probe.

Add a new probe in your Grafana instance

  1. Navigate to Observability > Synthetics > Probes.
  2. Click New.
  3. Enter a Probe Name.
  4. Enter a Latitude and Longitude to see an accurate location on the map panel.
  5. Enter a Region. Defaults are AMER, EMEA, and APAC, but you may define your own.
  6. Optionally enter up to three custom labels to identify your probe.
  7. Click Save.
  8. Copy the Probe Authentication Token and save it for next step.

Probe API Server URL

Based on the region of your stack, you need to use a different API server URL when setting up a private probe. If you are unsure about the region of your stack, go to your Grafana instance and visit the Config page under Synthetic Monitoring and search for backend address.

Note

The config expects API server URls without https://. If you receive an error, verify that you haved omitted https://.
RegionAPI server URLbackend address
Amsterdam (Azure)synthetic-monitoring-grpc-eu-west-3.grafana.net:443synthetic-monitoring-api-eu-west-3.grafana.net
Australiasynthetic-monitoring-grpc-au-southeast.grafana.net:443synthetic-monitoring-api-au-southeast.grafana.net
Brazilsynthetic-monitoring-grpc-sa-east-0.grafana.net:443synthetic-monitoring-api-sa-east-0.grafana.net
Canada (AWS)synthetic-monitoring-grpc-ca-east-0.grafana.net:443synthetic-monitoring-api-ca-east-0.grafana.net
Europe (AWS)synthetic-monitoring-grpc-eu-west-2.grafana.net:443synthetic-monitoring-api-eu-west-2.grafana.net
Europe (Azure)synthetic-monitoring-grpc-eu-west-1.grafana.net:443synthetic-monitoring-api-eu-west-1.grafana.net
Europesynthetic-monitoring-grpc-eu-west.grafana.net:443synthetic-monitoring-api-eu-west.grafana.net
Indiasynthetic-monitoring-grpc-ap-south-0.grafana.net:443synthetic-monitoring-api-ap-south-0.grafana.net
Singaporesynthetic-monitoring-grpc-ap-southeast-0.grafana.net:443synthetic-monitoring-api-ap-southeast-0.grafana.net
Sweden (AWS)synthetic-monitoring-grpc-eu-north-0.grafana.net:443synthetic-monitoring-api-eu-north-0.grafana.net
UKsynthetic-monitoring-grpc-gb-south.grafana.net:443synthetic-monitoring-api-gb-south.grafana.net
United States East (AWS)synthetic-monitoring-grpc-us-east-0.grafana.net:443synthetic-monitoring-api-us-east-0.grafana.net
United States West (AWS)synthetic-monitoring-grpc-us-west-0.grafana.net:443synthetic-monitoring-api-us-west-0.grafana.net
United States (Azure)synthetic-monitoring-grpc-us-central2.grafana.net:443synthetic-monitoring-api-us-central2.grafana.net
United Statessynthetic-monitoring-grpc.grafana.net:443synthetic-monitoring-api.grafana.net

Install the agent on Ubuntu/Debian

You can install the agent using our official APT repository.

  1. Add the required packages, and the repository GPG key:

    bash
    sudo apt install -y wget gnupg
    sudo mkdir -p /etc/apt/keyrings/
    wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg
  2. Add the repository for stable releases.

    bash
    echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
  3. Run the following commands:

    bash
    sudo apt update
    sudo apt install synthetic-monitoring-agent
  4. Configure the probe.

    1. Open /etc/synthetic-monitoring/synthetic-monitoring-agent.conf.
    2. Replace <YOUR TOKEN HERE> with the probe authentication token.
    3. Repleace API_SERVER with Probe API Server URL based on your stack region. (defaults to US region)
    4. Save the file.
  5. Restart the agent:

    bash
    sudo service synthetic-monitoring-agent restart

The new probe appears on your Synthetic Monitoring > Probes page.

Install the agent on distributions using yum

You can install the agent using the official YUM repository.

  1. Add the repository and the GPG key to the configuration

    bash
    sudo yum-config-manager --add-repo https://rpm.grafana.com
    wget https://rpm.grafana.com/gpg.key
    sudo rpm --import gpg.key
  2. Install the agent

    bash
    sudo yum install synthetic-monitoring-agent
    sudo systemctl status synthetic-monitoring-agent.service

    This last command is going to tell you that the service is loaded but disabled. Before enabling it you need to add the probe in your Synthetic Monitoring configuration.

  3. Configure the agent

    Edit the file /etc/synthetic-monitoring/synthetic-monitoring-agent.conf and make the following changes:

    • Replace <YOUR TOKEN HERE> with the probe authentication token.
    • Repleace API_SERVER with API Server URL corresponding to your stack’s region (defaults to US region).
  4. Enable the service and start it

    bash
    sudo systemctl enable synthetic-monitoring-agent.service
    sudo systemctl start synthetic-monitoring-agent.service

Deploy the agent using Docker

You can find the Synthetic Monitoring Agent docker image on Docker Hub

  1. Pull image from Docker Hub.

    bash
    docker pull grafana/synthetic-monitoring-agent:latest
  2. Export required config as environment variables.

    Replace <YOUR TOKEN> with the probe authentication token. Replace <PROBE_API_SERVER_URL> with Probe API Server URL based on your stack region.

    bash
    export API_TOKEN=<YOUR_TOKEN>
    export API_SERVER="<PROBE_API_SERVER_URL>"
  3. Start the agent.

    bash
    docker run grafana/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --verbose=true

The new probe appears on your Synthetic Monitoring > Probes page.

Deployment with Kubernetes

Synthetic Monitoring Agent can be deployed to Kubernetes using Synthetic Monitoring Agent Docker image

We maintain a Kubernetes deployment example in agent repo.

We will be using these files to deploy Synthetic Monitoring Agent in a Kubernetes namespace.

  1. Get deployment files.

    bash
    wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/namespace.yaml
    wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/secret.yaml
    wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/deployment.yaml
  2. Create the synthetic-monitoring namespace.

    bash
    kubectl apply -f namespace.yaml
  3. Create a secret with Probe Authentication Token.

    Replace <YOUR_TOKEN> in secret.yaml with your Probe Authentication Token. Replace <PROBE_API_SERVER_URL> in secret.yaml with Probe API Server URL based on your stack region.

    bash
    kubectl apply -f secret.yaml
    
    # verify secret
    kubectl get secrets -n synthetic-monitoring
  4. Deploy the Synthetic Monitoring Agent.

    bash
    kubectl apply -f deployment.yaml

The new probe appears on your Synthetic Monitoring > Probes page.

Configure feature flags

You can use feature flags to enable features in the private probe.

There are two ways to enable features:

  • Edit the file /etc/synthetic-monitoring/synthetic-monitoring-agent.conf and make the following changes:

    ini
    FEATURES = <list of features comma separated>
  • Export the list of features as an environment variable.

    bash
    export FEATURES=<list of features comma separated>

Available feature flags

These features are disabled by default.

Feature flag nameDescription
adhocRun checks out of schedule, which is done when you click the Test button in the check editor UI. If this isn’t enabled the result of the test is a timeout
k6Run k6-based checks.
tracerouteRun traceroute checks, in addition to the already enabled ping checks. Depending on your environment you might need to grant specific permission to the probe process or container.

Troubleshooting Private Probes

If you are facing problems with your private probes, check the following:

  1. Check the probe version (use -version flag) and make sure you are running a supported version.
  2. Make sure the probe has a valid probe access token.
  3. Make sure the probe is able to connect to the Synthetic Monitoring API server.
  4. Run the Synthetic Monitoring Agent in debug mode (use the -debug and -verbose flags), and check for errors in the logs.
  5. Check the Synthetic Monitoring Agent metrics. The default address for Synthetic Monitoring Agent Prometheus metrics is localhost:4050/metrics

If you are still facing issues with Private Probes, ask for help in the community forum. See Account types and support available for Grafana Cloud users for more information.

Production Deployments

The Agent process exposes Prometheus-style metrics on an HTTP server running on port 4050 (/metrics endpoint).

You can scrape and monitor your private probe in this way using either Prometheus or the Grafana Agent.

Next steps

You must reconfigure any existing checks to use your new probe even if you selected all probes when initially creating the check.

Synthetic Monitoring Agent is open-source, and avaliable on GitHub.