--- title: "Set up private probes | Grafana Cloud documentation" description: "How to configure your internal synthetic monitoring agents (private probes) in Grafana Synthetic Monitoring" --- # Set up private probes Probes are blackbox agents responsible for executing the configured checks on specified targets, which are URLs or endpoints of web applications or services that you want to monitor. The probes simulate user interactions with these targets by sending requests and recording the responses. Each probe sends metrics and logs to the Synthetic Monitoring backend, including such information as target availability and health and response latencies. The Synthetic Monitoring dashboard then displays that information. In addition to the public probes run by Grafana Labs, you can also install private probes. These are only accessible to you and only write data to your Grafana Cloud account. Private probes are instances of the open source Grafana [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent). ## Before you begin > Note > > By default, private probes collect and report anonymous, non-personally identifiable information (PII) about the running probe to a remote statistics server operated by Grafana Labs. Synthetic Monitoring maintainers use that data to understand how the open source community operates private probes. > > To opt out, use the CLI flag `-disable-usage-reports=true`. Refer to [Anonymous usage statistics reporting](#anonymous-usage-statistics-reporting) for more information on what’s collected. - You must have the *Admin* role in Grafana to access the Synthetic Monitoring config page. - [Install and configure](/docs/grafana-cloud/testing/synthetic-monitoring/set-up/grafana-cloud/) Synthetic Monitoring before adding a private probe. - Installing from the Grafana APT repository on amd64-based Debian and Ubuntu systems is the only packaging option provided. On other systems, you need to build the agent from [source](https://www.github.com/grafana/synthetic-monitoring-agent). - You can install multiple private probes. Each private probe has its own unique **Probe Authentication Token**. ## Add a new probe in your Grafana instance 1. Navigate to **Testing & synthetics > Synthetics > Probes**. 2. Click **Add Private Probe**. 3. Enter a **Probe Name**. 4. Enter a **Latitude** and **Longitude** to see an accurate location on the map panel. 5. Enter a **Region**. Defaults are `AMER`, `EMEA`, and `APAC`, but you may define your own. 6. Optionally enter up to three custom labels to identify your probe. 7. Click **Save**. 8. Copy the **Probe Authentication Token** and save it for the next step. ## Probe API Server URL Based on the region of your stack, you need to use a different API server URL when setting up a private probe. If you are unsure about the region of your stack, go to your Grafana instance and the **Config** page under **Synthetic Monitoring** and search for `backend address`. > Note > > The config expects API server URLs without `https://`. If you receive an error, verify that you have omitted `https://`. Expand table | Region | Provider | API server URL | Backend address | Secrets proxy | |--------------------|----------|------------------------------------------------------------|-------------------------------------------------------|--------------------------------------------------------------------------------------| | Amsterdam | Azure | `synthetic-monitoring-grpc-eu-west-3.grafana.net:443` | `synthetic-monitoring-api-eu-west-3.grafana.net` | `gsm-proxy-prod-eu-west-3.grafana.net` | | Australia | GCP | `synthetic-monitoring-grpc-au-southeast.grafana.net:443` | `synthetic-monitoring-api-au-southeast.grafana.net` | `gsm-proxy-prod-au-southeast-0.grafana.net` | | Australia | AWS | `synthetic-monitoring-grpc-au-southeast-1.grafana.net:443` | `synthetic-monitoring-api-au-southeast-1.grafana.net` | `gsm-proxy-prod-au-southeast-1.grafana.net` | | Brazil | GCP | `synthetic-monitoring-grpc-sa-east-0.grafana.net:443` | `synthetic-monitoring-api-sa-east-0.grafana.net` | `gsm-proxy-prod-sa-east-0.grafana.net` | | Brazil | AWS | `synthetic-monitoring-grpc-sa-east-1.grafana.net:443` | `synthetic-monitoring-api-sa-east-1.grafana.net` | `gsm-proxy-prod-sa-east-1.grafana.net` | | Canada | AWS | `synthetic-monitoring-grpc-ca-east-0.grafana.net:443` | `synthetic-monitoring-api-ca-east-0.grafana.net` | `gsm-proxy-prod-ca-east-0.grafana.net` | | Europe | AWS | `synthetic-monitoring-grpc-eu-west-2.grafana.net:443` | `synthetic-monitoring-api-eu-west-2.grafana.net` | `gsm-proxy-prod-eu-west-2.grafana.net`, `gsm-proxy-prod-eu-west-4.grafana.net` | | Europe | AWS | `synthetic-monitoring-grpc-eu-west-6.grafana.net:443` | `synthetic-monitoring-api-eu-west-6.grafana.net` | `gsm-proxy-prod-eu-west-6.grafana.net` | | Europe | GCP | `synthetic-monitoring-grpc-eu-west.grafana.net:443` | `synthetic-monitoring-api-eu-west.grafana.net` | `gsm-proxy-prod-eu-west-0.grafana.net` | | India | GCP | `synthetic-monitoring-grpc-ap-south-0.grafana.net:443` | `synthetic-monitoring-api-ap-south-0.grafana.net` | `gsm-proxy-prod-ap-south-0.grafana.net` | | India | AWS | `synthetic-monitoring-grpc-ap-south-1.grafana.net:443` | `synthetic-monitoring-api-ap-south-1.grafana.net` | `gsm-proxy-prod-ap-south-1.grafana.net` | | Japan | AWS | `synthetic-monitoring-grpc-ap-northeast-0.grafana.net:443` | `synthetic-monitoring-api-ap-northeast-0.grafana.net` | `gsm-proxy-prod-ap-northeast-0.grafana.net` | | Saudi Arabia | GCP | `synthetic-monitoring-grpc-me-central-0.grafana.net:443` | `synthetic-monitoring-api-me-central-0.grafana.net` | `gsm-proxy-prod-me-central-0.grafana.net` | | Singapore | GCP | `synthetic-monitoring-grpc-ap-southeast-0.grafana.net:443` | `synthetic-monitoring-api-ap-southeast-0.grafana.net` | `gsm-proxy-prod-ap-southeast-0.grafana.net` | | Singapore | AWS | `synthetic-monitoring-grpc-ap-southeast-1.grafana.net:443` | `synthetic-monitoring-api-ap-southeast-1.grafana.net` | `gsm-proxy-prod-ap-southeast-1.grafana.net` | | Indonesia | AWS | `synthetic-monitoring-grpc-ap-southeast-2.grafana.net:443` | `synthetic-monitoring-api-ap-southeast-2.grafana.net` | `gsm-proxy-prod-ap-southeast-2.grafana.net` | | Sweden | AWS | `synthetic-monitoring-grpc-eu-north-0.grafana.net:443` | `synthetic-monitoring-api-eu-north-0.grafana.net` | `gsm-proxy-prod-eu-north-0.grafana.net` | | Switzerland | AWS | `synthetic-monitoring-grpc-eu-central-0.grafana.net:443` | `synthetic-monitoring-api-eu-central-0.grafana.net` | `gsm-proxy-prod-eu-central-0.grafana.net` | | UAE | AWS | `synthetic-monitoring-grpc-me-central-1.grafana.net:443` | `synthetic-monitoring-api-me-central-1.grafana.net` | `gsm-proxy-prod-me-central-1.grafana.net` | | UK | GCP | `synthetic-monitoring-grpc-gb-south.grafana.net:443` | `synthetic-monitoring-api-gb-south.grafana.net` | `gsm-proxy-prod-gb-south-0.grafana.net` | | UK | AWS | `synthetic-monitoring-grpc-gb-south-1.grafana.net:443` | `synthetic-monitoring-api-gb-south-1.grafana.net` | `gsm-proxy-prod-gb-south-1.grafana.net` | | United States East | AWS | `synthetic-monitoring-grpc-us-east-0.grafana.net:443` | `synthetic-monitoring-api-us-east-0.grafana.net` | `gsm-proxy-prod-us-east-0.grafana.net`, `gsm-proxy-prod-us-east-2.grafana.net` | | United States East | AWS | `synthetic-monitoring-grpc-us-east-3.grafana.net:443` | `synthetic-monitoring-api-us-east-3.grafana.net` | `gsm-proxy-prod-us-east-3.grafana.net` | | United States East | GCP | `synthetic-monitoring-grpc-us-east-1.grafana.net:443` | `synthetic-monitoring-api-us-east-1.grafana.net` | `gsm-proxy-prod-us-east-1.grafana.net` | | United States West | AWS | `synthetic-monitoring-grpc-us-west-0.grafana.net:443` | `synthetic-monitoring-api-us-west-0.grafana.net` | `gsm-proxy-prod-us-west-0.grafana.net` | | United States | Azure | `synthetic-monitoring-grpc-us-central-7.grafana.net:443` | `synthetic-monitoring-api-us-central-7.grafana.net` | `gsm-proxy-prod-us-central-7.grafana.net` | | United States | GCP | `synthetic-monitoring-grpc.grafana.net:443` | `synthetic-monitoring-api.grafana.net` | `gsm-proxy-prod-us-central-0.grafana.net`, `gsm-proxy-prod-us-central-4.grafana.net` | ## Firewall configuration To ensure successful communication between your private Synthetic Monitoring probes and the necessary services, adjust your firewall rules as follows: ### Synthetic Monitoring services - Allow outbound connections from the probe to the “API server URL” specified in the preceding table. - Allow outbound connections from the probe to the “Secrets proxy” specified in the preceding table. ### Prometheus and Loki instances Allow outbound connections from the Prometheus server URL and the Loki URL: 1. Log in to Grafana Cloud and open your stack. 2. Navigate to **Home** > **Testing & synthetics** > **Synthetics** > [**Config**](/launch/a/grafana-synthetic-monitoring-app/config). 3. Under the **General** tab, scroll down to **Data sources**. 4. For the Prometheus server URL: 1. Click the Prometheus data source. For example: `grafanacloud-STACK_NAME-prom`. 2. Copy the **Prometheus server URL** under the **Settings** tab -> **Connection**. 5. For the Loki URL: 1. Click the Loki data source. For example: `grafanacloud-STACK_NAME-logs`. 2. Copy the **URL** under the **Settings** tab -> **Connection**. 6. Update your firewall rules with both URLs. ### External script imports - If using k6 scripts that import external modules - for example k6 modules from `jslib.k6.io` - allow outbound connections to the module hosts. > Note > > MultiHTTP test executions make outbound connections to `jslib.k6.io` when run. If you run MultiHTTP checks on private probes behind a firewall, you must allow connections to `jslib.k6.io`. ### Monitored targets Consider the check type: the firewall configuration varies based on the specific checks you’ve configured. - ICMP: for ping and traceroute checks. - UDP or TCP: for DNS checks. - TCP: for GRPC, HTTP, and MultiHTTP checks. Allow inbound connections to the targets you want to monitor, ensuring compatibility with the required protocols. ## Install the agent on Ubuntu/Debian You can install the agent using the Grafana official APT repository. 1. Add the required packages and the repository GPG key. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo apt install -y wget gnupg sudo mkdir -p /etc/apt/keyrings/ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg ``` 2. Add the repository for stable releases. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list ``` 3. Run the following commands. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo apt update sudo apt install synthetic-monitoring-agent ``` 4. Configure the probe. 1. Open `/etc/synthetic-monitoring/synthetic-monitoring-agent.conf`. 2. Replace `` with the probe authentication token. 3. Repleace `API_SERVER` with Probe API Server URL based on your stack region. (defaults to US region) 4. Save the file. 5. Restart the agent. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo service synthetic-monitoring-agent restart ``` The new probe appears on your **Synthetics > Probes** page. ## Install the agent on distributions using yum You can install the agent using the official YUM repository. 1. Add the repository and the GPG key to the configuration. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo yum-config-manager --add-repo https://rpm.grafana.com wget https://rpm.grafana.com/gpg.key sudo rpm --import gpg.key ``` 2. Install the agent. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo yum install synthetic-monitoring-agent sudo systemctl status synthetic-monitoring-agent.service ``` This last command will tell you that the service is loaded but disabled. Before enabling, it you need to [add the probe to your Synthetic Monitoring configuration](#add-a-new-probe-in-your-grafana-instance). 3. Configure the agent. Edit the file `/etc/synthetic-monitoring/synthetic-monitoring-agent.conf` and make the following changes: - Replace `` with the probe authentication token. - Replace `API_SERVER` with the API Server URL corresponding to your stack’s region. Defaults to the US region. 4. Enable the service and start it. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo systemctl enable synthetic-monitoring-agent.service sudo systemctl start synthetic-monitoring-agent.service ``` ## Deploy the agent using Docker You can find the [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent) Docker image on [Docker Hub](https://hub.docker.com/r/grafana/synthetic-monitoring-agent) 1. Pull image from Docker Hub. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash docker pull grafana/synthetic-monitoring-agent:latest ``` 2. Export required configuration as environment variables. > Replace `` with the probe authentication token. Replace `` with Probe API Server URL based on your stack region. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash export API_TOKEN= export API_SERVER="" ``` 3. Start the agent. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash docker run grafana/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --verbose=true ``` The new probe appears on your **Synthetics > Probes** page. ## Deployment with Kubernetes You can deploy the [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent) to Kubernetes using the Synthetic Monitoring Agent Docker image. In the agent GitHub repository, you can also find a [Kubernetes Deployment example](https://github.com/grafana/synthetic-monitoring-agent/tree/main/examples/kubernetes). The following section uses the example files to deploy a Synthetic Monitoring agent in a Kubernetes namespace. 1. Get the deployment files. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/namespace.yaml wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/secret.yaml wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/deployment.yaml ``` 2. Create the `synthetic-monitoring` namespace. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f namespace.yaml ``` 3. Create a secret with the Probe Authentication Token. > Replace `YOUR_TOKEN_HERE` in `secret.yaml` with your Probe Authentication Token. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f secret.yaml # verify secret kubectl get secrets -n synthetic-monitoring ``` 4. Deploy the Synthetic Monitoring Agent. > Replace `PROBE_API_SERVER_URL` in `deployment.yaml` with Probe API Server URL based on your stack region. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f deployment.yaml ``` The new probe appears on your **Synthetics > Probes** page. ## Calculate resource requirements Resources used by private probes may vary based on the targets being monitored, so it can be hard to predict them in advance. Below, we provide a rough estimate to get you started. The baseline CPU and memory resources, if you want to run: - API endpoint checks: `50m`/`64Mi`. - The above, plus multi-step and/or scripted checks: `200m`/`256Mi`. - The above, plus browser checks: `1000m`/`1024Mi`. Then, for each check per minute of the following types, add: - API endpoint (HTTP, Ping, DNS, TCP, Traceroute): `5m`/`2Mi`. - Multi-step and/or scripted: `20m`/`32Mi` - Browser checks: `100m`/`256Mi`. As mentioned above, keep in mind that these are ballpark estimates. Monitor the resource usage of your probe, and stay on the lookout for CPU throttling and OOM events to see if you need to adjust any resources. ## Configure feature flags You can use feature flags to enable features in the private probe. There are two ways to enable features: - Edit the file `/etc/synthetic-monitoring/synthetic-monitoring-agent.conf` and make the following changes: ini ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```ini FEATURES = ``` - Export the list of features as an environment variable. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash export FEATURES= ``` ### Available feature flags These features are disabled by default. Expand table | Feature flag name | Description | |-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `adhoc` | Click the **Test** button in the check editor UI to run checks out of schedule. If this isn’t enabled, the test result is a timeout. | | `k6` | Run k6-based checks. | | `traceroute` | Run `traceroute` checks, in addition to `ping` checks. Depending on your environment, you might need to grant specific permission to the probe process or container. | ## Run k6 checks targeting private IP addresses By default, private probes block k6 request targets in the 10.0.0.0/8 CIDR range. You can override this behavior by setting the CLI flag `--blocked-nets=""`. - When running Docker, append `--blocked-nets=""` to your `docker run` command. - On Linux installations, run `systemctl edit synthetic-monitoring-agent` and append `--blocked-nets=""` to `ExecStart=`. ## Running k6 browser checks on a private probe To be able to run [k6 browser checks](/docs/grafana-cloud/testing/synthetic-monitoring/create-checks/checks/k6-browser/) on private probes, you will need to have a Chromium-compatible web browser available on the system where the probe is running. Additionally, you need to make sure that in the probe settings page in your Grafana stack (Testing & synthetics > Synthetics > Probes), both “Disable scripted checks” and “Disable browser checks” are *unchecked*. ### k6 browser checks on Ubuntu, Debian, and RPM-based distributions If you installed the [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent) on you distribution via your package manager, your distribution needs to have Chromium installed, and a `chromium` binary available in `$PATH`. Alternatively, if you do not want to make `chromium` widely available, or you wish to use a specific installation, you can add set `K6_BROWSER_EXECUTABLE_PATH` in the synthetic monitoring agent’s environment to the full path of a `chromium` binary. For advanced use cases, you can also define `K6_BROWSER_ARGS` with additional arguments that the agent will use when launching Chromium. ### k6 browser checks on Docker, Kubernetes, and other container-based environments For container-based environments, we provide a special image that already includes an installation of Chromium. These images are tagged with the `-browser` suffix, for example `grafana/synthetic-monitoring-agent:vX.Y.Z-browser`, as opposed to `grafana/synthetic-monitoring-agent:vX.Y.Z`. Agent images tagged *with* the `-browser` suffix can run all types of checks, while those tagged *without* the `-browser` suffix can run every check type except k6 browser checks. If you do not plan to use browser checks, we recommend using the standard image, as it is lighter and more minimal. Swapping to a `-browser` can be done at any time without any manual intervention, other than changing the tag. ## Anonymous usage statistics reporting > Note > > The anonymous usage statistics is *enabled by default*. You can opt out by setting the CLI flag `-disable-usage-reports=true`. ### The statistics server When usage statistics reporting is enabled, information is collect by a server that Grafana Labs operates. Statistics are sent to `https://stats.grafana.org`. ### Which information is collected When the usage statistics reporting is enabled, probes collects the following information: - Information about the **probe**, including: - A unique hash representation of the probe - The probe version, such as `0.39.0` - The installation source, such as `docker` - The probe id, such as `1100` - The tenant id, such as `1100` - The features enabled, such as `k6` - Information about the **environment**, including: - The operating system, such as `linux` - The architecture, such as `amd64` - The os details, such as `Ubuntu 24.04.2 LTS` ## Troubleshooting private probes If you are facing problems with your private probes, check the following: 1. Check the probe version using the `-version` flag and make sure you are running a supported version. 2. Make sure the probe has a valid probe access token. 3. Make sure the probe can connect to the Synthetic Monitoring API server. 4. Run the Synthetic Monitoring Agent in debug mode with the `-debug` and `-verbose` flags and check for errors in the logs. 5. Check the Synthetic Monitoring Agent metrics. The default address for Synthetic Monitoring Agent Prometheus metrics is `localhost:4050/metrics`. If you are still facing issues with Private Probes, ask for help in the [community forum](https://community.grafana.com/). Refer to [Account types and support available for Grafana Cloud users](/docs/grafana-cloud/account-management/support/#account-types-and-support-available-for-grafana-cloud-users) for more information. ## Production deployments The agent process exposes Prometheus-style metrics on an HTTP server running on port 4050 with the “/metrics” endpoint. You can scrape and monitor your private probe using Prometheus or [Grafana Alloy](https://github.com/grafana/alloy). ## Next steps You must reconfigure any [checks](/docs/grafana-cloud/testing/synthetic-monitoring/create-checks/checks/) to use your new probe even if you selected all probes when initially creating the check. The Synthetic Monitoring Agent is open source and [available on GitHub](https://www.github.com/grafana/synthetic-monitoring-agent).