Remote configuration on Grafana Labs

Scaling with remote configuration

Thu, 09 Apr 2026 06:03:04 +0000

What you learned in Module 2

How Fleet Management works — collectors register, poll for config, send health data
Access tokens authenticate collectors with Fleet Management
Attributes organize your fleet using key-value pairs
Health monitoring provides centralized visibility into fleet health

In this module, you’ll learn to configure collectors remotely at scale:

The scaling challenge — Why manual configuration doesn’t work
Configuration pipelines — Creating and managing remote configs
Attribute matching — Targeting pipelines to specific collectors
Configuration strategies — Balancing local and remote config
Safety safeguards — Experimenting with confidence
Hands-on practice — Categorize your fleet with remote attributes

Estimated time

20-25 minutes

The scaling challenge

Thu, 09 Apr 2026 06:03:04 +0000

The scaling challenge

As your fleet grows, manual configuration doesn’t scale.

The math

Approach	5 collectors	50 collectors	500 collectors
Manual	5 updates	50 updates	500 updates
Remote config	1 update	1 update	1 update

Configure once, deploy everywhere. One update reaches your entire fleet.

Configuration pipelines

Thu, 09 Apr 2026 06:03:04 +0000

Configuration pipelines

A configuration pipeline is the unit of remote configuration in Fleet Management. It’s configuration to perform one operation, such as monitoring MySQL. You can match multiple pipelines to a single collector and multiple collectors to a single pipeline.

Each pipeline has a name, configuration content, and matching attributes.

Key terms

Term	What it means
Remote configuration	All the configuration pipelines matched to a collector
Configuration pipeline	The pieces of config you create and assign, usually representing one monitoring operation

Pipeline components

Component	Description
Name	Unique identifier for the pipeline
Content	Standard configuration for one operation
Matching attributes	Rules that assign the pipeline to collectors

Pipelines are reusable. Create once, assign to many collectors using attribute matching.

Working with pipelines

Thu, 09 Apr 2026 06:03:04 +0000

Creating pipelines

Approach	Best for
Integration templates	Common use cases (Linux, MySQL, PostgreSQL)
Custom configuration	Specific requirements, custom scrapes

Pipeline operations

Action	What it does
Activate/Deactivate	Toggle collection on/off instantly
Edit	Update configuration content
View history	See what changed and when

Why this matters

Instant changes — No deployments, no restarts
Safe experimentation — Toggle off if something goes wrong
Audit trail — History shows what changed and when

Designing your attribute schema

Thu, 09 Apr 2026 06:03:04 +0000

How to choose your attributes

Ask yourself: “What groups of collectors should receive the same configuration?”

If you want to target…	You need an attribute like…
All production servers	`env=production`
Collectors in specific regions	`region=us-east`
Collectors owned by a team	`team=platform`
Collectors for a specific service	`service=api-gateway`

Common attribute patterns

Attribute	Example values	Use case
`env`	`production`, `staging`, `dev`	Environment-specific configs
`region`	`us-east`, `eu-west`, `ap-south`	Region-specific collection
`team`	`platform`, `backend`, `data`	Team ownership
`service`	`api`, `web`, `worker`	Service-specific monitoring

Start simple. Begin with env and one or two others. Add more as needed.

Attribute anti-patterns

Thu, 09 Apr 2026 06:03:04 +0000

What not to do

Anti-pattern	Why it’s a problem
Too many attributes	Hard to manage, attribute limits per collector or API call
Frequently changing values	Use remote attributes in UI instead
Sensitive information	Attributes are visible in the Fleet Management UI
Duplicating reserved attributes	`collector.os`, `collector.version`, and `collector.ID` already exist
Inconsistent naming	`myEnv` vs `my_env` vs `my-env` causes confusion
Overly specific values	`hostname=server-123` — use `collector.ID` instead

Tips for beginners

Tip	Why it helps
Start simple	Begin with `env` and maybe one or two others
Be consistent	Pick a naming convention and stick to it
Document your schema	Note down what each attribute means
Think about targeting	Every attribute should help you target pipelines

Attribute examples

Thu, 09 Apr 2026 06:03:04 +0000

Example 1: Minimal starting schema

This example shows the recommended starting point for new Fleet Management users: just two attributes:

remotecfg {
    // ... other configuration ...
    
    attributes = {
        "env"  = "production",   // Which environment: production, staging, or dev
        "team" = "platform"      // Who owns this collector
    }
}

Attribute	Value	What it means
`env`	`production`	This collector monitors production infrastructure
`team`	`platform`	The platform team owns and manages this collector

What you can target with this schema

All production collectors → env=production
All collectors owned by platform team → team=platform
Production collectors owned by platform → env=production, team=platform

Example 2: Expanded schema for larger deployments

As your fleet grows, you may need more precise targeting. This example adds a region attribute:

remotecfg {
    // ... other configuration ...
    
    attributes = {
        "env"    = "production",
        "region" = "us-east-1",
        "team"   = "platform"
    }
}

Attribute	Value	What it means
`env`	`production`	This collector monitors production infrastructure
`region`	`us-east-1`	This collector is located in the US East region
`team`	`platform`	The platform team owns and manages this collector

What you can target with this schema

All US East collectors → region=us-east-1
Production collectors in US East → env=production, region=us-east-1
Platform team’s US East production collectors → all three attributes

Start simple, expand as needed. Begin with Example 1. Add more attributes when you need that level of targeting precision.

Attribute matching

Thu, 09 Apr 2026 06:03:04 +0000

Attribute matching

Attribute matching is the mechanism that connects pipelines to collectors. When you create a pipeline, you define matchers, which are rules that specify which collectors should receive that pipeline based on their attributes.

Matchers determine which collectors receive a pipeline.

How matchers work

You define…	To target…
`env=production`	All collectors with `env` attribute set to `production`
`region=~"us-.*"`	All collectors in any US region

Matcher syntax

Operator	Meaning	Example
`=`	Equals	`env=production`
`!=`	Not equals	`env!=development`
`=~`	Regex match	`region=~"us-.*"`
`!~`	Regex not match	`team!~"test-.*"`

Matching rule: AND logic

All matchers must match for a collector to receive a pipeline.

Matchers defined	Collector receives pipeline if…
`env=prod`	`env` is `prod`
`env=prod`, `region=us-east`	`env` is `prod` AND `region` is `us-east`

Matching in practice

Thu, 09 Apr 2026 06:03:04 +0000

Common patterns

Pattern	Matcher examples
Environment-based	`env="production"`, `env="staging"`
Region-based	`region="us-east"`, `region="eu-west"`
Team-based	`team="platform"`, `team="backend"`
Catch-all	`collector.os=~".+"` (matches all collectors)

Example: Production platform pipeline

Matchers: env=~"prod-.*", team="platform"

This pipeline applies to collectors where:

env matches the regex prod-.* AND
team equals platform

Limits to know

Limit	Value
Matcher length	200 characters max (key + operator + value)
Local attributes per collector	22 max when polling

Configuration strategies

Thu, 09 Apr 2026 06:03:04 +0000

Configuration strategies

Fleet Management gives you flexibility in how you balance local and remote configuration. You don’t have to go all-in on either approach. Choose the strategy that fits your situation.

Remote-first

Minimal local config, everything else managed remotely.

remotecfg {
    url = "https://..."
    // ... auth config
    id = constants.hostname
    attributes = {"env" = "prod"}
}

// Everything else comes from Fleet Management

Local-first

Primary config stays local, Fleet Management for self-monitoring.

// Local config: full configuration
remotecfg {
    url = "https://..."
    id = constants.hostname
    attributes = {"env" = "prod"}
}

// All your standard collection is here
prometheus.scrape "node" { /* ... */ }
prometheus.scrape "app" { /* ... */ }
loki.source.file "logs" { /* ... */ }

// Fleet Management used only for collector self-monitoring

Hybrid

Top-level config blocks stay local. Everything else is remote.

// Local config: top-level blocks
logging { level = "info" }

livedebugging { enabled = true }

remotecfg {
    url = "https://..."
    id = constants.hostname
    attributes = {"env" = "prod"}
}

// All other config managed remotely in Fleet Management

Why it's safe to experiment

Thu, 09 Apr 2026 06:03:04 +0000

Safeguards

Fleet Management is designed with safety in mind.

Four safeguards

Safeguard	What it means	Why it matters
Isolation	Local and remote configs run independently	Remote errors don’t break local collection
Fallback	Invalid configs aren’t applied	Bad syntax doesn’t stop your collector
Instant rollback	Deactivate pipelines with one click	Fast recovery from any problem
Targeted testing	Use matchers to limit blast radius	Test on one collector before fleet-wide

What happens when things go wrong

Scenario	What Alloy does	Your action
Syntax error in pipeline	Rejects config, keeps last known good	Fix syntax, re-save pipeline
Pipeline causes high load	Runs but collector health dashboards show warning	Deactivate pipeline, investigate
Wrong collectors matched	Pipeline runs on unintended collectors	Update matchers, deactivate if needed

Safe experimentation workflow

Thu, 09 Apr 2026 06:03:04 +0000

Safe experimentation workflow

Create pipeline
Set matcher for ONE test collector (e.g., collector.ID=“test-collector-01”)
Activate and verify
Expand matcher to small group (e.g., env=“staging”)
Verify at scale
Expand to full target (e.g., env=“production”)

The confidence equation

Isolation + Fallback + Instant rollback + Targeted testing = Safe to experiment

You’re not gambling with production. You’re making controlled changes with multiple safety nets. Start small, verify, expand.

Coming up next

You’ll categorize your collector fleet in the learning path. You’ll assign attributes like team, service.name, and env—the foundation that makes targeted configuration and safe rollouts possible.

Categorize your collector fleet

Thu, 09 Apr 2026 06:03:04 +0000

Learning path

Categorize a collector fleet

Welcome to the Grafana Fleet Management learning journey that shows you how to organize and categorize your collector fleet using custom attributes for remote configuration. Learn to categorize your infrastructure with meaningful attributes that answer critical questions such as who owns this collector, which service does it support, and where is it running.

9 min

Intermediate

Docs

Open in Grafana Cloud

Complete this learning path directly in your Grafana Cloud stack, or in the Grafana Play stack, with an interactive learning experience.

Open in Grafana Cloud Try in Grafana Play

View on website

Fleet meaningfully categorized!

Thu, 09 Apr 2026 06:03:04 +0000

What you accomplished

Navigated to Fleet Management and selected collectors
Assigned team attributes to identify collector ownership
Assigned service.name attributes to identify monitored services
Assigned env attributes to identify deployment environments
Used bulk edit to apply attributes across multiple collectors

Skills unlocked

Skill	What you can do now
Fleet categorization	Organize collectors with meaningful attributes
Attribute assignment	Set remote attributes in the Fleet Management UI
Bulk operations	Apply attributes to multiple collectors at once

Categorize once, target precisely. Attributes are the foundation for remote configuration at scale.