Slide 11 of 14

Why it's safe to experiment

Safeguards

Fleet Management is designed with safety in mind.

Experiment with confidence

Four safeguards

SafeguardWhat it meansWhy it matters
IsolationLocal and remote configs run independentlyRemote errors don’t break local collection
FallbackInvalid configs aren’t appliedBad syntax doesn’t stop your collector
Instant rollbackDeactivate pipelines with one clickFast recovery from any problem
Targeted testingUse matchers to limit blast radiusTest on one collector before fleet-wide

What happens when things go wrong

ScenarioWhat Alloy doesYour action
Syntax error in pipelineRejects config, keeps last known goodFix syntax, re-save pipeline
Pipeline causes high loadRuns but collector health dashboards show warningDeactivate pipeline, investigate
Wrong collectors matchedPipeline runs on unintended collectorsUpdate matchers, deactivate if needed

Script

There’s something that holds people back from using remote configuration: fear. Fear of assigning a bad config and breaking production. Fear of accidentally stopping data collection. Fear of making things worse during an incident.

These fears are reasonable—configuration management can be risky. But Fleet Management is designed with safety in mind. Here are the safeguards that make it safe to experiment.

First, isolation. Local and remote configurations run on completely separate component controllers inside Alloy. They don’t interfere with each other. If your remote configuration has a bug, your local configuration keeps running normally.

Second, fallback behavior. When Alloy polls for remote configuration and receives something invalid (maybe there’s a syntax error or a component that doesn’t exist), it doesn’t apply that config. Instead, it keeps running the last known good configuration.

Third, instant rollback. If you push a config that causes problems, you can deactivate the pipeline with one click. The collector stops running that pipeline on the next poll cycle.

Fourth, targeted testing. Because of attribute matching, you can assign a new pipeline to a single test collector first. Verify it works. Then expand to more collectors.