Manage telemetry costs and usage

Grafana Fleet Management is designed to give you maximum control over your telemetry collection. The guides in this section show you how to use its remote configuration and data flow features to estimate costs, control expensive telemetry collection, and triage unexpected spikes.

Reduce costs with data on demand

High verbosity telemetry, such as info and debug logs or continuous profiles, can become expensive if you’re collecting them all the time. But when there’s an incident, you might need these signals. With Fleet Management, you can create configuration pipelines to collect this telemetry but keep the pipelines disabled until you need them.

If you’re sampling traces from your collectors, you can also increase the sampling percentage remotely when it’s time to debug an issue.

Example: Collect on-demand profiles to diagnose excessive resource usage

For example, you can use on-demand continuous profiles to find the cause of excessive resource usage:

Create a profiling configuration pipeline using the Fleet Management application or the Pipeline API, but leave the pipeline disabled by turning off the UI switch or setting the enabled key to false in the API call.
When you notice an issue with high resource consumption, enable the profiling pipeline and make sure to add matching attributes so it’s matching to the correct collectors.
After finding the offending code, disable the profiling pipeline, remove the matching attributes, and leave the pipeline ready for the next time you need to troubleshoot an incident.

Example: Automate data collection during an incident response

Consider automating higher-resolution data collection as part of your incident response:

Create deactivated configuration pipelines that collect the “must gather” data specified in your runbooks. Leave them disabled.
When an incident is declared, the process triggers an API call that enables the necessary pipelines and sets matching attributes.
Closing the incident triggers another API call to disable the pipelines and remove matching attributes.

Example: Debug an issue with more tracing data

Increase your sampling percentage to get more tracing data when you need to debug an issue.

Alloy sampling configuration

Create a configuration pipeline that collects traces and samples 10% of them using probabilistic sampling.

otelcol.processor.probabilistic_sampler "default" {
    // Keep 10% of traces
  sampling_percentage = 10

  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

Match the pipeline to your collectors using attributes and activate it to begin receiving sampled traces.
If an issue occurs, return to the Fleet Management interface and edit the configuration pipeline to increase the sampling percentage to 100% so you can see all tracing data.
Alloy
```
otelcol.processor.probabilistic_sampler "default" {
    // Keep 100% of traces
  sampling_percentage = 100

  output {
    traces = [otelcol.processor.batch.default.input]
  }
}
```
When the issue is resolved, edit the pipeline again to return the sampling percentage to 10%.

OpenTelemetry Collector sampling configuration

Create a configuration pipeline that collects traces and samples 10% of them with the probabilistic sampler processor.

exporters:
  otlp_http:
    endpoint: ${env:OTLP_EXPORT_ENDPOINT}

receivers:
  otlp:
    protocols:
      http:
        endpoint: ${env:OTLP_INGEST_ENDPOINT}

processors:
  probabilistic_sampler:
    sampling_percentage: 10

service:
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - probabilistic_sampler
      exporters:
        - otlp_http

Match the pipeline to your collectors using attributes and activate it to begin receiving sampled traces.
If an issue occurs, return to the Fleet Management interface and edit the configuration pipeline to increase the sampling percentage to 100% so you can see all tracing data. The unchanged portion of the following configuration is omitted for brevity.
YAML
```
processors:
  probabilistic_sampler:
    sampling_percentage: 100

service:
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - probabilistic_sampler
      exporters:
        - otlp_http
```
When the issue is resolved, edit the pipeline again to return the sampling percentage to 10%.