DocumentationGrafana AgentFlow modeTasksSet up meta-monitoring

Set up meta-monitoring

You can configure Grafana Agent Flow to collect its own telemetry and forward it to the backend of your choosing.

This topic describes how to collect and forward Grafana Agent Flow’s metrics, logs and traces data.

Components and configuration blocks used in this topic

• prometheus.exporter.self
• prometheus.scrape
• logging
• tracing

Before you begin

• Identify where to send Grafana Agent Flow’s telemetry data.
• Be familiar with the concept of Components in Grafana Agent Flow.

Meta-monitoring metrics

Grafana Agent Flow exposes its internal metrics using the Prometheus exposition format.

In this task, you will use the prometheus.exporter.self and prometheus.scrape components to scrape Grafana Agent Flow’s internal metrics and forward it to compatible Grafana Agent Flow components.

1. Add the following prometheus.exporter.self component to your configuration. The component accepts no arguments.

   river Copy

   prometheus.exporter.self "<SELF_LABEL>" {
   }

2. Add the following prometheus.scrape component to your configuration file.

   river Copy

   prometheus.scrape "<SCRAPE_LABEL>" {
     targets    = prometheus.exporter.<SELF_LABEL>.default.targets
     forward_to = [<METRICS_RECEIVER_LIST>]
   }

   Replace the following:

   • <SELF_LABEL>: The label for the component such as default or metamonitoring. The label must be unique across all prometheus.exporter.self components in the same configuration file.
   • <SCRAPE_LABEL>: The label for the scrape component such as default. The label must be unique across all prometheus.scrape components in the same configuration file.
   • <METRICS_RECEIVER_LIST>: A comma-delimited list of component receivers to forward metrics to. For example, to send to an existing remote write component, use prometheus.remote_write.WRITE_LABEL.receiver. Similarly, to send data to an existing relabeling component, use prometheus.relabel.PROCESS_LABEL.receiver. To use data in the OTLP format, you can send data to an existing converter component, like otelcol.receiver.prometheus.OTEL.receiver.

The following example demonstrates configuring a possible sequence of components.

river Copy

prometheus.exporter.self "default" {
}

prometheus.scrape "metamonitoring" {
  targets    = prometheus.exporter.self.default.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

Meta-monitoring logs

The logging block defines the logging behavior of Grafana Agent Flow.

In this task, you will use the logging block to forward Grafana Agent Flow’s logs to a compatible component. The block is specified without a label and can only be provided once per configuration file.

1. Add the following logging configuration block to the top level of your configuration file.

   river Copy

   logging {
     level    = "<LOG_LEVEL>"
     format   = "<LOG_FORMAT>"
     write_to = [<LOGS_RECEIVER_LIST>]
   }

   Replace the following:

   • <LOG_LEVEL>: The log level to use for Grafana Agent Flow’s logs. If the attribute isn’t set, it defaults to info.
   • <LOG_FORMAT>: The log format to use for Grafana Agent Flow’s logs. If the attribute isn’t set, it defaults to logfmt.
   • <LOGS_RECEIVER_LIST>: A comma-delimited list of component receivers to forward logs to. For example, to send to an existing processing component, use loki.process.PROCESS_LABEL.receiver. Similarly, to send data to an existing relabeling component, use loki.relabel.PROCESS_LABEL.receiver. To use data in the OTLP format, you can send data to an existing converter component, like otelcol.receiver.loki.OTEL.receiver.

The following example demonstrates configuring the logging block and sending to a compatible component.

river Copy

logging {
  level    = "warn"  
  format   = "json"
  write_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Meta-monitoring traces

The tracing block defines the tracing behavior of Grafana Agent Flow.

In this task you will use the tracing block to forward Grafana Agent Flow internal traces to a compatible component. The block is specified without a label and can only be provided once per configuration file.

1. Add the following tracing configuration block to the top level of your configuration file.

   river Copy

   tracing {
     sampling_fraction = <SAMPLING_FRACTION>
     write_to          = [<TRACES_RECEIVER_LIST>]
   }

   Replace the following:

   • <SAMPLING_FRACTION>: The fraction of traces to keep. If the attribute isn’t set, it defaults to 0.1.
   • <TRACES_RECEIVER_LIST>: A comma-delimited list of component receivers to forward traces to. For example, to send to an existing OpenTelemetry exporter component use otelcol.exporter.otlp.EXPORT_LABEL.input.

The following example demonstrates configuring the tracing block and sending to a compatible component.

river Copy

tracing {
  sampling_fraction = 0.1
  write_to          = [otelcol.exporter.otlp.default.input]
}

otelcol.exporter.otlp "default" {
    client {
        endpoint = "tempo:4317"
    }
}