Talos Logs Dashboard

Talos Logs Dashboard

Grafana dashboard for visualizing Talos Linux service and kernel logs collected via Vector and stored in VictoriaLogs.

This dashboard uses the native VictoriaMetrics Logs datasource plugin and LogsQL queries. It is not compatible with Loki or Grafana's built-in Loki datasource.

Requirements

Pipeline architecture

Talos node (TCP :6050)
    └─► Vector agent (DaemonSet)
            └─► Vector aggregator (vector protocol)
                    └─► VictoriaLogs (Elasticsearch ingest API)
                            └─► Grafana (victoriametrics-logs-datasource)

Vector agent — Talos source

The Vector agent runs as a DaemonSet on every node and reads Talos logs from the local TCP socket configured in the Talos machine config:

# talos/patches/global/logging.yaml
machine:
  logging:
    destinations:
      - endpoint: "tcp://127.0.0.1:6050/"
        format: json_lines
        extraTags:
          cluster: main
---
apiVersion: v1alpha1
kind: KmsgLogConfig
name: vector
url: tcp://127.0.0.1:6050/

The agent parses the JSON log lines, normalizes fields (msg → message, talos-level → level, talos-service → service), drops debug logs, and forwards to the aggregator via the Vector protocol.

Vector aggregator — VictoriaLogs sink

The aggregator receives logs and ships them to VictoriaLogs using the Elasticsearch ingest API:

sinks:
  talos:
    type: elasticsearch
    endpoints: ["http://victoria-logs:9428/insert/elasticsearch/"]
    api_version: v8
    compression: gzip
    query:
      _msg_field: message
      _stream_fields: node,service,stream,cluster,source,level
      _time_field: timestamp

The _stream_fields parameter defines which fields are indexed as stream labels, making them available for efficient LogsQL stream filtering.

Stream labels

Every Talos log entry is stored with the following stream labels:

Panels

Template variables

Note: The node list in the node variable is static. Update it to match your cluster's node names after import.

Datasource

On import, Grafana will prompt you to map the VictoriaLogs input to an installed datasource. Select your VictoriaLogs datasource (type: VictoriaMetrics Logs).

The dashboard was built and tested against VictoriaLogs v1.50.0. The Loki-compatible query API is not used — all queries use native LogsQL.

LogsQL query examples

# All Talos logs from all nodes
_stream:{source="talos"}
Only service logs from a specific node
_stream:{source="talos", stream="service", node="k8s-0"}
Only warnings and errors
_stream:{source="talos", node="k8s-.*"} level:"warn|error|crit"
Log rate over time per node
sum by (node) (count_over_time(_stream:{source="talos"}[5m]))