Grafana Cloud Data configuration Grafana integrations Integrations reference ClickHouse integration

ClickHouse integration for Grafana Cloud

ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real-time. The ClickHouse integration uses the Grafana agent to collect metrics to monitor a ClickHouse cluster that can be used to easily select metrics via cluster instance. The integration also supports the ClickHouse error logs being scraped by the agent using Promtail. An accompanying dashboard is provided to visualize these metrics and logs.

This integration supports ClickHouse 22.6.8.35+.

Install ClickHouse integration for Grafana Cloud

1. In your Grafana Cloud instance, click on Connections from the Home menu located near the top left-hand side of the page.
2. Navigate to the ClickHouse tile and click on it to review the prerequisites.
3. When you are ready, follow the steps on the Configuration Details page to setup Grafana Agent and start sending ClickHouse metrics to your Grafana Cloud instance.

Pre-install configuration for the ClickHouse integration

ClickHouse supports multi-file configurations using either a /etc/clickhouse-server/config.xml or /etc/clickhouse-server/config.yaml. This example shows config.xml snippets required to collect metrics and logs.

Prometheus metrics are scraped using the ClickHouse Exporter that must be configured in the config.xml. Some ClickHouse installations have the prometheus exporter commented out, else include this snippet:

    <prometheus>
        <endpoint>/metrics</endpoint>
        <port>9363</port>
        <metrics>true</metrics>
        <events>true</events>
        <asynchronous_metrics>true</asynchronous_metrics>
        <status_info>true</status_info>
    </prometheus>

For more information please check out Prometheus.

ClickHouse error logs are scraped using a Promtail agent that targets the default /var/log/clickhouse-server/clickhouse-server.err.log path. Logs are enabled by including the target path inside the <errorlog> within the ClickHouse config.xml. Some ClickHouse installations have the error log by default, which would be located at the top of the config.xml file.

    <logger>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
    </logger>

For more information please check ClickHouse logger settings.

After editing the config.xml, restart the ClickHouse server:

sudo service clickhouse-server restart

Post-install configuration for the ClickHouse integration

This integration supports metrics and logs from a ClickHouse instance. The Grafana Agent config is stored in /etc/grafana-agent.yaml. In order to see these signals correlated on the same dashboard, job and instance labels must match for metrics scrape config and the logs scrape config in the Agent configuration file (replace with an identifier for your instance):

metrics:
  wal_directory: /tmp/wal
  configs:
    - name: integrations
      scrape_configs:
        - job_name: integrations/clickhouse
          relabel_configs:
            - replacement: "<your-instance-name>"
              target_label: instance
          static_configs:
            - targets: ['localhost:9363']
      remote_write:
        - url: http://cortex:9009/api/prom/push

logs:
  configs:
  - name: integrations
    clients:
      - url: http://loki:3100/loki/api/v1/push
    positions:
      filename: /var/lib/grafana-agent/logs/positions.yaml
    scrape_configs:
    - job_name: integrations/clickhouse
      static_configs:
      - targets:
        - localhost
        labels:
          instance: "<your-instance-name>"
          job: integrations/clickhouse
          __path__: /var/log/clickhouse-server/clickhouse-server.err.log
    pipeline_stages:
      - multiline:
          firstline: '^([\d]{4}).([\d]{1,2}).([\d]{1,2}) (([\d]+):([\d]+):([\d]+).([\d]+))'
      - regex:
          expression: '(?P<timestamp>(([\d]{4}).([\d]{1,2}).([\d]{1,2}) (([\d]+):([\d]+):([\d]+).([\d]+))) (?P<message>(?s:.*))$)'

Additionally, to gather logs from a Linux host using the Grafana Agent, add a Grafana Agent user as an owner to collect from the log location as mentioned here. For example, if you are running on Debian:

sudo usermod -a -G clickhouse grafana-agent

Dashboards

This integration includes the following dashboards.

• ClickHouse latency
• ClickHouse overview
• ClickHouse replica

ClickHouse Overview Dashboard 1

ClickHouse Overview Dashboard 2

ClickHouse Latency Dashboard

ClickHouse Replica Dashboard

Metrics

Below is a list of the metrics used on the dashboards and in alerts of the integration and are automatically written to your Grafana Cloud instance.

• ClickHouseAsyncMetrics_ReplicasMaxQueueSize
• ClickHouseMetrics_HTTPConnection
• ClickHouseMetrics_InterserverConnection
• ClickHouseMetrics_MemoryTracking
• ClickHouseMetrics_MySQLConnection
• ClickHouseMetrics_PostgreSQLConnection
• ClickHouseMetrics_ReadonlyReplica
• ClickHouseMetrics_TCPConnection
• ClickHouseMetrics_ZooKeeperRequest
• ClickHouseMetrics_ZooKeeperSession
• ClickHouseMetrics_ZooKeeperWatch
• ClickHouseProfileEvents_AsyncInsertQuery
• ClickHouseProfileEvents_DiskReadElapsedMicroseconds
• ClickHouseProfileEvents_DiskWriteElapsedMicroseconds
• ClickHouseProfileEvents_FailedInsertQuery
• ClickHouseProfileEvents_FailedQuery
• ClickHouseProfileEvents_FailedSelectQuery
• ClickHouseProfileEvents_InsertQuery
• ClickHouseProfileEvents_NetworkReceiveBytes
• ClickHouseProfileEvents_NetworkSendBytes
• ClickHouseProfileEvents_NetworkSendElapsedMicroseconds
• ClickHouseProfileEvents_Query
• ClickHouseProfileEvents_RejectedInserts
• ClickHouseProfileEvents_ReplicatedPartChecks
• ClickHouseProfileEvents_ReplicatedPartFetches
• ClickHouseProfileEvents_ReplicatedPartMerges
• ClickHouseProfileEvents_ReplicatedPartMutations
• ClickHouseProfileEvents_SelectQuery
• ClickHouseProfileEvents_ZooKeeperWaitMicroseconds

Alerts

The integration also comes packaged with a number of handy alerts.

Group: ClickHouseAlerts

Changelog

# 0.0.1 - February 2023

* Initial release

Cost

By connecting your ClickHouse instance to Grafana Cloud you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.