Apache Solr integration for Grafana Cloud

Before you begin

In order for the integration to properly work, it is essential to implement the built-in Prometheus Exporter on each cluster instance.

Initiating the Prometheus Exporter

Initiate the Prometheus Exporter in Apache Solr either in a Standalone or SolrCloud mode by navigating to contrib/prometheus-exporter on your Solr instance and applying relevant command line parameters as demonstrated below.

For Linux, initiate the Prometheus Exporter in SolrCloud mode with:
./bin/solr-exporter -p 9854 -z localhost:2181/solr -f ./conf/solr-exporter-config.xml -n 16

For Windows, initiate the Prometheus Exporter in SolrCloud mode with:
.\bin\solr-exporter -p 9854 -z localhost:2181/solr -f .\conf\solr-exporter-config.xml -n 16

In cases where SolrCloud mode operates with a ZooKeeper ensemble, set the -z zkhost value as a string zk1,zk2,zk3.

Validate the Prometheus Exporter

Upon activation, the Prometheus Exporter will report the metrics. The full metric list will be shown when there is an active ZooKeeper ensemble setup and a Solr Collection created. To validate that the Prometheus Exporter is setup correctly, the Prometheus metrics are accessible locally via curl:

sh Copy

curl http://localhost:9854/metrics

Configure logs location.

No additional configuration is required for the collection of Apache Solr logs. By default, logs are written to the SOLR_LOGS_DIR, located at /var/solr/logs on Linux or sever/logs on Windows.

Install Apache Solr integration for Grafana Cloud

1. In your Grafana Cloud stack, click Connections in the left-hand menu.
2. Find Apache Solr and click its tile to open the integration.
3. Review the prerequisites in the Configuration Details tab and set up Grafana Agent to send Apache Solr metrics and logs to your Grafana Cloud instance.
4. Click Install to add this integration’s pre-built dashboards and alerts to your Grafana Cloud instance, and you can start monitoring your Apache Solr setup.

Configuration snippets for Grafana Alloy

Advanced mode

The following snippets provide examples to guide you through the configuration process.

To instruct Grafana Alloy to scrape your Apache Solr instances, manually copy and append the snippets to your alloy configuration file, then follow subsequent instructions.

Advanced metrics snippets

river Copy

prometheus.scrape "metrics_integrations_integrations_apache_solr" {
	targets = [{
		__address__  = "localhost:<your-instance-port>",
		solr_cluster = "<your-solr-cluster-name>",
	}]
	forward_to = [prometheus.remote_write.metrics_service.receiver]
	job_name   = "integrations/apache-solr"
}

To monitor your Apache Solr instance, you must use a discovery.relabel component to discover your Apache Solr Prometheus endpoint and apply appropriate labels, followed by a prometheus.scrape component to scrape it.

Configure the following properties within each discovery.relabel component:

• __address__: The address to your Apache Solr Prometheus metrics endpoint.
• instance label: constants.hostname sets the instance label to your Grafana Alloy server hostname. If that is not suitable, change it to a value uniquely identifies this Apache Solr instance. Make sure this label value is the same for all telemetry data collected for this instance.
• solr_cluster must be the value that identifies the Solr cluster this instance belongs to.

If you have multiple Apache Solr servers to scrape, configure one discovery.relabel for each and scrape them by including each under targets within the prometheus.scrape component.

Advanced logs snippets

darwin

river Copy

local.file_match "logs_integrations_integrations_apache_solr" {
	path_targets = [{
		__address__  = "localhost",
		__path__     = "/var/solr/logs/*.log",
		instance     = constants.hostname,
		job          = "integrations/apache-solr",
		solr_cluster = "<your-solr-cluster-name>",
	}]
}

loki.process "logs_integrations_integrations_apache_solr" {
	forward_to = [loki.write.grafana_cloud_loki.receiver]

	stage.multiline {
		firstline     = "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}"
		max_lines     = 0
		max_wait_time = "3s"
	}

	stage.regex {
		expression = "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3} (?P<level>\\w+)"
	}

	stage.labels {
		values = {
			level = null,
		}
	}
}

loki.source.file "logs_integrations_integrations_apache_solr" {
	targets    = local.file_match.logs_integrations_integrations_apache_solr.targets
	forward_to = [loki.process.logs_integrations_integrations_apache_solr.receiver]
}

To monitor your Apache Solr instance logs, you will use a combination of the following components:

• local.file_match defines where to find the log file to be scraped. Change the following properties according to your environment:

  • __address__: The Apache Solr instance address
  • __path__: is the Solr logs location or where SOLR_LOGS_DIR is set. By default this value should be either /var/solr/logs/*.logs on Linux or sever/logs/*.logs on Windows.
  • instance label: constants.hostname sets the instance label to your Grafana Alloy server hostname. If that is not suitable, change it to a value uniquely identifies this Apache Solr instance. Make sure this label value is the same for all telemetry data collected for this instance.
  • solr_cluster must be the value that identifies the Solr cluster this instance belongs to.

• loki.process defines how to process logs before sending it to Loki.

• loki.source.file sends logs to Loki.

linux

river Copy

local.file_match "logs_integrations_integrations_apache_solr" {
	path_targets = [{
		__address__  = "localhost",
		__path__     = "/var/solr/logs/*.log",
		instance     = constants.hostname,
		job          = "integrations/apache-solr",
		solr_cluster = "<your-solr-cluster-name>",
	}]
}

loki.process "logs_integrations_integrations_apache_solr" {
	forward_to = [loki.write.grafana_cloud_loki.receiver]

	stage.multiline {
		firstline     = "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}"
		max_lines     = 0
		max_wait_time = "3s"
	}

	stage.regex {
		expression = "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3} (?P<level>\\w+)"
	}

	stage.labels {
		values = {
			level = null,
		}
	}
}

loki.source.file "logs_integrations_integrations_apache_solr" {
	targets    = local.file_match.logs_integrations_integrations_apache_solr.targets
	forward_to = [loki.process.logs_integrations_integrations_apache_solr.receiver]
}

To monitor your Apache Solr instance logs, you will use a combination of the following components:

• local.file_match defines where to find the log file to be scraped. Change the following properties according to your environment:

  • __address__: The Apache Solr instance address
  • __path__: is the Solr logs location or where SOLR_LOGS_DIR is set. By default this value should be either /var/solr/logs/*.logs on Linux or sever/logs/*.logs on Windows.
  • instance label: constants.hostname sets the instance label to your Grafana Alloy server hostname. If that is not suitable, change it to a value uniquely identifies this Apache Solr instance. Make sure this label value is the same for all telemetry data collected for this instance.
  • solr_cluster must be the value that identifies the Solr cluster this instance belongs to.

• loki.process defines how to process logs before sending it to Loki.

• loki.source.file sends logs to Loki.

windows

river Copy

local.file_match "logs_integrations_integrations_apache_solr" {
	path_targets = [{
		__address__  = "localhost",
		__path__     = "/var/solr/logs/*.log",
		instance     = constants.hostname,
		job          = "integrations/apache-solr",
		solr_cluster = "<your-solr-cluster-name>",
	}]
}

loki.process "logs_integrations_integrations_apache_solr" {
	forward_to = [loki.write.grafana_cloud_loki.receiver]

	stage.multiline {
		firstline     = "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}"
		max_lines     = 0
		max_wait_time = "3s"
	}

	stage.regex {
		expression = "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3} (?P<level>\\w+)"
	}

	stage.labels {
		values = {
			level = null,
		}
	}
}

loki.source.file "logs_integrations_integrations_apache_solr" {
	targets    = local.file_match.logs_integrations_integrations_apache_solr.targets
	forward_to = [loki.process.logs_integrations_integrations_apache_solr.receiver]
}

To monitor your Apache Solr instance logs, you will use a combination of the following components:

• local.file_match defines where to find the log file to be scraped. Change the following properties according to your environment:

  • __address__: The Apache Solr instance address
  • __path__: is the Solr logs location or where SOLR_LOGS_DIR is set. By default this value should be either /var/solr/logs/*.logs on Linux or sever/logs/*.logs on Windows.
  • instance label: constants.hostname sets the instance label to your Grafana Alloy server hostname. If that is not suitable, change it to a value uniquely identifies this Apache Solr instance. Make sure this label value is the same for all telemetry data collected for this instance.
  • solr_cluster must be the value that identifies the Solr cluster this instance belongs to.

• loki.process defines how to process logs before sending it to Loki.

• loki.source.file sends logs to Loki.

Before you begin

In order for the integration to properly work, it is essential to implement the built-in Prometheus Exporter on each cluster instance.

Initiating the Prometheus Exporter

Initiate the Prometheus Exporter in Apache Solr either in a Standalone or SolrCloud mode by navigating to contrib/prometheus-exporter on your Solr instance and applying relevant command line parameters as demonstrated below.

For Linux, initiate the Prometheus Exporter in SolrCloud mode with:
./bin/solr-exporter -p 9854 -z localhost:2181/solr -f ./conf/solr-exporter-config.xml -n 16

For Windows, initiate the Prometheus Exporter in SolrCloud mode with:
.\bin\solr-exporter -p 9854 -z localhost:2181/solr -f .\conf\solr-exporter-config.xml -n 16

In cases where SolrCloud mode operates with a ZooKeeper ensemble, set the -z zkhost value as a string zk1,zk2,zk3.

Validate the Prometheus Exporter

Upon activation, the Prometheus Exporter will report the metrics. The full metric list will be shown when there is an active ZooKeeper ensemble setup and a Solr Collection created. To validate that the Prometheus Exporter is setup correctly, the Prometheus metrics are accessible locally via curl:

sh Copy

curl http://localhost:9854/metrics

Configure logs location.

No additional configuration is required for the collection of Apache Solr logs. By default, logs are written to the SOLR_LOGS_DIR, located at /var/solr/logs on Linux or sever/logs on Windows.

Install Apache Solr integration for Grafana Cloud

1. In your Grafana Cloud stack, click Connections in the left-hand menu.
2. Find Apache Solr and click its tile to open the integration.
3. Review the prerequisites in the Configuration Details tab and set up Grafana Agent to send Apache Solr metrics and logs to your Grafana Cloud instance.
4. Click Install to add this integration’s pre-built dashboards and alerts to your Grafana Cloud instance, and you can start monitoring your Apache Solr setup.

Post-install configuration for the Apache Solr integration

After enabling metric collection through the Prometheus Exporter, instruct the Grafana Agent to scrape these Apache Solr metrics. The Grafana Agent config is stored in the default location for each respective operating system. More documentation for the Grafana Agent can be found here: Grafana Agent documentation.

Enable the integration by adding the suggested snippets to your agent configuration file.

To show show logs and metrics signals correlated in your dashboards as a single pane of glass, ensure the following:

• job and instance label values must match for the Solr integration and logs scrape config in your agent configuration file.
• job must be set to integrations/apache-solr (already configured in the snippets).
• instance must be set to a value that uniquely identifies your Solr instance.
• solr_cluster must be the value that identifies the Solr cluster this instance belongs to.
• __path__ is the Solr logs location or where SOLR_LOGS_DIR is set. By default this value should be either /var/solr/logs/*.logs on Linux or sever/logs/*.logs on Windows.
• Make sure to change targets in the snippet according to your host name and exposed Prometheus port.

Configuration snippets for Grafana Agent

Below metrics.configs.scrape_configs, insert the following lines and change the URLs according to your environment:

yaml Copy

    - job_name: integrations/apache-solr
      metrics_path: /metrics
      static_configs:
        - targets: ["<your-instance-hostname>:<your-instance-port>"]
          labels:
            solr_cluster: '<your-solr-cluster-name>'

Below logs.configs.scrape_configs, insert the following lines according to your environment.

yaml Copy

    - job_name: integrations/apache-solr
      static_configs:
        - targets: [localhost]
          labels:
            job: integrations/apache-solr
            instance: '<your-instance-name>'
            solr_cluster: '<your-solr-cluster-name>'
            __path__: /var/solr/logs/*.log
      pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}'
      - regex:
          expression: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} (?P<level>\w+)'
      - labels:
          level:

Full example configuration for Grafana Agent

Refer to the following Grafana Agent configuration for a complete example that contains all the snippets used for the Apache Solr integration. This example also includes metrics that are sent to monitor your Grafana Agent instance.

yaml Copy

integrations:
  prometheus_remote_write:
  - basic_auth:
      password: <your_prom_pass>
      username: <your_prom_user>
    url: <your_prom_url>
  agent:
    enabled: true
    relabel_configs:
    - action: replace
      source_labels:
      - agent_hostname
      target_label: instance
    - action: replace
      target_label: job
      replacement: "integrations/agent-check"
    metric_relabel_configs:
    - action: keep
      regex: (prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
      source_labels:
      - __name__
  # Add here any snippet that belongs to the `integrations` section.
  # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
logs:
  configs:
  - clients:
    - basic_auth:
        password: <your_loki_pass>
        username: <your_loki_user>
      url: <your_loki_url>
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      # Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
    - job_name: integrations/apache-solr
      static_configs:
        - targets: [localhost]
          labels:
            job: integrations/apache-solr
            instance: '<your-instance-name>'
            solr_cluster: '<your-solr-cluster-name>'
            __path__: /var/solr/logs/*.log
      pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}'
      - regex:
          expression: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} (?P<level>\w+)'
      - labels:
          level:
metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: <your_prom_pass>
        username: <your_prom_user>
      url: <your_prom_url>
    scrape_configs:
      # Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
    - job_name: integrations/apache-solr
      metrics_path: /metrics
      static_configs:
        - targets: ["<your-instance-hostname>:<your-instance-port>"]
          labels:
            solr_cluster: '<your-solr-cluster-name>'
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal