Menu

GitLab integration for Grafana Cloud

GitLab is an open-source DevOps platform for software development. The GitLab integration uses the Grafana agent to collect metrics for monitoring a GitLab EE (Enterprise Edition) instance, including aspects such as HTTP request rates, HTTP request latencies, and CI pipeline creation rates. The integration also supports the rails error logs being scraped by the agent using Promtail. An accompanying dashboard is provided to visualize these metrics and logs.

This integration supports GitLab EE 15.3.3+.

Pre-install configuration for the GitLab integration

This integration monitors a GitLab EE instance that exposes its metrics through GitLab’s built in prometheus metrics server.

In order for the integration to work, you must enable the prometheus metrics as described in the GitLab documentation.

In order to collect metrics using the agent, the local IP must be added to the monitoring whitelist in /etc/gitlab/gitlab.rb:

gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8']

Then, GitLab must be restarted for the above change to take effect:

sudo gitlab-ctl restart

Install GitLab integration for Grafana Cloud

  1. In your Grafana Cloud instance, click Integrations and Connections (lightning bolt icon).
  2. Navigate to the GitLab tile and review the prerequisites. Then click Install integration.
  3. Once the integration is installed, follow the steps on the Configuration Details page to setup Grafana Agent and start sending GitLab metrics to your Grafana Cloud instance.

Post-install configuration for the GitLab integration

This integration supports metrics and logs from a GitLab EE instance. In order to see these signals correlated on the same dashboard, the job and instance labels must match for the metrics scrape config and the logs scrape config in the Agent configuration file (replace <your-instance-name> with an identifier for your instance):

metrics:
  wal_directory: /tmp/wal
  configs:
    - name: integrations
      scrape_configs:
        - job_name: integrations/gitlab
          metrics_path: "/-/metrics"
          relabel_configs:
            - replacement: "<your-instance-name>"
              target_label: instance
          static_configs:
            - targets: ['localhost']
      remote_write:
        - url: http://cortex:9009/api/prom/push
logs:
  configs:
  - name: integrations
    clients:
      - url: http://loki:3100/loki/api/v1/push
    positions:
      filename: /var/lib/grafana-agent/logs/positions.yaml
    scrape_configs:
    - job_name: integrations/gitlab
      static_configs:
        - targets:
          - localhost
          labels:
            instance: "<your-instance-name>"
            job: integrations/gitlab
            __path__: /var/log/gitlab/gitlab-rails/exceptions_json.log
      pipeline_stages:
        - match:
            selector: '{job="integrations/gitlab"} |~ "^#"'
            action: drop
            drop_counter_reason: gitlab_comment_line

By default, the agent scrapes the rails error logs at this path: /var/log/gitlab/gitlab-rails/exceptions_json.log.

Dashboards

The GitLab integration installs the following dashboards in your Grafana Cloud instance to help monitor your metrics.

  • GitLab Overview

GitLab overview dashboard 1

image

GitLab overview dashboard 2

image

GitLab overview dashboard 3

image

Alerts

The GitLab integration includes the following useful alerts:

Group: GitLabAlerts

AlertDescription
GitLabHighJobRegistrationFailuresWarning: Large percentage of failed attempts to register a job.
GitLabHighRunnerAuthFailureWarning: Large percentage of runner authentication failures.
GitLabHigh5xxResponsesCritical: Large rate of HTTP 5XX errors.
GitLabHigh4xxResponsesWarning: Large rate of HTTP 4XX errors.

Metrics

The following metrics are automatically written to your Grafana Cloud instance by connecting your GitLab instance through this integration:

  • gitlab_ci_active_jobs_sum
  • gitlab_ci_pipeline_size_builds_sum
  • gitlab_ci_runner_authentication_failure_total
  • gitlab_ci_runner_authentication_success_total
  • gitlab_ci_trace_operations_total
  • gitlab_rails_boot_time_seconds
  • http_request_duration_seconds_count
  • http_request_duration_seconds_sum
  • http_requests_total
  • job_register_attempts_failed_total
  • job_register_attempts_total
  • pipelines_created_total
  • user_session_logins_total

Changelog

# 0.0.1 - November 2022

* Initial Release

Cost

By connecting your GitLab instance to Grafana Cloud you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.