Set up a Grafana Enterprise Logs cluster on Grafana Labs

Get a GEL license

Tue, 16 Jul 2024 15:42:20 +0000

Get a GEL license

To get a license, contact a Grafana Labs representative and give them the cluster name that you want to associate with the license.

GEL hardware requirements

Tue, 16 Jul 2024 15:42:20 +0000

GEL hardware requirements

This page outlines the current hardware requirements for running Grafana Enterprise Logs (GEL). Grafana Labs reserves the right to mark a support issue as ‘unresolvable’ if these requirements are not followed. See the Grafana Labs Enterprise Support SLA for more details.

CPU and memory

GEL should be deployed on machines with a 1:4 ratio of CPU to memory, so for every CPU core there should be 4 gigabytes of memory. For most clusters, Grafana Labs recommends deploying GEL onto machines with 16 CPU cores and 64 gigabytes of memory. All the nodes in the cluster should be of the same type. This is a good mix of CPU to memory for the type of workloads that GEL usually performs.

Disk

Various components of GEL (ingester, alertmanager) require fast, persistent disk resources to be available to the host machine. For example, in the case of the ingester and ruler components, all incoming data is sent to a write-ahead log (WAL) to help withstand unexpected node termination. The following are supported configurations for several cloud providers as well as guidance for custom hardware:

Amazon Web Services (AWS)

GEL is tested to run with io1 Provisioned IOPS SSD EBS volumes to ensure adequate performance to run the system correctly. The io1 storage must be provisioned at 50 IOPS per gigabyte, with a minimum of 150Gi allocated to ensure performant I/O.

Google Cloud Platform (GCP)

GEL is tested to run with pd-ssd SSD persistent disks to ensure adequate performance to run the system correctly.

Microsoft Azure

GEL is tested to run with Premium SSD SSD persistent disks to ensure adequate performance to run the system correctly.

Custom cluster hardware

GEL requires fast disks to run. Build your cluster with fast, locally attached SSD-based disks.

Network

All components of GEL require fast network access. Nodes on which the software runs should be connected by 10 gigabit/second or faster network connection speed.

Object storage

Various GEL components require object storage for config storage as well as long-term data storage.

Amazon Web Services (AWS)

GEL is tested to run with AWS’s S3 object storage service using the Standard storage class.

Google Cloud Platform (GCP)

GEL is tested to run with GCP’s GCS object storage service using the STANDARD storage class in both regional and dual regional storage locations.

Microsoft Azure

GEL is tested to run Azure’s Blob Storage object storage service using the Standard storage class with replication type LRS.

Unmanaged object storage

GEL generally works with object storage installations which support the popular AWS S3 API. However, vendors have various performance characteristics for each solution and installation, so performance testing with your individual solution will be necessary to determine if the performance profile will work for your use case. Slower storage backed by hard disks might be acceptable for less intensive workloads, but more intensive workloads will likely require more performant object storage solutions backed by SSDs.

Deploy GEL on Kubernetes with Helm

Tue, 16 Jul 2024 15:42:20 +0000

Deploy GEL on Kubernetes with Helm

The Helm charts for Grafana Enterprise Logs allows you to configure, install, and upgrade Grafana Enterprise Logs within a Kubernetes cluster.

To install Grafana Enterprise Logs on Kubernetes, use the Grafana Helm chart documentation.

License

You need a license to run Grafana Enterprise Logs. Your Grafana Labs representative should have provided you with a license.jwt license file for the cluster name you provided. The following section details various ways of setting the license for the Helm chart. Choose the one most appropriate for you.

The text “found a valid license” can be found in the logs of the Grafana Enterprise Logs components if the license has been specified correctly.

Once you have obtained your license, do either of the following:

Configure the license in the Helm values file
Store the license in a Kubernetes secret

Configure the license in the Helm values file

Add the following section to the Helm chart values file:

enterprise:
  enabled: true
  cluster_name: <cluster name>
license:
  contents: <content of license.jwt>

Store the license in a Kubernetes secret

Run the following command to load your GEL license file (license.jwt) as a Kubernetes secret.
Bash
```
kubectl create secret generic ge-logs-license --from-file license.jwt
```
Verify you have successfully created the secret by running the following command:
Bash
```
kubectl get secret ge-logs-license -oyaml
```
The preceding command prints a Kubernetes Secret object with a license.jwt field that contains a long base64-encoded value string.

Add the following section to the Helm chart values file:

enterprise:
  enabled: true
  cluster_name: <cluster name>
  useExternalLicense: true
  externalLicenseName: ge-logs-license

Set up the Grafana Enterprise Logs plugin for Grafana

Mon, 14 Apr 2025 21:05:47 +0000

Set up the Grafana Enterprise Logs plugin for Grafana

Requirements

Grafana Enterprise 7.3.0 or higher.

If you are using Kubernetes, refer to Deploy Grafana Enterprise on Kubernetes. Otherwise, refer to Install Grafana.

Install the plugin in your GEL instance

There are multiple ways to install the plugin in your local Grafana Enterprise instance. For more information, refer to Grafana Enterprise Logs app installation.

Enable and configure the plugin

Log in to your Grafana Enterprise Logs.
Go to the Config/Plugins page and select the Grafana Enterprise Logs (GEL) plugin from list.
From the configuration page of the plugin, enable the plugin by clicking on the “Enable plugin” button.
Provide the necessary API settings so that the plugin can connect to your cluster:

Access Token: Enter the admin-scoped access token that you generated when setting up your GEL cluster.
Grafana Enterprise Logs URL: Enter the URL of your GEL cluster. For single-process clusters, this is any node in the cluster. For microservice deployments, the URL is the GEL gateway. If you followed the Deploy on Kubernetes guide, your URL is http://ge-logs.<namespace>.svc.cluster.local:8100/, where <namespace> is the namespace that you used. For example, if you used the default namespace, then your URL would be http://ge-logs.default.svc.cluster.local:8100/.

Configuration GEL plugin page

Click Save API settings.
Verify that the plugin loads and can communicate with the GEL admin API endpoints.
Navigate to the GEL plugin through the main menu to see the default access policy under the Access Policies tab.

Now that you have correctly configured the GEL app plugin, follow directions in Set up a GEL tenant and visualize your data.

Set up a Grafana Enterprise Logs tenant

Mon, 14 Apr 2025 21:05:47 +0000

Set up a Grafana Enterprise Logs tenant

Tenants provide a mechanism for log stream isolation. Access policies may be set on a per-tenant basis. Authorization of requests is based on specified access policies.

These instructions assume that you have the Grafana Enterprise Logs administrative plugin installed. Use this plugin to create tenants, access policies, and tokens for your GEL cluster.

Create a tenant

Once a cluster is running, you can create new tenants.

Navigate to Grafana Enterprise Logs > Tenants.
Click Create tenant.
Enter a chosen display name and name for this tenant.
Choose the cluster for this tenant.
Click Save changes.

Create an access policy

Access policies are used to authorize actions and operations by specified tenants. Access policies have a realm, which defines the set of tenants they apply to, and a scope which defines the set of actions that they confer permissions to use.

Navigate to Grafana Enterprise Logs > Access Policies.
Click Create access policy.
Enter a chosen display name and name for access policy.
To enter the scopes for this access policy, click on either the Yes or No box, as appropriate to answer the question, under the Scopes heading to bring up a list of clickable scopes. Place check marks next to those scopes that correspond to operations that will be authorized under this access policy.
Sequentially select all tenants this access policy will grant access to.
Click Create.

Create tokens for the access policies

A token will be needed by any entity requesting actions or operations. One or more tokens may be created for each access policy. Tokens can be created with an expiration date, if the administrator wishes access granted to the system for a specific length of time.

Navigate to Grafana Enterprise Logs > Access Policies.
Click Add token for the access policy.
Enter a chosen name for the token and specify the expiration details.
Click Create.
Copy and save the token displayed.

Create a Grafana data source

To allow Grafana to read logs from GEL, you must create a Loki data source with the proper credentials.

Create an access policy with scope logs:read for the tenant you want to read logs from. Create and save a token for this access policy.
In Grafana Enterprise, navigate to Configuration > Data Sources.
Click Add data source.
Specify a name for this data source. Set the URL to http://<GEL host>:3100.
Enable Basic Auth.
The User differs based on use case. Set the User to one of:
- For single tenant access, set the User to the name of the tenant you want to read from.
- For explicitly-specified, multiple-tenant access, set the User to include the names of the each tenant you want to read from; delimit the tenant names with a pipe character (|). As an example, for the two tenants named team-engineering and team-finance, the User will be team-engineering|team-finance. This data source explicitly limits the tenants. The data source must be modified to add or remove a tenant.
- For multiple tenant access by all tenants specified in an access policy, set the User to *. If the access policy changes, the data source will not need to be modified to honor the modified access policy.
Set the Password to your saved token for the access policy with logs:read access to the tenant(s).
Click Save & Test.

Promtail access policy and token

Promtail will need an access policy with logs:write scope in order to push logs to a GEL cluster. Create an access policy and token to be used by Promtail. Capture the token and specify it in the Promtail configuration.

Use label-based access control with GEL

Mon, 14 Apr 2025 21:05:47 +0000

Use label-based access control with GEL

Note
This feature is available from v1.1.0. For the latest releases, refer to Download Grafana Enterprise Logs.

Label-based access control creates access policies that allow you to query only the logs that meet specific label requirements. The feature allows you to associate multiple sets of Prometheus label selectors with a policy. As a result, queries only return data from the logs that match at least one of the provided selectors. This correlates to disjunctive normal form, which allows you to express any required policy.

Setting up a label policy

Label policies are set when you create an access policy on a per-tenant basis. This means that each tenant that is associated with an access policy can have a unique label policy.

Click Create access policy.
Fill in the Display name field with the access policy name.
Select the logs:read scope.
Select a tenant.
Click Add label selector and add a label selector.
Click Create.

Alertmanager and ruler

Label policies are not enforced by the alertmanager and ruler. This means that the requests that they serve contain everything for a particular tenant without applying label-based access control. For example, listing all rule groups in the ruler will return all rule groups for the tenant even if a label selector in the access policy excludes some of the labels on the rules.

In the case of the ruler this only applies to the HTTP endpoints. All metrics that the ruler generates for alerting or recording rules (alerting rules generate the ALERTS and ALERTS_FOR_STATE metrics) are subject to label-based access control when queried in GEM. Note that GEL recording rules generate metrics which can be sent to a Prometheus-compatible metrics system.

Writing log lines

GEL does not enforce label-based access control on the write requests. This means a logs:write scope in the access policy allows clients to push any log lines without restrictions regarding the labels.

Label policy scope

Label policies are only applied to the log stream selector of the Loki query. They do not apply to label filter expressions. For more information, refer to the Log stream selector and Label filter expression sections in the Loki documentation about Log queries.

Exclude a label

One common use case for creating an LBAC policy is to exclude logs that have a specific label. For example, you can create a label policy that excludes all log lines with the label secret=true by adding a selector with secret!="true" when you create an access policy:

Use multiple selectors

To create a policy that allows someone to access the production and development environments and excludes logs with the label secret=true in the production environment, use multiple selectors.

The selectors {secret!="true", env="prod"} and {env="dev"} enforce the policy:

The selector {secret!="true", env="prod"} matches and returns log lines from the production environment that do not have the secret: true label.
The selector {env="dev"} matches and returns log lines from the development environment, even if they have the secret: true label.

Metamonitoring using Grafana Cloud

Tue, 16 Jul 2024 15:42:20 +0000

Metamonitoring using Grafana Cloud

Grafana Labs champions the value of observability for your software applications and infrastructure. This extends to the Database solutions offered as part of the Grafana Enterprise Stack: namely Grafana Enterprise Metrics, Logs, and Traces. Successful operation of these databases requires comprehensive monitoring and alerting. This allows the teams managing these Databases to swiftly identify issues and diagnose their root causes. Grafana Labs itself maintains robust observability for the Grafana Enterprise Metrics, Logs, and Traces Databases it operates to power Grafana Cloud, our hosted observability-as-a-service offering.

Our recommended approach for monitoring your Grafana Enterprise Databases is to use Grafana Cloud. This approach is the easiest, most robust way to observe your database in accordance with Grafana Labs best practices and is included free of charge with your purchase of a Grafana Enterprise Database.

How it works

Each Grafana Enterprise Database is instrumented for observability by default. Each Database component exposes metrics on a scrapable Prometheus compatible /metrics endpoint and emits logs and traces.

The Helm chart for each Enterprise Database includes an option for meta monitoring. When enabled, the Helm chart deploys Grafana Alloy alongside the Enterprise Database. Alloy is configured to collect metrics and logs from all database components and apply additional metadata, such as extra labels to indicate where the metrics or logs were scraped from. Alloy ONLY collects logs and metrics relevant to the internal system state of the Grafana Enterprise Databases; no other information is collected.

From there, Grafana Alloy forwards the telemetry data to your Grafana Cloud account, where it can be stored and queried to help you understand the health of your Grafana Enterprise Database. You can access this account at any time, and have complete control over who has access to it using Grafana Cloud’s identity and access management functionality. Availability and uptime adhere to the Grafana Cloud SLA.

Additionally, this Grafana Cloud account is preconfigured with Grafana Labs’ recommended dashboards for visualizing the telemetry data collected from your Grafana Enterprise Database. Should you choose, you can also create alerts on the data in this account to notify you when the telemetry data collected from your Grafana Enterprise Database is no longer within expected parameters. Grafana Labs can provide a recommended set of alerts to configure.

If you need to file a support escalation, you can choose to give the Grafana Labs support team access to the telemetry data in your Grafana Cloud account. Historically, Grafana Labs has found that faster access to telemetry data radically reduces the time needed to resolve escalations.