This is documentation for the next version of Enterprise logs. For the latest stable release, go to the latest version.
Deploy on Linux
This guide provides a step by step process for installing Grafana Enterprise Logs (GEL) on a Linux machine. It assumes you have access to a Linux machine and the permissions required to deploy a service with network and filesystem access. At the end of this guide you will have deployed a single GEL instance on a single node.
You will need:
- A valid Grafana Labs license with an associated GEL cluster name.
- An s3 compatible object store
Setup an object storage bucket
GEL uses object storage as the backend for its logs storage (both the index and the log content). It also uses object storage for storing various administrative credentials and data related to the state of the system. (GEL and Loki support using the local filesystem as the backend for logs storage as well, but as this is not recommended for production deployments and is not supported for storing admin credentials, this guide will focus on setup with an object storage.)
This guide assumes that you are using Amazon S3 on the AWS
us-east-1 region as your object store. If you plan on using a different region or object storage service, update the storage fields in the configuration file below. Currently, the supported object storage backends are AWS S3, other s3 compliant object stores, Google GCS, or Microsoft Azure’s object storage.
After you have provisioned an object storage backend, be sure to pre-create two buckets:
grafana-logs-data. Those buckets will be referenced in the configuration file of this guide.
Run the following commands on every node as the root user:
# # Add dedicated user and group # groupadd --system enterprise-logs # useradd --system --home-dir /var/lib/enterprise-logs -g enterprise-logs enterprise-logs # # Create directories # mkdir -p /etc/enterprise-logs /var/lib/enterprise-logs /var/lib/enterprise-logs/rules-temp # chown root:enterprise-logs /etc/enterprise-logs # chown enterprise-logs:enterprise-logs /var/lib/enterprise-logs /var/lib/enterprise-logs/rules-temp # chmod 0750 /etc/enterprise-logs /var/lib/enterprise-logs # # Download enterprise-logs binary # curl -Lo /usr/local/bin/enterprise-logs \ https://dl.grafana.com/gel/releases/enterprise-logs-v1.0.0-linux-amd64 # echo "d123d211d40ebbf41af9d7e6c87030bacdc6b62e8490b872aef970a22bb44b04" \ "/usr/local/bin/enterprise-logs" | sha256sum -c # chmod 0755 /usr/local/bin/enterprise-logs # setcap 'cap_net_bind_service=+ep' /usr/local/bin/enterprise-logs # # Set up systemd unit and enable startup on boot # cat > /etc/systemd/system/enterprise-logs.service <<EOF [Unit] After=network.target [Service] User=enterprise-logs Group=enterprise-logs WorkingDirectory=/var/lib/enterprise-logs ExecStart=/usr/local/bin/enterprise-logs \ -config.file=/etc/enterprise-logs/enterprise-logs.yaml \ -log.level=warn \ [Install] WantedBy=default.target EOF # systemctl daemon-reload # systemctl enable enterprise-logs.service
Create a GEL configuration file
Copy the following YAML config to a file called
cluster_name field with the name of the cluster your license was issued for and paste in your s3 credentials for the admin_client.
auth_enabled: true auth: type: enterprise server: http_listen_port: 3100 # TODO: Ensure the cluster name is set to match your Grafana Labs License file cluster_name: # TODO: enter your cluster name license: path: /etc/enterprise-logs/license.jwt ingester: lifecycler: address: 127.0.0.1 ring: kvstore: store: inmemory replication_factor: 1 final_sleep: 0s chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m) max_transfer_retries: 0 # Chunk transfers disabled schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: aws schema: v11 index: prefix: index_ period: 24h storage_config: boltdb_shipper: active_index_directory: /var/lib/enterprise-logs/boltdb-shipper-active cache_location: /var/lib/enterprise-logs/boltdb-shipper-cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space shared_store: s3 aws: region: us-east-1 bucketnames: grafana-logs-data access_key_id: # TODO: insert your key id secret_access_key: # TODO: insert your secret key limits_config: reject_old_samples: true reject_old_samples_max_age: 168h chunk_store_config: max_look_back_period: 0s table_manager: retention_deletes_enabled: false retention_period: 0s ruler: storage: type: local local: directory: /var/lib/enterprise-logs/rules rule_path: /var/lib/enterprise-logs/rules-temp alertmanager_url: http://localhost:9093 ring: kvstore: store: inmemory enable_api: true compactor: working_directory: /var/lib/enterprise-logs/compactor shared_store: s3 admin_client: storage: s3: endpoint: s3.us-east-1.amazonaws.com bucket_name: grafana-logs-admin access_key_id: # TODO: insert your key id secret_access_key: # TODO: insert your secret key type: s3
Move the configuration file and license to the proper directory
Copy the configuration and the license files to all nodes in the GEL cluster:
enterprise-logs.yamlshould be copied to
license.jwtshould be copied to
Generate an admin token
Generate an admin token by running the following on a single node in the cluster:
# su enterprise-logs -c "/usr/local/bin/enterprise-logs \ --config.file=/etc/enterprise-logs/enterprise-logs.yaml \ --bootstrap.license.path=/etc/enterprise-logs/license.jwt \ --log.level=warn \ --target=tokengen" Token created: YWRtaW4tcG9saWN5LWJvb3RzdHJhcC10b2tlbjo8Ujc1IzQyfXBfMjd7fDIwMDRdYVxgeXw=
Start the enterprise-logs service
# systemctl start enterprise-logs.service
Verify your cluster is working
To verify your cluster is working, you can run the following command using the token you generated in the previous step.
curl -u :$API_TOKEN localhost:3100/ready
After running the above command, you should see the following output:
This indicates the ingester component is ready to receive log data.
To integrate your logs cluster with Grafana and a UI to interact with the Admin API, refer to Set up the GEL plugin for Grafana.
Related Enterprise Logs resources
Grafana Enterprise Logs: Logging with security and scale
Join us for this webinar, which will cover: Challenges with logging as organizations scale and the volume of logs explodes, how Grafana Enterprise Logs enables organizations to make logs available to any team members who need them, features available in GEL and how to get access, a live product demo so you can see GEL for the first time
VIDEO: Watch this first-look demo of the new Grafana Enterprise Logs
Based on Loki, Grafana Enterprise Logs is part of the Grafana Enterprise Stack for composing and scaling observability on your own infrastructure.
Introducing Grafana Enterprise Logs, a core part of the Grafana Enterprise Stack integrated observability solution
Powered by the Loki open source project, the Enterprise Logs offering joins metrics and dashboards in our enterprise-ready stack for self-managed observability.