Deploy on Linux
This guide provides a step by step process for installing Grafana Enterprise Logs (GEL) on a Linux machine. It assumes you have access to a Linux machine and the permissions required to deploy a service with network and filesystem access. At the end of this guide you will have deployed a single GEL instance on a single node.
Prerequisites
You will need:
- A valid Grafana Labs license with an associated GEL cluster name.
- An s3 compatible object store
Setup an object storage bucket
GEL uses object storage as the backend for its logs storage (both the index and the log content). It also uses object storage for storing various administrative credentials and data related to the state of the system. (GEL and Loki support using the local filesystem as the backend for logs storage as well, but as this is not recommended for production deployments and is not supported for storing admin credentials, this guide will focus on setup with an object storage.)
This guide assumes that you are using Amazon S3 on the AWS us-east-1
region as your object store. If you plan on using a different region or object storage service, update the storage fields in the configuration file below. Currently, the supported object storage backends are AWS S3, other s3 compliant object stores, Google GCS, or Microsoft Azure’s object storage.
After you have provisioned an object storage backend, be sure to pre-create two buckets: grafana-logs-admin
and grafana-logs-data
. Those buckets will be referenced in the configuration file of this guide.
Install GEL
Run the following commands on every node as the root user:
# Add dedicated user and group
groupadd --system enterprise-logs
useradd --system --home-dir /var/lib/enterprise-logs -g enterprise-logs enterprise-logs
# Create directories
mkdir -p /etc/enterprise-logs /var/lib/enterprise-logs /var/lib/enterprise-logs/rules-temp
chown root:enterprise-logs /etc/enterprise-logs
chown enterprise-logs:enterprise-logs /var/lib/enterprise-logs /var/lib/enterprise-logs/rules-temp
chmod 0750 /etc/enterprise-logs /var/lib/enterprise-logs
# Download enterprise-logs binary
curl -Lo /usr/local/bin/enterprise-logs \
https://dl.grafana.com/gel/releases/enterprise-logs-v1.7.2-linux-amd64
echo "f21e8b13d28235c4f4ffb552ac6cfae4203f488543dc56d95fb00169caf7a2c7" \
"/usr/local/bin/enterprise-logs" | sha256sum -c
chmod 0755 /usr/local/bin/enterprise-logs
setcap 'cap_net_bind_service=+ep' /usr/local/bin/enterprise-logs
# Set up systemd unit and enable startup on boot
cat > /etc/systemd/system/enterprise-logs.service <<EOF
[Unit]
After=network.target
[Service]
User=enterprise-logs
Group=enterprise-logs
WorkingDirectory=/var/lib/enterprise-logs
ExecStart=/usr/local/bin/enterprise-logs \
-config.file=/etc/enterprise-logs/enterprise-logs.yaml \
-log.level=warn \
[Install]
WantedBy=default.target
EOF
systemctl daemon-reload
systemctl enable enterprise-logs.service
Create a GEL configuration file
Copy the following YAML config to a file called enterprise-logs.yaml
.
Update the cluster_name
field with the name of the cluster your license was issued for and paste in your s3 credentials for the admin_client.
auth_enabled: true
auth:
type: enterprise
server:
http_listen_port: 3100
# TODO: Ensure the cluster name is set to match your Grafana Labs License file
cluster_name: # TODO: enter your cluster name
license:
path: /etc/enterprise-logs/license.jwt
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /var/lib/enterprise-logs/boltdb-shipper-active
cache_location: /var/lib/enterprise-logs/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: s3
aws:
region: us-east-1
bucketnames: grafana-logs-data
access_key_id: # TODO: insert your key id
secret_access_key: # TODO: insert your secret key
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
ruler:
storage:
type: local
local:
directory: /var/lib/enterprise-logs/rules
rule_path: /var/lib/enterprise-logs/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
compactor:
working_directory: /var/lib/enterprise-logs/compactor
shared_store: s3
admin_client:
storage:
s3:
endpoint: s3.us-east-1.amazonaws.com
bucket_name: grafana-logs-admin
access_key_id: # TODO: insert your key id
secret_access_key: # TODO: insert your secret key
type: s3
Move the configuration file and license to the proper directory
Copy the configuration and the license files to all nodes in the GEL cluster:
enterprise-logs.yaml
should be copied to/etc/enterprise-logs/enterprise-logs.yaml
license.jwt
should be copied to/etc/enterprise-logs/license.jwt
Generate an admin token
Generate an admin token by running the following on a single node in the cluster:
# su enterprise-logs -c "/usr/local/bin/enterprise-logs \
--config.file=/etc/enterprise-logs/enterprise-logs.yaml \
--bootstrap.license.path=/etc/enterprise-logs/license.jwt \
--log.level=warn \
--target=tokengen"
Token created: YWRtaW4tcG9saWN5LWJvb3RzdHJhcC10b2tlbjo8Ujc1IzQyfXBfMjd7fDIwMDRdYVxgeXw=
Start the enterprise-logs service
# systemctl start enterprise-logs.service
Verify your cluster is working
To verify your cluster is working, you can run the following command using the token you generated in the previous step.
curl -u :$API_TOKEN localhost:3100/ready
After running the above command, you should see the following output:
ready
This indicates the ingester component is ready to receive log data.
Next steps
To integrate your logs cluster with Grafana and a UI to interact with the Admin API, refer to Set up the GEL plugin for Grafana.