Observability in under 5 seconds: Reflecting on a year of grafana/otel-lgtm

With grafana/otel-lgtm, observability is just one Docker command away.

Over the past year, grafana/otel-lgtm has simplified observability setups, helping developers get a complete OpenTelemetry stack running in under five seconds. With integrations for metrics, logs, traces, and now profiles via Grafana Pyroscope, it has become a go-to solution for demos, development, and testing, as evidenced by its growing community (1k stars on GitHub and growing!) and notable adopters.

In this blog, we’ll tell you all you need to know about grafana/otel-lgtm, from how to get started to what’s changed over the past 15 months.

What is grafana/otel-lgtm?

grafana/otel-lgtm is a Docker image that provides a complete open source OpenTelemetry solution for observability by bundling several key tools:

• OpenTelemetry Collector: Receives and forwards telemetry signals to observability backends.
• Prometheus: Stores and queries your application’s metrics (e.g., request rates, error counts, system health).
• Grafana Loki: Stores and queries your application’s logs.
• Grafana Tempo: Stores and queries traces, which show the path of a request as it journeys through your different services.
• Grafana Pyroscope: Stores and queries profiles, helping you understand which parts of your code are consuming the most resources (like CPU time or memory).
• Grafana: Visualizes all this data in dashboards, allowing you to see metrics, logs, traces, and profiles in one place.

Quickstart

Getting started is easy. The OpenTelemetry Collector receives signals on ports 4317 (gRPC) and 4318 (HTTP). It then forwards:

• Metrics to Prometheus
• Logs to Loki
• Traces to Tempo
• Profiles to Pyroscope

Grafana is pre-configured with these four data sources and exposes its Web UI on port 3000. Just follow these steps:

1. Run the following command to start the stack:

docker run --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

1. Wait for the log message The OpenTelemetry collector and the Grafana LGTM stack are up and running. or for the /tmp/ready file to appear, which will appear in a couple of seconds.
2. Start your OpenTelemetry-exporting application. Example applications are available in the repository.
3. Open http://localhost:3000/ in your browser to view metrics, logs, traces, and profiles.

You can read more about using grafana/otel-lgtm in the announcement blog post from last year.

The grafana/otel-lgtm Docker image is for development, demo, and testing. For production, refer to Grafana Cloud Application Observability.

One year later

grafana/otel-lgtm has made it easier for projects to create a demo setup for observability. Some notable adopters include:

• Deno (JavaScript)
• Quarkus (Java, features programmatic end-to-end tests)
• Bootzooka (Scala)
• Roadster (Rust)
• Embrace Observability (Kotlin)

Even a free course from Aalto University includes lessons on the LGTM stack.

A big thank you to the community for contributions like:

• Node.js example by @GhassenRjab
• PowerShell support by @tebeco 
• Publishing on GitHub Registry @Rajpratik71
• … and every one of the 34 contributors 

The fast startup time is not just great for demonstration and manual troubleshooting—but also for automated testing. Therefore, we created OATs, a no-code test framework based on grafana/otel-lgtm to test your application using YAML. For example:

Copy

expected:
traces:
  - traceql: '{ span.http.route = "/rolldice/{player?}" }'
    spans:
      - name: "GET /rolldice/{player?}"
        attributes:
          otel.library.name: Microsoft.AspNetCore

What’s new

We’re currently up to version 0.11.5, and we’ve worked hard to incorporate multiple improvements to grafana/otel-lgtm. Here are just some of the changes we’ve made since the initial release in March 2024.

Faster startup time

The most requested feature was faster startup. We reduced the startup time from ~60 seconds to under 5 seconds through these key improvements:

• Reducing the Prometheus health check scrape interval from 1 minute to 1 second.
• Using the health check extension for faster readiness detection.
• Optimizing database configurations for DNS lookups and readiness.

These changes are optimized for fast startup, not for high scalability.

Grafana Pyroscope integration

Profiling is the latest OpenTelemetry signal, and we’ve added Grafana Pyroscope as the fourth database. Profiling helps you understand which parts of your code are consuming the most resources (like CPU time or memory). This is crucial for identifying performance bottlenecks and optimizing your application. This allows you to seamlessly blend traces and profiles to optimize your critical paths.

To try the eBPF profiler example:

1. Change to the example directory: cd examples/ebpf-profiler
2. Start the stack: docker compose up --remove-orphans --build
3. Open Profiles Drilldown in Grafana.
4. Filter by process_executable_name = rolldice and explore the flame graph.

Profiles Drilldown provides an intuitive interface for exploring your profile data to view service performance, optimize processes, and diagnose issues. Learn more here.

In the example flame graph above, we can see that “rolldice” takes 1.2 seconds to complete. Of that, 1.15 seconds are spent in the “roll” function, which in turn is caused by the “Since” function from Go standard library. As “Since” is supposed to be fast, it’s a good idea to check out this particular piece of code, which in this case has been made deliberately slow for demonstration purposes. 

Copy

start := time.Now()
for time.Since(start) < 1*time.Second {
  // busy wait
}

Get started today

Grafana Labs remains dedicated to the OpenTelemetry project, continuously enhancing grafana/otel-lgtm and other tools to empower users in their observability journey.

Try grafana/docker-otel-lgtm today and see for yourself. And if you’re looking to help this project grow, feel free to jump in and start contributing today!

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!