What are traces?

A user on your website enters their email address into a form to sign up for your mailing list. They click Enter.

The user’s email address is data that flows through your system. In a cloud computing world, it is possible that clicking that one button causes data to touch multiple nodes across your cluster of microservices.

The email address may be sent to a verification algorithm sitting in a microservice that exists solely for that purpose. If it passes the check, the information is stored in a database.

Along the way an anonymization node strips personally-identifying data from the address and sends metadata collected to a marketing qualifying algorithm to determine whether the request was sent from a targeted part of the internet.

Services respond and data flows back from each, sometimes triggering new events across the system. Along the way, logs are written in various nodes with a time stamp showing when the info passed through.

Finally, the request and response activity ends.

Setting up tracing adds a label to all of these events. The label is generated when the request is initiated and that same label is applied to every single event as the request and response generate activity across the system.

That label enables one to trace, or follow a request as it flows from node to node, service to microservice to lambda function to wherever it goes in your chaotic, cloud computing system and back again. This is recorded and displayed as spans. Each span is the amount of time between a request and response from a specific service in the request’s journey through your system.

Here’s an example showing two pages in Grafana Cloud. The first, on the left, shows a query using the Explore feature. In the query results you can see a traceID field that was added to an application. That field contains a Tempo label. The second page, on the right, uses the same Explore feature to perform a Tempo search using that traceID label. It then shows a set of spans as horizontal bars, each bar denoting a different part of the system.

The label is applied to activities recorded as metrics and as logs. A trace can be visualized to give a graphic representation of how long it takes for each step in the data flow pathway to complete, where new requests are initiated and end, and how your system responds. You can see bottlenecks, often in places you never would have anticipated or found without this ability to trace the request flow.

See Set up and use Grafana Cloud Traces to get started.