Sometimes using a tracing system is intimidating because it seems like you need complex application instrumentation or a span ingestion pipeline in order to push spans. This guide aims to show an extremely basic technique for pushing spans with http/json from a Bash script using the Zipkin receiver.
Starting Tempo
Let’s first start Tempo with the Zipkin receiver configured. In order to do this create a config file like so:
server:
http_listen_port: 3200
distributor:
receivers:
zipkin:
storage:
trace:
backend: local
local:
path: /tmp/tempo/blocks
and run Tempo using it:
docker run -p 9411:9411 -p 3200:3200 -v $(pwd)/config.yaml:/config.yaml grafana/tempo:latest -config.file /config.yaml
Pushing Spans
Now that Tempo is running and listening on port 9411 for Zipkin spans let’s push a span to it using curl
.
curl -X POST http://localhost:9411 -H 'Content-Type: application/json' -d '[{
"id": "1234",
"traceId": "0123456789abcdef",
"timestamp": 1608239395286533,
"duration": 100000,
"name": "span from bash!",
"tags": {
"http.method": "GET",
"http.path": "/api"
},
"localEndpoint": {
"serviceName": "shell script"
}
}]'
Note that the timestamp
field is in microseconds and was obtained by running date +%s%6N
. The duration
field is also in microseconds and so 100000 is 100 milliseconds.
Retrieving Traces
The easiest way to get the trace is to execute a simple curl command to Tempo. The returned format is OTLP.
curl http://localhost:3200/api/traces/0123456789abcdef
{"batches":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"shell script"}}]},"instrumentationLibrarySpans":[{"spans":[{"traceId":"AAAAAAAAAAABI0VniavN7w==","spanId":"AAAAAAAAEjQ=","name":"span from bash!","startTimeUnixNano":"1608239395286533000","endTimeUnixNano":"1608239395386533000","attributes":[{"key":"http.path","value":{"stringValue":"/api"}},{"key":"http.method","value":{"stringValue":"GET"}}]}]}]}]}
However, staring at a json blob in bash is not very fun. Let’s start up Tempo query so we can visualize our trace. Tempo query is Jaeger Query with a GRPC Plugin that allows it to query Tempo.
docker run --env BACKEND=localhost:3200 --net host grafana/tempo-query:latest
And open http://localhost:16686/trace/0123456789abcdef
in the browser of your choice to see:
More Spans!
Now that we have the basics down it’s easy to continue building our trace. By specifying the same trace id and a parent span id we can start building a trace.
curl -X POST http://localhost:9411 -H 'Content-Type: application/json' -d '[{
"id": "5678",
"traceId": "0123456789abcdef",
"parentId": "1234",
"timestamp": 1608239395316533,
"duration": 100000,
"name": "child span from bash!",
"localEndpoint": {
"serviceName": "shell script"
}
}]'
And now the UI shows:
Spans from everything!
Tracing is not limited to enterprise languages with complex frameworks. As you can see it’s easy to store and track events from your js, python or bash scripts. You can use Tempo/distributed tracing today to trace CI pipelines, long running bash processes, python data processing flows or anything else you can think of.
Happy tracing!
Related Tempo resources
GrafanaCONline 2021
Be the first to learn about exciting next-generation features in Grafana 8.0, be inspired by what community members are building, and attend expert-led sessions and workshops on Grafana, Prometheus, Loki logs, and more.
Getting started with tracing and Grafana Tempo
In this session, we will use an instrumented application to demonstrate how to use logs and Prometheus exemplars to find traces effectively in Tempo. The demo will hit on the basics of operating Tempo and reveal how Tempo allows us to scale tracing as far as possible with less operational cost and complexity than ever before.
Tracing made simple with Grafana
Dive into the new options for viewing tracing data inside Grafana and learn how to make tracing an integral part of your observability strategy.