# Traces

The spans GoRunner sends for a run, and every attribute they carry.

A trace is the journey of one request or run; each bar in it is a span. GoRunner sends them when an OTLP address is set ([Connect a monitoring tool](https://docs.gorunner.io/self-host/monitoring-tools)). One run reads like this:

`A trace`

```
POST /api/webhooks/{token}            api     server     18 ms
└─ enqueue run                        api     producer    2 ms
   └─ run New lead → CRM              worker  consumer  2.24 s   queue.wait_ms = 142
      ├─ step enrich_lead             worker  internal   325 ms
      │  └─ HTTP GET                  worker  client     310 ms   api.clearbit.com:443 · 200
      ├─ step create_contact          worker  internal   607 ms
      └─ step notify_sales            worker  internal    1.3 s   failed · attempt 2
```

GoRunner’s own attributes start with `gorunner.`; HTTP and database spans use OpenTelemetry’s names. Error messages are scrubbed of anything like a credential and cut to 256 characters.

## The spans

| Span | Kind, where | Attributes |
| --- | --- | --- |
| `GET /api/workflows/{id}` | server, API | `http.request.method`, `http.route` (the pattern), `http.response.status_code`, `gorunner.request_id`, and the organization and workspace once known. A 5xx is an error. An incoming `traceparent` is honoured. |
| `enqueue run` | producer, API or worker | `gorunner.run.id`, `gorunner.queue`. `error.type` `enqueue_failed` when refused. |
| `run <workflow name>` | consumer, worker (internal for a sub-flow) | `gorunner.workflow.id`, `.name`, `.version`; `gorunner.run.id`, `.trigger`, `.environment`, `.status`, `.retry`; `gorunner.queue.wait_ms`; the organization and workspace. |
| `step <step name>` | internal, worker | `gorunner.step.name`, `.type`, `.attempt`, `.status`; `gorunner.piece.id`, `.action`, `.version`. A loop adds `gorunner.loop.iterations`. |
| `iteration` | internal, worker | `gorunner.loop.index`, for a loop’s first 50 iterations. |
| `HTTP GET` | client, worker or API | `server.address`, `server.port`, the status, and `error.type` on failure. The host only. |
| `pieceops <operation>` | consumer, worker | Editor work: a Test step, a connection test, a dropdown. `gorunner.pieceops.op`, the queue and its wait. |
| Postgres, Redis | client, API and worker | Only with `TELEMETRY_DB_SPANS=true`: the statement’s first word or the command, never the statement. |

A failed run or step has status Error and `gorunner.error.message`.

## How runs join their traces

- **Started by a request**: A webhook, Run or the API: the request, the enqueue and the run are one trace.
- **Scheduled or polled**: A trace of its own.
- **Resumed**: After an approval or a delay: a new trace, linked to the one it paused in.
- **A sub-flow**: A child of the step that called it.
- **Handed back in an update**: Keeps the trace it was born in.

A span never carries a step’s input or output, trigger data, a setting’s value, a Code step’s source, a call’s path, query, headers or body, or a database statement. See [What is sent, and what never is](https://docs.gorunner.io/self-host/monitoring-tools#what-is-sent-and-what-never-is).

## Where it came from

Every span carries `service.name`, `service.version`, `service.instance.id`, `gorunner.process` (`api` or `worker`), `deployment.environment.name` and your `OTEL_RESOURCE_ATTRIBUTES`. A TraceQL search for failed runs on the workers:

`TraceQL`

```
{ resource.gorunner.process = "worker" && name =~ "run .*" && status = error }
```

Sampling keeps or drops a trace whole; log lines carry the `trace_id` either way. Outbound HTTP spans need tracing on when a process starts.

Related: [Metrics](https://docs.gorunner.io/self-host/metrics) · [Logs](https://docs.gorunner.io/self-host/logs) · [Sampling and cost](https://docs.gorunner.io/self-host/monitoring-tools#sampling-and-cost)

Source: https://docs.gorunner.io/self-host/traces