GoRunner docs
Search or askSearch⌘K

Connect a monitoring tool

Send GoRunner’s metrics, traces, logs and crash reports to the tool you already use.

10 min · Self-hosted

Everything here is off until you set it. A self-hosted GoRunner sends nothing to GoRunner: no usage figures, no version check, no ping. The license is checked on your server.

What is sent, and what never is

Monitoring says how a workflow ran, never what it carried.

Metrics
Counts and timings: runs, steps by app, waiting jobs, busy worker slots, API times by route. Metrics
Traces
One trace per run: the request, the wait, each step and each call out. Traces
Logs
One line per event, with the ids of its organization, workflow, run and step. Logs
Crash reports
GoRunner’s own errors, to Sentry, with the line of code.
Never
Step inputs and outputs, trigger data, secrets, tokens, passwords, cookies, Authorization headers, full addresses and query strings, what people asked Keel, a Code step’s source.

A trace shows that the step notify_sales failed with channel_not_found. It never shows the message.

Turn it on

Either write the settings in .env and apply them with ./deploy.sh "$(cat VERSION)", or set them on the Monitoring page, where a save reaches every process within seconds. A setting in .env always wins.

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net
OTEL_EXPORTER_OTLP_HEADERS=api-key=<your license key>
OTEL_SERVICE_NAME=gorunner

OpenTelemetry

Most tools take OpenTelemetry (OTLP). Copy your tool’s address and key into the two settings above, apply, and press Send test on the Monitoring page.

ToolEndpointHeaders
Grafana CloudYour stack’s OTLP gateway, such as https://otlp-gateway-prod-eu-west-2.grafana.net/otlpAuthorization=Basic%20<base64 of instance-id:token>
Honeycombhttps://api.honeycomb.io (EU: api.eu1.honeycomb.io)x-honeycomb-team=<key>; metrics also need x-honeycomb-dataset in OTEL_EXPORTER_OTLP_METRICS_HEADERS
New Relichttps://otlp.nr-data.net (EU: otlp.eu01.nr-data.net)api-key=<license key>
ElasticYour deployment’s OTLP address, from KibanaAuthorization=ApiKey%20<key>
DatadogYour Datadog Agent, such as http://datadog-agent:4318None. Turn on the Agent’s OTLP receiver and logs.
A collectorhttp://otel-collector:4318As your collector needs

Headers are key=value pairs separated by commas, a space written %20. For gRPC, set OTEL_EXPORTER_OTLP_PROTOCOL=grpc and port 4317.

The Datadog Agent’s settings

Give the Agent these, so it takes OTLP and logs:

Datadog Agent environment
DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318
DD_LOGS_ENABLED=true
DD_OTLP_CONFIG_LOGS_ENABLED=true
A collector’s receiver

Your OpenTelemetry Collector receives GoRunner’s data with:

otel-collector.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317
Send one kind somewhere else, or turn it off?

Give it its own address: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, …_METRICS_ENDPOINT or …_LOGS_ENDPOINT. It is used as written, so it ends in its path (/v1/traces); the shared one has the paths added for you.

To stop one kind, set OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER or OTEL_LOGS_EXPORTER to none.

The API and the worker call themselves gorunner-api and gorunner-worker. With OTEL_SERVICE_NAME they share it, and gorunner.process tells them apart.

Prometheus

To scrape instead, turn on a /metrics page on port 9464 of every API and worker container (9465 for the API’s second process), inside the Docker network only. It needs a token of 16 characters or more (the settings above); without one it answers 503.

prometheus.yml
scrape_configs:
  - job_name: gorunner
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/gorunner-metrics-token
    dns_sd_configs:
      - names: [api, worker]
        type: A
        port: 9464
      - names: [api]          # the API's second process
        type: A
        port: 9465

The Monitoring page can make the token for you, and gives this config with your processes filled in.

Sentry

Crashes and errors, from the server and from people’s browsers, relayed through GoRunner. Requests, headers, cookies and addresses are removed first. Set SENTRY_DSN to your project’s DSN. The environment is production unless you set SENTRY_ENVIRONMENT.

Logs only

No tool at all: LOG_FORMAT=json makes each line one JSON object for any log collector, such as Vector, Fluent Bit, Promtail or CloudWatch. Docker keeps at most 50 MB per container, so ship them if you need them longer.

Sampling and cost

Traces
One per run, one span per step. On a busy installation keep one in ten: set OTEL_TRACES_SAMPLER to parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG to 0.1. Metrics and logs still count every run.
Metrics
Cost per series. METRICS_WORKFLOW_LABELS=true adds each workflow’s id: 2,000 workflows make each run metric 2,000 series. Off by default.
Logs
LOG_LEVEL=warn sends only what needs a look.
See where time goes in the database?

TELEMETRY_DB_SPANS=true adds a span for every Postgres query and Redis command, without the query’s text. It multiplies the spans, so turn it on while you look.

TELEMETRY_PROPAGATE_OUTBOUND=true adds traceparent to the calls steps make, so your own services can join the trace. Both need a restart.

Access logs

The app logs each request’s route, status and time, but not who asked. Both of these are off by default; apply with ./deploy.sh "$(cat VERSION)".

ACCESS_LOG=true
Adds the caller’s address (from a trusted proxy) and browser to that line.
GORUNNER_ACCESS_LOG=on
With automatic HTTPS, Caddy logs each request’s caller too. Never a query string, a header, or more than 8 characters of a webhook’s token.

Check it works

Press Send test on the Monitoring page for each connection. At start, each process logs telemetry started, with each connection off, on or its host:

bash
docker compose logs api worker | grep "telemetry started"

GoRunner never waits for your tool. A batch it can’t send is dropped, the next one tries again, and runs carry on.

You seeIt means
otlp-traces=off in the logNo address reached the containers. Check .env, then ./deploy.sh "$(cat VERSION)".
traces export to … failed: …Your tool refused or didn’t answer. The end of the line says why, such as 401 Unauthorized.
Metrics, but no gorunner_queue_…The queue figures come from the API, worker slots from each worker: check that both send.
401 or 403The key or header is wrong. A space is %20.
404A wrong path. The shared address has none; a per-kind one ends in /v1/….
connection refused, no such hostWrong address, or the tool isn’t on the Docker network. localhost is the container itself.
misconfigured and stays offA setting can’t be used. The log names it.
/metrics answers 503 or 401No METRICS_TOKEN, or the scraper sends the wrong one.
Try it on your server first?

The install folder’s observability/ kit runs Grafana’s all-in-one image: a collector, Prometheus, Tempo, Loki and Grafana, with a ready GoRunner dashboard. It is for trying: it has no sign-in and keeps nothing.

Add OTEL_EXPORTER_OTLP_ENDPOINT=http://lgtm:4318 to .env, then start it with GoRunner:

bash
docker compose -f docker-compose.yml \
  -f observability/docker-compose.observability.yml up -d

Its ports are on 127.0.0.1 only. Open a tunnel, then http://localhost:3000, which opens on the GoRunner dashboard:

bash
ssh -L 3000:127.0.0.1:3000 you@your-server

Never publish the port: anyone who opens it is Grafana’s admin.

Checked against 12d85b3b