# Connect a monitoring tool Send GoRunner’s metrics, traces, logs and crash reports to the tool you already use. Everything here is off until you set it. A self-hosted GoRunner sends nothing to GoRunner: no usage figures, no version check, no ping. The license is checked on your server. ## What is sent, and what never is Monitoring says **how** a workflow ran, never **what it carried**. - **Metrics**: Counts and timings: runs, steps by app, waiting jobs, busy worker slots, API times by route. [Metrics](https://docs.gorunner.io/self-host/metrics) - **Traces**: One trace per run: the request, the wait, each step and each call out. [Traces](https://docs.gorunner.io/self-host/traces) - **Logs**: One line per event, with the ids of its organization, workflow, run and step. [Logs](https://docs.gorunner.io/self-host/logs) - **Crash reports**: GoRunner’s own errors, to Sentry, with the line of code. - **Never**: Step inputs and outputs, trigger data, secrets, tokens, passwords, cookies, `Authorization` headers, full addresses and query strings, what people asked Keel, a Code step’s source. A trace shows that the step `notify_sales` failed with `channel_not_found`. It never shows the message. ## Turn it on Either write the settings in `.env` and apply them with `./deploy.sh "$(cat VERSION)"`, or set them on [the Monitoring page](https://docs.gorunner.io/self-host/monitoring-page), where a save reaches every process within seconds. A setting in `.env` always wins. **OpenTelemetry** ``` OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net OTEL_EXPORTER_OTLP_HEADERS=api-key= OTEL_SERVICE_NAME=gorunner ``` **Prometheus** ``` METRICS_ENABLED=true METRICS_TOKEN= ``` **Sentry** ``` SENTRY_DSN=https://@o123456.ingest.sentry.io/7890 ``` **Logs only** ``` LOG_FORMAT=json ``` ## OpenTelemetry Most tools take OpenTelemetry (OTLP). Copy your tool’s address and key into the two settings above, apply, and press **Send test** on the Monitoring page. | Tool | Endpoint | Headers | | --- | --- | --- | | Grafana Cloud | Your stack’s OTLP gateway, such as `https://otlp-gateway-prod-eu-west-2.grafana.net/otlp` | `Authorization=Basic%20` | | Honeycomb | `https://api.honeycomb.io` (EU: `api.eu1.honeycomb.io`) | `x-honeycomb-team=`; metrics also need `x-honeycomb-dataset` in `OTEL_EXPORTER_OTLP_METRICS_HEADERS` | | New Relic | `https://otlp.nr-data.net` (EU: `otlp.eu01.nr-data.net`) | `api-key=` | | Elastic | Your deployment’s OTLP address, from Kibana | `Authorization=ApiKey%20` | | Datadog | Your Datadog Agent, such as `http://datadog-agent:4318` | None. Turn on the Agent’s OTLP receiver and logs. | | A collector | `http://otel-collector:4318` | As your collector needs | Headers are `key=value` pairs separated by commas, a space written `%20`. For gRPC, set `OTEL_EXPORTER_OTLP_PROTOCOL=grpc` and port `4317`. **The Datadog Agent’s settings** Give the Agent these, so it takes OTLP and logs: `Datadog Agent environment` ``` DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318 DD_LOGS_ENABLED=true DD_OTLP_CONFIG_LOGS_ENABLED=true ``` **A collector’s receiver** Your OpenTelemetry Collector receives GoRunner’s data with: `otel-collector.yaml` ``` receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 grpc: endpoint: 0.0.0.0:4317 ``` **Send one kind somewhere else, or turn it off?** Give it its own address: `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `…_METRICS_ENDPOINT` or `…_LOGS_ENDPOINT`. It is used as written, so it ends in its path (`/v1/traces`); the shared one has the paths added for you. To stop one kind, set `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER` or `OTEL_LOGS_EXPORTER` to `none`. The API and the worker call themselves `gorunner-api` and `gorunner-worker`. With `OTEL_SERVICE_NAME` they share it, and `gorunner.process` tells them apart. ## Prometheus To scrape instead, turn on a `/metrics` page on port `9464` of every API and worker container (`9465` for the API’s second process), inside the Docker network only. It needs a token of 16 characters or more (the settings above); without one it answers `503`. `prometheus.yml` ``` scrape_configs: - job_name: gorunner authorization: type: Bearer credentials_file: /etc/prometheus/gorunner-metrics-token dns_sd_configs: - names: [api, worker] type: A port: 9464 - names: [api] # the API's second process type: A port: 9465 ``` The Monitoring page can make the token for you, and gives this config with your processes filled in. ## Sentry Crashes and errors, from the server and from people’s browsers, relayed through GoRunner. Requests, headers, cookies and addresses are removed first. Set `SENTRY_DSN` to your project’s DSN. The environment is `production` unless you set `SENTRY_ENVIRONMENT`. ## Logs only No tool at all: `LOG_FORMAT=json` makes each line one JSON object for any log collector, such as Vector, Fluent Bit, Promtail or CloudWatch. Docker keeps at most 50 MB per container, so ship them if you need them longer. ## Sampling and cost - **Traces**: One per run, one span per step. On a busy installation keep one in ten: set `OTEL_TRACES_SAMPLER` to `parentbased_traceidratio` and `OTEL_TRACES_SAMPLER_ARG` to `0.1`. Metrics and logs still count every run. - **Metrics**: Cost per series. `METRICS_WORKFLOW_LABELS=true` adds each workflow’s id: 2,000 workflows make each run metric 2,000 series. Off by default. - **Logs**: `LOG_LEVEL=warn` sends only what needs a look. **See where time goes in the database?** `TELEMETRY_DB_SPANS=true` adds a span for every Postgres query and Redis command, without the query’s text. It multiplies the spans, so turn it on while you look. `TELEMETRY_PROPAGATE_OUTBOUND=true` adds `traceparent` to the calls steps make, so your own services can join the trace. Both need a restart. ## Access logs The app logs each request’s route, status and time, but not who asked. Both of these are off by default; apply with `./deploy.sh "$(cat VERSION)"`. - `ACCESS_LOG=true`: Adds the caller’s address (from a trusted proxy) and browser to that line. - `GORUNNER_ACCESS_LOG=on`: With automatic HTTPS, Caddy logs each request’s caller too. Never a query string, a header, or more than 8 characters of a webhook’s token. ## Check it works Press **Send test** on the Monitoring page for each connection. At start, each process logs `telemetry started`, with each connection `off`, `on` or its host: ```bash docker compose logs api worker | grep "telemetry started" ``` GoRunner never waits for your tool. A batch it can’t send is dropped, the next one tries again, and runs carry on. | You see | It means | | --- | --- | | `otlp-traces=off` in the log | No address reached the containers. Check `.env`, then `./deploy.sh "$(cat VERSION)"`. | | `traces export to … failed: …` | Your tool refused or didn’t answer. The end of the line says why, such as `401 Unauthorized`. | | Metrics, but no `gorunner_queue_…` | The queue figures come from the API, worker slots from each worker: check that both send. | | `401` or `403` | The key or header is wrong. A space is `%20`. | | `404` | A wrong path. The shared address has none; a per-kind one ends in `/v1/…`. | | `connection refused`, `no such host` | Wrong address, or the tool isn’t on the Docker network. `localhost` is the container itself. | | `misconfigured and stays off` | A setting can’t be used. The log names it. | | `/metrics` answers `503` or `401` | No `METRICS_TOKEN`, or the scraper sends the wrong one. | **Try it on your server first?** The install folder’s `observability/` kit runs Grafana’s all-in-one image: a collector, Prometheus, Tempo, Loki and Grafana, with a ready **GoRunner** dashboard. It is for trying: it has no sign-in and keeps nothing. Add `OTEL_EXPORTER_OTLP_ENDPOINT=http://lgtm:4318` to `.env`, then start it with GoRunner: ```bash docker compose -f docker-compose.yml \ -f observability/docker-compose.observability.yml up -d ``` Its ports are on `127.0.0.1` only. Open a tunnel, then `http://localhost:3000`, which opens on the GoRunner dashboard: ```bash ssh -L 3000:127.0.0.1:3000 you@your-server ``` Never publish the port: anyone who opens it is Grafana’s admin. ## Related - [The Monitoring page](https://docs.gorunner.io/self-host/monitoring-page): Set it up in Platform, and send tests · 4 min - [Metrics](https://docs.gorunner.io/self-host/metrics): Every metric, and alerts to copy - [Health checks](https://docs.gorunner.io/self-host/health-checks): /api/ready for a load balancer Source: https://docs.gorunner.io/self-host/monitoring-tools