# Metrics

Every metric GoRunner records, with its type and labels, and alerts to copy.

The same metrics, pushed over OpenTelemetry or scraped from `/metrics`: [Connect a monitoring tool](https://docs.gorunner.io/self-host/monitoring-tools). These are the Prometheus names. Tools that keep OpenTelemetry’s names (Datadog, Honeycomb, New Relic, Elastic) drop a counter’s `_total`.

| Metric | Type | What it counts |
| --- | --- | --- |
| **Runs and steps** |   |   |
| `gorunner_runs_started_total` | counter | Runs that started, by `trigger` and `environment`. `trigger`: `webhook`, `schedule`, `poll`, `form`, `manual`, `subflow`, `error_trigger`, `run_finished`, `other`, or `test` for a run from the editor. `environment`: `production` or `test`. |
| `gorunner_runs_finished_total` | counter | Runs that ended or paused, by `status` (`completed`, `failed`, `cancelled`, `paused`) and `environment`. |
| `gorunner_run_duration_seconds` | histogram | How long a run took, to its end or its pause, by `status`. Buckets: 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600, 1800, 3600. |
| `gorunner_steps_finished_total` | counter | Steps that finished, by `piece` (the app, such as `http`; `branch`, `loop` or `code`) and `status`. |
| `gorunner_step_duration_seconds` | histogram | How long a step took, by `piece`. Buckets: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600. |
| `gorunner_runs_paused` | gauge | Runs waiting now, by `reason`: `delay`, `approval` or `webhook`. Counted once a minute. |
| **Queue and workers** |   |   |
| `gorunner_queue_waiting_jobs` | gauge | Jobs waiting for a worker, by `queue`: `jobs` (runs) or `rpc` (editor work). |
| `gorunner_queue_oldest_wait_seconds` | gauge | How long the oldest waiting job has waited, by `queue`. |
| `gorunner_dead_letter_jobs` | gauge | Jobs that never reached a worker. |
| `gorunner_worker_slots_busy` | gauge | This worker’s run slots in use. |
| `gorunner_worker_slots_total` | gauge | This worker’s run slots (`WORKER_CONCURRENCY`). |
| `gorunner_redis_memory_used_bytes` | gauge | The memory Redis uses, read by each API process every 15 seconds. |
| `gorunner_redis_memory_limit_bytes` | gauge | Redis’s `maxmemory`. 0 means no limit. |
| **The API and webhooks** |   |   |
| `gorunner_http_server_request_duration_seconds` | histogram | API requests, by `http_route` (the pattern), `http_request_method`, `http_response_status_code` and `outcome`. `outcome` is empty for an ordinary request; `workflow_failed` or `workflow_response` when a workflow chose the answer; `not_ready` for a readiness 503. A path no route matched is `unmatched`. Health checks and live streams aren’t measured. Buckets: 0.005 to 30. |
| `gorunner_webhooks_received_total` | counter | Webhook deliveries, by `outcome`: `accepted`, `rejected` or `sync_reply`. |
| `gorunner_client_errors_total` | counter | Error reports from people’s browsers. |
| **Connections, events and the audit log** |   |   |
| `gorunner_connection_refresh_failures_total` | counter | OAuth connections whose token couldn’t be renewed, by `piece`. |
| `gorunner_oauth_token_resource_mismatch_total` | counter | OAuth token requests for a resource the grant doesn’t cover (logged, not refused), by `grant_type`. |
| `gorunner_events_streamed_total` | counter | Tries to deliver streamed events, by `kind` (`webhook`, `datadog`, `splunk`, `syslog`, `otlp`) and `outcome` (`delivered`, `failed`). |
| `gorunner_audit_events_total` | counter | Audit log rows written, by `action`. |
| `gorunner_payload_deletes_total` | counter | Stored step data the cleanup deleted, or failed to and will retry, by `outcome`: `deleted`, `failed`. |

## Add them up

The queue, dead letters and paused runs describe the whole installation and come from every API process: take `max`. Worker slots come from each worker: take `sum`. With `METRICS_WORKFLOW_LABELS=true`, the run and step metrics also carry `workflow_id`.

Each process also reports the Go runtime (`go_memory_used_bytes`, `go_goroutine_count`…) and, scraped, the `process_` family. It describes itself in `target_info`: `service.name`, `service.instance.id`, `service.version` and `gorunner.process` (`api` or `worker`).

## Queries

`PromQL`

```
# Share of production runs that failed
sum(rate(gorunner_runs_finished_total{status="failed", environment="production"}[10m]))
  / sum(rate(gorunner_runs_finished_total{status=~"completed|failed|cancelled", environment="production"}[10m]))

# GoRunner's own error rate: 5xx with an empty outcome
sum by (http_route) (rate(gorunner_http_server_request_duration_seconds_count{http_response_status_code=~"5..",outcome=""}[5m]))

# Slowest apps: step duration p95
histogram_quantile(0.95, sum by (le, piece) (rate(gorunner_step_duration_seconds_bucket[10m])))

# Worker saturation
sum(gorunner_worker_slots_busy) / sum(gorunner_worker_slots_total)
```

## Alerts to copy

Prometheus rules; Grafana takes the same expressions. Over OpenTelemetry metrics arrive once a minute, so give rates a window of 5 minutes or more.

`gorunner-alerts.yml`

```
groups:
  - name: gorunner
    rules:
      - alert: GoRunnerFailedRunsHigh      # over 2% of production runs failing
        expr: |
          (
            sum(rate(gorunner_runs_finished_total{status="failed", environment="production"}[10m]))
              /
            sum(rate(gorunner_runs_finished_total{status=~"completed|failed|cancelled", environment="production"}[10m]))
          ) > 0.02
          and sum(rate(gorunner_runs_finished_total{environment="production"}[10m])) * 60 > 1
        for: 10m
      - alert: GoRunnerQueueBacklog        # a run waited over 5 minutes
        expr: max(gorunner_queue_oldest_wait_seconds{queue="jobs"}) > 300
        for: 5m
      - alert: GoRunnerWorkersSaturated    # slots over 90% busy
        expr: sum(gorunner_worker_slots_busy) / sum(gorunner_worker_slots_total) > 0.9
        for: 15m
      - alert: GoRunnerNoWorkers
        expr: absent(gorunner_worker_slots_total) or sum(gorunner_worker_slots_total) == 0
        for: 5m
      - alert: GoRunnerStepLatencyHigh     # an app's slowest 5% over 30 s
        expr: |
          histogram_quantile(0.95,
            sum by (le, piece) (rate(gorunner_step_duration_seconds_bucket[10m]))
          ) > 30
        for: 15m
      - alert: GoRunnerDeadLetters
        expr: max(gorunner_dead_letter_jobs) > 0
        for: 5m
      - alert: GoRunnerEventStreamFailures # a streaming destination refusing
        expr: sum by (kind) (rate(gorunner_events_streamed_total{outcome="failed"}[15m])) > 0
        for: 30m
```

Related: [Traces](https://docs.gorunner.io/self-host/traces) · [Logs](https://docs.gorunner.io/self-host/logs) · [The Health page](https://docs.gorunner.io/self-host/health) · [Connect a monitoring tool](https://docs.gorunner.io/self-host/monitoring-tools)

Source: https://docs.gorunner.io/self-host/metrics