Metrics

Every metric GoRunner records, with its type and labels, and alerts to copy.

Self-hosted

The same metrics, pushed over OpenTelemetry or scraped from /metrics: Connect a monitoring tool. These are the Prometheus names. Tools that keep OpenTelemetry’s names (Datadog, Honeycomb, New Relic, Elastic) drop a counter’s _total.

Filter GoRunner metrics, by group

GoRunner metrics, by group
Metric	Type	What it counts
Runs and steps6
`gorunner_runs_started_total`	counter	Runs that started, by `trigger` and `environment`. More `trigger`: `webhook`, `schedule`, `poll`, `form`, `manual`, `subflow`, `error_trigger`, `run_finished`, `other`, or `test` for a run from the editor. `environment`: `production` or `test`.
`gorunner_runs_finished_total`	counter	Runs that ended or paused, by `status` (`completed`, `failed`, `cancelled`, `paused`) and `environment`.
`gorunner_run_duration_seconds`	histogram	How long a run took, to its end or its pause, by `status`. More Buckets: 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600, 1800, 3600.
`gorunner_steps_finished_total`	counter	Steps that finished, by `piece` (the app, such as `http`; `branch`, `loop` or `code`) and `status`.
`gorunner_step_duration_seconds`	histogram	How long a step took, by `piece`. More Buckets: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600.
`gorunner_runs_paused`	gauge	Runs waiting now, by `reason`: `delay`, `approval` or `webhook`. Counted once a minute.
Queue and workers7
`gorunner_queue_waiting_jobs`	gauge	Jobs waiting for a worker, by `queue`: `jobs` (runs) or `rpc` (editor work).
`gorunner_queue_oldest_wait_seconds`	gauge	How long the oldest waiting job has waited, by `queue`.
`gorunner_dead_letter_jobs`	gauge	Jobs that never reached a worker.
`gorunner_worker_slots_busy`	gauge	This worker’s run slots in use.
`gorunner_worker_slots_total`	gauge	This worker’s run slots (`WORKER_CONCURRENCY`).
`gorunner_redis_memory_used_bytes`	gauge	The memory Redis uses, read by each API process every 15 seconds.
`gorunner_redis_memory_limit_bytes`	gauge	Redis’s `maxmemory`. 0 means no limit.
The API and webhooks3
`gorunner_http_server_request_duration_seconds`	histogram	API requests, by `http_route` (the pattern), `http_request_method`, `http_response_status_code` and `outcome`. More `outcome` is empty for an ordinary request; `workflow_failed` or `workflow_response` when a workflow chose the answer; `not_ready` for a readiness 503. A path no route matched is `unmatched`. Health checks and live streams aren’t measured. Buckets: 0.005 to 30.
`gorunner_webhooks_received_total`	counter	Webhook deliveries, by `outcome`: `accepted`, `rejected` or `sync_reply`.
`gorunner_client_errors_total`	counter	Error reports from people’s browsers.
Connections, events and the audit log5
`gorunner_connection_refresh_failures_total`	counter	OAuth connections whose token couldn’t be renewed, by `piece`.
`gorunner_oauth_token_resource_mismatch_total`	counter	OAuth token requests for a resource the grant doesn’t cover (logged, not refused), by `grant_type`.
`gorunner_events_streamed_total`	counter	Tries to deliver streamed events, by `kind` (`webhook`, `datadog`, `splunk`, `syslog`, `otlp`) and `outcome` (`delivered`, `failed`).
`gorunner_audit_events_total`	counter	Audit log rows written, by `action`.
`gorunner_payload_deletes_total`	counter	Stored step data the cleanup deleted, or failed to and will retry, by `outcome`: `deleted`, `failed`.

Add them up

The queue, dead letters and paused runs describe the whole installation and come from every API process: take max. Worker slots come from each worker: take sum. With METRICS_WORKFLOW_LABELS=true, the run and step metrics also carry workflow_id.

Each process also reports the Go runtime (go_memory_used_bytes, go_goroutine_count…) and, scraped, the process_ family. It describes itself in target_info: service.name, service.instance.id, service.version and gorunner.process (api or worker).

Queries

PromQL

# Share of production runs that failed
sum(rate(gorunner_runs_finished_total{status="failed", environment="production"}[10m]))
  / sum(rate(gorunner_runs_finished_total{status=~"completed|failed|cancelled", environment="production"}[10m]))

# GoRunner's own error rate: 5xx with an empty outcome
sum by (http_route) (rate(gorunner_http_server_request_duration_seconds_count{http_response_status_code=~"5..",outcome=""}[5m]))

# Slowest apps: step duration p95
histogram_quantile(0.95, sum by (le, piece) (rate(gorunner_step_duration_seconds_bucket[10m])))

# Worker saturation
sum(gorunner_worker_slots_busy) / sum(gorunner_worker_slots_total)

Alerts to copy

Prometheus rules; Grafana takes the same expressions. Over OpenTelemetry metrics arrive once a minute, so give rates a window of 5 minutes or more.

gorunner-alerts.yml

groups:
  - name: gorunner
    rules:
      - alert: GoRunnerFailedRunsHigh      # over 2% of production runs failing
        expr: |
          (
            sum(rate(gorunner_runs_finished_total{status="failed", environment="production"}[10m]))
              /
            sum(rate(gorunner_runs_finished_total{status=~"completed|failed|cancelled", environment="production"}[10m]))
          ) > 0.02
          and sum(rate(gorunner_runs_finished_total{environment="production"}[10m])) * 60 > 1
        for: 10m
      - alert: GoRunnerQueueBacklog        # a run waited over 5 minutes
        expr: max(gorunner_queue_oldest_wait_seconds{queue="jobs"}) > 300
        for: 5m
      - alert: GoRunnerWorkersSaturated    # slots over 90% busy
        expr: sum(gorunner_worker_slots_busy) / sum(gorunner_worker_slots_total) > 0.9
        for: 15m
      - alert: GoRunnerNoWorkers
        expr: absent(gorunner_worker_slots_total) or sum(gorunner_worker_slots_total) == 0
        for: 5m
      - alert: GoRunnerStepLatencyHigh     # an app's slowest 5% over 30 s
        expr: |
          histogram_quantile(0.95,
            sum by (le, piece) (rate(gorunner_step_duration_seconds_bucket[10m]))
          ) > 30
        for: 15m
      - alert: GoRunnerDeadLetters
        expr: max(gorunner_dead_letter_jobs) > 0
        for: 5m
      - alert: GoRunnerEventStreamFailures # a streaming destination refusing
        expr: sum by (kind) (rate(gorunner_events_streamed_total{outcome="failed"}[15m])) > 0
        for: 30m

Metrics

Add them up

Queries

Alerts to copy

Panel