By Orch8 Engineering · Editorial policy
Background Job Observability: Metrics That Find Silent Failures
The most useful background-job signals are oldest pending age, time since each worker last polled, claim-to-heartbeat delay, execution duration by handler, retry age, lease conflicts, and unknown external outcomes. Alert on missing progress and breached service objectives, not just exceptions or queue depth.
What is the practical answer?
Background-job observability must detect missing progress, not only thrown errors. Track whether expected jobs were created, the oldest pending age for each queue and handler, time since every worker last polled, claim-to-heartbeat delay, execution duration, retry age, lease conflicts, and unknown external outcomes. Queue depth alone is ambiguous: zero may mean healthy completion or a scheduler that stopped producing work, while a large queue may be normal during a planned batch. Define thresholds per handler because webhook validation and a nightly export have different service expectations. Page when age or failure threatens a user-facing objective; create lower-urgency work for retry drift or capacity pressure. Every alert should include workflow ID, task ID, handler, queue, attempt, lease owner, and a link to the execution record. The monitoring system is complete only when an operator can move from a breached objective to the responsible task boundary without reconstructing the incident from unstructured logs.
Evidence: OpenTelemetry: messaging spans · Google SRE: monitoring distributed systems
A silent queue may emit no error
A scheduler can stop enqueueing, a worker can poll the wrong queue, or every task can wait behind one poisoned item. None of those failures must throw an exception. Queue depth alone also lies: a depth of zero may mean healthy completion or no production at all.
Track the age of expected progress. For scheduled work, record the last successful run. For queues, record oldest pending age and last poll by queue and handler.
Cover the full task lifecycle
| Boundary | Signal | Why it matters |
|---|---|---|
| Enqueue | expected minus created | Find missing production |
| Queue | oldest pending age | Find stalled work |
| Claim | last poll and claim rate | Find routing failures |
| Execute | duration and heartbeat age | Find hangs |
| Complete | success, retry, lease conflict | Find ownership problems |
| Business | unknown or duplicate effects | Find customer harm |
Page on user impact; ticket the rest
Define service objectives per handler because a webhook and a nightly export do not share a useful threshold. Page when age or failure threatens a user-facing promise. Create lower-urgency work for growing retry rates, capacity drift, or missing labels.
Attach workflow ID, task ID, queue, handler, attempt, lease owner, and a run URL to every alert. Orch8 exposes execution boundaries, but operators still need dashboards and paging rules that match their services.
Which signals describe a background job system?
Start with the lifecycle: scheduled, eligible, leased, started, heartbeating, completed, failed, retry-delayed, dead-lettered, cancelled, and reconciled. Emit a counter for transitions and a histogram for time spent between boundaries. A single jobs processed metric mixes admission, execution, retry, and success, so it can rise during an outage.
Measure backlog as both count and age. Count predicts capacity demand; oldest eligible age shows user delay. Track schedule lag separately because a scheduler can fail before a job reaches the queue. For running work, report start latency, execution duration, heartbeat age, lease expirations, and stale completion conflicts. For outcomes, report first-attempt success, eventual success, attempts per completion, permanent failure, and recovery action.
Use stable low-cardinality dimensions: environment, queue, handler, outcome, version, and bounded tenant tier. Never put workflow IDs, customer IDs, URLs, exception messages, or arbitrary provider codes in metric labels. Those belong in traces or logs, where they can be sampled, secured, and queried without exploding time-series cost.
| Question | Primary signal | Important split |
|---|---|---|
| Are jobs being created? | schedule lag and admitted count | scheduler |
| Can workers keep up? | oldest eligible age | queue and handler |
| Are handlers healthy? | duration and outcome | version |
| Is recovery safe? | lease conflicts and unknown effects | effect provider |
How should traces, logs, and workflow history connect?
Create one trace or linked span set per execution attempt, not one unbounded span for a workflow that may wait for weeks. Carry workflow ID, task ID, attempt, definition version, queue, and business intent as structured attributes, subject to privacy policy. Link a retry span to the prior attempt and link an external-effect span to its durable intent record.
Logs should explain decisions the metrics reveal: why a retry was scheduled, which lease token lost a compare-and-swap, whether a timeout left an effect unknown, and which version rejected a stored state. Emit machine-readable error classes and boundary names. Redact tokens, prompt bodies, payment details, and personal data before export, not only in the log viewer.
The workflow history remains the authoritative accepted state. Traces can be sampled or missing; logs can arrive late. An incident view should join all three while clearly labeling which record proves state. Provide direct links from a high-cardinality task search to the trace and from the trace back to the durable execution.
workflow.id
workflow.version
task.id
task.attempt
lease.epoch
queue.name
handler.name
effect.intent_id
release.digestWhich SLOs reflect user-visible reliability?
Define timely completion by workload class. For a webhook, success may mean 99.9% of eligible tasks begin within seconds; for a nightly export, it may mean completion before a business deadline. Exclude planned waits but include platform delay, retries, and capacity shortage. Publish both the threshold and the exact population so a dashboard cannot quietly omit failures.
Pair availability with correctness indicators. Duplicate-prevention conflicts, unknown external effects, lost schedules, and manual force-completions can harm users while completion latency looks healthy. Track a small set as safety SLIs and page on changes that imply irreversible effects. Error budgets should constrain risky releases and capacity reductions, not merely decorate a monthly report.
Test the measurement path. Intentionally create one failed schedule, expired lease, stale completion, and reconciled unknown effect in a non-production environment. Confirm the metric, alert, dashboard, and runbook all identify the same boundary. Telemetry that has never observed a controlled failure is an assumption.
- Timely admission by workload class
- Oldest eligible age below a declared bound
- Eventual completion without operator mutation
- Unknown-effect age below provider reconciliation target
- No unapproved duplicate or missed consequential effect
- Detection and recovery time for injected failures
What should an operator see first?
The overview should answer impact, boundary, and ownership in one screen: delayed jobs and affected workloads, oldest age, schedule lag, worker capacity, failure and retry changes, expired leases, unknown effects, and the current release. Avoid a wall of infrastructure charts before user impact. Every panel should link to the filtered task list or runbook that advances the investigation.
The task detail should show the durable transition timeline, current lease, attempt history, heartbeats, handler version, error classification, external intent status, and guarded controls. Controls must preview the transition and explain duplicate-effect risk. Record every operator action in the same audit trail as worker transitions.
Review dashboards during real incidents and remove panels that never change a decision. Add annotations for deploys, configuration changes, provider incidents, and backfills. The mature outcome is not more telemetry; it is a shorter path from an alert to the failed boundary and a safer recovery choice.
How do you keep telemetry trustworthy over time?
Assign an owner and definition to every SLI. Store the query, unit, population, exclusions, labels, source system, expected delay, retention, and alert consumers in version control. Review changes to task states and retry semantics alongside dashboards; otherwise a release can alter the denominator while the chart continues to look familiar.
Monitor the monitoring path with synthetic transitions and freshness indicators. Alert when metrics stop arriving, cardinality jumps, trace sampling changes, or dashboard queries exceed their time budget. During incident review, note which signals changed a decision and which were misleading. Remove redundant panels and turn missing evidence into a structured field or durable event.
- Version metric and state definitions.
- Expose collection delay and last successful sample.
- Budget label cardinality before rollout.
- Test alert routing and runbook links.
- Audit sensitive fields in logs and traces.
Sources and further reading
Official references support technical claims; community discussions are used only as problem signals.