Skip to content

Cluster & Multi-Node

Scale Orch8 horizontally by running multiple engine nodes. Work is distributed automatically via database-level coordination — no external queue or service mesh required. Graceful drain ensures zero-downtime deployments.

Architecture

Orch8 uses a shared-nothing, database-coordinated architecture. Each node is an independent process that claims work from the same database. No leader election, no consensus protocol, no external coordination service.

Shared Database

PostgreSQL serves as the single source of truth. Nodes claim work using row-level locks (SELECT FOR UPDATE SKIP LOCKED).

No Single Point of Failure

Any node can go down. Unclaimed work is picked up by remaining nodes. No work is lost.

Heartbeat Registration

Each node registers itself and sends periodic heartbeats. Stale nodes (missed heartbeats) have their claimed work reclaimed.

Horizontal Scaling

Add nodes to handle more concurrent executions. Remove nodes when load decreases. No reconfiguration needed.

Node Lifecycle

1

Startup

Node generates a unique ID, registers in the cluster_nodes table, and starts its scheduler loop.

2

Heartbeat

Periodic heartbeat updates the node's last_seen_at timestamp. Other nodes can detect if this node becomes unresponsive.

3

Claim & Execute

Scheduler claims instances (SELECT FOR UPDATE SKIP LOCKED), executes steps, and writes outputs. Multiple nodes claim different instances.

4

Stale Detection

If a node misses heartbeats beyond the threshold, its claimed work is released back to the queue for other nodes to pick up.

5

Drain (optional)

Before shutdown, a drain signal tells the node to stop claiming new work and finish in-flight executions.

6

Shutdown

Node completes in-flight work, deregisters from cluster_nodes, and exits cleanly.

Work Distribution

Work distribution is pull-based. Each node's scheduler loop independently queries for claimable instances and processes them.

Claim Mechanics

-- Simplified claim query (actual implementation is more sophisticated)
SELECT id FROM instances
WHERE state = 'scheduled'
  AND next_fire_at <= NOW()
ORDER BY priority DESC, next_fire_at ASC
FOR UPDATE SKIP LOCKED
LIMIT {batch_size}

SKIP LOCKED ensures no two nodes ever claim the same instance. Higher priority instances are claimed first.

Concurrency Control

  • ●Per-node batch size — each node claims up to N instances per tick (configurable)
  • ●Concurrency keys — instances sharing a concurrency_key are limited to max_concurrency active at once, across all nodes
  • ●Rate limits — per-handler rate limiting is enforced globally via database counters

Graceful Drain

Before taking a node offline (for deployment, maintenance, or scaling down), initiate a drain. The node stops claiming new work but finishes everything in progress.

✓

Signal

Send drain signal via API or SIGTERM. Node enters draining state.

✓

Stop Claiming

Scheduler stops pulling new instances from the queue.

✓

Complete In-Flight

All currently executing steps run to completion. No work is abandoned.

✓

Deregister

Once all in-flight work finishes, node removes itself from cluster_nodes and exits.

Zero-Downtime Deployment

# Rolling deployment example (3 nodes)

# 1. Drain node-1
curl -X POST http://node-1:8080/cluster/nodes/node-1/drain

# 2. Wait for node-1 to finish in-flight work (health endpoint returns 503)
while curl -s http://node-1:8080/health/ready | grep -q "ok"; do sleep 2; done

# 3. Deploy new version to node-1, start it
systemctl restart orch8-node-1

# 4. Repeat for node-2, node-3
# At all times, at least 2 nodes are serving traffic

Cluster API

List Nodes

GET /cluster/nodes

Response:
[
  {
    "id": "node-abc123",
    "hostname": "orch8-prod-1",
    "started_at": "2024-01-15T08:00:00Z",
    "last_heartbeat": "2024-01-15T12:34:56Z",
    "state": "active",
    "instances_claimed": 42
  },
  {
    "id": "node-def456",
    "hostname": "orch8-prod-2",
    "started_at": "2024-01-15T08:00:00Z",
    "last_heartbeat": "2024-01-15T12:34:55Z",
    "state": "draining",
    "instances_claimed": 3
  }
]

Drain Node

POST /cluster/nodes/{node_id}/drain

Response: 200 OK (no body)

// Node transitions to "draining" state
// Stops claiming new work
// Completes in-flight executions
// Exits when idle

Health endpoints: UseGET /health/live (is the process running?) andGET /health/ready (is it accepting work?) for load balancer health checks. A draining node returns 503 on/health/ready but 200 on/health/live.

Deployment Patterns

Kubernetes

Deploy as a Deployment with multiple replicas. Use/health/ready as the readiness probe and/health/live as the liveness probe. SetterminationGracePeriodSeconds high enough for in-flight work to complete (e.g., 300s). PreStop hook sends the drain signal.

Docker Compose / VMs

Run multiple instances pointing to the same PostgreSQL. Each process self-registers. Use SIGTERM for graceful shutdown (triggers drain automatically).

Auto-Scaling

Scale based on queue depth (instances in Scheduled state withnext_fire_at <= now()). When queue grows, add nodes. When idle, drain and remove. No state migration needed — work rebalances automatically.

Single Node (Development)

Orch8 works perfectly with a single node — no cluster configuration required. The same binary scales from laptop to production fleet.

Environment Variables

# Required
DATABASE_URL=postgres://user:pass@host:5432/orch8

# Optional cluster tuning
ORCH8_NODE_ID=node-1              # Auto-generated if not set
ORCH8_HEARTBEAT_INTERVAL_SECS=10  # How often to heartbeat (default: 10)
ORCH8_STALE_THRESHOLD_SECS=60    # When to consider a node dead (default: 60)
ORCH8_BATCH_SIZE=50               # Instances claimed per scheduler tick
ORCH8_TICK_INTERVAL_MS=100        # Scheduler loop interval (default: 100ms)