How to Stream LLM Output Without Corrupting Workflow State

Separate the execution plane from the presentation plane

Use cursors and explicit completion

Persisting every token creates write amplification and still does not prove the provider completed the response. A compact final result plus optional archived stream is easier to reason about.

• Assign one model-call ID before dispatch.
• Emit ordered events with a cursor for the client.
• Mark tool requests as structured events, not text parsing.
• Commit the final response or result reference once.
• If the stream breaks, query call status before launching another call.

Design reconnect behavior before launch

A client may reconnect to a live stream, replay buffered deltas from a cursor, or fetch the committed final result. Specify which behavior your transport supports and how long buffers live.

Orch8 streaming can expose workflow events while the durable run remains the source of truth. The UI should label an interrupted presentation differently from a failed model call.

Why are streaming and durable state different clocks?

A token stream is a low-latency presentation channel; workflow state is an authoritative recovery record. Committing every token creates excessive writes and still may not produce a semantically valid checkpoint. Committing only the final answer makes reconnects and long generations opaque. Define coarse durable milestones while the gateway delivers smaller transient chunks.

Useful milestones include request accepted, model call started, tool call proposed, tool result accepted, response segment completed, response finalized, and usage recorded. Give each segment a monotonic sequence and store its final text or object when accepted. The client can render live deltas, then reconcile with the latest durable segment after reconnect.

Do not treat socket closure as model cancellation or workflow failure. The browser may sleep, a proxy may time out, or the network may switch. The workflow continues according to its own deadline and cancellation policy; the client reconnects with run ID and last acknowledged sequence.

What should the reconnect protocol guarantee?

Return a stable run ID before streaming begins. Every event carries run ID, segment ID, sequence, event type, and payload. The client acknowledges or remembers the highest contiguous sequence it rendered. On reconnect it requests events after that cursor, and the server returns retained durable events plus any currently available live stream.

Events must be idempotent for the UI. Repeating sequence 42 should replace or be ignored, never append duplicate text. Detect gaps and request replay rather than guessing. Define retention and the response when a cursor is older than retained events: return a durable snapshot or explicit expired result instead of silently starting in the middle.

Apply backpressure and bounded buffers. A slow or disconnected client must not consume unbounded worker memory or block the model handler. Coalesce token deltas into timed or sized chunks, persist semantic segments, and let the transport drop regenerable live chunks when a durable snapshot can restore the view.

{ run_id, segment_id, sequence, type, payload_digest, payload }
reconnect(run_id, after_sequence)
client: ignore sequence <= rendered; detect sequence gaps

How should common failures resolve?

If the model call fails before any accepted segment, retry under the model policy and budget. If it fails after partial prose, choose explicitly whether to preserve that segment, mark it superseded, or start a new segment; never concatenate two generations as though they were one. If a tool call was proposed, its validation and effect state control recovery independently of visible tokens.

When the client disconnects, keep or cancel the run according to product semantics. For an expensive report, continuing may be useful; for interactive autocomplete, cancellation may save cost. Cancellation is a durable request with acknowledgement, not an assumption based on a broken TCP connection. Record usage even when no client remains connected.

If the gateway restarts, reconstruct from the durable run and segments, then attach to the worker or wait for the next accepted event. Do not ask the model to regenerate already accepted content simply because the streaming node lost memory. This preserves cost, meaning, and citations.

• Browser refresh: resume after the last rendered sequence.
• Out-of-order event: buffer briefly, then request the missing range.
• Duplicate event: ignore by run and sequence.
• Model timeout: close or supersede the partial segment explicitly.
• Gateway restart: rebuild the view from accepted segments.
• Cancellation: stop future work and report whether the model call acknowledged it.

Which tests prove the user experience?

Automate disconnects after the first token, between segments, during a tool call, and immediately before finalization. Restart the gateway and worker separately. Deliver events twice and out of order, slow the client, expire retained deltas, and send cancellation while the provider response is delayed. Assert rendered text, durable state, provider call count, cost record, and final status.

Measure time to first event, inter-chunk delay, time to durable segment, reconnect success, replay bytes, sequence gaps, abandoned runs, cancellation acknowledgement, model latency, and cost. Separate transport latency from model and workflow waiting so optimization targets the real boundary.

Tell users when output is provisional, reconnecting, waiting for a tool, awaiting approval, or final. A polished typing animation cannot substitute for state clarity. The interface should recover to the same accepted response after refresh and make partial or superseded output visually distinct.

How do privacy and retention affect the stream?

Decide whether token deltas, completed segments, prompts, tool data, and usage records have different retention periods. Avoid copying sensitive output into gateway logs or analytics events. Encrypt retained segments and authorize every reconnect against the current user and tenant; possession of a run ID is not sufficient access.

When users delete or export a conversation, include durable workflow state and derived streaming stores in the policy. A transient buffer should expire quickly, while a final segment may follow the product's conversation retention. Document whether provider logs retain content independently and configure them where controls exist.

Test reconnect after logout, account transfer, link sharing, and permission revocation. The server should deny stale credentials without revealing whether another tenant's run exists. Audit access to consequential tool results and redact diagnostic payloads while preserving hashes and identifiers needed to investigate delivery.

• Authorize every initial and resumed connection.
• Keep run IDs unguessable but never treat them as credentials.
• Separate content retention from operational metrics.
• Apply deletion to caches, replicas, and exports.