Conversation
🦋 Changeset detectedLatest commit: 08318f6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 164555ms → this run 155884ms (Δ -8671ms, -5%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (1)9ef9fadFri, 11 Sep 2026 08:55:58 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
🧪 E2E Test Results✅ All tests passed 🛠 Infra Events (absorbed by the harness)Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 🌐 Cross-language Conformance
✅ vercel-http-transport
✅ vercel-multi-region
✅ vercel-ws-transport
|
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
There was a problem hiding this comment.
🟡 Changes recommended
Moderate concerns remain around repeated wake publishes, the inline-ownership kill switch, and treating unstamped starts as queue-owned.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds delayed backstop handling for queue-owned running steps on capable Worlds, with configuration, telemetry, tests, and documentation.
Changes:
- Adds queue redelivery capability support for Vercel.
- Updates core ownership classification and dispatch behavior.
- Adds tests, runtime tuning guidance, changelog updates, and a changeset.
File summaries
| File | Summary / final review comment |
|---|---|
packages/world/src/interfaces.ts |
Adds the optional queue redelivery capability contract. |
packages/world-vercel/src/index.ts |
Declares Vercel queue redelivery support. |
packages/world-vercel/src/capabilities.test.ts |
Tests the Vercel capability declaration. |
packages/core/src/telemetry/semantic-conventions.ts |
Adds backstop wake telemetry. |
packages/core/src/runtime/step-ownership.ts |
Adds queue-owned running step classification. Moderate (2 votes): unstamped starts may represent legacy inline execution, so provenance or a rollout guard is needed. |
packages/core/src/runtime/step-ownership.test.ts |
Tests ownership classification and exclusivity. |
packages/core/src/runtime/constants.ts |
Adds the backstop feature switch. |
packages/core/src/runtime.ts |
Integrates delayed backstop dispatch. Moderate (1 vote): repeated replays still publish continuations; coalescing or durable pre-send deduplication is needed. Moderate (1 vote): the branch bypasses WORKFLOW_INLINE_OWNERSHIP=0, contrary to the existing kill-switch behavior. |
packages/core/src/runtime.test.ts |
Tests enabled, disabled, and fallback dispatch scenarios. |
docs/content/worlds/v5/upgrading-to-v5.mdx |
Documents the capability upgrade guidance. |
docs/content/worlds/v5/building-a-world.mdx |
Documents World capability implementation. |
docs/content/docs/v5/configuration/runtime-tuning.mdx |
Documents the backstop configuration. |
docs/content/docs/v5/changelog/step-message-ownership.mdx |
Updates the ownership decision table and observability guidance. |
docs/content/docs/v5/changelog/eager-processing.mdx |
Documents the queueing exception. |
.changeset/queue-owned-backstop.md |
Records the package release changes. |
Review details
Suppressed comments (2)
packages/core/src/runtime.ts:3975
- This optimization still publishes one delayed continuation for every queue-owned step on every replay; it does not remove the duplicate
world.queuecalls described in the motivation.backstopIdempotencyKey(step)is stable, but world-vercel accepts repeated idempotency keys as fresh sends and deduplicates only delivery-side (packages/world-vercel/src/queue.ts:493-503), so the 8-connection publish traffic remains (just with a different payload). Please coalesce these wakes or add a pre-send/durable dedupe mechanism if reducing queue traffic is the goal.
if (queueOwnedRunning) {
queueOwnedBackstopWakesArmed++;
} else {
backstopWakesArmed++;
packages/core/src/runtime.ts:3877
- This new branch is not gated by
inlineOwnership, so setting the existingWORKFLOW_INLINE_OWNERSHIP=0still arms delayed backstops for bare steps. That contradicts the existing kill-switch contract that dispatch reverts to unconditional immediate requeue; unless that public behavior is intentionally changing, the queue-owned optimization should honor the existing switch as well.
const queueOwnedBackstop =
isQueueOwnedBackstopEnabled() &&
world.capabilities?.queueRedeliversUnacked?.active ===
true;
- Files reviewed: 15/15 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| export function isQueueOwnedRunning(step: StepInvocationQueueItem): boolean { | ||
| return ( | ||
| step.hasCreatedEvent === true && | ||
| step.lastStartedAt !== undefined && | ||
| step.ownerMessageId === undefined && | ||
| step.sawRetrying !== true | ||
| ); |
There was a problem hiding this comment.
Agreed: a bare start is only proof of a queue delivery on a run whose runtime stamps inline starts. Guarded in 08318f6.
No durable provenance marker exists today (a bare step_started carries nothing that says "queue delivery"), so the guard is a spec-version gate plus the capability being declared only by @workflow/world-vercel. isQueueOwnedRunning(step, runSpecVersion) now also requires the run's specVersion >= SPEC_VERSION_SUPPORTS_SLOT_IDENTITY (6). Ownership stamps shipped in #2848 (2026-07-10) while runs were still minted at spec 5, so a spec-5 run may predate them; spec 6 (#3389, 2026-08-11) shipped after, so every runtime that mints a spec-6+ run also stamps its inline starts. A run's spec version is fixed at start(), and on Vercel its queue messages and replays are deployment-pinned, so within one run the runtime is one version: the run's spec version stands for the version of every start in its log. Older or unknown spec versions keep the immediate re-enqueue.
Tests: step-ownership.test.ts covers a bare start at spec 5, spec 1 and undefined being treated as not queue-owned, and runtime.test.ts drives a legacy-spec run with a bare start against a redelivering World and asserts the immediate step re-enqueue.
| const queueOwnedRunning = | ||
| queueOwnedBackstop && isQueueOwnedRunning(step); | ||
| const backstopDelaySeconds = | ||
| ownershipActive || queueOwnedRunning | ||
| ? stepLeaseRemainingSeconds(step, dispatchNowMs) |
There was a problem hiding this comment.
[P2] This substitutes a backstop publish for each immediate step publish, but still sends one queue request per running step per replay. backstopIdempotencyKey(step) deduplicates server-side, so these repeated sends still use the same client connection pool—the bottleneck identified in the motivation. A targeted test with three replays over one unchanged running step produced three backstop sends.
The backstop key is also distinct from the existing step-dispatch key, so the first send creates an additional delayed run delivery, even if the step completes normally before it fires.
Could we avoid the repeated backstop sends, or demonstrate a measured benefit that outweighs the extra delayed deliveries, before enabling this by default as a queue-traffic optimization?
Local agent review (`openai/gpt-6-astra`)
There was a problem hiding this comment.
Fair on both counts, and thanks for the targeted test. Reworked in 08318f6.
New shape. The queue-owned row no longer sends anything per step. The dispatch pass folds every queue-owned running step it sees into one delayed run continuation for the run, due when the latest bare start's lease expires (each step's expiry is its own start + lease, so the latest start bounds them all), keyed ${runId}:queue-backstop:${latestStartedAt}. The invocation keeps the set of epochs it has armed and skips the send outright on a later pass over an unchanged log, so repeats never reach the client pool; concurrent invocations derive the same key and collapse onto one pending wake server-side. workflow.queue_ownership.backstop_wakes_armed is now 0/1 per pass.
Measured send counts (runtime.test.ts, "queue-owned running step dispatch"):
| scenario | before | after |
|---|---|---|
| 3 replay passes over one unchanged running step (your case) | 3 backstop sends | 1 |
| 3 running steps, one pass | 3 backstop sends | 1 |
| wake fires on a finished run | run_started reports terminal, handler returns | same: 0 reads, 0 sends, 0 bodies |
| lease spent when the wake fires | immediate step re-enqueue | same |
Why the epoch and not a coarse window. A key rounded to e.g. the lease length would be deduped against a wake already in flight, while that wake fires before a newer step's lease expires (the newer step landed in the same window); widening the delay so it always fires after every expiry in the window pushes delaySeconds past the 900s per-message cap that stepLeaseRemainingSeconds clamps to. The latest-start timestamp is the finest key that is still replay-stable, and a newer bare start moving it is exactly what keeps the new step covered.
On the extra delayed delivery. Yes, the first send still creates one delayed run delivery per epoch even when the steps complete normally, and I want to be honest that the benefit here is queue-traffic reduction, not TTLS: N sends per replay pass on a wide fan-out become at most one per lease window per run, and the wake on a completed run is the ordinary already-terminal exit. On the 32-branch durabench fan-out that was 19-28 sends per post-inline replay, but on that same shape the inline delta path is what moves latency; this PR only stops the orchestrator paying for re-sends the queue was going to dedupe anyway.
…ed running steps The pending-step dispatch pass re-enqueued every pending step that is not inline-owned on every replay. That is needed for a step that is created but never started (step_created proves nothing about whether its message was sent), but not for a step whose bare step_started is already in the log with no terminal event: a queue delivery is executing its body and has not acked its message, and a queue that redelivers unacked messages will redeliver it if that consumer dies. On such Worlds the immediate re-send was duplicate traffic, one send per pending step per replay of a fan-out. Add `capabilities.queueRedeliversUnacked` to @workflow/world, declare it in @workflow/world-vercel, and have the dispatch pass arm the same delayed backstop wake an inline-owned step gets (lease remainder, epoch-scoped key) for such steps instead of the immediate step enqueue. Once the lease is spent the backstop falls through to the immediate enqueue, so a wrong guess costs at most one lease. `WORKFLOW_QUEUE_OWNED_BACKSTOP=0` restores the previous behaviour; the count is reported on the span as `workflow.queue_ownership.backstop_wakes_armed`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…on ownership-stamping runs Review follow-ups on #4100: - The queue-owned row no longer sends a backstop per running step per replay. The dispatch pass folds every queue-owned running step into ONE delayed run continuation, due when the latest bare start's lease expires and keyed `${runId}:queue-backstop:${latestStartedAt}`. The invocation remembers the epochs it armed and skips the send on later passes over an unchanged log, so three replay passes over one running step (or one pass over N running steps) cost a single publish; concurrent invocations collapse onto one pending wake server-side via the shared key. A coarser bucket key would either be deduped against an in-flight wake that fires before a newer step's lease expires, or need a delay past the queue's per-message cap, so the epoch is the key. A wake on a finished run is the ordinary already-terminal exit. - `isQueueOwnedRunning` additionally requires the run's specVersion to be at or above SPEC_VERSION_SUPPORTS_SLOT_IDENTITY (6), the first spec version every ownership-stamping runtime mints (#2848 shipped under spec 5). A bare start on an older or unknown run is not proof of a queue delivery, since the replay contract tolerates a legacy unstamped inline start, so such runs keep the immediate re-enqueue. - `workflow.queue_ownership.backstop_wakes_armed` is now 0/1 per pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
9ef9fad to
08318f6
Compare
Motivation
The pending-step dispatch pass in
packages/core/src/runtime.tsre-enqueues every pending step that is not inline-owned, immediately, on every replay. The rationale (see the eager-processing changelog, "Queueing is unconditional") is crash recovery:step_createdin the log does not prove the step's message was ever sent. That holds for a step that is created but never started. It does not hold for a step that is already queue-owned and running: a bare (unstamped)step_startedwith no terminal event means a queue delivery of the step message is executing the body right now and has not acked its message. On a World whose queue redelivers unacked messages, the queue itself re-runs that step if its consumer dies, so the immediate re-enqueue is pure duplicate traffic. The queue dedupes it on the idempotency key, but each one is still a send on the 8-connection pool. Measured on a 32-branch durabench fan-out, the orchestrator's post-inline replay re-sent 19-28 messages, most of them for steps whosestep_startedwas already in the log.Safety argument
step_startedwith no terminal event and nostep_retryingmeans the step message is still unacked. On Vercel Queues an unacked message is redelivered after the visibility timeout; that redelivery writes another barestep_startedand runs the body, which is exactly what today's immediate re-enqueue would eventually do.delaySecondsequal to the ownership lease remainder anchored at the bare start,idempotencyKeyscoped to that start's timestamp. When it fires the lease is spent,stepLeaseRemainingSecondsreturns 0, and the same decision table falls through to the immediate enqueue. A wrong guess costs at most one lease (860s by default), which is the degradation inline ownership already accepts.isQueueOwnedRunningrequireslastStartedAt, so a step with onlystep_createdin the log still gets the immediate enqueue on every replay. Nothing in its log proves its message was sent.step_retryingis excluded, mirroringisStepOwnershipActive: from there the step rides its delayed retry handoff, which stays on the immediate re-enqueue path.capabilities.queueRedeliversUnacked.active === trueandWORKFLOW_QUEUE_OWNED_BACKSTOPis not0/false. World-local declares nothing and keeps today's behaviour.What changed
@workflow/world: new optionalWorldCapabilities.queueRedeliversUnacked?: { active: boolean }, documented in the existing capability style.@workflow/world-vercel: declaresqueueRedeliversUnacked: { active: true }(VQS redelivers after the visibility timeout). Addscapabilities.test.tsasserting the declaration.@workflow/core:runtime/step-ownership.ts:isQueueOwnedRunning(step)(created, started, unstamped, nostep_retrying); mutually exclusive withisStepOwnershipActive.runtime/constants.ts:isQueueOwnedBackstopEnabled()readingWORKFLOW_QUEUE_OWNED_BACKSTOP(default on, same shape asisBatchTransitionsEnabled).runtime.ts: the dispatch pass arms the delayed backstop for queue-owned running steps on a declaring World; the decision-table comment gains the new row. Count reported on the span asworkflow.queue_ownership.backstop_wakes_armed(new semantic convention), separate from the inline-ownership counter.WORKFLOW_QUEUE_OWNED_BACKSTOPin runtime tuning; the capability in the v5 world-authoring guide and the v5 upgrade table; the decision table and observability list in the ownership changelog; the "queueing is unconditional" wording in the eager-processing changelog now names the exception.@workflow/world-postgresis deliberately not declared. Its embedded Graphile worker executes each message over HTTP and holds the job lock for the call; if the app dies but the worker survives, Graphile retries the failed job, but if the worker process itself dies the job stays locked until Graphile's stale-lock sweep (hours), and the World's real crash recovery for the usual single-process deployment isreenqueueActiveRuns()onstart(), an immediate run wake that today produces the immediate re-enqueue. Declaring the capability would turn that into a lease-length delay.Tests
packages/core/src/runtime/step-ownership.test.ts:isQueueOwnedRunning(created+started+bare => true; stamped,sawRetrying, never started, uncreated => false; exclusive with inline ownership).packages/core/src/runtime.test.ts, new describe "queue-owned running step dispatch": with the capability active the replay sends one delayed run continuation (nostepId, epoch-scoped key,0 < delaySeconds <= lease); with the capability absent, declared inactive, or underWORKFLOW_QUEUE_OWNED_BACKSTOP=0it sends the immediate step message understepDispatchIdempotencyKey; a created-but-never-started step is still enqueued immediately with the capability active. The step's correlation ID is discovered by a first delivery whose lazystep_startedis refused, since IDs are minted from a run-seeded PRNG and are stable across replays.packages/world-vercel/src/capabilities.test.ts: the declaration.Results:
packages/core(FORCE_COLOR=0 pnpm test)packages/world-vercel(pnpm test)pnpm typecheckfor core, world, world-vercel, world-postgres, world-localDocs Preview
Preview from the
workflow-docsproject (behind deployment protection, Vercel team access required):WORKFLOW_QUEUE_OWNED_BACKSTOPNotes and open questions
runtime/quickjs-entrypoint.ts) mirrors the ownership table with its own dispatch loop, but its backstop is a delayed step message rather than a run continuation. Left unchanged here; it can pick up the same row in a follow-up.🤖 Generated with Claude Code