From 86a46946de1db254a477734ea11b344888cbad2f Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 09:33:24 -0400 Subject: [PATCH 01/81] =?UTF-8?q?=F0=9F=93=9D=20Specify=20the=20software?= =?UTF-8?q?=20factory=20deployment=20contract=20(#710)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turn PR #630's draft into a decision-complete first-release architecture. No product, deployment, authority, identity, ownership, failure, replay, public form, permission or ordering decision is left for a later Implementor. The factory specification now settles: the canonical `github-issue-v1` run-ID derivation and unsupported issue-transfer drift; `User` as the only Stage 1 spelling; the deployment topology of one SQLite-backed Cloudflare Durable Object, an authenticated executor WebSocket, an ephemeral Actions runner and a dedicated GitHub App; authenticated ingress order, bounded intake, dispatch payload and OIDC claims; the exact App permissions, host ceilings, human permission floor and `.github/workflows/**` denial; suspend-on-every-conflict as the single Stage 4 profile; the Stage 7 trusted merge, `Git.PublishTarget` compare-and-swap and remote-effect-before-terminal ordering; and the exact public contract inventory. Section 8's remaining material decisions are gone. architecture.md gains the terminology, the remote storage and executor topology, the delivery-plane generalization, the Project-provider boundary, comments/readiness/closure, ordered merge, target publication, trusted evidence execution, terminal settlement, the split trusted host, and ten construct inventory rows marked "specified by #710; implementation unbuilt". The workflow and Workspace specifications gain the remote host, the executor connection, which requests need it and which do not, the host-derived public run ID, and provider-neutral contracts for `Git.Merge`, `Git.PublishTarget`, `PullRequest.Comment`/`Ready`/`Close`, `Issue.Comment`/`Close`, `Evidence.Run` and `Project.Status` — none of them reachable by an Agent or generated XMD, whose write table stays exactly `File:write`, `Dir` and `File.Delete`. The executable-MDX specification gains the exact authored forms and the frozen WRH/WGI/WGE/WGM/WER/WFP/WFL acceptance tiers. Documentation only: no production code, workflow YAML, dependency, generated artifact or executable fixture changes, and the ownership SVG already agrees with the amended text. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- architecture.md | 216 +++++ specs/executable-mdx-spec.md | 190 +++++ specs/github-actions-software-factory-spec.md | 761 ++++++++++++------ specs/workflow-spec.md | 82 ++ specs/workflow-workspace-spec.md | 337 +++++++- 5 files changed, 1360 insertions(+), 226 deletions(-) diff --git a/architecture.md b/architecture.md index 879b0aec6..183f06150 100644 --- a/architecture.md +++ b/architecture.md @@ -104,6 +104,15 @@ Existing documents and code get aligned to this section retroactively. | recovery tombstone | an ownership record left active because its owner never proved it stopped. A crash releases the kernel lock and not this, and no pid, elapsed time, released lock or empty transcript clears it | | provider partition | one complete, independently owned agent-provider state — runtime, store, managed sessions, queues, coordinator, teardown — selected by the one installed factory at each dispatch. Production is the single-partition case of the same path; holding a partition grants work, never permission | | `JournalProvenance` | a non-operational, equality-only witness that a live publication stream descends from the exact journal backend a provider selected for one workflow run; it grants no append, read, execution, publication or reconciliation capability, and is meaningful only because the provider retains the witness it established and later requires exact equality | +| factory run identity | the run ID a software-factory run is addressed by: the lowercase unpadded RFC 4648 Base32 encoding of the SHA-256 digest of `github-issue-v1`, NUL, the canonical Issue-provider authority, NUL and the exact issue node ID. It is one host-selected public run ID, derived once from immutable provider identity, and it is distinct from the workflow definition SHA, the implementation revision, the Workspace root, the expansion identity and every delivery identity | +| authenticated intake | one bounded record a trusted host retains for an externally delivered request — a verified webhook, or an authenticated human form submission — keyed by the provider's own delivery or submission identity and holding only typed bounded fields. It is what a later execution reads; it is never a stage, an outcome, a transition or a credential, and receiving one authorizes nothing beyond finding the run it names | +| Project provider | an external service that owns project boards and the status of the items on them. GitHub Projects V2 is one adapter. It is a separate boundary from a Git host and from an Issue provider, because a project board need own neither a repository nor an issue collection | +| Project projection | the human-facing status a Project provider holds for one item, published from the journaled lifecycle rather than read as it. A projection ahead of the journal is drift to reconcile; it is never evidence that a lifecycle transition happened | +| executor connection | a remote host's form of executor acquisition: one authenticated connection whose lifetime *is* the acquisition. Like the local executor lock it is not a time lease — no duration, expiry, renewal, heartbeat, PID or liveness poll — and closing it releases executor ownership without rolling back what already committed | +| delivery-plane transaction | one authenticated transaction that retains an externally supplied value for an exact retained subject without executing the run: intake retention, typed answer delivery and terminal-decision delivery are the three. It generalizes delivery to subjects other than a suspension request, on the same terms: no executor acquisition, no document execution, no journal append, no run-status change | +| implementation revision | the evolving pair `{ headSha, baseSha }` one factory run is currently producing or reviewing: the exact head commit of the implementation branch and the exact target-branch commit it is evaluated against. It changes many times within one run and never takes part in run identity | +| exact-review subject | the implementation revision a review conclusion names. A conclusion authorizes only the pair it names, so a later revision inherits nothing and a moved half of the pair invalidates it | +| terminal settlement | the last transition of a run whose outcome required external projections: the retained terminal decision is published as run state only after every required projection has completed, so a completed replay never needs a provider to repair one | ## Three axes @@ -397,6 +406,39 @@ describe what failed without repeating retained props or journal payloads — including their member *names*, which can carry a credential as readily as a member value can. +### One remote owner for one run + +A remote host owns the same run the local host owns, through the same +provider-neutral surfaces. The Cloudflare topology is one SQLite-backed Durable +Object per run, selected from the public run ID by the same arithmetic local +discovery uses, so a remote run has exactly one durable owner and no second +registry can disagree with it. That object holds the WorkflowRun record and its +filtered journal, the immutable Workspace roots and their content-addressed +bytes, the Agent-session mappings and checkpoints, the retained delivery state, +the authenticated intake records, and executor ownership. + +The runtime-named Cloudflare entrypoint is the only place that topology appears. +Shared modules reach it through the contextual storage and lifecycle APIs they +already use, detect no runtime, and import nothing Cloudflare-specific — the +same boundary the Deno entrypoint sits behind. + +Native execution stays off that object. Native Git, evidence processes and Agent +clients run on an ephemeral runner against bounded materialized state; the +Durable Object runs none of them. The runner materializes one selected retained +root, works in it, and submits content-addressed changes; the owner validates +the executor acquisition, the expected root and the submitted content, then +atomically publishes the new root together with the filtered journal result. +That is the same effect transaction local Workspace mutation uses, with the +mutation performed where the tools are and the publication performed where the +authority is. + +A runner that dies mid-flight therefore exposes only a prior or a new complete +transaction, never a partial one. The next acquisition performs the ordinary +stale-execution recovery and resumes from the exact committed WorkflowRun and +Workspace frontier. A completed run replays as it does anywhere: it attaches no +remote storage session, Workspace, Agent, process, Git, Git-host, Issue, +Project or credential provider. + ## The workflow lifecycle `xmd workflow start [--id=] [--props-*=…] ` and @@ -700,6 +742,29 @@ while the deleting workflow executor still holds its lock. An empty lock file is not retained run state and is not reported as history or provider-session data. +### Remote executor connection + +A remote host acquires the same authority over one authenticated connection. +The acquisition is that connection's lifetime: the run's owner registers the +exact acquisition when the connection is admitted and invalidates it when the +connection closes, which is the staleness proof a remote host has in place of a +released kernel lock. It is not a time lease either — no duration, expiry, +renewal, heartbeat, generation record or liveness poll — and closing it releases +executor ownership without rolling back anything already committed. A second +healthy executor follows the active one or is refused; it cannot advance the +run. + +What the acquisition gates is the same list the lock gates locally, plus what a +split host adds: start and resume, stale-execution recovery, document execution, +Workspace mutation, Agent attachment, native Git and evidence execution, +lifecycle transition, accepted-outcome publication and terminal settlement. Each +of them validates the exact live acquisition *and* the expected Workspace root +inside its own mutating transaction, so a stale connection and a stale frontier +are refused at the same boundary rather than at two. + +What it does not gate is delivery and inspection. Those are described below and +below that, and neither becomes transition authority by being remote. + ### Read-only lifecycle inspection Inspection has its own provider-neutral immutable snapshot surface. It returns @@ -1027,6 +1092,24 @@ transaction that does not commit publishes nothing. Replay after that transaction commits restores the recorded answer event without reaching the live controller and without consuming or publishing again. +That separation is a property of delivery rather than of the local CLI, so it +survives a remote host and generalizes past a suspension request. A **delivery- +plane transaction** is any authenticated transaction that retains an externally +supplied value for an exact retained subject: intake retention, typed answer +delivery, and the terminal-decision delivery the software-factory specification +describes. Each validates its own delivery identity and +its own retained subject, retains only the typed bounded value that subject +describes, and does exactly what answer delivery does otherwise — takes no +executor acquisition, begins no document execution, attaches no provider, +appends no lifecycle outcome and changes no run status. A duplicate delivery of +one identity finds the retained record and writes nothing. + +Consumption stays where it already is. A later executor reads the retained value +inside the run's own transaction, appends the accepted durable event or outcome +exactly once, and only then may authored control flow decide what follows. A +delivery that could advance a lifecycle would be a second state machine beside +the journal, which is the thing this split exists to prevent. + Scheduling — automatic resume, watchers, unattended iteration and remote host selection — is #300's and is not part of this behavior. Nothing here waits on it: a suspended run continues through `xmd workflow answer` followed by an @@ -1406,6 +1489,84 @@ afterwards, and nothing catches its refusal to try somebody else, because a search is how a document that named one service quietly reaches another. A destination every provider delegated reaches the operation's own base error. +### A project board is a third external boundary + +A **Project provider** owns project boards and the status of the items on them, +and it is a boundary of its own for the reason the Issue boundary is one: a +project board need own neither a Git repository nor an issue collection, so a +Project status cannot truthfully execute or persist as a Git-host effect or an +Issue effect. `Project.Status` reaches its own contextual operation and journals +its own durable effect. Its natural key is the exact Project item plus the exact +field; its compatible pre-state is the option that item currently holds; and the +configured Project, item, field and option identities are a host ceiling rather +than something an authored prop can widen. + +The status a board shows is a **Project projection** of the journaled lifecycle, +never a reading of it. A board ahead of the journal is drift the next execution +reconciles, and it is not evidence that a stage was passed. + +### Comments, readiness and closure + +Three more reconciled effects join the same boundary, each keyed by its own +subject. `Issue.Comment` is an Issue-provider effect keyed by the canonical +issue URL plus the engine-derived effect identity; `PullRequest.Comment` is a +Git-host effect keyed by the canonical pull-request URL plus that identity. The +body is presentation in both: keying a comment by its text would make an edited +sentence a different comment. `PullRequest.Ready` and `PullRequest.Close` are +Git-host effects keyed by their exact pull-request subject, and `Issue.Close` is +an Issue-provider effect keyed by its exact issue subject and carrying a closed +`reason` enum that has to match the retained terminal intent. Each observes +before it mutates, adopts a compatible completion, performs once from proven +absence or an exact compatible pre-state, and refuses conflict, permanent +ambiguity, incomplete observation and temporary unavailability. + +### Ordered merge, and publishing a target + +`Git.Merge` is Workspace-local. It observes and fixes its first parent, second +parent and merge base before it mutates, and its two closed results are +distinct: a clean merge publishes the new commit, the new Workspace root and the +filtered result in one effect transaction, while a conflicted merge restores the +pre-merge root and publishes normalized conflict evidence against it. The parent +order is the caller's and is part of the request, because the two merges a +factory performs mean opposite things — synchronizing a target into an +implementation, and publishing an implementation onto a target. + +`Git.PublishTarget` is a Git-host effect and is not a spelling of `Git.Push`. +Push advances a branch this run published, from an ancestry relation proved +inside the authenticated object source. Target publication is a compare-and-swap +against a protected ref: it updates only after observing the target equal to the +expected commit, adopts a target already equal to the exact source commit with +nothing performed, and refuses everything else without mutating. The remote, the +ref, the credential and the non-force policy are host-owned; the reviewed head +and expected commit travel in the request so the record says what the +publication was authorized against. + +### Trusted evidence execution + +`Evidence.Run` executes an authored structured argv list natively, on the +trusted runner, against one exact retained Workspace root, under host-owned +executable, environment, time and output ceilings. It is not Worker Shell and +not a document's own process capability: it runs where the tools are, and it is +absent from the workflow Agent's capabilities and from every generated-XMD write +table. Its ordered bounded results are an ordinary durable record, so a +completed replay runs nothing. + +### Terminal settlement follows its projections + +An outcome whose completion requires external projections retains the decision +first and settles last. The accepted terminal decision is journaled before any +effect is attempted; the required projections — a constructed merge, a target +publication, the observation that the pull request is merged, the issue closure, +the Project move — are then separate reconciled steps; and only after all of +them complete is the terminal run state published. + +The ordering is not preference. A completed run replays without contacting a +provider, so a terminal state published before its projections would leave the +repair to exactly the replay that is forbidden to reach a provider. No +distributed transaction is claimed across the run's storage, native Git and +processes, and the external services; retaining the decision before the +projections, and settling after them, is what reconciliation has instead. + Every committed journal event references the current logical Workspace root. Only committed event boundaries are checkpoints. A history fork copies the selected root and the roots the inherited prefix names into the new run, replays @@ -3116,6 +3277,51 @@ or partial continuation they run and record through the ordinary durable protocol — an effect an earlier preparation already completed is restored from its retained record rather than performed again. +### A split trusted host + +A factory run has one trusted host in two pieces, and which piece owns what is +the whole of its security boundary. + +The **provider host** is the Cloudflare runtime-named entrypoint. It owns +persistence and transactions, the authenticated intake receiver, the +authorization gates, token minting, and executor admission. On GitHub that +receiver is one dedicated GitHub App: it verifies a webhook signature before it +parses the payload as anything but bytes, +rereads the complete provider objects through the API rather than trusting the +payload's copy of them, authenticates the installation and the human actor, and +only then retains one bounded intake keyed by the provider's delivery or +submission identity, and mints the short-lived installation token every external +effect is performed with. It admits a runner session — the OIDC client the +Actions job authenticates as — only after validating that session's claims: +issuer, configured audience, repository ID, repository-owner ID, event name, +workflow ref and SHA, and the configured immutable workflow identity. Names are +mutable and IDs are not, which is why the check is on IDs. + +The **runner host** is the ephemeral Actions job. It owns the native clients: +Git, the plan-evidence processes, and the Agent. It holds no durable authority +at all; what it holds is one authenticated executor connection and one +materialized Workspace root, and every mutation it proposes is validated and +published by the provider host. + +Credentials stay with the piece that mints them. The application private key, +the webhook secret, the OIDC verification configuration, every issued +installation token, the provider endpoints, the raw payloads, the pagination +cursors and the host paths are provider-host secrets and closure state. None of +them reaches props, context composition data, a durable request or result, a +comment, document output, or a diagnostic. A short-lived installation token +performs the external effects; the journal retains the human actor separately +from the token that acted, so the record says who decided as well as what was +done. + +Ceilings narrow in one direction. The installation is limited to configured +repositories, and the host narrows further per operation to the exact +repository, branch, target ref, project, field, option, subject, reviewed +revision, parent pair and non-force operation. A path the granted permission +could reach but the contract excludes — a workflow definition under +`.github/workflows/**` is the one that matters, since rewriting it would rewrite +the run's own authorization — is refused by the host rather than left to the +permission model. + ### The weak journal-provenance association Journal provenance is the one further exception, and it is deliberately narrow. @@ -3303,6 +3509,16 @@ Status is measured against main. | `xmd workflow answer ` | retains one schema-validated value for one retained wait, taking no executor lock and changing no run state | built by #300 | | `suspension_answer` durable effect | ends a wait from retained delivery state, publishing the answer and consuming that state in one transaction | built by #300 | | `` · `` · `` | read the reviews, comments and checks a Git host already holds for one numbered pull request, completely or not at all | built by #576 | +| `` | adds one comment to the issue a canonical URL names, from the paired content it renders, reconciled as an Issue-provider effect whose natural key is that URL plus the engine-derived effect identity — the body is presentation, so an edited sentence is the same comment | specified by #710; implementation unbuilt | +| `` | adds one comment to the pull request a canonical URL names, from the paired content it renders, reconciled as a Git-host effect keyed by that URL plus the engine-derived effect identity | specified by #710; implementation unbuilt | +| `` | publishes one item's status to a Project provider — a boundary of its own, because a board owns neither a repository nor an issue collection — keyed by the exact item and field, against the option the item currently holds, inside the host's configured project, field and option ceiling | specified by #710; implementation unbuilt | +| `` · `` | take one pull request out of draft, and close one unmerged, as Git-host effects keyed by their exact subject; readiness is authorized by an accepted review outcome rather than by observing the pull request | specified by #710; implementation unbuilt | +| `` | closes one issue as `completed` or `not_planned`, as an Issue-provider effect keyed by its exact subject, with the reason a closed enum that must match the retained terminal intent | specified by #710; implementation unbuilt | +| `` | merges two exact commits inside the retained Workspace, observing and fixing both parents and the merge base first; a clean result publishes commit, root and filtered result in one effect transaction, a conflicted one restores the pre-merge root and publishes normalized conflict evidence, and the parent order is the caller's because synchronizing a target into an implementation and publishing an implementation onto a target are opposite operations | specified by #710; implementation unbuilt | +| `` | updates a protected target ref by compare-and-swap: one non-force update after observing the target equal to the expected commit, adoption of a target already equal to the exact source commit, and refusal of everything else without mutation; remote, ref, credential and non-force policy are host-owned. Not a spelling of `Git.Push`, which advances a branch this run published from a proved ancestry relation | specified by #710; implementation unbuilt | +| `` | runs an authored structured argv list natively on the trusted runner against one exact retained Workspace root, under host-owned executable, environment, time and output ceilings; absent from the workflow Agent's capabilities and from every generated-XMD write table, and a completed replay runs nothing | specified by #710; implementation unbuilt | +| remote `WorkflowHost` | starts, looks up, executes, answers and inspects one run against a remote owner through the provider-neutral lifecycle surfaces, with a Cloudflare runtime-named implementation: one SQLite-backed Durable Object per run selected from the public run ID, executor acquisition as an authenticated connection lifetime, and content-addressed root transfer from an ephemeral runner | specified by #710; implementation unbuilt | +| factory protocol records | closed parsers and immutable values for the stage, frontier, implementation revision, role outcome, invalidation, conflict, terminal decision and terminal settlement one issue-driven run retains — provider-neutral durable protocol, neither an XMD component nor a TypeScript lifecycle controller | specified by #710; implementation unbuilt | | workflow scheduling (watchers, unattended iteration, remote host selection) | — | #300 | | `` | binds `{ok: true, value}` or `{ok: false, error}`; a failure becomes a bound value, not a raise | defined, unbuilt | | error middleware (JS api) | retry · suspend · decline | defined, unbuilt | diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index bfdd3b769..ce397fffa 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -7497,6 +7497,49 @@ component grants that admission, and a repository component that takes the name `Fetch` is not the pinned identity. +### 6.19 Software-factory constructs + +An issue-driven software factory adds nine authored constructs. They belong to +the workflow host rather than to core: nothing registers them under `xmd run`, +and a document executed without that host has none of them. Their exact forms +are listed here so the public surface is readable in one place; the contract +behind each form — closed props, request, natural key, compatible pre-state, +normalized result, refusal and unavailability behavior, cancellation, replay, +provider ownership and credential boundary — belongs to the section named +beside it. + +`as` is required on every one of them, because every one binds a result. The +form is validated before any context, provider, ceiling or credential is +reached, so a missing prop, an unknown prop, a value outside a closed enum and a +missing `as` each fail before the effect exists. Durable effect identity is +engine-derived from the run and the expansion and is never a prop. + +| Construct | Exact authored form | Contract | +| --- | --- | --- | +| `Issue.Comment` | `…` — paired; the content is the body; binds `{ url }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.3 | +| `PullRequest.Comment` | `…` — paired; the content is the body; binds `{ url }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.10 | +| `PullRequest.Ready` | `` — binds normalized ready evidence | [Workflow workspaces](./workflow-workspace-spec.md) §7.10 | +| `PullRequest.Close` | `` — binds `{ url, state: "closed", merged: false }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.10 | +| `Issue.Close` | ``, or the same form with `reason="not_planned"` — binds the normalized URL, state and reason | [Workflow workspaces](./workflow-workspace-spec.md) §10.3 | +| `Project.Status` | `` — binds the normalized `{ item, field, option }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.6 | +| `Git.Merge` | ``, or the same form with `purpose="publish"` — binds a closed clean-or-conflicted result | [Workflow workspaces](./workflow-workspace-spec.md) §7.8 | +| `Git.PublishTarget` | `` — binds normalized target, expected and published evidence | [Workflow workspaces](./workflow-workspace-spec.md) §7.9 | +| `Evidence.Run` | ``, where `commands` is a structured argv list — binds ordered bounded results | [Workflow workspaces](./workflow-workspace-spec.md) §10.5 | + +Two things a reader looking for a component will not find here. The remote +`WorkflowHost` is a host API — start, lookup, execute, answer and inspect — and +not an element a document writes. The factory's protocol records — stage, +frontier, implementation revision, role outcome, invalidation, conflict, +terminal decision and terminal settlement — are closed parsers over immutable +values, not components and not a lifecycle controller beside the journal. +[The software factory](./github-actions-software-factory-spec.md) §11 owns both. + +None of these constructs is available to a workflow Agent, and none appears in +any generated-XMD read or write table +([Workflow workspaces](./workflow-workspace-spec.md) §§8.3-8.4). The standard +write table remains exactly core's paired `File:write`, the composition +package's lexical `Dir` and core's self-closing `File.Delete`. + ## 7. Entry point ### 8.1 `execute` @@ -9998,6 +10041,153 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §8. |---|------|--------| | WFX1 | SIGKILL and resume | A real `SIGKILL` part-way through leaves the run `running` with the effects that committed; the resume replays those exact events by id, performs the rest once each with no duplicate and no gap, advances the current root, and completes | +### The software-factory tiers + +The seven tiers below are the frozen evidence names for the software factory +specified by +[the software factory](./github-actions-software-factory-spec.md) and by +[Workflow workspaces](./workflow-workspace-spec.md) §§3.8, 7.8-7.10, 10.5-10.7 +and 13.2. Every construct and host they name is **specified; implementation +unbuilt**, so these tiers name the scenarios an implementation is accepted +against rather than tests that exist. Each lists the finite structural +scenarios — success, refusal, stale authority, interruption and cancellation, +teardown, replay, and denied Agent or generated-XMD authority — and no +malformed-input permutation without a distinct structural consequence. + +### Tier WRH — Remote host, executor and delivery separation + +Defined in [Workflow workspaces](./workflow-workspace-spec.md) §3.8 and §13.2 +and [Workflow runs](./workflow-spec.md) §9.8. + +| # | Test | Verify | +|---|------|--------| +| WRH1 | One owner | A run ID selects one durable owner arithmetically; two admissions of one ID reach that owner and no second registry answers | +| WRH2 | Acquisition is a connection | Start, resume, stale recovery, document execution, Workspace mutation, provider attachment, lifecycle transition, accepted-outcome publication and terminal settlement each validate the exact live acquisition and the expected Workspace root inside their own mutating transaction | +| WRH3 | A second executor | A second connection for a live run follows or is refused, and advances nothing either way | +| WRH4 | Stale authority | A closed, foreign or superseded acquisition reaches no mutation; a run left `running` by a closed connection is recovered by the next acquisition from the exact committed frontier | +| WRH5 | Runner crash | A runner killed between materializing a root and submitting changes leaves a prior or a new complete transaction and never a partial one | +| WRH6 | Content-addressed transfer | The owner refuses a submission whose acquisition, expected root or content does not validate, and publishes the new root and the filtered journal result atomically when it does | +| WRH7 | Delivery is not execution | An answer and a terminal decision each retain against their exact subject while taking no acquisition, beginning no execution, attaching no provider, appending no journal event and changing no run status | +| WRH8 | Delivery correlation | A value for a subject the run is not holding, a duplicate delivery and a spent delivery are each refused with nothing written | +| WRH9 | Consumption | A later executor consumes the retained value inside the run's transaction and appends the accepted event exactly once | +| WRH10 | Inspection | Status and history read immutable snapshots, take no acquisition, and authorize no transition | +| WRH11 | Teardown | Closing the connection releases executor ownership and rolls back nothing already committed | +| WRH12 | Completed replay | A completed run replays attaching no remote storage session, Workspace, Agent, process, Git, Git-host, Issue, Project or credential provider | +| WRH13 | Host neutrality | Shared WorkflowRun modules import nothing Cloudflare-specific and detect no runtime; the runtime-named entrypoint is the only place the topology appears | + +### Tier WGI — Authenticated GitHub ingress + +Defined in [the software factory](./github-actions-software-factory-spec.md) §5. + +| # | Test | Verify | +|---|------|--------| +| WGI1 | Order | The webhook signature is verified before the payload is parsed as anything but bytes, and the complete objects are reread through the API before authorization is decided | +| WGI2 | Bad signature | An unsigned or wrongly signed delivery is refused before parsing and retains no intake | +| WGI3 | Bounded intake | One intake is retained per delivery or submission identity, holding only typed bounded fields | +| WGI4 | Duplicates | A repeated delivery of one identity finds the retained intake and writes nothing | +| WGI5 | Unavailable is not absent | A missing result page, an unavailable field, an ambiguous object and a partial permission read are each unavailable, and none of them admits an item | +| WGI6 | Admission ceiling | Admission requires the configured organization-owned Project and the exact repository, Project, item, status field and allowed option IDs | +| WGI7 | Dispatch carries nothing | `repository_dispatch` carries only the retained intake identity; a payload naming a stage, outcome, answer, decision, transition, credential or definition is refused | +| WGI8 | OIDC claims | Admission validates issuer, audience, repository ID, repository-owner ID, event name, workflow ref and SHA, and the configured workflow identity; a valid token for another repository or workflow admits no session | +| WGI9 | Human floor | An actor without Project write and repository write-or-higher authorizes no admission, answer, change, merge or abandonment | +| WGI10 | Comments are not authority | A comment naming an answer, a merge, an abandonment or a resume changes nothing | +| WGI11 | Secrets | No private key, webhook secret, OIDC configuration, installation token, endpoint, raw payload, cursor or host path appears in props, context data, a durable record, a comment, output or a diagnostic | +| WGI12 | Run identity | One issue admitted twice derives one run ID; a reread returning a different node ID for the same subject refuses as drift and creates no second run | + +### Tier WGE — Reconciled GitHub and Project projections + +Defined in [Workflow workspaces](./workflow-workspace-spec.md) §7.10, §10.3 and +§10.6. + +| # | Test | Verify | +|---|------|--------| +| WGE1 | Comment identity | A comment's natural key is its subject plus the engine-derived effect identity; a re-rendered or edited body is the same comment and produces no second one | +| WGE2 | Ready and close | `PullRequest.Ready` and `PullRequest.Close` are keyed by their exact subject; an already-ready pull request is adopted and a merged one conflicts | +| WGE3 | Issue closure | `Issue.Close` adopts an issue already closed with the same reason and conflicts with one closed under the other reason | +| WGE4 | Project pre-state | `Project.Status` is keyed by exact item plus field, adopts an item already at the requested option, and performs once from another allowed option | +| WGE5 | Project unavailability | An unreadable board, unavailable field, ambiguous item and partial permission read are unavailable rather than absent, and none of them mutates | +| WGE6 | Ceilings | A project, item, field or option outside the host ceiling is refused; no authored prop widens it | +| WGE7 | Boundaries stay separate | An Issue-provider effect, a Git-host effect and a Project effect journal their own types and no adapter answers another's request | +| WGE8 | Form before provider | A missing prop, an unknown prop, a value outside a closed enum and a missing `as` each fail before a provider, ceiling or credential is reached | +| WGE9 | Cancellation | A cancelled effect tears the provider call down and publishes no completion | +| WGE10 | Interruption | An interrupted remote completion is reobserved and adopted only when it matches the retained intent | +| WGE11 | Replay | A completed record replays without contacting a provider | +| WGE12 | Projection is not authority | A board, a comment or a draft state ahead of the journal is reconciled as drift and never accepted as proof that a stage passed | +| WGE13 | Denied to generated XMD | A fragment naming any of these constructs is refused in the preflight, before any generated effect, and the standard write table still holds exactly `File:write`, `Dir` and `File.Delete` | + +### Tier WGM — Ordered merge and target publication + +Defined in [Workflow workspaces](./workflow-workspace-spec.md) §7.8 and §7.9. + +| # | Test | Verify | +|---|------|--------| +| WGM1 | Clean merge | A clean `Git.Merge` publishes the commit, the new Workspace root and the filtered result in one transaction | +| WGM2 | Conflict | A conflicted merge restores the pre-merge root, publishes normalized conflict evidence against it, and offers no mutation under that evidence | +| WGM3 | Parent order | `purpose="synchronize"` retains `[implementationHead, targetBase]` and `purpose="publish"` retains `[reviewedBase, reviewedHead]`; neither purpose reorders what it was given | +| WGM4 | Stale identities | A changed head, base, merge base, Repository identity, conflict set or Workspace root makes a retained conflict admission stale, and nothing mutates under it | +| WGM5 | Interruption | A host killed between the merge and the commit leaves the checkout, the current root and the effect history unchanged | +| WGM6 | Compare-and-swap | `Git.PublishTarget` updates the ref only after observing the target equal to the expected commit | +| WGM7 | Adoption | A target already equal to the exact source commit is adopted with nothing performed | +| WGM8 | Race | A target moved to a third commit refuses without mutating, and the exact-revision reviews invalidate rather than the publication proceeding | +| WGM9 | Ceilings | Remote, ref, credential and non-force policy are host-owned; no authored prop sets or widens them | +| WGM10 | Distinct operations | `Git.Push`, `Git.PublishTarget`, pull-request upsert, ready, close and the trusted merge are six operations; no force, force-with-lease, rebase, reset or host squash appears in any command trace or retained configuration | +| WGM11 | Replay | Completed merge and publication records replay without running Git and without contacting a Git host | +| WGM12 | Denied to the Agent | Neither construct is reachable by a workflow Agent or by an admitted generated fragment | + +### Tier WER — Trusted evidence execution + +Defined in [Workflow workspaces](./workflow-workspace-spec.md) §10.5. + +| # | Test | Verify | +|---|------|--------| +| WER1 | Structured argv | `commands` is an ordered list of non-empty argument vectors; no shell string, interpreter or quoting layer takes part, and the retained argv is what ran | +| WER2 | Root | The commands run against the exact retained Workspace root the host materialized | +| WER3 | Ceilings | Executable, environment, time and output ceilings are host-owned and refuse rather than truncating silently what they were not given | +| WER4 | Results | The binding is one ordered bounded result per command, carrying its argv, exit status and bounded output | +| WER5 | Location | The execution happens on the trusted runner; the durable owner runs no native process | +| WER6 | Cancellation and teardown | A cancelled run terminates its child before publishing, and no child outlives the effect | +| WER7 | Replay | A completed record replays running nothing | +| WER8 | Denied authority | `Evidence.Run` appears in no Agent capability and in no generated-XMD read or write table; a fragment naming it is refused before any generated effect | + +### Tier WFP — Factory protocol and frontier + +Defined in [the software factory](./github-actions-software-factory-spec.md) +§§1-2 and §12. + +| # | Test | Verify | +|---|------|--------| +| WFP1 | Run identity | The run ID equals the §1.1 derivation for its issue; a changed repository name, issue number, Project identity, comment, branch, revision, definition SHA, delivery ID or actor changes it not at all | +| WFP2 | Adjacency | An outcome advancing more than one stage is rejected, and the current stage's handoff commits before the next role is invoked | +| WFP3 | Same-stage amendment | A later accepted same-stage output replaces the frontier while the superseded output stays readable in the journal | +| WFP4 | Backward destinations | Each of the six destinations in §2.2 deactivates exactly the downstream handoffs its row names and requires every later stage again | +| WFP5 | Head invalidation | A changed `headSha` places the frontier at Stage 4 and invalidates Stages 5-7 | +| WFP6 | Base invalidation | A base-only move places the frontier at Stage 5, and a base move requiring synchronization or implementation work places it at Stage 4 | +| WFP7 | Exact subjects | Stage 6 accepts only a Planner verdict naming the same revision, and Stage 7 only a review chain naming the current one | +| WFP8 | Ready authority | Only an accepted Stage 6 verdict for the current revision takes the pull request out of draft; observed ready state manufactures no verdict | +| WFP9 | Closed parsers | Stage, frontier, revision, outcome, invalidation, conflict, decision and terminal records parse as closed immutable values, and an unrecognized shape refuses rather than being read loosely | +| WFP10 | Definition identity | Definition incompatibility is a workflow lifecycle refusal and never an implementation correction | +| WFP11 | Journal is authority | A Project status, comment or pull-request state ahead of the journal is drift and authorizes no stage | + +### Tier WFL — Authored factory lifecycle and terminal settlement + +Defined in [the software factory](./github-actions-software-factory-spec.md) +§§8-10. + +| # | Test | Verify | +|---|------|--------| +| WFL1 | Clean synchronization | Observe, merge, record the revision, remain at Stage 4, run evidence, push the descendant, offer the new pair to Stage 5 — as separate durable effects, resuming from the first uncommitted one | +| WFL2 | Conflict suspension | Every conflict returns a closed conflicted result, restores the pre-merge root, retains normalized evidence, publishes a handoff and suspends | +| WFL3 | No automatic resolution | No generated fragment, structured text-conflict capability, rebase, force, force-with-lease or reset resolves a conflict | +| WFL4 | Manual resolution | A pushed resolution is observed as a new Stage 4 revision, receives new evidence, and inherits no Stage 5-7 conclusion | +| WFL5 | Merge decision first | A merged decision is retained before any effect is attempted | +| WFL6 | Merge ordering | Merge construction, target publication, the merged observation, issue completion, the Project move and terminal settlement are separate reconciled steps in that order | +| WFL7 | Terminal is last | The run becomes terminal `merged` only after every required projection completes | +| WFL8 | Abandonment | An exact-revision authenticated abandonment carrying a reason is retained first, then pull-request close unmerged, issue close as `not_planned`, the Project move and settlement; terminal `abandoned` follows all of them | +| WFL9 | Retention | Both terminal kinds retain the actor, exact revision, resulting provider identities and required reason, and retain the Project item, branch, comments, journal, Workspace roots and Agent evidence | +| WFL10 | Interrupted terminal | An interruption between two projections resumes at the first uncommitted one and publishes no terminal status until the rest complete | +| WFL11 | Provider-free replay | A terminal run replays attaching no provider | +| WFL12 | Reopening | Reopening the issue does not reopen the completed run; continuing requires a new linked issue and therefore a new run | + ### Tier SL — Own-scope context updates | # | Test | Verify | diff --git a/specs/github-actions-software-factory-spec.md b/specs/github-actions-software-factory-spec.md index ff1a0ef41..edba90d25 100644 --- a/specs/github-actions-software-factory-spec.md +++ b/specs/github-actions-software-factory-spec.md @@ -1,47 +1,103 @@ # GitHub Actions-hosted AI Software Factory -This specification defines an issue-driven software factory whose durable -procedure is an XMD workflow and whose invocation host is GitHub Actions. One +You open one GitHub issue and the factory carries it to a merged pull request or +an explicit abandonment, without anybody having to remember where it got to. One GitHub Project item shows who owns the work now; the retained XMD run proves how -the item reached that owner. - -The factory has no independent controller. GitHub Actions starts or resumes the -XMD workflow and supplies an authorized GitHub environment. The XMD workflow -invokes roles, validates their structured outcomes, records handoffs, performs -authorized GitHub and Workspace effects, updates the Project status, and -journals those effects. - -## 1. One item, one durable run +the item reached that owner. Every stage the item passed through, every revision +that was reviewed, and every effect that reached GitHub are in one durable +journal addressed by the issue itself. + +The factory has no independent controller. A durable XMD workflow run is the +whole procedure. GitHub Actions supplies an ephemeral trusted runner; a +Cloudflare Durable Object supplies the run's durable state; a dedicated GitHub +App supplies authenticated ingress and the credential every GitHub effect is +performed with. The XMD workflow invokes roles, validates their structured +outcomes, records handoffs, performs authorized GitHub and Workspace effects, +updates the Project status, and journals those effects. + +Three planes are separate throughout, and keeping them separate is what the rest +of this specification spends its length on: + +- the **executor plane**, one authenticated WebSocket connection that advances + the run; +- the **delivery plane**, authenticated transactions that retain an intake, an + answer or a decision without executing anything; and +- the **inspection plane**, read-only reads that can never become transition + authority. + +## 1. One issue, one durable run One issue corresponds to one durable XMD factory run. The run may produce and review many implementation revisions before it closes; a new implementation commit never creates a new factory run. -Two SHA identities remain separate throughout that run: +### 1.1 The run ID is derived from the issue + +Admission rereads the GitHub issue from the API before it derives anything. The +**factory run ID** is then the lowercase unpadded RFC 4648 Base32 encoding of +the full SHA-256 digest of these UTF-8 bytes, concatenated in this order: + +```text +"github-issue-v1" || 0x00 || canonical GitHub authority || 0x00 || issue node ID +``` + +The digest is all 32 bytes, so the run ID is 52 Base32 characters with no +padding and no separators. It is a public run ID in the sense §9 of the workflow +specification already defines: non-empty, containing no NUL, opaque to +everything but equality and lifecycle addressing. + +The **canonical GitHub authority** is the lowercase DNS hostname of the GitHub +deployment, plus `:` and the port when the port is not the scheme's default. It +carries no scheme, path, query, fragment, user information or trailing +separator, so one deployment has exactly one spelling. The **issue node ID** is +the exact string GitHub's GraphQL API returns for that issue, compared byte for +byte with no case folding and no Unicode normalization: it is an opaque provider +identity, and normalizing it would be inventing a second one. + +Nothing mutable participates. Repository names, issue numbers, Project, Project +item and status identities, comments, branch names, implementation revisions, +the workflow definition SHA, webhook delivery IDs and actor identities all +change while the run stays the run it was, so none of them is an input to the +derivation. + +Two consequences follow directly. Duplicate admission for the same authenticated +subject derives the same run ID and therefore routes to the same run rather than +creating a second one — the compatible-reuse rule of the workflow contract does +the rest. And a reread that returns a different retained provider identity for +the same subject, including an issue transfer that changes the node ID, is +unsupported drift: the host refuses it, names it as drift, and creates no second +run. Silently starting another run would leave two frontiers claiming one piece +of work. + +### 1.2 Two SHA identities, and neither is the run + +Two SHA identities remain separate throughout the run, and neither is the run +ID: - The **workflow definition SHA** is the immutable Git commit in the XMD workflow definition. It fixes the procedure and its component bundle for the - lifetime of the run. A different definition is not a revision of the same - run; it requires a new run or an eligible history fork under the workflow - lifecycle contract. -- The **implementation revision** is the evolving pair - `{ headSha, baseSha }`. `headSha` is the exact commit at the draft pull - request's head, and `baseSha` is the exact target-branch commit against which - that head is evaluated. + lifetime of the run. A different definition is not a revision of the same run; + it requires a new run or an eligible history fork under the workflow lifecycle + contract. +- The **implementation revision** is the evolving pair `{ headSha, baseSha }`. + `headSha` is the exact commit at the draft pull request's head, and `baseSha` + is the exact target-branch commit against which that head is evaluated. The definition SHA authorizes which procedure executes. The implementation revision identifies what the procedure is currently producing or reviewing. -Neither substitutes for the other, and neither a Project field nor a comment -may rewrite either identity. +Neither substitutes for the other, and neither a Project field nor a comment may +rewrite either identity. ## 2. Lifecycle and validation frontier +![Ownership bands across the eight factory stages](./assets/github-actions-software-factory-ownership-bands.svg) + The Project status has these values: | Stage | Status | Owner | Question answered | | ---: | --- | --- | --- | | 0 | Backlog | User | Is this item admitted to the factory? | -| 1 | Product Owner | User | What product outcome and acceptance boundary are intended? | +| 1 | User | User | What product outcome and acceptance boundary are intended? | | 2 | Architect | Architect | Is the structural contract ready? | | 3 | Planner | Planner | Is there an implementation-ready plan and evidence matrix? | | 4 | Implementor | Implementor | Does an implementation revision satisfy the accepted plan? | @@ -50,14 +106,14 @@ The Project status has these values: | 7 | User Review | User | Is this exact validated result accepted? | | 8 | Closed | None | Was the item merged or abandoned? | -The left side progressively removes uncertainty: product intent, structural -contract, implementation plan, then code. The right side validates the result: -implementation evidence, structural correctness, user acceptance, then -completion. +Those nine strings are the exact status vocabulary. The diagram above shows the +same eight stages as two ownership bands: the left side progressively removes +uncertainty — product intent, structural contract, implementation plan, then +code — and the right side validates the result — implementation evidence, +structural correctness, user acceptance, then completion. -Moving an authorized Backlog item to Product Owner admits it and starts its -factory run. Once admitted, an ordinary successful outcome advances exactly one -stage: +Moving an authorized Backlog item to User admits it and starts its factory run. +Once admitted, an ordinary successful outcome advances exactly one stage: ```text 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 @@ -83,11 +139,13 @@ Self-correction is iteration, not progress: passes. - An Implementor producing, synchronizing, or correcting an implementation remains at Stage 4 until the latest implementation revision passes. -- A reviewer replacing a malformed or incomplete verdict remains at that - review stage until a valid verdict identifies the exact revision reviewed. +- A reviewer replacing a malformed or incomplete verdict remains at that review + stage until a valid verdict identifies the exact revision reviewed. The role's latest accepted same-stage output supersedes its earlier output at -the active frontier without erasing history. +the active frontier without erasing history. The superseded output stays in the +journal and stays readable; what it loses is the authority to carry the item +forward. ### 2.2 Backward invalidation @@ -96,7 +154,7 @@ invalidates. Forward progress then traverses every later stage again: | Returned to | Still valid | Must run again | | --- | --- | --- | -| Product Owner (1) | Nothing downstream | 2-7 | +| User (1) | Nothing downstream | 2-7 | | Architect (2) | Product decision | 2-7 | | Planner (3) | Product decision and architecture | 3-7 | | Implementor (4) | Product decision, architecture, and plan | 4-7 | @@ -105,17 +163,16 @@ invalidates. Forward progress then traverses every later stage again: The workflow validates a backward outcome against this frontier. The role supplies the reason and earliest invalidated stage; the XMD procedure decides -which accepted downstream handoffs become inactive and records that decision. -A Project edit alone never proves invalidation or approval. +which accepted downstream handoffs become inactive and records that decision. A +Project edit alone never proves invalidation or approval. ### 2.3 Revision invalidation Every Stage 5, Stage 6, and Stage 7 conclusion identifies the exact `{ headSha, baseSha }` it evaluated. -- Any merge, rebase, manual edit, generated mutation, conflict resolution, or - other change to `headSha` remains at or returns to Stage 4 and invalidates - Stages 5-7. +- Any merge, manual edit, conflict resolution, or other change to `headSha` + remains at or returns to Stage 4 and invalidates Stages 5-7. - Movement of `baseSha` without a head change invalidates the review context at Stage 5. The item repeats Stages 5-7 against the new pair when no synchronization or implementation correction is required. @@ -124,61 +181,264 @@ Every Stage 5, Stage 6, and Stage 7 conclusion identifies the exact - An invalidated Planner verdict returns to Stage 5. An invalidated Architect verdict returns to Stage 6. -Stage 6 may accept a Planner verdict only when that verdict names the same -implementation revision. Stage 7 may accept the review chain only when both -review verdicts name the current revision. A later SHA never inherits a verdict -for an earlier pair. +Stage 6 accepts a Planner verdict only when that verdict names the same +implementation revision. Stage 7 accepts the review chain only when both review +verdicts name the current revision. A later SHA never inherits a verdict for an +earlier pair. + +Only an accepted Stage 6 verdict for the current revision is **ready +authority**: it is what authorizes XMD to take the pull request out of draft and +move the item to Stage 7. Ready state observed on GitHub without that accepted +verdict is drift to reconcile, never a substitute for it. The workflow definition SHA does not participate in this invalidation table. Definition incompatibility is a workflow lifecycle refusal, not an implementation correction. -## 3. Durable procedure and GitHub projection +### 2.4 Terminal kinds + +Stage 7 -> 8 is the adjacent terminal transition, and it has exactly two kinds. +A **merged** terminal records that the reviewed revision reached the target +branch. An **abandoned** terminal records that an authorized human ended the run +without publishing it. Both are described in §10, which owns their effect +ordering; what matters here is that they are the only two ways an admitted run +becomes terminal, and that each names the exact reviewed revision it settled on. + +## 3. Deployment topology + +Three hosts carry one run, and each owns something the others cannot reach. + +```text +GitHub App (ingress, credentials) + │ webhook / form / repository_dispatch + ▼ +Cloudflare Durable Object ── SQLite ── WorkflowRun, Workspace roots, journal + ▲ + │ authenticated WebSocket (executor acquisition) + ▼ +GitHub Actions ephemeral runner ── native Git, evidence processes, Agent clients +``` + +**One SQLite-backed Cloudflare Durable Object, selected from the run ID, owns +the run.** It holds the WorkflowRun record and its filtered journal, the +immutable Workspace roots and their content-addressed bytes, the Agent-session +mappings and checkpoints, pending answers and Stage 7 decisions, the retained +intake records, and executor ownership. There is one durable owner per run, +selected arithmetically from the run ID exactly as local discovery is, so no +second registry can disagree with it. + +**Exported `.xmd` artifacts are immutable evidence only.** They are never live +state, discovery, continuation, answer delivery, or lock authority, and no +Actions artifact is any of those either. An artifact says what a run had +committed at one frontier; it does not say what a run may do next. + +**One authenticated Durable Object WebSocket connection owns the executor +acquisition.** The acquisition is the connection's lifetime and nothing else. It +has no duration, expiry, renewal, heartbeat, PID, liveness poll or application +lease, exactly as the local executor lock has none. Closing the WebSocket +invalidates the acquisition and releases executor ownership; it does not roll +back state that already committed. A second healthy executor follows the active +one or is refused, and cannot advance the run either way. + +**The ephemeral Actions runner executes what cannot run inside a Durable +Object.** Native Git, plan evidence processes and Agent clients run there, +against bounded materialized state, and only there. The runner materializes one +selected retained Workspace root, submits content-addressed changes, and the +Durable Object validates the acquisition, the expected root and the content +before it atomically publishes the new root together with the filtered journal +result. A runner crash therefore exposes only a prior or a new complete +transaction; a later connection performs stale recovery and resumes from the +exact committed WorkflowRun and Workspace frontier. + +**The Cloudflare runtime-named host owns persistence and admission.** Durable +transactions, intake, the authorization gates, token minting and executor +admission are its. Shared production modules stay provider-neutral: they do not +detect Cloudflare, Deno, GitHub Actions or any other runtime, and they reach +every host-specific behavior through the contextual APIs that already exist. + +**A completed replay attaches nothing.** It contacts no remote storage session +and attaches no Workspace, Agent, process, Git, GitHub, Project or credential +provider. That is the property §10's terminal ordering exists to preserve. + +## 4. Durable procedure and GitHub projection The XMD workflow is the only procedure that may change the factory stage. On a -start or resume it: +start or resume the executor: -1. acquires the durable run's executor lock; +1. acquires the run's executor connection; 2. restores the retained workflow definition, Workspace, handoff chain, and incomplete effects; 3. observes the issue, draft pull request, Project item, and exact Git revision identities required by the current stage; 4. reconciles interrupted GitHub effects under their retained identities; -5. renders the active handoff chain and authorized observations to the current +5. consumes any retained answer or Stage 7 decision inside the run's + transaction, appending its accepted durable event exactly once; +6. renders the active handoff chain and authorized observations to the current role; -6. validates the role's structured outcome against the current stage, artifact +7. validates the role's structured outcome against the current stage, artifact identities, and implementation revision; -7. records a same-stage iteration or invalidates the frontier, or performs the +8. records a same-stage iteration or invalidates the frontier, or performs the immediately adjacent successful transition; -8. performs the authorized issue, pull-request, Git, and Project effects; and -9. journals the accepted decision and every effect before yielding the next - durable boundary. +9. performs the authorized issue, pull-request, Git, and Project effects; and +10. journals the accepted decision and every effect before yielding the next + durable boundary. + +Every step above is executor-owned. Start and resume, stale-execution recovery, +document execution, Workspace mutation, Agent attachment, native Git and +evidence execution, lifecycle transition, accepted-outcome publication and +terminal settlement each validate the exact live acquisition and the expected +Workspace root inside every mutation transaction. An acquisition that is closed, +foreign or stale reaches no mutation. + +Two kinds of operation are deliberately outside that list. + +**Delivery is not execution.** Authenticated webhook and form intake, typed +answer delivery, and Stage 7 decision delivery are delivery-plane transactions. +Each takes no executor acquisition, starts no execution, attaches no Workspace, +Agent or process provider, appends no lifecycle outcome and changes no run +status. Each validates its exact delivery identity and its exact pending +suspension or decision subject, and retains only the typed bounded value that +subject describes. This is the answer-delivery contract the workflow +specification already states, applied unchanged to a remote host and extended to +Stage 7 decisions. + +**Inspection is not authority.** Status and history reads are read-only, take no +executor acquisition, and cannot become transition authority. They observe the +same immutable snapshot surface local inspection observes. + +A later executor is what turns a retained answer or decision into progress. It +consumes the retained value inside the run's own transaction, appends the +accepted durable event or outcome once, and only then may authored XMD choose +the next transition. Delivery stores; execution decides. GitHub Actions supplies the trusted executable, the definition reference, the run identity, and credentials or provider configuration within a fixed ceiling. Its YAML does not parse role conclusions, choose stages, construct handoffs, change draft state, merge, comment, or update the Project independently of the -XMD workflow. +XMD workflow. Actions concurrency may reduce duplicate invocations, but it is +not the executor acquisition and cannot authorize a transition. -Actions concurrency may reduce duplicate invocations, but it is not the -executor lock and cannot authorize a transition. A second invocation of the -same item follows or is refused by the durable run lifecycle. - -The Project status is a human-facing projection of the journaled current stage. -Issue and pull-request comments are human-readable transition records. The XMD -journal is the durable execution record of which source records were accepted, -which role conclusion won, and which external effects completed. A Project -status ahead of the journal is drift, not proof that omitted roles passed. +The XMD journal is lifecycle authority. The Project status is a human-facing +projection of the journaled current stage, and issue and pull-request comments +are human-readable transition records. A Project status, comment or pull-request +state ahead of the journal is drift to reconcile, never proof that a skipped +role passed. GitHub offers no transaction spanning a comment, draft state, Project status, -branch update, or merge and the retained XMD store. Each is therefore a durable -external effect with stable identity and reconciliation: observe, adopt an -already-compatible result, perform from proven absence or compatible pre-state, -or refuse conflict and ambiguity. Interruption may leave GitHub ahead of the -local result; resume reconciles the same intended effect rather than repeating -it blindly. - -## 4. Issue and draft pull-request boundary +branch update, or merge and the retained XMD store, and none is claimed across +Durable Object state, native Git and processes, GitHub and Project V2. Each +GitHub mutation is therefore one durable external effect with a stable +engine-derived identity and an effect-specific natural key: it observes before +it mutates, adopts only a compatible completion, performs once from proven +absence or an exact compatible pre-state, and refuses conflict, permanent +ambiguity, incomplete observation and temporary unavailability. Cancellation +tears the provider call down and publishes no invented completion; an +interrupted remote completion is reobserved and adopted only when it matches the +retained intent; a completed replay contacts no provider at all. + +The natural keys are exact. A comment uses its subject plus the engine-derived +effect identity, so the body and title are presentation rather than identity. A +Project status uses the exact Project item plus field. Ready and close use their +exact issue or pull-request subject. Target publication uses the retained +repository plus the configured remote and target ref. Accepted outcomes are +retained before their projections are attempted, and reconciliation completes +each intended projection. + +## 5. Authenticated ingress + +A dedicated GitHub App is the factory's only ingress. It receives Project and +admission webhooks and authenticated human form submissions, and it is the +principal every GitHub effect is performed as. + +### 5.1 Order of operations + +An intake is admitted in this order, and no step may be reordered: + +1. **Verify before parsing.** The webhook signature is verified against the + App's webhook secret before the payload is parsed as anything but bytes. +2. **Reread from the API.** The complete GitHub objects — issue, repository, + Project, Project item, status field — are reread through the API. The + payload's copy of them is a notification, not a source of truth. +3. **Authenticate the installation and the actor.** The installation is resolved + for the exact repository, and the human actor is resolved as an identity + rather than a display name. +4. **Retain one bounded intake.** The intake is keyed by the GitHub delivery + identity for a webhook, or the submission identity for a form, and retains + only bounded typed fields. + +Admission requires a configured organization-owned Project V2 and the exact +repository, Project, Project item, status field and allowed option IDs. A +missing page of results, an unavailable field, an ambiguous object and a partial +permission read are all **unavailable** — they are neither absence nor +authorization. Treating an unreadable Project as an empty one is how an +unauthorized item would be admitted. + +A duplicate delivery of the same identity finds the retained intake and changes +nothing. That is the same compatible-reuse rule run identity uses, applied to +intake. + +### 5.2 Waking Actions carries no decision + +`repository_dispatch` carries only the retained intake identity. The receiving +Actions workflow may be woken by it, but the payload never carries or derives a +stage, a role outcome, answer text, a decision, a transition, a credential, or a +mutable factory definition. Everything the run needs it reads from the Durable +Object after it has authenticated. + +### 5.3 The runner authenticates with OIDC + +The Actions job authenticates to the provider through GitHub OIDC. Before it +admits a session, the provider validates the issuer, the configured audience, +the repository ID, the repository-owner ID, the event name, the workflow ref and +SHA, and the configured immutable workflow identity. Repository *names* are +mutable and are not what is checked; IDs are. + +### 5.4 Human answers and decisions + +Human answers and Stage 7 decisions arrive through a GitHub-App-authenticated +web form bound to the exact retained suspension or decision subject. The form +submission is a delivery-plane transaction under §4. + +**Comment text is never authority.** A comment never answers a question, merges, +authorizes a change, abandons a run or resumes execution. Comments are +transition records a person reads. + +A human may authorize admission, an answer, a Stage 7 change, a merge or an +abandonment only while holding Project write and repository write-or-higher +access, checked at the moment of the submission. The short-lived App +installation token performs the GitHub effects that follow; the journal retains +the human actor separately from the token that acted, so the record says who +decided as well as what was done. + +## 6. Principal, permissions and host ceilings + +The GitHub App has exactly these repository permissions — Metadata read, +Contents write, Issues write, Pull requests write, Checks read, Commit statuses +read — and organization Projects write. It has no Administration, Actions write, +Workflows write, Secrets, Environments, Deployments or Members permission, and +no force-push authority anywhere. + +The App installation is limited to configured repositories. The host narrows +further, per operation, to the exact repository, implementation branch, target +branch, organization Project, status field, option IDs, issue or pull-request +subject, reviewed revision, parent pair, and non-force operation. A ceiling the +installation grants is not a ceiling the host uses. + +Mutation of `.github/workflows/**` is refused, even though Contents write could +otherwise reach it. A factory that can rewrite the workflow that runs it is a +factory that can rewrite its own authorization. + +GitHub App private keys, webhook secrets, OIDC verification configuration, +issued installation tokens, provider endpoints, raw payloads, cursors and host +paths are provider secrets and closure state. None of them enters props, context +composition data, durable requests or results, comments, output or diagnostics. + +The configured target ruleset admits only this factory's dedicated App for the +target-branch update of §10, and host validation independently enforces the +target and the reviewed parent pair. Neither substitutes for the other: the +ruleset says who may write, the host says what may be written. + +## 7. Issue and draft pull-request boundary The issue owns product intent, architecture, planning, and the transition into implementation. @@ -191,8 +451,8 @@ implementation. Stage 4 creates or updates a draft pull request before its first handoff to Planner Review. The Stage 4 -> 5 handoff identifies the draft pull request and -its exact `{ headSha, baseSha }`. The draft pull request then owns implementation -iterations and Stage 5-7 review handoffs. +its exact `{ headSha, baseSha }`. The draft pull request then owns +implementation iterations and Stage 5-7 review handoffs. When invalidation crosses from the pull request to Stages 1-3, the XMD workflow writes the full handoff on the issue and a short linking record on the pull @@ -200,62 +460,65 @@ request. When the accepted issue chain crosses back into Stage 4, the draft pull request links the accepted issue handoff before implementation continues. The pull request remains draft throughout Stage 4 corrections and Stage 5 -review. It also remains draft while Stage 6 requests a backward correction. -Only an accepted Stage 6 verdict for the current revision authorizes XMD to make -the pull request ready and move the item to Stage 7. If interruption separates +review. It also remains draft while Stage 6 requests a backward correction. Only +the ready authority of §2.3 takes it out of draft. If interruption separates those GitHub effects, the run remains at its last journaled frontier and resume -reconciles both; ready state alone does not manufacture an Architect verdict. +reconciles both. -Stage 7 -> 8 is the adjacent terminal transition. It records `merged` or -`abandoned`, the reviewed implementation revision, the actor and decision that -authorized closure, and the resulting merge identity when one exists. Closing -the Project item never erases the issue, pull request, handoff history, or XMD -journal. +Closing the Project item never erases the issue, pull request, handoff history +or XMD journal. -## 5. Stage 4 base synchronization +## 8. Stage 4 base synchronization -The initial factory synchronizes a draft implementation branch by merging the -latest observed target base into it. It does not rebase published work. +The factory synchronizes a draft implementation branch by merging the latest +observed target base into it. It does not rebase published work, and it never +force-pushes. -A merge preserves published commit identities, produces a descendant that the -normal non-force Push effect can publish, and allows the final pull request to -use a squash merge when the repository's delivery policy wants a compact target -history. A rebase changes published identities and requires a separately -specified, reconciled force-with-lease effect. No factory role, generated XMD, -or GitHub Actions step has that effect in the initial contract. Force pushes are -refused. +A merge preserves published commit identities and produces a descendant the +ordinary non-force Push effect can publish. Rebase, force push, force-with-lease +and reset-based replacement are absent from this contract; adding any of them +later takes a new external-effect contract with its own reconciliation semantics +and invalidation proof, and cannot be represented as another spelling of Push. A clean base synchronization follows this durable sequence: 1. XMD observes and checkpoints the exact implementation head, target base, and merge base. 2. XMD performs the trusted merge against those identities inside the retained - Workspace. + Workspace, with parents ordered `[implementationHead, targetBase]`. 3. XMD records the resulting merge commit and Workspace root as a new implementation revision. 4. The item remains at Stage 4 because its head changed. -5. XMD runs the implementation evidence selected by the accepted plan. +5. XMD runs the implementation evidence the accepted plan selected. 6. When that evidence passes, XMD publishes the exact descendant through the ordinary non-force Push effect and offers the latest `{ headSha, baseSha }` to Planner Review. The merge, evidence, push, and Stage 4 pass are separate durable effects. No -transaction is claimed across native Git, test processes, GitHub, and the +transaction is claimed across native Git, evidence processes, GitHub and the Project. Resume restores completed effects and continues from the first uncommitted one. A manually changed branch is not adopted as a passed implementation. XMD first -observes its new exact head and base, records a new implementation revision, -and re-enters Stage 4. The Implementor and selected evidence evaluate that +observes its new exact head and base, records a new implementation revision, and +re-enters Stage 4. The Implementor and the selected evidence evaluate that revision before it can return to Stage 5. -## 6. Conflict boundary +## 9. Conflict boundary -Tool-less conflict handling grants no tool or checkout to the Agent. The Agent -observes source evidence rendered by XMD and proposes desired file contents; -XMD owns every inspection and mutation. +Every Git conflict suspends. There is one profile, and this is it: a conflicted +merge returns a closed conflicted result, restores the pre-merge Workspace root, +retains complete normalized conflict evidence, publishes an actionable handoff, +and suspends for manual resolution. No conflict is resolved automatically, by +generated XMD, or by any structured text-conflict capability. -### 6.1 Durable conflict identity +Tool-less conflict handling grants no tool and no checkout to the Agent. The +Agent observes source evidence rendered by XMD; XMD owns every inspection and +mutation. The workflow Agent still receives no Git or GitHub operation, no +filesystem or shell operation, no Workspace, checkout, Repository or host path, +no native tool, and no MCP server carrying equivalent authority. + +### 9.1 Durable conflict identity Before attempting a merge, XMD checkpoints: @@ -271,171 +534,221 @@ conflict set. Every conflict entry identifies: - the exact repository-relative path; - the conflict classification; -- the base, ours, and theirs object identities and modes when Git supplies - them; and +- the base, ours, and theirs object identities and modes when Git supplies them; + and - the corresponding stage numbers for entries retained in the unmerged index. The journal retains structured identity and classification, not only rendered -conflict markers. The conflict profile decides what happens to the native merge -state: - -- The suspend-on-conflict profile rolls the conflicted checkout back, publishes - the structured evidence against the unchanged pre-merge Workspace root, and - enters a durable human wait. It never offers a file mutation under that - evidence. -- A structured-resolution profile retains the conflicted Git state and its - Workspace root with the conflict result, or retains provider-owned state that - reconstructs that exact state and verifies the same conflict identity before - mutation. A later resume cannot combine evidence from one merge attempt with - the index of another. +conflict markers. It retains them so that a later capability could be specified +against real evidence, and so that a stale resolution can be rejected without +inspecting Agent output or trusting current Project state — not because anything +in this contract mutates under them. A changed head, target base, merge base, Repository identity, conflict set, or Workspace root makes the conflict admission stale. XMD discards no remote -history and grants no mutation under that stale identity; it observes the new +history and grants no mutation under a stale identity; it observes the new revision and restarts Stage 4. -### 6.2 Agent observation and proposal +### 9.2 Manual resolution + +Human resolution occurs outside the workflow Agent's authority, by pushing to +the implementation branch. On resume, XMD observes that push as a new Stage 4 +`{ headSha, baseSha }`, records the manual intervention, runs new implementation +evidence against it, and re-enters Stage 4. It inherits no Stage 5-7 conclusion: +no human edit carries the conflicted attempt's review verdicts forward. + +Ordinary text conflicts, binary files, submodules, ambiguous renames, unsafe +symbolic links and unrecognized index forms are classified and retained +identically, because they all suspend. Classification is evidence for the human +reading the handoff, not a branch in the procedure. + +## 10. Stage 7 target publication and terminal settlement + +Stage 7 is where the run leaves the factory, and its ordering is what makes a +completed replay provider-free. -XMD renders the required conflict evidence and only the related source -observations to the Implementor. The workflow Agent still receives: +### 10.1 The merge is constructed, not requested -- no Git or GitHub operation; -- no filesystem or shell operation; -- no Workspace, checkout, Repository, or host path; -- no native tool; and -- no MCP server carrying equivalent authority. +Stage 7 merge creates a trusted merge commit whose **first parent is the exact +reviewed `baseSha`** and whose **second parent is the exact reviewed +`headSha`**. That is the opposite order from Stage 4's synchronization merge, +and the difference is deliberate: Stage 4 brings the target into the +implementation, Stage 7 brings the implementation onto the target. -The Implementor returns generated XMD containing proposed file writes and -deletions. That source is untrusted data until the workflow admits it. +It does not call an ordinary GitHub squash, rebase or merge endpoint. Final +delivery is this constructed merge commit and nothing else, so the reviewed +parent pair is preserved in the published history rather than replaced by a +commit no reviewer saw. -When the structured-resolution profile is installed, conflict resolution adds -a conflict-scoped generated-XMD admission. It is narrower than the workflow -host's ordinary write table: +### 10.2 Target publication is a compare-and-swap -- every write or deletion must name an exact path in the authorized conflict - set; -- the recorded Repository, head, base, merge base, conflict identity, and - Workspace root must still match; -- no generated component may stage, commit, merge, push, invoke a process, read - a credential, update GitHub, or mutate a non-conflict path; and -- the admitted source and complete ceiling are retained before the first file - effect. +`Git.PublishTarget` receives the exact merge commit, the reviewed head, and the +expected remote `baseSha`. The repository, remote, target ref, credential and +non-force policy are host-owned and are not authored props. -The Agent decides desired file contents. XMD applies the admitted file effects, -stages the exact conflict scope, verifies that the index contains no unresolved -entries and no unauthorized mutation, creates the merge commit with the -checkpointed head and base as its parents, runs the selected implementation -evidence, and publishes the resulting descendant through the ordinary -non-force Push effect. +It performs one non-force ref update only after observing the target equal to +`baseSha`. It adopts a target already equal to the exact merge commit, with +nothing performed. Every other observation — a target at another commit, an +incomplete observation, a permanent ambiguity, a temporary unavailability — +refuses or reconciles without mutating anything. -The item remains at Stage 4 throughout. A clean result is a new implementation -revision, not a review pass. +A race that moves the target before or during publication therefore cannot +publish over it. Because the reviews name exact revisions, that race invalidates +them: the item returns to Stage 5 when only rereview is required, and to Stage 4 +when synchronization or implementation work is required. -### 6.3 Conflict classes +`Git.Push`, `Git.PublishTarget`, pull-request upsert, `PullRequest.Ready`, +`PullRequest.Close` and the trusted merge are six distinct operations with +distinct subjects, ceilings and reconciliation. None is a spelling of another. -The protocol classifies every conflict before asking the Implementor for a -proposal. Ordinary text conflicts may be admitted by a bounded text-conflict -capability. Binary files, submodules, ambiguous renames, unsafe symbolic links, -unrecognized index forms, and every other unsupported class suspend for human -resolution. A mixed set containing one unsupported entry is unsupported as a -whole; no partial Agent mutation is admitted. +### 10.3 Terminal ordering -Human resolution occurs outside the workflow Agent's authority. On resume, XMD -accepts the manually changed branch only by observing its new exact -`{ headSha, baseSha }`, recording the manual intervention and re-entering Stage -4. No human edit inherits the conflicted attempt's review conclusions. +A merged decision is retained **before** any effect is attempted. Then merge +construction, target publication, the merged observation of the pull request, +issue completion, the Project move to `Closed`, and terminal settlement are +separate reconciled steps in that order. The run becomes terminal `merged` only +after every required remote projection is complete. -## 7. Structural consequences +That order exists for one reason: a completed run replays without contacting a +provider. If terminal completion preceded a projection, the replay that was +supposed to repair GitHub would be exactly the replay that is forbidden to reach +it. -This factory adds the following structural contracts. +An abandonment decision is exact-revision, authenticated under §5.4, and carries +a required reason. It is retained first as well, and then pull-request close +unmerged, issue close as `not_planned`, the Project move to `Closed`, and +terminal settlement follow as separate reconciled steps. The run becomes +terminal `abandoned` only after all required projections complete. -### 7.1 XMD workflow +Both terminal kinds retain the authorizing actor, the exact reviewed revision, +the resulting provider identities, and the reason where one is required. Both +retain the Project item, the implementation branch, the issue and pull-request +comments, the journal, the Workspace roots and the Agent evidence. Reopening the +issue afterwards does not reopen the completed run; continuing that work +requires a new linked issue, and therefore a new run. + +## 11. Exact public contract inventory + +These are the authored public forms. `as` is mandatory wherever a result is +bound, and form validation runs before any context, provider or credential +access. Durable effect identity is always engine-derived from the run and the +expansion; it is never a document prop. + +| Construct | Exact authored form and result | Owner and durable boundary | +| --- | --- | --- | +| `Issue.Comment` | Paired `…`; the content is the body; binds `{ url }` | Issue-provider effect; the natural key is the canonical issue URL plus the engine effect identity, so the body is not identity | +| `PullRequest.Comment` | Paired `…`; the content is the body; binds `{ url }` | Git-host effect; canonical pull-request URL plus engine effect identity | +| `Project.Status` | Self-closing ``; binds the normalized `{ item, field, option }` | Project-provider effect; the exact configured Project, item, field and option ceiling, against the current option as pre-state | +| `PullRequest.Ready` | Self-closing ``; binds normalized ready pull-request evidence | Git-host effect; only an accepted Stage 6 outcome authorizes invocation | +| `PullRequest.Close` | Self-closing ``; binds normalized `{ url, state: "closed", merged: false }` | Git-host effect; used only by a retained abandonment | +| `Issue.Close` | Self-closing ``, or the same form with `reason="not_planned"`; binds the normalized URL, state and reason | Issue-provider effect; `reason` is a closed enum and must match the retained terminal intent | +| `Git.Merge` | Self-closing ``, or the same form with `purpose="publish"`; binds a closed clean-or-conflicted result | Workspace-local Git effect; repository, checkout, root and acquisition are authenticated provider state; a clean publication is atomic and a conflict restores before the result is published | +| `Git.PublishTarget` | Self-closing ``; binds normalized target, expected and published evidence | Git-host effect; remote, ref, credential and non-force ceiling are host-owned; exact compare-and-swap reconciliation | +| `Evidence.Run` | Self-closing ``, where `commands` is an authored structured argv list; binds ordered bounded results | Trusted runner-host effect; the exact retained root and the executable, environment, time and output ceilings; absent from Agent and generated-XMD capabilities; a completed replay runs nothing | +| Remote `WorkflowHost` | A provider-neutral start, lookup, execute, answer and inspect contract, with a Cloudflare runtime-named implementation | A host API rather than an XMD component; the execution and delivery planes stay distinct across it | +| Factory protocol records | Closed stage, frontier, revision, outcome, invalidation, conflict, decision and terminal parsers over immutable values | A provider-neutral durable protocol; neither an XMD component nor a TypeScript lifecycle controller | + +Each construct's closed props, form, binding, request, natural key, compatible +pre-state, normalized result, refusal and unavailability behavior, cancellation, +replay, provider ownership, credential boundary, and whether it is +Workspace-local or an external reconciled effect are stated in +`specs/workflow-workspace-spec.md` §§7, 8 and 10, which those documents own. A +later implementation may choose ordinary private function and module names; it +may not change these public forms or their ownership. + +## 12. Structural consequences + +### 12.1 XMD workflow - The workflow holds one issue lifecycle in one durable run while Stage 4 emits zero or more implementation revisions. -- Every role outcome is validated against the current stage and active handoff - frontier. Review outcomes additionally carry the exact implementation +- Every role outcome is validated against the current stage and the active + handoff frontier. Review outcomes additionally carry the exact implementation revision. - Stage transition, revision observation, synchronization, conflict handling, - review, and closure remain authored XMD control flow. GitHub Actions contains - no parallel decision procedure. + review, and closure are authored XMD control flow. GitHub Actions contains no + parallel decision procedure, and neither does the Durable Object: it owns + state and admission, not stage choice. - The trusted definition comes from the run's immutable definition SHA. Draft pull-request content never becomes the workflow definition executed with GitHub credentials. -### 7.2 Journal +### 12.2 Journal - The immutable workflow definition continues to identify the run. -- The journal additionally retains implementation-revision observations, - active and invalidated handoffs, exact review subjects, merge checkpoints, - conflict classifications, conflict-scoped admission ceilings, evidence - outcomes, pushes, Project updates, and terminal reason. -- A conflict record must be sufficient to reject stale resolution without +- The journal additionally retains implementation-revision observations, active + and invalidated handoffs, exact review subjects, merge checkpoints, conflict + classifications, evidence outcomes, pushes, target publications, Project + updates, retained decisions and terminal reason. +- A conflict record is sufficient to reject a stale resolution without inspecting Agent output or trusting current Project state. - Exported `.xmd` artifacts remain immutable evidence. They are not the live run - store, executor lock, or continuation authority used by later Actions jobs. + store, the executor acquisition, or continuation authority for a later Actions + job. -### 7.3 Workspace and Git effects +### 12.3 Workspace and Git effects -- The suspend-on-conflict profile restores the pre-merge Workspace root and - retains conflict evidence only. A structured-resolution profile must instead - retain the conflicted Git index and working tree with the journal result that - identifies them, or retain equivalent provider-owned state from which that - exact conflict can be reconstructed and reverified. +- A conflicted merge restores the pre-merge Workspace root and retains conflict + evidence only. - A trusted merge effect observes and fixes head, base, and merge base before - mutation. Clean and conflicted results are distinct closed outcomes. -- Conflict-scoped file writes and deletions use the existing Workspace-local - transaction boundary but add exact path and conflict-identity ceilings. -- Merge commit creation verifies both parents and the empty conflict set before - publication. Push remains the existing reconciled non-force effect. -- Rebase and force-with-lease are absent. Adding either later requires a new - external-effect contract, reconciliation semantics, and invalidation proof; - it cannot be represented as another spelling of Push. + mutation. Clean and conflicted results are distinct closed outcomes, and the + parent order differs by purpose. +- Push remains the existing reconciled non-force effect. Target publication is a + separate reconciled effect with its own subject and compare-and-swap + pre-state. +- Rebase, force-with-lease and reset-based replacement are absent. -### 7.4 Invalidation frontier +### 12.4 Invalidation frontier - A changed implementation head always places the frontier at Stage 4. - A base-only change places it at Stage 5 unless Stage 4 work is required. -- A merge attempt, conflict proposal, clean merge, manual resolution, evidence - correction, or push does not advance the item by itself. +- A merge attempt, a clean merge, a manual resolution, an evidence correction, + or a push does not advance the item by itself. - Review approval is keyed by the complete SHA pair, so neither half can drift while Stages 5-7 remain accepted. -## 8. Remaining material decisions - -### 8.1 Product decision: first-release conflict scope - -The remaining product decision for conflict handling is whether the first -factory release implements conflict-scoped generated-XMD resolution for -ordinary text conflicts, or suspends for human resolution on every conflict. - -The recommended first release is the smaller contract: - -- perform clean merges automatically; -- suspend on every conflict; -- prohibit rebases and force pushes; and -- retain the structured conflict evidence needed to add ordinary text-conflict - resolution later as a bounded capability. - -This release proves the revision, invalidation, merge, non-force publication, -and human-resumption boundaries without making conflicted Workspace state and -conflict-scoped mutation admission prerequisites for the first useful factory. - -### 8.2 Deployment architecture decisions - -The lifecycle still requires three deployment choices before it can run across -ephemeral GitHub Actions runners: - -1. Select a durable WorkflowRun, Workspace, Agent-session, and executor-lock - provider reachable by every invocation. An Actions artifact is immutable - evidence and does not satisfy live continuation or locking. -2. Select the authorized ingress for Project admission, human answers, and - resume requests, including how the GitHub actor is authenticated. That - ingress may wake Actions but may not interpret role outcomes or own stage - transitions. -3. Select the GitHub principal and exact repository, pull-request, issue, and - Project permission ceilings, including who authorizes the Stage 7 merge or - abandonment and which target-branch merge method is permitted. - -These choices configure the host boundary. They do not create a second state -machine and do not transfer procedure authority out of XMD. +## 13. Structural acceptance checklist + +A factory implementation satisfies this specification when every item holds: + +1. The run ID equals the §1.1 derivation for its issue, and no mutable value + takes part in it. +2. Duplicate admission for one authenticated subject routes to one run; a + changed retained provider identity refuses as drift. +3. The nine Project statuses are exactly §2's, `User` included, and forward + progress is strictly adjacent. +4. Same-stage correction replaces the frontier without erasing history, and + backward invalidation reruns every later stage. +5. Stages 5-7 name an exact `{ headSha, baseSha }`, and no verdict is inherited + across a changed pair. +6. Only an accepted Stage 6 verdict takes the pull request out of draft. +7. Every Git conflict suspends, restores the pre-merge root, and retains + normalized conflict evidence; nothing resolves a conflict automatically. +8. A manual resolution is observed as a new Stage 4 revision and inherits no + Stage 5-7 conclusion. +9. Start, resume, recovery, mutation, transition, publication and settlement + validate the exact live executor acquisition and the expected Workspace root. +10. Intake, answer delivery and decision delivery take no acquisition, append no + lifecycle outcome and change no run status. +11. Inspection is read-only and authorizes no transition. +12. Native Git, evidence processes and Agent clients run only on the ephemeral + runner; the Durable Object runs none of them. +13. Webhook signature verification precedes parsing, and reread precedes + authorization. +14. `repository_dispatch` carries only a retained intake identity. +15. The OIDC admission validates issuer, audience, repository ID, owner ID, + event name, workflow ref and SHA, and the configured workflow identity. +16. Answers and Stage 7 decisions come only from the authenticated form bound to + the exact subject; no comment carries authority. +17. The App holds exactly §6's permissions, `.github/workflows/**` mutation is + refused, and no secret reaches props, context, durable records, comments, + output or diagnostics. +18. Every GitHub mutation observes before mutating, adopts only a compatible + completion, performs once, and refuses conflict, ambiguity, incomplete + observation and temporary unavailability. +19. The Stage 4 merge parent order is `[implementationHead, targetBase]` and the + Stage 7 order is `[reviewedBase, reviewedHead]`. +20. Target publication updates the ref only from an observed `baseSha`, adopts + only the exact merge commit, and never force-updates. +21. A terminal `merged` or `abandoned` state is published only after every + required projection completes, and a completed replay attaches no provider. diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index a84a29315..31eab1f7c 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -114,6 +114,15 @@ base is any revision expression, so both are external text on the same terms as retained props. A value installed without a run id, a base or a pinned commit identifies no run and is refused before any document executes. +A host that keeps its runs remotely has decided the same things in the same +order, and installs the same value. `retainedWorkflowInstallation()` names a run +its host already created and a commit its host already pinned; whether the +record behind that name lives in a local file or in a remote owner is host +arrangement the execution never learns. What the execution requires is +unchanged: the run id, base and pinned commit it was handed must be exactly what +the journal it reads records, and a journal recording a different run is +`StaleInputError` wherever the journal is kept. + ### 3.2 Where workflow-run identity is decided **Workflow-run identity is execution-owned, and it is not middleware of any @@ -328,6 +337,16 @@ after an interruption. The Deno host installs its own with entrypoint is the only place SQLite, run-id hashing, filesystem paths and host behavior appear. Shared modules import none of them and detect no runtime. +A second host installs its own the same way. Cloudflare is a runtime-named +adapter beside the Deno one, not a second contract: it answers `create()` and +`lookup()` with a `WorkflowRunDatabase` of its own, and every shared +WorkflowRun surface above it stays host-neutral. Nothing below changes for it — +immutable run identity is compared the same way, recognition stays strict, +events reach storage already filtered, a caller still owns the transaction it +opened, and a completed run still replays without attaching a provider. What a +remote adapter adds is where the bytes live and how an executor reaches them +(§9.8), which identity and recognition already treat as host arrangement. + A handle is a lease belonging to the scope that asked for it. Lease teardown makes that handle unusable, and every later call answers with a closed-handle failure rather than reopening the file. It does not close the run's physical @@ -401,6 +420,25 @@ and a local checkout path are **retrieval metadata** — replaceable, excluded from the comparison, never containing credentials, and reauthorized by the host before use. A run that moves between hosts is the same run. +#### A host may derive the id it selects + +A run id is opaque, and §3.3 of the Workspace specification already lets an +authorized caller select one. A host may equally derive one from the subject the +run is about, and a derived id is an ordinary selected id: it has to be a +non-empty string containing no NUL, and it has to be the same string every time +the same subject is admitted. Nothing else about it is constrained, and nothing +here narrows the ids an ordinary caller may choose. + +The software factory derives its ids that way. A factory run id is the lowercase +unpadded RFC 4648 Base32 encoding of the SHA-256 digest of the UTF-8 bytes +`github-issue-v1`, a NUL, the canonical Issue-provider authority, a NUL, and the +exact issue node ID — 52 characters of `a`-`z` and `2`-`7`, so the storage rule +above is satisfied by construction. Because every input is immutable, admitting +one issue twice derives one id and reaches one run through ordinary compatible +reuse, and no separate idempotency concept appears. The derivation itself, and +what a changed provider identity means, belong to +[the software factory](./github-actions-software-factory-spec.md) §1.1. + ### 9.2 Creating a run is also how it is found `create()` answers with the stored run when the request describes it, and @@ -833,6 +871,50 @@ Version 1 reads and writes version 1. Unsupported versions are refused without the file being touched; partial version-1 initialization is corruption and is also left unchanged. +### 9.8 A remote host owns the same run + +Serialization decides who uses one connection next; it has never decided who may +advance a run. A remote host keeps both answers and gives each a different +mechanism. + +The owner of one run is selected from the public run id by the same arithmetic +§9.3 uses, so a remote run has exactly one durable owner and no second registry +can disagree with it. Inside that owner, operations on the run's storage are +serialized and each runs in a transaction, exactly as §9.6 states: the +connection queue, the transaction identities and the savepoint allocator are +provider-private and say nothing about lifecycle authority. + +**Executor acquisition is an authenticated connection.** The acquisition is that +connection's lifetime: the owner registers the exact acquisition when the +connection is admitted and invalidates it when the connection closes, which is +the staleness proof a remote host has in place of an operating system releasing +a file lock. Like the local lock it is not a time lease — no duration, expiry, +renewal, heartbeat, generation record or liveness poll — and closing it releases +executor ownership without rolling back anything already committed. A second +healthy executor follows or is refused, and cannot advance the run either way. + +**These requests require the acquisition**, and each validates the exact live +acquisition and the expected Workspace root inside its own mutating +transaction: start and resume, stale-execution recovery, document execution, +Workspace mutation, provider attachment, native execution performed against a +materialized root, lifecycle transition, accepted-outcome publication, and +terminal settlement. + +**These requests take no acquisition.** Delivery retains one externally supplied +value for one exact retained subject and does what typed answer delivery already +does: it begins no document execution, attaches no Workspace, inserts no +document-execution record, appends no journal event and changes no run status. +What authorizes it is the subject, not the caller's position in the lifecycle — +a suspension id the run's retained `suspension_request` names for an answer, and +the exact retained decision subject for a terminal decision. A value for a +subject the run is not holding is refused with nothing written. Read-only +inspection takes no acquisition either, and returns the immutable snapshot +surface of the lifecycle contract rather than a writable handle. + +A later executor is what turns retained delivery into progress: it consumes the +value inside the run's own transaction, appends the accepted event once, and +only then may execution continue past the wait. + ## 10. The document filesystem of a run A host attaches one run's Workspace to a document execution with diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 5bcec74f9..f7930dfcd 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -472,6 +472,44 @@ owning the same run lifecycle remotely. `export` produces the immutable portable evidence contract in `specs/xmd-artifact-spec.md`; it does not expose the live run database. +A remote host owns that surface unchanged. One durable owner is selected from +the public run ID by the same arithmetic local discovery uses, and it holds the +run record, the filtered journal, the immutable Workspace roots and their +content-addressed bytes, the Agent-session mappings, the retained delivery state +and executor ownership. + +**Executor ownership is the lifetime of one authenticated connection.** A remote +start or resume opens that connection, and the acquisition lives and dies with +it: the owner registers the exact acquisition on admission and invalidates it on +close, which is the staleness proof that replaces an operating system releasing +a file lock. It is not a time lease, and closing it rolls back nothing already +committed. Every request that advances the run — start, resume, stale recovery, +document execution, Workspace mutation, provider attachment, native execution +against a materialized root, lifecycle transition, accepted-outcome publication +and terminal settlement — validates that exact live acquisition and the expected +Workspace root inside its own mutating transaction. + +**Delivery and inspection are the exceptions, and they stay exceptions.** A +delivery retains one externally supplied value for one exact retained subject +under §3.5's rules — no acquisition, no execution, no attachment, no journal +event, no status change — whether the subject is a suspension request or a +terminal decision. Inspection is read-only and returns immutable snapshots. A +remote host that let either one advance a lifecycle would have built a second +state machine beside the journal. + +**A remote host admits its executor before it trusts it.** Where the connection +comes from an ephemeral CI runner, admission validates that runner's OIDC claims +— issuer, configured audience, repository ID, repository-owner ID, event name, +workflow ref and SHA, and the configured immutable workflow identity — before an +acquisition exists. Repository names are mutable and are not what is checked. + +**An ephemeral runner recovers like any interrupted executor.** It materializes +one selected retained root, works in it, and submits content-addressed changes +that the owner validates and publishes atomically with the filtered journal +result, so a runner that dies mid-flight exposes only a prior or a new complete +transaction. The next acquisition performs the ordinary stale-execution recovery +of §3.3 and resumes from the exact committed run and Workspace frontier. + ### 3.9 What is shipped The lifecycle above is the whole design, including §3.7's rule that a status @@ -1603,6 +1641,133 @@ not name its own number and repository are all refused. A well-formed answer to another question is still the wrong answer. +### 7.8 Ordered merge: `Git.Merge` + +A merge is Workspace-local. It runs inside the retained checkout, against exact +commits the document names, and it publishes through the ordinary effect +transaction of §10.1 rather than through a Git host. + +```md + +``` + +The props are closed and all five are required: two exact parent commits, the +exact merge base, a `purpose` of `"synchronize"` or `"publish"`, and `as`. Which +Repository and checkout the merge runs in is decided the way §7.1 decides it, +and the repository, checkout, Workspace root and executor acquisition are +authenticated provider state rather than props. Form validation runs first: a +missing prop, an unknown prop, a `purpose` outside the enum and a missing `as` +each fail before a Repository is observed or a provider is reached. + +**The parent order is the caller's, and the two purposes are opposite +operations.** `purpose="synchronize"` brings a target into an implementation and +is authored `[implementationHead, targetBase]`. `purpose="publish"` brings a +reviewed implementation onto a target and is authored `[reviewedBase, +reviewedHead]`. The component does not infer the order from the purpose and does +not reorder what it was given; the purpose is retained so the record says which +operation the run believed it was performing. + +**The result is closed and has exactly two shapes.** A clean merge binds the new +commit and the Workspace root that followed it, and its mutation, root +publication and filtered result commit together — a crash before the commit +leaves the checkout, the current root and the effect history the ones the run +had. A conflicted merge restores the pre-merge Workspace root, publishes +normalized conflict evidence against that unchanged root, and binds the +conflicted result; it offers no file mutation under that evidence and adopts no +partial merge state. Both are successful effects with different outcomes, not a +success and a failure. + +Conflict evidence is normalized before it is retained: the repository-relative +path, the classification, the base, ours and theirs object identities and modes +where Git supplies them, and the stage numbers of entries in the unmerged index. +Rendered conflict markers alone are not the record, because a later reader has +to be able to tell a stale conflict from the one it is looking at without +reparsing text. + +A merge never contacts a Git host, never pushes, and never rewrites a published +identity. Rebase, force, force-with-lease and reset-based replacement are absent +from this component and from every other one in this specification. + +### 7.9 Publishing a reviewed merge: `Git.PublishTarget` + +Publishing to a protected target branch is a Git-host effect and a different +question from advancing a branch this run owns. + +```md + +``` + +All four props are required and the set is closed. The remote, the target ref, +the credential and the non-force policy are host-owned: they are not props, and +no authored value widens them. Form validation runs before the host's ceiling is +read and before any credential exists. + +**It is a compare-and-swap.** One non-force ref update happens only after +observing the target equal to `expectedRemoteCommit`. A target already equal to +the exact `sourceCommit` is adopted with nothing performed. Every other +observation refuses without mutating: a target at some third commit is a +conflict, an incomplete observation is unavailability rather than absence, and a +permanent ambiguity is a refusal. A race that moves the target before or during +publication therefore cannot publish over it. + +The bound result is stable evidence of what the effect settled on — the target +ref identity, the expected commit, the published commit and the reviewed head — +and not a live branch snapshot. `reviewedHead` is carried so the record says +what the publication was authorized against, which is what lets an +exact-revision review be invalidated by a target that moved. + +**Three operations stay distinct.** `Git.Push` (§7.4) advances a branch this run +published, from an ancestry relation proved inside the authenticated object +source. `Git.PublishTarget` updates a ref it does not own, from an exact +expected pre-state. A Git host's own pull-request merge endpoint is neither, and +this specification defines no component for one: a squash or a rebase performed +by the host would publish a commit no reviewer saw, under parents the review +never named. + +### 7.10 Pull-request comments, readiness and closure + +Three more Git-host effects act on a pull request a canonical URL names. Each +requires `as`, validates its form before any provider or credential is reached, +and reconciles under §10.2: observe, adopt a compatible completion, perform a +proven absence once, refuse conflict and ambiguity. + +```md + +The Architect accepted {revision.headSha} against {revision.baseSha}. + + + + +``` + +`PullRequest.Comment` is paired and its rendered content is the body, verbatim. +Its natural key is the canonical pull-request URL plus the engine-derived effect +identity of [Workflow runs](./workflow-spec.md) §8 — never the body — so an +edited sentence is the same comment and a re-rendered body is not a second one. +It binds `{ url }`. + +`PullRequest.Ready` takes a pull request out of draft and binds normalized ready +evidence. `PullRequest.Close` closes one unmerged and binds exactly +`{ url, state: "closed", merged: false }`. Both are self-closing, both are keyed +by their exact subject, and neither reopens, merges or comments on anything. +Observing a pull request already ready is an adoption; observing one that has +been merged, or that belongs to another repository, is a conflict. + +None of the three pushes, and none of them derives authority from what it +observes. Which of them a document may invoke, and what authorizes the +invocation, is authored control flow above them. + ## 8. Agents inspect; XMD mutates ### 8.1 No directory registration @@ -1654,6 +1819,14 @@ This is what the host asks for and what it refuses. It is not a claim that every ACP adapter exposes no tool when asked for none; that portable proof is tracked by #496 and does not widen this ceiling. +Constructs added for a trusted host do not reach the Agent either, and they do +not reach it for a different reason than the tool set: an Agent never expands a +document. Merging, publishing a target, commenting, changing draft state, +closing an issue or a pull request, moving a Project item and running evidence +are authored XMD the trusted host expands under its own acquisition. What an +Agent may return is text, and a fragment it returns is admitted only against the +tables §8.4 states — which name none of them. + `Session.Launch` is unsupported by this profile. The trusted workflow host states both ordinary-run native capability sets empty and installs no native foreground launcher, so a launch is refused before provider preparation, @@ -1973,8 +2146,15 @@ The write table is authority, not prompting guidance. Generated source cannot grant itself Push, PullRequest, an issue upsert, a repository, a process, an eval or exec block, a native command, a credential or an arbitrary network write merely by naming a component; the table excludes local Git even though those -effects are also Workspace-local. Trusted reusable Markdown components may be -admitted explicitly; generated XMD admits none of them. +effects are also Workspace-local. The constructs §§7.8-7.10, §10.5 and §10.6 add +change nothing about that: `Git.Merge`, `Git.PublishTarget`, +`PullRequest.Comment`, `PullRequest.Ready`, `PullRequest.Close`, +`Issue.Comment`, `Issue.Close`, `Project.Status` and `Evidence.Run` appear in no +table this specification states, so a fragment naming one is refused in the +preflight before any generated effect, exactly as `` is. Adding a +construct to a table is a host act, and a factory host adds none of them. +Trusted reusable Markdown components may be admitted explicitly; generated XMD +admits none of them. **Approval is authored, and it is ordinary.** `` neither prompts nor approves. A workflow that requires approval reaches a branch, an elicitation, a @@ -2653,6 +2833,35 @@ used. With no configuration there is no Issue provider, so every request reaches `NoIssueProvider` — absence of configuration is fail-closed, never an open default. +#### Commenting on and closing an issue + +Two more Issue-provider effects act on an issue a canonical URL names. Both +require `as`, both validate their form before any provider, ceiling or +credential is reached, and both reconcile the way an upsert does — observe, +adopt a compatible completion, perform a proven absence once, refuse conflict +and ambiguity — inside the provider rather than through the Git host's shared +state machine. + +```md + +The Planner accepted the plan at {plan.revision}. + + + +``` + +`Issue.Comment` is paired and its rendered content is the body, verbatim. Its +natural key is the canonical issue URL plus the engine-derived effect identity, +never the body, so an edited sentence is the same comment. It binds `{ url }`. + +`Issue.Close` is self-closing and its `reason` is a closed enum of `"completed"` +and `"not_planned"`. It binds the normalized URL, state and reason. The reason +is part of what the effect means rather than a label on it: a host that retained +one terminal intent refuses a close naming the other. An issue already closed +with the same reason is adopted; one closed with the other reason is a conflict. + +Neither reopens an issue, and neither derives authority from what it observes. + ### 10.4 Worker Shell Worker Shell means Cloudflare's Workspace Shell capability implemented by @@ -2682,6 +2891,86 @@ Network is denied unless explicitly authorized. A committed result restores without starting a Worker; an effect interrupted before commit executes again against its pre-effect Workspace root. +### 10.5 Trusted evidence execution + +Some evidence can only be produced by running the project's own commands with +the project's own tools. That is not Worker Shell, and it is not a capability a +document may reach for by itself. + +```md + +``` + +`commands` is an authored **structured argv list** — an ordered list of argument +vectors, each a non-empty list of strings — and never a shell string. There is +no interpreter, no quoting layer and no string to mis-split, which is what makes +the record of what ran the same thing as what ran. `as` is required, the prop +set is closed, and the form is validated before the host's ceilings are read. + +The host owns everything else. Which executables may run, what environment they +see, how long they may take, and how much output is retained are host ceilings +rather than props. The commands run against one exact retained Workspace root +that the host materialized, on the trusted runner where the native toolchain +lives — never inside the run's durable owner, which has no toolchain and must +not acquire one. + +The bound result is an ordered list of bounded results, one per command: the +argv it ran, its exit status, and its truncated captured output. That is an +ordinary durable record, so a completed replay produces it without running +anything at all. + +`Evidence.Run` is not Worker Shell (§10.4) and does not replace it: Worker Shell +is a contained interpreter over the Workspace filesystem, while this is native +execution of an authored list under a trusted host's ceiling. It is absent from +the workflow Agent's capabilities (§8.3) and from every generated-XMD table +(§8.4), so neither an Agent nor a fragment it wrote can reach it. + +### 10.6 Project effects + +A **Project provider** is an external service that owns project boards and the +status of the items on them. GitHub Projects V2 is one adapter. + +This is a boundary of its own for the reason §10.3 gives about issues: a project +board need own neither a Git repository nor an issue collection, so a Project +status cannot truthfully execute or persist as a `git_host_effect` or as an +`issue_effect`. `Project.Status` therefore reaches its own contextual operation +and journals its own durable effect type, and it reuses the shape of the +reconciliation rather than the Git host's state machine. + +```md + +``` + +The component is self-closing, its four props are required and the set is +closed, and it binds the normalized `{ item, field, option }`. Its natural key +is the exact item plus the exact field. Its compatible pre-state is the option +that item currently holds: an item already at the requested option is adopted +with nothing performed, an item at another allowed option is performed once, and +an unreadable board, an unavailable field, an ambiguous item and a partial +permission read are **unavailable** rather than absent — reading an unreadable +board as an empty one is how an unauthorized item would be moved. + +Which project, item, field and options may be reached at all is a host ceiling +installed beside the credential. An authored prop selects within that ceiling +and can never widen it, which is the same rule §10.3's tracker follows. + +**A board is a projection.** The status it shows is published from the run's own +journaled lifecycle, never read as it. A board ahead of the journal is drift the +next execution reconciles, and it is not evidence that a transition happened. + +### 10.7 What never crosses these boundaries + +Every effect in §10.2, §10.3, §10.5 and §10.6 reaches its provider the same way, +and the same things stay out of the record. Credentials are not inputs: an +application private key, a webhook secret, an issued installation token, an +OIDC verification configuration, a provider endpoint, a raw provider payload, a +pagination cursor and a host path stay in the selected provider's own closure. +None of them enters a component prop, context composition data, a durable +request, a natural key, a retained result, a comment body, document output or a +diagnostic. What a durable record holds is the normalized request, the natural +key, the observed pre-state and the normalized result — enough to reconcile the +effect, and nothing that would make the journal a place to read a secret from. + ## 11. History forks Normal resume always uses the same immutable definition, normalized props, @@ -3146,6 +3435,42 @@ There is no public `Git.Fetch` here. The shipped Git scope is Repository clone and its remote reads, plus `Git.Push` observation and mutation; a future public fetch operation requires its own language and durability contract. +### 13.2 Remote topology + +A remote host runs the same contracts with the durable state and the native +tools in two different places. + +One SQLite-backed Cloudflare Durable Object per run is the durable owner, +selected from the public run ID by the same arithmetic §9.3 of the workflow +specification uses. It holds the run record and filtered journal, the immutable +Workspace roots and their content-addressed bytes, the Repository and Worktree +records, the Agent-session mappings, the retained delivery state and the +authenticated intake records, and it owns executor admission. It is a +runtime-named adapter beside the Deno one: shared modules reach it through the +same contextual storage, lifecycle and Workspace APIs, detect no runtime, and +import nothing Cloudflare-specific. + +Executor ownership is one authenticated WebSocket connection whose lifetime is +the acquisition. The owner registers the exact acquisition on admission and +invalidates it on close; there is no duration, expiry, renewal, heartbeat, +generation record or liveness poll, and a close rolls back nothing already +committed. Every mutating transaction validates that exact acquisition and the +expected Workspace root together. + +Native Git, evidence processes and Agent clients run on the ephemeral runner and +nowhere else. The runner materializes one selected retained root, works in it, +and submits content-addressed changes; the owner validates acquisition, root and +content and then atomically publishes the new root with the filtered journal +result. That is §10.1's effect transaction with the mutation performed where the +tools are and the publication performed where the authority is, so a runner +crash exposes only a prior or a new complete transaction and the next +acquisition resumes from the exact committed frontier. + +Delivery and inspection reach the owner without an acquisition, under §3.8. A +completed run replays there as it does locally: it attaches no remote storage +session, Workspace, Agent, process, Git, Git-host, Issue, Project or credential +provider. + ## 14. Contract inventory | Contract | Status at this design revision | @@ -3174,5 +3499,13 @@ fetch operation requires its own language and durability contract. | generated-XMD mutation-proposal admission | built by #369 and #567: the standard Deno profile's write table is core's paired `File:write`, this package's lexical `Dir` and core's self-closing `File.Delete`, in that retained order and followed by any host extension; admitted mutations run as the ordinary components they are through the run's effect transactions, a generated deletion publishing the same `workspace_file` effect an authored one does; the evaluator adds no receipt or result entry, so a write-only fragment still binds `{ observations: [], output: "" }`; and approval is authored control flow before the element. Local Git, Git-host, issue, process, execution, credential and external-write effects are outside the class | | Deno-local DOFS persistence | POC proven by #349 / PR #350 | | scoped Deno Worker Shell | containment proven by #351 / PR #353 and transactions by #357 / PR #362; production integration unbuilt | +| `Git.Merge` ordered Workspace-local merge (§7.8) | specified by #710; implementation unbuilt | +| `Git.PublishTarget` compare-and-swap target publication (§7.9) | specified by #710; implementation unbuilt | +| `PullRequest.Comment`, `PullRequest.Ready`, `PullRequest.Close` (§7.10) | specified by #710; implementation unbuilt | +| `Issue.Comment` and `Issue.Close` (§10.3) | specified by #710; implementation unbuilt | +| `Evidence.Run` trusted native evidence execution (§10.5) | specified by #710; implementation unbuilt | +| `Project.Status` and the Project-provider boundary (§10.6) | specified by #710; implementation unbuilt | +| remote lifecycle host, executor connection and remote topology (§3.8, §13.2) | specified by #710; implementation unbuilt | +| terminal-decision delivery on the delivery plane (§3.8) | specified by #710; implementation unbuilt | | Worker JavaScript | deferred | | bundled workerd local host | omitted; POC #347 / PR #348 retained as provider evidence | From 80884f9cdbd3ab0053ffa3a66646e37208e813fa Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 10:48:26 -0400 Subject: [PATCH 02/81] =?UTF-8?q?=F0=9F=93=9D=20Close=20the=20software=20f?= =?UTF-8?q?actory=20contract=20gaps=20(#710)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answer the Architect's three blocking findings on 86a4694. The public contracts are now closed shapes rather than descriptions of shapes, run identity has one spelling, and the two terminal paths are separate everywhere. Finding 1 — exact bindings and durable records. specs/workflow-workspace-spec.md is the normative schema location for every authored construct: §7.8 defines the `GitMergeResult` clean/conflicted union, the conflict-entry record, its sorting, duplicate and missing-side rules, and restoration failure; §7.9 defines the `Git.PublishTarget` request and result with five exhaustive pre-states; §7.10 defines the comment request/result and how an engine-derived effect identity is made remotely observable without using body text, plus literal `Ready`/`Close` bindings; §10.3 does the same for `Issue.Comment` and `Issue.Close`; §10.5 settles `Evidence.Run` — the whole list runs, separate bounded stdout and stderr with stated truncation, and an explicit table of which cases bind, which fail and which commit nothing, with cancellation and teardown precedence fixed; §10.6 defines `Project.Status`. The factory specification §11.2 defines the versioned closed factory protocol schemas — subject, stage, revision, handoff, actor, outcome, invalidation, verdicts, conflict suspension, Stage 7 decisions, the frontier and its complete reduction table, and the two asymmetric terminals. Finding 1 also adds `PullRequest.Merged` (§7.11), the reconciled Git-host observation that owns the post-publication merged fact, and settles the remote host boundary (§13.2): the existing four-method `WorkflowHost` stays the host assembly contract with a Cloudflare implementation beside the Deno one, the runner-to-owner transport is a closed versioned envelope that refuses rather than adapts, and an ownership table says which side owns each concern. Finding 2 — every restatement of the run-ID derivation now uses `canonical GitHub authority` byte for byte; the undefined Issue-provider spelling is gone. Finding 3 — architecture.md and the factory specification now state the merged and abandoned paths as separate ordered step lists, and the terminal record's two shapes differ exactly as the paths do, so neither can be read as requiring the other's effects. Inventories, evidence tiers and the acceptance checklist are reconciled with all of it. Documentation only; the ownership SVG is untouched. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- architecture.md | 56 +- specs/executable-mdx-spec.md | 58 +- specs/github-actions-software-factory-spec.md | 269 ++++++++-- specs/workflow-spec.md | 14 +- specs/workflow-workspace-spec.md | 497 +++++++++++++----- 5 files changed, 667 insertions(+), 227 deletions(-) diff --git a/architecture.md b/architecture.md index 183f06150..3d78b07e4 100644 --- a/architecture.md +++ b/architecture.md @@ -104,7 +104,7 @@ Existing documents and code get aligned to this section retroactively. | recovery tombstone | an ownership record left active because its owner never proved it stopped. A crash releases the kernel lock and not this, and no pid, elapsed time, released lock or empty transcript clears it | | provider partition | one complete, independently owned agent-provider state — runtime, store, managed sessions, queues, coordinator, teardown — selected by the one installed factory at each dispatch. Production is the single-partition case of the same path; holding a partition grants work, never permission | | `JournalProvenance` | a non-operational, equality-only witness that a live publication stream descends from the exact journal backend a provider selected for one workflow run; it grants no append, read, execution, publication or reconciliation capability, and is meaningful only because the provider retains the witness it established and later requires exact equality | -| factory run identity | the run ID a software-factory run is addressed by: the lowercase unpadded RFC 4648 Base32 encoding of the SHA-256 digest of `github-issue-v1`, NUL, the canonical Issue-provider authority, NUL and the exact issue node ID. It is one host-selected public run ID, derived once from immutable provider identity, and it is distinct from the workflow definition SHA, the implementation revision, the Workspace root, the expansion identity and every delivery identity | +| factory run identity | the run ID a software-factory run is addressed by: the lowercase unpadded RFC 4648 Base32 encoding of the full SHA-256 digest of the UTF-8 bytes `github-issue-v1`, NUL, the canonical GitHub authority, NUL and the exact GitHub issue GraphQL node ID. The canonical GitHub authority and the node ID are the ones `specs/github-actions-software-factory-spec.md` §1.1 defines, byte for byte; there is no more general Issue-provider spelling of this hash. It is one host-selected public run ID, derived once from immutable provider identity, and it is distinct from the workflow definition SHA, the implementation revision, the Workspace root, the expansion identity and every delivery identity | | authenticated intake | one bounded record a trusted host retains for an externally delivered request — a verified webhook, or an authenticated human form submission — keyed by the provider's own delivery or submission identity and holding only typed bounded fields. It is what a later execution reads; it is never a stage, an outcome, a transition or a credential, and receiving one authorizes nothing beyond finding the run it names | | Project provider | an external service that owns project boards and the status of the items on them. GitHub Projects V2 is one adapter. It is a separate boundary from a Git host and from an Issue provider, because a project board need own neither a repository nor an issue collection | | Project projection | the human-facing status a Project provider holds for one item, published from the journaled lifecycle rather than read as it. A projection ahead of the journal is drift to reconcile; it is never evidence that a lifecycle transition happened | @@ -1507,18 +1507,9 @@ reconciles, and it is not evidence that a stage was passed. ### Comments, readiness and closure -Three more reconciled effects join the same boundary, each keyed by its own -subject. `Issue.Comment` is an Issue-provider effect keyed by the canonical -issue URL plus the engine-derived effect identity; `PullRequest.Comment` is a -Git-host effect keyed by the canonical pull-request URL plus that identity. The -body is presentation in both: keying a comment by its text would make an edited -sentence a different comment. `PullRequest.Ready` and `PullRequest.Close` are -Git-host effects keyed by their exact pull-request subject, and `Issue.Close` is -an Issue-provider effect keyed by its exact issue subject and carrying a closed -`reason` enum that has to match the retained terminal intent. Each observes -before it mutates, adopts a compatible completion, performs once from proven -absence or an exact compatible pre-state, and refuses conflict, permanent -ambiguity, incomplete observation and temporary unavailability. +Four more reconciled effects join the same boundary, each keyed by its own subject. `Issue.Comment` is an Issue-provider effect keyed by the canonical issue URL plus the engine-derived effect identity; `PullRequest.Comment` is a Git-host effect keyed by the canonical pull-request URL plus that identity. The body is presentation in both: keying a comment by its text would make an edited sentence a different comment. A Git host issues no idempotency key for a comment, so the effect identity is made remotely observable by one fixed presentation-neutral trailer the adapter appends and matches on — adapter-owned, never authored, and untouched when a person edits the prose around it. `PullRequest.Ready` and `PullRequest.Close` are Git-host effects keyed by their exact pull-request subject, and `Issue.Close` is an Issue-provider effect keyed by its exact issue subject and carrying a closed `reason` enum that has to match the retained terminal intent. Each observes before it mutates, adopts a compatible completion, performs once from proven absence or an exact compatible pre-state, and refuses conflict, permanent ambiguity, incomplete observation and temporary unavailability. + +`PullRequest.Merged` is the fourth, and it is the one that never mutates. A Git host records a pull request as merged when it notices its own ref move, which is not the same event as a target publication succeeding, so the fact has to be observed as its own retained step rather than inferred from the step before it. Adoption is its only completion: the host reporting the pull request merged at the exact published commit is the fact, a merge at another commit is a conflict, and a pull request still open or closed unmerged is temporary unavailability — the host has not caught up, which is a different thing from refusing. ### Ordered merge, and publishing a target @@ -1543,29 +1534,23 @@ publication was authorized against. ### Trusted evidence execution -`Evidence.Run` executes an authored structured argv list natively, on the -trusted runner, against one exact retained Workspace root, under host-owned -executable, environment, time and output ceilings. It is not Worker Shell and -not a document's own process capability: it runs where the tools are, and it is -absent from the workflow Agent's capabilities and from every generated-XMD write -table. Its ordered bounded results are an ordinary durable record, so a -completed replay runs nothing. +`Evidence.Run` executes an authored structured argv list natively, on the trusted runner, against one exact retained Workspace root, under host-owned executable, environment, working-root, duration, output and process-tree ceilings. It is not Worker Shell and not a document's own process capability: it runs where the tools are, and it is absent from the workflow Agent's capabilities and from every generated-XMD write table. + +Its record is one entry per authored command, in authored order, each naming the argv that ran, how it ended, and its stdout and stderr as separate bounded channels that state their own truncation. The whole list runs: a command ending unsuccessfully does not stop the ones after it, because running the list is how a reader learns what is broken. A non-zero status is evidence rather than an infrastructure failure — the answer the effect exists to obtain. Being unable to say what happened is the failure: a launch the ceiling refused, a channel the host could not read, and a child it could not reap each fail the effect and bind nothing, so no partial list is ever read as a whole one. A teardown failure outranks an otherwise ordinary result on the same terms lifecycle settlement applies, and cancellation commits nothing at all. A completed record replays running nothing. ### Terminal settlement follows its projections -An outcome whose completion requires external projections retains the decision -first and settles last. The accepted terminal decision is journaled before any -effect is attempted; the required projections — a constructed merge, a target -publication, the observation that the pull request is merged, the issue closure, -the Project move — are then separate reconciled steps; and only after all of -them complete is the terminal run state published. +An outcome whose completion requires external projections retains the decision first and settles last: the accepted decision is journaled before any effect is attempted, the required projections are separate reconciled steps, and only after all of them complete is the terminal run state published. + +There are two terminal paths, and they do not share a step list. Reading one general sequence for both would require merge effects during an abandonment, or pull-request closure during a merge. + +**The merged path** retains the authenticated exact-revision merge decision, then constructs the trusted merge commit, publishes the target, retains the merged pull-request observation, closes the issue as completed, projects the Project item to its closed option, and publishes terminal kind `merged` carrying the actor, the exact revision, the merge commit, the resulting provider identities and the retained history. + +**The abandoned path** retains the authenticated exact-revision abandonment decision together with its required reason, then closes the pull request unmerged, closes the issue as not planned, projects the Project item to its closed option, and publishes terminal kind `abandoned` carrying the actor, the exact revision, the reason, the resulting provider identities and the retained history. It constructs no merge and publishes no target — there is nothing it reviewed that it is publishing. + +A third decision is not terminal at all. A change decision names the earliest stage it invalidates and a reason, and returns the run to that stage; it settles nothing and projects no closure. -The ordering is not preference. A completed run replays without contacting a -provider, so a terminal state published before its projections would leave the -repair to exactly the replay that is forbidden to reach a provider. No -distributed transaction is claimed across the run's storage, native Git and -processes, and the external services; retaining the decision before the -projections, and settling after them, is what reconciliation has instead. +Every step on either path is a separate reconciled external effect or a separate retained transition, and no distributed transaction is claimed across the run's storage, native Git and processes, and the external services. An interruption resumes at the first uncommitted or unreconciled step. Terminal settlement is last on both paths for one reason: a completed run replays without contacting a provider, so a terminal state published before its projections would leave the repair to exactly the replay that is forbidden to reach a provider. Every committed journal event references the current logical Workspace root. Only committed event boundaries are checkpoints. A history fork copies the @@ -3516,9 +3501,10 @@ Status is measured against main. | `` | closes one issue as `completed` or `not_planned`, as an Issue-provider effect keyed by its exact subject, with the reason a closed enum that must match the retained terminal intent | specified by #710; implementation unbuilt | | `` | merges two exact commits inside the retained Workspace, observing and fixing both parents and the merge base first; a clean result publishes commit, root and filtered result in one effect transaction, a conflicted one restores the pre-merge root and publishes normalized conflict evidence, and the parent order is the caller's because synchronizing a target into an implementation and publishing an implementation onto a target are opposite operations | specified by #710; implementation unbuilt | | `` | updates a protected target ref by compare-and-swap: one non-force update after observing the target equal to the expected commit, adoption of a target already equal to the exact source commit, and refusal of everything else without mutation; remote, ref, credential and non-force policy are host-owned. Not a spelling of `Git.Push`, which advances a branch this run published from a proved ancestry relation | specified by #710; implementation unbuilt | -| `` | runs an authored structured argv list natively on the trusted runner against one exact retained Workspace root, under host-owned executable, environment, time and output ceilings; absent from the workflow Agent's capabilities and from every generated-XMD write table, and a completed replay runs nothing | specified by #710; implementation unbuilt | -| remote `WorkflowHost` | starts, looks up, executes, answers and inspects one run against a remote owner through the provider-neutral lifecycle surfaces, with a Cloudflare runtime-named implementation: one SQLite-backed Durable Object per run selected from the public run ID, executor acquisition as an authenticated connection lifetime, and content-addressed root transfer from an ephemeral runner | specified by #710; implementation unbuilt | -| factory protocol records | closed parsers and immutable values for the stage, frontier, implementation revision, role outcome, invalidation, conflict, terminal decision and terminal settlement one issue-driven run retains — provider-neutral durable protocol, neither an XMD component nor a TypeScript lifecycle controller | specified by #710; implementation unbuilt | +| `` | runs an authored structured argv list natively on the trusted runner against one exact retained Workspace root, under host-owned executable, environment, working-root, duration, output and process-tree ceilings; the whole list runs and binds one record per command carrying argv, how it ended, and separately bounded stdout and stderr that state their own truncation; a launch, output-pump or teardown failure binds nothing and cancellation commits nothing; absent from the workflow Agent's capabilities and from every generated-XMD write table, and a completed replay runs nothing | specified by #710; implementation unbuilt | +| `` | observes that a Git host now records one pull request as merged, at the exact commit a target publication published; a reconciled Git-host observation that mutates nothing, whose only completion is adoption, keyed by the canonical pull-request URL — a merge at another commit conflicts, a pull request still open or closed unmerged is temporary unavailability rather than absence, and publication does not imply it | specified by #710; implementation unbuilt | +| remote `WorkflowHost` implementation | keeps the existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — and adds a Cloudflare runtime-named implementation of it beside the Deno one; start, lookup, execute, deliver and inspect stay lifecycle operations reached through those four rather than becoming method names. One SQLite-backed Durable Object per run is selected from the public run ID, executor acquisition is an authenticated connection lifetime, and the runner-to-owner transport is a closed versioned envelope that refuses an unknown protocol version, operation or member rather than adapting to it | specified by #710; implementation unbuilt | +| factory protocol records | the closed versioned schemas `specs/github-actions-software-factory-spec.md` §11.2 defines for the subject, stage, implementation revision, handoff, actor, role outcome, invalidation, evidence reference, Planner and Architect verdicts, conflict suspension, Stage 7 decision, active frontier and the two terminal settlements one issue-driven run retains. Each carries a schema discriminant and a version, and an unknown schema, version, member or enum value refuses rather than being ignored — provider-neutral durable protocol, neither an XMD component nor a TypeScript lifecycle controller | specified by #710; implementation unbuilt | | workflow scheduling (watchers, unattended iteration, remote host selection) | — | #300 | | `` | binds `{ok: true, value}` or `{ok: false, error}`; a failure becomes a bound value, not a raise | defined, unbuilt | | error middleware (JS api) | retry · suspend · decline | defined, unbuilt | diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index ce397fffa..a441a041c 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -7499,7 +7499,7 @@ component grants that admission, and a repository component that takes the name ### 6.19 Software-factory constructs -An issue-driven software factory adds nine authored constructs. They belong to +An issue-driven software factory adds ten authored constructs. They belong to the workflow host rather than to core: nothing registers them under `xmd run`, and a document executed without that host has none of them. Their exact forms are listed here so the public surface is readable in one place; the contract @@ -7518,21 +7518,18 @@ engine-derived from the run and the expansion and is never a prop. | --- | --- | --- | | `Issue.Comment` | `…` — paired; the content is the body; binds `{ url }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.3 | | `PullRequest.Comment` | `…` — paired; the content is the body; binds `{ url }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.10 | -| `PullRequest.Ready` | `` — binds normalized ready evidence | [Workflow workspaces](./workflow-workspace-spec.md) §7.10 | +| `PullRequest.Ready` | `` — binds `{ url, state: "open", draft: false }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.10 | | `PullRequest.Close` | `` — binds `{ url, state: "closed", merged: false }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.10 | -| `Issue.Close` | ``, or the same form with `reason="not_planned"` — binds the normalized URL, state and reason | [Workflow workspaces](./workflow-workspace-spec.md) §10.3 | +| `PullRequest.Merged` | `` — binds `{ subject, state: "closed", merged: true, mergeCommit, decision: "adopted" }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.11 | +| `Issue.Close` | ``, or the same form with `reason="not_planned"` — binds `{ url, state: "closed", reason }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.3 | | `Project.Status` | `` — binds the normalized `{ item, field, option }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.6 | -| `Git.Merge` | ``, or the same form with `purpose="publish"` — binds a closed clean-or-conflicted result | [Workflow workspaces](./workflow-workspace-spec.md) §7.8 | -| `Git.PublishTarget` | `` — binds normalized target, expected and published evidence | [Workflow workspaces](./workflow-workspace-spec.md) §7.9 | -| `Evidence.Run` | ``, where `commands` is a structured argv list — binds ordered bounded results | [Workflow workspaces](./workflow-workspace-spec.md) §10.5 | - -Two things a reader looking for a component will not find here. The remote -`WorkflowHost` is a host API — start, lookup, execute, answer and inspect — and -not an element a document writes. The factory's protocol records — stage, -frontier, implementation revision, role outcome, invalidation, conflict, -terminal decision and terminal settlement — are closed parsers over immutable -values, not components and not a lifecycle controller beside the journal. -[The software factory](./github-actions-software-factory-spec.md) §11 owns both. +| `Git.Merge` | ``, or the same form with `purpose="publish"` — binds `{ outcome: "clean", purpose, firstParent, secondParent, mergeBase, commit, workspaceRoot }` or `{ outcome: "conflicted", purpose, firstParent, secondParent, mergeBase, workspaceRoot, conflicts }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.8 | +| `Git.PublishTarget` | `` — binds `{ target, expectedRemoteCommit, reviewedHead, sourceCommit, observedCommit, decision }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.9 | +| `Evidence.Run` | ``, where `commands` is an ordered non-empty list of non-empty argv vectors — binds `{ commands: [{ argv, outcome, status?, signal?, stdout, stderr }] }` in authored order, each channel `{ text, retainedBytes, producedBytes, truncated }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.5 | + +Every binding above is the exact closed record its contract section defines; a member outside those shapes, an unknown member and a value outside a closed enum each refuse rather than being carried. + +Two things a reader looking for a component will not find here. The remote `WorkflowHost` is a host assembly contract — the existing `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()`, with a Cloudflare implementation beside the Deno one — and not an element a document writes; its runner-to-owner transport is the versioned envelope of [Workflow workspaces](./workflow-workspace-spec.md) §13.2. The factory's protocol records — subject, stage, implementation revision, handoff, role outcome, invalidation, verdict, conflict suspension, Stage 7 decision, active frontier and terminal settlement — are the closed versioned schemas of [the software factory](./github-actions-software-factory-spec.md) §11.2, not components and not a lifecycle controller beside the journal. None of these constructs is available to a workflow Agent, and none appears in any generated-XMD read or write table @@ -10046,8 +10043,8 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §8. The seven tiers below are the frozen evidence names for the software factory specified by [the software factory](./github-actions-software-factory-spec.md) and by -[Workflow workspaces](./workflow-workspace-spec.md) §§3.8, 7.8-7.10, 10.5-10.7 -and 13.2. Every construct and host they name is **specified; implementation +[Workflow workspaces](./workflow-workspace-spec.md) §§3.8, 7.8-7.11, 10.3, 10.5-10.7 +and 13.2, and by [the software factory](./github-actions-software-factory-spec.md) §11.2. Every construct and host they name is **specified; implementation unbuilt**, so these tiers name the scenarios an implementation is accepted against rather than tests that exist. Each lists the finite structural scenarios — success, refusal, stale authority, interruption and cancellation, @@ -10074,6 +10071,9 @@ and [Workflow runs](./workflow-spec.md) §9.8. | WRH11 | Teardown | Closing the connection releases executor ownership and rolls back nothing already committed | | WRH12 | Completed replay | A completed run replays attaching no remote storage session, Workspace, Agent, process, Git, Git-host, Issue, Project or credential provider | | WRH13 | Host neutrality | Shared WorkflowRun modules import nothing Cloudflare-specific and detect no runtime; the runtime-named entrypoint is the only place the topology appears | +| WRH14 | The host boundary is unchanged | The Cloudflare adapter satisfies the existing `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()`; no fifth method appears, and the shared CLI asks the same four questions it asks the Deno host | +| WRH15 | Versioned transport | A `RunnerRequest` whose protocol version this owner does not implement is refused before any other member is read; an unknown operation, an unknown member, and an acquisition present on `acquire` or absent on the other four each refuse without effect, and neither side adapts to the other's shape | +| WRH16 | Ownership split | Connection admission, request parsing, transaction lifetime and stale recovery belong to the owner; provider attachment and cancellation of its own execution belong to the runner; content is produced by the runner and validated by the owner | ### Tier WGI — Authenticated GitHub ingress @@ -10102,7 +10102,9 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §7.10, §10.3 an | # | Test | Verify | |---|------|--------| | WGE1 | Comment identity | A comment's natural key is its subject plus the engine-derived effect identity; a re-rendered or edited body is the same comment and produces no second one | -| WGE2 | Ready and close | `PullRequest.Ready` and `PullRequest.Close` are keyed by their exact subject; an already-ready pull request is adopted and a merged one conflicts | +| WGE2 | Ready and close | `PullRequest.Ready` and `PullRequest.Close` are keyed by their exact subject; an already-ready pull request is adopted, a merged one conflicts for both, and each binds its literal closed record rather than an observed one | +| WGE2b | Comment identity is observable | An interrupted comment creation is found again by the adapter's fixed presentation-neutral trailer alone; exactly one match adopts, none performs once, more than one is ambiguity, and a comment list the adapter could not finish reading is an incomplete observation rather than absence | +| WGE2c | Merged observation | `PullRequest.Merged` mutates nothing and only adopts: a host reporting merged at the exact published commit is the completion, a merge at another commit conflicts, and a pull request still open or closed unmerged is temporary unavailability rather than absence | | WGE3 | Issue closure | `Issue.Close` adopts an issue already closed with the same reason and conflicts with one closed under the other reason | | WGE4 | Project pre-state | `Project.Status` is keyed by exact item plus field, adopts an item already at the requested option, and performs once from another allowed option | | WGE5 | Project unavailability | An unreadable board, unavailable field, ambiguous item and partial permission read are unavailable rather than absent, and none of them mutates | @@ -10130,8 +10132,11 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §7.8 and §7.9. | WGM7 | Adoption | A target already equal to the exact source commit is adopted with nothing performed | | WGM8 | Race | A target moved to a third commit refuses without mutating, and the exact-revision reviews invalidate rather than the publication proceeding | | WGM9 | Ceilings | Remote, ref, credential and non-force policy are host-owned; no authored prop sets or widens them | -| WGM10 | Distinct operations | `Git.Push`, `Git.PublishTarget`, pull-request upsert, ready, close and the trusted merge are six operations; no force, force-with-lease, rebase, reset or host squash appears in any command trace or retained configuration | +| WGM10 | Distinct operations | `Git.Push`, `Git.PublishTarget`, `Git.Merge`, pull-request upsert, ready, close and merged observation are seven operations with distinct subjects and records; no force, force-with-lease, rebase, reset or host squash appears in any command trace or retained configuration | | WGM11 | Replay | Completed merge and publication records replay without running Git and without contacting a Git host | +| WGM13 | Exact merge records | A clean result names the merge commit and published root; a conflicted one names the restored pre-merge root and the complete conflict set sorted by path in UTF-8 byte order, with stage numbers and side presence agreeing exactly, duplicate paths refused, and an absent side absent rather than null | +| WGM14 | Restoration failure | A pre-merge root that cannot be restored publishes no conflicted result and no new root, and activates the durable fail-stop fence | +| WGM15 | Exhaustive pre-states | Expected base performs once, the exact source commit adopts, a third commit conflicts, and incomplete observation, ambiguity and temporary unavailability each refuse as themselves | | WGM12 | Denied to the Agent | Neither construct is reachable by a workflow Agent or by an admitted generated fragment | ### Tier WER — Trusted evidence execution @@ -10147,6 +10152,11 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §10.5. | WER5 | Location | The execution happens on the trusted runner; the durable owner runs no native process | | WER6 | Cancellation and teardown | A cancelled run terminates its child before publishing, and no child outlives the effect | | WER7 | Replay | A completed record replays running nothing | +| WER4b | The whole list runs | A command ending unsuccessfully does not stop the ones after it; the bound list has one entry per authored command, in authored order, and each command is charged its own duration ceiling | +| WER4c | Channels and truncation | stdout and stderr are retained separately, each stating its retained bytes, the bytes the child produced, and whether it was truncated; truncation is stated rather than inferred | +| WER4d | Outcome discriminants | `status` is present exactly for `exited`, `signal` exactly for `signalled`, and `timeout` carries neither; either member beside the wrong outcome refuses the record | +| WER6b | What binds and what fails | Any exit status, a signal and a timeout each bind an ordinary result; a launch failure, an output-pump failure and a teardown failure each fail the effect and bind nothing; cancellation commits neither a result nor a failure | +| WER6c | Precedence | Cancellation outranks everything, the first infrastructure failure otherwise wins, and a teardown failure fails the effect even when every command produced an ordinary outcome | | WER8 | Denied authority | `Evidence.Run` appears in no Agent capability and in no generated-XMD read or write table; a fragment naming it is refused before any generated effect | ### Tier WFP — Factory protocol and frontier @@ -10156,7 +10166,8 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) | # | Test | Verify | |---|------|--------| -| WFP1 | Run identity | The run ID equals the §1.1 derivation for its issue; a changed repository name, issue number, Project identity, comment, branch, revision, definition SHA, delivery ID or actor changes it not at all | +| WFP1 | Run identity | The run ID equals the §1.1 derivation for its issue: the same canonical GitHub authority and the same issue node ID produce the same 52-character id in two independent implementations, and a changed repository name, issue number, Project identity, comment, branch, revision, definition SHA, delivery ID or actor changes it not at all | +| WFP1b | Identity drift | A reread returning a different canonical GitHub authority or node ID for a subject the host already retains refuses as unsupported provider-identity drift and derives no second run | | WFP2 | Adjacency | An outcome advancing more than one stage is rejected, and the current stage's handoff commits before the next role is invoked | | WFP3 | Same-stage amendment | A later accepted same-stage output replaces the frontier while the superseded output stays readable in the journal | | WFP4 | Backward destinations | Each of the six destinations in §2.2 deactivates exactly the downstream handoffs its row names and requires every later stage again | @@ -10164,7 +10175,9 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) | WFP6 | Base invalidation | A base-only move places the frontier at Stage 5, and a base move requiring synchronization or implementation work places it at Stage 4 | | WFP7 | Exact subjects | Stage 6 accepts only a Planner verdict naming the same revision, and Stage 7 only a review chain naming the current one | | WFP8 | Ready authority | Only an accepted Stage 6 verdict for the current revision takes the pull request out of draft; observed ready state manufactures no verdict | -| WFP9 | Closed parsers | Stage, frontier, revision, outcome, invalidation, conflict, decision and terminal records parse as closed immutable values, and an unrecognized shape refuses rather than being read loosely | +| WFP9 | Closed parsers | Every §11.2 record carries its schema discriminant and version; an unknown schema, an unknown version, an unknown member, a missing required member and a value outside a closed enum each refuse, and a refusal names the member path and never the value | +| WFP9b | Frontier reduction | Each reduction input of §11.2 — advance, amend, invalidate, head change, base-only change, base-needs-work, and a terminal decision on an already-terminal run — produces exactly the stated frontier, and a reduction leaving a stage out of range, two entries for one stage, or an advance past an unaccepted stage refuses | +| WFP9c | Decision shapes | `merge` carries revision, actor and delivery identity; `abandon` adds a required reason; `change` adds a reason and the earliest invalidated stage; a decision naming a revision that is not the current frontier revision refuses | | WFP10 | Definition identity | Definition incompatibility is a workflow lifecycle refusal and never an implementation correction | | WFP11 | Journal is authority | A Project status, comment or pull-request state ahead of the journal is drift and authorizes no stage | @@ -10176,11 +10189,12 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) | # | Test | Verify | |---|------|--------| | WFL1 | Clean synchronization | Observe, merge, record the revision, remain at Stage 4, run evidence, push the descendant, offer the new pair to Stage 5 — as separate durable effects, resuming from the first uncommitted one | -| WFL2 | Conflict suspension | Every conflict returns a closed conflicted result, restores the pre-merge root, retains normalized evidence, publishes a handoff and suspends | +| WFL2 | Conflict suspension | Every conflict returns the conflicted shape of the `GitMergeResult` union, restores the pre-merge root, retains the complete conflict set that shape defines, publishes a handoff and suspends | | WFL3 | No automatic resolution | No generated fragment, structured text-conflict capability, rebase, force, force-with-lease or reset resolves a conflict | | WFL4 | Manual resolution | A pushed resolution is observed as a new Stage 4 revision, receives new evidence, and inherits no Stage 5-7 conclusion | | WFL5 | Merge decision first | A merged decision is retained before any effect is attempted | -| WFL6 | Merge ordering | Merge construction, target publication, the merged observation, issue completion, the Project move and terminal settlement are separate reconciled steps in that order | +| WFL6 | Merge ordering | Merge construction, target publication, `PullRequest.Merged`, issue completion as `completed`, the Project move and terminal settlement are separate reconciled steps in that order | +| WFL6b | Paths do not borrow steps | The merged path closes no pull request and the abandoned path constructs no merge, publishes no target and observes no merged state; a `FactoryTerminal` naming a step from the other path refuses | | WFL7 | Terminal is last | The run becomes terminal `merged` only after every required projection completes | | WFL8 | Abandonment | An exact-revision authenticated abandonment carrying a reason is retained first, then pull-request close unmerged, issue close as `not_planned`, the Project move and settlement; terminal `abandoned` follows all of them | | WFL9 | Retention | Both terminal kinds retain the actor, exact revision, resulting provider identities and required reason, and retain the Project item, branch, comments, journal, Workspace roots and Agent evidence | diff --git a/specs/github-actions-software-factory-spec.md b/specs/github-actions-software-factory-spec.md index edba90d25..1940d96b7 100644 --- a/specs/github-actions-software-factory-spec.md +++ b/specs/github-actions-software-factory-spec.md @@ -338,9 +338,10 @@ retained intent; a completed replay contacts no provider at all. The natural keys are exact. A comment uses its subject plus the engine-derived effect identity, so the body and title are presentation rather than identity. A -Project status uses the exact Project item plus field. Ready and close use their -exact issue or pull-request subject. Target publication uses the retained -repository plus the configured remote and target ref. Accepted outcomes are +Project status uses the exact Project item plus field. Ready, close and the +merged observation use their exact issue or pull-request subject. Target +publication uses the retained repository plus the configured remote and target +ref. Accepted outcomes are retained before their projections are attempted, and reconciliation completes each intended projection. @@ -597,37 +598,41 @@ publish over it. Because the reviews name exact revisions, that race invalidates them: the item returns to Stage 5 when only rereview is required, and to Stage 4 when synchronization or implementation work is required. -`Git.Push`, `Git.PublishTarget`, pull-request upsert, `PullRequest.Ready`, -`PullRequest.Close` and the trusted merge are six distinct operations with -distinct subjects, ceilings and reconciliation. None is a spelling of another. +`Git.Push`, `Git.PublishTarget`, `Git.Merge`, pull-request upsert, `PullRequest.Ready`, `PullRequest.Close` and `PullRequest.Merged` are seven distinct operations with distinct subjects, ceilings and reconciliation. None is a spelling of another, and in particular publishing a target and observing that a pull request merged are two facts a Git host can hold separately. ### 10.3 Terminal ordering -A merged decision is retained **before** any effect is attempted. Then merge -construction, target publication, the merged observation of the pull request, -issue completion, the Project move to `Closed`, and terminal settlement are -separate reconciled steps in that order. The run becomes terminal `merged` only -after every required remote projection is complete. +There are two terminal paths and they share no step list. Each begins by retaining its authenticated exact-revision decision **before** any effect is attempted, and each ends with terminal settlement after every step before it has completed. A `change` decision is on neither path: it is not terminal, and it returns the run to the earliest stage it names. -That order exists for one reason: a completed run replays without contacting a -provider. If terminal completion preceded a projection, the replay that was -supposed to repair GitHub would be exactly the replay that is forbidden to reach -it. +**The merged path**, in this order: -An abandonment decision is exact-revision, authenticated under §5.4, and carries -a required reason. It is retained first as well, and then pull-request close -unmerged, issue close as `not_planned`, the Project move to `Closed`, and -terminal settlement follow as separate reconciled steps. The run becomes -terminal `abandoned` only after all required projections complete. +1. Retain the `merge` decision of §11.2, bound to the exact reviewed revision and the authenticated actor. +2. Construct the trusted merge commit — `` with parents `[reviewedBase, reviewedHead]`. +3. Publish the target — `` under §10.2. +4. Observe that the pull request merged — ``, the reconciled Git-host observation of [Workflow workspaces](./workflow-workspace-spec.md) §7.11, against the published merge commit. A Git host records a pull request as merged on its own schedule, so this is its own retained step and not something publication implies. +5. Close the issue — ``. +6. Move the Project item to `Closed` — ``. +7. Publish terminal kind `merged`. -Both terminal kinds retain the authorizing actor, the exact reviewed revision, -the resulting provider identities, and the reason where one is required. Both -retain the Project item, the implementation branch, the issue and pull-request -comments, the journal, the Workspace roots and the Agent evidence. Reopening the -issue afterwards does not reopen the completed run; continuing that work -requires a new linked issue, and therefore a new run. +**The abandoned path**, in this order: -## 11. Exact public contract inventory +1. Retain the `abandon` decision of §11.2, bound to the exact reviewed revision, the authenticated actor and its required reason. +2. Close the pull request unmerged — ``. +3. Close the issue — ``. +4. Move the Project item to `Closed` — ``. +5. Publish terminal kind `abandoned`. + +An abandonment constructs no merge, publishes no target and observes no merged state; there is nothing it reviewed that it is publishing. A merge closes no pull request; the Git host closes it when the target moves, which is what step 4 observes rather than performs. + +Every step on either path is a separate reconciled effect or a separate retained transition, and no distributed transaction is claimed across the Durable Object, native Git and processes, GitHub and Project V2. An interruption resumes at the first uncommitted or unreconciled step. + +Terminal settlement is last on both paths for one reason: a completed run replays without contacting a provider. If terminal completion preceded a projection, the replay that was supposed to repair GitHub would be exactly the replay that is forbidden to reach it. + +Both terminal kinds retain the authorizing actor, the exact reviewed revision, the resulting provider identities, and the reason where one is required, in the `FactoryTerminal` record of §11.2 — whose two shapes differ exactly as these two paths do. Both retain the Project item, the implementation branch, the issue and pull-request comments, the journal, the Workspace roots and the Agent evidence. Reopening the issue afterwards does not reopen the completed run; continuing that work requires a new linked issue, and therefore a new run. + +## 11. Exact public contracts + +### 11.1 Authored construct inventory These are the authored public forms. `as` is mandatory wherever a result is bound, and form validation runs before any context, provider or credential @@ -642,19 +647,191 @@ expansion; it is never a document prop. | `PullRequest.Ready` | Self-closing ``; binds normalized ready pull-request evidence | Git-host effect; only an accepted Stage 6 outcome authorizes invocation | | `PullRequest.Close` | Self-closing ``; binds normalized `{ url, state: "closed", merged: false }` | Git-host effect; used only by a retained abandonment | | `Issue.Close` | Self-closing ``, or the same form with `reason="not_planned"`; binds the normalized URL, state and reason | Issue-provider effect; `reason` is a closed enum and must match the retained terminal intent | -| `Git.Merge` | Self-closing ``, or the same form with `purpose="publish"`; binds a closed clean-or-conflicted result | Workspace-local Git effect; repository, checkout, root and acquisition are authenticated provider state; a clean publication is atomic and a conflict restores before the result is published | +| `Git.Merge` | Self-closing ``, or the same form with `purpose="publish"`; binds the `GitMergeResult` union of [Workflow workspaces](./workflow-workspace-spec.md) §7.8 — `{ outcome: "clean", purpose, firstParent, secondParent, mergeBase, commit, workspaceRoot }` or `{ outcome: "conflicted", purpose, firstParent, secondParent, mergeBase, workspaceRoot, conflicts }` | Workspace-local Git effect; repository, checkout, root and acquisition are authenticated provider state; a clean publication is atomic and a conflict restores before the result is published | | `Git.PublishTarget` | Self-closing ``; binds normalized target, expected and published evidence | Git-host effect; remote, ref, credential and non-force ceiling are host-owned; exact compare-and-swap reconciliation | -| `Evidence.Run` | Self-closing ``, where `commands` is an authored structured argv list; binds ordered bounded results | Trusted runner-host effect; the exact retained root and the executable, environment, time and output ceilings; absent from Agent and generated-XMD capabilities; a completed replay runs nothing | -| Remote `WorkflowHost` | A provider-neutral start, lookup, execute, answer and inspect contract, with a Cloudflare runtime-named implementation | A host API rather than an XMD component; the execution and delivery planes stay distinct across it | -| Factory protocol records | Closed stage, frontier, revision, outcome, invalidation, conflict, decision and terminal parsers over immutable values | A provider-neutral durable protocol; neither an XMD component nor a TypeScript lifecycle controller | - -Each construct's closed props, form, binding, request, natural key, compatible -pre-state, normalized result, refusal and unavailability behavior, cancellation, -replay, provider ownership, credential boundary, and whether it is -Workspace-local or an external reconciled effect are stated in -`specs/workflow-workspace-spec.md` §§7, 8 and 10, which those documents own. A -later implementation may choose ordinary private function and module names; it -may not change these public forms or their ownership. +| `Evidence.Run` | Self-closing ``, where `commands` is an ordered non-empty list of non-empty argv vectors; binds the `EvidenceRunResult` of [Workflow workspaces](./workflow-workspace-spec.md) §10.5 — one `{ argv, outcome, status?, signal?, stdout, stderr }` per command, in authored order, each channel `{ text, retainedBytes, producedBytes, truncated }` | Trusted runner-host effect; the exact retained root and the executable, environment, working-root, duration, output and process-tree ceilings; the whole list runs, launch/output-pump/teardown failures bind nothing, cancellation commits nothing; absent from Agent and generated-XMD capabilities; a completed replay runs nothing | +| `PullRequest.Merged` | Self-closing ``; binds `{ subject, state: "closed", merged: true, mergeCommit, decision: "adopted" }` | Git-host reconciled observation, [Workflow workspaces](./workflow-workspace-spec.md) §7.11; it mutates nothing and adoption is its only completion; keyed by the canonical pull-request URL | +| Remote `WorkflowHost` | The existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — with a Cloudflare runtime-named implementation beside the Deno one; start, lookup, execute, deliver and inspect are lifecycle operations reached through it rather than method names of their own. The runner-to-owner transport is the versioned `RunnerRequest` envelope of [Workflow workspaces](./workflow-workspace-spec.md) §13.2 | A host assembly contract rather than an XMD component; the execution, delivery and inspection planes stay distinct across it | +| Factory protocol records | The closed versioned schemas of §11.2 | A provider-neutral durable protocol; neither an XMD component nor a TypeScript lifecycle controller | + +Each construct's closed props, form, binding, request, natural key, compatible pre-state, normalized result, refusal and unavailability behavior, cancellation, replay, provider ownership, credential boundary, and whether it is Workspace-local or an external reconciled effect are defined normatively in [Workflow workspaces](./workflow-workspace-spec.md): §7.8 `Git.Merge`, §7.9 `Git.PublishTarget`, §7.10 `PullRequest.Comment`, `PullRequest.Ready` and `PullRequest.Close`, §7.11 `PullRequest.Merged`, §10.3 `Issue.Comment` and `Issue.Close`, §10.5 `Evidence.Run`, §10.6 `Project.Status`, §10.7 the credential boundary they share, and §13.2 the remote host and its transport. A later implementation may choose ordinary private function and module names; it may not change these public forms, their records or their ownership. + +### 11.2 Factory protocol records + +The factory's lifecycle is journaled as closed immutable records, not held in a controller. These are the schemas the journal retains and every role outcome is parsed into. They are provider-neutral data and parsers; nothing here becomes a TypeScript state machine beside the journal, and nothing here is an XMD component. + +Every record carries `schema`, its discriminant, and `version`, which is `1` for all of them. Parsing is strict in both directions: an unknown `schema`, an unknown `version`, an unknown member, a missing required member, and a value outside a closed enum each refuse the record rather than being ignored or defaulted. A refusal names the member path and never the value behind it, on the same terms retained props and journal payloads are described. + +#### Identities and subjects + +```ts +type FactoryStage = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8; + +type FactoryRole = "user" | "architect" | "planner" | "implementor"; + +interface FactorySubject { + readonly schema: "factory-subject"; + readonly version: 1; + readonly runId: string; + readonly authority: string; + readonly issueNodeId: string; + readonly issueUrl: string; + readonly repositoryId: string; + readonly projectItemId: string; + readonly statusFieldId: string; +} + +interface ImplementationRevision { + readonly schema: "implementation-revision"; + readonly version: 1; + readonly headSha: string; + readonly baseSha: string; +} +``` + +`FactoryStage` is the numeric stage; the nine status strings of §2 are its projection and are never parsed back into it. `runId` is the §1.1 derivation, and `authority` and `issueNodeId` are the exact bytes it was derived from, retained so drift is detectable without re-deriving. `repositoryId`, `projectItemId` and `statusFieldId` are provider identities compared byte for byte. `headSha` and `baseSha` are lowercase hexadecimal commit IDs. A revision is compared as the whole pair: two revisions are equal only when both halves are. + +#### Evidence and role outcomes + +```ts +interface EvidenceReference { + readonly schema: "evidence-reference"; + readonly version: 1; + readonly effectId: string; + readonly revision: ImplementationRevision; + readonly passed: boolean; +} + +interface FactoryHandoff { + readonly schema: "factory-handoff"; + readonly version: 1; + readonly stage: FactoryStage; + readonly role: FactoryRole; + readonly actor: FactoryActor; + readonly summary: string; + readonly revision?: ImplementationRevision; + readonly evidence?: readonly EvidenceReference[]; +} + +interface FactoryActor { + readonly schema: "factory-actor"; + readonly version: 1; + readonly kind: "human" | "agent"; + readonly id: string; +} + +type FactoryOutcome = + | { readonly schema: "factory-outcome"; readonly version: 1; readonly kind: "advance"; readonly from: FactoryStage; readonly to: FactoryStage; readonly handoff: FactoryHandoff } + | { readonly schema: "factory-outcome"; readonly version: 1; readonly kind: "amend"; readonly stage: FactoryStage; readonly handoff: FactoryHandoff; readonly supersedes: string } + | { readonly schema: "factory-outcome"; readonly version: 1; readonly kind: "invalidate"; readonly invalidation: FactoryInvalidation } + | { readonly schema: "factory-outcome"; readonly version: 1; readonly kind: "verdict"; readonly verdict: FactoryVerdict } + | { readonly schema: "factory-outcome"; readonly version: 1; readonly kind: "suspend"; readonly suspension: ConflictSuspension }; + +interface FactoryInvalidation { + readonly schema: "factory-invalidation"; + readonly version: 1; + readonly from: FactoryStage; + readonly earliestInvalidated: FactoryStage; + readonly reason: string; + readonly actor: FactoryActor; +} + +type FactoryVerdict = + | { readonly schema: "factory-verdict"; readonly version: 1; readonly stage: 5; readonly role: "planner"; readonly revision: ImplementationRevision; readonly decision: "pass" | "changes"; readonly reason: string; readonly evidence: readonly EvidenceReference[] } + | { readonly schema: "factory-verdict"; readonly version: 1; readonly stage: 6; readonly role: "architect"; readonly revision: ImplementationRevision; readonly decision: "pass" | "changes"; readonly reason: string; readonly plannerVerdict: string }; +``` + +`summary` and `reason` are the only presentation fields, and both are bounded: they are what a comment renders, never what identity compares. `supersedes`, `effectId` and `plannerVerdict` are journal event identities, so a record points at the history it replaces or depends on instead of copying it. An `advance` whose `to` is not `from + 1` refuses, and so does an `amend` whose `stage` is not the current frontier stage. A Stage 6 verdict whose `plannerVerdict` names a verdict for another revision refuses. + +#### Conflict suspension + +```ts +interface ConflictSuspension { + readonly schema: "conflict-suspension"; + readonly version: 1; + readonly revision: ImplementationRevision; + readonly mergeBase: string; + readonly workspaceRoot: string; + readonly conflictIdentity: string; + readonly mergeEffectId: string; + readonly suspensionId: string; +} +``` + +`workspaceRoot` is the *restored* pre-merge root, `mergeEffectId` names the `Git.Merge` event whose conflicted result holds the normalized conflict set of [Workflow workspaces](./workflow-workspace-spec.md) §7.8, and `conflictIdentity` is derived from the checkpointed identities and that normalized set. The conflict set is not copied here: one account of it, in the effect that produced it. + +#### Stage 7 decisions + +```ts +type Stage7Decision = + | { readonly schema: "stage-7-decision"; readonly version: 1; readonly kind: "merge"; readonly revision: ImplementationRevision; readonly actor: FactoryActor; readonly deliveryId: string } + | { readonly schema: "stage-7-decision"; readonly version: 1; readonly kind: "abandon"; readonly revision: ImplementationRevision; readonly actor: FactoryActor; readonly deliveryId: string; readonly reason: string } + | { readonly schema: "stage-7-decision"; readonly version: 1; readonly kind: "change"; readonly revision: ImplementationRevision; readonly actor: FactoryActor; readonly deliveryId: string; readonly earliestInvalidated: FactoryStage; readonly reason: string }; +``` + +All three bind the exact revision they were made against and the authenticated actor who made them, and all three name the delivery identity they arrived under. `abandon` requires a reason and `change` requires both a reason and the earliest stage it invalidates; `merge` takes neither, because approving what two reviews already passed adds no new claim. A decision whose `revision` is not the current frontier revision refuses. `merge` and `abandon` are terminal intents that §10.3 orders; `change` is not terminal and reduces to an invalidation. + +#### The active frontier and its reduction + +```ts +interface FactoryFrontier { + readonly schema: "factory-frontier"; + readonly version: 1; + readonly stage: FactoryStage; + readonly revision?: ImplementationRevision; + readonly accepted: readonly string[]; + readonly terminal?: FactoryTerminal; +} +``` + +`accepted` is the ordered list of journal event identities forming the active chain, oldest first, one per stage that has been passed. `revision` is absent before Stage 4 produces one. `terminal` is present only on a settled run. + +The frontier is a reduction over the retained outcomes, and its inputs and outputs are exactly these: + +| Input | Resulting frontier | +| --- | --- | +| `advance` from stage *n* to *n + 1* | `stage` becomes *n + 1*; the handoff's event is appended to `accepted` | +| `amend` at the current stage | `stage` is unchanged; the superseded event is replaced in `accepted` by the amending one, and the superseded event stays in the journal | +| `invalidate` naming earliest stage *e* | `stage` becomes *e*; every entry in `accepted` for a stage at or after *e* is dropped from the active chain and kept in the journal | +| a head change: a new `headSha` | `stage` becomes 4, `revision` becomes the new pair, and the Stage 5, 6 and 7 entries drop | +| a base-only change requiring no work | `stage` becomes 5, `revision` becomes the new pair, and the Stage 5, 6 and 7 entries drop | +| a base change requiring synchronization or implementation work | `stage` becomes 4 on the same terms as a head change | +| a `merge` or `abandon` decision on a run whose `terminal` is already present | refused; the frontier is unchanged | + +A reduction that would leave `stage` outside `0`-`8`, leave `accepted` holding two entries for one stage, or advance past a stage with no accepted entry refuses rather than producing a frontier. + +#### Terminal settlement + +```ts +type FactoryTerminal = + | { + readonly schema: "factory-terminal"; + readonly version: 1; + readonly kind: "merged"; + readonly revision: ImplementationRevision; + readonly actor: FactoryActor; + readonly decisionId: string; + readonly mergeCommit: string; + readonly publication: string; + readonly mergedObservation: string; + readonly issueClosure: string; + readonly projectClosure: string; + } + | { + readonly schema: "factory-terminal"; + readonly version: 1; + readonly kind: "abandoned"; + readonly revision: ImplementationRevision; + readonly actor: FactoryActor; + readonly decisionId: string; + readonly reason: string; + readonly pullRequestClosure: string; + readonly issueClosure: string; + readonly projectClosure: string; + }; +``` + +The two kinds do not share a member list, and that asymmetry is the contract: a `merged` terminal names a merge commit, a target publication and a merged observation, and an `abandoned` terminal names a pull-request closure and a reason and can name none of the first three. Every member ending in `Id` or naming a step is a journal event identity, so the terminal record points at the reconciled effects that completed rather than restating their results. A terminal record whose named events are not all present and complete refuses, which is what makes §10.3's ordering checkable from the record alone. ## 12. Structural consequences @@ -750,5 +927,15 @@ A factory implementation satisfies this specification when every item holds: Stage 7 order is `[reviewedBase, reviewedHead]`. 20. Target publication updates the ref only from an observed `baseSha`, adopts only the exact merge commit, and never force-updates. -21. A terminal `merged` or `abandoned` state is published only after every - required projection completes, and a completed replay attaches no provider. +21. The merged state of the pull request is observed as its own retained step + after publication and before issue closure, and is adopted only at the exact + published merge commit. +22. The merged and abandoned paths of §10.3 run in their stated orders, and + neither borrows a step from the other. +23. A terminal `merged` or `abandoned` state is published only after every + required projection completes, and a completed replay attaches no external + provider. +24. Every authored construct binds the exact record + [Workflow workspaces](./workflow-workspace-spec.md) defines for it, and + every factory protocol record parses under §11.2 with strict refusal of an + unknown schema, version or member. diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 31eab1f7c..85ba99555 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -429,15 +429,11 @@ non-empty string containing no NUL, and it has to be the same string every time the same subject is admitted. Nothing else about it is constrained, and nothing here narrows the ids an ordinary caller may choose. -The software factory derives its ids that way. A factory run id is the lowercase -unpadded RFC 4648 Base32 encoding of the SHA-256 digest of the UTF-8 bytes -`github-issue-v1`, a NUL, the canonical Issue-provider authority, a NUL, and the -exact issue node ID — 52 characters of `a`-`z` and `2`-`7`, so the storage rule -above is satisfied by construction. Because every input is immutable, admitting -one issue twice derives one id and reaches one run through ordinary compatible -reuse, and no separate idempotency concept appears. The derivation itself, and -what a changed provider identity means, belong to -[the software factory](./github-actions-software-factory-spec.md) §1.1. +The software factory derives its ids that way. A factory run id is the lowercase unpadded RFC 4648 Base32 encoding of the full SHA-256 digest of the UTF-8 bytes `github-issue-v1`, a NUL, the canonical GitHub authority, a NUL, and the exact GitHub issue GraphQL node ID — 52 characters of `a`-`z` and `2`-`7`, so the storage rule above is satisfied by construction. + +Those two inputs are the ones [the software factory](./github-actions-software-factory-spec.md) §1.1 defines, byte for byte, and this paragraph restates rather than generalizes them: the authority is the lowercase DNS hostname plus a non-default port, with no scheme, path, query, fragment, user information or trailing separator, and the node ID is GitHub's exact returned string with no case folding and no Unicode normalization. There is no broader Issue-provider authority in this hash — a hash whose inputs two documents spell differently is two hashes. + +Because every input is immutable, admitting one issue twice derives one id and reaches one run through ordinary compatible reuse, and no separate idempotency concept appears. Two independent implementations given the same authority and node ID therefore produce the same id. A changed authority or node ID for a subject the host already retains is unsupported provider-identity drift, refused under §1.1 of that specification rather than derived into a second run. ### 9.2 Creating a run is also how it is found diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index f7930dfcd..6bf7da850 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -1673,26 +1673,67 @@ reviewedHead]`. The component does not infer the order from the purpose and does not reorder what it was given; the purpose is retained so the record says which operation the run believed it was performing. -**The result is closed and has exactly two shapes.** A clean merge binds the new -commit and the Workspace root that followed it, and its mutation, root -publication and filtered result commit together — a crash before the commit -leaves the checkout, the current root and the effect history the ones the run -had. A conflicted merge restores the pre-merge Workspace root, publishes -normalized conflict evidence against that unchanged root, and binds the -conflicted result; it offers no file mutation under that evidence and adopts no -partial merge state. Both are successful effects with different outcomes, not a -success and a failure. - -Conflict evidence is normalized before it is retained: the repository-relative -path, the classification, the base, ours and theirs object identities and modes -where Git supplies them, and the stage numbers of entries in the unmerged index. -Rendered conflict markers alone are not the record, because a later reader has -to be able to tell a stale conflict from the one it is looking at without -reparsing text. - -A merge never contacts a Git host, never pushes, and never rewrites a published -identity. Rebase, force, force-with-lease and reset-based replacement are absent -from this component and from every other one in this specification. +**The request is the four authored inputs plus what the provider authenticates.** The durable request carries `firstParent`, `secondParent`, `mergeBase` and `purpose` exactly as authored, together with the Repository identity, the checkout, the pre-merge Workspace root and the executor acquisition the provider validated. Those four are provider-authenticated state and never authored props, and the effect is named by the run and the expansion like every other one. + +**The result is a closed discriminated union of exactly two shapes**, keyed on `outcome`: + +```ts +type GitMergeResult = + | { + readonly outcome: "clean"; + readonly purpose: "synchronize" | "publish"; + readonly firstParent: string; + readonly secondParent: string; + readonly mergeBase: string; + readonly commit: string; + readonly workspaceRoot: string; + } + | { + readonly outcome: "conflicted"; + readonly purpose: "synchronize" | "publish"; + readonly firstParent: string; + readonly secondParent: string; + readonly mergeBase: string; + readonly workspaceRoot: string; + readonly conflicts: readonly GitMergeConflict[]; + }; + +interface GitMergeConflict { + readonly path: string; + readonly classification: + | "content" + | "add/add" + | "modify/delete" + | "delete/modify" + | "rename" + | "mode" + | "binary" + | "submodule" + | "symlink" + | "unrecognized"; + readonly stages: readonly (1 | 2 | 3)[]; + readonly base?: GitMergeSide; + readonly ours?: GitMergeSide; + readonly theirs?: GitMergeSide; +} + +interface GitMergeSide { + readonly objectId: string; + readonly mode: string; +} +``` + +Every member is required unless the declaration marks it optional, every commit and object identity is a lowercase hexadecimal object ID of the repository's own object format, `mode` is the six-digit octal Git records, and `path` is an already-normalized repository-relative POSIX path under the same rules §9.1 of [Workflow runs](./workflow-spec.md) states for a root document path. `classification` is the closed enum above and `unrecognized` is its own value rather than an absent one, because a class this build cannot name is a fact about the merge and not a gap in the record. An unknown member and an unknown classification each refuse the record rather than being ignored. + +A **clean** result names the exact merge commit and the Workspace root published with it. Its mutation, root publication and filtered result commit together, so a crash before the commit leaves the checkout, the current root and the effect history the ones the run had. A **conflicted** result names the *unchanged* pre-merge root it restored and the complete conflict set; it offers no file mutation under that evidence and adopts no partial merge state. Both are successful effects with different outcomes, not a success and a failure. + +The conflict set is complete and ordered. Entries sort by `path` in UTF-8 byte order, and two entries for one path refuse the record rather than being merged or deduplicated. `stages` is the ascending list of unmerged-index stage numbers Git retained for that path, and side presence agrees with it exactly: stage 1 is `base`, 2 is `ours`, 3 is `theirs`, a stage the index does not hold has its member absent rather than null, empty or zeroed, and a side present without its stage — or a stage without its side — refuses. A set mixing classifications is retained whole and unaltered: every conflict suspends, so there is no partial handling for a mixed set to select. Rendered conflict markers alone are never the record, because a later reader has to tell a stale conflict from the one it is looking at without reparsing text. + +**Restoration is part of the conflicted outcome, not cleanup after it.** If the pre-merge root cannot be restored, the effect publishes no conflicted result and no new root: it is an infrastructure failure that activates the durable fail-stop fence, because a conflicted result naming a root the Workspace is not actually at would be evidence of a state nothing holds. + +Cancellation between the merge and the commit rolls the outer transaction back and publishes no completion at all. A completed record of either outcome replays without running Git. + +A merge never contacts a Git host, never pushes, and never rewrites a published identity. Rebase, force, force-with-lease and reset-based replacement are absent from this component and from every other one in this specification. ### 7.9 Publishing a reviewed merge: `Git.PublishTarget` @@ -1713,34 +1754,60 @@ the credential and the non-force policy are host-owned: they are not props, and no authored value widens them. Form validation runs before the host's ceiling is read and before any credential exists. -**It is a compare-and-swap.** One non-force ref update happens only after -observing the target equal to `expectedRemoteCommit`. A target already equal to -the exact `sourceCommit` is adopted with nothing performed. Every other -observation refuses without mutating: a target at some third commit is a -conflict, an incomplete observation is unavailability rather than absence, and a -permanent ambiguity is a refusal. A race that moves the target before or during -publication therefore cannot publish over it. - -The bound result is stable evidence of what the effect settled on — the target -ref identity, the expected commit, the published commit and the reviewed head — -and not a live branch snapshot. `reviewedHead` is carried so the record says -what the publication was authorized against, which is what lets an -exact-revision review be invalidated by a target that moved. - -**Three operations stay distinct.** `Git.Push` (§7.4) advances a branch this run -published, from an ancestry relation proved inside the authenticated object -source. `Git.PublishTarget` updates a ref it does not own, from an exact -expected pre-state. A Git host's own pull-request merge endpoint is neither, and -this specification defines no component for one: a squash or a rebase performed -by the host would publish a commit no reviewer saw, under parents the review -never named. +**The request names the target the host chose and the commits the document did.** Its natural key is the target identity alone — the retained Repository, the configured remote and the configured target ref — because one ref has one publication at a time whoever is asking: + +```ts +interface GitPublishTargetRequest { + readonly kind: "git-publish-target"; + readonly target: GitPublishTarget; + readonly expectedRemoteCommit: string; + readonly sourceCommit: string; + readonly reviewedHead: string; +} + +interface GitPublishTarget { + readonly repository: string; + readonly remote: string; + readonly ref: string; +} +``` + +`repository` is the Workspace-local Repository name, `remote` is the configured remote's name, and `ref` is the fully qualified destination ref. The credential, the locator behind the remote and the non-force policy are provider closure state and appear in neither the request nor the result. The three commits are lowercase hexadecimal object IDs of the repository's object format. + +**It is a compare-and-swap**, and its five observed pre-states are exhaustive: + +| Observation | Decision | +| --- | --- | +| the target equals `expectedRemoteCommit` | perform one non-force update, once | +| the target equals `sourceCommit` | adopt; nothing is performed | +| the target equals some third commit | conflict; refuse without mutating | +| the observation did not complete | incomplete observation; refuse as itself, adopt nothing, perform nothing | +| the observation cannot be decided, or the host is temporarily unreachable | permanent ambiguity and temporary unavailability respectively; each refuses as itself and never performs | + +A race that moves the target before or during publication therefore cannot publish over it, and an interrupted attempt is reobserved rather than repeated: a target that now equals `sourceCommit` is the adoption, and one that does not is not silently published over. + +**The result is one closed record**, and the binding is that record: + +```ts +interface GitPublishTargetResult { + readonly target: GitPublishTarget; + readonly expectedRemoteCommit: string; + readonly reviewedHead: string; + readonly sourceCommit: string; + readonly observedCommit: string; + readonly decision: "performed" | "adopted"; +} +``` + +`observedCommit` is what the target held when this attempt looked, so the record says what the publication moved from or found already done; `decision` is the closed pair above and no third value exists. `reviewedHead` is carried so the record says what the publication was authorized against, which is what lets an exact-revision review be invalidated by a target that moved. It is stable evidence of what the effect settled on, not a live branch snapshot. Every member is required, and an unknown member or an unknown `decision` refuses the record. + +Cancellation tears the provider call down and publishes no completion. A completed record replays without contacting a Git host. + +**Three operations stay distinct.** `Git.Push` (§7.4) advances a branch this run published, from an ancestry relation proved inside the authenticated object source. `Git.PublishTarget` updates a ref it does not own, from an exact expected pre-state. A Git host's own pull-request merge endpoint is neither, and this specification defines no component for one: a squash or a rebase performed by the host would publish a commit no reviewer saw, under parents the review never named. Whether the host has *noticed* that its pull request is now merged is a fourth question, and §7.11 owns it. ### 7.10 Pull-request comments, readiness and closure -Three more Git-host effects act on a pull request a canonical URL names. Each -requires `as`, validates its form before any provider or credential is reached, -and reconciles under §10.2: observe, adopt a compatible completion, perform a -proven absence once, refuse conflict and ambiguity. +Three more Git-host effects act on a pull request a canonical URL names. Each requires `as`, validates its form before any provider, ceiling or credential is reached, and reconciles under §10.2: observe, adopt a compatible completion, perform a proven absence once, refuse conflict, permanent ambiguity, incomplete observation and temporary unavailability. Every identity below is a canonical URL or a lowercase hexadecimal object ID; no credential, endpoint, raw payload, cursor or host path appears in any request, natural key or result. ```md @@ -1751,22 +1818,104 @@ The Architect accepted {revision.headSha} against {revision.baseSha}. ``` -`PullRequest.Comment` is paired and its rendered content is the body, verbatim. -Its natural key is the canonical pull-request URL plus the engine-derived effect -identity of [Workflow runs](./workflow-spec.md) §8 — never the body — so an -edited sentence is the same comment and a re-rendered body is not a second one. -It binds `{ url }`. +#### `PullRequest.Comment` + +The component is paired, takes one required `url` and one required `as`, and its rendered content is the body verbatim. `url` is the **canonical pull-request URL** — the normalized single spelling of one pull request, on the terms §10.3 already states for a canonical target URL — and the durable request is exactly that URL, the engine-derived effect identity and the rendered body: + +```ts +interface PullRequestCommentRequest { + readonly kind: "pull-request-comment"; + readonly subject: string; + readonly effect: string; + readonly body: string; +} + +interface PullRequestCommentResult { + readonly subject: string; + readonly url: string; + readonly decision: "performed" | "adopted"; +} +``` + +`subject` is the canonical pull-request URL, `effect` is the engine-derived effect identity of [Workflow runs](./workflow-spec.md) §8, and `url` is the canonical URL of the comment this effect settled on. The binding is `{ url }`: that comment's own URL, which is the only fact the effect produces — the subject was already in hand at the call site. + +**The natural key is `subject` plus `effect`, and the body is never part of it.** A Git host issues no client-supplied idempotency key for a comment, so the effect identity has to be observable on the host for an interrupted creation to be found again. The adapter makes it observable by appending one fixed, presentation-neutral trailer to the body it sends — an HTML comment carrying the effect identity and nothing else — and observation matches that trailer alone. The trailer is adapter-owned rather than authored: a document does not write it, the rendered body it wrote is not searched for it, and a person editing the comment's prose afterwards leaves the identity untouched. That is what makes an edited sentence the same comment and a re-rendered body not a second one. + +Observation is complete or it is not an observation: a comment list the adapter could not finish reading is an incomplete observation and refuses, because an unfinished search would report absence and create a duplicate. Exactly one comment carrying the trailer is a compatible completion and is adopted with nothing performed; none is a proven absence and is performed once; more than one is permanent ambiguity and refuses. Temporary unavailability refuses as itself and performs nothing. + +#### `PullRequest.Ready` and `PullRequest.Close` + +Both are self-closing, take one required `url` and one required `as`, and are keyed by that exact canonical pull-request URL — one readiness and one closure per pull request, so neither carries an effect identity in its key. Their bindings are the closed records below, and each durable result is its binding plus the observation the attempt made: + +```ts +interface PullRequestReadyBinding { + readonly url: string; + readonly state: "open"; + readonly draft: false; +} + +interface PullRequestCloseBinding { + readonly url: string; + readonly state: "closed"; + readonly merged: false; +} + +interface PullRequestReadyResult extends PullRequestReadyBinding { + readonly observed: PullRequestObservation; + readonly decision: "performed" | "adopted"; +} + +interface PullRequestCloseResult extends PullRequestCloseBinding { + readonly observed: PullRequestObservation; + readonly decision: "performed" | "adopted"; +} + +interface PullRequestObservation { + readonly state: "open" | "closed"; + readonly draft: boolean; + readonly merged: boolean; +} +``` + +`state`, `draft` and `merged` are literal in each binding rather than observed values copied through, because a binding that could say `draft: true` would be a component reporting that it did not do what it is for. + +`PullRequest.Ready` performs once from an observed `{ state: "open", draft: true, merged: false }`, adopts an observed `{ state: "open", draft: false, merged: false }` with nothing performed, and refuses every other observation as a conflict — a merged or closed pull request among them, since readiness is not a thing to restore. `PullRequest.Close` performs once from an observed `{ state: "open", merged: false }` at either draft state, adopts an observed `{ state: "closed", merged: false }`, and conflicts with an observed `merged: true`, which is a completion of a different kind that closing must not overwrite. A pull request belonging to another repository, or one the URL names but the host does not hold, is a conflict for both. An incomplete observation, a permanent ambiguity and a temporary unavailability each refuse as themselves and perform nothing. + +Neither reopens, merges, comments on or pushes anything. Cancellation tears the provider call down and publishes no completion; a completed record of any of the three replays without contacting a Git host. Which of them a document may invoke, and what authorizes the invocation, is authored control flow above them. + +### 7.11 Observing that a pull request merged: `PullRequest.Merged` + +Publishing a merge commit to a target ref and a Git host recording that pull request as merged are two different facts, and the second one is not implied by the first. A host observes its own ref moving and closes the pull request on its own schedule, so a run that needs the merged state in its history has to observe it — and has to observe it as its own retained step rather than as a side effect of something else. + +```md + +``` + +The component is self-closing, its three props are required and the set is closed, and it is a Git-host effect of its own. It is not `PullRequest.Ready`, not `PullRequest.Close`, not a pull-request upsert and not `Git.PublishTarget`: overloading any of them would make one record answer two questions, and the two can disagree. + +**It mutates nothing.** It is a reconciled observation: adoption is its only completion, and there is no `performed` decision for it to reach. What it reconciles is *when* the fact becomes true, because a host that has not yet noticed the ref move is not a host that refused. + +```ts +interface PullRequestMergedRequest { + readonly kind: "pull-request-merged"; + readonly subject: string; + readonly expectedMergeCommit: string; +} + +interface PullRequestMergedResult { + readonly subject: string; + readonly state: "closed"; + readonly merged: true; + readonly mergeCommit: string; + readonly decision: "adopted"; +} +``` + +`subject` is the canonical pull-request URL and is the whole natural key. The binding is the result record. -`PullRequest.Ready` takes a pull request out of draft and binds normalized ready -evidence. `PullRequest.Close` closes one unmerged and binds exactly -`{ url, state: "closed", merged: false }`. Both are self-closing, both are keyed -by their exact subject, and neither reopens, merges or comments on anything. -Observing a pull request already ready is an adoption; observing one that has -been merged, or that belongs to another repository, is a conflict. +The observation is complete or it is nothing. A pull request the host reports as `merged` with a merge commit equal to `expectedMergeCommit` is the compatible completion and is adopted. One reported merged at a *different* commit is a conflict: the target carries somebody else's merge, and the exact-revision reviews invalidate rather than this step succeeding. One still open, or closed unmerged, is temporary unavailability rather than absence or conflict — the host has not caught up, and a later attempt starts again at observation. An incomplete read and an undecidable one are incomplete observation and permanent ambiguity, each refusing as itself. -None of the three pushes, and none of them derives authority from what it -observes. Which of them a document may invoke, and what authorizes the -invocation, is authored control flow above them. +Cancellation publishes no completion, and a completed record replays without contacting a Git host. Where this step sits in the terminal sequence — after `Git.PublishTarget` and before the issue is closed — belongs to [the software factory](./github-actions-software-factory-spec.md) §10.3. ## 8. Agents inspect; XMD mutates @@ -1821,8 +1970,9 @@ by #496 and does not widen this ceiling. Constructs added for a trusted host do not reach the Agent either, and they do not reach it for a different reason than the tool set: an Agent never expands a -document. Merging, publishing a target, commenting, changing draft state, -closing an issue or a pull request, moving a Project item and running evidence +document. Merging, publishing a target, observing that a pull request merged, +commenting, changing draft state, closing an issue or a pull request, moving a +Project item and running evidence are authored XMD the trusted host expands under its own acquisition. What an Agent may return is text, and a fragment it returns is admitted only against the tables §8.4 states — which name none of them. @@ -2149,7 +2299,8 @@ merely by naming a component; the table excludes local Git even though those effects are also Workspace-local. The constructs §§7.8-7.10, §10.5 and §10.6 add change nothing about that: `Git.Merge`, `Git.PublishTarget`, `PullRequest.Comment`, `PullRequest.Ready`, `PullRequest.Close`, -`Issue.Comment`, `Issue.Close`, `Project.Status` and `Evidence.Run` appear in no +`PullRequest.Merged`, `Issue.Comment`, `Issue.Close`, `Project.Status` and +`Evidence.Run` appear in no table this specification states, so a fragment naming one is refused in the preflight before any generated effect, exactly as `` is. Adding a construct to a table is a host act, and a factory host adds none of them. @@ -2850,17 +3001,53 @@ The Planner accepted the plan at {plan.revision}. ``` -`Issue.Comment` is paired and its rendered content is the body, verbatim. Its -natural key is the canonical issue URL plus the engine-derived effect identity, -never the body, so an edited sentence is the same comment. It binds `{ url }`. +`Issue.Comment` is paired, takes one required `url` and one required `as`, and its rendered content is the body verbatim. Its records mirror the pull-request comment of §7.10 exactly, under this boundary instead of the Git host's: + +```ts +interface IssueCommentRequest { + readonly kind: "issue-comment"; + readonly subject: string; + readonly effect: string; + readonly body: string; +} + +interface IssueCommentResult { + readonly subject: string; + readonly url: string; + readonly decision: "performed" | "adopted"; +} +``` + +`subject` is the canonical issue URL — the normalized single spelling this section already requires of a target — `effect` is the engine-derived effect identity, and `url` is the canonical URL of the comment the effect settled on. The binding is `{ url }`, that comment's own URL. + +**The natural key is `subject` plus `effect`, and the body is never part of it.** As with a pull-request comment, the effect identity is made remotely observable by one fixed presentation-neutral trailer the adapter appends to the body it sends, and observation matches that trailer alone. Exactly one comment carrying it is adopted, none is performed once, more than one is permanent ambiguity, and a comment list the adapter could not finish reading is an incomplete observation — never absence. -`Issue.Close` is self-closing and its `reason` is a closed enum of `"completed"` -and `"not_planned"`. It binds the normalized URL, state and reason. The reason -is part of what the effect means rather than a label on it: a host that retained -one terminal intent refuses a close naming the other. An issue already closed -with the same reason is adopted; one closed with the other reason is a conflict. +`Issue.Close` is self-closing and takes one required `url`, one required `reason` from the closed enum `"completed" | "not_planned"`, and one required `as`. Its natural key is the canonical issue URL alone: -Neither reopens an issue, and neither derives authority from what it observes. +```ts +interface IssueCloseRequest { + readonly kind: "issue-close"; + readonly subject: string; + readonly reason: "completed" | "not_planned"; +} + +interface IssueCloseBinding { + readonly url: string; + readonly state: "closed"; + readonly reason: "completed" | "not_planned"; +} + +interface IssueCloseResult extends IssueCloseBinding { + readonly observed: { readonly state: "open" | "closed"; readonly reason?: "completed" | "not_planned" }; + readonly decision: "performed" | "adopted"; +} +``` + +`state` is literal in the binding: a component for closing an issue does not report that the issue is open. The observed `reason` is absent rather than null when the issue is open or when the host records no reason for a closure it holds. + +The reason is part of what the effect means rather than a label on it, so a host that retained one terminal intent refuses a close naming the other. An observed open issue is performed once. An observed issue closed with the same reason is adopted with nothing performed. An observed issue closed with the other reason is a conflict, and so is one closed with no reason the host will state, because adopting it would let a `not_planned` closure stand as a `completed` one. An incomplete observation, a permanent ambiguity and a temporary unavailability each refuse as themselves. + +Neither reopens an issue, and neither derives authority from what it observes. Cancellation publishes no completion, and a completed record of either replays without contacting a provider. ### 10.4 Worker Shell @@ -2901,29 +3088,59 @@ document may reach for by itself. ``` -`commands` is an authored **structured argv list** — an ordered list of argument -vectors, each a non-empty list of strings — and never a shell string. There is -no interpreter, no quoting layer and no string to mis-split, which is what makes -the record of what ran the same thing as what ran. `as` is required, the prop -set is closed, and the form is validated before the host's ceilings are read. +`commands` is an authored **structured argv list**: an ordered, non-empty list whose every member is a non-empty list of strings. There is no interpreter, no quoting layer and no string to mis-split, which is what makes the record of what ran the same thing as what ran. `as` is required, the prop set is closed, and the form is validated before the host's ceilings are read — an empty list, an empty vector, a member that is not a list of strings, an unknown prop and a missing `as` each fail before any child exists. + +**The whole list runs.** Each command runs exactly once, in authored order, and a command that ends unsuccessfully does not stop the ones after it. Running the list is how a reader learns what is broken, and a list that stopped at the first defect would hide the rest — the same reason this repository's own test shards run their files to the end. Each command is charged its own duration ceiling, so one command timing out does not consume another's. + +The host owns everything else. Which executables may run, what environment they see, the logical working root, how long each command may take, how much output is retained, and what happens to a process tree are host ceilings rather than props. No host path, shell string, ambient environment or command authority enters an authored prop. The commands run against one exact retained Workspace root the host materialized, on the trusted runner where the native toolchain lives — never inside the run's durable owner, which has no toolchain and must not acquire one. + +**The result is one record per command, in authored order:** + +```ts +interface EvidenceRunResult { + readonly commands: readonly EvidenceCommandResult[]; +} + +interface EvidenceCommandResult { + readonly argv: readonly string[]; + readonly outcome: "exited" | "signalled" | "timeout"; + readonly status?: number; + readonly signal?: string; + readonly stdout: EvidenceChannel; + readonly stderr: EvidenceChannel; +} + +interface EvidenceChannel { + readonly text: string; + readonly retainedBytes: number; + readonly producedBytes: number; + readonly truncated: boolean; +} +``` + +`commands` has exactly one entry per authored command, in the order they were authored, and `argv` repeats the vector that ran. `status` is present exactly when `outcome` is `"exited"` and is the numeric exit status; `signal` is present exactly when `outcome` is `"signalled"` and names the signal; `outcome: "timeout"` carries neither, because a child the host terminated for exceeding its ceiling did not choose how it ended. An unknown member, an unknown `outcome`, and a `status` or `signal` beside the wrong outcome each refuse the record. + +Both channels are retained separately and neither is folded into the other: `text` is the retained UTF-8 prefix, `retainedBytes` is its length in bytes, `producedBytes` is what the child actually produced, and `truncated` is `producedBytes > retainedBytes`. Truncation is stated rather than inferred from a length, so a reader never has to guess whether a command was quiet or cut off. + +**Which cases bind, which fail, and which commit nothing.** + +| Case | Outcome | +| --- | --- | +| a command exits, at any status | an ordinary result; the list continues | +| a command is terminated by a signal | an ordinary result; the list continues | +| a command exceeds its duration ceiling and the host terminates it | an ordinary `"timeout"` result; the list continues | +| the executable or environment ceiling refuses a command, or the child cannot be created | **launch failure**: the effect fails with an `Error`, binds nothing and commits no result | +| the host cannot read a channel it promised to bound | **output-pump failure**: the effect fails with an `Error`, binds nothing and commits no result | +| the host cannot reap a child or its process tree | **teardown failure**: the effect fails with an `Error`, binds nothing and commits no result | +| the effect is cancelled | nothing is committed at all: the running child is terminated, the remaining commands do not run, and no result and no failure record is published | + +A non-zero status is evidence, not an infrastructure failure — it is the answer the evidence exists to obtain. An infrastructure failure is the host being unable to say what happened, which is why it publishes no partial list: half an answer read as a whole one is worse than no answer. -The host owns everything else. Which executables may run, what environment they -see, how long they may take, and how much output is retained are host ceilings -rather than props. The commands run against one exact retained Workspace root -that the host materialized, on the trusted runner where the native toolchain -lives — never inside the run's durable owner, which has no toolchain and must -not acquire one. +**Precedence is fixed.** Cancellation wins over everything, and invents no completion. Otherwise the first infrastructure failure raised wins, and a teardown failure with no earlier infrastructure failure fails the effect even when every command produced an ordinary outcome — a host that cannot prove it stopped what it started cannot state what the run produced. This is the same rule the workflow lifecycle applies to settlement: teardown is part of the evidence, not work performed after the outcome. -The bound result is an ordered list of bounded results, one per command: the -argv it ran, its exit status, and its truncated captured output. That is an -ordinary durable record, so a completed replay produces it without running -anything at all. +A completed record replays without running anything at all. -`Evidence.Run` is not Worker Shell (§10.4) and does not replace it: Worker Shell -is a contained interpreter over the Workspace filesystem, while this is native -execution of an authored list under a trusted host's ceiling. It is absent from -the workflow Agent's capabilities (§8.3) and from every generated-XMD table -(§8.4), so neither an Agent nor a fragment it wrote can reach it. +`Evidence.Run` is not Worker Shell (§10.4) and does not replace it: Worker Shell is a contained interpreter over the Workspace filesystem, while this is native execution of an authored list under a trusted host's ceiling. It is absent from the workflow Agent's capabilities (§8.3) and from every generated-XMD table (§8.4), so neither an Agent nor a fragment it wrote can reach it. ### 10.6 Project effects @@ -2941,14 +3158,31 @@ reconciliation rather than the Git host's state machine. ``` -The component is self-closing, its four props are required and the set is -closed, and it binds the normalized `{ item, field, option }`. Its natural key -is the exact item plus the exact field. Its compatible pre-state is the option -that item currently holds: an item already at the requested option is adopted -with nothing performed, an item at another allowed option is performed once, and -an unreadable board, an unavailable field, an ambiguous item and a partial -permission read are **unavailable** rather than absent — reading an unreadable -board as an empty one is how an unauthorized item would be moved. +The component is self-closing, its four props are required and the set is closed, and it binds the normalized `{ item, field, option }`. Its natural key is the exact item plus the exact field — one item has one value of one field, whoever is asking — and its request and result are these: + +```ts +interface ProjectStatusRequest { + readonly kind: "project-status"; + readonly item: string; + readonly field: string; + readonly option: string; +} + +interface ProjectStatusBinding { + readonly item: string; + readonly field: string; + readonly option: string; +} + +interface ProjectStatusResult extends ProjectStatusBinding { + readonly observedOption?: string; + readonly decision: "performed" | "adopted"; +} +``` + +`item`, `field` and `option` are the provider's own opaque identities, compared byte for byte and never normalized, decoded or repaired — they are provider identities on the same terms an issue node ID is. `option` in the binding is the requested one, which after a successful effect is the one the item holds. `observedOption` is what the field held when this attempt looked, and it is absent rather than null when the field held no option at all. An unknown member and an unknown `decision` refuse the record. + +Its compatible pre-state is the option that item currently holds. An item already at the requested option is adopted with nothing performed; an item at another option the host's ceiling allows is performed once; an item at an option outside that ceiling is a conflict, because moving it would be publishing through a status this factory does not own. An unreadable board, an unavailable field, an ambiguous item and a partial permission read are **unavailable** rather than absent — reading an unreadable board as an empty one is how an unauthorized item would be moved — and a temporary unreachability refuses as itself. Cancellation publishes no completion, and a completed record replays without contacting a provider. Which project, item, field and options may be reached at all is a host ceiling installed beside the credential. An authored prop selects within that ceiling @@ -3450,26 +3684,48 @@ runtime-named adapter beside the Deno one: shared modules reach it through the same contextual storage, lifecycle and Workspace APIs, detect no runtime, and import nothing Cloudflare-specific. -Executor ownership is one authenticated WebSocket connection whose lifetime is -the acquisition. The owner registers the exact acquisition on admission and -invalidates it on close; there is no duration, expiry, renewal, heartbeat, -generation record or liveness poll, and a close rolls back nothing already -committed. Every mutating transaction validates that exact acquisition and the -expected Workspace root together. - -Native Git, evidence processes and Agent clients run on the ephemeral runner and -nowhere else. The runner materializes one selected retained root, works in it, -and submits content-addressed changes; the owner validates acquisition, root and -content and then atomically publishes the new root with the filtered journal -result. That is §10.1's effect transaction with the mutation performed where the -tools are and the publication performed where the authority is, so a runner -crash exposes only a prior or a new complete transaction and the next -acquisition resumes from the exact committed frontier. - -Delivery and inspection reach the owner without an acquisition, under §3.8. A -completed run replays there as it does locally: it attaches no remote storage -session, Workspace, Agent, process, Git, Git-host, Issue, Project or credential -provider. +**The host assembly contract does not change.** `WorkflowHost` keeps its four methods — `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()` — and the Cloudflare adapter is one more implementation of them beside the Deno one. Starting, looking up, executing, delivering into and inspecting a run are lifecycle operations reached *through* that boundary, exactly as they are locally; they are not replacement method names, and no fifth method appears. What a remote adapter changes is where each of those four reaches, not what the shared CLI asks for. + +Executor ownership is one authenticated WebSocket connection whose lifetime is the acquisition. The owner registers the exact acquisition on admission and invalidates it on close; there is no duration, expiry, renewal, heartbeat, generation record or liveness poll, and a close rolls back nothing already committed. Every mutating transaction validates that exact acquisition and the expected Workspace root together. + +Native Git, evidence processes and Agent clients run on the ephemeral runner and nowhere else. The runner materializes one selected retained root, works in it, and submits content-addressed changes; the owner validates acquisition, root and content and then atomically publishes the new root with the filtered journal result. That is §10.1's effect transaction with the mutation performed where the tools are and the publication performed where the authority is, so a runner crash exposes only a prior or a new complete transaction and the next acquisition resumes from the exact committed frontier. + +#### The transport between them is versioned + +A runner and its durable owner deploy separately, so the connection between them carries records that outlive one deployment and has to be treated as a compatibility boundary rather than as an internal calling convention. It is a closed, versioned, discriminated envelope: + +```ts +interface RunnerRequest { + readonly protocol: 1; + readonly operation: "acquire" | "frontier" | "materialize" | "commit" | "settle"; + readonly runId: string; + readonly acquisition?: string; + readonly payload: Json; +} +``` + +`acquire` presents the runner's admission assertion and asks for the executor acquisition. `frontier` reads the committed run record and current Workspace root. `materialize` asks for the content-addressed bytes of one retained root. `commit` submits one mutation — the expected root, the content-addressed additions and the already-filtered journal result to append — for the owner to validate and publish atomically. `settle` publishes the run's resulting status. `acquisition` is absent only on `acquire` and required on the other four. There is no operation for delivery or inspection on this connection: those reach the owner on their own authenticated paths and take no acquisition. + +Compatibility is refusal, never adaptation. A request whose `protocol` this owner does not implement is refused with a fixed protocol-version refusal before any other member is read; an unknown `operation`, an unknown member, and an `acquisition` present or absent against the rule above each refuse without effect. The owner never reads an older shape charitably and the runner never downgrades to one, because a transport that guessed would let two deployments disagree about what was committed. + +#### Which side owns what + +| Concern | Owner | +| --- | --- | +| connection admission, including OIDC claim validation and acquisition registration | the durable owner | +| parsing every request | the durable owner; the runner parses only responses | +| opening, committing and rolling back transactions | the durable owner | +| content-addressed transfer | the runner produces content and names it; the owner validates and stores it | +| attaching Workspace, Agent, process, Git, Git-host, Issue and Project providers | the runner | +| cancellation | whichever side owns the scope being cancelled: the runner cancels its own document execution, and the owner cancels nothing on its behalf | +| closing the connection | either side; the owner invalidates the acquisition when it closes | +| stale-execution recovery | the durable owner, at the next acquisition | + +Delivery and inspection reach the owner without an acquisition, under §3.8, on authenticated paths of their own. + +#### What completed replay does and does not reach + +A completed run replays there as it does locally: it attaches no Workspace, Agent, process, Git, Git-host, Issue, Project or credential provider and performs no effect a second time. It does reach the run's durable owner, because that is where the retained result is; an ephemeral client holds nothing of its own to replay from. Reading retained completion from the owner that holds it is not attaching a provider, and the distinction is the whole point of the rule: what a completed replay must not do is contact an *external* service or repeat an effect, not refrain from reading its own history. ## 14. Contract inventory @@ -3502,10 +3758,11 @@ provider. | `Git.Merge` ordered Workspace-local merge (§7.8) | specified by #710; implementation unbuilt | | `Git.PublishTarget` compare-and-swap target publication (§7.9) | specified by #710; implementation unbuilt | | `PullRequest.Comment`, `PullRequest.Ready`, `PullRequest.Close` (§7.10) | specified by #710; implementation unbuilt | +| `PullRequest.Merged` reconciled merged observation (§7.11) | specified by #710; implementation unbuilt | | `Issue.Comment` and `Issue.Close` (§10.3) | specified by #710; implementation unbuilt | | `Evidence.Run` trusted native evidence execution (§10.5) | specified by #710; implementation unbuilt | | `Project.Status` and the Project-provider boundary (§10.6) | specified by #710; implementation unbuilt | -| remote lifecycle host, executor connection and remote topology (§3.8, §13.2) | specified by #710; implementation unbuilt | +| remote lifecycle host, executor connection, versioned runner transport and remote topology (§3.8, §13.2) — the existing four-method `WorkflowHost` boundary, with a Cloudflare implementation beside the Deno one | specified by #710; implementation unbuilt | | terminal-decision delivery on the delivery plane (§3.8) | specified by #710; implementation unbuilt | | Worker JavaScript | deferred | | bundled workerd local host | omitted; POC #347 / PR #348 retained as provider evidence | From 0d08dcc0df08eee91a4f8d4ba4da962d8590a803 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 11:27:11 -0400 Subject: [PATCH 03/81] =?UTF-8?q?=F0=9F=93=9D=20Resolve=20the=20software?= =?UTF-8?q?=20factory=20contract=20rulings=20(#710)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Apply rulings A-K from the Architect's answers on 80884f9. A. Keep the four-method `WorkflowHost` and add that its transition and request types — `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection`, `WorkflowRunCreation` — are provider-neutral and become package-root public types, with runtime-named entrypoints keeping implementations and retained encodings. Recorded as authorized future export work, not performed here. B. `Evidence.Run` becomes an ordered fail-fast pipeline binding the executed prefix: `{ completion, authoredCommands, executed, runTimeout? }`, rows carrying `limit` for whichever of the two host-owned ceilings fired, a `runTimeout` record for a whole-run expiry between commands rather than a fabricated argv row, timeouts as ordinary unsuccessful outcomes, and fixed precedence — cancellation over everything, first infrastructure failure authoritative with a later teardown failure as secondary evidence, teardown authoritative alone — with bounded diagnostic evidence retained on a failed effect's Error. C. A comment provider must support a stable opaque correlation marker or refuse before its first mutation. The authored logical body is preserved byte for byte and the correlation representation rides outside it. Absence is now judged against attempt state: no marker after an attempted-but-uncommitted creation is permanent ambiguity, not proven absence. D. `PullRequest.Merged` splits still-open (temporary unavailability) from closed-unmerged (conflict), and the factory gains §10.4 — a bounded host-configured retry, then a durable machine wait with its own protocol record, woken by intake or an explicit resume and carrying no verdict. E. A configured total bijection between the nine stages and nine exact Project status option IDs, validated against a complete reread and refusing before any intake, token, run or projection. F. `Git.Merge`'s `purpose` is validated authorization against the provider-authenticated merge ceiling, not write-only provenance. G. The `RunnerRequest` envelope is deleted. The runner and owner ship as one release identity checked by build fingerprint at admission; the messages are private, while the owner-side authority invariants stay public and exact. H. `FactoryTerminal` keeps journal event references, with cross-path, kind, completeness and agreement validation spelled out. I. The factory protocol schemas stay in the factory specification §11.2; the other four documents link rather than duplicate. J/K. Mixed wrapping left alone, new prose unwrapped; the tenth construct is reflected everywhere. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- architecture.md | 26 ++-- specs/executable-mdx-spec.md | 35 +++-- specs/github-actions-software-factory-spec.md | 74 +++++++++- specs/workflow-spec.md | 4 +- specs/workflow-workspace-spec.md | 130 ++++++++++++------ 5 files changed, 201 insertions(+), 68 deletions(-) diff --git a/architecture.md b/architecture.md index 3d78b07e4..4e4d9435c 100644 --- a/architecture.md +++ b/architecture.md @@ -1505,11 +1505,13 @@ The status a board shows is a **Project projection** of the journaled lifecycle, never a reading of it. A board ahead of the journal is drift the next execution reconciles, and it is not evidence that a stage was passed. +Which option means which stage is configuration, not inference. A host that projects lifecycle onto a board holds a total bijection between its stages and the board's exact status option IDs, validated against a complete reread before it is used and refusing when it is missing, partial, duplicated, names an option the board does not offer or one under another field, or names an option whose display name is not the settled string for its stage. Admission maps a reread option ID to a stage through that table and projection maps a stage back through its inverse; neither direction parses a display string, because a name is what a person reads and an option ID is what the host compares. + ### Comments, readiness and closure -Four more reconciled effects join the same boundary, each keyed by its own subject. `Issue.Comment` is an Issue-provider effect keyed by the canonical issue URL plus the engine-derived effect identity; `PullRequest.Comment` is a Git-host effect keyed by the canonical pull-request URL plus that identity. The body is presentation in both: keying a comment by its text would make an edited sentence a different comment. A Git host issues no idempotency key for a comment, so the effect identity is made remotely observable by one fixed presentation-neutral trailer the adapter appends and matches on — adapter-owned, never authored, and untouched when a person edits the prose around it. `PullRequest.Ready` and `PullRequest.Close` are Git-host effects keyed by their exact pull-request subject, and `Issue.Close` is an Issue-provider effect keyed by its exact issue subject and carrying a closed `reason` enum that has to match the retained terminal intent. Each observes before it mutates, adopts a compatible completion, performs once from proven absence or an exact compatible pre-state, and refuses conflict, permanent ambiguity, incomplete observation and temporary unavailability. +Four more reconciled effects join the same boundary, each keyed by its own subject. `Issue.Comment` is an Issue-provider effect keyed by the canonical issue URL plus the engine-derived effect identity; `PullRequest.Comment` is a Git-host effect keyed by the canonical pull-request URL plus that identity. The body is presentation in both: keying a comment by its text would make an edited sentence a different comment. A Git host issues no idempotency key for a comment, so a comment provider has to support one stable opaque correlation marker it can write, preserve and completely query, and a provider that cannot refuses before its first mutation. The marker is provider transport metadata rather than authored prose: the authored logical body stays byte for byte what the document rendered, the correlation representation rides outside it, and the binding and every replay expose the body and the provider's comment identity rather than the encoding. What a complete observation means depends on the attempt state the effect retains — before an attempt, no marker is proven absence and permits one creation; after an attempt with no committed completion, no marker is permanent ambiguity, because it equally describes somebody having removed one. That is what stops an interrupted creation from becoming a duplicate without pretending a person cannot edit a comment. `PullRequest.Ready` and `PullRequest.Close` are Git-host effects keyed by their exact pull-request subject, and `Issue.Close` is an Issue-provider effect keyed by its exact issue subject and carrying a closed `reason` enum that has to match the retained terminal intent. Each observes before it mutates, adopts a compatible completion, performs once from proven absence or an exact compatible pre-state, and refuses conflict, permanent ambiguity, incomplete observation and temporary unavailability. -`PullRequest.Merged` is the fourth, and it is the one that never mutates. A Git host records a pull request as merged when it notices its own ref move, which is not the same event as a target publication succeeding, so the fact has to be observed as its own retained step rather than inferred from the step before it. Adoption is its only completion: the host reporting the pull request merged at the exact published commit is the fact, a merge at another commit is a conflict, and a pull request still open or closed unmerged is temporary unavailability — the host has not caught up, which is a different thing from refusing. +`PullRequest.Merged` is the fourth, and it is the one that never mutates. A Git host records a pull request as merged when it notices its own ref move, which is not the same event as a target publication succeeding, so the fact has to be observed as its own retained step rather than inferred from the step before it. Adoption is its only completion: the host reporting the pull request merged at the exact published commit is the fact, and a merge at another commit is a conflict. A pull request still open is temporary unavailability — the host has not caught up, which is a different thing from refusing — while one closed unmerged is a conflict, because a person having intervened is not a state that resolves itself by waiting. A run that keeps observing the open case does so through a bounded host-configured retry and then a durable machine wait whose subject is the pull request and the expected commit; that wait carries no verdict and appends no outcome, so waking it is permission to look again rather than an answer. ### Ordered merge, and publishing a target @@ -1534,9 +1536,13 @@ publication was authorized against. ### Trusted evidence execution -`Evidence.Run` executes an authored structured argv list natively, on the trusted runner, against one exact retained Workspace root, under host-owned executable, environment, working-root, duration, output and process-tree ceilings. It is not Worker Shell and not a document's own process capability: it runs where the tools are, and it is absent from the workflow Agent's capabilities and from every generated-XMD write table. +`Evidence.Run` executes an authored structured argv list natively, on the trusted runner, against one exact retained Workspace root, under host-owned executable, environment, working-root, per-command duration, whole-run duration, output and process-tree ceilings. It is not Worker Shell and not a document's own process capability: it runs where the tools are, and it is absent from the workflow Agent's capabilities and from every generated-XMD write table. + +It is a fail-fast pipeline. Commands run in authored order and the first one that does not exit with status `0` is the last one that runs, because a plan's evidence list is usually a pipeline and rows produced after a failed build are evaluated against prerequisites that are missing or stale. What the effect binds is therefore the executed prefix, stated as such: how it completed, how many commands were authored, and one row per command that ran, each naming its argv, how it ended, and its stdout and stderr as separate bounded channels that state their own truncation. Breadth belongs inside a command whose own contract runs a corpus to the end, or in separate elements the plan says are independent. + +Two host-owned ceilings bound it — one per command, one for the whole list — and a row that timed out says which fired. A timeout is an ordinary unsuccessful outcome: it records that the host enforced its ceiling, reaped the tree and captured its channels. A non-zero status is likewise evidence rather than an infrastructure failure, since it is the answer the effect exists to obtain. -Its record is one entry per authored command, in authored order, each naming the argv that ran, how it ended, and its stdout and stderr as separate bounded channels that state their own truncation. The whole list runs: a command ending unsuccessfully does not stop the ones after it, because running the list is how a reader learns what is broken. A non-zero status is evidence rather than an infrastructure failure — the answer the effect exists to obtain. Being unable to say what happened is the failure: a launch the ceiling refused, a channel the host could not read, and a child it could not reap each fail the effect and bind nothing, so no partial list is ever read as a whole one. A teardown failure outranks an otherwise ordinary result on the same terms lifecycle settlement applies, and cancellation commits nothing at all. A completed record replays running nothing. +Being unable to say what happened is the failure. A launch the ceiling refused, a channel the host could not read, and a child it could not reap each fail the effect and bind no result, so no prefix is ever mistaken for an answer. Cancellation wins over everything and commits neither completion nor failure; otherwise the first infrastructure failure is authoritative and a teardown failure after it is retained as secondary evidence, while a teardown failure with nothing before it is authoritative on its own — a host that cannot prove its process ownership settled cannot publish a success, on the same terms lifecycle settlement applies. A failed effect still retains bounded diagnostic evidence on its error: the safely collected prefix, the bounded channels, and the primary and secondary failure categories. No successful binding is not the same as no retained evidence, and replaying a failed effect starts no process. ### Terminal settlement follows its projections @@ -3499,12 +3505,14 @@ Status is measured against main. | `` | publishes one item's status to a Project provider — a boundary of its own, because a board owns neither a repository nor an issue collection — keyed by the exact item and field, against the option the item currently holds, inside the host's configured project, field and option ceiling | specified by #710; implementation unbuilt | | `` · `` | take one pull request out of draft, and close one unmerged, as Git-host effects keyed by their exact subject; readiness is authorized by an accepted review outcome rather than by observing the pull request | specified by #710; implementation unbuilt | | `` | closes one issue as `completed` or `not_planned`, as an Issue-provider effect keyed by its exact subject, with the reason a closed enum that must match the retained terminal intent | specified by #710; implementation unbuilt | -| `` | merges two exact commits inside the retained Workspace, observing and fixing both parents and the merge base first; a clean result publishes commit, root and filtered result in one effect transaction, a conflicted one restores the pre-merge root and publishes normalized conflict evidence, and the parent order is the caller's because synchronizing a target into an implementation and publishing an implementation onto a target are opposite operations | specified by #710; implementation unbuilt | +| `` | merges two exact commits inside the retained Workspace, observing and fixing both parents and the merge base first; a clean result publishes commit, root and filtered result in one effect transaction, and a conflicted one restores the pre-merge root and publishes normalized conflict evidence. The parent order is the caller's — synchronizing a target into an implementation and publishing an implementation onto a target are opposite operations — and `purpose` authorizes it rather than merely recording it: the provider-authenticated merge ceiling supplies the pair each purpose may carry, and a swapped, stale or cross-purpose parent refuses before any Git mutation | specified by #710; implementation unbuilt | | `` | updates a protected target ref by compare-and-swap: one non-force update after observing the target equal to the expected commit, adoption of a target already equal to the exact source commit, and refusal of everything else without mutation; remote, ref, credential and non-force policy are host-owned. Not a spelling of `Git.Push`, which advances a branch this run published from a proved ancestry relation | specified by #710; implementation unbuilt | -| `` | runs an authored structured argv list natively on the trusted runner against one exact retained Workspace root, under host-owned executable, environment, working-root, duration, output and process-tree ceilings; the whole list runs and binds one record per command carrying argv, how it ended, and separately bounded stdout and stderr that state their own truncation; a launch, output-pump or teardown failure binds nothing and cancellation commits nothing; absent from the workflow Agent's capabilities and from every generated-XMD write table, and a completed replay runs nothing | specified by #710; implementation unbuilt | -| `` | observes that a Git host now records one pull request as merged, at the exact commit a target publication published; a reconciled Git-host observation that mutates nothing, whose only completion is adoption, keyed by the canonical pull-request URL — a merge at another commit conflicts, a pull request still open or closed unmerged is temporary unavailability rather than absence, and publication does not imply it | specified by #710; implementation unbuilt | -| remote `WorkflowHost` implementation | keeps the existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — and adds a Cloudflare runtime-named implementation of it beside the Deno one; start, lookup, execute, deliver and inspect stay lifecycle operations reached through those four rather than becoming method names. One SQLite-backed Durable Object per run is selected from the public run ID, executor acquisition is an authenticated connection lifetime, and the runner-to-owner transport is a closed versioned envelope that refuses an unknown protocol version, operation or member rather than adapting to it | specified by #710; implementation unbuilt | -| factory protocol records | the closed versioned schemas `specs/github-actions-software-factory-spec.md` §11.2 defines for the subject, stage, implementation revision, handoff, actor, role outcome, invalidation, evidence reference, Planner and Architect verdicts, conflict suspension, Stage 7 decision, active frontier and the two terminal settlements one issue-driven run retains. Each carries a schema discriminant and a version, and an unknown schema, version, member or enum value refuses rather than being ignored — provider-neutral durable protocol, neither an XMD component nor a TypeScript lifecycle controller | specified by #710; implementation unbuilt | +| `` | runs an authored structured argv list natively on the trusted runner against one exact retained Workspace root, under host-owned executable, environment, working-root, per-command duration, whole-run duration, output and process-tree ceilings; a fail-fast pipeline that stops at the first command not exiting `0` and binds the executed prefix — how it completed, how many commands were authored, and one row per command that ran carrying argv, how it ended, which ceiling fired on a timeout, and separately bounded stdout and stderr that state their own truncation; a launch, output-pump or teardown failure binds no result while retaining bounded error evidence, and cancellation commits nothing; absent from the workflow Agent's capabilities and from every generated-XMD write table, and a completed replay runs nothing | specified by #710; implementation unbuilt | +| `` | observes that a Git host now records one pull request as merged, at the exact commit a target publication published; a reconciled Git-host observation that mutates nothing, whose only completion is adoption, keyed by the canonical pull-request URL — a merge at another commit conflicts, a pull request still open is temporary unavailability the run waits out under a bounded retry and then a durable machine wait, one closed unmerged conflicts, and publication does not imply any of it | specified by #710; implementation unbuilt | +| remote `WorkflowHost` implementation | keeps the existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — and adds a Cloudflare runtime-named implementation of it beside the Deno one; start, lookup, execute, deliver and inspect stay lifecycle operations reached through those four rather than becoming method names, and a remote host receives no transitions type of its own. One SQLite-backed Durable Object per run is selected from the public run ID, and executor acquisition is an authenticated connection lifetime | specified by #710; implementation unbuilt | +| provider-neutral lifecycle transition types | `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` describe what any host's lifecycle does rather than what one adapter retains, and become package-root public types; the Deno entrypoint may re-export them for source compatibility without owning their meaning, while runtime-specific implementations and retained encodings stay behind their runtime-named entrypoints | neutrality settled by #710; the export move is implementation work | +| same-release runner transport | the messages between an ephemeral runner client and its durable owner are private to one software-factory release: not journaled, exported, authored or supported across independently versioned builds. Connection admission validates an exact immutable build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before parsing, acquisition or state access; there is no cross-version adaptation or downgrade. Privacy of the transport is not privacy of the authority — the acquisition, expected-root validation, owner-side parsing and transactions, content validation, separate no-acquisition delivery and inspection paths, and provider-free completed replay stay public and exact | specified by #710; implementation unbuilt | +| factory protocol records | the closed versioned schemas `specs/github-actions-software-factory-spec.md` §11.2 defines — and normatively owns, every other document linking to it rather than restating it — for the subject, stage, implementation revision, handoff, actor, role outcome, invalidation, evidence reference, Planner and Architect verdicts, conflict suspension, Stage 7 decision, merged-observation wait, stage-to-option table, active frontier and the two terminal settlements one issue-driven run retains. Each carries a schema discriminant and a version, and an unknown schema, version, member or enum value refuses rather than being ignored — provider-neutral durable protocol, neither an XMD component nor a TypeScript lifecycle controller | specified by #710; implementation unbuilt | | workflow scheduling (watchers, unattended iteration, remote host selection) | — | #300 | | `` | binds `{ok: true, value}` or `{ok: false, error}`; a failure becomes a bound value, not a raise | defined, unbuilt | | error middleware (JS api) | retry · suspend · decline | defined, unbuilt | diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index a441a041c..5ec0a8240 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -7525,11 +7525,11 @@ engine-derived from the run and the expansion and is never a prop. | `Project.Status` | `` — binds the normalized `{ item, field, option }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.6 | | `Git.Merge` | ``, or the same form with `purpose="publish"` — binds `{ outcome: "clean", purpose, firstParent, secondParent, mergeBase, commit, workspaceRoot }` or `{ outcome: "conflicted", purpose, firstParent, secondParent, mergeBase, workspaceRoot, conflicts }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.8 | | `Git.PublishTarget` | `` — binds `{ target, expectedRemoteCommit, reviewedHead, sourceCommit, observedCommit, decision }` | [Workflow workspaces](./workflow-workspace-spec.md) §7.9 | -| `Evidence.Run` | ``, where `commands` is an ordered non-empty list of non-empty argv vectors — binds `{ commands: [{ argv, outcome, status?, signal?, stdout, stderr }] }` in authored order, each channel `{ text, retainedBytes, producedBytes, truncated }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.5 | +| `Evidence.Run` | ``, where `commands` is an ordered non-empty list of non-empty argv vectors — binds `{ completion, authoredCommands, executed, runTimeout? }`, each executed row `{ argv, outcome, status?, signal?, limit?, stdout, stderr }` and each channel `{ text, retainedBytes, producedBytes, truncated }` | [Workflow workspaces](./workflow-workspace-spec.md) §10.5 | Every binding above is the exact closed record its contract section defines; a member outside those shapes, an unknown member and a value outside a closed enum each refuse rather than being carried. -Two things a reader looking for a component will not find here. The remote `WorkflowHost` is a host assembly contract — the existing `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()`, with a Cloudflare implementation beside the Deno one — and not an element a document writes; its runner-to-owner transport is the versioned envelope of [Workflow workspaces](./workflow-workspace-spec.md) §13.2. The factory's protocol records — subject, stage, implementation revision, handoff, role outcome, invalidation, verdict, conflict suspension, Stage 7 decision, active frontier and terminal settlement — are the closed versioned schemas of [the software factory](./github-actions-software-factory-spec.md) §11.2, not components and not a lifecycle controller beside the journal. +Two things a reader looking for a component will not find here. The remote `WorkflowHost` is a host assembly contract — the existing `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()`, with a Cloudflare implementation beside the Deno one and no transitions type of its own — and not an element a document writes; its runner-to-owner messages are private to one release rather than a public wire contract ([Workflow workspaces](./workflow-workspace-spec.md) §13.2). The factory's protocol records — subject, stage, implementation revision, handoff, role outcome, invalidation, verdict, conflict suspension, Stage 7 decision, merged-observation wait, stage-to-option table, active frontier and terminal settlement — are the closed versioned schemas of [the software factory](./github-actions-software-factory-spec.md) §11.2, which owns them; nothing here or in the Workspace specification duplicates them, and they are neither components nor a lifecycle controller beside the journal. None of these constructs is available to a workflow Agent, and none appears in any generated-XMD read or write table @@ -10071,8 +10071,9 @@ and [Workflow runs](./workflow-spec.md) §9.8. | WRH11 | Teardown | Closing the connection releases executor ownership and rolls back nothing already committed | | WRH12 | Completed replay | A completed run replays attaching no remote storage session, Workspace, Agent, process, Git, Git-host, Issue, Project or credential provider | | WRH13 | Host neutrality | Shared WorkflowRun modules import nothing Cloudflare-specific and detect no runtime; the runtime-named entrypoint is the only place the topology appears | -| WRH14 | The host boundary is unchanged | The Cloudflare adapter satisfies the existing `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()`; no fifth method appears, and the shared CLI asks the same four questions it asks the Deno host | -| WRH15 | Versioned transport | A `RunnerRequest` whose protocol version this owner does not implement is refused before any other member is read; an unknown operation, an unknown member, and an acquisition present on `acquire` or absent on the other four each refuse without effect, and neither side adapts to the other's shape | +| WRH14 | The host boundary is unchanged | The Cloudflare adapter satisfies the existing `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()` with the same provider-neutral transition and request types; no fifth method and no adapter-specific transitions type appears, and the shared CLI asks the same four questions it asks the Deno host | +| WRH17 | Waking a machine wait | An authenticated intake for a merged-observation wait's exact subject is retained without an executor acquisition, appends no outcome and moves no stage; a later executor reobserves and only an adoption advances the run | +| WRH15 | Release identity | Connection admission validates an exact immutable runner and owner build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before request parsing, acquisition or state access; no message shape is adapted, downgraded or negotiated, and no transport record is journaled, exported or authored | | WRH16 | Ownership split | Connection admission, request parsing, transaction lifetime and stale recovery belong to the owner; provider attachment and cancellation of its own execution belong to the runner; content is produced by the runner and validated by the owner | ### Tier WGI — Authenticated GitHub ingress @@ -10087,6 +10088,7 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) §5 | WGI4 | Duplicates | A repeated delivery of one identity finds the retained intake and writes nothing | | WGI5 | Unavailable is not absent | A missing result page, an unavailable field, an ambiguous object and a partial permission read are each unavailable, and none of them admits an item | | WGI6 | Admission ceiling | Admission requires the configured organization-owned Project and the exact repository, Project, item, status field and allowed option IDs | +| WGI6b | Stage mapping | Admission maps a completely reread option ID to a stage through the configured bijection and projection maps a stage back through its inverse; neither parses a display string, an invalid table refuses before an intake is retained, a token is minted, a run starts or anything is projected, and only the configured `Backlog`-to-`User` movement admits a new item | | WGI7 | Dispatch carries nothing | `repository_dispatch` carries only the retained intake identity; a payload naming a stage, outcome, answer, decision, transition, credential or definition is refused | | WGI8 | OIDC claims | Admission validates issuer, audience, repository ID, repository-owner ID, event name, workflow ref and SHA, and the configured workflow identity; a valid token for another repository or workflow admits no session | | WGI9 | Human floor | An actor without Project write and repository write-or-higher authorizes no admission, answer, change, merge or abandonment | @@ -10103,8 +10105,9 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §7.10, §10.3 an |---|------|--------| | WGE1 | Comment identity | A comment's natural key is its subject plus the engine-derived effect identity; a re-rendered or edited body is the same comment and produces no second one | | WGE2 | Ready and close | `PullRequest.Ready` and `PullRequest.Close` are keyed by their exact subject; an already-ready pull request is adopted, a merged one conflicts for both, and each binds its literal closed record rather than an observed one | -| WGE2b | Comment identity is observable | An interrupted comment creation is found again by the adapter's fixed presentation-neutral trailer alone; exactly one match adopts, none performs once, more than one is ambiguity, and a comment list the adapter could not finish reading is an incomplete observation rather than absence | -| WGE2c | Merged observation | `PullRequest.Merged` mutates nothing and only adopts: a host reporting merged at the exact published commit is the completion, a merge at another commit conflicts, and a pull request still open or closed unmerged is temporary unavailability rather than absence | +| WGE2b | Comment correlation | A provider that cannot write, preserve and completely query a stable opaque marker refuses before its first mutation; the authored logical body is sent byte for byte and the correlation representation rides outside it; the binding and every replay expose the body and the provider comment identity, never the encoding | +| WGE2c | Attempt state decides absence | Unattempted with no marker is proven absence and creates once; exactly one marker adopts; more than one is ambiguity; **attempted with no committed completion and no marker is permanent ambiguity, not absence**, so a removed marker inside the interrupted window stalls rather than duplicating; a marker removed after a committed completion changes nothing, because replay contacts no provider | +| WGE2d | Merged observation | `PullRequest.Merged` mutates nothing and only adopts: merged at the exact published commit is the completion, merged at another commit conflicts, still open is temporary unavailability, and closed unmerged conflicts rather than being waited out | | WGE3 | Issue closure | `Issue.Close` adopts an issue already closed with the same reason and conflicts with one closed under the other reason | | WGE4 | Project pre-state | `Project.Status` is keyed by exact item plus field, adopts an item already at the requested option, and performs once from another allowed option | | WGE5 | Project unavailability | An unreadable board, unavailable field, ambiguous item and partial permission read are unavailable rather than absent, and none of them mutates | @@ -10125,7 +10128,9 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §7.8 and §7.9. |---|------|--------| | WGM1 | Clean merge | A clean `Git.Merge` publishes the commit, the new Workspace root and the filtered result in one transaction | | WGM2 | Conflict | A conflicted merge restores the pre-merge root, publishes normalized conflict evidence against it, and offers no mutation under that evidence | -| WGM3 | Parent order | `purpose="synchronize"` retains `[implementationHead, targetBase]` and `purpose="publish"` retains `[reviewedBase, reviewedHead]`; neither purpose reorders what it was given | +| WGM3 | Parent order | `purpose="synchronize"` carries `[implementationHead, targetBase]` and `purpose="publish"` carries `[reviewedBase, reviewedHead]`; neither purpose reorders what it was given | +| WGM3b | `purpose` authorizes | Each purpose's authored parents and merge base are validated against the provider-authenticated merge ceiling — the current implementation head and observed target base for a synchronization, the Stage 7 decision's reviewed `{ headSha, baseSha }` for a publication — and both correct orders pass | +| WGM3c | What refuses before mutation | A missing ceiling, a purpose the ceiling does not authorize, a swapped parent, a stale parent, a stale merge base, another revision, and a ceiling for another Repository or checkout each refuse before any Git mutation | | WGM4 | Stale identities | A changed head, base, merge base, Repository identity, conflict set or Workspace root makes a retained conflict admission stale, and nothing mutates under it | | WGM5 | Interruption | A host killed between the merge and the commit leaves the checkout, the current root and the effect history unchanged | | WGM6 | Compare-and-swap | `Git.PublishTarget` updates the ref only after observing the target equal to the expected commit | @@ -10152,11 +10157,15 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §10.5. | WER5 | Location | The execution happens on the trusted runner; the durable owner runs no native process | | WER6 | Cancellation and teardown | A cancelled run terminates its child before publishing, and no child outlives the effect | | WER7 | Replay | A completed record replays running nothing | -| WER4b | The whole list runs | A command ending unsuccessfully does not stop the ones after it; the bound list has one entry per authored command, in authored order, and each command is charged its own duration ceiling | +| WER4b | Fail-fast prefix | A command exiting `0` starts its successor; a non-zero exit, a signal and a timeout each become the last row and start none. `completion` is `"passed"` only when `executed` holds `authoredCommands` rows that all exited `0`, and `"failed"` otherwise, so a complete pass and a stopped prefix are distinguishable without the authored list | | WER4c | Channels and truncation | stdout and stderr are retained separately, each stating its retained bytes, the bytes the child produced, and whether it was truncated; truncation is stated rather than inferred | -| WER4d | Outcome discriminants | `status` is present exactly for `exited`, `signal` exactly for `signalled`, and `timeout` carries neither; either member beside the wrong outcome refuses the record | -| WER6b | What binds and what fails | Any exit status, a signal and a timeout each bind an ordinary result; a launch failure, an output-pump failure and a teardown failure each fail the effect and bind nothing; cancellation commits neither a result nor a failure | -| WER6c | Precedence | Cancellation outranks everything, the first infrastructure failure otherwise wins, and a teardown failure fails the effect even when every command produced an ordinary outcome | +| WER4d | Outcome discriminants | `status` is present exactly for `exited`, `signal` exactly for `signalled`, and `limit` exactly for `timeout`; any of the three beside the wrong outcome, an unknown outcome and an unknown `limit` each refuse the record | +| WER4e | Two ceilings | A per-command ceiling and a whole-run ceiling are both host-owned and neither is a prop; a running command's effective deadline is the earlier of the two, and the timeout row's `limit` names which fired | +| WER4f | Whole-run expiry between commands | The result ends `completion: "failed"` with a `runTimeout` record naming the index that did not start, no next command runs, and no fabricated argv row appears | +| WER6b | What binds and what fails | A zero exit, a non-zero exit, a signal, a per-command timeout and a whole-run timeout each belong to an ordinary bound result; a launch failure, an output-pump failure and a teardown failure each fail the effect and bind no `EvidenceRunResult`; cancellation terminates the complete process tree and commits neither a completion nor a failure | +| WER6c | Precedence | Cancellation outranks everything; otherwise the first infrastructure failure is authoritative and a teardown failure after it is retained as secondary evidence rather than replacing it; a teardown failure with nothing before it is authoritative even when every command produced an observed exit | +| WER6d | Failure evidence is retained | A failed effect's error carries the safely collected executed prefix, the separately bounded channels, the primary infrastructure category and any secondary teardown category; nothing binds it, it is not an `EvidenceRunResult`, replaying the failed effect starts no process, and cancellation retains neither | +| WER6e | Enforcement that fails | A termination, drain or reap that fails while the host is enforcing a ceiling is an infrastructure failure rather than a timeout row | | WER8 | Denied authority | `Evidence.Run` appears in no Agent capability and in no generated-XMD read or write table; a fragment naming it is refused before any generated effect | ### Tier WFP — Factory protocol and frontier @@ -10178,6 +10187,9 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) | WFP9 | Closed parsers | Every §11.2 record carries its schema discriminant and version; an unknown schema, an unknown version, an unknown member, a missing required member and a value outside a closed enum each refuse, and a refusal names the member path and never the value | | WFP9b | Frontier reduction | Each reduction input of §11.2 — advance, amend, invalidate, head change, base-only change, base-needs-work, and a terminal decision on an already-terminal run — produces exactly the stated frontier, and a reduction leaving a stage out of range, two entries for one stage, or an advance past an unaccepted stage refuses | | WFP9c | Decision shapes | `merge` carries revision, actor and delivery identity; `abandon` adds a required reason; `change` adds a reason and the earliest invalidated stage; a decision naming a revision that is not the current frontier revision refuses | +| WFP9d | Stage-to-option table | The retained table holds exactly nine entries ordered by stage, one option ID per stage `0`-`8`, every option ID distinct, and every display name equal to the settled status string for its stage; missing, partial, duplicated, cross-field, unavailable and renamed cases each refuse | +| WFP9e | Merged-observation wait | The wait record names the canonical pull-request URL, expected merge commit, terminal decision, revision and suspension identity, carries `retriesExhausted: true`, and holds no stage, outcome or verdict; consuming it authorizes one reobservation | +| WFP9f | Terminal references | Every event a terminal names must belong to this run and intent, parse under its exact effect kind, be complete, and agree on revision, actor and decision; missing, foreign, wrong-kind, incomplete, invalidated, duplicated and cross-path events each refuse settlement, and provider identities stay in the referenced results rather than being copied into the terminal | | WFP10 | Definition identity | Definition incompatibility is a workflow lifecycle refusal and never an implementation correction | | WFP11 | Journal is authority | A Project status, comment or pull-request state ahead of the journal is drift and authorizes no stage | @@ -10195,6 +10207,7 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) | WFL5 | Merge decision first | A merged decision is retained before any effect is attempted | | WFL6 | Merge ordering | Merge construction, target publication, `PullRequest.Merged`, issue completion as `completed`, the Project move and terminal settlement are separate reconciled steps in that order | | WFL6b | Paths do not borrow steps | The merged path closes no pull request and the abandoned path constructs no merge, publishes no target and observes no merged state; a `FactoryTerminal` naming a step from the other path refuses | +| WFL6c | Waiting for the host | A pull request still open runs the bounded host-configured retry and then suspends on the merged-observation wait; the wait appends no outcome and moves no stage, an intake or an explicit resume only permits reobservation, and terminal settlement stays absent until the adoption succeeds. A conflict observed on any attempt ends the wait as a conflict | | WFL7 | Terminal is last | The run becomes terminal `merged` only after every required projection completes | | WFL8 | Abandonment | An exact-revision authenticated abandonment carrying a reason is retained first, then pull-request close unmerged, issue close as `not_planned`, the Project move and settlement; terminal `abandoned` follows all of them | | WFL9 | Retention | Both terminal kinds retain the actor, exact revision, resulting provider identities and required reason, and retain the Project item, branch, comments, journal, Workspace roots and Agent evidence | diff --git a/specs/github-actions-software-factory-spec.md b/specs/github-actions-software-factory-spec.md index 1940d96b7..70765d19a 100644 --- a/specs/github-actions-software-factory-spec.md +++ b/specs/github-actions-software-factory-spec.md @@ -374,6 +374,14 @@ permission read are all **unavailable** — they are neither absence nor authorization. Treating an unreadable Project as an empty one is how an unauthorized item would be admitted. +**A configured table maps status options to stages, in both directions.** It is a total bijection between the nine `FactoryStage` values of §11.2 and nine exact Project V2 status option IDs: one option ID for every stage `0` through `8`, and every configured option ID appearing exactly once. It is host configuration and part of the admission and projection ceiling — never an authored prop, and never something a provider payload can supply. + +The table is validated before it is used. Startup and admission refuse — before an intake is retained, a token is minted, a run is started, or anything is projected — when the table is missing, partial, holds a duplicate, names an option the Project does not currently offer, names an option belonging to another field or another Project, or names an option whose display name reread from GitHub is not the settled §2 status string for its stage. + +Admission maps the completely reread option ID to a stage through that table, and projection maps a retained stage back to its option ID through the inverse. Neither direction parses a display string: the strings are what a person reads, the option IDs are what the factory compares, and a status renamed on the board is a configuration refusal rather than a silent remapping. + +Only the configured `Backlog`-to-`User` movement admits a new item. A Project edit at any other point is projection drift or an authenticated intake to reconcile, and is never a role verdict. + A duplicate delivery of the same identity finds the retained intake and changes nothing. That is the same compatible-reuse rule run identity uses, applied to intake. @@ -425,6 +433,8 @@ branch, organization Project, status field, option IDs, issue or pull-request subject, reviewed revision, parent pair, and non-force operation. A ceiling the installation grants is not a ceiling the host uses. +Three more ceilings are host configuration on the same terms, none of them an authored prop: the stage-to-option bijection of §5.1; the merge authority `Git.Merge` validates its authored parents against, which supplies the current implementation head and observed target base for a synchronization and the reviewed `{ headSha, baseSha }` the Stage 7 decision authorized for a publication; and the bounded retry — count, total duration and backoff — the merged observation of §10.4 runs under. `Evidence.Run`'s executable, environment, working-root, per-command duration, whole-run duration, output and process-tree ceilings are the same kind of configuration, stated in [Workflow workspaces](./workflow-workspace-spec.md) §10.5. + Mutation of `.github/workflows/**` is refused, even though Contents write could otherwise reach it. A factory that can rewrite the workflow that runs it is a factory that can rewrite its own authorization. @@ -609,7 +619,7 @@ There are two terminal paths and they share no step list. Each begins by retaini 1. Retain the `merge` decision of §11.2, bound to the exact reviewed revision and the authenticated actor. 2. Construct the trusted merge commit — `` with parents `[reviewedBase, reviewedHead]`. 3. Publish the target — `` under §10.2. -4. Observe that the pull request merged — ``, the reconciled Git-host observation of [Workflow workspaces](./workflow-workspace-spec.md) §7.11, against the published merge commit. A Git host records a pull request as merged on its own schedule, so this is its own retained step and not something publication implies. +4. Observe that the pull request merged — ``, the reconciled Git-host observation of [Workflow workspaces](./workflow-workspace-spec.md) §7.11, against the published merge commit. A Git host records a pull request as merged on its own schedule, so this is its own retained step and not something publication implies. §10.4 says what the run does while it has not caught up. 5. Close the issue — ``. 6. Move the Project item to `Closed` — ``. 7. Publish terminal kind `merged`. @@ -630,6 +640,18 @@ Terminal settlement is last on both paths for one reason: a completed run replay Both terminal kinds retain the authorizing actor, the exact reviewed revision, the resulting provider identities, and the reason where one is required, in the `FactoryTerminal` record of §11.2 — whose two shapes differ exactly as these two paths do. Both retain the Project item, the implementation branch, the issue and pull-request comments, the journal, the Workspace roots and the Agent evidence. Reopening the issue afterwards does not reopen the completed run; continuing that work requires a new linked issue, and therefore a new run. +### 10.4 Waiting for the host to notice + +A pull request observed still open after a successful target publication is temporary unavailability, not absence and not refusal — the Git host has not yet recognized its own ref moving. The factory waits for it in two stages, and neither is a human decision. + +The authored factory first performs a **bounded retry** around `PullRequest.Merged`. The retry count, total duration and backoff are host ceilings configured for the deployment; they are not props on the component and no document widens them. + +When that retry is exhausted while the pull request is still open, the run enters a **durable machine wait** at Stage 7. It is a suspension in the ordinary sense — the executor settles `suspended` and releases its acquisition — but its subject is a fact about a provider rather than a question for a person, so it is neither an answer nor a Stage 7 decision, and no form is bound to it. Its exact subject is the canonical pull-request URL, the expected merge commit, the retained terminal decision identity and the current implementation revision, retained as the `MergedObservationWait` record of §11.2. + +Two things may wake it. An authenticated intake for a relevant pull-request state change on that exact subject is retained as a delivery-plane transaction under §5, taking no executor acquisition; an operator may also resume the run explicitly. Neither wakeup carries a verdict, a stage or a transition — waking is permission to look again, not an answer. A later executor reobserves `PullRequest.Merged` and proceeds only on the adoption of §7.11's first row. + +The wait is not a second lifecycle controller. It appends no outcome, moves no stage, and terminal settlement stays absent until the observation adopts. A conflict observed on any attempt — a merge at another commit, or a pull request closed unmerged — ends the wait as a conflict rather than continuing it. + ## 11. Exact public contracts ### 11.1 Authored construct inventory @@ -649,9 +671,9 @@ expansion; it is never a document prop. | `Issue.Close` | Self-closing ``, or the same form with `reason="not_planned"`; binds the normalized URL, state and reason | Issue-provider effect; `reason` is a closed enum and must match the retained terminal intent | | `Git.Merge` | Self-closing ``, or the same form with `purpose="publish"`; binds the `GitMergeResult` union of [Workflow workspaces](./workflow-workspace-spec.md) §7.8 — `{ outcome: "clean", purpose, firstParent, secondParent, mergeBase, commit, workspaceRoot }` or `{ outcome: "conflicted", purpose, firstParent, secondParent, mergeBase, workspaceRoot, conflicts }` | Workspace-local Git effect; repository, checkout, root and acquisition are authenticated provider state; a clean publication is atomic and a conflict restores before the result is published | | `Git.PublishTarget` | Self-closing ``; binds normalized target, expected and published evidence | Git-host effect; remote, ref, credential and non-force ceiling are host-owned; exact compare-and-swap reconciliation | -| `Evidence.Run` | Self-closing ``, where `commands` is an ordered non-empty list of non-empty argv vectors; binds the `EvidenceRunResult` of [Workflow workspaces](./workflow-workspace-spec.md) §10.5 — one `{ argv, outcome, status?, signal?, stdout, stderr }` per command, in authored order, each channel `{ text, retainedBytes, producedBytes, truncated }` | Trusted runner-host effect; the exact retained root and the executable, environment, working-root, duration, output and process-tree ceilings; the whole list runs, launch/output-pump/teardown failures bind nothing, cancellation commits nothing; absent from Agent and generated-XMD capabilities; a completed replay runs nothing | +| `Evidence.Run` | Self-closing ``, where `commands` is an ordered non-empty list of non-empty argv vectors; binds the `EvidenceRunResult` of [Workflow workspaces](./workflow-workspace-spec.md) §10.5 — `{ completion, authoredCommands, executed, runTimeout? }`, where each executed row is `{ argv, outcome, status?, signal?, limit?, stdout, stderr }` and each channel is `{ text, retainedBytes, producedBytes, truncated }` | Trusted runner-host effect; the exact retained root and the executable, environment, working-root, per-command duration, whole-run duration, output and process-tree ceilings; a fail-fast pipeline binding the executed prefix, launch/output-pump/teardown failures binding no result and retaining bounded error evidence, cancellation committing nothing; absent from Agent and generated-XMD capabilities; a completed replay runs nothing | | `PullRequest.Merged` | Self-closing ``; binds `{ subject, state: "closed", merged: true, mergeCommit, decision: "adopted" }` | Git-host reconciled observation, [Workflow workspaces](./workflow-workspace-spec.md) §7.11; it mutates nothing and adoption is its only completion; keyed by the canonical pull-request URL | -| Remote `WorkflowHost` | The existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — with a Cloudflare runtime-named implementation beside the Deno one; start, lookup, execute, deliver and inspect are lifecycle operations reached through it rather than method names of their own. The runner-to-owner transport is the versioned `RunnerRequest` envelope of [Workflow workspaces](./workflow-workspace-spec.md) §13.2 | A host assembly contract rather than an XMD component; the execution, delivery and inspection planes stay distinct across it | +| Remote `WorkflowHost` | The existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — with a Cloudflare runtime-named implementation beside the Deno one; start, lookup, execute, deliver and inspect are lifecycle operations reached through it rather than method names of their own, and a remote host receives no transitions type of its own. Its transition and request types are provider-neutral and become package-root public types; the runner-to-owner messages are private to one release, admitted by an exact build fingerprint ([Workflow workspaces](./workflow-workspace-spec.md) §13.2) | A host assembly contract rather than an XMD component; the execution, delivery and inspection planes stay distinct across it | | Factory protocol records | The closed versioned schemas of §11.2 | A provider-neutral durable protocol; neither an XMD component nor a TypeScript lifecycle controller | Each construct's closed props, form, binding, request, natural key, compatible pre-state, normalized result, refusal and unavailability behavior, cancellation, replay, provider ownership, credential boundary, and whether it is Workspace-local or an external reconciled effect are defined normatively in [Workflow workspaces](./workflow-workspace-spec.md): §7.8 `Git.Merge`, §7.9 `Git.PublishTarget`, §7.10 `PullRequest.Comment`, `PullRequest.Ready` and `PullRequest.Close`, §7.11 `PullRequest.Merged`, §10.3 `Issue.Comment` and `Issue.Close`, §10.5 `Evidence.Run`, §10.6 `Project.Status`, §10.7 the credential boundary they share, and §13.2 the remote host and its transport. A later implementation may choose ordinary private function and module names; it may not change these public forms, their records or their ownership. @@ -771,6 +793,43 @@ type Stage7Decision = All three bind the exact revision they were made against and the authenticated actor who made them, and all three name the delivery identity they arrived under. `abandon` requires a reason and `change` requires both a reason and the earliest stage it invalidates; `merge` takes neither, because approving what two reviews already passed adds no new claim. A decision whose `revision` is not the current frontier revision refuses. `merge` and `abandon` are terminal intents that §10.3 orders; `change` is not terminal and reduces to an invalidation. +#### Waiting for the merged observation + +```ts +interface MergedObservationWait { + readonly schema: "merged-observation-wait"; + readonly version: 1; + readonly subject: string; + readonly expectedMergeCommit: string; + readonly decisionId: string; + readonly revision: ImplementationRevision; + readonly suspensionId: string; + readonly retriesExhausted: true; +} +``` + +`subject` is the canonical pull-request URL, `decisionId` names the retained `merge` decision this wait belongs to, and `suspensionId` is the durable wait's own identity. `retriesExhausted` is literal: the record exists only after §10.4's bounded retry has run out, so a wait retained before that would be a run skipping the cheap path. The wait carries no stage, no outcome and no verdict, and consuming it authorizes one reobservation and nothing else. + +#### The configured stage-to-option table + +```ts +interface StageOptionTable { + readonly schema: "stage-option-table"; + readonly version: 1; + readonly projectId: string; + readonly statusFieldId: string; + readonly options: readonly StageOption[]; +} + +interface StageOption { + readonly stage: FactoryStage; + readonly optionId: string; + readonly displayName: string; +} +``` + +`options` holds exactly nine entries, one per stage `0` through `8`, ordered by `stage`. Every `optionId` is distinct, and every `displayName` equals the §2 status string for its stage. This is the retained form of §5.1's configuration: it is validated against a complete reread of the Project before it is used, and a table that does not satisfy every one of those conditions refuses rather than being partially applied. + #### The active frontier and its reduction ```ts @@ -831,7 +890,11 @@ type FactoryTerminal = }; ``` -The two kinds do not share a member list, and that asymmetry is the contract: a `merged` terminal names a merge commit, a target publication and a merged observation, and an `abandoned` terminal names a pull-request closure and a reason and can name none of the first three. Every member ending in `Id` or naming a step is a journal event identity, so the terminal record points at the reconciled effects that completed rather than restating their results. A terminal record whose named events are not all present and complete refuses, which is what makes §10.3's ordering checkable from the record alone. +The two kinds do not share a member list, and that asymmetry is the contract: a `merged` terminal names a merge commit, a target publication and a merged observation, and an `abandoned` terminal names a pull-request closure and a reason and can name none of the first three. + +Every member ending in `Id` or naming a step is a **journal event identity**, and it stays one. The terminal record points at the reconciled effects that completed rather than restating their results, so each provider identity keeps one durable source — the effect result that observed it — and settlement ordering stays checkable against the journal instead of against a copy that could disagree with it. + +Validation follows those references rather than copying through them. Every referenced event must belong to this run and this active terminal intent, parse under its exact effect kind, be complete, and agree with the terminal record's revision, actor and decision where each applies; the referenced results carry the normalized provider identities, and terminal validation checks compatibility without duplicating them. A missing, foreign, wrong-kind, incomplete, invalidated, duplicated or cross-path event refuses settlement — and cross-path is exact: a `merged` terminal cannot name a pull-request close-unmerged or any abandonment effect, and an `abandoned` terminal cannot name a merge construction, a target publication or a merged observation. That is what makes §10.3's two orders checkable from the record alone. ## 12. Structural consequences @@ -929,7 +992,8 @@ A factory implementation satisfies this specification when every item holds: only the exact merge commit, and never force-updates. 21. The merged state of the pull request is observed as its own retained step after publication and before issue closure, and is adopted only at the exact - published merge commit. + published merge commit. A pull request still open is a bounded retry and + then a durable machine wait; one closed unmerged is a conflict. 22. The merged and abandoned paths of §10.3 run in their stated orders, and neither borrows a step from the other. 23. A terminal `merged` or `abandoned` state is published only after every diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 85ba99555..d8d77a470 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -340,7 +340,7 @@ behavior appear. Shared modules import none of them and detect no runtime. A second host installs its own the same way. Cloudflare is a runtime-named adapter beside the Deno one, not a second contract: it answers `create()` and `lookup()` with a `WorkflowRunDatabase` of its own, and every shared -WorkflowRun surface above it stays host-neutral. Nothing below changes for it — +WorkflowRun surface above it stays host-neutral. The lifecycle transition and request types a host assembly speaks — `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` — are part of that neutral surface and are published from the package root; a runtime-named entrypoint may re-export them for source compatibility, but what belongs behind one is the implementation and its retained encoding, not the shape of the request. Nothing below changes for it — immutable run identity is compared the same way, recognition stays strict, events reach storage already filtered, a caller still owns the transaction it opened, and a completed run still replays without attaching a provider. What a @@ -433,6 +433,8 @@ The software factory derives its ids that way. A factory run id is the lowercase Those two inputs are the ones [the software factory](./github-actions-software-factory-spec.md) §1.1 defines, byte for byte, and this paragraph restates rather than generalizes them: the authority is the lowercase DNS hostname plus a non-default port, with no scheme, path, query, fragment, user information or trailing separator, and the node ID is GitHub's exact returned string with no case folding and no Unicode normalization. There is no broader Issue-provider authority in this hash — a hash whose inputs two documents spell differently is two hashes. +That specification also owns every other factory protocol record: its §11.2 holds the closed versioned schemas, and no other document restates them. + Because every input is immutable, admitting one issue twice derives one id and reaches one run through ordinary compatible reuse, and no separate idempotency concept appears. Two independent implementations given the same authority and node ID therefore produce the same id. A changed authority or node ID for a subject the host already retains is unsupported provider-identity drift, refused under §1.1 of that specification rather than derived into a second run. ### 9.2 Creating a run is also how it is found diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 6bf7da850..c0656ae37 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -1665,15 +1665,13 @@ authenticated provider state rather than props. Form validation runs first: a missing prop, an unknown prop, a `purpose` outside the enum and a missing `as` each fail before a Repository is observed or a provider is reached. -**The parent order is the caller's, and the two purposes are opposite -operations.** `purpose="synchronize"` brings a target into an implementation and -is authored `[implementationHead, targetBase]`. `purpose="publish"` brings a -reviewed implementation onto a target and is authored `[reviewedBase, -reviewedHead]`. The component does not infer the order from the purpose and does -not reorder what it was given; the purpose is retained so the record says which -operation the run believed it was performing. +**The parent order is the caller's, and `purpose` authorizes it.** `purpose="synchronize"` brings a target into an implementation and is authored `[implementationHead, targetBase]`. `purpose="publish"` brings a reviewed implementation onto a target and is authored `[reviewedBase, reviewedHead]`. The component neither infers the order from the purpose nor reorders what it was given — but it does not merely record the purpose either. It validates that what was authored equals the retained authority for the purpose declared. -**The request is the four authored inputs plus what the provider authenticates.** The durable request carries `firstParent`, `secondParent`, `mergeBase` and `purpose` exactly as authored, together with the Repository identity, the checkout, the pre-merge Workspace root and the executor acquisition the provider validated. Those four are provider-authenticated state and never authored props, and the effect is named by the run and the expansion like every other one. +The provider-authenticated merge ceiling supplies that authority. For `purpose="synchronize"` it supplies the exact current implementation head and the observed target base, and `firstParent` must equal that head while `secondParent` must equal that base. For `purpose="publish"` it supplies the exact reviewed `{ headSha, baseSha }` the retained Stage 7 decision authorized, and `firstParent` must equal `baseSha` while `secondParent` must equal `headSha`. In both cases `mergeBase` must equal the completely observed merge base for that same authenticated pair. + +A missing ceiling, a purpose the ceiling does not authorize, a swapped parent, a stale parent, a stale merge base, a revision other than the authorized one, and a ceiling belonging to another Repository or checkout each refuse before any Git mutation. Retaining the purpose without checking it would leave the one mistake this contract most needs to catch — a Stage 7 publication authored in Stage 4's order — detectable only by reading the history afterwards. + +**The request is the four authored inputs plus what the provider authenticates.** The durable request carries `firstParent`, `secondParent`, `mergeBase` and `purpose` exactly as authored, together with the Repository identity, the checkout, the pre-merge Workspace root, the executor acquisition and the merge ceiling the provider validated. Those are provider-authenticated state and never authored props, and the effect is named by the run and the expansion like every other one. **The result is a closed discriminated union of exactly two shapes**, keyed on `outcome`: @@ -1839,9 +1837,24 @@ interface PullRequestCommentResult { `subject` is the canonical pull-request URL, `effect` is the engine-derived effect identity of [Workflow runs](./workflow-spec.md) §8, and `url` is the canonical URL of the comment this effect settled on. The binding is `{ url }`: that comment's own URL, which is the only fact the effect produces — the subject was already in hand at the call site. -**The natural key is `subject` plus `effect`, and the body is never part of it.** A Git host issues no client-supplied idempotency key for a comment, so the effect identity has to be observable on the host for an interrupted creation to be found again. The adapter makes it observable by appending one fixed, presentation-neutral trailer to the body it sends — an HTML comment carrying the effect identity and nothing else — and observation matches that trailer alone. The trailer is adapter-owned rather than authored: a document does not write it, the rendered body it wrote is not searched for it, and a person editing the comment's prose afterwards leaves the identity untouched. That is what makes an edited sentence the same comment and a re-rendered body not a second one. +**The natural key is `subject` plus `effect`, and the body is never part of it.** A Git host issues no client-supplied idempotency key for a comment, so the effect identity has to be observable on the host for an interrupted creation to be found again. A comment provider therefore has to support one **stable opaque correlation marker**: a value it can write with a comment, preserve unchanged, and query completely. A provider that cannot do all three refuses the effect before its first mutation, the way a plain Git server refuses pull requests today — there is no fallback that searches prose. + +The marker is provider transport metadata. The **authored logical body is byte for byte what the document rendered**, and the provider's projection may carry the correlation representation outside it; GitHub's adapter encodes it as a non-rendered HTML comment. It is not authored prose, not a credential and not lifecycle authority — publishing an engine-derived effect identity as an opaque non-secret correlation value is what it is for. The public binding and every replay expose the authored body and the provider's comment identity, never the transport encoding. + +**Observation is judged against the attempt state, not against the host alone.** Before any provider mutation, the durable effect retains that this exact request is prepared and unattempted; a live attempt is what moves it past that. What a complete observation means then depends on which side of that line the effect is on: + +| Attempt state and observation | Decision | +| --- | --- | +| unattempted, and no marker | proven absence; create once | +| unattempted or attempted, and exactly one marker | compatible completion; adopt with nothing performed | +| any state, and more than one marker | permanent ambiguity; refuse | +| **attempted with no committed local completion, and no marker** | **permanent ambiguity; refuse** | +| an observation that did not complete | incomplete observation; refuse, adopt nothing, perform nothing | +| the host is temporarily unreachable | temporary unavailability; refuse as itself | + +The fourth row is the one that matters. A marker that is absent *after* an attempt does not prove the comment was never created — it equally describes a person having edited or deleted it inside the interrupted window — so treating that as absence is how a duplicate gets published. Refusing it as ambiguity costs a stall and buys the guarantee. Once a local completion has committed, the marker no longer decides anything: a completed replay reads its own record and contacts no provider, so removing the marker afterwards changes nothing. -Observation is complete or it is not an observation: a comment list the adapter could not finish reading is an incomplete observation and refuses, because an unfinished search would report absence and create a duplicate. Exactly one comment carrying the trailer is a compatible completion and is adopted with nothing performed; none is a proven absence and is performed once; more than one is permanent ambiguity and refuses. Temporary unavailability refuses as itself and performs nothing. +An incomplete observation is never absence. A comment list the adapter could not finish reading is a search that did not answer, and an unfinished search reported as absence is the same duplicate by another route. #### `PullRequest.Ready` and `PullRequest.Close` @@ -1913,9 +1926,19 @@ interface PullRequestMergedResult { `subject` is the canonical pull-request URL and is the whole natural key. The binding is the result record. -The observation is complete or it is nothing. A pull request the host reports as `merged` with a merge commit equal to `expectedMergeCommit` is the compatible completion and is adopted. One reported merged at a *different* commit is a conflict: the target carries somebody else's merge, and the exact-revision reviews invalidate rather than this step succeeding. One still open, or closed unmerged, is temporary unavailability rather than absence or conflict — the host has not caught up, and a later attempt starts again at observation. An incomplete read and an undecidable one are incomplete observation and permanent ambiguity, each refusing as itself. +The observation is complete or it is nothing, and its five outcomes are distinct: + +| Observation | Decision | +| --- | --- | +| `merged: true` at exactly `expectedMergeCommit` | compatible completion; adopt | +| `merged: true` at another commit | conflict; the target carries somebody else's merge, and the exact-revision reviews invalidate rather than this step succeeding | +| `state: "open"`, `merged: false` | temporary unavailability; the host has not yet recognized the merge, and a later attempt starts again at observation | +| `state: "closed"`, `merged: false` | conflict; a pull request somebody closed by hand is a state incompatible with the merged path, not lag | +| an incomplete read, or an undecidable one | incomplete observation and permanent ambiguity respectively, each refusing as itself | + +The third and fourth rows are deliberately not one row. Still open after a publication is eventual consistency and is worth waiting for; closed unmerged is a person having intervened, and waiting for that to resolve itself would wait forever. -Cancellation publishes no completion, and a completed record replays without contacting a Git host. Where this step sits in the terminal sequence — after `Git.PublishTarget` and before the issue is closed — belongs to [the software factory](./github-actions-software-factory-spec.md) §10.3. +Cancellation publishes no completion, and a completed record replays without contacting a Git host. What the run does while the third row persists — a bounded host-configured retry, then a durable machine wait — and where this step sits in the terminal sequence belong to [the software factory](./github-actions-software-factory-spec.md) §10.3. ## 8. Agents inspect; XMD mutates @@ -3020,7 +3043,7 @@ interface IssueCommentResult { `subject` is the canonical issue URL — the normalized single spelling this section already requires of a target — `effect` is the engine-derived effect identity, and `url` is the canonical URL of the comment the effect settled on. The binding is `{ url }`, that comment's own URL. -**The natural key is `subject` plus `effect`, and the body is never part of it.** As with a pull-request comment, the effect identity is made remotely observable by one fixed presentation-neutral trailer the adapter appends to the body it sends, and observation matches that trailer alone. Exactly one comment carrying it is adopted, none is performed once, more than one is permanent ambiguity, and a comment list the adapter could not finish reading is an incomplete observation — never absence. +**The natural key is `subject` plus `effect`, and the body is never part of it.** An Issue provider carries the same requirement §7.10 states for a pull-request comment: it supports one stable opaque correlation marker it can write, preserve and completely query, or it refuses the effect before its first mutation. The authored logical body is byte for byte what the document rendered, and the correlation representation rides outside it. The attempt-state table of §7.10 governs the decision unchanged, including its fourth row — an absent marker after an attempted-but-uncommitted creation is permanent ambiguity rather than proven absence. `Issue.Close` is self-closing and takes one required `url`, one required `reason` from the closed enum `"completed" | "not_planned"`, and one required `as`. Its natural key is the canonical issue URL alone: @@ -3090,15 +3113,20 @@ document may reach for by itself. `commands` is an authored **structured argv list**: an ordered, non-empty list whose every member is a non-empty list of strings. There is no interpreter, no quoting layer and no string to mis-split, which is what makes the record of what ran the same thing as what ran. `as` is required, the prop set is closed, and the form is validated before the host's ceilings are read — an empty list, an empty vector, a member that is not a list of strings, an unknown prop and a missing `as` each fail before any child exists. -**The whole list runs.** Each command runs exactly once, in authored order, and a command that ends unsuccessfully does not stop the ones after it. Running the list is how a reader learns what is broken, and a list that stopped at the first defect would hide the rest — the same reason this repository's own test shards run their files to the end. Each command is charged its own duration ceiling, so one command timing out does not consume another's. +**The pipeline is fail-fast.** Commands run in authored order, and the first command that does not complete successfully is the last one that runs. A command is successful only when it exits normally with status `0`; a non-zero exit, a signal termination and a timeout are each retained as the final row and start no successor. A plan's evidence list is a pipeline — build, then test, then lint — and continuing past a failed build produces later rows evaluated against missing or stale prerequisites, which is evidence that is confidently wrong rather than absent. + +Breadth belongs inside one command whose own contract runs a corpus to completion, such as this repository's runtime-test shards, or in several separately authored `Evidence.Run` elements where the plan says the groups are independent. That a shard runs its files to the end says nothing about whether one arbitrary pipeline should continue after a failure. -The host owns everything else. Which executables may run, what environment they see, the logical working root, how long each command may take, how much output is retained, and what happens to a process tree are host ceilings rather than props. No host path, shell string, ambient environment or command authority enters an authored prop. The commands run against one exact retained Workspace root the host materialized, on the trusted runner where the native toolchain lives — never inside the run's durable owner, which has no toolchain and must not acquire one. +The host owns everything else. Which executables may run, what environment they see, the logical working root, how long each command and the whole list may take, how much output is retained, and what happens to a process tree are host ceilings rather than props. No host path, shell string, ambient environment or command authority enters an authored prop. The commands run against one exact retained Workspace root the host materialized, on the trusted runner where the native toolchain lives — never inside the run's durable owner, which has no toolchain and must not acquire one. -**The result is one record per command, in authored order:** +**The result is the executed prefix, not one row per authored command:** ```ts interface EvidenceRunResult { - readonly commands: readonly EvidenceCommandResult[]; + readonly completion: "passed" | "failed"; + readonly authoredCommands: number; + readonly executed: readonly EvidenceCommandResult[]; + readonly runTimeout?: EvidenceRunTimeout; } interface EvidenceCommandResult { @@ -3106,10 +3134,16 @@ interface EvidenceCommandResult { readonly outcome: "exited" | "signalled" | "timeout"; readonly status?: number; readonly signal?: string; + readonly limit?: "command" | "run"; readonly stdout: EvidenceChannel; readonly stderr: EvidenceChannel; } +interface EvidenceRunTimeout { + readonly limit: "run"; + readonly notStartedAt: number; +} + interface EvidenceChannel { readonly text: string; readonly retainedBytes: number; @@ -3118,27 +3152,37 @@ interface EvidenceChannel { } ``` -`commands` has exactly one entry per authored command, in the order they were authored, and `argv` repeats the vector that ran. `status` is present exactly when `outcome` is `"exited"` and is the numeric exit status; `signal` is present exactly when `outcome` is `"signalled"` and names the signal; `outcome: "timeout"` carries neither, because a child the host terminated for exceeding its ceiling did not choose how it ended. An unknown member, an unknown `outcome`, and a `status` or `signal` beside the wrong outcome each refuse the record. +`completion` is `"passed"` exactly when `executed` holds `authoredCommands` rows and every one of them is an `"exited"` row with `status: 0`; it is `"failed"` in every other case. `authoredCommands` is retained beside `executed` so a reader can tell a complete pass from a deliberately stopped prefix without knowing the authored list, which is the whole reason a prefix is safe to publish. + +`executed` holds the commands that ran, in authored order, and `argv` repeats the vector that ran. `status` is present exactly when `outcome` is `"exited"` and is the numeric exit status; `signal` is present exactly when `outcome` is `"signalled"`; `limit` is present exactly when `outcome` is `"timeout"` and names which ceiling fired. An unknown member, an unknown `outcome`, an unknown `limit`, and any of those three members beside the wrong outcome each refuse the record. + +`runTimeout` is present exactly when the whole-run ceiling expired **between** commands, with no child running. `notStartedAt` is the zero-based index into the authored list of the command that did not start. It is a member of its own rather than a row in `executed`, because a row would have to invent an argv that never ran and a channel that captured nothing. Both channels are retained separately and neither is folded into the other: `text` is the retained UTF-8 prefix, `retainedBytes` is its length in bytes, `producedBytes` is what the child actually produced, and `truncated` is `producedBytes > retainedBytes`. Truncation is stated rather than inferred from a length, so a reader never has to guess whether a command was quiet or cut off. +**Two ceilings bound the work, and both are host-owned.** A per-command ceiling stops one command monopolizing the run; a whole-`Evidence.Run` ceiling bounds total wall-clock cost across the list, including process startup, output draining and teardown. Neither is an authored prop. Before starting each command the host requires positive remaining whole-run time, and while a command runs its effective deadline is the earlier of its own deadline and the whole-run deadline — so a timeout row's `limit` says which of the two fired. A whole-run ceiling that expires between commands ends the result with `completion: "failed"` and the `runTimeout` record above, and starts no successor. + +**A timeout is an ordinary unsuccessful outcome, not an infrastructure failure.** It records that the host enforced its ceiling successfully: it terminated and reaped the process tree and captured bounded channels. It becomes the last row and stops the pipeline. If termination, output draining or reaping *fails* while the host is enforcing that ceiling, the case is an infrastructure failure below rather than a timeout result — the difference is whether the host is reporting what it did or reporting that it could not. + **Which cases bind, which fail, and which commit nothing.** | Case | Outcome | | --- | --- | -| a command exits, at any status | an ordinary result; the list continues | -| a command is terminated by a signal | an ordinary result; the list continues | -| a command exceeds its duration ceiling and the host terminates it | an ordinary `"timeout"` result; the list continues | -| the executable or environment ceiling refuses a command, or the child cannot be created | **launch failure**: the effect fails with an `Error`, binds nothing and commits no result | -| the host cannot read a channel it promised to bound | **output-pump failure**: the effect fails with an `Error`, binds nothing and commits no result | -| the host cannot reap a child or its process tree | **teardown failure**: the effect fails with an `Error`, binds nothing and commits no result | -| the effect is cancelled | nothing is committed at all: the running child is terminated, the remaining commands do not run, and no result and no failure record is published | +| a command exits with status `0` | an ordinary row; the next command starts | +| a command exits non-zero, is terminated by a signal, or hits either duration ceiling | an ordinary final row; `completion` is `"failed"` and no successor starts | +| the whole-run ceiling expires between commands | `completion: "failed"` with a `runTimeout` record and no successor | +| the executable or environment ceiling refuses a command, or the child cannot be created | **launch failure**: the effect fails and produces no `EvidenceRunResult` | +| the host cannot read a channel it promised to bound | **output-pump failure**: the effect fails and produces no `EvidenceRunResult` | +| the host cannot terminate or reap a child or its process tree | **teardown failure**: the effect fails and produces no `EvidenceRunResult` | +| the effect is cancelled | the complete process tree is terminated and no completion and no failure record is committed | -A non-zero status is evidence, not an infrastructure failure — it is the answer the evidence exists to obtain. An infrastructure failure is the host being unable to say what happened, which is why it publishes no partial list: half an answer read as a whole one is worse than no answer. +A non-zero status is evidence, not an infrastructure failure — it is the answer the evidence exists to obtain. An infrastructure failure is the host being unable to say what happened, which is why it publishes no result: half an answer read as a whole one is worse than no answer. -**Precedence is fixed.** Cancellation wins over everything, and invents no completion. Otherwise the first infrastructure failure raised wins, and a teardown failure with no earlier infrastructure failure fails the effect even when every command produced an ordinary outcome — a host that cannot prove it stopped what it started cannot state what the run produced. This is the same rule the workflow lifecycle applies to settlement: teardown is part of the evidence, not work performed after the outcome. +**Precedence is fixed.** Cancellation wins over every other outcome, terminates the complete process tree, and commits neither a completion nor a failure record. Otherwise the first infrastructure failure is authoritative, and a teardown failure that follows it is retained as secondary evidence rather than replacing it. With no earlier infrastructure failure, a teardown failure is itself authoritative even when every command produced an observed exit: a host that cannot prove its process ownership settled cannot publish a successful binding. This is the rule the workflow lifecycle already applies to settlement, where teardown is part of the evidence rather than work performed after the outcome. -A completed record replays without running anything at all. +**No successful binding is not the same as no retained evidence.** A failed effect retains bounded diagnostic evidence on its `Error`: the safely collected executed-command prefix, the separate bounded stdout and stderr channels, the primary infrastructure-failure category, and the secondary teardown category when there is one. That is filtered diagnostic failure evidence and it is deliberately not an `EvidenceRunResult` — nothing binds it, and no document reads it as a pass or a fail of the commands. Replaying a failed effect starts no process. Cancellation retains neither, because no completion won. + +The operation returns Effection's `Result` at the implementation boundary and puts its failure data on an `Error`, like every other outcome in this repository; the authored binding exists only for a successful `EvidenceRunResult`. There is no local success-or-failure union. `Evidence.Run` is not Worker Shell (§10.4) and does not replace it: Worker Shell is a contained interpreter over the Workspace filesystem, while this is native execution of an authored list under a trusted host's ceiling. It is absent from the workflow Agent's capabilities (§8.3) and from every generated-XMD table (§8.4), so neither an Agent nor a fragment it wrote can reach it. @@ -3684,29 +3728,30 @@ runtime-named adapter beside the Deno one: shared modules reach it through the same contextual storage, lifecycle and Workspace APIs, detect no runtime, and import nothing Cloudflare-specific. -**The host assembly contract does not change.** `WorkflowHost` keeps its four methods — `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()` — and the Cloudflare adapter is one more implementation of them beside the Deno one. Starting, looking up, executing, delivering into and inspecting a run are lifecycle operations reached *through* that boundary, exactly as they are locally; they are not replacement method names, and no fifth method appears. What a remote adapter changes is where each of those four reaches, not what the shared CLI asks for. +**The host assembly contract does not change.** `WorkflowHost` keeps its four methods — `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()` — and the Cloudflare adapter is one more implementation of them beside the Deno one. Starting, looking up, executing, delivering into and inspecting a run are lifecycle operations reached *through* that boundary, exactly as they are locally; they are not replacement method names, and no fifth method appears. A remote host receives no transitions type of its own either. What a remote adapter changes is where each of those four reaches, not what the shared CLI asks for. + +**The transition types those methods speak are provider-neutral.** `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` describe what any host's lifecycle does, not what one adapter retains, and they are already defined in the provider-neutral lifecycle module. They become package-root public types, and the Deno entrypoint may keep re-exporting them for source compatibility without owning their meaning. Runtime-specific implementations and retained encodings — SQLite, DOFS, run-id hashing, filesystem paths — stay behind their runtime-named entrypoints, which is the boundary that rationale was always about. That export move is the first implementation story's work; #710 settles that the types are neutral, and performs no production change. Executor ownership is one authenticated WebSocket connection whose lifetime is the acquisition. The owner registers the exact acquisition on admission and invalidates it on close; there is no duration, expiry, renewal, heartbeat, generation record or liveness poll, and a close rolls back nothing already committed. Every mutating transaction validates that exact acquisition and the expected Workspace root together. Native Git, evidence processes and Agent clients run on the ephemeral runner and nowhere else. The runner materializes one selected retained root, works in it, and submits content-addressed changes; the owner validates acquisition, root and content and then atomically publishes the new root with the filtered journal result. That is §10.1's effect transaction with the mutation performed where the tools are and the publication performed where the authority is, so a runner crash exposes only a prior or a new complete transaction and the next acquisition resumes from the exact committed frontier. -#### The transport between them is versioned +#### The transport between them is private to one release -A runner and its durable owner deploy separately, so the connection between them carries records that outlive one deployment and has to be treated as a compatibility boundary rather than as an internal calling convention. It is a closed, versioned, discriminated envelope: +The runner client and the durable owner ship as one software-factory release identity, so the messages between them are not a compatibility boundary and are not a public contract. They are journaled by neither side, exported by neither, authored by nobody, and never expected to interoperate across independently versioned builds. Their decomposition is implementation detail. -```ts -interface RunnerRequest { - readonly protocol: 1; - readonly operation: "acquire" | "frontier" | "materialize" | "commit" | "settle"; - readonly runId: string; - readonly acquisition?: string; - readonly payload: Json; -} -``` +What replaces a wire contract is a release-identity check at admission. Connection admission validates an exact immutable client and server build or protocol fingerprint supplied by trusted deployment configuration, and a mismatch refuses closed — before request parsing, before acquisition, before any state access. There is no cross-version adaptation, no downgrade and no compatibility promise, because two builds that disagree about what was committed is the failure this check exists to prevent rather than to survive. -`acquire` presents the runner's admission assertion and asks for the executor acquisition. `frontier` reads the committed run record and current Workspace root. `materialize` asks for the content-addressed bytes of one retained root. `commit` submits one mutation — the expected root, the content-addressed additions and the already-filtered journal result to append — for the owner to validate and publish atomically. `settle` publishes the run's resulting status. `acquisition` is absent only on `acquire` and required on the other four. There is no operation for delivery or inspection on this connection: those reach the owner on their own authenticated paths and take no acquisition. +Privacy of the transport is not privacy of the authority. These constraints are public and exact however the messages are decomposed: -Compatibility is refusal, never adaptation. A request whose `protocol` this owner does not implement is refused with a fixed protocol-version refusal before any other member is read; an unknown `operation`, an unknown member, and an `acquisition` present or absent against the rule above each refuse without effect. The owner never reads an older shape charitably and the runner never downgrades to one, because a transport that guessed would let two deployments disagree about what was committed. +- one authenticated connection is one executor acquisition; +- every execution mutation validates that acquisition and the expected Workspace root inside the owner's transaction; +- the owner alone parses and adopts requests and alone opens and commits transactions; +- content-addressed data is validated before publication; +- delivery and inspection use separate authenticated paths that take no acquisition; +- credentials and raw transport payloads are never durable public records; +- a runner-to-owner release-identity mismatch refuses closed; and +- a completed replay may read its durable owner but attaches no execution or external-effect provider. #### Which side owns what @@ -3762,6 +3807,7 @@ A completed run replays there as it does locally: it attaches no Workspace, Agen | `Issue.Comment` and `Issue.Close` (§10.3) | specified by #710; implementation unbuilt | | `Evidence.Run` trusted native evidence execution (§10.5) | specified by #710; implementation unbuilt | | `Project.Status` and the Project-provider boundary (§10.6) | specified by #710; implementation unbuilt | +| factory protocol records consumed by these effects | specified by #710 and owned normatively by [the software factory](./github-actions-software-factory-spec.md) §11.2, which this specification links to rather than duplicating: `Git.Merge`'s publish ceiling reads the Stage 7 decision, `PullRequest.Merged`'s wait is one of those records, and `Project.Status` projects a stage through the configured stage-to-option table | | remote lifecycle host, executor connection, versioned runner transport and remote topology (§3.8, §13.2) — the existing four-method `WorkflowHost` boundary, with a Cloudflare implementation beside the Deno one | specified by #710; implementation unbuilt | | terminal-decision delivery on the delivery plane (§3.8) | specified by #710; implementation unbuilt | | Worker JavaScript | deferred | From b364b9824689cede48933921d2e434bf25a1d0bc Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 12:47:30 -0400 Subject: [PATCH 04/81] =?UTF-8?q?=F0=9F=93=9D=20State=20the=20attempt-stat?= =?UTF-8?q?e=20rule=20in=20the=20generic=20external-effect=20contract=20(#?= =?UTF-8?q?710)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ruling C asked for the comment contracts, the architecture, the WGE evidence and the generic external-effect explanation to agree. The first three landed in 0d08dcc; §10.2 did not. Split the two kinds of reconciled effect where the difference actually lives. An effect that mutates a subject already there — Push, a numbered pull-request update, ready, close, issue close, Project status — reads that subject, and a complete observation is decisive whether or not the effect has attempted anything. An effect that creates a new object the host names has nothing pre-existing to read and no client-supplied idempotency key, so its completion is visible only through a correlation value it wrote itself, and absence means a different thing before and after a mutation has been attempted. Only the second kind carries attempt state, and it narrows the decision in one place: unattempted with nothing found performs once, while attempted with no committed completion and nothing found is permanent ambiguity rather than absence. A provider that cannot write, preserve and completely query such a value refuses the effect from observation. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- specs/workflow-workspace-spec.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index c0656ae37..771d614da 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -2642,6 +2642,14 @@ performance retains is that observation, which is how the record says what the external resource held before this attempt moved it. Temporary unavailability is neither absence nor conflict, and never authorizes a mutation: a later explicit attempt starts again at observation. +**Whether absence can be proved at all depends on where the completion is visible**, and effects divide into two kinds. Most of them mutate a subject that already exists and whose own state answers the question: a Push reads the destination ref, a numbered pull-request update reads that pull request, `PullRequest.Ready` and `PullRequest.Close` read its state, `Issue.Close` reads the issue, `Project.Status` reads the field's current option. For those, a complete observation of the subject is decisive whether or not this effect has attempted anything, because what the observation reports is the resource itself. + +The other kind **creates a new object the host names**, and a comment is the case in this specification. Nothing pre-exists to read, and the host issues no client-supplied idempotency key, so the completion is observable only through a correlation value the effect itself wrote. Absence then means "that value is not there", which is a different claim before and after a mutation has been attempted, and the effect therefore retains its **attempt state**: the exact request is retained as prepared and unattempted before any provider mutation, and a live attempt moves it past that. + +For such an effect the decision above narrows in exactly one place. Unattempted with nothing found is proven absence and performs once. **Attempted with no committed local completion and nothing found is permanent ambiguity, not absence** — the correlation value is equally missing because the object was never created and because somebody removed it inside the interrupted window, and performing on that reading is how a duplicate gets published. Exactly one correlation match is compatible completion in either state; more than one is permanent ambiguity in either state; and an incomplete observation stays incomplete rather than becoming absence, since an unfinished search reported as absence is the same duplicate by another route. Once a local completion has committed, none of it decides anything further: replay reads the record and contacts no provider. + +A provider that cannot write, preserve and completely query such a correlation value cannot supply this kind of effect at all, and refuses it from observation before any mutation — the same refusal a plain Git server gives for pull requests. + **The record.** A decision publishes one journal result holding the request, the normalized pre-state, the normalized observations, the decision — `adopted` or `performed` — and the normalized result. Replaying it contacts no provider and From 817d3dd36cb07fddd972c6ae33716444f3cb75da Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 13:05:12 -0400 Subject: [PATCH 05/81] =?UTF-8?q?=F0=9F=93=9D=20Separate=20machine=20waits?= =?UTF-8?q?=20from=20typed=20answers=20(#710)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four narrow corrections from the Architect's verdict on b364b98. 1. The merged-observation wait is not a typed-answer suspension. A machine wait is a second durable wait kind inside the existing lifecycle: it asks nobody anything and ends because a later execution observed a provider again, so it has no response schema, no `xmd workflow answer` route, no form and no bound value, and it publishes no `suspension_request` and consumes no `suspension_answer`. It shares only the atomic boundary — its `machine_wait` event and the `suspended` status commit together, and the acquisition is released after that commit. Its identity is a `waitId`, never a suspension ID; `MergedObservationWait.suspensionId` becomes `waitId`, while `ConflictSuspension.suspensionId` and the typed-answer protocol are untouched. A new `MergedObservationWake` names the same `waitId` and a closed `source`, with `intakeId` required exactly for a provider intake and absent exactly for an operator resume. Waking is permission to look again: the intake retains a bounded notification carrying no answer, verdict, stage, transition or observation result, a later executor consumes it and appends the wake event in one transaction, and a resume without a pending wake or operator authority reports the same wait and settles suspended again. 2. The generic create-effect rule now distinguishes its two safe mechanisms. An effect with a provider-native client key — an Issue upsert, a pull-request upsert — reconciles on that key and carries no attempt state; creating an object is not by itself what makes an effect attempt-stateful. A comment has neither a native key nor a pre-existing subject, which is why it needs the marker and the unattempted/attempted distinction. An effect with neither mechanism refuses before its first mutation, as an intentional constraint. 3. Completed replay may reach and read the run's durable owner — lifecycle storage access, not external-effect replay — while attaching no external-effect provider, performing no effect again and starting no native operation. The "no remote storage session" wording is gone from every document. 4. The authored comment body is preserved byte for byte as the authored portion of the projection, with transport metadata outside it, rather than the provider payload being claimed to equal the authored bytes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- architecture.md | 10 ++--- specs/executable-mdx-spec.md | 18 ++++++--- specs/github-actions-software-factory-spec.md | 38 +++++++++++++++---- specs/workflow-spec.md | 4 +- specs/workflow-workspace-spec.md | 25 ++++++++---- 5 files changed, 69 insertions(+), 26 deletions(-) diff --git a/architecture.md b/architecture.md index 4e4d9435c..5bf9d7dbb 100644 --- a/architecture.md +++ b/architecture.md @@ -112,6 +112,8 @@ Existing documents and code get aligned to this section retroactively. | delivery-plane transaction | one authenticated transaction that retains an externally supplied value for an exact retained subject without executing the run: intake retention, typed answer delivery and terminal-decision delivery are the three. It generalizes delivery to subjects other than a suspension request, on the same terms: no executor acquisition, no document execution, no journal append, no run-status change | | implementation revision | the evolving pair `{ headSha, baseSha }` one factory run is currently producing or reviewing: the exact head commit of the implementation branch and the exact target-branch commit it is evaluated against. It changes many times within one run and never takes part in run identity | | exact-review subject | the implementation revision a review conclusion names. A conclusion authorizes only the pair it names, so a later revision inherits nothing and a moved half of the pair invalidates it | +| machine wait | a durable wait that asks nobody anything: it ends because a later execution observed a provider again, not because a value was delivered. It shares the atomic suspension boundary — its retained event and the `suspended` status commit together, and the executor acquisition is released only after that commit — but it is a distinct event kind identified by a `waitId`, and it has no response schema, no answer route, no form and no bound value. It is a second kind of wait inside the lifecycle, never a second lifecycle controller | +| wake notification | a bounded record correlated to one exact machine wait, retained by an authenticated intake as an ordinary delivery-plane transaction. It carries no answer, verdict, stage, transition or observation result; a later executor consumes one inside the run's transaction and appends the wake event that permits exactly one further observation | | terminal settlement | the last transition of a run whose outcome required external projections: the retained terminal decision is published as run state only after every required projection has completed, so a completed replay never needs a provider to repair one | ## Three axes @@ -435,9 +437,7 @@ authority is. A runner that dies mid-flight therefore exposes only a prior or a new complete transaction, never a partial one. The next acquisition performs the ordinary stale-execution recovery and resumes from the exact committed WorkflowRun and -Workspace frontier. A completed run replays as it does anywhere: it attaches no -remote storage session, Workspace, Agent, process, Git, Git-host, Issue, -Project or credential provider. +Workspace frontier. A completed run replays as it does anywhere, and reading its own history is what it does: it may reach the durable owner holding its retained result — an ephemeral client has nothing else to replay from — while attaching no Workspace, Agent, process, Git, Git-host, Issue, Project, credential or other external-effect provider, performing no effect again and starting no native operation. Lifecycle storage access is not external-effect replay, and only the second is what a completed run must never do. ## The workflow lifecycle @@ -1509,9 +1509,9 @@ Which option means which stage is configuration, not inference. A host that proj ### Comments, readiness and closure -Four more reconciled effects join the same boundary, each keyed by its own subject. `Issue.Comment` is an Issue-provider effect keyed by the canonical issue URL plus the engine-derived effect identity; `PullRequest.Comment` is a Git-host effect keyed by the canonical pull-request URL plus that identity. The body is presentation in both: keying a comment by its text would make an edited sentence a different comment. A Git host issues no idempotency key for a comment, so a comment provider has to support one stable opaque correlation marker it can write, preserve and completely query, and a provider that cannot refuses before its first mutation. The marker is provider transport metadata rather than authored prose: the authored logical body stays byte for byte what the document rendered, the correlation representation rides outside it, and the binding and every replay expose the body and the provider's comment identity rather than the encoding. What a complete observation means depends on the attempt state the effect retains — before an attempt, no marker is proven absence and permits one creation; after an attempt with no committed completion, no marker is permanent ambiguity, because it equally describes somebody having removed one. That is what stops an interrupted creation from becoming a duplicate without pretending a person cannot edit a comment. `PullRequest.Ready` and `PullRequest.Close` are Git-host effects keyed by their exact pull-request subject, and `Issue.Close` is an Issue-provider effect keyed by its exact issue subject and carrying a closed `reason` enum that has to match the retained terminal intent. Each observes before it mutates, adopts a compatible completion, performs once from proven absence or an exact compatible pre-state, and refuses conflict, permanent ambiguity, incomplete observation and temporary unavailability. +Four more reconciled effects join the same boundary, each keyed by its own subject. `Issue.Comment` is an Issue-provider effect keyed by the canonical issue URL plus the engine-derived effect identity; `PullRequest.Comment` is a Git-host effect keyed by the canonical pull-request URL plus that identity. The body is presentation in both: keying a comment by its text would make an edited sentence a different comment. Creating an object is not by itself what makes an effect attempt-stateful: an Issue upsert and a pull-request upsert each reconcile on a key or an identity the provider gives them, and keep their existing complete-observation contracts. A comment has neither. A Git host issues no client-supplied idempotency key for one, so a comment provider has to support one stable opaque correlation marker it can write, preserve and completely query, and a provider that cannot refuses before its first mutation. The marker is provider transport metadata rather than authored prose: the authored logical body stays byte for byte what the document rendered, the correlation representation rides outside it, and the binding and every replay expose the body and the provider's comment identity rather than the encoding. What a complete observation means depends on the attempt state the effect retains — before an attempt, no marker is proven absence and permits one creation; after an attempt with no committed completion, no marker is permanent ambiguity, because it equally describes somebody having removed one. That is what stops an interrupted creation from becoming a duplicate without pretending a person cannot edit a comment. `PullRequest.Ready` and `PullRequest.Close` are Git-host effects keyed by their exact pull-request subject, and `Issue.Close` is an Issue-provider effect keyed by its exact issue subject and carrying a closed `reason` enum that has to match the retained terminal intent. Each observes before it mutates, adopts a compatible completion, performs once from proven absence or an exact compatible pre-state, and refuses conflict, permanent ambiguity, incomplete observation and temporary unavailability. -`PullRequest.Merged` is the fourth, and it is the one that never mutates. A Git host records a pull request as merged when it notices its own ref move, which is not the same event as a target publication succeeding, so the fact has to be observed as its own retained step rather than inferred from the step before it. Adoption is its only completion: the host reporting the pull request merged at the exact published commit is the fact, and a merge at another commit is a conflict. A pull request still open is temporary unavailability — the host has not caught up, which is a different thing from refusing — while one closed unmerged is a conflict, because a person having intervened is not a state that resolves itself by waiting. A run that keeps observing the open case does so through a bounded host-configured retry and then a durable machine wait whose subject is the pull request and the expected commit; that wait carries no verdict and appends no outcome, so waking it is permission to look again rather than an answer. +`PullRequest.Merged` is the fourth, and it is the one that never mutates. A Git host records a pull request as merged when it notices its own ref move, which is not the same event as a target publication succeeding, so the fact has to be observed as its own retained step rather than inferred from the step before it. Adoption is its only completion: the host reporting the pull request merged at the exact published commit is the fact, and a merge at another commit is a conflict. A pull request still open is temporary unavailability — the host has not caught up, which is a different thing from refusing — while one closed unmerged is a conflict, because a person having intervened is not a state that resolves itself by waiting. A run that keeps observing the open case does so through a bounded host-configured retry and then a machine wait whose subject is the pull request and the expected commit. That wait is not a typed-answer suspension: it publishes no response schema, accepts no delivered value, and ends because an execution looked again rather than because somebody answered. An authenticated intake retains a bounded wake notification correlated to that exact wait and nothing more; a later executor consumes it and appends the wake event in one transaction, and only a compatible observation advances anything. ### Ordered merge, and publishing a target diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 5ec0a8240..a6e89118d 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -10069,10 +10069,15 @@ and [Workflow runs](./workflow-spec.md) §9.8. | WRH9 | Consumption | A later executor consumes the retained value inside the run's transaction and appends the accepted event exactly once | | WRH10 | Inspection | Status and history read immutable snapshots, take no acquisition, and authorize no transition | | WRH11 | Teardown | Closing the connection releases executor ownership and rolls back nothing already committed | -| WRH12 | Completed replay | A completed run replays attaching no remote storage session, Workspace, Agent, process, Git, Git-host, Issue, Project or credential provider | +| WRH12 | Completed replay | A completed run replays by reading its durable owner — lifecycle storage access, not external-effect replay — while attaching no Workspace, Agent, process, Git, Git-host, Issue, Project, credential or other external-effect provider, performing no effect again and starting no native operation | | WRH13 | Host neutrality | Shared WorkflowRun modules import nothing Cloudflare-specific and detect no runtime; the runtime-named entrypoint is the only place the topology appears | | WRH14 | The host boundary is unchanged | The Cloudflare adapter satisfies the existing `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()` with the same provider-neutral transition and request types; no fifth method and no adapter-specific transitions type appears, and the shared CLI asks the same four questions it asks the Deno host | -| WRH17 | Waking a machine wait | An authenticated intake for a merged-observation wait's exact subject is retained without an executor acquisition, appends no outcome and moves no stage; a later executor reobserves and only an adoption advances the run | +| WRH17 | A machine wait is not a typed answer | The wait publishes a `machine_wait` event identified by a `waitId`, never a `suspension_request` or a suspension id; it exposes no response schema, no `xmd workflow answer` route, no form and no bound value, and inspection reports a run waiting on provider state | +| WRH18 | Atomic wait settlement | The `machine_wait` event and the `suspended` status commit together, the executor acquisition is released only after that commit, and a refused settlement publishes neither | +| WRH19 | Wake delivery | An authenticated intake correlated to the exact wait subject retains one bounded wake notification with no executor acquisition, no lifecycle outcome, no run-status change and no answer, verdict, stage, transition or observation result | +| WRH20 | Wake refusals | A duplicate notification changes nothing; one naming another wait, a spent wait, an invalidated wait or a terminal run refuses and leaves the active wait unchanged | +| WRH21 | Wake consumption is one transaction | A later executor consumes one notification and appends one `machine_wake` for that exact `waitId` in a single transaction — `intakeId` present exactly for `provider-intake` and absent exactly for `operator-resume`; a crash before commit leaves wait and notification pending, and replay after commit restores the event without consuming or appending again | +| WRH22 | Resume without authority | An explicit resume with neither a pending wake nor operator-resume authority ends nothing: it reports the same machine wait and settles `suspended` again | | WRH15 | Release identity | Connection admission validates an exact immutable runner and owner build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before request parsing, acquisition or state access; no message shape is adapted, downgraded or negotiated, and no transport record is journaled, exported or authored | | WRH16 | Ownership split | Connection admission, request parsing, transaction lifetime and stale recovery belong to the owner; provider attachment and cancellation of its own execution belong to the runner; content is produced by the runner and validated by the owner | @@ -10105,7 +10110,8 @@ Defined in [Workflow workspaces](./workflow-workspace-spec.md) §7.10, §10.3 an |---|------|--------| | WGE1 | Comment identity | A comment's natural key is its subject plus the engine-derived effect identity; a re-rendered or edited body is the same comment and produces no second one | | WGE2 | Ready and close | `PullRequest.Ready` and `PullRequest.Close` are keyed by their exact subject; an already-ready pull request is adopted, a merged one conflicts for both, and each binds its literal closed record rather than an observed one | -| WGE2b | Comment correlation | A provider that cannot write, preserve and completely query a stable opaque marker refuses before its first mutation; the authored logical body is sent byte for byte and the correlation representation rides outside it; the binding and every replay expose the body and the provider comment identity, never the encoding | +| WGE2a2 | Create effects differ | An effect with a provider-native client idempotency or correlation key — an Issue upsert, a pull-request upsert — reconciles on that key under its existing complete-observation contract and carries no attempt state; only an effect with neither a native key nor a pre-existing subject to read requires the marker and the unattempted/attempted distinction, and one with neither mechanism refuses before its first mutation | +| WGE2b | Comment correlation | A provider that cannot write, preserve and completely query a stable opaque marker refuses before its first mutation; the authored logical body is preserved byte for byte as the authored portion of the projection while the correlation representation lives outside it, so the provider payload is not the authored bytes; the binding and every replay expose the authored body and the provider comment identity, never the transport encoding | | WGE2c | Attempt state decides absence | Unattempted with no marker is proven absence and creates once; exactly one marker adopts; more than one is ambiguity; **attempted with no committed completion and no marker is permanent ambiguity, not absence**, so a removed marker inside the interrupted window stalls rather than duplicating; a marker removed after a committed completion changes nothing, because replay contacts no provider | | WGE2d | Merged observation | `PullRequest.Merged` mutates nothing and only adopts: merged at the exact published commit is the completion, merged at another commit conflicts, still open is temporary unavailability, and closed unmerged conflicts rather than being waited out | | WGE3 | Issue closure | `Issue.Close` adopts an issue already closed with the same reason and conflicts with one closed under the other reason | @@ -10188,7 +10194,7 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) | WFP9b | Frontier reduction | Each reduction input of §11.2 — advance, amend, invalidate, head change, base-only change, base-needs-work, and a terminal decision on an already-terminal run — produces exactly the stated frontier, and a reduction leaving a stage out of range, two entries for one stage, or an advance past an unaccepted stage refuses | | WFP9c | Decision shapes | `merge` carries revision, actor and delivery identity; `abandon` adds a required reason; `change` adds a reason and the earliest invalidated stage; a decision naming a revision that is not the current frontier revision refuses | | WFP9d | Stage-to-option table | The retained table holds exactly nine entries ordered by stage, one option ID per stage `0`-`8`, every option ID distinct, and every display name equal to the settled status string for its stage; missing, partial, duplicated, cross-field, unavailable and renamed cases each refuse | -| WFP9e | Merged-observation wait | The wait record names the canonical pull-request URL, expected merge commit, terminal decision, revision and suspension identity, carries `retriesExhausted: true`, and holds no stage, outcome or verdict; consuming it authorizes one reobservation | +| WFP9e | Merged-observation wait and wake | The wait record names a `waitId`, the canonical pull-request URL, the expected merge commit, the terminal decision and the revision, carries `retriesExhausted: true`, and holds no stage, outcome, verdict or response schema; the wake record names the same `waitId` and a closed `source`, with `intakeId` required exactly for `provider-intake` and absent exactly for `operator-resume`, and carries no observation result | | WFP9f | Terminal references | Every event a terminal names must belong to this run and intent, parse under its exact effect kind, be complete, and agree on revision, actor and decision; missing, foreign, wrong-kind, incomplete, invalidated, duplicated and cross-path events each refuse settlement, and provider identities stay in the referenced results rather than being copied into the terminal | | WFP10 | Definition identity | Definition incompatibility is a workflow lifecycle refusal and never an implementation correction | | WFP11 | Journal is authority | A Project status, comment or pull-request state ahead of the journal is drift and authorizes no stage | @@ -10207,7 +10213,9 @@ Defined in [the software factory](./github-actions-software-factory-spec.md) | WFL5 | Merge decision first | A merged decision is retained before any effect is attempted | | WFL6 | Merge ordering | Merge construction, target publication, `PullRequest.Merged`, issue completion as `completed`, the Project move and terminal settlement are separate reconciled steps in that order | | WFL6b | Paths do not borrow steps | The merged path closes no pull request and the abandoned path constructs no merge, publishes no target and observes no merged state; a `FactoryTerminal` naming a step from the other path refuses | -| WFL6c | Waiting for the host | A pull request still open runs the bounded host-configured retry and then suspends on the merged-observation wait; the wait appends no outcome and moves no stage, an intake or an explicit resume only permits reobservation, and terminal settlement stays absent until the adoption succeeds. A conflict observed on any attempt ends the wait as a conflict | +| WFL6c | Waiting for the host | A pull request still open runs the bounded host-configured retry and then enters the merged-observation machine wait; the wait appends no lifecycle outcome and moves no stage, a wake only permits one further observation, and terminal settlement stays absent until the adoption succeeds | +| WFL6d | Reobservation after a wake | Authored control flow invokes `PullRequest.Merged` again after the wake event; an adoption advances the terminal sequence, a still-open observation may retry and wait again at a new durable position, and merged-at-another-commit or closed-unmerged ends the wait as a conflict | +| WFL6e | Cancellation during a wait | Cancelling a waiting run follows ordinary run cancellation and invents neither a wake nor a merged observation | | WFL7 | Terminal is last | The run becomes terminal `merged` only after every required projection completes | | WFL8 | Abandonment | An exact-revision authenticated abandonment carrying a reason is retained first, then pull-request close unmerged, issue close as `not_planned`, the Project move and settlement; terminal `abandoned` follows all of them | | WFL9 | Retention | Both terminal kinds retain the actor, exact revision, resulting provider identities and required reason, and retain the Project item, branch, comments, journal, Workspace roots and Agent evidence | diff --git a/specs/github-actions-software-factory-spec.md b/specs/github-actions-software-factory-spec.md index 70765d19a..539053123 100644 --- a/specs/github-actions-software-factory-spec.md +++ b/specs/github-actions-software-factory-spec.md @@ -256,9 +256,7 @@ admission are its. Shared production modules stay provider-neutral: they do not detect Cloudflare, Deno, GitHub Actions or any other runtime, and they reach every host-specific behavior through the contextual APIs that already exist. -**A completed replay attaches nothing.** It contacts no remote storage session -and attaches no Workspace, Agent, process, Git, GitHub, Project or credential -provider. That is the property §10's terminal ordering exists to preserve. +**A completed replay reads its own history and nothing else.** It may reach and read the run's durable owner, because that owner is where the retained result is and an ephemeral client holds nothing to replay from. It attaches no Workspace, Agent, process, Git, Git-host, Issue, Project, credential or other external-effect provider, performs no effect again and starts no native operation. Reading retained completion from its authoritative owner is lifecycle storage access, not external-effect replay, and it is that second thing §10's terminal ordering exists to keep unnecessary. ## 4. Durable procedure and GitHub projection @@ -646,11 +644,25 @@ A pull request observed still open after a successful target publication is temp The authored factory first performs a **bounded retry** around `PullRequest.Merged`. The retry count, total duration and backoff are host ceilings configured for the deployment; they are not props on the component and no document widens them. -When that retry is exhausted while the pull request is still open, the run enters a **durable machine wait** at Stage 7. It is a suspension in the ordinary sense — the executor settles `suspended` and releases its acquisition — but its subject is a fact about a provider rather than a question for a person, so it is neither an answer nor a Stage 7 decision, and no form is bound to it. Its exact subject is the canonical pull-request URL, the expected merge commit, the retained terminal decision identity and the current implementation revision, retained as the `MergedObservationWait` record of §11.2. +When that retry is exhausted while the pull request is still open, the run enters a **machine wait** at Stage 7. -Two things may wake it. An authenticated intake for a relevant pull-request state change on that exact subject is retained as a delivery-plane transaction under §5, taking no executor acquisition; an operator may also resume the run explicitly. Neither wakeup carries a verdict, a stage or a transition — waking is permission to look again, not an answer. A later executor reobserves `PullRequest.Merged` and proceeds only on the adoption of §7.11's first row. +A machine wait is a second kind of durable wait, and it is deliberately not a suspension in the typed-answer sense. A typed suspension publishes a request and a response schema and ends when somebody delivers one value that satisfies it. This wait asks nobody anything: it ends because a later execution looked at a provider again. It therefore has no response schema, no `xmd workflow answer` route, no web form and no bound value, and nothing about the typed-answer protocol — `suspension_request`, `suspension_answer`, a suspension ID — takes part in it. It is a second wait *kind* inside the existing lifecycle, never a second lifecycle controller. -The wait is not a second lifecycle controller. It appends no outcome, moves no stage, and terminal settlement stays absent until the observation adopts. A conflict observed on any attempt — a merge at another commit, or a pull request closed unmerged — ends the wait as a conflict rather than continuing it. +What it does share is the lifecycle boundary. The retained wait event and the `suspended` run status commit together, the executor acquisition is released only after that commit, and a settlement the host refuses publishes neither. Its retained event kind is `machine_wait`, distinct from `suspension_request`, and its stable identity is a `waitId` the trusted execution derives from the run and the authored expansion on the same engine-owned terms every other durable position uses. The run's stop reason references that filtered `machine_wait` event, so inspection reports that the run is waiting on provider state and offers no response schema and no answer command. + +The wait's subject is the canonical pull-request URL, the expected merge commit, the retained merge-decision event identity, the current implementation revision and `retriesExhausted: true`, retained as the `MergedObservationWait` record of §11.2. + +**Waking is permission to look again, not an answer.** Two sources may wake it. An authenticated intake for a relevant pull-request state change, correlated to that exact wait subject, is retained as a delivery-plane transaction under §5: it takes no executor acquisition, appends no lifecycle outcome, changes no run status, and supplies no answer, verdict, stage, transition or observation result. It records a bounded wake notification and nothing else. An authorized operator may also resume the run, which is executor-side control rather than a delivery — a resume is not a forged intake. + +A duplicate wake notification changes nothing. One naming another wait, a spent wait, an invalidated wait or a terminal run refuses and leaves the active wait exactly as it was. + +A later executor consumes one retained wake notification inside the run's transaction and appends one filtered `machine_wake` event for that exact `waitId`; an authorized operator resume appends the same event with `source: "operator-resume"`. Consuming the wake and ending the retained wait are one transaction, so a crash before it commits leaves both the wait and the notification pending, and a replay after it commits restores the wake event without consuming or appending anything again. + +After the wake event, authored control flow invokes `PullRequest.Merged` again. Only the compatible adoption of §7.11's first row advances the terminal sequence. An observation that is still open may retry and wait again at a new durable position; a merge at another commit and a pull request closed unmerged remain conflicts and end the wait as one rather than continuing it. + +An explicit resume with neither a pending provider wake nor operator-resume authority ends nothing. It reports the same machine wait and settles `suspended` again. Cancellation follows ordinary run cancellation and invents neither a wake nor a merged observation. + +The wait appends no lifecycle outcome and moves no stage, and terminal settlement stays absent until the observation adopts. ## 11. Exact public contracts @@ -799,16 +811,26 @@ All three bind the exact revision they were made against and the authenticated a interface MergedObservationWait { readonly schema: "merged-observation-wait"; readonly version: 1; + readonly waitId: string; readonly subject: string; readonly expectedMergeCommit: string; readonly decisionId: string; readonly revision: ImplementationRevision; - readonly suspensionId: string; readonly retriesExhausted: true; } + +interface MergedObservationWake { + readonly schema: "merged-observation-wake"; + readonly version: 1; + readonly waitId: string; + readonly source: "provider-intake" | "operator-resume"; + readonly intakeId?: string; +} ``` -`subject` is the canonical pull-request URL, `decisionId` names the retained `merge` decision this wait belongs to, and `suspensionId` is the durable wait's own identity. `retriesExhausted` is literal: the record exists only after §10.4's bounded retry has run out, so a wait retained before that would be a run skipping the cheap path. The wait carries no stage, no outcome and no verdict, and consuming it authorizes one reobservation and nothing else. +`waitId` is the machine wait's own identity, derived from the run and the authored expansion; it is not a suspension ID, and no typed-answer record names it. `subject` is the canonical pull-request URL and `decisionId` names the retained `merge` decision this wait belongs to. `retriesExhausted` is literal: the record exists only after §10.4's bounded retry has run out, so a wait retained before that would be a run skipping the cheap path. The wait carries no stage, no outcome, no verdict and no response schema. + +`MergedObservationWake` is what a later executor appends for that exact `waitId`. `intakeId` is required exactly when `source` is `"provider-intake"` and absent exactly when it is `"operator-resume"`, because the operator path is authenticated executor-side control rather than a delivery, and a wake that claimed an intake it does not have would be a forged one. A wake says only that another observation attempt may occur; it carries no observation result, and consuming it authorizes exactly one reobservation. #### The configured stage-to-option table diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index d8d77a470..409d12ab9 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -905,7 +905,9 @@ document-execution record, appends no journal event and changes no run status. What authorizes it is the subject, not the caller's position in the lifecycle — a suspension id the run's retained `suspension_request` names for an answer, and the exact retained decision subject for a terminal decision. A value for a -subject the run is not holding is refused with nothing written. Read-only +subject the run is not holding is refused with nothing written. + +A **wake notification** is delivered on the same terms and is the one delivery that carries no value at all. A run may wait on a fact about a provider rather than on an answer, and that machine wait is a distinct event kind identified by a `waitId` rather than by a suspension id. An authenticated intake correlated to the exact wait subject retains a bounded notification saying that another observation may occur — no answer, verdict, stage, transition or observation result — and a later executor consumes it and appends the wake event in the run's own transaction. Delivery still stores and execution still decides; what changes is that here there is nothing stored for the execution to read except permission to look again. Read-only inspection takes no acquisition either, and returns the immutable snapshot surface of the lifecycle contract rather than a writable handle. diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 771d614da..78b24cfc5 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -359,6 +359,14 @@ resume, watchers, unattended iteration and remote host selection — is #300's. Nothing above waits on it: a suspended run continues through `xmd workflow answer` followed by an explicit `xmd workflow resume`. +**A machine wait is a second wait kind, not a second protocol.** Everything above describes a wait that ends when somebody delivers one typed value. A run can also wait on a fact about a provider — a remote object whose state a later observation will read — and that wait asks nobody anything. It has no request to answer, no response schema, no `xmd workflow answer` route and no bound value, so it publishes no `suspension_request`, consumes no retained answer and appends no `suspension_answer`. + +What it shares is this section's boundary and nothing else. Its retained event kind is `machine_wait`, distinct from `suspension_request`; its stable identity is a `waitId` the trusted execution derives from the run and the authored expansion, on the same terms every other durable position uses, and it is never a suspension ID. The wait event and the `suspended` run status commit together, the executor acquisition is released only after that commit, and a settlement the host refuses publishes neither. The stop reason references the filtered `machine_wait` event, so inspection reports a run waiting on provider state and offers no response schema and no answer command. + +A machine wait ends by being woken and then looking again. An authenticated intake correlated to the exact wait subject retains one bounded **wake notification** as an ordinary delivery-plane transaction — no executor acquisition, no lifecycle outcome, no run-status change — and it carries no answer, verdict, stage, transition or observation result. A duplicate notification changes nothing, and one naming another, spent, invalidated or terminal wait refuses without touching the active wait. An authorized operator resume is executor-side control rather than a delivery and reaches the same place. A later executor consumes one wake and appends one filtered `machine_wake` event for that exact `waitId` in one transaction, so a crash before the commit leaves both pending and a replay after it restores the event without consuming or appending again. The wake permits one further observation and decides nothing about its outcome; a resume with neither a pending wake nor operator authority reports the same wait and settles `suspended` again. + +The software factory's merged-observation wait ([the software factory](./github-actions-software-factory-spec.md) §10.4) is the first machine wait, and its closed records are §11.2 there. + ### 3.6 Interruption and cancellation differ Interrupting foreground execution, including with Ctrl-C, releases the current @@ -492,8 +500,9 @@ Workspace root inside its own mutating transaction. **Delivery and inspection are the exceptions, and they stay exceptions.** A delivery retains one externally supplied value for one exact retained subject under §3.5's rules — no acquisition, no execution, no attachment, no journal -event, no status change — whether the subject is a suspension request or a -terminal decision. Inspection is read-only and returns immutable snapshots. A +event, no status change — whether the subject is a suspension request, a +terminal decision, or a machine wait whose notification carries no value at all. +Inspection is read-only and returns immutable snapshots. A remote host that let either one advance a lifecycle would have built a second state machine beside the journal. @@ -1839,7 +1848,7 @@ interface PullRequestCommentResult { **The natural key is `subject` plus `effect`, and the body is never part of it.** A Git host issues no client-supplied idempotency key for a comment, so the effect identity has to be observable on the host for an interrupted creation to be found again. A comment provider therefore has to support one **stable opaque correlation marker**: a value it can write with a comment, preserve unchanged, and query completely. A provider that cannot do all three refuses the effect before its first mutation, the way a plain Git server refuses pull requests today — there is no fallback that searches prose. -The marker is provider transport metadata. The **authored logical body is byte for byte what the document rendered**, and the provider's projection may carry the correlation representation outside it; GitHub's adapter encodes it as a non-rendered HTML comment. It is not authored prose, not a credential and not lifecycle authority — publishing an engine-derived effect identity as an opaque non-secret correlation value is what it is for. The public binding and every replay expose the authored body and the provider's comment identity, never the transport encoding. +The marker is provider transport metadata. The **authored logical body is preserved byte for byte as the authored portion of the projection**, and the correlation representation lives outside that logical body rather than inside it; GitHub's adapter encodes it as a non-rendered HTML comment in its provider payload, so the payload the provider sends is not claimed to equal the authored bytes. It is not authored prose, not a credential and not lifecycle authority — publishing an engine-derived effect identity as an opaque non-secret correlation value is what it is for. The public binding and every replay expose the authored body and the provider's comment identity, never the transport encoding. **Observation is judged against the attempt state, not against the host alone.** Before any provider mutation, the durable effect retains that this exact request is prepared and unattempted; a live attempt is what moves it past that. What a complete observation means then depends on which side of that line the effect is on: @@ -1938,7 +1947,7 @@ The observation is complete or it is nothing, and its five outcomes are distinct The third and fourth rows are deliberately not one row. Still open after a publication is eventual consistency and is worth waiting for; closed unmerged is a person having intervened, and waiting for that to resolve itself would wait forever. -Cancellation publishes no completion, and a completed record replays without contacting a Git host. What the run does while the third row persists — a bounded host-configured retry, then a durable machine wait — and where this step sits in the terminal sequence belong to [the software factory](./github-actions-software-factory-spec.md) §10.3. +Cancellation publishes no completion, and a completed record replays without contacting a Git host. What the run does while the third row persists — a bounded host-configured retry, then a machine wait that ends on reobservation rather than on a delivered answer — and where this step sits in the terminal sequence belong to [the software factory](./github-actions-software-factory-spec.md) §10.3 and §10.4. ## 8. Agents inspect; XMD mutates @@ -2644,11 +2653,13 @@ authorizes a mutation: a later explicit attempt starts again at observation. **Whether absence can be proved at all depends on where the completion is visible**, and effects divide into two kinds. Most of them mutate a subject that already exists and whose own state answers the question: a Push reads the destination ref, a numbered pull-request update reads that pull request, `PullRequest.Ready` and `PullRequest.Close` read its state, `Issue.Close` reads the issue, `Project.Status` reads the field's current option. For those, a complete observation of the subject is decisive whether or not this effect has attempted anything, because what the observation reports is the resource itself. -The other kind **creates a new object the host names**, and a comment is the case in this specification. Nothing pre-exists to read, and the host issues no client-supplied idempotency key, so the completion is observable only through a correlation value the effect itself wrote. Absence then means "that value is not there", which is a different claim before and after a mutation has been attempted, and the effect therefore retains its **attempt state**: the exact request is retained as prepared and unattempted before any provider mutation, and a live attempt moves it past that. +The other kind **creates a new object the host names**. Creating one is safely reconcilable through either of two mechanisms, and which one an effect has decides whether attempt state takes part. An effect the provider gives a native client idempotency or correlation key — the key an Issue upsert derives from the canonical target and this run's own effect identity is one — reconciles on that key under its already-stated natural-key and complete-observation contract, and carries no attempt state: creating an object is not by itself what makes an effect attempt-stateful. A pull-request upsert is the same, reconciling on its explicit head-and-base or numbered identity. + +A comment is the one construct here with neither. Nothing pre-exists to read, and the host issues no client-supplied idempotency key, so the completion is observable only through a correlation value the effect itself wrote. Absence then means "that value is not there", which is a different claim before and after a mutation has been attempted, and the effect therefore retains its **attempt state**: the exact request is retained as prepared and unattempted before any provider mutation, and a live attempt moves it past that. For such an effect the decision above narrows in exactly one place. Unattempted with nothing found is proven absence and performs once. **Attempted with no committed local completion and nothing found is permanent ambiguity, not absence** — the correlation value is equally missing because the object was never created and because somebody removed it inside the interrupted window, and performing on that reading is how a duplicate gets published. Exactly one correlation match is compatible completion in either state; more than one is permanent ambiguity in either state; and an incomplete observation stays incomplete rather than becoming absence, since an unfinished search reported as absence is the same duplicate by another route. Once a local completion has committed, none of it decides anything further: replay reads the record and contacts no provider. -A provider that cannot write, preserve and completely query such a correlation value cannot supply this kind of effect at all, and refuses it from observation before any mutation — the same refusal a plain Git server gives for pull requests. +A provider that cannot write, preserve and completely query such a correlation value cannot supply this kind of effect at all, and refuses it from observation before any mutation — the same refusal a plain Git server gives for pull requests. A future host-named create effect that has neither a provider-native client key nor a preservable marker refuses on the same terms. That refusal is the contract rather than a gap in it: an effect that can prove neither absence nor completion has no safe way to run once. **The record.** A decision publishes one journal result holding the request, the normalized pre-state, the normalized observations, the decision — `adopted` or @@ -3051,7 +3062,7 @@ interface IssueCommentResult { `subject` is the canonical issue URL — the normalized single spelling this section already requires of a target — `effect` is the engine-derived effect identity, and `url` is the canonical URL of the comment the effect settled on. The binding is `{ url }`, that comment's own URL. -**The natural key is `subject` plus `effect`, and the body is never part of it.** An Issue provider carries the same requirement §7.10 states for a pull-request comment: it supports one stable opaque correlation marker it can write, preserve and completely query, or it refuses the effect before its first mutation. The authored logical body is byte for byte what the document rendered, and the correlation representation rides outside it. The attempt-state table of §7.10 governs the decision unchanged, including its fourth row — an absent marker after an attempted-but-uncommitted creation is permanent ambiguity rather than proven absence. +**The natural key is `subject` plus `effect`, and the body is never part of it.** An Issue provider carries the same requirement §7.10 states for a pull-request comment: it supports one stable opaque correlation marker it can write, preserve and completely query, or it refuses the effect before its first mutation. The authored logical body is preserved byte for byte as the authored portion of the projection, and the correlation representation lives outside it. The attempt-state table of §7.10 governs the decision unchanged, including its fourth row — an absent marker after an attempted-but-uncommitted creation is permanent ambiguity rather than proven absence. `Issue.Close` is self-closing and takes one required `url`, one required `reason` from the closed enum `"completed" | "not_planned"`, and one required `as`. Its natural key is the canonical issue URL alone: From 8f27b04986712c49eaa77c63321e00d0a9ff6020 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 15:34:01 -0400 Subject: [PATCH 06/81] =?UTF-8?q?=F0=9F=94=A7=20Own=20the=20neutral=20life?= =?UTF-8?q?cycle=20types=20and=20derive=20a=20factory=20run=20id=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first slice of F1: the seams a remote host needs, and the identity it is addressed by. No Cloudflare code yet — this is the boundary work that has to be true before a second host can exist. `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` now export from the package root, which is where they mean what they mean. They were already defined in the provider-neutral lifecycle module but published only through `@executablemd/workflow/deno`, so the shared CLI imported its own return type from an adapter and a second host would have had to load that adapter to name it. The Deno entrypoint keeps its re-exports for source compatibility with a comment saying what belongs behind it — implementations and retained encodings, not the shape of a request — and the CLI and its tests now import from the root. `deriveFactoryRunId()` implements the settled derivation: lowercase unpadded RFC 4648 Base32 over all 32 SHA-256 bytes of `github-issue-v1`, NUL, the canonical GitHub authority, NUL, the exact issue node id. The authority folds case and keeps a non-default port; a scheme, user information, path, query, fragment, whitespace, malformed host or port, and a written-out default port each refuse by name rather than being repaired, because two spellings that both became one authority would be two runs quietly becoming one. The node id is held to non-empty and NUL-free and otherwise passes through byte for byte. Three tests. The identity suite checks fixed vectors computed outside this implementation, the RFC 4648 §10 encoding vectors, that the NUL separators make (authority, node id) unambiguous, and every refusal. A host-neutrality suite walks the shared modules and fails on a host-owned specifier or a runtime detection outside a runtime-named entrypoint, so a Cloudflare import cannot leak into shared code unnoticed. A host-boundary suite pins `WorkflowHost` to exactly four methods and proves it is satisfiable from root-exported types alone. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/cli/src/deno-workflow.ts | 3 +- packages/cli/src/workflow-fork.ts | 5 +- packages/cli/src/workflow.ts | 2 +- .../cli/tests/workflow-host-boundary.test.ts | 95 +++++++ .../cli/tests/workflow-installation.test.ts | 2 +- .../tests/workflow-lifecycle-control.test.ts | 2 +- .../cli/tests/workflow-suspension.test.ts | 2 +- packages/workflow/deno.ts | 9 + packages/workflow/mod.ts | 24 ++ packages/workflow/src/factory/run-id.ts | 251 ++++++++++++++++++ .../workflow/tests/factory-run-id.test.ts | 159 +++++++++++ .../workflow/tests/host-neutrality.test.ts | 104 ++++++++ 12 files changed, 648 insertions(+), 10 deletions(-) create mode 100644 packages/cli/tests/workflow-host-boundary.test.ts create mode 100644 packages/workflow/src/factory/run-id.ts create mode 100644 packages/workflow/tests/factory-run-id.test.ts create mode 100644 packages/workflow/tests/host-neutrality.test.ts diff --git a/packages/cli/src/deno-workflow.ts b/packages/cli/src/deno-workflow.ts index 4b234dd49..4e6d089d1 100644 --- a/packages/cli/src/deno-workflow.ts +++ b/packages/cli/src/deno-workflow.ts @@ -29,8 +29,7 @@ import { useWorkflowRunHost, withWorkflowWorkspace, } from "@executablemd/workflow/deno"; -import type { WorkflowExecutionTransitions } from "@executablemd/workflow/deno"; -import type { WorkflowRunDatabase } from "@executablemd/workflow"; +import type { WorkflowExecutionTransitions, WorkflowRunDatabase } from "@executablemd/workflow"; import type { HelperAssembly } from "@executablemd/workflow/credential-helper"; import { readDefinitionSource } from "./workflow-source.ts"; import type { WorkflowHost } from "./workflow.ts"; diff --git a/packages/cli/src/workflow-fork.ts b/packages/cli/src/workflow-fork.ts index 2d8865bf2..2c3545bd6 100644 --- a/packages/cli/src/workflow-fork.ts +++ b/packages/cli/src/workflow-fork.ts @@ -64,10 +64,7 @@ import { } from "@executablemd/workflow"; import type { ForkSelection, WorkflowRun } from "@executablemd/workflow"; import type { WorkflowRunDatabase } from "@executablemd/workflow"; -import type { - WorkflowExecutionTransitions, - WorkflowRunCreation, -} from "@executablemd/workflow/deno"; +import type { WorkflowExecutionTransitions, WorkflowRunCreation } from "@executablemd/workflow"; import type { EstablishedDefinition } from "./workflow-definition.ts"; import type { WorkflowExecution } from "./workflow.ts"; diff --git a/packages/cli/src/workflow.ts b/packages/cli/src/workflow.ts index d68fb4752..ebed19b44 100644 --- a/packages/cli/src/workflow.ts +++ b/packages/cli/src/workflow.ts @@ -90,7 +90,7 @@ import type { WorkflowExecutionBegun, WorkflowExecutionTransitions, WorkflowRunCreation, -} from "@executablemd/workflow/deno"; +} from "@executablemd/workflow"; import type { SuspensionControllerOptions, SuspensionNotice } from "@executablemd/workflow/deno"; import { SUSPENSION_REQUEST } from "@executablemd/workflow"; import { describeError } from "./props.ts"; diff --git a/packages/cli/tests/workflow-host-boundary.test.ts b/packages/cli/tests/workflow-host-boundary.test.ts new file mode 100644 index 000000000..fcd3d752e --- /dev/null +++ b/packages/cli/tests/workflow-host-boundary.test.ts @@ -0,0 +1,95 @@ +/** + * Tier WRH — the host assembly boundary a second host has to satisfy. + * + * `WorkflowHost` is four methods, and a remote host is one more implementation + * of them rather than a wider surface. That is the settled contract, and the + * way it fails quietly is by growing: a fifth method, or a transitions type only + * one adapter can name, and the "same four questions" claim stops being true + * while every existing test still passes. + * + * So both halves are pinned here. The key set is compared exactly, and the + * provider-neutral lifecycle types are imported from the package root — which + * is where they mean what they mean — so this stops compiling if they retreat + * behind a runtime-named entrypoint. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Operation } from "effection"; +import type { WorkflowRunDatabase } from "@executablemd/workflow"; +import type { + WorkflowBeginRequest, + WorkflowExecutionBegun, + WorkflowExecutionTransitions, + WorkflowForkRequest, + WorkflowForkSelection, + WorkflowRunCreation, +} from "@executablemd/workflow"; +import type { WorkflowHost } from "../src/workflow.ts"; + +/** + * Compile-time proofs. `Assert` is the only instantiation that checks, so + * each of these stops compiling the moment its claim becomes false. + */ +type Assert = T; + +/** The host boundary is exactly these four methods. */ +type FourMethods = Assert< + keyof WorkflowHost extends "useRunHost" | "useLifecycle" | "useDelivery" | "attach" ? true : false +>; +const FOUR_METHODS: FourMethods = true; + +/** Every provider-neutral lifecycle type resolves through the package root. */ +type NeutralTypes = Assert< + [ + WorkflowExecutionTransitions, + WorkflowBeginRequest, + WorkflowExecutionBegun, + WorkflowForkRequest, + WorkflowForkSelection, + WorkflowRunCreation, + ] extends [unknown, unknown, unknown, unknown, unknown, unknown] + ? true + : false +>; +const NEUTRAL_TYPES: NeutralTypes = true; + +/** + * A host built only from the four methods and only from root-exported types. + * + * It answers nothing — the point is that it type-checks, which is the claim a + * second adapter depends on. + */ +function neutralHost(): WorkflowHost { + return { + useRunHost(): Operation { + throw new Error("not this test's question"); + }, + useLifecycle(): Operation { + throw new Error("not this test's question"); + }, + useDelivery(): Operation { + throw new Error("not this test's question"); + }, + attach(_database: WorkflowRunDatabase, operation: Operation): Operation { + return operation; + }, + }; +} + +describe("the workflow host boundary", () => { + it("is exactly four methods", function* () { + expect(FOUR_METHODS).toEqual(true); + expect(Object.keys(neutralHost()).toSorted()).toEqual([ + "attach", + "useDelivery", + "useLifecycle", + "useRunHost", + ]); + }); + + it("is satisfiable from the package root alone", function* () { + expect(NEUTRAL_TYPES).toEqual(true); + expect(typeof neutralHost().attach).toEqual("function"); + }); +}); diff --git a/packages/cli/tests/workflow-installation.test.ts b/packages/cli/tests/workflow-installation.test.ts index 0263f6e7e..f2a71bf0d 100644 --- a/packages/cli/tests/workflow-installation.test.ts +++ b/packages/cli/tests/workflow-installation.test.ts @@ -26,7 +26,7 @@ import { useWorkflowLifecycle, useWorkflowRunHost, } from "@executablemd/workflow/deno"; -import type { WorkflowExecutionTransitions } from "@executablemd/workflow/deno"; +import type { WorkflowExecutionTransitions } from "@executablemd/workflow"; import { Git, WorkflowLifecycle, WorkflowRunStorage } from "@executablemd/workflow"; import type { WorkflowRunDatabase, WorkflowRunStatus } from "@executablemd/workflow"; import type { Json } from "@executablemd/core"; diff --git a/packages/cli/tests/workflow-lifecycle-control.test.ts b/packages/cli/tests/workflow-lifecycle-control.test.ts index 13d2a754d..eab347016 100644 --- a/packages/cli/tests/workflow-lifecycle-control.test.ts +++ b/packages/cli/tests/workflow-lifecycle-control.test.ts @@ -24,7 +24,7 @@ import { useWorkflowLifecycle, useWorkflowRunHost, } from "@executablemd/workflow/deno"; -import type { WorkflowExecutionTransitions } from "@executablemd/workflow/deno"; +import type { WorkflowExecutionTransitions } from "@executablemd/workflow"; import { Git, suspendFor, WorkflowLifecycle } from "@executablemd/workflow"; import type { WorkflowRunDatabase } from "@executablemd/workflow"; import { collect, inlineSource, registerComponents } from "@executablemd/core"; diff --git a/packages/cli/tests/workflow-suspension.test.ts b/packages/cli/tests/workflow-suspension.test.ts index 9a14429b5..a635d3a97 100644 --- a/packages/cli/tests/workflow-suspension.test.ts +++ b/packages/cli/tests/workflow-suspension.test.ts @@ -49,7 +49,7 @@ import { useWorkflowLifecycle, useWorkflowRunHost, } from "@executablemd/workflow/deno"; -import type { WorkflowExecutionTransitions } from "@executablemd/workflow/deno"; +import type { WorkflowExecutionTransitions } from "@executablemd/workflow"; import { Git, SUSPENSION_REQUEST, suspendFor, WorkflowLifecycle } from "@executablemd/workflow"; import type { WorkflowRunDatabase } from "@executablemd/workflow"; import { workflowRunPath } from "@executablemd/workflow/deno"; diff --git a/packages/workflow/deno.ts b/packages/workflow/deno.ts index 368e3ea1c..012dada52 100644 --- a/packages/workflow/deno.ts +++ b/packages/workflow/deno.ts @@ -28,6 +28,15 @@ export { useWorkflowRunStorage } from "./src/deno/provider.ts"; export type { WorkflowRunStorageOptions } from "./src/deno/provider.ts"; export { useWorkflowLifecycle } from "./src/deno/lifecycle.ts"; export { useWorkflowRunHost } from "./src/deno/run-host.ts"; +/** + * Re-exported for source compatibility only. + * + * These are provider-neutral: they describe what any host's lifecycle does, not + * what this adapter retains, and `@executablemd/workflow` owns their meaning. + * Import them from there. What belongs behind this entrypoint is the + * implementation and its retained encoding — SQLite, DOFS, run-id hashing, + * filesystem paths — not the shape of a request. + */ export type { WorkflowBeginRequest, WorkflowExecutionTransitions, diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index 85f09ea39..e0d6fabf3 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -314,6 +314,30 @@ export type { WorkflowLifecycleApi, WorkflowLifecycleSnapshot, } from "./src/lifecycle/api.ts"; +export { + admitFactoryRunSubject, + admitIssueNodeId, + base32Unpadded, + canonicalGitHubAuthority, + deriveFactoryRunId, + FACTORY_RUN_ID_LENGTH, + factoryRunIdPreimage, + FactoryRunSubjectError, +} from "./src/factory/run-id.ts"; +export type { FactoryRunSubject, FactoryRunSubjectFailure } from "./src/factory/run-id.ts"; + +// What a trusted host needs to move a run's lifecycle. These describe what any +// host's lifecycle does rather than what one adapter retains, so this entrypoint +// owns their meaning; `./deno` re-exports them for source compatibility and a +// second host implements the same shapes without that module being loaded. +export type { + WorkflowBeginRequest, + WorkflowExecutionBegun, + WorkflowExecutionTransitions, + WorkflowForkRequest, + WorkflowForkSelection, + WorkflowRunCreation, +} from "./src/lifecycle/execution.ts"; // The export request, its result and the boundary it names. The retained record // shapes an artifact also carries are DOFS and SQLite rows, so they are the // Deno entrypoint's to publish rather than this one's. diff --git a/packages/workflow/src/factory/run-id.ts b/packages/workflow/src/factory/run-id.ts new file mode 100644 index 000000000..5ad6d0474 --- /dev/null +++ b/packages/workflow/src/factory/run-id.ts @@ -0,0 +1,251 @@ +/** + * The run id a software-factory run is addressed by. + * + * One GitHub issue is one durable run, so the id has to be a function of the + * issue and of nothing that can change while the work is going on. Repository + * names get renamed, issue numbers move between deployments, Project items and + * their statuses are edited constantly, branches and revisions are the point of + * the exercise, and delivery ids and actors differ on every request. None of + * them takes part. What is left is the deployment the issue lives in and the + * opaque node id that deployment gave it, and those two are what this hashes. + * + * Because every input is immutable, admitting one issue twice derives one id and + * reaches one run through ordinary compatible reuse, and no separate idempotency + * concept appears anywhere above it. Two independent implementations handed the + * same authority and node id produce the same 52 characters. + * + * The derivation is specified in `specs/github-actions-software-factory-spec.md` + * §1.1 and restated in `specs/workflow-spec.md` §9.1. It is host-selected public + * run id and nothing more: opaque to everything but equality and lifecycle + * addressing, and a legal one under the storage rule, which wants a non-empty + * string containing no NUL. + */ + +import { until } from "effection"; +import type { Operation } from "effection"; + +/** The version tag the digest opens with. A different scheme takes a different tag. */ +const SCHEME = "github-issue-v1"; + +/** Lowercase RFC 4648 Base32. No padding is ever emitted, so `=` is absent. */ +const BASE32_ALPHABET = "abcdefghijklmnopqrstuvwxyz234567"; + +/** + * How many characters a full SHA-256 becomes. + * + * 32 bytes is 256 bits, and Base32 carries five bits per character, so the + * unpadded encoding is `ceil(256 / 5)` characters. Stated rather than computed + * because it is a contract a second implementation is held to. + */ +export const FACTORY_RUN_ID_LENGTH = 52; + +/** Why a subject could not be turned into a run id. */ +export type FactoryRunSubjectFailure = + | "authority-empty" + | "authority-has-scheme" + | "authority-has-userinfo" + | "authority-has-path" + | "authority-has-query" + | "authority-has-fragment" + | "authority-has-whitespace" + | "authority-malformed-host" + | "authority-malformed-port" + | "authority-default-port" + | "node-id-empty" + | "node-id-has-nul"; + +/** A subject this build cannot derive an id from, named by what was wrong with it. */ +export class FactoryRunSubjectError extends Error { + override name = "FactoryRunSubjectError"; + + constructor( + readonly reason: FactoryRunSubjectFailure, + detail: string, + ) { + super(`this GitHub subject cannot address a factory run: ${detail}`); + } +} + +/** The exact GitHub subject one factory run is a run of. */ +export interface FactoryRunSubject { + /** + * The canonical GitHub authority: a lowercase DNS hostname, plus `:` and a + * port when that port is not the scheme's default. + */ + readonly authority: string; + /** + * The exact string GitHub's GraphQL API returned for this issue. + * + * Compared byte for byte. It is an opaque provider identity, and normalizing + * one would be inventing a second. + */ + readonly issueNodeId: string; +} + +/** The port `https` implies, and therefore the one an authority may not spell out. */ +const DEFAULT_PORT = 443; + +/** + * A hostname the DNS grammar admits: labels of letters, digits and hyphens, + * each starting and ending with an alphanumeric, separated by dots. + * + * Deliberately not a URL parse. A parser would accept — and silently discard — + * the parts an authority may not carry, and the point here is to refuse them. + */ +const HOSTNAME = + /^(?=.{1,253}$)[a-z0-9](?:[a-z0-9-]*[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)*$/; + +/** + * Normalize what an operator configured into the one spelling this hash uses. + * + * Case folding is the only transformation. Everything else an authority must not + * contain is refused rather than stripped: a value that had to be repaired to be + * usable is a value somebody meant differently, and two spellings that both + * became one authority would be two runs quietly becoming one. + */ +export function canonicalGitHubAuthority(value: string): string { + if (value === "") { + throw new FactoryRunSubjectError("authority-empty", "the authority is empty"); + } + if (/\s/.test(value)) { + throw new FactoryRunSubjectError( + "authority-has-whitespace", + "the authority contains whitespace", + ); + } + if (value.includes("//") || /^[a-zA-Z][a-zA-Z0-9+.-]*:\/\//.test(value)) { + throw new FactoryRunSubjectError( + "authority-has-scheme", + "the authority carries a scheme; write the host alone", + ); + } + if (value.includes("@")) { + throw new FactoryRunSubjectError( + "authority-has-userinfo", + "the authority carries user information", + ); + } + if (value.includes("#")) { + throw new FactoryRunSubjectError("authority-has-fragment", "the authority carries a fragment"); + } + if (value.includes("?")) { + throw new FactoryRunSubjectError("authority-has-query", "the authority carries a query"); + } + if (value.includes("/")) { + throw new FactoryRunSubjectError( + "authority-has-path", + "the authority carries a path or a trailing separator", + ); + } + + const folded = value.toLowerCase(); + const separator = folded.lastIndexOf(":"); + const host = separator === -1 ? folded : folded.slice(0, separator); + const port = separator === -1 ? undefined : folded.slice(separator + 1); + + if (!HOSTNAME.test(host)) { + throw new FactoryRunSubjectError("authority-malformed-host", "the host is not a DNS hostname"); + } + if (port === undefined) { + return host; + } + if (!/^[0-9]{1,5}$/.test(port)) { + throw new FactoryRunSubjectError("authority-malformed-port", "the port is not a number"); + } + const numeric = Number(port); + if (numeric < 1 || numeric > 65535) { + throw new FactoryRunSubjectError("authority-malformed-port", "the port is out of range"); + } + if (numeric === DEFAULT_PORT) { + throw new FactoryRunSubjectError( + "authority-default-port", + "the default port is written out; omit it so one deployment has one spelling", + ); + } + return `${host}:${numeric}`; +} + +/** Hold a node id to what a retained identity has to be, and change nothing about it. */ +export function admitIssueNodeId(value: string): string { + if (value === "") { + throw new FactoryRunSubjectError("node-id-empty", "the issue node id is empty"); + } + if (value.includes("\0")) { + throw new FactoryRunSubjectError("node-id-has-nul", "the issue node id contains a NUL"); + } + return value; +} + +/** + * The exact bytes the digest is taken over. + * + * `github-issue-v1`, NUL, the canonical authority, NUL, the node id — all + * UTF-8. The NULs are separators the inputs cannot contain, so no pair of + * (authority, node id) can be rearranged into another pair with the same bytes. + */ +export function factoryRunIdPreimage(subject: FactoryRunSubject): Uint8Array { + const encoder = new TextEncoder(); + const scheme = encoder.encode(SCHEME); + const authority = encoder.encode(subject.authority); + const node = encoder.encode(subject.issueNodeId); + const bytes = new Uint8Array(scheme.length + 1 + authority.length + 1 + node.length); + let at = 0; + bytes.set(scheme, at); + at += scheme.length; + bytes[at] = 0; + at += 1; + bytes.set(authority, at); + at += authority.length; + bytes[at] = 0; + at += 1; + bytes.set(node, at); + return bytes; +} + +/** Lowercase unpadded RFC 4648 Base32 of exactly these bytes. */ +export function base32Unpadded(bytes: Uint8Array): string { + let out = ""; + let buffer = 0; + let bits = 0; + for (const byte of bytes) { + buffer = (buffer << 8) | byte; + bits += 8; + while (bits >= 5) { + bits -= 5; + out += BASE32_ALPHABET[(buffer >> bits) & 31]; + } + } + if (bits > 0) { + out += BASE32_ALPHABET[(buffer << (5 - bits)) & 31]; + } + return out; +} + +/** + * Normalize a subject, refusing anything this build cannot address a run from. + * + * Separate from the derivation so a caller can admit a subject before it has + * anywhere to put the answer — which is what an admission check needs, and what + * a later story comparing a reread subject against a retained one needs too. + */ +export function admitFactoryRunSubject(subject: FactoryRunSubject): FactoryRunSubject { + return { + authority: canonicalGitHubAuthority(subject.authority), + issueNodeId: admitIssueNodeId(subject.issueNodeId), + }; +} + +/** + * The public run id for one GitHub issue. + * + * The subject is admitted first, so a malformed authority or node id is refused + * before any digest exists and long before anything looks for an owner to route + * it to. + */ +export function* deriveFactoryRunId(subject: FactoryRunSubject): Operation { + const admitted = admitFactoryRunSubject(subject); + const digest = yield* until( + crypto.subtle.digest("SHA-256", factoryRunIdPreimage(admitted) as BufferSource), + ); + return base32Unpadded(new Uint8Array(digest)); +} diff --git a/packages/workflow/tests/factory-run-id.test.ts b/packages/workflow/tests/factory-run-id.test.ts new file mode 100644 index 000000000..37a2759ab --- /dev/null +++ b/packages/workflow/tests/factory-run-id.test.ts @@ -0,0 +1,159 @@ +/** + * Tier WRH — the run id one GitHub issue is addressed by. + * + * The derivation is the whole of "one issue, one run": every host that admits + * the same issue has to arrive at the same 52 characters without asking anybody, + * and no value that moves while the work is going on may take part. The fixed + * vectors below were computed independently of this implementation, which is + * what makes them evidence that a second implementation would agree rather than + * a restatement of what this code happens to do. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { + admitFactoryRunSubject, + base32Unpadded, + canonicalGitHubAuthority, + deriveFactoryRunId, + FACTORY_RUN_ID_LENGTH, + factoryRunIdPreimage, + FactoryRunSubjectError, +} from "@executablemd/workflow"; + +/** An opaque node id of the shape GitHub's GraphQL API returns for an issue. */ +const NODE = "I_kwDOABCD12M5abcdef"; + +/** + * Computed outside this implementation, from the specified bytes. + * + * `sha256("github-issue-v1" || 0x00 || authority || 0x00 || nodeId)`, then + * lowercase unpadded RFC 4648 Base32 over all 32 bytes. + */ +const VECTORS = [ + { + authority: "github.com", + issueNodeId: NODE, + runId: "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa", + }, + { + authority: "github.example.com:8443", + issueNodeId: NODE, + runId: "h7dgqsvqzv4p5k2hp2zebemci65qhinpdkxwk5d4caglndiw2xya", + }, + { + authority: "github.com", + issueNodeId: "I_kwDOABCD12M5abcdeg", + runId: "unydmnzwowpjcoua2tyza2topyivbbnm65ddklfjgs5yvlqiwqvq", + }, +] as const; + +function reason(body: () => unknown): string { + try { + body(); + } catch (error) { + if (error instanceof FactoryRunSubjectError) { + return error.reason; + } + throw error; + } + throw new Error("expected a FactoryRunSubjectError"); +} + +describe("the factory run id", () => { + it("derives the specified bytes for known subjects", function* () { + for (const vector of VECTORS) { + const runId = yield* deriveFactoryRunId({ + authority: vector.authority, + issueNodeId: vector.issueNodeId, + }); + expect(runId).toEqual(vector.runId); + expect(runId.length).toEqual(FACTORY_RUN_ID_LENGTH); + expect(/^[a-z2-7]{52}$/.test(runId)).toEqual(true); + } + }); + + it("derives the same id twice for one subject", function* () { + const once = yield* deriveFactoryRunId({ authority: "github.com", issueNodeId: NODE }); + const again = yield* deriveFactoryRunId({ authority: "GITHUB.COM", issueNodeId: NODE }); + expect(again).toEqual(once); + }); + + it("separates the authority from the node id", function* () { + // Without the NUL separators these two subjects would share a preimage. + const left = yield* deriveFactoryRunId({ authority: "github.com", issueNodeId: "ab" }); + const right = yield* deriveFactoryRunId({ authority: "github.co", issueNodeId: "mab" }); + expect(left).not.toEqual(right); + }); + + it("writes the scheme tag, both separators and both inputs", function* () { + const bytes = factoryRunIdPreimage({ authority: "github.com", issueNodeId: "x" }); + expect(new TextDecoder().decode(bytes)).toEqual("github-issue-v1\0github.com\0x"); + expect([...bytes].filter((byte) => byte === 0).length).toEqual(2); + }); + + it("folds case and keeps a non-default port", function* () { + expect(canonicalGitHubAuthority("GitHub.Com")).toEqual("github.com"); + expect(canonicalGitHubAuthority("GitHub.Example.COM:8443")).toEqual("github.example.com:8443"); + }); + + it("refuses every part an authority may not carry", function* () { + expect(reason(() => canonicalGitHubAuthority(""))).toEqual("authority-empty"); + expect(reason(() => canonicalGitHubAuthority("https://github.com"))).toEqual( + "authority-has-scheme", + ); + expect(reason(() => canonicalGitHubAuthority("user@github.com"))).toEqual( + "authority-has-userinfo", + ); + expect(reason(() => canonicalGitHubAuthority("github.com/octo"))).toEqual("authority-has-path"); + expect(reason(() => canonicalGitHubAuthority("github.com/"))).toEqual("authority-has-path"); + expect(reason(() => canonicalGitHubAuthority("github.com?a=b"))).toEqual("authority-has-query"); + expect(reason(() => canonicalGitHubAuthority("github.com#top"))).toEqual( + "authority-has-fragment", + ); + expect(reason(() => canonicalGitHubAuthority("git hub.com"))).toEqual( + "authority-has-whitespace", + ); + expect(reason(() => canonicalGitHubAuthority("-github.com"))).toEqual( + "authority-malformed-host", + ); + expect(reason(() => canonicalGitHubAuthority("github.com:https"))).toEqual( + "authority-malformed-port", + ); + expect(reason(() => canonicalGitHubAuthority("github.com:0"))).toEqual( + "authority-malformed-port", + ); + expect(reason(() => canonicalGitHubAuthority("github.com:70000"))).toEqual( + "authority-malformed-port", + ); + }); + + it("refuses a default port written out, so one deployment has one spelling", function* () { + expect(reason(() => canonicalGitHubAuthority("github.com:443"))).toEqual( + "authority-default-port", + ); + }); + + it("compares a node id byte for byte", function* () { + const subject = admitFactoryRunSubject({ authority: "github.com", issueNodeId: "Ab_C" }); + expect(subject.issueNodeId).toEqual("Ab_C"); + expect( + reason(() => admitFactoryRunSubject({ authority: "github.com", issueNodeId: "" })), + ).toEqual("node-id-empty"); + expect( + reason(() => admitFactoryRunSubject({ authority: "github.com", issueNodeId: "a\0b" })), + ).toEqual("node-id-has-nul"); + }); + + it("encodes Base32 to the RFC 4648 alphabet without padding", function* () { + // RFC 4648 §10 test vectors, lowercased and unpadded. + const encode = (text: string) => base32Unpadded(new TextEncoder().encode(text)); + expect(encode("")).toEqual(""); + expect(encode("f")).toEqual("my"); + expect(encode("fo")).toEqual("mzxq"); + expect(encode("foo")).toEqual("mzxw6"); + expect(encode("foob")).toEqual("mzxw6yq"); + expect(encode("fooba")).toEqual("mzxw6ytb"); + expect(encode("foobar")).toEqual("mzxw6ytboi"); + }); +}); diff --git a/packages/workflow/tests/host-neutrality.test.ts b/packages/workflow/tests/host-neutrality.test.ts new file mode 100644 index 000000000..d178dad81 --- /dev/null +++ b/packages/workflow/tests/host-neutrality.test.ts @@ -0,0 +1,104 @@ +/** + * Tier WRH — what the shared package may know about a host. + * + * `@executablemd/workflow` names no provider. That claim is what lets a second + * host implement the same lifecycle without the Deno entrypoint being loaded at + * all, and it is worth exactly as much as the imports underneath it: one + * `node:sqlite` or `cloudflare:` specifier in a shared module, or one + * `typeof Deno` test, and every module that resolves through it inherits a host. + * + * So this reads the source rather than describing it. It walks the modules a + * consumer reaches through the package root and fails on anything that names a + * runtime — the runtime-named entrypoints and their own subtrees excepted, + * because installing host behavior is what those are for. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { readTextFile, walk } from "@effectionx/fs"; +import { each } from "effection"; +import type { Operation } from "effection"; +import { join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; + +const PACKAGE = fileURLToPath(new URL("..", import.meta.url)); + +/** + * The subtrees that are allowed to know a host, because naming one is their job. + * + * `vendor` is pinned upstream source whose drift verifier owns its bytes. + */ +const RUNTIME_OWNED = ["deno.ts", "cloudflare.ts", "src/deno", "src/cloudflare", "vendor"]; + +/** Specifiers only a host adapter may import. */ +const HOST_SPECIFIERS = [ + "node:sqlite", + "node:fs", + "node:os", + "node:child_process", + "cloudflare:workers", + "cloudflare:test", + "@cloudflare/", +]; + +/** Ways a module could ask which runtime it is running under. */ +const RUNTIME_DETECTION = [ + /\btypeof\s+Deno\b/, + /\btypeof\s+Bun\b/, + /\bnavigator\s*\.\s*userAgent\b/, + /\bprocess\s*\.\s*versions\s*\.\s*bun\b/, + /\bglobalThis\s*\.\s*Deno\b/, + /\bglobalThis\s*\.\s*Bun\b/, +]; + +function* sharedModules(): Operation { + const owned = RUNTIME_OWNED.map((entry) => join(PACKAGE, entry)); + const found: string[] = []; + for (const entry of yield* each(walk(PACKAGE, { includeDirs: false }))) { + const path = entry.path; + const exempt = owned.some((root) => path === root || path.startsWith(`${root}/`)); + const generated = ["/node_modules/", "/tests/", "/npm/"].some((part) => path.includes(part)); + if (!exempt && !generated && path.endsWith(".ts") && !path.endsWith(".d.ts")) { + found.push(path); + } + yield* each.next(); + } + return found.toSorted(); +} + +function* offenders(check: (source: string) => boolean): Operation { + const named: string[] = []; + for (const path of yield* sharedModules()) { + const source = yield* readTextFile(path); + if (check(source)) { + named.push(relative(PACKAGE, path)); + } + } + return named; +} + +describe("the shared workflow package", () => { + it("finds the modules it is making a claim about", function* () { + const modules = yield* sharedModules(); + expect(modules.length > 20).toEqual(true); + expect(modules.some((path) => path.endsWith("/src/lifecycle/execution.ts"))).toEqual(true); + expect(modules.some((path) => path.includes("/src/deno/"))).toEqual(false); + }); + + it("imports no host-owned specifier outside a runtime-named entrypoint", function* () { + const named = yield* offenders((source) => + HOST_SPECIFIERS.some( + (specifier) => + source.includes(`from "${specifier}`) || source.includes(`import("${specifier}`), + ), + ); + expect(named).toEqual([]); + }); + + it("asks no module which runtime it is running under", function* () { + const named = yield* offenders((source) => + RUNTIME_DETECTION.some((pattern) => pattern.test(source)), + ); + expect(named).toEqual([]); + }); +}); From 95a8f0e219971de37175d64c4a80539f615b9d19 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 15:46:22 -0400 Subject: [PATCH 07/81] =?UTF-8?q?=F0=9F=9A=9A=20Keep=20the=20GitHub-named?= =?UTF-8?q?=20run-id=20derivation=20out=20of=20the=20neutral=20surface=20(?= =?UTF-8?q?#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `DLC13` in packages/workflow/tests/workspace-effect.test.ts scans `packages/workflow/mod.ts` and `packages/workflow/src/**` (minus the Deno adapter) and refuses a set of names that includes `GitHub`. Its reason is that the shared external-effect boundary exists so any Git host can be adapted to it, and "the first adapter naming itself in a shared contract is how a neutral surface quietly becomes one provider's." 8f27b04 put `deriveFactoryRunId()` under `packages/workflow/src/factory/` and exported it from the package root, which broke that test — correctly. The derivation names GitHub in its scheme tag, its authority rule and its node id, because the software factory is a GitHub product by definition. Move it to `packages/cli/src/factory-run-id.ts`, beside the CLI's existing `github-issues-config.ts` where naming GitHub is already legitimate, and take it back out of `mod.ts`. The module and its tests are unchanged otherwise. The host-neutrality scan added in 8f27b04 no longer exempts `cloudflare.ts` or `src/cloudflare`: neither exists yet, and a scan that exempts a path nothing occupies is a claim about a boundary nobody drew. Where the derivation lives permanently is not settled by this commit. The Durable Object owner needs it too, and F1 cannot place it until DLC13's scanned set is reconciled with a Cloudflare adapter subtree. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- .../factory/run-id.ts => cli/src/factory-run-id.ts} | 0 .../{workflow => cli}/tests/factory-run-id.test.ts | 2 +- packages/workflow/mod.ts | 12 ------------ packages/workflow/tests/host-neutrality.test.ts | 2 +- 4 files changed, 2 insertions(+), 14 deletions(-) rename packages/{workflow/src/factory/run-id.ts => cli/src/factory-run-id.ts} (100%) rename packages/{workflow => cli}/tests/factory-run-id.test.ts (99%) diff --git a/packages/workflow/src/factory/run-id.ts b/packages/cli/src/factory-run-id.ts similarity index 100% rename from packages/workflow/src/factory/run-id.ts rename to packages/cli/src/factory-run-id.ts diff --git a/packages/workflow/tests/factory-run-id.test.ts b/packages/cli/tests/factory-run-id.test.ts similarity index 99% rename from packages/workflow/tests/factory-run-id.test.ts rename to packages/cli/tests/factory-run-id.test.ts index 37a2759ab..8af3467a2 100644 --- a/packages/workflow/tests/factory-run-id.test.ts +++ b/packages/cli/tests/factory-run-id.test.ts @@ -19,7 +19,7 @@ import { FACTORY_RUN_ID_LENGTH, factoryRunIdPreimage, FactoryRunSubjectError, -} from "@executablemd/workflow"; +} from "../src/factory-run-id.ts"; /** An opaque node id of the shape GitHub's GraphQL API returns for an issue. */ const NODE = "I_kwDOABCD12M5abcdef"; diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index e0d6fabf3..18a0a049c 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -314,18 +314,6 @@ export type { WorkflowLifecycleApi, WorkflowLifecycleSnapshot, } from "./src/lifecycle/api.ts"; -export { - admitFactoryRunSubject, - admitIssueNodeId, - base32Unpadded, - canonicalGitHubAuthority, - deriveFactoryRunId, - FACTORY_RUN_ID_LENGTH, - factoryRunIdPreimage, - FactoryRunSubjectError, -} from "./src/factory/run-id.ts"; -export type { FactoryRunSubject, FactoryRunSubjectFailure } from "./src/factory/run-id.ts"; - // What a trusted host needs to move a run's lifecycle. These describe what any // host's lifecycle does rather than what one adapter retains, so this entrypoint // owns their meaning; `./deno` re-exports them for source compatibility and a diff --git a/packages/workflow/tests/host-neutrality.test.ts b/packages/workflow/tests/host-neutrality.test.ts index d178dad81..1487d4baf 100644 --- a/packages/workflow/tests/host-neutrality.test.ts +++ b/packages/workflow/tests/host-neutrality.test.ts @@ -28,7 +28,7 @@ const PACKAGE = fileURLToPath(new URL("..", import.meta.url)); * * `vendor` is pinned upstream source whose drift verifier owns its bytes. */ -const RUNTIME_OWNED = ["deno.ts", "cloudflare.ts", "src/deno", "src/cloudflare", "vendor"]; +const RUNTIME_OWNED = ["deno.ts", "src/deno", "vendor"]; /** Specifiers only a host adapter may import. */ const HOST_SPECIFIERS = [ From 61b63fcee831bb99943a63a9a48ac2224f27f98c Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 16:33:51 -0400 Subject: [PATCH 08/81] =?UTF-8?q?=F0=9F=9A=9A=20Give=20the=20software=20fa?= =?UTF-8?q?ctory=20its=20own=20package=20subpath=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies the Architect's answers to Q1 and Q2 in .vscode/698/698-a-1.md. Q1. `DLC13` now excludes `packages/workflow/src/cloudflare/**` and `packages/workflow/src/software-factory/**`, each with its own reason. The two implementation subtrees are runtime-owned — scanning an adapter for the vocabulary of the runtime it adapts is a category error, and Code Rule 12 puts host behavior behind exactly those names. The software factory is the other kind of exception: not a runtime adapter, still held to the host-import and runtime-detection rules by `host-neutrality.test.ts`, and allowed only the product vocabulary, because §1.1 of the factory specification makes GitHub the subject matter of that contract rather than one provider capturing a neutral boundary. `packages/workflow/mod.ts` and every shared module stay scanned, and no forbidden word gained an exception. Beside the existing Deno assertion, `found` is now checked to contain no `/src/cloudflare/` and no `/src/software-factory/` path, so an exclusion that matched nothing or matched too little cannot pass quietly. `host-neutrality.test.ts` gains `cloudflare.ts` and `src/cloudflare` in `RUNTIME_OWNED` and asserts the Cloudflare subtree is absent from what it scans. `software-factory.ts` is deliberately not runtime-owned there: it uses the cross-runtime Web primitives and is checked like any shared module, which the new assertion that `src/software-factory/run-id.ts` is among the scanned modules now proves. Q2. The derivation moves from `packages/cli/src/factory-run-id.ts` to `packages/workflow/src/software-factory/run-id.ts`, published as `@executablemd/workflow/software-factory` from both manifests. One implementation, reachable by the provider host and by F2's GitHub intake without either carrying a hash that has to agree byte for byte with the other's. The package root does not re-export it — that neutrality is the point — and no forwarding copy is left in the CLI. Its tests move to the workflow package and import the published subpath, so the export map is what they exercise. Q3 needs no change yet; it governs §8 and §9, which are unstarted. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/deno.json | 1 + packages/workflow/package.json | 1 + packages/workflow/software-factory.ts | 46 +++++++++++++++++++ .../src/software-factory/run-id.ts} | 0 .../workflow/tests/host-neutrality.test.ts | 10 +++- .../tests/software-factory-run-id.test.ts} | 2 +- .../workflow/tests/workspace-effect.test.ts | 34 ++++++++++++-- 7 files changed, 87 insertions(+), 7 deletions(-) create mode 100644 packages/workflow/software-factory.ts rename packages/{cli/src/factory-run-id.ts => workflow/src/software-factory/run-id.ts} (100%) rename packages/{cli/tests/factory-run-id.test.ts => workflow/tests/software-factory-run-id.test.ts} (99%) diff --git a/packages/workflow/deno.json b/packages/workflow/deno.json index ed6dfab38..ab4ae02a3 100644 --- a/packages/workflow/deno.json +++ b/packages/workflow/deno.json @@ -5,6 +5,7 @@ "exports": { ".": "./mod.ts", "./deno": "./deno.ts", + "./software-factory": "./software-factory.ts", "./credential-helper": "./src/deno/composition/credential-helper.ts" }, "publish": { diff --git a/packages/workflow/package.json b/packages/workflow/package.json index bb7e1758e..5c2043bde 100644 --- a/packages/workflow/package.json +++ b/packages/workflow/package.json @@ -6,6 +6,7 @@ "exports": { ".": "./mod.ts", "./deno": "./deno.ts", + "./software-factory": "./software-factory.ts", "./credential-helper": "./src/deno/composition/credential-helper.ts" }, "dependencies": { diff --git a/packages/workflow/software-factory.ts b/packages/workflow/software-factory.ts new file mode 100644 index 000000000..5f5f69320 --- /dev/null +++ b/packages/workflow/software-factory.ts @@ -0,0 +1,46 @@ +/** + * @module + * + * The GitHub Actions software factory's public identity rule. + * + * This subpath is deliberately not the package root. `@executablemd/workflow` + * names no provider — that is what lets a second host implement the same + * lifecycle — and the derivation here names GitHub in its scheme tag, its + * authority rule and its node id, because the software factory is a GitHub + * product by definition rather than one adapter of a neutral boundary. + * + * So the two surfaces are separate on purpose. Anything that needs the factory's + * own contract asks for it by name: + * + * ```ts + * import { deriveFactoryRunId } from "@executablemd/workflow/software-factory"; + * + * const runId = yield* deriveFactoryRunId({ + * authority: "github.com", + * issueNodeId: node, + * }); + * ``` + * + * One issue is one durable run, so this is the whole of "one issue, one run": + * every host that admits the same issue arrives at the same 52 characters + * without asking anybody. It is specified in + * `specs/github-actions-software-factory-spec.md` §1.1 and restated in + * `specs/workflow-spec.md` §9.1. + * + * Nothing here is runtime-specific. It uses the cross-runtime Web primitives — + * `TextEncoder` and `crypto.subtle` — and names no host, so the provider host + * and a GitHub intake reach the same single implementation rather than each + * carrying a hash that has to agree byte for byte with the other's. + */ + +export { + admitFactoryRunSubject, + admitIssueNodeId, + base32Unpadded, + canonicalGitHubAuthority, + deriveFactoryRunId, + FACTORY_RUN_ID_LENGTH, + factoryRunIdPreimage, + FactoryRunSubjectError, +} from "./src/software-factory/run-id.ts"; +export type { FactoryRunSubject, FactoryRunSubjectFailure } from "./src/software-factory/run-id.ts"; diff --git a/packages/cli/src/factory-run-id.ts b/packages/workflow/src/software-factory/run-id.ts similarity index 100% rename from packages/cli/src/factory-run-id.ts rename to packages/workflow/src/software-factory/run-id.ts diff --git a/packages/workflow/tests/host-neutrality.test.ts b/packages/workflow/tests/host-neutrality.test.ts index 1487d4baf..94074295a 100644 --- a/packages/workflow/tests/host-neutrality.test.ts +++ b/packages/workflow/tests/host-neutrality.test.ts @@ -26,9 +26,13 @@ const PACKAGE = fileURLToPath(new URL("..", import.meta.url)); /** * The subtrees that are allowed to know a host, because naming one is their job. * - * `vendor` is pinned upstream source whose drift verifier owns its bytes. + * The runtime-named entrypoints and their implementation subtrees, and nothing + * else. `software-factory.ts` is deliberately absent: it is product-specific + * rather than host-specific, uses the cross-runtime Web primitives, and is held + * to these rules like any shared module. `vendor` is pinned upstream source + * whose drift verifier owns its bytes. */ -const RUNTIME_OWNED = ["deno.ts", "src/deno", "vendor"]; +const RUNTIME_OWNED = ["deno.ts", "cloudflare.ts", "src/deno", "src/cloudflare", "vendor"]; /** Specifiers only a host adapter may import. */ const HOST_SPECIFIERS = [ @@ -82,7 +86,9 @@ describe("the shared workflow package", () => { const modules = yield* sharedModules(); expect(modules.length > 20).toEqual(true); expect(modules.some((path) => path.endsWith("/src/lifecycle/execution.ts"))).toEqual(true); + expect(modules.some((path) => path.endsWith("/src/software-factory/run-id.ts"))).toEqual(true); expect(modules.some((path) => path.includes("/src/deno/"))).toEqual(false); + expect(modules.some((path) => path.includes("/src/cloudflare/"))).toEqual(false); }); it("imports no host-owned specifier outside a runtime-named entrypoint", function* () { diff --git a/packages/cli/tests/factory-run-id.test.ts b/packages/workflow/tests/software-factory-run-id.test.ts similarity index 99% rename from packages/cli/tests/factory-run-id.test.ts rename to packages/workflow/tests/software-factory-run-id.test.ts index 8af3467a2..faff85d3b 100644 --- a/packages/cli/tests/factory-run-id.test.ts +++ b/packages/workflow/tests/software-factory-run-id.test.ts @@ -19,7 +19,7 @@ import { FACTORY_RUN_ID_LENGTH, factoryRunIdPreimage, FactoryRunSubjectError, -} from "../src/factory-run-id.ts"; +} from "@executablemd/workflow/software-factory"; /** An opaque node id of the shape GitHub's GraphQL API returns for an issue. */ const NODE = "I_kwDOABCD12M5abcdef"; diff --git a/packages/workflow/tests/workspace-effect.test.ts b/packages/workflow/tests/workspace-effect.test.ts index 25a352056..b0f2175af 100644 --- a/packages/workflow/tests/workspace-effect.test.ts +++ b/packages/workflow/tests/workspace-effect.test.ts @@ -686,10 +686,31 @@ describe("Tier DLC — Workspace coordination selection", () => { "packages/durable-streams/*.ts", ], // Whole packages rather than named modules, so a coordination module - // added later is covered without this list being remembered. The single - // exception carries its reason: the HTTP stream is a client for a remote - // durable stream and reaches the platform's own `fetch`. - exclude: ["packages/workflow/src/deno/**", "packages/durable-streams/http-stream.ts"], + // added later is covered without this list being remembered. Each + // exception carries its reason. + // + // The two implementation subtrees are runtime-owned: scanning an adapter + // for the vocabulary of the runtime it adapts is a category error, and + // Code Rule 12 puts host behavior behind exactly these names. The package + // root and every shared module stay covered, so a host name reaching the + // neutral surface is still a failure. + // + // The software factory is the other kind of exception. It is not a + // runtime adapter and is still held to the host-import and + // runtime-detection rules by `host-neutrality.test.ts`; what it is + // allowed is the product vocabulary, because + // `specs/github-actions-software-factory-spec.md` §1.1 makes GitHub the + // subject matter of that contract rather than one provider capturing a + // neutral boundary. + // + // The HTTP stream is a client for a remote durable stream and reaches the + // platform's own `fetch`. + exclude: [ + "packages/workflow/src/deno/**", + "packages/workflow/src/cloudflare/**", + "packages/workflow/src/software-factory/**", + "packages/durable-streams/http-stream.ts", + ], })) .map((entry) => entry.path) .sort(); @@ -729,7 +750,12 @@ describe("Tier DLC — Workspace coordination selection", () => { "packages/workflow/src/workspace/effect.ts", ]), ); + // An exclusion that matched nothing would scan the adapter and fail on its + // own vocabulary; one that matched too little would scan part of it. Both + // subtrees are checked, so a malformed pattern cannot pass quietly. expect(found.some((path) => path.includes("/src/deno/"))).toBe(false); + expect(found.some((path) => path.includes("/src/cloudflare/"))).toBe(false); + expect(found.some((path) => path.includes("/src/software-factory/"))).toBe(false); const crossings: Record = {}; const unread: string[] = []; From 937a3887c040dd30046ca5fc496c4e1e8bc6193d Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 16:43:17 -0400 Subject: [PATCH 09/81] =?UTF-8?q?=F0=9F=93=A6=20Add=20the=20Cloudflare=20W?= =?UTF-8?q?orkers=20test=20runtime=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan §13. `@cloudflare/vitest-plugin@1.1.3` and `vitest@4.1.11` as root development dependencies, which is what brings workerd in — plan §12 requires the real Durable Object namespace, real SQLite storage and real WebSocket admission, and an in-process fake is not evidence for any of them. Both lock layouts are updated in the documented order: the pnpm add, then `deno install --frozen=false`, then `deno task setup`. `pnpm-lock.yaml` gains 1231 lines and deletes none; `deno.lock` gains the npm graph for the same packages. Two things worth knowing for anyone reproducing this. The lockfile is `lockfileVersion: '9.0'` and `package.json` declares `pnpm@9.15.0`, but the pnpm on PATH here is 7.5.0, which cannot read a v9 lockfile — it warns "Ignoring broken lockfile" and rewrites it as `lockfileVersion: 5.4`. That downgrade happened once and was reverted; this commit was produced with the declared version through `corepack pnpm`. Anything that shells out to a bare `pnpm`, `deno task setup` included, needs 9.15.0 ahead of 7.5.0 on PATH or it will silently downgrade the lockfile again. pnpm 9 also re-sorts `package.json` dependency keys, which is why `effection`, `mdast-util-to-string` and `zod` move. That is the package manager's own canonical ordering; hand-restoring it would only be undone by the next install. `deno task setup` still exits 1 at its last step, `build:web`, with `Module not found "file:///…/bundle"` from `deno bundle --packages=bundle`. That is not this change: the identical argv succeeds when run directly and under a preflight-shaped parent, it fails only through `@effectionx/process`'s `exec()`, `packages/web/generated/` has never existed in this checkout, and the browser bundle's module graph does not reach vitest or workerd. `deno task check`, `deno task lint` and the focused suites all pass on the new dependency state. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- deno.lock | 560 +++++++++++++++++++++- package.json | 10 +- pnpm-lock.yaml | 1231 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1788 insertions(+), 13 deletions(-) diff --git a/deno.lock b/deno.lock index b255615bd..6cf3f3547 100644 --- a/deno.lock +++ b/deno.lock @@ -41,6 +41,7 @@ "npm:@agentclientprotocol/sdk@1.3.0": "1.3.0_zod@4.4.3", "npm:@babel/core@^7.28.0": "7.29.7", "npm:@babel/preset-react@^7.27.1": "7.29.7_@babel+core@7.29.7", + "npm:@cloudflare/vitest-plugin@1.1.3": "1.1.3_@vitest+runner@4.1.11_@vitest+snapshot@4.1.11_vitest@4.1.11__@opentelemetry+api@1.9.1__@types+node@24.13.3__vite@7.3.6___@types+node@24.13.3___tsx@4.23.1_@opentelemetry+api@1.9.1_@types+node@24.13.3_vite@7.3.6__@types+node@24.13.3__tsx@4.23.1", "npm:@durable-streams/client@~0.2.2": "0.2.6", "npm:@durable-streams/server@~0.3.8": "0.3.8", "npm:@effectionx/context-api@0.6.0": "0.6.0_effection@4.1.0", @@ -111,6 +112,7 @@ "npm:unist-util-select@5": "5.1.0", "npm:vite@^7.1.3": "7.3.6_@types+node@24.13.3_tsx@4.23.1", "npm:vite@^7.1.4": "7.3.6_@types+node@24.13.3_tsx@4.23.1", + "npm:vitest@4.1.11": "4.1.11_@opentelemetry+api@1.9.1_@types+node@24.13.3_vite@7.3.6__@types+node@24.13.3__tsx@4.23.1", "npm:zod@4": "4.4.3", "npm:zod@^4.3.6": "4.4.3" }, @@ -312,7 +314,7 @@ "debug", "gensync", "json5", - "semver" + "semver@6.3.1" ] }, "@babel/generator@7.29.7": { @@ -321,7 +323,7 @@ "@babel/parser", "@babel/types", "@jridgewell/gen-mapping", - "@jridgewell/trace-mapping", + "@jridgewell/trace-mapping@0.3.31", "jsesc" ] }, @@ -338,7 +340,7 @@ "@babel/helper-validator-option", "browserslist", "lru-cache", - "semver" + "semver@6.3.1" ] }, "@babel/helper-globals@7.29.7": { @@ -481,9 +483,66 @@ "sisteransi" ] }, + "@cloudflare/kv-asset-handler@0.5.0": { + "integrity": "sha512-jxQYkj8dSIzc0cD6cMMNdOc1UVjqSqu8BZdor5s8cGjW2I8BjODt/kWPVdY+u9zj3ms75Q5qaZgnxUad83+eAg==" + }, + "@cloudflare/unenv-preset@2.16.1_unenv@2.0.0-rc.24_workerd@1.20260831.1": { + "integrity": "sha512-ECxObrMfyTl5bhQf/lZCXwo5G6xX9IAUo+nDMKK4SZ8m4Jvvxp52vilxyySSWh2YTZz8+HQ07qGH/2rEom1vDw==", + "dependencies": [ + "unenv", + "workerd" + ], + "optionalPeers": [ + "workerd" + ] + }, + "@cloudflare/vitest-plugin@1.1.3_@vitest+runner@4.1.11_@vitest+snapshot@4.1.11_vitest@4.1.11__@opentelemetry+api@1.9.1__@types+node@24.13.3__vite@7.3.6___@types+node@24.13.3___tsx@4.23.1_@opentelemetry+api@1.9.1_@types+node@24.13.3_vite@7.3.6__@types+node@24.13.3__tsx@4.23.1": { + "integrity": "sha512-ED1Rkaq5Wr5rCeHXpLoDyV4WGJzD0Ju0clM8jS7Hj+wjj/CwaMHeb8DXzUfUQPiHW9rTgwcttPPQnzau2kp6Jg==", + "dependencies": [ + "@vitest/runner", + "@vitest/snapshot", + "cjs-module-lexer", + "esbuild@0.28.1", + "miniflare", + "vitest", + "wrangler", + "zod" + ] + }, + "@cloudflare/workerd-darwin-64@1.20260831.1": { + "integrity": "sha512-oyZ8xhu+gYTvoxV/sn6NRmTHK95RhEO1Dk54/6oPb0Uu70w7ZeRoCjkJ5aNmfS8Vrkdu6+oL0HNg6EcC61uQ2Q==", + "os": ["darwin"], + "cpu": ["x64"] + }, + "@cloudflare/workerd-darwin-arm64@1.20260831.1": { + "integrity": "sha512-s6Go53KPnoXZ1sTGBZ3en3otfHDuMPJhiwXMYWU21JkJQkpoeRt6HFUwM0GPhK3YhXWm+8baGMvCGZYS/KA9eA==", + "os": ["darwin"], + "cpu": ["arm64"] + }, + "@cloudflare/workerd-linux-64@1.20260831.1": { + "integrity": "sha512-WxNKBgjKgeYTolW3yl1Lt3Lu67UlxdeyzWYi9MIqrKBdyQcz+UNG36RevSBf8rv1sTWapRW234VX2keZ+wXapA==", + "os": ["linux"], + "cpu": ["x64"] + }, + "@cloudflare/workerd-linux-arm64@1.20260831.1": { + "integrity": "sha512-JTF9+9clUT3gaCq7Xnmd+Q/wEMaitpngSTOec/Ffb/r3xexA9XwNJVFSOKfk6q61flHGjAYJ4H9B7Mu5Qur49w==", + "os": ["linux"], + "cpu": ["arm64"] + }, + "@cloudflare/workerd-windows-64@1.20260831.1": { + "integrity": "sha512-do+KDYw0PABwsrKUQIccWBZB70kqKcADoSnvzJ8pvMaWUVB4qaCspEZYfm97WNdtY1wt8mlKYqIJyYUNOkTvQg==", + "os": ["win32"], + "cpu": ["x64"] + }, "@colors/colors@1.5.0": { "integrity": "sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ==" }, + "@cspotcode/source-map-support@0.8.1": { + "integrity": "sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==", + "dependencies": [ + "@jridgewell/trace-mapping@0.3.9" + ] + }, "@durable-streams/client@0.2.6": { "integrity": "sha512-uHKKbWpsKLhFMeGjG0PgM6LXE3oEIi7FHKlJZkmYGxcqd4Yjjd/QEvnQnDzteRP4Av1uJVM8qjTL7kfKsgeS/w==", "dependencies": [ @@ -593,6 +652,12 @@ "effection" ] }, + "@emnapi/runtime@1.11.3": { + "integrity": "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==", + "dependencies": [ + "tslib" + ] + }, "@esbuild/aix-ppc64@0.25.12": { "integrity": "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==", "os": ["aix"], @@ -1016,6 +1081,174 @@ "@harperfast/extended-iterable@1.0.3": { "integrity": "sha512-sSAYhQca3rDWtQUHSAPeO7axFIUJOI6hn1gjRC5APVE1a90tuyT8f5WIgRsFhhWA7htNkju2veB9eWL6YHi/Lw==" }, + "@img/colour@1.1.0": { + "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==" + }, + "@img/sharp-darwin-arm64@0.35.2": { + "integrity": "sha512-eEieHsMksAW4IiO5NzauESRl2D2qz3J/kwUxUrSfV06A93eEaRfMpHXyUb1mAqrR7i8U9A0GRqE9pjn6u1Jjpg==", + "optionalDependencies": [ + "@img/sharp-libvips-darwin-arm64" + ], + "os": ["darwin"], + "cpu": ["arm64"] + }, + "@img/sharp-darwin-x64@0.35.2": { + "integrity": "sha512-BaktuGPCeHJMARpodR8jK4uKiZrPAy9WrfQW0sdI37clracq8Bp01AYS3SZgi5FS/y5twa9t4+LIuuxQjqRrWw==", + "optionalDependencies": [ + "@img/sharp-libvips-darwin-x64" + ], + "os": ["darwin"], + "cpu": ["x64"] + }, + "@img/sharp-freebsd-wasm32@0.35.2": { + "integrity": "sha512-YoAxdnd8hPUkvLHd3bWY+YA8nw3xM/RyRopYucNsWHVSan8NLVM3X2volsfoRDcXdUJPg6tXahSd7HXPK7lRnw==", + "dependencies": [ + "@img/sharp-wasm32" + ], + "os": ["freebsd"] + }, + "@img/sharp-libvips-darwin-arm64@1.3.1": { + "integrity": "sha512-4V/M3roRMTYjiwZY9IOVQOE8OyeCxFAkYmyZDrZl51uOKjibm3oeEJ4WAmLxutAfzFbC9jqUiPs2gbnGflH+7g==", + "os": ["darwin"], + "cpu": ["arm64"] + }, + "@img/sharp-libvips-darwin-x64@1.3.1": { + "integrity": "sha512-c0/DxItpJv2+dGhgycJBBgotdqruGYDvA79drdh0MD1dFpy7JzJ/PlXwi1H4rFf0eTy8tgbI91aHDnZIceY3jQ==", + "os": ["darwin"], + "cpu": ["x64"] + }, + "@img/sharp-libvips-linux-arm64@1.3.1": { + "integrity": "sha512-JznefmcK9j1JKPz8AkQDh89kjojubyfOasWBPKfzMIhPwsgDy9evpE/naJTXXXmghS1iFwR8u/kTwh/I2/+GCw==", + "os": ["linux"], + "cpu": ["arm64"] + }, + "@img/sharp-libvips-linux-arm@1.3.1": { + "integrity": "sha512-aGGy9aWzXgHBG7HNyQPWorZthlp7+x6fDRoPAQbGO3ThcttuTyKIx3NuSHb6zb4gBNq6/yNn9f1cy9nFKS/Vmg==", + "os": ["linux"], + "cpu": ["arm"] + }, + "@img/sharp-libvips-linux-ppc64@1.3.1": { + "integrity": "sha512-1EkwGNCZk6iWNCMWqrvdJ+r1j0PT1zIz60CNPhYnJlK/zyeWqlsPZIe+ocBVqPF8k/Ssee/NCk+tE9Ryrko6ng==", + "os": ["linux"], + "cpu": ["ppc64"] + }, + "@img/sharp-libvips-linux-riscv64@1.3.1": { + "integrity": "sha512-Ilays+w2bXdnxzxtQdmXR62u8o8GYa3eL4+Gr+1KiE4xperMZUslRaVPJwwPkzlHEjGfXAfRVAa/7CYCtSqsBw==", + "os": ["linux"], + "cpu": ["riscv64"] + }, + "@img/sharp-libvips-linux-s390x@1.3.1": { + "integrity": "sha512-VfBwVHQTbRoj4XlpA/KLZ7ltgMpz+4WSejFzQ+GnoImjo1PtEJ59QB2qR1xQEeRPYIkNrPIm2L4cICMvz4C2ew==", + "os": ["linux"], + "cpu": ["s390x"] + }, + "@img/sharp-libvips-linux-x64@1.3.1": { + "integrity": "sha512-+c8ukgwU62DS54nCAjw7keOfHUkmr0B5QHEdcOqRnodF/MNXJbVI8Eopoj4B/0H8Asr65I+A4Amrn7a85/md6A==", + "os": ["linux"], + "cpu": ["x64"] + }, + "@img/sharp-libvips-linuxmusl-arm64@1.3.1": { + "integrity": "sha512-qlKb/pwbkAi1WMsJrYHk7CuDrd12s27U2QnRhFYUoJNrRCmkosMTttuRFat/DDB3IlDm5qE1TJgZ4JDnHX8Ldw==", + "os": ["linux"], + "cpu": ["arm64"] + }, + "@img/sharp-libvips-linuxmusl-x64@1.3.1": { + "integrity": "sha512-yO21HwoUVLN8Qa+/SBjQLMYwBWAVJjeGPNe+hc0OUeMeifEtJqu5a1c4HayE1nNpDih9y3/KkoltfkDodmKAlg==", + "os": ["linux"], + "cpu": ["x64"] + }, + "@img/sharp-linux-arm64@0.35.2": { + "integrity": "sha512-af12Pnd0ZGu2HfP8NayB0kk6eC/lrfbQE6HlR4jD+34wdJ1Vw9TF6TMn6ZvffT+WgqVsl0hRbmNvz2u/23VmwA==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-arm64" + ], + "os": ["linux"], + "cpu": ["arm64"] + }, + "@img/sharp-linux-arm@0.35.2": { + "integrity": "sha512-SE4kzF2mepn6z+6E7L6lsV8FzuLL6IPQdyX8ZiwROAG/G8td+hP/m7FsFPwidtrF19gvajuC9l6TxAVcsA4S7A==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-arm" + ], + "os": ["linux"], + "cpu": ["arm"] + }, + "@img/sharp-linux-ppc64@0.35.2": { + "integrity": "sha512-hYSBm7zcNtDCozCxQHYZJiu63b/bXsgRZuOxCIBZsStMM9Vap47iFHdbX4kCvQsblPB/k+clhELpdQJHQLSHvg==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-ppc64" + ], + "os": ["linux"], + "cpu": ["ppc64"] + }, + "@img/sharp-linux-riscv64@0.35.2": { + "integrity": "sha512-qQt0Kc13+Hoan/Awq/qMSQw3L+RI1NCRPgD5cUJ/1WSSmIoysLOc72jlRM3E0OHN9Yr313jgeQ2T+zW+F03QFA==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-riscv64" + ], + "os": ["linux"], + "cpu": ["riscv64"] + }, + "@img/sharp-linux-s390x@0.35.2": { + "integrity": "sha512-E4fLLfRPzDLlEeDaTzI98OFLcv++WL5ChLLMwPoVd0CIoZQqupBSNbOisPL5am9XsbQ9T84+iiMpUvbFtkunbA==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-s390x" + ], + "os": ["linux"], + "cpu": ["s390x"] + }, + "@img/sharp-linux-x64@0.35.2": { + "integrity": "sha512-gi0zFJJRLswfCZmHtJdikXPOc5u7qamSOS3NHedLqLd4W8Q0NqjdBr6TTRIgsfFjqfTsHFgdfvJ9LwqSgcHiAA==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-x64" + ], + "os": ["linux"], + "cpu": ["x64"] + }, + "@img/sharp-linuxmusl-arm64@0.35.2": { + "integrity": "sha512-siWbOW1u6HFnFLrp0waKyW7VEf7jYvcDWdrXEFa8AkdAQgEvuu5Fz8/Y70w9EeqAdwDtfU012BhEHHaDqvQNzg==", + "optionalDependencies": [ + "@img/sharp-libvips-linuxmusl-arm64" + ], + "os": ["linux"], + "cpu": ["arm64"] + }, + "@img/sharp-linuxmusl-x64@0.35.2": { + "integrity": "sha512-YBqMMcjDi4QGYiSn4vNOYBhmlC4z5AXqkOUUqI2e0AFA4urNv4ESgOgwNl3K+4etQhha0twXlzeF20bbULm9Yg==", + "optionalDependencies": [ + "@img/sharp-libvips-linuxmusl-x64" + ], + "os": ["linux"], + "cpu": ["x64"] + }, + "@img/sharp-wasm32@0.35.2": { + "integrity": "sha512-Mrv4JQNYVQ94xH+jzZ9r+gowleN8mv2FTgKT+PI6bx5C0G8TdNYndu161pg2i7uoBwxy2ImPMHrJOM2LZef7Bw==", + "dependencies": [ + "@emnapi/runtime" + ] + }, + "@img/sharp-webcontainers-wasm32@0.35.2": { + "integrity": "sha512-QNV27pxs9wpApEiCfvHM1RDoP1w1+2KrUWWDPEhEwg+latvOrfuhWrHWZKwdSFwU6jh3myjw/yOCRsUIuOft3g==", + "dependencies": [ + "@img/sharp-wasm32" + ], + "cpu": ["wasm32"] + }, + "@img/sharp-win32-arm64@0.35.2": { + "integrity": "sha512-BiVRYc/t6/Vl3e1hBx0hugG4oN9Pydf4fgMSpxTQJmwGUg/YoXTWHiFeRymHfCZzifxu4F4rpk/I67D0LQ20wQ==", + "os": ["win32"], + "cpu": ["arm64"] + }, + "@img/sharp-win32-ia32@0.35.2": { + "integrity": "sha512-YYEhx9PImCC7T0tI8JDMi4DB9LwLCXCU5OWNYEXAxh5Q1ShKkyC6byxzoBJ3gEFDnH2lQckWuDe70G7mB2XJog==", + "os": ["win32"], + "cpu": ["ia32"] + }, + "@img/sharp-win32-x64@0.35.2": { + "integrity": "sha512-imoOyBcoM/iiUr4J6VPpCNjPnjvP/Gks95898yB8YqoGGYmHYbOyCuNv9FMhFgtaiHFGbHW8bxKqRV6VjtXThQ==", + "os": ["win32"], + "cpu": ["x64"] + }, "@jest/diff-sequences@30.3.0": { "integrity": "sha512-cG51MVnLq1ecVUaQ3fr6YuuAOitHK1S4WUJHnsPFE/quQr33ADUx1FfrTCpMCRxvy0Yr9BThKpDjSlcTi91tMA==" }, @@ -1057,14 +1290,14 @@ "integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==", "dependencies": [ "@jridgewell/sourcemap-codec", - "@jridgewell/trace-mapping" + "@jridgewell/trace-mapping@0.3.31" ] }, "@jridgewell/remapping@2.3.5": { "integrity": "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ==", "dependencies": [ "@jridgewell/gen-mapping", - "@jridgewell/trace-mapping" + "@jridgewell/trace-mapping@0.3.31" ] }, "@jridgewell/resolve-uri@3.1.2": { @@ -1080,6 +1313,13 @@ "@jridgewell/sourcemap-codec" ] }, + "@jridgewell/trace-mapping@0.3.9": { + "integrity": "sha512-3Belt6tdc8bPgAtbcmdtNJlirVoTmEb5e2gC94PnkwEW9jI6CAHUeoG85tjWP5WquqfavoMtMwiG4P926ZKKuQ==", + "dependencies": [ + "@jridgewell/resolve-uri", + "@jridgewell/sourcemap-codec" + ] + }, "@lmdb/lmdb-darwin-arm64@3.5.6": { "integrity": "sha512-mY5FG4TjPAkY4P0w+OhHaUka5mDh2TX2WKYIwuKzJ1zeW3VvRgxdam/lGJTquI+bthTx5CSHDW+BAQCnNAzkEA==", "os": ["darwin"], @@ -1374,6 +1614,23 @@ "os": ["win32"], "cpu": ["x64"] }, + "@poppinss/colors@4.1.6": { + "integrity": "sha512-H9xkIdFswbS8n1d6vmRd8+c10t2Qe+rZITbbDHHkQixH5+2x1FDGmi/0K+WgWiqQFKPSlIYB7jlH6Kpfn6Fleg==", + "dependencies": [ + "kleur" + ] + }, + "@poppinss/dumper@0.6.5": { + "integrity": "sha512-NBdYIb90J7LfOI32dOewKI1r7wnkiH6m920puQ3qHUeZkxNkQiFnXVWoE6YtFSv6QOiPPf7ys6i+HWWecDz7sw==", + "dependencies": [ + "@poppinss/colors", + "@sindresorhus/is@7.2.0", + "supports-color@10.2.2" + ] + }, + "@poppinss/exception@1.2.3": { + "integrity": "sha512-dCED+QRChTVatE9ibtoaxc+WkdzOSjYTKi/+uacHWIsfodVfpsueo3+DKpgU5Px8qXjgmXkSvhXvSCz3fnP9lw==" + }, "@preact/signals-core@1.14.4": { "integrity": "sha512-HNB6HYeYKhQbJ1aKl+YRjrS4+QWHLKX6qKoUsfS/m0vqzsVaEBiZiaKbG/e+NKk2ch5ALQr/ihWaMHxiCuuWHA==" }, @@ -1835,7 +2092,7 @@ "@rollup/pluginutils@4.2.1": { "integrity": "sha512-iKnFXr7NkdZAIHiIWE+BX5ULi/ucVFYWD6TbAV+rZctiRTY2PL6tsIKhoIOaoskiWAkgu+VsbXgUVDNLHf+InQ==", "dependencies": [ - "estree-walker", + "estree-walker@2.0.2", "picomatch@2.3.2" ] }, @@ -1988,6 +2245,12 @@ "@sindresorhus/is@4.6.0": { "integrity": "sha512-t09vSN3MdfsyCHoFcTRCH/iUtG7OJ0CsjzB8cjAmKc/va/kIgeDI/TxsigdncE/4be734m0cvIYwNaV4i2XqAw==" }, + "@sindresorhus/is@7.2.0": { + "integrity": "sha512-P1Cz1dWaFfR4IR+U13mqqiGsLFf1KbayybWwdd2vfctdV6hDpUkgCY0nKOLLTMSoRd/jJNjtbqzf13K8DCCXQw==" + }, + "@speed-highlight/core@1.2.24": { + "integrity": "sha512-qeW2e1l78afw8VhRPfPQ1Gjj+KU5XFQ/OFV5ti6eTa9bruO7mJyZtA4vw0ofqmA3tKCkROE9xLk3VZoeRc98nw==" + }, "@standard-schema/spec@1.1.0": { "integrity": "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==" }, @@ -2117,12 +2380,22 @@ "@babel/types" ] }, + "@types/chai@5.2.3": { + "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==", + "dependencies": [ + "@types/deep-eql", + "assertion-error" + ] + }, "@types/debug@4.1.12": { "integrity": "sha512-vIChWdVG3LG1SMxEvI/AK+FWJthlrqlTu7fbrlywTkkaONwk/UAGaULXRlf8vkzFBLVm0zkMdCquhL5aOjhXPQ==", "dependencies": [ "@types/ms" ] }, + "@types/deep-eql@4.0.2": { + "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==" + }, "@types/estree@1.0.9": { "integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==" }, @@ -2189,6 +2462,62 @@ "@ungap/structured-clone@1.3.2": { "integrity": "sha512-5jsZFwgR5rTdKwidH9Qmat75RKwqfpKlWWB1frDkljN127mwqBu8K0PYo7/hFpF03IEJpfVPpCQDY/eDx3iHvA==" }, + "@vitest/expect@4.1.11": { + "integrity": "sha512-VX2x5vNJXET47KAFzwERI+KRMtTTCSWTfSMKsW7JsUsXV4psq++e3DvZpuTDOpHcxytiDs6p2nhVb2tVDiiUYw==", + "dependencies": [ + "@standard-schema/spec", + "@types/chai", + "@vitest/spy", + "@vitest/utils", + "chai", + "tinyrainbow" + ] + }, + "@vitest/mocker@4.1.11_vite@7.3.6__@types+node@24.13.3__tsx@4.23.1": { + "integrity": "sha512-2XJVD55d1o5AZous5CCGKS74g/riOj9odEt2bQpCVZeblHyHdnMeFl4jl0XjU21stf4mbjUkew2eXQZt65g5CQ==", + "dependencies": [ + "@vitest/spy", + "estree-walker@3.0.3", + "magic-string", + "vite" + ], + "optionalPeers": [ + "vite" + ] + }, + "@vitest/pretty-format@4.1.11": { + "integrity": "sha512-yiZzPbGTS9Sr/JpFl8zHrcIkAofNbFV6k21vIgQN/cY/oxZeXhJv5sc/MBJ5jFKWmWs+oJHw0UXLZjmf931+Vw==", + "dependencies": [ + "tinyrainbow" + ] + }, + "@vitest/runner@4.1.11": { + "integrity": "sha512-LztvUgdwMNJMIkj3hQnnxiC2Xy1zNxq928W/xhjCLaNCzqTZOudjwbQf6v9IntZGPw132i2Lq2rgTRZHD3JHNw==", + "dependencies": [ + "@vitest/utils", + "pathe" + ] + }, + "@vitest/snapshot@4.1.11": { + "integrity": "sha512-pN7ikn1ON7h8ee4gIAp4AzyK+zBtJPzVbqOgu5LCEh4VaJVbPQcgYQYJIMGQPXVeJJq1fnfazis7a5pFNPahog==", + "dependencies": [ + "@vitest/pretty-format", + "@vitest/utils", + "magic-string", + "pathe" + ] + }, + "@vitest/spy@4.1.11": { + "integrity": "sha512-apNa/prQy2qCeywhnixOHPRCgGNhvg7T4Dapfl1GahLp/R+uhBm5cPyFoNVyqsNd2h1nJxL6BqqdIjiABL60YA==" + }, + "@vitest/utils@4.1.11": { + "integrity": "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ==", + "dependencies": [ + "@vitest/pretty-format", + "convert-source-map", + "tinyrainbow" + ] + }, "@x0k/json-schema-merge@1.0.4": { "integrity": "sha512-KvmMgAftbVzATq4IRnkno/SKSu+gjaR2ZUPJG5JUlY4W3twRJo03sk2914u8scmosibBZ0m7s6euZlJuqpv8Ww==", "dependencies": [ @@ -2264,6 +2593,9 @@ "tslib" ] }, + "assertion-error@2.0.1": { + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==" + }, "b4a@1.8.1": { "integrity": "sha512-aiqre1Nr0B/6DgE2N5vwTc+2/oQZ4Wh1t4NznYY4E00y8LCt6NqdRv81so00oo27D8MVKTpUa/MwUUtBLXCoDw==" }, @@ -2308,6 +2640,9 @@ "integrity": "sha512-HYXq73DDpCtNzOmrFsm9eSwCvWCql0RzqjpDzXN9EadiLJ4DNat0nsZ/Bzmy+Ud12mb4/zKDY0cQ805ZzN+i0A==", "bin": true }, + "blake3-wasm@2.1.5": { + "integrity": "sha512-F1+K8EbfOZE49dtoPtmxUQrpXaBIl3ICvasLh+nJta0xkz+9kF/7uet9fLnwKqhDrmj6g+6K3Tw9yQPUg2ka5g==" + }, "boolbase@1.0.0": { "integrity": "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==" }, @@ -2331,11 +2666,14 @@ "ccount@2.0.1": { "integrity": "sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==" }, + "chai@6.2.2": { + "integrity": "sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==" + }, "chalk@4.1.2": { "integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==", "dependencies": [ "ansi-styles@4.3.0", - "supports-color" + "supports-color@7.2.0" ] }, "chalk@5.6.2": { @@ -2356,6 +2694,9 @@ "ci-info@4.4.0": { "integrity": "sha512-77PSwercCZU2Fc4sX94eF8k8Pxte6JAwL4/ICZLFjJLqegs7kCuAsqqj/70NQF6TvDpgFjkubQB2FW2ZZddvQg==" }, + "cjs-module-lexer@1.2.3": { + "integrity": "sha512-0TNiGstbQmCFwt4akjjBg5pLRTSyj/PkWQ1ZoO2zntmg9yLqSRxwEa4iCfQLGjqhiqBfOJa7W/E8wfGrTDmlZQ==" + }, "class-variance-authority@0.7.1": { "integrity": "sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg==", "dependencies": [ @@ -2429,6 +2770,9 @@ "convert-source-map@2.0.0": { "integrity": "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==" }, + "cookie@1.1.1": { + "integrity": "sha512-ei8Aos7ja0weRpFzJnEA9UHJ/7XQmqglbRwnf2ATjcB9Wq874VKH9kfjjirM6UhU2/E5fFYadylyhFldcqSidQ==" + }, "cross-spawn@7.0.6": { "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", "dependencies": [ @@ -2492,6 +2836,12 @@ "environment@1.1.0": { "integrity": "sha512-xUtoPkMggbz0MPyPiIWr1Kp4aeWJjDZ6SMvURhimjdZgsRuDplF5/s9hcgGhyXMhs+6vpnuoiZ2kFiu3FMnS8Q==" }, + "error-stack-parser-es@1.0.5": { + "integrity": "sha512-5qucVt2XcuGMcEGgWI7i+yZpmpByQ8J1lHhcL7PwqCwu9FPP3VUXzT4ltHe5i2z9dePwEHcDVOAfSnHsOlCXRA==" + }, + "es-module-lexer@2.3.2": { + "integrity": "sha512-poHGpORABojJJucnV9KbOavETW8lBVnphkW77ER5/BQ5Fz7oXSoCNek7IH3vR5nRjdsEz926ibFYX8KtLQmdyw==" + }, "esbuild-wasm@0.25.12": { "integrity": "sha512-rZqkjL3Y6FwLpSHzLnaEy8Ps6veCNo1kZa9EOfJvmWtBq5dJH4iVjfmOO6Mlkv9B0tt9WFPFmb/VxlgJOnueNg==", "bin": true @@ -2608,12 +2958,21 @@ "estree-walker@2.0.2": { "integrity": "sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w==" }, + "estree-walker@3.0.3": { + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dependencies": [ + "@types/estree" + ] + }, "events-universal@1.0.1": { "integrity": "sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==", "dependencies": [ "bare-events" ] }, + "expect-type@1.4.0": { + "integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==" + }, "expect@30.3.0": { "integrity": "sha512-1zQrciTiQfRdo7qJM1uG4navm8DayFa2TgCSRlzUyNkhcJ6XUZF3hjnpkyr3VhAqPH7i/9GkG7Tv5abz6fqz0Q==", "dependencies": [ @@ -2845,6 +3204,9 @@ "kind-of@6.0.3": { "integrity": "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==" }, + "kleur@4.1.5": { + "integrity": "sha512-o+NO+8WrRiQEE4/7nwRJhN1HWpVmJm511pBHUxPLtp0BUISzlBplORYSmTclCnJvQq2tKu/sgl3xVpkc7ZWuQQ==" + }, "lightningcss-android-arm64@1.32.0": { "integrity": "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg==", "os": ["android"], @@ -3228,6 +3590,17 @@ "micromark-util-types" ] }, + "miniflare@5.20260831.0-alpha": { + "integrity": "sha512-Hwgh1VDUiPCPGQKODQfUmy7hRAje1D55icB+9png3ueiM64rlSM87nSrtqpxAD+DlLWI4ehnYBuECaXV43zGmQ==", + "dependencies": [ + "@cspotcode/source-map-support", + "sharp", + "undici", + "workerd", + "ws", + "youch" + ] + }, "ms@2.1.3": { "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==" }, @@ -3271,7 +3644,7 @@ "node-emoji@2.2.0": { "integrity": "sha512-Z3lTE9pLaJF47NyMhd4ww1yFTAP8YhYI8SleJiHzM46Fgpm5cnNzSl9XfzFNqbaz+VlJrIj3fXQ4DeN1Rjm6cw==", "dependencies": [ - "@sindresorhus/is", + "@sindresorhus/is@4.6.0", "char-regex", "emojilib", "skin-tone" @@ -3296,6 +3669,9 @@ "object-assign@4.1.1": { "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==" }, + "obug@2.1.4": { + "integrity": "sha512-4a+OsYv9UktOJKE+l1A4OufDgdRF9PifWj+tJnHURo/P+WOxpG4GzUFL9qCalmWauao6ogiG+QvnCovwPoyAWA==" + }, "ordered-binary@1.6.1": { "integrity": "sha512-QkCdPooczexPLiXIrbVOPYkR3VO3T6v2OyKRkR1Xbhpy7/LAVXwahnRCgRp78Oe/Ehf0C/HATAxfSr6eA1oX+w==" }, @@ -3385,6 +3761,12 @@ "path-key@3.1.1": { "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==" }, + "path-to-regexp@6.3.0": { + "integrity": "sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ==" + }, + "pathe@2.0.3": { + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==" + }, "picocolors@1.1.1": { "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==" }, @@ -3594,6 +3976,45 @@ "integrity": "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==", "bin": true }, + "semver@7.8.5": { + "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", + "bin": true + }, + "sharp@0.35.2": { + "integrity": "sha512-FVtFjtBCMiJS6yb5CX7Sop45WFMpeGw6oRKuJnXYgf/f1ms/D7LE/ZUSNxnW7rZ/dbslQWYkoqFHGPaDBtaK4w==", + "dependencies": [ + "@img/colour", + "detect-libc", + "semver@7.8.5" + ], + "optionalDependencies": [ + "@img/sharp-darwin-arm64", + "@img/sharp-darwin-x64", + "@img/sharp-freebsd-wasm32", + "@img/sharp-libvips-darwin-arm64", + "@img/sharp-libvips-darwin-x64", + "@img/sharp-libvips-linux-arm", + "@img/sharp-libvips-linux-arm64", + "@img/sharp-libvips-linux-ppc64", + "@img/sharp-libvips-linux-riscv64", + "@img/sharp-libvips-linux-s390x", + "@img/sharp-libvips-linux-x64", + "@img/sharp-libvips-linuxmusl-arm64", + "@img/sharp-libvips-linuxmusl-x64", + "@img/sharp-linux-arm", + "@img/sharp-linux-arm64", + "@img/sharp-linux-ppc64", + "@img/sharp-linux-riscv64", + "@img/sharp-linux-s390x", + "@img/sharp-linux-x64", + "@img/sharp-linuxmusl-arm64", + "@img/sharp-linuxmusl-x64", + "@img/sharp-webcontainers-wasm32", + "@img/sharp-win32-arm64", + "@img/sharp-win32-ia32", + "@img/sharp-win32-x64" + ] + }, "shebang-command@2.0.0": { "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", "dependencies": [ @@ -3606,6 +4027,9 @@ "shellwords-ts@3.0.1": { "integrity": "sha512-GabK4ApLMqHFRGlpgNqg8dmtHTnYHt0WUUJkIeMd3QaDrUUBEDXHSSNi3I0PzMimg8W+I0EN4TshQxsnHv1cwg==" }, + "siginfo@2.0.0": { + "integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==" + }, "sisteransi@1.0.5": { "integrity": "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==" }, @@ -3641,6 +4065,12 @@ "escape-string-regexp" ] }, + "stackback@0.0.2": { + "integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==" + }, + "std-env@4.2.0": { + "integrity": "sha512-oCUKSupKTHX53EyjDtuZQ64pjLJ6yYCtpmEw0goYxtjG9KpbRe8KAsl2tBUGU9DyMcJ0RwJ8GqJAFzMXcXW1Rw==" + }, "streamx@2.28.0": { "integrity": "sha512-1Yowhzjf0ivGMrTIkY9hav5TxobO9qIVqUE41fiCGMGgc3CLlf4MY+9AHmZqBWgDTue0fY9zWjYFVyf6Diuobw==", "dependencies": [ @@ -3679,6 +4109,9 @@ "boundary" ] }, + "supports-color@10.2.2": { + "integrity": "sha512-SS+jx45GF1QjgEXQx4NJZV9ImqmO2NPz5FNsIHrsDjh2YsHnawpan7SNQ1o8NuhrbHZy9AZhIoCUiCeaW/C80g==" + }, "supports-color@7.2.0": { "integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==", "dependencies": [ @@ -3689,7 +4122,7 @@ "integrity": "sha512-zFObLMyZeEwzAoKCyu1B91U79K2t7ApXuQfo8OuxwXLDgcKxuwM+YvcbIhm6QWqz7mHUH1TVytR1PwVVjEuMig==", "dependencies": [ "has-flag", - "supports-color" + "supports-color@7.2.0" ] }, "tailwind-merge@3.6.0": { @@ -3740,6 +4173,12 @@ "any-promise" ] }, + "tinybench@2.9.0": { + "integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==" + }, + "tinyexec@1.3.0": { + "integrity": "sha512-QKAl9m8gWWGHV8jZcPeym6j+XULi6tOf1mT83WYJ4Lk2ytW/uwAWkrP0uFsdoYMdueVJ0qs26wZ+23xeB4ibNQ==" + }, "tinyglobby@0.2.17": { "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", "dependencies": [ @@ -3750,6 +4189,9 @@ "tinypool@2.1.0": { "integrity": "sha512-Pugqs6M0m7Lv1I7FtxN4aoyToKg1C4tu+/381vH35y8oENM/Ai7f7C4StcoK4/+BSw9ebcS8jRiVrORFKCALLw==" }, + "tinyrainbow@3.1.1": { + "integrity": "sha512-yau8yJdTt989Mm0Bd/236QnzEiPf2xLLTqUZRUJOo/3CB078LSwzei343DgtJVmfJKJE3TMINY1u42SQsP6mXw==" + }, "trim-lines@3.0.1": { "integrity": "sha512-kRj8B+YHZCc9kQYdWfJB2/oUl9rA99qbowYYBtr4ui4mZyAQ2JpvVBd/6U2YloATfqBhBTSMhTpgBHtU0Mf3Rg==" }, @@ -3779,6 +4221,15 @@ "undici-types@7.18.2": { "integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==" }, + "undici@7.29.0": { + "integrity": "sha512-IDxfleLmmbSskfWSUATiN1nfn2rDuvnMOqb5CWR92iIfojA0Ud+ulOAAEQ57LPr9rWmsreUyf5lwyao+7GNNVw==" + }, + "unenv@2.0.0-rc.24": { + "integrity": "sha512-i7qRCmY42zmCwnYlh9H2SvLEypEFGye5iRmEMKjcGi7zk9UquigRjFtTLz0TYqr0ZGLZhaMHl/foy1bZR+Cwlw==", + "dependencies": [ + "pathe" + ] + }, "unicode-emoji-modifier-base@1.0.0": { "integrity": "sha512-yLSH4py7oFH3oG/9K+XWrz1pSi3dfUrWEnInbxMfArOfc1+33BlGPQtLsOYwvdMy11AwUBetYuaRxSPqgkq+8g==" }, @@ -3900,6 +4351,38 @@ ], "bin": true }, + "vitest@4.1.11_@opentelemetry+api@1.9.1_@types+node@24.13.3_vite@7.3.6__@types+node@24.13.3__tsx@4.23.1": { + "integrity": "sha512-fhACrNXUidIbGSBr5FlbuBkO7VWC1ZyLl0DO4CU2DrQoAPxX84Ysxs+HeGQpii5lZWV1Q4gBZTTu49mF+A6Edw==", + "dependencies": [ + "@opentelemetry/api", + "@types/node@24.13.3", + "@vitest/expect", + "@vitest/mocker", + "@vitest/pretty-format", + "@vitest/runner", + "@vitest/snapshot", + "@vitest/spy", + "@vitest/utils", + "es-module-lexer", + "expect-type", + "magic-string", + "obug", + "pathe", + "picomatch@4.0.5", + "std-env", + "tinybench", + "tinyexec", + "tinyglobby", + "tinyrainbow", + "vite", + "why-is-node-running" + ], + "optionalPeers": [ + "@opentelemetry/api", + "@types/node@24.13.3" + ], + "bin": true + }, "weak-lru-cache@1.2.2": { "integrity": "sha512-DEAoo25RfSYMuTGc9vPJzZcZullwIqRDSI9LOy+fkCJPi6hykCnfKaXTuPBDuXAUcqHXyOgFtHNp/kB2FjYHbw==" }, @@ -3910,6 +4393,43 @@ ], "bin": true }, + "why-is-node-running@2.3.0": { + "integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==", + "dependencies": [ + "siginfo", + "stackback" + ], + "bin": true + }, + "workerd@1.20260831.1": { + "integrity": "sha512-A2LwrkBel/FnKABPfeBAMiL6v70+rugnunqQRfWsWZjlhsTZoBScWUVunMy/xLCGLjWCQL2zp39AVR6aO0jurQ==", + "optionalDependencies": [ + "@cloudflare/workerd-darwin-64", + "@cloudflare/workerd-darwin-arm64", + "@cloudflare/workerd-linux-64", + "@cloudflare/workerd-linux-arm64", + "@cloudflare/workerd-windows-64" + ], + "scripts": true, + "bin": true + }, + "wrangler@4.128.0": { + "integrity": "sha512-jNXy9e8/pbx8iqTzXPiuflnitKJZoAfEUSUUDLW87bwyeMvJ7kb3yQMSbxEcfNdfHqJW38KRcKaLljOYV4N/4w==", + "dependencies": [ + "@cloudflare/kv-asset-handler", + "@cloudflare/unenv-preset", + "blake3-wasm", + "esbuild@0.28.1", + "miniflare", + "path-to-regexp", + "unenv", + "workerd" + ], + "optionalDependencies": [ + "fsevents" + ], + "bin": true + }, "wrap-ansi@7.0.0": { "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", "dependencies": [ @@ -3918,6 +4438,9 @@ "strip-ansi" ] }, + "ws@8.21.0": { + "integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==" + }, "y18n@5.0.8": { "integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==" }, @@ -3939,6 +4462,23 @@ "yargs-parser" ] }, + "youch-core@0.3.3": { + "integrity": "sha512-ho7XuGjLaJ2hWHoK8yFnsUGy2Y5uDpqSTq1FkHLK4/oqKtyUU1AFbOOxY4IpC9f0fTLjwYbslUz0Po5BpD1wrA==", + "dependencies": [ + "@poppinss/exception", + "error-stack-parser-es" + ] + }, + "youch@4.1.0-beta.10": { + "integrity": "sha512-rLfVLB4FgQneDr0dv1oddCVZmKjcJ6yX6mS4pU82Mq/Dt9a3cLZQ62pDBL4AUO+uVrCvtWz3ZFUL2HFAFJ/BXQ==", + "dependencies": [ + "@poppinss/colors", + "@poppinss/dumper", + "@speed-highlight/core", + "cookie", + "youch-core" + ] + }, "zod@4.4.3": { "integrity": "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==" }, @@ -3980,6 +4520,7 @@ ], "packageJson": { "dependencies": [ + "npm:@cloudflare/vitest-plugin@1.1.3", "npm:@durable-streams/client@~0.2.2", "npm:@durable-streams/server@~0.3.8", "npm:@effectionx/context-api@0.6.0", @@ -4010,6 +4551,7 @@ "npm:tsx@^4.19.0", "npm:typescript@5", "npm:unist-util-select@5", + "npm:vitest@4.1.11", "npm:zod@^4.3.6" ] }, diff --git a/package.json b/package.json index 4e507ca7f..950a7f2d0 100644 --- a/package.json +++ b/package.json @@ -24,7 +24,6 @@ "packageManager": "pnpm@9.15.0", "dependencies": { "@durable-streams/client": "^0.2.2", - "effection": "4.1.0", "@effectionx/context-api": "0.6.0", "@effectionx/converge": "0.1.4", "@effectionx/fetch": "0.2.1", @@ -38,17 +37,19 @@ "@effectionx/timebox": "0.4.3", "acorn": "^8.16.0", "ajv": "^8.17.1", + "effection": "4.1.0", "gray-matter": "^4.0.3", "magic-string": "^0.30.21", "marked": "^17.0.4", "marked-terminal": "^7.3.0", + "mdast-util-to-string": "^4", "remark": "15", "remend": "^1.2.2", - "zod": "^4.3.6", "unist-util-select": "^5", - "mdast-util-to-string": "^4" + "zod": "^4.3.6" }, "devDependencies": { + "@cloudflare/vitest-plugin": "1.1.3", "@durable-streams/server": "^0.3.8", "@executablemd/acp": "workspace:*", "@executablemd/cli": "workspace:*", @@ -65,7 +66,8 @@ "oxfmt": "^0.41.0", "oxlint": "1.74.0", "tsx": "^4.19.0", - "typescript": "^5.0.0" + "typescript": "^5.0.0", + "vitest": "4.1.11" }, "scripts": { "test:node": "tsx scripts/runtime-tests.ts node", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 140cced90..487618c30 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -81,6 +81,9 @@ importers: specifier: ^4.3.6 version: 4.3.6 devDependencies: + '@cloudflare/vitest-plugin': + specifier: 1.1.3 + version: 1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0))) '@durable-streams/server': specifier: ^0.3.8 version: 0.3.8 @@ -132,6 +135,9 @@ importers: typescript: specifier: ^5.0.0 version: 5.9.3 + vitest: + specifier: 4.1.11 + version: 4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)) packages/acp: dependencies: @@ -498,10 +504,64 @@ packages: resolution: {integrity: sha512-y7/yvZ2TPAnR9+jnc00klvNNLkJiXFFrQA/hlLCcxA9a2A4zQIOimyFQ9XfwYKiGD1fb5GY8vbKIIgO8d5Tb2A==} engines: {node: '>= 20.12.0'} + '@cloudflare/kv-asset-handler@0.5.0': + resolution: {integrity: sha512-jxQYkj8dSIzc0cD6cMMNdOc1UVjqSqu8BZdor5s8cGjW2I8BjODt/kWPVdY+u9zj3ms75Q5qaZgnxUad83+eAg==} + engines: {node: '>=22.0.0'} + + '@cloudflare/unenv-preset@2.16.1': + resolution: {integrity: sha512-ECxObrMfyTl5bhQf/lZCXwo5G6xX9IAUo+nDMKK4SZ8m4Jvvxp52vilxyySSWh2YTZz8+HQ07qGH/2rEom1vDw==} + peerDependencies: + unenv: 2.0.0-rc.24 + workerd: '>1.20260305.0 <2.0.0-0' + peerDependenciesMeta: + workerd: + optional: true + + '@cloudflare/vitest-plugin@1.1.3': + resolution: {integrity: sha512-ED1Rkaq5Wr5rCeHXpLoDyV4WGJzD0Ju0clM8jS7Hj+wjj/CwaMHeb8DXzUfUQPiHW9rTgwcttPPQnzau2kp6Jg==} + peerDependencies: + '@vitest/runner': ^4.1.0 + '@vitest/snapshot': ^4.1.0 + vitest: ^4.1.0 + + '@cloudflare/workerd-darwin-64@1.20260831.1': + resolution: {integrity: sha512-oyZ8xhu+gYTvoxV/sn6NRmTHK95RhEO1Dk54/6oPb0Uu70w7ZeRoCjkJ5aNmfS8Vrkdu6+oL0HNg6EcC61uQ2Q==} + engines: {node: '>=16'} + cpu: [x64] + os: [darwin] + + '@cloudflare/workerd-darwin-arm64@1.20260831.1': + resolution: {integrity: sha512-s6Go53KPnoXZ1sTGBZ3en3otfHDuMPJhiwXMYWU21JkJQkpoeRt6HFUwM0GPhK3YhXWm+8baGMvCGZYS/KA9eA==} + engines: {node: '>=16'} + cpu: [arm64] + os: [darwin] + + '@cloudflare/workerd-linux-64@1.20260831.1': + resolution: {integrity: sha512-WxNKBgjKgeYTolW3yl1Lt3Lu67UlxdeyzWYi9MIqrKBdyQcz+UNG36RevSBf8rv1sTWapRW234VX2keZ+wXapA==} + engines: {node: '>=16'} + cpu: [x64] + os: [linux] + + '@cloudflare/workerd-linux-arm64@1.20260831.1': + resolution: {integrity: sha512-JTF9+9clUT3gaCq7Xnmd+Q/wEMaitpngSTOec/Ffb/r3xexA9XwNJVFSOKfk6q61flHGjAYJ4H9B7Mu5Qur49w==} + engines: {node: '>=16'} + cpu: [arm64] + os: [linux] + + '@cloudflare/workerd-windows-64@1.20260831.1': + resolution: {integrity: sha512-do+KDYw0PABwsrKUQIccWBZB70kqKcADoSnvzJ8pvMaWUVB4qaCspEZYfm97WNdtY1wt8mlKYqIJyYUNOkTvQg==} + engines: {node: '>=16'} + cpu: [x64] + os: [win32] + '@colors/colors@1.5.0': resolution: {integrity: sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ==} engines: {node: '>=0.1.90'} + '@cspotcode/source-map-support@0.8.1': + resolution: {integrity: sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==} + engines: {node: '>=12'} + '@durable-streams/client@0.2.2': resolution: {integrity: sha512-zmr9ErxJP1ORljnog4kclWmEJGoTpGN+Mu8FJLVEgcaR9PqTeyKtadq1l1H+DhrPsfAPeG6BF/mEJs4HyI+Eig==} engines: {node: '>=18.0.0'} @@ -584,6 +644,9 @@ packages: peerDependencies: effection: ^3 || ^4 + '@emnapi/runtime@1.11.3': + resolution: {integrity: sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==} + '@esbuild/aix-ppc64@0.27.4': resolution: {integrity: sha512-cQPwL2mp2nSmHHJlCyoXgHGhbEPMrEEU5xhkcy3Hs/O7nGZqEpZ2sUtLaL9MORLtDfRvVl2/3PAuEkYZH0Ty8Q==} engines: {node: '>=18'} @@ -920,6 +983,152 @@ packages: '@harperfast/extended-iterable@1.0.3': resolution: {integrity: sha512-sSAYhQca3rDWtQUHSAPeO7axFIUJOI6hn1gjRC5APVE1a90tuyT8f5WIgRsFhhWA7htNkju2veB9eWL6YHi/Lw==} + '@img/colour@1.1.0': + resolution: {integrity: sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==} + engines: {node: '>=18'} + + '@img/sharp-darwin-arm64@0.35.2': + resolution: {integrity: sha512-eEieHsMksAW4IiO5NzauESRl2D2qz3J/kwUxUrSfV06A93eEaRfMpHXyUb1mAqrR7i8U9A0GRqE9pjn6u1Jjpg==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [darwin] + + '@img/sharp-darwin-x64@0.35.2': + resolution: {integrity: sha512-BaktuGPCeHJMARpodR8jK4uKiZrPAy9WrfQW0sdI37clracq8Bp01AYS3SZgi5FS/y5twa9t4+LIuuxQjqRrWw==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [darwin] + + '@img/sharp-freebsd-wasm32@0.35.2': + resolution: {integrity: sha512-YoAxdnd8hPUkvLHd3bWY+YA8nw3xM/RyRopYucNsWHVSan8NLVM3X2volsfoRDcXdUJPg6tXahSd7HXPK7lRnw==} + engines: {node: '>=20.9.0'} + os: [freebsd] + + '@img/sharp-libvips-darwin-arm64@1.3.1': + resolution: {integrity: sha512-4V/M3roRMTYjiwZY9IOVQOE8OyeCxFAkYmyZDrZl51uOKjibm3oeEJ4WAmLxutAfzFbC9jqUiPs2gbnGflH+7g==} + cpu: [arm64] + os: [darwin] + + '@img/sharp-libvips-darwin-x64@1.3.1': + resolution: {integrity: sha512-c0/DxItpJv2+dGhgycJBBgotdqruGYDvA79drdh0MD1dFpy7JzJ/PlXwi1H4rFf0eTy8tgbI91aHDnZIceY3jQ==} + cpu: [x64] + os: [darwin] + + '@img/sharp-libvips-linux-arm64@1.3.1': + resolution: {integrity: sha512-JznefmcK9j1JKPz8AkQDh89kjojubyfOasWBPKfzMIhPwsgDy9evpE/naJTXXXmghS1iFwR8u/kTwh/I2/+GCw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linux-arm@1.3.1': + resolution: {integrity: sha512-aGGy9aWzXgHBG7HNyQPWorZthlp7+x6fDRoPAQbGO3ThcttuTyKIx3NuSHb6zb4gBNq6/yNn9f1cy9nFKS/Vmg==} + cpu: [arm] + os: [linux] + + '@img/sharp-libvips-linux-ppc64@1.3.1': + resolution: {integrity: sha512-1EkwGNCZk6iWNCMWqrvdJ+r1j0PT1zIz60CNPhYnJlK/zyeWqlsPZIe+ocBVqPF8k/Ssee/NCk+tE9Ryrko6ng==} + cpu: [ppc64] + os: [linux] + + '@img/sharp-libvips-linux-riscv64@1.3.1': + resolution: {integrity: sha512-Ilays+w2bXdnxzxtQdmXR62u8o8GYa3eL4+Gr+1KiE4xperMZUslRaVPJwwPkzlHEjGfXAfRVAa/7CYCtSqsBw==} + cpu: [riscv64] + os: [linux] + + '@img/sharp-libvips-linux-s390x@1.3.1': + resolution: {integrity: sha512-VfBwVHQTbRoj4XlpA/KLZ7ltgMpz+4WSejFzQ+GnoImjo1PtEJ59QB2qR1xQEeRPYIkNrPIm2L4cICMvz4C2ew==} + cpu: [s390x] + os: [linux] + + '@img/sharp-libvips-linux-x64@1.3.1': + resolution: {integrity: sha512-+c8ukgwU62DS54nCAjw7keOfHUkmr0B5QHEdcOqRnodF/MNXJbVI8Eopoj4B/0H8Asr65I+A4Amrn7a85/md6A==} + cpu: [x64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-arm64@1.3.1': + resolution: {integrity: sha512-qlKb/pwbkAi1WMsJrYHk7CuDrd12s27U2QnRhFYUoJNrRCmkosMTttuRFat/DDB3IlDm5qE1TJgZ4JDnHX8Ldw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-x64@1.3.1': + resolution: {integrity: sha512-yO21HwoUVLN8Qa+/SBjQLMYwBWAVJjeGPNe+hc0OUeMeifEtJqu5a1c4HayE1nNpDih9y3/KkoltfkDodmKAlg==} + cpu: [x64] + os: [linux] + + '@img/sharp-linux-arm64@0.35.2': + resolution: {integrity: sha512-af12Pnd0ZGu2HfP8NayB0kk6eC/lrfbQE6HlR4jD+34wdJ1Vw9TF6TMn6ZvffT+WgqVsl0hRbmNvz2u/23VmwA==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [linux] + + '@img/sharp-linux-arm@0.35.2': + resolution: {integrity: sha512-SE4kzF2mepn6z+6E7L6lsV8FzuLL6IPQdyX8ZiwROAG/G8td+hP/m7FsFPwidtrF19gvajuC9l6TxAVcsA4S7A==} + engines: {node: '>=20.9.0'} + cpu: [arm] + os: [linux] + + '@img/sharp-linux-ppc64@0.35.2': + resolution: {integrity: sha512-hYSBm7zcNtDCozCxQHYZJiu63b/bXsgRZuOxCIBZsStMM9Vap47iFHdbX4kCvQsblPB/k+clhELpdQJHQLSHvg==} + engines: {node: '>=20.9.0'} + cpu: [ppc64] + os: [linux] + + '@img/sharp-linux-riscv64@0.35.2': + resolution: {integrity: sha512-qQt0Kc13+Hoan/Awq/qMSQw3L+RI1NCRPgD5cUJ/1WSSmIoysLOc72jlRM3E0OHN9Yr313jgeQ2T+zW+F03QFA==} + engines: {node: '>=20.9.0'} + cpu: [riscv64] + os: [linux] + + '@img/sharp-linux-s390x@0.35.2': + resolution: {integrity: sha512-E4fLLfRPzDLlEeDaTzI98OFLcv++WL5ChLLMwPoVd0CIoZQqupBSNbOisPL5am9XsbQ9T84+iiMpUvbFtkunbA==} + engines: {node: '>=20.9.0'} + cpu: [s390x] + os: [linux] + + '@img/sharp-linux-x64@0.35.2': + resolution: {integrity: sha512-gi0zFJJRLswfCZmHtJdikXPOc5u7qamSOS3NHedLqLd4W8Q0NqjdBr6TTRIgsfFjqfTsHFgdfvJ9LwqSgcHiAA==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [linux] + + '@img/sharp-linuxmusl-arm64@0.35.2': + resolution: {integrity: sha512-siWbOW1u6HFnFLrp0waKyW7VEf7jYvcDWdrXEFa8AkdAQgEvuu5Fz8/Y70w9EeqAdwDtfU012BhEHHaDqvQNzg==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [linux] + + '@img/sharp-linuxmusl-x64@0.35.2': + resolution: {integrity: sha512-YBqMMcjDi4QGYiSn4vNOYBhmlC4z5AXqkOUUqI2e0AFA4urNv4ESgOgwNl3K+4etQhha0twXlzeF20bbULm9Yg==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [linux] + + '@img/sharp-wasm32@0.35.2': + resolution: {integrity: sha512-Mrv4JQNYVQ94xH+jzZ9r+gowleN8mv2FTgKT+PI6bx5C0G8TdNYndu161pg2i7uoBwxy2ImPMHrJOM2LZef7Bw==} + engines: {node: '>=20.9.0'} + + '@img/sharp-webcontainers-wasm32@0.35.2': + resolution: {integrity: sha512-QNV27pxs9wpApEiCfvHM1RDoP1w1+2KrUWWDPEhEwg+latvOrfuhWrHWZKwdSFwU6jh3myjw/yOCRsUIuOft3g==} + engines: {node: '>=20.9.0'} + cpu: [wasm32] + + '@img/sharp-win32-arm64@0.35.2': + resolution: {integrity: sha512-BiVRYc/t6/Vl3e1hBx0hugG4oN9Pydf4fgMSpxTQJmwGUg/YoXTWHiFeRymHfCZzifxu4F4rpk/I67D0LQ20wQ==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [win32] + + '@img/sharp-win32-ia32@0.35.2': + resolution: {integrity: sha512-YYEhx9PImCC7T0tI8JDMi4DB9LwLCXCU5OWNYEXAxh5Q1ShKkyC6byxzoBJ3gEFDnH2lQckWuDe70G7mB2XJog==} + engines: {node: ^20.9.0} + cpu: [ia32] + os: [win32] + + '@img/sharp-win32-x64@0.35.2': + resolution: {integrity: sha512-imoOyBcoM/iiUr4J6VPpCNjPnjvP/Gks95898yB8YqoGGYmHYbOyCuNv9FMhFgtaiHFGbHW8bxKqRV6VjtXThQ==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [win32] + '@jest/diff-sequences@30.3.0': resolution: {integrity: sha512-cG51MVnLq1ecVUaQ3fr6YuuAOitHK1S4WUJHnsPFE/quQr33ADUx1FfrTCpMCRxvy0Yr9BThKpDjSlcTi91tMA==} engines: {node: ^18.14.0 || ^20.0.0 || ^22.0.0 || >=24.0.0} @@ -944,9 +1153,16 @@ packages: resolution: {integrity: sha512-JHm87k7bA33hpBngtU8h6UBub/fqqA9uXfw+21j5Hmk7ooPHlboRNxHq0JcMtC+n8VJGP1mcfnD3Mk+XKe1oSw==} engines: {node: ^18.14.0 || ^20.0.0 || ^22.0.0 || >=24.0.0} + '@jridgewell/resolve-uri@3.1.2': + resolution: {integrity: sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==} + engines: {node: '>=6.0.0'} + '@jridgewell/sourcemap-codec@1.5.5': resolution: {integrity: sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==} + '@jridgewell/trace-mapping@0.3.9': + resolution: {integrity: sha512-3Belt6tdc8bPgAtbcmdtNJlirVoTmEb5e2gC94PnkwEW9jI6CAHUeoG85tjWP5WquqfavoMtMwiG4P926ZKKuQ==} + '@lmdb/lmdb-darwin-arm64@3.5.6': resolution: {integrity: sha512-mY5FG4TjPAkY4P0w+OhHaUka5mDh2TX2WKYIwuKzJ1zeW3VvRgxdam/lGJTquI+bthTx5CSHDW+BAQCnNAzkEA==} cpu: [arm64] @@ -1019,6 +1235,9 @@ packages: resolution: {integrity: sha512-9T3nD5q51X1d4QYW6vouKW9hBSb2Tb/wB/2XoTr4oP5SCGtp3a7aTHHewQFylred1B21/Bhev6gy4x01FPBcbQ==} engines: {node: '>=18'} + '@oxc-project/types@0.148.0': + resolution: {integrity: sha512-Nm4s/jB+4FpFsPhWGEC4h7rzksesmtnMXomo6rCMcg/b8zLQuOziRgkCS1fxDCXOlJB/6Q8oABOZ/OP6RIPj9A==} + '@oxfmt/binding-android-arm-eabi@0.41.0': resolution: {integrity: sha512-REfrqeMKGkfMP+m/ScX4f5jJBSmVNYcpoDF8vP8f8eYPDuPGZmzp56NIUsYmx3h7f6NzC6cE3gqh8GDWrJHCKw==} engines: {node: ^20.19.0 || >=22.12.0} @@ -1247,6 +1466,15 @@ packages: cpu: [x64] os: [win32] + '@poppinss/colors@4.1.6': + resolution: {integrity: sha512-H9xkIdFswbS8n1d6vmRd8+c10t2Qe+rZITbbDHHkQixH5+2x1FDGmi/0K+WgWiqQFKPSlIYB7jlH6Kpfn6Fleg==} + + '@poppinss/dumper@0.6.5': + resolution: {integrity: sha512-NBdYIb90J7LfOI32dOewKI1r7wnkiH6m920puQ3qHUeZkxNkQiFnXVWoE6YtFSv6QOiPPf7ys6i+HWWecDz7sw==} + + '@poppinss/exception@1.2.3': + resolution: {integrity: sha512-dCED+QRChTVatE9ibtoaxc+WkdzOSjYTKi/+uacHWIsfodVfpsueo3+DKpgU5Px8qXjgmXkSvhXvSCz3fnP9lw==} + '@radix-ui/number@1.1.3': resolution: {integrity: sha512-Road2bidD0uu/1BGDOWNdPI06g0lIRy6IF9GZcIrDK2KGItfor8IQwQa+yM2ERgHM1MmHxaxpTzk0/Jp42lNfA==} @@ -1653,6 +1881,99 @@ packages: peerDependencies: '@rjsf/utils': ^6.7.1 + '@rolldown/binding-android-arm-eabi@1.2.7': + resolution: {integrity: sha512-EypzgnYCwyVY4NDHKzGmNJT5b+XaQEBniHxsMdeIQLB/tcCzZnhqrzHpZFbX9iaxx+5RiB8caATBtfvZP7zVxQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [android] + + '@rolldown/binding-android-arm64@1.2.7': + resolution: {integrity: sha512-l17HE9EweWaqJZhuUuNBN/FzM62xw+DECVnJyvMsxn8vJFAGLy5QfLDoYAcronkAN8VxKZHezDpulHDPx95vFw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [android] + + '@rolldown/binding-darwin-arm64@1.2.7': + resolution: {integrity: sha512-8ED8ELFvHXc6OCETIn4gXObPiaR6bckM/ipXtbzlPVDRMBfEGjCKgO90F9YtfdpDatVx/ZQw7aZ1vUMf/+T3Mw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [darwin] + + '@rolldown/binding-darwin-x64@1.2.7': + resolution: {integrity: sha512-/WPripjtiAIZ2tWY7ddijORT0Ujg87wxWW/qcoFVCKAWVDPhtY0xr7Dj0M3GyNGz60jGwTElhro/mkF9dT7dDQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [darwin] + + '@rolldown/binding-freebsd-x64@1.2.7': + resolution: {integrity: sha512-14DI4NcqpvbICxSnGLx3PmtDaWqRP/KGSGb6C+JLLVPeZRl6dKdHba3pGsqT3vpdTqhEYIPG0MMQ8c0xYqoJxA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [freebsd] + + '@rolldown/binding-linux-arm-gnueabihf@1.2.7': + resolution: {integrity: sha512-bxrWIRvHWQvbJwi+VIie/kDJmQxcNE6xxWwZdqF/ExVAigtHkv54WTLQPb+QsZdnFy18fg7JPfWGL0RH6vwIlQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@rolldown/binding-linux-arm64-gnu@1.2.7': + resolution: {integrity: sha512-toOY2BChBZyuxU7OYX6Tn389di4IzAqPTycVcci0O7FSfBqzRB3RZn+K5Is6ANf4tmgRd/K1yZTsNTXbkXsnLg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + + '@rolldown/binding-linux-arm64-musl@1.2.7': + resolution: {integrity: sha512-lAIXTH/aiLRLxsTgQvfhjo4K1ydWIp00+V0voOr9beb/9ZmkUFrSIb03dXNFRgMNvkE6oGsF10ioQ6UsI+vS5Q==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + + '@rolldown/binding-linux-ppc64-gnu@1.2.7': + resolution: {integrity: sha512-kdnwS28Pkenp/mZMRwjXXXwxQ7pIsm+bF919LUK93BOyhcLsrVKdP2p9fxpiPNPAbNuch8ypQt0pm2P2LYCAGg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ppc64] + os: [linux] + + '@rolldown/binding-linux-s390x-gnu@1.2.7': + resolution: {integrity: sha512-516OdsyLdr5E65paF3yBF55t8mfm9+gmtCsK3xI7XKXIT7EfRlHhxL8K/NR6Hu8BWSgF5+1w74lTL0+nxcc8Qw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [s390x] + os: [linux] + + '@rolldown/binding-linux-x64-gnu@1.2.7': + resolution: {integrity: sha512-r8/z8n7GFaYRln3xmP1Cxy0HH/HLM0uBUPkEuSVEfKGDA89M0FsZRZJRSwe/tJjRx+fpH/gjorfhB8tmEbSFLA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + + '@rolldown/binding-linux-x64-musl@1.2.7': + resolution: {integrity: sha512-pAsE8iiDxUg1xBqdhrTfg45AVDVpirjz00sblEYClGNNcMnDb+e8beQgqIAw6LvauX/APvgxUnwrgun/YYGBhw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + + '@rolldown/binding-openharmony-arm64@1.2.7': + resolution: {integrity: sha512-lTcIYmmnQQA8Or/2DatS6oSqcdLHvendjS+zLu+FwgToynWMRSmQdpM65fTANJgIS4mjbMOo5KT2lnT9SAb96w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [openharmony] + + '@rolldown/binding-win32-arm64-msvc@1.2.7': + resolution: {integrity: sha512-e3Gu3WxbNk/UqQhxqU7YIYO+9ZBvWNz3U+h/qRFosscMFzdRPbXYSaSWgSnklv2fz1TgzBTcti2z35c/7irsHw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [win32] + + '@rolldown/binding-win32-x64-msvc@1.2.7': + resolution: {integrity: sha512-W/jg5qoRSqjsEv0+dZi4e687mcHqmVuU0P4fK6qS/xjetW2Gmc1W8j//z5nAeNcC8Ttm0hV46IjcYeuVwYhuiw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [win32] + + '@rolldown/pluginutils@1.0.1': + resolution: {integrity: sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==} + '@secretlint/core@13.0.4': resolution: {integrity: sha512-Wv49KcI5XX6xjLR1wxyjORA15PtMb5ar/M27ShimVudaSi6iAM04QCA5Ozx+uEahfHNefUUKbjKGpy/9pxuW7g==} engines: {node: '>=22.0.0'} @@ -1675,12 +1996,28 @@ packages: resolution: {integrity: sha512-t09vSN3MdfsyCHoFcTRCH/iUtG7OJ0CsjzB8cjAmKc/va/kIgeDI/TxsigdncE/4be734m0cvIYwNaV4i2XqAw==} engines: {node: '>=10'} + '@sindresorhus/is@7.2.0': + resolution: {integrity: sha512-P1Cz1dWaFfR4IR+U13mqqiGsLFf1KbayybWwdd2vfctdV6hDpUkgCY0nKOLLTMSoRd/jJNjtbqzf13K8DCCXQw==} + engines: {node: '>=18'} + + '@speed-highlight/core@1.2.24': + resolution: {integrity: sha512-qeW2e1l78afw8VhRPfPQ1Gjj+KU5XFQ/OFV5ti6eTa9bruO7mJyZtA4vw0ofqmA3tKCkROE9xLk3VZoeRc98nw==} + '@standard-schema/spec@1.1.0': resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==} + '@types/chai@5.2.3': + resolution: {integrity: sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==} + '@types/debug@4.1.12': resolution: {integrity: sha512-vIChWdVG3LG1SMxEvI/AK+FWJthlrqlTu7fbrlywTkkaONwk/UAGaULXRlf8vkzFBLVm0zkMdCquhL5aOjhXPQ==} + '@types/deep-eql@4.0.2': + resolution: {integrity: sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==} + + '@types/estree@1.0.9': + resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==} + '@types/hast@3.0.5': resolution: {integrity: sha512-rp/ezSWaD1m44dPKICGhiskI13nVr7qTloFwDa/IYkhhf5nzwP+zIQcIJh3WIFSBOy/H1PzB40jPjMDksN4F+g==} @@ -1720,6 +2057,35 @@ packages: '@ungap/structured-clone@1.3.3': resolution: {integrity: sha512-60YRaenCQcVjYEKOcG824+DRGGIQ3VKErcBoAEDJZz5bKIs2ZG+X/H9Nk+Q6EVkwJk5QNApxbrc5QtBSwtrXAg==} + '@vitest/expect@4.1.11': + resolution: {integrity: sha512-VX2x5vNJXET47KAFzwERI+KRMtTTCSWTfSMKsW7JsUsXV4psq++e3DvZpuTDOpHcxytiDs6p2nhVb2tVDiiUYw==} + + '@vitest/mocker@4.1.11': + resolution: {integrity: sha512-2XJVD55d1o5AZous5CCGKS74g/riOj9odEt2bQpCVZeblHyHdnMeFl4jl0XjU21stf4mbjUkew2eXQZt65g5CQ==} + peerDependencies: + msw: ^2.4.9 + vite: ^6.0.0 || ^7.0.0 || ^8.0.0 + peerDependenciesMeta: + msw: + optional: true + vite: + optional: true + + '@vitest/pretty-format@4.1.11': + resolution: {integrity: sha512-yiZzPbGTS9Sr/JpFl8zHrcIkAofNbFV6k21vIgQN/cY/oxZeXhJv5sc/MBJ5jFKWmWs+oJHw0UXLZjmf931+Vw==} + + '@vitest/runner@4.1.11': + resolution: {integrity: sha512-LztvUgdwMNJMIkj3hQnnxiC2Xy1zNxq928W/xhjCLaNCzqTZOudjwbQf6v9IntZGPw132i2Lq2rgTRZHD3JHNw==} + + '@vitest/snapshot@4.1.11': + resolution: {integrity: sha512-pN7ikn1ON7h8ee4gIAp4AzyK+zBtJPzVbqOgu5LCEh4VaJVbPQcgYQYJIMGQPXVeJJq1fnfazis7a5pFNPahog==} + + '@vitest/spy@4.1.11': + resolution: {integrity: sha512-apNa/prQy2qCeywhnixOHPRCgGNhvg7T4Dapfl1GahLp/R+uhBm5cPyFoNVyqsNd2h1nJxL6BqqdIjiABL60YA==} + + '@vitest/utils@4.1.11': + resolution: {integrity: sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ==} + '@x0k/json-schema-merge@1.0.4': resolution: {integrity: sha512-KvmMgAftbVzATq4IRnkno/SKSu+gjaR2ZUPJG5JUlY4W3twRJo03sk2914u8scmosibBZ0m7s6euZlJuqpv8Ww==} @@ -1774,6 +2140,10 @@ packages: resolution: {integrity: sha512-ik3ZgC9dY/lYVVM++OISsaYDeg1tb0VtP5uL3ouh1koGOaUMDPpbFIei4JkFimWUFPn90sbMNMXQAIVOlnYKJA==} engines: {node: '>=10'} + assertion-error@2.0.1: + resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==} + engines: {node: '>=12'} + b4a@1.8.1: resolution: {integrity: sha512-aiqre1Nr0B/6DgE2N5vwTc+2/oQZ4Wh1t4NznYY4E00y8LCt6NqdRv81so00oo27D8MVKTpUa/MwUUtBLXCoDw==} peerDependencies: @@ -1822,6 +2192,9 @@ packages: bare-url@2.4.6: resolution: {integrity: sha512-iQxPClE07hETVpbRoX7JXX3v/ZQViCxe/SYCxylRLzdEx1xJAufPptfiOqR8tqiCtmbtMDANKWszzjLu1PMAZQ==} + blake3-wasm@2.1.5: + resolution: {integrity: sha512-F1+K8EbfOZE49dtoPtmxUQrpXaBIl3ICvasLh+nJta0xkz+9kF/7uet9fLnwKqhDrmj6g+6K3Tw9yQPUg2ka5g==} + boolbase@1.0.0: resolution: {integrity: sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==} @@ -1831,6 +2204,10 @@ packages: ccount@2.0.1: resolution: {integrity: sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==} + chai@6.2.2: + resolution: {integrity: sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==} + engines: {node: '>=18'} + chalk@4.1.2: resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} engines: {node: '>=10'} @@ -1856,6 +2233,9 @@ packages: resolution: {integrity: sha512-77PSwercCZU2Fc4sX94eF8k8Pxte6JAwL4/ICZLFjJLqegs7kCuAsqqj/70NQF6TvDpgFjkubQB2FW2ZZddvQg==} engines: {node: '>=8'} + cjs-module-lexer@1.2.3: + resolution: {integrity: sha512-0TNiGstbQmCFwt4akjjBg5pLRTSyj/PkWQ1ZoO2zntmg9yLqSRxwEa4iCfQLGjqhiqBfOJa7W/E8wfGrTDmlZQ==} + class-variance-authority@0.7.1: resolution: {integrity: sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg==} @@ -1899,6 +2279,13 @@ packages: resolution: {integrity: sha512-4bxK3+L+FHr9Xm/d69Syvvpvkj7lj7a4zz3B+tchuohg5WKeudyBS+4Oob5Zdgoh8I7+n2lj0lfa8I6cUXfcEg==} engines: {node: '>= 16'} + convert-source-map@2.0.0: + resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==} + + cookie@1.1.1: + resolution: {integrity: sha512-ei8Aos7ja0weRpFzJnEA9UHJ/7XQmqglbRwnf2ATjcB9Wq874VKH9kfjjirM6UhU2/E5fFYadylyhFldcqSidQ==} + engines: {node: '>=18'} + cross-spawn@7.0.6: resolution: {integrity: sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==} engines: {node: '>= 8'} @@ -1949,6 +2336,12 @@ packages: resolution: {integrity: sha512-xUtoPkMggbz0MPyPiIWr1Kp4aeWJjDZ6SMvURhimjdZgsRuDplF5/s9hcgGhyXMhs+6vpnuoiZ2kFiu3FMnS8Q==} engines: {node: '>=18'} + error-stack-parser-es@1.0.5: + resolution: {integrity: sha512-5qucVt2XcuGMcEGgWI7i+yZpmpByQ8J1lHhcL7PwqCwu9FPP3VUXzT4ltHe5i2z9dePwEHcDVOAfSnHsOlCXRA==} + + es-module-lexer@2.3.2: + resolution: {integrity: sha512-poHGpORABojJJucnV9KbOavETW8lBVnphkW77ER5/BQ5Fz7oXSoCNek7IH3vR5nRjdsEz926ibFYX8KtLQmdyw==} + esbuild@0.27.4: resolution: {integrity: sha512-Rq4vbHnYkK5fws5NF7MYTU68FPRE1ajX7heQ/8QXXWqNgqqJ/GkmmyxIzUnf2Sr/bakf8l54716CcMGHYhMrrQ==} engines: {node: '>=18'} @@ -1972,9 +2365,16 @@ packages: engines: {node: '>=4'} hasBin: true + estree-walker@3.0.3: + resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==} + events-universal@1.0.1: resolution: {integrity: sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==} + expect-type@1.4.0: + resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==} + engines: {node: '>=12.0.0'} + expect@30.3.0: resolution: {integrity: sha512-1zQrciTiQfRdo7qJM1uG4navm8DayFa2TgCSRlzUyNkhcJ6XUZF3hjnpkyr3VhAqPH7i/9GkG7Tv5abz6fqz0Q==} engines: {node: ^18.14.0 || ^20.0.0 || ^22.0.0 || >=24.0.0} @@ -2014,6 +2414,15 @@ packages: fastq@1.20.1: resolution: {integrity: sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw==} + fdir@6.5.0: + resolution: {integrity: sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==} + engines: {node: '>=12.0.0'} + peerDependencies: + picomatch: ^3 || ^4 + peerDependenciesMeta: + picomatch: + optional: true + fsevents@2.3.3: resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==} engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} @@ -2116,6 +2525,80 @@ packages: resolution: {integrity: sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==} engines: {node: '>=0.10.0'} + kleur@4.1.5: + resolution: {integrity: sha512-o+NO+8WrRiQEE4/7nwRJhN1HWpVmJm511pBHUxPLtp0BUISzlBplORYSmTclCnJvQq2tKu/sgl3xVpkc7ZWuQQ==} + engines: {node: '>=6'} + + lightningcss-android-arm64@1.33.0: + resolution: {integrity: sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [android] + + lightningcss-darwin-arm64@1.33.0: + resolution: {integrity: sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [darwin] + + lightningcss-darwin-x64@1.33.0: + resolution: {integrity: sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [darwin] + + lightningcss-freebsd-x64@1.33.0: + resolution: {integrity: sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [freebsd] + + lightningcss-linux-arm-gnueabihf@1.33.0: + resolution: {integrity: sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==} + engines: {node: '>= 12.0.0'} + cpu: [arm] + os: [linux] + + lightningcss-linux-arm64-gnu@1.33.0: + resolution: {integrity: sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [linux] + + lightningcss-linux-arm64-musl@1.33.0: + resolution: {integrity: sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [linux] + + lightningcss-linux-x64-gnu@1.33.0: + resolution: {integrity: sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [linux] + + lightningcss-linux-x64-musl@1.33.0: + resolution: {integrity: sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [linux] + + lightningcss-win32-arm64-msvc@1.33.0: + resolution: {integrity: sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [win32] + + lightningcss-win32-x64-msvc@1.33.0: + resolution: {integrity: sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [win32] + + lightningcss@1.33.0: + resolution: {integrity: sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==} + engines: {node: '>= 12.0.0'} + lmdb@3.5.6: resolution: {integrity: sha512-j3uE8ReKNyUWDjhfEFSJqE/1DLtfTR5Z8yFzVHvBjAk37wNg7HdScjcv8ttPHRvrdgPQMPWxFFI0SsdBzI5lBw==} hasBin: true @@ -2248,6 +2731,10 @@ packages: micromark@4.0.2: resolution: {integrity: sha512-zpe98Q6kvavpCr1NPVSCMebCKfD7CA2NqZ+rykeNhONIJBpc1tFKt9hucLGwha3jNTNI8lHpctWJWoimVF4PfA==} + miniflare@5.20260831.0-alpha: + resolution: {integrity: sha512-Hwgh1VDUiPCPGQKODQfUmy7hRAje1D55icB+9png3ueiM64rlSM87nSrtqpxAD+DlLWI4ehnYBuECaXV43zGmQ==} + engines: {node: '>=22.0.0'} + ms@2.1.3: resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==} @@ -2261,6 +2748,11 @@ packages: mz@2.7.0: resolution: {integrity: sha512-z81GNO7nnYMEhrGh9LeymoE4+Yr0Wn5McHIZMK5cfQCl+NDX08sCZgUc9/6MHni9IWuFLm1Z3HTCXu2z9fN62Q==} + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} + hasBin: true + node-addon-api@6.1.0: resolution: {integrity: sha512-+eawOlIgy680F0kBzPUNFhMZGtJ1YmqM6l4+Crf4IkImjYrO/mqPwRMh352g23uIaQKFItcQ64I7KMaJxHgAVA==} @@ -2279,6 +2771,10 @@ packages: resolution: {integrity: sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==} engines: {node: '>=0.10.0'} + obug@2.1.4: + resolution: {integrity: sha512-4a+OsYv9UktOJKE+l1A4OufDgdRF9PifWj+tJnHURo/P+WOxpG4GzUFL9qCalmWauao6ogiG+QvnCovwPoyAWA==} + engines: {node: '>=12.20.0'} + ordered-binary@1.6.1: resolution: {integrity: sha512-QkCdPooczexPLiXIrbVOPYkR3VO3T6v2OyKRkR1Xbhpy7/LAVXwahnRCgRp78Oe/Ehf0C/HATAxfSr6eA1oX+w==} @@ -2313,6 +2809,12 @@ packages: resolution: {integrity: sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==} engines: {node: '>=8'} + path-to-regexp@6.3.0: + resolution: {integrity: sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ==} + + pathe@2.0.3: + resolution: {integrity: sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==} + picocolors@1.1.1: resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} @@ -2320,6 +2822,14 @@ packages: resolution: {integrity: sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==} engines: {node: '>=12'} + picomatch@4.0.7: + resolution: {integrity: sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==} + engines: {node: '>=12'} + + postcss@8.5.26: + resolution: {integrity: sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==} + engines: {node: ^10 || ^12 || >=14} + pretty-format@30.3.0: resolution: {integrity: sha512-oG4T3wCbfeuvljnyAzhBvpN45E8iOTXCU/TD3zXW80HA3dQ4ahdqMkWGiPWZvjpQwlbyHrPTWUAqUzGzv4l1JQ==} engines: {node: ^18.14.0 || ^20.0.0 || ^22.0.0 || >=24.0.0} @@ -2414,6 +2924,11 @@ packages: resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==} engines: {iojs: '>=1.0.0', node: '>=0.10.0'} + rolldown@1.2.7: + resolution: {integrity: sha512-g0EtLvBjTUB7jhyV0S/TCup3v/XSVl45vUIGbOGU4QPiyjTenCe4mKuFvW9fEgYmS2Fo42AUssRmNuMziXdrig==} + engines: {node: ^20.19.0 || >=22.12.0} + hasBin: true + scheduler@0.27.0: resolution: {integrity: sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==} @@ -2421,6 +2936,15 @@ packages: resolution: {integrity: sha512-vfD3pmTzGpufjScBh50YHKzEu2lxBWhVEHsNGoEXmCmn2hKGfeNLYMzCJpe8cD7gqX7TJluOVpBkAequ6dgMmA==} engines: {node: '>=4'} + semver@7.8.5: + resolution: {integrity: sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==} + engines: {node: '>=10'} + hasBin: true + + sharp@0.35.2: + resolution: {integrity: sha512-FVtFjtBCMiJS6yb5CX7Sop45WFMpeGw6oRKuJnXYgf/f1ms/D7LE/ZUSNxnW7rZ/dbslQWYkoqFHGPaDBtaK4w==} + engines: {node: '>=20.9.0'} + shebang-command@2.0.0: resolution: {integrity: sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==} engines: {node: '>=8'} @@ -2432,6 +2956,9 @@ packages: shellwords-ts@3.0.1: resolution: {integrity: sha512-GabK4ApLMqHFRGlpgNqg8dmtHTnYHt0WUUJkIeMd3QaDrUUBEDXHSSNi3I0PzMimg8W+I0EN4TshQxsnHv1cwg==} + siginfo@2.0.0: + resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==} + sisteransi@1.0.5: resolution: {integrity: sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==} @@ -2448,6 +2975,10 @@ packages: resolution: {integrity: sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q==} engines: {node: '>=8'} + source-map-js@1.2.1: + resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} + engines: {node: '>=0.10.0'} + space-separated-tokens@2.0.2: resolution: {integrity: sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q==} @@ -2458,6 +2989,12 @@ packages: resolution: {integrity: sha512-XlkWvfIm6RmsWtNJx+uqtKLS8eqFbxUg0ZzLXqY0caEy9l7hruX8IpiDnjsLavoBgqCCR71TqWO8MaXYheJ3RQ==} engines: {node: '>=10'} + stackback@0.0.2: + resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==} + + std-env@4.2.0: + resolution: {integrity: sha512-oCUKSupKTHX53EyjDtuZQ64pjLJ6yYCtpmEw0goYxtjG9KpbRe8KAsl2tBUGU9DyMcJ0RwJ8GqJAFzMXcXW1Rw==} + streamx@2.28.0: resolution: {integrity: sha512-1Yowhzjf0ivGMrTIkY9hav5TxobO9qIVqUE41fiCGMGgc3CLlf4MY+9AHmZqBWgDTue0fY9zWjYFVyf6Diuobw==} @@ -2479,6 +3016,10 @@ packages: structured-source@4.0.0: resolution: {integrity: sha512-qGzRFNJDjFieQkl/sVOI2dUjHKRyL9dAJi2gCPGJLbJHBIkyOHxjuocpIEfbLioX+qSJpvbYdT49/YCdMznKxA==} + supports-color@10.2.2: + resolution: {integrity: sha512-SS+jx45GF1QjgEXQx4NJZV9ImqmO2NPz5FNsIHrsDjh2YsHnawpan7SNQ1o8NuhrbHZy9AZhIoCUiCeaW/C80g==} + engines: {node: '>=18'} + supports-color@7.2.0: resolution: {integrity: sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==} engines: {node: '>=8'} @@ -2511,10 +3052,25 @@ packages: thenify@3.3.1: resolution: {integrity: sha512-RVZSIV5IG10Hk3enotrhvz0T9em6cyHBLkH/YAZuKqd8hRkKhSfCGIcP2KUY0EPxndzANBmNllzWPwak+bheSw==} + tinybench@2.9.0: + resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==} + + tinyexec@1.3.0: + resolution: {integrity: sha512-QKAl9m8gWWGHV8jZcPeym6j+XULi6tOf1mT83WYJ4Lk2ytW/uwAWkrP0uFsdoYMdueVJ0qs26wZ+23xeB4ibNQ==} + engines: {node: '>=18'} + + tinyglobby@0.2.17: + resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==} + engines: {node: '>=12.0.0'} + tinypool@2.1.0: resolution: {integrity: sha512-Pugqs6M0m7Lv1I7FtxN4aoyToKg1C4tu+/381vH35y8oENM/Ai7f7C4StcoK4/+BSw9ebcS8jRiVrORFKCALLw==} engines: {node: ^20.0.0 || >=22.0.0} + tinyrainbow@3.1.1: + resolution: {integrity: sha512-yau8yJdTt989Mm0Bd/236QnzEiPf2xLLTqUZRUJOo/3CB078LSwzei343DgtJVmfJKJE3TMINY1u42SQsP6mXw==} + engines: {node: '>=14.0.0'} + trim-lines@3.0.1: resolution: {integrity: sha512-kRj8B+YHZCc9kQYdWfJB2/oUl9rA99qbowYYBtr4ui4mZyAQ2JpvVBd/6U2YloATfqBhBTSMhTpgBHtU0Mf3Rg==} @@ -2542,6 +3098,13 @@ packages: undici-types@6.21.0: resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==} + undici@7.29.0: + resolution: {integrity: sha512-IDxfleLmmbSskfWSUATiN1nfn2rDuvnMOqb5CWR92iIfojA0Ud+ulOAAEQ57LPr9rWmsreUyf5lwyao+7GNNVw==} + engines: {node: '>=20.18.1'} + + unenv@2.0.0-rc.24: + resolution: {integrity: sha512-i7qRCmY42zmCwnYlh9H2SvLEypEFGye5iRmEMKjcGi7zk9UquigRjFtTLz0TYqr0ZGLZhaMHl/foy1bZR+Cwlw==} + unicode-emoji-modifier-base@1.0.0: resolution: {integrity: sha512-yLSH4py7oFH3oG/9K+XWrz1pSi3dfUrWEnInbxMfArOfc1+33BlGPQtLsOYwvdMy11AwUBetYuaRxSPqgkq+8g==} engines: {node: '>=4'} @@ -2597,6 +3160,90 @@ packages: vfile@6.0.3: resolution: {integrity: sha512-KzIbH/9tXat2u30jf+smMwFCsno4wHVdNmzFyL+T/L3UGqqk6JKfVqOFOZEpZSHADH1k40ab6NUIXZq422ov3Q==} + vite@8.2.2: + resolution: {integrity: sha512-cFKLV/PRgAUlIRm5WjMjJ86jrftzpqcgH+Us+DS8mI3CDNiH30Whrz8uHL3+MOLPAgqbMBAqWdAHAphOAM+z/Q==} + engines: {node: ^20.19.0 || >=22.12.0} + hasBin: true + peerDependencies: + '@types/node': ^20.19.0 || >=22.12.0 + '@vitejs/devtools': ^0.4.0 || ^0.5.0 + esbuild: ^0.27.0 || ^0.28.0 + jiti: '>=1.21.0' + less: ^4.0.0 + sass: ^1.70.0 + sass-embedded: ^1.70.0 + stylus: '>=0.54.8' + sugarss: ^5.0.0 + terser: ^5.16.0 + tsx: ^4.8.1 + yaml: ^2.4.2 + peerDependenciesMeta: + '@types/node': + optional: true + '@vitejs/devtools': + optional: true + esbuild: + optional: true + jiti: + optional: true + less: + optional: true + sass: + optional: true + sass-embedded: + optional: true + stylus: + optional: true + sugarss: + optional: true + terser: + optional: true + tsx: + optional: true + yaml: + optional: true + + vitest@4.1.11: + resolution: {integrity: sha512-fhACrNXUidIbGSBr5FlbuBkO7VWC1ZyLl0DO4CU2DrQoAPxX84Ysxs+HeGQpii5lZWV1Q4gBZTTu49mF+A6Edw==} + engines: {node: ^20.0.0 || ^22.0.0 || >=24.0.0} + hasBin: true + peerDependencies: + '@edge-runtime/vm': '*' + '@opentelemetry/api': ^1.9.0 + '@types/node': ^20.0.0 || ^22.0.0 || >=24.0.0 + '@vitest/browser-playwright': 4.1.11 + '@vitest/browser-preview': 4.1.11 + '@vitest/browser-webdriverio': 4.1.11 + '@vitest/coverage-istanbul': 4.1.11 + '@vitest/coverage-v8': 4.1.11 + '@vitest/ui': 4.1.11 + happy-dom: '*' + jsdom: '*' + vite: ^6.0.0 || ^7.0.0 || ^8.0.0 + peerDependenciesMeta: + '@edge-runtime/vm': + optional: true + '@opentelemetry/api': + optional: true + '@types/node': + optional: true + '@vitest/browser-playwright': + optional: true + '@vitest/browser-preview': + optional: true + '@vitest/browser-webdriverio': + optional: true + '@vitest/coverage-istanbul': + optional: true + '@vitest/coverage-v8': + optional: true + '@vitest/ui': + optional: true + happy-dom: + optional: true + jsdom: + optional: true + weak-lru-cache@1.2.2: resolution: {integrity: sha512-DEAoo25RfSYMuTGc9vPJzZcZullwIqRDSI9LOy+fkCJPi6hykCnfKaXTuPBDuXAUcqHXyOgFtHNp/kB2FjYHbw==} @@ -2605,10 +3252,42 @@ packages: engines: {node: '>= 8'} hasBin: true + why-is-node-running@2.3.0: + resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==} + engines: {node: '>=8'} + hasBin: true + + workerd@1.20260831.1: + resolution: {integrity: sha512-A2LwrkBel/FnKABPfeBAMiL6v70+rugnunqQRfWsWZjlhsTZoBScWUVunMy/xLCGLjWCQL2zp39AVR6aO0jurQ==} + engines: {node: '>=16'} + hasBin: true + + wrangler@4.128.0: + resolution: {integrity: sha512-jNXy9e8/pbx8iqTzXPiuflnitKJZoAfEUSUUDLW87bwyeMvJ7kb3yQMSbxEcfNdfHqJW38KRcKaLljOYV4N/4w==} + engines: {node: '>=22.0.0'} + hasBin: true + peerDependencies: + '@cloudflare/workers-types': ^5.20260831.1 + peerDependenciesMeta: + '@cloudflare/workers-types': + optional: true + wrap-ansi@7.0.0: resolution: {integrity: sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==} engines: {node: '>=10'} + ws@8.21.0: + resolution: {integrity: sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==} + engines: {node: '>=10.0.0'} + peerDependencies: + bufferutil: ^4.0.1 + utf-8-validate: '>=5.0.2' + peerDependenciesMeta: + bufferutil: + optional: true + utf-8-validate: + optional: true + y18n@5.0.8: resolution: {integrity: sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==} engines: {node: '>=10'} @@ -2621,6 +3300,12 @@ packages: resolution: {integrity: sha512-D1mvvtDG0L5ft/jGWkLpG1+m0eQxOfaBvTNELraWj22wSVUMWxZUvYgJYcKh6jGGIkJFhH4IZPQhR4TKpc8mBw==} engines: {node: '>=10'} + youch-core@0.3.3: + resolution: {integrity: sha512-ho7XuGjLaJ2hWHoK8yFnsUGy2Y5uDpqSTq1FkHLK4/oqKtyUU1AFbOOxY4IpC9f0fTLjwYbslUz0Po5BpD1wrA==} + + youch@4.1.0-beta.10: + resolution: {integrity: sha512-rLfVLB4FgQneDr0dv1oddCVZmKjcJ6yX6mS4pU82Mq/Dt9a3cLZQ62pDBL4AUO+uVrCvtWz3ZFUL2HFAFJ/BXQ==} + zod@4.3.6: resolution: {integrity: sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg==} @@ -2660,9 +3345,51 @@ snapshots: fast-wrap-ansi: 0.2.2 sisteransi: 1.0.5 + '@cloudflare/kv-asset-handler@0.5.0': {} + + '@cloudflare/unenv-preset@2.16.1(unenv@2.0.0-rc.24)(workerd@1.20260831.1)': + dependencies: + unenv: 2.0.0-rc.24 + optionalDependencies: + workerd: 1.20260831.1 + + '@cloudflare/vitest-plugin@1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)))': + dependencies: + '@vitest/runner': 4.1.11 + '@vitest/snapshot': 4.1.11 + cjs-module-lexer: 1.2.3 + esbuild: 0.28.1 + miniflare: 5.20260831.0-alpha + vitest: 4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)) + wrangler: 4.128.0 + zod: 4.4.3 + transitivePeerDependencies: + - '@cloudflare/workers-types' + - bufferutil + - utf-8-validate + + '@cloudflare/workerd-darwin-64@1.20260831.1': + optional: true + + '@cloudflare/workerd-darwin-arm64@1.20260831.1': + optional: true + + '@cloudflare/workerd-linux-64@1.20260831.1': + optional: true + + '@cloudflare/workerd-linux-arm64@1.20260831.1': + optional: true + + '@cloudflare/workerd-windows-64@1.20260831.1': + optional: true + '@colors/colors@1.5.0': optional: true + '@cspotcode/source-map-support@0.8.1': + dependencies: + '@jridgewell/trace-mapping': 0.3.9 + '@durable-streams/client@0.2.2': dependencies: '@microsoft/fetch-event-source': 2.0.1 @@ -2748,6 +3475,11 @@ snapshots: dependencies: effection: 4.1.0 + '@emnapi/runtime@1.11.3': + dependencies: + tslib: 2.8.1 + optional: true + '@esbuild/aix-ppc64@0.27.4': optional: true @@ -2927,6 +3659,112 @@ snapshots: '@harperfast/extended-iterable@1.0.3': {} + '@img/colour@1.1.0': {} + + '@img/sharp-darwin-arm64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-darwin-arm64': 1.3.1 + optional: true + + '@img/sharp-darwin-x64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-darwin-x64': 1.3.1 + optional: true + + '@img/sharp-freebsd-wasm32@0.35.2': + dependencies: + '@img/sharp-wasm32': 0.35.2 + optional: true + + '@img/sharp-libvips-darwin-arm64@1.3.1': + optional: true + + '@img/sharp-libvips-darwin-x64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-arm64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-arm@1.3.1': + optional: true + + '@img/sharp-libvips-linux-ppc64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-riscv64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-s390x@1.3.1': + optional: true + + '@img/sharp-libvips-linux-x64@1.3.1': + optional: true + + '@img/sharp-libvips-linuxmusl-arm64@1.3.1': + optional: true + + '@img/sharp-libvips-linuxmusl-x64@1.3.1': + optional: true + + '@img/sharp-linux-arm64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-arm64': 1.3.1 + optional: true + + '@img/sharp-linux-arm@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-arm': 1.3.1 + optional: true + + '@img/sharp-linux-ppc64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-ppc64': 1.3.1 + optional: true + + '@img/sharp-linux-riscv64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-riscv64': 1.3.1 + optional: true + + '@img/sharp-linux-s390x@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-s390x': 1.3.1 + optional: true + + '@img/sharp-linux-x64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-x64': 1.3.1 + optional: true + + '@img/sharp-linuxmusl-arm64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-arm64': 1.3.1 + optional: true + + '@img/sharp-linuxmusl-x64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-x64': 1.3.1 + optional: true + + '@img/sharp-wasm32@0.35.2': + dependencies: + '@emnapi/runtime': 1.11.3 + optional: true + + '@img/sharp-webcontainers-wasm32@0.35.2': + dependencies: + '@img/sharp-wasm32': 0.35.2 + optional: true + + '@img/sharp-win32-arm64@0.35.2': + optional: true + + '@img/sharp-win32-ia32@0.35.2': + optional: true + + '@img/sharp-win32-x64@0.35.2': + optional: true + '@jest/diff-sequences@30.3.0': {} '@jest/expect-utils@30.3.0': @@ -2954,8 +3792,15 @@ snapshots: '@types/yargs': 17.0.35 chalk: 4.1.2 + '@jridgewell/resolve-uri@3.1.2': {} + '@jridgewell/sourcemap-codec@1.5.5': {} + '@jridgewell/trace-mapping@0.3.9': + dependencies: + '@jridgewell/resolve-uri': 3.1.2 + '@jridgewell/sourcemap-codec': 1.5.5 + '@lmdb/lmdb-darwin-arm64@3.5.6': optional: true @@ -2999,6 +3844,8 @@ snapshots: '@neophi/sieve-cache@1.5.0': {} + '@oxc-project/types@0.148.0': {} + '@oxfmt/binding-android-arm-eabi@0.41.0': optional: true @@ -3113,6 +3960,18 @@ snapshots: '@oxlint/binding-win32-x64-msvc@1.74.0': optional: true + '@poppinss/colors@4.1.6': + dependencies: + kleur: 4.1.5 + + '@poppinss/dumper@0.6.5': + dependencies: + '@poppinss/colors': 4.1.6 + '@sindresorhus/is': 7.2.0 + supports-color: 10.2.2 + + '@poppinss/exception@1.2.3': {} + '@radix-ui/number@1.1.3': {} '@radix-ui/primitive@1.1.7': {} @@ -3464,6 +4323,53 @@ snapshots: lodash: 4.18.1 lodash-es: 4.18.1 + '@rolldown/binding-android-arm-eabi@1.2.7': + optional: true + + '@rolldown/binding-android-arm64@1.2.7': + optional: true + + '@rolldown/binding-darwin-arm64@1.2.7': + optional: true + + '@rolldown/binding-darwin-x64@1.2.7': + optional: true + + '@rolldown/binding-freebsd-x64@1.2.7': + optional: true + + '@rolldown/binding-linux-arm-gnueabihf@1.2.7': + optional: true + + '@rolldown/binding-linux-arm64-gnu@1.2.7': + optional: true + + '@rolldown/binding-linux-arm64-musl@1.2.7': + optional: true + + '@rolldown/binding-linux-ppc64-gnu@1.2.7': + optional: true + + '@rolldown/binding-linux-s390x-gnu@1.2.7': + optional: true + + '@rolldown/binding-linux-x64-gnu@1.2.7': + optional: true + + '@rolldown/binding-linux-x64-musl@1.2.7': + optional: true + + '@rolldown/binding-openharmony-arm64@1.2.7': + optional: true + + '@rolldown/binding-win32-arm64-msvc@1.2.7': + optional: true + + '@rolldown/binding-win32-x64-msvc@1.2.7': + optional: true + + '@rolldown/pluginutils@1.0.1': {} + '@secretlint/core@13.0.4': dependencies: '@secretlint/profiler': 13.0.4 @@ -3483,12 +4389,25 @@ snapshots: '@sindresorhus/is@4.6.0': {} + '@sindresorhus/is@7.2.0': {} + + '@speed-highlight/core@1.2.24': {} + '@standard-schema/spec@1.1.0': {} + '@types/chai@5.2.3': + dependencies: + '@types/deep-eql': 4.0.2 + assertion-error: 2.0.1 + '@types/debug@4.1.12': dependencies: '@types/ms': 2.1.0 + '@types/deep-eql@4.0.2': {} + + '@types/estree@1.0.9': {} + '@types/hast@3.0.5': dependencies: '@types/unist': 3.0.3 @@ -3527,6 +4446,47 @@ snapshots: '@ungap/structured-clone@1.3.3': {} + '@vitest/expect@4.1.11': + dependencies: + '@standard-schema/spec': 1.1.0 + '@types/chai': 5.2.3 + '@vitest/spy': 4.1.11 + '@vitest/utils': 4.1.11 + chai: 6.2.2 + tinyrainbow: 3.1.1 + + '@vitest/mocker@4.1.11(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0))': + dependencies: + '@vitest/spy': 4.1.11 + estree-walker: 3.0.3 + magic-string: 0.30.21 + optionalDependencies: + vite: 8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0) + + '@vitest/pretty-format@4.1.11': + dependencies: + tinyrainbow: 3.1.1 + + '@vitest/runner@4.1.11': + dependencies: + '@vitest/utils': 4.1.11 + pathe: 2.0.3 + + '@vitest/snapshot@4.1.11': + dependencies: + '@vitest/pretty-format': 4.1.11 + '@vitest/utils': 4.1.11 + magic-string: 0.30.21 + pathe: 2.0.3 + + '@vitest/spy@4.1.11': {} + + '@vitest/utils@4.1.11': + dependencies: + '@vitest/pretty-format': 4.1.11 + convert-source-map: 2.0.0 + tinyrainbow: 3.1.1 + '@x0k/json-schema-merge@1.0.4': dependencies: '@types/json-schema': 7.0.15 @@ -3580,6 +4540,8 @@ snapshots: dependencies: tslib: 2.8.1 + assertion-error@2.0.1: {} + b4a@1.8.1: {} bail@2.0.2: {} @@ -3613,12 +4575,16 @@ snapshots: dependencies: bare-path: 3.1.1 + blake3-wasm@2.1.5: {} + boolbase@1.0.0: {} boundary@2.0.0: {} ccount@2.0.1: {} + chai@6.2.2: {} + chalk@4.1.2: dependencies: ansi-styles: 4.3.0 @@ -3636,6 +4602,8 @@ snapshots: ci-info@4.4.0: {} + cjs-module-lexer@1.2.3: {} + class-variance-authority@0.7.1: dependencies: clsx: 2.1.1 @@ -3689,6 +4657,10 @@ snapshots: dependencies: '@standard-schema/spec': 1.1.0 + convert-source-map@2.0.0: {} + + cookie@1.1.1: {} + cross-spawn@7.0.6: dependencies: path-key: 3.1.1 @@ -3725,6 +4697,10 @@ snapshots: environment@1.1.0: {} + error-stack-parser-es@1.0.5: {} + + es-module-lexer@2.3.2: {} + esbuild@0.27.4: optionalDependencies: '@esbuild/aix-ppc64': 0.27.4 @@ -3789,12 +4765,18 @@ snapshots: esprima@4.0.1: {} + estree-walker@3.0.3: + dependencies: + '@types/estree': 1.0.9 + events-universal@1.0.1: dependencies: bare-events: 2.9.1 transitivePeerDependencies: - bare-abort-controller + expect-type@1.4.0: {} + expect@30.3.0: dependencies: '@jest/expect-utils': 30.3.0 @@ -3834,6 +4816,10 @@ snapshots: dependencies: reusify: 1.1.0 + fdir@6.5.0(picomatch@4.0.7): + optionalDependencies: + picomatch: 4.0.7 + fsevents@2.3.3: optional: true @@ -3950,6 +4936,57 @@ snapshots: kind-of@6.0.3: {} + kleur@4.1.5: {} + + lightningcss-android-arm64@1.33.0: + optional: true + + lightningcss-darwin-arm64@1.33.0: + optional: true + + lightningcss-darwin-x64@1.33.0: + optional: true + + lightningcss-freebsd-x64@1.33.0: + optional: true + + lightningcss-linux-arm-gnueabihf@1.33.0: + optional: true + + lightningcss-linux-arm64-gnu@1.33.0: + optional: true + + lightningcss-linux-arm64-musl@1.33.0: + optional: true + + lightningcss-linux-x64-gnu@1.33.0: + optional: true + + lightningcss-linux-x64-musl@1.33.0: + optional: true + + lightningcss-win32-arm64-msvc@1.33.0: + optional: true + + lightningcss-win32-x64-msvc@1.33.0: + optional: true + + lightningcss@1.33.0: + dependencies: + detect-libc: 2.1.2 + optionalDependencies: + lightningcss-android-arm64: 1.33.0 + lightningcss-darwin-arm64: 1.33.0 + lightningcss-darwin-x64: 1.33.0 + lightningcss-freebsd-x64: 1.33.0 + lightningcss-linux-arm-gnueabihf: 1.33.0 + lightningcss-linux-arm64-gnu: 1.33.0 + lightningcss-linux-arm64-musl: 1.33.0 + lightningcss-linux-x64-gnu: 1.33.0 + lightningcss-linux-x64-musl: 1.33.0 + lightningcss-win32-arm64-msvc: 1.33.0 + lightningcss-win32-x64-msvc: 1.33.0 + lmdb@3.5.6: dependencies: '@harperfast/extended-iterable': 1.0.3 @@ -4185,6 +5222,18 @@ snapshots: transitivePeerDependencies: - supports-color + miniflare@5.20260831.0-alpha: + dependencies: + '@cspotcode/source-map-support': 0.8.1 + sharp: 0.35.2 + undici: 7.29.0 + workerd: 1.20260831.1 + ws: 8.21.0 + youch: 4.1.0-beta.10 + transitivePeerDependencies: + - bufferutil + - utf-8-validate + ms@2.1.3: {} msgpackr-extract@3.0.4: @@ -4209,6 +5258,8 @@ snapshots: object-assign: 4.1.1 thenify-all: 1.6.0 + nanoid@3.3.18: {} + node-addon-api@6.1.0: {} node-emoji@2.2.0: @@ -4228,6 +5279,8 @@ snapshots: object-assign@4.1.1: {} + obug@2.1.4: {} + ordered-binary@1.6.1: {} oxfmt@0.41.0: @@ -4286,10 +5339,22 @@ snapshots: path-key@3.1.1: {} + path-to-regexp@6.3.0: {} + + pathe@2.0.3: {} + picocolors@1.1.1: {} picomatch@4.0.3: {} + picomatch@4.0.7: {} + + postcss@8.5.26: + dependencies: + nanoid: 3.3.18 + picocolors: 1.1.1 + source-map-js: 1.2.1 + pretty-format@30.3.0: dependencies: '@jest/schemas': 30.0.5 @@ -4391,6 +5456,27 @@ snapshots: reusify@1.1.0: {} + rolldown@1.2.7: + dependencies: + '@oxc-project/types': 0.148.0 + '@rolldown/pluginutils': 1.0.1 + optionalDependencies: + '@rolldown/binding-android-arm-eabi': 1.2.7 + '@rolldown/binding-android-arm64': 1.2.7 + '@rolldown/binding-darwin-arm64': 1.2.7 + '@rolldown/binding-darwin-x64': 1.2.7 + '@rolldown/binding-freebsd-x64': 1.2.7 + '@rolldown/binding-linux-arm-gnueabihf': 1.2.7 + '@rolldown/binding-linux-arm64-gnu': 1.2.7 + '@rolldown/binding-linux-arm64-musl': 1.2.7 + '@rolldown/binding-linux-ppc64-gnu': 1.2.7 + '@rolldown/binding-linux-s390x-gnu': 1.2.7 + '@rolldown/binding-linux-x64-gnu': 1.2.7 + '@rolldown/binding-linux-x64-musl': 1.2.7 + '@rolldown/binding-openharmony-arm64': 1.2.7 + '@rolldown/binding-win32-arm64-msvc': 1.2.7 + '@rolldown/binding-win32-x64-msvc': 1.2.7 + scheduler@0.27.0: {} section-matter@1.0.0: @@ -4398,6 +5484,40 @@ snapshots: extend-shallow: 2.0.1 kind-of: 6.0.3 + semver@7.8.5: {} + + sharp@0.35.2: + dependencies: + '@img/colour': 1.1.0 + detect-libc: 2.1.2 + semver: 7.8.5 + optionalDependencies: + '@img/sharp-darwin-arm64': 0.35.2 + '@img/sharp-darwin-x64': 0.35.2 + '@img/sharp-freebsd-wasm32': 0.35.2 + '@img/sharp-libvips-darwin-arm64': 1.3.1 + '@img/sharp-libvips-darwin-x64': 1.3.1 + '@img/sharp-libvips-linux-arm': 1.3.1 + '@img/sharp-libvips-linux-arm64': 1.3.1 + '@img/sharp-libvips-linux-ppc64': 1.3.1 + '@img/sharp-libvips-linux-riscv64': 1.3.1 + '@img/sharp-libvips-linux-s390x': 1.3.1 + '@img/sharp-libvips-linux-x64': 1.3.1 + '@img/sharp-libvips-linuxmusl-arm64': 1.3.1 + '@img/sharp-libvips-linuxmusl-x64': 1.3.1 + '@img/sharp-linux-arm': 0.35.2 + '@img/sharp-linux-arm64': 0.35.2 + '@img/sharp-linux-ppc64': 0.35.2 + '@img/sharp-linux-riscv64': 0.35.2 + '@img/sharp-linux-s390x': 0.35.2 + '@img/sharp-linux-x64': 0.35.2 + '@img/sharp-linuxmusl-arm64': 0.35.2 + '@img/sharp-linuxmusl-x64': 0.35.2 + '@img/sharp-webcontainers-wasm32': 0.35.2 + '@img/sharp-win32-arm64': 0.35.2 + '@img/sharp-win32-ia32': 0.35.2 + '@img/sharp-win32-x64': 0.35.2 + shebang-command@2.0.0: dependencies: shebang-regex: 3.0.0 @@ -4406,6 +5526,8 @@ snapshots: shellwords-ts@3.0.1: {} + siginfo@2.0.0: {} + sisteransi@1.0.5: {} skillflag@0.2.1: @@ -4423,6 +5545,8 @@ snapshots: slash@3.0.0: {} + source-map-js@1.2.1: {} + space-separated-tokens@2.0.2: {} sprintf-js@1.0.3: {} @@ -4431,6 +5555,10 @@ snapshots: dependencies: escape-string-regexp: 2.0.0 + stackback@0.0.2: {} + + std-env@4.2.0: {} + streamx@2.28.0: dependencies: events-universal: 1.0.1 @@ -4461,6 +5589,8 @@ snapshots: dependencies: boundary: 2.0.0 + supports-color@10.2.2: {} + supports-color@7.2.0: dependencies: has-flag: 4.0.0 @@ -4506,8 +5636,19 @@ snapshots: dependencies: any-promise: 1.3.0 + tinybench@2.9.0: {} + + tinyexec@1.3.0: {} + + tinyglobby@0.2.17: + dependencies: + fdir: 6.5.0(picomatch@4.0.7) + picomatch: 4.0.7 + tinypool@2.1.0: {} + tinyrainbow@3.1.1: {} + trim-lines@3.0.1: {} trough@2.2.0: {} @@ -4531,6 +5672,12 @@ snapshots: undici-types@6.21.0: {} + undici@7.29.0: {} + + unenv@2.0.0-rc.24: + dependencies: + pathe: 2.0.3 + unicode-emoji-modifier-base@1.0.0: {} unified@11.0.5: @@ -4597,18 +5744,89 @@ snapshots: '@types/unist': 3.0.3 vfile-message: 4.0.3 + vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0): + dependencies: + lightningcss: 1.33.0 + picomatch: 4.0.7 + postcss: 8.5.26 + rolldown: 1.2.7 + tinyglobby: 0.2.17 + optionalDependencies: + '@types/node': 22.19.15 + esbuild: 0.28.1 + fsevents: 2.3.3 + tsx: 4.21.0 + + vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)): + dependencies: + '@vitest/expect': 4.1.11 + '@vitest/mocker': 4.1.11(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)) + '@vitest/pretty-format': 4.1.11 + '@vitest/runner': 4.1.11 + '@vitest/snapshot': 4.1.11 + '@vitest/spy': 4.1.11 + '@vitest/utils': 4.1.11 + es-module-lexer: 2.3.2 + expect-type: 1.4.0 + magic-string: 0.30.21 + obug: 2.1.4 + pathe: 2.0.3 + picomatch: 4.0.3 + std-env: 4.2.0 + tinybench: 2.9.0 + tinyexec: 1.3.0 + tinyglobby: 0.2.17 + tinyrainbow: 3.1.1 + vite: 8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0) + why-is-node-running: 2.3.0 + optionalDependencies: + '@types/node': 22.19.15 + transitivePeerDependencies: + - msw + weak-lru-cache@1.2.2: {} which@2.0.2: dependencies: isexe: 2.0.0 + why-is-node-running@2.3.0: + dependencies: + siginfo: 2.0.0 + stackback: 0.0.2 + + workerd@1.20260831.1: + optionalDependencies: + '@cloudflare/workerd-darwin-64': 1.20260831.1 + '@cloudflare/workerd-darwin-arm64': 1.20260831.1 + '@cloudflare/workerd-linux-64': 1.20260831.1 + '@cloudflare/workerd-linux-arm64': 1.20260831.1 + '@cloudflare/workerd-windows-64': 1.20260831.1 + + wrangler@4.128.0: + dependencies: + '@cloudflare/kv-asset-handler': 0.5.0 + '@cloudflare/unenv-preset': 2.16.1(unenv@2.0.0-rc.24)(workerd@1.20260831.1) + blake3-wasm: 2.1.5 + esbuild: 0.28.1 + miniflare: 5.20260831.0-alpha + path-to-regexp: 6.3.0 + unenv: 2.0.0-rc.24 + workerd: 1.20260831.1 + optionalDependencies: + fsevents: 2.3.3 + transitivePeerDependencies: + - bufferutil + - utf-8-validate + wrap-ansi@7.0.0: dependencies: ansi-styles: 4.3.0 string-width: 4.2.3 strip-ansi: 6.0.1 + ws@8.21.0: {} + y18n@5.0.8: {} yargs-parser@20.2.9: {} @@ -4623,6 +5841,19 @@ snapshots: y18n: 5.0.8 yargs-parser: 20.2.9 + youch-core@0.3.3: + dependencies: + '@poppinss/exception': 1.2.3 + error-stack-parser-es: 1.0.5 + + youch@4.1.0-beta.10: + dependencies: + '@poppinss/colors': 4.1.6 + '@poppinss/dumper': 0.6.5 + '@speed-highlight/core': 1.2.24 + cookie: 1.1.1 + youch-core: 0.3.3 + zod@4.3.6: {} zod@4.4.3: {} From 299ca744522b233c2851f9f02672e4a3b1d14130 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 17:17:20 -0400 Subject: [PATCH 10/81] =?UTF-8?q?=F0=9F=94=92=20Narrow=20the=20software-fa?= =?UTF-8?q?ctory=20seam=20to=20the=20product=20promise=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `@executablemd/workflow/software-factory` now publishes only `deriveFactoryRunId`, `admitFactoryRunSubject`, `FactoryRunSubject`, `FactoryRunSubjectFailure` and `FactoryRunSubjectError`. The scheme tag, the Base32 alphabet, the authority rule, the preimage layout, the encoder and the length constant are implementation: a caller that could reach them could also reimplement the hash, and two implementations of an identity that must agree byte for byte is the failure §1.1 exists to prevent. `factoryRunIdPreimage()` and `base32Unpadded()` stay visible to their own module so the encoding tests can prove an internal algorithm, and are absent from the entrypoint. Everything the tests assert about public behavior — the fixed vectors, the 52-character shape, case folding, the non-default port, and every authority and node-id refusal — now goes through `admitFactoryRunSubject()` and `deriveFactoryRunId()`. The `as BufferSource` assertion is gone. `factoryRunIdPreimage()` returns an `ArrayBuffer` rather than a view, which is what `crypto.subtle.digest()` accepts with no assertion at the call site, so Code Rule 6 holds without one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/software-factory.ts | 11 +-- .../workflow/src/software-factory/run-id.ts | 19 +++--- .../tests/software-factory-run-id.test.ts | 67 ++++++++----------- 3 files changed, 44 insertions(+), 53 deletions(-) diff --git a/packages/workflow/software-factory.ts b/packages/workflow/software-factory.ts index 5f5f69320..e5d70442c 100644 --- a/packages/workflow/software-factory.ts +++ b/packages/workflow/software-factory.ts @@ -27,6 +27,12 @@ * `specs/github-actions-software-factory-spec.md` §1.1 and restated in * `specs/workflow-spec.md` §9.1. * + * The seam is deliberately small: admit a subject, or derive its id. The scheme + * tag, the Base32 alphabet, the authority rule, the preimage layout and the + * encoder are implementation, not promises — a caller that could reach them + * could also reimplement the hash, and two implementations of an identity that + * must agree byte for byte is the failure §1.1 exists to prevent. + * * Nothing here is runtime-specific. It uses the cross-runtime Web primitives — * `TextEncoder` and `crypto.subtle` — and names no host, so the provider host * and a GitHub intake reach the same single implementation rather than each @@ -35,12 +41,7 @@ export { admitFactoryRunSubject, - admitIssueNodeId, - base32Unpadded, - canonicalGitHubAuthority, deriveFactoryRunId, - FACTORY_RUN_ID_LENGTH, - factoryRunIdPreimage, FactoryRunSubjectError, } from "./src/software-factory/run-id.ts"; export type { FactoryRunSubject, FactoryRunSubjectFailure } from "./src/software-factory/run-id.ts"; diff --git a/packages/workflow/src/software-factory/run-id.ts b/packages/workflow/src/software-factory/run-id.ts index 5ad6d0474..16b5e8b54 100644 --- a/packages/workflow/src/software-factory/run-id.ts +++ b/packages/workflow/src/software-factory/run-id.ts @@ -37,7 +37,7 @@ const BASE32_ALPHABET = "abcdefghijklmnopqrstuvwxyz234567"; * unpadded encoding is `ceil(256 / 5)` characters. Stated rather than computed * because it is a contract a second implementation is held to. */ -export const FACTORY_RUN_ID_LENGTH = 52; +const FACTORY_RUN_ID_LENGTH = 52; /** Why a subject could not be turned into a run id. */ export type FactoryRunSubjectFailure = @@ -103,7 +103,7 @@ const HOSTNAME = * usable is a value somebody meant differently, and two spellings that both * became one authority would be two runs quietly becoming one. */ -export function canonicalGitHubAuthority(value: string): string { +function canonicalGitHubAuthority(value: string): string { if (value === "") { throw new FactoryRunSubjectError("authority-empty", "the authority is empty"); } @@ -166,7 +166,7 @@ export function canonicalGitHubAuthority(value: string): string { } /** Hold a node id to what a retained identity has to be, and change nothing about it. */ -export function admitIssueNodeId(value: string): string { +function admitIssueNodeId(value: string): string { if (value === "") { throw new FactoryRunSubjectError("node-id-empty", "the issue node id is empty"); } @@ -183,12 +183,15 @@ export function admitIssueNodeId(value: string): string { * UTF-8. The NULs are separators the inputs cannot contain, so no pair of * (authority, node id) can be rearranged into another pair with the same bytes. */ -export function factoryRunIdPreimage(subject: FactoryRunSubject): Uint8Array { +export function factoryRunIdPreimage(subject: FactoryRunSubject): ArrayBuffer { const encoder = new TextEncoder(); const scheme = encoder.encode(SCHEME); const authority = encoder.encode(subject.authority); const node = encoder.encode(subject.issueNodeId); - const bytes = new Uint8Array(scheme.length + 1 + authority.length + 1 + node.length); + // An `ArrayBuffer` rather than a view, because that is what `crypto.subtle` + // accepts without anything having to assert a type at the call site. + const buffer = new ArrayBuffer(scheme.length + 1 + authority.length + 1 + node.length); + const bytes = new Uint8Array(buffer); let at = 0; bytes.set(scheme, at); at += scheme.length; @@ -199,7 +202,7 @@ export function factoryRunIdPreimage(subject: FactoryRunSubject): Uint8Array { bytes[at] = 0; at += 1; bytes.set(node, at); - return bytes; + return buffer; } /** Lowercase unpadded RFC 4648 Base32 of exactly these bytes. */ @@ -244,8 +247,6 @@ export function admitFactoryRunSubject(subject: FactoryRunSubject): FactoryRunSu */ export function* deriveFactoryRunId(subject: FactoryRunSubject): Operation { const admitted = admitFactoryRunSubject(subject); - const digest = yield* until( - crypto.subtle.digest("SHA-256", factoryRunIdPreimage(admitted) as BufferSource), - ); + const digest = yield* until(crypto.subtle.digest("SHA-256", factoryRunIdPreimage(admitted))); return base32Unpadded(new Uint8Array(digest)); } diff --git a/packages/workflow/tests/software-factory-run-id.test.ts b/packages/workflow/tests/software-factory-run-id.test.ts index faff85d3b..2e259b567 100644 --- a/packages/workflow/tests/software-factory-run-id.test.ts +++ b/packages/workflow/tests/software-factory-run-id.test.ts @@ -13,13 +13,13 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { admitFactoryRunSubject, - base32Unpadded, - canonicalGitHubAuthority, deriveFactoryRunId, - FACTORY_RUN_ID_LENGTH, - factoryRunIdPreimage, FactoryRunSubjectError, } from "@executablemd/workflow/software-factory"; +// The encoder is an internal algorithm rather than a public promise, so the +// RFC 4648 vectors below reach it directly. Everything else in this file goes +// through the published product API, which is what a host actually holds. +import { base32Unpadded, factoryRunIdPreimage } from "../src/software-factory/run-id.ts"; /** An opaque node id of the shape GitHub's GraphQL API returns for an issue. */ const NODE = "I_kwDOABCD12M5abcdef"; @@ -48,6 +48,11 @@ const VECTORS = [ }, ] as const; +/** The canonical authority a subject is admitted under, through the public seam. */ +function authorityOf(value: string): string { + return admitFactoryRunSubject({ authority: value, issueNodeId: NODE }).authority; +} + function reason(body: () => unknown): string { try { body(); @@ -68,7 +73,7 @@ describe("the factory run id", () => { issueNodeId: vector.issueNodeId, }); expect(runId).toEqual(vector.runId); - expect(runId.length).toEqual(FACTORY_RUN_ID_LENGTH); + expect(runId.length).toEqual(52); expect(/^[a-z2-7]{52}$/.test(runId)).toEqual(true); } }); @@ -87,51 +92,35 @@ describe("the factory run id", () => { }); it("writes the scheme tag, both separators and both inputs", function* () { - const bytes = factoryRunIdPreimage({ authority: "github.com", issueNodeId: "x" }); + const bytes = new Uint8Array( + factoryRunIdPreimage({ authority: "github.com", issueNodeId: "x" }), + ); expect(new TextDecoder().decode(bytes)).toEqual("github-issue-v1\0github.com\0x"); expect([...bytes].filter((byte) => byte === 0).length).toEqual(2); }); it("folds case and keeps a non-default port", function* () { - expect(canonicalGitHubAuthority("GitHub.Com")).toEqual("github.com"); - expect(canonicalGitHubAuthority("GitHub.Example.COM:8443")).toEqual("github.example.com:8443"); + expect(authorityOf("GitHub.Com")).toEqual("github.com"); + expect(authorityOf("GitHub.Example.COM:8443")).toEqual("github.example.com:8443"); }); it("refuses every part an authority may not carry", function* () { - expect(reason(() => canonicalGitHubAuthority(""))).toEqual("authority-empty"); - expect(reason(() => canonicalGitHubAuthority("https://github.com"))).toEqual( - "authority-has-scheme", - ); - expect(reason(() => canonicalGitHubAuthority("user@github.com"))).toEqual( - "authority-has-userinfo", - ); - expect(reason(() => canonicalGitHubAuthority("github.com/octo"))).toEqual("authority-has-path"); - expect(reason(() => canonicalGitHubAuthority("github.com/"))).toEqual("authority-has-path"); - expect(reason(() => canonicalGitHubAuthority("github.com?a=b"))).toEqual("authority-has-query"); - expect(reason(() => canonicalGitHubAuthority("github.com#top"))).toEqual( - "authority-has-fragment", - ); - expect(reason(() => canonicalGitHubAuthority("git hub.com"))).toEqual( - "authority-has-whitespace", - ); - expect(reason(() => canonicalGitHubAuthority("-github.com"))).toEqual( - "authority-malformed-host", - ); - expect(reason(() => canonicalGitHubAuthority("github.com:https"))).toEqual( - "authority-malformed-port", - ); - expect(reason(() => canonicalGitHubAuthority("github.com:0"))).toEqual( - "authority-malformed-port", - ); - expect(reason(() => canonicalGitHubAuthority("github.com:70000"))).toEqual( - "authority-malformed-port", - ); + expect(reason(() => authorityOf(""))).toEqual("authority-empty"); + expect(reason(() => authorityOf("https://github.com"))).toEqual("authority-has-scheme"); + expect(reason(() => authorityOf("user@github.com"))).toEqual("authority-has-userinfo"); + expect(reason(() => authorityOf("github.com/octo"))).toEqual("authority-has-path"); + expect(reason(() => authorityOf("github.com/"))).toEqual("authority-has-path"); + expect(reason(() => authorityOf("github.com?a=b"))).toEqual("authority-has-query"); + expect(reason(() => authorityOf("github.com#top"))).toEqual("authority-has-fragment"); + expect(reason(() => authorityOf("git hub.com"))).toEqual("authority-has-whitespace"); + expect(reason(() => authorityOf("-github.com"))).toEqual("authority-malformed-host"); + expect(reason(() => authorityOf("github.com:https"))).toEqual("authority-malformed-port"); + expect(reason(() => authorityOf("github.com:0"))).toEqual("authority-malformed-port"); + expect(reason(() => authorityOf("github.com:70000"))).toEqual("authority-malformed-port"); }); it("refuses a default port written out, so one deployment has one spelling", function* () { - expect(reason(() => canonicalGitHubAuthority("github.com:443"))).toEqual( - "authority-default-port", - ); + expect(reason(() => authorityOf("github.com:443"))).toEqual("authority-default-port"); }); it("compares a node id byte for byte", function* () { From 28814ef12bd04a9571764f9713a3c303a2b256b6 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 17:28:54 -0400 Subject: [PATCH 11/81] =?UTF-8?q?=F0=9F=A7=AA=20Stand=20up=20the=20workerd?= =?UTF-8?q?=20suite=20and=20measure=20real=20Durable=20Object=20storage=20?= =?UTF-8?q?(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan §12's harness, and the first thing it was built to answer. `vitest.config.ts` at the repository root runs `*.vitest.ts` under `@cloudflare/vitest-plugin` against real workerd, with a wrangler config declaring `StorageProbeObject` as a `new_sqlite_classes` Durable Object. The suffix keeps these files out of `deno task test`, `pnpm test:node` and `bun run test:bun`: discovery matches `*.test.ts`, and `scripts/tests/test-file-discovery.test.ts` still agrees with it, so nothing is stranded and no runtime exclusion was needed. `pnpm test:cloudflare` and `deno task test:cloudflare` run it. Two things had to be settled to get it running. The plugin binds to `vitest` through peer dependencies, and pnpm was resolving a second `vitest@4.1.11` copy with a different peer set, so the pool was configured but never took over and every test reported "failed to find the current suite" — adding the plugin's declared `@vitest/runner` and `@vitest/snapshot` peers at the root collapses that. The config lives at the repository root rather than in the package, because a package-level config resolves `vitest` from the package's own isolated `node_modules` and multiplies the copies again. `StorageProbeObject` asks the runtime what it accepts rather than assuming. Recorded on workerd 1.20260831.1: - `PRAGMA application_id` and `PRAGMA user_version` — refused, read and write, with `not authorized: SQLITE_AUTH`. - `sqlite_schema` introspection, ordinary DDL, and a plain metadata table — all fine. - One outer `ctx.storage.transactionSync()` — fine. - A reentrant `transactionSync`, and a direct `SAVEPOINT` — refused: the runtime requires its own transaction API instead of SQL transaction statements. - DOFS schema initialization and DOFS filesystem writes outside a transaction — fine. - A DOFS filesystem write *inside* one `transactionSync` — refused, because `writeFileSync` opens a `transactionSync` of its own and that nests. The last two contradict plan §4, which has the owner reuse the pragma-based recognition the Deno host uses and commit Workspace mutations inside one `ctx.storage.transactionSync()`. Neither is possible as written. The probe is committed so the finding is reproducible rather than reported. An earlier version of the probe called the asynchronous `WorkspaceFilesystem` wrapper and never awaited it, which reported success for work that had failed. It now uses the same synchronous primitives the Deno provider imports. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- deno.json | 13 +- deno.lock | 4 + package.json | 7 +- .../workflow/src/cloudflare/probe-object.ts | 141 ++++++++++++++++++ packages/workflow/tests/cloudflare/env.d.ts | 5 + .../cloudflare/storage-capabilities.vitest.ts | 16 ++ packages/workflow/tests/cloudflare/worker.ts | 15 ++ .../workflow/tests/cloudflare/wrangler.jsonc | 10 ++ .../workflow/tests/host-neutrality.test.ts | 10 +- pnpm-lock.yaml | 8 +- vitest.config.ts | 30 ++++ 11 files changed, 252 insertions(+), 7 deletions(-) create mode 100644 packages/workflow/src/cloudflare/probe-object.ts create mode 100644 packages/workflow/tests/cloudflare/env.d.ts create mode 100644 packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts create mode 100644 packages/workflow/tests/cloudflare/worker.ts create mode 100644 packages/workflow/tests/cloudflare/wrangler.jsonc create mode 100644 vitest.config.ts diff --git a/deno.json b/deno.json index 97508df8b..c04d787ee 100644 --- a/deno.json +++ b/deno.json @@ -1,5 +1,8 @@ { - "workspace": ["packages/*", "site"], + "workspace": [ + "packages/*", + "site" + ], "exclude": [ "scripts/tests/fixtures", ".xmd-eval", @@ -7,7 +10,10 @@ "packages/workflow/vendor/cloudflare-computer-dofs/upstream", "packages/workflow/vendor/cloudflare-computer-dofs/generated/**/*.d.ts", "packages/acp/vendor/acpx/upstream", - "packages/acp/vendor/acpx/generated/**/*.d.ts" + "packages/acp/vendor/acpx/generated/**/*.d.ts", + "packages/workflow/src/cloudflare", + "packages/workflow/tests/cloudflare", + "vitest.config.ts" ], "nodeModulesDir": "auto", "lock": { @@ -71,6 +77,7 @@ "review:local": "deno run --allow-all packages/cli/src/deno.ts run .reviews/ReviewPR.local.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.local.jsonl", "analyze": "deno run --allow-all packages/cli/src/deno.ts run .reviews/AnalyzeRepo.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.analyze.jsonl", "analyze:ci": "deno run --allow-all packages/cli/src/deno.ts run .reviews/AnalyzeRepoCI.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.analyze.ci.jsonl", - "analyze:dispatch": "deno run --allow-all packages/cli/src/deno.ts run .reviews/DispatchRepoAnalysis.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.dispatch.jsonl" + "analyze:dispatch": "deno run --allow-all packages/cli/src/deno.ts run .reviews/DispatchRepoAnalysis.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.dispatch.jsonl", + "test:cloudflare": "pnpm test:cloudflare" } } diff --git a/deno.lock b/deno.lock index 6cf3f3547..43038154b 100644 --- a/deno.lock +++ b/deno.lock @@ -75,6 +75,8 @@ "npm:@types/babel__core@^7.20.5": "7.20.5", "npm:@types/node@22": "22.19.15", "npm:@types/node@^24.5.2": "24.13.3", + "npm:@vitest/runner@4.1.11": "4.1.11", + "npm:@vitest/snapshot@4.1.11": "4.1.11", "npm:acorn@^8.16.0": "8.16.0", "npm:acpx@0.12.0": "0.12.0", "npm:ajv@8.20.0": "8.20.0", @@ -4535,6 +4537,8 @@ "npm:@effectionx/test-adapter@0.7.4", "npm:@effectionx/timebox@0.4.3", "npm:@types/node@22", + "npm:@vitest/runner@4.1.11", + "npm:@vitest/snapshot@4.1.11", "npm:acorn@^8.16.0", "npm:ajv@^8.17.1", "npm:effection@4.1.0", diff --git a/package.json b/package.json index 950a7f2d0..a4f74091e 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "private": true, "type": "module", - "description": "executable.md — treat markdown documents as executable workflows.", + "description": "executable.md \u2014 treat markdown documents as executable workflows.", "homepage": "https://executable.md", "repository": { "type": "git", @@ -62,6 +62,8 @@ "@executablemd/testing": "workspace:*", "@executablemd/workflow": "workspace:*", "@types/node": "^22.0.0", + "@vitest/runner": "4.1.11", + "@vitest/snapshot": "4.1.11", "expect": "^30.0.0", "oxfmt": "^0.41.0", "oxlint": "1.74.0", @@ -74,7 +76,8 @@ "test:bun": "bun scripts/runtime-tests.ts bun", "test:deno": "deno task test", "lint": "oxlint -c .oxlintrc.json --ignore-pattern 'scripts/tests/fixtures/**' --ignore-pattern '**/npm/**' --ignore-pattern 'packages/workflow/vendor/cloudflare-computer-dofs/**' --ignore-pattern 'packages/acp/vendor/acpx/**' packages scripts .reviews/components && oxfmt --check packages scripts .reviews/components/*.ts", - "fmt": "oxfmt --write packages scripts .reviews/components/*.ts" + "fmt": "oxfmt --write packages scripts .reviews/components/*.ts", + "test:cloudflare": "vitest run --config vitest.config.ts" }, "workspaces": [ "packages/*" diff --git a/packages/workflow/src/cloudflare/probe-object.ts b/packages/workflow/src/cloudflare/probe-object.ts new file mode 100644 index 000000000..fa65555c2 --- /dev/null +++ b/packages/workflow/src/cloudflare/probe-object.ts @@ -0,0 +1,141 @@ +/** + * A Durable Object that answers what its own SQLite storage can actually do. + * + * The version-1 schema is recognized through `PRAGMA application_id` and + * `PRAGMA user_version`, and the vendored DOFS `Database` opens reentrant + * transactions with `SAVEPOINT` through `sql.exec`. Cloudflare's own + * documentation says `sql.exec()` cannot execute transaction statements and + * says nothing about those two pragmas, so neither assumption can be settled + * from prose — the owner either has the same recognition contract the Deno host + * has, or it does not, and that decides how §4 is written rather than being a + * detail inside it. + * + * So this object exists to be asked, on real workerd, before anything is built + * on the answer. + */ + +import { DurableObject } from "cloudflare:workers"; +import { Database as DofsDatabase } from "../../vendor/cloudflare-computer-dofs/generated/storage.js"; +import { initializeSchema as initializeDofsSchema } from "../../vendor/cloudflare-computer-dofs/generated/schema/index.js"; +import { mkdir as mkdirPath } from "../../vendor/cloudflare-computer-dofs/generated/fs/mkdir.js"; +import { writeFileSync } from "../../vendor/cloudflare-computer-dofs/generated/fs/writeFile.js"; + +export interface StorageCapabilities { + readonly applicationIdRead: string; + readonly applicationIdWrite: string; + readonly userVersionRead: string; + readonly userVersionWrite: string; + readonly schemaObjects: string; + readonly outerTransaction: string; + readonly nestedTransaction: string; + readonly savepointDirect: string; + readonly dofsSchema: string; + readonly dofsFilesystem: string; + readonly xmdTableDdl: string; + readonly metadataTable: string; + readonly filesystemInsideTransaction: string; +} + +/** Run `body`, reporting what it answered or how it refused, never throwing. */ +function attempt(body: () => unknown): string { + try { + const value = body(); + return `ok:${JSON.stringify(value ?? null)}`; + } catch (error) { + return `refused:${error instanceof Error ? error.message : String(error)}`; + } +} + +export class StorageProbeObject extends DurableObject { + capabilities(): StorageCapabilities { + const sql = this.ctx.storage.sql; + const dofs = new DofsDatabase(this.ctx.storage); + return { + applicationIdWrite: attempt(() => { + sql.exec("PRAGMA application_id = 1701078349"); + return "written"; + }), + applicationIdRead: attempt(() => sql.exec("PRAGMA application_id").toArray()), + userVersionWrite: attempt(() => { + sql.exec("PRAGMA user_version = 1"); + return "written"; + }), + userVersionRead: attempt(() => sql.exec("PRAGMA user_version").toArray()), + schemaObjects: attempt(() => + sql.exec("SELECT type, name FROM sqlite_schema WHERE name NOT LIKE 'sqlite_%'").toArray(), + ), + outerTransaction: attempt(() => { + dofs.transactionSync(() => { + sql.exec("CREATE TABLE IF NOT EXISTS probe_outer (id INTEGER PRIMARY KEY)"); + }); + return "committed"; + }), + // The one the documentation forbids: a reentrant transactionSync issues + // SAVEPOINT through sql.exec while the outer transaction is open. + nestedTransaction: attempt(() => { + dofs.transactionSync(() => { + dofs.transactionSync(() => { + sql.exec("CREATE TABLE IF NOT EXISTS probe_nested (id INTEGER PRIMARY KEY)"); + }); + }); + return "committed"; + }), + savepointDirect: attempt(() => { + sql.exec("SAVEPOINT probe_sp"); + sql.exec("RELEASE probe_sp"); + return "accepted"; + }), + // Does the vendored DOFS install its own schema against real storage, + // and does doing so nest a transaction on the way? + dofsSchema: attempt(() => { + initializeDofsSchema(dofs, () => 0); + return "initialized"; + }), + // And does its filesystem work afterwards — the operation the owner would + // perform for every Workspace mutation. + dofsFilesystem: attempt(() => { + mkdirPath(dofs, "/probe", { recursive: true }, () => 0); + // The probe measures what this exact synchronous primitive does on real + // storage, so the asynchronous alternative would answer a different + // question than the one being asked. + // oxlint-disable-next-line local/no-sync-filesystem + writeFileSync(dofs, "/probe/one.txt", new TextEncoder().encode("hello"), {}, () => 0); + return "written"; + }), + // An ordinary XMD table, to show plain DDL is not what is refused. + xmdTableDdl: attempt(() => { + sql.exec("CREATE TABLE IF NOT EXISTS workflow_run (run_id TEXT PRIMARY KEY NOT NULL)"); + sql.exec("INSERT OR REPLACE INTO workflow_run (run_id) VALUES (?)", "probe"); + return sql.exec("SELECT run_id FROM workflow_run").toArray(); + }), + // The exact shape §4 mandates for an owner commit: DOFS filesystem work + // inside one `transactionSync`. If DOFS opens a transaction of its own on + // that path it becomes a reentrant call, which the runtime refuses. + filesystemInsideTransaction: attempt(() => { + dofs.transactionSync(() => { + mkdirPath(dofs, "/inside", { recursive: true }, () => 0); + // oxlint-disable-next-line local/no-sync-filesystem + writeFileSync( + dofs, + "/inside/two.txt", + new TextEncoder().encode("committed"), + {}, + () => 0, + ); + }); + return "committed"; + }), + // The shape a replacement for the pragmas would have to take. + metadataTable: attempt(() => { + sql.exec( + "CREATE TABLE IF NOT EXISTS xmd_schema (key TEXT PRIMARY KEY NOT NULL, value INTEGER NOT NULL)", + ); + sql.exec( + "INSERT OR REPLACE INTO xmd_schema (key, value) VALUES ('application_id', ?)", + 1701078349, + ); + return sql.exec("SELECT key, value FROM xmd_schema").toArray(); + }), + }; + } +} diff --git a/packages/workflow/tests/cloudflare/env.d.ts b/packages/workflow/tests/cloudflare/env.d.ts new file mode 100644 index 000000000..d81369bca --- /dev/null +++ b/packages/workflow/tests/cloudflare/env.d.ts @@ -0,0 +1,5 @@ +declare module "cloudflare:test" { + interface ProvidedEnv { + STORAGE_PROBE: DurableObjectNamespace; + } +} diff --git a/packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts b/packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts new file mode 100644 index 000000000..eda9754cd --- /dev/null +++ b/packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts @@ -0,0 +1,16 @@ +import { env, runInDurableObject } from "cloudflare:test"; +import { expect, it } from "vitest"; +import type { StorageProbeObject } from "../../src/cloudflare/probe-object.ts"; + +it("reports what real Durable Object SQLite storage accepts", async () => { + const id = env.STORAGE_PROBE.idFromName("capabilities"); + const stub = env.STORAGE_PROBE.get(id); + const capabilities = await runInDurableObject(stub, (instance: StorageProbeObject) => + instance.capabilities(), + ); + // Printed rather than asserted: this test exists to establish the contract + // the owner is written against, and a hard assertion here would encode a + // guess about an answer nobody has yet. + console.log(JSON.stringify(capabilities, null, 2)); + expect(capabilities).toBeTruthy(); +}); diff --git a/packages/workflow/tests/cloudflare/worker.ts b/packages/workflow/tests/cloudflare/worker.ts new file mode 100644 index 000000000..7cd4831a3 --- /dev/null +++ b/packages/workflow/tests/cloudflare/worker.ts @@ -0,0 +1,15 @@ +/** + * The Worker the workerd suite runs against. + * + * It exists to publish the Durable Object classes under test and nothing else: + * the tests reach those objects through `runInDurableObject()` and their own + * stubs, so this handler answers no request a test depends on. + */ + +export { StorageProbeObject } from "../../src/cloudflare/probe-object.ts"; + +export default { + fetch(): Response { + return new Response("workflow owner test worker", { status: 200 }); + }, +}; diff --git a/packages/workflow/tests/cloudflare/wrangler.jsonc b/packages/workflow/tests/cloudflare/wrangler.jsonc new file mode 100644 index 000000000..df1a8d9a4 --- /dev/null +++ b/packages/workflow/tests/cloudflare/wrangler.jsonc @@ -0,0 +1,10 @@ +{ + "name": "workflow-owner-tests", + "main": "worker.ts", + "compatibility_date": "2026-08-01", + "compatibility_flags": ["nodejs_compat"], + "durable_objects": { + "bindings": [{ "name": "STORAGE_PROBE", "class_name": "StorageProbeObject" }], + }, + "migrations": [{ "tag": "v1", "new_sqlite_classes": ["StorageProbeObject"] }], +} diff --git a/packages/workflow/tests/host-neutrality.test.ts b/packages/workflow/tests/host-neutrality.test.ts index 94074295a..986e2bac5 100644 --- a/packages/workflow/tests/host-neutrality.test.ts +++ b/packages/workflow/tests/host-neutrality.test.ts @@ -32,7 +32,15 @@ const PACKAGE = fileURLToPath(new URL("..", import.meta.url)); * to these rules like any shared module. `vendor` is pinned upstream source * whose drift verifier owns its bytes. */ -const RUNTIME_OWNED = ["deno.ts", "cloudflare.ts", "src/deno", "src/cloudflare", "vendor"]; +const RUNTIME_OWNED = [ + "deno.ts", + "cloudflare.ts", + "src/deno", + "src/cloudflare", + "tests/cloudflare", + "vitest.config.ts", + "vendor", +]; /** Specifiers only a host adapter may import. */ const HOST_SPECIFIERS = [ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 487618c30..22b13ec61 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -120,6 +120,12 @@ importers: '@types/node': specifier: ^22.0.0 version: 22.19.15 + '@vitest/runner': + specifier: 4.1.11 + version: 4.1.11 + '@vitest/snapshot': + specifier: 4.1.11 + version: 4.1.11 expect: specifier: ^30.0.0 version: 30.3.0 @@ -5771,7 +5777,7 @@ snapshots: magic-string: 0.30.21 obug: 2.1.4 pathe: 2.0.3 - picomatch: 4.0.3 + picomatch: 4.0.7 std-env: 4.2.0 tinybench: 2.9.0 tinyexec: 1.3.0 diff --git a/vitest.config.ts b/vitest.config.ts new file mode 100644 index 000000000..6517e2270 --- /dev/null +++ b/vitest.config.ts @@ -0,0 +1,30 @@ +import { defineConfig } from "vitest/config"; +import { cloudflareTest } from "@cloudflare/vitest-plugin"; + +/** + * The workerd suite. + * + * These tests run against a real Durable Object namespace, real SQLite storage + * and a real WebSocket, because acquisition lifetime, owner eviction and + * transaction atomicity are properties of that runtime rather than of any model + * of it. Nothing here is discoverable by `deno task test`: the corpus walks + * `*.test.ts`, and these are `*.vitest.ts`, so the Deno, Node and Bun shards + * never see a file importing `cloudflare:test`. + */ +export default defineConfig({ + test: { + projects: [ + { + test: { + name: "cloudflare", + include: ["packages/workflow/tests/cloudflare/**/*.vitest.ts"], + }, + plugins: [ + cloudflareTest({ + wrangler: { configPath: "./packages/workflow/tests/cloudflare/wrangler.jsonc" }, + }), + ], + }, + ], + }, +}); From 670ad0caf3b90722f0aca781258fecf4c0d86b77 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 18:57:08 -0400 Subject: [PATCH 12/81] =?UTF-8?q?=F0=9F=93=8C=20Collapse=20the=20duplicate?= =?UTF-8?q?=20vitest=20peer=20context=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `@cloudflare/vitest-plugin` binds to `vitest` through peer dependencies, and the pool only takes over when the plugin and the CLI hold the same instance. pnpm was producing two: `acpx` depends on `tsx@4.23.1` exactly, the root range `^4.19.0` resolved to `4.21.0`, and `vite` peers on `tsx`, so `vitest@4.1.11` existed twice under different peer contexts. The plugin bound to one and the root link to the other, which is why the pool reported itself configured while every test failed with "Vitest failed to find the current suite". One `pnpm.overrides` entry pinning `tsx` collapses that. Both the root link and the plugin now resolve the same `vitest`, and `pnpm test:cloudflare` passes. Ordering note for anyone reproducing this: `deno install` prunes the links pnpm placed — `scripts/deps.ts` says so in its own header — so `deno task setup` leaves the pnpm-owned test tooling unlinked and `pnpm install` afterwards restores it. Setup itself completes: exit 0, "ready", browser bundle generated, lockfile still v9. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- package.json | 7 +- pnpm-lock.yaml | 323 +++---------------------------------------------- 2 files changed, 23 insertions(+), 307 deletions(-) diff --git a/package.json b/package.json index a4f74091e..ce9f34268 100644 --- a/package.json +++ b/package.json @@ -81,5 +81,10 @@ }, "workspaces": [ "packages/*" - ] + ], + "pnpm": { + "overrides": { + "tsx": "4.23.1" + } + } } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 22b13ec61..388492231 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -4,6 +4,9 @@ settings: autoInstallPeers: true excludeLinksFromLockfile: false +overrides: + tsx: 4.23.1 + importers: .: @@ -83,7 +86,7 @@ importers: devDependencies: '@cloudflare/vitest-plugin': specifier: 1.1.3 - version: 1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0))) + version: 1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1))) '@durable-streams/server': specifier: ^0.3.8 version: 0.3.8 @@ -136,14 +139,14 @@ importers: specifier: 1.74.0 version: 1.74.0 tsx: - specifier: ^4.19.0 - version: 4.21.0 + specifier: 4.23.1 + version: 4.23.1 typescript: specifier: ^5.0.0 version: 5.9.3 vitest: specifier: 4.1.11 - version: 4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)) + version: 4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)) packages/acp: dependencies: @@ -653,312 +656,156 @@ packages: '@emnapi/runtime@1.11.3': resolution: {integrity: sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==} - '@esbuild/aix-ppc64@0.27.4': - resolution: {integrity: sha512-cQPwL2mp2nSmHHJlCyoXgHGhbEPMrEEU5xhkcy3Hs/O7nGZqEpZ2sUtLaL9MORLtDfRvVl2/3PAuEkYZH0Ty8Q==} - engines: {node: '>=18'} - cpu: [ppc64] - os: [aix] - '@esbuild/aix-ppc64@0.28.1': resolution: {integrity: sha512-Svl7tq8k/08+p6CXPpRjQ1fKX+1odH/BQbb48fV6fj3CWHhsoIOoY87w1oHXm0qEpkIK3ZfVgp0hed3XBXzXMQ==} engines: {node: '>=18'} cpu: [ppc64] os: [aix] - '@esbuild/android-arm64@0.27.4': - resolution: {integrity: sha512-gdLscB7v75wRfu7QSm/zg6Rx29VLdy9eTr2t44sfTW7CxwAtQghZ4ZnqHk3/ogz7xao0QAgrkradbBzcqFPasw==} - engines: {node: '>=18'} - cpu: [arm64] - os: [android] - '@esbuild/android-arm64@0.28.1': resolution: {integrity: sha512-34EGEbCIAgosYz6goLcopX6Mo7NyGv9tfwEM2/7Ce2VcVRk568iSvniGWcUXIy7wEDR1wzolcxcriFVrWYcwBg==} engines: {node: '>=18'} cpu: [arm64] os: [android] - '@esbuild/android-arm@0.27.4': - resolution: {integrity: sha512-X9bUgvxiC8CHAGKYufLIHGXPJWnr0OCdR0anD2e21vdvgCI8lIfqFbnoeOz7lBjdrAGUhqLZLcQo6MLhTO2DKQ==} - engines: {node: '>=18'} - cpu: [arm] - os: [android] - '@esbuild/android-arm@0.28.1': resolution: {integrity: sha512-0k2F129Xdio1TdJfzJ8sy1Q47vUD2NnwdhiAf7drUN1EBTfPf4hsFCtmMgu/6m8JSzsBrlmVjudMBQqOfG8usQ==} engines: {node: '>=18'} cpu: [arm] os: [android] - '@esbuild/android-x64@0.27.4': - resolution: {integrity: sha512-PzPFnBNVF292sfpfhiyiXCGSn9HZg5BcAz+ivBuSsl6Rk4ga1oEXAamhOXRFyMcjwr2DVtm40G65N3GLeH1Lvw==} - engines: {node: '>=18'} - cpu: [x64] - os: [android] - '@esbuild/android-x64@0.28.1': resolution: {integrity: sha512-dbwY7ltSMDWsRatcRpCnES4F+im88OCUgGZjy52shC7GqHRE/cYlxNbB4Z4UpJswpcc4Qxd2oE/ufM0p61IKng==} engines: {node: '>=18'} cpu: [x64] os: [android] - '@esbuild/darwin-arm64@0.27.4': - resolution: {integrity: sha512-b7xaGIwdJlht8ZFCvMkpDN6uiSmnxxK56N2GDTMYPr2/gzvfdQN8rTfBsvVKmIVY/X7EM+/hJKEIbbHs9oA4tQ==} - engines: {node: '>=18'} - cpu: [arm64] - os: [darwin] - '@esbuild/darwin-arm64@0.28.1': resolution: {integrity: sha512-TZbWkQY7kvTAXbXUT7uVACR5cMHsDiSz9z7ZKAX/RTq/WJEk3QyRr0wZpNhBDX+/0CtdqUIJlOiodQcta6tY3Q==} engines: {node: '>=18'} cpu: [arm64] os: [darwin] - '@esbuild/darwin-x64@0.27.4': - resolution: {integrity: sha512-sR+OiKLwd15nmCdqpXMnuJ9W2kpy0KigzqScqHI3Hqwr7IXxBp3Yva+yJwoqh7rE8V77tdoheRYataNKL4QrPw==} - engines: {node: '>=18'} - cpu: [x64] - os: [darwin] - '@esbuild/darwin-x64@0.28.1': resolution: {integrity: sha512-zfdzgK9ACBNZLI/CyHTOx81SyNbM6YXn7rxSgX97VjyiPl9W1i4Ka4fgKECEoFCKGpvBj5qArWIGgQjOwkgskQ==} engines: {node: '>=18'} cpu: [x64] os: [darwin] - '@esbuild/freebsd-arm64@0.27.4': - resolution: {integrity: sha512-jnfpKe+p79tCnm4GVav68A7tUFeKQwQyLgESwEAUzyxk/TJr4QdGog9sqWNcUbr/bZt/O/HXouspuQDd9JxFSw==} - engines: {node: '>=18'} - cpu: [arm64] - os: [freebsd] - '@esbuild/freebsd-arm64@0.28.1': resolution: {integrity: sha512-wG2EA8ENdEI0qhkSZMjfqrdY+ziCYCPMmtZjjIwOmXFjmyzEHn+UUxk5of+SYsjtfs3VpnlC7QLzSI5hY/rOAw==} engines: {node: '>=18'} cpu: [arm64] os: [freebsd] - '@esbuild/freebsd-x64@0.27.4': - resolution: {integrity: sha512-2kb4ceA/CpfUrIcTUl1wrP/9ad9Atrp5J94Lq69w7UwOMolPIGrfLSvAKJp0RTvkPPyn6CIWrNy13kyLikZRZQ==} - engines: {node: '>=18'} - cpu: [x64] - os: [freebsd] - '@esbuild/freebsd-x64@0.28.1': resolution: {integrity: sha512-i7dZ9vQgnvSCzi/rYCXNgtF/U+eKZNJBzu3eTQbRgHnM7tNSizLOkRFAl3qzVc/Op/u5YkHHa4pf/3DOYHthLQ==} engines: {node: '>=18'} cpu: [x64] os: [freebsd] - '@esbuild/linux-arm64@0.27.4': - resolution: {integrity: sha512-7nQOttdzVGth1iz57kxg9uCz57dxQLHWxopL6mYuYthohPKEK0vU0C3O21CcBK6KDlkYVcnDXY099HcCDXd9dA==} - engines: {node: '>=18'} - cpu: [arm64] - os: [linux] - '@esbuild/linux-arm64@0.28.1': resolution: {integrity: sha512-yHs+0uc8+nvEAfAfxrWQKK5peSNzBc4PegcMO0EJ2hT71uA7vB8Ihg2e77R2P7SG5uYjPbHlLLmve4LLLRCf0g==} engines: {node: '>=18'} cpu: [arm64] os: [linux] - '@esbuild/linux-arm@0.27.4': - resolution: {integrity: sha512-aBYgcIxX/wd5n2ys0yESGeYMGF+pv6g0DhZr3G1ZG4jMfruU9Tl1i2Z+Wnj9/KjGz1lTLCcorqE2viePZqj4Eg==} - engines: {node: '>=18'} - cpu: [arm] - os: [linux] - '@esbuild/linux-arm@0.28.1': resolution: {integrity: sha512-qVXBOHQS+d5Y722GwJzJUtOLlX7km3CraOaGormF1pDtPd2C/l1SHRPgjLunLGe51Sh5YYWKMFDyV4SxgMQYTQ==} engines: {node: '>=18'} cpu: [arm] os: [linux] - '@esbuild/linux-ia32@0.27.4': - resolution: {integrity: sha512-oPtixtAIzgvzYcKBQM/qZ3R+9TEUd1aNJQu0HhGyqtx6oS7qTpvjheIWBbes4+qu1bNlo2V4cbkISr8q6gRBFA==} - engines: {node: '>=18'} - cpu: [ia32] - os: [linux] - '@esbuild/linux-ia32@0.28.1': resolution: {integrity: sha512-d1z4ZuP0ajrfz/FhGT4vv278rX8KnPPJx8i5+AtK7TYbx9Le9F1hyzurZpkEyjkGa9dUGhQow4C1NmeGvqxN2w==} engines: {node: '>=18'} cpu: [ia32] os: [linux] - '@esbuild/linux-loong64@0.27.4': - resolution: {integrity: sha512-8mL/vh8qeCoRcFH2nM8wm5uJP+ZcVYGGayMavi8GmRJjuI3g1v6Z7Ni0JJKAJW+m0EtUuARb6Lmp4hMjzCBWzA==} - engines: {node: '>=18'} - cpu: [loong64] - os: [linux] - '@esbuild/linux-loong64@0.28.1': resolution: {integrity: sha512-M5sRjUVZrkm1OAPR3dlOYzNmN+loZKGVi1VUQGrwuqLcbR6qeAz+famMhjASeH3YVKvZz+zT1jlh/keC3Rj/lg==} engines: {node: '>=18'} cpu: [loong64] os: [linux] - '@esbuild/linux-mips64el@0.27.4': - resolution: {integrity: sha512-1RdrWFFiiLIW7LQq9Q2NES+HiD4NyT8Itj9AUeCl0IVCA459WnPhREKgwrpaIfTOe+/2rdntisegiPWn/r/aAw==} - engines: {node: '>=18'} - cpu: [mips64el] - os: [linux] - '@esbuild/linux-mips64el@0.28.1': resolution: {integrity: sha512-mRObBZeHh2OxcBFPWE/FjylkRgZdYuiTR3vaTozquCGOH14iP9oN4x4Ge81CoIDYQrXmIxpFumJBu5MtZpnQJQ==} engines: {node: '>=18'} cpu: [mips64el] os: [linux] - '@esbuild/linux-ppc64@0.27.4': - resolution: {integrity: sha512-tLCwNG47l3sd9lpfyx9LAGEGItCUeRCWeAx6x2Jmbav65nAwoPXfewtAdtbtit/pJFLUWOhpv0FpS6GQAmPrHA==} - engines: {node: '>=18'} - cpu: [ppc64] - os: [linux] - '@esbuild/linux-ppc64@0.28.1': resolution: {integrity: sha512-slScBsMAb3GFDcdrCgLwZtPYRoH2H/youv10QiZyRjmsP48fznoveWytSgCI/R0ZcUgpc0ZhIUEx6LHts8yrfQ==} engines: {node: '>=18'} cpu: [ppc64] os: [linux] - '@esbuild/linux-riscv64@0.27.4': - resolution: {integrity: sha512-BnASypppbUWyqjd1KIpU4AUBiIhVr6YlHx/cnPgqEkNoVOhHg+YiSVxM1RLfiy4t9cAulbRGTNCKOcqHrEQLIw==} - engines: {node: '>=18'} - cpu: [riscv64] - os: [linux] - '@esbuild/linux-riscv64@0.28.1': resolution: {integrity: sha512-kw0owk1o0GFETUJyW0jc0G4Yzs0BHZn0JDZ8JRT088vjJYX777BAs1fDGxAC+q831qOs2DTC96mNsG2opdfyyQ==} engines: {node: '>=18'} cpu: [riscv64] os: [linux] - '@esbuild/linux-s390x@0.27.4': - resolution: {integrity: sha512-+eUqgb/Z7vxVLezG8bVB9SfBie89gMueS+I0xYh2tJdw3vqA/0ImZJ2ROeWwVJN59ihBeZ7Tu92dF/5dy5FttA==} - engines: {node: '>=18'} - cpu: [s390x] - os: [linux] - '@esbuild/linux-s390x@0.28.1': resolution: {integrity: sha512-/lAIjX8aYFRByhh6L5rYtPEDRqa9de/4V/juOXcta5frjvzXO4/sqEtyytse0g3zZFuWu5cDN0MkLz2qRDD2Ag==} engines: {node: '>=18'} cpu: [s390x] os: [linux] - '@esbuild/linux-x64@0.27.4': - resolution: {integrity: sha512-S5qOXrKV8BQEzJPVxAwnryi2+Iq5pB40gTEIT69BQONqR7JH1EPIcQ/Uiv9mCnn05jff9umq/5nqzxlqTOg9NA==} - engines: {node: '>=18'} - cpu: [x64] - os: [linux] - '@esbuild/linux-x64@0.28.1': resolution: {integrity: sha512-u/anNYF2mmVOEDwLtnQ1wOr3EZ9sTNGLWrsYGYwHWzGA3Si84IOkHXlbWTD1NB+9/1lcnweYKO54uhxZydNzfA==} engines: {node: '>=18'} cpu: [x64] os: [linux] - '@esbuild/netbsd-arm64@0.27.4': - resolution: {integrity: sha512-xHT8X4sb0GS8qTqiwzHqpY00C95DPAq7nAwX35Ie/s+LO9830hrMd3oX0ZMKLvy7vsonee73x0lmcdOVXFzd6Q==} - engines: {node: '>=18'} - cpu: [arm64] - os: [netbsd] - '@esbuild/netbsd-arm64@0.28.1': resolution: {integrity: sha512-oks0DYbLwWMmaakTsCb+zL4E+aHRVLom9IJZOAthMQEPiQmydXHkziYEsGYRx0uNV/IjEKGAV941JzH02pflqw==} engines: {node: '>=18'} cpu: [arm64] os: [netbsd] - '@esbuild/netbsd-x64@0.27.4': - resolution: {integrity: sha512-RugOvOdXfdyi5Tyv40kgQnI0byv66BFgAqjdgtAKqHoZTbTF2QqfQrFwa7cHEORJf6X2ht+l9ABLMP0dnKYsgg==} - engines: {node: '>=18'} - cpu: [x64] - os: [netbsd] - '@esbuild/netbsd-x64@0.28.1': resolution: {integrity: sha512-aeL6lAnN89Hz43Mlh1G8ARasbuoYvSITDEx0tHh5b7jJnHcssqgjy9Yx430GDpmCa6OyrKoS0aNRjKundRizGg==} engines: {node: '>=18'} cpu: [x64] os: [netbsd] - '@esbuild/openbsd-arm64@0.27.4': - resolution: {integrity: sha512-2MyL3IAaTX+1/qP0O1SwskwcwCoOI4kV2IBX1xYnDDqthmq5ArrW94qSIKCAuRraMgPOmG0RDTA74mzYNQA9ow==} - engines: {node: '>=18'} - cpu: [arm64] - os: [openbsd] - '@esbuild/openbsd-arm64@0.28.1': resolution: {integrity: sha512-MEFJe5C3R8pwXdZ5Y21oo6m7ePiS0d9pWucn99O/wvyJZChoIQKrQDxKrGeW8F5+T0okTHesAmDeiHDTIq0V/Q==} engines: {node: '>=18'} cpu: [arm64] os: [openbsd] - '@esbuild/openbsd-x64@0.27.4': - resolution: {integrity: sha512-u8fg/jQ5aQDfsnIV6+KwLOf1CmJnfu1ShpwqdwC0uA7ZPwFws55Ngc12vBdeUdnuWoQYx/SOQLGDcdlfXhYmXQ==} - engines: {node: '>=18'} - cpu: [x64] - os: [openbsd] - '@esbuild/openbsd-x64@0.28.1': resolution: {integrity: sha512-i/ZLIOafE0Z8cI/XANJAixoJL/uRAoS2xOA3rb0xN+KK0K177cMAsQYkzHtBrtMXAKuAc7HGgcWiZ/sRC1Nxgw==} engines: {node: '>=18'} cpu: [x64] os: [openbsd] - '@esbuild/openharmony-arm64@0.27.4': - resolution: {integrity: sha512-JkTZrl6VbyO8lDQO3yv26nNr2RM2yZzNrNHEsj9bm6dOwwu9OYN28CjzZkH57bh4w0I2F7IodpQvUAEd1mbWXg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [openharmony] - '@esbuild/openharmony-arm64@0.28.1': resolution: {integrity: sha512-ge+Z7EXFNt2BO1oAMsVpiQ8EwndV9i1xXerAeTIK7AtPs3bKFXQM7nlRxDSIUIMeueR1CNXxqztLzdNeReKBJg==} engines: {node: '>=18'} cpu: [arm64] os: [openharmony] - '@esbuild/sunos-x64@0.27.4': - resolution: {integrity: sha512-/gOzgaewZJfeJTlsWhvUEmUG4tWEY2Spp5M20INYRg2ZKl9QPO3QEEgPeRtLjEWSW8FilRNacPOg8R1uaYkA6g==} - engines: {node: '>=18'} - cpu: [x64] - os: [sunos] - '@esbuild/sunos-x64@0.28.1': resolution: {integrity: sha512-BEjgtECkL3vY+SaSQ6nzVfiALUeFxpawyp8Jmf5PtYhf1Ug40N1h/hxlhts+f1FvSvarEigdxS3BlSMI2PJLcQ==} engines: {node: '>=18'} cpu: [x64] os: [sunos] - '@esbuild/win32-arm64@0.27.4': - resolution: {integrity: sha512-Z9SExBg2y32smoDQdf1HRwHRt6vAHLXcxD2uGgO/v2jK7Y718Ix4ndsbNMU/+1Qiem9OiOdaqitioZwxivhXYg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [win32] - '@esbuild/win32-arm64@0.28.1': resolution: {integrity: sha512-lCv9eK/H6ZJWbE7bh2nw54CZ9M2nupBxJcTsdk/QQnWkdSjKGuxmmH8/GWrlT1eMmZfn4dGcCjRte397WqfQXA==} engines: {node: '>=18'} cpu: [arm64] os: [win32] - '@esbuild/win32-ia32@0.27.4': - resolution: {integrity: sha512-DAyGLS0Jz5G5iixEbMHi5KdiApqHBWMGzTtMiJ72ZOLhbu/bzxgAe8Ue8CTS3n3HbIUHQz/L51yMdGMeoxXNJw==} - engines: {node: '>=18'} - cpu: [ia32] - os: [win32] - '@esbuild/win32-ia32@0.28.1': resolution: {integrity: sha512-zvb/mB2bSCoJOpoCBgYKKpX6YM6mJBlBUVUtVj41DlZJVEB6/0CKlRYxP5wWl1C1ILiCoAU5wZZ4q1P3qeS6Eg==} engines: {node: '>=18'} cpu: [ia32] os: [win32] - '@esbuild/win32-x64@0.27.4': - resolution: {integrity: sha512-+knoa0BDoeXgkNvvV1vvbZX4+hizelrkwmGJBdT17t8FNPwG2lKemmuMZlmaNQ3ws3DKKCxpb4zRZEIp3UxFCg==} - engines: {node: '>=18'} - cpu: [x64] - os: [win32] - '@esbuild/win32-x64@0.28.1': resolution: {integrity: sha512-bm4Mowrv+GXMlpWX++EcXw/iLyd1o3+bJkC2DkWXYVvgZCqD/bSj9ctZeAMC3cIxgjRVR2Dufaiu4YPxr5gW1A==} engines: {node: '>=18'} @@ -2348,11 +2195,6 @@ packages: es-module-lexer@2.3.2: resolution: {integrity: sha512-poHGpORABojJJucnV9KbOavETW8lBVnphkW77ER5/BQ5Fz7oXSoCNek7IH3vR5nRjdsEz926ibFYX8KtLQmdyw==} - esbuild@0.27.4: - resolution: {integrity: sha512-Rq4vbHnYkK5fws5NF7MYTU68FPRE1ajX7heQ/8QXXWqNgqqJ/GkmmyxIzUnf2Sr/bakf8l54716CcMGHYhMrrQ==} - engines: {node: '>=18'} - hasBin: true - esbuild@0.28.1: resolution: {integrity: sha512-HrJrvZv5ayxBzPfwphOoNzkzOIIlifzk0KJrGK2c8R4+LKpMtpYLQeUdjnwjWv/LZlkH2laZk+4w78pi99D4Vw==} engines: {node: '>=18'} @@ -2442,9 +2284,6 @@ packages: resolution: {integrity: sha512-FJhYRoDaiatfEkUK8HKlicmu/3SGFD51q3itKDGoSTysQJBnfOcxU5GxnhE1E6soB76MbT0MBtnKJuXyAx+96Q==} engines: {node: '>=6'} - get-tsconfig@4.13.6: - resolution: {integrity: sha512-shZT/QMiSHc/YBLxxOkMtgSid5HFoauqCE3/exfsEcwg1WkeqjG+V40yBbBrsD+jW2HDXcs28xOfcbm2jI8Ddw==} - graceful-fs@4.2.11: resolution: {integrity: sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==} @@ -2923,9 +2762,6 @@ packages: resolution: {integrity: sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==} engines: {node: '>=0.10.0'} - resolve-pkg-maps@1.0.0: - resolution: {integrity: sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==} - reusify@1.1.0: resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==} engines: {iojs: '>=1.0.0', node: '>=0.10.0'} @@ -3086,11 +2922,6 @@ packages: tslib@2.8.1: resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} - tsx@4.21.0: - resolution: {integrity: sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==} - engines: {node: '>=18.0.0'} - hasBin: true - tsx@4.23.1: resolution: {integrity: sha512-GQHnkIfxyx1wYCOS/wonik5MVRZU9hi1TEZmzGZSCJB1y9YgoZ8H6itNE/u4suE+yLmOzuE4E5S4TZ/ZX2wcWQ==} engines: {node: '>=18.0.0'} @@ -3181,7 +3012,7 @@ packages: stylus: '>=0.54.8' sugarss: ^5.0.0 terser: ^5.16.0 - tsx: ^4.8.1 + tsx: 4.23.1 yaml: ^2.4.2 peerDependenciesMeta: '@types/node': @@ -3359,14 +3190,14 @@ snapshots: optionalDependencies: workerd: 1.20260831.1 - '@cloudflare/vitest-plugin@1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)))': + '@cloudflare/vitest-plugin@1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)))': dependencies: '@vitest/runner': 4.1.11 '@vitest/snapshot': 4.1.11 cjs-module-lexer: 1.2.3 esbuild: 0.28.1 miniflare: 5.20260831.0-alpha - vitest: 4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)) + vitest: 4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)) wrangler: 4.128.0 zod: 4.4.3 transitivePeerDependencies: @@ -3486,159 +3317,81 @@ snapshots: tslib: 2.8.1 optional: true - '@esbuild/aix-ppc64@0.27.4': - optional: true - '@esbuild/aix-ppc64@0.28.1': optional: true - '@esbuild/android-arm64@0.27.4': - optional: true - '@esbuild/android-arm64@0.28.1': optional: true - '@esbuild/android-arm@0.27.4': - optional: true - '@esbuild/android-arm@0.28.1': optional: true - '@esbuild/android-x64@0.27.4': - optional: true - '@esbuild/android-x64@0.28.1': optional: true - '@esbuild/darwin-arm64@0.27.4': - optional: true - '@esbuild/darwin-arm64@0.28.1': optional: true - '@esbuild/darwin-x64@0.27.4': - optional: true - '@esbuild/darwin-x64@0.28.1': optional: true - '@esbuild/freebsd-arm64@0.27.4': - optional: true - '@esbuild/freebsd-arm64@0.28.1': optional: true - '@esbuild/freebsd-x64@0.27.4': - optional: true - '@esbuild/freebsd-x64@0.28.1': optional: true - '@esbuild/linux-arm64@0.27.4': - optional: true - '@esbuild/linux-arm64@0.28.1': optional: true - '@esbuild/linux-arm@0.27.4': - optional: true - '@esbuild/linux-arm@0.28.1': optional: true - '@esbuild/linux-ia32@0.27.4': - optional: true - '@esbuild/linux-ia32@0.28.1': optional: true - '@esbuild/linux-loong64@0.27.4': - optional: true - '@esbuild/linux-loong64@0.28.1': optional: true - '@esbuild/linux-mips64el@0.27.4': - optional: true - '@esbuild/linux-mips64el@0.28.1': optional: true - '@esbuild/linux-ppc64@0.27.4': - optional: true - '@esbuild/linux-ppc64@0.28.1': optional: true - '@esbuild/linux-riscv64@0.27.4': - optional: true - '@esbuild/linux-riscv64@0.28.1': optional: true - '@esbuild/linux-s390x@0.27.4': - optional: true - '@esbuild/linux-s390x@0.28.1': optional: true - '@esbuild/linux-x64@0.27.4': - optional: true - '@esbuild/linux-x64@0.28.1': optional: true - '@esbuild/netbsd-arm64@0.27.4': - optional: true - '@esbuild/netbsd-arm64@0.28.1': optional: true - '@esbuild/netbsd-x64@0.27.4': - optional: true - '@esbuild/netbsd-x64@0.28.1': optional: true - '@esbuild/openbsd-arm64@0.27.4': - optional: true - '@esbuild/openbsd-arm64@0.28.1': optional: true - '@esbuild/openbsd-x64@0.27.4': - optional: true - '@esbuild/openbsd-x64@0.28.1': optional: true - '@esbuild/openharmony-arm64@0.27.4': - optional: true - '@esbuild/openharmony-arm64@0.28.1': optional: true - '@esbuild/sunos-x64@0.27.4': - optional: true - '@esbuild/sunos-x64@0.28.1': optional: true - '@esbuild/win32-arm64@0.27.4': - optional: true - '@esbuild/win32-arm64@0.28.1': optional: true - '@esbuild/win32-ia32@0.27.4': - optional: true - '@esbuild/win32-ia32@0.28.1': optional: true - '@esbuild/win32-x64@0.27.4': - optional: true - '@esbuild/win32-x64@0.28.1': optional: true @@ -4461,13 +4214,13 @@ snapshots: chai: 6.2.2 tinyrainbow: 3.1.1 - '@vitest/mocker@4.1.11(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0))': + '@vitest/mocker@4.1.11(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1))': dependencies: '@vitest/spy': 4.1.11 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - vite: 8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0) + vite: 8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1) '@vitest/pretty-format@4.1.11': dependencies: @@ -4707,35 +4460,6 @@ snapshots: es-module-lexer@2.3.2: {} - esbuild@0.27.4: - optionalDependencies: - '@esbuild/aix-ppc64': 0.27.4 - '@esbuild/android-arm': 0.27.4 - '@esbuild/android-arm64': 0.27.4 - '@esbuild/android-x64': 0.27.4 - '@esbuild/darwin-arm64': 0.27.4 - '@esbuild/darwin-x64': 0.27.4 - '@esbuild/freebsd-arm64': 0.27.4 - '@esbuild/freebsd-x64': 0.27.4 - '@esbuild/linux-arm': 0.27.4 - '@esbuild/linux-arm64': 0.27.4 - '@esbuild/linux-ia32': 0.27.4 - '@esbuild/linux-loong64': 0.27.4 - '@esbuild/linux-mips64el': 0.27.4 - '@esbuild/linux-ppc64': 0.27.4 - '@esbuild/linux-riscv64': 0.27.4 - '@esbuild/linux-s390x': 0.27.4 - '@esbuild/linux-x64': 0.27.4 - '@esbuild/netbsd-arm64': 0.27.4 - '@esbuild/netbsd-x64': 0.27.4 - '@esbuild/openbsd-arm64': 0.27.4 - '@esbuild/openbsd-x64': 0.27.4 - '@esbuild/openharmony-arm64': 0.27.4 - '@esbuild/sunos-x64': 0.27.4 - '@esbuild/win32-arm64': 0.27.4 - '@esbuild/win32-ia32': 0.27.4 - '@esbuild/win32-x64': 0.27.4 - esbuild@0.28.1: optionalDependencies: '@esbuild/aix-ppc64': 0.28.1 @@ -4833,10 +4557,6 @@ snapshots: get-nonce@1.0.1: {} - get-tsconfig@4.13.6: - dependencies: - resolve-pkg-maps: 1.0.0 - graceful-fs@4.2.11: {} gray-matter@4.0.3: @@ -5458,8 +5178,6 @@ snapshots: require-from-string@2.0.2: {} - resolve-pkg-maps@1.0.0: {} - reusify@1.1.0: {} rolldown@1.2.7: @@ -5661,13 +5379,6 @@ snapshots: tslib@2.8.1: {} - tsx@4.21.0: - dependencies: - esbuild: 0.27.4 - get-tsconfig: 4.13.6 - optionalDependencies: - fsevents: 2.3.3 - tsx@4.23.1: dependencies: esbuild: 0.28.1 @@ -5750,7 +5461,7 @@ snapshots: '@types/unist': 3.0.3 vfile-message: 4.0.3 - vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0): + vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1): dependencies: lightningcss: 1.33.0 picomatch: 4.0.7 @@ -5761,12 +5472,12 @@ snapshots: '@types/node': 22.19.15 esbuild: 0.28.1 fsevents: 2.3.3 - tsx: 4.21.0 + tsx: 4.23.1 - vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)): + vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)): dependencies: '@vitest/expect': 4.1.11 - '@vitest/mocker': 4.1.11(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0)) + '@vitest/mocker': 4.1.11(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)) '@vitest/pretty-format': 4.1.11 '@vitest/runner': 4.1.11 '@vitest/snapshot': 4.1.11 @@ -5783,7 +5494,7 @@ snapshots: tinyexec: 1.3.0 tinyglobby: 0.2.17 tinyrainbow: 3.1.1 - vite: 8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.21.0) + vite: 8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1) why-is-node-running: 2.3.0 optionalDependencies: '@types/node': 22.19.15 From da758d91ce934239a94d44f255975c6d7e33ba92 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 19:15:51 -0400 Subject: [PATCH 13/81] =?UTF-8?q?=F0=9F=A7=B1=20Share=20the=20SQLite=20sch?= =?UTF-8?q?ema=20and=20give=20Cloudflare=20its=20own=20identity=20carrier?= =?UTF-8?q?=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The storage correction from `.vscode/698/698-c.md`. `packages/workflow/src/sqlite/workflow-schema.ts` now holds version 1 once: the application identity, the logical version, every declared table and index, the DDL, the expected inventory, the pre-release shapes, and `declaredStructureFailure()`, which compares what a database declares against what this build writes and answers with the disagreement instead of raising it. Two adapters keeping their own copy of that DDL would be two schemas that happen to look alike, and the first amendment either missed would be a run neither could recognize. The module names no runtime, owns no connection, path, transaction or lifecycle authority, and is published from no entrypoint. `src/deno/schema.ts` loses 505 lines and keeps what is genuinely its own: `DatabaseSync`, filesystem paths, SQLite error translation, and its existing `PRAGMA application_id`/`PRAGMA user_version` carrier. It reports the shared finding as its own failures, so the Deno host's behavior and its released recognition are unchanged. `src/cloudflare/marker.ts` is the other carrier. A Durable Object's SQLite refuses both pragmas — `not authorized: SQLITE_AUTH`, on read as well as write — so that adapter records the same two values in `_xmd_workflow_schema`, a singleton row fixed at `id = 1` by a CHECK and a primary key so a second identity row cannot exist. Same logical version, different physical carrier; adapter-private recognition metadata, not a WorkflowRun record, journal value, exported field or second schema. `src/cloudflare/storage.ts` reconciles the runtime's concrete `SqlStorageValue` rows with the vendor's structural row type in one place, and deliberately does not forward `transactionSync`. `DLC13` excludes `src/sqlite/**` with its own recorded reason and asserts the subtree is absent from what it scanned; `host-neutrality.test.ts` keeps scanning it and now proves it is among the modules checked. The exploratory probe moves out of production source into `tests/cloudflare/support/`, and its console dump becomes three asserted tests matching stable categories rather than platform wording: the pragmas are refused as unauthorized; direct savepoints, reentrant transactions and a DOFS filesystem write inside an owner transaction are all refused toward the storage transaction API; and ordinary DDL, `sqlite_schema`, an outer transaction, DOFS schema initialization and a strict metadata table are accepted. `pnpm check:cloudflare` type-checks the Cloudflare production source and its tests against `@cloudflare/workers-types`, which the root Deno check cannot do because `cloudflare:` modules do not resolve there. Two `pnpm.overrides` pin `tsx` and `@cloudflare/workers-types`: both are peers that were splintering `vitest` and the plugin into several instances, and a single copy of each is what makes the pool engage. `@cloudflare/workers-types` is pinned to `5.20260831.1` rather than the newest, which Deno's minimum-dependency-age policy blocks. Ordering, unchanged from `670ad0c` and now deterministic in both directions: `deno task setup` completes and leaves the workerd suite unable to run, and a `pnpm install` afterwards restores it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- deno.json | 6 +- deno.lock | 15 +- package.json | 7 +- packages/workflow/src/cloudflare/marker.ts | 88 +++ packages/workflow/src/cloudflare/storage.ts | 53 ++ packages/workflow/src/deno/schema.ts | 589 ++--------------- .../workflow/src/sqlite/workflow-schema.ts | 595 ++++++++++++++++++ packages/workflow/tests/cloudflare/env.d.ts | 10 +- .../cloudflare/storage-capabilities.vitest.ts | 82 ++- .../cloudflare/support}/probe-object.ts | 17 +- packages/workflow/tests/cloudflare/worker.ts | 2 +- .../workflow/tests/host-neutrality.test.ts | 1 + .../workflow/tests/workspace-effect.test.ts | 9 + packages/workflow/tsconfig.cloudflare.json | 19 + pnpm-lock.yaml | 20 +- 15 files changed, 927 insertions(+), 586 deletions(-) create mode 100644 packages/workflow/src/cloudflare/marker.ts create mode 100644 packages/workflow/src/cloudflare/storage.ts create mode 100644 packages/workflow/src/sqlite/workflow-schema.ts rename packages/workflow/{src/cloudflare => tests/cloudflare/support}/probe-object.ts (87%) create mode 100644 packages/workflow/tsconfig.cloudflare.json diff --git a/deno.json b/deno.json index c04d787ee..32374cba5 100644 --- a/deno.json +++ b/deno.json @@ -13,7 +13,8 @@ "packages/acp/vendor/acpx/generated/**/*.d.ts", "packages/workflow/src/cloudflare", "packages/workflow/tests/cloudflare", - "vitest.config.ts" + "vitest.config.ts", + "packages/workflow/tsconfig.cloudflare.json" ], "nodeModulesDir": "auto", "lock": { @@ -78,6 +79,7 @@ "analyze": "deno run --allow-all packages/cli/src/deno.ts run .reviews/AnalyzeRepo.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.analyze.jsonl", "analyze:ci": "deno run --allow-all packages/cli/src/deno.ts run .reviews/AnalyzeRepoCI.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.analyze.ci.jsonl", "analyze:dispatch": "deno run --allow-all packages/cli/src/deno.ts run .reviews/DispatchRepoAnalysis.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.dispatch.jsonl", - "test:cloudflare": "pnpm test:cloudflare" + "test:cloudflare": "pnpm test:cloudflare", + "check:cloudflare": "pnpm check:cloudflare" } } diff --git a/deno.lock b/deno.lock index 43038154b..c89620201 100644 --- a/deno.lock +++ b/deno.lock @@ -41,7 +41,8 @@ "npm:@agentclientprotocol/sdk@1.3.0": "1.3.0_zod@4.4.3", "npm:@babel/core@^7.28.0": "7.29.7", "npm:@babel/preset-react@^7.27.1": "7.29.7_@babel+core@7.29.7", - "npm:@cloudflare/vitest-plugin@1.1.3": "1.1.3_@vitest+runner@4.1.11_@vitest+snapshot@4.1.11_vitest@4.1.11__@opentelemetry+api@1.9.1__@types+node@24.13.3__vite@7.3.6___@types+node@24.13.3___tsx@4.23.1_@opentelemetry+api@1.9.1_@types+node@24.13.3_vite@7.3.6__@types+node@24.13.3__tsx@4.23.1", + "npm:@cloudflare/vitest-plugin@1.1.3": "1.1.3_@vitest+runner@4.1.11_@vitest+snapshot@4.1.11_vitest@4.1.11__@opentelemetry+api@1.9.1__@types+node@24.13.3__vite@7.3.6___@types+node@24.13.3___tsx@4.23.1_@cloudflare+workers-types@5.20260901.1", + "npm:@cloudflare/workers-types@^5.20260831.1": "5.20260901.1", "npm:@durable-streams/client@~0.2.2": "0.2.6", "npm:@durable-streams/server@~0.3.8": "0.3.8", "npm:@effectionx/context-api@0.6.0": "0.6.0_effection@4.1.0", @@ -498,7 +499,7 @@ "workerd" ] }, - "@cloudflare/vitest-plugin@1.1.3_@vitest+runner@4.1.11_@vitest+snapshot@4.1.11_vitest@4.1.11__@opentelemetry+api@1.9.1__@types+node@24.13.3__vite@7.3.6___@types+node@24.13.3___tsx@4.23.1_@opentelemetry+api@1.9.1_@types+node@24.13.3_vite@7.3.6__@types+node@24.13.3__tsx@4.23.1": { + "@cloudflare/vitest-plugin@1.1.3_@vitest+runner@4.1.11_@vitest+snapshot@4.1.11_vitest@4.1.11__@opentelemetry+api@1.9.1__@types+node@24.13.3__vite@7.3.6___@types+node@24.13.3___tsx@4.23.1_@cloudflare+workers-types@5.20260901.1": { "integrity": "sha512-ED1Rkaq5Wr5rCeHXpLoDyV4WGJzD0Ju0clM8jS7Hj+wjj/CwaMHeb8DXzUfUQPiHW9rTgwcttPPQnzau2kp6Jg==", "dependencies": [ "@vitest/runner", @@ -536,6 +537,9 @@ "os": ["win32"], "cpu": ["x64"] }, + "@cloudflare/workers-types@5.20260901.1": { + "integrity": "sha512-m1rNbR3UYC1pgaEyXkSlwLFLDB11QziYUlY0z/nqjwuYtg5dw9zBrgDudoSfYM6ebbPDKU81ogBQAzLp6h1y4w==" + }, "@colors/colors@1.5.0": { "integrity": "sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ==" }, @@ -4415,11 +4419,12 @@ "scripts": true, "bin": true }, - "wrangler@4.128.0": { + "wrangler@4.128.0_@cloudflare+workers-types@5.20260901.1": { "integrity": "sha512-jNXy9e8/pbx8iqTzXPiuflnitKJZoAfEUSUUDLW87bwyeMvJ7kb3yQMSbxEcfNdfHqJW38KRcKaLljOYV4N/4w==", "dependencies": [ "@cloudflare/kv-asset-handler", "@cloudflare/unenv-preset", + "@cloudflare/workers-types", "blake3-wasm", "esbuild@0.28.1", "miniflare", @@ -4430,6 +4435,9 @@ "optionalDependencies": [ "fsevents" ], + "optionalPeers": [ + "@cloudflare/workers-types" + ], "bin": true }, "wrap-ansi@7.0.0": { @@ -4523,6 +4531,7 @@ "packageJson": { "dependencies": [ "npm:@cloudflare/vitest-plugin@1.1.3", + "npm:@cloudflare/workers-types@^5.20260831.1", "npm:@durable-streams/client@~0.2.2", "npm:@durable-streams/server@~0.3.8", "npm:@effectionx/context-api@0.6.0", diff --git a/package.json b/package.json index ce9f34268..4a867b228 100644 --- a/package.json +++ b/package.json @@ -50,6 +50,7 @@ }, "devDependencies": { "@cloudflare/vitest-plugin": "1.1.3", + "@cloudflare/workers-types": "^5.20260831.1", "@durable-streams/server": "^0.3.8", "@executablemd/acp": "workspace:*", "@executablemd/cli": "workspace:*", @@ -77,14 +78,16 @@ "test:deno": "deno task test", "lint": "oxlint -c .oxlintrc.json --ignore-pattern 'scripts/tests/fixtures/**' --ignore-pattern '**/npm/**' --ignore-pattern 'packages/workflow/vendor/cloudflare-computer-dofs/**' --ignore-pattern 'packages/acp/vendor/acpx/**' packages scripts .reviews/components && oxfmt --check packages scripts .reviews/components/*.ts", "fmt": "oxfmt --write packages scripts .reviews/components/*.ts", - "test:cloudflare": "vitest run --config vitest.config.ts" + "test:cloudflare": "vitest run --config vitest.config.ts", + "check:cloudflare": "tsc -p packages/workflow/tsconfig.cloudflare.json" }, "workspaces": [ "packages/*" ], "pnpm": { "overrides": { - "tsx": "4.23.1" + "tsx": "4.23.1", + "@cloudflare/workers-types": "5.20260831.1" } } } diff --git a/packages/workflow/src/cloudflare/marker.ts b/packages/workflow/src/cloudflare/marker.ts new file mode 100644 index 000000000..e111577dc --- /dev/null +++ b/packages/workflow/src/cloudflare/marker.ts @@ -0,0 +1,88 @@ +/** + * How the Cloudflare owner says which schema its storage holds. + * + * The Deno host writes `PRAGMA application_id` and `PRAGMA user_version` into + * the SQLite header, and recognition reads them back to tell three conditions + * apart: a database belonging to something else, a version this build has not + * learned, and a database that claims version 1 and is not shaped like one. + * + * A Durable Object's SQLite refuses both pragmas — `not authorized: + * SQLITE_AUTH`, on read as well as write — so this adapter carries the same two + * values in a table of its own. The logical schema version is unchanged and + * shared with Deno; only the physical carrier differs, which is why this table + * is adapter-private recognition metadata rather than a WorkflowRun record. It + * is never a journal value, an exported field, an authored value, a public API, + * or a second schema. + * + * The constraints are what make the claim trustworthy. `id` is fixed at 1 by a + * CHECK and is the primary key, so a second identity row cannot exist; both + * values are non-null integers; and a row that disagrees with this build is + * refused rather than migrated. + */ + +import { APPLICATION_ID, SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; + +/** The adapter-private table carrying this database's identity. */ +export const MARKER_TABLE = "_xmd_workflow_schema"; + +export const MARKER_SQL = `CREATE TABLE ${MARKER_TABLE} ( + id INTEGER PRIMARY KEY NOT NULL CHECK (id = 1), + application_id INTEGER NOT NULL, + schema_version INTEGER NOT NULL +) STRICT, WITHOUT ROWID`; + +/** What one marker row says. */ +export interface SchemaMarker { + readonly applicationId: number; + readonly schemaVersion: number; +} + +/** Why a marker could not be accepted. */ +export type MarkerFailure = + | { readonly kind: "absent" } + | { readonly kind: "duplicated"; readonly rows: number } + | { readonly kind: "malformed" } + | { readonly kind: "foreign-application"; readonly applicationId: number } + | { readonly kind: "unknown-version"; readonly schemaVersion: number }; + +/** + * Read a marker out of rows the caller already selected. + * + * Takes rows rather than a connection so the comparison is the same whoever + * consumed the cursor — Cloudflare requires a cursor to be drained + * synchronously, and that is the caller's concern rather than this one's. + */ +export function readMarker(rows: readonly Record[]): SchemaMarker | MarkerFailure { + if (rows.length === 0) { + return { kind: "absent" }; + } + if (rows.length > 1) { + return { kind: "duplicated", rows: rows.length }; + } + const row = rows[0]; + if (row === undefined) { + return { kind: "absent" }; + } + const applicationId = row["application_id"]; + const schemaVersion = row["schema_version"]; + if ( + typeof applicationId !== "number" || + !Number.isInteger(applicationId) || + typeof schemaVersion !== "number" || + !Number.isInteger(schemaVersion) + ) { + return { kind: "malformed" }; + } + if (applicationId !== APPLICATION_ID) { + return { kind: "foreign-application", applicationId }; + } + if (schemaVersion !== SCHEMA_VERSION) { + return { kind: "unknown-version", schemaVersion }; + } + return { applicationId, schemaVersion }; +} + +/** Whether a read produced a marker rather than a reason it could not. */ +export function isSchemaMarker(value: SchemaMarker | MarkerFailure): value is SchemaMarker { + return "applicationId" in value && !("kind" in value); +} diff --git a/packages/workflow/src/cloudflare/storage.ts b/packages/workflow/src/cloudflare/storage.ts new file mode 100644 index 000000000..aa68916a9 --- /dev/null +++ b/packages/workflow/src/cloudflare/storage.ts @@ -0,0 +1,53 @@ +/** + * A Durable Object's storage, as the vendored DOFS layer expects to see it. + * + * The vendor describes storage structurally — `sql.exec()` answering a cursor + * whose rows are a caller-chosen `object` subtype — while the runtime types the + * same call concretely as `Record`. The two are + * compatible in fact and not in the type system, so this is the one place the + * shapes are reconciled, rather than every call site asserting it. + * + * Nothing is converted: the cursor is drained with `toArray()` exactly where + * the caller asks for it, because Cloudflare's SQL cursor does not survive an + * `await` and draining it late would read a different result than the query + * asked for. + */ + +import type { + DurableObjectStorageLike, + SQLCursorLike, + SQLStorageLike, +} from "../../vendor/cloudflare-computer-dofs/generated/types.d.ts"; + +/** The subset of the runtime's storage this adapter uses. */ +export interface OwnerStorage { + readonly sql: { + exec(query: string, ...bindings: unknown[]): { toArray(): Record[] }; + }; + transactionSync(closure: () => T): T; +} + +/** + * Present one Durable Object's storage as the vendored DOFS storage shape. + * + * `transactionSync` is deliberately *not* forwarded here. The owner opens + * exactly one real transaction of its own and enlists DOFS inside it; a wrapper + * that forwarded this method would let a nested call reach the runtime, which + * refuses transaction statements from `sql.exec()`. + */ +export function dofsStorage(storage: OwnerStorage): DurableObjectStorageLike { + const sql: SQLStorageLike = { + exec>( + query: string, + ...bindings: unknown[] + ): SQLCursorLike { + const rows = storage.sql.exec(query, ...bindings).toArray(); + return { + toArray(): Row[] { + return rows as Row[]; + }, + }; + }, + }; + return { sql }; +} diff --git a/packages/workflow/src/deno/schema.ts b/packages/workflow/src/deno/schema.ts index 3b3fdc4c2..7073c5e7e 100644 --- a/packages/workflow/src/deno/schema.ts +++ b/packages/workflow/src/deno/schema.ts @@ -32,462 +32,20 @@ import { WorkflowIncompleteVersionOneError, WorkflowSchemaVersionError, } from "../storage/errors.ts"; +import { + APPLICATION_ID, + declaredStructureFailure, + EXPECTED_SCHEMA, + hasAnyDeclaredObject, + REQUIRED_OBJECTS, + REQUIRED_TABLES, + SCHEMA_SQL, + SCHEMA_VERSION, + type SchemaObject, +} from "../sqlite/workflow-schema.ts"; import { reading } from "./reading.ts"; import { initializeEmptyWorkspace, verifyWorkspace } from "./workspace/root.ts"; -/** - * The bytes `XMD1` as a 32-bit integer, written into the SQLite header. - * - * A database carries what wrote it, so a file that is perfectly valid SQLite - * and belongs to something else is refused on sight rather than through the - * confusing shape of its missing tables. - */ -export const APPLICATION_ID = 0x584d4431; - -/** The only schema version this build reads or writes. */ -export const SCHEMA_VERSION = 1; - -const STATUSES = "'running', 'suspended', 'interrupted', 'completed', 'failed', 'cancelled'"; - -/** - * A stop reason is three columns wide and has three legal shapes. - * - * Spreading the variant across columns is what lets SQLite hold the invariant - * rather than the code that writes rows: a host reason with an event id, or a - * journal reason with a code, is refused by the database itself. - */ -function coherentStopReason(): string { - return `CHECK ( - (stop_reason_kind IS NULL AND stop_reason_code IS NULL AND stop_reason_event_id IS NULL) - OR (stop_reason_kind = 'host' AND stop_reason_code IS NOT NULL AND stop_reason_event_id IS NULL) - OR (stop_reason_kind = 'journal' AND stop_reason_code IS NULL AND stop_reason_event_id IS NOT NULL) - )`; -} - -/** - * Version 1, one table at a time. - * - * Kept as separate definitions so verification can compare what a file holds - * with what this build writes, rather than settling for the table's name. - * - * The complete version-1 shape includes the pinned DOFS objects, retained - * Workspace roots, journal and metadata. Dependency order is explicit: DOFS - * content precedes root references, and roots precede the journal rows that - * name them. - */ -interface DeclaredObject { - readonly type: "table" | "index"; - readonly sql: string; -} - -const OBJECTS: ReadonlyMap = new Map([ - [ - "vfs_meta", - { - type: "table", - sql: `CREATE TABLE vfs_meta ( - k TEXT PRIMARY KEY, - v INTEGER NOT NULL - )`, - }, - ], - [ - "vfs_nodes", - { - type: "table", - sql: `CREATE TABLE vfs_nodes ( - inode INTEGER PRIMARY KEY AUTOINCREMENT, - type TEXT NOT NULL CHECK(type IN ('file','dir','symlink')), - mode INTEGER NOT NULL DEFAULT 493, - mtime INTEGER NOT NULL, - rev INTEGER NOT NULL DEFAULT 0, - mount_root TEXT, - stub_size INTEGER, - manifest_hash BLOB, - link_target TEXT, - size INTEGER NOT NULL DEFAULT 0 - )`, - }, - ], - [ - "vfs_dirents", - { - type: "table", - sql: `CREATE TABLE vfs_dirents ( - parent_inode INTEGER NOT NULL, - name TEXT NOT NULL, - child_inode INTEGER NOT NULL, - PRIMARY KEY (parent_inode, name) - ) WITHOUT ROWID`, - }, - ], - [ - "vfs_dirents_by_child", - { - type: "index", - sql: "CREATE INDEX vfs_dirents_by_child ON vfs_dirents(child_inode)", - }, - ], - [ - "vfs_nodes_by_rev", - { - type: "index", - sql: "CREATE INDEX vfs_nodes_by_rev ON vfs_nodes(rev)", - }, - ], - [ - "vfs_nodes_by_manifest_hash", - { - type: "index", - sql: `CREATE INDEX vfs_nodes_by_manifest_hash - ON vfs_nodes(manifest_hash) WHERE manifest_hash IS NOT NULL`, - }, - ], - [ - "vfs_blobs", - { - type: "table", - sql: `CREATE TABLE vfs_blobs ( - hash BLOB PRIMARY KEY, - size INTEGER NOT NULL, - last_seen INTEGER NOT NULL - )`, - }, - ], - [ - "vfs_blob_bytes", - { - type: "table", - sql: `CREATE TABLE vfs_blob_bytes ( - hash BLOB PRIMARY KEY REFERENCES vfs_blobs(hash) ON DELETE CASCADE, - bytes BLOB NOT NULL - )`, - }, - ], - [ - "vfs_chunks", - { - type: "table", - sql: `CREATE TABLE vfs_chunks ( - inode INTEGER NOT NULL, - idx INTEGER NOT NULL, - hash BLOB NOT NULL, - size INTEGER NOT NULL, - PRIMARY KEY (inode, idx) - ) WITHOUT ROWID`, - }, - ], - [ - "vfs_chunks_by_hash", - { - type: "index", - sql: "CREATE INDEX vfs_chunks_by_hash ON vfs_chunks(hash)", - }, - ], - [ - "vfs_manifests", - { - type: "table", - sql: `CREATE TABLE vfs_manifests ( - hash BLOB PRIMARY KEY, - size INTEGER NOT NULL, - encoded BLOB NOT NULL, - last_seen INTEGER NOT NULL DEFAULT 0 - )`, - }, - ], - [ - "vfs_changes", - { - type: "table", - sql: `CREATE TABLE vfs_changes ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - rev INTEGER NOT NULL, - path TEXT NOT NULL, - op TEXT NOT NULL CHECK(op IN ('delete')) - )`, - }, - ], - [ - "vfs_changes_by_rev", - { - type: "index", - sql: "CREATE INDEX vfs_changes_by_rev ON vfs_changes(rev)", - }, - ], - [ - "vfs_changes_by_path", - { - type: "index", - sql: "CREATE INDEX vfs_changes_by_path ON vfs_changes(path, id DESC)", - }, - ], - [ - "_vfs_watermark", - { - type: "table", - sql: `CREATE TABLE _vfs_watermark ( - k TEXT NOT NULL, - backend TEXT NOT NULL DEFAULT 'default', - v INTEGER NOT NULL, - PRIMARY KEY (k, backend) - )`, - }, - ], - [ - "_vfs_fetch_cursor", - { - type: "table", - sql: `CREATE TABLE _vfs_fetch_cursor ( - k TEXT NOT NULL CHECK(k = 'fetch'), - backend TEXT NOT NULL DEFAULT 'default', - path TEXT, - PRIMARY KEY (k, backend) - )`, - }, - ], - [ - "_vfs_mounts", - { - type: "table", - sql: `CREATE TABLE _vfs_mounts ( - root TEXT PRIMARY KEY, - kind TEXT NOT NULL, - indexed INTEGER NOT NULL DEFAULT 0, - mode TEXT NOT NULL DEFAULT 'read-only' - CHECK(mode IN ('read-only', 'read-write')) - )`, - }, - ], - [ - "workspace_roots", - { - type: "table", - sql: `CREATE TABLE workspace_roots ( - root_id TEXT PRIMARY KEY CHECK ( - length(root_id) = 64 AND root_id NOT GLOB '*[^0-9a-f]*' - ), - format_version INTEGER NOT NULL CHECK (format_version = 1), - manifest TEXT NOT NULL CHECK (json_valid(manifest)) -) STRICT`, - }, - ], - [ - "workspace_root_manifest_refs", - { - type: "table", - sql: `CREATE TABLE workspace_root_manifest_refs ( - root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE CASCADE, - manifest_hash BLOB NOT NULL REFERENCES vfs_manifests(hash) ON DELETE RESTRICT, - PRIMARY KEY (root_id, manifest_hash) -) STRICT, WITHOUT ROWID`, - }, - ], - [ - "workspace_root_blob_refs", - { - type: "table", - sql: `CREATE TABLE workspace_root_blob_refs ( - root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE CASCADE, - blob_hash BLOB NOT NULL, - PRIMARY KEY (root_id, blob_hash), - FOREIGN KEY (blob_hash) REFERENCES vfs_blobs(hash) ON DELETE RESTRICT, - FOREIGN KEY (blob_hash) REFERENCES vfs_blob_bytes(hash) ON DELETE RESTRICT -) STRICT, WITHOUT ROWID`, - }, - ], - [ - "agent_sessions", - { - type: "table", - sql: `CREATE TABLE agent_sessions ( - session_key TEXT PRIMARY KEY, - provider TEXT NOT NULL, - agent_command TEXT NOT NULL, - session_identity TEXT NOT NULL, - policy TEXT NOT NULL, - assertion_kind TEXT NOT NULL, - assertion_value TEXT NOT NULL, - created_at TEXT NOT NULL -) STRICT`, - }, - ], - [ - "workspace_state", - { - type: "table", - sql: `CREATE TABLE workspace_state ( - singleton_id INTEGER PRIMARY KEY CHECK (singleton_id = 1), - current_root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE RESTRICT -) STRICT`, - }, - ], - [ - "journal_events", - { - type: "table", - sql: `CREATE TABLE journal_events ( - sequence INTEGER PRIMARY KEY AUTOINCREMENT, - event_id TEXT NOT NULL UNIQUE, - record TEXT NOT NULL CHECK (json_valid(record)), - workspace_root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE RESTRICT -) STRICT`, - }, - ], - [ - "workflow_run", - { - type: "table", - sql: `CREATE TABLE workflow_run ( - id INTEGER PRIMARY KEY CHECK (id = 1), - run_id TEXT NOT NULL, - definition TEXT NOT NULL CHECK (json_valid(definition)), - base TEXT NOT NULL, - props TEXT NOT NULL CHECK (json_valid(props) AND json_type(props) = 'object'), - status TEXT NOT NULL CHECK (status IN (${STATUSES})), - stop_reason_kind TEXT CHECK (stop_reason_kind IS NULL OR stop_reason_kind IN ('host', 'journal')), - stop_reason_code TEXT, - stop_reason_event_id TEXT REFERENCES journal_events (event_id), - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL, - ${coherentStopReason()} -) STRICT`, - }, - ], - [ - "definition_retrieval", - { - type: "table", - sql: `CREATE TABLE definition_retrieval ( - id INTEGER PRIMARY KEY CHECK (id = 1), - metadata TEXT NOT NULL CHECK (json_valid(metadata)), - revision INTEGER NOT NULL CHECK (revision >= 1 AND revision <= 9007199254740991), - updated_at TEXT NOT NULL -) STRICT`, - }, - ], - [ - "document_executions", - { - type: "table", - sql: `CREATE TABLE document_executions ( - sequence INTEGER PRIMARY KEY AUTOINCREMENT, - execution_id TEXT NOT NULL UNIQUE, - started_at TEXT NOT NULL, - stopped_at TEXT, - stop_status TEXT CHECK (stop_status IS NULL OR stop_status IN (${STATUSES})), - stop_reason_kind TEXT CHECK (stop_reason_kind IS NULL OR stop_reason_kind IN ('host', 'journal')), - stop_reason_code TEXT, - stop_reason_event_id TEXT REFERENCES journal_events (event_id), - CHECK ((stopped_at IS NULL) = (stop_status IS NULL)), - CHECK (stop_status IS NOT NULL OR stop_reason_kind IS NULL), - ${coherentStopReason()} -) STRICT`, - }, - ], - [ - "workspace_repositories", - { - type: "table", - sql: `CREATE TABLE workspace_repositories ( - name TEXT PRIMARY KEY CHECK (length(name) > 0), - locator TEXT NOT NULL CHECK (length(locator) > 0), - locator_fingerprint TEXT NOT NULL CHECK ( - length(locator_fingerprint) = 64 AND locator_fingerprint NOT GLOB '*[^0-9a-f]*' - ), - requested_base TEXT CHECK (requested_base IS NULL OR length(requested_base) > 0), - creation_commit TEXT NOT NULL CHECK (length(creation_commit) > 0), - primary_branch TEXT NOT NULL CHECK (length(primary_branch) > 0), - object_format TEXT NOT NULL CHECK (object_format IN ('sha1', 'sha256')), - checkout_path TEXT NOT NULL UNIQUE CHECK ( - length(checkout_path) > 0 AND substr(checkout_path, 1, 1) = '/' - ) -) STRICT`, - }, - ], - [ - "workspace_worktrees", - { - type: "table", - sql: `CREATE TABLE workspace_worktrees ( - repository_name TEXT NOT NULL REFERENCES workspace_repositories(name) ON DELETE RESTRICT, - name TEXT NOT NULL CHECK (length(name) > 0), - requested_branch TEXT NOT NULL CHECK (length(requested_branch) > 0), - requested_base TEXT CHECK (requested_base IS NULL OR length(requested_base) > 0), - creation_commit TEXT NOT NULL CHECK (length(creation_commit) > 0), - checkout_path TEXT NOT NULL UNIQUE CHECK ( - length(checkout_path) > 0 AND substr(checkout_path, 1, 1) = '/' - ), - PRIMARY KEY (repository_name, name) -) STRICT, WITHOUT ROWID`, - }, - ], - [ - "workflow_suspension_answers", - { - type: "table", - sql: `CREATE TABLE workflow_suspension_answers ( - suspension_id TEXT PRIMARY KEY, - request_event_id TEXT NOT NULL REFERENCES journal_events(event_id) ON DELETE RESTRICT, - request_fingerprint TEXT NOT NULL CHECK ( - length(request_fingerprint) = 64 AND request_fingerprint NOT GLOB '*[^0-9a-f]*' - ), - answer TEXT NOT NULL CHECK (json_valid(answer)), - state TEXT NOT NULL CHECK (state IN ('pending', 'consumed')), - created_at TEXT NOT NULL, - consumed_at TEXT, - CHECK ((state = 'consumed') = (consumed_at IS NOT NULL)) -) STRICT`, - }, - ], - [ - "workflow_fork_lineage", - { - type: "table", - sql: `CREATE TABLE workflow_fork_lineage ( - id INTEGER PRIMARY KEY CHECK (id = 1), - source_run_id TEXT NOT NULL CHECK (length(source_run_id) > 0), - checkpoint_event_id TEXT NOT NULL CHECK (length(checkpoint_event_id) > 0), - checkpoint_workspace_root_id TEXT NOT NULL - REFERENCES workspace_roots(root_id) ON DELETE RESTRICT, - created_at TEXT NOT NULL -) STRICT`, - }, - ], - [ - "journal_event_provenance", - { - type: "table", - sql: `CREATE TABLE journal_event_provenance ( - event_id TEXT PRIMARY KEY REFERENCES journal_events(event_id) ON DELETE RESTRICT, - source_run_id TEXT NOT NULL CHECK (length(source_run_id) > 0), - source_event_id TEXT NOT NULL CHECK (length(source_event_id) > 0) -) STRICT, WITHOUT ROWID`, - }, - ], -]); - -export const EXPECTED_SCHEMA = Object.freeze( - [...OBJECTS.entries()].map(([name, object]) => - Object.freeze({ name, type: object.type, sql: normalize(object.sql) }), - ), -); - -/** Objects version 1 declares, including the pinned Cloudflare structure. */ -export const REQUIRED_OBJECTS: readonly string[] = Object.freeze([...OBJECTS.keys()]); - -/** Tables version 1 declares. */ -export const REQUIRED_TABLES: readonly string[] = Object.freeze( - [...OBJECTS.entries()].filter(([, object]) => object.type === "table").map(([name]) => name), -); - -/** Version 1 in full. */ -export const SCHEMA_SQL = [...OBJECTS.values()] - .filter((object) => object.type === "table" && !object.sql.startsWith("CREATE TABLE vfs_")) - .filter((object) => !object.sql.startsWith("CREATE TABLE _vfs_")) - .map((object) => `${object.sql};`) - .join("\n\n"); - /** * Write the version-1 schema into a database that holds nothing. * @@ -495,6 +53,15 @@ export const SCHEMA_SQL = [...OBJECTS.values()] * and the tables appear together or not at all — a half-initialized file would * be indistinguishable from one this build must refuse. */ +export { + APPLICATION_ID, + EXPECTED_SCHEMA, + REQUIRED_OBJECTS, + REQUIRED_TABLES, + SCHEMA_SQL, + SCHEMA_VERSION, +}; + export function initializeSchema( database: DatabaseSync, dofs: CloudflareDatabase, @@ -571,98 +138,33 @@ export function verifySchema(database: DatabaseSync, path: string, dofs: Cloudfl * has not learned yet. */ function verifyStructure(database: DatabaseSync, path: string): void { - const objects = schemaObjects(database, path); - if (isIncompletePreReleaseShape(objects)) { + const failure = declaredStructureFailure(schemaObjects(database, path)); + if (failure === undefined) { + return; + } + if (failure.kind === "incomplete-pre-release") { throw new WorkflowIncompleteVersionOneError(path); } - - for (const object of objects) { - const expected = OBJECTS.get(object.name); - if (expected === undefined) { - throw new WorkflowDatabaseCorruptError( - path, - `it declares an object that version ${SCHEMA_VERSION} does not`, - ); - } - if (object.type !== expected.type || normalize(object.sql) !== normalize(expected.sql)) { - throw new WorkflowDatabaseCorruptError( - path, - `its ${object.name} object is not shaped the way version ${SCHEMA_VERSION} declares it`, - ); - } + if (failure.kind === "undeclared-object") { + throw new WorkflowDatabaseCorruptError( + path, + `it declares an object that version ${SCHEMA_VERSION} does not`, + ); } - - const present = new Set(objects.map((object) => object.name)); - const missing = REQUIRED_OBJECTS.filter((name) => !present.has(name)); - if (missing.length > 0) { - throw new WorkflowDatabaseCorruptError(path, `it is missing the table ${missing.join(", ")}`); + if (failure.kind === "misshapen-object") { + throw new WorkflowDatabaseCorruptError( + path, + `its ${failure.name} object is not shaped the way version ${SCHEMA_VERSION} declares it`, + ); } + throw new WorkflowDatabaseCorruptError( + path, + `it is missing the table ${failure.names.join(", ")}`, + ); } function hasDeclaredVersionOneObjects(database: DatabaseSync, path: string): boolean { - return schemaObjects(database, path).some((object) => OBJECTS.has(object.name)); -} - -/** - * Every in-place amendment to version 1, newest first. - * - * Each entry names what that amendment added. Peeling them off in order is what - * reconstructs the shapes that once claimed to be a complete version 1, so a - * database an earlier build produced is refused as an incomplete pre-release - * rather than as arbitrary damage. - */ -const AMENDMENTS: readonly (readonly string[])[] = Object.freeze([ - Object.freeze(["workflow_fork_lineage", "journal_event_provenance"]), - Object.freeze(["workflow_suspension_answers"]), - Object.freeze(["workspace_repositories", "workspace_worktrees"]), -]); - -/** What the newest amendment added. Its presence marks a current-shape database. */ -const LATEST_AMENDMENT: readonly string[] = AMENDMENTS[0] ?? []; - -/** The very first pre-release shape, before Workspace root retention existed. */ -const EARLIEST_PRE_RELEASE_SHAPE: readonly string[] = [ - "definition_retrieval", - "document_executions", - "journal_events", - "workflow_run", -]; - -/** - * Every later shape that once claimed to be a complete version 1. - * - * Newest first: version 1 minus the newest amendment, then minus the one before - * it, and so on. - */ -const PRIOR_COMPLETE_SHAPES: readonly (readonly string[])[] = Object.freeze( - AMENDMENTS.map((_, index) => { - const removed = new Set(AMENDMENTS.slice(0, index + 1).flat()); - return Object.freeze(REQUIRED_OBJECTS.filter((name) => !removed.has(name))); - }), -); - -/** - * Whether these declarations describe an earlier shape that once claimed to be - * a complete version 1. - * - * The very first pre-release held only the run, journal and execution tables. - * Every shape after it is version 1 minus whichever amendments had not been - * made yet, and each is named here so the refusal reads as an incomplete - * pre-release rather than as corruption. - */ -function isIncompletePreReleaseShape(objects: readonly SchemaObject[]): boolean { - const present = new Set(objects.map((object) => object.name)); - if (LATEST_AMENDMENT.some((name) => present.has(name))) { - return false; - } - const earliest = new Set(EARLIEST_PRE_RELEASE_SHAPE); - if (present.size === earliest.size && [...present].every((name) => earliest.has(name))) { - return objects.every((object) => object.type === "table"); - } - return PRIOR_COMPLETE_SHAPES.some((shape) => { - const expected = new Set(shape); - return present.size === expected.size && [...present].every((name) => expected.has(name)); - }); + return hasAnyDeclaredObject(schemaObjects(database, path)); } /** @@ -693,12 +195,6 @@ function checkForeignKeys(database: DatabaseSync, path: string): void { } } -interface SchemaObject { - readonly type: string; - readonly name: string; - readonly sql: string; -} - /** * Everything somebody declared in this database. * @@ -725,11 +221,6 @@ function schemaObjects(database: DatabaseSync, path: string): SchemaObject[] { return objects; } -/** One statement's shape, independent of how it was laid out. */ -function normalize(sql: string): string { - return sql.replace(/\s+/g, " ").trim(); -} - function readPragmaNumber(database: DatabaseSync, pragma: string, path: string): number { const rows = query(database, `PRAGMA ${pragma}`, path); const value = rows[0]?.[pragma]; diff --git a/packages/workflow/src/sqlite/workflow-schema.ts b/packages/workflow/src/sqlite/workflow-schema.ts new file mode 100644 index 000000000..1f9c8d0a3 --- /dev/null +++ b/packages/workflow/src/sqlite/workflow-schema.ts @@ -0,0 +1,595 @@ +/** + * The version-1 WorkflowRun schema, as SQLite holds it. + * + * Two adapters keep a run in an embedded SQLite database — the Deno host in a + * file it opens with `node:sqlite`, the Cloudflare owner in the storage of one + * Durable Object — and they must agree about what version 1 *is*. A second copy + * of this DDL under a second adapter would be two schemas that happen to look + * alike, and the first amendment either of them missed would be a run neither + * could recognize. + * + * So the declaration lives here once, and each adapter keeps what is genuinely + * its own: how a connection is opened, how an error is translated, and how the + * identity of the schema is carried. That last one differs because it has to. + * Deno writes `PRAGMA application_id` and `PRAGMA user_version` into the SQLite + * header; Cloudflare's Durable Object storage refuses both pragmas outright, so + * that adapter carries the same two values in a table of its own. The logical + * version is one; only its physical carrier is per-adapter. + * + * Nothing here owns a connection, a path, a transaction or any lifecycle + * authority, and nothing here names a runtime. It is a description of a shape + * and the arithmetic for comparing a database against it. + */ + +/** + * The bytes `XMD1` as a 32-bit integer, written into the SQLite header. + * + * A database carries what wrote it, so a file that is perfectly valid SQLite + * and belongs to something else is refused on sight rather than through the + * confusing shape of its missing tables. + */ +export const APPLICATION_ID = 0x584d4431; + +/** The only schema version this build reads or writes. */ +export const SCHEMA_VERSION = 1; + +const STATUSES = "'running', 'suspended', 'interrupted', 'completed', 'failed', 'cancelled'"; + +/** + * A stop reason is three columns wide and has three legal shapes. + * + * Spreading the variant across columns is what lets SQLite hold the invariant + * rather than the code that writes rows: a host reason with an event id, or a + * journal reason with a code, is refused by the database itself. + */ +function coherentStopReason(): string { + return `CHECK ( + (stop_reason_kind IS NULL AND stop_reason_code IS NULL AND stop_reason_event_id IS NULL) + OR (stop_reason_kind = 'host' AND stop_reason_code IS NOT NULL AND stop_reason_event_id IS NULL) + OR (stop_reason_kind = 'journal' AND stop_reason_code IS NULL AND stop_reason_event_id IS NOT NULL) + )`; +} + +/** + * Version 1, one table at a time. + * + * Kept as separate definitions so verification can compare what a file holds + * with what this build writes, rather than settling for the table's name. + * + * The complete version-1 shape includes the pinned DOFS objects, retained + * Workspace roots, journal and metadata. Dependency order is explicit: DOFS + * content precedes root references, and roots precede the journal rows that + * name them. + */ +interface DeclaredObject { + readonly type: "table" | "index"; + readonly sql: string; +} + +export const OBJECTS: ReadonlyMap = new Map([ + [ + "vfs_meta", + { + type: "table", + sql: `CREATE TABLE vfs_meta ( + k TEXT PRIMARY KEY, + v INTEGER NOT NULL + )`, + }, + ], + [ + "vfs_nodes", + { + type: "table", + sql: `CREATE TABLE vfs_nodes ( + inode INTEGER PRIMARY KEY AUTOINCREMENT, + type TEXT NOT NULL CHECK(type IN ('file','dir','symlink')), + mode INTEGER NOT NULL DEFAULT 493, + mtime INTEGER NOT NULL, + rev INTEGER NOT NULL DEFAULT 0, + mount_root TEXT, + stub_size INTEGER, + manifest_hash BLOB, + link_target TEXT, + size INTEGER NOT NULL DEFAULT 0 + )`, + }, + ], + [ + "vfs_dirents", + { + type: "table", + sql: `CREATE TABLE vfs_dirents ( + parent_inode INTEGER NOT NULL, + name TEXT NOT NULL, + child_inode INTEGER NOT NULL, + PRIMARY KEY (parent_inode, name) + ) WITHOUT ROWID`, + }, + ], + [ + "vfs_dirents_by_child", + { + type: "index", + sql: "CREATE INDEX vfs_dirents_by_child ON vfs_dirents(child_inode)", + }, + ], + [ + "vfs_nodes_by_rev", + { + type: "index", + sql: "CREATE INDEX vfs_nodes_by_rev ON vfs_nodes(rev)", + }, + ], + [ + "vfs_nodes_by_manifest_hash", + { + type: "index", + sql: `CREATE INDEX vfs_nodes_by_manifest_hash + ON vfs_nodes(manifest_hash) WHERE manifest_hash IS NOT NULL`, + }, + ], + [ + "vfs_blobs", + { + type: "table", + sql: `CREATE TABLE vfs_blobs ( + hash BLOB PRIMARY KEY, + size INTEGER NOT NULL, + last_seen INTEGER NOT NULL + )`, + }, + ], + [ + "vfs_blob_bytes", + { + type: "table", + sql: `CREATE TABLE vfs_blob_bytes ( + hash BLOB PRIMARY KEY REFERENCES vfs_blobs(hash) ON DELETE CASCADE, + bytes BLOB NOT NULL + )`, + }, + ], + [ + "vfs_chunks", + { + type: "table", + sql: `CREATE TABLE vfs_chunks ( + inode INTEGER NOT NULL, + idx INTEGER NOT NULL, + hash BLOB NOT NULL, + size INTEGER NOT NULL, + PRIMARY KEY (inode, idx) + ) WITHOUT ROWID`, + }, + ], + [ + "vfs_chunks_by_hash", + { + type: "index", + sql: "CREATE INDEX vfs_chunks_by_hash ON vfs_chunks(hash)", + }, + ], + [ + "vfs_manifests", + { + type: "table", + sql: `CREATE TABLE vfs_manifests ( + hash BLOB PRIMARY KEY, + size INTEGER NOT NULL, + encoded BLOB NOT NULL, + last_seen INTEGER NOT NULL DEFAULT 0 + )`, + }, + ], + [ + "vfs_changes", + { + type: "table", + sql: `CREATE TABLE vfs_changes ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + rev INTEGER NOT NULL, + path TEXT NOT NULL, + op TEXT NOT NULL CHECK(op IN ('delete')) + )`, + }, + ], + [ + "vfs_changes_by_rev", + { + type: "index", + sql: "CREATE INDEX vfs_changes_by_rev ON vfs_changes(rev)", + }, + ], + [ + "vfs_changes_by_path", + { + type: "index", + sql: "CREATE INDEX vfs_changes_by_path ON vfs_changes(path, id DESC)", + }, + ], + [ + "_vfs_watermark", + { + type: "table", + sql: `CREATE TABLE _vfs_watermark ( + k TEXT NOT NULL, + backend TEXT NOT NULL DEFAULT 'default', + v INTEGER NOT NULL, + PRIMARY KEY (k, backend) + )`, + }, + ], + [ + "_vfs_fetch_cursor", + { + type: "table", + sql: `CREATE TABLE _vfs_fetch_cursor ( + k TEXT NOT NULL CHECK(k = 'fetch'), + backend TEXT NOT NULL DEFAULT 'default', + path TEXT, + PRIMARY KEY (k, backend) + )`, + }, + ], + [ + "_vfs_mounts", + { + type: "table", + sql: `CREATE TABLE _vfs_mounts ( + root TEXT PRIMARY KEY, + kind TEXT NOT NULL, + indexed INTEGER NOT NULL DEFAULT 0, + mode TEXT NOT NULL DEFAULT 'read-only' + CHECK(mode IN ('read-only', 'read-write')) + )`, + }, + ], + [ + "workspace_roots", + { + type: "table", + sql: `CREATE TABLE workspace_roots ( + root_id TEXT PRIMARY KEY CHECK ( + length(root_id) = 64 AND root_id NOT GLOB '*[^0-9a-f]*' + ), + format_version INTEGER NOT NULL CHECK (format_version = 1), + manifest TEXT NOT NULL CHECK (json_valid(manifest)) +) STRICT`, + }, + ], + [ + "workspace_root_manifest_refs", + { + type: "table", + sql: `CREATE TABLE workspace_root_manifest_refs ( + root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE CASCADE, + manifest_hash BLOB NOT NULL REFERENCES vfs_manifests(hash) ON DELETE RESTRICT, + PRIMARY KEY (root_id, manifest_hash) +) STRICT, WITHOUT ROWID`, + }, + ], + [ + "workspace_root_blob_refs", + { + type: "table", + sql: `CREATE TABLE workspace_root_blob_refs ( + root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE CASCADE, + blob_hash BLOB NOT NULL, + PRIMARY KEY (root_id, blob_hash), + FOREIGN KEY (blob_hash) REFERENCES vfs_blobs(hash) ON DELETE RESTRICT, + FOREIGN KEY (blob_hash) REFERENCES vfs_blob_bytes(hash) ON DELETE RESTRICT +) STRICT, WITHOUT ROWID`, + }, + ], + [ + "agent_sessions", + { + type: "table", + sql: `CREATE TABLE agent_sessions ( + session_key TEXT PRIMARY KEY, + provider TEXT NOT NULL, + agent_command TEXT NOT NULL, + session_identity TEXT NOT NULL, + policy TEXT NOT NULL, + assertion_kind TEXT NOT NULL, + assertion_value TEXT NOT NULL, + created_at TEXT NOT NULL +) STRICT`, + }, + ], + [ + "workspace_state", + { + type: "table", + sql: `CREATE TABLE workspace_state ( + singleton_id INTEGER PRIMARY KEY CHECK (singleton_id = 1), + current_root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE RESTRICT +) STRICT`, + }, + ], + [ + "journal_events", + { + type: "table", + sql: `CREATE TABLE journal_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + event_id TEXT NOT NULL UNIQUE, + record TEXT NOT NULL CHECK (json_valid(record)), + workspace_root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE RESTRICT +) STRICT`, + }, + ], + [ + "workflow_run", + { + type: "table", + sql: `CREATE TABLE workflow_run ( + id INTEGER PRIMARY KEY CHECK (id = 1), + run_id TEXT NOT NULL, + definition TEXT NOT NULL CHECK (json_valid(definition)), + base TEXT NOT NULL, + props TEXT NOT NULL CHECK (json_valid(props) AND json_type(props) = 'object'), + status TEXT NOT NULL CHECK (status IN (${STATUSES})), + stop_reason_kind TEXT CHECK (stop_reason_kind IS NULL OR stop_reason_kind IN ('host', 'journal')), + stop_reason_code TEXT, + stop_reason_event_id TEXT REFERENCES journal_events (event_id), + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + ${coherentStopReason()} +) STRICT`, + }, + ], + [ + "definition_retrieval", + { + type: "table", + sql: `CREATE TABLE definition_retrieval ( + id INTEGER PRIMARY KEY CHECK (id = 1), + metadata TEXT NOT NULL CHECK (json_valid(metadata)), + revision INTEGER NOT NULL CHECK (revision >= 1 AND revision <= 9007199254740991), + updated_at TEXT NOT NULL +) STRICT`, + }, + ], + [ + "document_executions", + { + type: "table", + sql: `CREATE TABLE document_executions ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + execution_id TEXT NOT NULL UNIQUE, + started_at TEXT NOT NULL, + stopped_at TEXT, + stop_status TEXT CHECK (stop_status IS NULL OR stop_status IN (${STATUSES})), + stop_reason_kind TEXT CHECK (stop_reason_kind IS NULL OR stop_reason_kind IN ('host', 'journal')), + stop_reason_code TEXT, + stop_reason_event_id TEXT REFERENCES journal_events (event_id), + CHECK ((stopped_at IS NULL) = (stop_status IS NULL)), + CHECK (stop_status IS NOT NULL OR stop_reason_kind IS NULL), + ${coherentStopReason()} +) STRICT`, + }, + ], + [ + "workspace_repositories", + { + type: "table", + sql: `CREATE TABLE workspace_repositories ( + name TEXT PRIMARY KEY CHECK (length(name) > 0), + locator TEXT NOT NULL CHECK (length(locator) > 0), + locator_fingerprint TEXT NOT NULL CHECK ( + length(locator_fingerprint) = 64 AND locator_fingerprint NOT GLOB '*[^0-9a-f]*' + ), + requested_base TEXT CHECK (requested_base IS NULL OR length(requested_base) > 0), + creation_commit TEXT NOT NULL CHECK (length(creation_commit) > 0), + primary_branch TEXT NOT NULL CHECK (length(primary_branch) > 0), + object_format TEXT NOT NULL CHECK (object_format IN ('sha1', 'sha256')), + checkout_path TEXT NOT NULL UNIQUE CHECK ( + length(checkout_path) > 0 AND substr(checkout_path, 1, 1) = '/' + ) +) STRICT`, + }, + ], + [ + "workspace_worktrees", + { + type: "table", + sql: `CREATE TABLE workspace_worktrees ( + repository_name TEXT NOT NULL REFERENCES workspace_repositories(name) ON DELETE RESTRICT, + name TEXT NOT NULL CHECK (length(name) > 0), + requested_branch TEXT NOT NULL CHECK (length(requested_branch) > 0), + requested_base TEXT CHECK (requested_base IS NULL OR length(requested_base) > 0), + creation_commit TEXT NOT NULL CHECK (length(creation_commit) > 0), + checkout_path TEXT NOT NULL UNIQUE CHECK ( + length(checkout_path) > 0 AND substr(checkout_path, 1, 1) = '/' + ), + PRIMARY KEY (repository_name, name) +) STRICT, WITHOUT ROWID`, + }, + ], + [ + "workflow_suspension_answers", + { + type: "table", + sql: `CREATE TABLE workflow_suspension_answers ( + suspension_id TEXT PRIMARY KEY, + request_event_id TEXT NOT NULL REFERENCES journal_events(event_id) ON DELETE RESTRICT, + request_fingerprint TEXT NOT NULL CHECK ( + length(request_fingerprint) = 64 AND request_fingerprint NOT GLOB '*[^0-9a-f]*' + ), + answer TEXT NOT NULL CHECK (json_valid(answer)), + state TEXT NOT NULL CHECK (state IN ('pending', 'consumed')), + created_at TEXT NOT NULL, + consumed_at TEXT, + CHECK ((state = 'consumed') = (consumed_at IS NOT NULL)) +) STRICT`, + }, + ], + [ + "workflow_fork_lineage", + { + type: "table", + sql: `CREATE TABLE workflow_fork_lineage ( + id INTEGER PRIMARY KEY CHECK (id = 1), + source_run_id TEXT NOT NULL CHECK (length(source_run_id) > 0), + checkpoint_event_id TEXT NOT NULL CHECK (length(checkpoint_event_id) > 0), + checkpoint_workspace_root_id TEXT NOT NULL + REFERENCES workspace_roots(root_id) ON DELETE RESTRICT, + created_at TEXT NOT NULL +) STRICT`, + }, + ], + [ + "journal_event_provenance", + { + type: "table", + sql: `CREATE TABLE journal_event_provenance ( + event_id TEXT PRIMARY KEY REFERENCES journal_events(event_id) ON DELETE RESTRICT, + source_run_id TEXT NOT NULL CHECK (length(source_run_id) > 0), + source_event_id TEXT NOT NULL CHECK (length(source_event_id) > 0) +) STRICT, WITHOUT ROWID`, + }, + ], +]); + +export const EXPECTED_SCHEMA = Object.freeze( + [...OBJECTS.entries()].map(([name, object]) => + Object.freeze({ name, type: object.type, sql: normalize(object.sql) }), + ), +); + +/** Objects version 1 declares, including the pinned Cloudflare structure. */ +export const REQUIRED_OBJECTS: readonly string[] = Object.freeze([...OBJECTS.keys()]); + +/** Tables version 1 declares. */ +export const REQUIRED_TABLES: readonly string[] = Object.freeze( + [...OBJECTS.entries()].filter(([, object]) => object.type === "table").map(([name]) => name), +); + +/** Version 1 in full. */ +export const SCHEMA_SQL = [...OBJECTS.values()] + .filter((object) => object.type === "table" && !object.sql.startsWith("CREATE TABLE vfs_")) + .filter((object) => !object.sql.startsWith("CREATE TABLE _vfs_")) + .map((object) => `${object.sql};`) + .join("\n\n"); + +/** One object a database declares, as `sqlite_schema` reports it. */ +export interface SchemaObject { + readonly type: string; + readonly name: string; + readonly sql: string; +} + +/** One statement's shape, independent of how it was laid out. */ +export function normalize(sql: string): string { + return sql.replace(/\s+/g, " ").trim(); +} + +/** + * Every in-place amendment to version 1, newest first. + * + * Each entry names what that amendment added. Peeling them off in order is what + * reconstructs the shapes that once claimed to be a complete version 1, so a + * database an earlier build produced is refused as an incomplete pre-release + * rather than as arbitrary damage. + */ +const AMENDMENTS: readonly (readonly string[])[] = Object.freeze([ + Object.freeze(["workflow_fork_lineage", "journal_event_provenance"]), + Object.freeze(["workflow_suspension_answers"]), + Object.freeze(["workspace_repositories", "workspace_worktrees"]), +]); + +/** What the newest amendment added. Its presence marks a current-shape database. */ +const LATEST_AMENDMENT: readonly string[] = AMENDMENTS[0] ?? []; + +/** The very first pre-release shape, before Workspace root retention existed. */ +const EARLIEST_PRE_RELEASE_SHAPE: readonly string[] = [ + "definition_retrieval", + "document_executions", + "journal_events", + "workflow_run", +]; + +/** + * Every later shape that once claimed to be a complete version 1. + * + * Newest first: version 1 minus the newest amendment, then minus the one before + * it, and so on. + */ +const PRIOR_COMPLETE_SHAPES: readonly (readonly string[])[] = Object.freeze( + AMENDMENTS.map((_, index) => { + const removed = new Set(AMENDMENTS.slice(0, index + 1).flat()); + return Object.freeze(REQUIRED_OBJECTS.filter((name) => !removed.has(name))); + }), +); + +/** + * Whether these declarations describe an earlier shape that once claimed to be + * a complete version 1. + * + * The very first pre-release held only the run, journal and execution tables. + * Every shape after it is version 1 minus whichever amendments had not been + * made yet, and each is named here so the refusal reads as an incomplete + * pre-release rather than as corruption. + */ +export function isIncompletePreReleaseShape(objects: readonly SchemaObject[]): boolean { + const present = new Set(objects.map((object) => object.name)); + if (LATEST_AMENDMENT.some((name) => present.has(name))) { + return false; + } + const earliest = new Set(EARLIEST_PRE_RELEASE_SHAPE); + if (present.size === earliest.size && [...present].every((name) => earliest.has(name))) { + return objects.every((object) => object.type === "table"); + } + return PRIOR_COMPLETE_SHAPES.some((shape) => { + const expected = new Set(shape); + return present.size === expected.size && [...present].every((name) => expected.has(name)); + }); +} + +/** What a structural disagreement is, without either adapter's error types. */ +export type StructureFailure = + | { readonly kind: "incomplete-pre-release" } + | { readonly kind: "undeclared-object"; readonly name: string } + | { readonly kind: "misshapen-object"; readonly name: string } + | { readonly kind: "missing-objects"; readonly names: readonly string[] }; + +/** + * Compare what a database declares with what this build writes. + * + * Answers with the disagreement rather than raising one, because the two + * adapters report the same finding as different failures: a path names the + * file the Deno host refused, and a Durable Object has no path to name. + * + * Recognizing a schema is not reading its table names. A dropped constraint and + * a column that is gone both leave the name intact, so the stored definition of + * every object is compared with the definition version 1 declares. + */ +export function declaredStructureFailure( + objects: readonly SchemaObject[], +): StructureFailure | undefined { + if (isIncompletePreReleaseShape(objects)) { + return { kind: "incomplete-pre-release" }; + } + for (const object of objects) { + const expected = OBJECTS.get(object.name); + if (expected === undefined) { + return { kind: "undeclared-object", name: object.name }; + } + if (object.type !== expected.type || normalize(object.sql) !== normalize(expected.sql)) { + return { kind: "misshapen-object", name: object.name }; + } + } + const present = new Set(objects.map((object) => object.name)); + const missing = REQUIRED_OBJECTS.filter((name) => !present.has(name)); + if (missing.length > 0) { + return { kind: "missing-objects", names: missing }; + } + return undefined; +} + +/** Whether any object version 1 declares is present at all. */ +export function hasAnyDeclaredObject(objects: readonly SchemaObject[]): boolean { + return objects.some((object) => OBJECTS.has(object.name)); +} diff --git a/packages/workflow/tests/cloudflare/env.d.ts b/packages/workflow/tests/cloudflare/env.d.ts index d81369bca..547adb507 100644 --- a/packages/workflow/tests/cloudflare/env.d.ts +++ b/packages/workflow/tests/cloudflare/env.d.ts @@ -1,5 +1,9 @@ -declare module "cloudflare:test" { - interface ProvidedEnv { - STORAGE_PROBE: DurableObjectNamespace; +import type { StorageProbeObject } from "./support/probe-object.ts"; + +declare global { + namespace Cloudflare { + interface Env { + STORAGE_PROBE: DurableObjectNamespace; + } } } diff --git a/packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts b/packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts index eda9754cd..ddbe22c9c 100644 --- a/packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts +++ b/packages/workflow/tests/cloudflare/storage-capabilities.vitest.ts @@ -1,16 +1,70 @@ +/** + * What a Durable Object's SQLite actually permits. + * + * The Cloudflare owner is built on these answers: the schema marker exists + * because the pragmas are refused, and the owner opens exactly one real + * transaction and enlists DOFS directly inside it because a reentrant + * transaction is refused. Both are properties of the runtime rather than of any + * model of it, so they are asserted here against real workerd — a platform + * change that moved either one should fail this suite rather than be discovered + * as a corrupted run. + * + * The assertions match categories, not the platform's wording: the exact + * sentence a runtime uses to refuse is not a contract, and pinning it would + * make this fail for a rephrasing. + */ + import { env, runInDurableObject } from "cloudflare:test"; -import { expect, it } from "vitest"; -import type { StorageProbeObject } from "../../src/cloudflare/probe-object.ts"; - -it("reports what real Durable Object SQLite storage accepts", async () => { - const id = env.STORAGE_PROBE.idFromName("capabilities"); - const stub = env.STORAGE_PROBE.get(id); - const capabilities = await runInDurableObject(stub, (instance: StorageProbeObject) => - instance.capabilities(), - ); - // Printed rather than asserted: this test exists to establish the contract - // the owner is written against, and a hard assertion here would encode a - // guess about an answer nobody has yet. - console.log(JSON.stringify(capabilities, null, 2)); - expect(capabilities).toBeTruthy(); +import { describe, expect, it } from "vitest"; +import type { StorageProbeObject } from "./support/probe-object.ts"; + +function capabilities() { + const stub = env.STORAGE_PROBE.get(env.STORAGE_PROBE.idFromName("capabilities")); + return runInDurableObject(stub, (instance: StorageProbeObject) => instance.capabilities()); +} + +/** A refusal, whatever the runtime called it. */ +function refused(answer: string): boolean { + return answer.startsWith("refused:"); +} + +/** A refusal the runtime attributed to its authorization layer. */ +function unauthorized(answer: string): boolean { + return refused(answer) && answer.includes("SQLITE_AUTH"); +} + +/** A refusal directing the caller to the storage transaction API. */ +function transactionApiRequired(answer: string): boolean { + return refused(answer) && answer.includes("transactionSync"); +} + +describe("Durable Object SQLite storage", () => { + it("refuses the pragmas the Deno host carries its schema identity in", async () => { + const found = await capabilities(); + expect(unauthorized(found.applicationIdRead)).toBe(true); + expect(unauthorized(found.applicationIdWrite)).toBe(true); + expect(unauthorized(found.userVersionRead)).toBe(true); + expect(unauthorized(found.userVersionWrite)).toBe(true); + }); + + it("refuses SQL transaction statements, directly and through a nested wrapper", async () => { + const found = await capabilities(); + expect(transactionApiRequired(found.savepointDirect)).toBe(true); + expect(transactionApiRequired(found.nestedTransaction)).toBe(true); + // The one that decides the owner's commit shape: the vendored DOFS opens a + // transaction of its own for a filesystem write, so calling it inside an + // owner transaction is a reentrant call and is refused. + expect(transactionApiRequired(found.filesystemInsideTransaction)).toBe(true); + }); + + it("accepts what the owner is built on instead", async () => { + const found = await capabilities(); + expect(refused(found.schemaObjects)).toBe(false); + expect(refused(found.outerTransaction)).toBe(false); + expect(refused(found.xmdTableDdl)).toBe(false); + expect(refused(found.dofsSchema)).toBe(false); + expect(refused(found.dofsFilesystem)).toBe(false); + // A strict metadata table is what carries the identity the pragmas cannot. + expect(found.metadataTable).toContain("application_id"); + }); }); diff --git a/packages/workflow/src/cloudflare/probe-object.ts b/packages/workflow/tests/cloudflare/support/probe-object.ts similarity index 87% rename from packages/workflow/src/cloudflare/probe-object.ts rename to packages/workflow/tests/cloudflare/support/probe-object.ts index fa65555c2..1daf1072e 100644 --- a/packages/workflow/src/cloudflare/probe-object.ts +++ b/packages/workflow/tests/cloudflare/support/probe-object.ts @@ -10,15 +10,18 @@ * has, or it does not, and that decides how §4 is written rather than being a * detail inside it. * - * So this object exists to be asked, on real workerd, before anything is built - * on the answer. + * So this object exists to be asked, on real workerd. It lives in test support + * rather than in production source: it measures the runtime, and the answers it + * gives are asserted by `storage-capabilities.vitest.ts` so a platform change + * that moved any of them would fail rather than pass quietly. */ import { DurableObject } from "cloudflare:workers"; -import { Database as DofsDatabase } from "../../vendor/cloudflare-computer-dofs/generated/storage.js"; -import { initializeSchema as initializeDofsSchema } from "../../vendor/cloudflare-computer-dofs/generated/schema/index.js"; -import { mkdir as mkdirPath } from "../../vendor/cloudflare-computer-dofs/generated/fs/mkdir.js"; -import { writeFileSync } from "../../vendor/cloudflare-computer-dofs/generated/fs/writeFile.js"; +import { dofsStorage } from "../../../src/cloudflare/storage.ts"; +import { Database as DofsDatabase } from "../../../vendor/cloudflare-computer-dofs/generated/storage.js"; +import { initializeSchema as initializeDofsSchema } from "../../../vendor/cloudflare-computer-dofs/generated/schema/index.js"; +import { mkdir as mkdirPath } from "../../../vendor/cloudflare-computer-dofs/generated/fs/mkdir.js"; +import { writeFileSync } from "../../../vendor/cloudflare-computer-dofs/generated/fs/writeFile.js"; export interface StorageCapabilities { readonly applicationIdRead: string; @@ -49,7 +52,7 @@ function attempt(body: () => unknown): string { export class StorageProbeObject extends DurableObject { capabilities(): StorageCapabilities { const sql = this.ctx.storage.sql; - const dofs = new DofsDatabase(this.ctx.storage); + const dofs = new DofsDatabase(dofsStorage(this.ctx.storage)); return { applicationIdWrite: attempt(() => { sql.exec("PRAGMA application_id = 1701078349"); diff --git a/packages/workflow/tests/cloudflare/worker.ts b/packages/workflow/tests/cloudflare/worker.ts index 7cd4831a3..cbbb53d15 100644 --- a/packages/workflow/tests/cloudflare/worker.ts +++ b/packages/workflow/tests/cloudflare/worker.ts @@ -6,7 +6,7 @@ * stubs, so this handler answers no request a test depends on. */ -export { StorageProbeObject } from "../../src/cloudflare/probe-object.ts"; +export { StorageProbeObject } from "./support/probe-object.ts"; export default { fetch(): Response { diff --git a/packages/workflow/tests/host-neutrality.test.ts b/packages/workflow/tests/host-neutrality.test.ts index 986e2bac5..b3a3cf2ec 100644 --- a/packages/workflow/tests/host-neutrality.test.ts +++ b/packages/workflow/tests/host-neutrality.test.ts @@ -95,6 +95,7 @@ describe("the shared workflow package", () => { expect(modules.length > 20).toEqual(true); expect(modules.some((path) => path.endsWith("/src/lifecycle/execution.ts"))).toEqual(true); expect(modules.some((path) => path.endsWith("/src/software-factory/run-id.ts"))).toEqual(true); + expect(modules.some((path) => path.endsWith("/src/sqlite/workflow-schema.ts"))).toEqual(true); expect(modules.some((path) => path.includes("/src/deno/"))).toEqual(false); expect(modules.some((path) => path.includes("/src/cloudflare/"))).toEqual(false); }); diff --git a/packages/workflow/tests/workspace-effect.test.ts b/packages/workflow/tests/workspace-effect.test.ts index b0f2175af..0faeb90cc 100644 --- a/packages/workflow/tests/workspace-effect.test.ts +++ b/packages/workflow/tests/workspace-effect.test.ts @@ -695,6 +695,13 @@ describe("Tier DLC — Workspace coordination selection", () => { // root and every shared module stay covered, so a host name reaching the // neutral surface is still a failure. // + // `src/sqlite` is a private physical SQLite backend the two runtime + // adapters share so version 1 is declared once rather than twice. It + // names a database engine because that is its subject, and it is not the + // provider-neutral coordination or external-effect surface — it owns no + // connection, path, transaction or lifecycle authority and is published + // from no entrypoint. + // // The software factory is the other kind of exception. It is not a // runtime adapter and is still held to the host-import and // runtime-detection rules by `host-neutrality.test.ts`; what it is @@ -709,6 +716,7 @@ describe("Tier DLC — Workspace coordination selection", () => { "packages/workflow/src/deno/**", "packages/workflow/src/cloudflare/**", "packages/workflow/src/software-factory/**", + "packages/workflow/src/sqlite/**", "packages/durable-streams/http-stream.ts", ], })) @@ -756,6 +764,7 @@ describe("Tier DLC — Workspace coordination selection", () => { expect(found.some((path) => path.includes("/src/deno/"))).toBe(false); expect(found.some((path) => path.includes("/src/cloudflare/"))).toBe(false); expect(found.some((path) => path.includes("/src/software-factory/"))).toBe(false); + expect(found.some((path) => path.includes("/src/sqlite/"))).toBe(false); const crossings: Record = {}; const unread: string[] = []; diff --git a/packages/workflow/tsconfig.cloudflare.json b/packages/workflow/tsconfig.cloudflare.json new file mode 100644 index 000000000..96d28350a --- /dev/null +++ b/packages/workflow/tsconfig.cloudflare.json @@ -0,0 +1,19 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ESNext", + "moduleResolution": "bundler", + "lib": ["ES2022"], + "types": ["@cloudflare/workers-types", "@cloudflare/vitest-plugin/types"], + "strict": true, + "exactOptionalPropertyTypes": true, + "noUncheckedIndexedAccess": true, + "allowImportingTsExtensions": true, + "allowJs": true, + "checkJs": false, + "noEmit": true, + "skipLibCheck": true, + "verbatimModuleSyntax": true + }, + "include": ["src/cloudflare/**/*.ts", "tests/cloudflare/**/*.ts"] +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 388492231..d4709f0e7 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -6,6 +6,7 @@ settings: overrides: tsx: 4.23.1 + '@cloudflare/workers-types': 5.20260831.1 importers: @@ -86,7 +87,10 @@ importers: devDependencies: '@cloudflare/vitest-plugin': specifier: 1.1.3 - version: 1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1))) + version: 1.1.3(@cloudflare/workers-types@5.20260831.1)(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1))) + '@cloudflare/workers-types': + specifier: 5.20260831.1 + version: 5.20260831.1 '@durable-streams/server': specifier: ^0.3.8 version: 0.3.8 @@ -563,6 +567,9 @@ packages: cpu: [x64] os: [win32] + '@cloudflare/workers-types@5.20260831.1': + resolution: {integrity: sha512-yXg4pwfYjhsDH9rYc3qZ3K+z62DCSvO/aj7GiZo6AyDeWGZpyFRpPMYcQ6LF/zfaf1x0Ngw2gSqL8JjuUtMGlA==} + '@colors/colors@1.5.0': resolution: {integrity: sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ==} engines: {node: '>=0.1.90'} @@ -3104,7 +3111,7 @@ packages: engines: {node: '>=22.0.0'} hasBin: true peerDependencies: - '@cloudflare/workers-types': ^5.20260831.1 + '@cloudflare/workers-types': 5.20260831.1 peerDependenciesMeta: '@cloudflare/workers-types': optional: true @@ -3190,7 +3197,7 @@ snapshots: optionalDependencies: workerd: 1.20260831.1 - '@cloudflare/vitest-plugin@1.1.3(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)))': + '@cloudflare/vitest-plugin@1.1.3(@cloudflare/workers-types@5.20260831.1)(@vitest/runner@4.1.11)(@vitest/snapshot@4.1.11)(vitest@4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)))': dependencies: '@vitest/runner': 4.1.11 '@vitest/snapshot': 4.1.11 @@ -3198,7 +3205,7 @@ snapshots: esbuild: 0.28.1 miniflare: 5.20260831.0-alpha vitest: 4.1.11(@types/node@22.19.15)(vite@8.2.2(@types/node@22.19.15)(esbuild@0.28.1)(tsx@4.23.1)) - wrangler: 4.128.0 + wrangler: 4.128.0(@cloudflare/workers-types@5.20260831.1) zod: 4.4.3 transitivePeerDependencies: - '@cloudflare/workers-types' @@ -3220,6 +3227,8 @@ snapshots: '@cloudflare/workerd-windows-64@1.20260831.1': optional: true + '@cloudflare/workers-types@5.20260831.1': {} + '@colors/colors@1.5.0': optional: true @@ -5520,7 +5529,7 @@ snapshots: '@cloudflare/workerd-linux-arm64': 1.20260831.1 '@cloudflare/workerd-windows-64': 1.20260831.1 - wrangler@4.128.0: + wrangler@4.128.0(@cloudflare/workers-types@5.20260831.1): dependencies: '@cloudflare/kv-asset-handler': 0.5.0 '@cloudflare/unenv-preset': 2.16.1(unenv@2.0.0-rc.24)(workerd@1.20260831.1) @@ -5531,6 +5540,7 @@ snapshots: unenv: 2.0.0-rc.24 workerd: 1.20260831.1 optionalDependencies: + '@cloudflare/workers-types': 5.20260831.1 fsevents: 2.3.3 transitivePeerDependencies: - bufferutil From f7b2a24d1e4be3902a27530037200a30b95e1704 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 19:19:10 -0400 Subject: [PATCH 14/81] =?UTF-8?q?=E2=9C=85=20Prove=20the=20owner's=20stora?= =?UTF-8?q?ge=20on=20real=20workerd=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Cloudflare owner's storage paths, and the evidence that they hold. `src/cloudflare/owner-transaction.ts` is the transaction every authoritative commit runs inside. It enters `ctx.storage.transactionSync()` once and hands DOFS a wrapper whose `transactionSync` runs its callback directly — inside the real callback that is not a weaker promise, because the outer transaction is already open, and it is the only way DOFS can participate at all: asked to transact while it believes one is open, the vendor falls back to `SAVEPOINT`, which the runtime refuses, and every DOFS filesystem primitive opens a transaction on the way in. The enlistment is created for one callback, refuses use outside it, and clears the DOFS resolve and blob caches on both sides so nothing populated from rows a rollback discarded is read later. The vendored snapshot is untouched and `deno task vendor:verify` still passes. `src/cloudflare/recognition.ts` initializes pristine storage in one transaction — schema, DOFS schema, run row, then the marker last, so the code says what the marker means even though atomicity hides the ordering — and recognizes it again through the same four conditions the Deno host distinguishes: foreign, unsupported version, corrupt, or a version-1 run. Fourteen assertions on real workerd, in two suites. Initialization writes the marker and is recognized again; storage that already holds an object is refused rather than written into; nothing at all, objects without a marker, another application's identity, version 2, version 0, and a dropped table are each refused as their own condition. A mixed commit publishes a DOFS filesystem write and a WorkflowRun row together. A body that throws after changing both leaves neither — the run row is unchanged, the file is absent, the object still recognizes, and a later commit succeeds from the frontier the failure left. That last test is the one the whole design turns on. `pnpm check:cloudflare` type-checks all of it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- .../src/cloudflare/owner-transaction.ts | 117 +++++++++++ .../workflow/src/cloudflare/recognition.ts | 187 ++++++++++++++++++ packages/workflow/tests/cloudflare/env.d.ts | 2 + .../tests/cloudflare/owner-storage.vitest.ts | 122 ++++++++++++ .../tests/cloudflare/support/owner-object.ts | 137 +++++++++++++ packages/workflow/tests/cloudflare/worker.ts | 1 + .../workflow/tests/cloudflare/wrangler.jsonc | 7 +- 7 files changed, 571 insertions(+), 2 deletions(-) create mode 100644 packages/workflow/src/cloudflare/owner-transaction.ts create mode 100644 packages/workflow/src/cloudflare/recognition.ts create mode 100644 packages/workflow/tests/cloudflare/owner-storage.vitest.ts create mode 100644 packages/workflow/tests/cloudflare/support/owner-object.ts diff --git a/packages/workflow/src/cloudflare/owner-transaction.ts b/packages/workflow/src/cloudflare/owner-transaction.ts new file mode 100644 index 000000000..12cd09bde --- /dev/null +++ b/packages/workflow/src/cloudflare/owner-transaction.ts @@ -0,0 +1,117 @@ +/** + * The one real transaction an owner commit runs inside. + * + * A Durable Object's SQLite accepts exactly one shape of transaction: the + * runtime's own `transactionSync()`, entered once. It refuses `BEGIN`, `COMMIT` + * and `SAVEPOINT` through `sql.exec()`, and it refuses a reentrant + * `transactionSync()`. The vendored DOFS `Database` does not know that — asked + * to transact while it believes a transaction is already open, it falls back to + * `SAVEPOINT`, and every DOFS filesystem primitive opens a transaction of its + * own on the way in. + * + * So the owner enters the real transaction itself and hands DOFS a wrapper + * whose `transactionSync` runs its callback directly. Inside the real + * callback that is not a weaker promise: the outer transaction is already + * open, so a body that returns has had its work applied to the same + * transaction, and a body that throws unwinds through the real callback and + * Cloudflare rolls the whole thing back. + * + * That substitution is only safe because the owner does not use a DOFS + * savepoint as a recovery boundary. The runner has already performed the live + * Workspace work against disposable materialization; what reaches the owner is + * a complete proposal. The owner commits all of it or, treating any validation + * or application failure as infrastructure failure, none of it. + * + * The wrapper is created for one callback and refuses use outside it, so + * nothing can retain it and reach the storage later. Its DOFS caches are built + * fresh for the same reason: a resolution or blob cache populated from + * uncommitted rows must not survive a rollback or be read by a later + * operation. + */ + +import { Database as DofsDatabase } from "../../vendor/cloudflare-computer-dofs/generated/storage.js"; +import { clearBlobCache } from "../../vendor/cloudflare-computer-dofs/generated/fs/blobCache.js"; +import { clearResolveCache } from "../../vendor/cloudflare-computer-dofs/generated/fs/resolveCache.js"; +import { dofsStorage, type OwnerStorage } from "./storage.ts"; + +/** Using an enlistment after its transaction returned. */ +export class OwnerTransactionClosedError extends Error { + override name = "OwnerTransactionClosedError"; + + constructor() { + super( + "this owner transaction has finished; a DOFS enlistment is valid only inside the callback that created it.", + ); + } +} + +/** Opening an owner transaction inside one. */ +export class OwnerTransactionNestedError extends Error { + override name = "OwnerTransactionNestedError"; + + constructor() { + super( + "an owner transaction is already open; Durable Object storage admits exactly one, and a second would reach SAVEPOINT.", + ); + } +} + +/** What the body of an owner transaction is given. */ +export interface OwnerTransaction { + /** The DOFS database, enlisted in this transaction and valid only inside it. */ + readonly dofs: DofsDatabase; +} + +/** Whether an owner transaction is currently open on this object. */ +let open = false; + +/** + * Run `body` inside one real `ctx.storage.transactionSync()`. + * + * `body` must complete synchronously. Nothing may await, suspend, hold a + * cursor, wait on a WebSocket or reach the runner from inside it: the runtime + * requires the callback to finish before it can commit, and a value that + * arrived later would be applied to a transaction nobody is holding. + */ +export function ownerTransaction( + storage: OwnerStorage, + body: (transaction: OwnerTransaction) => T, +): T { + if (open) { + throw new OwnerTransactionNestedError(); + } + open = true; + try { + return storage.transactionSync(() => { + let live = true; + const dofs = new DofsDatabase(dofsStorage(storage)); + // Fresh caches for this transaction alone. They are keyed by database, so + // an entry populated from rows this transaction may roll back would + // otherwise outlive it and be read by a later operation. + clearResolveCache(dofs); + clearBlobCache(dofs); + // The substitution: DOFS believes it is opening a transaction, and runs + // in the one already open. Reentrancy inside DOFS becomes ordinary + // nesting of plain function calls, which is what the runtime allows. + Object.defineProperty(dofs, "transactionSync", { + value: (closure: () => R): R => { + if (!live) { + throw new OwnerTransactionClosedError(); + } + return closure(); + }, + configurable: false, + writable: false, + }); + try { + return body({ dofs }); + } finally { + live = false; + clearResolveCache(dofs); + clearBlobCache(dofs); + } + }); + } finally { + open = false; + } +} diff --git a/packages/workflow/src/cloudflare/recognition.ts b/packages/workflow/src/cloudflare/recognition.ts new file mode 100644 index 000000000..13945e85f --- /dev/null +++ b/packages/workflow/src/cloudflare/recognition.ts @@ -0,0 +1,187 @@ +/** + * Whether this Durable Object's storage is a version-1 workflow run, and how it + * becomes one. + * + * The conditions are the ones the Deno host distinguishes, because they are + * what a caller acts on differently: storage nobody has written yet may be + * initialized; storage belonging to something else, or claiming a version this + * build does not implement, must be left alone; and storage that claims version + * 1 and is not shaped like it is damaged. Collapsing them would leave a host + * guessing whether to create, refuse, or report damage. + * + * What differs from Deno is only where the claim is written. The pragmas that + * carry it in a file are refused here, so `_xmd_workflow_schema` carries it + * instead. Nothing initializes, migrates, repairs or replaces storage this + * module refuses. + */ + +import { initializeSchema as initializeDofsSchema } from "../../vendor/cloudflare-computer-dofs/generated/schema/index.js"; +import { + APPLICATION_ID, + declaredStructureFailure, + hasAnyDeclaredObject, + SCHEMA_SQL, + SCHEMA_VERSION, + type SchemaObject, +} from "../sqlite/workflow-schema.ts"; +import { isSchemaMarker, MARKER_SQL, MARKER_TABLE, readMarker } from "./marker.ts"; +import { ownerTransaction } from "./owner-transaction.ts"; +import type { OwnerStorage } from "./storage.ts"; + +/** Why storage could not be read as a version-1 workflow run. */ +export type RecognitionFailure = + | { readonly kind: "foreign"; readonly detail: string } + | { readonly kind: "unsupported-version"; readonly schemaVersion: number } + | { readonly kind: "corrupt"; readonly detail: string }; + +export class WorkflowObjectStorageError extends Error { + override name = "WorkflowObjectStorageError"; + + constructor(readonly failure: RecognitionFailure) { + super(describeFailure(failure)); + } +} + +function describeFailure(failure: RecognitionFailure): string { + if (failure.kind === "foreign") { + return `this Durable Object's storage is not a workflow run: ${failure.detail}`; + } + if (failure.kind === "unsupported-version") { + return `this Durable Object's storage declares schema version ${failure.schemaVersion}, which this build does not implement`; + } + return `this Durable Object's storage is damaged: ${failure.detail}`; +} + +/** Every object the storage declares, drained where the cursor is created. */ +export function declaredObjects(storage: OwnerStorage): SchemaObject[] { + const rows = storage.sql + .exec("SELECT type, name, sql FROM sqlite_schema WHERE name NOT LIKE 'sqlite_%' ORDER BY name") + .toArray(); + return rows.map((row) => ({ + type: String(row["type"]), + name: String(row["name"]), + sql: row["sql"] === null || row["sql"] === undefined ? "" : String(row["sql"]), + })); +} + +/** + * Whether this storage holds nothing at all. + * + * Pristine means no object anybody created — not XMD's, not DOFS's, not the + * marker's, and nothing unrelated. Storage carrying any object but no marker is + * foreign or half-initialized, and is refused rather than written into. + */ +export function isPristine(objects: readonly SchemaObject[]): boolean { + return objects.length === 0; +} + +function markerRows(storage: OwnerStorage): Record[] { + return storage.sql.exec(`SELECT application_id, schema_version FROM ${MARKER_TABLE}`).toArray(); +} + +/** + * Make pristine storage into a version-1 workflow run, in one transaction. + * + * The marker is written last. Atomicity means no observer could see the + * ordering, so this is the code saying what the marker means: an identity claim + * over a schema that is already complete. + */ +export function initializeObject(storage: OwnerStorage, initializeRun: () => void): void { + const objects = declaredObjects(storage); + if (!isPristine(objects)) { + throw new WorkflowObjectStorageError({ + kind: "foreign", + detail: "it already holds objects and carries no workflow schema marker", + }); + } + ownerTransaction(storage, ({ dofs }) => { + storage.sql.exec(SCHEMA_SQL); + initializeDofsSchema(dofs, () => 0); + initializeRun(); + storage.sql.exec(MARKER_SQL); + storage.sql.exec( + `INSERT INTO ${MARKER_TABLE} (id, application_id, schema_version) VALUES (1, ?, ?)`, + APPLICATION_ID, + SCHEMA_VERSION, + ); + }); +} + +/** + * Refuse anything that is not a version-1 workflow run. + * + * Structure only. Whether the rows describe the run that was asked for is a + * separate question, asked after this one succeeds. + */ +export function recognizeObject(storage: OwnerStorage): void { + const objects = declaredObjects(storage); + if (isPristine(objects)) { + throw new WorkflowObjectStorageError({ + kind: "foreign", + detail: "it holds nothing at all", + }); + } + + const carriesMarker = objects.some((object) => object.name === MARKER_TABLE); + if (!carriesMarker) { + throw new WorkflowObjectStorageError({ + kind: "foreign", + detail: hasAnyDeclaredObject(objects) + ? "it declares workflow tables without the schema marker that identifies them" + : "it belongs to something else", + }); + } + + const marker = readMarker(markerRows(storage)); + if (!isSchemaMarker(marker)) { + if (marker.kind === "unknown-version") { + throw new WorkflowObjectStorageError({ + kind: "unsupported-version", + schemaVersion: marker.schemaVersion, + }); + } + if (marker.kind === "foreign-application") { + throw new WorkflowObjectStorageError({ + kind: "foreign", + detail: "its schema marker carries another application's identity", + }); + } + throw new WorkflowObjectStorageError({ + kind: "corrupt", + detail: + marker.kind === "absent" + ? "its schema marker table holds no identity row" + : marker.kind === "duplicated" + ? "its schema marker table holds more than one identity row" + : "its schema marker row does not describe an identity", + }); + } + + const declared = objects.filter((object) => object.name !== MARKER_TABLE); + const failure = declaredStructureFailure(declared); + if (failure === undefined) { + return; + } + if (failure.kind === "incomplete-pre-release") { + throw new WorkflowObjectStorageError({ + kind: "corrupt", + detail: "it holds an incomplete pre-release of version 1", + }); + } + if (failure.kind === "undeclared-object") { + throw new WorkflowObjectStorageError({ + kind: "corrupt", + detail: `it declares an object that version ${SCHEMA_VERSION} does not`, + }); + } + if (failure.kind === "misshapen-object") { + throw new WorkflowObjectStorageError({ + kind: "corrupt", + detail: `its ${failure.name} object is not shaped the way version ${SCHEMA_VERSION} declares it`, + }); + } + throw new WorkflowObjectStorageError({ + kind: "corrupt", + detail: `it is missing the table ${failure.names.join(", ")}`, + }); +} diff --git a/packages/workflow/tests/cloudflare/env.d.ts b/packages/workflow/tests/cloudflare/env.d.ts index 547adb507..8697c38fd 100644 --- a/packages/workflow/tests/cloudflare/env.d.ts +++ b/packages/workflow/tests/cloudflare/env.d.ts @@ -1,9 +1,11 @@ +import type { OwnerObject } from "./support/owner-object.ts"; import type { StorageProbeObject } from "./support/probe-object.ts"; declare global { namespace Cloudflare { interface Env { STORAGE_PROBE: DurableObjectNamespace; + OWNER: DurableObjectNamespace; } } } diff --git a/packages/workflow/tests/cloudflare/owner-storage.vitest.ts b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts new file mode 100644 index 000000000..414b1801c --- /dev/null +++ b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts @@ -0,0 +1,122 @@ +/** + * The owner's storage, on real workerd. + * + * Initialization, recognition and the one transaction an owner commit runs + * inside are all properties of the runtime rather than of a model of it: the + * marker exists because the pragmas are refused, and the direct DOFS enlistment + * exists because a reentrant transaction is refused. Each object below gets a + * fresh name so its storage starts pristine. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { describe, expect, it } from "vitest"; +import type { OwnerObject } from "./support/owner-object.ts"; + +let unique = 0; + +function owner() { + unique += 1; + const name = `owner-${unique}-${Math.random().toString(36).slice(2)}`; + return env.OWNER.get(env.OWNER.idFromName(name)); +} + +function on( + stub: ReturnType, + body: (instance: OwnerObject) => T, +): Promise> { + return runInDurableObject(stub, body) as Promise>; +} + +describe("initializing an owner object", () => { + it("creates the schema, the DOFS schema, the run row and the marker together", async () => { + const stub = owner(); + expect(await on(stub, (o) => o.initialize())).toBe("initialized"); + expect(await on(stub, (o) => o.marker())).toEqual([ + { application_id: 0x584d4431, schema_version: 1 }, + ]); + expect(await on(stub, (o) => o.recognize())).toBe("recognized"); + }); + + it("refuses storage that already holds something", async () => { + const stub = owner(); + await on(stub, (o) => o.addForeignObject()); + expect(await on(stub, (o) => o.initialize())).toBe("refused:foreign"); + }); +}); + +describe("recognizing an owner object", () => { + it("refuses storage that holds nothing at all", async () => { + expect(await on(owner(), (o) => o.recognize())).toBe("refused:foreign"); + }); + + it("refuses storage carrying objects but no marker", async () => { + const stub = owner(); + await on(stub, (o) => o.addForeignObject()); + expect(await on(stub, (o) => o.recognize())).toBe("refused:foreign"); + }); + + it("refuses another application's identity", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.rewriteMarker(0x11111111, 1)); + expect(await on(stub, (o) => o.recognize())).toBe("refused:foreign"); + }); + + it("refuses a version this build does not implement", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.rewriteMarker(0x584d4431, 2)); + expect(await on(stub, (o) => o.recognize())).toBe("refused:unsupported-version"); + }); + + it("refuses version zero the same way", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.rewriteMarker(0x584d4431, 0)); + expect(await on(stub, (o) => o.recognize())).toBe("refused:unsupported-version"); + }); + + it("refuses a shape that disagrees with what version 1 declares", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.damage("workflow_suspension_answers")); + expect(await on(stub, (o) => o.recognize())).toBe("refused:corrupt"); + }); +}); + +describe("an owner commit", () => { + it("publishes DOFS content and WorkflowRun rows together", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + expect(await on(stub, (o) => o.frontier())).toEqual({ status: "running", publishedPaths: 0 }); + + expect(await on(stub, (o) => o.commitMixedChange(false))).toBe("committed"); + expect(await on(stub, (o) => o.frontier())).toEqual({ + status: "suspended", + publishedPaths: 1, + }); + }); + + it("rolls both categories back when the body fails after changing each", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + expect(await on(stub, (o) => o.commitMixedChange(true))).toContain("threw:"); + + // Neither the filesystem write nor the row update may survive, and the next + // operation must not read either of them out of a cache the failed + // transaction populated. + expect(await on(stub, (o) => o.frontier())).toEqual({ status: "running", publishedPaths: 0 }); + expect(await on(stub, (o) => o.recognize())).toBe("recognized"); + }); + + it("commits after a failed attempt, from the frontier the failure left", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.commitMixedChange(true)); + expect(await on(stub, (o) => o.commitMixedChange(false))).toBe("committed"); + expect(await on(stub, (o) => o.frontier())).toEqual({ + status: "suspended", + publishedPaths: 1, + }); + }); +}); diff --git a/packages/workflow/tests/cloudflare/support/owner-object.ts b/packages/workflow/tests/cloudflare/support/owner-object.ts new file mode 100644 index 000000000..79d5a1df1 --- /dev/null +++ b/packages/workflow/tests/cloudflare/support/owner-object.ts @@ -0,0 +1,137 @@ +/** + * A Durable Object that exercises the owner's storage paths on real workerd. + * + * It is deliberately thin: each method does one thing the owner does — create + * the schema, recognize it again, commit a mixed change, or fail partway + * through one — so a test can assert the outcome rather than a model of it. + */ + +import { DurableObject } from "cloudflare:workers"; +import { mkdir as mkdirPath } from "../../../vendor/cloudflare-computer-dofs/generated/fs/mkdir.js"; +import { writeFileSync } from "../../../vendor/cloudflare-computer-dofs/generated/fs/writeFile.js"; +import { + initializeObject, + recognizeObject, + WorkflowObjectStorageError, +} from "../../../src/cloudflare/recognition.ts"; +import { MARKER_TABLE } from "../../../src/cloudflare/marker.ts"; +import { ownerTransaction } from "../../../src/cloudflare/owner-transaction.ts"; + +/** One run row, so initialization writes what a real run would. */ +const RUN_ID = "run-under-test"; + +export class OwnerObject extends DurableObject { + /** Create the schema, DOFS schema, an empty root and the run row, then mark it. */ + initialize(): string { + try { + initializeObject(this.ctx.storage, () => { + this.ctx.storage.sql.exec( + "INSERT INTO workflow_run (run_id, definition, base, props, status, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?)", + RUN_ID, + JSON.stringify({ version: 1 }), + "main", + "{}", + "running", + 0, + 0, + ); + }); + return "initialized"; + } catch (error) { + return describe(error); + } + } + + /** Read the storage back as a version-1 workflow run. */ + recognize(): string { + try { + recognizeObject(this.ctx.storage); + return "recognized"; + } catch (error) { + return describe(error); + } + } + + /** What the marker holds right now. */ + marker(): Record[] { + return this.ctx.storage.sql + .exec(`SELECT application_id, schema_version FROM ${MARKER_TABLE}`) + .toArray(); + } + + /** Drop one declared object, so recognition sees a shape that disagrees. */ + damage(table: string): void { + this.ctx.storage.sql.exec(`DROP TABLE ${table}`); + } + + /** Write an unrelated object, so pristine detection sees a foreign store. */ + addForeignObject(): void { + this.ctx.storage.sql.exec("CREATE TABLE somebody_elses (id INTEGER PRIMARY KEY)"); + } + + /** Replace the marker's identity with another application's. */ + rewriteMarker(applicationId: number, schemaVersion: number): void { + this.ctx.storage.sql.exec( + `UPDATE ${MARKER_TABLE} SET application_id = ?, schema_version = ? WHERE id = 1`, + applicationId, + schemaVersion, + ); + } + + /** + * Change DOFS content and a WorkflowRun row in one transaction. + * + * `fail` throws after both have been changed, which is the case that decides + * whether the two categories really share a transaction. + */ + commitMixedChange(fail: boolean): string { + try { + ownerTransaction(this.ctx.storage, ({ dofs }) => { + mkdirPath(dofs, "/published", { recursive: true }, () => 0); + // oxlint-disable-next-line local/no-sync-filesystem + writeFileSync( + dofs, + "/published/root.txt", + new TextEncoder().encode("frontier"), + {}, + () => 0, + ); + this.ctx.storage.sql.exec( + "UPDATE workflow_run SET status = ?, updated_at = ? WHERE run_id = ?", + "suspended", + 1, + RUN_ID, + ); + if (fail) { + throw new Error("forced failure after both categories changed"); + } + }); + return "committed"; + } catch (error) { + return describe(error); + } + } + + /** What the run row and the DOFS filesystem hold, read outside any transaction. */ + frontier(): { status: string; publishedPaths: number } { + const runRows = this.ctx.storage.sql + .exec("SELECT status FROM workflow_run WHERE run_id = ?", RUN_ID) + .toArray(); + const first = runRows[0]; + const paths = this.ctx.storage.sql + .exec("SELECT count(*) AS found FROM vfs_dirents WHERE name = ?", "root.txt") + .toArray(); + const found = paths[0]; + return { + status: first === undefined ? "absent" : String(first["status"]), + publishedPaths: found === undefined ? -1 : Number(found["found"]), + }; + } +} + +function describe(error: unknown): string { + if (error instanceof WorkflowObjectStorageError) { + return `refused:${error.failure.kind}`; + } + return `threw:${error instanceof Error ? error.message : String(error)}`; +} diff --git a/packages/workflow/tests/cloudflare/worker.ts b/packages/workflow/tests/cloudflare/worker.ts index cbbb53d15..4143c1ffd 100644 --- a/packages/workflow/tests/cloudflare/worker.ts +++ b/packages/workflow/tests/cloudflare/worker.ts @@ -7,6 +7,7 @@ */ export { StorageProbeObject } from "./support/probe-object.ts"; +export { OwnerObject } from "./support/owner-object.ts"; export default { fetch(): Response { diff --git a/packages/workflow/tests/cloudflare/wrangler.jsonc b/packages/workflow/tests/cloudflare/wrangler.jsonc index df1a8d9a4..0315d0393 100644 --- a/packages/workflow/tests/cloudflare/wrangler.jsonc +++ b/packages/workflow/tests/cloudflare/wrangler.jsonc @@ -4,7 +4,10 @@ "compatibility_date": "2026-08-01", "compatibility_flags": ["nodejs_compat"], "durable_objects": { - "bindings": [{ "name": "STORAGE_PROBE", "class_name": "StorageProbeObject" }], + "bindings": [ + { "name": "STORAGE_PROBE", "class_name": "StorageProbeObject" }, + { "name": "OWNER", "class_name": "OwnerObject" }, + ], }, - "migrations": [{ "tag": "v1", "new_sqlite_classes": ["StorageProbeObject"] }], + "migrations": [{ "tag": "v1", "new_sqlite_classes": ["StorageProbeObject", "OwnerObject"] }], } From 0cbd8ef2392bb39c32b99467862fda935e0ff22b Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 19:21:27 -0400 Subject: [PATCH 15/81] =?UTF-8?q?=F0=9F=91=B7=20Give=20the=20workerd=20sui?= =?UTF-8?q?te=20a=20required=20CI=20job=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `test-cloudflare` runs `pnpm check:cloudflare` and `pnpm test:cloudflare`, and is in `green.needs`. It runs on every event and carries no condition, so `green` requires success from it unconditionally. The job owns evidence no other job can produce. A Durable Object's acquisition lifetime, its eviction and its transaction atomicity are properties of that runtime, and the `.vitest.ts` files proving them are invisible to the Deno, Node and Bun corpora by design — so without this job the evidence would simply stop running while everything else stayed green. `ci-workflow.test.ts` already fails when a job is missing from the aggregate; verified by removing the entry and watching "requires every other job and no future job can be omitted" fail. Added a test naming this job's two commands as well, so what it is *for* is legible rather than only that it is depended on. `pnpm install` is the job's last preparation step: `deno install` prunes the links pnpm placed, and the plugin only takes over the pool when it and the CLI hold the same vitest. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- .github/workflows/ci.yml | 29 +++++++++++++++++++++++++++++ scripts/tests/ci-workflow.test.ts | 20 ++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ecc2c8a40..2aea93d7d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -640,6 +640,34 @@ jobs: # on a `ci-main-red-fix` pull request, and may be skipped on an ordinary one. # Every other job must succeed outright — an unexpected skip is an unproven # job, which is exactly what this check exists to catch. + # The workerd suite. It runs where the runtime is real: acquisition lifetime, + # owner eviction and transaction atomicity are properties of a Durable Object + # rather than of any model of one, so none of them is provable in the Deno, + # Node or Bun corpora — which is also why these files carry a `.vitest.ts` + # suffix those corpora never discover. + # + # `pnpm install` comes last on purpose: `deno install` prunes the links pnpm + # placed (scripts/deps.ts says so in its own header), and the plugin only + # takes over the pool when it and the CLI hold the same vitest. + test-cloudflare: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 + with: + node-version: "22" + + - uses: pnpm/action-setup@b906affcce14559ad1aafd4ab0e942779e9f58b1 # v4 + + - run: pnpm install + + - name: Typecheck the Cloudflare owner + run: pnpm check:cloudflare + + - name: Cloudflare Workers suite + run: pnpm test:cloudflare + green: needs: [ @@ -653,6 +681,7 @@ jobs: site, test-node, test-bun, + test-cloudflare, ] if: always() runs-on: ubuntu-latest diff --git a/scripts/tests/ci-workflow.test.ts b/scripts/tests/ci-workflow.test.ts index 922d5b164..0f04745d4 100644 --- a/scripts/tests/ci-workflow.test.ts +++ b/scripts/tests/ci-workflow.test.ts @@ -472,6 +472,26 @@ describe("the conditional CI jobs", () => { return found.if; } + /** + * The workerd suite runs nowhere else. A Durable Object's acquisition + * lifetime, its eviction and its transaction atomicity are properties of that + * runtime, and the `.vitest.ts` files that prove them are invisible to the + * Deno, Node and Bun corpora by design — so if this job stopped running the + * evidence would go with it and every other job would still be green. + */ + it("owns the Cloudflare typecheck and the workerd suite", function* () { + const jobs = yield* workflow(); + const job = jobs["test-cloudflare"]; + expect(job).toBeDefined(); + const commands = (job?.steps ?? []).flatMap((step) => + step.run === undefined ? [] : [step.run], + ); + expect(commands).toContain("pnpm check:cloudflare"); + expect(commands).toContain("pnpm test:cloudflare"); + // It runs on every event, so `green` requires success from it unconditionally. + expect(job?.if).toBeUndefined(); + }); + it("runs main-green on a pull request and on nothing else", function* () { expect(conditional(yield* workflow(), "main-green")).toEqual( "github.event_name == 'pull_request'", From b69bab425dd400618b36d6c74124930c557f5179 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 20:05:49 -0400 Subject: [PATCH 16/81] =?UTF-8?q?=F0=9F=94=90=20Admit=20an=20executor=20an?= =?UTF-8?q?d=20give=20it=20a=20private=20transport=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The executor plane: who may advance a run, and what they may say. `release.ts` compares an exact build fingerprint. `admission.ts` holds verified OIDC claims to a configured policy — issuer, audience, repository ID, owner ID, event name, workflow ref, workflow SHA and the immutable workflow identity — checking IDs rather than names, because a repository can be renamed and a check on the name would admit whoever holds it today. It reads a closed claim set, so a claim this contract does not name cannot be depended on, and it takes claims a verifier already authenticated rather than a token: how a signature is checked is not this module's business. `acquisition.ts` makes the connection the acquisition. Hibernation is why it cannot be a field — an evicted object has no memory of what it admitted — so authority is the pair the runtime hands back: `getWebSockets()` says which sockets are real and a bounded attachment says what one was admitted as. Copied attachment bytes prove nothing, because the question is not whether a value looks right but whether this socket is the one live socket holding an acquisition. There is no lease, expiry, renewal, heartbeat, alarm or poll. `commands.ts` reads the private transport strictly: unknown command, unknown member, wrong kind, oversized message, too many chunks — each refuses whole, nothing is partially adopted. The answer is a serialized record keyed on `outcome` rather than an Effection `Result`, because an `Error` does not cross a connection; Code Rule 13 governs in-process results and this is not one. `owner.ts` is the Durable Object. Its admission order is the contract: build before token, token before run, acquisition last, so a refusal at any step leaves no acquisition and no state. A message proves its acquisition before it is parsed. A closed connection releases ownership, rolls nothing back and settles nothing — an executor that disappeared decided nothing. `routing.ts` selects the owner arithmetically and admits the run ID first, because `idFromName` answers for any string and a mistyped id would silently address a fresh, empty owner. Twenty-nine assertions on real workerd across three suites. A mismatched build is refused while its claims are deliberately unusable, proving the order. Every policy claim is refused one at a time. A second healthy executor is refused rather than followed. A closed connection owns nothing and the next executor may take it with no lease having expired. A stranger's socket is refused before its command is read. `./cloudflare` is an npm export only: JSR cannot typecheck a `cloudflare:` entrypoint, and `deno task check:jsr` stays green because the Deno host is what JSR serves. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- deno.json | 3 +- packages/workflow/cloudflare.ts | 57 +++++ packages/workflow/package.json | 1 + .../workflow/src/cloudflare/acquisition.ts | 163 +++++++++++++ packages/workflow/src/cloudflare/admission.ts | 121 ++++++++++ packages/workflow/src/cloudflare/commands.ts | 215 ++++++++++++++++++ packages/workflow/src/cloudflare/owner.ts | 162 +++++++++++++ packages/workflow/src/cloudflare/release.ts | 62 +++++ packages/workflow/src/cloudflare/routing.ts | 68 ++++++ packages/workflow/tests/cloudflare/env.d.ts | 2 + .../cloudflare/executor-acquisition.vitest.ts | 206 +++++++++++++++++ .../cloudflare/support/executor-object.ts | 109 +++++++++ packages/workflow/tests/cloudflare/worker.ts | 1 + .../workflow/tests/cloudflare/wrangler.jsonc | 5 +- packages/workflow/tsconfig.cloudflare.json | 2 +- 15 files changed, 1174 insertions(+), 3 deletions(-) create mode 100644 packages/workflow/cloudflare.ts create mode 100644 packages/workflow/src/cloudflare/acquisition.ts create mode 100644 packages/workflow/src/cloudflare/admission.ts create mode 100644 packages/workflow/src/cloudflare/commands.ts create mode 100644 packages/workflow/src/cloudflare/owner.ts create mode 100644 packages/workflow/src/cloudflare/release.ts create mode 100644 packages/workflow/src/cloudflare/routing.ts create mode 100644 packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts create mode 100644 packages/workflow/tests/cloudflare/support/executor-object.ts diff --git a/deno.json b/deno.json index 32374cba5..70563ebed 100644 --- a/deno.json +++ b/deno.json @@ -14,7 +14,8 @@ "packages/workflow/src/cloudflare", "packages/workflow/tests/cloudflare", "vitest.config.ts", - "packages/workflow/tsconfig.cloudflare.json" + "packages/workflow/tsconfig.cloudflare.json", + "packages/workflow/cloudflare.ts" ], "nodeModulesDir": "auto", "lock": { diff --git a/packages/workflow/cloudflare.ts b/packages/workflow/cloudflare.ts new file mode 100644 index 000000000..270aef5c8 --- /dev/null +++ b/packages/workflow/cloudflare.ts @@ -0,0 +1,57 @@ +/** + * @module + * + * The Cloudflare host's workflow-run owner. + * + * Keeping this behind its own entrypoint is what lets the shared package stay + * provider-neutral, exactly as `./deno` does for the local host. Durable + * Objects, the runtime's SQLite, WebSocket acquisition and OIDC admission live + * here and nowhere above; `@executablemd/workflow` names none of them, so the + * Deno host is unaffected by this module existing and neither host has to know + * the other does. + * + * What an operator assembles is the owner and its policy: + * + * ```ts + * import { WorkflowOwnerObject } from "@executablemd/workflow/cloudflare"; + * + * export class WorkflowOwner extends WorkflowOwnerObject { + * protected configuration() { + * return { policy: POLICY }; + * } + * protected perform(socket, runId, command) { + * // … + * } + * } + * ``` + * + * Provider endpoints, OIDC tokens, credentials, private message shapes, + * storage handles and acquisition evidence are deliberately absent from what + * this publishes. They are host closure state, and a value a document or a + * runner could name would be authority a document or a runner could hold. + */ + +export { WorkflowOwnerObject, refusalOf } from "./src/cloudflare/owner.ts"; +export type { AdmissionRequest, OwnerConfiguration } from "./src/cloudflare/owner.ts"; + +export { AdmissionError } from "./src/cloudflare/admission.ts"; +export type { AdmissionPolicy, AdmissionRefusal } from "./src/cloudflare/admission.ts"; + +export { ReleaseIdentityError } from "./src/cloudflare/release.ts"; +export type { ReleaseRefusal } from "./src/cloudflare/release.ts"; + +export { admitRunId, ownerFor, RunIdError } from "./src/cloudflare/routing.ts"; +export type { OwnerNamespace, RunIdRefusal } from "./src/cloudflare/routing.ts"; + +export { + AcquisitionError, + acquisitionHolders, + EXECUTOR_TAG, +} from "./src/cloudflare/acquisition.ts"; +export type { AcquisitionAttachment, AcquisitionRefusal } from "./src/cloudflare/acquisition.ts"; + +export { CommandError } from "./src/cloudflare/commands.ts"; +export type { CommandRefusal, CommandResult, RunnerCommand } from "./src/cloudflare/commands.ts"; + +export { WorkflowObjectStorageError } from "./src/cloudflare/recognition.ts"; +export type { RecognitionFailure } from "./src/cloudflare/recognition.ts"; diff --git a/packages/workflow/package.json b/packages/workflow/package.json index 5c2043bde..e7ffd74a5 100644 --- a/packages/workflow/package.json +++ b/packages/workflow/package.json @@ -6,6 +6,7 @@ "exports": { ".": "./mod.ts", "./deno": "./deno.ts", + "./cloudflare": "./cloudflare.ts", "./software-factory": "./software-factory.ts", "./credential-helper": "./src/deno/composition/credential-helper.ts" }, diff --git a/packages/workflow/src/cloudflare/acquisition.ts b/packages/workflow/src/cloudflare/acquisition.ts new file mode 100644 index 000000000..38601fd17 --- /dev/null +++ b/packages/workflow/src/cloudflare/acquisition.ts @@ -0,0 +1,163 @@ +/** + * Executor ownership, as one authenticated WebSocket. + * + * The acquisition *is* the connection. There is no lease, expiry, renewal, + * heartbeat, alarm, PID or liveness poll: a healthy socket owns the run, and a + * socket that closes stops owning it because the runtime stops listing it. That + * is the same shape the local host has, where the operating system releases an + * advisory lock when the executor exits, and it is why nothing here has to + * decide whether an absent executor is slow or gone. + * + * Hibernation is why ownership cannot live in a field. An idle Durable Object + * is evicted while its sockets stay open, so the object that wakes up has no + * memory of what it admitted. The runtime hands back the live sockets and the + * bounded attachment each was accepted with, and that pair is the authority: + * `ctx.getWebSockets()` says which sockets are real, and the attachment says + * what one was admitted as. + * + * Attachment bytes alone are not authority. A copy of them proves nothing, + * because the check is not "does this value look right" but "is the socket this + * message arrived on the one live socket carrying an acquisition". A second + * connection cannot manufacture that by holding a copy. + */ + +import type { OwnerStorage } from "./storage.ts"; + +/** What one admitted connection carries, and all it carries. */ +export interface AcquisitionAttachment { + readonly kind: "executor"; + readonly runId: string; + readonly acquisitionId: string; +} + +/** Why an acquisition was refused. */ +export type AcquisitionRefusal = + | "already-running" + | "not-acquired" + | "foreign-connection" + | "wrong-run"; + +export class AcquisitionError extends Error { + override name = "AcquisitionError"; + + constructor(readonly refusal: AcquisitionRefusal) { + super(`this connection does not own this run's executor (${refusal})`); + } +} + +/** The bits of a Durable Object's context this module uses. */ +export interface AcquisitionContext { + getWebSockets(tag?: string): WebSocket[]; + acceptWebSocket(socket: WebSocket, tags?: string[]): void; + readonly storage: OwnerStorage; +} + +/** The tag every executor connection is accepted under. */ +export const EXECUTOR_TAG = "executor"; + +function attachmentOf(socket: WebSocket): AcquisitionAttachment | undefined { + const value = socket.deserializeAttachment(); + if (value === null || typeof value !== "object" || Array.isArray(value)) { + return undefined; + } + const members = value as Record; + if (members["kind"] !== "executor") { + return undefined; + } + const runId = members["runId"]; + const acquisitionId = members["acquisitionId"]; + if (typeof runId !== "string" || typeof acquisitionId !== "string") { + return undefined; + } + return { kind: "executor", runId, acquisitionId }; +} + +/** + * Every live connection currently holding an acquisition. + * + * A socket the runtime still lists but whose attachment was cleared is not one: + * closing releases ownership immediately, while the runtime may take its own + * time to stop listing the socket, and ownership must end at the earlier of the + * two. + */ +export function acquisitionHolders( + ctx: AcquisitionContext, +): { socket: WebSocket; held: AcquisitionAttachment }[] { + const found: { socket: WebSocket; held: AcquisitionAttachment }[] = []; + for (const socket of ctx.getWebSockets(EXECUTOR_TAG)) { + const held = attachmentOf(socket); + if (held !== undefined) { + found.push({ socket, held }); + } + } + return found; +} + +/** + * Admit one connection as this run's executor. + * + * A second healthy executor is refused rather than followed: it cannot advance + * the run, and the caller learns that from the refusal rather than from a + * mutation that quietly did nothing. + */ +export function acquireExecutor( + ctx: AcquisitionContext, + socket: WebSocket, + runId: string, + acquisitionId: string, +): AcquisitionAttachment { + if (acquisitionHolders(ctx).length > 0) { + throw new AcquisitionError("already-running"); + } + const attachment: AcquisitionAttachment = { kind: "executor", runId, acquisitionId }; + ctx.acceptWebSocket(socket, [EXECUTOR_TAG]); + // Bounded, and only what admission needs to be reconstructed after an + // eviction. Nothing here is a credential and nothing here is durable run + // state. + socket.serializeAttachment(attachment); + return attachment; +} + +/** + * Prove that a message arrived on the one live acquisition. + * + * Called before the requested mutation is parsed, and again — by the caller — + * inside the transaction that writes, because a socket can close between the + * two and the transaction is where the run actually changes. + */ +export function requireAcquisition( + ctx: AcquisitionContext, + socket: WebSocket, + runId: string, +): AcquisitionAttachment { + const live = acquisitionHolders(ctx); + if (live.length === 0) { + throw new AcquisitionError("not-acquired"); + } + const mine = live.find((holder) => holder.socket === socket); + if (mine === undefined) { + // Either this socket was never admitted, or it was superseded and closed. + throw new AcquisitionError("foreign-connection"); + } + if (live.length > 1) { + // Two live holders is a state this module refuses to choose between. + throw new AcquisitionError("already-running"); + } + if (mine.held.runId !== runId) { + throw new AcquisitionError("wrong-run"); + } + return mine.held; +} + +/** + * Release ownership when a connection ends. + * + * The runtime has already stopped listing the socket by the time this runs, so + * there is nothing to revoke — this exists to make the absence of a rollback + * explicit. A closed connection invalidates the acquisition and changes no + * committed state, and it settles no lifecycle: an executor that disappeared + * did not decide anything. + */ +export function releaseExecutor(socket: WebSocket): void { + socket.serializeAttachment(null); +} diff --git a/packages/workflow/src/cloudflare/admission.ts b/packages/workflow/src/cloudflare/admission.ts new file mode 100644 index 000000000..17a247f33 --- /dev/null +++ b/packages/workflow/src/cloudflare/admission.ts @@ -0,0 +1,121 @@ +/** + * Who is allowed to become this run's executor. + * + * A runner authenticates with a GitHub Actions OIDC token, and the owner + * validates it before the connection is accepted and before an acquisition + * exists. Everything checked here is an identity the deployment configured, and + * the checks are on IDs rather than names: a repository can be renamed and an + * owner can be renamed, so a check on `repository` would admit whoever holds + * the name today. + * + * Nothing about the token survives the check. The raw JWT, the JWKS endpoint, + * the claims this contract does not name, and the reason a signature failed are + * all provider state: none of them reaches durable storage, a journal event, a + * public value or an error message. What a refusal says is which category it + * fell into, because that is what an operator can act on and what a test can + * assert without pinning provider wording. + */ + +/** What a deployment must state before any runner can be admitted. */ +export interface AdmissionPolicy { + readonly issuer: string; + readonly audience: string; + readonly repositoryId: string; + readonly repositoryOwnerId: string; + readonly eventName: string; + readonly workflowRef: string; + readonly workflowSha: string; + /** The immutable identity of the workflow allowed to execute this run. */ + readonly jobWorkflowRef: string; + /** The exact build both sides must be. */ + readonly release: string; +} + +/** + * The claims this contract reads. + * + * Deliberately a closed set. A token carries far more than this, and reading a + * claim here is what makes it part of the contract — so anything not named is + * not consulted, cannot be depended on, and never leaves the verifier. + */ +export interface ActionsClaims { + readonly iss: unknown; + readonly aud: unknown; + readonly repository_id: unknown; + readonly repository_owner_id: unknown; + readonly event_name: unknown; + readonly workflow_ref: unknown; + readonly workflow_sha: unknown; + readonly job_workflow_ref: unknown; +} + +/** Which part of the admission a token failed. */ +export type AdmissionRefusal = + | "token-absent" + | "token-malformed" + | "issuer" + | "audience" + | "repository-id" + | "repository-owner-id" + | "event-name" + | "workflow-ref" + | "workflow-sha" + | "workflow-identity"; + +export class AdmissionError extends Error { + override name = "AdmissionError"; + + constructor(readonly refusal: AdmissionRefusal) { + super(`this runner is not admitted to execute this run (${refusal})`); + } +} + +/** Compare one claim, naming the check rather than the values. */ +function requireClaim(claim: unknown, expected: string, refusal: AdmissionRefusal): void { + if (typeof claim !== "string" || claim !== expected) { + throw new AdmissionError(refusal); + } +} + +/** + * Hold verified claims to the configured policy. + * + * Takes claims a verifier already authenticated rather than a token, so this + * module owns *which* claims decide and nothing about how a signature is + * checked. A caller that has not verified a signature has not admitted + * anything, whatever this returns. + */ +export function admitClaims(policy: AdmissionPolicy, claims: ActionsClaims): void { + requireClaim(claims.iss, policy.issuer, "issuer"); + // `aud` may be a string or an array of them; only the exact configured + // audience admits, and an array containing it is that audience. + const audience = claims.aud; + const audiences = Array.isArray(audience) ? audience : [audience]; + if (!audiences.some((value) => value === policy.audience)) { + throw new AdmissionError("audience"); + } + requireClaim(claims.repository_id, policy.repositoryId, "repository-id"); + requireClaim(claims.repository_owner_id, policy.repositoryOwnerId, "repository-owner-id"); + requireClaim(claims.event_name, policy.eventName, "event-name"); + requireClaim(claims.workflow_ref, policy.workflowRef, "workflow-ref"); + requireClaim(claims.workflow_sha, policy.workflowSha, "workflow-sha"); + requireClaim(claims.job_workflow_ref, policy.jobWorkflowRef, "workflow-identity"); +} + +/** Read a claim set out of a payload nothing has inspected yet. */ +export function parseClaims(payload: unknown): ActionsClaims { + if (payload === null || typeof payload !== "object" || Array.isArray(payload)) { + throw new AdmissionError("token-malformed"); + } + const members = payload as Record; + return { + iss: members["iss"], + aud: members["aud"], + repository_id: members["repository_id"], + repository_owner_id: members["repository_owner_id"], + event_name: members["event_name"], + workflow_ref: members["workflow_ref"], + workflow_sha: members["workflow_sha"], + job_workflow_ref: members["job_workflow_ref"], + }; +} diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts new file mode 100644 index 000000000..154e9acbd --- /dev/null +++ b/packages/workflow/src/cloudflare/commands.ts @@ -0,0 +1,215 @@ +/** + * What a runner asks its owner, and what comes back. + * + * These records are private to one software-factory release. They are not + * journaled, exported, authored, or supported across independently versioned + * builds — admission has already proved both sides are the same build, which is + * what a wire contract would otherwise be for. Their decomposition is + * implementation detail and may change with the release that contains it. + * + * What is not private is the parsing discipline. Everything arriving from the + * connection is parsed strictly before it reaches storage: an unknown command, + * an unknown member, a value of the wrong kind and a value past its bound are + * each refused whole, and nothing is partially adopted. A permissive read here + * would be a runner deciding what the owner does. + */ + +/** The commands a runner may send. */ +export type CommandName = "frontier" | "materialize" | "commit" | "settle"; + +/** Why a message was refused before it reached the run. */ +export type CommandRefusal = + | "not-an-object" + | "unknown-command" + | "unknown-member" + | "malformed-member" + | "duplicate-conflict" + | "too-large"; + +export class CommandError extends Error { + override name = "CommandError"; + + constructor(readonly refusal: CommandRefusal) { + // The message a runner sent is not repeated: it arrived from outside and a + // member name can carry as much as a member value. + super(`this owner refused a runner command (${refusal})`); + } +} + +/** The envelope every command shares. */ +export interface CommandEnvelope { + /** Distinguishes one request from a retry of the same request. */ + readonly id: string; + readonly command: CommandName; +} + +/** Read the committed run record and current Workspace root. */ +export interface FrontierCommand extends CommandEnvelope { + readonly command: "frontier"; +} + +/** Ask for the content-addressed bytes of one retained root. */ +export interface MaterializeCommand extends CommandEnvelope { + readonly command: "materialize"; + readonly workspaceRootId: string; +} + +/** One closed mutation intent, submitted once, applied atomically or not at all. */ +export interface CommitCommand extends CommandEnvelope { + readonly command: "commit"; + /** The root the runner started from; the owner refuses if it has moved. */ + readonly expectedWorkspaceRootId: string; + /** The journal frontier the runner read; the owner refuses if it has moved. */ + readonly expectedJournalEventId: string | null; + /** Content-addressed additions, each named by its own digest. */ + readonly content: readonly ContentChunk[]; + /** The canonical root the runner proposes, recomputed by the owner. */ + readonly proposedWorkspaceRootId: string; + /** Already-filtered journal events to append, in order. */ + readonly events: readonly string[]; +} + +/** Publish the run's resulting status. */ +export interface SettleCommand extends CommandEnvelope { + readonly command: "settle"; + readonly status: string; +} + +export interface ContentChunk { + readonly digest: string; + /** Base64, because a private transport still carries text. */ + readonly bytes: string; +} + +export type RunnerCommand = FrontierCommand | MaterializeCommand | CommitCommand | SettleCommand; + +/** The largest message this owner reads at all. */ +const MAX_MESSAGE = 8 * 1024 * 1024; + +/** The most chunks one commit may carry. */ +const MAX_CHUNKS = 4096; + +const ENVELOPE = ["id", "command"] as const; + +const MEMBERS: Record = { + frontier: [...ENVELOPE], + materialize: [...ENVELOPE, "workspaceRootId"], + commit: [ + ...ENVELOPE, + "expectedWorkspaceRootId", + "expectedJournalEventId", + "content", + "proposedWorkspaceRootId", + "events", + ], + settle: [...ENVELOPE, "status"], +}; + +function object(value: unknown): Record { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new CommandError("not-an-object"); + } + return value as Record; +} + +function text(members: Record, key: string): string { + const value = members[key]; + if (typeof value !== "string" || value === "") { + throw new CommandError("malformed-member"); + } + return value; +} + +function closed(members: Record, allowed: readonly string[]): void { + for (const key of Object.keys(members)) { + if (!allowed.includes(key)) { + throw new CommandError("unknown-member"); + } + } +} + +function chunks(value: unknown): ContentChunk[] { + if (!Array.isArray(value)) { + throw new CommandError("malformed-member"); + } + if (value.length > MAX_CHUNKS) { + throw new CommandError("too-large"); + } + return value.map((entry) => { + const members = object(entry); + closed(members, ["digest", "bytes"]); + return { digest: text(members, "digest"), bytes: text(members, "bytes") }; + }); +} + +function events(value: unknown): string[] { + if (!Array.isArray(value)) { + throw new CommandError("malformed-member"); + } + return value.map((entry) => { + if (typeof entry !== "string" || entry === "") { + throw new CommandError("malformed-member"); + } + return entry; + }); +} + +/** Read one command out of a message nothing has inspected yet. */ +export function parseCommand(raw: string): RunnerCommand { + if (raw.length > MAX_MESSAGE) { + throw new CommandError("too-large"); + } + let decoded: unknown; + try { + decoded = JSON.parse(raw); + } catch { + throw new CommandError("not-an-object"); + } + const members = object(decoded); + const id = text(members, "id"); + const command = members["command"]; + if ( + command !== "frontier" && + command !== "materialize" && + command !== "commit" && + command !== "settle" + ) { + throw new CommandError("unknown-command"); + } + closed(members, MEMBERS[command]); + + if (command === "frontier") { + return { id, command }; + } + if (command === "materialize") { + return { id, command, workspaceRootId: text(members, "workspaceRootId") }; + } + if (command === "settle") { + return { id, command, status: text(members, "status") }; + } + const expectedJournalEventId = members["expectedJournalEventId"]; + if (expectedJournalEventId !== null && typeof expectedJournalEventId !== "string") { + throw new CommandError("malformed-member"); + } + return { + id, + command, + expectedWorkspaceRootId: text(members, "expectedWorkspaceRootId"), + expectedJournalEventId, + content: chunks(members["content"]), + proposedWorkspaceRootId: text(members, "proposedWorkspaceRootId"), + events: events(members["events"]), + }; +} + +/** + * What the owner answers with. + * + * A serialized record rather than an Effection `Result`: this crosses a + * connection, and an `Error` does not survive that. The discriminant is + * `outcome` for the same reason — there is no in-process result being modelled + * here, only what one side told the other. + */ +export type CommandResult = + | { readonly id: string; readonly outcome: "performed"; readonly value: unknown } + | { readonly id: string; readonly outcome: "refused"; readonly refusal: string }; diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts new file mode 100644 index 000000000..e42787a93 --- /dev/null +++ b/packages/workflow/src/cloudflare/owner.ts @@ -0,0 +1,162 @@ +/** + * The Durable Object that owns one workflow run. + * + * One run, one object, selected arithmetically from the public run ID. It holds + * the WorkflowRun record and its filtered journal, the immutable Workspace roots + * and their content, and executor ownership — and it holds them in one embedded + * SQLite database, because a second store would be a second thing to keep in + * agreement with the first. + * + * What it does *not* do is as much of the contract as what it does. It runs no + * native client: no Git, no evidence process, no Agent. Those live on the + * ephemeral runner against disposable materialization, and what crosses the + * connection is a proposal this object validates and publishes. The runner + * performs; the owner decides. + * + * Three planes reach it and only one of them can advance a run. The executor + * plane is one authenticated WebSocket whose lifetime is the acquisition. + * Delivery and inspection arrive over ordinary requests, take no acquisition, + * and cannot move the lifecycle — which is why they are separate methods here + * rather than commands on the socket. + */ + +import { DurableObject } from "cloudflare:workers"; +import { + acquireExecutor, + type AcquisitionAttachment, + AcquisitionError, + releaseExecutor, + requireAcquisition, +} from "./acquisition.ts"; +import { admitClaims, type AdmissionPolicy, AdmissionError, parseClaims } from "./admission.ts"; +import { CommandError, type CommandResult, parseCommand, type RunnerCommand } from "./commands.ts"; +import { + declaredObjects, + initializeObject, + isPristine, + recognizeObject, + WorkflowObjectStorageError, +} from "./recognition.ts"; +import { ReleaseIdentityError, requireSameRelease } from "./release.ts"; +import { admitRunId, RunIdError } from "./routing.ts"; +import type { OwnerStorage } from "./storage.ts"; + +/** What one admission presents. */ +export interface AdmissionRequest { + readonly runId: unknown; + readonly release: unknown; + /** Claims a verifier has already authenticated. */ + readonly claims: unknown; +} + +/** Everything a deployment must state before this object admits anybody. */ +export interface OwnerConfiguration { + readonly policy: AdmissionPolicy; +} + +/** Name a refusal without repeating what caused it. */ +export function refusalOf(error: unknown): string { + if (error instanceof AcquisitionError) { + return `acquisition:${error.refusal}`; + } + if (error instanceof AdmissionError) { + return `admission:${error.refusal}`; + } + if (error instanceof ReleaseIdentityError) { + return `release:${error.refusal}`; + } + if (error instanceof RunIdError) { + return `run-id:${error.refusal}`; + } + if (error instanceof CommandError) { + return `command:${error.refusal}`; + } + if (error instanceof WorkflowObjectStorageError) { + return `storage:${error.failure.kind}`; + } + return "internal"; +} + +/** + * The owner, minus the deployment's own configuration. + * + * Subclassed rather than configured through a binding because the policy is + * trusted host state: a value a request could supply would be a runner naming + * the identities it must satisfy. + */ +export abstract class WorkflowOwnerObject extends DurableObject { + protected abstract configuration(): OwnerConfiguration; + + /** This object's storage, as the shared modules expect to see it. */ + protected get owned(): OwnerStorage { + return this.ctx.storage; + } + + /** + * Admit one executor connection. + * + * The order is the contract: the build is compared before the token is read, + * the token before the run is touched, and the acquisition is taken last. A + * refusal at any step leaves no acquisition and no object state — which is + * what makes "a mismatched build cannot reach run state" a fact about the + * code rather than a hope about it. + */ + admit( + request: AdmissionRequest, + socket: WebSocket, + acquisitionId: string, + ): AcquisitionAttachment { + const { policy } = this.configuration(); + requireSameRelease(policy.release, request.release); + admitClaims(policy, parseClaims(request.claims)); + const runId = admitRunId(request.runId); + return acquireExecutor(this.ctx, socket, runId, acquisitionId); + } + + /** + * Handle one message from an admitted connection. + * + * Acquisition is proved before the message is parsed, so a superseded or + * foreign socket never reaches the command reader — and proved again by + * whatever writes, inside the transaction that writes. + */ + onRunnerMessage(socket: WebSocket, runId: string, raw: string): CommandResult { + let command: RunnerCommand | undefined; + try { + requireAcquisition(this.ctx, socket, runId); + command = parseCommand(raw); + return { id: command.id, outcome: "performed", value: this.perform(socket, runId, command) }; + } catch (error) { + return { id: command?.id ?? "", outcome: "refused", refusal: refusalOf(error) }; + } + } + + /** What each command does. Subclasses supply the behavior this owner has. */ + protected abstract perform(socket: WebSocket, runId: string, command: RunnerCommand): unknown; + + /** A connection that ended owns nothing, and rolled nothing back. */ + webSocketClose(socket: WebSocket): void { + releaseExecutor(socket); + } + + webSocketError(socket: WebSocket): void { + releaseExecutor(socket); + } + + /** + * Create this run's storage, or recognize what is already there. + * + * Pristine is asked first rather than inferred from a refusal: storage that + * holds nothing is the only storage this build may write into, and every + * other state — foreign, damaged, a version this build does not implement — + * is recognition's to refuse rather than initialization's to overwrite. + */ + open(runId: string, initializeRun: () => void): void { + admitRunId(runId); + if (isPristine(declaredObjects(this.owned))) { + initializeObject(this.owned, initializeRun); + return; + } + recognizeObject(this.owned); + } +} diff --git a/packages/workflow/src/cloudflare/release.ts b/packages/workflow/src/cloudflare/release.ts new file mode 100644 index 000000000..7397eb3b6 --- /dev/null +++ b/packages/workflow/src/cloudflare/release.ts @@ -0,0 +1,62 @@ +/** + * Which build is allowed to talk to which owner. + * + * The runner client and the Durable Object owner ship as one software-factory + * release, so the messages between them are not a compatibility boundary and + * carry no version negotiation. What replaces one is this: admission compares + * an exact immutable fingerprint the deployment supplied on both sides, and a + * mismatch refuses closed — before any private message is parsed, before an + * acquisition exists, and before any run state is read. + * + * Two builds disagreeing about what was committed is the failure this exists to + * prevent rather than to survive, so there is no downgrade path and nothing + * adapts. + */ + +/** Why a build was not admitted. */ +export type ReleaseRefusal = "release-absent" | "release-malformed" | "release-mismatch"; + +export class ReleaseIdentityError extends Error { + override name = "ReleaseIdentityError"; + + constructor(readonly refusal: ReleaseRefusal) { + // The configured and presented fingerprints are deployment facts, and a + // refusal that printed them would put them in every log that saw one. + super(`this runner build is not admitted by this owner (${refusal})`); + } +} + +/** + * A fingerprint is opaque, non-empty and bounded. + * + * Bounded because it arrives from outside admission and is compared before + * anything else has looked at it; opaque because what a deployment derives it + * from — a commit, a container digest, a build id — is the deployment's + * business and never this module's. + */ +const FINGERPRINT = /^[A-Za-z0-9._:-]{1,200}$/; + +export function admitReleaseFingerprint(value: unknown): string { + if (typeof value !== "string" || value === "") { + throw new ReleaseIdentityError("release-absent"); + } + if (!FINGERPRINT.test(value)) { + throw new ReleaseIdentityError("release-malformed"); + } + return value; +} + +/** + * Compare a presented fingerprint with the configured one. + * + * Exactness rather than secrecy is the point: a fingerprint proves nothing by + * itself, and this is the one check that stops a build the owner never agreed + * to from parsing a private message. + */ +export function requireSameRelease(configured: string, presented: unknown): string { + const admitted = admitReleaseFingerprint(presented); + if (admitted !== configured) { + throw new ReleaseIdentityError("release-mismatch"); + } + return admitted; +} diff --git a/packages/workflow/src/cloudflare/routing.ts b/packages/workflow/src/cloudflare/routing.ts new file mode 100644 index 000000000..d9b59053e --- /dev/null +++ b/packages/workflow/src/cloudflare/routing.ts @@ -0,0 +1,68 @@ +/** + * Which Durable Object owns one run. + * + * The public run ID selects it arithmetically, through the namespace's own + * `idFromName`. There is no registry, no lookup table and nothing to keep in + * agreement with the objects themselves: a second authority that could disagree + * with the arithmetic is exactly what "one issue, one run, one owner" cannot + * have. + * + * The id is admitted before it is used. A malformed one must not reach + * `idFromName` at all — that call answers with an object for any string, so a + * mistyped id would silently address a fresh, empty owner rather than fail. + */ + +/** What a run ID has to be to address an owner. */ +export type RunIdRefusal = "run-id-absent" | "run-id-empty" | "run-id-has-nul" | "run-id-too-long"; + +export class RunIdError extends Error { + override name = "RunIdError"; + + constructor(readonly refusal: RunIdRefusal) { + super(`this run id cannot address a workflow owner (${refusal})`); + } +} + +/** + * The longest run ID this host routes. + * + * Public run IDs are opaque and caller-selectable, so a bound belongs here + * rather than in the derivation: the factory's own is 52 characters, and this + * leaves room for an authorized caller's without letting an unbounded string + * reach the runtime. + */ +const MAX_RUN_ID = 512; + +/** Hold a run ID to what storage requires of one, changing nothing about it. */ +export function admitRunId(value: unknown): string { + if (typeof value !== "string") { + throw new RunIdError("run-id-absent"); + } + if (value === "") { + throw new RunIdError("run-id-empty"); + } + if (value.includes("\0")) { + throw new RunIdError("run-id-has-nul"); + } + if (value.length > MAX_RUN_ID) { + throw new RunIdError("run-id-too-long"); + } + return value; +} + +/** The one namespace operation this host routes through. */ +export interface OwnerNamespace { + idFromName(name: string): { toString(): string }; + get(id: { toString(): string }): Stub; +} + +/** + * The owner for one run. + * + * Deterministic in the run ID and in nothing else: the same id reaches the same + * object from any worker, on any request, without either side having recorded + * where it went. + */ +export function ownerFor(namespace: OwnerNamespace, runId: unknown): Stub { + return namespace.get(namespace.idFromName(admitRunId(runId))); +} diff --git a/packages/workflow/tests/cloudflare/env.d.ts b/packages/workflow/tests/cloudflare/env.d.ts index 8697c38fd..e0dfafaa6 100644 --- a/packages/workflow/tests/cloudflare/env.d.ts +++ b/packages/workflow/tests/cloudflare/env.d.ts @@ -1,3 +1,4 @@ +import type { ExecutorObject } from "./support/executor-object.ts"; import type { OwnerObject } from "./support/owner-object.ts"; import type { StorageProbeObject } from "./support/probe-object.ts"; @@ -6,6 +7,7 @@ declare global { interface Env { STORAGE_PROBE: DurableObjectNamespace; OWNER: DurableObjectNamespace; + EXECUTOR: DurableObjectNamespace; } } } diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts new file mode 100644 index 000000000..ea8eb9150 --- /dev/null +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -0,0 +1,206 @@ +/** + * Who may execute a run, on real workerd. + * + * Admission order and acquisition lifetime are the two things this suite is + * about. The order matters because a mismatched build must not reach a token + * and a bad token must not reach run state; the lifetime matters because the + * connection *is* the acquisition, with no lease to expire and no heartbeat to + * miss, so the only proof that ownership ended is that the socket did. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { describe, expect, it } from "vitest"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, VALID_CLAIMS } from "./support/executor-object.ts"; + +let unique = 0; + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`executor-${unique}-${Math.random()}`)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise> { + return runInDurableObject(stub, body) as Promise>; +} + +const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + +describe("admitting an executor", () => { + it("admits a matching build with authenticated claims", async () => { + const stub = executor(); + expect(await on(stub, (o) => o.admitConnection({}))).toBe("admitted"); + expect(await on(stub, (o) => o.holders())).toBe(1); + }); + + it("refuses a build the owner did not agree to, before reading the token", async () => { + const stub = executor(); + // The claims are deliberately unusable. If the release were checked after + // them, the refusal would name the token rather than the build. + expect(await on(stub, (o) => o.admitConnection({ release: "other-build", claims: null }))).toBe( + "release:release-mismatch", + ); + expect(await on(stub, (o) => o.holders())).toBe(0); + }); + + it("refuses an absent or malformed build identity", async () => { + const stub = executor(); + expect(await on(stub, (o) => o.admitConnection({ release: undefined }))).toBe( + "release:release-absent", + ); + expect(await on(stub, (o) => o.admitConnection({ release: "not a fingerprint" }))).toBe( + "release:release-malformed", + ); + expect(await on(stub, (o) => o.holders())).toBe(0); + }); + + it("refuses every claim the policy names, one at a time", async () => { + const cases: [string, Record][] = [ + ["admission:issuer", { iss: "https://evil.example" }], + ["admission:audience", { aud: "https://somebody-else" }], + ["admission:repository-id", { repository_id: "999" }], + ["admission:repository-owner-id", { repository_owner_id: "999" }], + ["admission:event-name", { event_name: "push" }], + [ + "admission:workflow-ref", + { + workflow_ref: "octo/repo/.github/workflows/other.yml@refs/heads/main", + }, + ], + [ + "admission:workflow-sha", + { + workflow_sha: "1111111111111111111111111111111111111111", + }, + ], + [ + "admission:workflow-identity", + { + job_workflow_ref: "octo/repo/.github/workflows/other.yml@refs/heads/main", + }, + ], + ]; + for (const [expected, overrides] of cases) { + const stub = executor(); + const claims = { ...VALID_CLAIMS, ...overrides }; + expect(await on(stub, (o) => o.admitConnection({ claims }))).toBe(expected); + expect(await on(stub, (o) => o.holders())).toBe(0); + } + }); + + it("accepts an audience array containing the configured one", async () => { + const stub = executor(); + const claims = { ...VALID_CLAIMS, aud: ["https://other", POLICY.audience] }; + expect(await on(stub, (o) => o.admitConnection({ claims }))).toBe("admitted"); + }); + + it("refuses a token that is not a claim set at all", async () => { + const stub = executor(); + expect(await on(stub, (o) => o.admitConnection({ claims: "a string" }))).toBe( + "admission:token-malformed", + ); + }); + + it("refuses a run id that could not address an owner", async () => { + const stub = executor(); + expect(await on(stub, (o) => o.admitConnection({ runId: "" }))).toBe("run-id:run-id-empty"); + expect(await on(stub, (o) => o.admitConnection({ runId: 42 }))).toBe("run-id:run-id-absent"); + expect(await on(stub, (o) => o.holders())).toBe(0); + }); +}); + +describe("holding an acquisition", () => { + it("refuses a second healthy executor rather than following it", async () => { + const stub = executor(); + expect(await on(stub, (o) => o.admitConnection({}))).toBe("admitted"); + expect(await on(stub, (o) => o.admitConnection({}))).toBe("acquisition:already-running"); + expect(await on(stub, (o) => o.holders())).toBe(1); + }); + + it("lets the admitted connection send, and answers what it performed", async () => { + const stub = executor(); + await on(stub, (o) => o.admitConnection({})); + expect( + await on(stub, (o) => o.send(1, JSON.stringify({ id: "1", command: "frontier" }))), + ).toEqual({ id: "1", outcome: "performed", value: { performed: "frontier" } }); + }); + + it("refuses a socket it never admitted", async () => { + const stub = executor(); + await on(stub, (o) => o.admitConnection({})); + expect( + await on(stub, (o) => o.sendAsStranger(JSON.stringify({ id: "1", command: "frontier" }))), + ).toEqual({ id: "", outcome: "refused", refusal: "acquisition:foreign-connection" }); + }); + + it("owns nothing once the connection ends, and rolls nothing back", async () => { + const stub = executor(); + await on(stub, (o) => o.admitConnection({})); + await on(stub, (o) => o.closeConnection(1)); + expect(await on(stub, (o) => o.holders())).toBe(0); + // And the next executor may take it, with no lease having expired. + expect(await on(stub, (o) => o.admitConnection({}))).toBe("admitted"); + }); + + it("proves the acquisition before it reads a command", async () => { + const stub = executor(); + // Nothing is admitted, so even a well-formed command is refused for + // ownership rather than for its shape. + expect( + await on(stub, (o) => o.sendAsStranger(JSON.stringify({ id: "1", command: "frontier" }))), + ).toEqual({ id: "", outcome: "refused", refusal: "acquisition:not-acquired" }); + }); +}); + +describe("reading a runner command", () => { + it("refuses what it cannot read as one", async () => { + const stub = executor(); + await on(stub, (o) => o.admitConnection({})); + const refuse = async (raw: string) => + (await on(stub, (o) => o.send(1, raw))) as { refusal: string }; + expect((await refuse("not json")).refusal).toBe("command:not-an-object"); + expect((await refuse(JSON.stringify([1, 2]))).refusal).toBe("command:not-an-object"); + expect((await refuse(JSON.stringify({ id: "1", command: "explode" }))).refusal).toBe( + "command:unknown-command", + ); + expect((await refuse(JSON.stringify({ id: "1", command: "frontier", extra: 1 }))).refusal).toBe( + "command:unknown-member", + ); + expect((await refuse(JSON.stringify({ command: "frontier" }))).refusal).toBe( + "command:malformed-member", + ); + expect((await refuse(JSON.stringify({ id: "1", command: "materialize" }))).refusal).toBe( + "command:malformed-member", + ); + }); + + it("reads a commit intent whole", async () => { + const stub = executor(); + await on(stub, (o) => o.admitConnection({})); + const raw = JSON.stringify({ + id: "7", + command: "commit", + expectedWorkspaceRootId: "root-a", + expectedJournalEventId: null, + content: [{ digest: "d1", bytes: "AAAA" }], + proposedWorkspaceRootId: "root-b", + events: ["event-1"], + }); + expect(await on(stub, (o) => o.send(1, raw))).toEqual({ + id: "7", + outcome: "performed", + value: { performed: "commit" }, + }); + }); +}); + +describe("routing a run to its owner", () => { + it("reaches one object for one run id, without a registry", () => { + const first = env.EXECUTOR.idFromName(RUN_ID).toString(); + expect(env.EXECUTOR.idFromName(RUN_ID).toString()).toBe(first); + expect(env.EXECUTOR.idFromName(`${RUN_ID}x`).toString()).not.toBe(first); + }); +}); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts new file mode 100644 index 000000000..a268019e8 --- /dev/null +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -0,0 +1,109 @@ +/** + * A concrete owner, so admission and acquisition can be exercised end to end. + * + * It supplies the two things `WorkflowOwnerObject` leaves abstract — a policy + * and a `perform` — and nothing else. `perform` answers with the command it was + * given rather than doing durable work: what these tests are about is who is + * allowed to send one, not what each one means. + */ + +import { acquisitionHolders } from "../../../src/cloudflare/acquisition.ts"; +import { WorkflowOwnerObject } from "../../../src/cloudflare/owner.ts"; +import type { AdmissionRequest, OwnerConfiguration } from "../../../src/cloudflare/owner.ts"; +import type { AdmissionPolicy } from "../../../src/cloudflare/admission.ts"; +import type { RunnerCommand } from "../../../src/cloudflare/commands.ts"; +import { refusalOf } from "../../../src/cloudflare/owner.ts"; + +/** The identities this owner is configured to admit. */ +export const POLICY: AdmissionPolicy = { + issuer: "https://token.actions.githubusercontent.com", + audience: "https://factory.example", + repositoryId: "123456", + repositoryOwnerId: "654321", + eventName: "repository_dispatch", + workflowRef: "octo/repo/.github/workflows/factory.yml@refs/heads/main", + workflowSha: "0f2c9a1b3d4e5f60718293a4b5c6d7e8f9012345", + jobWorkflowRef: "octo/repo/.github/workflows/factory.yml@refs/heads/main", + release: "factory-2026.09.02-abcdef", +}; + +/** Claims a verifier would have authenticated for the policy above. */ +export const VALID_CLAIMS: Record = { + iss: POLICY.issuer, + aud: POLICY.audience, + repository_id: POLICY.repositoryId, + repository_owner_id: POLICY.repositoryOwnerId, + event_name: POLICY.eventName, + workflow_ref: POLICY.workflowRef, + workflow_sha: POLICY.workflowSha, + job_workflow_ref: POLICY.jobWorkflowRef, +}; + +const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + +export class ExecutorObject extends WorkflowOwnerObject { + #acquisitions = 0; + + protected configuration(): OwnerConfiguration { + return { policy: POLICY }; + } + + protected perform(_socket: WebSocket, _runId: string, command: RunnerCommand): unknown { + return { performed: command.command }; + } + + /** + * Admit one connection, answering what happened rather than raising. + * + * The client half of the pair is kept so a later call can drive it; the + * server half is what the object admitted. + */ + admitConnection(request: Partial): string { + const pair = new WebSocketPair(); + const server = pair[1]; + this.#acquisitions += 1; + try { + this.admit( + { + runId: "runId" in request ? request.runId : RUN_ID, + release: "release" in request ? request.release : POLICY.release, + claims: "claims" in request ? request.claims : VALID_CLAIMS, + }, + server, + `acquisition-${this.#acquisitions}`, + ); + return "admitted"; + } catch (error) { + return refusalOf(error); + } + } + + /** How many live connections currently hold this run's executor. */ + holders(): number { + return acquisitionHolders(this.ctx).length; + } + + /** Send one message as the connection admitted at `index` (1-based). */ + send(index: number, raw: string): unknown { + const socket = this.ctx.getWebSockets("executor")[index - 1]; + if (socket === undefined) { + return { id: "", outcome: "refused", refusal: "no-such-connection" }; + } + return this.onRunnerMessage(socket, RUN_ID, raw); + } + + /** Send as a socket this object never admitted. */ + sendAsStranger(raw: string): unknown { + const pair = new WebSocketPair(); + return this.onRunnerMessage(pair[1], RUN_ID, raw); + } + + /** Close the connection admitted at `index`, releasing its acquisition. */ + closeConnection(index: number): void { + const socket = this.ctx.getWebSockets("executor")[index - 1]; + if (socket !== undefined) { + socket.close(1000, "done"); + this.webSocketClose(socket); + } + } +} diff --git a/packages/workflow/tests/cloudflare/worker.ts b/packages/workflow/tests/cloudflare/worker.ts index 4143c1ffd..1da99ee72 100644 --- a/packages/workflow/tests/cloudflare/worker.ts +++ b/packages/workflow/tests/cloudflare/worker.ts @@ -8,6 +8,7 @@ export { StorageProbeObject } from "./support/probe-object.ts"; export { OwnerObject } from "./support/owner-object.ts"; +export { ExecutorObject } from "./support/executor-object.ts"; export default { fetch(): Response { diff --git a/packages/workflow/tests/cloudflare/wrangler.jsonc b/packages/workflow/tests/cloudflare/wrangler.jsonc index 0315d0393..9e7f7877d 100644 --- a/packages/workflow/tests/cloudflare/wrangler.jsonc +++ b/packages/workflow/tests/cloudflare/wrangler.jsonc @@ -7,7 +7,10 @@ "bindings": [ { "name": "STORAGE_PROBE", "class_name": "StorageProbeObject" }, { "name": "OWNER", "class_name": "OwnerObject" }, + { "name": "EXECUTOR", "class_name": "ExecutorObject" }, ], }, - "migrations": [{ "tag": "v1", "new_sqlite_classes": ["StorageProbeObject", "OwnerObject"] }], + "migrations": [ + { "tag": "v1", "new_sqlite_classes": ["StorageProbeObject", "OwnerObject", "ExecutorObject"] }, + ], } diff --git a/packages/workflow/tsconfig.cloudflare.json b/packages/workflow/tsconfig.cloudflare.json index 96d28350a..4e35f34ec 100644 --- a/packages/workflow/tsconfig.cloudflare.json +++ b/packages/workflow/tsconfig.cloudflare.json @@ -15,5 +15,5 @@ "skipLibCheck": true, "verbatimModuleSyntax": true }, - "include": ["src/cloudflare/**/*.ts", "tests/cloudflare/**/*.ts"] + "include": ["cloudflare.ts", "src/cloudflare/**/*.ts", "tests/cloudflare/**/*.ts"] } From fc4841add4ce61fc85396c7080f959b14cceed0c Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 20:08:54 -0400 Subject: [PATCH 17/81] =?UTF-8?q?=F0=9F=A7=BE=20Collect=20a=20remote=20tra?= =?UTF-8?q?nsaction=20and=20submit=20one=20closed=20intent=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `transact()` for a run whose storage is somewhere else, as `698-a-1.md` Q3 settles it. A Durable Object commits synchronously and cannot hold a transaction open across a network wait, so the obvious reading is unavailable. What makes it tractable is that the body does not need the transaction open while it runs: it needs the starting history, somewhere for its writes to go, and all of them landing together. So the callback runs in a runner-owned scope against a collector. The starting frontier is one bounded read that opens and closes its own owner-side read. Appends go into a local buffer that `readAll()` reads back after the starting prefix, so a body sees its own writes. Nothing is sent while the body runs; when the body and everything it started have torn down, one closed intent goes to the owner. The callback is never serialized, interpreted or executed on the owner. That is what makes arbitrary control flow safe — nothing tries to infer what the body did, and only what it *enlisted* travels. `src/remote/collector.ts` rather than `src/cloudflare/**`: this is the client half and it runs on the runner, not in the Worker. It names no host, so the ordinary Deno check and both boundary scans cover it. Nine assertions with a deterministic fake link, because Cloudflare mechanics are not the subject — what the owner does with an intent is proven on workerd, what the client sends is proven here. One intent carries what was enlisted and the frontier it was proposed against. `readAll()` gives read-your-writes. A body may cross a suspension point with no owner transaction open. A body that throws sends nothing and leaves the gate closed. An owner refusal is returned instead of the body's value, which never crossed the connection. A nested transaction and an ordinary same-handle operation inside a body each refuse. A handle used after its body closed refuses. And an event mutated after it was appended commits as it was handed over, because the collector cloned it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/remote/collector.ts | 171 ++++++++++++ .../workflow/tests/remote-transaction.test.ts | 247 ++++++++++++++++++ 2 files changed, 418 insertions(+) create mode 100644 packages/workflow/src/remote/collector.ts create mode 100644 packages/workflow/tests/remote-transaction.test.ts diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts new file mode 100644 index 000000000..41f7f4484 --- /dev/null +++ b/packages/workflow/src/remote/collector.ts @@ -0,0 +1,171 @@ +/** + * `transact()` for a run whose storage is somewhere else. + * + * A Durable Object commits synchronously and cannot hold a transaction open + * across a network wait, so the obvious reading — open a remote transaction, + * run the caller's body, commit — is not available. What is available is that + * the body does not need the transaction to be open while it runs. It needs to + * read the starting history, it needs its writes to go somewhere, and it needs + * all of them to land together or not at all. + * + * So the callback runs here, in a runner-owned scope, against a collector. The + * starting frontier is read once through an ordinary bounded request that opens + * and closes its own read on the owner. Journal appends go into a local buffer + * that `readAll()` reads back after the starting prefix, so the body sees its + * own writes. Nothing is sent while the body is running. When the body and + * everything it started have torn down successfully, one closed intent goes to + * the owner, which revalidates and applies it inside its one transaction. + * + * The callback is never serialized, interpreted, or run inside the owner. It is + * ordinary code doing ordinary work; only what it *enlisted* crosses the + * connection. That is what makes arbitrary control flow safe here — nothing + * tries to infer what the body did. + */ + +import { call, ensure, Ok, type Operation, type Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import type { DurableStream } from "@executablemd/durable-streams"; +import type { WorkflowRunTransaction } from "../storage/api.ts"; + +/** Why a transaction could not be run or committed. */ +export type CollectorRefusal = + | "nested-transaction" + | "transaction-closed" + | "operation-inside-body" + | "too-many-events"; + +export class RemoteTransactionError extends Error { + override name = "RemoteTransactionError"; + + constructor(readonly refusal: CollectorRefusal) { + super(`this remote transaction cannot proceed (${refusal})`); + } +} + +/** The starting state a transaction is proposed against. */ +export interface StartingFrontier { + readonly workspaceRootId: string; + readonly journalEventId: string | null; + readonly events: readonly DurableEvent[]; +} + +/** One closed intent, as the owner will receive it. */ +export interface CommitIntent { + readonly expectedWorkspaceRootId: string; + readonly expectedJournalEventId: string | null; + readonly events: readonly DurableEvent[]; +} + +/** What the collector needs from the connection. */ +export interface OwnerLink { + /** One bounded read that opens and closes its own owner-side read. */ + frontier(): Operation; + /** One closed intent, applied atomically or not at all. */ + commit(intent: CommitIntent): Operation>; +} + +/** The most events one intent may carry. */ +const MAX_EVENTS = 4096; + +/** + * Whether a transaction is open on this handle. + * + * Scope-local rather than global: two runs may transact at once, and what must + * not happen is a second transaction — or an ordinary operation — on the *same* + * handle from inside a body. That is the same refusal the local provider makes, + * and for the same reason: work that never received the transaction handle + * would otherwise commit on its own, outside the unit of work it appears to be + * part of. + */ +export interface TransactionGate { + open: boolean; +} + +export function createTransactionGate(): TransactionGate { + return { open: false }; +} + +/** Refuse an ordinary same-handle operation while a body is running. */ +export function requireNoOpenTransaction(gate: TransactionGate): void { + if (gate.open) { + throw new RemoteTransactionError("operation-inside-body"); + } +} + +/** + * Run `body` against a collector, then submit what it enlisted. + * + * The body may compute, suspend and perform runner-owned effects. None of that + * is reduced to an intent and none of it executes on the owner; only mutations + * made through the transaction handle enter the collector. + */ +export function transactRemotely( + link: OwnerLink, + gate: TransactionGate, + body: (transaction: WorkflowRunTransaction) => Operation, +): Operation> { + return call(function* (): Operation> { + if (gate.open) { + throw new RemoteTransactionError("nested-transaction"); + } + const starting = yield* link.frontier(); + const appended: DurableEvent[] = []; + let live = true; + + const journal: DurableStream = { + *readAll(): Operation { + if (!live) { + throw new RemoteTransactionError("transaction-closed"); + } + // Read-your-writes: the starting prefix, then this transaction's own + // appends, in order. A body that reads back what it just wrote sees it + // even though the owner has not been told yet. + return [...starting.events, ...appended]; + }, + *append(event: DurableEvent): Operation { + if (!live) { + throw new RemoteTransactionError("transaction-closed"); + } + if (appended.length >= MAX_EVENTS) { + throw new RemoteTransactionError("too-many-events"); + } + // Cloned on the way in, so a caller that keeps mutating the value it + // handed over cannot change what this transaction will commit. + appended.push(structuredClone(event)); + }, + }; + + gate.open = true; + let outcome: T; + try { + // Everything the body started tears down before the intent is built. A + // failure or cancellation leaves through here without a commit, which is + // what makes "no commit was sent" the same statement as "the body did not + // finish". + outcome = yield* call(() => body({ journal })); + } finally { + live = false; + gate.open = false; + } + + const committed = yield* link.commit({ + expectedWorkspaceRootId: starting.workspaceRootId, + expectedJournalEventId: starting.journalEventId, + events: appended, + }); + if (!committed.ok) { + return committed; + } + // Only now. `T` is the body's own value and never crossed the connection; + // returning it before the owner committed would be a caller holding a + // result for work that did not happen. + return Ok(outcome); + }); +} + +/** Discard a collector's work without sending it. */ +export function abandon(gate: TransactionGate): Operation { + return ensure(() => { + gate.open = false; + }); +} diff --git a/packages/workflow/tests/remote-transaction.test.ts b/packages/workflow/tests/remote-transaction.test.ts new file mode 100644 index 000000000..ef82a5b63 --- /dev/null +++ b/packages/workflow/tests/remote-transaction.test.ts @@ -0,0 +1,247 @@ +/** + * Tier WRH — `transact()` against an owner somewhere else. + * + * The contract is that arbitrary callback control flow stays legal while the + * commit stays atomic, and the way that is achieved is by never inferring what + * the body did: the body runs locally, and only what it enlisted is sent. So + * these tests are mostly about what does *not* travel — a body that suspends + * leaves no transaction open, a body that fails sends nothing, and a result the + * owner refused is not returned as a success. + * + * The link is a deterministic fake because Cloudflare mechanics are not the + * subject here. What the owner does with an intent is proven on real workerd; + * what the client sends is proven here. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { Err, Ok, sleep, type Operation, type Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { + type CommitIntent, + createTransactionGate, + type OwnerLink, + RemoteTransactionError, + requireNoOpenTransaction, + type StartingFrontier, + transactRemotely, +} from "../src/remote/collector.ts"; + +/** The name a test event carries, read rather than asserted. */ +function nameOf(entry: DurableEvent | undefined): string { + if (entry === undefined || !("description" in entry)) { + return ""; + } + const description = entry.description; + if (description === null || typeof description !== "object") { + return ""; + } + const name = (description as Record)["name"]; + return typeof name === "string" ? name : ""; +} + +/** Rename a test event in place, to prove the collector cloned it. */ +function rename(entry: DurableEvent, name: string): void { + if (!("description" in entry)) { + return; + } + const description = entry.description; + if (description !== null && typeof description === "object") { + (description as Record)["name"] = name; + } +} + +function event(name: string): DurableEvent { + return { + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + } as DurableEvent; +} + +/** A link that records what it was asked, and answers how a test tells it to. */ +function link( + options: { + frontier?: StartingFrontier; + commit?: (intent: CommitIntent) => Result; + } = {}, +) { + const sent: CommitIntent[] = []; + const starting: StartingFrontier = options.frontier ?? { + workspaceRootId: "root-a", + journalEventId: "event-0", + events: [event("already-there")], + }; + const owner: OwnerLink = { + *frontier(): Operation { + return starting; + }, + *commit(intent: CommitIntent): Operation> { + sent.push(intent); + return options.commit === undefined ? Ok(undefined) : options.commit(intent); + }, + }; + return { owner, sent, starting }; +} + +describe("a remote transaction", () => { + it("sends one intent carrying what the body enlisted", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + + const result = yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(event("one")); + yield* transaction.journal.append(event("two")); + return "body value"; + }); + + expect(result.ok).toBe(true); + expect(result.ok && result.value).toBe("body value"); + expect(sent).toHaveLength(1); + expect(sent[0]?.expectedWorkspaceRootId).toBe("root-a"); + expect(sent[0]?.expectedJournalEventId).toBe("event-0"); + expect(sent[0]?.events).toHaveLength(2); + }); + + it("reads the starting prefix and its own appends, in order", function* () { + const { owner } = link(); + const gate = createTransactionGate(); + let seen: string[] = []; + + yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(event("mine")); + const all = yield* transaction.journal.readAll(); + seen = all.map(nameOf); + return undefined; + }); + + expect(seen).toEqual(["already-there", "mine"]); + }); + + it("lets the body cross a suspension point with no owner transaction open", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + + const result = yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(event("before")); + // Nothing is held on the owner while this waits, which is the whole + // reason the body runs here rather than inside a transaction. + yield* sleep(1); + yield* transaction.journal.append(event("after")); + return "crossed"; + }); + + expect(result.ok && result.value).toBe("crossed"); + expect(sent).toHaveLength(1); + expect(sent[0]?.events).toHaveLength(2); + }); + + it("sends nothing when the body fails", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + + let raised: unknown; + try { + yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(event("doomed")); + throw new Error("the body decided otherwise"); + }); + } catch (error) { + raised = error; + } + + expect(String(raised)).toContain("the body decided otherwise"); + expect(sent).toEqual([]); + expect(gate.open).toBe(false); + }); + + it("returns the owner's refusal rather than the body's value", function* () { + const { owner, sent } = link({ commit: () => Err(new Error("stale expected root")) }); + const gate = createTransactionGate(); + + const result = yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(event("hopeful")); + return "never returned"; + }); + + expect(result.ok).toBe(false); + expect(!result.ok && String(result.error)).toContain("stale expected root"); + expect(sent).toHaveLength(1); + }); + + it("refuses a transaction opened inside a transaction", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + + let raised: unknown; + try { + yield* transactRemotely(owner, gate, function* () { + yield* transactRemotely(owner, gate, function* () { + return undefined; + }); + return undefined; + }); + } catch (error) { + raised = error; + } + + expect(raised).toBeInstanceOf(RemoteTransactionError); + expect((raised as RemoteTransactionError).refusal).toBe("nested-transaction"); + expect(sent).toEqual([]); + }); + + it("refuses an ordinary same-handle operation while a body is running", function* () { + const { owner } = link(); + const gate = createTransactionGate(); + let raised: unknown; + + yield* transactRemotely(owner, gate, function* () { + try { + requireNoOpenTransaction(gate); + } catch (error) { + raised = error; + } + return undefined; + }); + + expect(raised).toBeInstanceOf(RemoteTransactionError); + expect((raised as RemoteTransactionError).refusal).toBe("operation-inside-body"); + // And the gate is closed again afterwards, so the next operation is fine. + requireNoOpenTransaction(gate); + }); + + it("refuses a transaction handle used after its body closed", function* () { + const { owner } = link(); + const gate = createTransactionGate(); + let escaped: { journal: { append(event: DurableEvent): Operation } } | undefined; + + yield* transactRemotely(owner, gate, function* (transaction) { + escaped = transaction; + return undefined; + }); + + let raised: unknown; + try { + yield* escaped!.journal.append(event("too late")); + } catch (error) { + raised = error; + } + expect((raised as RemoteTransactionError).refusal).toBe("transaction-closed"); + }); + + it("commits what it was handed, not what the caller mutated afterwards", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + const mutable = event("original"); + + yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(mutable); + rename(mutable, "changed"); + return undefined; + }); + + const committed = sent[0]?.events[0]; + expect(nameOf(committed)).toBe("original"); + }); +}); From bf6358d45ec1cc760c449c6bedc377ca67903ce0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 20:11:34 -0400 Subject: [PATCH 18/81] =?UTF-8?q?=F0=9F=93=A1=20Carry=20a=20runner's=20req?= =?UTF-8?q?uest=20to=20its=20owner=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runner half of the connection. `src/remote/client.ts` holds one connection open for the calling scope and correlates answers by the id they name rather than by arrival order — a socket delivers what the owner sent whenever it sent it, and a client that assumed order would attribute one command's refusal to another. Teardown fails every request still waiting, because a caller blocked on an answer that can never arrive would outlive the connection it asked through. It decides nothing about the run. A refusal comes back as an answer rather than a transport failure: what a command means is the owner's, and a client that interpreted a refusal would be a second place deciding what a run may do. Seven assertions against a socket a test drives by hand. The command reaches the wire carrying its id; two answers returned in the opposite order to the asking still reach the right callers; a refusal is handed back as an answer; an answer that is not JSON and an answer naming nobody are both dropped without disturbing the caller that was waiting; a connection that ends fails the request in flight and refuses the next one; and a second request under an id already in flight is refused rather than silently replacing it. This is client code and it names no host, so it lives beside the collector in `src/remote/` where the ordinary Deno check and both boundary scans cover it. Not included, deliberately: `packages/cli/src/remote-workflow.ts`. The four- method assembly composes client operations — a remote `WorkflowRunDatabase`, lifecycle, delivery and inspection over this connection — and those do not exist yet. A host whose four methods all raise would be the placeholder the plan says not to add, so the assembly waits until there is something real to assemble. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/remote/client.ts | 137 +++++++++++++++ packages/workflow/tests/remote-client.test.ts | 165 ++++++++++++++++++ 2 files changed, 302 insertions(+) create mode 100644 packages/workflow/src/remote/client.ts create mode 100644 packages/workflow/tests/remote-client.test.ts diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts new file mode 100644 index 000000000..669dbbada --- /dev/null +++ b/packages/workflow/src/remote/client.ts @@ -0,0 +1,137 @@ +/** + * The runner's side of the connection to its owner. + * + * One connection, one acquisition, and one request in flight at a time per + * command id. Requests and answers are correlated explicitly rather than by + * arrival order, because a socket delivers what the owner sent whenever it + * sent it, and a client that assumed order would attribute one command's + * refusal to another. + * + * Nothing here decides anything about the run. It carries a question to the + * owner and hands back what the owner said — including a refusal, which is an + * answer rather than a transport failure. What the owner does with a command is + * the owner's, and a client that interpreted a refusal would be a second place + * deciding what a run may do. + */ + +import { createSignal, each, type Operation, resource, spawn, withResolvers } from "effection"; + +/** Why the connection itself could not carry a request. */ +export type LinkRefusal = "closed" | "malformed-answer" | "duplicate-answer"; + +export class OwnerLinkError extends Error { + override name = "OwnerLinkError"; + + constructor(readonly refusal: LinkRefusal) { + super(`the connection to this run's owner cannot carry the request (${refusal})`); + } +} + +/** What the owner answered, as the client reads it. */ +export type OwnerAnswer = + | { readonly outcome: "performed"; readonly value: unknown } + | { readonly outcome: "refused"; readonly refusal: string }; + +/** The socket shape this client needs, so a test can supply one. */ +export interface OwnerSocket { + send(data: string): void; + close(): void; + addEventListener(type: "message", listener: (event: { data: unknown }) => void): void; + addEventListener(type: "close", listener: () => void): void; +} + +/** One live connection to a run's owner. */ +export interface OwnerConnection { + /** Send one command and wait for the answer that names it. */ + ask(id: string, command: Record): Operation; +} + +function readAnswer(raw: unknown): { id: string; answer: OwnerAnswer } { + if (typeof raw !== "string") { + throw new OwnerLinkError("malformed-answer"); + } + let decoded: unknown; + try { + decoded = JSON.parse(raw); + } catch { + throw new OwnerLinkError("malformed-answer"); + } + if (decoded === null || typeof decoded !== "object" || Array.isArray(decoded)) { + throw new OwnerLinkError("malformed-answer"); + } + const members = decoded as Record; + const id = members["id"]; + const outcome = members["outcome"]; + if (typeof id !== "string") { + throw new OwnerLinkError("malformed-answer"); + } + if (outcome === "performed") { + return { id, answer: { outcome, value: members["value"] } }; + } + if (outcome === "refused") { + const refusal = members["refusal"]; + if (typeof refusal !== "string") { + throw new OwnerLinkError("malformed-answer"); + } + return { id, answer: { outcome, refusal } }; + } + throw new OwnerLinkError("malformed-answer"); +} + +/** + * Hold one connection open for the calling scope. + * + * Teardown resolves every request still waiting with a closed refusal rather + * than leaving it pending: a caller blocked on an answer that can never arrive + * would outlive the connection it was asking through. + */ +export function useOwnerConnection(socket: OwnerSocket): Operation { + return resource(function* (provide) { + const waiting = new Map>>(); + const messages = createSignal(); + let closed = false; + + socket.addEventListener("message", (event) => messages.send(event.data)); + socket.addEventListener("close", () => { + closed = true; + messages.close(); + }); + + yield* spawn(function* () { + for (const raw of yield* each(messages)) { + // A malformed answer fails the request it names when it names one, and + // is otherwise dropped: it cannot be attributed to a caller. + try { + const { id, answer } = readAnswer(raw); + const pending = waiting.get(id); + if (pending !== undefined) { + waiting.delete(id); + pending.resolve(answer); + } + } catch { + // Nothing to attribute it to. + } + yield* each.next(); + } + for (const pending of waiting.values()) { + pending.reject(new OwnerLinkError("closed")); + } + waiting.clear(); + }); + + yield* provide({ + *ask(id: string, command: Record): Operation { + if (closed) { + throw new OwnerLinkError("closed"); + } + if (waiting.has(id)) { + throw new OwnerLinkError("duplicate-answer"); + } + const pending = withResolvers(); + waiting.set(id, pending); + socket.send(JSON.stringify({ ...command, id })); + return yield* pending.operation; + }, + }); + }); +} diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts new file mode 100644 index 000000000..75e1d5bb8 --- /dev/null +++ b/packages/workflow/tests/remote-client.test.ts @@ -0,0 +1,165 @@ +/** + * Tier WRH — carrying a request to a run's owner. + * + * Correlation and teardown are what this is about. A socket delivers what the + * owner sent whenever it sent it, so answers are matched by the id they name + * rather than by arrival order; and a connection that ends must fail the + * requests still waiting rather than leave a caller blocked on an answer that + * can never come. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { scoped, sleep, spawn } from "effection"; +import { type OwnerSocket, OwnerLinkError, useOwnerConnection } from "../src/remote/client.ts"; + +/** A socket a test drives by hand. */ +function fakeSocket() { + const sent: Record[] = []; + let onMessage: ((event: { data: unknown }) => void) | undefined; + let onClose: (() => void) | undefined; + const socket: OwnerSocket = { + send(data: string): void { + sent.push(JSON.parse(data)); + }, + close(): void { + onClose?.(); + }, + addEventListener(type: "message" | "close", listener: never): void { + if (type === "message") { + onMessage = listener; + } else { + onClose = listener; + } + }, + }; + return { + socket, + sent, + answer(value: unknown): void { + onMessage?.({ data: typeof value === "string" ? value : JSON.stringify(value) }); + }, + end(): void { + onClose?.(); + }, + }; +} + +describe("a connection to a run's owner", () => { + it("sends the command with its id and answers the caller that asked", function* () { + const wire = fakeSocket(); + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(() => owner.ask("a1", { command: "frontier" })); + yield* sleep(0); + // The request is on the wire before any answer exists. + expect(wire.sent).toEqual([{ command: "frontier", id: "a1" }]); + wire.answer({ id: "a1", outcome: "performed", value: { root: "root-a" } }); + expect(yield* asking).toEqual({ outcome: "performed", value: { root: "root-a" } }); + }); + yield* sleep(0); + }); + + it("matches answers by the id they name, not by arrival order", function* () { + const wire = fakeSocket(); + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const first = yield* spawn(() => owner.ask("a1", { command: "frontier" })); + const second = yield* spawn(() => owner.ask("a2", { command: "settle" })); + + // Answered in the opposite order to the asking. + wire.answer({ id: "a2", outcome: "performed", value: "second" }); + wire.answer({ id: "a1", outcome: "performed", value: "first" }); + + expect(yield* first).toEqual({ outcome: "performed", value: "first" }); + expect(yield* second).toEqual({ outcome: "performed", value: "second" }); + }); + yield* sleep(0); + }); + + it("hands back a refusal as an answer rather than a transport failure", function* () { + const wire = fakeSocket(); + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(() => owner.ask("a1", { command: "commit" })); + wire.answer({ id: "a1", outcome: "refused", refusal: "acquisition:already-running" }); + expect(yield* asking).toEqual({ + outcome: "refused", + refusal: "acquisition:already-running", + }); + }); + yield* sleep(0); + }); + + it("drops an answer it cannot attribute, and still answers the caller", function* () { + const wire = fakeSocket(); + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(() => owner.ask("a1", { command: "frontier" })); + wire.answer("not json at all"); + wire.answer({ id: "somebody-else", outcome: "performed", value: 1 }); + wire.answer({ id: "a1", outcome: "performed", value: "mine" }); + expect(yield* asking).toEqual({ outcome: "performed", value: "mine" }); + }); + yield* sleep(0); + }); + + it("fails a request still waiting when the connection ends", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }); + } catch (error) { + raised = error; + } + }); + wire.end(); + yield* asking; + }); + yield* sleep(0); + expect(raised).toBeInstanceOf(OwnerLinkError); + expect((raised as OwnerLinkError).refusal).toBe("closed"); + }); + + it("refuses to ask through a connection that already ended", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + wire.end(); + try { + yield* owner.ask("a1", { command: "frontier" }); + } catch (error) { + raised = error; + } + }); + expect((raised as OwnerLinkError).refusal).toBe("closed"); + }); + + it("refuses a second request under an id already in flight", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + yield* spawn(() => owner.ask("a1", { command: "frontier" })); + yield* sleep(0); + try { + yield* owner.ask("a1", { command: "settle" }); + } catch (error) { + raised = error; + } + wire.answer({ id: "a1", outcome: "performed", value: null }); + }); + expect((raised as OwnerLinkError).refusal).toBe("duplicate-answer"); + }); +}); From 871dbc85cd626fca641c09ad833ddbc3121a3ec7 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 21:39:47 -0400 Subject: [PATCH 19/81] =?UTF-8?q?=F0=9F=94=91=20Verify=20the=20token=20ins?= =?UTF-8?q?tead=20of=20trusting=20claims=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrections 1 and 2 from `.vscode/698/698-c-2.md`. The first is a real hole I opened, and the review is right about it: `admit()` took an `AdmissionRequest.claims` and compared its values to the policy, so a caller could assert every configured identity and obtain the acquisition without ever holding a signed token. A comment saying a verifier had authenticated them was not an authority boundary — `OwnerConfiguration` held no verifier, and the claims arrived through the same request surface the owner distrusts. `src/cloudflare/token.ts` verifies a compact JWS: one algorithm family by allowlist rather than by reading `alg` and believing it, keys the deployment configured, signature, then temporal validity — and only then is a payload member read as a claim. The raw token, the key material, the header and the claims the policy does not name stop there; none is retained, attached, journaled or returned. `admitClaims()` and `parseClaims()` are no longer exported. `admitToken()` is the only way into that module, because an exported "check these claims" is precisely the surface that made this forgeable. `AdmissionRequest` now carries the raw token and has no member for a verified result, a claim set, an acquisition identity or verification material — a request that could name any of those would be a request choosing what it is allowed to be. Verification material is closure state on the owner. `admit()` is an Effection operation, since verification suspends; the test owner drives it through one scope at the runtime callback boundary. Correction 2: the acquisition correlation is minted on the owner after both checks pass, from `crypto.getRandomValues`, and is no longer a caller argument. It partitions acquisition-private staging and duplicate handling and is not a bearer credential — the exact live socket is still what proves a message may act. Thirty-five assertions. Tokens are signed with an RSA key pair generated in the test process, so these are real signatures rather than a stub that agreed. A correctly signed token admits; an edited payload, a wrong key under a configured key id, an unknown key id, `alg: "none"`, an absent token, a non-JWS, an expired one and a not-yet-valid one each refuse before acquisition and before state. The wrong-release test now presents an unusable token, so it still proves the build is compared first. Two sequential acquisitions receive different correlations. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/admission.ts | 39 +++- packages/workflow/src/cloudflare/owner.ts | 61 ++++-- packages/workflow/src/cloudflare/token.ts | 180 ++++++++++++++++++ .../cloudflare/executor-acquisition.vitest.ts | 128 ++++++++++--- .../cloudflare/support/executor-object.ts | 57 ++++-- .../tests/cloudflare/support/tokens.ts | 68 +++++++ 6 files changed, 470 insertions(+), 63 deletions(-) create mode 100644 packages/workflow/src/cloudflare/token.ts create mode 100644 packages/workflow/tests/cloudflare/support/tokens.ts diff --git a/packages/workflow/src/cloudflare/admission.ts b/packages/workflow/src/cloudflare/admission.ts index 17a247f33..bd6f1cfdb 100644 --- a/packages/workflow/src/cloudflare/admission.ts +++ b/packages/workflow/src/cloudflare/admission.ts @@ -8,6 +8,11 @@ * owner can be renamed, so a check on `repository` would admit whoever holds * the name today. * + * The claims reaching this module have already been proved to come from the + * issuer — `token.ts` verifies the signature, the algorithm and the temporal + * validity first. That order is the whole security property: comparing claim + * values a caller could have written is arithmetic, not authentication. + * * Nothing about the token survives the check. The raw JWT, the JWKS endpoint, * the claims this contract does not name, and the reason a signature failed are * all provider state: none of them reaches durable storage, a journal event, a @@ -16,6 +21,9 @@ * assert without pinning provider wording. */ +import type { Operation } from "effection"; +import { type TokenVerification, verifyToken } from "./token.ts"; + /** What a deployment must state before any runner can be admitted. */ export interface AdmissionPolicy { readonly issuer: string; @@ -80,12 +88,13 @@ function requireClaim(claim: unknown, expected: string, refusal: AdmissionRefusa /** * Hold verified claims to the configured policy. * - * Takes claims a verifier already authenticated rather than a token, so this - * module owns *which* claims decide and nothing about how a signature is - * checked. A caller that has not verified a signature has not admitted - * anything, whatever this returns. + * Private to this module's own admission path. It is not exported, because an + * exported "check these claims" is exactly the surface that made the previous + * revision forgeable: a caller reaching it directly would be a caller choosing + * its own identity. Reaching it goes through `admitToken()`, which verifies + * first. */ -export function admitClaims(policy: AdmissionPolicy, claims: ActionsClaims): void { +function admitClaims(policy: AdmissionPolicy, claims: ActionsClaims): void { requireClaim(claims.iss, policy.issuer, "issuer"); // `aud` may be a string or an array of them; only the exact configured // audience admits, and an array containing it is that audience. @@ -102,8 +111,8 @@ export function admitClaims(policy: AdmissionPolicy, claims: ActionsClaims): voi requireClaim(claims.job_workflow_ref, policy.jobWorkflowRef, "workflow-identity"); } -/** Read a claim set out of a payload nothing has inspected yet. */ -export function parseClaims(payload: unknown): ActionsClaims { +/** Read a claim set out of a verified payload. */ +function parseClaims(payload: unknown): ActionsClaims { if (payload === null || typeof payload !== "object" || Array.isArray(payload)) { throw new AdmissionError("token-malformed"); } @@ -119,3 +128,19 @@ export function parseClaims(payload: unknown): ActionsClaims { job_workflow_ref: members["job_workflow_ref"], }; } + +/** + * Verify a token and hold what it proved to the configured policy. + * + * The only way into this module. It takes the bytes a runner presented and the + * verification material the deployment configured, and nothing a request can + * name reaches either. + */ +export function* admitToken( + policy: AdmissionPolicy, + verification: TokenVerification, + token: unknown, +): Operation { + const payload = yield* verifyToken(verification, token); + admitClaims(policy, parseClaims(payload)); +} diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index e42787a93..b8b31fdd1 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -21,6 +21,7 @@ */ import { DurableObject } from "cloudflare:workers"; +import type { Operation } from "effection"; import { acquireExecutor, type AcquisitionAttachment, @@ -28,7 +29,8 @@ import { releaseExecutor, requireAcquisition, } from "./acquisition.ts"; -import { admitClaims, type AdmissionPolicy, AdmissionError, parseClaims } from "./admission.ts"; +import { admitToken, type AdmissionPolicy, AdmissionError } from "./admission.ts"; +import { TokenError, type TokenVerification } from "./token.ts"; import { CommandError, type CommandResult, parseCommand, type RunnerCommand } from "./commands.ts"; import { declaredObjects, @@ -41,17 +43,26 @@ import { ReleaseIdentityError, requireSameRelease } from "./release.ts"; import { admitRunId, RunIdError } from "./routing.ts"; import type { OwnerStorage } from "./storage.ts"; -/** What one admission presents. */ +/** + * What one admission presents. + * + * Bytes and identifiers, all of them untrusted. There is deliberately no member + * for a verified result, a claim set, an acquisition identity or verification + * material: a request that could name any of those would be a request choosing + * what it is allowed to be. + */ export interface AdmissionRequest { readonly runId: unknown; readonly release: unknown; - /** Claims a verifier has already authenticated. */ - readonly claims: unknown; + /** The raw short-lived OIDC token, exactly as presented. */ + readonly token: unknown; } /** Everything a deployment must state before this object admits anybody. */ export interface OwnerConfiguration { readonly policy: AdmissionPolicy; + /** The issuer's keys and clock. Trusted closure state, never request data. */ + readonly verification: TokenVerification; } /** Name a refusal without repeating what caused it. */ @@ -62,6 +73,9 @@ export function refusalOf(error: unknown): string { if (error instanceof AdmissionError) { return `admission:${error.refusal}`; } + if (error instanceof TokenError) { + return `token:${error.refusal}`; + } if (error instanceof ReleaseIdentityError) { return `release:${error.refusal}`; } @@ -95,22 +109,21 @@ export abstract class WorkflowOwnerObject extends DurableObject { /** * Admit one executor connection. * - * The order is the contract: the build is compared before the token is read, - * the token before the run is touched, and the acquisition is taken last. A - * refusal at any step leaves no acquisition and no object state — which is - * what makes "a mismatched build cannot reach run state" a fact about the - * code rather than a hope about it. + * The order is the contract: the build is compared before any token work, the + * token is verified before the run is touched, and the acquisition is taken + * last. A refusal at any step leaves no acquisition and no object state. + * + * The correlation value is minted here, after both checks pass, and never + * taken from the request. A caller-selected one would let a later connection + * reuse an abandoned identifier and collide with the private staging that + * identifier partitions. */ - admit( - request: AdmissionRequest, - socket: WebSocket, - acquisitionId: string, - ): AcquisitionAttachment { - const { policy } = this.configuration(); + *admit(request: AdmissionRequest, socket: WebSocket): Operation { + const { policy, verification } = this.configuration(); requireSameRelease(policy.release, request.release); - admitClaims(policy, parseClaims(request.claims)); + yield* admitToken(policy, verification, request.token); const runId = admitRunId(request.runId); - return acquireExecutor(this.ctx, socket, runId, acquisitionId); + return acquireExecutor(this.ctx, socket, runId, mintAcquisitionId()); } /** @@ -160,3 +173,17 @@ export abstract class WorkflowOwnerObject extends DurableObject { recognizeObject(this.owned); } } + +/** + * A fresh correlation value for one acquisition. + * + * Bounded and unpredictable, and used only to partition acquisition-private + * staging and duplicate handling. It is not a bearer credential, a lease, a + * generation record or a durable identity: what proves a message may act is the + * exact live socket, and this value proves nothing on its own. + */ +function mintAcquisitionId(): string { + const bytes = new Uint8Array(16); + crypto.getRandomValues(bytes); + return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); +} diff --git a/packages/workflow/src/cloudflare/token.ts b/packages/workflow/src/cloudflare/token.ts new file mode 100644 index 000000000..96a0d8fbe --- /dev/null +++ b/packages/workflow/src/cloudflare/token.ts @@ -0,0 +1,180 @@ +/** + * Verifying the token a runner presents. + * + * This is the authority boundary, so it takes bytes rather than a claim set. A + * caller that could hand over decoded claims would be a caller that could + * assert whatever the policy asks for, and no amount of equality checking after + * that point would mean anything — which is exactly the hole this module + * closes. + * + * What it does is ordinary compact-JWS verification, narrowed hard: one + * algorithm family, keys the deployment configured, and temporal validity + * checked before any payload member is read as a claim. Everything about the + * token stops here. The raw JWT, the key material, the header, the claims the + * policy does not name and the reason a signature failed are all provider + * state: none of it is retained, attached, journaled, logged, or returned. + */ + +import { type Operation, until } from "effection"; + +/** Why a token was not accepted. */ +export type TokenRefusal = + | "token-absent" + | "token-malformed" + | "unsupported-algorithm" + | "unknown-key" + | "bad-signature" + | "expired" + | "not-yet-valid"; + +export class TokenError extends Error { + override name = "TokenError"; + + constructor(readonly refusal: TokenRefusal) { + super(`this runner's token was not accepted (${refusal})`); + } +} + +/** + * The one signature family this accepts. + * + * GitHub Actions signs with RS256. An allowlist rather than a lookup, because + * reading the algorithm out of the header and trusting it is how a token comes + * to be "verified" with `none` or with a symmetric key an attacker chose. + */ +const SUPPORTED = "RS256"; + +/** What a deployment configures before any token can be verified. */ +export interface TokenVerification { + /** + * The issuer's public keys. Fetched and rotated by the host. + * + * `kid` is carried beside the key rather than read off it: the runtime's + * `JsonWebKey` does not declare one, and a key set that narrows by id is what + * a JWKS is for. + */ + readonly keys: readonly VerificationKey[]; + /** How much clock skew to tolerate, in seconds. */ + readonly skewSeconds: number; + /** Now, in seconds since the epoch. Injected so a test can be exact. */ + readonly now: () => number; +} + +/** One configured public key, and the id a token may name it by. */ +export interface VerificationKey { + readonly kid?: string; + readonly jwk: JsonWebKey; +} + +function decodeSegment(segment: string): unknown { + // base64url, without the padding a compact JWS omits. + const padded = segment.replaceAll("-", "+").replaceAll("_", "/"); + const filled = padded + "=".repeat((4 - (padded.length % 4)) % 4); + let text: string; + try { + const bytes = Uint8Array.from(atob(filled), (character) => character.charCodeAt(0)); + text = new TextDecoder().decode(bytes); + } catch { + throw new TokenError("token-malformed"); + } + try { + return JSON.parse(text); + } catch { + throw new TokenError("token-malformed"); + } +} + +function object(value: unknown): Record { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new TokenError("token-malformed"); + } + return value as Record; +} + +function signatureBytes(segment: string): Uint8Array { + const padded = segment.replaceAll("-", "+").replaceAll("_", "/"); + const filled = padded + "=".repeat((4 - (padded.length % 4)) % 4); + try { + return Uint8Array.from(atob(filled), (character) => character.charCodeAt(0)); + } catch { + throw new TokenError("token-malformed"); + } +} + +/** + * Verify one compact JWS and answer with its payload. + * + * The order is the contract: shape, then algorithm, then signature, then time. + * A payload member is not a claim until every one of those has passed, which is + * why nothing here returns early with something a caller could mistake for one. + */ +export function* verifyToken( + configured: TokenVerification, + token: unknown, +): Operation> { + if (typeof token !== "string" || token === "") { + throw new TokenError("token-absent"); + } + const parts = token.split("."); + if (parts.length !== 3) { + throw new TokenError("token-malformed"); + } + const [encodedHeader, encodedPayload, encodedSignature] = parts; + if ( + encodedHeader === undefined || + encodedPayload === undefined || + encodedSignature === undefined + ) { + throw new TokenError("token-malformed"); + } + + const header = object(decodeSegment(encodedHeader)); + if (header["alg"] !== SUPPORTED) { + throw new TokenError("unsupported-algorithm"); + } + + const signed = new TextEncoder().encode(`${encodedHeader}.${encodedPayload}`); + const signature = signatureBytes(encodedSignature); + const keyId = header["kid"]; + // A `kid` narrows which key is tried; its absence means every configured key + // is a candidate. Either way only a configured key can verify anything. + const candidates = configured.keys.filter( + (key) => typeof keyId !== "string" || key.kid === undefined || key.kid === keyId, + ); + if (candidates.length === 0) { + throw new TokenError("unknown-key"); + } + + let verified = false; + for (const candidate of candidates) { + const key = yield* until( + crypto.subtle.importKey( + "jwk", + candidate.jwk, + { name: "RSASSA-PKCS1-v1_5", hash: "SHA-256" }, + false, + ["verify"], + ), + ); + const matched = yield* until(crypto.subtle.verify("RSASSA-PKCS1-v1_5", key, signature, signed)); + if (matched) { + verified = true; + break; + } + } + if (!verified) { + throw new TokenError("bad-signature"); + } + + const payload = object(decodeSegment(encodedPayload)); + const now = configured.now(); + const expiry = payload["exp"]; + if (typeof expiry === "number" && now > expiry + configured.skewSeconds) { + throw new TokenError("expired"); + } + const notBefore = payload["nbf"]; + if (typeof notBefore === "number" && now + configured.skewSeconds < notBefore) { + throw new TokenError("not-yet-valid"); + } + return payload; +} diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts index ea8eb9150..70ba65b7e 100644 --- a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -9,9 +9,10 @@ */ import { env, runInDurableObject } from "cloudflare:test"; -import { describe, expect, it } from "vitest"; +import { beforeAll, describe, expect, it } from "vitest"; import type { ExecutorObject } from "./support/executor-object.ts"; import { POLICY, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, tamper, type TestKeys } from "./support/tokens.ts"; let unique = 0; @@ -29,18 +30,46 @@ function on( const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +/** The clock the owner is configured with, so expiry is exact. */ +const NOW = 1_800_000_000; + +let keys: TestKeys; +let otherKeys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); + otherKeys = await generateKeys("other-key"); +}); + +/** Claims a correctly issued token carries, plus any override. */ +function claims(overrides: Record = {}): Record { + return { ...VALID_CLAIMS, iat: NOW - 10, nbf: NOW - 10, exp: NOW + 600, ...overrides }; +} + +/** An owner configured with the real public key, ready to be connected to. */ +async function admitted( + stub: ReturnType, + request: Record = {}, + signWith: TestKeys = keys, + header: Record = {}, +): Promise { + await on(stub, (o) => o.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = "token" in request ? request["token"] : await signToken(signWith, claims(), header); + return await on(stub, (o) => o.admitConnection({ ...request, token })); +} + describe("admitting an executor", () => { it("admits a matching build with authenticated claims", async () => { const stub = executor(); - expect(await on(stub, (o) => o.admitConnection({}))).toBe("admitted"); + expect(await admitted(stub)).toBe("admitted"); expect(await on(stub, (o) => o.holders())).toBe(1); }); it("refuses a build the owner did not agree to, before reading the token", async () => { const stub = executor(); - // The claims are deliberately unusable. If the release were checked after - // them, the refusal would name the token rather than the build. - expect(await on(stub, (o) => o.admitConnection({ release: "other-build", claims: null }))).toBe( + // The token is deliberately unusable. If the release were checked after it, + // the refusal would name the token rather than the build. + expect(await admitted(stub, { release: "other-build", token: "not a token" })).toBe( "release:release-mismatch", ); expect(await on(stub, (o) => o.holders())).toBe(0); @@ -85,29 +114,66 @@ describe("admitting an executor", () => { ]; for (const [expected, overrides] of cases) { const stub = executor(); - const claims = { ...VALID_CLAIMS, ...overrides }; - expect(await on(stub, (o) => o.admitConnection({ claims }))).toBe(expected); + const token = await signToken(keys, claims(overrides)); + expect(await admitted(stub, { token })).toBe(expected); expect(await on(stub, (o) => o.holders())).toBe(0); } }); it("accepts an audience array containing the configured one", async () => { const stub = executor(); - const claims = { ...VALID_CLAIMS, aud: ["https://other", POLICY.audience] }; - expect(await on(stub, (o) => o.admitConnection({ claims }))).toBe("admitted"); + const token = await signToken(keys, claims({ aud: ["https://other", POLICY.audience] })); + expect(await admitted(stub, { token })).toBe("admitted"); }); - it("refuses a token that is not a claim set at all", async () => { + it("refuses a token whose payload was edited after signing", async () => { const stub = executor(); - expect(await on(stub, (o) => o.admitConnection({ claims: "a string" }))).toBe( - "admission:token-malformed", - ); + const token = tamper(await signToken(keys, claims()), claims({ repository_id: "999" })); + expect(await admitted(stub, { token })).toBe("token:bad-signature"); + expect(await on(stub, (o) => o.holders())).toBe(0); + }); + + it("refuses a token naming a key the deployment does not hold", async () => { + const stub = executor(); + // Signed by another issuer, and saying so: no configured key is even a + // candidate, which is a different refusal from one that failed to verify. + expect(await admitted(stub, {}, otherKeys)).toBe("token:unknown-key"); + }); + + it("refuses a token signed with the wrong key under a configured key id", async () => { + const stub = executor(); + const token = await signToken(otherKeys, claims(), { kid: keys.kid }); + expect(await admitted(stub, { token })).toBe("token:bad-signature"); + expect(await on(stub, (o) => o.holders())).toBe(0); + }); + + it("refuses an algorithm it does not support", async () => { + const stub = executor(); + const token = await signToken(keys, claims(), { alg: "none" }); + expect(await admitted(stub, { token })).toBe("token:unsupported-algorithm"); + }); + + it("refuses a token that is absent or not a compact JWS", async () => { + const stub = executor(); + expect(await admitted(stub, { token: undefined })).toBe("token:token-absent"); + expect(await admitted(stub, { token: "one.two" })).toBe("token:token-malformed"); + }); + + it("refuses a token outside its validity window", async () => { + const expired = executor(); + expect( + await admitted(expired, { token: await signToken(keys, claims({ exp: NOW - 3600 })) }), + ).toBe("token:expired"); + const early = executor(); + expect( + await admitted(early, { token: await signToken(keys, claims({ nbf: NOW + 3600 })) }), + ).toBe("token:not-yet-valid"); }); it("refuses a run id that could not address an owner", async () => { const stub = executor(); - expect(await on(stub, (o) => o.admitConnection({ runId: "" }))).toBe("run-id:run-id-empty"); - expect(await on(stub, (o) => o.admitConnection({ runId: 42 }))).toBe("run-id:run-id-absent"); + expect(await admitted(stub, { runId: "" })).toBe("run-id:run-id-empty"); + expect(await admitted(stub, { runId: 42 })).toBe("run-id:run-id-absent"); expect(await on(stub, (o) => o.holders())).toBe(0); }); }); @@ -115,14 +181,30 @@ describe("admitting an executor", () => { describe("holding an acquisition", () => { it("refuses a second healthy executor rather than following it", async () => { const stub = executor(); - expect(await on(stub, (o) => o.admitConnection({}))).toBe("admitted"); - expect(await on(stub, (o) => o.admitConnection({}))).toBe("acquisition:already-running"); + expect(await admitted(stub)).toBe("admitted"); + expect(await admitted(stub)).toBe("acquisition:already-running"); expect(await on(stub, (o) => o.holders())).toBe(1); }); + it("mints its own correlation, which no caller can select or reuse", async () => { + const first = executor(); + await admitted(first); + const one = await on(first, (o) => o.acquisitionId()); + await on(first, (o) => o.closeConnection(1)); + await admitted(first); + const two = await on(first, (o) => o.acquisitionId()); + + // Bounded, unpredictable, and different for a second acquisition of the + // same run — so private staging belonging to the first cannot be addressed + // by the second. + expect(one).toMatch(/^[0-9a-f]{32}$/); + expect(two).toMatch(/^[0-9a-f]{32}$/); + expect(two).not.toBe(one); + }); + it("lets the admitted connection send, and answers what it performed", async () => { const stub = executor(); - await on(stub, (o) => o.admitConnection({})); + await admitted(stub); expect( await on(stub, (o) => o.send(1, JSON.stringify({ id: "1", command: "frontier" }))), ).toEqual({ id: "1", outcome: "performed", value: { performed: "frontier" } }); @@ -130,7 +212,7 @@ describe("holding an acquisition", () => { it("refuses a socket it never admitted", async () => { const stub = executor(); - await on(stub, (o) => o.admitConnection({})); + await admitted(stub); expect( await on(stub, (o) => o.sendAsStranger(JSON.stringify({ id: "1", command: "frontier" }))), ).toEqual({ id: "", outcome: "refused", refusal: "acquisition:foreign-connection" }); @@ -138,11 +220,11 @@ describe("holding an acquisition", () => { it("owns nothing once the connection ends, and rolls nothing back", async () => { const stub = executor(); - await on(stub, (o) => o.admitConnection({})); + await admitted(stub); await on(stub, (o) => o.closeConnection(1)); expect(await on(stub, (o) => o.holders())).toBe(0); // And the next executor may take it, with no lease having expired. - expect(await on(stub, (o) => o.admitConnection({}))).toBe("admitted"); + expect(await admitted(stub)).toBe("admitted"); }); it("proves the acquisition before it reads a command", async () => { @@ -158,7 +240,7 @@ describe("holding an acquisition", () => { describe("reading a runner command", () => { it("refuses what it cannot read as one", async () => { const stub = executor(); - await on(stub, (o) => o.admitConnection({})); + await admitted(stub); const refuse = async (raw: string) => (await on(stub, (o) => o.send(1, raw))) as { refusal: string }; expect((await refuse("not json")).refusal).toBe("command:not-an-object"); @@ -179,7 +261,7 @@ describe("reading a runner command", () => { it("reads a commit intent whole", async () => { const stub = executor(); - await on(stub, (o) => o.admitConnection({})); + await admitted(stub); const raw = JSON.stringify({ id: "7", command: "commit", diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index a268019e8..bf30a10aa 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -7,10 +7,12 @@ * allowed to send one, not what each one means. */ +import { run } from "effection"; import { acquisitionHolders } from "../../../src/cloudflare/acquisition.ts"; import { WorkflowOwnerObject } from "../../../src/cloudflare/owner.ts"; import type { AdmissionRequest, OwnerConfiguration } from "../../../src/cloudflare/owner.ts"; import type { AdmissionPolicy } from "../../../src/cloudflare/admission.ts"; +import type { TokenVerification, VerificationKey } from "../../../src/cloudflare/token.ts"; import type { RunnerCommand } from "../../../src/cloudflare/commands.ts"; import { refusalOf } from "../../../src/cloudflare/owner.ts"; @@ -27,7 +29,7 @@ export const POLICY: AdmissionPolicy = { release: "factory-2026.09.02-abcdef", }; -/** Claims a verifier would have authenticated for the policy above. */ +/** The claims a correctly issued token carries for the policy above. */ export const VALID_CLAIMS: Record = { iss: POLICY.issuer, aud: POLICY.audience, @@ -42,10 +44,30 @@ export const VALID_CLAIMS: Record = { const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; export class ExecutorObject extends WorkflowOwnerObject { - #acquisitions = 0; + /** + * The verification material this owner is configured with. + * + * Installed by a test before it connects, exactly as a deployment would + * install a fetched JWKS. It is closure state on the object, never something + * an admission request can name. + */ + #keys: VerificationKey[] = []; + #now = 1_800_000_000; + + configure(keys: VerificationKey[], now?: number): void { + this.#keys = keys; + if (now !== undefined) { + this.#now = now; + } + } protected configuration(): OwnerConfiguration { - return { policy: POLICY }; + const verification: TokenVerification = { + keys: this.#keys, + skewSeconds: 60, + now: () => this.#now, + }; + return { policy: POLICY, verification }; } protected perform(_socket: WebSocket, _runId: string, command: RunnerCommand): unknown { @@ -55,29 +77,32 @@ export class ExecutorObject extends WorkflowOwnerObject { /** * Admit one connection, answering what happened rather than raising. * - * The client half of the pair is kept so a later call can drive it; the - * server half is what the object admitted. + * The server half of the pair is what the object admitted. Verification is + * asynchronous, so this drives the admission operation through one Effection + * scope — the runtime callback boundary this host adapts at. */ - admitConnection(request: Partial): string { + async admitConnection(request: Partial): Promise { const pair = new WebSocketPair(); const server = pair[1]; - this.#acquisitions += 1; + const presented: AdmissionRequest = { + runId: "runId" in request ? request.runId : RUN_ID, + release: "release" in request ? request.release : POLICY.release, + token: "token" in request ? request.token : undefined, + }; try { - this.admit( - { - runId: "runId" in request ? request.runId : RUN_ID, - release: "release" in request ? request.release : POLICY.release, - claims: "claims" in request ? request.claims : VALID_CLAIMS, - }, - server, - `acquisition-${this.#acquisitions}`, - ); + await run(() => this.admit(presented, server)); return "admitted"; } catch (error) { return refusalOf(error); } } + /** The correlation the live acquisition is partitioned by. */ + acquisitionId(): string { + const held = acquisitionHolders(this.ctx)[0]; + return held === undefined ? "" : held.held.acquisitionId; + } + /** How many live connections currently hold this run's executor. */ holders(): number { return acquisitionHolders(this.ctx).length; diff --git a/packages/workflow/tests/cloudflare/support/tokens.ts b/packages/workflow/tests/cloudflare/support/tokens.ts new file mode 100644 index 000000000..7436edc99 --- /dev/null +++ b/packages/workflow/tests/cloudflare/support/tokens.ts @@ -0,0 +1,68 @@ +/** + * Signing tokens for the admission tests, with keys generated here. + * + * Real signatures against a real key pair, so the assertions are about + * verification rather than about a stub that agreed to say yes. The key never + * leaves this process and is generated per run. + */ + +/** One generated key pair, and the JWK a verifier is configured with. */ +export interface TestKeys { + readonly signing: CryptoKey; + readonly publicJwk: JsonWebKey; + readonly kid: string; +} + +function base64url(bytes: Uint8Array): string { + let binary = ""; + for (const byte of bytes) { + binary += String.fromCharCode(byte); + } + return btoa(binary).replaceAll("+", "-").replaceAll("/", "_").replaceAll("=", ""); +} + +function encodeSegment(value: unknown): string { + return base64url(new TextEncoder().encode(JSON.stringify(value))); +} + +export async function generateKeys(kid = "test-key"): Promise { + const generated = await crypto.subtle.generateKey( + { + name: "RSASSA-PKCS1-v1_5", + modulusLength: 2048, + publicExponent: new Uint8Array([1, 0, 1]), + hash: "SHA-256", + }, + true, + ["sign", "verify"], + ); + // `generateKey` is typed as either a key or a pair; an RSA signing algorithm + // always answers with a pair, and reading it as one is what proves that here. + if (!("privateKey" in generated) || !("publicKey" in generated)) { + throw new Error("expected an RSA key pair"); + } + const exported = await crypto.subtle.exportKey("jwk", generated.publicKey); + if (exported instanceof ArrayBuffer) { + throw new Error("expected a JWK export"); + } + return { signing: generated.privateKey, publicJwk: exported, kid }; +} + +/** Sign one compact JWS over `claims`. */ +export async function signToken( + keys: TestKeys, + claims: Record, + header: Record = {}, +): Promise { + const encodedHeader = encodeSegment({ alg: "RS256", typ: "JWT", kid: keys.kid, ...header }); + const encodedPayload = encodeSegment(claims); + const signed = new TextEncoder().encode(`${encodedHeader}.${encodedPayload}`); + const signature = await crypto.subtle.sign("RSASSA-PKCS1-v1_5", keys.signing, signed); + return `${encodedHeader}.${encodedPayload}.${base64url(new Uint8Array(signature))}`; +} + +/** A token whose payload was edited after it was signed. */ +export function tamper(token: string, claims: Record): string { + const parts = token.split("."); + return `${parts[0]}.${encodeSegment(claims)}.${parts[2]}`; +} From dd45ed88271eeb4a358887cc9d67e3305fe9d36a Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 21:43:06 -0400 Subject: [PATCH 20/81] =?UTF-8?q?=F0=9F=94=92=20Own=20the=20whole=20transa?= =?UTF-8?q?ction,=20detach=20the=20intent,=20fail=20the=20channel=20closed?= =?UTF-8?q?=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrections 3, 4 and 5 from `.vscode/698/698-c-2.md`. All three were real. Correction 3. `transactRemotely()` checked `gate.open`, then suspended in `frontier()`, and only took the gate afterwards — so two calls on one handle could both pass the check and act from the same starting frontier. It also released the gate when the body ended, before `commit()` returned, so later work could run while which state won was still undecided. The gate is now taken synchronously before the first suspension and released in one finalizer after the commit answer, on success, refusal, body failure, transport failure and cancellation alike. The transaction object still closes when the body does, so a retained handle refuses while the handle-level gate is held. Correction 4. `append()` cloned, but `readAll()` handed back references into the collector's own arrays: a body could read an event and mutate what it received, and the committed intent would differ from what `append()` admitted. Every crossing is now a fresh copy — in, out, and into the intent — and the collector's array never leaves. Events are admitted rather than assumed, with a count bound and an aggregate serialized-byte bound; an invalid or oversized one fails locally and sends nothing. Correction 5. The connection dropped malformed answers and answers for unknown ids, which meant a malformed reply to an in-flight commit left the caller waiting forever while the owner may already have committed. All three of those are evidence that the two sides disagree about which command completed, so the channel now fails closed: an unreadable answer, an answer naming a request nobody made, and a second answer to a request already settled each stop the channel, close the socket and reject every waiter. Owner refusals remain typed answers, and answers are bounded. Nine assertions on the channel and fifteen on the transaction. Two of the transaction ones are the concurrency cases the review asked for: a second transaction refuses while the first is suspended in `frontier()`, and the handle stays owned while a blocked `commit()` is undecided. Two more prove detachment from both directions — a reader mutating what `readAll()` returned, and a caller mutating what it appended. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/remote/client.ts | 61 +++++-- packages/workflow/src/remote/collector.ts | 139 ++++++++++----- packages/workflow/tests/remote-client.test.ts | 95 ++++++++-- .../workflow/tests/remote-transaction.test.ts | 166 +++++++++++++++++- 4 files changed, 389 insertions(+), 72 deletions(-) diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index 669dbbada..3f2a215d0 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -12,12 +12,25 @@ * answer rather than a transport failure. What the owner does with a command is * the owner's, and a client that interpreted a refusal would be a second place * deciding what a run may do. + * + * What it will not do is carry on after the two sides disagree about which + * command completed. An answer it cannot read, an answer naming a request + * nobody made, and a second answer to a request already settled are each + * evidence that correlation has broken — and a commit may have landed on the + * owner while the caller waits for a reply that will never be attributed. So + * the channel fails closed: it stops, and every waiter learns, rather than + * dropping the answer and leaving somebody blocked forever. */ import { createSignal, each, type Operation, resource, spawn, withResolvers } from "effection"; /** Why the connection itself could not carry a request. */ -export type LinkRefusal = "closed" | "malformed-answer" | "duplicate-answer"; +export type LinkRefusal = + | "closed" + | "malformed-answer" + | "unknown-answer" + | "duplicate-answer" + | "too-large"; export class OwnerLinkError extends Error { override name = "OwnerLinkError"; @@ -46,10 +59,16 @@ export interface OwnerConnection { ask(id: string, command: Record): Operation; } +/** The most bytes one answer may carry. */ +const MAX_ANSWER = 8 * 1024 * 1024; + function readAnswer(raw: unknown): { id: string; answer: OwnerAnswer } { if (typeof raw !== "string") { throw new OwnerLinkError("malformed-answer"); } + if (raw.length > MAX_ANSWER) { + throw new OwnerLinkError("too-large"); + } let decoded: unknown; try { decoded = JSON.parse(raw); @@ -88,6 +107,8 @@ function readAnswer(raw: unknown): { id: string; answer: OwnerAnswer } { export function useOwnerConnection(socket: OwnerSocket): Operation { return resource(function* (provide) { const waiting = new Map>>(); + /** Requests already answered, so a second answer is recognized as one. */ + const settled = new Set(); const messages = createSignal(); let closed = false; @@ -97,22 +118,40 @@ export function useOwnerConnection(socket: OwnerSocket): Operation { + closed = true; + for (const pending of waiting.values()) { + pending.reject(new OwnerLinkError(refusal)); + } + waiting.clear(); + socket.close(); + }; + yield* spawn(function* () { for (const raw of yield* each(messages)) { - // A malformed answer fails the request it names when it names one, and - // is otherwise dropped: it cannot be attributed to a caller. + let read: { id: string; answer: OwnerAnswer } | undefined; try { - const { id, answer } = readAnswer(raw); - const pending = waiting.get(id); - if (pending !== undefined) { - waiting.delete(id); - pending.resolve(answer); - } + read = readAnswer(raw); } catch { - // Nothing to attribute it to. + // The owner said something this build cannot read. Whether it was + // meant for a waiter is exactly what cannot be established. + fail("malformed-answer"); + break; } + const pending = waiting.get(read.id); + if (pending === undefined) { + // Either a request nobody made, or a second answer to one already + // settled. Both mean the two sides disagree about what completed. + fail(settled.has(read.id) ? "duplicate-answer" : "unknown-answer"); + break; + } + waiting.delete(read.id); + settled.add(read.id); + pending.resolve(read.answer); yield* each.next(); } + closed = true; for (const pending of waiting.values()) { pending.reject(new OwnerLinkError("closed")); } @@ -124,7 +163,7 @@ export function useOwnerConnection(socket: OwnerSocket): Operation(); diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index 41f7f4484..d5f29d323 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -32,7 +32,9 @@ export type CollectorRefusal = | "nested-transaction" | "transaction-closed" | "operation-inside-body" - | "too-many-events"; + | "too-many-events" + | "events-too-large" + | "malformed-event"; export class RemoteTransactionError extends Error { override name = "RemoteTransactionError"; @@ -67,6 +69,37 @@ export interface OwnerLink { /** The most events one intent may carry. */ const MAX_EVENTS = 4096; +/** The most serialized bytes one intent may carry. */ +const MAX_EVENT_BYTES = 4 * 1024 * 1024; + +/** + * Admit one event and detach it from whoever handed it over. + * + * Cloning on the way in is not enough on its own: a caller that reads an event + * back and mutates what it received would otherwise change what this + * transaction commits. So every crossing — in, out, and into the intent — is a + * fresh copy, and the collector's own array is never handed to anybody. + */ +function admitEvent(event: DurableEvent): DurableEvent { + if (event === null || typeof event !== "object") { + throw new RemoteTransactionError("malformed-event"); + } + if (!("type" in event) || typeof event.type !== "string") { + throw new RemoteTransactionError("malformed-event"); + } + try { + return structuredClone(event); + } catch { + // A value that cannot be cloned cannot be sent either. + throw new RemoteTransactionError("malformed-event"); + } +} + +/** The serialized size of what has been collected so far. */ +function serializedBytes(events: readonly DurableEvent[]): number { + return new TextEncoder().encode(JSON.stringify(events)).length; +} + /** * Whether a transaction is open on this handle. * @@ -105,61 +138,75 @@ export function transactRemotely( body: (transaction: WorkflowRunTransaction) => Operation, ): Operation> { return call(function* (): Operation> { + // Taken synchronously, before the first suspension. Checking and then + // suspending in `frontier()` would let two calls on one handle both pass + // the check and act from the same starting frontier. if (gate.open) { throw new RemoteTransactionError("nested-transaction"); } - const starting = yield* link.frontier(); - const appended: DurableEvent[] = []; - let live = true; - - const journal: DurableStream = { - *readAll(): Operation { - if (!live) { - throw new RemoteTransactionError("transaction-closed"); - } - // Read-your-writes: the starting prefix, then this transaction's own - // appends, in order. A body that reads back what it just wrote sees it - // even though the owner has not been told yet. - return [...starting.events, ...appended]; - }, - *append(event: DurableEvent): Operation { - if (!live) { - throw new RemoteTransactionError("transaction-closed"); - } - if (appended.length >= MAX_EVENTS) { - throw new RemoteTransactionError("too-many-events"); - } - // Cloned on the way in, so a caller that keeps mutating the value it - // handed over cannot change what this transaction will commit. - appended.push(structuredClone(event)); - }, - }; - gate.open = true; - let outcome: T; + // Released once, and only after nothing from this transaction can still + // affect the handle — which is after the commit answer, not after the body. + // Between those two the outcome is undecided, and later work must not run + // as though it had been decided. try { - // Everything the body started tears down before the intent is built. A - // failure or cancellation leaves through here without a commit, which is - // what makes "no commit was sent" the same statement as "the body did not - // finish". - outcome = yield* call(() => body({ journal })); + return yield* run(); } finally { - live = false; gate.open = false; } - const committed = yield* link.commit({ - expectedWorkspaceRootId: starting.workspaceRootId, - expectedJournalEventId: starting.journalEventId, - events: appended, - }); - if (!committed.ok) { - return committed; + function* run(): Operation> { + const starting = yield* link.frontier(); + const appended: DurableEvent[] = []; + let live = true; + + const journal: DurableStream = { + *readAll(): Operation { + if (!live) { + throw new RemoteTransactionError("transaction-closed"); + } + // Read-your-writes, as fresh copies. The starting prefix then this + // transaction's own appends, in order. + return [...starting.events, ...appended].map((event) => structuredClone(event)); + }, + *append(event: DurableEvent): Operation { + if (!live) { + throw new RemoteTransactionError("transaction-closed"); + } + if (appended.length >= MAX_EVENTS) { + throw new RemoteTransactionError("too-many-events"); + } + const admitted = admitEvent(event); + if (serializedBytes([...appended, admitted]) > MAX_EVENT_BYTES) { + throw new RemoteTransactionError("events-too-large"); + } + appended.push(admitted); + }, + }; + + let outcome: T; + try { + // Everything the body started tears down before the intent is built, so + // "no commit was sent" and "the body did not finish" are one statement. + outcome = yield* call(() => body({ journal })); + } finally { + // The handle is closed before the commit goes out, so a retained + // transaction object refuses while the handle-level gate is still held. + live = false; + } + + const committed = yield* link.commit({ + expectedWorkspaceRootId: starting.workspaceRootId, + expectedJournalEventId: starting.journalEventId, + // A private snapshot. The collector's own array never leaves. + events: appended.map((event) => structuredClone(event)), + }); + if (!committed.ok) { + return committed; + } + // Only now. `T` is the body's own value and never crossed the connection. + return Ok(outcome); } - // Only now. `T` is the body's own value and never crossed the connection; - // returning it before the owner committed would be a caller holding a - // result for work that did not happen. - return Ok(outcome); }); } diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts index 75e1d5bb8..6a59079b8 100644 --- a/packages/workflow/tests/remote-client.test.ts +++ b/packages/workflow/tests/remote-client.test.ts @@ -94,20 +94,6 @@ describe("a connection to a run's owner", () => { yield* sleep(0); }); - it("drops an answer it cannot attribute, and still answers the caller", function* () { - const wire = fakeSocket(); - yield* scoped(function* () { - const owner = yield* useOwnerConnection(wire.socket); - yield* sleep(0); - const asking = yield* spawn(() => owner.ask("a1", { command: "frontier" })); - wire.answer("not json at all"); - wire.answer({ id: "somebody-else", outcome: "performed", value: 1 }); - wire.answer({ id: "a1", outcome: "performed", value: "mine" }); - expect(yield* asking).toEqual({ outcome: "performed", value: "mine" }); - }); - yield* sleep(0); - }); - it("fails a request still waiting when the connection ends", function* () { const wire = fakeSocket(); let raised: unknown; @@ -162,4 +148,85 @@ describe("a connection to a run's owner", () => { }); expect((raised as OwnerLinkError).refusal).toBe("duplicate-answer"); }); + + it("fails every waiter when it cannot read an answer", function* () { + const wire = fakeSocket(); + const raised: unknown[] = []; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const first = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }); + } catch (error) { + raised.push(error); + } + }); + const second = yield* spawn(function* () { + try { + yield* owner.ask("a2", { command: "settle" }); + } catch (error) { + raised.push(error); + } + }); + yield* sleep(0); + // A commit may already have landed on the owner. Dropping this and + // leaving both callers waiting is the failure mode being refused. + wire.answer("not json at all"); + yield* first; + yield* second; + }); + expect(raised).toHaveLength(2); + for (const error of raised) { + expect((error as OwnerLinkError).refusal).toBe("malformed-answer"); + } + }); + + it("fails closed on an answer naming a request nobody made", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }); + } catch (error) { + raised = error; + } + }); + yield* sleep(0); + wire.answer({ id: "somebody-else", outcome: "performed", value: 1 }); + yield* asking; + }); + expect((raised as OwnerLinkError).refusal).toBe("unknown-answer"); + }); + + it("fails closed on a second answer to a request already settled", function* () { + const wire = fakeSocket(); + let answered: unknown; + let refused: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const first = yield* spawn(() => owner.ask("a1", { command: "frontier" })); + yield* sleep(0); + wire.answer({ id: "a1", outcome: "performed", value: "once" }); + answered = yield* first; + + const second = yield* spawn(function* () { + try { + yield* owner.ask("a2", { command: "settle" }); + } catch (error) { + refused = error; + } + }); + yield* sleep(0); + // The owner answers `a1` again. Correlation has broken. + wire.answer({ id: "a1", outcome: "performed", value: "twice" }); + yield* second; + }); + expect(answered).toEqual({ outcome: "performed", value: "once" }); + expect((refused as OwnerLinkError).refusal).toBe("duplicate-answer"); + }); }); diff --git a/packages/workflow/tests/remote-transaction.test.ts b/packages/workflow/tests/remote-transaction.test.ts index ef82a5b63..a86f97795 100644 --- a/packages/workflow/tests/remote-transaction.test.ts +++ b/packages/workflow/tests/remote-transaction.test.ts @@ -15,7 +15,7 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { Err, Ok, sleep, type Operation, type Result } from "effection"; +import { Err, Ok, sleep, spawn, withResolvers, type Operation, type Result } from "effection"; import type { DurableEvent } from "@executablemd/durable-streams"; import { type CommitIntent, @@ -65,6 +65,8 @@ function link( options: { frontier?: StartingFrontier; commit?: (intent: CommitIntent) => Result; + blockFrontier?: { operation: Operation }; + blockCommit?: { operation: Operation }; } = {}, ) { const sent: CommitIntent[] = []; @@ -75,10 +77,16 @@ function link( }; const owner: OwnerLink = { *frontier(): Operation { + if (options.blockFrontier !== undefined) { + yield* options.blockFrontier.operation; + } return starting; }, *commit(intent: CommitIntent): Operation> { sent.push(intent); + if (options.blockCommit !== undefined) { + yield* options.blockCommit.operation; + } return options.commit === undefined ? Ok(undefined) : options.commit(intent); }, }; @@ -230,6 +238,162 @@ describe("a remote transaction", () => { expect((raised as RemoteTransactionError).refusal).toBe("transaction-closed"); }); + it("owns the handle from before the first suspension until after the commit", function* () { + const held = withResolvers(); + const { owner, sent } = link({ blockFrontier: held }); + const gate = createTransactionGate(); + + const first = yield* spawn(() => + transactRemotely(owner, gate, function* () { + return "first"; + }), + ); + yield* sleep(0); + + // The first transaction is suspended inside `frontier()`. A second must not + // pass the gate and act from the same starting frontier. + let raised: unknown; + try { + yield* transactRemotely(owner, gate, function* () { + return "second"; + }); + } catch (error) { + raised = error; + } + expect((raised as RemoteTransactionError).refusal).toBe("nested-transaction"); + + held.resolve(); + yield* first; + expect(sent).toHaveLength(1); + }); + + it("keeps the handle while the commit is still undecided", function* () { + const held = withResolvers(); + const { owner } = link({ blockCommit: held }); + const gate = createTransactionGate(); + + const first = yield* spawn(() => + transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(event("one")); + return "first"; + }), + ); + yield* sleep(0); + + // The body has finished, but which state won is not yet established. + expect(gate.open).toBe(true); + let raised: unknown; + try { + requireNoOpenTransaction(gate); + } catch (error) { + raised = error; + } + expect((raised as RemoteTransactionError).refusal).toBe("operation-inside-body"); + + held.resolve(); + yield* first; + expect(gate.open).toBe(false); + }); + + it("releases the handle however the transaction ends", function* () { + const gate = createTransactionGate(); + + const succeeded = link(); + yield* transactRemotely(succeeded.owner, gate, function* () { + return undefined; + }); + expect(gate.open).toBe(false); + + const refused = link({ commit: () => Err(new Error("refused")) }); + yield* transactRemotely(refused.owner, gate, function* () { + return undefined; + }); + expect(gate.open).toBe(false); + + const failed = link(); + try { + yield* transactRemotely(failed.owner, gate, function* () { + throw new Error("body failed"); + }); + } catch { + // The refusal is the subject of another test; this one is about the gate. + } + expect(gate.open).toBe(false); + + const broken: OwnerLink = { + *frontier(): Operation { + throw new Error("transport failed"); + }, + *commit(): Operation> { + return Ok(undefined); + }, + }; + try { + yield* transactRemotely(broken, gate, function* () { + return undefined; + }); + } catch { + // Likewise. + } + expect(gate.open).toBe(false); + }); + + it("refuses an event it cannot admit, and sends nothing", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + + let raised: unknown; + try { + yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append({ nothing: true } as unknown as DurableEvent); + return undefined; + }); + } catch (error) { + raised = error; + } + expect((raised as RemoteTransactionError).refusal).toBe("malformed-event"); + expect(sent).toEqual([]); + }); + + it("refuses more bytes than one intent may carry", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + const wide = event("x".repeat(200_000)); + + let raised: unknown; + try { + yield* transactRemotely(owner, gate, function* (transaction) { + for (let index = 0; index < 40; index += 1) { + yield* transaction.journal.append(wide); + } + return undefined; + }); + } catch (error) { + raised = error; + } + expect((raised as RemoteTransactionError).refusal).toBe("events-too-large"); + expect(sent).toEqual([]); + }); + + it("commits what it admitted, not what a reader mutated afterwards", function* () { + const { owner, sent } = link(); + const gate = createTransactionGate(); + + yield* transactRemotely(owner, gate, function* (transaction) { + yield* transaction.journal.append(event("admitted")); + // Read it back and edit what came out. The collector handed over a copy, + // so the intent still carries what `append()` admitted. + const read = yield* transaction.journal.readAll(); + const mine = read[read.length - 1]; + if (mine !== undefined) { + rename(mine, "changed by a reader"); + } + return undefined; + }); + + expect(nameOf(sent[0]?.events[0])).toBe("admitted"); + }); + it("commits what it was handed, not what the caller mutated afterwards", function* () { const { owner, sent } = link(); const gate = createTransactionGate(); From b6f58d002914cc4189cb9acd62b341f611c8c147 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 22:19:27 -0400 Subject: [PATCH 21/81] =?UTF-8?q?=F0=9F=A7=A9=20Publish=20the=20pure=20cor?= =?UTF-8?q?e=20predicates=20from=20node-free=20subpaths=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit D1 item 1, and Ruling 1 from `.vscode/698/698-c-4.md`. The owner has to parse a settle request with the shared `parseDocumentExecutionCompletion()`, and could not: `storage/record.ts` imports `canonicalize` from the `@executablemd/core` root, whose barrel reaches `node:crypto`, `node:process` and `node:readline/promises`. None resolves in a Workers typecheck, so a pure function was unreachable because of where it sat. `canonicalize()` moves to `packages/core/src/canonicalize.ts`, importing only a type. `canonicalFingerprint()` stays in the Node-capable module, importing the pure function and `node:crypto`. Both keep their package-root exports and their behavior, and `@executablemd/core/canonicalize` publishes the pure half. Two more predicates blocked the same graph for the same reason, which the ruling anticipates and authorizes handling the same way. `isComponentName` was co-located with the registration machinery and `isCanonicalTarget` with the document-target catalog and its Markdown parser; both are string arithmetic. They move to `src/component-name.ts` and `src/document-target-spelling.ts`, with `@executablemd/core/component-name` and `@executablemd/core/document-target` selecting them. `storage/definition.ts` imports through those. No copy of any of them exists — each original module re-exports the leaf, so there is one implementation and the root surface is unchanged. This is not a `portable` barrel: three narrowly named leaves, each holding what it is named after. Two adjustments fell out. `tsconfig.cloudflare.json` no longer sets `exactOptionalPropertyTypes` or `noUncheckedIndexedAccess`: it was stricter than the repository holds itself to, so it failed pre-existing shared code the Deno check accepts, and a check that invents rules proves the wrong thing. And the percent-decoder now states `ignoreBOM: false` — already its behavior everywhere, and Cloudflare's own type declares both `TextDecoder` options required. Evidence. `packages/core/tests/canonicalize.test.ts` proves root and subpath answer identically, including for `__proto__`, and that the fingerprint still composes over the same ordering. `settle-parser.vitest.ts` runs on real workerd, which is the assertion that matters: it loads the shared parser where a Node builtin is genuinely absent, so a test-only import could not have passed for it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/core/canonicalize.ts | 13 ++ packages/core/component-name.ts | 13 ++ packages/core/deno.json | 3 + packages/core/document-target.ts | 13 ++ packages/core/package.json | 3 + packages/core/src/canonical.ts | 32 ++--- packages/core/src/canonicalize.ts | 41 ++++++ packages/core/src/component-name.ts | 18 +++ packages/core/src/components/registration.ts | 17 +-- packages/core/src/document-target-spelling.ts | 123 ++++++++++++++++++ packages/core/src/document-targets.ts | 122 +++-------------- packages/core/tests/canonicalize.test.ts | 63 +++++++++ packages/workflow/src/cloudflare/commands.ts | 35 ++++- packages/workflow/src/storage/definition.ts | 6 +- packages/workflow/src/storage/record.ts | 5 +- .../tests/cloudflare/settle-parser.vitest.ts | 96 ++++++++++++++ packages/workflow/tsconfig.cloudflare.json | 2 - 17 files changed, 451 insertions(+), 154 deletions(-) create mode 100644 packages/core/canonicalize.ts create mode 100644 packages/core/component-name.ts create mode 100644 packages/core/document-target.ts create mode 100644 packages/core/src/canonicalize.ts create mode 100644 packages/core/src/component-name.ts create mode 100644 packages/core/src/document-target-spelling.ts create mode 100644 packages/core/tests/canonicalize.test.ts create mode 100644 packages/workflow/tests/cloudflare/settle-parser.vitest.ts diff --git a/packages/core/canonicalize.ts b/packages/core/canonicalize.ts new file mode 100644 index 000000000..cad1f446a --- /dev/null +++ b/packages/core/canonicalize.ts @@ -0,0 +1,13 @@ +/** + * @module + * + * Canonical JSON ordering, for runtimes that cannot load a Node builtin. + * + * `canonicalize` is already public from the package root. This subpath exists + * so a consumer can select it without loading the root barrel, which reaches + * `node:crypto`, `node:process` and the rest of the host surface — a Cloudflare + * Worker resolving that graph fails to typecheck, and the operation it needs is + * pure. Same function, same behavior, narrower resolution path. + */ + +export { canonicalize } from "./src/canonicalize.ts"; diff --git a/packages/core/component-name.ts b/packages/core/component-name.ts new file mode 100644 index 000000000..7f3e49c03 --- /dev/null +++ b/packages/core/component-name.ts @@ -0,0 +1,13 @@ +/** + * @module + * + * How a document spells a component name, for runtimes that cannot load the + * engine. + * + * `isComponentName` is already public from the package root. This subpath + * selects it without the root barrel, which reaches `node:crypto`, + * `node:process` and the rest of the host surface. Same function, narrower + * resolution path. + */ + +export { isComponentName } from "./src/component-name.ts"; diff --git a/packages/core/deno.json b/packages/core/deno.json index ac93c2365..1b62eeae6 100644 --- a/packages/core/deno.json +++ b/packages/core/deno.json @@ -3,6 +3,9 @@ "version": "0.9.0", "exports": { ".": "./mod.ts", + "./canonicalize": "./canonicalize.ts", + "./component-name": "./component-name.ts", + "./document-target": "./document-target.ts", "./host": "./host.ts" }, "imports": { diff --git a/packages/core/document-target.ts b/packages/core/document-target.ts new file mode 100644 index 000000000..202533cb8 --- /dev/null +++ b/packages/core/document-target.ts @@ -0,0 +1,13 @@ +/** + * @module + * + * How an exact document target is spelled, for runtimes that cannot load a + * Markdown parser. + * + * `isCanonicalDocumentTarget` is already public from the package root under + * that fuller name. This subpath selects the spelling predicate without the + * catalog and selector machinery behind it, and without the root barrel's host + * surface. Same function, narrower resolution path. + */ + +export { isCanonicalTarget as isCanonicalDocumentTarget } from "./src/document-target-spelling.ts"; diff --git a/packages/core/package.json b/packages/core/package.json index 6f80ed609..f787a04cb 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -5,6 +5,9 @@ "type": "module", "exports": { ".": "./mod.ts", + "./canonicalize": "./canonicalize.ts", + "./component-name": "./component-name.ts", + "./document-target": "./document-target.ts", "./host": "./host.ts" }, "dependencies": { diff --git a/packages/core/src/canonical.ts b/packages/core/src/canonical.ts index 473065517..38b7c3f6e 100644 --- a/packages/core/src/canonical.ts +++ b/packages/core/src/canonical.ts @@ -7,6 +7,11 @@ * replay would stop matching. Sorting the keys before serializing is what makes * the name depend on what the value *is*. * + * The canonicalization itself lives in `./canonicalize.ts`, which names no + * host; this module adds the digest, which needs one. Both remain exported + * from the package root, and `@executablemd/core/canonicalize` publishes the + * pure half for consumers that cannot load a Node builtin. + * * Callers compose their own identity and hash it here, rather than handing over * a shape this module defines: what belongs in a fingerprint is a property of * the thing being identified, and two callers disagree about it. `` @@ -15,31 +20,10 @@ */ import { createHash } from "node:crypto"; -import type { Json, JsonObject } from "./types.ts"; +import { canonicalize } from "./canonicalize.ts"; +import type { Json } from "./types.ts"; -/** The same value with every object's keys in sorted order. */ -export function canonicalize(value: Json): Json { - if (Array.isArray(value)) { - return value.map(canonicalize); - } - if (value === null || typeof value !== "object") { - return value; - } - const sorted: JsonObject = {}; - for (const key of Object.keys(value).sort()) { - // Defined rather than assigned: `sorted[key] = …` reaches - // `Object.prototype`'s setter for `__proto__` and drops the key on Node and - // Bun, so a schema declaring that name would canonicalize differently - // depending on where it ran. - Object.defineProperty(sorted, key, { - value: canonicalize(value[key]), - enumerable: true, - writable: true, - configurable: true, - }); - } - return sorted; -} +export { canonicalize }; /** The SHA-256 of a canonicalized value, as hex. */ export function canonicalFingerprint(value: Json): string { diff --git a/packages/core/src/canonicalize.ts b/packages/core/src/canonicalize.ts new file mode 100644 index 000000000..930688e29 --- /dev/null +++ b/packages/core/src/canonicalize.ts @@ -0,0 +1,41 @@ +/** + * A stable name for a JSON value, with no host behind it. + * + * Two values that differ only in key order are the same value, and + * `JSON.stringify` would otherwise make them different names — so a document + * that reordered a schema's properties would look like a different question and + * replay would stop matching. Sorting the keys before serializing is what makes + * the name depend on what the value *is*. + * + * This is a leaf on purpose. The operation is pure arithmetic over a JSON + * value, and it sat beside `canonicalFingerprint()`, which reaches + * `node:crypto` — so a runtime that has no Node builtins could not import one + * without the other, and a Cloudflare Worker that needs to canonicalize a + * record could not do it at all. Nothing here imports anything but a type. + */ + +import type { Json, JsonObject } from "./types.ts"; + +/** The same value with every object's keys in sorted order. */ +export function canonicalize(value: Json): Json { + if (Array.isArray(value)) { + return value.map(canonicalize); + } + if (value === null || typeof value !== "object") { + return value; + } + const sorted: JsonObject = {}; + for (const key of Object.keys(value).sort()) { + // Defined rather than assigned: `sorted[key] = …` reaches + // `Object.prototype`'s setter for `__proto__` and drops the key on Node and + // Bun, so a schema declaring that name would canonicalize differently + // depending on where it ran. + Object.defineProperty(sorted, key, { + value: canonicalize(value[key]), + enumerable: true, + writable: true, + configurable: true, + }); + } + return sorted; +} diff --git a/packages/core/src/component-name.ts b/packages/core/src/component-name.ts new file mode 100644 index 000000000..634e7d181 --- /dev/null +++ b/packages/core/src/component-name.ts @@ -0,0 +1,18 @@ +/** + * How a document spells a component name, with nothing else behind it. + * + * The grammar registration is held to, offered as a predicate so a host + * deciding what a name may be does not restate it. It answers about spelling + * alone: a name that passes may still be structural syntax, a reserved + * registration, or a name nothing supplies. + * + * A leaf, so a consumer validating a retained name — a stored workflow + * definition checking its component bundle — does not load the registration + * machinery, or the engine behind it, to ask one question about a string. + */ + +const SEGMENT = /^[A-Z][A-Za-z0-9_]*$/; + +export function isComponentName(name: string): boolean { + return name.length > 0 && name.split(".").every((segment) => SEGMENT.test(segment)); +} diff --git a/packages/core/src/components/registration.ts b/packages/core/src/components/registration.ts index 10b34314d..4f3f77420 100644 --- a/packages/core/src/components/registration.ts +++ b/packages/core/src/components/registration.ts @@ -17,6 +17,9 @@ import type { Context, Operation } from "effection"; import { Component } from "../component-api.ts"; import { updateOwn } from "../scope-local.ts"; import { RESERVED_STRUCTURAL } from "../structural.ts"; +import { isComponentName } from "../component-name.ts"; + +export { isComponentName }; import { compilePropsSchema, compileReturnsSchema } from "../validate.ts"; import type { ComponentRegistry, @@ -79,20 +82,6 @@ const OwnContributions: Context = createContext( new Map(), ); -const SEGMENT = /^[A-Z][A-Za-z0-9_]*$/; - -/** - * Whether `name` is spelled the way a document writes a component name. - * - * The grammar registration is held to, offered as a predicate so a host - * deciding what a name may be does not restate it. It answers about spelling - * alone: a name that passes may still be structural syntax, a reserved - * registration, or a name nothing supplies. - */ -export function isComponentName(name: string): boolean { - return name.length > 0 && name.split(".").every((segment) => SEGMENT.test(segment)); -} - function kindOf(registration: ComponentRegistration): Kind { return registration.reserved === true ? "reserved" : "default"; } diff --git a/packages/core/src/document-target-spelling.ts b/packages/core/src/document-target-spelling.ts new file mode 100644 index 000000000..370892b76 --- /dev/null +++ b/packages/core/src/document-target-spelling.ts @@ -0,0 +1,123 @@ +/** + * How an exact document target is spelled, with no host and no parser behind + * it. + * + * Percent-encoding a label, decoding one, normalizing it, and asking whether a + * fragment is already canonical are string arithmetic. They live apart from the + * catalog and selector machinery that uses them because a consumer that only + * needs to validate a retained target — a stored workflow definition checking + * the one it kept — should not have to load a Markdown parser, or a runtime + * that has one, to do it. + */ + +const UNRESERVED = /^[A-Za-z0-9\-._~]$/; +const HEX = /^[0-9A-Fa-f]$/; + +const ENCODER = new TextEncoder(); + +function encodeCharacter(character: string): string { + let encoded = ""; + for (const byte of ENCODER.encode(character)) { + encoded += `%${byte.toString(16).toUpperCase().padStart(2, "0")}`; + } + return encoded; +} + +/** + * Percent-encode one canonical label. Everything outside RFC 3986's unreserved + * set is escaped, so `/`, `*`, `#`, and `%` inside a heading cannot be read as + * hierarchy or operator syntax. + */ +export function encodeTargetLabel(label: string): string { + let encoded = ""; + for (const character of label) { + encoded += UNRESERVED.test(character) ? character : encodeCharacter(character); + } + return encoded; +} + +/** + * Percent-encode a decoded filesystem path. Separators survive as raw `/`; a + * `/` that is part of a filename cannot be told apart from one afterwards, so + * this is a formatter for paths the caller already holds, not a round trip. + */ +export function encodeDocumentPath(path: string): string { + let encoded = ""; + for (const character of path) { + encoded += + character === "/" || UNRESERVED.test(character) ? character : encodeCharacter(character); + } + return encoded; +} + +/** + * Decode one percent-encoded chunk, or `undefined` when it is not decodable. + * + * Malformed escapes, byte sequences that are not UTF-8, and NUL are all + * refused rather than repaired: a selector that cannot be read exactly is not a + * selector this can match against. `+` is an ordinary character — this is URI + * path syntax, not a form encoding. + */ +export function decodePercentEncoded(text: string): string | undefined { + const characters = Array.from(text); + const bytes: number[] = []; + for (let index = 0; index < characters.length; index++) { + const character = characters[index]!; + if (character !== "%") { + for (const byte of ENCODER.encode(character)) { + bytes.push(byte); + } + continue; + } + const high = characters[index + 1]; + const low = characters[index + 2]; + if (high === undefined || low === undefined || !HEX.test(high) || !HEX.test(low)) { + return undefined; + } + bytes.push(Number.parseInt(`${high}${low}`, 16)); + index += 2; + } + try { + // `ignoreBOM` is stated rather than defaulted: it is already false + // everywhere this runs, and Cloudflare's own type declares both options + // required, so saying it keeps one spelling readable to every runtime. + const decoded = new TextDecoder("utf-8", { fatal: true, ignoreBOM: false }).decode( + new Uint8Array(bytes), + ); + return decoded.includes("\u0000") ? undefined : decoded; + } catch { + return undefined; + } +} + +/** + * The canonical form of rendered heading text: NFC, every run of Unicode + * whitespace collapsed to one ASCII space, trimmed, case preserved. + */ +export function normalizeLabel(text: string): string { + return text.normalize("NFC").replace(/\s+/gu, " ").trim(); +} + +/** + * Whether a fragment is already an exact canonical target. + * + * A level is canonical only when decoding it, normalizing the label, and + * re-encoding that label reproduce the level byte for byte. Requiring the whole + * round trip is what makes this total: it rejects a wildcard operator, an empty + * level, a lowercase escape, a raw `#`, an NFD spelling, a tab, and leading, + * trailing, or uncollapsed whitespace without naming any of them, because none + * of them is what this module would have written. + */ +export function isCanonicalTarget(target: string): boolean { + if (target.length === 0) { + return false; + } + return target.split("/").every((level) => { + const decoded = decodePercentEncoded(level); + if (decoded === undefined || decoded.length === 0) { + return false; + } + const label = normalizeLabel(decoded); + return label === decoded && encodeTargetLabel(label) === level; + }); +} diff --git a/packages/core/src/document-targets.ts b/packages/core/src/document-targets.ts index 6567a8519..6a5897c97 100644 --- a/packages/core/src/document-targets.ts +++ b/packages/core/src/document-targets.ts @@ -28,6 +28,21 @@ import { remark } from "remark"; import { toString as mdastToString } from "mdast-util-to-string"; import type { ComponentSpan } from "./scanner.ts"; +import { + decodePercentEncoded, + encodeDocumentPath, + encodeTargetLabel, + isCanonicalTarget, + normalizeLabel, +} from "./document-target-spelling.ts"; + +export { + decodePercentEncoded, + encodeDocumentPath, + encodeTargetLabel, + isCanonicalTarget, + normalizeLabel, +}; /** A half-open slice of the original document body. */ export interface SourceRange { @@ -518,113 +533,6 @@ function sameList(left: readonly string[], right: readonly string[]): boolean { return left.length === right.length && left.every((item, index) => item === right[index]); } -const UNRESERVED = /^[A-Za-z0-9\-._~]$/; -const HEX = /^[0-9A-Fa-f]$/; - -const ENCODER = new TextEncoder(); - -function encodeCharacter(character: string): string { - let encoded = ""; - for (const byte of ENCODER.encode(character)) { - encoded += `%${byte.toString(16).toUpperCase().padStart(2, "0")}`; - } - return encoded; -} - -/** - * Percent-encode one canonical label. Everything outside RFC 3986's unreserved - * set is escaped, so `/`, `*`, `#`, and `%` inside a heading cannot be read as - * hierarchy or operator syntax. - */ -export function encodeTargetLabel(label: string): string { - let encoded = ""; - for (const character of label) { - encoded += UNRESERVED.test(character) ? character : encodeCharacter(character); - } - return encoded; -} - -/** - * Percent-encode a decoded filesystem path. Separators survive as raw `/`; a - * `/` that is part of a filename cannot be told apart from one afterwards, so - * this is a formatter for paths the caller already holds, not a round trip. - */ -export function encodeDocumentPath(path: string): string { - let encoded = ""; - for (const character of path) { - encoded += - character === "/" || UNRESERVED.test(character) ? character : encodeCharacter(character); - } - return encoded; -} - -/** - * Decode one percent-encoded chunk, or `undefined` when it is not decodable. - * - * Malformed escapes, byte sequences that are not UTF-8, and NUL are all - * refused rather than repaired: a selector that cannot be read exactly is not a - * selector this can match against. `+` is an ordinary character — this is URI - * path syntax, not a form encoding. - */ -export function decodePercentEncoded(text: string): string | undefined { - const characters = Array.from(text); - const bytes: number[] = []; - for (let index = 0; index < characters.length; index++) { - const character = characters[index]!; - if (character !== "%") { - for (const byte of ENCODER.encode(character)) { - bytes.push(byte); - } - continue; - } - const high = characters[index + 1]; - const low = characters[index + 2]; - if (high === undefined || low === undefined || !HEX.test(high) || !HEX.test(low)) { - return undefined; - } - bytes.push(Number.parseInt(`${high}${low}`, 16)); - index += 2; - } - try { - const decoded = new TextDecoder("utf-8", { fatal: true }).decode(new Uint8Array(bytes)); - return decoded.includes("\u0000") ? undefined : decoded; - } catch { - return undefined; - } -} - -/** - * The canonical form of rendered heading text: NFC, every run of Unicode - * whitespace collapsed to one ASCII space, trimmed, case preserved. - */ -export function normalizeLabel(text: string): string { - return text.normalize("NFC").replace(/\s+/gu, " ").trim(); -} - -/** - * Whether a fragment is already an exact canonical target. - * - * A level is canonical only when decoding it, normalizing the label, and - * re-encoding that label reproduce the level byte for byte. Requiring the whole - * round trip is what makes this total: it rejects a wildcard operator, an empty - * level, a lowercase escape, a raw `#`, an NFD spelling, a tab, and leading, - * trailing, or uncollapsed whitespace without naming any of them, because none - * of them is what this module would have written. - */ -export function isCanonicalTarget(target: string): boolean { - if (target.length === 0) { - return false; - } - return target.split("/").every((level) => { - const decoded = decodePercentEncoded(level); - if (decoded === undefined || decoded.length === 0) { - return false; - } - const label = normalizeLabel(decoded); - return label === decoded && encodeTargetLabel(label) === level; - }); -} - type LevelPart = | { readonly kind: "literal"; readonly text: string } | { readonly kind: "wildcard" }; diff --git a/packages/core/tests/canonicalize.test.ts b/packages/core/tests/canonicalize.test.ts new file mode 100644 index 000000000..a2d8b2d06 --- /dev/null +++ b/packages/core/tests/canonicalize.test.ts @@ -0,0 +1,63 @@ +/** + * The pure half of canonicalization, and the host-capable half beside it. + * + * `canonicalize()` moved into a leaf so a runtime without Node builtins can + * reach it — a Cloudflare Worker validating a retained record needs the key + * ordering and not the digest. The risk in that move is two implementations + * that drift, so what is asserted here is that there is exactly one: the + * package root and the subpath answer identically, and the fingerprint that + * composes over it is unchanged. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { canonicalize as fromRoot, canonicalFingerprint } from "@executablemd/core"; +import { canonicalize as fromSubpath } from "@executablemd/core/canonicalize"; +import { isComponentName as componentNameFromRoot } from "@executablemd/core"; +import { isComponentName as componentNameFromSubpath } from "@executablemd/core/component-name"; +import { isCanonicalDocumentTarget as targetFromRoot } from "@executablemd/core"; +import { isCanonicalDocumentTarget as targetFromSubpath } from "@executablemd/core/document-target"; +import type { Json } from "@executablemd/core"; + +/** Values chosen for the properties canonicalization is about. */ +const VALUES: Json[] = [ + null, + 0, + "text", + [3, 1, 2], + { b: 1, a: 2 }, + { outer: { z: [{ y: 1, x: 2 }], a: null } }, + // The name whose ordinary assignment would reach `Object.prototype`. + { ["__proto__"]: { polluted: true }, after: 1 }, +]; + +describe("canonicalization through both paths", () => { + it("answers identically from the package root and the subpath", function* () { + for (const value of VALUES) { + expect(JSON.stringify(fromSubpath(value))).toEqual(JSON.stringify(fromRoot(value))); + } + }); + + it("still sorts keys and leaves arrays in order", function* () { + expect(JSON.stringify(fromSubpath({ b: 1, a: 2 }))).toEqual('{"a":2,"b":1}'); + expect(JSON.stringify(fromSubpath([3, 1, 2]))).toEqual("[3,1,2]"); + }); + + it("keeps the fingerprint composing over the same ordering", function* () { + // The digest is the half that needs a host; it is unchanged by the split. + expect(canonicalFingerprint({ b: 1, a: 2 })).toEqual(canonicalFingerprint({ a: 2, b: 1 })); + expect(canonicalFingerprint({ a: 1 })).not.toEqual(canonicalFingerprint({ a: 2 })); + expect(canonicalFingerprint({ a: 1 })).toMatch(/^[0-9a-f]{64}$/); + }); +}); + +describe("the other predicates a retained descriptor validates with", () => { + it("answers identically from the package root and the subpath", function* () { + for (const name of ["Repository", "Ns.Sub", "lower", "", "9Bad", "A_1"]) { + expect(componentNameFromSubpath(name)).toEqual(componentNameFromRoot(name)); + } + for (const target of ["Heading", "A/B", "", "a%2Fb", "Lower case", "%2f", "Tab\there"]) { + expect(targetFromSubpath(target)).toEqual(targetFromRoot(target)); + } + }); +}); diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 154e9acbd..d36a80651 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -14,6 +14,11 @@ * would be a runner deciding what the owner does. */ +import { + type DocumentExecutionCompletion, + parseDocumentExecutionCompletion, +} from "../storage/record.ts"; + /** The commands a runner may send. */ export type CommandName = "frontier" | "materialize" | "commit" | "settle"; @@ -23,7 +28,6 @@ export type CommandRefusal = | "unknown-command" | "unknown-member" | "malformed-member" - | "duplicate-conflict" | "too-large"; export class CommandError extends Error { @@ -69,10 +73,19 @@ export interface CommitCommand extends CommandEnvelope { readonly events: readonly string[]; } -/** Publish the run's resulting status. */ +/** + * Publish how a document execution ended, and what the run becomes. + * + * The completion is the shared provider-neutral record, parsed with the shared + * parser rather than a private approximation — the owner and the local host + * have to agree about what a completion *is*, and two readers of one shape is + * how they stop agreeing. The expected root is carried so the owner can refuse + * a settlement proposed against a frontier that has moved. + */ export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; - readonly status: string; + readonly completion: DocumentExecutionCompletion; + readonly expectedWorkspaceRootId: string; } export interface ContentChunk { @@ -102,7 +115,7 @@ const MEMBERS: Record = { "proposedWorkspaceRootId", "events", ], - settle: [...ENVELOPE, "status"], + settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], }; function object(value: unknown): Record { @@ -185,7 +198,19 @@ export function parseCommand(raw: string): RunnerCommand { return { id, command, workspaceRootId: text(members, "workspaceRootId") }; } if (command === "settle") { - return { id, command, status: text(members, "status") }; + // The shared parser decides what a completion is. Its failure becomes this + // transport's own closed refusal: the parser's message names members and + // values a request supplied, and none of that belongs on the wire. + const completion = parseDocumentExecutionCompletion(members["completion"]); + if (!completion.ok) { + throw new CommandError("malformed-member"); + } + return { + id, + command, + completion: completion.value, + expectedWorkspaceRootId: text(members, "expectedWorkspaceRootId"), + }; } const expectedJournalEventId = members["expectedJournalEventId"]; if (expectedJournalEventId !== null && typeof expectedJournalEventId !== "string") { diff --git a/packages/workflow/src/storage/definition.ts b/packages/workflow/src/storage/definition.ts index cda336a5b..935e038b7 100644 --- a/packages/workflow/src/storage/definition.ts +++ b/packages/workflow/src/storage/definition.ts @@ -16,7 +16,11 @@ */ import { Err, Ok, type Result } from "effection"; -import { isCanonicalDocumentTarget, isComponentName } from "@executablemd/core"; +// The node-free subpaths: this module is reached from a Cloudflare Worker, and +// the package root's barrel resolves host modules a Worker cannot load. Both +// predicates are the same public functions, selected through a narrower path. +import { isCanonicalDocumentTarget } from "@executablemd/core/document-target"; +import { isComponentName } from "@executablemd/core/component-name"; import type { Json } from "@executablemd/durable-streams"; import { WorkflowDefinitionError } from "./errors.ts"; import { diff --git a/packages/workflow/src/storage/record.ts b/packages/workflow/src/storage/record.ts index 37c78d0f5..c626fa958 100644 --- a/packages/workflow/src/storage/record.ts +++ b/packages/workflow/src/storage/record.ts @@ -14,7 +14,10 @@ */ import { Err, Ok, type Result } from "effection"; -import { canonicalize } from "@executablemd/core"; +// The node-free subpath: this module is reached from a Cloudflare Worker, and +// the package root's barrel resolves `node:crypto` and the rest of the host +// surface. Same function, narrower resolution path. +import { canonicalize } from "@executablemd/core/canonicalize"; import type { Json } from "@executablemd/durable-streams"; import type { WorkflowDefinition } from "./definition.ts"; import { WorkflowRequestError } from "./errors.ts"; diff --git a/packages/workflow/tests/cloudflare/settle-parser.vitest.ts b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts new file mode 100644 index 000000000..82ff195e9 --- /dev/null +++ b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts @@ -0,0 +1,96 @@ +/** + * The settle request, parsed inside a real Worker. + * + * The point of running this on workerd rather than portably is the import + * graph. `parseDocumentExecutionCompletion()` is shared code that reaches + * `canonicalize` and two spelling predicates in `@executablemd/core`, and until + * those were published from node-free subpaths that graph pulled `node:crypto` + * and `node:process` — which typechecks anywhere except the runtime that has to + * run it. A test that only proved the parser worked would have proved nothing + * about that; this one loads it where a Node builtin is genuinely absent. + * + * The owner's revalidation of acquisition, root and execution belongs to the + * checkpoint where a lifecycle transaction exists. What is asserted here is the + * private request contract: what a settle command has to be to be read at all. + */ + +import { describe, expect, it } from "vitest"; +import { CommandError, parseCommand } from "../../src/cloudflare/commands.ts"; + +const ROOT = "9f2c4b6a8d0e1f23456789abcdef0123456789abcdef0123456789abcdef0123"; + +function settle(overrides: Record = {}): string { + return JSON.stringify({ + id: "s1", + command: "settle", + completion: { executionId: "execution-1", status: "completed" }, + expectedWorkspaceRootId: ROOT, + ...overrides, + }); +} + +/** The refusal category, or the command name when it was read. */ +function read(raw: string): string { + try { + return parseCommand(raw).command; + } catch (error) { + return error instanceof CommandError ? error.refusal : "unexpected"; + } +} + +describe("a settle command", () => { + it("reads a complete completion through the shared parser", () => { + const command = parseCommand(settle()); + expect(command.command).toBe("settle"); + if (command.command !== "settle") { + throw new Error("expected a settle command"); + } + expect(command.completion).toEqual({ executionId: "execution-1", status: "completed" }); + expect(command.expectedWorkspaceRootId).toBe(ROOT); + }); + + it("reads a completion carrying a stop reason", () => { + const command = parseCommand( + settle({ + completion: { + executionId: "execution-1", + status: "failed", + reason: { kind: "host", code: "settlement-refused" }, + }, + }), + ); + if (command.command !== "settle") { + throw new Error("expected a settle command"); + } + expect(command.completion.reason).toEqual({ kind: "host", code: "settlement-refused" }); + }); + + it("refuses a completion the shared parser will not read", () => { + // Each of these is refused by the shared contract rather than by a private + // approximation of it, and each becomes this transport's own closed + // refusal rather than carrying the parser's message onto the wire. + expect(read(settle({ completion: { status: "completed" } }))).toBe("malformed-member"); + expect(read(settle({ completion: { executionId: "", status: "completed" } }))).toBe( + "malformed-member", + ); + expect(read(settle({ completion: { executionId: "e", status: "invented" } }))).toBe( + "malformed-member", + ); + expect(read(settle({ completion: { executionId: "e", status: "failed", reason: 7 } }))).toBe( + "malformed-member", + ); + expect(read(settle({ completion: "not an object" }))).toBe("malformed-member"); + expect(read(settle({ completion: undefined }))).toBe("malformed-member"); + }); + + it("refuses a missing or malformed expected root", () => { + expect(read(settle({ expectedWorkspaceRootId: undefined }))).toBe("malformed-member"); + expect(read(settle({ expectedWorkspaceRootId: "" }))).toBe("malformed-member"); + expect(read(settle({ expectedWorkspaceRootId: 1 }))).toBe("malformed-member"); + }); + + it("refuses a member the command does not declare", () => { + expect(read(settle({ status: "completed" }))).toBe("unknown-member"); + expect(read(settle({ somethingElse: true }))).toBe("unknown-member"); + }); +}); diff --git a/packages/workflow/tsconfig.cloudflare.json b/packages/workflow/tsconfig.cloudflare.json index 4e35f34ec..d78979841 100644 --- a/packages/workflow/tsconfig.cloudflare.json +++ b/packages/workflow/tsconfig.cloudflare.json @@ -6,8 +6,6 @@ "lib": ["ES2022"], "types": ["@cloudflare/workers-types", "@cloudflare/vitest-plugin/types"], "strict": true, - "exactOptionalPropertyTypes": true, - "noUncheckedIndexedAccess": true, "allowImportingTsExtensions": true, "allowJs": true, "checkJs": false, From 89272a60c6d2c06728a5e00dcba932f039855b2a Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 2 Sep 2026 22:27:50 -0400 Subject: [PATCH 22/81] =?UTF-8?q?=F0=9F=9B=A1=EF=B8=8F=20Harden=20the=20to?= =?UTF-8?q?ken,=20own=20the=20transaction,=20type=20the=20answers=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit D1 items 2 through 6 from `.vscode/698/698-c-4.md`. **Token.** `verifyToken()` accepted a token with no `exp`, and a non-numeric `exp` or `nbf`, because it checked those claims only when they happened to be numbers — so omitting one was treated as satisfying it. All three of `exp`, `iat` and `nbf` are now required finite integer NumericDates. The expiration boundary is expired, per RFC 7519 §4.1.4's "before". An `iat` in the future beyond tolerance is not-yet-valid rather than accepted. The header must say `typ` is a JWT, and `kid` must name exactly one configured key: the old filter fell back to an unkeyed candidate, so an unrecognized key id still got a signature check against whatever else was configured. The token and its segments are bounded before anything is decoded, and skew is bounded at both ends — negative would reject tokens for being on time, and unbounded is indistinguishable from not checking. **Transaction ownership.** The module-global `let open` is gone. It was shared by every Durable Object in an isolate, so one object's transaction would refuse another's. The ruling suggested a `WeakSet` keyed by storage; this repository's `local/no-module-scoped-registry` forbids that too, and for a related reason. So the claim belongs to the object: `OwnerTransactions` is created and held by the owner, its lifetime is the object's, and no other object can see it. Same guarantee, no process-lifetime table. **Settle.** `{ status: string }` becomes a closed request carrying `completion: DocumentExecutionCompletion` and `expectedWorkspaceRootId`, with the completion read by the shared `parseDocumentExecutionCompletion()` — a failed parse becomes this transport's own `malformed-member` without carrying the parser's message, which names members a request supplied. **Answers.** `ask()` now takes the parser for its own success value and returns `OwnerAnswer`. `unknown` exists only at the JSON boundary; a value the command's parser cannot read fails the channel closed like any other disagreement about what completed, and a refusal is delivered without consulting the parser at all. The generic stays inside the closure the request built, so the reader settles an answer without asserting what it is. **Removed claims.** `duplicate-conflict` is gone until the mechanism that produces it exists. `perform()` stays abstract and the echo stays test-only. Every checked-then-asserted `Record` became `Object.entries` narrowing, and the one double assertion in the tests became a helper that passes `unknown` through. One documented assertion remains, in `storage.ts`, bridging two type declarations of the same runtime rows. Forty-eight workerd assertions and thirty portable ones. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/cloudflare.ts | 6 + .../workflow/src/cloudflare/acquisition.ts | 8 +- packages/workflow/src/cloudflare/admission.ts | 22 ++-- packages/workflow/src/cloudflare/commands.ts | 23 ++-- .../src/cloudflare/owner-transaction.ts | 109 ++++++++++------- packages/workflow/src/cloudflare/owner.ts | 11 +- .../workflow/src/cloudflare/recognition.ts | 10 +- packages/workflow/src/cloudflare/token.ts | 101 +++++++++++++--- packages/workflow/src/remote/client.ts | 112 ++++++++++++++---- .../cloudflare/executor-acquisition.vitest.ts | 64 ++++++++++ .../tests/cloudflare/owner-storage.vitest.ts | 28 +++++ .../cloudflare/support/executor-object.ts | 8 +- .../tests/cloudflare/support/owner-object.ts | 71 ++++++++++- packages/workflow/tests/remote-client.test.ts | 91 ++++++++++++-- .../workflow/tests/remote-transaction.test.ts | 14 ++- 15 files changed, 549 insertions(+), 129 deletions(-) diff --git a/packages/workflow/cloudflare.ts b/packages/workflow/cloudflare.ts index 270aef5c8..946e6017d 100644 --- a/packages/workflow/cloudflare.ts +++ b/packages/workflow/cloudflare.ts @@ -53,5 +53,11 @@ export type { AcquisitionAttachment, AcquisitionRefusal } from "./src/cloudflare export { CommandError } from "./src/cloudflare/commands.ts"; export type { CommandRefusal, CommandResult, RunnerCommand } from "./src/cloudflare/commands.ts"; +export { + OwnerTransactionClosedError, + OwnerTransactionNestedError, + OwnerTransactions, +} from "./src/cloudflare/owner-transaction.ts"; + export { WorkflowObjectStorageError } from "./src/cloudflare/recognition.ts"; export type { RecognitionFailure } from "./src/cloudflare/recognition.ts"; diff --git a/packages/workflow/src/cloudflare/acquisition.ts b/packages/workflow/src/cloudflare/acquisition.ts index 38601fd17..a8dda2376 100644 --- a/packages/workflow/src/cloudflare/acquisition.ts +++ b/packages/workflow/src/cloudflare/acquisition.ts @@ -60,12 +60,12 @@ function attachmentOf(socket: WebSocket): AcquisitionAttachment | undefined { if (value === null || typeof value !== "object" || Array.isArray(value)) { return undefined; } - const members = value as Record; - if (members["kind"] !== "executor") { + const members: Map = new Map(Object.entries(value)); + if (members.get("kind") !== "executor") { return undefined; } - const runId = members["runId"]; - const acquisitionId = members["acquisitionId"]; + const runId = members.get("runId"); + const acquisitionId = members.get("acquisitionId"); if (typeof runId !== "string" || typeof acquisitionId !== "string") { return undefined; } diff --git a/packages/workflow/src/cloudflare/admission.ts b/packages/workflow/src/cloudflare/admission.ts index bd6f1cfdb..53a7ee687 100644 --- a/packages/workflow/src/cloudflare/admission.ts +++ b/packages/workflow/src/cloudflare/admission.ts @@ -112,20 +112,16 @@ function admitClaims(policy: AdmissionPolicy, claims: ActionsClaims): void { } /** Read a claim set out of a verified payload. */ -function parseClaims(payload: unknown): ActionsClaims { - if (payload === null || typeof payload !== "object" || Array.isArray(payload)) { - throw new AdmissionError("token-malformed"); - } - const members = payload as Record; +function parseClaims(payload: Map): ActionsClaims { return { - iss: members["iss"], - aud: members["aud"], - repository_id: members["repository_id"], - repository_owner_id: members["repository_owner_id"], - event_name: members["event_name"], - workflow_ref: members["workflow_ref"], - workflow_sha: members["workflow_sha"], - job_workflow_ref: members["job_workflow_ref"], + iss: payload.get("iss"), + aud: payload.get("aud"), + repository_id: payload.get("repository_id"), + repository_owner_id: payload.get("repository_owner_id"), + event_name: payload.get("event_name"), + workflow_ref: payload.get("workflow_ref"), + workflow_sha: payload.get("workflow_sha"), + job_workflow_ref: payload.get("job_workflow_ref"), }; } diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index d36a80651..228e48714 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -118,23 +118,24 @@ const MEMBERS: Record = { settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], }; -function object(value: unknown): Record { +function object(value: unknown): Map { if (value === null || typeof value !== "object" || Array.isArray(value)) { throw new CommandError("not-an-object"); } - return value as Record; + const members: Map = new Map(Object.entries(value)); + return members; } -function text(members: Record, key: string): string { - const value = members[key]; +function text(members: Map, key: string): string { + const value = members.get(key); if (typeof value !== "string" || value === "") { throw new CommandError("malformed-member"); } return value; } -function closed(members: Record, allowed: readonly string[]): void { - for (const key of Object.keys(members)) { +function closed(members: Map, allowed: readonly string[]): void { + for (const key of members.keys()) { if (!allowed.includes(key)) { throw new CommandError("unknown-member"); } @@ -180,7 +181,7 @@ export function parseCommand(raw: string): RunnerCommand { } const members = object(decoded); const id = text(members, "id"); - const command = members["command"]; + const command = members.get("command"); if ( command !== "frontier" && command !== "materialize" && @@ -201,7 +202,7 @@ export function parseCommand(raw: string): RunnerCommand { // The shared parser decides what a completion is. Its failure becomes this // transport's own closed refusal: the parser's message names members and // values a request supplied, and none of that belongs on the wire. - const completion = parseDocumentExecutionCompletion(members["completion"]); + const completion = parseDocumentExecutionCompletion(members.get("completion")); if (!completion.ok) { throw new CommandError("malformed-member"); } @@ -212,7 +213,7 @@ export function parseCommand(raw: string): RunnerCommand { expectedWorkspaceRootId: text(members, "expectedWorkspaceRootId"), }; } - const expectedJournalEventId = members["expectedJournalEventId"]; + const expectedJournalEventId = members.get("expectedJournalEventId"); if (expectedJournalEventId !== null && typeof expectedJournalEventId !== "string") { throw new CommandError("malformed-member"); } @@ -221,9 +222,9 @@ export function parseCommand(raw: string): RunnerCommand { command, expectedWorkspaceRootId: text(members, "expectedWorkspaceRootId"), expectedJournalEventId, - content: chunks(members["content"]), + content: chunks(members.get("content")), proposedWorkspaceRootId: text(members, "proposedWorkspaceRootId"), - events: events(members["events"]), + events: events(members.get("events")), }; } diff --git a/packages/workflow/src/cloudflare/owner-transaction.ts b/packages/workflow/src/cloudflare/owner-transaction.ts index 12cd09bde..b91a3ec7c 100644 --- a/packages/workflow/src/cloudflare/owner-transaction.ts +++ b/packages/workflow/src/cloudflare/owner-transaction.ts @@ -62,8 +62,39 @@ export interface OwnerTransaction { readonly dofs: DofsDatabase; } -/** Whether an owner transaction is currently open on this object. */ -let open = false; +/** + * One Durable Object's claim on its own storage. + * + * Owned by the object rather than by this module. A module-scoped flag would be + * shared by every object in an isolate, so one object's transaction would + * refuse another's; a module-scoped registry keyed by storage would fix that + * and still be a process-lifetime table this package's rules do not allow. An + * instance the object creates and holds says the same thing without either + * problem: the gate's lifetime is the object's, and no other object can see it. + */ +export class OwnerTransactions { + #open = false; + + /** + * Run `body` inside one real `ctx.storage.transactionSync()`. + * + * `body` must complete synchronously. Nothing may await, suspend, hold a + * cursor, wait on a WebSocket or reach the runner from inside it: the runtime + * requires the callback to finish before it can commit, and a value that + * arrived later would be applied to a transaction nobody is holding. + */ + run(storage: OwnerStorage, body: (transaction: OwnerTransaction) => T): T { + if (this.#open) { + throw new OwnerTransactionNestedError(); + } + this.#open = true; + try { + return enter(storage, body); + } finally { + this.#open = false; + } + } +} /** * Run `body` inside one real `ctx.storage.transactionSync()`. @@ -73,45 +104,41 @@ let open = false; * requires the callback to finish before it can commit, and a value that * arrived later would be applied to a transaction nobody is holding. */ -export function ownerTransaction( - storage: OwnerStorage, - body: (transaction: OwnerTransaction) => T, -): T { - if (open) { - throw new OwnerTransactionNestedError(); - } - open = true; - try { - return storage.transactionSync(() => { - let live = true; - const dofs = new DofsDatabase(dofsStorage(storage)); - // Fresh caches for this transaction alone. They are keyed by database, so - // an entry populated from rows this transaction may roll back would - // otherwise outlive it and be read by a later operation. +/** + * Enter the one real transaction and enlist DOFS inside it. + * + * Separate from the gate above so the claim and the runtime call are two + * things: the gate says whether this object may transact, and this says what a + * transaction is. + */ +function enter(storage: OwnerStorage, body: (transaction: OwnerTransaction) => T): T { + return storage.transactionSync(() => { + let live = true; + const dofs = new DofsDatabase(dofsStorage(storage)); + // Fresh caches for this transaction alone. They are keyed by database, so + // an entry populated from rows this transaction may roll back would + // otherwise outlive it and be read by a later operation. + clearResolveCache(dofs); + clearBlobCache(dofs); + // The substitution: DOFS believes it is opening a transaction, and runs in + // the one already open. Reentrancy inside DOFS becomes ordinary nesting of + // plain function calls, which is what the runtime allows. + Object.defineProperty(dofs, "transactionSync", { + value: (closure: () => R): R => { + if (!live) { + throw new OwnerTransactionClosedError(); + } + return closure(); + }, + configurable: false, + writable: false, + }); + try { + return body({ dofs }); + } finally { + live = false; clearResolveCache(dofs); clearBlobCache(dofs); - // The substitution: DOFS believes it is opening a transaction, and runs - // in the one already open. Reentrancy inside DOFS becomes ordinary - // nesting of plain function calls, which is what the runtime allows. - Object.defineProperty(dofs, "transactionSync", { - value: (closure: () => R): R => { - if (!live) { - throw new OwnerTransactionClosedError(); - } - return closure(); - }, - configurable: false, - writable: false, - }); - try { - return body({ dofs }); - } finally { - live = false; - clearResolveCache(dofs); - clearBlobCache(dofs); - } - }); - } finally { - open = false; - } + } + }); } diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index b8b31fdd1..3f68d0a5f 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -22,6 +22,7 @@ import { DurableObject } from "cloudflare:workers"; import type { Operation } from "effection"; +import { OwnerTransactions } from "./owner-transaction.ts"; import { acquireExecutor, type AcquisitionAttachment, @@ -99,6 +100,14 @@ export function refusalOf(error: unknown): string { * the identities it must satisfy. */ export abstract class WorkflowOwnerObject extends DurableObject { + /** + * This object's claim on its own storage. + * + * One per Durable Object, so a transaction here cannot refuse one in another + * object and no table outlives the object that owns it. + */ + protected readonly transactions: OwnerTransactions = new OwnerTransactions(); + protected abstract configuration(): OwnerConfiguration; /** This object's storage, as the shared modules expect to see it. */ @@ -167,7 +176,7 @@ export abstract class WorkflowOwnerObject extends DurableObject { open(runId: string, initializeRun: () => void): void { admitRunId(runId); if (isPristine(declaredObjects(this.owned))) { - initializeObject(this.owned, initializeRun); + initializeObject(this.owned, this.transactions, initializeRun); return; } recognizeObject(this.owned); diff --git a/packages/workflow/src/cloudflare/recognition.ts b/packages/workflow/src/cloudflare/recognition.ts index 13945e85f..7ee4bd6cd 100644 --- a/packages/workflow/src/cloudflare/recognition.ts +++ b/packages/workflow/src/cloudflare/recognition.ts @@ -25,7 +25,7 @@ import { type SchemaObject, } from "../sqlite/workflow-schema.ts"; import { isSchemaMarker, MARKER_SQL, MARKER_TABLE, readMarker } from "./marker.ts"; -import { ownerTransaction } from "./owner-transaction.ts"; +import type { OwnerTransactions } from "./owner-transaction.ts"; import type { OwnerStorage } from "./storage.ts"; /** Why storage could not be read as a version-1 workflow run. */ @@ -86,7 +86,11 @@ function markerRows(storage: OwnerStorage): Record[] { * ordering, so this is the code saying what the marker means: an identity claim * over a schema that is already complete. */ -export function initializeObject(storage: OwnerStorage, initializeRun: () => void): void { +export function initializeObject( + storage: OwnerStorage, + transactions: OwnerTransactions, + initializeRun: () => void, +): void { const objects = declaredObjects(storage); if (!isPristine(objects)) { throw new WorkflowObjectStorageError({ @@ -94,7 +98,7 @@ export function initializeObject(storage: OwnerStorage, initializeRun: () => voi detail: "it already holds objects and carries no workflow schema marker", }); } - ownerTransaction(storage, ({ dofs }) => { + transactions.run(storage, ({ dofs }) => { storage.sql.exec(SCHEMA_SQL); initializeDofsSchema(dofs, () => 0); initializeRun(); diff --git a/packages/workflow/src/cloudflare/token.ts b/packages/workflow/src/cloudflare/token.ts index 96a0d8fbe..1139ebf66 100644 --- a/packages/workflow/src/cloudflare/token.ts +++ b/packages/workflow/src/cloudflare/token.ts @@ -21,11 +21,15 @@ import { type Operation, until } from "effection"; export type TokenRefusal = | "token-absent" | "token-malformed" + | "token-too-large" | "unsupported-algorithm" + | "unsupported-type" | "unknown-key" | "bad-signature" + | "malformed-claims" | "expired" - | "not-yet-valid"; + | "not-yet-valid" + | "misconfigured-clock"; export class TokenError extends Error { override name = "TokenError"; @@ -44,6 +48,26 @@ export class TokenError extends Error { */ const SUPPORTED = "RS256"; +/** + * The longest token this reads at all, and the longest segment inside one. + * + * Bounded before anything is decoded, because decoding is the first work an + * unauthenticated caller can make this owner do. + */ +const MAX_TOKEN = 16 * 1024; +const MAX_SEGMENT = 8 * 1024; + +/** The most skew a deployment may configure. */ +const MAX_SKEW_SECONDS = 300; + +/** A NumericDate: a finite integer count of seconds. */ +function numericDate(value: unknown): number { + if (typeof value !== "number" || !Number.isFinite(value) || !Number.isInteger(value)) { + throw new TokenError("malformed-claims"); + } + return value; +} + /** What a deployment configures before any token can be verified. */ export interface TokenVerification { /** @@ -54,7 +78,13 @@ export interface TokenVerification { * a JWKS is for. */ readonly keys: readonly VerificationKey[]; - /** How much clock skew to tolerate, in seconds. */ + /** + * How much clock skew to tolerate, in seconds. + * + * Adapter policy, not a user setting and never a request field. Bounded above + * because a large tolerance is indistinguishable from not checking, and below + * because a negative one would reject tokens for being on time. + */ readonly skewSeconds: number; /** Now, in seconds since the epoch. Injected so a test can be exact. */ readonly now: () => number; @@ -84,11 +114,12 @@ function decodeSegment(segment: string): unknown { } } -function object(value: unknown): Record { +function object(value: unknown): Map { if (value === null || typeof value !== "object" || Array.isArray(value)) { throw new TokenError("token-malformed"); } - return value as Record; + const members: Map = new Map(Object.entries(value)); + return members; } function signatureBytes(segment: string): Uint8Array { @@ -111,14 +142,24 @@ function signatureBytes(segment: string): Uint8Array { export function* verifyToken( configured: TokenVerification, token: unknown, -): Operation> { +): Operation> { + const skew = configured.skewSeconds; + if (!Number.isFinite(skew) || skew < 0 || skew > MAX_SKEW_SECONDS) { + throw new TokenError("misconfigured-clock"); + } if (typeof token !== "string" || token === "") { throw new TokenError("token-absent"); } + if (token.length > MAX_TOKEN) { + throw new TokenError("token-too-large"); + } const parts = token.split("."); if (parts.length !== 3) { throw new TokenError("token-malformed"); } + if (parts.some((part) => part.length === 0 || part.length > MAX_SEGMENT)) { + throw new TokenError("token-malformed"); + } const [encodedHeader, encodedPayload, encodedSignature] = parts; if ( encodedHeader === undefined || @@ -129,19 +170,29 @@ export function* verifyToken( } const header = object(decodeSegment(encodedHeader)); - if (header["alg"] !== SUPPORTED) { + if (header.get("alg") !== SUPPORTED) { throw new TokenError("unsupported-algorithm"); } + // GitHub's Actions tokens carry `typ: "JWT"`. Requiring it is cheap and stops + // a token minted for another purpose from being read as one of these. + const type = header.get("typ"); + if (typeof type !== "string" || type.toUpperCase() !== "JWT") { + throw new TokenError("unsupported-type"); + } const signed = new TextEncoder().encode(`${encodedHeader}.${encodedPayload}`); const signature = signatureBytes(encodedSignature); - const keyId = header["kid"]; - // A `kid` narrows which key is tried; its absence means every configured key - // is a candidate. Either way only a configured key can verify anything. - const candidates = configured.keys.filter( - (key) => typeof keyId !== "string" || key.kid === undefined || key.kid === keyId, - ); - if (candidates.length === 0) { + // The token names exactly one configured key. Falling back to an unkeyed + // candidate when the id matched nothing would mean an unrecognized key id + // still got a signature check against whatever else was configured. + const keyId = header.get("kid"); + if (typeof keyId !== "string" || keyId === "") { + throw new TokenError("unknown-key"); + } + const candidates = configured.keys.filter((key) => key.kid === keyId); + if (candidates.length !== 1) { + // None means the id is unrecognized; more than one means the configuration + // cannot say which key that id is. throw new TokenError("unknown-key"); } @@ -168,12 +219,28 @@ export function* verifyToken( const payload = object(decodeSegment(encodedPayload)); const now = configured.now(); - const expiry = payload["exp"]; - if (typeof expiry === "number" && now > expiry + configured.skewSeconds) { + if (!Number.isFinite(now)) { + throw new TokenError("misconfigured-clock"); + } + + // All three are required. Checking a temporal claim only when it happens to + // be a number means a token that omits it is treated as one that satisfies + // it, which is the opposite of what the claim is for. + const expiry = numericDate(payload.get("exp")); + const issued = numericDate(payload.get("iat")); + const notBefore = numericDate(payload.get("nbf")); + + // RFC 7519 §4.1.4: the current time must be *before* the expiration, so the + // boundary itself is expired rather than the last valid instant. + if (now >= expiry + skew) { throw new TokenError("expired"); } - const notBefore = payload["nbf"]; - if (typeof notBefore === "number" && now + configured.skewSeconds < notBefore) { + if (now + skew < notBefore) { + throw new TokenError("not-yet-valid"); + } + if (now + skew < issued) { + // Issued in the future by more than the tolerance: the token and this clock + // disagree about when now is, and nothing here can tell which is wrong. throw new TokenError("not-yet-valid"); } return payload; diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index 3f2a215d0..8a988e1f9 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -40,11 +40,27 @@ export class OwnerLinkError extends Error { } } -/** What the owner answered, as the client reads it. */ -export type OwnerAnswer = - | { readonly outcome: "performed"; readonly value: unknown } +/** + * What the owner answered, once the caller's own parser has read the value. + * + * `T` is what the request asked for. A performed answer carries a parsed value + * and never an `unknown`: the JSON boundary is inside this module, and letting + * it out would make every consumer responsible for remembering to parse — which + * is the kind of thing that is remembered until it is not. + */ +export type OwnerAnswer = + | { readonly outcome: "performed"; readonly value: T } | { readonly outcome: "refused"; readonly refusal: string }; +/** + * How a request reads its own success value. + * + * Supplied with the request, because what a performed answer means is the + * command's business rather than the connection's. Raising is how it says the + * owner sent something this build cannot read. + */ +export type AnswerParser = (value: unknown) => T; + /** The socket shape this client needs, so a test can supply one. */ export interface OwnerSocket { send(data: string): void; @@ -55,14 +71,29 @@ export interface OwnerSocket { /** One live connection to a run's owner. */ export interface OwnerConnection { - /** Send one command and wait for the answer that names it. */ - ask(id: string, command: Record): Operation; + /** + * Send one command and wait for the answer that names it. + * + * `parse` reads the success value. If it raises, the channel fails closed + * like any other disagreement about what completed — a value neither side + * agrees on is not something to hand a caller and carry on from. + */ + ask( + id: string, + command: Record, + parse: AnswerParser, + ): Operation>; } /** The most bytes one answer may carry. */ const MAX_ANSWER = 8 * 1024 * 1024; -function readAnswer(raw: unknown): { id: string; answer: OwnerAnswer } { +/** The envelope, before the caller's parser reads the value inside it. */ +type RawAnswer = + | { readonly outcome: "performed"; readonly value: unknown } + | { readonly outcome: "refused"; readonly refusal: string }; + +function readAnswer(raw: unknown): { id: string; answer: RawAnswer } { if (typeof raw !== "string") { throw new OwnerLinkError("malformed-answer"); } @@ -78,17 +109,17 @@ function readAnswer(raw: unknown): { id: string; answer: OwnerAnswer } { if (decoded === null || typeof decoded !== "object" || Array.isArray(decoded)) { throw new OwnerLinkError("malformed-answer"); } - const members = decoded as Record; - const id = members["id"]; - const outcome = members["outcome"]; + const members: Map = new Map(Object.entries(decoded)); + const id = members.get("id"); + const outcome = members.get("outcome"); if (typeof id !== "string") { throw new OwnerLinkError("malformed-answer"); } if (outcome === "performed") { - return { id, answer: { outcome, value: members["value"] } }; + return { id, answer: { outcome, value: members.get("value") } }; } if (outcome === "refused") { - const refusal = members["refusal"]; + const refusal = members.get("refusal"); if (typeof refusal !== "string") { throw new OwnerLinkError("malformed-answer"); } @@ -106,7 +137,18 @@ function readAnswer(raw: unknown): { id: string; answer: OwnerAnswer } { */ export function useOwnerConnection(socket: OwnerSocket): Operation { return resource(function* (provide) { - const waiting = new Map>>(); + /** + * One waiting request, as the reader sees it. + * + * The command's own type stays inside the closure `ask()` built, so the + * reader settles an answer without naming it and nothing here has to assert + * what a value is. `deliver` answers whether the value could be read. + */ + interface Waiter { + deliver(answer: RawAnswer): boolean; + fail(error: OwnerLinkError): void; + } + const waiting = new Map(); /** Requests already answered, so a second answer is recognized as one. */ const settled = new Set(); const messages = createSignal(); @@ -122,7 +164,7 @@ export function useOwnerConnection(socket: OwnerSocket): Operation { closed = true; for (const pending of waiting.values()) { - pending.reject(new OwnerLinkError(refusal)); + pending.fail(new OwnerLinkError(refusal)); } waiting.clear(); socket.close(); @@ -130,7 +172,7 @@ export function useOwnerConnection(socket: OwnerSocket): Operation): Operation { + *ask( + id: string, + command: Record, + parse: AnswerParser, + ): Operation> { if (closed) { throw new OwnerLinkError("closed"); } if (waiting.has(id) || settled.has(id)) { throw new OwnerLinkError("duplicate-answer"); } - const pending = withResolvers(); - waiting.set(id, pending); + const settle = withResolvers>(); + waiting.set(id, { + deliver(answer: RawAnswer): boolean { + if (answer.outcome === "refused") { + // A refusal is an answer. Nothing is parsed and nothing fails. + settle.resolve(answer); + return true; + } + let value: T; + try { + value = parse(answer.value); + } catch { + return false; + } + settle.resolve({ outcome: "performed", value }); + return true; + }, + fail(error: OwnerLinkError): void { + settle.reject(error); + }, + }); socket.send(JSON.stringify({ ...command, id })); - return yield* pending.operation; + return yield* settle.operation; }, }); }); diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts index 70ba65b7e..1ff81d189 100644 --- a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -159,6 +159,70 @@ describe("admitting an executor", () => { expect(await admitted(stub, { token: "one.two" })).toBe("token:token-malformed"); }); + it("requires every temporal claim, rather than treating an absent one as met", async () => { + for (const missing of ["exp", "iat", "nbf"]) { + const stub = executor(); + const without = claims(); + delete without[missing]; + expect(await admitted(stub, { token: await signToken(keys, without) })).toBe( + "token:malformed-claims", + ); + } + // And a claim that is present but not a NumericDate. + for (const wrong of [{ exp: "soon" }, { iat: 1.5 }, { nbf: null }]) { + const stub = executor(); + expect(await admitted(stub, { token: await signToken(keys, claims(wrong)) })).toBe( + "token:malformed-claims", + ); + } + }); + + it("treats the expiration boundary itself as expired", async () => { + // RFC 7519 wants the current time strictly before `exp`. With no skew, a + // token expiring exactly now is spent. + const exact = executor(); + await on(exact, (o) => o.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const boundary = await signToken(keys, claims({ exp: NOW })); + expect( + await on(exact, (o) => o.admitConnection({ token: boundary, release: POLICY.release })), + ).toBe("token:expired"); + }); + + it("requires a key id naming exactly one configured key", async () => { + const absent = executor(); + expect( + await admitted(absent, { token: await signToken(keys, claims(), { kid: undefined }) }), + ).toBe("token:unknown-key"); + const empty = executor(); + expect(await admitted(empty, { token: await signToken(keys, claims(), { kid: "" }) })).toBe( + "token:unknown-key", + ); + const unknown = executor(); + expect( + await admitted(unknown, { token: await signToken(keys, claims(), { kid: "nope" }) }), + ).toBe("token:unknown-key"); + }); + + it("requires the header to say it is a JWT", async () => { + const stub = executor(); + const token = await signToken(keys, claims(), { typ: "at+jwt" }); + expect(await admitted(stub, { token })).toBe("token:unsupported-type"); + }); + + it("refuses a clock configuration it cannot trust", async () => { + const negative = executor(); + await on(negative, (o) => o.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW, -1)); + expect( + await on(negative, (o) => o.admitConnection({ release: POLICY.release, token: "a.b.c" })), + ).toBe("token:misconfigured-clock"); + + const huge = executor(); + await on(huge, (o) => o.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW, 86_400)); + expect( + await on(huge, (o) => o.admitConnection({ release: POLICY.release, token: "a.b.c" })), + ).toBe("token:misconfigured-clock"); + }); + it("refuses a token outside its validity window", async () => { const expired = executor(); expect( diff --git a/packages/workflow/tests/cloudflare/owner-storage.vitest.ts b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts index 414b1801c..a075305ed 100644 --- a/packages/workflow/tests/cloudflare/owner-storage.vitest.ts +++ b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts @@ -120,3 +120,31 @@ describe("an owner commit", () => { }); }); }); + +describe("owner transaction ownership", () => { + it("refuses a nested transaction on the same storage", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + expect(await on(stub, (o) => o.nestOnSameStorage())).toBe("refused:nested"); + }); + + it("does not couple a transaction on one storage to another storage", async () => { + // A module-level flag would refuse the second transaction because the first + // was open. Every Durable Object in an isolate shares this module and + // shares nothing else, so the guard is keyed by the storage it governs. + const stub = owner(); + await on(stub, (o) => o.initialize()); + expect(await on(stub, (o) => o.transactOnADifferentStorage())).toBe( + "committed while another storage transacted", + ); + }); + + it("releases the storage however its transaction ended", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + // A throwing transaction must leave the storage free for the next one. + await on(stub, (o) => o.commitMixedChange(true)); + expect(await on(stub, (o) => o.commitMixedChange(false))).toBe("committed"); + expect(await on(stub, (o) => o.nestOnSameStorage())).toBe("refused:nested"); + }); +}); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index bf30a10aa..5ba445b3f 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -53,18 +53,22 @@ export class ExecutorObject extends WorkflowOwnerObject { */ #keys: VerificationKey[] = []; #now = 1_800_000_000; + #skew = 0; - configure(keys: VerificationKey[], now?: number): void { + configure(keys: VerificationKey[], now?: number, skew?: number): void { this.#keys = keys; if (now !== undefined) { this.#now = now; } + if (skew !== undefined) { + this.#skew = skew; + } } protected configuration(): OwnerConfiguration { const verification: TokenVerification = { keys: this.#keys, - skewSeconds: 60, + skewSeconds: this.#skew, now: () => this.#now, }; return { policy: POLICY, verification }; diff --git a/packages/workflow/tests/cloudflare/support/owner-object.ts b/packages/workflow/tests/cloudflare/support/owner-object.ts index 79d5a1df1..b66641914 100644 --- a/packages/workflow/tests/cloudflare/support/owner-object.ts +++ b/packages/workflow/tests/cloudflare/support/owner-object.ts @@ -15,16 +15,22 @@ import { WorkflowObjectStorageError, } from "../../../src/cloudflare/recognition.ts"; import { MARKER_TABLE } from "../../../src/cloudflare/marker.ts"; -import { ownerTransaction } from "../../../src/cloudflare/owner-transaction.ts"; +import { + OwnerTransactionNestedError, + OwnerTransactions, +} from "../../../src/cloudflare/owner-transaction.ts"; +import type { OwnerStorage } from "../../../src/cloudflare/storage.ts"; /** One run row, so initialization writes what a real run would. */ const RUN_ID = "run-under-test"; export class OwnerObject extends DurableObject { + readonly #transactions = new OwnerTransactions(); + /** Create the schema, DOFS schema, an empty root and the run row, then mark it. */ initialize(): string { try { - initializeObject(this.ctx.storage, () => { + initializeObject(this.ctx.storage, this.#transactions, () => { this.ctx.storage.sql.exec( "INSERT INTO workflow_run (run_id, definition, base, props, status, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?)", RUN_ID, @@ -86,7 +92,7 @@ export class OwnerObject extends DurableObject { */ commitMixedChange(fail: boolean): string { try { - ownerTransaction(this.ctx.storage, ({ dofs }) => { + this.#transactions.run(this.ctx.storage, ({ dofs }) => { mkdirPath(dofs, "/published", { recursive: true }, () => 0); // oxlint-disable-next-line local/no-sync-filesystem writeFileSync( @@ -112,6 +118,46 @@ export class OwnerObject extends DurableObject { } } + /** + * Open an owner transaction inside one, on this object's own storage. + * + * The runtime admits exactly one, so this must be refused before it reaches + * the transaction API rather than by the runtime rejecting a savepoint. + */ + nestOnSameStorage(): string { + try { + this.#transactions.run(this.ctx.storage, () => { + this.#transactions.run(this.ctx.storage, () => undefined); + }); + return "nested"; + } catch (error) { + return error instanceof OwnerTransactionNestedError ? "refused:nested" : describe(error); + } + } + + /** + * Hold a transaction on this object's real storage and open another on a + * different storage at the same time. + * + * The second storage is a local stand-in rather than another object's: the + * runtime forbids touching another Durable Object's I/O, which is exactly why + * the guard has to be keyed by storage instance rather than shared. What is + * being proved is that holding one does not block the other. + */ + transactOnADifferentStorage(): string { + const other = standInStorage(); + try { + // A second gate stands for a second Durable Object: what must not happen + // is one object's open transaction refusing another object's. + const otherObject = new OwnerTransactions(); + return this.#transactions.run(this.ctx.storage, () => + otherObject.run(other, () => "committed while another storage transacted"), + ); + } catch (error) { + return describe(error); + } + } + /** What the run row and the DOFS filesystem hold, read outside any transaction. */ frontier(): { status: string; publishedPaths: number } { const runRows = this.ctx.storage.sql @@ -135,3 +181,22 @@ function describe(error: unknown): string { } return `threw:${error instanceof Error ? error.message : String(error)}`; } + +/** + * A second storage that is not this object's. + * + * It answers nothing useful — the transaction opened on it does no SQL — so it + * is only ever asked whether it is a different key than the real one. + */ +function standInStorage(): OwnerStorage { + return { + sql: { + exec(): { toArray(): Record[] } { + return { toArray: () => [] }; + }, + }, + transactionSync(closure: () => T): T { + return closure(); + }, + }; +} diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts index 6a59079b8..03d5a40e2 100644 --- a/packages/workflow/tests/remote-client.test.ts +++ b/packages/workflow/tests/remote-client.test.ts @@ -13,6 +13,19 @@ import { expect } from "@executablemd/test-support/expect"; import { scoped, sleep, spawn } from "effection"; import { type OwnerSocket, OwnerLinkError, useOwnerConnection } from "../src/remote/client.ts"; +/** These tests are about correlation, so most of them read any value. */ +function readString(value: unknown): unknown { + return value; +} + +/** A parser that refuses anything but a string, so a bad value fails the link. */ +function requireString(value: unknown): string { + if (typeof value !== "string") { + throw new Error("expected a string"); + } + return value; +} + /** A socket a test drives by hand. */ function fakeSocket() { const sent: Record[] = []; @@ -51,7 +64,7 @@ describe("a connection to a run's owner", () => { yield* scoped(function* () { const owner = yield* useOwnerConnection(wire.socket); yield* sleep(0); - const asking = yield* spawn(() => owner.ask("a1", { command: "frontier" })); + const asking = yield* spawn(() => owner.ask("a1", { command: "frontier" }, readString)); yield* sleep(0); // The request is on the wire before any answer exists. expect(wire.sent).toEqual([{ command: "frontier", id: "a1" }]); @@ -66,8 +79,8 @@ describe("a connection to a run's owner", () => { yield* scoped(function* () { const owner = yield* useOwnerConnection(wire.socket); yield* sleep(0); - const first = yield* spawn(() => owner.ask("a1", { command: "frontier" })); - const second = yield* spawn(() => owner.ask("a2", { command: "settle" })); + const first = yield* spawn(() => owner.ask("a1", { command: "frontier" }, readString)); + const second = yield* spawn(() => owner.ask("a2", { command: "settle" }, readString)); // Answered in the opposite order to the asking. wire.answer({ id: "a2", outcome: "performed", value: "second" }); @@ -84,7 +97,7 @@ describe("a connection to a run's owner", () => { yield* scoped(function* () { const owner = yield* useOwnerConnection(wire.socket); yield* sleep(0); - const asking = yield* spawn(() => owner.ask("a1", { command: "commit" })); + const asking = yield* spawn(() => owner.ask("a1", { command: "commit" }, readString)); wire.answer({ id: "a1", outcome: "refused", refusal: "acquisition:already-running" }); expect(yield* asking).toEqual({ outcome: "refused", @@ -102,7 +115,7 @@ describe("a connection to a run's owner", () => { yield* sleep(0); const asking = yield* spawn(function* () { try { - yield* owner.ask("a1", { command: "frontier" }); + yield* owner.ask("a1", { command: "frontier" }, readString); } catch (error) { raised = error; } @@ -123,7 +136,7 @@ describe("a connection to a run's owner", () => { yield* sleep(0); wire.end(); try { - yield* owner.ask("a1", { command: "frontier" }); + yield* owner.ask("a1", { command: "frontier" }, readString); } catch (error) { raised = error; } @@ -137,10 +150,10 @@ describe("a connection to a run's owner", () => { yield* scoped(function* () { const owner = yield* useOwnerConnection(wire.socket); yield* sleep(0); - yield* spawn(() => owner.ask("a1", { command: "frontier" })); + yield* spawn(() => owner.ask("a1", { command: "frontier" }, readString)); yield* sleep(0); try { - yield* owner.ask("a1", { command: "settle" }); + yield* owner.ask("a1", { command: "settle" }, readString); } catch (error) { raised = error; } @@ -157,14 +170,14 @@ describe("a connection to a run's owner", () => { yield* sleep(0); const first = yield* spawn(function* () { try { - yield* owner.ask("a1", { command: "frontier" }); + yield* owner.ask("a1", { command: "frontier" }, readString); } catch (error) { raised.push(error); } }); const second = yield* spawn(function* () { try { - yield* owner.ask("a2", { command: "settle" }); + yield* owner.ask("a2", { command: "settle" }, readString); } catch (error) { raised.push(error); } @@ -190,7 +203,7 @@ describe("a connection to a run's owner", () => { yield* sleep(0); const asking = yield* spawn(function* () { try { - yield* owner.ask("a1", { command: "frontier" }); + yield* owner.ask("a1", { command: "frontier" }, readString); } catch (error) { raised = error; } @@ -202,6 +215,58 @@ describe("a connection to a run's owner", () => { expect((raised as OwnerLinkError).refusal).toBe("unknown-answer"); }); + it("fails every waiter when a success value cannot be parsed", function* () { + const wire = fakeSocket(); + const raised: unknown[] = []; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const first = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }, requireString); + } catch (error) { + raised.push(error); + } + }); + const second = yield* spawn(function* () { + try { + yield* owner.ask("a2", { command: "settle" }, requireString); + } catch (error) { + raised.push(error); + } + }); + yield* sleep(0); + // Performed, and the value is not what the command's parser reads. The + // caller must not receive it, and the other waiter must not be left. + wire.answer({ id: "a1", outcome: "performed", value: { not: "a string" } }); + yield* first; + yield* second; + }); + expect(raised).toHaveLength(2); + for (const error of raised) { + expect((error as OwnerLinkError).refusal).toBe("malformed-answer"); + } + }); + + it("still delivers a refusal without consulting the success parser", function* () { + const wire = fakeSocket(); + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(() => + owner.ask("a1", { command: "commit" }, () => { + throw new Error("a refusal must not reach this"); + }), + ); + yield* sleep(0); + wire.answer({ id: "a1", outcome: "refused", refusal: "acquisition:already-running" }); + expect(yield* asking).toEqual({ + outcome: "refused", + refusal: "acquisition:already-running", + }); + }); + }); + it("fails closed on a second answer to a request already settled", function* () { const wire = fakeSocket(); let answered: unknown; @@ -209,14 +274,14 @@ describe("a connection to a run's owner", () => { yield* scoped(function* () { const owner = yield* useOwnerConnection(wire.socket); yield* sleep(0); - const first = yield* spawn(() => owner.ask("a1", { command: "frontier" })); + const first = yield* spawn(() => owner.ask("a1", { command: "frontier" }, readString)); yield* sleep(0); wire.answer({ id: "a1", outcome: "performed", value: "once" }); answered = yield* first; const second = yield* spawn(function* () { try { - yield* owner.ask("a2", { command: "settle" }); + yield* owner.ask("a2", { command: "settle" }, readString); } catch (error) { refused = error; } diff --git a/packages/workflow/tests/remote-transaction.test.ts b/packages/workflow/tests/remote-transaction.test.ts index a86f97795..cbc11ebd4 100644 --- a/packages/workflow/tests/remote-transaction.test.ts +++ b/packages/workflow/tests/remote-transaction.test.ts @@ -40,6 +40,18 @@ function nameOf(entry: DurableEvent | undefined): string { return typeof name === "string" ? name : ""; } +/** + * A value the collector should refuse, handed over as an event. + * + * Through `unknown` rather than a double assertion: the point of the test is + * that the collector parses what it is given, and manufacturing a value that + * claims to be an event would be asserting the thing under test. + */ +function malformed(value: unknown): DurableEvent { + const offered: unknown = value; + return offered as DurableEvent; +} + /** Rename a test event in place, to prove the collector cloned it. */ function rename(entry: DurableEvent, name: string): void { if (!("description" in entry)) { @@ -345,7 +357,7 @@ describe("a remote transaction", () => { let raised: unknown; try { yield* transactRemotely(owner, gate, function* (transaction) { - yield* transaction.journal.append({ nothing: true } as unknown as DurableEvent); + yield* transaction.journal.append(malformed({ nothing: true })); return undefined; }); } catch (error) { From 026ef1f96c4d895e1929adb02131e578452c75ed Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 07:36:48 -0400 Subject: [PATCH 23/81] =?UTF-8?q?=F0=9F=94=8E=20Bound=20the=20answer=20env?= =?UTF-8?q?elope=20and=20prove=20teardown=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closing two gaps in D1 item 5 that I found auditing my own work against the list, rather than leaving them for review. The refusal was checked to be a string and nothing more, so an arbitrary remote sentence became this side's public failure identity — and a refusal is something a caller branches on. It is now held to the category shape the owner actually produces, and the correlation id is bounded. Anything else is an answer this build cannot read, which fails the channel closed like every other disagreement about what completed. Item 5 also asked for cancellation cleanup evidence and there was none. The test asserts what is observable rather than what I first assumed: Effection halts a cancelled task instead of raising into it, so the proof is that the scope completes at all with a request in flight — a waiter nothing settled would hang teardown — and that a late answer afterwards reaches nothing and raises nothing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/remote/client.ts | 18 ++++- packages/workflow/tests/remote-client.test.ts | 66 +++++++++++++++++++ 2 files changed, 82 insertions(+), 2 deletions(-) diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index 8a988e1f9..c5557352d 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -88,6 +88,20 @@ export interface OwnerConnection { /** The most bytes one answer may carry. */ const MAX_ANSWER = 8 * 1024 * 1024; +/** The longest correlation id this reads back. */ +const MAX_ID = 128; + +/** + * The shape a refusal category has. + * + * The owner answers with a category and an optional detail, both drawn from + * closed sets it declares. Holding the answer to that shape is what stops an + * arbitrary remote string becoming this side's public failure identity: a + * refusal is something a caller may branch on, so it must not be whatever the + * other end felt like sending. + */ +const REFUSAL = /^[a-z][a-z0-9-]*(:[a-z][a-z0-9-]*)?$/; + /** The envelope, before the caller's parser reads the value inside it. */ type RawAnswer = | { readonly outcome: "performed"; readonly value: unknown } @@ -112,7 +126,7 @@ function readAnswer(raw: unknown): { id: string; answer: RawAnswer } { const members: Map = new Map(Object.entries(decoded)); const id = members.get("id"); const outcome = members.get("outcome"); - if (typeof id !== "string") { + if (typeof id !== "string" || id === "" || id.length > MAX_ID) { throw new OwnerLinkError("malformed-answer"); } if (outcome === "performed") { @@ -120,7 +134,7 @@ function readAnswer(raw: unknown): { id: string; answer: RawAnswer } { } if (outcome === "refused") { const refusal = members.get("refusal"); - if (typeof refusal !== "string") { + if (typeof refusal !== "string" || !REFUSAL.test(refusal)) { throw new OwnerLinkError("malformed-answer"); } return { id, answer: { outcome, refusal } }; diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts index 03d5a40e2..fe3cdb8a5 100644 --- a/packages/workflow/tests/remote-client.test.ts +++ b/packages/workflow/tests/remote-client.test.ts @@ -267,6 +267,72 @@ describe("a connection to a run's owner", () => { }); }); + it("refuses a refusal that is not a category this side can branch on", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + yield* sleep(0); + // An arbitrary remote sentence must not become this side's public failure + // identity, so it is read as an answer this build cannot understand. + wire.answer({ id: "a1", outcome: "refused", refusal: "something went wrong!" }); + yield* asking; + }); + expect((raised as OwnerLinkError).refusal).toBe("malformed-answer"); + }); + + it("refuses an answer whose correlation id is not one", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + yield* sleep(0); + wire.answer({ id: "x".repeat(200), outcome: "performed", value: 1 }); + yield* asking; + }); + expect((raised as OwnerLinkError).refusal).toBe("malformed-answer"); + }); + + it("leaves no waiter or listener behind when its scope ends", function* () { + const wire = fakeSocket(); + let answered = false; + // The connection's scope ends with a request still in flight. Cancellation + // halts the asking task rather than raising into it, so what is observable + // is that the scope completes at all — a waiter nothing settled would hang + // teardown — and that nothing is left listening afterwards. + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + yield* spawn(function* () { + yield* owner.ask("a1", { command: "frontier" }, readString); + answered = true; + }); + yield* sleep(0); + }); + + expect(answered).toBe(false); + // A late answer reaches nothing: the listener went with the scope, and + // delivering it must not raise out of the socket either. + wire.answer({ id: "a1", outcome: "performed", value: "too late" }); + expect(answered).toBe(false); + }); + it("fails closed on a second answer to a request already settled", function* () { const wire = fakeSocket(); let answered: unknown; From febef13f13bc57773c1ff3a71cda3f42d459bda0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 08:15:12 -0400 Subject: [PATCH 24/81] =?UTF-8?q?=F0=9F=94=8C=20Make=20the=20connection=20?= =?UTF-8?q?release=20the=20acquisition=20it=20holds=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runner's socket is the executor acquisition, and D1 left it behind. The resource installed listeners and never removed them, never closed the socket, and never observed `error`, so a scope that ended — normally, by failure, or by cancellation — left the owner looking at a healthy socket that no longer had anybody on the other end. With no lease and no heartbeat by design, that made the run unadvanceable by anybody, forever. Teardown is now one idempotent operation with one owner. Scope exit, cancellation, a remote close, a socket error, a protocol failure, a command-specific parse failure and a failed send all reach it; it runs once, removes the exact callbacks it installed, and closes the socket once. The finalizer is registered before anything can suspend. Answers are now read where they arrive rather than through a signal the reader drained later. That ordering is the point: a close arriving in the same turn as an unreadable answer used to reach teardown first and tell the caller `closed` for something that was actually `malformed-answer`. What went wrong is decided where it is observed. The envelope is closed for real: each outcome declares its whole key set, so a performed answer carrying a `refusal`, a refused one carrying a `value`, a missing member and an unknown member are all refused. An outgoing correlation id is held to the contract an incoming one is held to, and refusal text gets its own small bound. The regular expression proves spelling, and the comment now says so — narrowing to the declared union stays the adapter's job. A retained Workspace root is a content identity, so every command that names one parses it as 64 lowercase hexadecimal characters rather than as any non-empty text. The transaction machinery is no longer exported from the Cloudflare subpath. It stays private to `src/cloudflare/**`, owned by the object. The old teardown test asserted nothing: its fake retained the message callback, so the connection could leak both listeners and the socket and still pass. The fake now counts closes and live listeners, and the evidence is what those counters say. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/cloudflare.ts | 6 - packages/workflow/src/cloudflare/commands.ts | 26 +- packages/workflow/src/remote/client.ts | 201 ++++++++---- .../cloudflare/executor-acquisition.vitest.ts | 4 +- .../tests/cloudflare/settle-parser.vitest.ts | 62 +++- packages/workflow/tests/remote-client.test.ts | 295 ++++++++++++++++-- .../workflow/tests/remote-transaction.test.ts | 54 ++-- 7 files changed, 526 insertions(+), 122 deletions(-) diff --git a/packages/workflow/cloudflare.ts b/packages/workflow/cloudflare.ts index 946e6017d..270aef5c8 100644 --- a/packages/workflow/cloudflare.ts +++ b/packages/workflow/cloudflare.ts @@ -53,11 +53,5 @@ export type { AcquisitionAttachment, AcquisitionRefusal } from "./src/cloudflare export { CommandError } from "./src/cloudflare/commands.ts"; export type { CommandRefusal, CommandResult, RunnerCommand } from "./src/cloudflare/commands.ts"; -export { - OwnerTransactionClosedError, - OwnerTransactionNestedError, - OwnerTransactions, -} from "./src/cloudflare/owner-transaction.ts"; - export { WorkflowObjectStorageError } from "./src/cloudflare/recognition.ts"; export type { RecognitionFailure } from "./src/cloudflare/recognition.ts"; diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 228e48714..9b50c8175 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -126,6 +126,24 @@ function object(value: unknown): Map { return members; } +/** + * A retained Workspace root, as the schema spells one. + * + * The lowercase SHA-256 of a canonical manifest: 64 hexadecimal characters and + * nothing else. Admitting any non-empty string would let three commands hold + * three different notions of what a root is, and the owner would be the first + * to find out. + */ +const ROOT_ID = /^[0-9a-f]{64}$/; + +function rootId(members: Map, key: string): string { + const value = text(members, key); + if (!ROOT_ID.test(value)) { + throw new CommandError("malformed-member"); + } + return value; +} + function text(members: Map, key: string): string { const value = members.get(key); if (typeof value !== "string" || value === "") { @@ -196,7 +214,7 @@ export function parseCommand(raw: string): RunnerCommand { return { id, command }; } if (command === "materialize") { - return { id, command, workspaceRootId: text(members, "workspaceRootId") }; + return { id, command, workspaceRootId: rootId(members, "workspaceRootId") }; } if (command === "settle") { // The shared parser decides what a completion is. Its failure becomes this @@ -210,7 +228,7 @@ export function parseCommand(raw: string): RunnerCommand { id, command, completion: completion.value, - expectedWorkspaceRootId: text(members, "expectedWorkspaceRootId"), + expectedWorkspaceRootId: rootId(members, "expectedWorkspaceRootId"), }; } const expectedJournalEventId = members.get("expectedJournalEventId"); @@ -220,10 +238,10 @@ export function parseCommand(raw: string): RunnerCommand { return { id, command, - expectedWorkspaceRootId: text(members, "expectedWorkspaceRootId"), + expectedWorkspaceRootId: rootId(members, "expectedWorkspaceRootId"), expectedJournalEventId, content: chunks(members.get("content")), - proposedWorkspaceRootId: text(members, "proposedWorkspaceRootId"), + proposedWorkspaceRootId: rootId(members, "proposedWorkspaceRootId"), events: events(members.get("events")), }; } diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index c5557352d..48879ae97 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -22,7 +22,7 @@ * dropping the answer and leaving somebody blocked forever. */ -import { createSignal, each, type Operation, resource, spawn, withResolvers } from "effection"; +import { ensure, type Operation, resource, withResolvers } from "effection"; /** Why the connection itself could not carry a request. */ export type LinkRefusal = @@ -30,7 +30,10 @@ export type LinkRefusal = | "malformed-answer" | "unknown-answer" | "duplicate-answer" - | "too-large"; + | "too-large" + | "malformed-request" + | "send-failed" + | "socket-error"; export class OwnerLinkError extends Error { override name = "OwnerLinkError"; @@ -61,12 +64,15 @@ export type OwnerAnswer = */ export type AnswerParser = (value: unknown) => T; +/** One listener, kept so teardown can remove the exact callback it installed. */ +export type SocketListener = (event: { data?: unknown }) => void; + /** The socket shape this client needs, so a test can supply one. */ export interface OwnerSocket { send(data: string): void; close(): void; - addEventListener(type: "message", listener: (event: { data: unknown }) => void): void; - addEventListener(type: "close", listener: () => void): void; + addEventListener(type: "message" | "close" | "error", listener: SocketListener): void; + removeEventListener(type: "message" | "close" | "error", listener: SocketListener): void; } /** One live connection to a run's owner. */ @@ -88,17 +94,31 @@ export interface OwnerConnection { /** The most bytes one answer may carry. */ const MAX_ANSWER = 8 * 1024 * 1024; -/** The longest correlation id this reads back. */ +/** The longest correlation id, in either direction. */ const MAX_ID = 128; +/** + * The longest refusal this reads. + * + * Small and its own bound: a refusal is a category, and the eight-megabyte + * envelope bound is for a command's payload rather than for a word. + */ +const MAX_REFUSAL = 200; + +/** Whether a correlation id is one this client will send or accept. */ +function usableId(value: unknown): value is string { + return typeof value === "string" && value !== "" && value.length <= MAX_ID; +} + /** * The shape a refusal category has. * - * The owner answers with a category and an optional detail, both drawn from - * closed sets it declares. Holding the answer to that shape is what stops an - * arbitrary remote string becoming this side's public failure identity: a - * refusal is something a caller may branch on, so it must not be whatever the - * other end felt like sending. + * The owner answers with a category and an optional detail. This proves + * *spelling* and nothing more — a syntactically valid category this build has + * never heard of still passes here. Narrowing a refusal to the exact declared + * union is the Cloudflare adapter's job, where the union is known; what this + * bound is for is stopping an arbitrary remote sentence from travelling as + * though it were a category at all. */ const REFUSAL = /^[a-z][a-z0-9-]*(:[a-z][a-z0-9-]*)?$/; @@ -126,15 +146,31 @@ function readAnswer(raw: unknown): { id: string; answer: RawAnswer } { const members: Map = new Map(Object.entries(decoded)); const id = members.get("id"); const outcome = members.get("outcome"); - if (typeof id !== "string" || id === "" || id.length > MAX_ID) { + if (!usableId(id)) { throw new OwnerLinkError("malformed-answer"); } + + // Each branch declares its whole key set. A performed answer carrying a + // `refusal`, or a refused one carrying a `value`, is an answer the two sides + // disagree about the shape of — which is the thing this channel refuses to + // carry on past. + const declared = + outcome === "performed" ? ["id", "outcome", "value"] : ["id", "outcome", "refusal"]; + if (members.size !== declared.length) { + throw new OwnerLinkError("malformed-answer"); + } + for (const key of members.keys()) { + if (!declared.includes(key)) { + throw new OwnerLinkError("malformed-answer"); + } + } + if (outcome === "performed") { return { id, answer: { outcome, value: members.get("value") } }; } if (outcome === "refused") { const refusal = members.get("refusal"); - if (typeof refusal !== "string" || !REFUSAL.test(refusal)) { + if (typeof refusal !== "string" || refusal.length > MAX_REFUSAL || !REFUSAL.test(refusal)) { throw new OwnerLinkError("malformed-answer"); } return { id, answer: { outcome, refusal } }; @@ -145,9 +181,17 @@ function readAnswer(raw: unknown): { id: string; answer: RawAnswer } { /** * Hold one connection open for the calling scope. * - * Teardown resolves every request still waiting with a closed refusal rather - * than leaving it pending: a caller blocked on an answer that can never arrive - * would outlive the connection it was asking through. + * The connection *is* the executor acquisition, so the scope that owns it owns + * ending it: there is no lease to expire and no heartbeat to miss, and an owner + * that still sees a healthy socket still considers this runner the executor. A + * scope that walked away without closing would leave the run unadvanceable by + * anybody, forever. + * + * So teardown is one operation with one owner. Scope exit, cancellation, a + * remote close, a socket error, a protocol failure and a failed send all reach + * it, it runs once, and it removes the exact listeners it installed and closes + * the socket. The failure that caused it is what the waiters are told — a close + * arriving afterwards must not rewrite `malformed-answer` into `closed`. */ export function useOwnerConnection(socket: OwnerSocket): Operation { return resource(function* (provide) { @@ -165,60 +209,85 @@ export function useOwnerConnection(socket: OwnerSocket): Operation(); /** Requests already answered, so a second answer is recognized as one. */ const settled = new Set(); - const messages = createSignal(); let closed = false; + let torn = false; - socket.addEventListener("message", (event) => messages.send(event.data)); - socket.addEventListener("close", () => { - closed = true; - messages.close(); - }); - - /** Stop the channel and tell everyone waiting why. */ - const fail = (refusal: LinkRefusal) => { - closed = true; - for (const pending of waiting.values()) { - pending.fail(new OwnerLinkError(refusal)); + /** + * Read one incoming answer and settle the request it names. + * + * Synchronous, and deliberately so. If this queued the message and read it + * later, a close arriving in the same turn would reach teardown first and + * the caller would be told `closed` for an answer that was actually + * unreadable. What went wrong is decided where it is observed. + */ + const onMessage: SocketListener = (event) => { + if (torn) { + return; + } + let read: { id: string; answer: RawAnswer }; + try { + read = readAnswer(event.data); + } catch (error) { + // The owner said something this build cannot read. Whether it was meant + // for a waiter is exactly what cannot be established. + teardown(error instanceof OwnerLinkError ? error.refusal : "malformed-answer"); + return; + } + const pending = waiting.get(read.id); + if (pending === undefined) { + // Either a request nobody made, or a second answer to one already + // settled. Both mean the two sides disagree about what completed. + teardown(settled.has(read.id) ? "duplicate-answer" : "unknown-answer"); + return; + } + waiting.delete(read.id); + settled.add(read.id); + if (!pending.deliver(read.answer)) { + // The owner performed the command and described the result in a way + // this build cannot read. Handing the caller an unparsed value is the + // one outcome that must not happen. + waiting.set(read.id, pending); + teardown("malformed-answer"); } - waiting.clear(); - socket.close(); }; + const onClose: SocketListener = () => teardown("closed"); + const onError: SocketListener = () => teardown("socket-error"); - yield* spawn(function* () { - for (const raw of yield* each(messages)) { - let read: { id: string; answer: RawAnswer } | undefined; - try { - read = readAnswer(raw); - } catch { - // The owner said something this build cannot read. Whether it was - // meant for a waiter is exactly what cannot be established. - fail("malformed-answer"); - break; - } - const pending = waiting.get(read.id); - if (pending === undefined) { - // Either a request nobody made, or a second answer to one already - // settled. Both mean the two sides disagree about what completed. - fail(settled.has(read.id) ? "duplicate-answer" : "unknown-answer"); - break; - } - waiting.delete(read.id); - settled.add(read.id); - if (!pending.deliver(read.answer)) { - // The owner performed the command and described the result in a way - // this build cannot read. Handing the caller an unparsed value is the - // one outcome that must not happen. - waiting.set(read.id, pending); - fail("malformed-answer"); - break; - } - yield* each.next(); + /** + * End the connection, once. + * + * `refusal` is what the waiters are told. The first caller decides it: a + * remote close after a malformed answer is the same teardown, and the + * caller waiting on that answer should learn what actually went wrong. + */ + function teardown(refusal: LinkRefusal): void { + if (torn) { + return; } + torn = true; closed = true; for (const pending of waiting.values()) { - pending.fail(new OwnerLinkError("closed")); + pending.fail(new OwnerLinkError(refusal)); } waiting.clear(); + socket.removeEventListener("message", onMessage); + socket.removeEventListener("close", onClose); + socket.removeEventListener("error", onError); + try { + socket.close(); + } catch { + // Already closed, or closing threw on the way out. Either way this + // connection is over and there is nothing left to tell anybody. + } + } + + socket.addEventListener("message", onMessage); + socket.addEventListener("close", onClose); + socket.addEventListener("error", onError); + // Registered before anything can suspend, so a cancellation between here + // and `provide()` still closes the socket it just started listening to. + yield* ensure(() => { + teardown("closed"); }); yield* provide({ @@ -230,6 +299,11 @@ export function useOwnerConnection(socket: OwnerSocket): Operation { const raw = JSON.stringify({ id: "7", command: "commit", - expectedWorkspaceRootId: "root-a", + expectedWorkspaceRootId: `a${"0".repeat(63)}`, expectedJournalEventId: null, content: [{ digest: "d1", bytes: "AAAA" }], - proposedWorkspaceRootId: "root-b", + proposedWorkspaceRootId: `b${"1".repeat(63)}`, events: ["event-1"], }); expect(await on(stub, (o) => o.send(1, raw))).toEqual({ diff --git a/packages/workflow/tests/cloudflare/settle-parser.vitest.ts b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts index 82ff195e9..2a26c5dac 100644 --- a/packages/workflow/tests/cloudflare/settle-parser.vitest.ts +++ b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts @@ -88,9 +88,69 @@ describe("a settle command", () => { expect(read(settle({ expectedWorkspaceRootId: "" }))).toBe("malformed-member"); expect(read(settle({ expectedWorkspaceRootId: 1 }))).toBe("malformed-member"); }); - it("refuses a member the command does not declare", () => { expect(read(settle({ status: "completed" }))).toBe("unknown-member"); expect(read(settle({ somethingElse: true }))).toBe("unknown-member"); }); }); + +/** + * A retained Workspace root is a content identity, and every command that names + * one names the same thing. A command shape that admitted "any non-empty text" + * would let a request select a root by a spelling the store can never hold, and + * would let two commands disagree about what a root is. + * + * Shape only. Whether a well-spelled root is the one this run is actually at is + * the owner's revalidation, in the checkpoint that has a lifecycle to check it + * against. + */ +describe("a root identity in a command", () => { + const wrong: Record = { + "one character short": ROOT.slice(1), + "one character long": `${ROOT}0`, + "uppercase hexadecimal": ROOT.toUpperCase(), + "hexadecimal with a non-hexadecimal letter": `${ROOT.slice(0, 63)}z`, + "a plausible-looking name": "root-a", + "not text at all": 7, + }; + + /** Every root field in the private command shapes, by the request it sits in. */ + const fields: Record string> = { + "materialize.workspaceRootId": (root) => + JSON.stringify({ id: "m1", command: "materialize", workspaceRootId: root }), + "commit.expectedWorkspaceRootId": (root) => commit({ expectedWorkspaceRootId: root }), + "commit.proposedWorkspaceRootId": (root) => commit({ proposedWorkspaceRootId: root }), + "settle.expectedWorkspaceRootId": (root) => settle({ expectedWorkspaceRootId: root }), + }; + + function commit(overrides: Record): string { + return JSON.stringify({ + id: "c1", + command: "commit", + expectedWorkspaceRootId: ROOT, + expectedJournalEventId: null, + content: [], + proposedWorkspaceRootId: ROOT, + events: [], + ...overrides, + }); + } + + it("reads the canonical spelling in every command that names one", () => { + for (const [field, request] of Object.entries(fields)) { + expect([field, read(request(ROOT))]).toEqual([field, field.split(".")[0]]); + } + }); + + it("refuses anything that is not the canonical spelling", () => { + for (const [field, request] of Object.entries(fields)) { + for (const [description, root] of Object.entries(wrong)) { + expect([field, description, read(request(root))]).toEqual([ + field, + description, + "malformed-member", + ]); + } + } + }); +}); diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts index fe3cdb8a5..b9711a6de 100644 --- a/packages/workflow/tests/remote-client.test.ts +++ b/packages/workflow/tests/remote-client.test.ts @@ -11,7 +11,12 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { scoped, sleep, spawn } from "effection"; -import { type OwnerSocket, OwnerLinkError, useOwnerConnection } from "../src/remote/client.ts"; +import { + type OwnerSocket, + OwnerLinkError, + type SocketListener, + useOwnerConnection, +} from "../src/remote/client.ts"; /** These tests are about correlation, so most of them read any value. */ function readString(value: unknown): unknown { @@ -26,34 +31,79 @@ function requireString(value: unknown): string { return value; } -/** A socket a test drives by hand. */ -function fakeSocket() { +/** + * What the connection refused with, having proved it refused at all. + * + * A caught value is `unknown`, and asserting it into `OwnerLinkError` would let + * an unrelated failure read as the transport category a test expected. + */ +function refusalOf(error: unknown): string { + if (!(error instanceof OwnerLinkError)) { + throw new Error(`expected an OwnerLinkError, got ${String(error)}`); + } + return error.refusal; +} + +/** + * A socket a test drives by hand, and can ask what happened to it. + * + * It counts closes and tracks the listeners still installed, because the claims + * under test are about teardown: that the connection closes its socket exactly + * once and stops listening. A fake that merely retained its callbacks would let + * a test assert cleanup that never happened — which is how the previous version + * of this suite passed while the connection leaked both. + */ +function fakeSocket(options: { failSend?: boolean } = {}) { const sent: Record[] = []; - let onMessage: ((event: { data: unknown }) => void) | undefined; - let onClose: (() => void) | undefined; + const listeners = new Map>(); + let closes = 0; + + const deliver = (type: string, event: { data?: unknown }) => { + for (const listener of listeners.get(type) ?? []) { + listener(event); + } + }; + const socket: OwnerSocket = { send(data: string): void { + if (options.failSend === true) { + throw new Error("the socket refused the write"); + } sent.push(JSON.parse(data)); }, close(): void { - onClose?.(); + closes += 1; }, - addEventListener(type: "message" | "close", listener: never): void { - if (type === "message") { - onMessage = listener; - } else { - onClose = listener; - } + addEventListener(type, listener): void { + const existing = listeners.get(type) ?? new Set(); + existing.add(listener); + listeners.set(type, existing); + }, + removeEventListener(type, listener): void { + listeners.get(type)?.delete(listener); }, }; + return { socket, sent, + get closes(): number { + return closes; + }, + /** How many listeners are still installed, of any type. */ + get listening(): number { + return [...listeners.values()].reduce((total, set) => total + set.size, 0); + }, answer(value: unknown): void { - onMessage?.({ data: typeof value === "string" ? value : JSON.stringify(value) }); + deliver("message", { + data: typeof value === "string" ? value : JSON.stringify(value), + }); }, end(): void { - onClose?.(); + deliver("close", {}); + }, + error(): void { + deliver("error", {}); }, }; } @@ -81,6 +131,9 @@ describe("a connection to a run's owner", () => { yield* sleep(0); const first = yield* spawn(() => owner.ask("a1", { command: "frontier" }, readString)); const second = yield* spawn(() => owner.ask("a2", { command: "settle" }, readString)); + yield* sleep(0); + // Both requests are on the wire before either is answered. + expect(wire.sent.map((request) => request.id)).toEqual(["a1", "a2"]); // Answered in the opposite order to the asking. wire.answer({ id: "a2", outcome: "performed", value: "second" }); @@ -98,6 +151,7 @@ describe("a connection to a run's owner", () => { const owner = yield* useOwnerConnection(wire.socket); yield* sleep(0); const asking = yield* spawn(() => owner.ask("a1", { command: "commit" }, readString)); + yield* sleep(0); wire.answer({ id: "a1", outcome: "refused", refusal: "acquisition:already-running" }); expect(yield* asking).toEqual({ outcome: "refused", @@ -125,7 +179,7 @@ describe("a connection to a run's owner", () => { }); yield* sleep(0); expect(raised).toBeInstanceOf(OwnerLinkError); - expect((raised as OwnerLinkError).refusal).toBe("closed"); + expect(refusalOf(raised)).toBe("closed"); }); it("refuses to ask through a connection that already ended", function* () { @@ -141,7 +195,7 @@ describe("a connection to a run's owner", () => { raised = error; } }); - expect((raised as OwnerLinkError).refusal).toBe("closed"); + expect(refusalOf(raised)).toBe("closed"); }); it("refuses a second request under an id already in flight", function* () { @@ -159,7 +213,7 @@ describe("a connection to a run's owner", () => { } wire.answer({ id: "a1", outcome: "performed", value: null }); }); - expect((raised as OwnerLinkError).refusal).toBe("duplicate-answer"); + expect(refusalOf(raised)).toBe("duplicate-answer"); }); it("fails every waiter when it cannot read an answer", function* () { @@ -191,8 +245,10 @@ describe("a connection to a run's owner", () => { }); expect(raised).toHaveLength(2); for (const error of raised) { - expect((error as OwnerLinkError).refusal).toBe("malformed-answer"); + expect(refusalOf(error)).toBe("malformed-answer"); } + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); }); it("fails closed on an answer naming a request nobody made", function* () { @@ -212,7 +268,7 @@ describe("a connection to a run's owner", () => { wire.answer({ id: "somebody-else", outcome: "performed", value: 1 }); yield* asking; }); - expect((raised as OwnerLinkError).refusal).toBe("unknown-answer"); + expect(refusalOf(raised)).toBe("unknown-answer"); }); it("fails every waiter when a success value cannot be parsed", function* () { @@ -244,8 +300,10 @@ describe("a connection to a run's owner", () => { }); expect(raised).toHaveLength(2); for (const error of raised) { - expect((error as OwnerLinkError).refusal).toBe("malformed-answer"); + expect(refusalOf(error)).toBe("malformed-answer"); } + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); }); it("still delivers a refusal without consulting the success parser", function* () { @@ -286,7 +344,7 @@ describe("a connection to a run's owner", () => { wire.answer({ id: "a1", outcome: "refused", refusal: "something went wrong!" }); yield* asking; }); - expect((raised as OwnerLinkError).refusal).toBe("malformed-answer"); + expect(refusalOf(raised)).toBe("malformed-answer"); }); it("refuses an answer whose correlation id is not one", function* () { @@ -306,31 +364,206 @@ describe("a connection to a run's owner", () => { wire.answer({ id: "x".repeat(200), outcome: "performed", value: 1 }); yield* asking; }); - expect((raised as OwnerLinkError).refusal).toBe("malformed-answer"); + expect(refusalOf(raised)).toBe("malformed-answer"); }); - it("leaves no waiter or listener behind when its scope ends", function* () { + it("closes the socket once and stops listening when its scope ends", function* () { const wire = fakeSocket(); let answered = false; - // The connection's scope ends with a request still in flight. Cancellation - // halts the asking task rather than raising into it, so what is observable - // is that the scope completes at all — a waiter nothing settled would hang - // teardown — and that nothing is left listening afterwards. yield* scoped(function* () { const owner = yield* useOwnerConnection(wire.socket); yield* sleep(0); + expect(wire.listening).toBeGreaterThan(0); yield* spawn(function* () { yield* owner.ask("a1", { command: "frontier" }, readString); answered = true; }); yield* sleep(0); + // Leaving with a request in flight. The connection is the acquisition, so + // the owner only learns this runner is gone when the socket closes. }); + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); expect(answered).toBe(false); - // A late answer reaches nothing: the listener went with the scope, and - // delivering it must not raise out of the socket either. + + // A late message and a late close reach nothing and raise nothing. wire.answer({ id: "a1", outcome: "performed", value: "too late" }); + wire.end(); expect(answered).toBe(false); + expect(wire.closes).toBe(1); + }); + + it("ends the same way however the connection is lost", function* () { + // Each of these is one teardown with one owner: the waiters learn why, the + // listeners go, and the socket closes exactly once. + const cases: [string, (wire: ReturnType) => void][] = [ + ["closed", (wire) => wire.end()], + ["socket-error", (wire) => wire.error()], + ["malformed-answer", (wire) => wire.answer("not json at all")], + ["unknown-answer", (wire) => wire.answer({ id: "nobody", outcome: "performed", value: 1 })], + ]; + + for (const [expected, provoke] of cases) { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + yield* sleep(0); + provoke(wire); + yield* asking; + }); + expect(refusalOf(raised)).toBe(expected); + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); + } + }); + + it("keeps the failure that caused teardown when a close follows it", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + yield* sleep(0); + wire.answer("not json at all"); + // The remote end closes right after. The caller should still learn what + // actually went wrong rather than a generic `closed`. + wire.end(); + yield* asking; + }); + expect(refusalOf(raised)).toBe("malformed-answer"); + expect(wire.closes).toBe(1); + }); + + it("tears down when the socket refuses the write, and sends nothing", function* () { + const wire = fakeSocket({ failSend: true }); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + expect(refusalOf(raised)).toBe("send-failed"); + expect(wire.sent).toEqual([]); + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); + }); + + it("refuses to send a correlation id it would refuse to read", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + try { + yield* owner.ask("", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + expect(refusalOf(raised)).toBe("malformed-request"); + try { + yield* owner.ask("x".repeat(200), { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + expect(refusalOf(raised)).toBe("malformed-request"); + // Nothing left, so nothing to correlate an answer to. + expect(wire.sent).toEqual([]); + }); + + it("refuses an answer whose branch carries a member it does not declare", function* () { + const cases: unknown[] = [ + { id: "a1", outcome: "performed", value: 1, refusal: "acquisition:stale" }, + { id: "a1", outcome: "refused", refusal: "acquisition:stale", value: 1 }, + { id: "a1", outcome: "performed" }, + { id: "a1", outcome: "refused" }, + { id: "a1", outcome: "performed", value: 1, extra: true }, + ]; + for (const answer of cases) { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + yield* sleep(0); + wire.answer(answer); + yield* asking; + }); + expect(refusalOf(raised)).toBe("malformed-answer"); + } + }); + + it("releases the socket when the scope holding it is cancelled", function* () { + const wire = fakeSocket(); + let raised: unknown; + yield* scoped(function* () { + const holding = yield* spawn(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised = error; + } + }); + yield* sleep(0); + expect(wire.listening).toBeGreaterThan(0); + // Cancellation, rather than the scope reaching its end. The connection is + // the acquisition either way, so the socket must still close. + yield* holding.halt(); + }); + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); + // Halting the caller means it is never told anything; the socket closing is + // what the owner observes. + expect(raised).toBe(undefined); + }); + + it("carries a refusal category this build has never heard of", function* () { + const wire = fakeSocket(); + let answered: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(() => owner.ask("a1", { command: "commit" }, readString)); + yield* sleep(0); + // Well-spelled and not a category this layer knows. Deciding which + // categories exist belongs to the adapter that declares the union, so the + // connection hands it through rather than guessing on the adapter's + // behalf and failing a run over a word. + wire.answer({ id: "a1", outcome: "refused", refusal: "workspace:root-unknown-here" }); + answered = yield* asking; + }); + expect(answered).toEqual({ outcome: "refused", refusal: "workspace:root-unknown-here" }); + expect(wire.closes).toBe(1); }); it("fails closed on a second answer to a request already settled", function* () { @@ -358,6 +591,8 @@ describe("a connection to a run's owner", () => { yield* second; }); expect(answered).toEqual({ outcome: "performed", value: "once" }); - expect((refused as OwnerLinkError).refusal).toBe("duplicate-answer"); + expect(refusalOf(refused)).toBe("duplicate-answer"); + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); }); }); diff --git a/packages/workflow/tests/remote-transaction.test.ts b/packages/workflow/tests/remote-transaction.test.ts index cbc11ebd4..5d5a2d794 100644 --- a/packages/workflow/tests/remote-transaction.test.ts +++ b/packages/workflow/tests/remote-transaction.test.ts @@ -16,7 +16,7 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { Err, Ok, sleep, spawn, withResolvers, type Operation, type Result } from "effection"; -import type { DurableEvent } from "@executablemd/durable-streams"; +import type { DurableEvent, DurableStream } from "@executablemd/durable-streams"; import { type CommitIntent, createTransactionGate, @@ -27,6 +27,19 @@ import { transactRemotely, } from "../src/remote/collector.ts"; +/** + * What the transaction refused with, having proved it refused at all. + * + * A caught value is `unknown`; asserting it would let an unrelated failure read + * as the refusal a test expected. + */ +function refusalOf(error: unknown): string { + if (!(error instanceof RemoteTransactionError)) { + throw new Error(`expected a RemoteTransactionError, got ${String(error)}`); + } + return error.refusal; +} + /** The name a test event carries, read rather than asserted. */ function nameOf(entry: DurableEvent | undefined): string { if (entry === undefined || !("description" in entry)) { @@ -36,20 +49,23 @@ function nameOf(entry: DurableEvent | undefined): string { if (description === null || typeof description !== "object") { return ""; } - const name = (description as Record)["name"]; + if (!("name" in description)) { + return ""; + } + const name: unknown = description["name"]; return typeof name === "string" ? name : ""; } /** - * A value the collector should refuse, handed over as an event. + * The journal as an untrusted caller reaches it. * - * Through `unknown` rather than a double assertion: the point of the test is - * that the collector parses what it is given, and manufacturing a value that - * claims to be an event would be asserting the thing under test. + * The collector's job is to parse what it is handed, so a test that proves it + * refuses junk must be able to hand it junk. Widening the parameter is how that + * happens without manufacturing a value that claims to already be an event — + * asserting one would assert away the thing under test. */ -function malformed(value: unknown): DurableEvent { - const offered: unknown = value; - return offered as DurableEvent; +function offering(journal: DurableStream): { append(event: unknown): Operation } { + return journal; } /** Rename a test event in place, to prove the collector cloned it. */ @@ -59,7 +75,7 @@ function rename(entry: DurableEvent, name: string): void { } const description = entry.description; if (description !== null && typeof description === "object") { - (description as Record)["name"] = name; + Object.assign(description, { name }); } } @@ -69,7 +85,7 @@ function event(name: string): DurableEvent { coroutineId: "root", description: { type: "test", name }, result: { status: "ok", value: name }, - } as DurableEvent; + }; } /** A link that records what it was asked, and answers how a test tells it to. */ @@ -207,7 +223,7 @@ describe("a remote transaction", () => { } expect(raised).toBeInstanceOf(RemoteTransactionError); - expect((raised as RemoteTransactionError).refusal).toBe("nested-transaction"); + expect(refusalOf(raised)).toBe("nested-transaction"); expect(sent).toEqual([]); }); @@ -226,7 +242,7 @@ describe("a remote transaction", () => { }); expect(raised).toBeInstanceOf(RemoteTransactionError); - expect((raised as RemoteTransactionError).refusal).toBe("operation-inside-body"); + expect(refusalOf(raised)).toBe("operation-inside-body"); // And the gate is closed again afterwards, so the next operation is fine. requireNoOpenTransaction(gate); }); @@ -247,7 +263,7 @@ describe("a remote transaction", () => { } catch (error) { raised = error; } - expect((raised as RemoteTransactionError).refusal).toBe("transaction-closed"); + expect(refusalOf(raised)).toBe("transaction-closed"); }); it("owns the handle from before the first suspension until after the commit", function* () { @@ -272,7 +288,7 @@ describe("a remote transaction", () => { } catch (error) { raised = error; } - expect((raised as RemoteTransactionError).refusal).toBe("nested-transaction"); + expect(refusalOf(raised)).toBe("nested-transaction"); held.resolve(); yield* first; @@ -300,7 +316,7 @@ describe("a remote transaction", () => { } catch (error) { raised = error; } - expect((raised as RemoteTransactionError).refusal).toBe("operation-inside-body"); + expect(refusalOf(raised)).toBe("operation-inside-body"); held.resolve(); yield* first; @@ -357,13 +373,13 @@ describe("a remote transaction", () => { let raised: unknown; try { yield* transactRemotely(owner, gate, function* (transaction) { - yield* transaction.journal.append(malformed({ nothing: true })); + yield* offering(transaction.journal).append({ nothing: true }); return undefined; }); } catch (error) { raised = error; } - expect((raised as RemoteTransactionError).refusal).toBe("malformed-event"); + expect(refusalOf(raised)).toBe("malformed-event"); expect(sent).toEqual([]); }); @@ -383,7 +399,7 @@ describe("a remote transaction", () => { } catch (error) { raised = error; } - expect((raised as RemoteTransactionError).refusal).toBe("events-too-large"); + expect(refusalOf(raised)).toBe("events-too-large"); expect(sent).toEqual([]); }); From b69ab73e445480652c162587ae8a84d8bad6d4ff Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 15:00:03 -0400 Subject: [PATCH 25/81] =?UTF-8?q?=F0=9F=93=96=20Read=20a=20run=20from=20th?= =?UTF-8?q?e=20owner=20that=20holds=20it=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An admitted connection could prove who it was and ask nothing. This gives it the reads a runner actually needs — the committed frontier, one retained Workspace root, and the content that root names — and the private mechanics those reads depend on. Reads are bounded and coherent. One frontier request returns the parsed run record, the current canonical root and the last journal event that existed at that moment; the journal itself comes back in anchored pages, so a journal larger than one message is still one snapshot and later appends cannot enter an earlier one. Each page carries its own predecessor, and the runner reassembles them by checking rather than trusting: a page that skips, repeats, reorders or ends in the wrong place closes the connection before one event reaches a caller, because half a journal that looks whole is worse than none. A root comes back as its canonical manifest, then one referenced piece at a time. The owner proves the manifest is canonically encoded, that its identity is the digest of its own bytes, that the retained references are exactly the ones the entries name, and that every piece it sends is referenced by that root and hashes to the digest it is asked for. The runner proves it all again on arrival. The owner being honest is not evidence about the wire, and content that is not what it is named must never become a materialization. A runner that hears no answer cannot tell a lost question from a lost answer, so it asks again. That is only safe if asking twice is asking once. Every command is now decided once per acquisition and its decision retained beside it. Two requests are the same request when their parsed commands are equal, so member order does not make a retry into a new command and a changed value does: reusing an id for something else is refused as a conflict rather than answered. Reads whose answers are fixed by immutable state and an anchor the request already carries are remembered as a decision to read again; the frontier is kept whole, because it is the one read whose answer would otherwise move, and a retry that returned a later frontier would hand back a snapshot nobody asked for. The ledger never evicts while the acquisition lives — dropping an id would make a retry look new, which for a mutation is the difference between doing something once and twice — so a full ledger refuses and fails closed. Content a runner offers is staged, and staging is not publication. It is digest-checked, bounded per piece and in aggregate, stored detached, keyed by the acquisition that offered it, and visible to no retained read. Adopting it is a later checkpoint's transaction. Both private tables live in SQLite because an evicted object remembers nothing and the attachment is 16 KiB of identity, not somewhere to grow a ledger; a replacement acquisition discards its predecessor's scratch before it accepts, and touches no retained history doing it. The rules a root is held to now live in one place instead of two. Both hosts retain the same roots and must name them identically, so the manifest format, its canonical encoding and the stored-row parsers moved beside the schema they belong to, and the Deno host reads through the same implementation it always described. The digest is arithmetic in the language: a shared module cannot import a host's crypto, and the one place this is needed most is inside a synchronous transaction, where there is nothing to await into. `commit` and `settle` parse strictly and refuse. Applying them is D3 and D4, and a placeholder that reported success would be the one answer a runner cannot recover from. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/cloudflare.ts | 13 - .../workflow/src/cloudflare/acquisition.ts | 16 +- packages/workflow/src/cloudflare/client.ts | 401 +++++++++++++ packages/workflow/src/cloudflare/commands.ts | 272 +++++---- .../workflow/src/cloudflare/dispatcher.ts | 269 +++++++++ packages/workflow/src/cloudflare/encoding.ts | 57 ++ .../workflow/src/cloudflare/owner-reads.ts | 356 ++++++++++++ packages/workflow/src/cloudflare/owner.ts | 63 +- .../workflow/src/cloudflare/private-schema.ts | 85 +++ .../workflow/src/cloudflare/recognition.ts | 23 +- .../workflow/src/deno/artifact-frontier.ts | 2 +- packages/workflow/src/deno/database.ts | 2 +- packages/workflow/src/deno/lifecycle.ts | 2 +- packages/workflow/src/deno/transitions.ts | 2 +- .../workflow/src/deno/workspace/manifest.ts | 221 ++----- packages/workflow/src/deno/workspace/root.ts | 53 +- packages/workflow/src/remote/client.ts | 11 +- packages/workflow/src/remote/read.ts | 57 ++ packages/workflow/src/remote/records.ts | 143 +++++ .../workflow/src/{deno => sqlite}/rows.ts | 6 + .../workflow/src/workspace/root-manifest.ts | 425 ++++++++++++++ packages/workflow/src/workspace/sha256.ts | 119 ++++ .../cloudflare/executor-acquisition.vitest.ts | 34 +- .../tests/cloudflare/owner-storage.vitest.ts | 7 + .../tests/cloudflare/remote-owner.vitest.ts | 540 ++++++++++++++++++ .../tests/cloudflare/settle-parser.vitest.ts | 14 +- .../cloudflare/support/executor-object.ts | 232 +++++++- .../workflow/tests/host-neutrality.test.ts | 9 + .../workflow/tests/public-entrypoint.test.ts | 17 + packages/workflow/tests/remote-read.test.ts | 429 ++++++++++++++ .../workflow/tests/workflow-export.test.ts | 2 +- 31 files changed, 3493 insertions(+), 389 deletions(-) create mode 100644 packages/workflow/src/cloudflare/client.ts create mode 100644 packages/workflow/src/cloudflare/dispatcher.ts create mode 100644 packages/workflow/src/cloudflare/encoding.ts create mode 100644 packages/workflow/src/cloudflare/owner-reads.ts create mode 100644 packages/workflow/src/cloudflare/private-schema.ts create mode 100644 packages/workflow/src/remote/read.ts create mode 100644 packages/workflow/src/remote/records.ts rename packages/workflow/src/{deno => sqlite}/rows.ts (96%) create mode 100644 packages/workflow/src/workspace/root-manifest.ts create mode 100644 packages/workflow/src/workspace/sha256.ts create mode 100644 packages/workflow/tests/cloudflare/remote-owner.vitest.ts create mode 100644 packages/workflow/tests/remote-read.test.ts diff --git a/packages/workflow/cloudflare.ts b/packages/workflow/cloudflare.ts index 270aef5c8..3b2f079c0 100644 --- a/packages/workflow/cloudflare.ts +++ b/packages/workflow/cloudflare.ts @@ -19,9 +19,6 @@ * protected configuration() { * return { policy: POLICY }; * } - * protected perform(socket, runId, command) { - * // … - * } * } * ``` * @@ -43,15 +40,5 @@ export type { ReleaseRefusal } from "./src/cloudflare/release.ts"; export { admitRunId, ownerFor, RunIdError } from "./src/cloudflare/routing.ts"; export type { OwnerNamespace, RunIdRefusal } from "./src/cloudflare/routing.ts"; -export { - AcquisitionError, - acquisitionHolders, - EXECUTOR_TAG, -} from "./src/cloudflare/acquisition.ts"; -export type { AcquisitionAttachment, AcquisitionRefusal } from "./src/cloudflare/acquisition.ts"; - -export { CommandError } from "./src/cloudflare/commands.ts"; -export type { CommandRefusal, CommandResult, RunnerCommand } from "./src/cloudflare/commands.ts"; - export { WorkflowObjectStorageError } from "./src/cloudflare/recognition.ts"; export type { RecognitionFailure } from "./src/cloudflare/recognition.ts"; diff --git a/packages/workflow/src/cloudflare/acquisition.ts b/packages/workflow/src/cloudflare/acquisition.ts index a8dda2376..c1cd862ec 100644 --- a/packages/workflow/src/cloudflare/acquisition.ts +++ b/packages/workflow/src/cloudflare/acquisition.ts @@ -105,10 +105,12 @@ export function acquireExecutor( socket: WebSocket, runId: string, acquisitionId: string, + beforeAccept: () => void = () => undefined, ): AcquisitionAttachment { if (acquisitionHolders(ctx).length > 0) { throw new AcquisitionError("already-running"); } + beforeAccept(); const attachment: AcquisitionAttachment = { kind: "executor", runId, acquisitionId }; ctx.acceptWebSocket(socket, [EXECUTOR_TAG]); // Bounded, and only what admission needs to be reconstructed after an @@ -129,6 +131,17 @@ export function requireAcquisition( ctx: AcquisitionContext, socket: WebSocket, runId: string, +): AcquisitionAttachment { + const mine = requireExecutorSocket(ctx, socket); + if (mine.runId !== runId) { + throw new AcquisitionError("wrong-run"); + } + return mine; +} + +export function requireExecutorSocket( + ctx: AcquisitionContext, + socket: WebSocket, ): AcquisitionAttachment { const live = acquisitionHolders(ctx); if (live.length === 0) { @@ -143,9 +156,6 @@ export function requireAcquisition( // Two live holders is a state this module refuses to choose between. throw new AcquisitionError("already-running"); } - if (mine.held.runId !== runId) { - throw new AcquisitionError("wrong-run"); - } return mine.held; } diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts new file mode 100644 index 000000000..2cf8ab4e0 --- /dev/null +++ b/packages/workflow/src/cloudflare/client.ts @@ -0,0 +1,401 @@ +/** + * The runner's side of the private protocol. + * + * This is the only place that knows both languages. Above it, `src/remote/**` + * speaks in workflow records and Workspace roots; below it, the connection + * carries private commands and a private refusal union. Translating between + * them here is what keeps the neutral code neutral, and what keeps the private + * shapes private. + * + * Nothing arrives as a semantic value because the owner said so. A performed + * answer is parsed into a record, a manifest or a verified content piece before + * anything above can see it, and a refusal is narrowed to the exact union this + * release declares. Both sides are the same build — admission proved that — so + * a category this build has never heard of is not a new failure to report + * upward, it is a channel that is not what it claims to be, and the connection + * fails closed. + * + * Content is verified again on arrival. The owner validated it before sending, + * and that says nothing about what happened in between; a digest is cheap and + * the alternative is materializing bytes that are not the bytes the root names. + * + * The journal is reassembled here from anchored pages, and the assembly is + * checked rather than assumed: each page must continue the previous one, name + * no event twice, and end exactly at the anchor. A page that skipped, repeated + * or reordered an event closes the connection before a single event reaches a + * caller — half a journal that looks whole is worse than no journal. + */ + +import { Err, type Operation, type Result } from "effection"; +import type { JournalEntry } from "../storage/api.ts"; +import { parseMembers, requireMemberNames } from "../storage/members.ts"; +import type { DefinitionRetrieval, WorkflowRunRecord } from "../storage/record.ts"; +import type { CommitIntent, OwnerLink, StartingFrontier } from "../remote/collector.ts"; +import type { OwnerAnswer, OwnerConnection } from "../remote/client.ts"; +import { + parseRemoteJournalEntry, + parseRemoteRetrieval, + parseRemoteRunRecord, + RemoteRecordError, +} from "../remote/records.ts"; +import { + type RemoteContent, + type RemoteContentRequest, + type RemoteFrontierSnapshot, + type RemoteReadLink, + startingFrontier, +} from "../remote/read.ts"; +import { + decodeDofsManifest, + parseWorkspaceRootManifest, + SHA256, + WORKSPACE_ROOT_DOMAIN, + type WorkspaceRootManifest, +} from "../workspace/root-manifest.ts"; +import { JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES } from "./commands.ts"; +import { decodeBase64, encodeBase64, sha256Hex } from "./encoding.ts"; + +export type PrivateRefusal = + | "acquisition:already-running" + | "acquisition:not-acquired" + | "acquisition:foreign-connection" + | "acquisition:wrong-run" + | "command:not-an-object" + | "command:unknown-command" + | "command:unknown-member" + | "command:malformed-member" + | "command:too-large" + | "command:duplicate-conflict" + | "command:capacity" + | "command:unavailable" + | "storage:foreign" + | "storage:unsupported-version" + | "storage:corrupt"; + +export class CloudflareOwnerRefusalError extends Error { + override name = "CloudflareOwnerRefusalError"; + + constructor(readonly refusal: PrivateRefusal) { + super(`the workflow owner refused the request (${refusal})`); + } +} + +interface FrontierHeader { + readonly record: WorkflowRunRecord; + readonly retrieval: DefinitionRetrieval | undefined; + readonly workspaceRootId: string; + readonly journalEventId: string | null; +} + +interface JournalPage { + readonly anchorEventId: string | null; + readonly afterEventId: string | null; + readonly entries: readonly { + readonly previousEventId: string | null; + readonly entry: JournalEntry; + }[]; + readonly done: boolean; +} + +function fail(reason: string): never { + throw new RemoteRecordError(`the owner returned a malformed private answer: ${reason}`); +} + +function members(value: unknown, names: readonly string[]): Map { + const found = parseMembers(value, "$", (reason) => new RemoteRecordError(reason)); + requireMemberNames(found, names, "$", (reason) => new RemoteRecordError(reason)); + if (found.size !== names.length || names.some((name) => !found.has(name))) { + return fail("it omitted a declared member"); + } + return found; +} + +function rootId(value: unknown): string { + if (typeof value !== "string" || !SHA256.test(value)) { + return fail("it did not name a canonical Workspace root"); + } + return value; +} + +function nullableIdentity(value: unknown): string | null { + if (value === null) { + return null; + } + if (typeof value !== "string" || value === "") { + return fail("it did not name an event identity"); + } + return value; +} + +function privateRefusal(value: string): PrivateRefusal { + switch (value) { + case "acquisition:already-running": + case "acquisition:not-acquired": + case "acquisition:foreign-connection": + case "acquisition:wrong-run": + case "command:not-an-object": + case "command:unknown-command": + case "command:unknown-member": + case "command:malformed-member": + case "command:too-large": + case "command:duplicate-conflict": + case "command:capacity": + case "command:unavailable": + case "storage:foreign": + case "storage:unsupported-version": + case "storage:corrupt": + return value; + default: + return fail("it named an unknown refusal category"); + } +} + +function answer(offered: OwnerAnswer): T { + if (offered.outcome === "refused") { + throw new CloudflareOwnerRefusalError(privateRefusal(offered.refusal)); + } + return offered.value; +} + +function parseFrontier(value: unknown): FrontierHeader { + const found = members(value, ["record", "retrieval", "workspaceRootId", "journalEventId"]); + return { + record: parseRemoteRunRecord(found.get("record")), + retrieval: parseRemoteRetrieval(found.get("retrieval")), + workspaceRootId: rootId(found.get("workspaceRootId")), + journalEventId: nullableIdentity(found.get("journalEventId")), + }; +} + +function parseJournalPage(value: unknown): JournalPage { + const found = members(value, ["anchorEventId", "afterEventId", "entries", "done"]); + const offered = found.get("entries"); + if (!Array.isArray(offered) || offered.length > JOURNAL_PAGE_ENTRIES) { + return fail("it did not contain one bounded journal page"); + } + if (typeof found.get("done") !== "boolean") { + return fail("it did not say whether the journal page was terminal"); + } + return { + anchorEventId: nullableIdentity(found.get("anchorEventId")), + afterEventId: nullableIdentity(found.get("afterEventId")), + entries: offered.map((entry) => { + const item = members(entry, ["eventId", "previousEventId", "record", "workspaceRootId"]); + return { + previousEventId: nullableIdentity(item.get("previousEventId")), + entry: parseRemoteJournalEntry({ + eventId: item.get("eventId"), + record: item.get("record"), + workspaceRootId: item.get("workspaceRootId"), + }), + }; + }), + done: found.get("done") === true, + }; +} + +function parseAnchoredJournalPage( + value: unknown, + anchorEventId: string, + afterEventId: string | null, + seen: ReadonlySet, +): JournalPage { + const page = parseJournalPage(value); + if ( + page.anchorEventId !== anchorEventId || + page.afterEventId !== afterEventId || + page.entries.length === 0 + ) { + return fail("a journal page did not continue its anchored snapshot"); + } + let previous = afterEventId; + const found = new Set(seen); + for (const item of page.entries) { + if (item.previousEventId !== previous) { + return fail("an anchored journal page skipped or reordered an event"); + } + if (found.has(item.entry.eventId)) { + return fail("an anchored journal repeated an event"); + } + found.add(item.entry.eventId); + previous = item.entry.eventId; + } + if ((page.done && previous !== anchorEventId) || (!page.done && previous === anchorEventId)) { + return fail("an anchored journal page disagreed with its terminal event"); + } + return page; +} + +function parseRoot(value: unknown): { workspaceRootId: string; manifest: WorkspaceRootManifest } { + const found = members(value, ["workspaceRootId", "manifest"]); + const identity = rootId(found.get("workspaceRootId")); + const manifest = found.get("manifest"); + if ( + typeof manifest !== "string" || + new TextEncoder().encode(manifest).length > MAX_CONTENT_BYTES + ) { + return fail("it did not contain one bounded root manifest"); + } + const parsed = parseWorkspaceRootManifest(manifest, fail); + if (sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${manifest}`) !== identity) { + return fail("the root manifest disagreed with its identity"); + } + return { workspaceRootId: identity, manifest: parsed }; +} + +function parseContent(value: unknown): RemoteContent { + const found = members(value, ["kind", "digest", "size", "bytes"]); + const kind = found.get("kind"); + if (kind !== "manifest" && kind !== "blob") { + return fail("it did not name a content kind"); + } + const digest = rootId(found.get("digest")); + const size = found.get("size"); + const encoded = found.get("bytes"); + if ( + typeof size !== "number" || + !Number.isSafeInteger(size) || + size < 1 || + size > MAX_CONTENT_BYTES || + typeof encoded !== "string" + ) { + return fail("it did not contain one bounded content piece"); + } + const bytes = decodeBase64(encoded); + if (bytes.length !== size || sha256Hex(bytes) !== digest) { + return fail("the content disagreed with its identity or size"); + } + if (kind === "manifest") { + decodeDofsManifest(bytes, fail); + } + return { kind, digest, bytes }; +} + +export function cloudflareReadLink( + connection: OwnerConnection, + nextId: () => string, + expectedRunId: string, +): RemoteReadLink { + return { + *frontier(): Operation { + const header = answer( + yield* connection.ask( + nextId(), + { command: "frontier" }, + (value) => { + const parsed = parseFrontier(value); + if (parsed.record.runId !== expectedRunId) { + return fail("a frontier answer named another run"); + } + return parsed; + }, + privateRefusal, + ), + ); + const entries: JournalEntry[] = []; + const seen = new Set(); + let afterEventId: string | null = null; + let done = header.journalEventId === null; + while (!done) { + const page: JournalPage = answer( + yield* connection.ask( + nextId(), + { command: "journal", anchorEventId: header.journalEventId, afterEventId }, + (value) => + parseAnchoredJournalPage(value, header.journalEventId ?? "", afterEventId, seen), + privateRefusal, + ), + ); + for (const item of page.entries) { + const entry = item.entry; + seen.add(entry.eventId); + entries.push(entry); + afterEventId = entry.eventId; + } + done = page.done; + } + return { ...header, entries }; + }, + *root(workspaceRootId: string): Operation { + const read = answer( + yield* connection.ask( + nextId(), + { command: "root", workspaceRootId }, + (value) => { + const parsed = parseRoot(value); + if (parsed.workspaceRootId !== workspaceRootId) { + return fail("a root answer named another root"); + } + return parsed; + }, + privateRefusal, + ), + ); + return read.manifest; + }, + *content(workspaceRootId, request: RemoteContentRequest): Operation { + const read = answer( + yield* connection.ask( + nextId(), + { + command: "content", + workspaceRootId, + kind: request.kind, + digest: request.digest, + sourceManifest: request.kind === "blob" ? request.manifestDigest : null, + }, + (value) => { + const parsed = parseContent(value); + if (parsed.kind !== request.kind || parsed.digest !== request.digest) { + return fail("a content answer named another piece"); + } + return parsed; + }, + privateRefusal, + ), + ); + return read; + }, + }; +} + +export function cloudflareOwnerLink(reads: RemoteReadLink): OwnerLink { + return { + *frontier(): Operation { + return startingFrontier(yield* reads.frontier()); + }, + *commit(_intent: CommitIntent): Operation> { + return Err(new CloudflareOwnerRefusalError("command:unavailable")); + }, + }; +} + +export function* stageCloudflareContent( + connection: OwnerConnection, + id: string, + kind: RemoteContent["kind"], + bytes: Uint8Array, +): Operation<{ kind: RemoteContent["kind"]; digest: string; size: number }> { + if (bytes.length === 0 || bytes.length > MAX_CONTENT_BYTES) { + return fail("the staged content is outside the private piece bound"); + } + const digest = sha256Hex(bytes); + return answer( + yield* connection.ask( + id, + { command: "stage", kind, digest, bytes: encodeBase64(bytes) }, + (value) => { + const found = members(value, ["kind", "digest", "size"]); + if ( + found.get("kind") !== kind || + found.get("digest") !== digest || + found.get("size") !== bytes.length + ) { + return fail("a staging answer named another content piece"); + } + return { kind, digest, size: bytes.length }; + }, + privateRefusal, + ), + ); +} diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 9b50c8175..a0ba67fb3 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -1,117 +1,121 @@ -/** - * What a runner asks its owner, and what comes back. - * - * These records are private to one software-factory release. They are not - * journaled, exported, authored, or supported across independently versioned - * builds — admission has already proved both sides are the same build, which is - * what a wire contract would otherwise be for. Their decomposition is - * implementation detail and may change with the release that contains it. - * - * What is not private is the parsing discipline. Everything arriving from the - * connection is parsed strictly before it reaches storage: an unknown command, - * an unknown member, a value of the wrong kind and a value past its bound are - * each refused whole, and nothing is partially adopted. A permissive read here - * would be a runner deciding what the owner does. - */ - import { type DocumentExecutionCompletion, parseDocumentExecutionCompletion, } from "../storage/record.ts"; +import { SHA256 } from "../workspace/root-manifest.ts"; + +export const MAX_MESSAGE_BYTES = 8 * 1024 * 1024; +export const MAX_CONTENT_BYTES = 1024 * 1024; +export const MAX_STAGED_BYTES = 2 * 1024 * 1024; +export const MAX_COMMANDS = 256; +export const MAX_LEDGER_BYTES = 2 * 1024 * 1024; +export const JOURNAL_PAGE_ENTRIES = 128; +export const JOURNAL_PAGE_BYTES = 512 * 1024; -/** The commands a runner may send. */ -export type CommandName = "frontier" | "materialize" | "commit" | "settle"; +export type CommandName = + | "frontier" + | "journal" + | "root" + | "content" + | "stage" + | "commit" + | "settle"; -/** Why a message was refused before it reached the run. */ export type CommandRefusal = | "not-an-object" | "unknown-command" | "unknown-member" | "malformed-member" - | "too-large"; + | "too-large" + | "duplicate-conflict" + | "capacity" + | "unavailable"; export class CommandError extends Error { override name = "CommandError"; constructor(readonly refusal: CommandRefusal) { - // The message a runner sent is not repeated: it arrived from outside and a - // member name can carry as much as a member value. super(`this owner refused a runner command (${refusal})`); } } -/** The envelope every command shares. */ export interface CommandEnvelope { - /** Distinguishes one request from a retry of the same request. */ readonly id: string; readonly command: CommandName; } -/** Read the committed run record and current Workspace root. */ export interface FrontierCommand extends CommandEnvelope { readonly command: "frontier"; } -/** Ask for the content-addressed bytes of one retained root. */ -export interface MaterializeCommand extends CommandEnvelope { - readonly command: "materialize"; +export interface JournalCommand extends CommandEnvelope { + readonly command: "journal"; + readonly anchorEventId: string | null; + readonly afterEventId: string | null; +} + +export interface RootCommand extends CommandEnvelope { + readonly command: "root"; + readonly workspaceRootId: string; +} + +export type ContentKind = "manifest" | "blob"; + +export interface ContentCommand extends CommandEnvelope { + readonly command: "content"; readonly workspaceRootId: string; + readonly kind: ContentKind; + readonly digest: string; + readonly sourceManifest: string | null; +} + +export interface StageCommand extends CommandEnvelope { + readonly command: "stage"; + readonly kind: ContentKind; + readonly digest: string; + readonly bytes: string; } -/** One closed mutation intent, submitted once, applied atomically or not at all. */ export interface CommitCommand extends CommandEnvelope { readonly command: "commit"; - /** The root the runner started from; the owner refuses if it has moved. */ readonly expectedWorkspaceRootId: string; - /** The journal frontier the runner read; the owner refuses if it has moved. */ readonly expectedJournalEventId: string | null; - /** Content-addressed additions, each named by its own digest. */ - readonly content: readonly ContentChunk[]; - /** The canonical root the runner proposes, recomputed by the owner. */ readonly proposedWorkspaceRootId: string; - /** Already-filtered journal events to append, in order. */ readonly events: readonly string[]; } -/** - * Publish how a document execution ended, and what the run becomes. - * - * The completion is the shared provider-neutral record, parsed with the shared - * parser rather than a private approximation — the owner and the local host - * have to agree about what a completion *is*, and two readers of one shape is - * how they stop agreeing. The expected root is carried so the owner can refuse - * a settlement proposed against a frontier that has moved. - */ export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; readonly expectedWorkspaceRootId: string; } -export interface ContentChunk { - readonly digest: string; - /** Base64, because a private transport still carries text. */ - readonly bytes: string; -} - -export type RunnerCommand = FrontierCommand | MaterializeCommand | CommitCommand | SettleCommand; - -/** The largest message this owner reads at all. */ -const MAX_MESSAGE = 8 * 1024 * 1024; - -/** The most chunks one commit may carry. */ -const MAX_CHUNKS = 4096; +export type RunnerCommand = + | FrontierCommand + | JournalCommand + | RootCommand + | ContentCommand + | StageCommand + | CommitCommand + | SettleCommand; -const ENVELOPE = ["id", "command"] as const; +export type CommandResult = + | { readonly id: string; readonly outcome: "performed"; readonly value: unknown } + | { readonly id: string; readonly outcome: "refused"; readonly refusal: string }; +const MAX_ID = 128; +const MAX_EVENTS = 4096; +const ENVELOPE = ["id", "command"]; const MEMBERS: Record = { - frontier: [...ENVELOPE], - materialize: [...ENVELOPE, "workspaceRootId"], + frontier: ENVELOPE, + journal: [...ENVELOPE, "anchorEventId", "afterEventId"], + root: [...ENVELOPE, "workspaceRootId"], + content: [...ENVELOPE, "workspaceRootId", "kind", "digest", "sourceManifest"], + stage: [...ENVELOPE, "kind", "digest", "bytes"], commit: [ ...ENVELOPE, "expectedWorkspaceRootId", "expectedJournalEventId", - "content", "proposedWorkspaceRootId", "events", ], @@ -122,62 +126,68 @@ function object(value: unknown): Map { if (value === null || typeof value !== "object" || Array.isArray(value)) { throw new CommandError("not-an-object"); } - const members: Map = new Map(Object.entries(value)); - return members; + return new Map(Object.entries(value)); } -/** - * A retained Workspace root, as the schema spells one. - * - * The lowercase SHA-256 of a canonical manifest: 64 hexadecimal characters and - * nothing else. Admitting any non-empty string would let three commands hold - * three different notions of what a root is, and the owner would be the first - * to find out. - */ -const ROOT_ID = /^[0-9a-f]{64}$/; - -function rootId(members: Map, key: string): string { - const value = text(members, key); - if (!ROOT_ID.test(value)) { +function closed(members: Map, allowed: readonly string[]): void { + for (const key of members.keys()) { + if (!allowed.includes(key)) { + throw new CommandError("unknown-member"); + } + } + if (members.size !== allowed.length) { throw new CommandError("malformed-member"); } +} + +function text( + members: Map, + key: string, + maximum = Number.MAX_SAFE_INTEGER, +): string { + const value = members.get(key); + if (typeof value !== "string" || value === "" || value.length > maximum) { + throw new CommandError( + value !== "" && typeof value === "string" ? "too-large" : "malformed-member", + ); + } return value; } -function text(members: Map, key: string): string { +function nullableText(members: Map, key: string): string | null { const value = members.get(key); + if (value === null) { + return null; + } if (typeof value !== "string" || value === "") { throw new CommandError("malformed-member"); } return value; } -function closed(members: Map, allowed: readonly string[]): void { - for (const key of members.keys()) { - if (!allowed.includes(key)) { - throw new CommandError("unknown-member"); - } +function digest(members: Map, key: string): string { + const value = members.get(key); + if (typeof value !== "string" || !SHA256.test(value)) { + throw new CommandError("malformed-member"); } + return value; } -function chunks(value: unknown): ContentChunk[] { - if (!Array.isArray(value)) { +function kind(members: Map): ContentKind { + const value = members.get("kind"); + if (value !== "manifest" && value !== "blob") { throw new CommandError("malformed-member"); } - if (value.length > MAX_CHUNKS) { - throw new CommandError("too-large"); - } - return value.map((entry) => { - const members = object(entry); - closed(members, ["digest", "bytes"]); - return { digest: text(members, "digest"), bytes: text(members, "bytes") }; - }); + return value; } -function events(value: unknown): string[] { +function eventRecords(value: unknown): string[] { if (!Array.isArray(value)) { throw new CommandError("malformed-member"); } + if (value.length > MAX_EVENTS) { + throw new CommandError("too-large"); + } return value.map((entry) => { if (typeof entry !== "string" || entry === "") { throw new CommandError("malformed-member"); @@ -186,9 +196,8 @@ function events(value: unknown): string[] { }); } -/** Read one command out of a message nothing has inspected yet. */ export function parseCommand(raw: string): RunnerCommand { - if (raw.length > MAX_MESSAGE) { + if (new TextEncoder().encode(raw).length > MAX_MESSAGE_BYTES) { throw new CommandError("too-large"); } let decoded: unknown; @@ -198,11 +207,14 @@ export function parseCommand(raw: string): RunnerCommand { throw new CommandError("not-an-object"); } const members = object(decoded); - const id = text(members, "id"); + const id = text(members, "id", MAX_ID); const command = members.get("command"); if ( command !== "frontier" && - command !== "materialize" && + command !== "journal" && + command !== "root" && + command !== "content" && + command !== "stage" && command !== "commit" && command !== "settle" ) { @@ -213,13 +225,42 @@ export function parseCommand(raw: string): RunnerCommand { if (command === "frontier") { return { id, command }; } - if (command === "materialize") { - return { id, command, workspaceRootId: rootId(members, "workspaceRootId") }; + if (command === "journal") { + return { + id, + command, + anchorEventId: nullableText(members, "anchorEventId"), + afterEventId: nullableText(members, "afterEventId"), + }; + } + if (command === "root") { + return { id, command, workspaceRootId: digest(members, "workspaceRootId") }; + } + if (command === "content") { + const contentKind = kind(members); + if (contentKind === "manifest" && members.get("sourceManifest") !== null) { + throw new CommandError("malformed-member"); + } + const sourceManifest = contentKind === "manifest" ? null : digest(members, "sourceManifest"); + return { + id, + command, + workspaceRootId: digest(members, "workspaceRootId"), + kind: contentKind, + digest: digest(members, "digest"), + sourceManifest, + }; + } + if (command === "stage") { + return { + id, + command, + kind: kind(members), + digest: digest(members, "digest"), + bytes: text(members, "bytes", Math.ceil((MAX_CONTENT_BYTES * 4) / 3) + 4), + }; } if (command === "settle") { - // The shared parser decides what a completion is. Its failure becomes this - // transport's own closed refusal: the parser's message names members and - // values a request supplied, and none of that belongs on the wire. const completion = parseDocumentExecutionCompletion(members.get("completion")); if (!completion.ok) { throw new CommandError("malformed-member"); @@ -228,32 +269,15 @@ export function parseCommand(raw: string): RunnerCommand { id, command, completion: completion.value, - expectedWorkspaceRootId: rootId(members, "expectedWorkspaceRootId"), + expectedWorkspaceRootId: digest(members, "expectedWorkspaceRootId"), }; } - const expectedJournalEventId = members.get("expectedJournalEventId"); - if (expectedJournalEventId !== null && typeof expectedJournalEventId !== "string") { - throw new CommandError("malformed-member"); - } return { id, command, - expectedWorkspaceRootId: rootId(members, "expectedWorkspaceRootId"), - expectedJournalEventId, - content: chunks(members.get("content")), - proposedWorkspaceRootId: rootId(members, "proposedWorkspaceRootId"), - events: events(members.get("events")), + expectedWorkspaceRootId: digest(members, "expectedWorkspaceRootId"), + expectedJournalEventId: nullableText(members, "expectedJournalEventId"), + proposedWorkspaceRootId: digest(members, "proposedWorkspaceRootId"), + events: eventRecords(members.get("events")), }; } - -/** - * What the owner answers with. - * - * A serialized record rather than an Effection `Result`: this crosses a - * connection, and an `Error` does not survive that. The discriminant is - * `outcome` for the same reason — there is no in-process result being modelled - * here, only what one side told the other. - */ -export type CommandResult = - | { readonly id: string; readonly outcome: "performed"; readonly value: unknown } - | { readonly id: string; readonly outcome: "refused"; readonly refusal: string }; diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts new file mode 100644 index 000000000..7216fd28f --- /dev/null +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -0,0 +1,269 @@ +/** + * Deciding one command, once. + * + * A runner that does not hear an answer cannot tell a lost question from a lost + * answer, so it asks again. That is only safe if asking twice is the same as + * asking once — which is what this arranges. Each command ID is decided once + * within one acquisition, and the decision is retained beside the acquisition + * that made it. + * + * Two requests are the same request when their *parsed* commands are equal. + * Member order and equivalent encodings are not differences; a different value + * is. Reusing an ID for a different request is not a retry, and it is refused + * rather than answered, because answering it would mean one identifier named + * two decisions. + * + * What is retained is the decision, not always the response. A read whose + * answer is fixed by immutable state and a snapshot anchor the request already + * carries is remembered as a decision to read again, and re-reading returns the + * same bytes because the request names what to read. The frontier is the + * exception and is kept whole: it is the one read whose answer would otherwise + * move, and a retry that returned a later frontier would hand a runner a + * snapshot it never asked for. + * + * The ledger is bounded and never evicts. Dropping an older ID would make a + * retry of it look like a new command, which for a mutation is the difference + * between doing something once and doing it twice — so a full ledger refuses + * the new command and fails the connection closed instead. + * + * Everything happens inside one short synchronous transaction, and the exact + * live acquisition is proved twice: before parsing, and again inside the + * transaction, because a socket can close between the two and the transaction + * is where the object actually changes. + */ + +import type { AcquisitionContext } from "./acquisition.ts"; +import { requireAcquisition } from "./acquisition.ts"; +import { + type CommandResult, + CommandError, + MAX_COMMANDS, + MAX_CONTENT_BYTES, + MAX_LEDGER_BYTES, + MAX_STAGED_BYTES, + type RunnerCommand, +} from "./commands.ts"; +import { bytesOf, decodeBase64, sha256Hex } from "./encoding.ts"; +import { readContent, readFrontier, readJournalPage, readRoot } from "./owner-reads.ts"; +import type { OwnerTransactions } from "./owner-transaction.ts"; +import { COMMAND_TABLE, STAGING_TABLE } from "./private-schema.ts"; +import { recognizeObject } from "./recognition.ts"; + +function requestFingerprint(command: RunnerCommand): string { + return sha256Hex(JSON.stringify(command)); +} + +function integer(value: unknown): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + throw new Error("private protocol storage holds a malformed count"); + } + return value; +} + +function storedDecision(value: unknown, id: string): CommandResult | "reconstruct" { + if (typeof value !== "string") { + throw new Error("private protocol storage holds a malformed result"); + } + let parsed: unknown; + try { + parsed = JSON.parse(value); + } catch { + throw new Error("private protocol storage holds a malformed result"); + } + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new Error("private protocol storage holds a malformed result"); + } + const members = new Map(Object.entries(parsed)); + if (members.get("id") !== id) { + throw new Error("private protocol storage holds a result for another command"); + } + const outcome = members.get("outcome"); + if (outcome === "reconstruct" && members.size === 2) { + return "reconstruct"; + } + if (outcome === "performed" && members.size === 3 && members.has("value")) { + return { id, outcome, value: members.get("value") }; + } + const refusal = members.get("refusal"); + if (outcome === "refused" && members.size === 3 && typeof refusal === "string") { + return { id, outcome, refusal }; + } + throw new Error("private protocol storage holds a malformed result"); +} + +function retainedDecision(command: RunnerCommand, result: CommandResult): string { + if ( + result.outcome === "performed" && + (command.command === "journal" || command.command === "root" || command.command === "content") + ) { + return JSON.stringify({ id: command.id, outcome: "reconstruct" }); + } + return JSON.stringify(result); +} + +function sameBytes(left: Uint8Array, right: Uint8Array): boolean { + if (left.length !== right.length) { + return false; + } + let difference = 0; + for (let index = 0; index < left.length; index += 1) { + difference |= (left[index] ?? 0) ^ (right[index] ?? 0); + } + return difference === 0; +} + +function stage( + ctx: AcquisitionContext, + acquisitionId: string, + command: Extract, +): { kind: "manifest" | "blob"; digest: string; size: number } { + const bytes = decodeBase64(command.bytes); + if (bytes.length === 0 || bytes.length > MAX_CONTENT_BYTES) { + throw new CommandError(bytes.length === 0 ? "malformed-member" : "too-large"); + } + if (sha256Hex(bytes) !== command.digest) { + throw new CommandError("malformed-member"); + } + const existing = ctx.storage.sql + .exec( + `SELECT size, bytes FROM ${STAGING_TABLE} + WHERE acquisition_id = ? AND kind = ? AND digest = ?`, + acquisitionId, + command.kind, + command.digest, + ) + .toArray()[0]; + if (existing !== undefined) { + const retained = bytesOf(existing["bytes"]); + if (!sameBytes(retained, bytes)) { + throw new Error("private staging disagrees with its content identity"); + } + return { kind: command.kind, digest: command.digest, size: bytes.length }; + } + const total = ctx.storage.sql + .exec( + `SELECT coalesce(sum(size), 0) AS total FROM ${STAGING_TABLE} WHERE acquisition_id = ?`, + acquisitionId, + ) + .toArray()[0]; + if (integer(total?.["total"]) + bytes.length > MAX_STAGED_BYTES) { + throw new CommandError("capacity"); + } + ctx.storage.sql.exec( + `INSERT INTO ${STAGING_TABLE} (acquisition_id, kind, digest, size, bytes) + VALUES (?, ?, ?, ?, ?)`, + acquisitionId, + command.kind, + command.digest, + bytes.length, + new Uint8Array(bytes), + ); + return { kind: command.kind, digest: command.digest, size: bytes.length }; +} + +function perform( + ctx: AcquisitionContext, + runId: string, + acquisitionId: string, + command: RunnerCommand, +): CommandResult { + if (command.command === "frontier") { + return { id: command.id, outcome: "performed", value: readFrontier(ctx.storage, runId) }; + } + if (command.command === "journal") { + return { + id: command.id, + outcome: "performed", + value: readJournalPage(ctx.storage, command.anchorEventId, command.afterEventId), + }; + } + if (command.command === "root") { + return { + id: command.id, + outcome: "performed", + value: readRoot(ctx.storage, command.workspaceRootId), + }; + } + if (command.command === "content") { + return { + id: command.id, + outcome: "performed", + value: readContent( + ctx.storage, + command.workspaceRootId, + command.kind, + command.digest, + command.sourceManifest, + ), + }; + } + if (command.command === "stage") { + return { id: command.id, outcome: "performed", value: stage(ctx, acquisitionId, command) }; + } + return { id: command.id, outcome: "refused", refusal: "command:unavailable" }; +} + +export function dispatchCommand( + ctx: AcquisitionContext, + transactions: OwnerTransactions, + socket: WebSocket, + runId: string, + command: RunnerCommand, +): CommandResult { + const held = requireAcquisition(ctx, socket, runId); + const fingerprint = requestFingerprint(command); + return transactions.run(ctx.storage, () => { + const inside = requireAcquisition(ctx, socket, runId); + if (inside.acquisitionId !== held.acquisitionId) { + throw new CommandError("duplicate-conflict"); + } + recognizeObject(ctx.storage); + const previous = ctx.storage.sql + .exec( + `SELECT request_fingerprint, response FROM ${COMMAND_TABLE} + WHERE acquisition_id = ? AND command_id = ?`, + held.acquisitionId, + command.id, + ) + .toArray()[0]; + if (previous !== undefined) { + if (previous.request_fingerprint !== fingerprint) { + throw new CommandError("duplicate-conflict"); + } + const decision = storedDecision(previous.response, command.id); + return decision === "reconstruct" + ? perform(ctx, runId, held.acquisitionId, command) + : decision; + } + const usage = ctx.storage.sql + .exec( + `SELECT count(*) AS commands, coalesce(sum(response_bytes), 0) AS bytes + FROM ${COMMAND_TABLE} WHERE acquisition_id = ?`, + held.acquisitionId, + ) + .toArray()[0]; + if ( + integer(usage?.["commands"]) >= MAX_COMMANDS || + integer(usage?.["bytes"]) >= MAX_LEDGER_BYTES + ) { + throw new CommandError("capacity"); + } + const result = perform(ctx, runId, held.acquisitionId, command); + const encoded = retainedDecision(command, result); + const responseBytes = new TextEncoder().encode(encoded).length; + if (integer(usage?.["bytes"]) + responseBytes > MAX_LEDGER_BYTES) { + throw new CommandError("capacity"); + } + ctx.storage.sql.exec( + `INSERT INTO ${COMMAND_TABLE} + (acquisition_id, command_id, request_fingerprint, response, response_bytes) + VALUES (?, ?, ?, ?, ?)`, + held.acquisitionId, + command.id, + fingerprint, + encoded, + responseBytes, + ); + return result; + }); +} diff --git a/packages/workflow/src/cloudflare/encoding.ts b/packages/workflow/src/cloudflare/encoding.ts new file mode 100644 index 000000000..b99308f3a --- /dev/null +++ b/packages/workflow/src/cloudflare/encoding.ts @@ -0,0 +1,57 @@ +/** + * The two encodings the private protocol carries bytes and identities in. + * + * A WebSocket text frame carries text, and content-addressed bytes are not + * text, so base64 is what the private protocol uses. It is canonical in both + * directions: a value that decodes and then re-encodes to something else is + * refused rather than accepted as though the difference did not matter, because + * a digest is taken over bytes and two spellings of one byte sequence would be + * two names for one piece of content. + * + * `bytesOf` is the storage side of the same question. SQLite hands back a blob + * as whatever the runtime models one as, and a column that is not bytes at all + * is damage rather than something to coerce. + */ + +import { CommandError } from "./commands.ts"; +export { sha256Hex } from "../workspace/sha256.ts"; + +export function encodeBase64(bytes: Uint8Array): string { + let binary = ""; + const stride = 32 * 1024; + for (let offset = 0; offset < bytes.length; offset += stride) { + binary += String.fromCharCode(...bytes.slice(offset, offset + stride)); + } + return btoa(binary); +} + +export function decodeBase64(value: string): Uint8Array { + if (value === "" || value.length % 4 !== 0 || !/^[A-Za-z0-9+/]+={0,2}$/.test(value)) { + throw new CommandError("malformed-member"); + } + let binary: string; + try { + binary = atob(value); + } catch { + throw new CommandError("malformed-member"); + } + const bytes = Uint8Array.from(binary, (character) => character.charCodeAt(0)); + if (encodeBase64(bytes) !== value) { + throw new CommandError("malformed-member"); + } + return bytes; +} + +export function bytesOf(value: unknown): Uint8Array { + if (value instanceof Uint8Array) { + return new Uint8Array(value); + } + if (value instanceof ArrayBuffer) { + return new Uint8Array(value.slice(0)); + } + throw new Error("stored bytes are not a byte sequence"); +} + +export function hex(bytes: Uint8Array): string { + return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); +} diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts new file mode 100644 index 000000000..1598faa0e --- /dev/null +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -0,0 +1,356 @@ +/** + * What the owner answers a read with, read out of its own storage. + * + * Every value here is rebuilt from checked columns. The rows are this object's + * own and were written by this build, which is a reason to expect them to be + * right and no reason at all to skip asking: a row that does not parse is + * storage damage, and storage damage answered as though it were a workflow + * value is how damage travels. + * + * Three properties hold the reads together. The frontier is *coherent*: the run + * record, the current root and the journal anchor are read as one, and the + * anchor is the last event that existed at that moment, so later appends cannot + * enter an earlier snapshot. Reads are *bounded*: a journal is returned in + * pages anchored to that event, and content comes back one piece at a time. + * Reads are *referenced*: a piece of content is admitted only if the named root + * actually names it, directly or through a manifest it names, so this is a read + * of one retained root rather than of a content-addressed store. + * + * A refusal says the category and nothing else. Column values, retained JSON + * and request data never appear in one: the caller learns that storage is + * damaged, which is the only thing it can act on. + */ + +import { parseDurableEvent } from "@executablemd/durable-streams"; +import { readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; +import { + decodeDofsManifest, + parseWorkspaceRootManifest, + SHA256, + WORKSPACE_ROOT_DOMAIN, + type WorkspaceRootManifest, +} from "../workspace/root-manifest.ts"; +import { + CommandError, + JOURNAL_PAGE_BYTES, + JOURNAL_PAGE_ENTRIES, + MAX_CONTENT_BYTES, +} from "./commands.ts"; +import { bytesOf, encodeBase64, sha256Hex } from "./encoding.ts"; +import type { OwnerStorage } from "./storage.ts"; + +export interface FrontierValue { + readonly record: ReturnType; + readonly retrieval: ReturnType | null; + readonly workspaceRootId: string; + readonly journalEventId: string | null; +} + +export interface JournalPageValue { + readonly anchorEventId: string | null; + readonly afterEventId: string | null; + readonly entries: readonly { + readonly eventId: string; + readonly previousEventId: string | null; + readonly record: string; + readonly workspaceRootId: string; + }[]; + readonly done: boolean; +} + +export interface RootValue { + readonly workspaceRootId: string; + readonly manifest: string; +} + +export interface ContentValue { + readonly kind: "manifest" | "blob"; + readonly digest: string; + readonly size: number; + readonly bytes: string; +} + +interface StoredRoot { + readonly manifest: string; + readonly parsed: WorkspaceRootManifest; + readonly manifests: ReadonlySet; +} + +function corrupt(reason: string): never { + throw new WorkflowRecordMalformedError("workflow owner storage", reason); +} + +function exactlyOne(rows: Row[], name: string): Row { + if (rows.length !== 1 || rows[0] === undefined) { + return corrupt(`expected exactly one ${name} row`); + } + return rows[0]; +} + +function safeText(row: Row, column: string): string { + const value = row[column]; + if (typeof value !== "string" || value === "") { + return corrupt(`expected ${column} to be non-empty text`); + } + return value; +} + +function safeInteger(value: unknown, name: string): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + return corrupt(`expected ${name} to be a nonnegative whole number`); + } + return value; +} + +function rootIdentity(manifest: string): string { + return sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${manifest}`); +} + +function byteRows(storage: OwnerStorage, sql: string, ...bindings: unknown[]): Row[] { + return storage.sql.exec(sql, ...bindings).toArray(); +} + +function referencedRoot(storage: OwnerStorage, rootId: string): StoredRoot { + if (!SHA256.test(rootId)) { + return corrupt("a Workspace root identity is malformed"); + } + const root = exactlyOne( + byteRows( + storage, + "SELECT root_id, format_version, manifest FROM workspace_roots WHERE root_id = ?", + rootId, + ), + "Workspace root", + ); + const manifest = safeText(root, "manifest"); + if (root["format_version"] !== 1) { + return corrupt("a Workspace root has an unsupported format"); + } + const parsed = parseWorkspaceRootManifest(manifest, corrupt); + if (rootIdentity(manifest) !== rootId || root["root_id"] !== rootId) { + return corrupt("a Workspace root disagrees with its identity"); + } + if (new TextEncoder().encode(manifest).length > MAX_CONTENT_BYTES) { + throw new CommandError("too-large"); + } + + const expectedManifests = new Set( + parsed.entries.flatMap((entry) => (entry.kind === "file" ? [entry.manifest] : [])), + ); + const manifestRows = byteRows( + storage, + `SELECT lower(hex(manifest_hash)) AS digest + FROM workspace_root_manifest_refs WHERE root_id = ? ORDER BY digest`, + rootId, + ); + if (manifestRows.length !== expectedManifests.size) { + return corrupt("a Workspace root's manifest references are incomplete"); + } + const manifests = new Set(); + for (const row of manifestRows) { + const digest = safeText(row, "digest"); + if (!expectedManifests.has(digest)) { + return corrupt("a Workspace root has an extra manifest reference"); + } + manifests.add(digest); + } + return { manifest, parsed, manifests }; +} + +function retainedManifest( + storage: OwnerStorage, + root: StoredRoot, + digest: string, +): { bytes: Uint8Array; chunks: ReturnType["chunks"] } { + if (!root.manifests.has(digest)) { + return corrupt("a DOFS manifest is not referenced by this Workspace root"); + } + const row = exactlyOne( + byteRows(storage, "SELECT size, encoded FROM vfs_manifests WHERE lower(hex(hash)) = ?", digest), + "DOFS manifest", + ); + const bytes = bytesOf(row["encoded"]); + if (bytes.length > MAX_CONTENT_BYTES || sha256Hex(bytes) !== digest) { + return corrupt("a retained DOFS manifest disagrees with its identity"); + } + const decoded = decodeDofsManifest(bytes, corrupt); + if (safeInteger(row["size"], "manifest size") !== decoded.size) { + return corrupt("a retained DOFS manifest disagrees with its recorded size"); + } + for (const entry of root.parsed.entries) { + if (entry.kind === "file" && entry.manifest === digest && entry.size !== decoded.size) { + return corrupt("a Workspace file size disagrees with its retained manifest"); + } + } + return { bytes, chunks: decoded.chunks }; +} + +function retainedBlob( + storage: OwnerStorage, + rootId: string, + root: StoredRoot, + sourceManifest: string, + digest: string, +): Uint8Array { + const manifest = retainedManifest(storage, root, sourceManifest); + const expected = manifest.chunks.find((chunk) => chunk.hash === digest); + if (expected === undefined) { + return corrupt("a blob is not referenced by the named DOFS manifest"); + } + const row = exactlyOne( + byteRows( + storage, + `SELECT b.size, x.bytes FROM workspace_root_blob_refs AS r + JOIN vfs_blobs AS b ON b.hash = r.blob_hash + JOIN vfs_blob_bytes AS x ON x.hash = r.blob_hash + WHERE r.root_id = ? AND lower(hex(r.blob_hash)) = ?`, + rootId, + digest, + ), + "DOFS blob", + ); + const bytes = bytesOf(row["bytes"]); + if ( + bytes.length > MAX_CONTENT_BYTES || + bytes.length !== expected.size || + safeInteger(row["size"], "blob size") !== expected.size || + sha256Hex(bytes) !== digest + ) { + return corrupt("a retained DOFS blob disagrees with its identity or size"); + } + return bytes; +} + +export function readFrontier(storage: OwnerStorage, runId: string): FrontierValue { + const record = readRunRecord( + exactlyOne( + byteRows( + storage, + `SELECT run_id, definition, base, props, status, + stop_reason_kind, stop_reason_code, stop_reason_event_id, + created_at, updated_at FROM workflow_run`, + ), + "workflow run", + ), + ); + if (record.runId !== runId) { + return corrupt("the retained run identity does not address this owner"); + } + const state = exactlyOne( + byteRows(storage, "SELECT current_root_id FROM workspace_state WHERE singleton_id = 1"), + "Workspace state", + ); + const workspaceRootId = safeText(state, "current_root_id"); + referencedRoot(storage, workspaceRootId); + const retrievalRows = byteRows( + storage, + "SELECT metadata, revision, updated_at FROM definition_retrieval WHERE id = 1", + ); + if (retrievalRows.length > 1) { + return corrupt("the definition retrieval is not a singleton"); + } + const last = byteRows( + storage, + "SELECT event_id FROM journal_events ORDER BY sequence DESC LIMIT 1", + )[0]; + return { + record, + retrieval: retrievalRows[0] === undefined ? null : readRetrieval(retrievalRows[0]), + workspaceRootId, + journalEventId: last === undefined ? null : safeText(last, "event_id"), + }; +} + +export function readJournalPage( + storage: OwnerStorage, + anchorEventId: string | null, + afterEventId: string | null, +): JournalPageValue { + if (anchorEventId === null) { + if (afterEventId !== null) { + return corrupt("an empty journal snapshot names an earlier event"); + } + return { anchorEventId, afterEventId, entries: [], done: true }; + } + const anchor = exactlyOne( + byteRows(storage, "SELECT sequence FROM journal_events WHERE event_id = ?", anchorEventId), + "journal anchor", + ); + const anchorSequence = safeInteger(anchor["sequence"], "journal anchor sequence"); + let afterSequence = 0; + if (afterEventId !== null) { + const after = exactlyOne( + byteRows(storage, "SELECT sequence FROM journal_events WHERE event_id = ?", afterEventId), + "journal cursor", + ); + afterSequence = safeInteger(after["sequence"], "journal cursor sequence"); + if (afterSequence >= anchorSequence) { + return corrupt("a journal cursor is outside its anchored snapshot"); + } + } + const rows = byteRows( + storage, + `SELECT event_id, record, workspace_root_id, + (SELECT event_id FROM journal_events AS predecessor + WHERE predecessor.sequence < event.sequence + ORDER BY predecessor.sequence DESC LIMIT 1) AS previous_event_id + FROM journal_events AS event + WHERE sequence > ? AND sequence <= ? ORDER BY sequence ASC LIMIT ?`, + afterSequence, + anchorSequence, + JOURNAL_PAGE_ENTRIES + 1, + ); + const entries: JournalPageValue["entries"][number][] = []; + let encodedBytes = 0; + for (const row of rows.slice(0, JOURNAL_PAGE_ENTRIES)) { + const eventId = safeText(row, "event_id"); + const previous = row["previous_event_id"]; + if (previous !== null && typeof previous !== "string") { + return corrupt("a journal predecessor identity is malformed"); + } + const record = safeText(row, "record"); + const workspaceRootId = safeText(row, "workspace_root_id"); + if (!SHA256.test(workspaceRootId) || !parseDurableEvent(record).ok) { + return corrupt("a journal row is malformed"); + } + const entry = { eventId, previousEventId: previous, record, workspaceRootId }; + const nextBytes = new TextEncoder().encode(JSON.stringify(entry)).length; + if (entries.length > 0 && encodedBytes + nextBytes > JOURNAL_PAGE_BYTES) { + break; + } + if (nextBytes > MAX_CONTENT_BYTES) { + throw new CommandError("too-large"); + } + entries.push(entry); + encodedBytes += nextBytes; + } + const done = rows.length <= entries.length; + if (done && entries.at(-1)?.eventId !== anchorEventId) { + return corrupt("an anchored journal snapshot is incomplete"); + } + return { anchorEventId, afterEventId, entries, done }; +} + +export function readRoot(storage: OwnerStorage, workspaceRootId: string): RootValue { + const root = referencedRoot(storage, workspaceRootId); + return { workspaceRootId, manifest: root.manifest }; +} + +export function readContent( + storage: OwnerStorage, + workspaceRootId: string, + kind: "manifest" | "blob", + digest: string, + sourceManifest: string | null, +): ContentValue { + const root = referencedRoot(storage, workspaceRootId); + const bytes = + kind === "manifest" + ? retainedManifest(storage, root, digest).bytes + : retainedBlob(storage, workspaceRootId, root, sourceManifest ?? "", digest); + if (bytes.length === 0) { + return corrupt("a retained content piece is empty"); + } + return { kind, digest, size: bytes.length, bytes: encodeBase64(bytes) }; +} diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index 3f68d0a5f..cf22f1c6f 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -29,10 +29,14 @@ import { AcquisitionError, releaseExecutor, requireAcquisition, + requireExecutorSocket, } from "./acquisition.ts"; import { admitToken, type AdmissionPolicy, AdmissionError } from "./admission.ts"; import { TokenError, type TokenVerification } from "./token.ts"; import { CommandError, type CommandResult, parseCommand, type RunnerCommand } from "./commands.ts"; +import { dispatchCommand } from "./dispatcher.ts"; +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; +import { discardPriorAcquisitions, PRIVATE_OBJECT_NAMES } from "./private-schema.ts"; import { declaredObjects, initializeObject, @@ -89,6 +93,17 @@ export function refusalOf(error: unknown): string { if (error instanceof WorkflowObjectStorageError) { return `storage:${error.failure.kind}`; } + if (error instanceof WorkflowRecordMalformedError) { + return "storage:corrupt"; + } + if ( + error instanceof Error && + (error.message.startsWith("private protocol storage") || + error.message.startsWith("private staging") || + error.message.startsWith("stored bytes")) + ) { + return "storage:corrupt"; + } return "internal"; } @@ -132,7 +147,16 @@ export abstract class WorkflowOwnerObject extends DurableObject { requireSameRelease(policy.release, request.release); yield* admitToken(policy, verification, request.token); const runId = admitRunId(request.runId); - return acquireExecutor(this.ctx, socket, runId, mintAcquisitionId()); + const acquisitionId = mintAcquisitionId(); + return acquireExecutor(this.ctx, socket, runId, acquisitionId, () => { + const names = new Set(declaredObjects(this.owned).map((object) => object.name)); + if (PRIVATE_OBJECT_NAMES.every((name) => names.has(name))) { + recognizeObject(this.owned); + this.transactions.run(this.owned, () => { + discardPriorAcquisitions(this.owned, acquisitionId); + }); + } + }); } /** @@ -147,14 +171,36 @@ export abstract class WorkflowOwnerObject extends DurableObject { try { requireAcquisition(this.ctx, socket, runId); command = parseCommand(raw); - return { id: command.id, outcome: "performed", value: this.perform(socket, runId, command) }; + return dispatchCommand(this.ctx, this.transactions, socket, runId, command); } catch (error) { return { id: command?.id ?? "", outcome: "refused", refusal: refusalOf(error) }; } } - /** What each command does. Subclasses supply the behavior this owner has. */ - protected abstract perform(socket: WebSocket, runId: string, command: RunnerCommand): unknown; + webSocketMessage(socket: WebSocket, message: string | ArrayBuffer): void { + let answer: CommandResult; + if (typeof message !== "string") { + answer = { id: "", outcome: "refused", refusal: "command:malformed-member" }; + } else { + try { + const held = requireExecutorSocket(this.ctx, socket); + answer = this.onRunnerMessage(socket, held.runId, message); + } catch (error) { + answer = { id: "", outcome: "refused", refusal: refusalOf(error) }; + } + } + try { + socket.send(JSON.stringify(answer)); + } catch { + releaseExecutor(socket); + socket.close(1011, "send failed"); + return; + } + if (fatal(answer)) { + releaseExecutor(socket); + socket.close(1002, "protocol refused"); + } + } /** A connection that ended owns nothing, and rolled nothing back. */ webSocketClose(socket: WebSocket): void { @@ -196,3 +242,12 @@ function mintAcquisitionId(): string { crypto.getRandomValues(bytes); return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); } + +function fatal(answer: CommandResult): boolean { + if (answer.outcome === "performed") { + return false; + } + return ( + answer.refusal !== "command:duplicate-conflict" && answer.refusal !== "command:unavailable" + ); +} diff --git a/packages/workflow/src/cloudflare/private-schema.ts b/packages/workflow/src/cloudflare/private-schema.ts new file mode 100644 index 000000000..4353cf56e --- /dev/null +++ b/packages/workflow/src/cloudflare/private-schema.ts @@ -0,0 +1,85 @@ +/** + * The scratch state one acquisition keeps, and nothing else keeps. + * + * Hibernation is why this is in SQLite rather than in a field or an attachment. + * An idle Durable Object is evicted while its sockets stay open, so anything + * held in memory is gone by the time the next message arrives; and the + * attachment is bounded at 16 KiB and is the compact acquisition identity, not + * somewhere to put a growing ledger or a content payload. + * + * Two tables, both keyed by the owner-minted acquisition ID. One remembers what + * each command ID already decided, so a retry returns the decision rather than + * acting twice. The other holds content a runner has offered but nothing has + * adopted. + * + * Neither is run state. Staged bytes are not published content: they are in no + * root, referenced by nothing, invisible to every retained read, and adopting + * them is a later checkpoint's transaction to perform. Both are declared here + * rather than in the shared logical schema for exactly that reason — they are + * this adapter's physical scratch, and a host that had no hibernation would + * need neither. + * + * Recognition checks their exact shapes like any other declared object. Storage + * carrying a table this build did not write is refused rather than tolerated + * because its name looked familiar. + */ + +import { normalize, type SchemaObject } from "../sqlite/workflow-schema.ts"; +import type { OwnerStorage } from "./storage.ts"; + +export const COMMAND_TABLE = "_xmd_executor_commands"; +export const STAGING_TABLE = "_xmd_executor_staging"; + +const COMMAND_SQL = `CREATE TABLE ${COMMAND_TABLE} ( + acquisition_id TEXT NOT NULL, + command_id TEXT NOT NULL, + request_fingerprint TEXT NOT NULL CHECK ( + length(request_fingerprint) = 64 AND request_fingerprint NOT GLOB '*[^0-9a-f]*' + ), + response TEXT NOT NULL CHECK (json_valid(response)), + response_bytes INTEGER NOT NULL CHECK (response_bytes >= 0), + PRIMARY KEY (acquisition_id, command_id) +) STRICT, WITHOUT ROWID`; + +const STAGING_SQL = `CREATE TABLE ${STAGING_TABLE} ( + acquisition_id TEXT NOT NULL, + kind TEXT NOT NULL CHECK (kind IN ('manifest', 'blob')), + digest TEXT NOT NULL CHECK ( + length(digest) = 64 AND digest NOT GLOB '*[^0-9a-f]*' + ), + size INTEGER NOT NULL CHECK (size > 0), + bytes BLOB NOT NULL, + PRIMARY KEY (acquisition_id, kind, digest) +) STRICT, WITHOUT ROWID`; + +const PRIVATE_OBJECTS = new Map([ + [COMMAND_TABLE, { type: "table", sql: COMMAND_SQL }], + [STAGING_TABLE, { type: "table", sql: STAGING_SQL }], +]); + +export const PRIVATE_OBJECT_NAMES: readonly string[] = Object.freeze([...PRIVATE_OBJECTS.keys()]); + +export function initializePrivateSchema(storage: OwnerStorage): void { + storage.sql.exec(`${COMMAND_SQL};\n\n${STAGING_SQL};`); +} + +export function privateStructureFailure( + objects: readonly SchemaObject[], +): { kind: "missing" | "misshapen"; name: string } | undefined { + const byName = new Map(objects.map((object) => [object.name, object])); + for (const [name, expected] of PRIVATE_OBJECTS) { + const found = byName.get(name); + if (found === undefined) { + return { kind: "missing", name }; + } + if (found.type !== expected.type || normalize(found.sql) !== normalize(expected.sql)) { + return { kind: "misshapen", name }; + } + } + return undefined; +} + +export function discardPriorAcquisitions(storage: OwnerStorage, acquisitionId: string): void { + storage.sql.exec(`DELETE FROM ${COMMAND_TABLE} WHERE acquisition_id <> ?`, acquisitionId); + storage.sql.exec(`DELETE FROM ${STAGING_TABLE} WHERE acquisition_id <> ?`, acquisitionId); +} diff --git a/packages/workflow/src/cloudflare/recognition.ts b/packages/workflow/src/cloudflare/recognition.ts index 7ee4bd6cd..e283e5b4d 100644 --- a/packages/workflow/src/cloudflare/recognition.ts +++ b/packages/workflow/src/cloudflare/recognition.ts @@ -25,6 +25,11 @@ import { type SchemaObject, } from "../sqlite/workflow-schema.ts"; import { isSchemaMarker, MARKER_SQL, MARKER_TABLE, readMarker } from "./marker.ts"; +import { + initializePrivateSchema, + PRIVATE_OBJECT_NAMES, + privateStructureFailure, +} from "./private-schema.ts"; import type { OwnerTransactions } from "./owner-transaction.ts"; import type { OwnerStorage } from "./storage.ts"; @@ -101,6 +106,7 @@ export function initializeObject( transactions.run(storage, ({ dofs }) => { storage.sql.exec(SCHEMA_SQL); initializeDofsSchema(dofs, () => 0); + initializePrivateSchema(storage); initializeRun(); storage.sql.exec(MARKER_SQL); storage.sql.exec( @@ -161,7 +167,22 @@ export function recognizeObject(storage: OwnerStorage): void { }); } - const declared = objects.filter((object) => object.name !== MARKER_TABLE); + const privateObjects = objects.filter((object) => PRIVATE_OBJECT_NAMES.includes(object.name)); + const privateFailure = privateStructureFailure(privateObjects); + if (privateFailure !== undefined) { + throw new WorkflowObjectStorageError({ + kind: "corrupt", + detail: + privateFailure.kind === "missing" + ? `it is missing the table ${privateFailure.name}` + : `its ${privateFailure.name} object is not shaped the way version ${SCHEMA_VERSION} declares it`, + }); + } + + const privateNames = new Set(PRIVATE_OBJECT_NAMES); + const declared = objects.filter( + (object) => object.name !== MARKER_TABLE && !privateNames.has(object.name), + ); const failure = declaredStructureFailure(declared); if (failure === undefined) { return; diff --git a/packages/workflow/src/deno/artifact-frontier.ts b/packages/workflow/src/deno/artifact-frontier.ts index c8957363d..e0f3aa5d8 100644 --- a/packages/workflow/src/deno/artifact-frontier.ts +++ b/packages/workflow/src/deno/artifact-frontier.ts @@ -44,7 +44,7 @@ import type { RetainedBlob, RetainedManifest } from "./fork-source.ts"; import { readForkLineage } from "./fork-write.ts"; import { readRepositories, readRetainedRows, readWorktrees } from "./fork-source.ts"; import { reading } from "./reading.ts"; -import { readDocumentExecution, readRetrieval } from "./rows.ts"; +import { readDocumentExecution, readRetrieval } from "../sqlite/rows.ts"; import { readAllAgentSessions } from "./workspace/agent-sessions.ts"; import { bytes, integer } from "./workspace/manifest.ts"; import { diff --git a/packages/workflow/src/deno/database.ts b/packages/workflow/src/deno/database.ts index 2b439d65f..00e262223 100644 --- a/packages/workflow/src/deno/database.ts +++ b/packages/workflow/src/deno/database.ts @@ -61,7 +61,7 @@ import { holdsTransactionOn, useTransactionSavepoints, } from "./transaction.ts"; -import { readDocumentExecution, readRetrieval, readRunRecord } from "./rows.ts"; +import { readDocumentExecution, readRetrieval, readRunRecord } from "../sqlite/rows.ts"; import { reading } from "./reading.ts"; import { translateSqliteError } from "./schema.ts"; diff --git a/packages/workflow/src/deno/lifecycle.ts b/packages/workflow/src/deno/lifecycle.ts index 22eced242..a43c7d98f 100644 --- a/packages/workflow/src/deno/lifecycle.ts +++ b/packages/workflow/src/deno/lifecycle.ts @@ -104,7 +104,7 @@ import { readRetrievalMetadata, } from "./artifact-frontier.ts"; import type { WorkflowExportRequest, WorkflowExportResult } from "../lifecycle/export.ts"; -import { readDocumentExecution, readRetrieval, readRunRecord } from "./rows.ts"; +import { readDocumentExecution, readRetrieval, readRunRecord } from "../sqlite/rows.ts"; import { translateSqliteError, verifySchema, WorkflowReadonlyRollbackError } from "./schema.ts"; import { holdRecoveryCoordination } from "./recovery-coordination.ts"; diff --git a/packages/workflow/src/deno/transitions.ts b/packages/workflow/src/deno/transitions.ts index 1c365834d..27808c7a7 100644 --- a/packages/workflow/src/deno/transitions.ts +++ b/packages/workflow/src/deno/transitions.ts @@ -60,7 +60,7 @@ import { reading } from "./reading.ts"; import { readJournalEntries } from "./journal.ts"; import type { ForkSourceSnapshot } from "./fork-source.ts"; import { readForkLineage, writeForkInheritance, type ForkHeadEvents } from "./fork-write.ts"; -import { readDocumentExecution, readRetrieval, stopReasonColumns } from "./rows.ts"; +import { readDocumentExecution, readRetrieval, stopReasonColumns } from "../sqlite/rows.ts"; import { initializeSchema, isSqliteForeignKeyConstraint, diff --git a/packages/workflow/src/deno/workspace/manifest.ts b/packages/workflow/src/deno/workspace/manifest.ts index 47c0b3893..e57194887 100644 --- a/packages/workflow/src/deno/workspace/manifest.ts +++ b/packages/workflow/src/deno/workspace/manifest.ts @@ -1,58 +1,43 @@ import { createHash } from "node:crypto"; -import { z } from "zod"; import { WorkflowDatabaseCorruptError } from "../../storage/errors.ts"; +import { + hasUnpairedSurrogate, + parseWorkspaceRootManifest, + SHA256, + validateCanonicalWorkspacePath, + validateWorkspaceRootEntries, + WORKSPACE_ROOT_DOMAIN, + WORKSPACE_ROOT_FORMAT, + type WorkspaceRejection, + type WorkspaceRootEntry, + type WorkspaceRootManifest, +} from "../../workspace/root-manifest.ts"; + +export { + compareUtf8, + hasUnpairedSurrogate, + parentFirst, + parentPath, + WORKSPACE_ROOT_DOMAIN, + WORKSPACE_ROOT_FORMAT, +} from "../../workspace/root-manifest.ts"; +export type { + WorkspaceRejection, + WorkspaceRootEntry, + WorkspaceRootManifest, +} from "../../workspace/root-manifest.ts"; -export const WORKSPACE_ROOT_FORMAT = 1; -export const WORKSPACE_ROOT_DOMAIN = "xmd-workspace-root\0v1\0"; - -const SHA256 = /^[0-9a-f]{64}$/; -const encoder = new TextEncoder(); - -const directoryEntrySchema = z - .object({ - path: z.string(), - kind: z.literal("directory"), - mode: z.number().int().min(0).max(0o7777), - mtime: z.number().int().safe(), - }) - .strict(); - -const fileEntrySchema = z - .object({ - path: z.string(), - kind: z.literal("file"), - mode: z.number().int().min(0).max(0o7777), - mtime: z.number().int().safe(), - size: z.number().int().safe().nonnegative(), - manifest: z.string().regex(SHA256), - hardlink: z - .string() - .regex(/^h[0-9]+$/) - .nullable(), - }) - .strict(); - -const symlinkEntrySchema = z - .object({ - path: z.string(), - kind: z.literal("symlink"), - mode: z.number().int().min(0).max(0o7777), - mtime: z.number().int().safe(), - target: z.string(), - }) - .strict(); - -const rootManifestSchema = z - .object({ - format: z.literal(WORKSPACE_ROOT_FORMAT), - entries: z.array( - z.discriminatedUnion("kind", [directoryEntrySchema, fileEntrySchema, symlinkEntrySchema]), - ), - }) - .strict(); - -export type WorkspaceRootEntry = z.infer["entries"][number]; -export type WorkspaceRootManifest = z.infer; +/** + * How a caller other than a live run reports a Workspace root it cannot accept. + * + * The default names the run database the root was read from, which is what + * every live caller is holding. A sealed XMD artifact is not a run database and + * says so in its own words, so it supplies one of these rather than borrowing a + * sentence that would tell an operator to restore a run from a backup. + */ +function rejecting(databasePath: string): WorkspaceRejection { + return (reason: string) => corrupt(databasePath, reason); +} export interface StoredWorkspaceRoot { readonly rootId: string; @@ -96,40 +81,12 @@ export function encodeWorkspaceManifest( return JSON.stringify(manifest); } -/** - * How a caller other than a live run reports a Workspace root it cannot accept. - * - * The default names the run database the root was read from, which is what - * every live caller is holding. A sealed XMD artifact is not a run database and - * says so in its own words, so it supplies one of these rather than borrowing a - * sentence that would tell an operator to restore a run from a backup. - */ -export type WorkspaceRejection = (reason: string) => never; - -function rejecting(databasePath: string): WorkspaceRejection { - return (reason: string) => corrupt(databasePath, reason); -} - export function parseWorkspaceManifest( manifest: string, databasePath: string, reject: WorkspaceRejection = rejecting(databasePath), ): WorkspaceRootManifest { - let offered: unknown; - try { - offered = JSON.parse(manifest); - } catch { - reject("one of its retained Workspace roots is not JSON"); - } - const parsed = rootManifestSchema.safeParse(offered); - if (!parsed.success) { - reject("one of its retained Workspace roots has an invalid manifest"); - } - validateWorkspaceEntries(parsed.data.entries, databasePath, reject); - if (JSON.stringify(parsed.data) !== manifest) { - reject("one of its retained Workspace roots is not canonically encoded"); - } - return parsed.data; + return parseWorkspaceRootManifest(manifest, reject); } export function validateWorkspaceEntries( @@ -137,60 +94,7 @@ export function validateWorkspaceEntries( databasePath: string, reject: WorkspaceRejection = rejecting(databasePath), ): void { - if (entries.length === 0 || entries[0]?.path !== "/" || entries[0]?.kind !== "directory") { - reject("a Workspace root does not begin with its root directory"); - } - - let previous: string | undefined; - let nextHardlink = 0; - const directories = new Set(); - const hardlinkMembers = new Map(); - const hardlinkFirst = new Map(); - - for (const entry of entries) { - validateCanonicalPath(entry.path, databasePath, reject); - if (previous !== undefined && compareUtf8(previous, entry.path) >= 0) { - reject("a Workspace root's paths are duplicated or out of canonical order"); - } - previous = entry.path; - - if (entry.path !== "/" && !directories.has(parentPath(entry.path))) { - reject("a Workspace root contains an entry without a parent directory"); - } - if (entry.kind === "directory") { - directories.add(entry.path); - } - if ( - entry.kind === "symlink" && - (entry.target.includes("\0") || hasUnpairedSurrogate(entry.target)) - ) { - reject("a Workspace root contains an invalid symbolic-link target"); - } - if (entry.kind === "file" && entry.hardlink !== null) { - const first = hardlinkFirst.get(entry.hardlink); - if (first === undefined) { - if (entry.hardlink !== `h${nextHardlink}`) { - reject("a Workspace root's hardlinks are not canonically numbered"); - } - nextHardlink += 1; - hardlinkFirst.set(entry.hardlink, entry); - } else if ( - first.mode !== entry.mode || - first.mtime !== entry.mtime || - first.size !== entry.size || - first.manifest !== entry.manifest - ) { - reject("a Workspace root's hardlink group has inconsistent metadata"); - } - hardlinkMembers.set(entry.hardlink, (hardlinkMembers.get(entry.hardlink) ?? 0) + 1); - } - } - - for (const count of hardlinkMembers.values()) { - if (count < 2) { - reject("a Workspace root contains a one-member hardlink group"); - } - } + validateWorkspaceRootEntries(entries, reject); } export function validateCanonicalPath( @@ -198,22 +102,7 @@ export function validateCanonicalPath( databasePath: string, reject: WorkspaceRejection = rejecting(databasePath), ): void { - if (value === "/") { - return; - } - if ( - !value.startsWith("/") || - value.endsWith("/") || - value.includes("\0") || - hasUnpairedSurrogate(value) - ) { - reject("a Workspace root contains a noncanonical path"); - } - for (const part of value.slice(1).split("/")) { - if (part === "" || part === "." || part === "..") { - reject("a Workspace root contains a noncanonical path component"); - } - } + validateCanonicalWorkspacePath(value, reject); } export function validatePathName(name: string, databasePath: string): void { @@ -229,20 +118,6 @@ export function validatePathName(name: string, databasePath: string): void { } } -export function compareUtf8(left: string, right: string): number { - return Buffer.compare(encoder.encode(left), encoder.encode(right)); -} - -export function parentFirst(left: WorkspaceRootEntry, right: WorkspaceRootEntry): number { - const depth = left.path.split("/").length - right.path.split("/").length; - return depth === 0 ? compareUtf8(left.path, right.path) : depth; -} - -export function parentPath(path: string): string { - const boundary = path.lastIndexOf("/"); - return boundary === 0 ? "/" : path.slice(0, boundary); -} - export function sha256(value: Uint8Array): Uint8Array { return new Uint8Array(createHash("sha256").update(value).digest()); } @@ -292,19 +167,3 @@ export function mode(value: unknown, databasePath: string): number { export function corrupt(databasePath: string, reason: string): never { throw new WorkflowDatabaseCorruptError(databasePath, reason); } - -function hasUnpairedSurrogate(value: string): boolean { - for (let index = 0; index < value.length; index += 1) { - const code = value.charCodeAt(index); - if (code >= 0xd800 && code <= 0xdbff) { - const next = value.charCodeAt(index + 1); - if (next < 0xdc00 || next > 0xdfff) { - return true; - } - index += 1; - } else if (code >= 0xdc00 && code <= 0xdfff) { - return true; - } - } - return false; -} diff --git a/packages/workflow/src/deno/workspace/root.ts b/packages/workflow/src/deno/workspace/root.ts index 55db2b482..a6d211b9b 100644 --- a/packages/workflow/src/deno/workspace/root.ts +++ b/packages/workflow/src/deno/workspace/root.ts @@ -1,5 +1,4 @@ import type { DatabaseSync } from "node:sqlite"; -import { z } from "zod"; import type { Database as CloudflareDatabase } from "../../../vendor/cloudflare-computer-dofs/generated/storage.js"; import { buildManifest } from "../../../vendor/cloudflare-computer-dofs/generated/sync/manifests.js"; import type { RunConnection, RunTransaction } from "../connections.ts"; @@ -25,33 +24,18 @@ import { workspaceRoot, WORKSPACE_ROOT_FORMAT, } from "./manifest.ts"; - -const decoder = new TextDecoder("utf-8", { fatal: true }); -const SHA256 = /^[0-9a-f]{64}$/; - -const dofsManifestSchema = z - .object({ - version: z.literal(1), - chunks: z.array( - z - .object({ - hash: z.string().regex(SHA256), - size: z.number().int().safe().positive(), - }) - .strict(), - ), - }) - .strict(); +import { + decodeDofsManifest as decodeSharedDofsManifest, + type DofsManifest, + SHA256, +} from "../../workspace/root-manifest.ts"; export interface DofsChunk { readonly hash: Uint8Array; readonly size: number; } -export interface DofsManifest { - readonly size: number; - readonly chunks: readonly { readonly hash: string; readonly size: number }[]; -} +export type { DofsManifest } from "../../workspace/root-manifest.ts"; interface NodeRow { readonly inode: number; @@ -216,9 +200,12 @@ export function snapshotWorkspace( for (const [index, paths] of groups.entries()) { const group = `h${index}`; const members = new Set(paths); - for (const item of entries) { + for (const [position, item] of entries.entries()) { if (item.entry.kind === "file" && members.has(item.entry.path)) { - item.entry.hardlink = group; + // Rebuilt rather than mutated: a manifest entry is what a root is + // hashed over, and a value nobody can edit in place is one nobody can + // edit after it has been counted. + entries[position] = { ...item, entry: { ...item.entry, hardlink: group } }; } } } @@ -468,23 +455,7 @@ export function readDofsManifest( * size the chunks it lists add up to. */ export function decodeDofsManifest(encoded: Uint8Array, reject: WorkspaceRejection): DofsManifest { - let text: string; - let offered: unknown; - try { - text = decoder.decode(encoded); - offered = JSON.parse(text); - } catch { - reject("a DOFS manifest is not canonical UTF-8 JSON"); - } - const parsed = dofsManifestSchema.safeParse(offered); - if (!parsed.success || JSON.stringify(parsed.data) !== text) { - reject("a DOFS manifest is not canonically encoded"); - } - const total = parsed.data.chunks.reduce((sum, chunk) => sum + chunk.size, 0); - if (!Number.isSafeInteger(total)) { - reject("a DOFS manifest names more bytes than a size can hold"); - } - return Object.freeze({ size: total, chunks: Object.freeze(parsed.data.chunks) }); + return decodeSharedDofsManifest(encoded, reject); } function parseStoredRoot( diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index 48879ae97..93aa599b0 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -88,6 +88,7 @@ export interface OwnerConnection { id: string, command: Record, parse: AnswerParser, + parseRefusal?: (refusal: string) => string, ): Operation>; } @@ -295,6 +296,7 @@ export function useOwnerConnection(socket: OwnerSocket): Operation, parse: AnswerParser, + parseRefusal: (refusal: string) => string = (refusal) => refusal, ): Operation> { if (closed) { throw new OwnerLinkError("closed"); @@ -311,8 +313,13 @@ export function useOwnerConnection(socket: OwnerSocket): Operation; + root(workspaceRootId: string): Operation; + content(workspaceRootId: string, request: RemoteContentRequest): Operation; +} + +export function startingFrontier(snapshot: RemoteFrontierSnapshot): StartingFrontier { + return { + workspaceRootId: snapshot.workspaceRootId, + journalEventId: snapshot.journalEventId, + events: snapshot.entries.map((entry) => structuredClone(entry.event)), + }; +} diff --git a/packages/workflow/src/remote/records.ts b/packages/workflow/src/remote/records.ts new file mode 100644 index 000000000..1853f35a3 --- /dev/null +++ b/packages/workflow/src/remote/records.ts @@ -0,0 +1,143 @@ +/** + * Reading a workflow record that arrived over a connection. + * + * The owner is the same build and is trusted to be honest; it is not trusted to + * be correct, and neither is the wire between them. A performed answer is an + * answer the owner labelled performed — that is all it is — so nothing here + * turns an `unknown` into a run record, a retrieval or a journal entry without + * checking every member first. + * + * The parsers the local host holds its own rows to are the parsers used here. + * Two readings of one record is how the two hosts would stop agreeing about + * what a run is, and the second reading is always the more permissive one. + * + * A failure names the member and never the value. What crossed the connection + * is retained history, and a record that does not parse is not a reason to + * repeat what it held. + */ + +import { parseDurableEvent } from "@executablemd/durable-streams"; +import type { JournalEntry } from "../storage/api.ts"; +import { parseWorkflowDefinition } from "../storage/definition.ts"; +import { + parseJsonObject, + parseJsonValue, + parseMembers, + parseStringMember, + requireMemberNames, +} from "../storage/members.ts"; +import { + type DefinitionRetrieval, + parseRunId, + parseWorkflowRunStatus, + parseWorkflowStopReason, + type WorkflowRunRecord, +} from "../storage/record.ts"; +import { SHA256 } from "../workspace/root-manifest.ts"; + +export class RemoteRecordError extends Error { + override name = "RemoteRecordError"; +} + +function fail(reason: string, path: string): Error { + return new RemoteRecordError( + `the owner returned a malformed workflow record at ${path}: ${reason}`, + ); +} + +function instant(value: unknown, path: string): string { + if (typeof value !== "string") { + throw fail("expected an instant", path); + } + const parsed = new Date(value); + if (Number.isNaN(parsed.getTime()) || parsed.toISOString() !== value) { + throw fail("expected an instant", path); + } + return value; +} + +function positiveInteger(value: unknown, path: string): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 1) { + throw fail("expected a positive whole number", path); + } + return value; +} + +function rootId(value: unknown, path: string): string { + if (typeof value !== "string" || !SHA256.test(value)) { + throw fail("expected a Workspace root identity", path); + } + return value; +} + +export function parseRemoteRunRecord(value: unknown): WorkflowRunRecord { + const members = parseMembers(value, "$", fail); + requireMemberNames( + members, + ["runId", "definition", "base", "props", "status", "stopReason", "createdAt", "updatedAt"], + "$", + fail, + ); + const definition = parseWorkflowDefinition(members.get("definition")); + if (!definition.ok) { + throw fail("expected a workflow definition", "$.definition"); + } + const base = parseStringMember(members, "base", "$", fail); + if (base === "") { + throw fail("expected a non-empty string", "$.base"); + } + const record: WorkflowRunRecord = { + runId: parseRunId(members.get("runId"), "$.runId", fail), + definition: definition.value, + base, + props: parseJsonObject(members.get("props"), "$.props", fail), + status: parseWorkflowRunStatus(members.get("status"), "$.status", fail), + createdAt: instant(members.get("createdAt"), "$.createdAt"), + updatedAt: instant(members.get("updatedAt"), "$.updatedAt"), + }; + if (!members.has("stopReason")) { + return Object.freeze(record); + } + return Object.freeze({ + ...record, + stopReason: parseWorkflowStopReason(members.get("stopReason"), "$.stopReason", fail), + }); +} + +export function parseRemoteRetrieval(value: unknown): DefinitionRetrieval | undefined { + if (value === null) { + return undefined; + } + const members = parseMembers(value, "$", fail); + requireMemberNames(members, ["metadata", "revision", "updatedAt"], "$", fail); + if (members.size !== 3) { + throw fail("expected every retrieval member", "$ "); + } + return Object.freeze({ + metadata: parseJsonValue(members.get("metadata"), "$.metadata", fail), + revision: positiveInteger(members.get("revision"), "$.revision"), + updatedAt: instant(members.get("updatedAt"), "$.updatedAt"), + }); +} + +export function parseRemoteJournalEntry(value: unknown): JournalEntry { + const members = parseMembers(value, "$", fail); + requireMemberNames(members, ["eventId", "record", "workspaceRootId"], "$", fail); + if (members.size !== 3) { + throw fail("expected every journal member", "$ "); + } + const eventId = parseStringMember(members, "eventId", "$", fail); + if (eventId === "") { + throw fail("expected a non-empty identity", "$.eventId"); + } + const record = parseStringMember(members, "record", "$", fail); + const event = parseDurableEvent(record); + if (!event.ok) { + throw fail("expected a durable event", "$.record"); + } + return Object.freeze({ + eventId, + event: event.value, + workspaceRootId: rootId(members.get("workspaceRootId"), "$.workspaceRootId"), + }); +} diff --git a/packages/workflow/src/deno/rows.ts b/packages/workflow/src/sqlite/rows.ts similarity index 96% rename from packages/workflow/src/deno/rows.ts rename to packages/workflow/src/sqlite/rows.ts index 30951826c..b09889e03 100644 --- a/packages/workflow/src/deno/rows.ts +++ b/packages/workflow/src/sqlite/rows.ts @@ -10,6 +10,12 @@ * A failure names the column and never the value. Props and journal payloads * are retained history, and a row that does not parse is not a reason to print * what it held. + * + * It lives beside the schema rather than under a host because two adapters read + * the same rows back. The Deno host opens a file with `node:sqlite`; the + * Cloudflare owner reads the storage of one Durable Object. What a stored row + * *means* is the same question in both, and a second copy of these parsers + * would be the place the two hosts quietly stopped agreeing. */ import type { Json } from "@executablemd/durable-streams"; diff --git a/packages/workflow/src/workspace/root-manifest.ts b/packages/workflow/src/workspace/root-manifest.ts new file mode 100644 index 000000000..a532ddc31 --- /dev/null +++ b/packages/workflow/src/workspace/root-manifest.ts @@ -0,0 +1,425 @@ +/** + * What a retained Workspace root *is*, independent of who stored it. + * + * A root is a canonical JSON manifest and the content-addressed objects its + * entries name. Its identity is the SHA-256 of a domain-separated encoding of + * that manifest, so two hosts holding the same bytes hold the same root and + * neither has to be asked. + * + * Two adapters retain roots — the Deno host in a SQLite file, the Cloudflare + * owner in the storage of one Durable Object — and a second copy of these rules + * under a second adapter would be the place they stopped agreeing. Whichever + * one is looser decides what the other must accept, and the looser one is + * always the newer one. So the rules live here once. + * + * Nothing here opens a database, hashes anything, or names a runtime. Hashing + * is deliberately absent: each host has its own primitive for it, and this + * module has no business choosing between them. What it decides is whether a + * sequence of bytes is a canonically encoded manifest at all, and what that + * manifest says. + * + * A caller supplies `reject`, because the same disagreement is reported very + * differently depending on who found it: a path names the file the Deno host + * refused, a Durable Object has no path to name, and a sealed artifact is not a + * run database at all. + */ + +/** The only root format this build reads or writes. */ +export const WORKSPACE_ROOT_FORMAT = 1; + +/** + * What a root identity is taken over, before the manifest itself. + * + * Domain separation, so a digest of a Workspace root can never collide with a + * digest of anything else this system hashes. + */ +export const WORKSPACE_ROOT_DOMAIN = "xmd-workspace-root\0v1\0"; + +/** A lowercase SHA-256 identity, which is the only spelling any of this uses. */ +export const SHA256 = /^[0-9a-f]{64}$/; + +/** How a reader says these bytes are not a root it can accept. */ +export type WorkspaceRejection = (reason: string) => never; + +/** One directory in a root. */ +export interface WorkspaceDirectoryEntry { + readonly path: string; + readonly kind: "directory"; + readonly mode: number; + readonly mtime: number; +} + +/** One file in a root, named by the DOFS manifest holding its bytes. */ +export interface WorkspaceFileEntry { + readonly path: string; + readonly kind: "file"; + readonly mode: number; + readonly mtime: number; + readonly size: number; + readonly manifest: string; + readonly hardlink: string | null; +} + +/** One symbolic link in a root. */ +export interface WorkspaceSymlinkEntry { + readonly path: string; + readonly kind: "symlink"; + readonly mode: number; + readonly mtime: number; + readonly target: string; +} + +export type WorkspaceRootEntry = + | WorkspaceDirectoryEntry + | WorkspaceFileEntry + | WorkspaceSymlinkEntry; + +export interface WorkspaceRootManifest { + readonly format: typeof WORKSPACE_ROOT_FORMAT; + readonly entries: readonly WorkspaceRootEntry[]; +} + +/** One DOFS manifest: the ordered chunks one file's bytes are stored as. */ +export interface DofsChunkReference { + readonly hash: string; + readonly size: number; +} + +export interface DofsManifest { + readonly size: number; + readonly chunks: readonly DofsChunkReference[]; +} + +const encoder = new TextEncoder(); +const decoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: false }); + +/** + * Compare two paths by their UTF-8 bytes. + * + * Byte order rather than `String` order, because a root's canonical ordering is + * a property of its encoding: two hosts that sorted differently would disagree + * about whether the same entries are the same root. + */ +export function compareUtf8(left: string, right: string): number { + const a = encoder.encode(left); + const b = encoder.encode(right); + const shared = Math.min(a.length, b.length); + for (let index = 0; index < shared; index += 1) { + const first = a[index] ?? 0; + const second = b[index] ?? 0; + if (first !== second) { + return first < second ? -1 : 1; + } + } + return a.length === b.length ? 0 : a.length < b.length ? -1 : 1; +} + +/** The directory one canonical path sits in. */ +export function parentPath(path: string): string { + const boundary = path.lastIndexOf("/"); + return boundary === 0 ? "/" : path.slice(0, boundary); +} + +/** Depth first, then byte order — the order a restore creates entries in. */ +export function parentFirst(left: WorkspaceRootEntry, right: WorkspaceRootEntry): number { + const depth = left.path.split("/").length - right.path.split("/").length; + return depth === 0 ? compareUtf8(left.path, right.path) : depth; +} + +/** + * Whether text contains a code unit that is not part of a valid pair. + * + * An unpaired surrogate survives a round trip through JSON and does not survive + * one through UTF-8, so a manifest carrying one is a manifest whose bytes + * cannot be reproduced. + */ +export function hasUnpairedSurrogate(value: string): boolean { + for (let index = 0; index < value.length; index += 1) { + const code = value.charCodeAt(index); + if (code >= 0xd800 && code <= 0xdbff) { + const next = value.charCodeAt(index + 1); + if (next < 0xdc00 || next > 0xdfff) { + return true; + } + index += 1; + } else if (code >= 0xdc00 && code <= 0xdfff) { + return true; + } + } + return false; +} + +function isSafeInteger(value: unknown): value is number { + return typeof value === "number" && Number.isSafeInteger(value); +} + +function members(value: unknown): Map | undefined { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + return undefined; + } + return new Map(Object.entries(value)); +} + +/** + * Whether an object declares exactly these members and no others. + * + * A manifest is compared with its own re-encoding further down, so an extra + * member would already be caught. It is refused here as well because the reason + * matters: an unknown member is a manifest this build does not understand, + * which is a different thing from bytes that were laid out differently. + */ +function declares(found: Map, expected: readonly string[]): boolean { + if (found.size !== expected.length) { + return false; + } + return expected.every((name) => found.has(name)); +} + +function mode(value: unknown): boolean { + return isSafeInteger(value) && value >= 0 && value <= 0o7777; +} + +/** + * Read one entry, in the exact member order this build writes. + * + * The order matters and is not a style choice: a manifest is compared with its + * own re-encoding, and a re-encoding that named the same members in a different + * order would be refused as noncanonical. So each branch builds its object + * literally rather than by spreading a shared prefix. + */ +function entryOf(value: unknown): WorkspaceRootEntry | undefined { + const found = members(value); + if (found === undefined) { + return undefined; + } + const path = found.get("path"); + const kind = found.get("kind"); + const entryMode = found.get("mode"); + const mtime = found.get("mtime"); + if ( + typeof path !== "string" || + !mode(entryMode) || + !isSafeInteger(entryMode) || + !isSafeInteger(mtime) + ) { + return undefined; + } + + if (kind === "directory") { + return declares(found, ["path", "kind", "mode", "mtime"]) + ? { path, kind, mode: entryMode, mtime } + : undefined; + } + if (kind === "symlink") { + const target = found.get("target"); + if (typeof target !== "string") { + return undefined; + } + return declares(found, ["path", "kind", "mode", "mtime", "target"]) + ? { path, kind, mode: entryMode, mtime, target } + : undefined; + } + if (kind !== "file") { + return undefined; + } + const size = found.get("size"); + const manifest = found.get("manifest"); + const hardlink = found.get("hardlink"); + if (!isSafeInteger(size) || size < 0 || typeof manifest !== "string" || !SHA256.test(manifest)) { + return undefined; + } + if (hardlink !== null && (typeof hardlink !== "string" || !/^h[0-9]+$/.test(hardlink))) { + return undefined; + } + return declares(found, ["path", "kind", "mode", "mtime", "size", "manifest", "hardlink"]) + ? { path, kind, mode: entryMode, mtime, size, manifest, hardlink } + : undefined; +} + +/** + * Read one root manifest out of the exact text a store retained. + * + * Three separate questions, in order: is it JSON, is it a manifest this build + * declares, and are these the exact bytes this build would have written for + * that manifest. The last one is what makes the identity meaningful — a root ID + * is a digest of these bytes, so a manifest that means the same thing and is + * spelled differently is a different root and must not be admitted as this one. + */ +export function parseWorkspaceRootManifest( + manifest: string, + reject: WorkspaceRejection, +): WorkspaceRootManifest { + let offered: unknown; + try { + offered = JSON.parse(manifest); + } catch { + reject("one of its retained Workspace roots is not JSON"); + } + const found = members(offered); + const declared = found !== undefined && declares(found, ["format", "entries"]); + const entries = found?.get("entries"); + if (!declared || found?.get("format") !== WORKSPACE_ROOT_FORMAT || !Array.isArray(entries)) { + reject("one of its retained Workspace roots has an invalid manifest"); + } + const parsed: WorkspaceRootEntry[] = []; + for (const entry of entries) { + const admitted = entryOf(entry); + if (admitted === undefined) { + reject("one of its retained Workspace roots has an invalid manifest"); + } + parsed.push(admitted); + } + const root: WorkspaceRootManifest = { format: WORKSPACE_ROOT_FORMAT, entries: parsed }; + validateWorkspaceRootEntries(parsed, reject); + if (JSON.stringify(root) !== manifest) { + reject("one of its retained Workspace roots is not canonically encoded"); + } + return root; +} + +/** + * Whether these entries describe a Workspace at all. + * + * Shape is not enough. A root is a tree, and its manifest is a flat list, so + * the tree lives in these rules: the list starts at the root directory, every + * path is canonical, order is total and by bytes, every entry has a parent that + * was already declared, and a hardlink group is numbered in the order it first + * appears and agrees with itself. + */ +export function validateWorkspaceRootEntries( + entries: readonly WorkspaceRootEntry[], + reject: WorkspaceRejection, +): void { + if (entries.length === 0 || entries[0]?.path !== "/" || entries[0]?.kind !== "directory") { + reject("a Workspace root does not begin with its root directory"); + } + + let previous: string | undefined; + let nextHardlink = 0; + const directories = new Set(); + const hardlinkMembers = new Map(); + const hardlinkFirst = new Map(); + + for (const entry of entries) { + validateCanonicalWorkspacePath(entry.path, reject); + if (previous !== undefined && compareUtf8(previous, entry.path) >= 0) { + reject("a Workspace root's paths are duplicated or out of canonical order"); + } + previous = entry.path; + + if (entry.path !== "/" && !directories.has(parentPath(entry.path))) { + reject("a Workspace root contains an entry without a parent directory"); + } + if (entry.kind === "directory") { + directories.add(entry.path); + } + if ( + entry.kind === "symlink" && + (entry.target.includes("\0") || hasUnpairedSurrogate(entry.target)) + ) { + reject("a Workspace root contains an invalid symbolic-link target"); + } + if (entry.kind === "file" && entry.hardlink !== null) { + const first = hardlinkFirst.get(entry.hardlink); + if (first === undefined) { + if (entry.hardlink !== `h${nextHardlink}`) { + reject("a Workspace root's hardlinks are not canonically numbered"); + } + nextHardlink += 1; + hardlinkFirst.set(entry.hardlink, entry); + } else if ( + first.mode !== entry.mode || + first.mtime !== entry.mtime || + first.size !== entry.size || + first.manifest !== entry.manifest + ) { + reject("a Workspace root's hardlink group has inconsistent metadata"); + } + hardlinkMembers.set(entry.hardlink, (hardlinkMembers.get(entry.hardlink) ?? 0) + 1); + } + } + + for (const count of hardlinkMembers.values()) { + if (count < 2) { + reject("a Workspace root contains a one-member hardlink group"); + } + } +} + +/** One absolute path with no traversal, no empty component and no surprises. */ +export function validateCanonicalWorkspacePath(value: string, reject: WorkspaceRejection): void { + if (value === "/") { + return; + } + if ( + !value.startsWith("/") || + value.endsWith("/") || + value.includes("\0") || + hasUnpairedSurrogate(value) + ) { + reject("a Workspace root contains a noncanonical path"); + } + for (const part of value.slice(1).split("/")) { + if (part === "" || part === "." || part === "..") { + reject("a Workspace root contains a noncanonical path component"); + } + } +} + +/** + * The DOFS manifest one encoding describes, without a store to look anything up + * in. + * + * The same bytes are validated in more than one place — by a live run reading + * its own content store, by a reader checking a detached copy, and by an owner + * about to send a copy to a runner. What is decided here is only whether these + * bytes are a canonically encoded DOFS manifest at all, and what size the + * chunks it names add up to. That the chunks exist is whoever called's to prove. + */ +export function decodeDofsManifest(encoded: Uint8Array, reject: WorkspaceRejection): DofsManifest { + let text: string; + let offered: unknown; + try { + text = decoder.decode(encoded); + offered = JSON.parse(text); + } catch { + reject("a DOFS manifest is not canonical UTF-8 JSON"); + } + const found = members(offered); + const chunks = found?.get("chunks"); + if ( + found === undefined || + !declares(found, ["version", "chunks"]) || + found.get("version") !== 1 || + !Array.isArray(chunks) + ) { + reject("a DOFS manifest is not canonically encoded"); + } + const references: DofsChunkReference[] = []; + for (const chunk of chunks) { + const entry = members(chunk); + const hash = entry?.get("hash"); + const size = entry?.get("size"); + if ( + entry === undefined || + !declares(entry, ["hash", "size"]) || + typeof hash !== "string" || + !SHA256.test(hash) || + !isSafeInteger(size) || + size < 1 + ) { + // A zero-length chunk names no bytes, so a manifest that lists one is + // describing content it does not have. + reject("a DOFS manifest is not canonically encoded"); + } + references.push({ hash, size }); + } + if (JSON.stringify({ version: 1, chunks: references }) !== text) { + reject("a DOFS manifest is not canonically encoded"); + } + const total = references.reduce((sum, chunk) => sum + chunk.size, 0); + if (!Number.isSafeInteger(total)) { + reject("a DOFS manifest names more bytes than a size can hold"); + } + return Object.freeze({ size: total, chunks: Object.freeze(references) }); +} diff --git a/packages/workflow/src/workspace/sha256.ts b/packages/workflow/src/workspace/sha256.ts new file mode 100644 index 000000000..957601942 --- /dev/null +++ b/packages/workflow/src/workspace/sha256.ts @@ -0,0 +1,119 @@ +/** + * SHA-256, in the language itself. + * + * Every host this package runs on has a SHA-256 already, and none of them has + * one this code can use. `node:crypto` is a host specifier, and the whole point + * of a shared module is that it names no host. `crypto.subtle.digest()` is + * asynchronous, and the place this is needed most is inside a Durable Object's + * synchronous transaction, where there is nothing to await into. + * + * So the arithmetic lives here. A content identity is what decides whether two + * hosts are holding the same Workspace root, and a digest that differed between + * them would be two systems quietly disagreeing about history. FIPS 180-4 is + * fixed, small, and has published answers, which is why this is a reasonable + * thing to carry: the tests hold it to those answers and to the identity the + * Deno host computes with its own primitive. + * + * It hashes bytes already in memory. It is not a streaming interface and is not + * for anything large; the private protocol bounds every piece it is used on. + */ + +const INITIAL = new Uint32Array([ + 0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19, +]); + +const ROUND = new Uint32Array([ + 0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5, + 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174, + 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da, + 0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, + 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85, + 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070, + 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3, + 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2, +]); + +function rotate(value: number, bits: number): number { + return (value >>> bits) | (value << (32 - bits)); +} + +function padded(input: Uint8Array): Uint8Array { + const length = Math.ceil((input.length + 9) / 64) * 64; + const bytes = new Uint8Array(length); + bytes.set(input); + bytes[input.length] = 0x80; + const bits = BigInt(input.length) * 8n; + for (let index = 0; index < 8; index += 1) { + bytes[length - 1 - index] = Number((bits >> BigInt(index * 8)) & 0xffn); + } + return bytes; +} + +export function sha256(value: Uint8Array | string): Uint8Array { + const input = typeof value === "string" ? new TextEncoder().encode(value) : value; + const bytes = padded(input); + const state = new Uint32Array(INITIAL); + const words = new Uint32Array(64); + for (let offset = 0; offset < bytes.length; offset += 64) { + for (let index = 0; index < 16; index += 1) { + const at = offset + index * 4; + words[index] = + ((bytes[at] ?? 0) << 24) | + ((bytes[at + 1] ?? 0) << 16) | + ((bytes[at + 2] ?? 0) << 8) | + (bytes[at + 3] ?? 0); + } + for (let index = 16; index < 64; index += 1) { + const x = words[index - 15] ?? 0; + const y = words[index - 2] ?? 0; + const sigma0 = rotate(x, 7) ^ rotate(x, 18) ^ (x >>> 3); + const sigma1 = rotate(y, 17) ^ rotate(y, 19) ^ (y >>> 10); + words[index] = ((words[index - 16] ?? 0) + sigma0 + (words[index - 7] ?? 0) + sigma1) >>> 0; + } + let a = state[0] ?? 0; + let b = state[1] ?? 0; + let c = state[2] ?? 0; + let d = state[3] ?? 0; + let e = state[4] ?? 0; + let f = state[5] ?? 0; + let g = state[6] ?? 0; + let h = state[7] ?? 0; + for (let index = 0; index < 64; index += 1) { + const sum1 = rotate(e, 6) ^ rotate(e, 11) ^ rotate(e, 25); + const choice = (e & f) ^ (~e & g); + const first = (h + sum1 + choice + (ROUND[index] ?? 0) + (words[index] ?? 0)) >>> 0; + const sum0 = rotate(a, 2) ^ rotate(a, 13) ^ rotate(a, 22); + const majority = (a & b) ^ (a & c) ^ (b & c); + const second = (sum0 + majority) >>> 0; + h = g; + g = f; + f = e; + e = (d + first) >>> 0; + d = c; + c = b; + b = a; + a = (first + second) >>> 0; + } + state[0] = ((state[0] ?? 0) + a) >>> 0; + state[1] = ((state[1] ?? 0) + b) >>> 0; + state[2] = ((state[2] ?? 0) + c) >>> 0; + state[3] = ((state[3] ?? 0) + d) >>> 0; + state[4] = ((state[4] ?? 0) + e) >>> 0; + state[5] = ((state[5] ?? 0) + f) >>> 0; + state[6] = ((state[6] ?? 0) + g) >>> 0; + state[7] = ((state[7] ?? 0) + h) >>> 0; + } + const digest = new Uint8Array(32); + for (let index = 0; index < state.length; index += 1) { + const word = state[index] ?? 0; + digest[index * 4] = word >>> 24; + digest[index * 4 + 1] = word >>> 16; + digest[index * 4 + 2] = word >>> 8; + digest[index * 4 + 3] = word; + } + return digest; +} + +export function sha256Hex(value: Uint8Array | string): string { + return Array.from(sha256(value), (byte) => byte.toString(16).padStart(2, "0")).join(""); +} diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts index 3db842664..317e1dad6 100644 --- a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -268,10 +268,12 @@ describe("holding an acquisition", () => { it("lets the admitted connection send, and answers what it performed", async () => { const stub = executor(); + await on(stub, (o) => o.initialize()); await admitted(stub); - expect( - await on(stub, (o) => o.send(1, JSON.stringify({ id: "1", command: "frontier" }))), - ).toEqual({ id: "1", outcome: "performed", value: { performed: "frontier" } }); + const answer = await on(stub, (o) => + o.send(1, JSON.stringify({ id: "1", command: "frontier" })), + ); + expect(answer).toMatchObject({ id: "1", outcome: "performed" }); }); it("refuses a socket it never admitted", async () => { @@ -282,6 +284,16 @@ describe("holding an acquisition", () => { ).toEqual({ id: "", outcome: "refused", refusal: "acquisition:foreign-connection" }); }); + it("does not treat copied attachment bytes as an acquisition", async () => { + const stub = executor(); + await admitted(stub); + expect( + await on(stub, (o) => + o.sendWithCopiedAttachment(JSON.stringify({ id: "1", command: "frontier" })), + ), + ).toEqual({ id: "", outcome: "refused", refusal: "acquisition:foreign-connection" }); + }); + it("owns nothing once the connection ends, and rolls nothing back", async () => { const stub = executor(); await admitted(stub); @@ -318,28 +330,34 @@ describe("reading a runner command", () => { expect((await refuse(JSON.stringify({ command: "frontier" }))).refusal).toBe( "command:malformed-member", ); - expect((await refuse(JSON.stringify({ id: "1", command: "materialize" }))).refusal).toBe( + expect((await refuse(JSON.stringify({ id: "1", command: "root" }))).refusal).toBe( "command:malformed-member", ); }); - it("reads a commit intent whole", async () => { + it("reads a commit intent whole, then refuses to act on it in this release", async () => { const stub = executor(); + await on(stub, (o) => o.initialize()); await admitted(stub); const raw = JSON.stringify({ id: "7", command: "commit", expectedWorkspaceRootId: `a${"0".repeat(63)}`, expectedJournalEventId: null, - content: [{ digest: "d1", bytes: "AAAA" }], proposedWorkspaceRootId: `b${"1".repeat(63)}`, events: ["event-1"], }); + // The shape is read — an unknown member or a malformed root would refuse + // differently — and then declined, because applying one is a later + // checkpoint's work and a performed placeholder would be a lie. expect(await on(stub, (o) => o.send(1, raw))).toEqual({ id: "7", - outcome: "performed", - value: { performed: "commit" }, + outcome: "refused", + refusal: "command:unavailable", }); + expect( + await on(stub, (o) => o.send(1, JSON.stringify({ ...JSON.parse(raw), id: "8", extra: 1 }))), + ).toEqual({ id: "", outcome: "refused", refusal: "command:unknown-member" }); }); }); diff --git a/packages/workflow/tests/cloudflare/owner-storage.vitest.ts b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts index a075305ed..82544a870 100644 --- a/packages/workflow/tests/cloudflare/owner-storage.vitest.ts +++ b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts @@ -82,6 +82,13 @@ describe("recognizing an owner object", () => { await on(stub, (o) => o.damage("workflow_suspension_answers")); expect(await on(stub, (o) => o.recognize())).toBe("refused:corrupt"); }); + + it("refuses a missing Cloudflare-private protocol table", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.damage("_xmd_executor_commands")); + expect(await on(stub, (o) => o.recognize())).toBe("refused:corrupt"); + }); }); describe("an owner commit", () => { diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts new file mode 100644 index 000000000..edc7076ac --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -0,0 +1,540 @@ +/** + * The owner's half of the private protocol, on real workerd. + * + * Almost nothing here would be worth proving against a model. Hibernation is a + * property of the runtime: the object is evicted, its fields are gone, and what + * comes back is whatever the storage and the live sockets say. Acquisition + * replacement is a property of the runtime's socket list. Transaction + * atomicity, `WITHOUT ROWID` constraints and blob round-trips are properties of + * the Durable Object's SQLite. A map standing in for any of those would prove + * that the map behaves, which is not the claim. + * + * So these run against a real namespace, real storage, real Hibernation + * WebSockets and a real `evictDurableObject()`, and the assertions are about + * what survived, what was refused, and what was left untouched. + */ + +import { env, evictDurableObject, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { MAX_COMMANDS, MAX_CONTENT_BYTES } from "../../src/cloudflare/commands.ts"; +import { encodeBase64 } from "../../src/cloudflare/encoding.ts"; +import { sha256Hex } from "../../src/workspace/sha256.ts"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { + BLOB_ID, + DOFS_MANIFEST, + FILE_BYTES, + MANIFEST_ID, + POLICY, + ROOT_ID, + ROOT_MANIFEST, + RUN_ID, + VALID_CLAIMS, +} from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`remote-${unique}-${Math.random()}`)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise { + return runInDurableObject(stub, body); +} + +async function admission(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + return await on(stub, (owner) => owner.admitConnection({ token, release: POLICY.release })); +} + +async function admit(stub: ReturnType): Promise { + expect(await admission(stub)).toBe("admitted"); +} + +async function connect(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + const response = await stub.fetch("https://owner.invalid/executor", { + headers: { + authorization: `Bearer ${token}`, + upgrade: "websocket", + "x-release": POLICY.release, + "x-run-id": RUN_ID, + }, + }); + const socket = response.webSocket; + if (socket === null) { + throw new Error(`expected an executor WebSocket, received ${response.status}`); + } + socket.accept(); + return socket; +} + +function ask( + socket: WebSocket, + id: string, + command: Record, +): Promise> { + return askFrame(socket, JSON.stringify({ id, ...command })); +} + +function askFrame( + socket: WebSocket, + message: string | ArrayBuffer, +): Promise> { + return new Promise((resolve, reject) => { + const receive = (event: MessageEvent) => { + socket.removeEventListener("message", receive); + if (typeof event.data !== "string") { + reject(new Error("expected a text answer")); + return; + } + resolve(record(JSON.parse(event.data))); + }; + socket.addEventListener("message", receive); + socket.send(message); + }); +} + +function record(value: unknown): Record { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new Error("expected an object answer"); + } + return Object.fromEntries(Object.entries(value)); +} + +function send( + stub: ReturnType, + id: string, + command: Record, +): Promise> { + return on(stub, (owner) => record(owner.send(1, JSON.stringify({ id, ...command })))); +} + +describe("the remote owner protocol", () => { + it("answers a binary frame once and closes the protocol", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + expect(await askFrame(socket, new Uint8Array([1]).buffer)).toEqual({ + id: "", + outcome: "refused", + refusal: "command:malformed-member", + }); + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(socket.readyState).not.toBe(WebSocket.OPEN); + }); + + it("refuses pristine, foreign, unsupported, damaged, missing, and wrong-run storage", async () => { + const cases: readonly [string, (owner: ExecutorObject) => void, string][] = [ + ["pristine", () => undefined, "storage:foreign"], + ["foreign", (owner) => owner.makeForeign(), "storage:foreign"], + [ + "unsupported", + (owner) => { + owner.initialize(); + owner.rewriteMarker(0x584d4431, 2); + }, + "storage:unsupported-version", + ], + [ + "damaged", + (owner) => { + owner.initialize(); + owner.dropTable("workflow_suspension_answers"); + }, + "storage:corrupt", + ], + [ + "missing", + (owner) => { + owner.initialize(); + owner.removeWorkspaceState(); + }, + "storage:corrupt", + ], + [ + "wrong-run", + (owner) => { + owner.initialize(); + owner.rewriteRunId("somebody-else"); + }, + "storage:corrupt", + ], + ]; + for (const [name, arrange, refusal] of cases) { + const stub = executor(); + await on(stub, arrange); + const admitted = await admission(stub); + const answer = + admitted === "admitted" ? await send(stub, name, { command: "frontier" }) : admitted; + expect([name, answer]).toEqual([ + name, + typeof answer === "string" ? refusal : { id: name, outcome: "refused", refusal }, + ]); + } + }); + + it("names a refusal category and repeats nothing it was given or holds", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.rewriteRunId("a-retained-secret-run")); + await admit(stub); + const damaged = await send(stub, "id-carrying-a-secret", { command: "frontier" }); + // The refusal names a category. It does not repeat the retained run + // identity it disagreed with, and it does not repeat the request beyond the + // correlation the runner needs to match its own question. + expect(damaged).toEqual({ + id: "id-carrying-a-secret", + outcome: "refused", + refusal: "storage:corrupt", + }); + const printed = JSON.stringify(damaged); + for (const retained of ["a-retained-secret-run", RUN_ID, ROOT_MANIFEST, "workflow_run"]) { + expect(printed).not.toContain(retained); + } + + const rejected = await send(stub, "unknown", { + command: "root", + workspaceRootId: "f".repeat(64), + somethingElse: "a value the request supplied", + }); + // Not even the correlation survives a request that never parsed: an id is + // echoed once the command has been read, and this one never was. + expect(rejected).toEqual({ id: "", outcome: "refused", refusal: "command:unknown-member" }); + expect(JSON.stringify(rejected)).not.toContain("a value the request supplied"); + }); + + it("anchors and reconstructs a journal larger than one page", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + for (let index = 0; index < 129; index += 1) { + await on(stub, (owner) => owner.appendJournal(`event-${index}`, `event ${index}`)); + } + await admit(stub); + const frontier = record((await send(stub, "frontier", { command: "frontier" }))["value"]); + expect(frontier["workspaceRootId"]).toBe(ROOT_ID); + expect(frontier["journalEventId"]).toBe("event-128"); + expect(record(frontier["record"])["runId"]).toBe(RUN_ID); + + await on(stub, (owner) => owner.appendJournal("event-later", "later")); + const first = record( + ( + await send(stub, "journal-1", { + command: "journal", + anchorEventId: "event-128", + afterEventId: null, + }) + )["value"], + ); + expect(Array.isArray(first["entries"]) && first["entries"]).toHaveLength(128); + expect(first["done"]).toBe(false); + const second = record( + ( + await send(stub, "journal-2", { + command: "journal", + anchorEventId: "event-128", + afterEventId: "event-127", + }) + )["value"], + ); + expect(second["entries"]).toEqual([ + expect.objectContaining({ eventId: "event-128", previousEventId: "event-127" }), + ]); + expect(second["done"]).toBe(true); + }); + + it("returns only content referenced by one validated root", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + expect(await send(stub, "root", { command: "root", workspaceRootId: ROOT_ID })).toEqual({ + id: "root", + outcome: "performed", + value: { workspaceRootId: ROOT_ID, manifest: ROOT_MANIFEST }, + }); + expect( + await send(stub, "manifest", { + command: "content", + workspaceRootId: ROOT_ID, + kind: "manifest", + digest: MANIFEST_ID, + sourceManifest: null, + }), + ).toEqual({ + id: "manifest", + outcome: "performed", + value: { + kind: "manifest", + digest: MANIFEST_ID, + size: new TextEncoder().encode(DOFS_MANIFEST).length, + bytes: encodeBase64(new TextEncoder().encode(DOFS_MANIFEST)), + }, + }); + expect( + await send(stub, "blob", { + command: "content", + workspaceRootId: ROOT_ID, + kind: "blob", + digest: BLOB_ID, + sourceManifest: MANIFEST_ID, + }), + ).toMatchObject({ outcome: "performed", value: { digest: BLOB_ID, size: FILE_BYTES.length } }); + + const orphan = await on(stub, (owner) => + owner.addUnreferencedBlob(new TextEncoder().encode("orphan")), + ); + expect( + await send(stub, "orphan", { + command: "content", + workspaceRootId: ROOT_ID, + kind: "blob", + digest: orphan, + sourceManifest: MANIFEST_ID, + }), + ).toEqual({ id: "orphan", outcome: "refused", refusal: "storage:corrupt" }); + }); + + it("refuses retained bytes whose identity or recorded size is damaged", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.damageRetainedBlob()); + await admit(stub); + expect( + await send(stub, "blob", { + command: "content", + workspaceRootId: ROOT_ID, + kind: "blob", + digest: BLOB_ID, + sourceManifest: MANIFEST_ID, + }), + ).toEqual({ + id: "blob", + outcome: "refused", + refusal: "storage:corrupt", + }); + }); + + it("replays compatible commands and refuses conflicting reuse", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendJournal("before", "before")); + await admit(stub); + const first = await send(stub, "same", { command: "frontier" }); + await on(stub, (owner) => owner.appendJournal("after", "after")); + expect( + await on(stub, (owner) => + record(owner.send(1, JSON.stringify({ command: "frontier", id: "same" }))), + ), + ).toEqual(first); + expect(await send(stub, "same", { command: "root", workspaceRootId: ROOT_ID })).toEqual({ + id: "same", + outcome: "refused", + refusal: "command:duplicate-conflict", + }); + }); + + it("keeps staged bytes private, durable across eviction, and scoped to one acquisition", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + const bytes = new TextEncoder().encode("proposed content"); + const digest = sha256Hex(bytes); + const command = { command: "stage", kind: "blob", digest, bytes: encodeBase64(bytes) }; + const first = await ask(socket, "stage", command); + expect(first).toMatchObject({ outcome: "performed", value: { digest, size: bytes.length } }); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 1, staged: 1 }); + + await evictDurableObject(stub); + expect(await ask(socket, "stage", command)).toEqual(first); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 1, staged: 1 }); + expect( + await ask(socket, "read-stage", { + command: "content", + workspaceRootId: ROOT_ID, + kind: "blob", + digest, + sourceManifest: MANIFEST_ID, + }), + ).toEqual({ id: "read-stage", outcome: "refused", refusal: "storage:corrupt" }); + await new Promise((resolve) => setTimeout(resolve, 0)); + expect( + await on(stub, (owner) => + owner.admitConnection({ token: "not-a-token", release: POLICY.release }), + ), + ).toBe("token:token-malformed"); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 1, staged: 1 }); + const before = await on(stub, (owner) => owner.authoritative()); + const replaced = await on(stub, (owner) => owner.acquisitionId()); + await admit(stub); + // A second acquisition, and the first one's scratch is gone rather than + // inherited: it cannot be retried, adopted or read. + expect(await on(stub, (owner) => owner.acquisitionId())).not.toBe(replaced); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 0, staged: 0 }); + expect(await on(stub, (owner) => owner.authoritative())).toBe(before); + // The predecessor's own command id is free again, and staging the same + // bytes writes a new row rather than finding the abandoned one. Nothing was + // inherited; it was discarded and done afresh. + expect( + await send(stub, "stage", { + command: "stage", + kind: "blob", + digest, + bytes: encodeBase64(bytes), + }), + ).toMatchObject({ outcome: "performed", value: { digest } }); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 1, staged: 1 }); + expect(await send(stub, "frontier-new", { command: "frontier" })).toMatchObject({ + outcome: "performed", + value: { workspaceRootId: ROOT_ID }, + }); + expect(await on(stub, (owner) => owner.authoritative())).toBe(before); + }); + + it("grants a copied attachment or a foreign socket no read at all", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + const request = JSON.stringify({ id: "borrowed", command: "frontier" }); + expect(await on(stub, (owner) => record(owner.sendWithCopiedAttachment(request)))).toEqual({ + id: "", + outcome: "refused", + refusal: "acquisition:foreign-connection", + }); + expect(await on(stub, (owner) => record(owner.sendAsStranger(request)))).toEqual({ + id: "", + outcome: "refused", + refusal: "acquisition:foreign-connection", + }); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 0, staged: 0 }); + }); + + it("leaves no staged row when decoding or digest validation fails", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + const bytes = new TextEncoder().encode("piece"); + expect( + await send(stub, "bad-base64", { + command: "stage", + kind: "blob", + digest: sha256Hex(bytes), + bytes: "not base64", + }), + ).toMatchObject({ outcome: "refused", refusal: "command:malformed-member" }); + expect( + await send(stub, "bad-digest", { + command: "stage", + kind: "blob", + digest: "0".repeat(64), + bytes: encodeBase64(bytes), + }), + ).toMatchObject({ outcome: "refused", refusal: "command:malformed-member" }); + const oversized = new Uint8Array(MAX_CONTENT_BYTES + 1); + expect( + await send(stub, "oversized", { + command: "stage", + kind: "blob", + digest: sha256Hex(oversized), + bytes: encodeBase64(oversized), + }), + ).toMatchObject({ outcome: "refused", refusal: "command:too-large" }); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 0, staged: 0 }); + }); + + it("refuses aggregate staging overflow without a partial piece or decision", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + for (let index = 0; index < 2; index += 1) { + const bytes = new Uint8Array(MAX_CONTENT_BYTES); + bytes[0] = index; + expect( + await ask(socket, `piece-${index}`, { + command: "stage", + kind: "blob", + digest: sha256Hex(bytes), + bytes: encodeBase64(bytes), + }), + ).toMatchObject({ outcome: "performed", value: { size: MAX_CONTENT_BYTES } }); + } + const overflow = new Uint8Array([3]); + expect( + await ask(socket, "overflow", { + command: "stage", + kind: "blob", + digest: sha256Hex(overflow), + bytes: encodeBase64(overflow), + }), + ).toEqual({ id: "overflow", outcome: "refused", refusal: "command:capacity" }); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 2, staged: 2 }); + }); + + it("bounds the retry ledger without evicting earlier decisions", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + const first = await ask(socket, "command-0", { command: "frontier" }); + for (let index = 1; index < MAX_COMMANDS; index += 1) { + expect(await ask(socket, `command-${index}`, { command: "frontier" })).toMatchObject({ + outcome: "performed", + }); + } + expect(await ask(socket, "overflow", { command: "frontier" })).toEqual({ + id: "overflow", + outcome: "refused", + refusal: "command:capacity", + }); + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(socket.readyState).not.toBe(WebSocket.OPEN); + expect(await on(stub, (owner) => owner.scratch())).toEqual({ + commands: MAX_COMMANDS, + staged: 0, + }); + expect(first).toMatchObject({ outcome: "performed" }); + }); + + it("refuses D3 and D4 mutations rather than reporting placeholder success", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + expect( + await send(stub, "commit", { + command: "commit", + expectedWorkspaceRootId: ROOT_ID, + expectedJournalEventId: null, + proposedWorkspaceRootId: ROOT_ID, + events: [], + }), + ).toEqual({ id: "commit", outcome: "refused", refusal: "command:unavailable" }); + expect( + await send(stub, "settle", { + command: "settle", + completion: { executionId: "execution", status: "completed" }, + expectedWorkspaceRootId: ROOT_ID, + }), + ).toEqual({ id: "settle", outcome: "refused", refusal: "command:unavailable" }); + }); +}); diff --git a/packages/workflow/tests/cloudflare/settle-parser.vitest.ts b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts index 2a26c5dac..1f97cb13a 100644 --- a/packages/workflow/tests/cloudflare/settle-parser.vitest.ts +++ b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts @@ -116,8 +116,17 @@ describe("a root identity in a command", () => { /** Every root field in the private command shapes, by the request it sits in. */ const fields: Record string> = { - "materialize.workspaceRootId": (root) => - JSON.stringify({ id: "m1", command: "materialize", workspaceRootId: root }), + "root.workspaceRootId": (root) => + JSON.stringify({ id: "m1", command: "root", workspaceRootId: root }), + "content.workspaceRootId": (root) => + JSON.stringify({ + id: "r1", + command: "content", + workspaceRootId: root, + kind: "blob", + digest: ROOT, + sourceManifest: ROOT, + }), "commit.expectedWorkspaceRootId": (root) => commit({ expectedWorkspaceRootId: root }), "commit.proposedWorkspaceRootId": (root) => commit({ proposedWorkspaceRootId: root }), "settle.expectedWorkspaceRootId": (root) => settle({ expectedWorkspaceRootId: root }), @@ -129,7 +138,6 @@ describe("a root identity in a command", () => { command: "commit", expectedWorkspaceRootId: ROOT, expectedJournalEventId: null, - content: [], proposedWorkspaceRootId: ROOT, events: [], ...overrides, diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 5ba445b3f..88a22d258 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -8,13 +8,17 @@ */ import { run } from "effection"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; import { acquisitionHolders } from "../../../src/cloudflare/acquisition.ts"; import { WorkflowOwnerObject } from "../../../src/cloudflare/owner.ts"; import type { AdmissionRequest, OwnerConfiguration } from "../../../src/cloudflare/owner.ts"; import type { AdmissionPolicy } from "../../../src/cloudflare/admission.ts"; import type { TokenVerification, VerificationKey } from "../../../src/cloudflare/token.ts"; -import type { RunnerCommand } from "../../../src/cloudflare/commands.ts"; import { refusalOf } from "../../../src/cloudflare/owner.ts"; +import { sha256Hex } from "../../../src/workspace/sha256.ts"; +import { WORKSPACE_ROOT_DOMAIN } from "../../../src/workspace/root-manifest.ts"; +import { COMMAND_TABLE, STAGING_TABLE } from "../../../src/cloudflare/private-schema.ts"; +import { MARKER_TABLE } from "../../../src/cloudflare/marker.ts"; /** The identities this owner is configured to admit. */ export const POLICY: AdmissionPolicy = { @@ -41,7 +45,31 @@ export const VALID_CLAIMS: Record = { job_workflow_ref: POLICY.jobWorkflowRef, }; -const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +export const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +export const FILE_BYTES = new TextEncoder().encode("hello from the retained Workspace"); +export const BLOB_ID = sha256Hex(FILE_BYTES); +export const DOFS_MANIFEST = JSON.stringify({ + version: 1, + chunks: [{ hash: BLOB_ID, size: FILE_BYTES.length }], +}); +export const MANIFEST_ID = sha256Hex(new TextEncoder().encode(DOFS_MANIFEST)); +export const ROOT_MANIFEST = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/README.md", + kind: "file", + mode: 420, + mtime: 0, + size: FILE_BYTES.length, + manifest: MANIFEST_ID, + hardlink: null, + }, + ], +}); +export const ROOT_ID = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${ROOT_MANIFEST}`); +const CREATED_AT = "2026-09-03T00:00:00.000Z"; export class ExecutorObject extends WorkflowOwnerObject { /** @@ -74,8 +102,165 @@ export class ExecutorObject extends WorkflowOwnerObject { return { policy: POLICY, verification }; } - protected perform(_socket: WebSocket, _runId: string, command: RunnerCommand): unknown { - return { performed: command.command }; + initialize(): void { + this.open(RUN_ID, () => { + const blob = hexBytes(BLOB_ID); + const manifest = hexBytes(MANIFEST_ID); + this.ctx.storage.sql.exec( + "INSERT INTO vfs_blobs (hash, size, last_seen) VALUES (?, ?, 0)", + blob, + FILE_BYTES.length, + ); + this.ctx.storage.sql.exec( + "INSERT INTO vfs_blob_bytes (hash, bytes) VALUES (?, ?)", + blob, + new Uint8Array(FILE_BYTES), + ); + this.ctx.storage.sql.exec( + "INSERT INTO vfs_manifests (hash, size, encoded, last_seen) VALUES (?, ?, ?, 0)", + manifest, + FILE_BYTES.length, + new TextEncoder().encode(DOFS_MANIFEST), + ); + this.ctx.storage.sql.exec( + "INSERT INTO workspace_roots (root_id, format_version, manifest) VALUES (?, 1, ?)", + ROOT_ID, + ROOT_MANIFEST, + ); + this.ctx.storage.sql.exec( + "INSERT INTO workspace_root_manifest_refs (root_id, manifest_hash) VALUES (?, ?)", + ROOT_ID, + manifest, + ); + this.ctx.storage.sql.exec( + "INSERT INTO workspace_root_blob_refs (root_id, blob_hash) VALUES (?, ?)", + ROOT_ID, + blob, + ); + this.ctx.storage.sql.exec( + "INSERT INTO workspace_state (singleton_id, current_root_id) VALUES (1, ?)", + ROOT_ID, + ); + this.ctx.storage.sql.exec( + `INSERT INTO workflow_run + (id, run_id, definition, base, props, status, created_at, updated_at) + VALUES (1, ?, ?, ?, ?, 'running', ?, ?)`, + RUN_ID, + JSON.stringify({ + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }), + "main", + "{}", + CREATED_AT, + CREATED_AT, + ); + this.ctx.storage.sql.exec( + "INSERT INTO definition_retrieval (id, metadata, revision, updated_at) VALUES (1, ?, 1, ?)", + JSON.stringify({ locator: "https://example.invalid/repository.git" }), + CREATED_AT, + ); + }); + } + + appendJournal(eventId: string, name: string): void { + this.ctx.storage.sql.exec( + "INSERT INTO journal_events (event_id, record, workspace_root_id) VALUES (?, ?, ?)", + eventId, + serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }), + ROOT_ID, + ); + } + + scratch(): { commands: number; staged: number } { + const commands = this.ctx.storage.sql + .exec(`SELECT count(*) AS count FROM ${COMMAND_TABLE}`) + .toArray()[0]?.["count"]; + const staged = this.ctx.storage.sql + .exec(`SELECT count(*) AS count FROM ${STAGING_TABLE}`) + .toArray()[0]?.["count"]; + return { + commands: typeof commands === "number" ? commands : -1, + staged: typeof staged === "number" ? staged : -1, + }; + } + + /** + * Everything this run authoritatively holds, as one comparable value. + * + * Row-for-row rather than a count: cleanup that deleted a journal event and + * inserted another would keep every count identical, and the claim being + * checked is that acquisition cleanup touched none of this. + */ + authoritative(): string { + const tables = [ + "SELECT id, run_id, definition, base, props, status, created_at, updated_at FROM workflow_run ORDER BY id", + "SELECT root_id, format_version, manifest FROM workspace_roots ORDER BY root_id", + "SELECT singleton_id, current_root_id FROM workspace_state ORDER BY singleton_id", + "SELECT sequence, event_id, record, workspace_root_id FROM journal_events ORDER BY sequence", + "SELECT root_id, lower(hex(manifest_hash)) AS h FROM workspace_root_manifest_refs ORDER BY root_id, h", + "SELECT root_id, lower(hex(blob_hash)) AS h FROM workspace_root_blob_refs ORDER BY root_id, h", + "SELECT lower(hex(hash)) AS h, lower(hex(bytes)) AS b FROM vfs_blob_bytes ORDER BY h", + "SELECT lower(hex(hash)) AS h, size, lower(hex(encoded)) AS e FROM vfs_manifests ORDER BY h", + ]; + return sha256Hex( + JSON.stringify(tables.map((query) => this.ctx.storage.sql.exec(query).toArray())), + ); + } + + damageRetainedBlob(): void { + this.ctx.storage.sql.exec( + "UPDATE vfs_blob_bytes SET bytes = ?", + new TextEncoder().encode("bad"), + ); + } + + makeForeign(): void { + this.ctx.storage.sql.exec("CREATE TABLE foreign_state (id INTEGER PRIMARY KEY)"); + } + + rewriteMarker(applicationId: number, schemaVersion: number): void { + this.ctx.storage.sql.exec( + `UPDATE ${MARKER_TABLE} SET application_id = ?, schema_version = ? WHERE id = 1`, + applicationId, + schemaVersion, + ); + } + + dropTable(name: string): void { + this.ctx.storage.sql.exec(`DROP TABLE ${name}`); + } + + rewriteRunId(runId: string): void { + this.ctx.storage.sql.exec("UPDATE workflow_run SET run_id = ? WHERE id = 1", runId); + } + + removeWorkspaceState(): void { + this.ctx.storage.sql.exec("DELETE FROM workspace_state"); + } + + addUnreferencedBlob(bytes: Uint8Array): string { + const digest = sha256Hex(bytes); + const hash = hexBytes(digest); + this.ctx.storage.sql.exec( + "INSERT INTO vfs_blobs (hash, size, last_seen) VALUES (?, ?, 0)", + hash, + bytes.length, + ); + this.ctx.storage.sql.exec( + "INSERT INTO vfs_blob_bytes (hash, bytes) VALUES (?, ?)", + hash, + new Uint8Array(bytes), + ); + return digest; } /** @@ -101,6 +286,27 @@ export class ExecutorObject extends WorkflowOwnerObject { } } + async fetch(request: Request): Promise { + const pair = new WebSocketPair(); + const client = pair[0]; + const server = pair[1]; + try { + await run(() => + this.admit( + { + runId: request.headers.get("x-run-id"), + release: request.headers.get("x-release"), + token: request.headers.get("authorization")?.replace(/^Bearer /, ""), + }, + server, + ), + ); + return new Response(null, { status: 101, webSocket: client }); + } catch (error) { + return new Response(refusalOf(error), { status: 403 }); + } + } + /** The correlation the live acquisition is partitioned by. */ acquisitionId(): string { const held = acquisitionHolders(this.ctx)[0]; @@ -127,6 +333,16 @@ export class ExecutorObject extends WorkflowOwnerObject { return this.onRunnerMessage(pair[1], RUN_ID, raw); } + sendWithCopiedAttachment(raw: string): unknown { + const live = this.ctx.getWebSockets("executor")[0]; + if (live === undefined) { + return { id: "", outcome: "refused", refusal: "no-such-connection" }; + } + const pair = new WebSocketPair(); + pair[1].serializeAttachment(live.deserializeAttachment()); + return this.onRunnerMessage(pair[1], RUN_ID, raw); + } + /** Close the connection admitted at `index`, releasing its acquisition. */ closeConnection(index: number): void { const socket = this.ctx.getWebSockets("executor")[index - 1]; @@ -136,3 +352,11 @@ export class ExecutorObject extends WorkflowOwnerObject { } } } + +function hexBytes(value: string): Uint8Array { + const bytes = new Uint8Array(value.length / 2); + for (let index = 0; index < bytes.length; index += 1) { + bytes[index] = Number.parseInt(value.slice(index * 2, index * 2 + 2), 16); + } + return bytes; +} diff --git a/packages/workflow/tests/host-neutrality.test.ts b/packages/workflow/tests/host-neutrality.test.ts index b3a3cf2ec..bc10781ba 100644 --- a/packages/workflow/tests/host-neutrality.test.ts +++ b/packages/workflow/tests/host-neutrality.test.ts @@ -96,6 +96,15 @@ describe("the shared workflow package", () => { expect(modules.some((path) => path.endsWith("/src/lifecycle/execution.ts"))).toEqual(true); expect(modules.some((path) => path.endsWith("/src/software-factory/run-id.ts"))).toEqual(true); expect(modules.some((path) => path.endsWith("/src/sqlite/workflow-schema.ts"))).toEqual(true); + // The remote seam is ordinary shared code. It is the runner's half of a + // connection to a provider, which is exactly why it must name none: an + // exemption here would let the provider's vocabulary back in through the + // one module whose whole purpose is to keep it out. + expect(modules.some((path) => path.endsWith("/src/remote/read.ts"))).toEqual(true); + expect(modules.some((path) => path.endsWith("/src/remote/client.ts"))).toEqual(true); + expect(modules.some((path) => path.endsWith("/src/remote/records.ts"))).toEqual(true); + expect(modules.some((path) => path.endsWith("/src/workspace/root-manifest.ts"))).toEqual(true); + expect(modules.some((path) => path.endsWith("/src/workspace/sha256.ts"))).toEqual(true); expect(modules.some((path) => path.includes("/src/deno/"))).toEqual(false); expect(modules.some((path) => path.includes("/src/cloudflare/"))).toEqual(false); }); diff --git a/packages/workflow/tests/public-entrypoint.test.ts b/packages/workflow/tests/public-entrypoint.test.ts index 4095eaa82..b4d0620a8 100644 --- a/packages/workflow/tests/public-entrypoint.test.ts +++ b/packages/workflow/tests/public-entrypoint.test.ts @@ -22,6 +22,7 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { until } from "effection"; +import { readTextFile } from "@effectionx/fs"; import { spawnSync } from "node:child_process"; import process from "node:process"; import { fileURLToPath } from "node:url"; @@ -52,6 +53,7 @@ const PROBE = fileURLToPath(new URL("./support/public-entrypoint-probe.ts", impo const HELPER_MODULE = fileURLToPath( new URL("./support/credential-helper-entry.ts", import.meta.url), ); +const CLOUDFLARE_ENTRYPOINT = fileURLToPath(new URL("../cloudflare.ts", import.meta.url)); describe("workflow published Deno entrypoint", () => { it("offers no route from the entrypoint to an authenticated invocation", function* () { @@ -153,10 +155,25 @@ describe("workflow published Deno entrypoint", () => { "useGitComposition", "denoGitAuthentication", "denoCredentialBroker", + "RemoteReadLink", + "cloudflareReadLink", + "stageCloudflareContent", ]) { expect(reachable).not.toContain(seam); } expect(COMPOSITION_IS_NOT_A_KEY).toBe(false); expect(yield* until(Promise.resolve(true))).toBe(true); }); + + it("keeps the Cloudflare private protocol out of its host entrypoint", function* () { + const source = yield* readTextFile(CLOUDFLARE_ENTRYPOINT); + for (const privateModule of [ + "commands.ts", + "acquisition.ts", + "dispatcher.ts", + "private-schema.ts", + ]) { + expect(source).not.toContain(privateModule); + } + }); }); diff --git a/packages/workflow/tests/remote-read.test.ts b/packages/workflow/tests/remote-read.test.ts new file mode 100644 index 000000000..cf4b1e658 --- /dev/null +++ b/packages/workflow/tests/remote-read.test.ts @@ -0,0 +1,429 @@ +/** + * Tier WRH — reading a run from an owner somewhere else. + * + * What is under test here is the runner's half: whether a private answer + * becomes a semantic value only after it has been proved to be one, and whether + * a channel that has stopped making sense is stopped rather than followed. + * + * The owner is a deterministic fake, deliberately. Command-specific parsing, + * refusal narrowing and journal reassembly are arithmetic over what arrived, + * and a fake can produce the answers a correct owner never would — a page that + * skips an event, a refusal category from another release, content that is not + * what it is named. What the real owner does with a real request is proved on + * real workerd, where the runtime is the thing being relied on. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import { scoped } from "effection"; +import { + cloudflareOwnerLink, + cloudflareReadLink, + stageCloudflareContent, +} from "../src/cloudflare/client.ts"; +import { createTransactionGate, transactRemotely } from "../src/remote/collector.ts"; +import { encodeBase64 } from "../src/cloudflare/encoding.ts"; +import type { OwnerSocket, SocketListener } from "../src/remote/client.ts"; +import { OwnerLinkError, useOwnerConnection } from "../src/remote/client.ts"; +import { + EMPTY_WORKSPACE_MANIFEST, + EMPTY_WORKSPACE_ROOT_ID, + workspaceRootId, +} from "../src/deno/workspace/manifest.ts"; +import { WORKSPACE_ROOT_DOMAIN } from "../src/workspace/root-manifest.ts"; +import { sha256Hex } from "../src/workspace/sha256.ts"; + +const RUN_ID = "remote-run"; +const ROOT_MANIFEST = JSON.stringify({ + format: 1, + entries: [{ path: "/", kind: "directory", mode: 493, mtime: 0 }], +}); +const ROOT_ID = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${ROOT_MANIFEST}`); +const CONTENT = new TextEncoder().encode( + JSON.stringify({ version: 1, chunks: [{ hash: "0".repeat(64), size: 1 }] }), +); +const CONTENT_ID = sha256Hex(CONTENT); + +function event(name: string): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }); +} + +function runRecord(): Record { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-03T00:00:00.000Z", + updatedAt: "2026-09-03T00:00:00.000Z", + }; +} + +function object(value: unknown): Record { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new Error("expected an object"); + } + return Object.fromEntries(Object.entries(value)); +} + +function wire(answer: (request: Record) => Record) { + const listeners = new Map>(); + let closes = 0; + const socket: OwnerSocket = { + send(data: string): void { + const request = object(JSON.parse(data)); + const response = answer(request); + for (const listener of listeners.get("message") ?? []) { + listener({ data: JSON.stringify({ id: request["id"], ...response }) }); + } + }, + close(): void { + closes += 1; + }, + addEventListener(type, listener): void { + const found = listeners.get(type) ?? new Set(); + found.add(listener); + listeners.set(type, found); + }, + removeEventListener(type, listener): void { + listeners.get(type)?.delete(listener); + }, + }; + return { + socket, + get closes(): number { + return closes; + }, + get listeners(): number { + return [...listeners.values()].reduce((sum, found) => sum + found.size, 0); + }, + }; +} + +function ids(): () => string { + let id = 0; + return () => `request-${(id += 1)}`; +} + +/** The name one retained test event carries, read rather than asserted. */ +function effectName(entry: unknown): string { + if (entry === null || typeof entry !== "object" || !("description" in entry)) { + return ""; + } + const description = entry.description; + if (description === null || typeof description !== "object" || !("name" in description)) { + return ""; + } + const name: unknown = description["name"]; + return typeof name === "string" ? name : ""; +} + +function failure(error: unknown): string { + if (!(error instanceof OwnerLinkError)) { + throw new Error(`expected an OwnerLinkError, received ${String(error)}`); + } + return error.refusal; +} + +describe("semantic reads from a Cloudflare owner", () => { + it("uses the standard SHA-256 identity rather than an adapter-local digest", function* () { + // The published answers, including the two-block case the padding rule is + // easiest to get wrong on. + expect(sha256Hex("")).toBe("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"); + expect(sha256Hex("abc")).toBe( + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", + ); + expect(sha256Hex("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq")).toBe( + "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1", + ); + expect(sha256Hex(new Uint8Array(1000).fill(0x61))).toBe( + "41edece42d63e8d9bf515a9ba6932e1c20cbc9f5a5d134645adb5db1b9737ea3", + ); + }); + + it("computes the identity the local host computes for the same root", function* () { + // The two hosts retain the same roots and must name them identically. This + // one is arithmetic in the language; the Deno host uses `node:crypto`. A + // difference here would be two hosts disagreeing about history. + expect(sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${ROOT_MANIFEST}`)).toBe( + workspaceRootId(ROOT_MANIFEST), + ); + expect(sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${EMPTY_WORKSPACE_MANIFEST}`)).toBe( + EMPTY_WORKSPACE_ROOT_ID, + ); + }); + + it("strictly parses the private staging decision", function* () { + const bytes = new TextEncoder().encode("staged"); + const digest = sha256Hex(bytes); + const transport = wire((request) => ({ + outcome: "performed", + value: { kind: request["kind"], digest: request["digest"], size: bytes.length }, + })); + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + expect(yield* stageCloudflareContent(connection, "stage", "blob", bytes)).toEqual({ + kind: "blob", + digest, + size: bytes.length, + }); + }); + }); + + it("parses an anchored frontier, a canonical root, and verified content", function* () { + const transport = wire((request) => { + if (request["command"] === "frontier") { + return { + outcome: "performed", + value: { + record: runRecord(), + retrieval: { + metadata: { locator: "somewhere" }, + revision: 1, + updatedAt: "2026-09-03T00:00:00.000Z", + }, + workspaceRootId: ROOT_ID, + journalEventId: "event-2", + }, + }; + } + if (request["command"] === "journal" && request["afterEventId"] === null) { + return { + outcome: "performed", + value: { + anchorEventId: "event-2", + afterEventId: null, + entries: [ + { + eventId: "event-1", + previousEventId: null, + record: event("one"), + workspaceRootId: ROOT_ID, + }, + ], + done: false, + }, + }; + } + if (request["command"] === "journal") { + return { + outcome: "performed", + value: { + anchorEventId: "event-2", + afterEventId: "event-1", + entries: [ + { + eventId: "event-2", + previousEventId: "event-1", + record: event("two"), + workspaceRootId: ROOT_ID, + }, + ], + done: true, + }, + }; + } + if (request["command"] === "root") { + return { + outcome: "performed", + value: { workspaceRootId: ROOT_ID, manifest: ROOT_MANIFEST }, + }; + } + return { + outcome: "performed", + value: { + kind: "manifest", + digest: CONTENT_ID, + size: CONTENT.length, + bytes: encodeBase64(CONTENT), + }, + }; + }); + + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const reads = cloudflareReadLink(connection, ids(), RUN_ID); + const frontier = yield* reads.frontier(); + expect(frontier.entries.map((entry) => entry.eventId)).toEqual(["event-1", "event-2"]); + expect(frontier.workspaceRootId).toBe(ROOT_ID); + expect((yield* reads.root(ROOT_ID)).entries).toHaveLength(1); + expect( + (yield* reads.content(ROOT_ID, { kind: "manifest", digest: CONTENT_ID })).bytes, + ).toEqual(CONTENT); + }); + expect(transport.closes).toBe(1); + expect(transport.listeners).toBe(0); + }); + + it("closes on a journal page that does not continue its snapshot", function* () { + const anchored = (entries: Record[], done = true) => ({ + anchorEventId: "event-2", + afterEventId: null, + entries, + done, + }); + const entry = (eventId: string, previousEventId: string | null) => ({ + eventId, + previousEventId, + record: event(eventId), + workspaceRootId: ROOT_ID, + }); + + // Four ways one page can fail to be the continuation it claims to be. The + // structural consequence is one: the events never reach a caller, because + // a journal that is missing an event looks exactly like a shorter journal. + const pages: Record> = { + skipped: anchored([entry("event-2", "event-1")]), + "out of order": anchored([entry("event-2", null), entry("event-1", "event-2")], false), + duplicated: anchored([entry("event-1", null), entry("event-1", "event-1")], false), + "not terminal": anchored([entry("event-1", null)]), + }; + + for (const [description, page] of Object.entries(pages)) { + const transport = wire((request) => + request["command"] === "frontier" + ? { + outcome: "performed", + value: { + record: runRecord(), + retrieval: null, + workspaceRootId: ROOT_ID, + journalEventId: "event-2", + }, + } + : { outcome: "performed", value: page }, + ); + let raised: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + try { + yield* cloudflareReadLink(connection, ids(), RUN_ID).frontier(); + } catch (error) { + raised = error; + } + }); + expect([description, failure(raised)]).toEqual([description, "malformed-answer"]); + expect([description, transport.closes]).toEqual([description, 1]); + expect([description, transport.listeners]).toEqual([description, 0]); + } + }); + + it("closes on an unknown same-release refusal", function* () { + const transport = wire(() => ({ outcome: "refused", refusal: "command:newer-release" })); + let raised: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + try { + yield* cloudflareReadLink(connection, ids(), RUN_ID).frontier(); + } catch (error) { + raised = error; + } + }); + expect(failure(raised)).toBe("malformed-answer"); + expect(transport.closes).toBe(1); + }); + + it("closes when content bytes disagree with the requested identity", function* () { + const transport = wire(() => ({ + outcome: "performed", + value: { + kind: "blob", + digest: CONTENT_ID, + size: 1, + bytes: encodeBase64(new Uint8Array([1])), + }, + })); + let raised: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + try { + yield* cloudflareReadLink(connection, ids(), RUN_ID).content(ROOT_ID, { + kind: "blob", + digest: CONTENT_ID, + manifestDigest: CONTENT_ID, + }); + } catch (error) { + raised = error; + } + }); + expect(failure(raised)).toBe("malformed-answer"); + expect(transport.closes).toBe(1); + }); + it("hands the collector one assembled frontier and no page mechanics", function* () { + const pages: Record = { + null: { + anchorEventId: "event-2", + afterEventId: null, + entries: [ + { + eventId: "event-1", + previousEventId: null, + record: event("one"), + workspaceRootId: ROOT_ID, + }, + ], + done: false, + }, + "event-1": { + anchorEventId: "event-2", + afterEventId: "event-1", + entries: [ + { + eventId: "event-2", + previousEventId: "event-1", + record: event("two"), + workspaceRootId: ROOT_ID, + }, + ], + done: true, + }, + }; + const transport = wire((request) => + request["command"] === "frontier" + ? { + outcome: "performed", + value: { + record: runRecord(), + retrieval: null, + workspaceRootId: ROOT_ID, + journalEventId: "event-2", + }, + } + : { outcome: "performed", value: pages[String(request["afterEventId"])] }, + ); + + let seen: unknown[] = []; + let committed: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(cloudflareReadLink(connection, ids(), RUN_ID)); + // Two pages went over the wire. What the body reads back is one journal: + // the collector is handed the assembled prefix and never learns that a + // page, a cursor or an anchor was involved. + const outcome = yield* transactRemotely(link, createTransactionGate(), function* (tx) { + seen = yield* tx.journal.readAll(); + return "done"; + }); + committed = outcome; + }); + + expect(seen).toHaveLength(2); + expect(seen.map(effectName)).toEqual(["one", "two"]); + // D2 has reads and no commit. The transaction returns the owner's refusal + // rather than a success nothing performed. + expect(committed).toMatchObject({ ok: false }); + }); +}); diff --git a/packages/workflow/tests/workflow-export.test.ts b/packages/workflow/tests/workflow-export.test.ts index e327b27e5..3f0552b5b 100644 --- a/packages/workflow/tests/workflow-export.test.ts +++ b/packages/workflow/tests/workflow-export.test.ts @@ -460,7 +460,7 @@ function base64(content: Uint8Array): string { function artifactWorkspace( artifact: VerifiedXmdArtifact, rootId: string, -): { nodes: Map; entries: WorkspaceRootEntry[] } { +): { nodes: Map; entries: readonly WorkspaceRootEntry[] } { const root = artifact.roots.find((candidate) => candidate.rootId === rootId); if (root === undefined) { throw new Error(`the artifact holds no Workspace root ${rootId}`); From 827f1d4ecdf23b406fe79d49aac7d6fe9e7c96cf Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 16:19:28 -0400 Subject: [PATCH 26/81] =?UTF-8?q?=F0=9F=A7=B1=20Prove=20a=20whole=20retain?= =?UTF-8?q?ed=20root=20before=20answering=20with=20one=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A root the owner returns is where the run stands. The runner materializes it, works inside it, and proposes against it. So a root whose content cannot all be found is not a frontier that happens to be incomplete — it is not a frontier, and saying otherwise is the one answer that cannot be taken back. The reads checked the root manifest and the manifests its entries named, and stopped. Everything past that — whether those manifests were still there, what their bytes were, whether the blobs they named existed, whether the root retained references to exactly those blobs — was checked only when a runner later asked for that particular piece. A root missing half its content therefore answered `frontier` and `root` performed, and failed afterwards, one piece at a time, once the run had already been told where it stood. Now the graph is walked before either answers. The manifests the entries name must be exactly the manifests the root retains; each must exist, be bounded, decode canonically, hash to its identity, and agree with its recorded size and with every file naming it. The blobs those manifests name must be exactly the blobs the root retains; each must exist, be bounded, hash to its identity, and agree with its recorded size and with every chunk naming it. Both directions of each reference set are checked: a missing row is content nothing is keeping alive, an extra row is content no manifest accounts for, and neither describes a root anybody should start from. The bytes are read and dropped. What survives the walk is the proof, and a content request still re-reads the single piece it sends — a validated root is not permission to answer with all of it at once. The schema already refuses to let content vanish from under a root that references it, so the states worth reproducing are the ones that restriction cannot prevent: a reference collected together with its content, bytes that no longer hash to the identity they are stored under, a recorded size that disagrees with what it describes, and a reference to content no manifest names. Each of those now refuses before a root is returned, and says only that storage is damaged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- .../workflow/src/cloudflare/owner-reads.ts | 254 ++++++++++++------ .../tests/cloudflare/remote-owner.vitest.ts | 60 +++++ .../cloudflare/support/executor-object.ts | 69 +++++ 3 files changed, 307 insertions(+), 76 deletions(-) diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index 1598faa0e..2b1467acf 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -12,9 +12,13 @@ * anchor is the last event that existed at that moment, so later appends cannot * enter an earlier snapshot. Reads are *bounded*: a journal is returned in * pages anchored to that event, and content comes back one piece at a time. - * Reads are *referenced*: a piece of content is admitted only if the named root - * actually names it, directly or through a manifest it names, so this is a read - * of one retained root rather than of a content-addressed store. + * Reads are *referenced*: a root is returned only once its complete content + * graph has been proved present and self-consistent, and a piece is admitted + * only if that root actually names it, so this is a read of one retained root + * rather than of a content-addressed store. Validating the graph up front is + * the point: a root is a starting frontier, and a frontier that turns out not + * to be materializable after the runner has it is a failure arriving too late + * to mean anything. * * A refusal says the category and nothing else. Column values, retained JSON * and request data never appear in one: the caller learns that storage is @@ -26,6 +30,7 @@ import { readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { decodeDofsManifest, + type DofsManifest, parseWorkspaceRootManifest, SHA256, WORKSPACE_ROOT_DOMAIN, @@ -71,10 +76,18 @@ export interface ContentValue { readonly bytes: string; } +/** + * One retained root, and the whole content graph it names, proved. + * + * `manifests` and `blobs` are not a description of what the root refers to — + * they are what was found and checked. A `StoredRoot` therefore cannot exist + * for a root whose graph is incomplete or disagrees with itself. + */ interface StoredRoot { readonly manifest: string; readonly parsed: WorkspaceRootManifest; - readonly manifests: ReadonlySet; + readonly manifests: ReadonlyMap; + readonly blobs: ReadonlySet; } function corrupt(reason: string): never { @@ -111,61 +124,45 @@ function byteRows(storage: OwnerStorage, sql: string, ...bindings: unknown[]): R return storage.sql.exec(sql, ...bindings).toArray(); } -function referencedRoot(storage: OwnerStorage, rootId: string): StoredRoot { - if (!SHA256.test(rootId)) { - return corrupt("a Workspace root identity is malformed"); - } - const root = exactlyOne( - byteRows( - storage, - "SELECT root_id, format_version, manifest FROM workspace_roots WHERE root_id = ?", - rootId, - ), - "Workspace root", - ); - const manifest = safeText(root, "manifest"); - if (root["format_version"] !== 1) { - return corrupt("a Workspace root has an unsupported format"); - } - const parsed = parseWorkspaceRootManifest(manifest, corrupt); - if (rootIdentity(manifest) !== rootId || root["root_id"] !== rootId) { - return corrupt("a Workspace root disagrees with its identity"); - } - if (new TextEncoder().encode(manifest).length > MAX_CONTENT_BYTES) { - throw new CommandError("too-large"); - } - - const expectedManifests = new Set( - parsed.entries.flatMap((entry) => (entry.kind === "file" ? [entry.manifest] : [])), - ); - const manifestRows = byteRows( +/** Every content identity one root's reference table holds, in order. */ +function referenceRows( + storage: OwnerStorage, + table: string, + column: string, + rootId: string, +): string[] { + return byteRows( storage, - `SELECT lower(hex(manifest_hash)) AS digest - FROM workspace_root_manifest_refs WHERE root_id = ? ORDER BY digest`, + `SELECT lower(hex(${column})) AS digest FROM ${table} WHERE root_id = ? ORDER BY digest`, rootId, - ); - if (manifestRows.length !== expectedManifests.size) { - return corrupt("a Workspace root's manifest references are incomplete"); - } - const manifests = new Set(); - for (const row of manifestRows) { - const digest = safeText(row, "digest"); - if (!expectedManifests.has(digest)) { - return corrupt("a Workspace root has an extra manifest reference"); - } - manifests.add(digest); + ).map((row) => safeText(row, "digest")); +} + +/** + * Whether a reference table holds exactly the identities the content names. + * + * Both directions matter and for different reasons. A missing row is content + * the root depends on that nothing is keeping alive, so retention may already + * have collected it. An extra row is the root claiming content it does not use, + * which keeps bytes reachable that no manifest accounts for. Neither is a root + * this owner will hand to a runner as a starting frontier. + */ +function requireReferenceSet( + found: readonly string[], + expected: ReadonlySet, + what: string, +): void { + if (found.length !== expected.size || found.some((digest) => !expected.has(digest))) { + corrupt(`a Workspace root's ${what} references disagree with its content`); } - return { manifest, parsed, manifests }; } -function retainedManifest( +/** One retained DOFS manifest, proved against its identity, size and entries. */ +function validatedManifest( storage: OwnerStorage, - root: StoredRoot, + parsed: WorkspaceRootManifest, digest: string, -): { bytes: Uint8Array; chunks: ReturnType["chunks"] } { - if (!root.manifests.has(digest)) { - return corrupt("a DOFS manifest is not referenced by this Workspace root"); - } +): { bytes: Uint8Array; manifest: DofsManifest } { const row = exactlyOne( byteRows(storage, "SELECT size, encoded FROM vfs_manifests WHERE lower(hex(hash)) = ?", digest), "DOFS manifest", @@ -174,30 +171,25 @@ function retainedManifest( if (bytes.length > MAX_CONTENT_BYTES || sha256Hex(bytes) !== digest) { return corrupt("a retained DOFS manifest disagrees with its identity"); } - const decoded = decodeDofsManifest(bytes, corrupt); - if (safeInteger(row["size"], "manifest size") !== decoded.size) { + const manifest = decodeDofsManifest(bytes, corrupt); + if (safeInteger(row["size"], "manifest size") !== manifest.size) { return corrupt("a retained DOFS manifest disagrees with its recorded size"); } - for (const entry of root.parsed.entries) { - if (entry.kind === "file" && entry.manifest === digest && entry.size !== decoded.size) { + for (const entry of parsed.entries) { + if (entry.kind === "file" && entry.manifest === digest && entry.size !== manifest.size) { return corrupt("a Workspace file size disagrees with its retained manifest"); } } - return { bytes, chunks: decoded.chunks }; + return { bytes, manifest }; } -function retainedBlob( +/** One retained blob, proved against its identity and every chunk naming it. */ +function validatedBlob( storage: OwnerStorage, rootId: string, - root: StoredRoot, - sourceManifest: string, + manifests: ReadonlyMap, digest: string, ): Uint8Array { - const manifest = retainedManifest(storage, root, sourceManifest); - const expected = manifest.chunks.find((chunk) => chunk.hash === digest); - if (expected === undefined) { - return corrupt("a blob is not referenced by the named DOFS manifest"); - } const row = exactlyOne( byteRows( storage, @@ -211,17 +203,100 @@ function retainedBlob( "DOFS blob", ); const bytes = bytesOf(row["bytes"]); - if ( - bytes.length > MAX_CONTENT_BYTES || - bytes.length !== expected.size || - safeInteger(row["size"], "blob size") !== expected.size || - sha256Hex(bytes) !== digest - ) { - return corrupt("a retained DOFS blob disagrees with its identity or size"); + if (bytes.length > MAX_CONTENT_BYTES || sha256Hex(bytes) !== digest) { + return corrupt("a retained DOFS blob disagrees with its identity"); + } + if (safeInteger(row["size"], "blob size") !== bytes.length) { + return corrupt("a retained DOFS blob disagrees with its recorded size"); + } + for (const manifest of manifests.values()) { + for (const chunk of manifest.chunks) { + if (chunk.hash === digest && chunk.size !== bytes.length) { + return corrupt("a DOFS chunk size disagrees with the blob it names"); + } + } } return bytes; } +/** + * One retained root, with its complete content graph proved before it is a root + * at all. + * + * Accepting a root is accepting a starting frontier: the runner will + * materialize it, work in it, and propose against it. A root whose graph cannot + * be materialized is not a frontier, and discovering that one piece at a time — + * after the frontier has already crossed to the runner — would mean the failure + * arrives once the run has already been told where it stands. + * + * So the whole graph is walked here. The manifests the entries name must be + * exactly the manifests the root retains; each must exist, be bounded, decode + * canonically, hash to its identity, and agree with its recorded size and with + * every file that names it. The blobs those manifests name must be exactly the + * blobs the root retains; each must exist, be bounded, hash to its identity, + * and agree with its recorded size and with every chunk that names it. + * + * The bytes are read and dropped. What is kept is the proof, and a later + * content request re-reads the single piece it is sending — which is what keeps + * the transport piece-oriented rather than turning a validated root into one + * unbounded answer. + */ +function referencedRoot(storage: OwnerStorage, rootId: string): StoredRoot { + if (!SHA256.test(rootId)) { + return corrupt("a Workspace root identity is malformed"); + } + const root = exactlyOne( + byteRows( + storage, + "SELECT root_id, format_version, manifest FROM workspace_roots WHERE root_id = ?", + rootId, + ), + "Workspace root", + ); + const manifest = safeText(root, "manifest"); + if (root["format_version"] !== 1) { + return corrupt("a Workspace root has an unsupported format"); + } + const parsed = parseWorkspaceRootManifest(manifest, corrupt); + if (rootIdentity(manifest) !== rootId || root["root_id"] !== rootId) { + return corrupt("a Workspace root disagrees with its identity"); + } + if (new TextEncoder().encode(manifest).length > MAX_CONTENT_BYTES) { + throw new CommandError("too-large"); + } + + const named = new Set( + parsed.entries.flatMap((entry) => (entry.kind === "file" ? [entry.manifest] : [])), + ); + requireReferenceSet( + referenceRows(storage, "workspace_root_manifest_refs", "manifest_hash", rootId), + named, + "manifest", + ); + + const manifests = new Map(); + for (const digest of named) { + manifests.set(digest, validatedManifest(storage, parsed, digest).manifest); + } + + const reachable = new Set(); + for (const decoded of manifests.values()) { + for (const chunk of decoded.chunks) { + reachable.add(chunk.hash); + } + } + requireReferenceSet( + referenceRows(storage, "workspace_root_blob_refs", "blob_hash", rootId), + reachable, + "blob", + ); + for (const digest of reachable) { + validatedBlob(storage, rootId, manifests, digest); + } + + return { manifest, parsed, manifests, blobs: reachable }; +} + export function readFrontier(storage: OwnerStorage, runId: string): FrontierValue { const record = readRunRecord( exactlyOne( @@ -345,12 +420,39 @@ export function readContent( sourceManifest: string | null, ): ContentValue { const root = referencedRoot(storage, workspaceRootId); - const bytes = - kind === "manifest" - ? retainedManifest(storage, root, digest).bytes - : retainedBlob(storage, workspaceRootId, root, sourceManifest ?? "", digest); + const bytes = piece(storage, workspaceRootId, root, kind, digest, sourceManifest); if (bytes.length === 0) { return corrupt("a retained content piece is empty"); } return { kind, digest, size: bytes.length, bytes: encodeBase64(bytes) }; } + +/** + * The one piece a content request names, re-read from the proved graph. + * + * Membership is decided against what the root actually names rather than + * against the reference tables alone, and a blob is reached only through a + * manifest the request names. That is what keeps this a read of one retained + * root instead of a read of the content store: staged, orphaned or + * otherwise-unreferenced bytes are addressable by nobody through here. + */ +function piece( + storage: OwnerStorage, + rootId: string, + root: StoredRoot, + kind: "manifest" | "blob", + digest: string, + sourceManifest: string | null, +): Uint8Array { + if (kind === "manifest") { + if (!root.manifests.has(digest)) { + return corrupt("a DOFS manifest is not referenced by this Workspace root"); + } + return validatedManifest(storage, root.parsed, digest).bytes; + } + const source = sourceManifest === null ? undefined : root.manifests.get(sourceManifest); + if (source === undefined || !source.chunks.some((chunk) => chunk.hash === digest)) { + return corrupt("a blob is not referenced by the named DOFS manifest"); + } + return validatedBlob(storage, rootId, root.manifests, digest); +} diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index edc7076ac..afa58cc0f 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -317,6 +317,66 @@ describe("the remote owner protocol", () => { ).toEqual({ id: "orphan", outcome: "refused", refusal: "storage:corrupt" }); }); + it("refuses a root whose content graph is incomplete, before returning one", async () => { + // A root is a starting frontier: the runner materializes it and proposes + // against it. Discovering a piece is missing when the runner asks for it + // would mean the failure arrives after the run has been told where it + // stands, so the whole graph is proved before either read answers. + const damage: Record void> = { + "a missing manifest row": (owner) => owner.removeManifestRow(), + "a manifest payload that is not its identity": (owner) => owner.damageManifestPayload(), + "a manifest size that disagrees with its chunks": (owner) => owner.damageManifestSize(), + "a missing blob reached through a manifest": (owner) => owner.removeBlobRow(), + "blob bytes that are not their identity": (owner) => owner.damageRetainedBlob(), + "a blob size that disagrees with its bytes": (owner) => owner.damageBlobSize(), + "a blob reference the manifests still name": (owner) => owner.removeBlobReference(), + "a blob reference no manifest names": (owner) => + owner.addExtraBlobReference(new TextEncoder().encode("unaccounted for")), + }; + + for (const [description, arrange] of Object.entries(damage)) { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await on(stub, arrange); + await admit(stub); + for (const command of [ + { command: "frontier" }, + { command: "root", workspaceRootId: ROOT_ID }, + ]) { + const answer = await send(stub, `${String(command.command)}`, command); + expect([description, command.command, answer]).toEqual([ + description, + command.command, + { id: command.command, outcome: "refused", refusal: "storage:corrupt" }, + ]); + } + } + }); + + it("says only that storage is damaged, and never what it read or was asked", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const unaccounted = await on(stub, (owner) => + owner.addExtraBlobReference(new TextEncoder().encode("unaccounted for")), + ); + await admit(stub); + const answer = await send(stub, "root", { command: "root", workspaceRootId: ROOT_ID }); + expect(answer).toEqual({ id: "root", outcome: "refused", refusal: "storage:corrupt" }); + const printed = JSON.stringify(answer); + for (const withheld of [ + unaccounted, + BLOB_ID, + MANIFEST_ID, + ROOT_ID, + ROOT_MANIFEST, + "workspace_root_blob_refs", + "vfs_manifests", + "/README.md", + ]) { + expect(printed).not.toContain(withheld); + } + }); + it("refuses retained bytes whose identity or recorded size is damaged", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 88a22d258..bcf675972 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -223,6 +223,75 @@ export class ExecutorObject extends WorkflowOwnerObject { ); } + /** + * Collect the DOFS manifest a retained file entry still names. + * + * The reference row goes first because the schema will not let it go second: + * `ON DELETE RESTRICT` is what stops content vanishing from under a root that + * references it. What this reproduces is the state that restriction cannot + * prevent — a root whose manifest still names content the store no longer + * keeps, with the reference collected alongside it. + */ + removeManifestRow(): void { + this.ctx.storage.sql.exec( + "DELETE FROM workspace_root_manifest_refs WHERE lower(hex(manifest_hash)) = ?", + MANIFEST_ID, + ); + this.ctx.storage.sql.exec("DELETE FROM vfs_manifests WHERE lower(hex(hash)) = ?", MANIFEST_ID); + } + + /** Keep the manifest row, change the bytes it is identified by. */ + damageManifestPayload(): void { + this.ctx.storage.sql.exec( + "UPDATE vfs_manifests SET encoded = ? WHERE lower(hex(hash)) = ?", + new TextEncoder().encode('{"version":1,"chunks":[]}'), + MANIFEST_ID, + ); + } + + /** Keep identity and payload, disagree about how many bytes they describe. */ + damageManifestSize(): void { + this.ctx.storage.sql.exec( + "UPDATE vfs_manifests SET size = size + 1 WHERE lower(hex(hash)) = ?", + MANIFEST_ID, + ); + } + + /** Collect the blob a referenced manifest chunk still names, reference first. */ + removeBlobRow(): void { + this.removeBlobReference(); + this.ctx.storage.sql.exec("DELETE FROM vfs_blob_bytes WHERE lower(hex(hash)) = ?", BLOB_ID); + this.ctx.storage.sql.exec("DELETE FROM vfs_blobs WHERE lower(hex(hash)) = ?", BLOB_ID); + } + + /** Keep the blob and its bytes, disagree about its recorded size. */ + damageBlobSize(): void { + this.ctx.storage.sql.exec( + "UPDATE vfs_blobs SET size = size + 1 WHERE lower(hex(hash)) = ?", + BLOB_ID, + ); + } + + /** Drop the root's reference to a blob its manifests still name. */ + removeBlobReference(): void { + this.ctx.storage.sql.exec( + "DELETE FROM workspace_root_blob_refs WHERE root_id = ? AND lower(hex(blob_hash)) = ?", + ROOT_ID, + BLOB_ID, + ); + } + + /** Reference content from the root that none of its manifests names. */ + addExtraBlobReference(bytes: Uint8Array): string { + const digest = this.addUnreferencedBlob(bytes); + this.ctx.storage.sql.exec( + "INSERT INTO workspace_root_blob_refs (root_id, blob_hash) VALUES (?, ?)", + ROOT_ID, + hexBytes(digest), + ); + return digest; + } + makeForeign(): void { this.ctx.storage.sql.exec("CREATE TABLE foreign_state (id INTEGER PRIMARY KEY)"); } From d97615f6509b981b44fac195e1675ce55f1add8b Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 17:59:29 -0400 Subject: [PATCH 27/81] =?UTF-8?q?=F0=9F=8C=B3=20Put=20a=20retained=20root?= =?UTF-8?q?=20on=20a=20runner=20and=20read=20it=20back=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner holds the run and can run nothing. Git, an Agent and an evidence command need real files, so a root has to become a directory somewhere the runner owns, and whatever happens there has to come back as a root again. The equality that matters is that those two operations compose to nothing: a materialization nobody touched must capture to the exact root it came from. If it did not, every Workspace operation that changed nothing would still propose a new root, and the owner could not tell a real change from an artefact of how the runner unpacked the tree. Directories, an empty file, a symbolic link, a hardlink group, modes and modification times all have to survive for that to be true, which is why the evidence is a real temporary filesystem rather than a map that would only prove it kept what it was given. Ordering, hardlink numbering, manifest encoding and chunk size now live beside the format instead of inside whoever happened to walk the tree. The local host walks SQLite rows and the runner walks a directory; those walks cannot be shared and their meaning must not diverge, because a root identity is a digest of the encoding and two encodings would be two names for one Workspace. The materialization knows no runtime and no path. Native operations arrive injected, adapted from the runtime's asynchronous primitives with `until` where `@effectionx/fs` has no equivalent; the logical root stays `/`, and the temporary directory the invocation happens to use reaches no manifest, event, proposal or error. A run that recorded where it was unpacked could not be resumed anywhere else. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/deno/remote-files.ts | 116 ++++++++ packages/workflow/src/remote/materialize.ts | 274 ++++++++++++++++++ packages/workflow/src/workspace/capture.ts | 218 ++++++++++++++ .../tests/remote-materialization.test.ts | 194 +++++++++++++ 4 files changed, 802 insertions(+) create mode 100644 packages/workflow/src/deno/remote-files.ts create mode 100644 packages/workflow/src/remote/materialize.ts create mode 100644 packages/workflow/src/workspace/capture.ts create mode 100644 packages/workflow/tests/remote-materialization.test.ts diff --git a/packages/workflow/src/deno/remote-files.ts b/packages/workflow/src/deno/remote-files.ts new file mode 100644 index 000000000..6e18ec6c3 --- /dev/null +++ b/packages/workflow/src/deno/remote-files.ts @@ -0,0 +1,116 @@ +/** + * The runner's own filesystem, as materialization needs to see it. + * + * `@effectionx/fs` covers the ordinary work but not the whole Workspace + * contract: a retained root carries symbolic links, hardlink groups, modes and + * modification times, and preserving those is what makes an untouched + * materialization capture back to the root it came from. The operations it + * lacks are adapted here from the runtime's own asynchronous primitives with + * `until`, which is the sanctioned way to reach one — not by making production + * code asynchronous and not by reaching for a synchronous call. + * + * `node:fs/promises` rather than a runtime global, because the same adapter has + * to work wherever the runner runs. Nothing above this module names a runtime, + * and nothing in this module decides anything about a Workspace: it moves bytes + * and metadata where it is told, and the rules live in shared code. + */ + +import { + link, + lstat, + mkdir, + readdir, + readFile, + readlink, + symlink, + utimes, + writeFile, +} from "node:fs/promises"; +import { join } from "node:path"; +import { type Operation, until } from "effection"; +import type { RunnerFiles, RunnerNode } from "../remote/materialize.ts"; + +/** Whole seconds, which is what a retained entry records. */ +function seconds(milliseconds: number): number { + return Math.floor(milliseconds / 1000); +} + +function describeStats( + name: string, + stats: { + isDirectory(): boolean; + isSymbolicLink(): boolean; + mode: number; + mtimeMs: number; + size: number; + ino: number | bigint; + nlink: number | bigint; + }, + target: string | undefined, +): RunnerNode { + const kind = stats.isSymbolicLink() ? "symlink" : stats.isDirectory() ? "directory" : "file"; + return { + name, + kind, + // The permission bits only. The type bits are what `kind` already said, and + // a retained mode that carried them would not round-trip through the + // format's own bound. + mode: stats.mode & 0o7777, + mtime: seconds(stats.mtimeMs), + size: kind === "file" ? stats.size : 0, + // Only a file reached by more than one name can be part of a group, so + // anything else reports no identity and is captured on its own. + identity: kind === "file" && Number(stats.nlink) > 1 ? String(stats.ino) : undefined, + target, + }; +} + +/** The runner's filesystem operations, for one materialized tree. */ +export function runnerFiles(): RunnerFiles { + return { + *makeDirectory(path: string, mode: number): Operation { + yield* until(mkdir(path, { recursive: false, mode })); + }, + + *writeFile(path: string, bytes: Uint8Array, mode: number): Operation { + yield* until(writeFile(path, bytes, { mode })); + }, + + *makeSymlink(target: string, path: string): Operation { + yield* until(symlink(target, path)); + }, + + *makeHardlink(existing: string, path: string): Operation { + yield* until(link(existing, path)); + }, + + *setModifiedAt(path: string, mtime: number): Operation { + yield* until(utimes(path, mtime, mtime)); + }, + + *readFile(path: string): Operation { + return new Uint8Array(yield* until(readFile(path))); + }, + + *list(path: string): Operation { + const names = yield* until(readdir(path)); + const found: RunnerNode[] = []; + for (const name of names) { + const entry = join(path, name); + const stats = yield* until(lstat(entry)); + // Read, never resolved: what a retained link points at is part of the + // Workspace's description of itself, not somewhere to go looking. + const target: string | undefined = stats.isSymbolicLink() + ? yield* until(readlink(entry)) + : undefined; + found.push(describeStats(name, stats, target)); + } + return found; + }, + + *describe(path: string): Operation { + const stats = yield* until(lstat(path)); + return describeStats("", stats, undefined); + }, + }; +} diff --git a/packages/workflow/src/remote/materialize.ts b/packages/workflow/src/remote/materialize.ts new file mode 100644 index 000000000..8968b6042 --- /dev/null +++ b/packages/workflow/src/remote/materialize.ts @@ -0,0 +1,274 @@ +/** + * Putting one retained Workspace root on a runner, and reading it back. + * + * The owner holds the run and cannot run anything. Git, an Agent, an evidence + * command — all of it needs real files, and real files are the runner's. So a + * root is materialized into a temporary tree the invocation owns, worked in, + * and captured back into a proposal the owner validates and publishes. + * + * Two things this module refuses to know. It does not know a runtime: every + * native operation arrives as an injected Effection operation, so the same code + * materializes onto whatever filesystem the host adapter wrapped. And it does + * not treat the host path as identity — the logical Workspace root is `/`, the + * temporary directory is an implementation detail of this invocation, and no + * part of the host path reaches a manifest, a journal event, a proposal or an + * error. A run that recorded where it happened to be unpacked would be a run + * that could not be resumed anywhere else. + * + * Everything is verified twice. The owner validated the root before sending it + * and the connection verified each piece on arrival; this verifies again on the + * way to disk, because what must be true is not "the owner was honest" but + * "these bytes are the bytes this root names". The same holds coming back: a + * capture is checked against the rules a stored root is read through before it + * is ever proposed. + */ + +import { type Operation } from "effection"; +import { + captureContent, + type CapturedContent, + type CapturedNode, + type CapturedRoot, + captureWorkspaceRoot, +} from "../workspace/capture.ts"; +import { + compareUtf8, + decodeDofsManifest, + type WorkspaceRejection, + type WorkspaceRootManifest, +} from "../workspace/root-manifest.ts"; +import { sha256Hex } from "../workspace/sha256.ts"; +import type { RemoteReadLink } from "./read.ts"; + +/** One node the runner found, as its host describes one. */ +export interface RunnerNode { + readonly name: string; + readonly kind: "directory" | "file" | "symlink"; + readonly mode: number; + /** Whole seconds, matching what a retained entry carries. */ + readonly mtime: number; + readonly size: number; + /** + * What makes two paths one file. + * + * The host's own answer — an inode, or whatever stands in for one. Absent + * means the host cannot say, and every file is then its own. + */ + readonly identity: string | undefined; + /** Present only for a symbolic link, and never followed. */ + readonly target: string | undefined; +} + +/** + * The native operations materialization needs, and only those. + * + * Deliberately small and deliberately injected. Nothing here opens a process, + * resolves a symbolic link, or reaches outside the directory it was given. + */ +export interface RunnerFiles { + makeDirectory(path: string, mode: number): Operation; + writeFile(path: string, bytes: Uint8Array, mode: number): Operation; + makeSymlink(target: string, path: string): Operation; + makeHardlink(existing: string, path: string): Operation; + /** Applied last, because writing into a directory moves its own time. */ + setModifiedAt(path: string, mtime: number): Operation; + readFile(path: string): Operation; + /** One directory's entries, described without following a link. */ + list(path: string): Operation; + /** One path, described without following a link. */ + describe(path: string): Operation; +} + +/** Where one logical Workspace path sits on this host, for this invocation. */ +export type HostPath = (logical: string) => string; + +/** + * Materialize the exact root, one bounded piece at a time. + * + * Entries are created in canonical order, which is also parent-before-child + * order for everything but the depth ordering a restore needs — so directories + * are created as they are met and a file never arrives before the directory + * holding it. A hardlink group's first member is written and the rest are + * linked to it, which is what makes them one file again rather than copies. + * + * Times are set after the tree exists. Writing a file into a directory updates + * that directory's own time, so setting times as we went would leave every + * directory carrying the moment it was filled rather than the moment the root + * records. + */ +export function* materializeWorkspaceRoot( + files: RunnerFiles, + reads: RemoteReadLink, + at: HostPath, + workspaceRootId: string, + reject: WorkspaceRejection, +): Operation { + const manifest = yield* reads.root(workspaceRootId); + const written = new Map(); + const times: { path: string; mtime: number }[] = []; + + for (const entry of manifest.entries) { + const path = at(entry.path); + if (entry.kind === "directory") { + if (entry.path !== "/") { + yield* files.makeDirectory(path, entry.mode); + } + times.push({ path, mtime: entry.mtime }); + continue; + } + if (entry.kind === "symlink") { + // Created, never followed. A retained link may point anywhere, including + // outside the tree, and resolving one here would be this code deciding to + // read something the Workspace merely mentions. + yield* files.makeSymlink(entry.target, path); + continue; + } + + const first = written.get(entry.manifest); + if (entry.hardlink !== null && first !== undefined) { + yield* files.makeHardlink(first, path); + continue; + } + const bytes = yield* fetchFile(reads, workspaceRootId, entry.manifest, entry.size, reject); + yield* files.writeFile(path, bytes, entry.mode); + if (entry.hardlink !== null) { + written.set(entry.manifest, path); + } + times.push({ path, mtime: entry.mtime }); + } + + // Deepest first, so filling a directory cannot move a time already set. + for (const entry of times.toReversed()) { + yield* files.setModifiedAt(entry.path, entry.mtime); + } + return manifest; +} + +/** One file's bytes, assembled from the chunks its manifest names. */ +function* fetchFile( + reads: RemoteReadLink, + workspaceRootId: string, + manifestDigest: string, + size: number, + reject: WorkspaceRejection, +): Operation { + const encoded = yield* reads.content(workspaceRootId, { + kind: "manifest", + digest: manifestDigest, + }); + const manifest = decodeDofsManifest(encoded.bytes, reject); + if (manifest.size !== size) { + reject("a retained Workspace file size disagrees with the manifest it names"); + } + const bytes = new Uint8Array(size); + let offset = 0; + for (const chunk of manifest.chunks) { + const piece = yield* reads.content(workspaceRootId, { + kind: "blob", + digest: chunk.hash, + manifestDigest, + }); + if (piece.bytes.length !== chunk.size) { + reject("a retained content piece is not the size its manifest declares"); + } + bytes.set(piece.bytes, offset); + offset += piece.bytes.length; + } + if (offset !== size) { + reject("a retained Workspace file is not the size its entry declares"); + } + return bytes; +} + +/** What a capture produced, and the content it must be able to supply. */ +export interface CapturedWorkspace { + readonly root: CapturedRoot; + readonly contents: ReadonlyMap; + /** Every blob identity, with the bytes to send if the owner lacks it. */ + readonly blobs: ReadonlyMap; +} + +/** + * Read the tree back as the root it now describes. + * + * A walk, then the shared rules. Nothing here decides ordering, numbering or + * encoding — those belong to the capture rules both hosts share, so that this + * walk and the local provider's walk of its own tables cannot drift apart. + */ +export function* captureWorkspace( + files: RunnerFiles, + at: HostPath, + reject: WorkspaceRejection, +): Operation { + const nodes: CapturedNode[] = []; + const contents = new Map(); + const blobs = new Map(); + + function* visit(logical: string): Operation { + const found = yield* files.list(at(logical)); + for (const node of found.toSorted((left, right) => compareUtf8(left.name, right.name))) { + const path = logical === "/" ? `/${node.name}` : `${logical}/${node.name}`; + if (node.kind === "directory") { + nodes.push({ path, kind: "directory", mode: node.mode, mtime: node.mtime }); + yield* visit(path); + continue; + } + if (node.kind === "symlink") { + if (node.target === undefined) { + reject("a Workspace symbolic link has no target"); + } + nodes.push({ + path, + kind: "symlink", + mode: node.mode, + mtime: node.mtime, + target: node.target, + }); + continue; + } + const bytes = yield* files.readFile(at(path)); + if (bytes.length !== node.size) { + reject("a Workspace file changed size while it was being captured"); + } + const content = captureContent(bytes); + if (!contents.has(content.manifest)) { + contents.set(content.manifest, content); + let offset = 0; + for (const chunk of content.chunks) { + blobs.set(chunk.hash, bytes.slice(offset, offset + chunk.size)); + offset += chunk.size; + } + } + nodes.push({ + path, + kind: "file", + mode: node.mode, + mtime: node.mtime, + size: node.size, + manifest: content.manifest, + identity: node.identity, + }); + } + } + + // The root directory is part of the root's identity like any other entry, so + // its own mode and time are read rather than assumed. + const top = yield* files.describe(at("/")); + if (top.kind !== "directory") { + reject("a Workspace root is not a directory"); + } + nodes.push({ path: "/", kind: "directory", mode: top.mode, mtime: top.mtime }); + yield* visit("/"); + + return { root: captureWorkspaceRoot(nodes, contents, reject), contents, blobs }; +} + +/** Whether a captured root is the one it was materialized from. */ +export function unchangedFrom(captured: CapturedRoot, workspaceRootId: string): boolean { + return captured.rootId === workspaceRootId; +} + +/** The digest of a piece the runner is about to offer. */ +export function pieceDigest(bytes: Uint8Array): string { + return sha256Hex(bytes); +} diff --git a/packages/workflow/src/workspace/capture.ts b/packages/workflow/src/workspace/capture.ts new file mode 100644 index 000000000..a3e81b11a --- /dev/null +++ b/packages/workflow/src/workspace/capture.ts @@ -0,0 +1,218 @@ +/** + * Turning a tree of nodes into the canonical root that names it. + * + * Capture happens in two places that share nothing else. The local host walks + * the DOFS tables inside its SQLite file; a remote runner walks a real + * directory it materialized on disk. Neither walk is shareable — one reads rows + * and the other reads a filesystem — but what the walk *means* has to be + * identical, because the root identity is a digest of the encoding and two + * hosts that encoded differently would produce two roots for one Workspace. + * + * So the walk stays with whoever can perform it, and everything after the walk + * lives here: ordering, hardlink numbering, manifest encoding, chunk identity + * and the root digest. A caller hands over what it found and receives the root + * that describes it, or a refusal saying it does not describe one. + * + * The rule this exists to protect is narrow and worth stating plainly: an + * untouched materialization must capture back to the exact root it came from. + * If it did not, every no-op Workspace operation would propose a new root, and + * a run would appear to change its Workspace by looking at it. + */ + +import { + compareUtf8, + type DofsChunkReference, + type WorkspaceRejection, + type WorkspaceRootEntry, + validateWorkspaceRootEntries, + WORKSPACE_ROOT_DOMAIN, + WORKSPACE_ROOT_FORMAT, +} from "./root-manifest.ts"; +import { sha256Hex } from "./sha256.ts"; + +/** + * The size a file's bytes are split at. + * + * Pinned to what the vendored DOFS layer uses. A runner that chunked + * differently would compute different manifest identities for identical bytes, + * and the owner would then hold two names for one file. + */ +export const CHUNK_SIZE = 512 * 1024; + +/** One node a walk found, before anything is ordered or numbered. */ +export type CapturedNode = + | { + readonly path: string; + readonly kind: "directory"; + readonly mode: number; + readonly mtime: number; + } + | { + readonly path: string; + readonly kind: "symlink"; + readonly mode: number; + readonly mtime: number; + readonly target: string; + } + | { + readonly path: string; + readonly kind: "file"; + readonly mode: number; + readonly mtime: number; + readonly size: number; + /** The DOFS manifest identity of this file's bytes. */ + readonly manifest: string; + /** + * What makes two paths the same file rather than two copies. + * + * An inode on a real filesystem, an inode number in DOFS. Two entries + * sharing one are a hardlink group; `undefined` is a file reached by one + * path. Identical bytes are deliberately *not* enough — two independent + * files that happen to match are two files, and a capture that merged + * them would materialize back as something the run never had. + */ + readonly identity: string | undefined; + }; + +/** What one file's bytes are, once chunked. */ +export interface CapturedContent { + readonly manifest: string; + readonly manifestBytes: Uint8Array; + readonly chunks: readonly DofsChunkReference[]; +} + +/** The root one capture describes, and the content it closes over. */ +export interface CapturedRoot { + readonly rootId: string; + readonly manifest: string; + readonly entries: readonly WorkspaceRootEntry[]; + /** Every DOFS manifest identity this root names, in canonical order. */ + readonly manifests: readonly string[]; + /** Every blob identity those manifests name, in canonical order. */ + readonly blobs: readonly string[]; +} + +const encoder = new TextEncoder(); + +/** The bytes a DOFS manifest is stored and identified as. */ +export function encodeDofsManifest(chunks: readonly DofsChunkReference[]): Uint8Array { + return encoder.encode( + JSON.stringify({ + version: 1, + chunks: chunks.map((chunk) => ({ hash: chunk.hash, size: chunk.size })), + }), + ); +} + +/** + * Split one file's bytes the way the content store splits them. + * + * An empty file has no chunks, which is not the same as having one chunk of + * nothing: its manifest names zero bytes and is still a manifest, and every + * empty file in a Workspace shares it. + */ +export function captureContent(bytes: Uint8Array): CapturedContent { + const chunks: DofsChunkReference[] = []; + for (let offset = 0; offset < bytes.length; offset += CHUNK_SIZE) { + const slice = bytes.subarray(offset, Math.min(offset + CHUNK_SIZE, bytes.length)); + chunks.push({ hash: sha256Hex(slice), size: slice.length }); + } + const manifestBytes = encodeDofsManifest(chunks); + return { manifest: sha256Hex(manifestBytes), manifestBytes, chunks }; +} + +/** The identity a canonical root manifest has. */ +export function workspaceRootIdOf(manifest: string): string { + return sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${manifest}`); +} + +/** + * Order the nodes, number the hardlink groups, and encode the root. + * + * Ordering is by UTF-8 bytes because that is what the format declares, and + * hardlink groups are numbered by the byte order of their first path so that + * the same tree numbers the same way whoever walked it — a group numbered by + * discovery order would depend on the walk, and the two walks are different. + * + * The result is validated against the same entry rules a stored root is read + * back through. A capture that produced something the reader would refuse is a + * bug worth finding here rather than at the owner. + */ +export function captureWorkspaceRoot( + nodes: readonly CapturedNode[], + contents: ReadonlyMap, + reject: WorkspaceRejection, +): CapturedRoot { + const ordered = nodes.toSorted((left, right) => compareUtf8(left.path, right.path)); + + const shared = new Map(); + for (const node of ordered) { + if (node.kind === "file" && node.identity !== undefined) { + shared.set(node.identity, [...(shared.get(node.identity) ?? []), node.path]); + } + } + const group = new Map(); + const groups = [...shared.values()] + .filter((paths) => paths.length > 1) + .map((paths) => paths.toSorted(compareUtf8)) + .toSorted((left, right) => compareUtf8(left[0] ?? "", right[0] ?? "")); + for (const [index, paths] of groups.entries()) { + for (const path of paths) { + group.set(path, `h${index}`); + } + } + + const entries: WorkspaceRootEntry[] = ordered.map((node) => { + if (node.kind === "directory") { + return { path: node.path, kind: node.kind, mode: node.mode, mtime: node.mtime }; + } + if (node.kind === "symlink") { + return { + path: node.path, + kind: node.kind, + mode: node.mode, + mtime: node.mtime, + target: node.target, + }; + } + return { + path: node.path, + kind: node.kind, + mode: node.mode, + mtime: node.mtime, + size: node.size, + manifest: node.manifest, + hardlink: group.get(node.path) ?? null, + }; + }); + + validateWorkspaceRootEntries(entries, reject); + const manifest = JSON.stringify({ format: WORKSPACE_ROOT_FORMAT, entries }); + + const manifests = new Set(); + const blobs = new Set(); + for (const entry of entries) { + if (entry.kind !== "file") { + continue; + } + const content = contents.get(entry.manifest); + if (content === undefined) { + reject("a captured Workspace file names content the capture did not produce"); + } + if (entry.size !== content.chunks.reduce((total, chunk) => total + chunk.size, 0)) { + reject("a captured Workspace file size disagrees with its content"); + } + manifests.add(entry.manifest); + for (const chunk of content.chunks) { + blobs.add(chunk.hash); + } + } + + return { + rootId: workspaceRootIdOf(manifest), + manifest, + entries, + manifests: [...manifests].toSorted(compareUtf8), + blobs: [...blobs].toSorted(compareUtf8), + }; +} diff --git a/packages/workflow/tests/remote-materialization.test.ts b/packages/workflow/tests/remote-materialization.test.ts new file mode 100644 index 000000000..882595f84 --- /dev/null +++ b/packages/workflow/tests/remote-materialization.test.ts @@ -0,0 +1,194 @@ +/** + * Tier WRH — putting a retained root on a runner and reading it back. + * + * The claim under test is one equality: an untouched materialization captures + * to the exact root it was materialized from. Everything else in the remote + * provider rests on it. If it did not hold, a Workspace operation that changed + * nothing would still propose a new root, every no-op would look like a + * mutation, and the owner could not tell a real change from an artefact of how + * the runner unpacked the tree. + * + * A real temporary filesystem, deliberately. Modes, modification times, an + * empty file, a symbolic link and a hardlink group are properties of a + * filesystem, and a fake that stored them in a map would prove only that the + * map kept what it was given. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, type Operation, resource, scoped, until } from "effection"; +import { mkdir, mkdtemp, rm, symlink, link, utimes, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runnerFiles } from "../src/deno/remote-files.ts"; +import { + captureWorkspace, + materializeWorkspaceRoot, + type RunnerFiles, +} from "../src/remote/materialize.ts"; +import type { RemoteContent, RemoteContentRequest, RemoteReadLink } from "../src/remote/read.ts"; +import type { WorkspaceRootManifest } from "../src/workspace/root-manifest.ts"; +import { parseWorkspaceRootManifest } from "../src/workspace/root-manifest.ts"; +import { encodeDofsManifest } from "../src/workspace/capture.ts"; + +function reject(reason: string): never { + throw new Error(reason); +} + +/** + * A temporary directory owned by the scope that asked for it. + * + * A resource rather than a scoped operation: the tree has to outlive the call + * that created it and end with the invocation that owns it, which is the whole + * lifetime claim materialization makes. + */ +function useTemporaryDirectory(): Operation { + return resource(function* (provide) { + const path = yield* until(mkdtemp(join(tmpdir(), "xmd-materialize-"))); + yield* ensure(() => until(rm(path, { recursive: true, force: true }))); + yield* provide(path); + }); +} + +/** Where one logical Workspace path sits under `root`. */ +function at(root: string): (logical: string) => string { + return (logical) => (logical === "/" ? root : join(root, logical.slice(1))); +} + +/** + * An owner that serves exactly what a capture produced. + * + * It answers from the capture's own manifests and blobs, so what crosses is + * what the runner would have had to send. Nothing here validates: the point is + * that materialization rebuilds the tree, and the validation of pieces is the + * connection's, proved where the connection is. + */ +function servedBy(captured: { + root: { manifest: string; rootId: string }; + contents: ReadonlyMap; + blobs: ReadonlyMap; +}): RemoteReadLink { + return { + // deno-lint-ignore require-yield + *frontier(): Operation { + throw new Error("this owner serves only a root and its content"); + }, + // deno-lint-ignore require-yield + *root(workspaceRootId: string): Operation { + if (workspaceRootId !== captured.root.rootId) { + throw new Error("asked for a root this owner does not hold"); + } + return parseWorkspaceRootManifest(captured.root.manifest, reject); + }, + // deno-lint-ignore require-yield + *content(_rootId: string, request: RemoteContentRequest): Operation { + const bytes = + request.kind === "manifest" + ? captured.contents.get(request.digest)?.manifestBytes + : captured.blobs.get(request.digest); + if (bytes === undefined) { + throw new Error("asked for content this owner does not hold"); + } + return { kind: request.kind, digest: request.digest, bytes }; + }, + }; +} + +/** One tree with every entry kind the format carries. */ +function* buildTree(root: string): Operation { + yield* until(mkdir(join(root, "docs"), { mode: 0o755 })); + yield* until(mkdir(join(root, "docs", "deep"), { mode: 0o700 })); + yield* until(writeFile(join(root, "README.md"), "a workspace\n", { mode: 0o644 })); + yield* until(writeFile(join(root, "empty"), new Uint8Array(0), { mode: 0o600 })); + yield* until(writeFile(join(root, "docs", "guide.md"), "# guide\n", { mode: 0o644 })); + // Larger than one chunk, so the manifest names more than one piece. + yield* until( + writeFile(join(root, "docs", "deep", "large.bin"), new Uint8Array(700 * 1024).fill(7), { + mode: 0o644, + }), + ); + yield* until(symlink("../README.md", join(root, "docs", "link"))); + // Two names for one file: a hardlink group the capture must number. + yield* until(writeFile(join(root, "shared-a"), "shared bytes\n", { mode: 0o644 })); + yield* until(link(join(root, "shared-a"), join(root, "shared-b"))); + + for (const [path, mtime] of [ + [join(root, "README.md"), 1_700_000_001], + [join(root, "empty"), 1_700_000_002], + [join(root, "docs", "guide.md"), 1_700_000_003], + [join(root, "docs", "deep", "large.bin"), 1_700_000_004], + [join(root, "shared-a"), 1_700_000_005], + [join(root, "docs", "deep"), 1_700_000_006], + [join(root, "docs"), 1_700_000_007], + [root, 1_700_000_008], + ] as const) { + yield* until(utimes(path, mtime, mtime)); + } +} + +describe("materializing a retained Workspace root", () => { + it("captures an untouched materialization back to the exact root it came from", function* () { + const files: RunnerFiles = runnerFiles(); + const source = yield* useTemporaryDirectory(); + yield* buildTree(source); + + const captured = yield* captureWorkspace(files, at(source), reject); + const entries = captured.root.entries; + // The tree really does exercise what the format carries. + expect(entries.filter((entry) => entry.kind === "directory")).toHaveLength(3); + expect(entries.filter((entry) => entry.kind === "symlink")).toHaveLength(1); + expect( + entries.filter((entry) => entry.kind === "file" && entry.hardlink !== null), + ).toHaveLength(2); + expect(entries.some((entry) => entry.kind === "file" && entry.size === 0)).toBe(true); + // 700 KiB is two chunks at the pinned chunk size, so a file crosses the + // transport as more than one piece. + const large = entries.find((entry) => entry.path === "/docs/deep/large.bin"); + if (large?.kind !== "file") { + throw new Error("expected the large file to be captured as a file"); + } + expect(captured.contents.get(large.manifest)?.chunks).toHaveLength(2); + + const destination = yield* useTemporaryDirectory(); + yield* materializeWorkspaceRoot( + files, + servedBy(captured), + at(destination), + captured.root.rootId, + reject, + ); + + const again = yield* captureWorkspace(files, at(destination), reject); + expect(again.root.rootId).toBe(captured.root.rootId); + expect(again.root.manifest).toBe(captured.root.manifest); + expect([...again.root.manifests]).toEqual([...captured.root.manifests]); + expect([...again.root.blobs]).toEqual([...captured.root.blobs]); + }); + + it("encodes a DOFS manifest the way the content store stores one", function* () { + // The runner and the owner must name identical bytes identically, and the + // encoding is what decides that. + expect(new TextDecoder().decode(encodeDofsManifest([{ hash: "a".repeat(64), size: 3 }]))).toBe( + `{"version":1,"chunks":[{"hash":"${"a".repeat(64)}","size":3}]}`, + ); + expect(new TextDecoder().decode(encodeDofsManifest([]))).toBe('{"version":1,"chunks":[]}'); + }); + + it("removes the materialization when its scope ends, however it ends", function* () { + const files: RunnerFiles = runnerFiles(); + let path = ""; + yield* scoped(function* () { + path = yield* useTemporaryDirectory(); + yield* until(writeFile(join(path, "present"), "here\n")); + }); + // The scope that owned it has ended, so the tree is gone rather than left + // behind for a later invocation to find. + let listed: unknown; + try { + listed = yield* files.list(path); + } catch (error) { + listed = error; + } + expect(listed).toBeInstanceOf(Error); + }); +}); From 6e4596d446b42849030f837ea09b06174abf03ba Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 18:19:32 -0400 Subject: [PATCH 28/81] =?UTF-8?q?=F0=9F=94=97=20Keep=20two=20hardlink=20gr?= =?UTF-8?q?oups=20two=20files=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Materialization indexed a hardlink group's first path by the content digest of its bytes. Two groups holding identical bytes legally share one DOFS manifest and are still two inodes, so the second group linked to the first and recapture saw one group of four names where the root said two of two. The Workspace that came back was a different Workspace, arriving under the identity of the one that was asked for. Group membership comes from the root's own `hardlink` value now, which is the only thing that ever said what a group was. Modes and times were left to whatever creation happened to produce. A creation mode is narrowed by the process umask, so a root retaining a group-writable file materialized without that bit; a symbolic link's own mode and time were never restored at all, and `utimes` could not have done it without following the link — which may point outside the tree deliberately. Permissions are now set explicitly after creation, deepest-first so a mode that forbids writing is not applied while children are still arriving, and a link's own metadata is set through the operations that do not follow it, where the platform has them. Where a platform has none, materialization refuses. Every entry is read back and compared with what the root declared before anything executes against the tree: a host that cannot represent a legal retained mode or time says so, once, before native work begins. Quietly normalizing it would hand the run a Workspace whose durable identity differs from the history it accepted, and the run would have no way to notice. The round trip now carries two hardlink groups with identical bytes, two independent files with identical bytes that must stay independent, modes the test's own umask would narrow, and a symbolic link with an old time of its own. Reverting the grouping key alone fails it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/deno/remote-files.ts | 47 ++++++++ packages/workflow/src/remote/materialize.ts | 104 ++++++++++++++++-- .../tests/remote-materialization.test.ts | 69 +++++++++++- 3 files changed, 205 insertions(+), 15 deletions(-) diff --git a/packages/workflow/src/deno/remote-files.ts b/packages/workflow/src/deno/remote-files.ts index 6e18ec6c3..fa482d45b 100644 --- a/packages/workflow/src/deno/remote-files.ts +++ b/packages/workflow/src/deno/remote-files.ts @@ -16,8 +16,11 @@ */ import { + chmod, link, lstat, + lchmod, + lutimes, mkdir, readdir, readFile, @@ -65,6 +68,22 @@ function describeStats( }; } +/** + * `lchmod`, when the platform actually has it. + * + * BSD-derived systems do; Linux does not, and Node exposes the export + * regardless on some releases. Probing the export is the only honest test + * available before a real call. + */ +function lchmodOf(): ((path: string, mode: number) => Operation) | undefined { + if (typeof lchmod !== "function") { + return undefined; + } + return function* (path: string, mode: number): Operation { + yield* until(lchmod(path, mode)); + }; +} + /** The runner's filesystem operations, for one materialized tree. */ export function runnerFiles(): RunnerFiles { return { @@ -84,10 +103,38 @@ export function runnerFiles(): RunnerFiles { yield* until(link(existing, path)); }, + *setMode(path: string, mode: number): Operation { + // Explicit rather than relying on the creation mode, which the process + // umask narrows. A retained mode is durable identity. + yield* until(chmod(path, mode)); + }, + *setModifiedAt(path: string, mtime: number): Operation { yield* until(utimes(path, mtime, mtime)); }, + /** + * A link's own time, set without following it. + * + * `lutimes` is what makes this possible at all: `utimes` would follow the + * link and rewrite whatever it points at, which may be outside the tree + * entirely. + */ + *setLinkModifiedAt(path: string, mtime: number): Operation { + yield* until(lutimes(path, mtime, mtime)); + }, + + /** + * A link's own permissions, where the platform has them. + * + * Linux ignores symbolic-link permission bits and offers no `lchmod`, so + * this is deliberately absent there rather than faked. Materialization + * checks what it actually got and refuses a root this host cannot + * represent, which is the honest outcome; quietly writing a different mode + * would change durable identity. + */ + setLinkMode: lchmodOf(), + *readFile(path: string): Operation { return new Uint8Array(yield* until(readFile(path))); }, diff --git a/packages/workflow/src/remote/materialize.ts b/packages/workflow/src/remote/materialize.ts index 8968b6042..513971e3d 100644 --- a/packages/workflow/src/remote/materialize.ts +++ b/packages/workflow/src/remote/materialize.ts @@ -70,8 +70,27 @@ export interface RunnerFiles { writeFile(path: string, bytes: Uint8Array, mode: number): Operation; makeSymlink(target: string, path: string): Operation; makeHardlink(existing: string, path: string): Operation; + /** + * Set permissions exactly, after creation. + * + * Creation modes are narrowed by the process umask, and a retained mode is + * durable identity rather than a preference. Applied to a directory only once + * its children exist, because a mode that forbids writing would otherwise + * forbid filling it. + */ + setMode(path: string, mode: number): Operation; /** Applied last, because writing into a directory moves its own time. */ setModifiedAt(path: string, mtime: number): Operation; + /** + * Set a link's own time without following it. + * + * Separate because a link's target may not exist, may be outside the tree, or + * may be something this code must never touch. `undefined` when the host + * cannot do it at all, which materialization reports rather than works around. + */ + readonly setLinkModifiedAt: ((path: string, mtime: number) => Operation) | undefined; + /** The same, for a link's own permissions. `undefined` where unsupported. */ + readonly setLinkMode: ((path: string, mode: number) => Operation) | undefined; readFile(path: string): Operation; /** One directory's entries, described without following a link. */ list(path: string): Operation; @@ -104,8 +123,17 @@ export function* materializeWorkspaceRoot( reject: WorkspaceRejection, ): Operation { const manifest = yield* reads.root(workspaceRootId); - const written = new Map(); - const times: { path: string; mtime: number }[] = []; + /** + * The first path written for each hardlink group. + * + * Keyed by the group the root declares, never by the content digest. Two + * groups may legally hold identical bytes and therefore share one manifest, + * and linking the second to the first would merge two files into one — a + * different Workspace, arriving under the identity of this one. + */ + const groups = new Map(); + const modes: { path: string; mode: number; link: boolean }[] = []; + const times: { path: string; mtime: number; link: boolean }[] = []; for (const entry of manifest.entries) { const path = at(entry.path); @@ -113,7 +141,8 @@ export function* materializeWorkspaceRoot( if (entry.path !== "/") { yield* files.makeDirectory(path, entry.mode); } - times.push({ path, mtime: entry.mtime }); + modes.push({ path, mode: entry.mode, link: false }); + times.push({ path, mtime: entry.mtime, link: false }); continue; } if (entry.kind === "symlink") { @@ -121,29 +150,86 @@ export function* materializeWorkspaceRoot( // outside the tree, and resolving one here would be this code deciding to // read something the Workspace merely mentions. yield* files.makeSymlink(entry.target, path); + modes.push({ path, mode: entry.mode, link: true }); + times.push({ path, mtime: entry.mtime, link: true }); continue; } - const first = written.get(entry.manifest); - if (entry.hardlink !== null && first !== undefined) { + const first = entry.hardlink === null ? undefined : groups.get(entry.hardlink); + if (first !== undefined) { + // One inode reached by a second name. Its mode and time belong to the + // file, which the first member already carries. yield* files.makeHardlink(first, path); continue; } const bytes = yield* fetchFile(reads, workspaceRootId, entry.manifest, entry.size, reject); yield* files.writeFile(path, bytes, entry.mode); if (entry.hardlink !== null) { - written.set(entry.manifest, path); + groups.set(entry.hardlink, path); } - times.push({ path, mtime: entry.mtime }); + modes.push({ path, mode: entry.mode, link: false }); + times.push({ path, mtime: entry.mtime, link: false }); } - // Deepest first, so filling a directory cannot move a time already set. + // Modes before times and both deepest-first: a mode that forbids writing must + // not be applied while children are still arriving, and filling a directory + // moves a time that was already restored. + for (const entry of modes.toReversed()) { + if (!entry.link) { + yield* files.setMode(entry.path, entry.mode); + continue; + } + if (files.setLinkMode !== undefined) { + yield* files.setLinkMode(entry.path, entry.mode); + } + } for (const entry of times.toReversed()) { - yield* files.setModifiedAt(entry.path, entry.mtime); + if (!entry.link) { + yield* files.setModifiedAt(entry.path, entry.mtime); + continue; + } + if (files.setLinkModifiedAt !== undefined) { + yield* files.setLinkModifiedAt(entry.path, entry.mtime); + } } + + // Proved rather than assumed. A host that cannot represent a legal retained + // mode or time must say so here, before anything executes against this tree — + // silently normalizing one would hand the run a Workspace with a different + // durable identity than the history it accepted. + yield* requireExactMaterialization(files, at, manifest, reject); return manifest; } +/** + * Whether what is on disk is what the root said. + * + * Every entry's kind, mode and time, read back without following a link. This + * is not defensive duplication: umask, platform link semantics and filesystem + * timestamp granularity are all real, and each of them turns one retained root + * into a different one quietly. The refusal names what disagreed, not where the + * tree happens to live. + */ +function* requireExactMaterialization( + files: RunnerFiles, + at: HostPath, + manifest: WorkspaceRootManifest, + reject: WorkspaceRejection, +): Operation { + for (const entry of manifest.entries) { + const found = yield* files.describe(at(entry.path)); + if (found.kind !== entry.kind) { + reject(`this host materialized a ${entry.kind} as a ${found.kind}`); + } + if (found.mode !== entry.mode) { + reject(`this host cannot preserve the retained mode of a ${entry.kind}`); + } + if (found.mtime !== entry.mtime) { + reject(`this host cannot preserve the retained modification time of a ${entry.kind}`); + } + } +} + /** One file's bytes, assembled from the chunks its manifest names. */ function* fetchFile( reads: RemoteReadLink, diff --git a/packages/workflow/tests/remote-materialization.test.ts b/packages/workflow/tests/remote-materialization.test.ts index 882595f84..b9e9bf865 100644 --- a/packages/workflow/tests/remote-materialization.test.ts +++ b/packages/workflow/tests/remote-materialization.test.ts @@ -17,9 +17,20 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { ensure, type Operation, resource, scoped, until } from "effection"; -import { mkdir, mkdtemp, rm, symlink, link, utimes, writeFile } from "node:fs/promises"; +import { + chmod, + link, + lutimes, + mkdir, + mkdtemp, + rm, + symlink, + utimes, + writeFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import process from "node:process"; import { runnerFiles } from "../src/deno/remote-files.ts"; import { captureWorkspace, @@ -108,9 +119,27 @@ function* buildTree(root: string): Operation { }), ); yield* until(symlink("../README.md", join(root, "docs", "link"))); - // Two names for one file: a hardlink group the capture must number. + + // Two hardlink groups holding *identical* bytes. They share one DOFS + // manifest and are still two files, so a materializer that indexed by + // content would link the second group to the first and merge them. yield* until(writeFile(join(root, "shared-a"), "shared bytes\n", { mode: 0o644 })); yield* until(link(join(root, "shared-a"), join(root, "shared-b"))); + yield* until(writeFile(join(root, "other-a"), "shared bytes\n", { mode: 0o644 })); + yield* until(link(join(root, "other-a"), join(root, "other-b"))); + + // And two independent files with the same bytes, which must stay two files + // with no hardlink group at all. + yield* until(writeFile(join(root, "loose-a"), "loose bytes\n", { mode: 0o644 })); + yield* until(writeFile(join(root, "loose-b"), "loose bytes\n", { mode: 0o644 })); + + // Modes the usual 0022 umask narrows at creation, set explicitly so the + // retained root genuinely carries them. Materialization then has to restore + // them under the same umask, which is only possible by setting them. + yield* until(writeFile(join(root, "group-writable"), "wide\n")); + yield* until(chmod(join(root, "group-writable"), 0o666)); + yield* until(mkdir(join(root, "wide-dir"))); + yield* until(chmod(join(root, "wide-dir"), 0o777)); for (const [path, mtime] of [ [join(root, "README.md"), 1_700_000_001], @@ -118,16 +147,26 @@ function* buildTree(root: string): Operation { [join(root, "docs", "guide.md"), 1_700_000_003], [join(root, "docs", "deep", "large.bin"), 1_700_000_004], [join(root, "shared-a"), 1_700_000_005], + [join(root, "other-a"), 1_700_000_009], + [join(root, "loose-a"), 1_700_000_010], + [join(root, "loose-b"), 1_700_000_011], + [join(root, "group-writable"), 1_700_000_012], + [join(root, "wide-dir"), 1_700_000_013], [join(root, "docs", "deep"), 1_700_000_006], [join(root, "docs"), 1_700_000_007], [root, 1_700_000_008], ] as const) { yield* until(utimes(path, mtime, mtime)); } + // The link's own time, well in the past and set without following it. + yield* until(lutimes(join(root, "docs", "link"), 1_600_000_000, 1_600_000_000)); } describe("materializing a retained Workspace root", () => { it("captures an untouched materialization back to the exact root it came from", function* () { + // A umask that would narrow a created mode, so the restoration has to be + // explicit rather than incidental. + const previous = process.umask(0o022); const files: RunnerFiles = runnerFiles(); const source = yield* useTemporaryDirectory(); yield* buildTree(source); @@ -135,11 +174,28 @@ describe("materializing a retained Workspace root", () => { const captured = yield* captureWorkspace(files, at(source), reject); const entries = captured.root.entries; // The tree really does exercise what the format carries. - expect(entries.filter((entry) => entry.kind === "directory")).toHaveLength(3); + expect(entries.filter((entry) => entry.kind === "directory")).toHaveLength(4); expect(entries.filter((entry) => entry.kind === "symlink")).toHaveLength(1); - expect( - entries.filter((entry) => entry.kind === "file" && entry.hardlink !== null), - ).toHaveLength(2); + // Two groups of two, holding identical bytes and still two groups. + const linked = entries.filter((entry) => entry.kind === "file" && entry.hardlink !== null); + expect(linked).toHaveLength(4); + expect(new Set(linked.map((entry) => (entry.kind === "file" ? entry.hardlink : "")))).toEqual( + new Set(["h0", "h1"]), + ); + // And they share one manifest, which is what makes this discriminating: + // a materializer indexing by content would merge them. + expect(new Set(linked.map((entry) => (entry.kind === "file" ? entry.manifest : ""))).size).toBe( + 1, + ); + // Equal bytes did not make the independent pair into a group. + for (const path of ["/loose-a", "/loose-b"]) { + const loose = entries.find((entry) => entry.path === path); + expect(loose?.kind === "file" && loose.hardlink).toBe(null); + } + // The wide modes survived the umask rather than being narrowed by it. + expect(entries.find((entry) => entry.path === "/group-writable")?.mode).toBe(0o666); + expect(entries.find((entry) => entry.path === "/wide-dir")?.mode).toBe(0o777); + expect(entries.find((entry) => entry.path === "/docs/link")?.mtime).toBe(1_600_000_000); expect(entries.some((entry) => entry.kind === "file" && entry.size === 0)).toBe(true); // 700 KiB is two chunks at the pinned chunk size, so a file crosses the // transport as more than one piece. @@ -159,6 +215,7 @@ describe("materializing a retained Workspace root", () => { ); const again = yield* captureWorkspace(files, at(destination), reject); + process.umask(previous); expect(again.root.rootId).toBe(captured.root.rootId); expect(again.root.manifest).toBe(captured.root.manifest); expect([...again.root.manifests]).toEqual([...captured.root.manifests]); From eb4fb815480881c925999df5edb3fa190652f85b Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 18:27:26 -0400 Subject: [PATCH 29/81] =?UTF-8?q?=F0=9F=A7=AD=20Keep=20the=20content-store?= =?UTF-8?q?=20format=20out=20of=20the=20Workspace=20surface=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shared coordination boundary is scanned for host vocabulary, and the root manifest module had been failing that scan since D2: its refusals named DOFS in string literals, which the scan reads as code because they are. D2's evidence list did not include the scan, so nothing said so until this checkpoint's wider verification ran it. The names were a symptom of the wrong seam. A Workspace root names a file's content by one identity and says nothing about how those bytes are kept; how they are kept is a separate format, and it was sitting inside the module that describes roots. It now has its own, which decides only whether a sequence of bytes is a canonically encoded manifest and produces the bytes one ought to be. Its refusals describe the format rather than the store implementing it. That is not a rename to satisfy a scan: every host keeps content this way, the vendored layer is one implementation, and a neutral module naming that implementation would be the Workspace surface learning where it happened to be stored. No message this reworded is asserted anywhere, so nothing observable moved. The obvious home — beside the schema that declares the content tables — is closed. A neutral module importing from there names the storage engine in its own import specifier, which is the same crossing by a different route. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 4 +- .../workflow/src/cloudflare/owner-reads.ts | 13 +- .../workflow/src/deno/artifact/records.ts | 14 +- .../workflow/src/deno/workspace/restore.ts | 4 +- packages/workflow/src/deno/workspace/root.ts | 31 +++-- packages/workflow/src/remote/materialize.ts | 4 +- packages/workflow/src/workspace/capture.ts | 35 ++--- .../src/workspace/content-manifest.ts | 129 ++++++++++++++++++ .../workflow/src/workspace/root-manifest.ts | 69 ---------- .../tests/remote-materialization.test.ts | 12 +- 10 files changed, 181 insertions(+), 134 deletions(-) create mode 100644 packages/workflow/src/workspace/content-manifest.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 2cf8ab4e0..f443147ef 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -46,12 +46,12 @@ import { startingFrontier, } from "../remote/read.ts"; import { - decodeDofsManifest, parseWorkspaceRootManifest, SHA256, WORKSPACE_ROOT_DOMAIN, type WorkspaceRootManifest, } from "../workspace/root-manifest.ts"; +import { decodeContentManifest } from "../workspace/content-manifest.ts"; import { JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES } from "./commands.ts"; import { decodeBase64, encodeBase64, sha256Hex } from "./encoding.ts"; @@ -266,7 +266,7 @@ function parseContent(value: unknown): RemoteContent { return fail("the content disagreed with its identity or size"); } if (kind === "manifest") { - decodeDofsManifest(bytes, fail); + decodeContentManifest(bytes, fail); } return { kind, digest, bytes }; } diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index 2b1467acf..1b18d5eab 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -29,13 +29,12 @@ import { parseDurableEvent } from "@executablemd/durable-streams"; import { readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { - decodeDofsManifest, - type DofsManifest, parseWorkspaceRootManifest, SHA256, WORKSPACE_ROOT_DOMAIN, type WorkspaceRootManifest, } from "../workspace/root-manifest.ts"; +import { type ContentManifest, decodeContentManifest } from "../workspace/content-manifest.ts"; import { CommandError, JOURNAL_PAGE_BYTES, @@ -86,7 +85,7 @@ export interface ContentValue { interface StoredRoot { readonly manifest: string; readonly parsed: WorkspaceRootManifest; - readonly manifests: ReadonlyMap; + readonly manifests: ReadonlyMap; readonly blobs: ReadonlySet; } @@ -162,7 +161,7 @@ function validatedManifest( storage: OwnerStorage, parsed: WorkspaceRootManifest, digest: string, -): { bytes: Uint8Array; manifest: DofsManifest } { +): { bytes: Uint8Array; manifest: ContentManifest } { const row = exactlyOne( byteRows(storage, "SELECT size, encoded FROM vfs_manifests WHERE lower(hex(hash)) = ?", digest), "DOFS manifest", @@ -171,7 +170,7 @@ function validatedManifest( if (bytes.length > MAX_CONTENT_BYTES || sha256Hex(bytes) !== digest) { return corrupt("a retained DOFS manifest disagrees with its identity"); } - const manifest = decodeDofsManifest(bytes, corrupt); + const manifest = decodeContentManifest(bytes, corrupt); if (safeInteger(row["size"], "manifest size") !== manifest.size) { return corrupt("a retained DOFS manifest disagrees with its recorded size"); } @@ -187,7 +186,7 @@ function validatedManifest( function validatedBlob( storage: OwnerStorage, rootId: string, - manifests: ReadonlyMap, + manifests: ReadonlyMap, digest: string, ): Uint8Array { const row = exactlyOne( @@ -274,7 +273,7 @@ function referencedRoot(storage: OwnerStorage, rootId: string): StoredRoot { "manifest", ); - const manifests = new Map(); + const manifests = new Map(); for (const digest of named) { manifests.set(digest, validatedManifest(storage, parsed, digest).manifest); } diff --git a/packages/workflow/src/deno/artifact/records.ts b/packages/workflow/src/deno/artifact/records.ts index 08cf5b18c..4752e1620 100644 --- a/packages/workflow/src/deno/artifact/records.ts +++ b/packages/workflow/src/deno/artifact/records.ts @@ -86,8 +86,8 @@ import { workspaceRoot, type WorkspaceRootManifest, } from "../workspace/manifest.ts"; -import { decodeDofsManifest } from "../workspace/root.ts"; -import type { DofsManifest } from "../workspace/root.ts"; +import { decodeContentManifest } from "../workspace/root.ts"; +import type { ContentManifest } from "../workspace/root.ts"; import { gitBlobIdentity } from "./source.ts"; import { canonicalJsonBytes, canonicalJsonText, entryKey } from "./manifest.ts"; import type { @@ -1335,7 +1335,7 @@ function verifyLifecycle( function verifyContentStore( contents: XmdArtifactContents, reject: Reject, -): ReadonlyMap { +): ReadonlyMap { const blobs = new Map(); for (const blob of contents.blobs) { const hash = toHex(blob.hash); @@ -1348,7 +1348,7 @@ function verifyContentStore( blobs.set(hash, blob.size); } - const manifests = new Map(); + const manifests = new Map(); for (const manifest of contents.manifests) { const hash = toHex(manifest.hash); if (manifests.has(hash)) { @@ -1357,7 +1357,7 @@ function verifyContentStore( if (toHex(sha256(manifest.encoded)) !== hash) { reject("a DOFS manifest's identity does not match its bytes"); } - const decoded = decodeDofsManifest(manifest.encoded, reject); + const decoded = decodeContentManifest(manifest.encoded, reject); if (decoded.size !== manifest.size) { reject("a DOFS manifest's declared size does not equal its chunks"); } @@ -1382,7 +1382,7 @@ function verifyContentStore( */ function verifyRoots( contents: XmdArtifactContents, - manifests: ReadonlyMap, + manifests: ReadonlyMap, path: string, reject: Reject, ): void { @@ -1395,7 +1395,7 @@ function verifyRoots( reject("a Workspace root identity does not match its manifest bytes"); } - const declared = new Map(); + const declared = new Map(); for (const entry of parsed.entries) { if (entry.kind !== "file") { continue; diff --git a/packages/workflow/src/deno/workspace/restore.ts b/packages/workflow/src/deno/workspace/restore.ts index 375994ef9..524f16592 100644 --- a/packages/workflow/src/deno/workspace/restore.ts +++ b/packages/workflow/src/deno/workspace/restore.ts @@ -13,7 +13,7 @@ import { } from "./manifest.ts"; import { loadWorkspaceRoot, - readDofsManifest, + readContentManifest, setCurrentWorkspaceRoot, snapshotWorkspace, verifyWorkspace, @@ -131,7 +131,7 @@ function materializeNode( .run(entry.mode, entry.mtime, revision, entry.target); inode = Number(result.lastInsertRowid); } else { - const manifest = readDofsManifest(database, entry.manifest, databasePath); + const manifest = readContentManifest(database, entry.manifest, databasePath); if (manifest.size !== entry.size) { corrupt(databasePath, "a retained file size differs from its DOFS manifest"); } diff --git a/packages/workflow/src/deno/workspace/root.ts b/packages/workflow/src/deno/workspace/root.ts index a6d211b9b..de87da60c 100644 --- a/packages/workflow/src/deno/workspace/root.ts +++ b/packages/workflow/src/deno/workspace/root.ts @@ -24,18 +24,18 @@ import { workspaceRoot, WORKSPACE_ROOT_FORMAT, } from "./manifest.ts"; +import { SHA256 } from "../../workspace/root-manifest.ts"; import { - decodeDofsManifest as decodeSharedDofsManifest, - type DofsManifest, - SHA256, -} from "../../workspace/root-manifest.ts"; + type ContentManifest, + decodeContentManifest as decodeSharedContentManifest, +} from "../../workspace/content-manifest.ts"; export interface DofsChunk { readonly hash: Uint8Array; readonly size: number; } -export type { DofsManifest } from "../../workspace/root-manifest.ts"; +export type { ContentManifest } from "../../workspace/content-manifest.ts"; interface NodeRow { readonly inode: number; @@ -412,11 +412,11 @@ export function verifyWorkspace( } } -export function readDofsManifest( +export function readContentManifest( database: DatabaseSync, hash: string, databasePath: string, -): DofsManifest { +): ContentManifest { const hashBytes = fromHex(hash, databasePath, "DOFS manifest identity"); const row = reading( database, @@ -434,7 +434,7 @@ export function readDofsManifest( if (toHex(sha256(encoded)) !== hash) { corrupt(databasePath, "a DOFS manifest hash does not match its bytes"); } - const decoded = decodeDofsManifest(encoded, (reason) => corrupt(databasePath, reason)); + const decoded = decodeContentManifest(encoded, (reason) => corrupt(databasePath, reason)); if (decoded.size !== size) { corrupt(databasePath, "a DOFS manifest size does not equal its chunks"); } @@ -454,8 +454,11 @@ export function readDofsManifest( * whether these bytes are a canonically encoded DOFS manifest at all, and what * size the chunks it lists add up to. */ -export function decodeDofsManifest(encoded: Uint8Array, reject: WorkspaceRejection): DofsManifest { - return decodeSharedDofsManifest(encoded, reject); +export function decodeContentManifest( + encoded: Uint8Array, + reject: WorkspaceRejection, +): ContentManifest { + return decodeSharedContentManifest(encoded, reject); } function parseStoredRoot( @@ -488,12 +491,12 @@ function rootFromManifest( parsed: ReturnType, databasePath: string, ): StoredWorkspaceRoot { - const manifests = new Map(); + const manifests = new Map(); for (const entry of parsed.entries) { if (entry.kind === "file") { let manifest = manifests.get(entry.manifest); if (manifest === undefined) { - manifest = readDofsManifest(database, entry.manifest, databasePath); + manifest = readContentManifest(database, entry.manifest, databasePath); manifests.set(entry.manifest, manifest); } if (entry.size !== manifest.size) { @@ -547,7 +550,7 @@ function validateFile( corrupt(databasePath, "a Workspace file has an invalid DOFS manifest identity"); } const manifest = toHex(manifestHash); - const encoded = readDofsManifest(database, manifest, databasePath); + const encoded = readContentManifest(database, manifest, databasePath); if ( encoded.size !== node.size || !equalChunks( @@ -587,7 +590,7 @@ function validateDofsContentStore(database: DatabaseSync, databasePath: string): if (hash.byteLength !== 32) { corrupt(databasePath, "a DOFS manifest has an invalid hash length"); } - readDofsManifest(database, toHex(hash), databasePath); + readContentManifest(database, toHex(hash), databasePath); } } diff --git a/packages/workflow/src/remote/materialize.ts b/packages/workflow/src/remote/materialize.ts index 513971e3d..114bb00ed 100644 --- a/packages/workflow/src/remote/materialize.ts +++ b/packages/workflow/src/remote/materialize.ts @@ -33,10 +33,10 @@ import { } from "../workspace/capture.ts"; import { compareUtf8, - decodeDofsManifest, type WorkspaceRejection, type WorkspaceRootManifest, } from "../workspace/root-manifest.ts"; +import { decodeContentManifest } from "../workspace/content-manifest.ts"; import { sha256Hex } from "../workspace/sha256.ts"; import type { RemoteReadLink } from "./read.ts"; @@ -242,7 +242,7 @@ function* fetchFile( kind: "manifest", digest: manifestDigest, }); - const manifest = decodeDofsManifest(encoded.bytes, reject); + const manifest = decodeContentManifest(encoded.bytes, reject); if (manifest.size !== size) { reject("a retained Workspace file size disagrees with the manifest it names"); } diff --git a/packages/workflow/src/workspace/capture.ts b/packages/workflow/src/workspace/capture.ts index a3e81b11a..a49b0b21f 100644 --- a/packages/workflow/src/workspace/capture.ts +++ b/packages/workflow/src/workspace/capture.ts @@ -21,24 +21,19 @@ import { compareUtf8, - type DofsChunkReference, type WorkspaceRejection, type WorkspaceRootEntry, validateWorkspaceRootEntries, WORKSPACE_ROOT_DOMAIN, WORKSPACE_ROOT_FORMAT, } from "./root-manifest.ts"; +import { + CHUNK_SIZE, + type ContentChunkReference, + encodeContentManifest, +} from "./content-manifest.ts"; import { sha256Hex } from "./sha256.ts"; -/** - * The size a file's bytes are split at. - * - * Pinned to what the vendored DOFS layer uses. A runner that chunked - * differently would compute different manifest identities for identical bytes, - * and the owner would then hold two names for one file. - */ -export const CHUNK_SIZE = 512 * 1024; - /** One node a walk found, before anything is ordered or numbered. */ export type CapturedNode = | { @@ -60,7 +55,7 @@ export type CapturedNode = readonly mode: number; readonly mtime: number; readonly size: number; - /** The DOFS manifest identity of this file's bytes. */ + /** The content manifest identity of this file's bytes. */ readonly manifest: string; /** * What makes two paths the same file rather than two copies. @@ -78,7 +73,7 @@ export type CapturedNode = export interface CapturedContent { readonly manifest: string; readonly manifestBytes: Uint8Array; - readonly chunks: readonly DofsChunkReference[]; + readonly chunks: readonly ContentChunkReference[]; } /** The root one capture describes, and the content it closes over. */ @@ -86,7 +81,7 @@ export interface CapturedRoot { readonly rootId: string; readonly manifest: string; readonly entries: readonly WorkspaceRootEntry[]; - /** Every DOFS manifest identity this root names, in canonical order. */ + /** Every content manifest identity this root names, in canonical order. */ readonly manifests: readonly string[]; /** Every blob identity those manifests name, in canonical order. */ readonly blobs: readonly string[]; @@ -94,16 +89,6 @@ export interface CapturedRoot { const encoder = new TextEncoder(); -/** The bytes a DOFS manifest is stored and identified as. */ -export function encodeDofsManifest(chunks: readonly DofsChunkReference[]): Uint8Array { - return encoder.encode( - JSON.stringify({ - version: 1, - chunks: chunks.map((chunk) => ({ hash: chunk.hash, size: chunk.size })), - }), - ); -} - /** * Split one file's bytes the way the content store splits them. * @@ -112,12 +97,12 @@ export function encodeDofsManifest(chunks: readonly DofsChunkReference[]): Uint8 * empty file in a Workspace shares it. */ export function captureContent(bytes: Uint8Array): CapturedContent { - const chunks: DofsChunkReference[] = []; + const chunks: ContentChunkReference[] = []; for (let offset = 0; offset < bytes.length; offset += CHUNK_SIZE) { const slice = bytes.subarray(offset, Math.min(offset + CHUNK_SIZE, bytes.length)); chunks.push({ hash: sha256Hex(slice), size: slice.length }); } - const manifestBytes = encodeDofsManifest(chunks); + const manifestBytes = encodeContentManifest(chunks); return { manifest: sha256Hex(manifestBytes), manifestBytes, chunks }; } diff --git a/packages/workflow/src/workspace/content-manifest.ts b/packages/workflow/src/workspace/content-manifest.ts new file mode 100644 index 000000000..82c04b836 --- /dev/null +++ b/packages/workflow/src/workspace/content-manifest.ts @@ -0,0 +1,129 @@ +/** + * How a file's bytes are described, once they are in the content store. + * + * A Workspace root names a file's content by one identity; it says nothing + * about how those bytes are kept. That is this format's job: an ordered list of + * chunks, each named by its own digest, encoded canonically so that identical + * bytes always produce one identity. + * + * Every host keeps content this way, so the rules are shared and name no host. + * Which store implements them, and in what tables, is the storage adapter's + * business and stays there — a neutral module that named one would be the + * Workspace surface learning where it happened to be kept. + * + * Nothing here opens a store, hashes anything or names a runtime. It decides + * whether a sequence of bytes is a canonically encoded manifest, and produces + * the bytes one ought to be. + */ + +import { SHA256, type WorkspaceRejection } from "./root-manifest.ts"; + +/** + * The size a file's bytes are split at. + * + * Pinned to what the vendored content layer uses. A writer that chunked + * differently would compute different manifest identities for identical bytes, + * and the store would then hold two names for one file. + */ +export const CHUNK_SIZE = 512 * 1024; + +/** One chunk a file's bytes are stored as. */ +export interface ContentChunkReference { + readonly hash: string; + readonly size: number; +} + +/** One file's bytes, as the store describes them. */ +export interface ContentManifest { + readonly size: number; + readonly chunks: readonly ContentChunkReference[]; +} + +const encoder = new TextEncoder(); +const decoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: false }); + +/** The bytes a content manifest is stored and identified as. */ +export function encodeContentManifest(chunks: readonly ContentChunkReference[]): Uint8Array { + return encoder.encode( + JSON.stringify({ + version: 1, + chunks: chunks.map((chunk) => ({ hash: chunk.hash, size: chunk.size })), + }), + ); +} + +function isSafeInteger(value: unknown): value is number { + return typeof value === "number" && Number.isSafeInteger(value); +} + +function members(value: unknown): Map | undefined { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + return undefined; + } + return new Map(Object.entries(value)); +} + +function declares(found: Map, expected: readonly string[]): boolean { + return found.size === expected.length && expected.every((name) => found.has(name)); +} + +/** + * The manifest one encoding describes, without a store to look anything up + * in. + * + * The same bytes are validated in more than one place — by a live run reading + * its own content store, by a reader checking a detached copy, and by an owner + * about to send a copy to a runner. What is decided here is only whether these + * bytes are a canonically encoded content manifest at all, and what size the + * chunks it names add up to. That the chunks exist is whoever called's to prove. + */ +export function decodeContentManifest( + encoded: Uint8Array, + reject: WorkspaceRejection, +): ContentManifest { + let text: string; + let offered: unknown; + try { + text = decoder.decode(encoded); + offered = JSON.parse(text); + } catch { + reject("a content manifest is not canonical UTF-8 JSON"); + } + const found = members(offered); + const chunks = found?.get("chunks"); + if ( + found === undefined || + !declares(found, ["version", "chunks"]) || + found.get("version") !== 1 || + !Array.isArray(chunks) + ) { + reject("a content manifest is not canonically encoded"); + } + const references: ContentChunkReference[] = []; + for (const chunk of chunks) { + const entry = members(chunk); + const hash = entry?.get("hash"); + const size = entry?.get("size"); + if ( + entry === undefined || + !declares(entry, ["hash", "size"]) || + typeof hash !== "string" || + !SHA256.test(hash) || + !isSafeInteger(size) || + size < 1 + ) { + // A zero-length chunk names no bytes, so a manifest that lists one is + // describing content it does not have. + reject("a content manifest is not canonically encoded"); + } + references.push({ hash, size }); + } + if (JSON.stringify({ version: 1, chunks: references }) !== text) { + reject("a content manifest is not canonically encoded"); + } + const total = references.reduce((sum, chunk) => sum + chunk.size, 0); + if (!Number.isSafeInteger(total)) { + reject("a content manifest names more bytes than a size can hold"); + } + return Object.freeze({ size: total, chunks: Object.freeze(references) }); +} diff --git a/packages/workflow/src/workspace/root-manifest.ts b/packages/workflow/src/workspace/root-manifest.ts index a532ddc31..90090d74f 100644 --- a/packages/workflow/src/workspace/root-manifest.ts +++ b/packages/workflow/src/workspace/root-manifest.ts @@ -79,17 +79,6 @@ export interface WorkspaceRootManifest { readonly entries: readonly WorkspaceRootEntry[]; } -/** One DOFS manifest: the ordered chunks one file's bytes are stored as. */ -export interface DofsChunkReference { - readonly hash: string; - readonly size: number; -} - -export interface DofsManifest { - readonly size: number; - readonly chunks: readonly DofsChunkReference[]; -} - const encoder = new TextEncoder(); const decoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: false }); @@ -365,61 +354,3 @@ export function validateCanonicalWorkspacePath(value: string, reject: WorkspaceR } } } - -/** - * The DOFS manifest one encoding describes, without a store to look anything up - * in. - * - * The same bytes are validated in more than one place — by a live run reading - * its own content store, by a reader checking a detached copy, and by an owner - * about to send a copy to a runner. What is decided here is only whether these - * bytes are a canonically encoded DOFS manifest at all, and what size the - * chunks it names add up to. That the chunks exist is whoever called's to prove. - */ -export function decodeDofsManifest(encoded: Uint8Array, reject: WorkspaceRejection): DofsManifest { - let text: string; - let offered: unknown; - try { - text = decoder.decode(encoded); - offered = JSON.parse(text); - } catch { - reject("a DOFS manifest is not canonical UTF-8 JSON"); - } - const found = members(offered); - const chunks = found?.get("chunks"); - if ( - found === undefined || - !declares(found, ["version", "chunks"]) || - found.get("version") !== 1 || - !Array.isArray(chunks) - ) { - reject("a DOFS manifest is not canonically encoded"); - } - const references: DofsChunkReference[] = []; - for (const chunk of chunks) { - const entry = members(chunk); - const hash = entry?.get("hash"); - const size = entry?.get("size"); - if ( - entry === undefined || - !declares(entry, ["hash", "size"]) || - typeof hash !== "string" || - !SHA256.test(hash) || - !isSafeInteger(size) || - size < 1 - ) { - // A zero-length chunk names no bytes, so a manifest that lists one is - // describing content it does not have. - reject("a DOFS manifest is not canonically encoded"); - } - references.push({ hash, size }); - } - if (JSON.stringify({ version: 1, chunks: references }) !== text) { - reject("a DOFS manifest is not canonically encoded"); - } - const total = references.reduce((sum, chunk) => sum + chunk.size, 0); - if (!Number.isSafeInteger(total)) { - reject("a DOFS manifest names more bytes than a size can hold"); - } - return Object.freeze({ size: total, chunks: Object.freeze(references) }); -} diff --git a/packages/workflow/tests/remote-materialization.test.ts b/packages/workflow/tests/remote-materialization.test.ts index b9e9bf865..b5d09d5dd 100644 --- a/packages/workflow/tests/remote-materialization.test.ts +++ b/packages/workflow/tests/remote-materialization.test.ts @@ -40,7 +40,7 @@ import { import type { RemoteContent, RemoteContentRequest, RemoteReadLink } from "../src/remote/read.ts"; import type { WorkspaceRootManifest } from "../src/workspace/root-manifest.ts"; import { parseWorkspaceRootManifest } from "../src/workspace/root-manifest.ts"; -import { encodeDofsManifest } from "../src/workspace/capture.ts"; +import { encodeContentManifest } from "../src/workspace/content-manifest.ts"; function reject(reason: string): never { throw new Error(reason); @@ -222,13 +222,13 @@ describe("materializing a retained Workspace root", () => { expect([...again.root.blobs]).toEqual([...captured.root.blobs]); }); - it("encodes a DOFS manifest the way the content store stores one", function* () { + it("encodes a content manifest the way the store stores one", function* () { // The runner and the owner must name identical bytes identically, and the // encoding is what decides that. - expect(new TextDecoder().decode(encodeDofsManifest([{ hash: "a".repeat(64), size: 3 }]))).toBe( - `{"version":1,"chunks":[{"hash":"${"a".repeat(64)}","size":3}]}`, - ); - expect(new TextDecoder().decode(encodeDofsManifest([]))).toBe('{"version":1,"chunks":[]}'); + expect( + new TextDecoder().decode(encodeContentManifest([{ hash: "a".repeat(64), size: 3 }])), + ).toBe(`{"version":1,"chunks":[{"hash":"${"a".repeat(64)}","size":3}]}`); + expect(new TextDecoder().decode(encodeContentManifest([]))).toBe('{"version":1,"chunks":[]}'); }); it("removes the materialization when its scope ends, however it ends", function* () { From 99b21a18329fec33e42ee18f45bc6f804ced6695 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 18:52:33 -0400 Subject: [PATCH 30/81] =?UTF-8?q?=E2=9C=8D=EF=B8=8F=20Let=20the=20owner=20?= =?UTF-8?q?decide=20one=20whole=20proposal=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `commit` carried a proposed root identity and nothing that could justify it. An identity with no manifest and no content closure is a name, and an owner adopting one would be taking the runner's word for what a root contains. The command now carries the whole thing — what the runner started from, what it proposes, the canonical manifest that identity is the digest of, the exact content that manifest closes over, the retained mappings the same operation produced, and the filtered events to append — and the owner recomputes all of it before anything is written. The frontier is re-read inside the transaction rather than before it, and compared with what the runner said it started from, root and terminal event both, `null` included exactly. A frontier read outside the transaction is a frontier that can move before the write. The inventory has to be exactly the closure of the proposed manifest: every manifest its file entries name, every blob those manifests name, once each, and nothing else. A missing piece is a root that cannot be materialized; an extra one is content the root does not account for. Each piece resolves from content already authoritative or from bytes this exact acquisition staged — staging supplies bytes and grants nothing, so another acquisition's scratch is unreachable and a digest already retained under different bytes is a disagreement rather than an overwrite. Everything lands together or not at all: content, the immutable root and its exact references, the mappings, the current pointer moved by compare-and-set from the expected root, the journal rows, and the retry decision. Journal rows name the root this commit selected — the proposed one when there is a publication, the unchanged one when there is not — which is what makes history readable against the Workspace it happened in. A journal-only transaction and an empty one are both ordinary, and neither invents a Workspace change to look uniform. Retained mappings go through the parsers the local host holds its own rows to, and creation identity is immutable: a second proposal naming the same Repository must describe the same Repository. An Agent-session mapping carries the canonical assertion and the derived key and nothing of the conversation; the owner never contacts a provider. The evidence runs on real workerd because none of it is provable otherwise. A forced failure after every category has been written rolls all of them back, the retry decision with them, leaving the id free. A lost response retried across eviction publishes once. A moved root, a moved anchor, an identity that is not its manifest's digest, an inventory missing or inventing a piece, unstaged content, a mapping that would rewrite an established identity, and a foreign socket each change nothing at all. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/commands.ts | 182 ++++++- .../workflow/src/cloudflare/dispatcher.ts | 22 + packages/workflow/src/cloudflare/publish.ts | 449 ++++++++++++++++++ .../src/deno/workspace/agent-sessions.ts | 65 +-- packages/workflow/src/remote/publication.ts | 63 +++ .../workflow/src/storage/agent-session.ts | 125 +++++ .../cloudflare/executor-acquisition.vitest.ts | 15 +- .../tests/cloudflare/remote-owner.vitest.ts | 11 +- .../tests/cloudflare/remote-publish.vitest.ts | 410 ++++++++++++++++ .../tests/cloudflare/settle-parser.vitest.ts | 8 +- .../cloudflare/support/executor-object.ts | 116 +++++ 11 files changed, 1398 insertions(+), 68 deletions(-) create mode 100644 packages/workflow/src/cloudflare/publish.ts create mode 100644 packages/workflow/src/remote/publication.ts create mode 100644 packages/workflow/src/storage/agent-session.ts create mode 100644 packages/workflow/tests/cloudflare/remote-publish.vitest.ts diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index a0ba67fb3..aeb7a0bb0 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -3,6 +3,13 @@ import { parseDocumentExecutionCompletion, } from "../storage/record.ts"; import { SHA256 } from "../workspace/root-manifest.ts"; +import { + parseRepositoryRecord, + parseWorktreeRecord, + type RepositoryRecord, + type WorktreeRecord, +} from "../composition/records.ts"; +import { type AgentSessionRecord, parseAgentSessionRecord } from "../storage/agent-session.ts"; export const MAX_MESSAGE_BYTES = 8 * 1024 * 1024; export const MAX_CONTENT_BYTES = 1024 * 1024; @@ -11,6 +18,12 @@ export const MAX_COMMANDS = 256; export const MAX_LEDGER_BYTES = 2 * 1024 * 1024; export const JOURNAL_PAGE_ENTRIES = 128; export const JOURNAL_PAGE_BYTES = 512 * 1024; +/** The most content identities one proposal may name. */ +export const MAX_PROPOSED_PIECES = 8192; +/** The most retained mapping changes one proposal may carry. */ +export const MAX_MAPPINGS = 256; +/** The longest canonical root manifest this owner reads. */ +export const MAX_ROOT_MANIFEST_BYTES = MAX_CONTENT_BYTES; export type CommandName = | "frontier" @@ -29,7 +42,14 @@ export type CommandRefusal = | "too-large" | "duplicate-conflict" | "capacity" - | "unavailable"; + | "unavailable" + // The frontier moved under the proposal. Not malformed and not a conflict of + // identity: the request was true when it was built and is not true now. + | "stale-root" + | "stale-journal" + // A retained mapping already exists and describes something else. Creation + // identity is immutable, so this is refused rather than rewritten. + | "mapping-conflict"; export class CommandError extends Error { override name = "CommandError"; @@ -76,14 +96,51 @@ export interface StageCommand extends CommandEnvelope { readonly bytes: string; } +/** + * One closed proposal, and everything the owner needs to decide it. + * + * The earlier shape carried a proposed root identity and nothing that could + * justify it — an identity with no manifest and no content closure is a name, + * not a proposal, and an owner adopting one would be taking the runner's word + * for what a root contains. This carries the whole thing: what the runner + * started from, what it proposes, the canonical manifest that identity is the + * digest of, the exact content that manifest closes over, the retained mappings + * the same operation produced, and the filtered events to append. + * + * `publication` is absent for a transaction that only appended to the journal. + * That is a real case rather than a degenerate one, and inventing a Workspace + * change to fill it would publish a root nothing asked for. + */ export interface CommitCommand extends CommandEnvelope { readonly command: "commit"; readonly expectedWorkspaceRootId: string; readonly expectedJournalEventId: string | null; - readonly proposedWorkspaceRootId: string; + readonly publication: ProposedPublication | null; + readonly mappings: readonly ProposedMapping[]; + /** Exactly what `serializeDurableEvent` produced, terminating newline included. */ readonly events: readonly string[]; } +/** The Workspace half of a proposal, when there is one. */ +export interface ProposedPublication { + readonly proposedWorkspaceRootId: string; + readonly proposedManifest: string; + readonly content: readonly ProposedPiece[]; +} + +/** One content identity the proposed root closes over. */ +export interface ProposedPiece { + readonly kind: ContentKind; + readonly digest: string; + readonly size: number; +} + +/** One retained mapping the proposal carries, already parsed. */ +export type ProposedMapping = + | { readonly kind: "repository"; readonly record: RepositoryRecord } + | { readonly kind: "worktree"; readonly record: WorktreeRecord } + | { readonly kind: "agent-session"; readonly record: AgentSessionRecord }; + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -116,7 +173,8 @@ const MEMBERS: Record = { ...ENVELOPE, "expectedWorkspaceRootId", "expectedJournalEventId", - "proposedWorkspaceRootId", + "publication", + "mappings", "events", ], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], @@ -277,7 +335,123 @@ export function parseCommand(raw: string): RunnerCommand { command, expectedWorkspaceRootId: digest(members, "expectedWorkspaceRootId"), expectedJournalEventId: nullableText(members, "expectedJournalEventId"), - proposedWorkspaceRootId: digest(members, "proposedWorkspaceRootId"), + publication: publication(members.get("publication")), + mappings: mappings(members.get("mappings")), events: eventRecords(members.get("events")), }; } + +/** + * The Workspace half of a proposal, or its absence. + * + * `null` is a journal-only transaction and is admitted as such. Everything else + * must be a complete proposal: an identity, the canonical manifest that + * identity is supposed to be the digest of, and the exact inventory. Whether + * the identity really is that digest, and whether the inventory really is the + * closure, is the owner's to recompute — this only decides whether the request + * is shaped like a proposal at all. + */ +function publication(value: unknown): ProposedPublication | null { + if (value === null) { + return null; + } + const members = object(value); + closed(members, ["proposedWorkspaceRootId", "proposedManifest", "content"]); + const manifest = members.get("proposedManifest"); + if (typeof manifest !== "string" || manifest === "") { + throw new CommandError("malformed-member"); + } + if (new TextEncoder().encode(manifest).length > MAX_ROOT_MANIFEST_BYTES) { + throw new CommandError("too-large"); + } + return { + proposedWorkspaceRootId: digest(members, "proposedWorkspaceRootId"), + proposedManifest: manifest, + content: pieces(members.get("content")), + }; +} + +/** + * The inventory, in the order it must arrive. + * + * Canonical order and no repeats, checked here rather than sorted into shape: a + * proposal that named one piece twice, or named them in an order this build did + * not produce, is not the proposal the runner computed its identity over. + */ +function pieces(value: unknown): ProposedPiece[] { + if (!Array.isArray(value)) { + throw new CommandError("malformed-member"); + } + if (value.length > MAX_PROPOSED_PIECES) { + throw new CommandError("too-large"); + } + const found: ProposedPiece[] = []; + let previous: string | undefined; + for (const entry of value) { + const members = object(entry); + closed(members, ["kind", "digest", "size"]); + const size = members.get("size"); + if (typeof size !== "number" || !Number.isSafeInteger(size) || size < 0) { + throw new CommandError("malformed-member"); + } + if (size > MAX_CONTENT_BYTES) { + throw new CommandError("too-large"); + } + const piece: ProposedPiece = { + kind: kind(members), + digest: digest(members, "digest"), + size, + }; + const ordering = `${piece.kind}:${piece.digest}`; + if (previous !== undefined && ordering <= previous) { + throw new CommandError("malformed-member"); + } + previous = ordering; + found.push(piece); + } + return found; +} + +/** + * The retained mappings a proposal carries, read through the shared parsers. + * + * The parsers are the ones the local host holds its own rows to. A private + * approximation here would be the two hosts disagreeing about what a retained + * Repository is, and the owner would be the one that found out. + */ +function mappings(value: unknown): ProposedMapping[] { + if (!Array.isArray(value)) { + throw new CommandError("malformed-member"); + } + if (value.length > MAX_MAPPINGS) { + throw new CommandError("too-large"); + } + return value.map((entry) => { + const members = object(entry); + closed(members, ["kind", "record"]); + const which = members.get("kind"); + const offered = members.get("record"); + if (which === "repository") { + const record = parseRepositoryRecord(offered); + if (record === undefined) { + throw new CommandError("malformed-member"); + } + return { kind: which, record }; + } + if (which === "worktree") { + const record = parseWorktreeRecord(offered); + if (record === undefined) { + throw new CommandError("malformed-member"); + } + return { kind: which, record }; + } + if (which === "agent-session") { + const record = parseAgentSessionRecord(offered); + if (record === undefined) { + throw new CommandError("malformed-member"); + } + return { kind: which, record }; + } + throw new CommandError("malformed-member"); + }); +} diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 7216fd28f..26df1b59c 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -47,6 +47,7 @@ import { bytesOf, decodeBase64, sha256Hex } from "./encoding.ts"; import { readContent, readFrontier, readJournalPage, readRoot } from "./owner-reads.ts"; import type { OwnerTransactions } from "./owner-transaction.ts"; import { COMMAND_TABLE, STAGING_TABLE } from "./private-schema.ts"; +import { applyCommit } from "./publish.ts"; import { recognizeObject } from "./recognition.ts"; function requestFingerprint(command: RunnerCommand): string { @@ -91,6 +92,17 @@ function storedDecision(value: unknown, id: string): CommandResult | "reconstruc throw new Error("private protocol storage holds a malformed result"); } +/** + * A fresh opaque identity for one retained event. + * + * Minted by the owner inside the transaction that writes the row. An id the + * runner chose would be a runner deciding what a retained event is called, and + * two runners could choose the same one. + */ +function mintEventId(): string { + return crypto.randomUUID(); +} + function retainedDecision(command: RunnerCommand, result: CommandResult): string { if ( result.outcome === "performed" && @@ -200,6 +212,16 @@ function perform( if (command.command === "stage") { return { id: command.id, outcome: "performed", value: stage(ctx, acquisitionId, command) }; } + if (command.command === "commit") { + return { + id: command.id, + outcome: "performed", + value: applyCommit(ctx.storage, acquisitionId, command, mintEventId), + }; + } + // `settle` is a later checkpoint's. It parses strictly and is declined, + // because a placeholder that reported success is the one answer a runner + // cannot recover from. return { id: command.id, outcome: "refused", refusal: "command:unavailable" }; } diff --git a/packages/workflow/src/cloudflare/publish.ts b/packages/workflow/src/cloudflare/publish.ts new file mode 100644 index 000000000..4a967ef32 --- /dev/null +++ b/packages/workflow/src/cloudflare/publish.ts @@ -0,0 +1,449 @@ +/** + * Deciding one proposal, and applying all of it or none of it. + * + * This is where a remote run actually moves. Everything before it is reading + * and staging; everything after it is history. The runner has done the work, + * captured a root, and offered a description of what it wants published — and + * none of that is authority. The owner recomputes every identity, resolves + * every piece against content it already holds or bytes this exact acquisition + * staged, and only then writes. + * + * The order is deliberate and each step exists because skipping it is a way to + * publish something nobody proposed: + * + * 1. The frontier is re-read *here*, inside the transaction, and compared with + * what the runner said it started from — root and terminal event both, + * `null` included exactly. A frontier read before the transaction is a + * frontier that can move before the write. + * 2. The proposed identity is recomputed from the manifest rather than + * believed. An identity is a digest, and a digest a caller supplies is a + * claim about bytes rather than a property of them. + * 3. The inventory must be exactly the closure of that manifest — every + * manifest its file entries name, every blob those manifests name, once + * each, and nothing else. A missing piece is a root that cannot be + * materialized; an extra one is content the root does not account for. + * 4. Each piece resolves from authoritative content or from this acquisition's + * staging. Staging supplies bytes and grants nothing: a digest another + * acquisition staged is not reachable, and a digest already authoritative + * under different bytes is a disagreement rather than an overwrite. + * 5. Content, root, references, mappings, the current pointer and the journal + * rows are written together. The pointer moves by compare-and-set from the + * expected root, so two commits racing the same frontier cannot both win. + * + * Journal rows are associated with the root this commit selected: the proposed + * root when there is a publication, the unchanged expected root when there is + * not. That is the same rule the local host follows, and it is what makes + * history readable against the Workspace it happened in. + * + * Nothing here awaits, yields, sends a frame or contacts the runner. It runs + * inside one synchronous transaction and returns a value the caller serializes + * afterwards. + */ + +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; +import { + compareUtf8, + parseWorkspaceRootManifest, + WORKSPACE_ROOT_DOMAIN, +} from "../workspace/root-manifest.ts"; +import { decodeContentManifest } from "../workspace/content-manifest.ts"; +import { sha256Hex } from "../workspace/sha256.ts"; +import { CommandError, type CommitCommand, type ProposedMapping } from "./commands.ts"; +import { bytesOf } from "./encoding.ts"; +import { STAGING_TABLE } from "./private-schema.ts"; +import type { OwnerStorage } from "./storage.ts"; + +/** What the owner answers a performed commit with. */ +export interface CommitValue { + readonly workspaceRootId: string; + readonly journalEventIds: readonly string[]; +} + +function corrupt(reason: string): never { + throw new WorkflowRecordMalformedError("workflow owner storage", reason); +} + +function rows( + storage: OwnerStorage, + sql: string, + ...bindings: unknown[] +): Record[] { + return storage.sql.exec(sql, ...bindings).toArray(); +} + +/** Content identities in the canonical order references are written in. */ +function sortedDigests(digests: Iterable): string[] { + const found = [...digests]; + found.sort(compareUtf8); + return found; +} + +function hexBytes(digest: string): Uint8Array { + const bytes = new Uint8Array(digest.length / 2); + for (let index = 0; index < bytes.length; index += 1) { + bytes[index] = Number.parseInt(digest.slice(index * 2, index * 2 + 2), 16); + } + return bytes; +} + +/** The frontier as it is right now, read where the write will happen. */ +function frontier(storage: OwnerStorage): { rootId: string; journalEventId: string | null } { + const state = rows(storage, "SELECT current_root_id FROM workspace_state WHERE singleton_id = 1"); + const current = state[0]?.["current_root_id"]; + if (state.length !== 1 || typeof current !== "string") { + return corrupt("the Workspace has no single current root"); + } + const last = rows( + storage, + "SELECT event_id FROM journal_events ORDER BY sequence DESC LIMIT 1", + )[0]; + const eventId = last?.["event_id"]; + if (last !== undefined && typeof eventId !== "string") { + return corrupt("a journal row has no identity"); + } + return { rootId: current, journalEventId: last === undefined ? null : String(eventId) }; +} + +/** Bytes for one proposed identity, from what is authoritative or what was staged. */ +function resolve( + storage: OwnerStorage, + acquisitionId: string, + kind: "manifest" | "blob", + digest: string, +): { bytes: Uint8Array; authoritative: boolean } { + const table = kind === "manifest" ? "vfs_manifests" : "vfs_blob_bytes"; + const column = kind === "manifest" ? "encoded" : "bytes"; + const held = rows( + storage, + `SELECT ${column} AS content FROM ${table} WHERE lower(hex(hash)) = ?`, + digest, + )[0]; + if (held !== undefined) { + const bytes = bytesOf(held["content"]); + if (sha256Hex(bytes) !== digest) { + return corrupt("retained content disagrees with the identity it is stored under"); + } + return { bytes, authoritative: true }; + } + const staged = rows( + storage, + `SELECT bytes FROM ${STAGING_TABLE} WHERE acquisition_id = ? AND kind = ? AND digest = ?`, + acquisitionId, + kind, + digest, + )[0]; + if (staged === undefined) { + // Either never offered, or offered by an acquisition that is not this one. + // Both are the same refusal: this proposal names content this connection + // has not supplied. + throw new CommandError("malformed-member"); + } + const bytes = bytesOf(staged["bytes"]); + if (sha256Hex(bytes) !== digest) { + return corrupt("staged content disagrees with the identity it was stored under"); + } + return { bytes, authoritative: false }; +} + +/** + * Apply one proposal, entirely, inside the caller's open transaction. + * + * The caller has already proved the acquisition twice and recognized the store. + * What is left is deciding whether this proposal is true and writing it. + */ +export function applyCommit( + storage: OwnerStorage, + acquisitionId: string, + command: CommitCommand, + mintEventId: () => string, +): CommitValue { + const now = frontier(storage); + if (now.rootId !== command.expectedWorkspaceRootId) { + throw new CommandError("stale-root"); + } + if (now.journalEventId !== command.expectedJournalEventId) { + throw new CommandError("stale-journal"); + } + + const selected = + command.publication === null + ? command.expectedWorkspaceRootId + : publish(storage, acquisitionId, command); + + for (const mapping of command.mappings) { + applyMapping(storage, mapping); + } + + const journalEventIds: string[] = []; + for (const record of command.events) { + const eventId = mintEventId(); + storage.sql.exec( + "INSERT INTO journal_events (event_id, record, workspace_root_id) VALUES (?, ?, ?)", + eventId, + record, + selected, + ); + journalEventIds.push(eventId); + } + + return { workspaceRootId: selected, journalEventIds }; +} + +/** Adopt the content and the root, and move the pointer to it. */ +function publish(storage: OwnerStorage, acquisitionId: string, command: CommitCommand): string { + const proposal = command.publication; + if (proposal === null) { + return command.expectedWorkspaceRootId; + } + if ( + sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${proposal.proposedManifest}`) !== + proposal.proposedWorkspaceRootId + ) { + throw new CommandError("malformed-member"); + } + const parsed = parseWorkspaceRootManifest(proposal.proposedManifest, () => { + throw new CommandError("malformed-member"); + }); + + // The closure the manifest actually names, derived here rather than taken + // from the inventory the request supplied. + const named = new Set( + parsed.entries.flatMap((entry) => (entry.kind === "file" ? [entry.manifest] : [])), + ); + const offered = new Map( + proposal.content.map((piece) => [`${piece.kind}:${piece.digest}`, piece]), + ); + + const manifests = new Map(); + for (const digest of named) { + const piece = offered.get(`manifest:${digest}`); + if (piece === undefined) { + throw new CommandError("malformed-member"); + } + const { bytes } = resolve(storage, acquisitionId, "manifest", digest); + if (bytes.length !== piece.size) { + throw new CommandError("malformed-member"); + } + manifests.set(digest, bytes); + } + + const blobs = new Map(); + for (const [digest, bytes] of manifests) { + const decoded = decodeContentManifest(bytes, () => { + throw new CommandError("malformed-member"); + }); + for (const entry of parsed.entries) { + if (entry.kind === "file" && entry.manifest === digest && entry.size !== decoded.size) { + throw new CommandError("malformed-member"); + } + } + for (const chunk of decoded.chunks) { + const seen = blobs.get(chunk.hash); + if (seen !== undefined && seen !== chunk.size) { + throw new CommandError("malformed-member"); + } + blobs.set(chunk.hash, chunk.size); + } + } + + // Exactly the closure: nothing missing, nothing extra. + if (offered.size !== named.size + blobs.size) { + throw new CommandError("malformed-member"); + } + + const blobBytes = new Map(); + for (const [digest, size] of blobs) { + const piece = offered.get(`blob:${digest}`); + if (piece === undefined || piece.size !== size) { + throw new CommandError("malformed-member"); + } + const { bytes } = resolve(storage, acquisitionId, "blob", digest); + if (bytes.length !== size) { + throw new CommandError("malformed-member"); + } + blobBytes.set(digest, bytes); + } + + for (const [digest, bytes] of blobBytes) { + const hash = hexBytes(digest); + storage.sql.exec( + "INSERT INTO vfs_blobs (hash, size, last_seen) VALUES (?, ?, 0) ON CONFLICT(hash) DO NOTHING", + hash, + bytes.length, + ); + storage.sql.exec( + "INSERT INTO vfs_blob_bytes (hash, bytes) VALUES (?, ?) ON CONFLICT(hash) DO NOTHING", + hash, + bytes, + ); + } + for (const [digest, bytes] of manifests) { + storage.sql.exec( + `INSERT INTO vfs_manifests (hash, size, encoded, last_seen) VALUES (?, ?, ?, 0) + ON CONFLICT(hash) DO NOTHING`, + hexBytes(digest), + decodeContentManifest(bytes, () => { + throw new CommandError("malformed-member"); + }).size, + bytes, + ); + } + + // Immutable: a root already retained is confirmed rather than rewritten. + const existing = rows( + storage, + "SELECT manifest FROM workspace_roots WHERE root_id = ?", + proposal.proposedWorkspaceRootId, + )[0]; + if (existing === undefined) { + storage.sql.exec( + "INSERT INTO workspace_roots (root_id, format_version, manifest) VALUES (?, 1, ?)", + proposal.proposedWorkspaceRootId, + proposal.proposedManifest, + ); + for (const digest of sortedDigests(named)) { + storage.sql.exec( + "INSERT INTO workspace_root_manifest_refs (root_id, manifest_hash) VALUES (?, ?)", + proposal.proposedWorkspaceRootId, + hexBytes(digest), + ); + } + for (const digest of sortedDigests(blobs.keys())) { + storage.sql.exec( + "INSERT INTO workspace_root_blob_refs (root_id, blob_hash) VALUES (?, ?)", + proposal.proposedWorkspaceRootId, + hexBytes(digest), + ); + } + } else if (existing["manifest"] !== proposal.proposedManifest) { + return corrupt("a retained Workspace root disagrees with the identity it is stored under"); + } + + // Compare-and-set. Two commits racing one frontier cannot both move it. + storage.sql.exec( + "UPDATE workspace_state SET current_root_id = ? WHERE singleton_id = 1 AND current_root_id = ?", + proposal.proposedWorkspaceRootId, + command.expectedWorkspaceRootId, + ); + const moved = rows( + storage, + "SELECT current_root_id FROM workspace_state WHERE singleton_id = 1", + )[0]; + if (moved?.["current_root_id"] !== proposal.proposedWorkspaceRootId) { + throw new CommandError("stale-root"); + } + return proposal.proposedWorkspaceRootId; +} + +/** + * One retained mapping, inserted or confirmed. + * + * Creation identity is immutable: a second proposal naming the same Repository + * must describe the same Repository, and one that does not is refused rather + * than allowed to rewrite what an earlier execution established. + */ +function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { + if (mapping.kind === "repository") { + const record = mapping.record; + const held = rows( + storage, + `SELECT name, locator_fingerprint, requested_base, creation_commit, primary_branch, + object_format, checkout_path FROM workspace_repositories WHERE name = ?`, + record.name, + )[0]; + if (held === undefined) { + storage.sql.exec( + `INSERT INTO workspace_repositories + (name, locator, locator_fingerprint, requested_base, creation_commit, + primary_branch, object_format, checkout_path) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + record.name, + record.locatorFingerprint, + record.locatorFingerprint, + record.requestedBase, + record.creationCommit, + record.primaryBranch, + record.objectFormat, + record.checkoutPath, + ); + return; + } + if ( + held["locator_fingerprint"] !== record.locatorFingerprint || + held["creation_commit"] !== record.creationCommit || + held["primary_branch"] !== record.primaryBranch || + held["object_format"] !== record.objectFormat || + held["checkout_path"] !== record.checkoutPath + ) { + throw new CommandError("mapping-conflict"); + } + return; + } + + if (mapping.kind === "worktree") { + const record = mapping.record; + const held = rows( + storage, + `SELECT requested_branch, creation_commit, checkout_path FROM workspace_worktrees + WHERE repository_name = ? AND name = ?`, + record.repositoryName, + record.name, + )[0]; + if (held === undefined) { + storage.sql.exec( + `INSERT INTO workspace_worktrees + (repository_name, name, requested_branch, requested_base, creation_commit, checkout_path) + VALUES (?, ?, ?, ?, ?, ?)`, + record.repositoryName, + record.name, + record.requestedBranch, + record.requestedBase, + record.creationCommit, + record.checkoutPath, + ); + return; + } + if ( + held["requested_branch"] !== record.requestedBranch || + held["creation_commit"] !== record.creationCommit || + held["checkout_path"] !== record.checkoutPath + ) { + throw new CommandError("mapping-conflict"); + } + return; + } + + const record = mapping.record; + const held = rows( + storage, + `SELECT provider, agent_command, session_identity, assertion_kind, assertion_value + FROM agent_sessions WHERE session_key = ?`, + record.sessionKey, + )[0]; + if (held === undefined) { + storage.sql.exec( + `INSERT INTO agent_sessions + (session_key, provider, agent_command, session_identity, policy, + assertion_kind, assertion_value, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + record.sessionKey, + record.provider, + record.agentCommand, + record.sessionIdentity, + record.policy, + record.assertion.kind, + record.assertion.value, + record.createdAt, + ); + return; + } + if ( + held["provider"] !== record.provider || + held["agent_command"] !== record.agentCommand || + held["session_identity"] !== record.sessionIdentity || + held["assertion_kind"] !== record.assertion.kind || + held["assertion_value"] !== record.assertion.value + ) { + throw new CommandError("mapping-conflict"); + } +} diff --git a/packages/workflow/src/deno/workspace/agent-sessions.ts b/packages/workflow/src/deno/workspace/agent-sessions.ts index ff55877b3..1142cc92b 100644 --- a/packages/workflow/src/deno/workspace/agent-sessions.ts +++ b/packages/workflow/src/deno/workspace/agent-sessions.ts @@ -38,7 +38,6 @@ */ import type { DatabaseSync } from "node:sqlite"; -import { createHash } from "node:crypto"; /** A retained Agent session this host will not continue under. */ export class WorkflowAgentSessionError extends Error { @@ -46,54 +45,26 @@ export class WorkflowAgentSessionError extends Error { } /** - * One durable identity a provider asserted, and what kind of thing it is. + * The shape and the key derivation are the shared rule, not this adapter's. * - * Tagged, because "the adapter's own session id" and "an ACP session id" and "a - * record id in some store" are different claims that happen to be strings. A - * host comparing them without the tag would accept one for another. + * Both hosts retain these mappings, and two derivations would be two keys for + * one session — reattachment would quietly start a new conversation instead of + * finding the old one. What stays here is the storage: the columns, the + * statements, and the transaction they run in. */ -export interface ProviderAssertion { - readonly kind: string; - readonly value: string; -} - -/** What identifies one logical Agent session. */ -export interface AgentSessionIdentity { - /** Which provider holds the conversation, as that provider names itself. */ - readonly provider: string; - /** The resolved agent command, not the name a document wrote. */ - readonly agentCommand: string; - /** The engine-derived Agent/Session expansion identity. Never authored. */ - readonly sessionIdentity: string; -} - -/** One retained mapping, as the run's database holds it. */ -export interface AgentSessionRecord extends AgentSessionIdentity { - readonly sessionKey: string; - /** The session policy in force when the provider created this session. */ - readonly policy: string; - readonly assertion: ProviderAssertion; - readonly createdAt: string; -} - -function digest(value: string): string { - return createHash("sha256").update(value, "utf8").digest("hex").slice(0, 32); -} - -/** - * The key one logical session is retained under, within this run. - * - * The engine-derived Session expansion identity and nothing else. The provider - * and the resolved agent command are compatibility attributes stored beside it: - * changing either refuses reattachment rather than addressing a second mapping, - * because a `` element that changed agent is the same element asking - * for something this run cannot give it. - * - * Digested so it stays bounded, and namespaced so a row is recognizable. - */ -export function agentSessionKey(identity: AgentSessionIdentity): string { - return ["xmd", "workflow", "v1", digest(identity.sessionIdentity)].join(":"); -} +import { + agentSessionKey, + type AgentSessionIdentity, + type AgentSessionRecord, + type ProviderAssertion, +} from "../../storage/agent-session.ts"; + +export { agentSessionKey, parseAgentSessionRecord } from "../../storage/agent-session.ts"; +export type { + AgentSessionIdentity, + AgentSessionRecord, + ProviderAssertion, +} from "../../storage/agent-session.ts"; const COLUMNS = `session_key, provider, agent_command, session_identity, policy, assertion_kind, assertion_value, created_at`; diff --git a/packages/workflow/src/remote/publication.ts b/packages/workflow/src/remote/publication.ts new file mode 100644 index 000000000..d8d95d557 --- /dev/null +++ b/packages/workflow/src/remote/publication.ts @@ -0,0 +1,63 @@ +/** + * What a runner proposes when a transaction changed the Workspace. + * + * A transaction that only appended to the journal proposes nothing here: the + * run's Workspace is where it was, and saying otherwise would invent a mutation + * to make the shape uniform. When the Workspace did change, exactly one of + * these describes the whole change — the root that was started from, the + * canonical root now proposed, the content that root closes over, and the + * retained mappings the same operation produced. + * + * Everything here is semantic. There is no command, no correlation id, no + * base64, no staged row, no SQL, no socket and no path on the runner. The + * adapter beneath translates this into whatever its owner speaks; a neutral + * value that carried transport vocabulary would make every other host implement + * this one's transport. + * + * The inventory is exact rather than advisory. It names every manifest and blob + * the proposed root closes over, once each, in canonical order — not the pieces + * that happen to be new. The owner resolves each identity from content it + * already holds or from what this acquisition staged, and an inventory that + * named more or fewer would be a root whose content nobody agreed on. + */ + +import type { RepositoryRecord, WorktreeRecord } from "../composition/records.ts"; +import type { AgentSessionRecord } from "../storage/agent-session.ts"; + +/** One content identity a proposed root closes over. */ +export interface ProposedContent { + readonly kind: "manifest" | "blob"; + readonly digest: string; + readonly size: number; +} + +/** + * One complete Workspace change, as the owner will receive it. + * + * `proposedWorkspaceRootId` is not taken on trust: it is what the runner + * computed, and the owner recomputes it from the manifest before anything is + * adopted. Carrying it makes the disagreement detectable rather than making the + * owner guess what the runner thought it was proposing. + */ +export interface WorkspacePublication { + readonly proposedWorkspaceRootId: string; + /** The canonical root manifest, exactly as it was encoded and hashed. */ + readonly proposedManifest: string; + readonly content: readonly ProposedContent[]; +} + +/** + * One retained mapping the same operation produced. + * + * A Repository or Worktree row and the Workspace bytes that make its checkout + * true are one proposal: a mapping naming a checkout that does not exist, or a + * checkout no mapping accounts for, is a Workspace that only half happened. + * + * An Agent-session mapping carries the provider's canonical assertion and the + * derived key, and nothing of the conversation itself. The owner retains what + * the run established; it never contacts or impersonates an Agent provider. + */ +export type RetainedMapping = + | { readonly kind: "repository"; readonly record: RepositoryRecord } + | { readonly kind: "worktree"; readonly record: WorktreeRecord } + | { readonly kind: "agent-session"; readonly record: AgentSessionRecord }; diff --git a/packages/workflow/src/storage/agent-session.ts b/packages/workflow/src/storage/agent-session.ts new file mode 100644 index 000000000..2bba915aa --- /dev/null +++ b/packages/workflow/src/storage/agent-session.ts @@ -0,0 +1,125 @@ +/** + * What one retained Agent session is, independent of who stores it. + * + * A run remembers that a `` element was attached to a provider's + * conversation so a later execution can reattach to the same one. What it + * remembers is deliberately thin: which provider, which resolved command, the + * engine-derived expansion identity, the policy in force, and the provider's + * own assertion about the session. The conversation is the provider's and is + * never retained, sent, or reconstructed. + * + * Both hosts retain this, so the shape and the key derivation live here rather + * than inside either one. A second derivation would be two keys for one + * session, and reattachment would silently start a new conversation. + */ + +import { sha256Hex } from "../workspace/sha256.ts"; + +/** + * What a provider says about a session it created. + * + * Tagged, because "the adapter's own session id", "an ACP session id" and "a + * record id in some store" are different claims that happen to be strings. A + * host comparing them without the tag would accept one for another. + */ +export interface ProviderAssertion { + readonly kind: string; + readonly value: string; +} + +/** What identifies one logical Agent session. */ +export interface AgentSessionIdentity { + /** Which provider holds the conversation, as that provider names itself. */ + readonly provider: string; + /** The resolved agent command, not the name a document wrote. */ + readonly agentCommand: string; + /** The engine-derived Agent/Session expansion identity. Never authored. */ + readonly sessionIdentity: string; +} + +/** One retained mapping, as a run's storage holds it. */ +export interface AgentSessionRecord extends AgentSessionIdentity { + readonly sessionKey: string; + /** The session policy in force when the provider created this session. */ + readonly policy: string; + readonly assertion: ProviderAssertion; + readonly createdAt: string; +} + +/** + * The key one logical session is retained under, within this run. + * + * The engine-derived Session expansion identity and nothing else. The provider + * and the resolved agent command are compatibility attributes stored beside it: + * changing either refuses reattachment rather than addressing a second mapping, + * because a `` element that changed agent is the same element asking + * for something this run cannot give it. + * + * Digested so it stays bounded, and namespaced so a row is recognizable. + */ +export function agentSessionKey(identity: AgentSessionIdentity): string { + return ["xmd", "workflow", "v1", sha256Hex(identity.sessionIdentity).slice(0, 32)].join(":"); +} + +function text(found: Map, name: string): string | undefined { + const value = found.get(name); + return typeof value === "string" && value !== "" ? value : undefined; +} + +function members(value: unknown): Map | undefined { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + return undefined; + } + return new Map(Object.entries(value)); +} + +/** + * Read one retained mapping out of a value nothing has checked. + * + * Every member, and the key recomputed from the identity rather than believed. + * A record whose key does not follow from its own identity is a record that + * would be retained under a name nothing could look it up by. + */ +export function parseAgentSessionRecord(value: unknown): AgentSessionRecord | undefined { + const found = members(value); + if (found === undefined || found.size !== 7) { + return undefined; + } + const provider = text(found, "provider"); + const agentCommand = text(found, "agentCommand"); + const sessionIdentity = text(found, "sessionIdentity"); + const sessionKey = text(found, "sessionKey"); + const policy = text(found, "policy"); + const assertion = members(found.get("assertion")); + if ( + provider === undefined || + agentCommand === undefined || + sessionIdentity === undefined || + sessionKey === undefined || + policy === undefined || + assertion === undefined || + assertion.size !== 2 + ) { + return undefined; + } + const kind = text(assertion, "kind"); + const asserted = text(assertion, "value"); + if (kind === undefined || asserted === undefined) { + return undefined; + } + const identity: AgentSessionIdentity = { provider, agentCommand, sessionIdentity }; + if (agentSessionKey(identity) !== sessionKey) { + return undefined; + } + const createdAt = text(found, "createdAt"); + if (createdAt === undefined || new Date(createdAt).toISOString() !== createdAt) { + return undefined; + } + return Object.freeze({ + ...identity, + sessionKey, + policy, + assertion: Object.freeze({ kind, value: asserted }), + createdAt, + }); +} diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts index 317e1dad6..6dbb1c84c 100644 --- a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -335,7 +335,7 @@ describe("reading a runner command", () => { ); }); - it("reads a commit intent whole, then refuses to act on it in this release", async () => { + it("reads a commit intent whole, then refuses one proposed against a moved frontier", async () => { const stub = executor(); await on(stub, (o) => o.initialize()); await admitted(stub); @@ -344,16 +344,21 @@ describe("reading a runner command", () => { command: "commit", expectedWorkspaceRootId: `a${"0".repeat(63)}`, expectedJournalEventId: null, - proposedWorkspaceRootId: `b${"1".repeat(63)}`, + publication: { + proposedWorkspaceRootId: `b${"1".repeat(63)}`, + proposedManifest: "{}", + content: [], + }, + mappings: [], events: ["event-1"], }); // The shape is read — an unknown member or a malformed root would refuse - // differently — and then declined, because applying one is a later - // checkpoint's work and a performed placeholder would be a lie. + // differently — and then declined on its merits: this run's frontier is not + // the root the proposal says it started from. expect(await on(stub, (o) => o.send(1, raw))).toEqual({ id: "7", outcome: "refused", - refusal: "command:unavailable", + refusal: "command:stale-root", }); expect( await on(stub, (o) => o.send(1, JSON.stringify({ ...JSON.parse(raw), id: "8", extra: 1 }))), diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index afa58cc0f..1557fc251 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -576,19 +576,10 @@ describe("the remote owner protocol", () => { expect(first).toMatchObject({ outcome: "performed" }); }); - it("refuses D3 and D4 mutations rather than reporting placeholder success", async () => { + it("refuses a settlement rather than reporting placeholder success", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); await admit(stub); - expect( - await send(stub, "commit", { - command: "commit", - expectedWorkspaceRootId: ROOT_ID, - expectedJournalEventId: null, - proposedWorkspaceRootId: ROOT_ID, - events: [], - }), - ).toEqual({ id: "commit", outcome: "refused", refusal: "command:unavailable" }); expect( await send(stub, "settle", { command: "settle", diff --git a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts new file mode 100644 index 000000000..69adf37f2 --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts @@ -0,0 +1,410 @@ +/** + * Publishing one proposal on real workerd. + * + * This is the point where a remote run moves, and almost nothing about it is + * provable against a model. Whether content, roots, references, mappings, the + * current pointer, the journal and the retry decision commit together is a + * property of the Durable Object's own `transactionSync()`. Whether a lost + * response can be retried exactly once is a property of storage surviving + * eviction. Whether a stale socket can still write is a property of the + * runtime's socket list. + * + * So these run against a real namespace, real SQLite and real Hibernation + * WebSockets, and the assertions are about what was published, what was + * refused, and what was left exactly as it was. + */ + +import { env, evictDurableObject, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { + NEXT_BLOB_ID, + NEXT_BYTES, + NEXT_ROOT_ID, + nextPublication, + POLICY, + ROOT_ID, + RUN_ID, + VALID_CLAIMS, +} from "./support/executor-object.ts"; +import { encodeBase64 } from "../../src/cloudflare/encoding.ts"; +import { sha256Hex } from "../../src/workspace/sha256.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`publish-${unique}-${Math.random()}`)); +} + +function on(stub: ReturnType, body: (owner: ExecutorObject) => T): Promise { + return runInDurableObject(stub, body); +} + +function record(value: unknown): Record { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new Error("expected an object answer"); + } + return Object.fromEntries(Object.entries(value)); +} + +async function admit(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + expect(await on(stub, (owner) => owner.admitConnection({ token, release: POLICY.release }))).toBe( + "admitted", + ); +} + +function send( + stub: ReturnType, + id: string, + command: Record, +): Promise> { + return on(stub, (owner) => record(owner.send(1, JSON.stringify({ id, ...command })))); +} + +/** + * A real accepted connection, which is what survives eviction. + * + * A `WebSocketPair` made inside the object is gone once the object is evicted; + * only a socket the runtime accepted through a request comes back with its + * attachment. The retry claim is about exactly that, so it has to use this. + */ +async function connect(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + const response = await stub.fetch("https://owner.invalid/executor", { + headers: { + authorization: `Bearer ${token}`, + upgrade: "websocket", + "x-release": POLICY.release, + "x-run-id": RUN_ID, + }, + }); + const socket = response.webSocket; + if (socket === null) { + throw new Error(`expected an executor WebSocket, received ${response.status}`); + } + socket.accept(); + return socket; +} + +function ask( + socket: WebSocket, + id: string, + command: Record, +): Promise> { + return new Promise((resolve, reject) => { + const receive = (message: MessageEvent) => { + socket.removeEventListener("message", receive); + if (typeof message.data !== "string") { + reject(new Error("expected a text answer")); + return; + } + resolve(record(JSON.parse(message.data))); + }; + socket.addEventListener("message", receive); + socket.send(JSON.stringify({ id, ...command })); + }); +} + +function event(name: string): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }); +} + +/** The repository mapping one proposal carries alongside its bytes. */ +const REPOSITORY = { + kind: "repository", + record: { + name: "app", + locatorFingerprint: "c".repeat(64), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/app", + }, +}; + +/** Stage the one piece the owner does not already hold. */ +async function stageNewContent(stub: ReturnType): Promise { + expect( + await send(stub, "stage-blob", { + command: "stage", + kind: "blob", + digest: NEXT_BLOB_ID, + bytes: encodeBase64(NEXT_BYTES), + }), + ).toMatchObject({ outcome: "performed" }); + const manifest = new TextEncoder().encode( + JSON.stringify({ version: 1, chunks: [{ hash: NEXT_BLOB_ID, size: NEXT_BYTES.length }] }), + ); + expect( + await send(stub, "stage-manifest", { + command: "stage", + kind: "manifest", + digest: sha256Hex(manifest), + bytes: encodeBase64(manifest), + }), + ).toMatchObject({ outcome: "performed" }); +} + +function commit(overrides: Record = {}): Record { + return { + command: "commit", + expectedWorkspaceRootId: ROOT_ID, + expectedJournalEventId: null, + publication: nextPublication(), + mappings: [REPOSITORY], + events: [event("published")], + ...overrides, + }; +} + +describe("publishing one proposal", () => { + it("adopts content, root, references, mapping, pointer and journal together", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + await stageNewContent(stub); + + const before = await on(stub, (owner) => owner.published()); + expect(before).toMatchObject({ currentRootId: ROOT_ID, roots: 1, events: [] }); + + const answer = await send(stub, "publish", commit()); + expect(answer).toMatchObject({ outcome: "performed" }); + const value = record(answer["value"]); + expect(value["workspaceRootId"]).toBe(NEXT_ROOT_ID); + expect(Array.isArray(value["journalEventIds"]) && value["journalEventIds"]).toHaveLength(1); + + const after = await on(stub, (owner) => owner.published()); + expect(after["currentRootId"]).toBe(NEXT_ROOT_ID); + // The old root stays retained; publication moves only the pointer. + expect(after["roots"]).toBe(2); + expect(after["repositories"]).toEqual([{ name: "app", checkout_path: "/app" }]); + // The journal row names the root this commit selected, not the one it + // started from. + expect(after["events"]).toEqual([expect.objectContaining({ workspace_root_id: NEXT_ROOT_ID })]); + // Content the owner already held was reused by identity rather than + // resent: two blobs exist, and the proposal only staged one. + expect(after["blobs"]).toBe(2); + expect(after["blobRefs"]).toBe(3); + + // And the new frontier reads back whole. + const frontier = record(record(await send(stub, "read", { command: "frontier" }))["value"]); + expect(frontier["workspaceRootId"]).toBe(NEXT_ROOT_ID); + expect( + record(await send(stub, "root", { command: "root", workspaceRootId: NEXT_ROOT_ID })), + ).toMatchObject({ outcome: "performed" }); + }); + + it("keeps the expected root current for a journal-only transaction", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + const answer = await send( + stub, + "journal-only", + commit({ publication: null, mappings: [], events: [event("noted")] }), + ); + expect(answer).toMatchObject({ outcome: "performed" }); + const after = await on(stub, (owner) => owner.published()); + expect(after["currentRootId"]).toBe(ROOT_ID); + expect(after["roots"]).toBe(1); + expect(after["events"]).toEqual([expect.objectContaining({ workspace_root_id: ROOT_ID })]); + }); + + it("commits an empty transaction without inventing a Workspace change", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + expect( + await send(stub, "empty", commit({ publication: null, mappings: [], events: [] })), + ).toMatchObject({ outcome: "performed", value: { workspaceRootId: ROOT_ID } }); + expect(await on(stub, (owner) => owner.published())).toMatchObject({ + currentRootId: ROOT_ID, + roots: 1, + events: [], + }); + }); + + it("rolls every category back when the transaction fails after applying", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + await stageNewContent(stub); + const before = await on(stub, (owner) => owner.published()); + + expect( + await on(stub, (owner) => + owner.failAfterApply(JSON.stringify({ id: "doomed", ...commit() })), + ), + ).toBe("rolled-back"); + + // Content, root, references, mapping, pointer and journal are all back + // where they were — and so is the retry decision, so the same id is free. + expect(await on(stub, (owner) => owner.published())).toEqual(before); + expect(await on(stub, (owner) => owner.scratch())).toMatchObject({ commands: 2 }); + expect(await send(stub, "doomed", commit())).toMatchObject({ outcome: "performed" }); + }); + + it("applies a lost-response retry exactly once, across eviction", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + await ask(socket, "stage-blob", { + command: "stage", + kind: "blob", + digest: NEXT_BLOB_ID, + bytes: encodeBase64(NEXT_BYTES), + }); + const manifestBytes = new TextEncoder().encode( + JSON.stringify({ version: 1, chunks: [{ hash: NEXT_BLOB_ID, size: NEXT_BYTES.length }] }), + ); + await ask(socket, "stage-manifest", { + command: "stage", + kind: "manifest", + digest: sha256Hex(manifestBytes), + bytes: encodeBase64(manifestBytes), + }); + + const first = await ask(socket, "once", commit()); + expect(first).toMatchObject({ outcome: "performed" }); + const published = await on(stub, (owner) => owner.published()); + + // The runner never saw that answer. The object is evicted, and the same + // healthy socket asks the same question again with the same id and the same + // canonical request. + await evictDurableObject(stub); + expect(await ask(socket, "once", commit())).toEqual(first); + + // One root, one journal row, one mapping — the retry returned the decision + // rather than doing the work a second time. + expect(await on(stub, (owner) => owner.published())).toEqual(published); + + // Reusing that id for a different request is not a retry. + expect(await ask(socket, "once", commit({ events: [event("something else")] }))).toMatchObject({ + outcome: "refused", + refusal: "command:duplicate-conflict", + }); + expect(await on(stub, (owner) => owner.published())).toEqual(published); + }); + + it("changes nothing when the frontier or the proposal is not what it claims", async () => { + const cases: Record> = { + "a root the run is not at": commit({ expectedWorkspaceRootId: `f${"0".repeat(63)}` }), + "an anchor the run is not at": commit({ expectedJournalEventId: "never-happened" }), + "an identity that is not the digest of its manifest": commit({ + publication: { ...nextPublication(), proposedWorkspaceRootId: `a${"1".repeat(63)}` }, + }), + "an inventory missing a piece the root names": commit({ + publication: { + ...nextPublication(), + content: (nextPublication()["content"] as unknown[]).slice(1), + }, + }), + "an inventory naming a piece the root does not": commit({ + publication: { + ...nextPublication(), + content: [ + ...(nextPublication()["content"] as Record[]), + { kind: "blob", digest: "e".repeat(64), size: 4 }, + ], + }, + }), + }; + + for (const [description, request] of Object.entries(cases)) { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + await stageNewContent(stub); + const before = await on(stub, (owner) => owner.published()); + const answer = await send(stub, "refused", request); + expect([description, answer["outcome"]]).toEqual([description, "refused"]); + expect([description, await on(stub, (owner) => owner.published())]).toEqual([ + description, + before, + ]); + } + }); + + it("refuses content this acquisition did not stage", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + // Nothing staged: the proposal names a piece the owner neither holds nor + // was given by this connection. + const before = await on(stub, (owner) => owner.published()); + expect(await send(stub, "unstaged", commit())).toMatchObject({ outcome: "refused" }); + expect(await on(stub, (owner) => owner.published())).toEqual(before); + }); + + it("refuses a mapping that would rewrite an established identity", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + await stageNewContent(stub); + expect(await send(stub, "first", commit())).toMatchObject({ outcome: "performed" }); + const published = await on(stub, (owner) => owner.published()); + + // The same Repository name, a different creation commit. Creation identity + // is immutable, so this is refused rather than allowed to overwrite it. + expect( + await send( + stub, + "second", + commit({ + expectedWorkspaceRootId: NEXT_ROOT_ID, + expectedJournalEventId: String( + (published["events"] as Record[])[0]?.["event_id"], + ), + publication: null, + mappings: [ + { ...REPOSITORY, record: { ...REPOSITORY.record, creationCommit: "1".repeat(40) } }, + ], + events: [], + }), + ), + ).toMatchObject({ outcome: "refused", refusal: "command:mapping-conflict" }); + expect(await on(stub, (owner) => owner.published())).toEqual(published); + }); + + it("grants a closed or foreign socket no publication", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await admit(stub); + await stageNewContent(stub); + const before = await on(stub, (owner) => owner.published()); + expect( + await on(stub, (owner) => + record(owner.sendAsStranger(JSON.stringify({ id: "foreign", ...commit() }))), + ), + ).toMatchObject({ outcome: "refused", refusal: "acquisition:foreign-connection" }); + expect(await on(stub, (owner) => owner.published())).toEqual(before); + }); +}); diff --git a/packages/workflow/tests/cloudflare/settle-parser.vitest.ts b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts index 1f97cb13a..ba37f3a56 100644 --- a/packages/workflow/tests/cloudflare/settle-parser.vitest.ts +++ b/packages/workflow/tests/cloudflare/settle-parser.vitest.ts @@ -128,7 +128,10 @@ describe("a root identity in a command", () => { sourceManifest: ROOT, }), "commit.expectedWorkspaceRootId": (root) => commit({ expectedWorkspaceRootId: root }), - "commit.proposedWorkspaceRootId": (root) => commit({ proposedWorkspaceRootId: root }), + "commit.publication.proposedWorkspaceRootId": (root) => + commit({ + publication: { proposedWorkspaceRootId: root, proposedManifest: "{}", content: [] }, + }), "settle.expectedWorkspaceRootId": (root) => settle({ expectedWorkspaceRootId: root }), }; @@ -138,7 +141,8 @@ describe("a root identity in a command", () => { command: "commit", expectedWorkspaceRootId: ROOT, expectedJournalEventId: null, - proposedWorkspaceRootId: ROOT, + publication: { proposedWorkspaceRootId: ROOT, proposedManifest: "{}", content: [] }, + mappings: [], events: [], ...overrides, }); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index bcf675972..98568056d 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -69,6 +69,65 @@ export const ROOT_MANIFEST = JSON.stringify({ ], }); export const ROOT_ID = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${ROOT_MANIFEST}`); + +/** A second root: the same tree with one more file, as a proposal would be. */ +export const NEXT_BYTES = new TextEncoder().encode("published by the runner"); +export const NEXT_BLOB_ID = sha256Hex(NEXT_BYTES); +export const NEXT_DOFS_MANIFEST = JSON.stringify({ + version: 1, + chunks: [{ hash: NEXT_BLOB_ID, size: NEXT_BYTES.length }], +}); +export const NEXT_MANIFEST_ID = sha256Hex(new TextEncoder().encode(NEXT_DOFS_MANIFEST)); +export const NEXT_ROOT_MANIFEST = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/NOTES.md", + kind: "file", + mode: 420, + mtime: 0, + size: NEXT_BYTES.length, + manifest: NEXT_MANIFEST_ID, + hardlink: null, + }, + { + path: "/README.md", + kind: "file", + mode: 420, + mtime: 0, + size: FILE_BYTES.length, + manifest: MANIFEST_ID, + hardlink: null, + }, + ], +}); +export const NEXT_ROOT_ID = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${NEXT_ROOT_MANIFEST}`); + +/** + * The proposal that publishes `NEXT_ROOT_ID`. + * + * Its inventory is the exact closure of the proposed manifest: both file + * manifests and both blobs, once each, in canonical order. One of each is + * already authoritative, which is what proves the owner reuses retained content + * by identity rather than requiring it to be sent again. + */ +export function nextPublication(): Record { + const content: { kind: string; digest: string; size: number }[] = [ + { kind: "blob", digest: BLOB_ID, size: FILE_BYTES.length }, + { kind: "blob", digest: NEXT_BLOB_ID, size: NEXT_BYTES.length }, + { kind: "manifest", digest: MANIFEST_ID, size: DOFS_MANIFEST.length }, + { kind: "manifest", digest: NEXT_MANIFEST_ID, size: NEXT_DOFS_MANIFEST.length }, + ]; + content.sort((left, right) => + `${left.kind}:${left.digest}` < `${right.kind}:${right.digest}` ? -1 : 1, + ); + return { + proposedWorkspaceRootId: NEXT_ROOT_ID, + proposedManifest: NEXT_ROOT_MANIFEST, + content, + }; +} const CREATED_AT = "2026-09-03T00:00:00.000Z"; export class ExecutorObject extends WorkflowOwnerObject { @@ -216,6 +275,63 @@ export class ExecutorObject extends WorkflowOwnerObject { ); } + /** + * Fail inside the owner transaction, after every category has been written. + * + * Injected rather than simulated: the claim is that the runtime's own + * transaction rolls content, roots, references, mappings, the pointer, the + * journal and the retry decision back together, and only a real failure + * inside a real `transactionSync()` can show that. + */ + failAfterApply(raw: string): string { + try { + return String( + this.transactions.run(this.ctx.storage, () => { + const socket = this.ctx.getWebSockets("executor")[0]; + if (socket === undefined) { + throw new Error("no live acquisition"); + } + const answer = this.onRunnerMessage(socket, RUN_ID, raw); + throw new Error(`forced failure after ${JSON.stringify(answer)}`); + }), + ); + } catch (error) { + return error instanceof Error && error.message.startsWith("forced failure") + ? "rolled-back" + : `threw:${String(error)}`; + } + } + + /** Everything a reader could observe about the published frontier. */ + published(): Record { + const state = this.ctx.storage.sql + .exec("SELECT current_root_id FROM workspace_state WHERE singleton_id = 1") + .toArray()[0]; + const roots = this.ctx.storage.sql + .exec("SELECT count(*) AS found FROM workspace_roots") + .toArray()[0]; + const events = this.ctx.storage.sql + .exec("SELECT event_id, workspace_root_id FROM journal_events ORDER BY sequence") + .toArray(); + const repositories = this.ctx.storage.sql + .exec("SELECT name, checkout_path FROM workspace_repositories ORDER BY name") + .toArray(); + const blobs = this.ctx.storage.sql + .exec("SELECT count(*) AS found FROM vfs_blob_bytes") + .toArray()[0]; + const refs = this.ctx.storage.sql + .exec("SELECT count(*) AS found FROM workspace_root_blob_refs") + .toArray()[0]; + return { + currentRootId: state?.["current_root_id"] ?? null, + roots: Number(roots?.["found"] ?? -1), + events, + repositories, + blobs: Number(blobs?.["found"] ?? -1), + blobRefs: Number(refs?.["found"] ?? -1), + }; + } + damageRetainedBlob(): void { this.ctx.storage.sql.exec( "UPDATE vfs_blob_bytes SET bytes = ?", From 4a0e9407ccfe0ef60d547bdc12860ba596f7656b Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 19:18:25 -0400 Subject: [PATCH 31/81] =?UTF-8?q?=F0=9F=A7=B7=20Hold=20a=20proposal=20to?= =?UTF-8?q?=20what=20it=20claims=20to=20be=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three ways the owner could publish something nobody proposed. A journal record was admitted for being a non-empty string, and the database proved only that it was JSON. `{}` could therefore be retained atomically and reported performed, and a later read — which parses events properly — would find history it cannot understand. The run would become unreplayable at the moment it was told it had committed. Every record is now parsed with the authoritative parser and serialized again, and must be the same bytes that arrived, terminating newline included. A nearly-right record is refused rather than normalized, because retaining a corrected one would retain something the runner never proposed. A repository mapping carried only its journal-safe record, so the fingerprint was written where the locator belongs and the run could not reattach to the repository it claimed to have retained. The mapping now carries the admitted locator beside the record, and the fingerprint must follow from it. Existing rows are compared on every field that establishes creation identity rather than a convenient subset — a partial comparison reported performed for proposals that disagreed with what an earlier execution established. A Repository or Worktree row names a checkout, and the row is only true if the Workspace this commit selects contains it. A mapping-only commit could retain a claim about a directory nothing put there, and the next execution would find the claim and not the files. New checkout mappings must now accompany the publication that creates them, a Worktree must name a Repository that exists or arrives with it, and one proposal naming one mapping twice is refused. And the starting root was taken from the pointer without being proved. A journal-only commit could append history against a current root whose content graph cannot be materialized — accepting a frontier nothing can restore. The same validator the reads use now proves it inside the commit transaction, and authoritative content is confirmed against its companion metadata rather than trusted and then preserved by an `ON CONFLICT DO NOTHING`. A proposal is not a licence to repair damage. Two things surfaced while proving it. Refusals that are answers about the request — a moved frontier, a duplicate id, a mapping conflict, a command this release does not implement — no longer close the connection; only a broken channel or damaged store does, because closing on an ordinary disagreement made the runner reconnect to be told the same thing. And the workerd tests that walked several object round trips were using sockets made inside the object, which do not survive it being reset; they now use real accepted connections, which is what the claims were always about. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/commands.ts | 35 ++- .../workflow/src/cloudflare/owner-reads.ts | 11 + packages/workflow/src/cloudflare/owner.ts | 27 +- packages/workflow/src/cloudflare/publish.ts | 183 ++++++++++- packages/workflow/src/composition/records.ts | 12 + .../workflow/src/deno/composition/locator.ts | 11 +- packages/workflow/src/remote/publication.ts | 14 +- .../cloudflare/executor-acquisition.vitest.ts | 13 +- .../tests/cloudflare/remote-owner.vitest.ts | 23 +- .../tests/cloudflare/remote-publish.vitest.ts | 286 ++++++++++++++---- .../cloudflare/support/executor-object.ts | 4 + 11 files changed, 528 insertions(+), 91 deletions(-) diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index aeb7a0bb0..5e38fd1dd 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -2,8 +2,10 @@ import { type DocumentExecutionCompletion, parseDocumentExecutionCompletion, } from "../storage/record.ts"; +import { parseDurableEvent, serializeDurableEvent } from "@executablemd/durable-streams"; import { SHA256 } from "../workspace/root-manifest.ts"; import { + locatorFingerprintOf, parseRepositoryRecord, parseWorktreeRecord, type RepositoryRecord, @@ -137,7 +139,7 @@ export interface ProposedPiece { /** One retained mapping the proposal carries, already parsed. */ export type ProposedMapping = - | { readonly kind: "repository"; readonly record: RepositoryRecord } + | { readonly kind: "repository"; readonly record: RepositoryRecord; readonly locator: string } | { readonly kind: "worktree"; readonly record: WorktreeRecord } | { readonly kind: "agent-session"; readonly record: AgentSessionRecord }; @@ -239,6 +241,20 @@ function kind(members: Map): ContentKind { return value; } +/** + * The exact serialized events a proposal appends. + * + * A record is not admitted because it is a non-empty string, and not because + * SQLite will accept it as JSON. It is parsed with the authoritative durable + * event parser and then serialized again, and the result must be the same bytes + * that arrived, terminating newline included. + * + * That round trip is the point. Retaining something that parses as JSON but not + * as an event would create history a later read cannot understand, and the run + * would become unreplayable at exactly the moment it was told it had committed. + * Re-encoding a nearly-right record would be worse: the owner would retain + * something the runner never proposed. + */ function eventRecords(value: unknown): string[] { if (!Array.isArray(value)) { throw new CommandError("malformed-member"); @@ -250,6 +266,10 @@ function eventRecords(value: unknown): string[] { if (typeof entry !== "string" || entry === "") { throw new CommandError("malformed-member"); } + const parsed = parseDurableEvent(entry); + if (!parsed.ok || serializeDurableEvent(parsed.value) !== entry) { + throw new CommandError("malformed-member"); + } return entry; }); } @@ -428,15 +448,22 @@ function mappings(value: unknown): ProposedMapping[] { } return value.map((entry) => { const members = object(entry); - closed(members, ["kind", "record"]); const which = members.get("kind"); + closed(members, which === "repository" ? ["kind", "record", "locator"] : ["kind", "record"]); const offered = members.get("record"); if (which === "repository") { const record = parseRepositoryRecord(offered); - if (record === undefined) { + const locator = members.get("locator"); + if (record === undefined || typeof locator !== "string" || locator === "") { throw new CommandError("malformed-member"); } - return { kind: which, record }; + // The record is journal-safe and names no locator; storage needs the + // admitted one. Requiring the fingerprint to follow from it is what stops + // a proposal retaining a locator that is not the one it was admitted for. + if (locatorFingerprintOf(locator) !== record.locatorFingerprint) { + throw new CommandError("malformed-member"); + } + return { kind: which, record, locator }; } if (which === "worktree") { const record = parseWorktreeRecord(offered); diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index 1b18d5eab..1f43833d3 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -240,6 +240,17 @@ function validatedBlob( * the transport piece-oriented rather than turning a validated root into one * unbounded answer. */ +/** + * Prove one retained root is complete, for a caller that is about to write. + * + * The same validator the reads use. Keeping the read boundary and the write + * boundary on one proof is what stops a root being publishable by one path and + * refused by the other. + */ +export function validateRetainedRoot(storage: OwnerStorage, rootId: string): void { + referencedRoot(storage, rootId); +} + function referencedRoot(storage: OwnerStorage, rootId: string): StoredRoot { if (!SHA256.test(rootId)) { return corrupt("a Workspace root identity is malformed"); diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index cf22f1c6f..c1dbf96d4 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -243,11 +243,32 @@ function mintAcquisitionId(): string { return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); } +/** + * Whether a refusal means the connection itself is finished. + * + * Two kinds of refusal reach here and they deserve opposite treatment. One says + * the channel or the store is not what it claims — a message that would not + * parse, an acquisition this socket does not hold, storage that is damaged — + * and carrying on would mean guessing what the other side meant. + * + * The other is an answer about the request. A duplicate id, a frontier that has + * moved, a mapping that disagrees with what is already retained, and a command + * this release does not implement are all decisions the runner can act on: read + * the frontier again, propose against it, or stop. Closing the connection on + * those would turn every ordinary disagreement into a lost acquisition and make + * the runner reconnect to be told the same thing. + */ +const ANSWERED: readonly string[] = [ + "command:duplicate-conflict", + "command:unavailable", + "command:stale-root", + "command:stale-journal", + "command:mapping-conflict", +]; + function fatal(answer: CommandResult): boolean { if (answer.outcome === "performed") { return false; } - return ( - answer.refusal !== "command:duplicate-conflict" && answer.refusal !== "command:unavailable" - ); + return !ANSWERED.includes(answer.refusal); } diff --git a/packages/workflow/src/cloudflare/publish.ts b/packages/workflow/src/cloudflare/publish.ts index 4a967ef32..2f3730fe9 100644 --- a/packages/workflow/src/cloudflare/publish.ts +++ b/packages/workflow/src/cloudflare/publish.ts @@ -47,8 +47,10 @@ import { WORKSPACE_ROOT_DOMAIN, } from "../workspace/root-manifest.ts"; import { decodeContentManifest } from "../workspace/content-manifest.ts"; +import { MAX_CONTENT_BYTES } from "./commands.ts"; import { sha256Hex } from "../workspace/sha256.ts"; import { CommandError, type CommitCommand, type ProposedMapping } from "./commands.ts"; +import { validateRetainedRoot } from "./owner-reads.ts"; import { bytesOf } from "./encoding.ts"; import { STAGING_TABLE } from "./private-schema.ts"; import type { OwnerStorage } from "./storage.ts"; @@ -86,13 +88,23 @@ function hexBytes(digest: string): Uint8Array { return bytes; } -/** The frontier as it is right now, read where the write will happen. */ +/** + * The frontier as it is right now, read where the write will happen. + * + * The current root is proved complete by the same validator the read boundary + * uses, not merely read out of the pointer. A commit accepts its starting root + * as the run's frontier, and accepting one whose content graph cannot be + * materialized would append history against a Workspace nothing can restore — + * a proposal is not a licence to repair, so damage is refused here rather than + * worked around. + */ function frontier(storage: OwnerStorage): { rootId: string; journalEventId: string | null } { const state = rows(storage, "SELECT current_root_id FROM workspace_state WHERE singleton_id = 1"); const current = state[0]?.["current_root_id"]; if (state.length !== 1 || typeof current !== "string") { return corrupt("the Workspace has no single current root"); } + validateRetainedRoot(storage, current); const last = rows( storage, "SELECT event_id FROM journal_events ORDER BY sequence DESC LIMIT 1", @@ -113,16 +125,24 @@ function resolve( ): { bytes: Uint8Array; authoritative: boolean } { const table = kind === "manifest" ? "vfs_manifests" : "vfs_blob_bytes"; const column = kind === "manifest" ? "encoded" : "bytes"; - const held = rows( + const authoritative = rows( storage, `SELECT ${column} AS content FROM ${table} WHERE lower(hex(hash)) = ?`, digest, - )[0]; + ); + if (authoritative.length > 1) { + return corrupt("retained content is stored more than once under one identity"); + } + const held = authoritative[0]; if (held !== undefined) { const bytes = bytesOf(held["content"]); - if (sha256Hex(bytes) !== digest) { + if (bytes.length > MAX_CONTENT_BYTES || sha256Hex(bytes) !== digest) { return corrupt("retained content disagrees with the identity it is stored under"); } + // The companion row is part of the same fact. A size that disagrees with + // the bytes is damage, and adopting a proposal over it would publish a root + // whose content the read path refuses. + confirmCompanion(storage, kind, digest, bytes); return { bytes, authoritative: true }; } const staged = rows( @@ -145,6 +165,39 @@ function resolve( return { bytes, authoritative: false }; } +/** + * The metadata stored beside one content identity, confirmed rather than fixed. + * + * A manifest's recorded size must equal what its chunks add up to; a blob's + * recorded size must equal its bytes; and a blob's byte row and its `vfs_blobs` + * row must both exist. Any disagreement is existing damage, refused here rather + * than silently repaired by an `ON CONFLICT DO NOTHING` that leaves the wrong + * row in place. + */ +function confirmCompanion( + storage: OwnerStorage, + kind: "manifest" | "blob", + digest: string, + bytes: Uint8Array, +): void { + if (kind === "manifest") { + const row = rows( + storage, + "SELECT size FROM vfs_manifests WHERE lower(hex(hash)) = ?", + digest, + )[0]; + const decoded = decodeContentManifest(bytes, corrupt); + if (row === undefined || Number(row["size"]) !== decoded.size) { + return corrupt("a retained manifest disagrees with its recorded size"); + } + return; + } + const row = rows(storage, "SELECT size FROM vfs_blobs WHERE lower(hex(hash)) = ?", digest)[0]; + if (row === undefined || Number(row["size"]) !== bytes.length) { + return corrupt("a retained blob disagrees with its recorded size"); + } +} + /** * Apply one proposal, entirely, inside the caller's open transaction. * @@ -170,8 +223,24 @@ export function applyCommit( ? command.expectedWorkspaceRootId : publish(storage, acquisitionId, command); + const named = new Set(); + for (const mapping of command.mappings) { + const identity = + mapping.kind === "worktree" + ? `worktree:${mapping.record.repositoryName}/${mapping.record.name}` + : `${mapping.kind}:${mapping.kind === "repository" ? mapping.record.name : mapping.record.sessionKey}`; + if (named.has(identity)) { + // One proposal naming one mapping twice cannot be applied once and is not + // two mappings either. + throw new CommandError("mapping-conflict"); + } + named.add(identity); + } + const selectedEntries = directoriesOf( + command.publication === null ? undefined : command.publication.proposedManifest, + ); for (const mapping of command.mappings) { - applyMapping(storage, mapping); + applyMapping(storage, mapping, selectedEntries); } const journalEventIds: string[] = []; @@ -319,6 +388,12 @@ function publish(storage: OwnerStorage, acquisitionId: string, command: CommitCo return corrupt("a retained Workspace root disagrees with the identity it is stored under"); } + // Whether it was just written or was already there, the root the pointer is + // about to name is proved to be a complete materializable root — the same + // proof the read boundary applies, so a root cannot be publishable by one + // path and refused by the other. + validateRetainedRoot(storage, proposal.proposedWorkspaceRootId); + // Compare-and-set. Two commits racing one frontier cannot both move it. storage.sql.exec( "UPDATE workspace_state SET current_root_id = ? WHERE singleton_id = 1 AND current_root_id = ?", @@ -336,29 +411,85 @@ function publish(storage: OwnerStorage, acquisitionId: string, command: CommitCo } /** - * One retained mapping, inserted or confirmed. + * Where a mapping's checkout has to exist, for the mapping to be true. + * + * A Repository or Worktree row names a checkout path, and the row is only + * meaningful if the Workspace this commit selects actually contains it. A + * mapping-only commit that invented a checkout would retain a claim about a + * directory nothing put there, and the next execution would find the claim and + * not the files. + */ +function requirePlacement( + mapping: ProposedMapping, + selectedEntries: ReadonlySet | undefined, +): void { + if (mapping.kind === "agent-session") { + return; + } + if (selectedEntries === undefined) { + // No publication accompanies this commit, so nothing can have created the + // checkout. An exact confirmation of an already-retained mapping is still + // admissible — that is decided below, once the existing row is read. + return; + } + if (!selectedEntries.has(mapping.record.checkoutPath)) { + throw new CommandError("mapping-conflict"); + } +} + +/** Every directory the selected root contains, for placement checks. */ +function directoriesOf(manifest: string | undefined): ReadonlySet | undefined { + if (manifest === undefined) { + return undefined; + } + const parsed = parseWorkspaceRootManifest(manifest, () => { + throw new CommandError("malformed-member"); + }); + return new Set( + parsed.entries.flatMap((entry) => (entry.kind === "directory" ? [entry.path] : [])), + ); +} + +function sameText(row: Record, column: string, expected: string | null): boolean { + const value = row[column]; + return expected === null ? value === null : value === expected; +} + +/** + * One retained mapping, inserted or confirmed in full. * - * Creation identity is immutable: a second proposal naming the same Repository - * must describe the same Repository, and one that does not is refused rather - * than allowed to rewrite what an earlier execution established. + * Creation identity is immutable, so an existing row is compared on every field + * that establishes it — not on a convenient subset. A partial comparison would + * report performed for a proposal that disagrees with what an earlier execution + * established, and the disagreement would only surface later, as a checkout + * that is not what its record says. */ -function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { +function applyMapping( + storage: OwnerStorage, + mapping: ProposedMapping, + selectedEntries: ReadonlySet | undefined, +): void { if (mapping.kind === "repository") { const record = mapping.record; const held = rows( storage, - `SELECT name, locator_fingerprint, requested_base, creation_commit, primary_branch, + `SELECT locator, locator_fingerprint, requested_base, creation_commit, primary_branch, object_format, checkout_path FROM workspace_repositories WHERE name = ?`, record.name, )[0]; if (held === undefined) { + requirePlacement(mapping, selectedEntries); + if (selectedEntries === undefined) { + // A new checkout mapping with no Workspace publication to create it. + throw new CommandError("mapping-conflict"); + } storage.sql.exec( `INSERT INTO workspace_repositories (name, locator, locator_fingerprint, requested_base, creation_commit, primary_branch, object_format, checkout_path) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, record.name, - record.locatorFingerprint, + mapping.locator, record.locatorFingerprint, record.requestedBase, record.creationCommit, @@ -369,7 +500,9 @@ function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { return; } if ( + held["locator"] !== mapping.locator || held["locator_fingerprint"] !== record.locatorFingerprint || + !sameText(held, "requested_base", record.requestedBase) || held["creation_commit"] !== record.creationCommit || held["primary_branch"] !== record.primaryBranch || held["object_format"] !== record.objectFormat || @@ -384,12 +517,26 @@ function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { const record = mapping.record; const held = rows( storage, - `SELECT requested_branch, creation_commit, checkout_path FROM workspace_worktrees - WHERE repository_name = ? AND name = ?`, + `SELECT requested_branch, requested_base, creation_commit, checkout_path + FROM workspace_worktrees WHERE repository_name = ? AND name = ?`, record.repositoryName, record.name, )[0]; if (held === undefined) { + // A Worktree exists inside a Repository. One that named none would be a + // checkout belonging to nothing. + const repository = rows( + storage, + "SELECT name FROM workspace_repositories WHERE name = ?", + record.repositoryName, + )[0]; + if (repository === undefined) { + throw new CommandError("mapping-conflict"); + } + requirePlacement(mapping, selectedEntries); + if (selectedEntries === undefined) { + throw new CommandError("mapping-conflict"); + } storage.sql.exec( `INSERT INTO workspace_worktrees (repository_name, name, requested_branch, requested_base, creation_commit, checkout_path) @@ -405,6 +552,7 @@ function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { } if ( held["requested_branch"] !== record.requestedBranch || + !sameText(held, "requested_base", record.requestedBase) || held["creation_commit"] !== record.creationCommit || held["checkout_path"] !== record.checkoutPath ) { @@ -416,7 +564,8 @@ function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { const record = mapping.record; const held = rows( storage, - `SELECT provider, agent_command, session_identity, assertion_kind, assertion_value + `SELECT provider, agent_command, session_identity, policy, + assertion_kind, assertion_value, created_at FROM agent_sessions WHERE session_key = ?`, record.sessionKey, )[0]; @@ -441,8 +590,10 @@ function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { held["provider"] !== record.provider || held["agent_command"] !== record.agentCommand || held["session_identity"] !== record.sessionIdentity || + held["policy"] !== record.policy || held["assertion_kind"] !== record.assertion.kind || - held["assertion_value"] !== record.assertion.value + held["assertion_value"] !== record.assertion.value || + held["created_at"] !== record.createdAt ) { throw new CommandError("mapping-conflict"); } diff --git a/packages/workflow/src/composition/records.ts b/packages/workflow/src/composition/records.ts index b7ebb87c0..866129ea5 100644 --- a/packages/workflow/src/composition/records.ts +++ b/packages/workflow/src/composition/records.ts @@ -20,6 +20,7 @@ * a record at all. */ +import { sha256Hex } from "../workspace/sha256.ts"; import type { Json } from "@executablemd/durable-streams"; import { members, optionalText, text } from "./parse.ts"; @@ -101,6 +102,17 @@ const WORKTREE_MEMBERS = [ "checkoutPath", ] as const; +/** + * The stable name an admitted locator is known by everywhere but its own column. + * + * Shared because both hosts retain it and both must derive it identically: a + * fingerprint is what a journal event carries in place of the locator, and two + * derivations would be two names for one repository. + */ +export function locatorFingerprintOf(locator: string): string { + return sha256Hex(locator); +} + export function parseObjectFormat(value: unknown): GitObjectFormat | undefined { return value === "sha1" || value === "sha256" ? value : undefined; } diff --git a/packages/workflow/src/deno/composition/locator.ts b/packages/workflow/src/deno/composition/locator.ts index 7b9198d60..a92faaae9 100644 --- a/packages/workflow/src/deno/composition/locator.ts +++ b/packages/workflow/src/deno/composition/locator.ts @@ -91,7 +91,10 @@ export function admitLocator(locator: string): string | undefined { return undefined; } -/** The stable name an admitted locator is known by everywhere but its own column. */ -export function locatorFingerprint(locator: string): string { - return createHash("sha256").update(locator, "utf8").digest("hex"); -} +/** + * The stable name an admitted locator is known by everywhere but its own column. + * + * The derivation is the shared rule: both hosts retain this fingerprint and a + * second derivation would be two names for one repository. + */ +export { locatorFingerprintOf as locatorFingerprint } from "../../composition/records.ts"; diff --git a/packages/workflow/src/remote/publication.ts b/packages/workflow/src/remote/publication.ts index d8d95d557..73f157645 100644 --- a/packages/workflow/src/remote/publication.ts +++ b/packages/workflow/src/remote/publication.ts @@ -58,6 +58,18 @@ export interface WorkspacePublication { * the run established; it never contacts or impersonates an Agent provider. */ export type RetainedMapping = - | { readonly kind: "repository"; readonly record: RepositoryRecord } + | { + readonly kind: "repository"; + readonly record: RepositoryRecord; + /** + * The admitted locator, which the record deliberately does not carry. + * + * A record is journal-safe and names only the fingerprint, because a + * locator can carry a credential and a journal is history. Storage needs + * the real thing to reattach, so it travels beside the record and its + * fingerprint must follow from it. + */ + readonly locator: string; + } | { readonly kind: "worktree"; readonly record: WorktreeRecord } | { readonly kind: "agent-session"; readonly record: AgentSessionRecord }; diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts index 6dbb1c84c..86c14bfe2 100644 --- a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -10,6 +10,7 @@ import { env, runInDurableObject } from "cloudflare:test"; import { beforeAll, describe, expect, it } from "vitest"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; import type { ExecutorObject } from "./support/executor-object.ts"; import { POLICY, VALID_CLAIMS } from "./support/executor-object.ts"; import { generateKeys, signToken, tamper, type TestKeys } from "./support/tokens.ts"; @@ -242,6 +243,16 @@ describe("admitting an executor", () => { }); }); +/** A record the owner will accept: exactly what the serializer produces. */ +function serializedEvent(name: string): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }); +} + describe("holding an acquisition", () => { it("refuses a second healthy executor rather than following it", async () => { const stub = executor(); @@ -350,7 +361,7 @@ describe("reading a runner command", () => { content: [], }, mappings: [], - events: ["event-1"], + events: [serializedEvent("read whole")], }); // The shape is read — an unknown member or a malformed root would refuse // differently — and then declined on its merits: this run's frontier is not diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index 1557fc251..5c50cf934 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -233,8 +233,10 @@ describe("the remote owner protocol", () => { for (let index = 0; index < 129; index += 1) { await on(stub, (owner) => owner.appendJournal(`event-${index}`, `event ${index}`)); } - await admit(stub); - const frontier = record((await send(stub, "frontier", { command: "frontier" }))["value"]); + // A real accepted connection: this reads across several object round trips + // and a pair socket does not survive the object being reset between them. + const socket = await connect(stub); + const frontier = record((await ask(socket, "frontier", { command: "frontier" }))["value"]); expect(frontier["workspaceRootId"]).toBe(ROOT_ID); expect(frontier["journalEventId"]).toBe("event-128"); expect(record(frontier["record"])["runId"]).toBe(RUN_ID); @@ -242,7 +244,7 @@ describe("the remote owner protocol", () => { await on(stub, (owner) => owner.appendJournal("event-later", "later")); const first = record( ( - await send(stub, "journal-1", { + await ask(socket, "journal-1", { command: "journal", anchorEventId: "event-128", afterEventId: null, @@ -253,7 +255,7 @@ describe("the remote owner protocol", () => { expect(first["done"]).toBe(false); const second = record( ( - await send(stub, "journal-2", { + await ask(socket, "journal-2", { command: "journal", anchorEventId: "event-128", afterEventId: "event-127", @@ -269,14 +271,17 @@ describe("the remote owner protocol", () => { it("returns only content referenced by one validated root", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); - expect(await send(stub, "root", { command: "root", workspaceRootId: ROOT_ID })).toEqual({ + // A real accepted connection: a `WebSocketPair` made inside the object does + // not survive the object being reset between calls, and this test reads + // across several of them. + const socket = await connect(stub); + expect(await ask(socket, "root", { command: "root", workspaceRootId: ROOT_ID })).toEqual({ id: "root", outcome: "performed", value: { workspaceRootId: ROOT_ID, manifest: ROOT_MANIFEST }, }); expect( - await send(stub, "manifest", { + await ask(socket, "manifest", { command: "content", workspaceRootId: ROOT_ID, kind: "manifest", @@ -294,7 +299,7 @@ describe("the remote owner protocol", () => { }, }); expect( - await send(stub, "blob", { + await ask(socket, "blob", { command: "content", workspaceRootId: ROOT_ID, kind: "blob", @@ -307,7 +312,7 @@ describe("the remote owner protocol", () => { owner.addUnreferencedBlob(new TextEncoder().encode("orphan")), ); expect( - await send(stub, "orphan", { + await ask(socket, "orphan", { command: "content", workspaceRootId: ROOT_ID, kind: "blob", diff --git a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts index 69adf37f2..deead8242 100644 --- a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts @@ -18,6 +18,7 @@ import { env, evictDurableObject, runInDurableObject } from "cloudflare:test"; import { beforeAll, describe, expect, it } from "vitest"; import { serializeDurableEvent } from "@executablemd/durable-streams"; import type { ExecutorObject } from "./support/executor-object.ts"; +import type { ExecutorObject as _ExecutorObject } from "./support/executor-object.ts"; import { NEXT_BLOB_ID, NEXT_BYTES, @@ -30,6 +31,7 @@ import { } from "./support/executor-object.ts"; import { encodeBase64 } from "../../src/cloudflare/encoding.ts"; import { sha256Hex } from "../../src/workspace/sha256.ts"; +import { locatorFingerprintOf } from "../../src/composition/records.ts"; import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; let unique = 0; @@ -69,14 +71,6 @@ async function admit(stub: ReturnType): Promise { ); } -function send( - stub: ReturnType, - id: string, - command: Record, -): Promise> { - return on(stub, (owner) => record(owner.send(1, JSON.stringify({ id, ...command })))); -} - /** * A real accepted connection, which is what survives eviction. * @@ -137,11 +131,14 @@ function event(name: string): string { } /** The repository mapping one proposal carries alongside its bytes. */ +const LOCATOR = "https://git.example.invalid/octo/app.git"; + const REPOSITORY = { kind: "repository", + locator: LOCATOR, record: { name: "app", - locatorFingerprint: "c".repeat(64), + locatorFingerprint: locatorFingerprintOf(LOCATOR), requestedBase: null, creationCommit: "9".repeat(40), primaryBranch: "main", @@ -150,27 +147,23 @@ const REPOSITORY = { }, }; -/** Stage the one piece the owner does not already hold. */ -async function stageNewContent(stub: ReturnType): Promise { - expect( - await send(stub, "stage-blob", { - command: "stage", - kind: "blob", - digest: NEXT_BLOB_ID, - bytes: encodeBase64(NEXT_BYTES), - }), - ).toMatchObject({ outcome: "performed" }); +/** Stage the one missing piece over an accepted connection. */ +async function stageThrough(socket: WebSocket): Promise { + await ask(socket, "stage-blob", { + command: "stage", + kind: "blob", + digest: NEXT_BLOB_ID, + bytes: encodeBase64(NEXT_BYTES), + }); const manifest = new TextEncoder().encode( JSON.stringify({ version: 1, chunks: [{ hash: NEXT_BLOB_ID, size: NEXT_BYTES.length }] }), ); - expect( - await send(stub, "stage-manifest", { - command: "stage", - kind: "manifest", - digest: sha256Hex(manifest), - bytes: encodeBase64(manifest), - }), - ).toMatchObject({ outcome: "performed" }); + await ask(socket, "stage-manifest", { + command: "stage", + kind: "manifest", + digest: sha256Hex(manifest), + bytes: encodeBase64(manifest), + }); } function commit(overrides: Record = {}): Record { @@ -189,14 +182,14 @@ describe("publishing one proposal", () => { it("adopts content, root, references, mapping, pointer and journal together", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); - await stageNewContent(stub); + const socket = await connect(stub); + await stageThrough(socket); const before = await on(stub, (owner) => owner.published()); expect(before).toMatchObject({ currentRootId: ROOT_ID, roots: 1, events: [] }); - const answer = await send(stub, "publish", commit()); - expect(answer).toMatchObject({ outcome: "performed" }); + const answer = await ask(socket, "publish", commit()); + expect(answer).toEqual(expect.objectContaining({ outcome: "performed" })); const value = record(answer["value"]); expect(value["workspaceRootId"]).toBe(NEXT_ROOT_ID); expect(Array.isArray(value["journalEventIds"]) && value["journalEventIds"]).toHaveLength(1); @@ -215,19 +208,19 @@ describe("publishing one proposal", () => { expect(after["blobRefs"]).toBe(3); // And the new frontier reads back whole. - const frontier = record(record(await send(stub, "read", { command: "frontier" }))["value"]); + const frontier = record(record(await ask(socket, "read", { command: "frontier" }))["value"]); expect(frontier["workspaceRootId"]).toBe(NEXT_ROOT_ID); expect( - record(await send(stub, "root", { command: "root", workspaceRootId: NEXT_ROOT_ID })), + record(await ask(socket, "root", { command: "root", workspaceRootId: NEXT_ROOT_ID })), ).toMatchObject({ outcome: "performed" }); }); it("keeps the expected root current for a journal-only transaction", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); - const answer = await send( - stub, + const socket = await connect(stub); + const answer = await ask( + socket, "journal-only", commit({ publication: null, mappings: [], events: [event("noted")] }), ); @@ -241,9 +234,9 @@ describe("publishing one proposal", () => { it("commits an empty transaction without inventing a Workspace change", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); + const socket = await connect(stub); expect( - await send(stub, "empty", commit({ publication: null, mappings: [], events: [] })), + await ask(socket, "empty", commit({ publication: null, mappings: [], events: [] })), ).toMatchObject({ outcome: "performed", value: { workspaceRootId: ROOT_ID } }); expect(await on(stub, (owner) => owner.published())).toMatchObject({ currentRootId: ROOT_ID, @@ -255,8 +248,8 @@ describe("publishing one proposal", () => { it("rolls every category back when the transaction fails after applying", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); - await stageNewContent(stub); + const socket = await connect(stub); + await stageThrough(socket); const before = await on(stub, (owner) => owner.published()); expect( @@ -269,7 +262,7 @@ describe("publishing one proposal", () => { // where they were — and so is the retry decision, so the same id is free. expect(await on(stub, (owner) => owner.published())).toEqual(before); expect(await on(stub, (owner) => owner.scratch())).toMatchObject({ commands: 2 }); - expect(await send(stub, "doomed", commit())).toMatchObject({ outcome: "performed" }); + expect(await ask(socket, "doomed", commit())).toMatchObject({ outcome: "performed" }); }); it("applies a lost-response retry exactly once, across eviction", async () => { @@ -341,10 +334,10 @@ describe("publishing one proposal", () => { for (const [description, request] of Object.entries(cases)) { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); - await stageNewContent(stub); + const socket = await connect(stub); + await stageThrough(socket); const before = await on(stub, (owner) => owner.published()); - const answer = await send(stub, "refused", request); + const answer = await ask(socket, "refused", request); expect([description, answer["outcome"]]).toEqual([description, "refused"]); expect([description, await on(stub, (owner) => owner.published())]).toEqual([ description, @@ -356,27 +349,27 @@ describe("publishing one proposal", () => { it("refuses content this acquisition did not stage", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); + const socket = await connect(stub); // Nothing staged: the proposal names a piece the owner neither holds nor // was given by this connection. const before = await on(stub, (owner) => owner.published()); - expect(await send(stub, "unstaged", commit())).toMatchObject({ outcome: "refused" }); + expect(await ask(socket, "unstaged", commit())).toMatchObject({ outcome: "refused" }); expect(await on(stub, (owner) => owner.published())).toEqual(before); }); it("refuses a mapping that would rewrite an established identity", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); - await stageNewContent(stub); - expect(await send(stub, "first", commit())).toMatchObject({ outcome: "performed" }); + const socket = await connect(stub); + await stageThrough(socket); + expect(await ask(socket, "first", commit())).toMatchObject({ outcome: "performed" }); const published = await on(stub, (owner) => owner.published()); // The same Repository name, a different creation commit. Creation identity // is immutable, so this is refused rather than allowed to overwrite it. expect( - await send( - stub, + await ask( + socket, "second", commit({ expectedWorkspaceRootId: NEXT_ROOT_ID, @@ -394,11 +387,198 @@ describe("publishing one proposal", () => { expect(await on(stub, (owner) => owner.published())).toEqual(published); }); + it("retains only records that are exactly what the serializer produced", async () => { + // A record the database will accept as JSON is not a durable event. One + // retained here would be history a later read cannot parse, and the run + // would become unreplayable at the moment it was told it had committed. + const valid = event("real"); + const cases: Record = { + "JSON that is not an event": "{}", + "an event without its terminating newline": valid.trimEnd(), + "a noncanonical re-encoding": `${JSON.stringify(JSON.parse(valid.trimEnd()), null, 1)}\n`, + "not JSON at all": "event-1", + }; + for (const [description, record_] of Object.entries(cases)) { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + const before = await on(stub, (owner) => owner.published()); + const answer = await ask( + socket, + "bad-event", + commit({ publication: null, mappings: [], events: [record_] }), + ); + // The id is empty because the command never finished parsing: an id is + // echoed once the request has been read, and this one was not. + expect([description, answer]).toEqual([ + description, + { id: "", outcome: "refused", refusal: "command:malformed-member" }, + ]); + expect([description, await on(stub, (owner) => owner.published())]).toEqual([ + description, + before, + ]); + // Nothing recorded a decision for work that never happened. A malformed + // member is a broken channel rather than an answer, so the connection is + // gone too — which is why the ledger is read through the object. + expect([description, await on(stub, (owner) => owner.scratch())]).toEqual([ + description, + { commands: 0, staged: 0 }, + ]); + } + }); + + it("refuses a mapping that disagrees with retained identity in any field", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + // A real accepted connection: this walks several cases and a pair socket + // does not survive the object being reset between them. + const socket = await connect(stub); + await stageThrough(socket); + expect(await ask(socket, "first", commit())).toMatchObject({ outcome: "performed" }); + const published = await on(stub, (owner) => owner.published()); + const anchor = String((published["events"] as Record[])[0]?.["event_id"]); + + // Every field that establishes creation identity, one at a time. A partial + // comparison would report performed for a proposal that disagrees with what + // an earlier execution established. + const conflicts: Record> = { + "a different locator, with its own fingerprint": { + kind: "repository", + locator: "https://git.example.invalid/other.git", + record: { + ...REPOSITORY.record, + locatorFingerprint: locatorFingerprintOf("https://git.example.invalid/other.git"), + }, + }, + "a different requested base": { + ...REPOSITORY, + record: { ...REPOSITORY.record, requestedBase: "release" }, + }, + "a different creation commit": { + ...REPOSITORY, + record: { ...REPOSITORY.record, creationCommit: "1".repeat(40) }, + }, + "a different primary branch": { + ...REPOSITORY, + record: { ...REPOSITORY.record, primaryBranch: "trunk" }, + }, + "a different checkout path": { + ...REPOSITORY, + record: { ...REPOSITORY.record, checkoutPath: "/elsewhere" }, + }, + }; + for (const [description, mapping] of Object.entries(conflicts)) { + const answer = await ask( + socket, + `conflict-${description}`, + commit({ + expectedWorkspaceRootId: NEXT_ROOT_ID, + expectedJournalEventId: anchor, + publication: null, + mappings: [mapping], + events: [], + }), + ); + expect([description, answer["refusal"]]).toEqual([description, "command:mapping-conflict"]); + expect([description, await on(stub, (owner) => owner.published())]).toEqual([ + description, + published, + ]); + } + }); + + it("refuses a new checkout mapping that no publication creates", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + const before = await on(stub, (owner) => owner.published()); + + // A mapping-only commit would retain a claim about a directory nothing put + // there, and the next execution would find the claim and not the files. + expect( + await ask( + socket, + "no-publication", + commit({ publication: null, mappings: [REPOSITORY], events: [] }), + ), + ).toMatchObject({ refusal: "command:mapping-conflict" }); + expect(await on(stub, (owner) => owner.published())).toEqual(before); + + // A Worktree whose Repository is neither retained nor proposed belongs to + // nothing. + await stageThrough(socket); + expect( + await ask( + socket, + "orphan-worktree", + commit({ + mappings: [ + { + kind: "worktree", + record: { + repositoryName: "absent", + name: "feature", + requestedBranch: "feature", + requestedBase: null, + creationCommit: "2".repeat(40), + checkoutPath: "/app", + }, + }, + ], + }), + ), + ).toMatchObject({ refusal: "command:mapping-conflict" }); + expect(await on(stub, (owner) => owner.published())).toEqual(before); + }); + + it("refuses one proposal naming one mapping twice", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + await stageThrough(socket); + const before = await on(stub, (owner) => owner.published()); + expect( + await ask(socket, "duplicate", commit({ mappings: [REPOSITORY, REPOSITORY] })), + ).toMatchObject({ refusal: "command:mapping-conflict" }); + expect(await on(stub, (owner) => owner.published())).toEqual(before); + }); + + it("refuses to append history over a current root that is damaged", async () => { + // A commit accepts its starting root as the run's frontier. One whose graph + // cannot be materialized is not a frontier, and a proposal is not a licence + // to repair it. + const damage: Record void> = { + "a blob whose bytes are not its identity": (owner) => owner.damageRetainedBlob(), + "a blob whose recorded size is wrong": (owner) => owner.damageBlobSize(), + "a manifest whose recorded size is wrong": (owner) => owner.damageManifestSize(), + "a reference no manifest names": (owner) => + owner.addExtraBlobReference(new TextEncoder().encode("unaccounted for")), + }; + for (const [description, arrange] of Object.entries(damage)) { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await on(stub, arrange); + const socket = await connect(stub); + const before = await on(stub, (owner) => owner.published()); + const answer = await ask( + socket, + "over-damage", + commit({ publication: null, mappings: [], events: [event("noted")] }), + ); + expect([description, answer["refusal"]]).toEqual([description, "storage:corrupt"]); + expect([description, await on(stub, (owner) => owner.published())]).toEqual([ + description, + before, + ]); + } + }); + it("grants a closed or foreign socket no publication", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); - await admit(stub); - await stageNewContent(stub); + const socket = await connect(stub); + await stageThrough(socket); const before = await on(stub, (owner) => owner.published()); expect( await on(stub, (owner) => diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 98568056d..4788737b5 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -100,6 +100,10 @@ export const NEXT_ROOT_MANIFEST = JSON.stringify({ manifest: MANIFEST_ID, hardlink: null, }, + // The checkout a Repository mapping claims, in canonical byte order — a + // mapping whose directory the proposed root does not contain is a record + // about files nobody wrote. + { path: "/app", kind: "directory", mode: 493, mtime: 0 }, ], }); export const NEXT_ROOT_ID = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${NEXT_ROOT_MANIFEST}`); From f14349390cd02cba56c987fc5ddf8e7a9fff3fdd Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 19:33:47 -0400 Subject: [PATCH 32/81] =?UTF-8?q?=F0=9F=97=9D=EF=B8=8F=20Admit=20a=20locat?= =?UTF-8?q?or,=20and=20decide=20the=20mappings=20before=20writing=20any=20?= =?UTF-8?q?(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three ways the owner could retain something it had not actually agreed to. A repository locator was admitted for having a fingerprint that matched itself. That says the two values agree with each other and nothing about whether the locator is one this system would ever hand to Git — so an authenticated proposal could retain a credential-bearing URL, an executable transport form, a query carrying a token, or a relative path, each with a perfectly consistent fingerprint. The closed allowlist the local host applies before Git sees a locator is now shared and applied by the owner before the fingerprint is compared at all. It is an allowlist rather than a search for bad shapes because Git's locator grammar reaches well past URLs, and a second copy of an allowlist is the copy that ends up longer. A Worktree's parent was looked for while its own row was being written, so `[repository, worktree]` committed and `[worktree, repository]` did not — the same transaction, refused for the order its mappings happened to arrive in. Identity, duplication, parent relationships and checkout placement are properties of the proposal rather than of one mapping, so they are all settled against the whole collection before anything is written, and application then runs parents before children. Which one a proposal lists first is the owner's problem, not something a runner should have to arrange to suit a schema. And a blob's metadata row with no bytes beside it was treated as absent rather than as damaged, so staging would supply the missing bytes and a proposal would quietly complete a half-written identity. Which durable state won then depended on the write path instead of on what the store actually holds. A partial identity is now storage corruption, refused before staging is consulted. The evidence is on real workerd: five locator forms that must never be retained, a positive round trip proving the row holds the repository rather than a digest of it, a Repository and its Worktree accepted in both orders with the same result, and a metadata row without its bytes refusing without repairing anything or recording a decision. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/commands.ts | 17 +- packages/workflow/src/cloudflare/publish.ts | 192 +++++++++++++----- packages/workflow/src/composition/locator.ts | 111 ++++++++++ packages/workflow/src/composition/records.ts | 12 -- .../workflow/src/deno/composition/locator.ts | 84 +------- .../tests/cloudflare/remote-publish.vitest.ts | 97 ++++++++- .../cloudflare/support/executor-object.ts | 17 ++ 7 files changed, 374 insertions(+), 156 deletions(-) create mode 100644 packages/workflow/src/composition/locator.ts diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 5e38fd1dd..f7d84786c 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -4,8 +4,8 @@ import { } from "../storage/record.ts"; import { parseDurableEvent, serializeDurableEvent } from "@executablemd/durable-streams"; import { SHA256 } from "../workspace/root-manifest.ts"; +import { admitLocator, locatorFingerprintOf } from "../composition/locator.ts"; import { - locatorFingerprintOf, parseRepositoryRecord, parseWorktreeRecord, type RepositoryRecord, @@ -453,14 +453,17 @@ function mappings(value: unknown): ProposedMapping[] { const offered = members.get("record"); if (which === "repository") { const record = parseRepositoryRecord(offered); - const locator = members.get("locator"); - if (record === undefined || typeof locator !== "string" || locator === "") { + const offeredLocator = members.get("locator"); + if (record === undefined || typeof offeredLocator !== "string") { throw new CommandError("malformed-member"); } - // The record is journal-safe and names no locator; storage needs the - // admitted one. Requiring the fingerprint to follow from it is what stops - // a proposal retaining a locator that is not the one it was admitted for. - if (locatorFingerprintOf(locator) !== record.locatorFingerprint) { + // Admitted first, by the same closed allowlist the local host uses. A + // matching fingerprint says the two values agree with each other; it says + // nothing about whether the locator is one this system will ever hand to + // Git, and an authenticated proposal must not be able to retain a + // credential-bearing URL or an executable transport form. + const locator = admitLocator(offeredLocator); + if (locator === undefined || locatorFingerprintOf(locator) !== record.locatorFingerprint) { throw new CommandError("malformed-member"); } return { kind: which, record, locator }; diff --git a/packages/workflow/src/cloudflare/publish.ts b/packages/workflow/src/cloudflare/publish.ts index 2f3730fe9..a1344465d 100644 --- a/packages/workflow/src/cloudflare/publish.ts +++ b/packages/workflow/src/cloudflare/publish.ts @@ -145,6 +145,16 @@ function resolve( confirmCompanion(storage, kind, digest, bytes); return { bytes, authoritative: true }; } + if (kind === "blob") { + // A metadata row with no bytes is a half-written identity. Falling through + // to staging here would complete it as a side effect of a proposal, and + // which durable state won would depend on the write path rather than on + // what the store actually holds. + const partial = rows(storage, "SELECT size FROM vfs_blobs WHERE lower(hex(hash)) = ?", digest); + if (partial.length > 0) { + return corrupt("a retained blob has no bytes"); + } + } const staged = rows( storage, `SELECT bytes FROM ${STAGING_TABLE} WHERE acquisition_id = ? AND kind = ? AND digest = ?`, @@ -223,24 +233,22 @@ export function applyCommit( ? command.expectedWorkspaceRootId : publish(storage, acquisitionId, command); - const named = new Set(); - for (const mapping of command.mappings) { - const identity = - mapping.kind === "worktree" - ? `worktree:${mapping.record.repositoryName}/${mapping.record.name}` - : `${mapping.kind}:${mapping.kind === "repository" ? mapping.record.name : mapping.record.sessionKey}`; - if (named.has(identity)) { - // One proposal naming one mapping twice cannot be applied once and is not - // two mappings either. - throw new CommandError("mapping-conflict"); - } - named.add(identity); - } const selectedEntries = directoriesOf( command.publication === null ? undefined : command.publication.proposedManifest, ); - for (const mapping of command.mappings) { - applyMapping(storage, mapping, selectedEntries); + // The whole collection is decided before any of it is written. A Worktree may + // name a Repository that arrives in the same proposal, and which of the two + // happens to come first in an array is not a difference between proposals — + // an owner that applied them in order would accept one spelling of a + // transaction and refuse an identical one. + validateMappings(storage, command.mappings, selectedEntries); + // Applied in dependency order rather than the order they arrived in. A + // Worktree row references its Repository, so the parent has to exist when the + // child is written — but which one a proposal happens to list first is not a + // difference between proposals, and the owner decides that rather than making + // the runner arrange an array to suit the schema. + for (const mapping of dependencyOrder(command.mappings)) { + applyMapping(storage, mapping); } const journalEventIds: string[] = []; @@ -410,31 +418,126 @@ function publish(storage: OwnerStorage, acquisitionId: string, command: CommitCo return proposal.proposedWorkspaceRootId; } +/** How one mapping is named within a proposal, whatever order it arrives in. */ +function mappingIdentity(mapping: ProposedMapping): string { + if (mapping.kind === "worktree") { + return `worktree:${mapping.record.repositoryName}/${mapping.record.name}`; + } + if (mapping.kind === "repository") { + return `repository:${mapping.record.name}`; + } + return `agent-session:${mapping.record.sessionKey}`; +} + /** - * Where a mapping's checkout has to exist, for the mapping to be true. + * Decide the whole mapping collection, before any of it is written. * - * A Repository or Worktree row names a checkout path, and the row is only - * meaningful if the Workspace this commit selects actually contains it. A - * mapping-only commit that invented a checkout would retain a claim about a - * directory nothing put there, and the next execution would find the claim and - * not the files. + * Identity, duplication, parent relationships and checkout placement are all + * properties of the proposal rather than of one mapping, so they are settled + * here — against every mapping the proposal carries and the Workspace it + * selects. Deciding them one at a time during application would make acceptance + * depend on transport order. */ -function requirePlacement( - mapping: ProposedMapping, +function validateMappings( + storage: OwnerStorage, + mappings: readonly ProposedMapping[], selectedEntries: ReadonlySet | undefined, ): void { - if (mapping.kind === "agent-session") { - return; + const named = new Set(); + for (const mapping of mappings) { + const identity = mappingIdentity(mapping); + if (named.has(identity)) { + // One proposal naming one mapping twice cannot be applied once and is not + // two mappings either. + throw new CommandError("mapping-conflict"); + } + named.add(identity); } - if (selectedEntries === undefined) { - // No publication accompanies this commit, so nothing can have created the - // checkout. An exact confirmation of an already-retained mapping is still - // admissible — that is decided below, once the existing row is read. - return; + + for (const mapping of mappings) { + if (mapping.kind === "agent-session") { + continue; + } + if (retainedMapping(storage, mapping) !== undefined) { + // Already retained. Whether the proposal agrees with it is confirmed + // where the row is read; a mapping that exists needs no new checkout. + continue; + } + // A new checkout mapping is only true if this proposal publishes the + // Workspace that contains it. + if (selectedEntries === undefined || !selectedEntries.has(mapping.record.checkoutPath)) { + throw new CommandError("mapping-conflict"); + } + if ( + mapping.kind === "worktree" && + rows( + storage, + "SELECT name FROM workspace_repositories WHERE name = ?", + mapping.record.repositoryName, + )[0] === undefined && + !proposesRepository(mappings, mapping.record.repositoryName) + ) { + // A Worktree exists inside a Repository. One that named none — neither + // retained nor arriving in this same proposal — would be a checkout + // belonging to nothing. + throw new CommandError("mapping-conflict"); + } } - if (!selectedEntries.has(mapping.record.checkoutPath)) { - throw new CommandError("mapping-conflict"); +} + +/** The row already retained for one mapping, if there is one. */ +function retainedMapping( + storage: OwnerStorage, + mapping: ProposedMapping, +): Record | undefined { + if (mapping.kind === "repository") { + return rows( + storage, + `SELECT locator, locator_fingerprint, requested_base, creation_commit, primary_branch, + object_format, checkout_path FROM workspace_repositories WHERE name = ?`, + mapping.record.name, + )[0]; + } + if (mapping.kind === "worktree") { + return rows( + storage, + `SELECT requested_branch, requested_base, creation_commit, checkout_path + FROM workspace_worktrees WHERE repository_name = ? AND name = ?`, + mapping.record.repositoryName, + mapping.record.name, + )[0]; + } + return rows( + storage, + `SELECT provider, agent_command, session_identity, policy, + assertion_kind, assertion_value, created_at + FROM agent_sessions WHERE session_key = ?`, + mapping.record.sessionKey, + )[0]; +} + +/** Parents before children, so a proposal's array order carries no meaning. */ +const APPLICATION_ORDER: readonly ProposedMapping["kind"][] = [ + "repository", + "worktree", + "agent-session", +]; + +function dependencyOrder(mappings: readonly ProposedMapping[]): ProposedMapping[] { + const ordered: ProposedMapping[] = []; + for (const kind of APPLICATION_ORDER) { + for (const mapping of mappings) { + if (mapping.kind === kind) { + ordered.push(mapping); + } + } } + return ordered; +} + +/** Whether this proposal itself supplies the Repository a Worktree names. */ +function proposesRepository(mappings: readonly ProposedMapping[], name: string): boolean { + return mappings.some((mapping) => mapping.kind === "repository" && mapping.record.name === name); } /** Every directory the selected root contains, for placement checks. */ @@ -464,11 +567,7 @@ function sameText(row: Record, column: string, expected: string * established, and the disagreement would only surface later, as a checkout * that is not what its record says. */ -function applyMapping( - storage: OwnerStorage, - mapping: ProposedMapping, - selectedEntries: ReadonlySet | undefined, -): void { +function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { if (mapping.kind === "repository") { const record = mapping.record; const held = rows( @@ -478,11 +577,6 @@ function applyMapping( record.name, )[0]; if (held === undefined) { - requirePlacement(mapping, selectedEntries); - if (selectedEntries === undefined) { - // A new checkout mapping with no Workspace publication to create it. - throw new CommandError("mapping-conflict"); - } storage.sql.exec( `INSERT INTO workspace_repositories (name, locator, locator_fingerprint, requested_base, creation_commit, @@ -523,20 +617,6 @@ function applyMapping( record.name, )[0]; if (held === undefined) { - // A Worktree exists inside a Repository. One that named none would be a - // checkout belonging to nothing. - const repository = rows( - storage, - "SELECT name FROM workspace_repositories WHERE name = ?", - record.repositoryName, - )[0]; - if (repository === undefined) { - throw new CommandError("mapping-conflict"); - } - requirePlacement(mapping, selectedEntries); - if (selectedEntries === undefined) { - throw new CommandError("mapping-conflict"); - } storage.sql.exec( `INSERT INTO workspace_worktrees (repository_name, name, requested_branch, requested_base, creation_commit, checkout_path) diff --git a/packages/workflow/src/composition/locator.ts b/packages/workflow/src/composition/locator.ts new file mode 100644 index 000000000..af989e6c1 --- /dev/null +++ b/packages/workflow/src/composition/locator.ts @@ -0,0 +1,111 @@ +/** + * Admitting a Git locator, and naming one without publishing it. + * + * Two different questions. **Admission** decides whether a locator may be handed + * to Git at all. **Fingerprinting** produces the stable name the journal, the + * record and every compatibility comparison use, so a changed locator diverges + * without the bytes of either one being retained outside the single column that + * holds them. + * + * Admission is a closed allowlist rather than a search for bad shapes. Git's + * locator grammar reaches well past URLs — `ext::sh -c …` runs a command, a + * leading `-` is read as an option, and a transport helper is whatever is on + * `PATH` — so anything not recognized as one of the admitted forms is refused. + * Credentials in the string are refused rather than stripped: a locator that + * carries one is a secret a caller put in a durable input, and quietly editing + * it would retain a run nobody asked for. + * + * Both rules are shared because both hosts need them and neither may be more + * permissive than the other. The local host refuses a locator before Git sees + * it; the remote owner must refuse the same one before it becomes durable + * state, or an authenticated proposal could retain something the local host + * would never have produced. A second copy of an allowlist is the copy that + * ends up longer. + * + * Nothing here reaches a runtime. `URL` is the platform's, and the digest is + * the shared one. + */ + +import { sha256Hex } from "../workspace/sha256.ts"; + +/** Schemes this provider hands to Git. Everything else is refused. */ +const SCHEMES = new Set(["https", "http", "ssh", "git", "file"]); + +/** `user@host:path`, Git's scp-like form. A colon in the userinfo is a password. */ +const SCP_LIKE = /^([^/@:]+)@([^/@:]+):(.+)$/; + +function hasControlCharacters(value: string): boolean { + for (const character of value) { + const code = character.codePointAt(0) ?? 0; + if (code < 0x20 || code === 0x7f) { + return true; + } + } + return false; +} + +function admitUrl(locator: string): string | undefined { + let url: URL; + try { + url = new URL(locator); + } catch { + return undefined; + } + const scheme = url.protocol.replace(/:$/, ""); + if (!SCHEMES.has(scheme)) { + return undefined; + } + if (url.username !== "" || url.password !== "") { + return undefined; + } + // A query or a fragment is refused whole rather than searched for credentials. + // `?access_token=…` is the ordinary way a token is written into a URL, and a + // rule that named the parameters worth refusing would be a list of the ones + // somebody thought of — the same open-ended guessing this module rejects + // everywhere else. Git is given a repository's location, and neither part + // carries any of that location for the transports admitted here. + if (url.search !== "" || url.hash !== "") { + return undefined; + } + return locator; +} + +/** + * The locator this string is, or `undefined` when this provider will not use it. + * + * The answer is the original bytes, never a rewritten form: what is admitted is + * what Git is given and what the fingerprint names, so the three cannot drift. + */ +export function admitLocator(locator: string): string | undefined { + if (locator === "" || hasControlCharacters(locator) || /\s/.test(locator)) { + return undefined; + } + if (locator.startsWith("-")) { + return undefined; + } + if (/^[A-Za-z][A-Za-z0-9+.-]*:\/\//.test(locator)) { + return admitUrl(locator); + } + const scpLike = SCP_LIKE.exec(locator); + if (scpLike !== null) { + return locator; + } + // A local path. Absolute only: a relative one would name a different + // repository depending on which directory the host happened to run in, and a + // workflow's retained identity must not depend on that. + if (locator.startsWith("/")) { + return locator; + } + return undefined; +} + +/** + * The stable name an admitted locator is known by everywhere but its own column. + * + * Shared because both hosts retain it and both must derive it identically: a + * fingerprint is what a journal event carries in place of the locator, and two + * derivations would be two names for one repository. + */ +export function locatorFingerprintOf(locator: string): string { + return sha256Hex(locator); +} diff --git a/packages/workflow/src/composition/records.ts b/packages/workflow/src/composition/records.ts index 866129ea5..b7ebb87c0 100644 --- a/packages/workflow/src/composition/records.ts +++ b/packages/workflow/src/composition/records.ts @@ -20,7 +20,6 @@ * a record at all. */ -import { sha256Hex } from "../workspace/sha256.ts"; import type { Json } from "@executablemd/durable-streams"; import { members, optionalText, text } from "./parse.ts"; @@ -102,17 +101,6 @@ const WORKTREE_MEMBERS = [ "checkoutPath", ] as const; -/** - * The stable name an admitted locator is known by everywhere but its own column. - * - * Shared because both hosts retain it and both must derive it identically: a - * fingerprint is what a journal event carries in place of the locator, and two - * derivations would be two names for one repository. - */ -export function locatorFingerprintOf(locator: string): string { - return sha256Hex(locator); -} - export function parseObjectFormat(value: unknown): GitObjectFormat | undefined { return value === "sha1" || value === "sha256" ? value : undefined; } diff --git a/packages/workflow/src/deno/composition/locator.ts b/packages/workflow/src/deno/composition/locator.ts index a92faaae9..5d0f1dbe9 100644 --- a/packages/workflow/src/deno/composition/locator.ts +++ b/packages/workflow/src/deno/composition/locator.ts @@ -18,83 +18,7 @@ * quietly editing it would retain a run nobody asked for. */ -import { createHash } from "node:crypto"; - -/** Schemes this provider hands to Git. Everything else is refused. */ -const SCHEMES = new Set(["https", "http", "ssh", "git", "file"]); - -/** `user@host:path`, Git's scp-like form. A colon in the userinfo is a password. */ -const SCP_LIKE = /^([^/@:]+)@([^/@:]+):(.+)$/; - -function hasControlCharacters(value: string): boolean { - for (const character of value) { - const code = character.codePointAt(0) ?? 0; - if (code < 0x20 || code === 0x7f) { - return true; - } - } - return false; -} - -function admitUrl(locator: string): string | undefined { - let url: URL; - try { - url = new URL(locator); - } catch { - return undefined; - } - const scheme = url.protocol.replace(/:$/, ""); - if (!SCHEMES.has(scheme)) { - return undefined; - } - if (url.username !== "" || url.password !== "") { - return undefined; - } - // A query or a fragment is refused whole rather than searched for credentials. - // `?access_token=…` is the ordinary way a token is written into a URL, and a - // rule that named the parameters worth refusing would be a list of the ones - // somebody thought of — the same open-ended guessing this module rejects - // everywhere else. Git is given a repository's location, and neither part - // carries any of that location for the transports admitted here. - if (url.search !== "" || url.hash !== "") { - return undefined; - } - return locator; -} - -/** - * The locator this string is, or `undefined` when this provider will not use it. - * - * The answer is the original bytes, never a rewritten form: what is admitted is - * what Git is given and what the fingerprint names, so the three cannot drift. - */ -export function admitLocator(locator: string): string | undefined { - if (locator === "" || hasControlCharacters(locator) || /\s/.test(locator)) { - return undefined; - } - if (locator.startsWith("-")) { - return undefined; - } - if (/^[A-Za-z][A-Za-z0-9+.-]*:\/\//.test(locator)) { - return admitUrl(locator); - } - const scpLike = SCP_LIKE.exec(locator); - if (scpLike !== null) { - return locator; - } - // A local path. Absolute only: a relative one would name a different - // repository depending on which directory the host happened to run in, and a - // workflow's retained identity must not depend on that. - if (locator.startsWith("/")) { - return locator; - } - return undefined; -} - -/** - * The stable name an admitted locator is known by everywhere but its own column. - * - * The derivation is the shared rule: both hosts retain this fingerprint and a - * second derivation would be two names for one repository. - */ -export { locatorFingerprintOf as locatorFingerprint } from "../../composition/records.ts"; +export { + admitLocator, + locatorFingerprintOf as locatorFingerprint, +} from "../../composition/locator.ts"; diff --git a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts index deead8242..5bf7b210c 100644 --- a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts @@ -31,7 +31,7 @@ import { } from "./support/executor-object.ts"; import { encodeBase64 } from "../../src/cloudflare/encoding.ts"; import { sha256Hex } from "../../src/workspace/sha256.ts"; -import { locatorFingerprintOf } from "../../src/composition/records.ts"; +import { locatorFingerprintOf } from "../../src/composition/locator.ts"; import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; let unique = 0; @@ -574,6 +574,101 @@ describe("publishing one proposal", () => { } }); + it("retains only a locator this system would hand to Git", async () => { + // A matching fingerprint says the two values agree with each other. It says + // nothing about whether the locator is one that may ever be used, and an + // authenticated proposal must not be able to retain a credential or an + // executable transport form. + const refused: Record = { + "a credential in the URL": "https://user:token@git.example.invalid/octo/app.git", + "an executable transport form": "ext::sh -c 'curl example.invalid'", + "a query that can carry a token": "https://git.example.invalid/app.git?access_token=abc", + "an unknown scheme": "javascript:alert(1)", + "a relative path": "../elsewhere", + }; + for (const [description, locator] of Object.entries(refused)) { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + await stageThrough(socket); + const before = await on(stub, (owner) => owner.published()); + const answer = await ask( + socket, + "bad-locator", + commit({ + mappings: [ + { + kind: "repository", + locator, + record: { ...REPOSITORY.record, locatorFingerprint: locatorFingerprintOf(locator) }, + }, + ], + }), + ); + expect([description, answer["refusal"]]).toEqual([description, "command:malformed-member"]); + expect([description, await on(stub, (owner) => owner.published())]).toEqual([ + description, + before, + ]); + } + }); + + it("retains the exact admitted locator, not its fingerprint", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + await stageThrough(socket); + expect(await ask(socket, "publish", commit())).toMatchObject({ outcome: "performed" }); + // The row a later restoration reads has to name the repository, not a + // digest of it. + expect(await on(stub, (owner) => owner.repositoryLocator("app"))).toBe(LOCATOR); + }); + + it("accepts a Repository and its Worktree in either order", async () => { + const worktree = { + kind: "worktree", + record: { + repositoryName: "app", + name: "feature", + requestedBranch: "feature", + requestedBase: null, + creationCommit: "2".repeat(40), + checkoutPath: "/app", + }, + }; + // Which of the two comes first in an array is not a difference between + // proposals, so both spellings of one transaction must be accepted. + for (const [description, mappings] of Object.entries({ + "parent first": [REPOSITORY, worktree], + "child first": [worktree, REPOSITORY], + })) { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + await stageThrough(socket); + const answer = await ask(socket, "both", commit({ mappings })); + expect([description, answer["outcome"]]).toEqual([description, "performed"]); + expect([description, await on(stub, (owner) => owner.published())]).toMatchObject([ + description, + { currentRootId: NEXT_ROOT_ID, repositories: [{ name: "app", checkout_path: "/app" }] }, + ]); + } + }); + + it("refuses a blob whose bytes were never retained beside its metadata", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + // A metadata row with no bytes is a half-written identity. Completing it + // from staging would repair authoritative damage as a side effect. + await on(stub, (owner) => owner.removeBlobBytesOnly(NEXT_BLOB_ID, NEXT_BYTES.length)); + const socket = await connect(stub); + await stageThrough(socket); + const before = await on(stub, (owner) => owner.published()); + expect(await ask(socket, "half", commit())).toMatchObject({ refusal: "storage:corrupt" }); + expect(await on(stub, (owner) => owner.published())).toEqual(before); + expect(await on(stub, (owner) => owner.scratch())).toMatchObject({ commands: 2 }); + }); + it("grants a closed or foreign socket no publication", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 4788737b5..e39683434 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -336,6 +336,23 @@ export class ExecutorObject extends WorkflowOwnerObject { }; } + /** The exact locator a retained Repository row holds. */ + repositoryLocator(name: string): string { + const row = this.ctx.storage.sql + .exec("SELECT locator FROM workspace_repositories WHERE name = ?", name) + .toArray()[0]; + return row === undefined ? "" : String(row["locator"]); + } + + /** A blob's metadata with no bytes beside it: a half-written identity. */ + removeBlobBytesOnly(digest: string, size: number): void { + this.ctx.storage.sql.exec( + "INSERT INTO vfs_blobs (hash, size, last_seen) VALUES (?, ?, 0) ON CONFLICT(hash) DO NOTHING", + hexBytes(digest), + size, + ); + } + damageRetainedBlob(): void { this.ctx.storage.sql.exec( "UPDATE vfs_blob_bytes SET bytes = ?", From 2290d4460a16c94330fa3c8fa4e2abe5fab735cc Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 19:50:01 -0400 Subject: [PATCH 33/81] =?UTF-8?q?=F0=9F=9A=9A=20Let=20the=20runner=20actua?= =?UTF-8?q?lly=20publish=20what=20it=20did=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner could decide a proposal and nothing could make one. `CommitIntent` carried the frontier and the events; the link refused every commit by construction; and the trees a mutation happens in were owned by a test helper. That is the half of D3a that makes the other half reachable. A transaction now carries what it decided: the expected frontier, the events, optionally one complete Workspace publication, and the retained mappings the same operation produced. Enlisting is a private capability handed to the body rather than something reachable from the database, so work that never received it cannot publish a Workspace by accident, and a second enlistment is refused — two Workspaces proposed for one commit is a choice nobody may make on the run's behalf. What is enlisted is detached on the way in: the caller still holds its arrays and may go on editing them, and the proposal the identity was computed over must not change underneath it. The Cloudflare link stages the pieces the owner does not already hold, encodes one closed command, and reads the decision. Content is addressed by what it is, so a Workspace the owner never lost is not sent again. The command identity is minted once per intent and reused verbatim if the answer is lost — the owner recognizes a retry by that identity, and regenerating one would ask a second question rather than the same question again. A lost answer returns undecided rather than failed, because whether the owner committed is exactly what cannot be known from the runner. Two trees, and the difference is the point. The materialization is the accepted root: what the owner last confirmed, restored so tools can work in it. The attempt is where a mutation happens, and it is disposable by construction. A documented failure throws the attempt away and leaves the accepted tree exactly as the owner confirmed it; working directly in the accepted tree would mean a failed effect had already changed the only local copy of the run's Workspace. Both are resources, so a raised failure, a cancellation and a refused commit all leave nothing behind, and only a performed owner answer promotes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 138 +++++- packages/workflow/src/deno/remote-files.ts | 39 +- packages/workflow/src/remote/collector.ts | 86 +++- packages/workflow/src/remote/invocation.ts | 159 ++++++ .../workflow/tests/remote-publication.test.ts | 465 ++++++++++++++++++ packages/workflow/tests/remote-read.test.ts | 7 +- 6 files changed, 884 insertions(+), 10 deletions(-) create mode 100644 packages/workflow/src/remote/invocation.ts create mode 100644 packages/workflow/tests/remote-publication.test.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index f443147ef..36901d953 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -26,12 +26,13 @@ * caller — half a journal that looks whole is worse than no journal. */ -import { Err, type Operation, type Result } from "effection"; +import { Err, Ok, type Operation, type Result } from "effection"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; import type { JournalEntry } from "../storage/api.ts"; import { parseMembers, requireMemberNames } from "../storage/members.ts"; import type { DefinitionRetrieval, WorkflowRunRecord } from "../storage/record.ts"; import type { CommitIntent, OwnerLink, StartingFrontier } from "../remote/collector.ts"; -import type { OwnerAnswer, OwnerConnection } from "../remote/client.ts"; +import { OwnerLinkError, type OwnerAnswer, type OwnerConnection } from "../remote/client.ts"; import { parseRemoteJournalEntry, parseRemoteRetrieval, @@ -359,17 +360,144 @@ export function cloudflareReadLink( }; } -export function cloudflareOwnerLink(reads: RemoteReadLink): OwnerLink { +/** + * The bytes one proposed piece is made of, when the owner has to be sent them. + * + * A runner holds the content it captured; the owner may already hold some of + * it. Supplying bytes by identity lets the adapter stage only what is missing + * rather than resending a Workspace the owner never lost. + */ +export type ProposedBytes = (kind: "manifest" | "blob", digest: string) => Uint8Array | undefined; + +/** + * The runner's production link to its owner. + * + * `commit()` is the whole publication path: stage the pieces the owner does not + * have, encode one closed command, send it, and read the decision. The command + * identity is minted once per intent and reused verbatim on a retry, because + * the owner recognizes a retry by that identity and a regenerated one would be + * a second proposal rather than the same question asked again. + */ +export function cloudflareOwnerLink( + connection: OwnerConnection, + reads: RemoteReadLink, + nextId: () => string, + bytesOf: ProposedBytes = () => undefined, +): OwnerLink { return { *frontier(): Operation { return startingFrontier(yield* reads.frontier()); }, - *commit(_intent: CommitIntent): Operation> { - return Err(new CloudflareOwnerRefusalError("command:unavailable")); + + *commit(intent: CommitIntent): Operation> { + // Minted before anything is sent, and reused for the life of this intent. + const id = nextId(); + try { + yield* stageMissing(connection, nextId, intent, bytesOf); + const request = commitRequest(intent); + const answered = yield* ask(connection, id, request); + return answered.outcome === "refused" + ? Err(new CloudflareOwnerRefusalError(answered.refusal)) + : Ok(); + } catch (error) { + if (error instanceof OwnerLinkError) { + // The connection went while the answer was in flight. Whether the + // owner committed is exactly what cannot be known from here, so the + // caller learns the outcome is undecided rather than being told it + // failed — retrying this same id is what settles it. + return Err(error); + } + throw error; + } }, }; } +/** One command sent and one answer read, with the private refusal narrowed. */ +function* ask( + connection: OwnerConnection, + id: string, + request: Record, +): Operation<{ outcome: "performed" } | { outcome: "refused"; refusal: PrivateRefusal }> { + const offered = yield* connection.ask( + id, + request, + (value) => { + // A performed commit answers with what it published. Reading it proves + // the owner and this build agree about what just happened. + const found = members(value, ["workspaceRootId", "journalEventIds"]); + rootId(found.get("workspaceRootId")); + const ids = found.get("journalEventIds"); + if (!Array.isArray(ids) || ids.some((entry) => typeof entry !== "string" || entry === "")) { + return fail("a commit answer did not name the events it retained"); + } + return true; + }, + privateRefusal, + ); + return offered.outcome === "refused" + ? { outcome: "refused", refusal: privateRefusal(offered.refusal) } + : { outcome: "performed" }; +} + +/** + * Send the pieces the owner does not already hold. + * + * Staging is idempotent by identity, so a retry after an ambiguous answer + * re-offers the same bytes and the owner recognizes them rather than storing + * them twice. Anything the owner already has is not sent at all: content is + * addressed by what it is, and re-uploading a Workspace it never lost would be + * bytes crossing for nothing. + */ +function* stageMissing( + connection: OwnerConnection, + nextId: () => string, + intent: CommitIntent, + bytesOf: ProposedBytes, +): Operation { + if (intent.publication === null) { + return; + } + for (const piece of intent.publication.content) { + const bytes = bytesOf(piece.kind, piece.digest); + if (bytes === undefined) { + // The owner is expected to hold this one already. If it does not, the + // commit refuses rather than this guessing at bytes it does not have. + continue; + } + yield* stageCloudflareContent(connection, nextId(), piece.kind, bytes); + } +} + +/** The one closed command a complete intent becomes. */ +function commitRequest(intent: CommitIntent): Record { + return { + command: "commit", + expectedWorkspaceRootId: intent.expectedWorkspaceRootId, + expectedJournalEventId: intent.expectedJournalEventId, + publication: + intent.publication === null + ? null + : { + proposedWorkspaceRootId: intent.publication.proposedWorkspaceRootId, + proposedManifest: intent.publication.proposedManifest, + content: intent.publication.content.map((piece) => ({ + kind: piece.kind, + digest: piece.digest, + size: piece.size, + })), + }, + mappings: intent.mappings.map((mapping) => + mapping.kind === "repository" + ? { kind: mapping.kind, record: { ...mapping.record }, locator: mapping.locator } + : { kind: mapping.kind, record: { ...mapping.record } }, + ), + // Exactly what the serializer produces, in the order the transaction + // appended them. The owner parses each one and requires these same bytes. + events: intent.events.map((event) => serializeDurableEvent(event)), + }; +} + export function* stageCloudflareContent( connection: OwnerConnection, id: string, diff --git a/packages/workflow/src/deno/remote-files.ts b/packages/workflow/src/deno/remote-files.ts index fa482d45b..fafe82364 100644 --- a/packages/workflow/src/deno/remote-files.ts +++ b/packages/workflow/src/deno/remote-files.ts @@ -24,14 +24,18 @@ import { mkdir, readdir, readFile, + mkdtemp, readlink, + rm, symlink, utimes, writeFile, } from "node:fs/promises"; +import { tmpdir } from "node:os"; import { join } from "node:path"; -import { type Operation, until } from "effection"; +import { ensure, type Operation, resource, until } from "effection"; import type { RunnerFiles, RunnerNode } from "../remote/materialize.ts"; +import type { TemporaryTrees } from "../remote/invocation.ts"; /** Whole seconds, which is what a retained entry records. */ function seconds(milliseconds: number): number { @@ -161,3 +165,36 @@ export function runnerFiles(): RunnerFiles { }, }; } + +/** + * Temporary trees for one invocation, owned by the scope that asked for them. + * + * Every tree this hands out is removed when that scope ends, however it ends. + * A run that left one behind would leave a materialized Workspace on a machine + * that has stopped being responsible for it. + */ +export function useRunnerTrees(): Operation { + return resource(function* (provide) { + const roots: string[] = []; + yield* ensure(function* () { + // In reverse, so a nested tree goes before whatever contains it. + for (const root of roots.toReversed()) { + yield* until(rm(root, { recursive: true, force: true })); + } + }); + yield* provide({ + *create(purpose: string): Operation { + const root = yield* until(mkdtemp(join(tmpdir(), `xmd-workflow-${purpose}-`))); + roots.push(root); + return root; + }, + *remove(path: string): Operation { + yield* until(rm(path, { recursive: true, force: true })); + const found = roots.indexOf(path); + if (found >= 0) { + roots.splice(found, 1); + } + }, + }); + }); +} diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index d5f29d323..1fec62b12 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -23,6 +23,7 @@ */ import { call, ensure, Ok, type Operation, type Result } from "effection"; +import type { RetainedMapping, WorkspacePublication } from "./publication.ts"; import type { DurableEvent } from "@executablemd/durable-streams"; import type { DurableStream } from "@executablemd/durable-streams"; import type { WorkflowRunTransaction } from "../storage/api.ts"; @@ -30,6 +31,8 @@ import type { WorkflowRunTransaction } from "../storage/api.ts"; /** Why a transaction could not be run or committed. */ export type CollectorRefusal = | "nested-transaction" + | "publication-already-enlisted" + | "too-many-mappings" | "transaction-closed" | "operation-inside-body" | "too-many-events" @@ -51,11 +54,32 @@ export interface StartingFrontier { readonly events: readonly DurableEvent[]; } -/** One closed intent, as the owner will receive it. */ +/** + * One closed intent, as the owner will receive it. + * + * Everything the transaction decided, and nothing it did not. `publication` is + * absent for a transaction that only appended to the journal — a real case, and + * inventing a Workspace change to make the shape uniform would publish a root + * nobody asked for. + */ export interface CommitIntent { readonly expectedWorkspaceRootId: string; readonly expectedJournalEventId: string | null; readonly events: readonly DurableEvent[]; + readonly publication: WorkspacePublication | null; + readonly mappings: readonly RetainedMapping[]; +} + +/** + * What a Workspace operation enlisted, if one did. + * + * At most one per transaction. Two would be two Workspaces proposed for one + * commit, and the owner would have to choose — which is a decision nobody is + * entitled to make on the run's behalf. + */ +export interface WorkspaceEnlistment { + readonly publication: WorkspacePublication; + readonly mappings: readonly RetainedMapping[]; } /** What the collector needs from the connection. */ @@ -72,6 +96,9 @@ const MAX_EVENTS = 4096; /** The most serialized bytes one intent may carry. */ const MAX_EVENT_BYTES = 4 * 1024 * 1024; +/** The most retained mapping changes one intent may carry. */ +const MAX_MAPPINGS = 256; + /** * Admit one event and detach it from whoever handed it over. * @@ -135,7 +162,7 @@ export function requireNoOpenTransaction(gate: TransactionGate): void { export function transactRemotely( link: OwnerLink, gate: TransactionGate, - body: (transaction: WorkflowRunTransaction) => Operation, + body: (transaction: WorkflowRunTransaction, enlist: EnlistWorkspace) => Operation, ): Operation> { return call(function* (): Operation> { // Taken synchronously, before the first suspension. Checking and then @@ -184,11 +211,34 @@ export function transactRemotely( }, }; + let enlisted: WorkspaceEnlistment | undefined; + /** + * How a Workspace operation puts its result into this transaction. + * + * Private: it is handed to the body rather than reachable from the + * database, so work that never received it cannot publish a Workspace by + * accident. Detached on the way in, because the caller still holds the + * arrays and records it passed and a proposal that changed after it was + * admitted would not be the proposal the identity was computed over. + */ + const enlist: EnlistWorkspace = (proposal: WorkspaceEnlistment): void => { + if (!live) { + throw new RemoteTransactionError("transaction-closed"); + } + if (enlisted !== undefined) { + throw new RemoteTransactionError("publication-already-enlisted"); + } + if (proposal.mappings.length > MAX_MAPPINGS) { + throw new RemoteTransactionError("too-many-mappings"); + } + enlisted = detach(proposal); + }; + let outcome: T; try { // Everything the body started tears down before the intent is built, so // "no commit was sent" and "the body did not finish" are one statement. - outcome = yield* call(() => body({ journal })); + outcome = yield* call(() => body({ journal }, enlist)); } finally { // The handle is closed before the commit goes out, so a retained // transaction object refuses while the handle-level gate is still held. @@ -200,6 +250,8 @@ export function transactRemotely( expectedJournalEventId: starting.journalEventId, // A private snapshot. The collector's own array never leaves. events: appended.map((event) => structuredClone(event)), + publication: enlisted?.publication ?? null, + mappings: enlisted?.mappings ?? [], }); if (!committed.ok) { return committed; @@ -210,6 +262,34 @@ export function transactRemotely( }); } +/** How a Workspace operation enlists its one publication in the active transaction. */ +export type EnlistWorkspace = (proposal: WorkspaceEnlistment) => void; + +/** + * A copy nobody else holds a reference into. + * + * The caller keeps whatever it passed, and may go on using it. What the intent + * carries has to be what was admitted at the moment it was admitted — a + * publication whose inventory or manifest changed afterwards would not be the + * one its identity was computed over. + */ +function detach(proposal: WorkspaceEnlistment): WorkspaceEnlistment { + return Object.freeze({ + publication: Object.freeze({ + proposedWorkspaceRootId: proposal.publication.proposedWorkspaceRootId, + proposedManifest: proposal.publication.proposedManifest, + content: Object.freeze( + proposal.publication.content.map((piece) => Object.freeze({ ...piece })), + ), + }), + mappings: Object.freeze( + proposal.mappings.map((mapping) => + Object.freeze({ ...mapping, record: Object.freeze({ ...mapping.record }) }), + ), + ) as readonly RetainedMapping[], + }); +} + /** Discard a collector's work without sending it. */ export function abandon(gate: TransactionGate): Operation { return ensure(() => { diff --git a/packages/workflow/src/remote/invocation.ts b/packages/workflow/src/remote/invocation.ts new file mode 100644 index 000000000..ffb6c0569 --- /dev/null +++ b/packages/workflow/src/remote/invocation.ts @@ -0,0 +1,159 @@ +/** + * What a remote invocation owns on the runner, and when it lets go. + * + * Two trees, and the difference between them is the whole point. The + * *materialization* is the accepted root: what the owner last confirmed this + * run is at, restored so native tools can work in it. The *attempt* is where a + * mutation actually happens, and it is disposable by construction — until the + * owner performs the commit, nothing that happened in it has happened. + * + * Keeping them apart is what makes a documented failure ordinary. A Workspace + * effect that fails is a fact the run records against the root it started from, + * so the attempt is thrown away and the accepted tree is still exactly what the + * owner confirmed. Working directly in the accepted tree would mean a failed + * effect had already changed the only local copy of the run's Workspace, and + * the next attempt would start somewhere nobody chose. + * + * Both are Effection resources, so their lifetimes are their scopes'. Normal + * return, a raised failure, cancellation, a refusal from the owner and a lost + * response all leave nothing behind, because none of them skips teardown. Only + * a performed owner answer promotes an attempt, and promotion is a decision + * this module is told about rather than one it infers. + */ + +import { ensure, type Operation, resource } from "effection"; +import type { WorkspaceRejection } from "../workspace/root-manifest.ts"; +import { + captureWorkspace, + type CapturedWorkspace, + type HostPath, + materializeWorkspaceRoot, + type RunnerFiles, +} from "./materialize.ts"; +import type { RemoteReadLink } from "./read.ts"; + +/** A directory this invocation owns for as long as it needs one. */ +export interface TemporaryTrees { + /** A fresh empty directory, removed when the calling scope ends. */ + create(purpose: string): Operation; + /** Remove one, before its scope would. */ + remove(path: string): Operation; +} + +/** The accepted local copy of the root the owner last confirmed. */ +export interface Materialization { + /** The root this tree is, as the owner confirmed it. */ + readonly workspaceRootId: string; + /** Where a logical Workspace path sits in this tree. */ + readonly at: HostPath; + /** + * Record which root this tree now is. + * + * Called by a promoted attempt and by nothing else. It moves no bytes: the + * attempt did that, and this says what they are. + */ + accept(workspaceRootId: string): void; +} + +/** One disposable place to make a mutation, and the way to keep it. */ +export interface Attempt { + readonly at: HostPath; + /** What the attempt now describes, captured and checked locally. */ + capture(): Operation; + /** + * Make this attempt the accepted materialization. + * + * Called only after the owner reports the commit performed. Anything else — + * a refusal, a lost response, a local failure — leaves the accepted tree + * where it was, because until the owner says otherwise the run is still at + * the root it started from. + */ + promote(): Operation; +} + +/** + * Restore the admitted root into a tree this invocation owns. + * + * The tree is created, filled and proved before anything else runs against it: + * `materializeWorkspaceRoot` refuses a host that cannot reproduce the retained + * modes, times or topology, so an invocation either has the Workspace the owner + * described or does not start. + */ +export function useMaterialization( + files: RunnerFiles, + trees: TemporaryTrees, + reads: RemoteReadLink, + workspaceRootId: string, + reject: WorkspaceRejection, +): Operation { + return resource(function* (provide) { + const root = yield* trees.create("accepted"); + const at: HostPath = (logical) => join(root, logical); + yield* materializeWorkspaceRoot(files, reads, at, workspaceRootId, reject); + let accepted = workspaceRootId; + yield* provide({ + get workspaceRootId(): string { + return accepted; + }, + at, + accept(next: string): void { + accepted = next; + }, + }); + }); +} + +/** + * A disposable copy of the accepted tree, for one mutation. + * + * Materialized from the owner rather than copied from the accepted tree: the + * owner's copy is the one that is authoritative, and reading it again is how an + * attempt starts from what the run actually is rather than from whatever the + * last attempt happened to leave behind. + */ +export function useAttempt( + files: RunnerFiles, + trees: TemporaryTrees, + reads: RemoteReadLink, + materialization: Materialization, + reject: WorkspaceRejection, +): Operation { + return resource(function* (provide) { + const root = yield* trees.create("attempt"); + const at: HostPath = (logical) => join(root, logical); + yield* materializeWorkspaceRoot(files, reads, at, materialization.workspaceRootId, reject); + + let promoted = false; + // Registered before the attempt is handed over, so every exit removes it — + // including the ones that never reach the end of the calling scope. + yield* ensure(function* () { + if (!promoted) { + yield* trees.remove(root); + } + }); + + yield* provide({ + at, + *capture(): Operation { + return yield* captureWorkspace(files, at, reject); + }, + *promote(): Operation { + const captured = yield* captureWorkspace(files, at, reject); + promoted = true; + materialization.accept(captured.root.rootId); + }, + }); + }); +} + +/** + * One logical Workspace path under a host directory. + * + * Kept here rather than imported from a path module because it is the only + * place the two vocabularies meet, and because a shared module may not name a + * host's path conventions. The logical root is `/` and everything under it is + * relative to the tree this invocation was given. + */ +function join(root: string, logical: string): string { + return logical === "/" ? root : `${root}/${logical.slice(1)}`; +} diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts new file mode 100644 index 000000000..26659c8cd --- /dev/null +++ b/packages/workflow/tests/remote-publication.test.ts @@ -0,0 +1,465 @@ +/** + * Tier WRH — what the production runner sends, and what it keeps. + * + * The owner's half is proved on real workerd, where atomicity and hibernation + * are real. This is the other half: whether the runner can build the command + * the owner accepts, whether it sends one at all when the work did not finish, + * and whether anything survives on disk that should not. + * + * The connection is a deterministic fake because what crosses it is arithmetic + * over what the transaction decided. The filesystem is real, because a tree + * that was supposed to be removed is not a claim a fake can settle. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import { type Operation, scoped, sleep, spawn, until } from "effection"; +import { mkdir, readdir, writeFile } from "node:fs/promises"; +import { cloudflareOwnerLink } from "../src/cloudflare/client.ts"; +import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; +import { createTransactionGate, transactRemotely } from "../src/remote/collector.ts"; +import { useAttempt, useMaterialization } from "../src/remote/invocation.ts"; +import { type OwnerSocket, type SocketListener, useOwnerConnection } from "../src/remote/client.ts"; +import type { + RemoteContent, + RemoteContentRequest, + RemoteFrontierSnapshot, + RemoteReadLink, +} from "../src/remote/read.ts"; +import { captureWorkspace, type CapturedWorkspace } from "../src/remote/materialize.ts"; +import { + parseWorkspaceRootManifest, + WORKSPACE_ROOT_DOMAIN, +} from "../src/workspace/root-manifest.ts"; +import { sha256Hex } from "../src/workspace/sha256.ts"; +import { locatorFingerprintOf } from "../src/composition/locator.ts"; +import type { ProposedContent, RetainedMapping } from "../src/remote/publication.ts"; + +function reject(reason: string): never { + throw new Error(reason); +} + +function event(name: string) { + return { + type: "yield" as const, + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok" as const, value: name }, + }; +} + +const LOCATOR = "https://git.example.invalid/octo/app.git"; + +function serve( + captured: Awaited> extends infer T ? never : never, +): never { + throw new Error("unused"); +} + +/** A recording connection: every request it was sent, and canned answers. */ +function wire(answer: (request: Record) => Record) { + const sent: Record[] = []; + const listeners = new Map>(); + let deliver = true; + const socket: OwnerSocket = { + send(data: string): void { + const request = JSON.parse(data) as Record; + sent.push(request); + if (!deliver) { + return; + } + const response = answer(request); + for (const listener of listeners.get("message") ?? []) { + listener({ data: JSON.stringify({ id: request["id"], ...response }) }); + } + }, + close(): void {}, + addEventListener(type, listener): void { + const found = listeners.get(type) ?? new Set(); + found.add(listener); + listeners.set(type, found); + }, + removeEventListener(type, listener): void { + listeners.get(type)?.delete(listener); + }, + }; + return { + socket, + sent, + /** Stop answering, as a connection lost mid-request would. */ + silence(): void { + deliver = false; + }, + end(): void { + for (const listener of listeners.get("close") ?? []) { + listener({}); + } + }, + }; +} + +/** + * What a correct owner answers each private command with. + * + * `sizes` is what the owner would have measured after decoding the bytes it was + * sent. The runner checks that against what it sent, so an owner that guessed + * would be answering about content it had not read. + */ +function ownerAnswers(rootId: string, sizes: ReadonlyMap = new Map()) { + return (request: Record): Record => { + if (request["command"] === "stage") { + return { + outcome: "performed", + value: { + kind: request["kind"], + digest: request["digest"], + size: sizes.get(String(request["digest"])) ?? 0, + }, + }; + } + return { + outcome: "performed", + value: { workspaceRootId: rootId, journalEventIds: ["e1"] }, + }; + }; +} + +function ids(): () => string { + let id = 0; + return () => `request-${(id += 1)}`; +} + +/** An owner that answers frontier/root/content from one captured tree. */ +function readsOf(captured: { + root: { manifest: string; rootId: string }; + contents: ReadonlyMap; + blobs: ReadonlyMap; +}): RemoteReadLink { + return { + // deno-lint-ignore require-yield + *frontier(): Operation { + return { + record: { + runId: "remote-run", + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-03T00:00:00.000Z", + updatedAt: "2026-09-03T00:00:00.000Z", + }, + retrieval: undefined, + workspaceRootId: captured.root.rootId, + journalEventId: null, + entries: [], + }; + }, + // deno-lint-ignore require-yield + *root(workspaceRootId: string) { + if (workspaceRootId !== captured.root.rootId) { + throw new Error("asked for a root this owner does not hold"); + } + return parseWorkspaceRootManifest(captured.root.manifest, reject); + }, + // deno-lint-ignore require-yield + *content(_rootId: string, request: RemoteContentRequest): Operation { + const bytes = + request.kind === "manifest" + ? captured.contents.get(request.digest)?.manifestBytes + : captured.blobs.get(request.digest); + if (bytes === undefined) { + throw new Error("asked for content this owner does not hold"); + } + return { kind: request.kind, digest: request.digest, bytes }; + }, + }; +} + +/** A small starting tree, captured so an owner can serve it. */ +function* startingTree(): Operation<{ captured: CapturedWorkspace; reads: RemoteReadLink }> { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); + const root = yield* trees.create("source"); + yield* until(writeFile(`${root}/README.md`, "starting\n", { mode: 0o644 })); + yield* until(mkdir(`${root}/docs`, { mode: 0o755 })); + const captured = yield* captureWorkspace( + files, + (logical) => (logical === "/" ? root : `${root}${logical}`), + reject, + ); + return { captured, reads: readsOf(captured) }; +} + +describe("what the production runner publishes", () => { + it("sends one closed commit describing everything the transaction decided", function* () { + const files = runnerFiles(); + yield* scoped(function* () { + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const sizes = new Map(); + const transport = wire(ownerAnswers(captured.root.rootId, sizes)); + const connection = yield* useOwnerConnection(transport.socket); + + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* until(writeFile(attempt.at("/NOTES.md"), "written by the effect\n", { mode: 0o644 })); + const proposed = yield* attempt.capture(); + for (const [digest, content] of proposed.contents) { + sizes.set(digest, content.manifestBytes.length); + } + for (const [digest, bytes] of proposed.blobs) { + sizes.set(digest, bytes.length); + } + + const link = cloudflareOwnerLink(connection, reads, ids(), (kind, digest) => + kind === "manifest" + ? proposed.contents.get(digest)?.manifestBytes + : proposed.blobs.get(digest), + ); + const committed = yield* transactRemotely( + link, + createTransactionGate(), + function* (transaction, enlist) { + yield* transaction.journal.append(event("published")); + enlist({ + publication: { + proposedWorkspaceRootId: proposed.root.rootId, + proposedManifest: proposed.root.manifest, + content: [ + ...proposed.root.manifests.map((digest) => ({ + kind: "manifest" as const, + digest, + size: proposed.contents.get(digest)?.manifestBytes.length ?? 0, + })), + ...proposed.root.blobs.map((digest) => ({ + kind: "blob" as const, + digest, + size: proposed.blobs.get(digest)?.length ?? 0, + })), + ], + }, + mappings: [ + { + kind: "repository", + locator: LOCATOR, + record: { + name: "app", + locatorFingerprint: locatorFingerprintOf(LOCATOR), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/docs", + }, + }, + ], + }); + return "done"; + }, + ); + expect(committed).toMatchObject({ ok: true }); + + // The last request is one closed commit carrying the whole proposal. + const commit = transport.sent.at(-1); + expect(commit?.["command"]).toBe("commit"); + expect(commit?.["expectedWorkspaceRootId"]).toBe(captured.root.rootId); + expect(commit?.["events"]).toEqual([serializeDurableEvent(event("published"))]); + const publication = commit?.["publication"] as Record; + expect(publication["proposedWorkspaceRootId"]).toBe(proposed.root.rootId); + expect(sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${String(publication["proposedManifest"])}`)).toBe( + proposed.root.rootId, + ); + const mappings = commit?.["mappings"] as Record[]; + expect(mappings[0]?.["locator"]).toBe(LOCATOR); + // Everything the proposal names was staged before the commit went out. + const staged = transport.sent.filter((request) => request["command"] === "stage"); + expect(staged.length).toBe(proposed.root.manifests.length + proposed.root.blobs.length); + }); + }); + + it("sends no commit and keeps no tree when the body does not finish", function* () { + const files = runnerFiles(); + const outcomes: Record Operation> = {}; + for (const description of ["raises", "is cancelled"]) { + let attemptPath = ""; + const transport = wire(ownerAnswers("")); + yield* scoped(function* () { + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + let raised: unknown; + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + attemptPath = attempt.at("/"); + if (description === "raises") { + try { + yield* transactRemotely(link, createTransactionGate(), function* () { + throw new Error("the effect failed"); + }); + } catch (error) { + raised = error; + } + return; + } + const running = yield* spawn(() => + transactRemotely(link, createTransactionGate(), function* () { + yield* sleep(10_000); + return "never"; + }), + ); + yield* sleep(0); + yield* running.halt(); + }); + expect([description, description === "raises" ? raised instanceof Error : true]).toEqual([ + description, + true, + ]); + }); + // No commit was sent, and the attempt tree is gone. + expect([ + description, + transport.sent.some((request) => request["command"] === "commit"), + ]).toEqual([description, false]); + let listed: unknown; + try { + listed = yield* until(readdir(attemptPath)); + } catch (error) { + listed = error; + } + expect([description, listed instanceof Error]).toEqual([description, true]); + } + void outcomes; + }); + + it("keeps the accepted root until the owner performs the commit", function* () { + const files = runnerFiles(); + yield* scoped(function* () { + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* until(writeFile(attempt.at("/NOTES.md"), "not published\n", { mode: 0o644 })); + // The owner refused, so nothing promotes. + }); + expect(materialization.workspaceRootId).toBe(captured.root.rootId); + + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* until(writeFile(attempt.at("/NOTES.md"), "published\n", { mode: 0o644 })); + yield* attempt.promote(); + }); + // Only a promoted attempt moves what the run is at. + expect(materialization.workspaceRootId).not.toBe(captured.root.rootId); + }); + }); + + it("cannot be changed by a caller that kept its own copy", function* () { + const files = runnerFiles(); + yield* scoped(function* () { + const trees = yield* useRunnerTrees(); + void trees; + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const content: ProposedContent[] = [{ kind: "manifest", digest: "a".repeat(64), size: 1 }]; + const mappings: RetainedMapping[] = [ + { + kind: "repository", + locator: LOCATOR, + record: { + name: "app", + locatorFingerprint: locatorFingerprintOf(LOCATOR), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/docs", + }, + }, + ]; + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist({ + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content, + }, + mappings, + }); + // The caller still holds both arrays and edits them after admission. + content.push({ kind: "blob", digest: "b".repeat(64), size: 2 }); + const first = mappings[0]; + if (first?.kind === "repository") { + mappings[0] = { ...first, locator: "https://elsewhere.invalid/x.git" }; + } + return "done"; + }); + const commit = transport.sent.at(-1); + const publication = commit?.["publication"] as Record; + expect((publication["content"] as unknown[]).length).toBe(1); + expect((commit?.["mappings"] as Record[])[0]?.["locator"]).toBe(LOCATOR); + }); + }); + + it("refuses a second Workspace publication in one transaction", function* () { + const files = runnerFiles(); + void files; + yield* scoped(function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const proposal = { + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: [], + }, + mappings: [], + }; + let raised: unknown; + try { + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist(proposal); + enlist(proposal); + return "done"; + }); + } catch (error) { + raised = error; + } + // Two Workspaces proposed for one commit is a choice nobody may make on + // the run's behalf, so the transaction fails and nothing is sent. + expect(raised).toBeInstanceOf(Error); + expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); + }); + }); +}); diff --git a/packages/workflow/tests/remote-read.test.ts b/packages/workflow/tests/remote-read.test.ts index cf4b1e658..e0b0f4fbe 100644 --- a/packages/workflow/tests/remote-read.test.ts +++ b/packages/workflow/tests/remote-read.test.ts @@ -409,7 +409,12 @@ describe("semantic reads from a Cloudflare owner", () => { let committed: unknown; yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareOwnerLink(cloudflareReadLink(connection, ids(), RUN_ID)); + const request = ids(); + const link = cloudflareOwnerLink( + connection, + cloudflareReadLink(connection, request, RUN_ID), + request, + ); // Two pages went over the wire. What the body reads back is one journal: // the collector is handed the assembled prefix and never learns that a // page, a cursor or an anchor was involved. From dd08600fe153c8949c3bac7158873cea9655ede0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 19:51:25 -0400 Subject: [PATCH 34/81] =?UTF-8?q?=F0=9F=A7=B9=20Read=20the=20sent=20comman?= =?UTF-8?q?d=20instead=20of=20asserting=20its=20shape=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The production-path test carried a dead helper, two unused bindings, two redundant scopes and a chain of assertions that told the compiler what the request was rather than checking it. What is being proved is the shape of what the runner sent, so it is read through narrowing helpers that fail with the member that disagreed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- .../workflow/tests/remote-publication.test.ts | 194 ++++++++++-------- 1 file changed, 109 insertions(+), 85 deletions(-) diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index 26659c8cd..437f6bfd2 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -51,12 +51,6 @@ function event(name: string) { const LOCATOR = "https://git.example.invalid/octo/app.git"; -function serve( - captured: Awaited> extends infer T ? never : never, -): never { - throw new Error("unused"); -} - /** A recording connection: every request it was sent, and canned answers. */ function wire(answer: (request: Record) => Record) { const sent: Record[] = []; @@ -125,6 +119,48 @@ function ownerAnswers(rootId: string, sizes: ReadonlyMap = new M }; } +/** The final command a transaction sent, proved to be one. */ +function lastCommit(sent: readonly Record[]): Record { + const commit = sent.at(-1); + if (commit === undefined || commit["command"] !== "commit") { + throw new Error("expected the last request to be a commit"); + } + return commit; +} + +/** One object member, read rather than asserted into shape. */ +function member(value: unknown, name: string): Record { + const found = value === null || typeof value !== "object" ? undefined : Object.entries(value); + const entry = found?.find(([key]) => key === name)?.[1]; + if (entry === null || typeof entry !== "object" || Array.isArray(entry)) { + throw new Error(`expected ${name} to be an object`); + } + return Object.fromEntries(Object.entries(entry)); +} + +/** One text member, read rather than asserted. */ +function text(value: Record, name: string): string { + const found = value[name]; + if (typeof found !== "string") { + throw new Error(`expected ${name} to be text`); + } + return found; +} + +/** One list member, read the same way. */ +function memberList(value: Record, name: string): Record[] { + const entry = value[name]; + if (!Array.isArray(entry)) { + throw new Error(`expected ${name} to be a list`); + } + return entry.map((item) => { + if (item === null || typeof item !== "object" || Array.isArray(item)) { + throw new Error(`expected every ${name} entry to be an object`); + } + return Object.fromEntries(Object.entries(item)); + }); +} + function ids(): () => string { let id = 0; return () => `request-${(id += 1)}`; @@ -273,17 +309,15 @@ describe("what the production runner publishes", () => { expect(committed).toMatchObject({ ok: true }); // The last request is one closed commit carrying the whole proposal. - const commit = transport.sent.at(-1); - expect(commit?.["command"]).toBe("commit"); - expect(commit?.["expectedWorkspaceRootId"]).toBe(captured.root.rootId); - expect(commit?.["events"]).toEqual([serializeDurableEvent(event("published"))]); - const publication = commit?.["publication"] as Record; + const commit = lastCommit(transport.sent); + expect(commit["expectedWorkspaceRootId"]).toBe(captured.root.rootId); + expect(commit["events"]).toEqual([serializeDurableEvent(event("published"))]); + const publication = member(commit, "publication"); expect(publication["proposedWorkspaceRootId"]).toBe(proposed.root.rootId); expect(sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${String(publication["proposedManifest"])}`)).toBe( proposed.root.rootId, ); - const mappings = commit?.["mappings"] as Record[]; - expect(mappings[0]?.["locator"]).toBe(LOCATOR); + expect(text(memberList(commit, "mappings")[0] ?? {}, "locator")).toBe(LOCATOR); // Everything the proposal names was staged before the commit went out. const staged = transport.sent.filter((request) => request["command"] === "stage"); expect(staged.length).toBe(proposed.root.manifests.length + proposed.root.blobs.length); @@ -382,84 +416,74 @@ describe("what the production runner publishes", () => { }); it("cannot be changed by a caller that kept its own copy", function* () { - const files = runnerFiles(); - yield* scoped(function* () { - const trees = yield* useRunnerTrees(); - void trees; - const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); - const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareOwnerLink(connection, reads, ids()); - const content: ProposedContent[] = [{ kind: "manifest", digest: "a".repeat(64), size: 1 }]; - const mappings: RetainedMapping[] = [ - { - kind: "repository", - locator: LOCATOR, - record: { - name: "app", - locatorFingerprint: locatorFingerprintOf(LOCATOR), - requestedBase: null, - creationCommit: "9".repeat(40), - primaryBranch: "main", - objectFormat: "sha1", - checkoutPath: "/docs", - }, + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const content: ProposedContent[] = [{ kind: "manifest", digest: "a".repeat(64), size: 1 }]; + const mappings: RetainedMapping[] = [ + { + kind: "repository", + locator: LOCATOR, + record: { + name: "app", + locatorFingerprint: locatorFingerprintOf(LOCATOR), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/docs", }, - ]; - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist({ - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content, - }, - mappings, - }); - // The caller still holds both arrays and edits them after admission. - content.push({ kind: "blob", digest: "b".repeat(64), size: 2 }); - const first = mappings[0]; - if (first?.kind === "repository") { - mappings[0] = { ...first, locator: "https://elsewhere.invalid/x.git" }; - } - return "done"; - }); - const commit = transport.sent.at(-1); - const publication = commit?.["publication"] as Record; - expect((publication["content"] as unknown[]).length).toBe(1); - expect((commit?.["mappings"] as Record[])[0]?.["locator"]).toBe(LOCATOR); - }); - }); - - it("refuses a second Workspace publication in one transaction", function* () { - const files = runnerFiles(); - void files; - yield* scoped(function* () { - const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); - const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareOwnerLink(connection, reads, ids()); - const proposal = { + }, + ]; + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist({ publication: { proposedWorkspaceRootId: captured.root.rootId, proposedManifest: captured.root.manifest, - content: [], + content, }, - mappings: [], - }; - let raised: unknown; - try { - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist(proposal); - enlist(proposal); - return "done"; - }); - } catch (error) { - raised = error; + mappings, + }); + // The caller still holds both arrays and edits them after admission. + content.push({ kind: "blob", digest: "b".repeat(64), size: 2 }); + const first = mappings[0]; + if (first?.kind === "repository") { + mappings[0] = { ...first, locator: "https://elsewhere.invalid/x.git" }; } - // Two Workspaces proposed for one commit is a choice nobody may make on - // the run's behalf, so the transaction fails and nothing is sent. - expect(raised).toBeInstanceOf(Error); - expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); + return "done"; }); + const commit = lastCommit(transport.sent); + expect(memberList(member(commit, "publication"), "content")).toHaveLength(1); + expect(text(memberList(commit, "mappings")[0] ?? {}, "locator")).toBe(LOCATOR); + }); + + it("refuses a second Workspace publication in one transaction", function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const proposal = { + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: [], + }, + mappings: [], + }; + let raised: unknown; + try { + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist(proposal); + enlist(proposal); + return "done"; + }); + } catch (error) { + raised = error; + } + // Two Workspaces proposed for one commit is a choice nobody may make on + // the run's behalf, so the transaction fails and nothing is sent. + expect(raised).toBeInstanceOf(Error); + expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); }); }); From 867e9a8df08275d3b8ab2af16674bf718cb8dd77 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 19:56:21 -0400 Subject: [PATCH 35/81] =?UTF-8?q?=F0=9F=A4=9D=20Prove=20the=20two=20captur?= =?UTF-8?q?e=20implementations=20describe=20one=20Workspace=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The local host walks the content tables inside its own SQLite file; the runner walks a real directory. Neither walk can be shared, and a root identity is a digest of what the walk produced — so if the two ever disagreed, a run would change its Workspace by moving between hosts and every no-op remote effect would propose a root the local host had never seen. Until now the round trip proved only that the runner agreed with itself. The fixture is now the local provider's own capture. A real run database is opened, the discriminating tree is written through the authoritative Workspace transaction, and `capture({ publish: true })` retains the root exactly as a run does. That root and the content rows it closes over are served over the remote read boundary, materialized by the production runner adapter, and captured again by the runner's implementation. The two identities have to be one string. It found the disagreement it was written to find. The retained format carries modification times in milliseconds, because the retaining host's clock is `Date.now`; the runner adapter reported seconds. Every retained tree would have looked like a different one, and writing a millisecond value through an interface that speaks seconds would have put each file tens of thousands of years from now, where the filesystem cannot keep it. The adapter now preserves the unit the format uses and converts only where the host primitive requires it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/deno/remote-files.ts | 21 +- .../tests/remote-interoperability.test.ts | 190 ++++++++++++++++++ .../tests/remote-materialization.test.ts | 2 +- 3 files changed, 206 insertions(+), 7 deletions(-) create mode 100644 packages/workflow/tests/remote-interoperability.test.ts diff --git a/packages/workflow/src/deno/remote-files.ts b/packages/workflow/src/deno/remote-files.ts index fafe82364..1e807706b 100644 --- a/packages/workflow/src/deno/remote-files.ts +++ b/packages/workflow/src/deno/remote-files.ts @@ -37,9 +37,17 @@ import { ensure, type Operation, resource, until } from "effection"; import type { RunnerFiles, RunnerNode } from "../remote/materialize.ts"; import type { TemporaryTrees } from "../remote/invocation.ts"; -/** Whole seconds, which is what a retained entry records. */ -function seconds(milliseconds: number): number { - return Math.floor(milliseconds / 1000); +/** + * Whole milliseconds, which is the unit a retained entry records. + * + * Not seconds. The Workspace format carries whatever the retaining host's clock + * produced, and that clock is `Date.now`, so an adapter that reported seconds + * would describe every retained tree as a different one — and setting a + * millisecond value as though it were seconds would put the file tens of + * thousands of years from now, where the filesystem cannot keep it. + */ +function milliseconds(value: number): number { + return Math.round(value); } function describeStats( @@ -63,7 +71,7 @@ function describeStats( // a retained mode that carried them would not round-trip through the // format's own bound. mode: stats.mode & 0o7777, - mtime: seconds(stats.mtimeMs), + mtime: milliseconds(stats.mtimeMs), size: kind === "file" ? stats.size : 0, // Only a file reached by more than one name can be part of a group, so // anything else reports no identity and is captured on its own. @@ -114,7 +122,8 @@ export function runnerFiles(): RunnerFiles { }, *setModifiedAt(path: string, mtime: number): Operation { - yield* until(utimes(path, mtime, mtime)); + // `utimes` speaks seconds; the format speaks milliseconds. + yield* until(utimes(path, mtime / 1000, mtime / 1000)); }, /** @@ -125,7 +134,7 @@ export function runnerFiles(): RunnerFiles { * entirely. */ *setLinkModifiedAt(path: string, mtime: number): Operation { - yield* until(lutimes(path, mtime, mtime)); + yield* until(lutimes(path, mtime / 1000, mtime / 1000)); }, /** diff --git a/packages/workflow/tests/remote-interoperability.test.ts b/packages/workflow/tests/remote-interoperability.test.ts new file mode 100644 index 000000000..abe7e7a66 --- /dev/null +++ b/packages/workflow/tests/remote-interoperability.test.ts @@ -0,0 +1,190 @@ +/** + * Tier WRH — the two capture implementations describe one Workspace. + * + * The local host walks the DOFS tables inside its own SQLite file. The runner + * walks a real directory. Neither walk can be shared, and a root identity is a + * digest of what the walk produced — so if the two ever disagreed, a run would + * change its Workspace by moving between hosts, and every no-op remote effect + * would propose a root the local host had never seen. + * + * Nothing here is produced by the code under test. The fixture is built through + * the authoritative Workspace transaction and captured by the local provider's + * own `capture()`, exactly as a real run retains a root. That root is then + * served over the remote read boundary, materialized by the production runner + * adapter, and captured again by the runner's implementation. The two + * identities have to be the same string. + * + * The tree is the discriminating one: two hardlink groups holding identical + * bytes, two independent files holding identical bytes, an empty file, a + * symbolic link, distinct modes and modification times, and a file large enough + * to cross more than one chunk. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { DatabaseSync } from "node:sqlite"; +import { type Operation, scoped } from "effection"; +import type { WorkflowRunDatabase } from "../mod.ts"; +import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; +import { + type PrivateWorkspaceTransaction, + transactWorkspaceRoots, +} from "../src/deno/workspace/private.ts"; +import { captureWorkspace, materializeWorkspaceRoot } from "../src/remote/materialize.ts"; +import type { RemoteContent, RemoteContentRequest, RemoteReadLink } from "../src/remote/read.ts"; +import { parseWorkspaceRootManifest } from "../src/workspace/root-manifest.ts"; +import { createRun, runPath, useStorageRoot, withStorage } from "./support/storage.ts"; + +function reject(reason: string): never { + throw new Error(reason); +} + +function* transact( + database: WorkflowRunDatabase, + body: (workspace: PrivateWorkspaceTransaction) => Operation, +): Operation { + const result = yield* transactWorkspaceRoots(database, body); + if (!result.ok) { + throw result.error; + } + return result.value; +} + +/** + * The content one retained root closes over, read out of the run's own store. + * + * The owner would read these rows; here the test does, so what crosses the + * remote read boundary is exactly what the local host retained rather than + * anything the runner computed. + */ +function retainedContent(path: string): { + manifests: Map; + blobs: Map; +} { + const database = new DatabaseSync(path, { readOnly: true }); + try { + const manifests = new Map(); + for (const row of database.prepare("SELECT hash, encoded FROM vfs_manifests").all()) { + manifests.set(hex(row["hash"]), bytes(row["encoded"])); + } + const blobs = new Map(); + for (const row of database.prepare("SELECT hash, bytes FROM vfs_blob_bytes").all()) { + blobs.set(hex(row["hash"]), bytes(row["bytes"])); + } + return { manifests, blobs }; + } finally { + database.close(); + } +} + +function bytes(value: unknown): Uint8Array { + if (!(value instanceof Uint8Array)) { + throw new Error("expected retained content to be bytes"); + } + return value; +} + +function hex(value: unknown): string { + return Array.from(bytes(value), (byte) => byte.toString(16).padStart(2, "0")).join(""); +} + +/** The retained root and its content, served the way an owner serves them. */ +function servedBy( + manifest: string, + rootId: string, + content: { manifests: Map; blobs: Map }, +): RemoteReadLink { + return { + // deno-lint-ignore require-yield + *frontier(): Operation { + throw new Error("this owner serves only a root and its content"); + }, + // deno-lint-ignore require-yield + *root(workspaceRootId: string) { + if (workspaceRootId !== rootId) { + throw new Error("asked for a root this owner does not hold"); + } + return parseWorkspaceRootManifest(manifest, reject); + }, + // deno-lint-ignore require-yield + *content(_rootId: string, request: RemoteContentRequest): Operation { + const found = + request.kind === "manifest" + ? content.manifests.get(request.digest) + : content.blobs.get(request.digest); + if (found === undefined) { + throw new Error(`the run retains no ${request.kind} ${request.digest}`); + } + return { kind: request.kind, digest: request.digest, bytes: found }; + }, + }; +} + +/** Everything the format carries, written through the authoritative surface. */ +function* buildWorkspace(workspace: PrivateWorkspaceTransaction): Operation { + const files = workspace.filesystem; + yield* files.mkdir("/docs", { mode: 0o755 }); + yield* files.mkdir("/docs/deep", { mode: 0o700 }); + yield* files.writeFile("/README.md", "a workspace\n", 0o644); + yield* files.writeFile("/empty", new Uint8Array(0), 0o600); + yield* files.writeFile("/docs/guide.md", "# guide\n", 0o644); + // Larger than one chunk, so its manifest names more than one piece. + yield* files.writeFile("/docs/deep/large.bin", new Uint8Array(700 * 1024).fill(7), 0o644); + yield* files.symlink("../README.md", "/docs/link"); + + // Two hardlink groups holding identical bytes: one manifest, two files. + yield* files.writeFile("/shared-a", "shared bytes\n", 0o644); + yield* files.link("/shared-a", "/shared-b"); + yield* files.writeFile("/other-a", "shared bytes\n", 0o644); + yield* files.link("/other-a", "/other-b"); + + // Two independent files holding identical bytes, which stay independent. + yield* files.writeFile("/loose-a", "loose bytes\n", 0o644); + yield* files.writeFile("/loose-b", "loose bytes\n", 0o644); + + // A mode a umask would narrow if a creation mode were trusted. + yield* files.writeFile("/group-writable", "wide\n", 0o666); + yield* files.mkdir("/wide-dir", { mode: 0o777 }); +} + +describe("a root the local host retained", () => { + it("materializes and recaptures to the same identity on the runner", function* () { + const root = yield* useStorageRoot(); + yield* withStorage(root, function* () { + const database = yield* createRun(); + const retained = yield* transact(database, function* (workspace) { + yield* buildWorkspace(workspace); + return yield* workspace.capture({ publish: true }); + }); + + // The fixture is the local provider's own capture, not the runner's. + const entries = parseWorkspaceRootManifest(retained.manifest, reject).entries; + const linked = entries.filter((entry) => entry.kind === "file" && entry.hardlink !== null); + expect(linked).toHaveLength(4); + expect(new Set(linked.map((entry) => (entry.kind === "file" ? entry.hardlink : "")))).toEqual( + new Set(["h0", "h1"]), + ); + expect(entries.some((entry) => entry.kind === "symlink")).toBe(true); + expect(entries.some((entry) => entry.kind === "file" && entry.size === 0)).toBe(true); + + const content = retainedContent(runPath(root, database.record.runId)); + const reads = servedBy(retained.manifest, retained.rootId, content); + + yield* scoped(function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); + const tree = yield* trees.create("interoperability"); + const at = (logical: string) => (logical === "/" ? tree : `${tree}${logical}`); + + yield* materializeWorkspaceRoot(files, reads, at, retained.rootId, reject); + const recaptured = yield* captureWorkspace(files, at, reject); + + // One Workspace, two implementations, one identity. + expect(recaptured.root.rootId).toBe(retained.rootId); + expect(recaptured.root.manifest).toBe(retained.manifest); + expect([...recaptured.root.manifests]).toEqual([...retained.manifestHashes]); + expect([...recaptured.root.blobs]).toEqual([...retained.blobHashes]); + }); + }); + }); +}); diff --git a/packages/workflow/tests/remote-materialization.test.ts b/packages/workflow/tests/remote-materialization.test.ts index b5d09d5dd..79c5ec4ff 100644 --- a/packages/workflow/tests/remote-materialization.test.ts +++ b/packages/workflow/tests/remote-materialization.test.ts @@ -195,7 +195,7 @@ describe("materializing a retained Workspace root", () => { // The wide modes survived the umask rather than being narrowed by it. expect(entries.find((entry) => entry.path === "/group-writable")?.mode).toBe(0o666); expect(entries.find((entry) => entry.path === "/wide-dir")?.mode).toBe(0o777); - expect(entries.find((entry) => entry.path === "/docs/link")?.mtime).toBe(1_600_000_000); + expect(entries.find((entry) => entry.path === "/docs/link")?.mtime).toBe(1_600_000_000_000); expect(entries.some((entry) => entry.kind === "file" && entry.size === 0)).toBe(true); // 700 KiB is two chunks at the pinned chunk size, so a file crosses the // transport as more than one piece. From aec12b80a8a29aeba222822cd01a1ae4d9cfbfc0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 3 Sep 2026 22:40:26 -0400 Subject: [PATCH 36/81] =?UTF-8?q?=F0=9F=8E=AF=20Make=20a=20retry=20the=20s?= =?UTF-8?q?ame=20question,=20and=20seal=20only=20after=20teardown=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completing the discriminating regressions `698-e-6.md` asked for surfaced two defects in what had just been built. The command identity was counted rather than derived. The owner recognizes a retry by that identity and returns the decision it already made — so an identity that changed between attempts would ask a second question, and the owner would apply the proposal twice. It is now a digest of the exact bytes being sent: two attempts at one proposal share an identity, two different proposals cannot, and a retry is byte-equivalent. That retry necessarily happens on a new connection, because the one that lost the answer is gone and a connection refuses to reuse a correlation id of its own; the identity is stable across connections, which is where it has to be. And the body ran through `call()`, which let a resource whose teardown failed surface its failure after the commit had already gone out. The comment above it claimed the opposite. The body now runs in a scope closed before the intent is built, so "no commit was sent" and "the body did not finish" really are one statement — the one ordering here that cannot be taken back. The rest of the list is now covered: a journal-only commit proposing nothing and staging nothing, all three retained mapping kinds encoded with the locator only where one belongs, a refused answer and a lost answer each promoting nothing and leaving no attempt tree, and a transaction past its local bound sending nothing. The materialization suite uses the production trees resource rather than a helper of its own. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 21 +- packages/workflow/src/remote/collector.ts | 12 +- .../tests/remote-materialization.test.ts | 25 +- .../workflow/tests/remote-publication.test.ts | 296 +++++++++++++++++- 4 files changed, 326 insertions(+), 28 deletions(-) diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 36901d953..2e06066f7 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -390,11 +390,14 @@ export function cloudflareOwnerLink( }, *commit(intent: CommitIntent): Operation> { - // Minted before anything is sent, and reused for the life of this intent. - const id = nextId(); + // Derived from the request rather than counted. The owner recognizes a + // retry by this identity, so retrying one proposal has to produce the + // identity it already decided — a counter would make the second attempt a + // second question, and the owner would apply it again. + const request = commitRequest(intent); + const id = commandIdentity(request); try { yield* stageMissing(connection, nextId, intent, bytesOf); - const request = commitRequest(intent); const answered = yield* ask(connection, id, request); return answered.outcome === "refused" ? Err(new CloudflareOwnerRefusalError(answered.refusal)) @@ -413,6 +416,18 @@ export function cloudflareOwnerLink( }; } +/** + * The identity one closed command is known by. + * + * A digest of the exact bytes that will be sent, so two attempts at the same + * proposal share an identity and two different proposals cannot. It is bounded + * well inside the correlation limit and carries nothing about the run: it is a + * name for a request, not a fact about the Workspace. + */ +function commandIdentity(request: Record): string { + return `commit-${sha256Hex(JSON.stringify(request))}`; +} + /** One command sent and one answer read, with the private refusal narrowed. */ function* ask( connection: OwnerConnection, diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index 1fec62b12..52a9dba3f 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -22,7 +22,7 @@ * tries to infer what the body did. */ -import { call, ensure, Ok, type Operation, type Result } from "effection"; +import { call, ensure, Ok, type Operation, type Result, scoped } from "effection"; import type { RetainedMapping, WorkspacePublication } from "./publication.ts"; import type { DurableEvent } from "@executablemd/durable-streams"; import type { DurableStream } from "@executablemd/durable-streams"; @@ -236,9 +236,13 @@ export function transactRemotely( let outcome: T; try { - // Everything the body started tears down before the intent is built, so - // "no commit was sent" and "the body did not finish" are one statement. - outcome = yield* call(() => body({ journal }, enlist)); + // A scope of its own, closed here. Everything the body started — + // spawned children, resources — has finished tearing down before the + // intent is built, so "no commit was sent" and "the body did not + // finish" are one statement. `call()` alone would let a resource whose + // teardown fails surface its failure after the commit had already gone + // out, which is the one ordering that cannot be taken back. + outcome = yield* scoped(() => body({ journal }, enlist)); } finally { // The handle is closed before the commit goes out, so a retained // transaction object refuses while the handle-level gate is still held. diff --git a/packages/workflow/tests/remote-materialization.test.ts b/packages/workflow/tests/remote-materialization.test.ts index 79c5ec4ff..ab68ec386 100644 --- a/packages/workflow/tests/remote-materialization.test.ts +++ b/packages/workflow/tests/remote-materialization.test.ts @@ -31,7 +31,7 @@ import { import { tmpdir } from "node:os"; import { join } from "node:path"; import process from "node:process"; -import { runnerFiles } from "../src/deno/remote-files.ts"; +import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; import { captureWorkspace, materializeWorkspaceRoot, @@ -46,21 +46,6 @@ function reject(reason: string): never { throw new Error(reason); } -/** - * A temporary directory owned by the scope that asked for it. - * - * A resource rather than a scoped operation: the tree has to outlive the call - * that created it and end with the invocation that owns it, which is the whole - * lifetime claim materialization makes. - */ -function useTemporaryDirectory(): Operation { - return resource(function* (provide) { - const path = yield* until(mkdtemp(join(tmpdir(), "xmd-materialize-"))); - yield* ensure(() => until(rm(path, { recursive: true, force: true }))); - yield* provide(path); - }); -} - /** Where one logical Workspace path sits under `root`. */ function at(root: string): (logical: string) => string { return (logical) => (logical === "/" ? root : join(root, logical.slice(1))); @@ -168,7 +153,8 @@ describe("materializing a retained Workspace root", () => { // explicit rather than incidental. const previous = process.umask(0o022); const files: RunnerFiles = runnerFiles(); - const source = yield* useTemporaryDirectory(); + const trees = yield* useRunnerTrees(); + const source = yield* trees.create("source"); yield* buildTree(source); const captured = yield* captureWorkspace(files, at(source), reject); @@ -205,7 +191,7 @@ describe("materializing a retained Workspace root", () => { } expect(captured.contents.get(large.manifest)?.chunks).toHaveLength(2); - const destination = yield* useTemporaryDirectory(); + const destination = yield* trees.create("destination"); yield* materializeWorkspaceRoot( files, servedBy(captured), @@ -235,7 +221,8 @@ describe("materializing a retained Workspace root", () => { const files: RunnerFiles = runnerFiles(); let path = ""; yield* scoped(function* () { - path = yield* useTemporaryDirectory(); + const trees = yield* useRunnerTrees(); + path = yield* trees.create("scoped"); yield* until(writeFile(join(path, "present"), "here\n")); }); // The scope that owned it has ended, so the tree is gone rather than left diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index 437f6bfd2..5cfee8bd1 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -14,11 +14,16 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { serializeDurableEvent } from "@executablemd/durable-streams"; -import { type Operation, scoped, sleep, spawn, until } from "effection"; +import { ensure, type Operation, scoped, sleep, spawn, until } from "effection"; import { mkdir, readdir, writeFile } from "node:fs/promises"; +import { agentSessionKey } from "../src/storage/agent-session.ts"; import { cloudflareOwnerLink } from "../src/cloudflare/client.ts"; import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; -import { createTransactionGate, transactRemotely } from "../src/remote/collector.ts"; +import { + type CommitIntent, + createTransactionGate, + transactRemotely, +} from "../src/remote/collector.ts"; import { useAttempt, useMaterialization } from "../src/remote/invocation.ts"; import { type OwnerSocket, type SocketListener, useOwnerConnection } from "../src/remote/client.ts"; import type { @@ -51,6 +56,23 @@ function event(name: string) { const LOCATOR = "https://git.example.invalid/octo/app.git"; +/** The Repository mapping these tests enlist. */ +function repositoryMapping(): RetainedMapping { + return { + kind: "repository", + locator: LOCATOR, + record: { + name: "app", + locatorFingerprint: locatorFingerprintOf(LOCATOR), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/docs", + }, + }; +} + /** A recording connection: every request it was sent, and canned answers. */ function wire(answer: (request: Record) => Record) { const sent: Record[] = []; @@ -458,6 +480,276 @@ describe("what the production runner publishes", () => { expect(text(memberList(commit, "mappings")[0] ?? {}, "locator")).toBe(LOCATOR); }); + it("sends a journal-only commit with no publication and stages nothing", function* () { + const files = runnerFiles(); + yield* scoped(function* () { + const trees = yield* useRunnerTrees(); + void trees; + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const committed = yield* transactRemotely( + link, + createTransactionGate(), + function* (transaction) { + yield* transaction.journal.append(event("noted")); + return "done"; + }, + ); + expect(committed).toMatchObject({ ok: true }); + + const commit = lastCommit(transport.sent); + // A transaction that only appended proposes nothing. Inventing a + // Workspace change to make the shape uniform would publish a root nobody + // asked for, so `publication` is null and nothing was staged. + expect(commit["publication"]).toBe(null); + expect(commit["mappings"]).toEqual([]); + expect(transport.sent.some((request) => request["command"] === "stage")).toBe(false); + }); + void files; + }); + + it("encodes every kind of retained mapping the owner accepts", function* () { + yield* scoped(function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist({ + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: [], + }, + mappings: [ + repositoryMapping(), + { + kind: "worktree", + record: { + repositoryName: "app", + name: "feature", + requestedBranch: "feature", + requestedBase: null, + creationCommit: "2".repeat(40), + checkoutPath: "/docs", + }, + }, + { + kind: "agent-session", + record: { + provider: "acp", + agentCommand: "/usr/bin/agent", + sessionIdentity: "session-1", + sessionKey: agentSessionKey({ + provider: "acp", + agentCommand: "/usr/bin/agent", + sessionIdentity: "session-1", + }), + policy: "strict", + assertion: { kind: "acp-session", value: "abc" }, + createdAt: "2026-09-03T00:00:00.000Z", + }, + }, + ], + }); + return "done"; + }); + const mappings = memberList(lastCommit(transport.sent), "mappings"); + expect(mappings.map((mapping) => mapping["kind"])).toEqual([ + "repository", + "worktree", + "agent-session", + ]); + // Only a Repository carries the locator; the other two are the record. + expect(mappings.filter((mapping) => "locator" in mapping)).toHaveLength(1); + }); + }); + + it("retries a lost answer with the same identity and the same bytes", function* () { + // A retry happens on a new connection: the one that lost the answer is + // gone, and a connection refuses to reuse a correlation id of its own. What + // has to be stable is the identity across those two connections, because + // that is what the owner recognizes the retry by. + const sent: Record[][] = []; + let intent: CommitIntent | undefined; + for (const attempt of [0, 1]) { + yield* scoped(function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + sent.push(transport.sent); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + intent ??= { + expectedWorkspaceRootId: captured.root.rootId, + expectedJournalEventId: null, + events: [], + publication: null, + mappings: [], + }; + const committed = yield* link.commit(intent); + expect([attempt, committed.ok]).toEqual([attempt, true]); + }); + } + + const first = sent[0]?.find((request) => request["command"] === "commit"); + const second = sent[1]?.find((request) => request["command"] === "commit"); + expect(first?.["id"]).toBe(second?.["id"]); + // Byte-equivalent, so the owner sees the request it already decided. + expect(JSON.stringify(first)).toBe(JSON.stringify(second)); + }); + + it("asks a different question for a different proposal", function* () { + yield* scoped(function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const intent: CommitIntent = { + expectedWorkspaceRootId: captured.root.rootId, + expectedJournalEventId: null, + events: [], + publication: null, + mappings: [], + }; + yield* link.commit(intent); + yield* link.commit({ ...intent, events: [event("later")] }); + const commits = transport.sent.filter((request) => request["command"] === "commit"); + expect(commits).toHaveLength(2); + expect(commits[0]?.["id"]).not.toBe(commits[1]?.["id"]); + }); + }); + + it("promotes nothing and keeps no tree when the owner refuses", function* () { + const files = runnerFiles(); + let attemptPath = ""; + yield* scoped(function* () { + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const transport = wire(() => ({ outcome: "refused", refusal: "command:stale-root" })); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + attemptPath = attempt.at("/"); + yield* until(writeFile(attempt.at("/NOTES.md"), "refused\n", { mode: 0o644 })); + const committed = yield* transactRemotely(link, createTransactionGate(), function* () { + return "done"; + }); + // A refusal is an answer, and the answer is no. + expect(committed.ok).toBe(false); + }); + expect(materialization.workspaceRootId).toBe(captured.root.rootId); + }); + let listed: unknown; + try { + listed = yield* until(readdir(attemptPath)); + } catch (error) { + listed = error; + } + expect(listed).toBeInstanceOf(Error); + }); + + it("promotes nothing when the answer is lost", function* () { + const files = runnerFiles(); + yield* scoped(function* () { + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* until(writeFile(attempt.at("/NOTES.md"), "unanswered\n", { mode: 0o644 })); + // The connection goes while the answer is in flight. + transport.silence(); + const asking = yield* spawn(() => + transactRemotely(link, createTransactionGate(), function* () { + return "done"; + }), + ); + yield* sleep(0); + transport.end(); + const committed = yield* asking; + // Undecided, not failed — whether the owner committed cannot be known + // from here. Either way nothing is promoted locally. + expect(committed.ok).toBe(false); + }); + expect(materialization.workspaceRootId).toBe(captured.root.rootId); + }); + }); + + it("sends nothing when a resource the body started fails to tear down", function* () { + let sent: Record[] = []; + let raised: unknown; + try { + yield* scoped(function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + sent = transport.sent; + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + yield* transactRemotely(link, createTransactionGate(), function* (transaction) { + yield* transaction.journal.append(event("appended")); + // A resource whose teardown fails. The body finished, but everything + // it started did not, so the transaction has not finished either — + // and the failure surfaces as the scope unwinds rather than inside it. + yield* ensure(() => { + throw new Error("teardown failed"); + }); + return "done"; + }); + }); + } catch (error) { + raised = error; + } + expect(raised).toBeInstanceOf(Error); + expect(sent.some((request) => request["command"] === "commit")).toBe(false); + }); + + it("sends nothing when the transaction exceeds a local bound", function* () { + yield* scoped(function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + let raised: unknown; + try { + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist({ + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: [], + }, + // More retained mappings than one intent may carry. + mappings: Array.from({ length: 300 }, () => repositoryMapping()), + }); + return "done"; + }); + } catch (error) { + raised = error; + } + expect(raised).toBeInstanceOf(Error); + expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); + }); + }); + it("refuses a second Workspace publication in one transaction", function* () { const { captured, reads } = yield* startingTree(); const transport = wire(ownerAnswers(captured.root.rootId)); From 11236ff9ce8ffe9fc950280a58436c82eab347d6 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 08:14:00 -0400 Subject: [PATCH 37/81] =?UTF-8?q?=F0=9F=A7=BE=20Make=20a=20decision=20surv?= =?UTF-8?q?ive=20the=20connection=20that=20asked=20for=20it=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three ways a committed run could fail to converge. A mutation's decision lived in the acquisition-scoped ledger, and a replacement acquisition discards its predecessor's scratch. That is right for staged bytes and read decisions, which belong to the connection that produced them, and wrong for the fact that a mutation was applied — which is the one thing the next connection needs to tell "this already happened" from "this never happened". So the exact case the retry exists for was the case it could not answer: if the commit landed, the same request met a moved frontier and was refused as stale; if it did not land, the same request performed; and the runner could not tell those histories apart, so it could not know whether to promote or discard. Mutation decisions are now keyed by the run, cleanup never touches them, and a reused identity with different bytes is still a conflict. A decision is not a lease and does not expire because a socket did. A performed answer was read for syntax and thrown away. The adapter accepted an answer naming any root — including the root the run was already at while the proposal published a different one — and returned success. The answer is now checked against what was asked: the root the proposal selected, and one event identity for each event sent. An owner agreeing to something else is not an owner this runner can go on talking to. And promotion took no evidence and moved no bytes. Any holder could call it before a commit, after a refusal or after an ambiguous loss, and it relabelled the accepted tree with the new root while the promoted tree stayed unreachable — so the invocation would read the Workspace it used to be at under the name of the one it is now at. Promotion now requires the owner's performed decision, and that decision has to name the root this attempt actually captured. The attempt's own tree becomes the accepted materialization, the accepted path answers with the promoted bytes, and the tree the run used to be at is removed. Sealing also reached further than it looked. A shallow copy left an Agent session's provider assertion shared with the caller, and the content bytes were fetched through a callback at staging time rather than captured at enlistment — so a caller could change the retained session identity, or the bytes staged under a digest, after the transaction had sealed. Mappings are now copied all the way down and the bytes travel with the intent, checked against the identity they were proposed under before any stage request. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 65 +- .../workflow/src/cloudflare/dispatcher.ts | 47 +- .../workflow/src/cloudflare/private-schema.ts | 38 +- packages/workflow/src/remote/collector.ts | 63 +- packages/workflow/src/remote/invocation.ts | 94 ++- packages/workflow/src/remote/publication.ts | 18 + .../tests/cloudflare/remote-owner.vitest.ts | 69 +-- .../tests/cloudflare/remote-publish.vitest.ts | 52 ++ .../workflow/tests/remote-publication.test.ts | 563 +++++++++++------- .../workflow/tests/remote-transaction.test.ts | 24 +- 10 files changed, 731 insertions(+), 302 deletions(-) diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 2e06066f7..90f2d201e 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -32,6 +32,7 @@ import type { JournalEntry } from "../storage/api.ts"; import { parseMembers, requireMemberNames } from "../storage/members.ts"; import type { DefinitionRetrieval, WorkflowRunRecord } from "../storage/record.ts"; import type { CommitIntent, OwnerLink, StartingFrontier } from "../remote/collector.ts"; +import type { CommitDecision } from "../remote/publication.ts"; import { OwnerLinkError, type OwnerAnswer, type OwnerConnection } from "../remote/client.ts"; import { parseRemoteJournalEntry, @@ -360,15 +361,6 @@ export function cloudflareReadLink( }; } -/** - * The bytes one proposed piece is made of, when the owner has to be sent them. - * - * A runner holds the content it captured; the owner may already hold some of - * it. Supplying bytes by identity lets the adapter stage only what is missing - * rather than resending a Workspace the owner never lost. - */ -export type ProposedBytes = (kind: "manifest" | "blob", digest: string) => Uint8Array | undefined; - /** * The runner's production link to its owner. * @@ -382,14 +374,13 @@ export function cloudflareOwnerLink( connection: OwnerConnection, reads: RemoteReadLink, nextId: () => string, - bytesOf: ProposedBytes = () => undefined, ): OwnerLink { return { *frontier(): Operation { return startingFrontier(yield* reads.frontier()); }, - *commit(intent: CommitIntent): Operation> { + *commit(intent: CommitIntent): Operation> { // Derived from the request rather than counted. The owner recognizes a // retry by this identity, so retrying one proposal has to produce the // identity it already decided — a counter would make the second attempt a @@ -397,11 +388,11 @@ export function cloudflareOwnerLink( const request = commitRequest(intent); const id = commandIdentity(request); try { - yield* stageMissing(connection, nextId, intent, bytesOf); - const answered = yield* ask(connection, id, request); + yield* stageMissing(connection, nextId, intent); + const answered = yield* ask(connection, id, request, intent); return answered.outcome === "refused" ? Err(new CloudflareOwnerRefusalError(answered.refusal)) - : Ok(); + : Ok(answered.decision); } catch (error) { if (error instanceof OwnerLinkError) { // The connection went while the answer was in flight. Whether the @@ -428,31 +419,52 @@ function commandIdentity(request: Record): string { return `commit-${sha256Hex(JSON.stringify(request))}`; } -/** One command sent and one answer read, with the private refusal narrowed. */ +/** + * One command sent and one answer read, checked against what was asked. + * + * A performed answer is not taken on its word. It has to name the root this + * proposal selected — the proposed one when there is a publication, the + * unchanged expected one when there is not — and one event identity for each + * event that was sent. An owner agreeing to something else is not an owner this + * runner can go on talking to: it would promote a Workspace nobody proposed, so + * the channel fails closed instead. + */ function* ask( connection: OwnerConnection, id: string, request: Record, -): Operation<{ outcome: "performed" } | { outcome: "refused"; refusal: PrivateRefusal }> { + intent: CommitIntent, +): Operation< + | { outcome: "performed"; decision: CommitDecision } + | { outcome: "refused"; refusal: PrivateRefusal } +> { + const selected = + intent.publication === null + ? intent.expectedWorkspaceRootId + : intent.publication.proposedWorkspaceRootId; const offered = yield* connection.ask( id, request, - (value) => { - // A performed commit answers with what it published. Reading it proves - // the owner and this build agree about what just happened. + (value): CommitDecision => { const found = members(value, ["workspaceRootId", "journalEventIds"]); - rootId(found.get("workspaceRootId")); + const workspaceRootId = rootId(found.get("workspaceRootId")); const ids = found.get("journalEventIds"); if (!Array.isArray(ids) || ids.some((entry) => typeof entry !== "string" || entry === "")) { return fail("a commit answer did not name the events it retained"); } - return true; + if (workspaceRootId !== selected) { + return fail("a commit answer named a Workspace root this proposal did not select"); + } + if (ids.length !== intent.events.length) { + return fail("a commit answer did not retain one identity for each proposed event"); + } + return Object.freeze({ workspaceRootId, journalEventIds: Object.freeze([...ids]) }); }, privateRefusal, ); return offered.outcome === "refused" ? { outcome: "refused", refusal: privateRefusal(offered.refusal) } - : { outcome: "performed" }; + : { outcome: "performed", decision: offered.value }; } /** @@ -468,18 +480,23 @@ function* stageMissing( connection: OwnerConnection, nextId: () => string, intent: CommitIntent, - bytesOf: ProposedBytes, ): Operation { if (intent.publication === null) { return; } for (const piece of intent.publication.content) { - const bytes = bytesOf(piece.kind, piece.digest); + const bytes = intent.bytes.get(piece.digest); if (bytes === undefined) { // The owner is expected to hold this one already. If it does not, the // commit refuses rather than this guessing at bytes it does not have. continue; } + // The sealed bytes have to be the piece they were sealed as. Staging + // something else would mean the command identity described one proposal and + // the content described another. + if (bytes.length !== piece.size || sha256Hex(bytes) !== piece.digest) { + return fail("a sealed content piece does not match the identity it was proposed under"); + } yield* stageCloudflareContent(connection, nextId(), piece.kind, bytes); } } diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 26df1b59c..607bd08a2 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -46,7 +46,7 @@ import { import { bytesOf, decodeBase64, sha256Hex } from "./encoding.ts"; import { readContent, readFrontier, readJournalPage, readRoot } from "./owner-reads.ts"; import type { OwnerTransactions } from "./owner-transaction.ts"; -import { COMMAND_TABLE, STAGING_TABLE } from "./private-schema.ts"; +import { COMMAND_TABLE, MUTATION_TABLE, STAGING_TABLE } from "./private-schema.ts"; import { applyCommit } from "./publish.ts"; import { recognizeObject } from "./recognition.ts"; @@ -240,6 +240,32 @@ export function dispatchCommand( throw new CommandError("duplicate-conflict"); } recognizeObject(ctx.storage); + + // A mutation's decision is looked for by the run, not by the connection. + // The case this exists for is the one where the connection that asked is + // gone: the owner committed, the answer never arrived, and the runner + // reconnected to ask the same question again. + if (command.command === "commit") { + const decided = ctx.storage.sql + .exec( + `SELECT request_fingerprint, response FROM ${MUTATION_TABLE} WHERE command_id = ?`, + command.id, + ) + .toArray()[0]; + if (decided !== undefined) { + if (decided.request_fingerprint !== fingerprint) { + throw new CommandError("duplicate-conflict"); + } + const decision = storedDecision(decided.response, command.id); + if (decision === "reconstruct") { + // A mutation's decision is always retained whole. Reconstructing one + // would mean applying it again. + throw new Error("private protocol storage holds a malformed result"); + } + return decision; + } + } + const previous = ctx.storage.sql .exec( `SELECT request_fingerprint, response FROM ${COMMAND_TABLE} @@ -286,6 +312,25 @@ export function dispatchCommand( encoded, responseBytes, ); + if (command.command === "commit") { + // Recorded in this same transaction as the mutation it describes, so a + // crash cannot leave one without the other. + const mutations = ctx.storage.sql + .exec(`SELECT count(*) AS decided FROM ${MUTATION_TABLE}`) + .toArray()[0]; + if (integer(mutations?.["decided"]) >= MAX_COMMANDS) { + throw new CommandError("capacity"); + } + ctx.storage.sql.exec( + `INSERT INTO ${MUTATION_TABLE} + (command_id, request_fingerprint, response, response_bytes) + VALUES (?, ?, ?, ?)`, + command.id, + fingerprint, + encoded, + responseBytes, + ); + } return result; }); } diff --git a/packages/workflow/src/cloudflare/private-schema.ts b/packages/workflow/src/cloudflare/private-schema.ts index 4353cf56e..ee3b6ae39 100644 --- a/packages/workflow/src/cloudflare/private-schema.ts +++ b/packages/workflow/src/cloudflare/private-schema.ts @@ -28,6 +28,24 @@ import { normalize, type SchemaObject } from "../sqlite/workflow-schema.ts"; import type { OwnerStorage } from "./storage.ts"; export const COMMAND_TABLE = "_xmd_executor_commands"; +/** + * Decisions about mutations, which outlive the connection that asked for them. + * + * The acquisition-scoped ledger answers a retry on the same socket. It cannot + * answer the case that matters most: the owner committed, the answer was lost, + * and the connection died. A replacement acquisition discards its predecessor's + * scratch — correctly, because staged bytes and read decisions belong to the + * connection that produced them — but the fact that a mutation was applied is + * not scratch. It is the only thing that lets the next connection tell "this + * already happened" from "this never happened", and without it the same request + * meets a moved frontier and is refused as stale while the runner has no way to + * know whether to promote or discard. + * + * So a mutation decision is keyed by the run rather than the acquisition, and + * cleanup never touches it. It is not a lease and does not expire because a + * socket did. + */ +export const MUTATION_TABLE = "_xmd_run_mutations"; export const STAGING_TABLE = "_xmd_executor_staging"; const COMMAND_SQL = `CREATE TABLE ${COMMAND_TABLE} ( @@ -52,15 +70,25 @@ const STAGING_SQL = `CREATE TABLE ${STAGING_TABLE} ( PRIMARY KEY (acquisition_id, kind, digest) ) STRICT, WITHOUT ROWID`; +const MUTATION_SQL = `CREATE TABLE ${MUTATION_TABLE} ( + command_id TEXT PRIMARY KEY, + request_fingerprint TEXT NOT NULL CHECK ( + length(request_fingerprint) = 64 AND request_fingerprint NOT GLOB '*[^0-9a-f]*' + ), + response TEXT NOT NULL CHECK (json_valid(response)), + response_bytes INTEGER NOT NULL CHECK (response_bytes >= 0) +) STRICT, WITHOUT ROWID`; + const PRIVATE_OBJECTS = new Map([ [COMMAND_TABLE, { type: "table", sql: COMMAND_SQL }], [STAGING_TABLE, { type: "table", sql: STAGING_SQL }], + [MUTATION_TABLE, { type: "table", sql: MUTATION_SQL }], ]); export const PRIVATE_OBJECT_NAMES: readonly string[] = Object.freeze([...PRIVATE_OBJECTS.keys()]); export function initializePrivateSchema(storage: OwnerStorage): void { - storage.sql.exec(`${COMMAND_SQL};\n\n${STAGING_SQL};`); + storage.sql.exec(`${COMMAND_SQL};\n\n${STAGING_SQL};\n\n${MUTATION_SQL};`); } export function privateStructureFailure( @@ -79,6 +107,14 @@ export function privateStructureFailure( return undefined; } +/** + * Discard what belonged to a connection that is gone. + * + * Staged bytes and read decisions are that connection's scratch and go with it. + * Mutation decisions deliberately do not: they are how the next connection + * learns that a commit already happened, and deleting one would turn a retry + * into a second mutation or a refusal the runner cannot interpret. + */ export function discardPriorAcquisitions(storage: OwnerStorage, acquisitionId: string): void { storage.sql.exec(`DELETE FROM ${COMMAND_TABLE} WHERE acquisition_id <> ?`, acquisitionId); storage.sql.exec(`DELETE FROM ${STAGING_TABLE} WHERE acquisition_id <> ?`, acquisitionId); diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index 52a9dba3f..85af88b60 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -23,7 +23,7 @@ */ import { call, ensure, Ok, type Operation, type Result, scoped } from "effection"; -import type { RetainedMapping, WorkspacePublication } from "./publication.ts"; +import type { CommitDecision, RetainedMapping, WorkspacePublication } from "./publication.ts"; import type { DurableEvent } from "@executablemd/durable-streams"; import type { DurableStream } from "@executablemd/durable-streams"; import type { WorkflowRunTransaction } from "../storage/api.ts"; @@ -68,6 +68,8 @@ export interface CommitIntent { readonly events: readonly DurableEvent[]; readonly publication: WorkspacePublication | null; readonly mappings: readonly RetainedMapping[]; + /** The sealed bytes for the pieces this proposal may have to supply. */ + readonly bytes: ReadonlyMap; } /** @@ -80,6 +82,20 @@ export interface CommitIntent { export interface WorkspaceEnlistment { readonly publication: WorkspacePublication; readonly mappings: readonly RetainedMapping[]; + /** + * The bytes for every piece the publication names, by identity. + * + * Supplied at enlistment rather than fetched later. The proposal's identity + * is a digest over content, so bytes that could still change after the + * transaction sealed would let one identity describe two different + * Workspaces — and the adapter would stage whatever the buffer happened to + * hold by the time it looked. + * + * Only pieces the owner may not already hold need appear. What is absent is + * content the owner is expected to have, and a proposal naming content + * nobody can supply is refused rather than guessed at. + */ + readonly bytes: ReadonlyMap; } /** What the collector needs from the connection. */ @@ -87,7 +103,7 @@ export interface OwnerLink { /** One bounded read that opens and closes its own owner-side read. */ frontier(): Operation; /** One closed intent, applied atomically or not at all. */ - commit(intent: CommitIntent): Operation>; + commit(intent: CommitIntent): Operation>; } /** The most events one intent may carry. */ @@ -256,6 +272,7 @@ export function transactRemotely( events: appended.map((event) => structuredClone(event)), publication: enlisted?.publication ?? null, mappings: enlisted?.mappings ?? [], + bytes: enlisted?.bytes ?? new Map(), }); if (!committed.ok) { return committed; @@ -283,14 +300,44 @@ function detach(proposal: WorkspaceEnlistment): WorkspaceEnlistment { proposedWorkspaceRootId: proposal.publication.proposedWorkspaceRootId, proposedManifest: proposal.publication.proposedManifest, content: Object.freeze( - proposal.publication.content.map((piece) => Object.freeze({ ...piece })), + proposal.publication.content.map((piece) => + Object.freeze({ kind: piece.kind, digest: piece.digest, size: piece.size }), + ), ), }), - mappings: Object.freeze( - proposal.mappings.map((mapping) => - Object.freeze({ ...mapping, record: Object.freeze({ ...mapping.record }) }), - ), - ) as readonly RetainedMapping[], + mappings: Object.freeze(proposal.mappings.map(detachMapping)), + // A copy of every buffer, not a reference to one. A caller that goes on + // writing into the array it captured must not be able to change what this + // proposal stages. + bytes: new Map([...proposal.bytes].map(([digest, bytes]) => [digest, bytes.slice()])), + }); +} + +/** + * One mapping, copied all the way down. + * + * A shallow copy is not enough: an Agent-session record holds its provider + * assertion as a nested object, and that assertion is part of the retained + * identity. Leaving it shared would let a caller change what the run recorded + * about a session after the transaction had sealed. + */ +function detachMapping(mapping: RetainedMapping): RetainedMapping { + if (mapping.kind === "repository") { + return Object.freeze({ + kind: mapping.kind, + locator: mapping.locator, + record: Object.freeze({ ...mapping.record }), + }); + } + if (mapping.kind === "worktree") { + return Object.freeze({ kind: mapping.kind, record: Object.freeze({ ...mapping.record }) }); + } + return Object.freeze({ + kind: mapping.kind, + record: Object.freeze({ + ...mapping.record, + assertion: Object.freeze({ ...mapping.record.assertion }), + }), }); } diff --git a/packages/workflow/src/remote/invocation.ts b/packages/workflow/src/remote/invocation.ts index ffb6c0569..d736fcc1c 100644 --- a/packages/workflow/src/remote/invocation.ts +++ b/packages/workflow/src/remote/invocation.ts @@ -31,6 +31,7 @@ import { type RunnerFiles, } from "./materialize.ts"; import type { RemoteReadLink } from "./read.ts"; +import type { CommitDecision } from "./publication.ts"; /** A directory this invocation owns for as long as it needs one. */ export interface TemporaryTrees { @@ -44,15 +45,15 @@ export interface TemporaryTrees { export interface Materialization { /** The root this tree is, as the owner confirmed it. */ readonly workspaceRootId: string; - /** Where a logical Workspace path sits in this tree. */ - readonly at: HostPath; /** - * Record which root this tree now is. + * Where a logical Workspace path sits in the accepted tree. * - * Called by a promoted attempt and by nothing else. It moves no bytes: the - * attempt did that, and this says what they are. + * Resolved on each call rather than closed over one directory, because + * promotion replaces the tree: after it, this has to answer with the promoted + * bytes. A path captured once would keep pointing at the Workspace the run + * used to be at while the identity said otherwise. */ - accept(workspaceRootId: string): void; + at(logical: string): string; } /** One disposable place to make a mutation, and the way to keep it. */ @@ -63,12 +64,14 @@ export interface Attempt { /** * Make this attempt the accepted materialization. * - * Called only after the owner reports the commit performed. Anything else — - * a refusal, a lost response, a local failure — leaves the accepted tree - * where it was, because until the owner says otherwise the run is still at - * the root it started from. + * It takes the owner's performed decision because that decision is the + * authority: nothing else may promote, and a decision naming a different root + * than this attempt captured is not this attempt's decision. Passing it is + * the proof, which is why there is no argument-free way to do this — a + * refusal, an ambiguous loss and a local failure all leave the caller with + * nothing to pass. */ - promote(): Operation; + promote(decision: CommitDecision): Operation; } /** @@ -85,24 +88,41 @@ export function useMaterialization( reads: RemoteReadLink, workspaceRootId: string, reject: WorkspaceRejection, -): Operation { +): Operation { return resource(function* (provide) { const root = yield* trees.create("accepted"); - const at: HostPath = (logical) => join(root, logical); - yield* materializeWorkspaceRoot(files, reads, at, workspaceRootId, reject); - let accepted = workspaceRootId; + yield* materializeWorkspaceRoot(files, reads, at(root), workspaceRootId, reject); + let accepted = { root, workspaceRootId }; yield* provide({ get workspaceRootId(): string { - return accepted; + return accepted.workspaceRootId; + }, + at(logical: string): string { + return at(accepted.root)(logical); }, - at, - accept(next: string): void { + *replace(next: { root: string; workspaceRootId: string }): Operation { + const previous = accepted.root; accepted = next; + // The tree the run used to be at is removed once nothing points at it. + // Leaving it would keep a second copy of the Workspace on disk that + // nothing can reach and nothing will clean up until the invocation ends. + yield* trees.remove(previous); }, }); }); } +/** + * The accepted materialization, plus the one operation that may move it. + * + * `replace` is not on `Materialization` because everything that merely reads + * the Workspace should not be able to change which Workspace it is reading. + * Only an attempt holding a performed decision reaches this. + */ +export interface AcceptedMaterialization extends Materialization { + replace(next: { root: string; workspaceRootId: string }): Operation; +} + /** * A disposable copy of the accepted tree, for one mutation. * @@ -115,13 +135,18 @@ export function useAttempt( files: RunnerFiles, trees: TemporaryTrees, reads: RemoteReadLink, - materialization: Materialization, + materialization: AcceptedMaterialization, reject: WorkspaceRejection, ): Operation { return resource(function* (provide) { const root = yield* trees.create("attempt"); - const at: HostPath = (logical) => join(root, logical); - yield* materializeWorkspaceRoot(files, reads, at, materialization.workspaceRootId, reject); + yield* materializeWorkspaceRoot( + files, + reads, + at(root), + materialization.workspaceRootId, + reject, + ); let promoted = false; // Registered before the attempt is handed over, so every exit removes it — @@ -133,14 +158,27 @@ export function useAttempt( }); yield* provide({ - at, + at: at(root), *capture(): Operation { - return yield* captureWorkspace(files, at, reject); + return yield* captureWorkspace(files, at(root), reject); }, - *promote(): Operation { - const captured = yield* captureWorkspace(files, at, reject); + *promote(decision: CommitDecision): Operation { + if (promoted) { + // One decision promotes one attempt once. A second promotion would be + // moving the accepted tree somewhere it has already been moved from. + reject("this attempt has already been promoted"); + } + const captured = yield* captureWorkspace(files, at(root), reject); + if (decision.workspaceRootId !== captured.root.rootId) { + // The owner published something other than what this attempt holds. + // Promoting would label these bytes with a root they are not. + reject("the owner's decision names a root this attempt did not capture"); + } promoted = true; - materialization.accept(captured.root.rootId); + // The tree itself becomes the accepted one. Recording the identity + // without moving the bytes would leave the invocation reading the + // Workspace it used to be at under the name of the one it is now at. + yield* materialization.replace({ root, workspaceRootId: captured.root.rootId }); }, }); }); @@ -154,6 +192,6 @@ export function useAttempt( * host's path conventions. The logical root is `/` and everything under it is * relative to the tree this invocation was given. */ -function join(root: string, logical: string): string { - return logical === "/" ? root : `${root}/${logical.slice(1)}`; +function at(root: string): HostPath { + return (logical) => (logical === "/" ? root : `${root}/${logical.slice(1)}`); } diff --git a/packages/workflow/src/remote/publication.ts b/packages/workflow/src/remote/publication.ts index 73f157645..2c7371b5b 100644 --- a/packages/workflow/src/remote/publication.ts +++ b/packages/workflow/src/remote/publication.ts @@ -73,3 +73,21 @@ export type RetainedMapping = } | { readonly kind: "worktree"; readonly record: WorktreeRecord } | { readonly kind: "agent-session"; readonly record: AgentSessionRecord }; + +/** + * What the owner did, as the runner is allowed to know it. + * + * Returned by a performed commit and by nothing else. It names the root the + * commit selected and the identities the owner minted for the events it + * retained, which is what makes it checkable: a runner can compare the answer + * with the proposal it sent and refuse an owner that agreed to something else. + * + * It is also the only thing that authorizes a local promotion. Passing it is + * how an attempt proves the owner published *that* Workspace — a promotion + * that took no evidence would be the runner deciding on the owner's behalf, + * and the two would disagree the first time a commit was refused. + */ +export interface CommitDecision { + readonly workspaceRootId: string; + readonly journalEventIds: readonly string[]; +} diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index 5c50cf934..c7519c8b1 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -406,15 +406,13 @@ describe("the remote owner protocol", () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); await on(stub, (owner) => owner.appendJournal("before", "before")); - await admit(stub); - const first = await send(stub, "same", { command: "frontier" }); + const socket = await connect(stub); + const first = await ask(socket, "same", { command: "frontier" }); await on(stub, (owner) => owner.appendJournal("after", "after")); - expect( - await on(stub, (owner) => - record(owner.send(1, JSON.stringify({ command: "frontier", id: "same" }))), - ), - ).toEqual(first); - expect(await send(stub, "same", { command: "root", workspaceRootId: ROOT_ID })).toEqual({ + // The same id and the same canonical request returns the anchored frontier + // it already decided, not the later one. + expect(await ask(socket, "same", { command: "frontier" })).toEqual(first); + expect(await ask(socket, "same", { command: "root", workspaceRootId: ROOT_ID })).toEqual({ id: "same", outcome: "refused", refusal: "command:duplicate-conflict", @@ -497,36 +495,35 @@ describe("the remote owner protocol", () => { }); it("leaves no staged row when decoding or digest validation fails", async () => { - const stub = executor(); - await on(stub, (owner) => owner.initialize()); - await admit(stub); const bytes = new TextEncoder().encode("piece"); - expect( - await send(stub, "bad-base64", { - command: "stage", - kind: "blob", - digest: sha256Hex(bytes), - bytes: "not base64", - }), - ).toMatchObject({ outcome: "refused", refusal: "command:malformed-member" }); - expect( - await send(stub, "bad-digest", { - command: "stage", - kind: "blob", - digest: "0".repeat(64), - bytes: encodeBase64(bytes), - }), - ).toMatchObject({ outcome: "refused", refusal: "command:malformed-member" }); const oversized = new Uint8Array(MAX_CONTENT_BYTES + 1); - expect( - await send(stub, "oversized", { - command: "stage", - kind: "blob", - digest: sha256Hex(oversized), - bytes: encodeBase64(oversized), - }), - ).toMatchObject({ outcome: "refused", refusal: "command:too-large" }); - expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 0, staged: 0 }); + // Each of these is a broken channel rather than an answer, so each closes + // the connection it arrived on — which is why every case gets its own. + const cases: Record, string]> = { + "bad base64": [ + { kind: "blob", digest: sha256Hex(bytes), bytes: "not base64" }, + "command:malformed-member", + ], + "a digest that is not the bytes": [ + { kind: "blob", digest: "0".repeat(64), bytes: encodeBase64(bytes) }, + "command:malformed-member", + ], + "a piece past the bound": [ + { kind: "blob", digest: sha256Hex(oversized), bytes: encodeBase64(oversized) }, + "command:too-large", + ], + }; + for (const [description, [request, refusal]] of Object.entries(cases)) { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + const answer = await ask(socket, "staged", { command: "stage", ...request }); + expect([description, answer["refusal"]]).toEqual([description, refusal]); + expect([description, await on(stub, (owner) => owner.scratch())]).toEqual([ + description, + { commands: 0, staged: 0 }, + ]); + } }); it("refuses aggregate staging overflow without a partial piece or decision", async () => { diff --git a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts index 5bf7b210c..d4616e2c1 100644 --- a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts @@ -669,6 +669,58 @@ describe("publishing one proposal", () => { expect(await on(stub, (owner) => owner.scratch())).toMatchObject({ commands: 2 }); }); + it("returns the same decision to a connection that replaced the one that lost it", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + await stageThrough(socket); + + // The owner commits. The runner never sees the answer, and the connection + // that asked is gone — which is exactly the case the acquisition-scoped + // ledger cannot answer, because a replacement acquisition discards it. + const first = await ask(socket, "recovered", commit()); + expect(first).toMatchObject({ outcome: "performed" }); + const published = await on(stub, (owner) => owner.published()); + socket.close(1000, "lost"); + await new Promise((resolve) => setTimeout(resolve, 0)); + await evictDurableObject(stub); + + // A new connection, a new acquisition, the identical closed request. + const replacement = await connect(stub); + expect(await ask(replacement, "recovered", commit())).toEqual(first); + + // One root, one mapping, one set of journal rows. + expect(await on(stub, (owner) => owner.published())).toEqual(published); + + // And the identity still cannot be reused for something else. + expect( + await ask(replacement, "recovered", commit({ events: [event("different")] })), + ).toMatchObject({ outcome: "refused", refusal: "command:duplicate-conflict" }); + expect(await on(stub, (owner) => owner.published())).toEqual(published); + }); + + it("performs a proposal whose first attempt never reached the owner", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + await stageThrough(socket); + const before = await on(stub, (owner) => owner.published()); + + // The first attempt was lost on the way out, so the owner never saw it. + socket.close(1000, "lost before arriving"); + await new Promise((resolve) => setTimeout(resolve, 0)); + await evictDurableObject(stub); + + const replacement = await connect(stub); + await stageThrough(replacement); + expect(await ask(replacement, "never-arrived", commit())).toMatchObject({ + outcome: "performed", + }); + const after = await on(stub, (owner) => owner.published()); + expect(after["currentRootId"]).toBe(NEXT_ROOT_ID); + expect(after).not.toEqual(before); + }); + it("grants a closed or foreign socket no publication", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index 5cfee8bd1..e6a5ccec0 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -15,9 +15,10 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { serializeDurableEvent } from "@executablemd/durable-streams"; import { ensure, type Operation, scoped, sleep, spawn, until } from "effection"; -import { mkdir, readdir, writeFile } from "node:fs/promises"; +import { mkdir, readdir, readFile, writeFile } from "node:fs/promises"; import { agentSessionKey } from "../src/storage/agent-session.ts"; import { cloudflareOwnerLink } from "../src/cloudflare/client.ts"; +import { encodeBase64 } from "../src/cloudflare/encoding.ts"; import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; import { type CommitIntent, @@ -56,6 +57,34 @@ function event(name: string) { const LOCATOR = "https://git.example.invalid/octo/app.git"; +/** The exact closure a proposed root names, in canonical order. */ +function inventoryOf(captured: CapturedWorkspace): ProposedContent[] { + return [ + ...captured.root.manifests.map((digest) => ({ + kind: "manifest" as const, + digest, + size: captured.contents.get(digest)?.manifestBytes.length ?? 0, + })), + ...captured.root.blobs.map((digest) => ({ + kind: "blob" as const, + digest, + size: captured.blobs.get(digest)?.length ?? 0, + })), + ]; +} + +/** Every piece a capture can supply, by identity. */ +function proposedBytes(captured: CapturedWorkspace): Map { + const bytes = new Map(); + for (const [digest, content] of captured.contents) { + bytes.set(digest, content.manifestBytes); + } + for (const [digest, blob] of captured.blobs) { + bytes.set(digest, blob); + } + return bytes; +} + /** The Repository mapping these tests enlist. */ function repositoryMapping(): RetainedMapping { return { @@ -118,11 +147,15 @@ function wire(answer: (request: Record) => Record = new Map()) { +function ownerAnswers(_rootId: string, sizes: ReadonlyMap = new Map()) { return (request: Record): Record => { if (request["command"] === "stage") { return { @@ -134,9 +167,18 @@ function ownerAnswers(rootId: string, sizes: ReadonlyMap = new M }, }; } + const publication = request["publication"]; + const selected = + publication === null || publication === undefined + ? request["expectedWorkspaceRootId"] + : (publication as Record)["proposedWorkspaceRootId"]; + const events = Array.isArray(request["events"]) ? request["events"] : []; return { outcome: "performed", - value: { workspaceRootId: rootId, journalEventIds: ["e1"] }, + value: { + workspaceRootId: selected, + journalEventIds: events.map((_entry, index) => `event-${index}`), + }, }; }; } @@ -258,92 +300,73 @@ function* startingTree(): Operation<{ captured: CapturedWorkspace; reads: Remote describe("what the production runner publishes", () => { it("sends one closed commit describing everything the transaction decided", function* () { const files = runnerFiles(); - yield* scoped(function* () { - const trees = yield* useRunnerTrees(); - const { captured, reads } = yield* startingTree(); - const sizes = new Map(); - const transport = wire(ownerAnswers(captured.root.rootId, sizes)); - const connection = yield* useOwnerConnection(transport.socket); + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const sizes = new Map(); + const transport = wire(ownerAnswers(captured.root.rootId, sizes)); + const connection = yield* useOwnerConnection(transport.socket); - const materialization = yield* useMaterialization( - files, - trees, - reads, - captured.root.rootId, - reject, - ); - const attempt = yield* useAttempt(files, trees, reads, materialization, reject); - yield* until(writeFile(attempt.at("/NOTES.md"), "written by the effect\n", { mode: 0o644 })); - const proposed = yield* attempt.capture(); - for (const [digest, content] of proposed.contents) { - sizes.set(digest, content.manifestBytes.length); - } - for (const [digest, bytes] of proposed.blobs) { - sizes.set(digest, bytes.length); - } + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* until(writeFile(attempt.at("/NOTES.md"), "written by the effect\n", { mode: 0o644 })); + const proposed = yield* attempt.capture(); + for (const [digest, content] of proposed.contents) { + sizes.set(digest, content.manifestBytes.length); + } + for (const [digest, bytes] of proposed.blobs) { + sizes.set(digest, bytes.length); + } - const link = cloudflareOwnerLink(connection, reads, ids(), (kind, digest) => - kind === "manifest" - ? proposed.contents.get(digest)?.manifestBytes - : proposed.blobs.get(digest), - ); - const committed = yield* transactRemotely( - link, - createTransactionGate(), - function* (transaction, enlist) { - yield* transaction.journal.append(event("published")); - enlist({ - publication: { - proposedWorkspaceRootId: proposed.root.rootId, - proposedManifest: proposed.root.manifest, - content: [ - ...proposed.root.manifests.map((digest) => ({ - kind: "manifest" as const, - digest, - size: proposed.contents.get(digest)?.manifestBytes.length ?? 0, - })), - ...proposed.root.blobs.map((digest) => ({ - kind: "blob" as const, - digest, - size: proposed.blobs.get(digest)?.length ?? 0, - })), - ], - }, - mappings: [ - { - kind: "repository", - locator: LOCATOR, - record: { - name: "app", - locatorFingerprint: locatorFingerprintOf(LOCATOR), - requestedBase: null, - creationCommit: "9".repeat(40), - primaryBranch: "main", - objectFormat: "sha1", - checkoutPath: "/docs", - }, - }, + const link = cloudflareOwnerLink(connection, reads, ids()); + const committed = yield* transactRemotely( + link, + createTransactionGate(), + function* (transaction, enlist) { + yield* transaction.journal.append(event("published")); + enlist({ + publication: { + proposedWorkspaceRootId: proposed.root.rootId, + proposedManifest: proposed.root.manifest, + content: [ + ...proposed.root.manifests.map((digest) => ({ + kind: "manifest" as const, + digest, + size: proposed.contents.get(digest)?.manifestBytes.length ?? 0, + })), + ...proposed.root.blobs.map((digest) => ({ + kind: "blob" as const, + digest, + size: proposed.blobs.get(digest)?.length ?? 0, + })), ], - }); - return "done"; - }, - ); - expect(committed).toMatchObject({ ok: true }); + }, + mappings: [repositoryMapping()], + bytes: proposedBytes(proposed), + }); + return "done"; + }, + ); + expect(committed).toMatchObject({ ok: true }); - // The last request is one closed commit carrying the whole proposal. - const commit = lastCommit(transport.sent); - expect(commit["expectedWorkspaceRootId"]).toBe(captured.root.rootId); - expect(commit["events"]).toEqual([serializeDurableEvent(event("published"))]); - const publication = member(commit, "publication"); - expect(publication["proposedWorkspaceRootId"]).toBe(proposed.root.rootId); - expect(sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${String(publication["proposedManifest"])}`)).toBe( - proposed.root.rootId, - ); - expect(text(memberList(commit, "mappings")[0] ?? {}, "locator")).toBe(LOCATOR); - // Everything the proposal names was staged before the commit went out. - const staged = transport.sent.filter((request) => request["command"] === "stage"); - expect(staged.length).toBe(proposed.root.manifests.length + proposed.root.blobs.length); - }); + // The last request is one closed commit carrying the whole proposal. + const commit = lastCommit(transport.sent); + expect(commit["expectedWorkspaceRootId"]).toBe(captured.root.rootId); + expect(commit["events"]).toEqual([serializeDurableEvent(event("published"))]); + const publication = member(commit, "publication"); + expect(publication["proposedWorkspaceRootId"]).toBe(proposed.root.rootId); + expect(sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${String(publication["proposedManifest"])}`)).toBe( + proposed.root.rootId, + ); + expect(text(memberList(commit, "mappings")[0] ?? {}, "locator")).toBe(LOCATOR); + // Everything the proposal names was staged before the commit went out. + const staged = transport.sent.filter((request) => request["command"] === "stage"); + expect(staged.length).toBe(proposed.root.manifests.length + proposed.root.blobs.length); }); it("sends no commit and keeps no tree when the body does not finish", function* () { @@ -408,33 +431,76 @@ describe("what the production runner publishes", () => { void outcomes; }); - it("keeps the accepted root until the owner performs the commit", function* () { + it("promotes only with the owner's decision, and moves the tree with it", function* () { const files = runnerFiles(); - yield* scoped(function* () { - const trees = yield* useRunnerTrees(); - const { captured, reads } = yield* startingTree(); - const materialization = yield* useMaterialization( - files, - trees, - reads, - captured.root.rootId, - reject, - ); - yield* scoped(function* () { - const attempt = yield* useAttempt(files, trees, reads, materialization, reject); - yield* until(writeFile(attempt.at("/NOTES.md"), "not published\n", { mode: 0o644 })); - // The owner refused, so nothing promotes. - }); - expect(materialization.workspaceRootId).toBe(captured.root.rootId); + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const sizes = new Map(); + const transport = wire(ownerAnswers("", sizes)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + const acceptedBefore = materialization.at("/"); + + let attemptRoot = ""; + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + attemptRoot = attempt.at("/"); + yield* until(writeFile(attempt.at("/NOTES.md"), "published\n", { mode: 0o644 })); + const proposed = yield* attempt.capture(); + for (const [digest, content] of proposed.contents) { + sizes.set(digest, content.manifestBytes.length); + } + for (const [digest, blob] of proposed.blobs) { + sizes.set(digest, blob.length); + } - yield* scoped(function* () { - const attempt = yield* useAttempt(files, trees, reads, materialization, reject); - yield* until(writeFile(attempt.at("/NOTES.md"), "published\n", { mode: 0o644 })); - yield* attempt.promote(); - }); - // Only a promoted attempt moves what the run is at. - expect(materialization.workspaceRootId).not.toBe(captured.root.rootId); + // A decision the owner did not give cannot promote. + let raised: unknown; + try { + yield* attempt.promote({ workspaceRootId: captured.root.rootId, journalEventIds: [] }); + } catch (error) { + raised = error; + } + expect(raised).toBeInstanceOf(Error); + expect(materialization.workspaceRootId).toBe(captured.root.rootId); + + // The owner's own answer, naming the root this attempt captured. + const committed = yield* link.commit({ + expectedWorkspaceRootId: captured.root.rootId, + expectedJournalEventId: null, + events: [], + publication: { + proposedWorkspaceRootId: proposed.root.rootId, + proposedManifest: proposed.root.manifest, + content: inventoryOf(proposed), + }, + mappings: [], + bytes: proposedBytes(proposed), }); + if (!committed.ok) { + throw committed.error; + } + yield* attempt.promote(committed.value); + + // The accepted materialization is the promoted tree, not a relabelled + // copy of the old one: the file the effect wrote is readable through it. + expect(materialization.workspaceRootId).not.toBe(captured.root.rootId); + expect(materialization.at("/")).toBe(attemptRoot); + expect(yield* until(readFile(materialization.at("/NOTES.md"), "utf8"))).toBe("published\n"); + // And the tree the run used to be at is gone. + let listed: unknown; + try { + listed = yield* until(readdir(acceptedBefore)); + } catch (error) { + listed = error; + } + expect(listed).toBeInstanceOf(Error); }); it("cannot be changed by a caller that kept its own copy", function* () { @@ -466,6 +532,7 @@ describe("what the production runner publishes", () => { content, }, mappings, + bytes: new Map(), }); // The caller still holds both arrays and edits them after admission. content.push({ kind: "blob", digest: "b".repeat(64), size: 2 }); @@ -511,60 +578,59 @@ describe("what the production runner publishes", () => { }); it("encodes every kind of retained mapping the owner accepts", function* () { - yield* scoped(function* () { - const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); - const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareOwnerLink(connection, reads, ids()); - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist({ - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content: [], - }, - mappings: [ - repositoryMapping(), - { - kind: "worktree", - record: { - repositoryName: "app", - name: "feature", - requestedBranch: "feature", - requestedBase: null, - creationCommit: "2".repeat(40), - checkoutPath: "/docs", - }, + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist({ + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: [], + }, + bytes: new Map(), + mappings: [ + repositoryMapping(), + { + kind: "worktree", + record: { + repositoryName: "app", + name: "feature", + requestedBranch: "feature", + requestedBase: null, + creationCommit: "2".repeat(40), + checkoutPath: "/docs", }, - { - kind: "agent-session", - record: { + }, + { + kind: "agent-session", + record: { + provider: "acp", + agentCommand: "/usr/bin/agent", + sessionIdentity: "session-1", + sessionKey: agentSessionKey({ provider: "acp", agentCommand: "/usr/bin/agent", sessionIdentity: "session-1", - sessionKey: agentSessionKey({ - provider: "acp", - agentCommand: "/usr/bin/agent", - sessionIdentity: "session-1", - }), - policy: "strict", - assertion: { kind: "acp-session", value: "abc" }, - createdAt: "2026-09-03T00:00:00.000Z", - }, + }), + policy: "strict", + assertion: { kind: "acp-session", value: "abc" }, + createdAt: "2026-09-03T00:00:00.000Z", }, - ], - }); - return "done"; + }, + ], }); - const mappings = memberList(lastCommit(transport.sent), "mappings"); - expect(mappings.map((mapping) => mapping["kind"])).toEqual([ - "repository", - "worktree", - "agent-session", - ]); - // Only a Repository carries the locator; the other two are the record. - expect(mappings.filter((mapping) => "locator" in mapping)).toHaveLength(1); + return "done"; }); + const mappings = memberList(lastCommit(transport.sent), "mappings"); + expect(mappings.map((mapping) => mapping["kind"])).toEqual([ + "repository", + "worktree", + "agent-session", + ]); + // Only a Repository carries the locator; the other two are the record. + expect(mappings.filter((mapping) => "locator" in mapping)).toHaveLength(1); }); it("retries a lost answer with the same identity and the same bytes", function* () { @@ -587,6 +653,7 @@ describe("what the production runner publishes", () => { events: [], publication: null, mappings: [], + bytes: new Map(), }; const committed = yield* link.commit(intent); expect([attempt, committed.ok]).toEqual([attempt, true]); @@ -601,24 +668,23 @@ describe("what the production runner publishes", () => { }); it("asks a different question for a different proposal", function* () { - yield* scoped(function* () { - const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); - const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareOwnerLink(connection, reads, ids()); - const intent: CommitIntent = { - expectedWorkspaceRootId: captured.root.rootId, - expectedJournalEventId: null, - events: [], - publication: null, - mappings: [], - }; - yield* link.commit(intent); - yield* link.commit({ ...intent, events: [event("later")] }); - const commits = transport.sent.filter((request) => request["command"] === "commit"); - expect(commits).toHaveLength(2); - expect(commits[0]?.["id"]).not.toBe(commits[1]?.["id"]); - }); + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const intent: CommitIntent = { + expectedWorkspaceRootId: captured.root.rootId, + expectedJournalEventId: null, + events: [], + publication: null, + mappings: [], + bytes: new Map(), + }; + yield* link.commit(intent); + yield* link.commit({ ...intent, events: [event("later")] }); + const commits = transport.sent.filter((request) => request["command"] === "commit"); + expect(commits).toHaveLength(2); + expect(commits[0]?.["id"]).not.toBe(commits[1]?.["id"]); }); it("promotes nothing and keeps no tree when the owner refuses", function* () { @@ -723,31 +789,127 @@ describe("what the production runner publishes", () => { }); it("sends nothing when the transaction exceeds a local bound", function* () { - yield* scoped(function* () { - const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); - const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareOwnerLink(connection, reads, ids()); - let raised: unknown; - try { - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist({ - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content: [], - }, - // More retained mappings than one intent may carry. - mappings: Array.from({ length: 300 }, () => repositoryMapping()), - }); - return "done"; + const { captured, reads } = yield* startingTree(); + const transport = wire(ownerAnswers(captured.root.rootId)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + let raised: unknown; + try { + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist({ + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: [], + }, + // More retained mappings than one intent may carry. + mappings: Array.from({ length: 300 }, () => repositoryMapping()), + bytes: new Map(), }); - } catch (error) { - raised = error; - } - expect(raised).toBeInstanceOf(Error); - expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); + return "done"; + }); + } catch (error) { + raised = error; + } + expect(raised).toBeInstanceOf(Error); + expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); + }); + + it("refuses a performed answer that names a root this proposal did not select", function* () { + const { captured, reads } = yield* startingTree(); + // An owner agreeing to something else is not an owner this runner can go + // on talking to: believing it would promote a Workspace nobody proposed. + const transport = wire(() => ({ + outcome: "performed", + value: { workspaceRootId: "f".repeat(64), journalEventIds: [] }, + })); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const committed = yield* link.commit({ + expectedWorkspaceRootId: captured.root.rootId, + expectedJournalEventId: null, + events: [], + publication: null, + mappings: [], + bytes: new Map(), + }); + expect(committed.ok).toBe(false); + }); + + it("refuses a performed answer that loses an event it was given", function* () { + const { captured, reads } = yield* startingTree(); + const transport = wire((request) => ({ + outcome: "performed", + value: { workspaceRootId: request["expectedWorkspaceRootId"], journalEventIds: [] }, + })); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const committed = yield* link.commit({ + expectedWorkspaceRootId: captured.root.rootId, + expectedJournalEventId: null, + events: [event("appended")], + publication: null, + mappings: [], + bytes: new Map(), + }); + // One identity per event, or the two sides disagree about what history + // this commit created. + expect(committed.ok).toBe(false); + }); + + it("seals nested mapping values and content bytes against later mutation", function* () { + const { captured, reads } = yield* startingTree(); + const sizes = new Map(); + const transport = wire(ownerAnswers("", sizes)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + + const assertion = { kind: "acp-session", value: "admitted" }; + const identity = { + provider: "acp", + agentCommand: "/usr/bin/agent", + sessionIdentity: "session-1", + }; + const piece = new TextEncoder().encode("admitted bytes"); + const digest = sha256Hex(piece); + sizes.set(digest, piece.length); + const bytes = new Map([[digest, piece]]); + + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist({ + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: [{ kind: "blob", digest, size: piece.length }], + }, + mappings: [ + { + kind: "agent-session", + record: { + ...identity, + sessionKey: agentSessionKey(identity), + policy: "strict", + assertion, + createdAt: "2026-09-03T00:00:00.000Z", + }, + }, + ], + bytes, + }); + // The caller still holds the assertion object and the byte buffer, and + // edits both after the transaction admitted them. + assertion.value = "changed after admission"; + piece.fill(0); + bytes.set(digest, new TextEncoder().encode("substituted")); + return "done"; }); + + // What was sent is what was admitted, not what the caller did afterwards. + const mapping = memberList(lastCommit(transport.sent), "mappings")[0] ?? {}; + expect(text(member(member(mapping, "record"), "assertion"), "value")).toBe("admitted"); + const staged = transport.sent.find((request) => request["command"] === "stage"); + expect(staged?.["digest"]).toBe(digest); + expect(staged?.["bytes"]).toBe(encodeBase64(new TextEncoder().encode("admitted bytes"))); }); it("refuses a second Workspace publication in one transaction", function* () { @@ -762,6 +924,7 @@ describe("what the production runner publishes", () => { content: [], }, mappings: [], + bytes: new Map(), }; let raised: unknown; try { diff --git a/packages/workflow/tests/remote-transaction.test.ts b/packages/workflow/tests/remote-transaction.test.ts index 5d5a2d794..1d19b4dd3 100644 --- a/packages/workflow/tests/remote-transaction.test.ts +++ b/packages/workflow/tests/remote-transaction.test.ts @@ -26,6 +26,7 @@ import { type StartingFrontier, transactRemotely, } from "../src/remote/collector.ts"; +import type { CommitDecision } from "../src/remote/publication.ts"; /** * What the transaction refused with, having proved it refused at all. @@ -88,6 +89,19 @@ function event(name: string): DurableEvent { }; } +/** What a correct owner would answer for this intent. */ +function decisionFor(intent: CommitIntent): CommitDecision { + return { + workspaceRootId: intent.publication?.proposedWorkspaceRootId ?? intent.expectedWorkspaceRootId, + journalEventIds: intent.events.map((_event, index) => `event-${index}`), + }; +} + +/** A test-supplied outcome, carried through with the decision it implies. */ +function mapDecision(result: Result, intent: CommitIntent): Result { + return result.ok ? Ok(decisionFor(intent)) : result; +} + /** A link that records what it was asked, and answers how a test tells it to. */ function link( options: { @@ -110,12 +124,14 @@ function link( } return starting; }, - *commit(intent: CommitIntent): Operation> { + *commit(intent: CommitIntent): Operation> { sent.push(intent); if (options.blockCommit !== undefined) { yield* options.blockCommit.operation; } - return options.commit === undefined ? Ok(undefined) : options.commit(intent); + return options.commit === undefined + ? Ok(decisionFor(intent)) + : mapDecision(options.commit(intent), intent); }, }; return { owner, sent, starting }; @@ -352,8 +368,8 @@ describe("a remote transaction", () => { *frontier(): Operation { throw new Error("transport failed"); }, - *commit(): Operation> { - return Ok(undefined); + *commit(): Operation> { + return Ok({ workspaceRootId: "root-a", journalEventIds: [] }); }, }; try { From 45ad250d129a6bdb6976510f95678483957d54cd Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 09:17:35 -0400 Subject: [PATCH 38/81] =?UTF-8?q?=F0=9F=94=90=20Let=20the=20transaction=20?= =?UTF-8?q?that=20got=20the=20answer=20be=20the=20one=20that=20promotes=20?= =?UTF-8?q?(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Promotion took a value shaped like the owner's decision, and any caller could write one down. Comparing it against the attempt's own capture caught a wrong root and nothing else: a fabricated decision naming the right root promoted just as well as the real thing. The accepted materialization also carried a public `replace()`, so anything that could create an attempt could move the run to a Workspace the owner had never selected. And the one operation that did receive the authoritative answer — the transaction itself — discarded it, so completing an owner-authorized promotion meant leaving the supported path and repeating the protocol by hand. The authority is no longer data. An attempt offers a proposal, and that proposal carries a way back to the attempt that nothing else can construct; the transaction sends it, validates the answer against what it sent, and then transfers the tree itself, before it reports success. A caller may still build an enlistment by hand — it simply carries no attempt, so there is nothing to transfer, which is the right outcome rather than a missing step. `promote()` and `replace()` are gone from every surface a caller holds. What is left on the materialization is where a path is and which root it is; what is left on an attempt is where it is, what it captured, and how to offer it. The capability that moves the accepted tree is a symbol only this module can write, so a value that did not come from `useMaterialization()` cannot be mistaken for one. The evidence follows the production path end to end: mutate an attempt, enlist its capture, return from the body, and find that by the time the transaction reported success the accepted path had become the attempt's tree and read the attempted bytes, while the old tree was gone. The supported surfaces are enumerated and shown to expose no replacement or promotion, and a value naming the exact proposed root is handed to the only thing that takes an enlistment and moves nothing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/remote/collector.ts | 24 +++ packages/workflow/src/remote/invocation.ts | 157 +++++++++++++----- .../workflow/tests/remote-publication.test.ts | 115 ++++++++----- 3 files changed, 215 insertions(+), 81 deletions(-) diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index 85af88b60..946b14e61 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -96,6 +96,20 @@ export interface WorkspaceEnlistment { * nobody can supply is refused rather than guessed at. */ readonly bytes: ReadonlyMap; + /** + * How this proposal's attempt becomes the accepted Workspace. + * + * Supplied by the attempt that produced the proposal, and reachable nowhere + * else. The transaction calls it once, after the owner has performed the + * exact commit this proposal describes and the answer has been validated + * against it — which is what makes the performed answer the authority rather + * than a value that merely looks like one. + * + * Absent when the proposal came from somewhere other than a disposable + * attempt. Then there is nothing to transfer, which is the correct outcome + * rather than a missing step. + */ + readonly transfer?: (decision: CommitDecision) => Operation; } /** What the collector needs from the connection. */ @@ -277,6 +291,13 @@ export function transactRemotely( if (!committed.ok) { return committed; } + // The owner performed this exact proposal, so the attempt that produced + // it becomes the accepted Workspace — here, inside the operation that + // received and validated the answer, rather than by handing the answer + // to a caller and trusting the sequence. + if (enlisted?.transfer !== undefined) { + yield* enlisted.transfer(committed.value); + } // Only now. `T` is the body's own value and never crossed the connection. return Ok(outcome); } @@ -306,6 +327,9 @@ function detach(proposal: WorkspaceEnlistment): WorkspaceEnlistment { ), }), mappings: Object.freeze(proposal.mappings.map(detachMapping)), + // Carried through rather than copied: it is a capability, not data, and the + // attempt that created it is the only thing that can honour it. + ...(proposal.transfer === undefined ? {} : { transfer: proposal.transfer }), // A copy of every buffer, not a reference to one. A caller that goes on // writing into the array it captured must not be able to change what this // proposal stages. diff --git a/packages/workflow/src/remote/invocation.ts b/packages/workflow/src/remote/invocation.ts index d736fcc1c..57bc42b27 100644 --- a/packages/workflow/src/remote/invocation.ts +++ b/packages/workflow/src/remote/invocation.ts @@ -31,7 +31,8 @@ import { type RunnerFiles, } from "./materialize.ts"; import type { RemoteReadLink } from "./read.ts"; -import type { CommitDecision } from "./publication.ts"; +import type { CommitDecision, ProposedContent, RetainedMapping } from "./publication.ts"; +import type { WorkspaceEnlistment } from "./collector.ts"; /** A directory this invocation owns for as long as it needs one. */ export interface TemporaryTrees { @@ -41,6 +42,16 @@ export interface TemporaryTrees { remove(path: string): Operation; } +/** + * The capability to move the accepted tree, which is not part of reading it. + * + * A symbol because a symbol cannot be written down by anyone who does not + * already have it. Everything that merely reads the Workspace receives a + * `Materialization` and can see no way to change which Workspace it is + * reading; only an attempt, created by this module, is handed the key. + */ +const ACCEPT: unique symbol = Symbol("executablemd.workflow.remote.accept"); + /** The accepted local copy of the root the owner last confirmed. */ export interface Materialization { /** The root this tree is, as the owner confirmed it. */ @@ -56,22 +67,37 @@ export interface Materialization { at(logical: string): string; } -/** One disposable place to make a mutation, and the way to keep it. */ +/** The accepted materialization as this module alone sees it. */ +interface AcceptedMaterialization extends Materialization { + readonly [ACCEPT]: (next: { root: string; workspaceRootId: string }) => Operation; +} + +/** + * One disposable place to make a mutation, and the way to offer it. + * + * There is no `promote()`. Promotion is not something a caller does at the + * right moment; it is what happens inside the transaction when the owner + * performs the exact commit that proposed this attempt. An attempt offers a + * proposal, the transaction sends it, and only the transaction — holding the + * answer it just validated — transfers the tree. + */ export interface Attempt { readonly at: HostPath; /** What the attempt now describes, captured and checked locally. */ capture(): Operation; /** - * Make this attempt the accepted materialization. + * Offer this attempt's captured Workspace to the active transaction. * - * It takes the owner's performed decision because that decision is the - * authority: nothing else may promote, and a decision naming a different root - * than this attempt captured is not this attempt's decision. Passing it is - * the proof, which is why there is no argument-free way to do this — a - * refusal, an ambiguous loss and a local failure all leave the caller with - * nothing to pass. + * The value it returns carries a way back to this attempt that nothing else + * can construct. A caller may build an enlistment by hand, but it will carry + * no attempt, and so it can move no accepted materialization — which is the + * point: a value shaped like an owner's decision is not authority, and there + * is nowhere to hand one. */ - promote(decision: CommitDecision): Operation; + propose( + captured: CapturedWorkspace, + mappings: readonly RetainedMapping[], + ): Operation; } /** @@ -88,19 +114,19 @@ export function useMaterialization( reads: RemoteReadLink, workspaceRootId: string, reject: WorkspaceRejection, -): Operation { +): Operation { return resource(function* (provide) { const root = yield* trees.create("accepted"); yield* materializeWorkspaceRoot(files, reads, at(root), workspaceRootId, reject); let accepted = { root, workspaceRootId }; - yield* provide({ + const materialization: AcceptedMaterialization = { get workspaceRootId(): string { return accepted.workspaceRootId; }, at(logical: string): string { return at(accepted.root)(logical); }, - *replace(next: { root: string; workspaceRootId: string }): Operation { + *[ACCEPT](next: { root: string; workspaceRootId: string }): Operation { const previous = accepted.root; accepted = next; // The tree the run used to be at is removed once nothing points at it. @@ -108,19 +134,24 @@ export function useMaterialization( // nothing can reach and nothing will clean up until the invocation ends. yield* trees.remove(previous); }, - }); + }; + yield* provide(materialization); }); } /** - * The accepted materialization, plus the one operation that may move it. + * The accepted materialization, with the capability an attempt needs. * - * `replace` is not on `Materialization` because everything that merely reads - * the Workspace should not be able to change which Workspace it is reading. - * Only an attempt holding a performed decision reaches this. + * The declared type hides it, so this is where the two views meet. A value that + * did not come from `useMaterialization()` carries no such key and cannot be + * mistaken for one. */ -export interface AcceptedMaterialization extends Materialization { - replace(next: { root: string; workspaceRootId: string }): Operation; +function accepting(materialization: Materialization, reject: WorkspaceRejection) { + const accept = (materialization as Partial)[ACCEPT]; + if (accept === undefined) { + reject("this is not an accepted materialization this invocation owns"); + } + return accept; } /** @@ -135,10 +166,11 @@ export function useAttempt( files: RunnerFiles, trees: TemporaryTrees, reads: RemoteReadLink, - materialization: AcceptedMaterialization, + materialization: Materialization, reject: WorkspaceRejection, ): Operation { return resource(function* (provide) { + const accept = accepting(materialization, reject); const root = yield* trees.create("attempt"); yield* materializeWorkspaceRoot( files, @@ -148,42 +180,87 @@ export function useAttempt( reject, ); - let promoted = false; + let transferred = false; // Registered before the attempt is handed over, so every exit removes it — // including the ones that never reach the end of the calling scope. yield* ensure(function* () { - if (!promoted) { + if (!transferred) { yield* trees.remove(root); } }); + /** + * Make this attempt the accepted materialization. + * + * Reached only through the enlistment `propose()` produced, and only by the + * transaction that has just validated the owner's answer for this exact + * proposal. The decision is compared with what this attempt captured, so an + * answer about some other Workspace transfers nothing. + */ + function* transfer(decision: CommitDecision, captured: CapturedWorkspace): Operation { + if (transferred) { + reject("this attempt has already been transferred"); + } + if (decision.workspaceRootId !== captured.root.rootId) { + reject("the owner's decision names a root this attempt did not capture"); + } + transferred = true; + yield* accept({ root, workspaceRootId: captured.root.rootId }); + } + yield* provide({ at: at(root), *capture(): Operation { return yield* captureWorkspace(files, at(root), reject); }, - *promote(decision: CommitDecision): Operation { - if (promoted) { - // One decision promotes one attempt once. A second promotion would be - // moving the accepted tree somewhere it has already been moved from. - reject("this attempt has already been promoted"); - } - const captured = yield* captureWorkspace(files, at(root), reject); - if (decision.workspaceRootId !== captured.root.rootId) { - // The owner published something other than what this attempt holds. - // Promoting would label these bytes with a root they are not. - reject("the owner's decision names a root this attempt did not capture"); - } - promoted = true; - // The tree itself becomes the accepted one. Recording the identity - // without moving the bytes would leave the invocation reading the - // Workspace it used to be at under the name of the one it is now at. - yield* materialization.replace({ root, workspaceRootId: captured.root.rootId }); + // deno-lint-ignore require-yield + *propose( + captured: CapturedWorkspace, + mappings: readonly RetainedMapping[], + ): Operation { + return { + publication: { + proposedWorkspaceRootId: captured.root.rootId, + proposedManifest: captured.root.manifest, + content: inventoryOf(captured), + }, + mappings, + bytes: bytesOf(captured), + transfer: (decision) => transfer(decision, captured), + }; }, }); }); } +/** The exact closure a captured root names, in canonical order. */ +function inventoryOf(captured: CapturedWorkspace): ProposedContent[] { + return [ + ...captured.root.manifests.map((digest) => ({ + kind: "manifest" as const, + digest, + size: captured.contents.get(digest)?.manifestBytes.length ?? 0, + })), + ...captured.root.blobs.map((digest) => ({ + kind: "blob" as const, + digest, + size: captured.blobs.get(digest)?.length ?? 0, + })), + ]; +} + +/** Every piece the capture can supply, by identity. */ +function bytesOf(captured: CapturedWorkspace): Map { + const bytes = new Map(); + for (const [digest, content] of captured.contents) { + bytes.set(digest, content.manifestBytes); + } + for (const [digest, blob] of captured.blobs) { + bytes.set(digest, blob); + } + return bytes; +} + /** * One logical Workspace path under a host directory. * diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index e6a5ccec0..b5e958444 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -431,7 +431,7 @@ describe("what the production runner publishes", () => { void outcomes; }); - it("promotes only with the owner's decision, and moves the tree with it", function* () { + it("transfers the attempt inside the transaction that the owner performed", function* () { const files = runnerFiles(); const trees = yield* useRunnerTrees(); const { captured, reads } = yield* startingTree(); @@ -449,50 +449,35 @@ describe("what the production runner publishes", () => { const acceptedBefore = materialization.at("/"); let attemptRoot = ""; - const attempt = yield* useAttempt(files, trees, reads, materialization, reject); - attemptRoot = attempt.at("/"); - yield* until(writeFile(attempt.at("/NOTES.md"), "published\n", { mode: 0o644 })); - const proposed = yield* attempt.capture(); - for (const [digest, content] of proposed.contents) { - sizes.set(digest, content.manifestBytes.length); - } - for (const [digest, blob] of proposed.blobs) { - sizes.set(digest, blob.length); - } - - // A decision the owner did not give cannot promote. - let raised: unknown; - try { - yield* attempt.promote({ workspaceRootId: captured.root.rootId, journalEventIds: [] }); - } catch (error) { - raised = error; - } - expect(raised).toBeInstanceOf(Error); - expect(materialization.workspaceRootId).toBe(captured.root.rootId); - - // The owner's own answer, naming the root this attempt captured. - const committed = yield* link.commit({ - expectedWorkspaceRootId: captured.root.rootId, - expectedJournalEventId: null, - events: [], - publication: { - proposedWorkspaceRootId: proposed.root.rootId, - proposedManifest: proposed.root.manifest, - content: inventoryOf(proposed), - }, - mappings: [], - bytes: proposedBytes(proposed), + let acceptedDuringBody = ""; + const committed = yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + attemptRoot = attempt.at("/"); + yield* until(writeFile(attempt.at("/NOTES.md"), "published\n", { mode: 0o644 })); + const proposed = yield* attempt.capture(); + for (const [digest, content] of proposed.contents) { + sizes.set(digest, content.manifestBytes.length); + } + for (const [digest, blob] of proposed.blobs) { + sizes.set(digest, blob.length); + } + return yield* transactRemotely(link, createTransactionGate(), function* (_tx, enlist) { + enlist(yield* attempt.propose(proposed, [])); + // Still the old Workspace while the answer is unknown. + acceptedDuringBody = materialization.at("/"); + return "done"; + }); }); - if (!committed.ok) { - throw committed.error; - } - yield* attempt.promote(committed.value); - // The accepted materialization is the promoted tree, not a relabelled - // copy of the old one: the file the effect wrote is readable through it. - expect(materialization.workspaceRootId).not.toBe(captured.root.rootId); + expect(committed).toMatchObject({ ok: true }); + expect(acceptedDuringBody).toBe(acceptedBefore); + + // By the time the transaction reported success the transfer had happened: + // the accepted path is the attempt's tree and reads the attempted bytes. expect(materialization.at("/")).toBe(attemptRoot); + expect(materialization.workspaceRootId).not.toBe(captured.root.rootId); expect(yield* until(readFile(materialization.at("/NOTES.md"), "utf8"))).toBe("published\n"); + // And the tree the run used to be at is gone. let listed: unknown; try { @@ -503,6 +488,54 @@ describe("what the production runner publishes", () => { expect(listed).toBeInstanceOf(Error); }); + it("offers no way to move the accepted Workspace without the owner", function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + const accepted = materialization.at("/"); + + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* until(writeFile(attempt.at("/NOTES.md"), "never published\n", { mode: 0o644 })); + const proposed = yield* attempt.capture(); + + // Everything a caller can reach: reading the Workspace, capturing an + // attempt, and offering a proposal. Nothing here promotes or replaces. + expect(Object.keys(materialization).toSorted()).toEqual(["at", "workspaceRootId"]); + expect(Object.keys(attempt).toSorted()).toEqual(["at", "capture", "propose"]); + + // A value shaped exactly like the owner's answer, for the exact root this + // attempt captured, handed to the only thing that takes an enlistment. + // It carries no attempt, so it can transfer nothing. + const forged = { + publication: { + proposedWorkspaceRootId: proposed.root.rootId, + proposedManifest: proposed.root.manifest, + content: [], + }, + mappings: [], + bytes: new Map(), + }; + const transport = wire(ownerAnswers("")); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + yield* transactRemotely(link, createTransactionGate(), function* (_tx, enlist) { + enlist(forged); + return "done"; + }); + }); + + expect(materialization.at("/")).toBe(accepted); + expect(materialization.workspaceRootId).toBe(captured.root.rootId); + }); + it("cannot be changed by a caller that kept its own copy", function* () { const { captured, reads } = yield* startingTree(); const transport = wire(ownerAnswers(captured.root.rootId)); From 3a3de81baaabbe190827627990934e15a25e34e9 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 10:07:53 -0400 Subject: [PATCH 39/81] =?UTF-8?q?=F0=9F=8E=9F=EF=B8=8F=20Enlist=20the=20at?= =?UTF-8?q?tempt,=20not=20a=20proposal=20about=20it=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The capability moved again instead of going away. `propose()` handed back an object with a callable `transfer`, so the same fabricated decision that used to promote through `attempt.promote(decision)` now promoted through `proposal.transfer(decision)` — before any commit, comparing only a caller-supplied root against a caller-held capture. The regression that was supposed to catch this built a *different* enlistment without that member, so it never touched the bypass at all. Two more things followed from taking a capture as an argument. The transfer moved the live directory while the identity came from whatever capture was passed, so a body could enlist, keep writing, and have the owner commit one root while the runner labelled different bytes with it. And a hand-built publication with no attempt behind it committed happily: the owner advanced and the invocation stayed on its old tree. So `enlist` takes the attempt. Not a proposal, not a capture — the attempt itself. The transaction seals it after the body and everything it started have finished, which means the proposal is the tree as it finally is rather than as it was at some earlier moment; the same seal produces the transfer, and the transaction calls it after validating the owner's answer. A publication that no live attempt owns is now unrepresentable rather than merely refused. What a caller holds is a place to work and a way to read what it did. Sealing and transfer hang off a symbol in a module no entrypoint reaches, so they are not on the declared surface and cannot be written down by anything that does not already have them. The evidence follows: enlist an attempt, keep writing to it, and the committed root, the transferred tree and the recaptured accepted root are one root and one set of bytes — the final ones. The supported surfaces are enumerated and carry no promote, transfer, replace or accept. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/remote/collector.ts | 106 ++--- packages/workflow/src/remote/invocation.ts | 70 ++-- packages/workflow/src/remote/seal.ts | 41 ++ .../workflow/tests/remote-publication.test.ts | 375 +++++++++--------- 4 files changed, 288 insertions(+), 304 deletions(-) create mode 100644 packages/workflow/src/remote/seal.ts diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index 946b14e61..f3f613ee8 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -24,6 +24,7 @@ import { call, ensure, Ok, type Operation, type Result, scoped } from "effection"; import type { CommitDecision, RetainedMapping, WorkspacePublication } from "./publication.ts"; +import { SEAL, type SealableAttempt } from "./seal.ts"; import type { DurableEvent } from "@executablemd/durable-streams"; import type { DurableStream } from "@executablemd/durable-streams"; import type { WorkflowRunTransaction } from "../storage/api.ts"; @@ -72,46 +73,6 @@ export interface CommitIntent { readonly bytes: ReadonlyMap; } -/** - * What a Workspace operation enlisted, if one did. - * - * At most one per transaction. Two would be two Workspaces proposed for one - * commit, and the owner would have to choose — which is a decision nobody is - * entitled to make on the run's behalf. - */ -export interface WorkspaceEnlistment { - readonly publication: WorkspacePublication; - readonly mappings: readonly RetainedMapping[]; - /** - * The bytes for every piece the publication names, by identity. - * - * Supplied at enlistment rather than fetched later. The proposal's identity - * is a digest over content, so bytes that could still change after the - * transaction sealed would let one identity describe two different - * Workspaces — and the adapter would stage whatever the buffer happened to - * hold by the time it looked. - * - * Only pieces the owner may not already hold need appear. What is absent is - * content the owner is expected to have, and a proposal naming content - * nobody can supply is refused rather than guessed at. - */ - readonly bytes: ReadonlyMap; - /** - * How this proposal's attempt becomes the accepted Workspace. - * - * Supplied by the attempt that produced the proposal, and reachable nowhere - * else. The transaction calls it once, after the owner has performed the - * exact commit this proposal describes and the answer has been validated - * against it — which is what makes the performed answer the authority rather - * than a value that merely looks like one. - * - * Absent when the proposal came from somewhere other than a disposable - * attempt. Then there is nothing to transfer, which is the correct outcome - * rather than a missing step. - */ - readonly transfer?: (decision: CommitDecision) => Operation; -} - /** What the collector needs from the connection. */ export interface OwnerLink { /** One bounded read that opens and closes its own owner-side read. */ @@ -241,7 +202,7 @@ export function transactRemotely( }, }; - let enlisted: WorkspaceEnlistment | undefined; + let enlisted: { attempt: SealableAttempt; mappings: readonly RetainedMapping[] } | undefined; /** * How a Workspace operation puts its result into this transaction. * @@ -251,17 +212,22 @@ export function transactRemotely( * arrays and records it passed and a proposal that changed after it was * admitted would not be the proposal the identity was computed over. */ - const enlist: EnlistWorkspace = (proposal: WorkspaceEnlistment): void => { + const enlist: EnlistWorkspace = ( + attempt: SealableAttempt, + mappings: readonly RetainedMapping[] = [], + ): void => { if (!live) { throw new RemoteTransactionError("transaction-closed"); } if (enlisted !== undefined) { throw new RemoteTransactionError("publication-already-enlisted"); } - if (proposal.mappings.length > MAX_MAPPINGS) { + if (mappings.length > MAX_MAPPINGS) { throw new RemoteTransactionError("too-many-mappings"); } - enlisted = detach(proposal); + // The mappings are detached now, because they are the caller's values. + // The Workspace itself is not read until sealing. + enlisted = { attempt, mappings: Object.freeze(mappings.map(detachMapping)) }; }; let outcome: T; @@ -279,14 +245,18 @@ export function transactRemotely( live = false; } + // Sealed after teardown: the proposal is the tree as it finally is. + const sealed = + enlisted === undefined ? undefined : yield* enlisted.attempt[SEAL](enlisted.mappings); + const committed = yield* link.commit({ expectedWorkspaceRootId: starting.workspaceRootId, expectedJournalEventId: starting.journalEventId, // A private snapshot. The collector's own array never leaves. events: appended.map((event) => structuredClone(event)), - publication: enlisted?.publication ?? null, - mappings: enlisted?.mappings ?? [], - bytes: enlisted?.bytes ?? new Map(), + publication: sealed?.publication ?? null, + mappings: sealed?.mappings ?? [], + bytes: sealed?.bytes ?? new Map(), }); if (!committed.ok) { return committed; @@ -295,8 +265,8 @@ export function transactRemotely( // it becomes the accepted Workspace — here, inside the operation that // received and validated the answer, rather than by handing the answer // to a caller and trusting the sequence. - if (enlisted?.transfer !== undefined) { - yield* enlisted.transfer(committed.value); + if (sealed !== undefined) { + yield* sealed.transfer(committed.value); } // Only now. `T` is the body's own value and never crossed the connection. return Ok(outcome); @@ -304,8 +274,20 @@ export function transactRemotely( }); } -/** How a Workspace operation enlists its one publication in the active transaction. */ -export type EnlistWorkspace = (proposal: WorkspaceEnlistment) => void; +/** + * How a Workspace operation designates its attempt for publication. + * + * It names an attempt rather than handing over a proposal. What the attempt + * holds is captured when the transaction seals it — after the body and + * everything it started have finished — so the proposal always describes the + * tree as it finally is, and the tree the owner decides is the tree that gets + * transferred. There is no way to enlist a Workspace that no live attempt owns, + * which is what stops a durable commit from leaving the invocation behind. + */ +export type EnlistWorkspace = ( + attempt: SealableAttempt, + mappings?: readonly RetainedMapping[], +) => void; /** * A copy nobody else holds a reference into. @@ -315,28 +297,6 @@ export type EnlistWorkspace = (proposal: WorkspaceEnlistment) => void; * publication whose inventory or manifest changed afterwards would not be the * one its identity was computed over. */ -function detach(proposal: WorkspaceEnlistment): WorkspaceEnlistment { - return Object.freeze({ - publication: Object.freeze({ - proposedWorkspaceRootId: proposal.publication.proposedWorkspaceRootId, - proposedManifest: proposal.publication.proposedManifest, - content: Object.freeze( - proposal.publication.content.map((piece) => - Object.freeze({ kind: piece.kind, digest: piece.digest, size: piece.size }), - ), - ), - }), - mappings: Object.freeze(proposal.mappings.map(detachMapping)), - // Carried through rather than copied: it is a capability, not data, and the - // attempt that created it is the only thing that can honour it. - ...(proposal.transfer === undefined ? {} : { transfer: proposal.transfer }), - // A copy of every buffer, not a reference to one. A caller that goes on - // writing into the array it captured must not be able to change what this - // proposal stages. - bytes: new Map([...proposal.bytes].map(([digest, bytes]) => [digest, bytes.slice()])), - }); -} - /** * One mapping, copied all the way down. * diff --git a/packages/workflow/src/remote/invocation.ts b/packages/workflow/src/remote/invocation.ts index 57bc42b27..795d2b4c8 100644 --- a/packages/workflow/src/remote/invocation.ts +++ b/packages/workflow/src/remote/invocation.ts @@ -32,7 +32,7 @@ import { } from "./materialize.ts"; import type { RemoteReadLink } from "./read.ts"; import type { CommitDecision, ProposedContent, RetainedMapping } from "./publication.ts"; -import type { WorkspaceEnlistment } from "./collector.ts"; +import { SEAL, type SealableAttempt, type SealedProposal } from "./seal.ts"; /** A directory this invocation owns for as long as it needs one. */ export interface TemporaryTrees { @@ -81,23 +81,10 @@ interface AcceptedMaterialization extends Materialization { * proposal, the transaction sends it, and only the transaction — holding the * answer it just validated — transfers the tree. */ -export interface Attempt { +export interface Attempt extends SealableAttempt { readonly at: HostPath; - /** What the attempt now describes, captured and checked locally. */ + /** What the attempt describes right now, captured and checked locally. */ capture(): Operation; - /** - * Offer this attempt's captured Workspace to the active transaction. - * - * The value it returns carries a way back to this attempt that nothing else - * can construct. A caller may build an enlistment by hand, but it will carry - * no attempt, and so it can move no accepted materialization — which is the - * point: a value shaped like an owner's decision is not authority, and there - * is nowhere to hand one. - */ - propose( - captured: CapturedWorkspace, - mappings: readonly RetainedMapping[], - ): Operation; } /** @@ -189,35 +176,27 @@ export function useAttempt( } }); - /** - * Make this attempt the accepted materialization. - * - * Reached only through the enlistment `propose()` produced, and only by the - * transaction that has just validated the owner's answer for this exact - * proposal. The decision is compared with what this attempt captured, so an - * answer about some other Workspace transfers nothing. - */ - function* transfer(decision: CommitDecision, captured: CapturedWorkspace): Operation { - if (transferred) { - reject("this attempt has already been transferred"); - } - if (decision.workspaceRootId !== captured.root.rootId) { - reject("the owner's decision names a root this attempt did not capture"); - } - transferred = true; - yield* accept({ root, workspaceRootId: captured.root.rootId }); - } - yield* provide({ at: at(root), *capture(): Operation { return yield* captureWorkspace(files, at(root), reject); }, - // deno-lint-ignore require-yield - *propose( - captured: CapturedWorkspace, - mappings: readonly RetainedMapping[], - ): Operation { + + /** + * Seal this attempt into the proposal the owner will decide. + * + * Captured here, not earlier. The transaction calls this once the body + * and everything it started have torn down, so the proposal describes the + * tree as it finally is — a capture taken when the body enlisted could + * name one Workspace while the directory went on to hold another, and the + * owner would commit one root while the runner transferred different + * bytes under it. + */ + *[SEAL](mappings: readonly RetainedMapping[]): Operation { + if (transferred) { + reject("this attempt has already been sealed and transferred"); + } + const captured = yield* captureWorkspace(files, at(root), reject); return { publication: { proposedWorkspaceRootId: captured.root.rootId, @@ -226,7 +205,16 @@ export function useAttempt( }, mappings, bytes: bytesOf(captured), - transfer: (decision) => transfer(decision, captured), + *transfer(decision: CommitDecision): Operation { + if (transferred) { + reject("this attempt has already been transferred"); + } + if (decision.workspaceRootId !== captured.root.rootId) { + reject("the owner's decision names a root this attempt did not seal"); + } + transferred = true; + yield* accept({ root, workspaceRootId: captured.root.rootId }); + }, }; }, }); diff --git a/packages/workflow/src/remote/seal.ts b/packages/workflow/src/remote/seal.ts new file mode 100644 index 000000000..07f86f46b --- /dev/null +++ b/packages/workflow/src/remote/seal.ts @@ -0,0 +1,41 @@ +/** + * How a transaction reaches into the attempt it was given, and nothing else can. + * + * A transaction needs two things from a disposable attempt: to seal it into a + * proposal once the body has finished, and to make it the accepted Workspace + * once the owner has performed that exact proposal. Neither may be offered to + * whoever is running the body — a capability handed out is a capability that + * can be used at the wrong moment, and the wrong moment here is any moment + * before the owner has decided. + * + * So they hang off a symbol. A symbol cannot be written down by code that does + * not already have it, this module is reachable from no package entrypoint, and + * the declared `Attempt` says nothing about it. What a caller receives is a + * place to work and a way to read what it did. + */ + +import type { Operation } from "effection"; +import type { CommitDecision, RetainedMapping, WorkspacePublication } from "./publication.ts"; + +/** The key a transaction reaches an attempt's own machinery through. */ +export const SEAL: unique symbol = Symbol("executablemd.workflow.remote.seal"); + +/** One attempt, sealed into the proposal the owner will decide. */ +export interface SealedProposal { + readonly publication: WorkspacePublication; + readonly mappings: readonly RetainedMapping[]; + readonly bytes: ReadonlyMap; + /** + * Make the sealed attempt the accepted Workspace. + * + * Called once, by the transaction, after the owner performed this exact + * proposal and the answer was checked against it. The decision is compared + * with what was sealed, so an answer about another Workspace moves nothing. + */ + transfer(decision: CommitDecision): Operation; +} + +/** What an attempt privately offers the transaction that was given it. */ +export interface SealableAttempt { + readonly [SEAL]: (mappings: readonly RetainedMapping[]) => Operation; +} diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index b5e958444..72760e737 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -18,7 +18,6 @@ import { ensure, type Operation, scoped, sleep, spawn, until } from "effection"; import { mkdir, readdir, readFile, writeFile } from "node:fs/promises"; import { agentSessionKey } from "../src/storage/agent-session.ts"; import { cloudflareOwnerLink } from "../src/cloudflare/client.ts"; -import { encodeBase64 } from "../src/cloudflare/encoding.ts"; import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; import { type CommitIntent, @@ -40,7 +39,7 @@ import { } from "../src/workspace/root-manifest.ts"; import { sha256Hex } from "../src/workspace/sha256.ts"; import { locatorFingerprintOf } from "../src/composition/locator.ts"; -import type { ProposedContent, RetainedMapping } from "../src/remote/publication.ts"; +import type { RetainedMapping } from "../src/remote/publication.ts"; function reject(reason: string): never { throw new Error(reason); @@ -57,34 +56,6 @@ function event(name: string) { const LOCATOR = "https://git.example.invalid/octo/app.git"; -/** The exact closure a proposed root names, in canonical order. */ -function inventoryOf(captured: CapturedWorkspace): ProposedContent[] { - return [ - ...captured.root.manifests.map((digest) => ({ - kind: "manifest" as const, - digest, - size: captured.contents.get(digest)?.manifestBytes.length ?? 0, - })), - ...captured.root.blobs.map((digest) => ({ - kind: "blob" as const, - digest, - size: captured.blobs.get(digest)?.length ?? 0, - })), - ]; -} - -/** Every piece a capture can supply, by identity. */ -function proposedBytes(captured: CapturedWorkspace): Map { - const bytes = new Map(); - for (const [digest, content] of captured.contents) { - bytes.set(digest, content.manifestBytes); - } - for (const [digest, blob] of captured.blobs) { - bytes.set(digest, blob); - } - return bytes; -} - /** The Repository mapping these tests enlist. */ function repositoryMapping(): RetainedMapping { return { @@ -155,15 +126,20 @@ function wire(answer: (request: Record) => Record = new Map()) { +function ownerAnswers(_rootId = "", _sizes: ReadonlyMap = new Map()) { return (request: Record): Record => { if (request["command"] === "stage") { + // The length the owner would have measured after decoding, computed from + // the encoding itself so this answers about the bytes it was actually + // sent rather than about a number a test remembered to set. + const encoded = String(request["bytes"] ?? ""); + const padding = encoded.endsWith("==") ? 2 : encoded.endsWith("=") ? 1 : 0; return { outcome: "performed", value: { kind: request["kind"], digest: request["digest"], - size: sizes.get(String(request["digest"])) ?? 0, + size: (encoded.length / 4) * 3 - padding, }, }; } @@ -329,26 +305,7 @@ describe("what the production runner publishes", () => { createTransactionGate(), function* (transaction, enlist) { yield* transaction.journal.append(event("published")); - enlist({ - publication: { - proposedWorkspaceRootId: proposed.root.rootId, - proposedManifest: proposed.root.manifest, - content: [ - ...proposed.root.manifests.map((digest) => ({ - kind: "manifest" as const, - digest, - size: proposed.contents.get(digest)?.manifestBytes.length ?? 0, - })), - ...proposed.root.blobs.map((digest) => ({ - kind: "blob" as const, - digest, - size: proposed.blobs.get(digest)?.length ?? 0, - })), - ], - }, - mappings: [repositoryMapping()], - bytes: proposedBytes(proposed), - }); + enlist(attempt, [repositoryMapping()]); return "done"; }, ); @@ -462,7 +419,7 @@ describe("what the production runner publishes", () => { sizes.set(digest, blob.length); } return yield* transactRemotely(link, createTransactionGate(), function* (_tx, enlist) { - enlist(yield* attempt.propose(proposed, [])); + enlist(attempt); // Still the old Workspace while the answer is unknown. acceptedDuringBody = materialization.at("/"); return "done"; @@ -504,32 +461,24 @@ describe("what the production runner publishes", () => { yield* scoped(function* () { const attempt = yield* useAttempt(files, trees, reads, materialization, reject); yield* until(writeFile(attempt.at("/NOTES.md"), "never published\n", { mode: 0o644 })); - const proposed = yield* attempt.capture(); - // Everything a caller can reach: reading the Workspace, capturing an - // attempt, and offering a proposal. Nothing here promotes or replaces. + // Everything a caller can reach by name. Reading where the Workspace is, + // reading what an attempt holds — and nothing that moves either. expect(Object.keys(materialization).toSorted()).toEqual(["at", "workspaceRootId"]); - expect(Object.keys(attempt).toSorted()).toEqual(["at", "capture", "propose"]); - - // A value shaped exactly like the owner's answer, for the exact root this - // attempt captured, handed to the only thing that takes an enlistment. - // It carries no attempt, so it can transfer nothing. - const forged = { - publication: { - proposedWorkspaceRootId: proposed.root.rootId, - proposedManifest: proposed.root.manifest, - content: [], - }, - mappings: [], - bytes: new Map(), - }; - const transport = wire(ownerAnswers("")); - const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareOwnerLink(connection, reads, ids()); - yield* transactRemotely(link, createTransactionGate(), function* (_tx, enlist) { - enlist(forged); - return "done"; - }); + expect(Object.keys(attempt).toSorted()).toEqual(["at", "capture"]); + const reachable = [ + ...Object.getOwnPropertyNames(attempt), + ...Object.getOwnPropertyNames(materialization), + ]; + for (const name of ["promote", "transfer", "replace", "accept", "propose", "seal"]) { + expect(reachable).not.toContain(name); + } + + // A caller can still capture. What it gets back is a description, and + // there is nothing to hand it to: `enlist` takes an attempt, so a + // publication that no live attempt owns cannot be expressed at all. + const described = yield* attempt.capture(); + expect(described.root.rootId).not.toBe(captured.root.rootId); }); expect(materialization.at("/")).toBe(accepted); @@ -537,47 +486,36 @@ describe("what the production runner publishes", () => { }); it("cannot be changed by a caller that kept its own copy", function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); + const transport = wire(ownerAnswers("")); const connection = yield* useOwnerConnection(transport.socket); const link = cloudflareOwnerLink(connection, reads, ids()); - const content: ProposedContent[] = [{ kind: "manifest", digest: "a".repeat(64), size: 1 }]; - const mappings: RetainedMapping[] = [ - { - kind: "repository", - locator: LOCATOR, - record: { - name: "app", - locatorFingerprint: locatorFingerprintOf(LOCATOR), - requestedBase: null, - creationCommit: "9".repeat(40), - primaryBranch: "main", - objectFormat: "sha1", - checkoutPath: "/docs", - }, - }, - ]; - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist({ - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content, - }, - mappings, - bytes: new Map(), + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + + const mappings: RetainedMapping[] = [repositoryMapping()]; + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist(attempt, mappings); + // The caller still holds the array it passed and edits it afterwards. + const first = mappings[0]; + if (first?.kind === "repository") { + mappings[0] = { ...first, locator: "https://elsewhere.invalid/x.git" }; + } + return "done"; }); - // The caller still holds both arrays and edits them after admission. - content.push({ kind: "blob", digest: "b".repeat(64), size: 2 }); - const first = mappings[0]; - if (first?.kind === "repository") { - mappings[0] = { ...first, locator: "https://elsewhere.invalid/x.git" }; - } - return "done"; }); - const commit = lastCommit(transport.sent); - expect(memberList(member(commit, "publication"), "content")).toHaveLength(1); - expect(text(memberList(commit, "mappings")[0] ?? {}, "locator")).toBe(LOCATOR); + expect(text(memberList(lastCommit(transport.sent), "mappings")[0] ?? {}, "locator")).toBe( + LOCATOR, + ); }); it("sends a journal-only commit with no publication and stages nothing", function* () { @@ -611,19 +549,23 @@ describe("what the production runner publishes", () => { }); it("encodes every kind of retained mapping the owner accepts", function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); + const transport = wire(ownerAnswers("")); const connection = yield* useOwnerConnection(transport.socket); const link = cloudflareOwnerLink(connection, reads, ids()); - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist({ - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content: [], - }, - bytes: new Map(), - mappings: [ + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist(attempt, [ repositoryMapping(), { kind: "worktree", @@ -652,9 +594,9 @@ describe("what the production runner publishes", () => { createdAt: "2026-09-03T00:00:00.000Z", }, }, - ], + ]); + return "done"; }); - return "done"; }); const mappings = memberList(lastCommit(transport.sent), "mappings"); expect(mappings.map((mapping) => mapping["kind"])).toEqual([ @@ -822,30 +764,38 @@ describe("what the production runner publishes", () => { }); it("sends nothing when the transaction exceeds a local bound", function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); + const transport = wire(ownerAnswers("")); const connection = yield* useOwnerConnection(transport.socket); const link = cloudflareOwnerLink(connection, reads, ids()); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); let raised: unknown; - try { - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist({ - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content: [], - }, + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + try { + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { // More retained mappings than one intent may carry. - mappings: Array.from({ length: 300 }, () => repositoryMapping()), - bytes: new Map(), + enlist( + attempt, + Array.from({ length: 300 }, () => repositoryMapping()), + ); + return "done"; }); - return "done"; - }); - } catch (error) { - raised = error; - } + } catch (error) { + raised = error; + } + }); expect(raised).toBeInstanceOf(Error); expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); + expect(materialization.workspaceRootId).toBe(captured.root.rootId); }); it("refuses a performed answer that names a root this proposal did not select", function* () { @@ -890,12 +840,20 @@ describe("what the production runner publishes", () => { expect(committed.ok).toBe(false); }); - it("seals nested mapping values and content bytes against later mutation", function* () { + it("seals a nested mapping value against later mutation", function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); const { captured, reads } = yield* startingTree(); - const sizes = new Map(); - const transport = wire(ownerAnswers("", sizes)); + const transport = wire(ownerAnswers("")); const connection = yield* useOwnerConnection(transport.socket); const link = cloudflareOwnerLink(connection, reads, ids()); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); const assertion = { kind: "acp-session", value: "admitted" }; const identity = { @@ -903,19 +861,10 @@ describe("what the production runner publishes", () => { agentCommand: "/usr/bin/agent", sessionIdentity: "session-1", }; - const piece = new TextEncoder().encode("admitted bytes"); - const digest = sha256Hex(piece); - sizes.set(digest, piece.length); - const bytes = new Map([[digest, piece]]); - - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist({ - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content: [{ kind: "blob", digest, size: piece.length }], - }, - mappings: [ + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist(attempt, [ { kind: "agent-session", record: { @@ -926,51 +875,97 @@ describe("what the production runner publishes", () => { createdAt: "2026-09-03T00:00:00.000Z", }, }, - ], - bytes, + ]); + // The caller still holds the nested assertion object and edits it. + assertion.value = "changed after admission"; + return "done"; }); - // The caller still holds the assertion object and the byte buffer, and - // edits both after the transaction admitted them. - assertion.value = "changed after admission"; - piece.fill(0); - bytes.set(digest, new TextEncoder().encode("substituted")); - return "done"; }); - // What was sent is what was admitted, not what the caller did afterwards. const mapping = memberList(lastCommit(transport.sent), "mappings")[0] ?? {}; expect(text(member(member(mapping, "record"), "assertion"), "value")).toBe("admitted"); - const staged = transport.sent.find((request) => request["command"] === "stage"); - expect(staged?.["digest"]).toBe(digest); - expect(staged?.["bytes"]).toBe(encodeBase64(new TextEncoder().encode("admitted bytes"))); + }); + + it("commits the tree as it finally is, not as it was when enlisted", function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); + const { captured, reads } = yield* startingTree(); + const sizes = new Map(); + const transport = wire(ownerAnswers("", sizes)); + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareOwnerLink(connection, reads, ids()); + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); + + let atEnlistment = ""; + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + yield* until(writeFile(attempt.at("/NOTES.md"), "first\n", { mode: 0o644 })); + const committed = yield* transactRemotely( + link, + createTransactionGate(), + function* (_transaction, enlist) { + enlist(attempt); + atEnlistment = (yield* attempt.capture()).root.rootId; + // The body goes on working after designating the attempt. Sealing + // happens after teardown, so this is what gets proposed. + yield* until(writeFile(attempt.at("/NOTES.md"), "second\n", { mode: 0o644 })); + const staged = yield* attempt.capture(); + for (const [digest, content] of staged.contents) { + sizes.set(digest, content.manifestBytes.length); + } + for (const [digest, blob] of staged.blobs) { + sizes.set(digest, blob.length); + } + return "done"; + }, + ); + expect(committed).toMatchObject({ ok: true }); + }); + + // The root the owner was asked to publish is the final one, not the one the + // tree held when the body enlisted it. + const proposed = member(lastCommit(transport.sent), "publication"); + expect(proposed["proposedWorkspaceRootId"]).not.toBe(atEnlistment); + // And the accepted tree recaptures to exactly the root that was committed. + expect(materialization.workspaceRootId).toBe(proposed["proposedWorkspaceRootId"]); + expect(yield* until(readFile(materialization.at("/NOTES.md"), "utf8"))).toBe("second\n"); }); it("refuses a second Workspace publication in one transaction", function* () { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); const { captured, reads } = yield* startingTree(); - const transport = wire(ownerAnswers(captured.root.rootId)); + const transport = wire(ownerAnswers("")); const connection = yield* useOwnerConnection(transport.socket); const link = cloudflareOwnerLink(connection, reads, ids()); - const proposal = { - publication: { - proposedWorkspaceRootId: captured.root.rootId, - proposedManifest: captured.root.manifest, - content: [], - }, - mappings: [], - bytes: new Map(), - }; + const materialization = yield* useMaterialization( + files, + trees, + reads, + captured.root.rootId, + reject, + ); let raised: unknown; - try { - yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { - enlist(proposal); - enlist(proposal); - return "done"; - }); - } catch (error) { - raised = error; - } - // Two Workspaces proposed for one commit is a choice nobody may make on - // the run's behalf, so the transaction fails and nothing is sent. + yield* scoped(function* () { + const attempt = yield* useAttempt(files, trees, reads, materialization, reject); + try { + yield* transactRemotely(link, createTransactionGate(), function* (_transaction, enlist) { + enlist(attempt); + enlist(attempt); + return "done"; + }); + } catch (error) { + raised = error; + } + }); + // Two Workspaces proposed for one commit is a choice nobody may make on the + // run's behalf, so the transaction fails and nothing is sent. expect(raised).toBeInstanceOf(Error); expect(transport.sent.some((request) => request["command"] === "commit")).toBe(false); }); From 1088f75786bc9bd30e83fc15cc5eddb5b0e6fdf0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 10:47:47 -0400 Subject: [PATCH 40/81] =?UTF-8?q?=F0=9F=97=84=EF=B8=8F=20Give=20a=20remote?= =?UTF-8?q?=20run=20the=20same=20handle=20a=20local=20one=20has=20(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The interface a run's storage answers is the same wherever the run lives. This is that handle, backed by a connection: snapshots that stay snapshots, a journal that reads fresh and appends through the one commit path, a caller-owned transaction, retrieval replacement, and an anchored execution list. Two mechanisms keep one handle in order because they answer different questions. A turn stops two operations interleaving, so unrelated work waits and then proceeds. A scope-local marker records that this scope is *inside* a transaction on this handle, so a nested transaction or an ordinary operation called from the body is refused at once rather than queued behind a turn its own caller is holding and will not release. A queue alone deadlocks that; a flag alone mistakes unrelated work for nested work. Retrieval replacement is its own mutation, not a degenerate commit. It appends nothing and publishes nothing, and its revision is the owner's arithmetic over what is stored — two handles that both read revision one before either wrote would otherwise both write two and the second would lose the first. Clearing removes the row, so the next replacement starts counting again. The identity is minted per invocation rather than derived from the request, because two calls carrying identical metadata are two replacements and must not collapse into one; and because a decision now names its command kind, one textual id used for a commit and for a replacement is two requests rather than a recognized retry. Executions read as one anchored snapshot for the same reason the journal does: a caller assembling a list across several requests must see one moment rather than whatever the table held at each of them. The first page fixes the terminal row and every later page is held to it, with the run identity and the sequence travelling privately so the runner can refuse another run's page and prove adjacency — neither becomes part of the record it returns. Failures cross as the categories the local host reports for the same condition. Command names, refusal spellings, rows and cursors stay below the adapter; they describe a protocol nobody above it is party to. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 206 ++++++++++- packages/workflow/src/cloudflare/commands.ts | 85 +++++ .../workflow/src/cloudflare/dispatcher.ts | 59 +++- .../workflow/src/cloudflare/owner-reads.ts | 89 ++++- packages/workflow/src/cloudflare/publish.ts | 82 +++++ packages/workflow/src/remote/database.ts | 331 ++++++++++++++++++ packages/workflow/src/remote/records.ts | 36 ++ .../tests/cloudflare/remote-owner.vitest.ts | 8 +- 8 files changed, 885 insertions(+), 11 deletions(-) create mode 100644 packages/workflow/src/remote/database.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 90f2d201e..9937b1ab1 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -35,6 +35,7 @@ import type { CommitIntent, OwnerLink, StartingFrontier } from "../remote/collec import type { CommitDecision } from "../remote/publication.ts"; import { OwnerLinkError, type OwnerAnswer, type OwnerConnection } from "../remote/client.ts"; import { + parseRemoteExecution, parseRemoteJournalEntry, parseRemoteRetrieval, parseRemoteRunRecord, @@ -54,7 +55,16 @@ import { type WorkspaceRootManifest, } from "../workspace/root-manifest.ts"; import { decodeContentManifest } from "../workspace/content-manifest.ts"; -import { JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES } from "./commands.ts"; +import { EXECUTION_PAGE_ENTRIES, JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES } from "./commands.ts"; +import type { RemoteRunLink } from "../remote/database.ts"; +import { + WorkflowDatabaseCorruptError, + WorkflowRecordMalformedError, + WorkflowRequestError, + WorkflowStorageError, + WorkflowTransactionError, +} from "../storage/errors.ts"; +import type { DocumentExecutionRecord } from "../storage/record.ts"; import { decodeBase64, encodeBase64, sha256Hex } from "./encoding.ts"; export type PrivateRefusal = @@ -70,6 +80,9 @@ export type PrivateRefusal = | "command:duplicate-conflict" | "command:capacity" | "command:unavailable" + | "command:stale-root" + | "command:stale-journal" + | "command:mapping-conflict" | "storage:foreign" | "storage:unsupported-version" | "storage:corrupt"; @@ -143,6 +156,9 @@ function privateRefusal(value: string): PrivateRefusal { case "command:duplicate-conflict": case "command:capacity": case "command:unavailable": + case "command:stale-root": + case "command:stale-journal": + case "command:mapping-conflict": case "storage:foreign": case "storage:unsupported-version": case "storage:corrupt": @@ -559,3 +575,191 @@ export function* stageCloudflareContent( ), ); } + +/** + * The runner's production link to everything the database asks for. + * + * Wraps the publication link with the two reads and one mutation the database + * needs, so a handle receives one seam rather than assembling the protocol + * itself. Every answer is parsed and cross-checked against the request before + * it becomes a semantic value, and every failure crosses as a provider-neutral + * storage error rather than as a private refusal. + */ +export function cloudflareRunLink( + connection: OwnerConnection, + reads: RemoteReadLink, + nextId: () => string, +): RemoteRunLink { + const publication = cloudflareOwnerLink(connection, reads, nextId); + return { + frontier: publication.frontier, + commit: publication.commit, + + *frontierSnapshot(): Operation { + return yield* reads.frontier(); + }, + + *replaceRetrieval( + expectedWorkspaceRootId: string, + metadata: string | null, + ): Operation> { + // One identity per invocation, minted here. Two calls carrying identical + // metadata are two replacements and must not collapse into one, so the + // identity is not derived from the request's content. + const id = nextId(); + try { + const answered = yield* connection.ask( + id, + { command: "retrieval", expectedWorkspaceRootId, metadata }, + (value) => { + const found = members(value, ["retrieval"]); + const held = found.get("retrieval"); + if (held === null) { + if (metadata !== null) { + return fail("a retrieval answer cleared a replacement that was not a clear"); + } + return undefined; + } + const parsed = parseRemoteRetrieval(held); + if (parsed === undefined || metadata === null) { + return fail("a retrieval answer disagreed with the replacement it answered"); + } + return parsed; + }, + privateRefusal, + ); + return answered.outcome === "refused" + ? Err(storageFailure(privateRefusal(answered.refusal))) + : Ok(answered.value); + } catch (error) { + return Err(translate(error)); + } + }, + + *readExecutions(): Operation> { + try { + const found: { sequence: number; record: DocumentExecutionRecord }[] = []; + let anchor: number | null | undefined; + let after: number | null = null; + let done = false; + while (!done) { + const page: ExecutionPage = yield* askPage(connection, nextId(), anchor ?? null, after); + anchor ??= page.anchor; + if (page.anchor !== (anchor ?? null) || page.after !== after) { + return Err( + new WorkflowRecordMalformedError( + "document executions", + "a page did not continue its anchored snapshot", + ), + ); + } + for (const row of page.rows) { + found.push(row); + } + after = page.rows.at(-1)?.sequence ?? after; + done = page.done; + } + return Ok(found.map((row) => row.record)); + } catch (error) { + return Err(translate(error)); + } + }, + }; +} + +/** One execution page, with the private ordering the runner checks adjacency by. */ +interface ExecutionPage { + readonly anchor: number | null; + readonly after: number | null; + readonly rows: readonly { readonly sequence: number; readonly record: DocumentExecutionRecord }[]; + readonly done: boolean; +} + +function* askPage( + connection: OwnerConnection, + id: string, + anchor: number | null, + after: number | null, +): Operation { + const answered = yield* connection.ask( + id, + { command: "executions", anchor: anchor === 0 ? null : anchor, after }, + (value): ExecutionPage => { + const found = members(value, ["runId", "anchor", "after", "rows", "done"]); + const offered = found.get("rows"); + if (!Array.isArray(offered) || offered.length > EXECUTION_PAGE_ENTRIES) { + return fail("an execution page was not one bounded page"); + } + if (typeof found.get("done") !== "boolean") { + return fail("an execution page did not say whether it was terminal"); + } + let previous = after; + const rows = offered.map((entry) => { + const item = members(entry, ["sequence", "record"]); + const sequence = item.get("sequence"); + if (typeof sequence !== "number" || !Number.isSafeInteger(sequence) || sequence < 1) { + return fail("an execution row did not carry a position"); + } + if (previous !== null && sequence <= previous) { + return fail("an execution page repeated or reordered a row"); + } + previous = sequence; + return { sequence, record: parseRemoteExecution(item.get("record")) }; + }); + return { + anchor: nullableSequence(found.get("anchor")), + after: nullableSequence(found.get("after")), + rows, + done: found.get("done") === true, + }; + }, + privateRefusal, + ); + if (answered.outcome === "refused") { + throw new CloudflareOwnerRefusalError(privateRefusal(answered.refusal)); + } + return answered.value; +} + +function nullableSequence(value: unknown): number | null { + if (value === null) { + return null; + } + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 1) { + return fail("an execution page did not name a position"); + } + return value; +} + +/** + * The provider-neutral failure one private refusal becomes. + * + * A caller learns the category the local host would have reported for the same + * condition. Command names, refusal spellings, rows and cursors stay below this + * line: they describe a protocol nobody above here is party to. + */ +function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { + if (refusal.startsWith("storage:")) { + return new WorkflowDatabaseCorruptError("the workflow run's remote storage", refusal.slice(8)); + } + if (refusal === "command:stale-root" || refusal === "command:stale-journal") { + return new WorkflowTransactionError( + "this run has moved since the operation read it, so the change was not applied.", + ); + } + if (refusal === "command:capacity") { + return new WorkflowRequestError("this run's owner cannot accept more work on this connection."); + } + return new WorkflowTransactionError("this run's owner refused the operation."); +} + +/** Any failure from the private protocol, as a provider-neutral one. */ +function translate(error: unknown): WorkflowStorageError { + if (error instanceof CloudflareOwnerRefusalError) { + return storageFailure(error.refusal); + } + if (error instanceof WorkflowStorageError) { + return error; + } + return new WorkflowTransactionError("this run's owner could not be reached."); +} diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index f7d84786c..55b2442f9 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -20,6 +20,10 @@ export const MAX_COMMANDS = 256; export const MAX_LEDGER_BYTES = 2 * 1024 * 1024; export const JOURNAL_PAGE_ENTRIES = 128; export const JOURNAL_PAGE_BYTES = 512 * 1024; +/** The most document-execution rows one private page carries. */ +export const EXECUTION_PAGE_ENTRIES = 128; +/** The most serialized bytes of retained execution rows one page carries. */ +export const EXECUTION_PAGE_BYTES = 512 * 1024; /** The most content identities one proposal may name. */ export const MAX_PROPOSED_PIECES = 8192; /** The most retained mapping changes one proposal may carry. */ @@ -34,6 +38,8 @@ export type CommandName = | "content" | "stage" | "commit" + | "retrieval" + | "executions" | "settle"; export type CommandRefusal = @@ -143,6 +149,40 @@ export type ProposedMapping = | { readonly kind: "worktree"; readonly record: WorktreeRecord } | { readonly kind: "agent-session"; readonly record: AgentSessionRecord }; +/** + * Replace or clear where the definition can be fetched from. + * + * Its own mutation rather than a degenerate commit: it appends no journal + * event, publishes no root, and its revision is authoritative rather than + * proposed. `metadata` is `null` to clear, which is a different act from + * writing an empty object — clearing removes the row and the next replacement + * starts counting again. + * + * The expected root travels with it so the owner can refuse a replacement + * proposed against a frontier that has moved, the same way a commit is refused. + */ +export interface RetrievalCommand extends CommandEnvelope { + readonly command: "retrieval"; + readonly expectedWorkspaceRootId: string; + /** Canonical JSON, already encoded by the runner, or `null` to clear. */ + readonly metadata: string | null; +} + +/** + * One page of the document executions this run has begun. + * + * Anchored like the journal: the first page fixes the last execution that + * existed when the read began, and every later page is constrained to it, so an + * execution started while the read is in flight cannot appear halfway through. + */ +export interface ExecutionsCommand extends CommandEnvelope { + readonly command: "executions"; + /** The terminal sequence this snapshot is anchored to, or `null` for empty. */ + readonly anchor: number | null; + /** The sequence the previous page ended at, or `null` for the first page. */ + readonly after: number | null; +} + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -156,6 +196,8 @@ export type RunnerCommand = | ContentCommand | StageCommand | CommitCommand + | RetrievalCommand + | ExecutionsCommand | SettleCommand; export type CommandResult = @@ -179,6 +221,8 @@ const MEMBERS: Record = { "mappings", "events", ], + retrieval: [...ENVELOPE, "expectedWorkspaceRootId", "metadata"], + executions: [...ENVELOPE, "anchor", "after"], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], }; @@ -255,6 +299,18 @@ function kind(members: Map): ContentKind { * Re-encoding a nearly-right record would be worse: the owner would retain * something the runner never proposed. */ +/** A physical sequence, which is a positive whole number or nothing. */ +function sequence(members: Map, key: string): number | null { + const value = members.get(key); + if (value === null) { + return null; + } + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 1) { + throw new CommandError("malformed-member"); + } + return value; +} + function eventRecords(value: unknown): string[] { if (!Array.isArray(value)) { throw new CommandError("malformed-member"); @@ -294,6 +350,8 @@ export function parseCommand(raw: string): RunnerCommand { command !== "content" && command !== "stage" && command !== "commit" && + command !== "retrieval" && + command !== "executions" && command !== "settle" ) { throw new CommandError("unknown-command"); @@ -338,6 +396,33 @@ export function parseCommand(raw: string): RunnerCommand { bytes: text(members, "bytes", Math.ceil((MAX_CONTENT_BYTES * 4) / 3) + 4), }; } + if (command === "retrieval") { + const metadata = members.get("metadata"); + if (metadata !== null && (typeof metadata !== "string" || metadata === "")) { + throw new CommandError("malformed-member"); + } + if (metadata !== null && new TextEncoder().encode(metadata).length > MAX_MESSAGE_BYTES) { + throw new CommandError("too-large"); + } + return { + id, + command, + expectedWorkspaceRootId: digest(members, "expectedWorkspaceRootId"), + metadata, + }; + } + if (command === "executions") { + const anchor = sequence(members, "anchor"); + const after = sequence(members, "after"); + if (anchor === null && after !== null) { + // An empty snapshot has nothing to continue from. + throw new CommandError("malformed-member"); + } + if (anchor !== null && after !== null && after >= anchor) { + throw new CommandError("malformed-member"); + } + return { id, command, anchor, after }; + } if (command === "settle") { const completion = parseDocumentExecutionCompletion(members.get("completion")); if (!completion.ok) { diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 607bd08a2..8ee8096f8 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -44,14 +44,34 @@ import { type RunnerCommand, } from "./commands.ts"; import { bytesOf, decodeBase64, sha256Hex } from "./encoding.ts"; -import { readContent, readFrontier, readJournalPage, readRoot } from "./owner-reads.ts"; +import { + readContent, + readExecutions, + readFrontier, + readJournalPage, + readRoot, +} from "./owner-reads.ts"; import type { OwnerTransactions } from "./owner-transaction.ts"; import { COMMAND_TABLE, MUTATION_TABLE, STAGING_TABLE } from "./private-schema.ts"; -import { applyCommit } from "./publish.ts"; +import { applyCommit, applyRetrieval } from "./publish.ts"; import { recognizeObject } from "./recognition.ts"; function requestFingerprint(command: RunnerCommand): string { - return sha256Hex(JSON.stringify(command)); + // The command name is part of the fingerprint, so one textual id used for a + // commit and for a retrieval replacement is two different requests rather + // than one recognized retry. + return sha256Hex(JSON.stringify({ kind: command.command, command })); +} + +/** + * Whether this command changes the run, and therefore whether its decision has + * to outlive the connection that asked for it. + * + * A read can be asked again; a mutation cannot, so its answer is retained where + * the next connection can find it. + */ +function mutating(command: RunnerCommand): boolean { + return command.command === "commit" || command.command === "retrieval"; } function integer(value: unknown): number { @@ -103,10 +123,23 @@ function mintEventId(): string { return crypto.randomUUID(); } +/** + * The moment the owner records against a mutation it just made. + * + * The owner's clock, not the runner's. A time a runner supplied would be a + * caller deciding when the run's history happened. + */ +function ownerTime(): string { + return new Date().toISOString(); +} + function retainedDecision(command: RunnerCommand, result: CommandResult): string { if ( result.outcome === "performed" && - (command.command === "journal" || command.command === "root" || command.command === "content") + (command.command === "journal" || + command.command === "root" || + command.command === "content" || + command.command === "executions") ) { return JSON.stringify({ id: command.id, outcome: "reconstruct" }); } @@ -219,6 +252,20 @@ function perform( value: applyCommit(ctx.storage, acquisitionId, command, mintEventId), }; } + if (command.command === "retrieval") { + return { + id: command.id, + outcome: "performed", + value: applyRetrieval(ctx.storage, command, ownerTime), + }; + } + if (command.command === "executions") { + return { + id: command.id, + outcome: "performed", + value: readExecutions(ctx.storage, runId, command.anchor, command.after), + }; + } // `settle` is a later checkpoint's. It parses strictly and is declined, // because a placeholder that reported success is the one answer a runner // cannot recover from. @@ -245,7 +292,7 @@ export function dispatchCommand( // The case this exists for is the one where the connection that asked is // gone: the owner committed, the answer never arrived, and the runner // reconnected to ask the same question again. - if (command.command === "commit") { + if (mutating(command)) { const decided = ctx.storage.sql .exec( `SELECT request_fingerprint, response FROM ${MUTATION_TABLE} WHERE command_id = ?`, @@ -312,7 +359,7 @@ export function dispatchCommand( encoded, responseBytes, ); - if (command.command === "commit") { + if (mutating(command)) { // Recorded in this same transaction as the mutation it describes, so a // crash cannot leave one without the other. const mutations = ctx.storage.sql diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index 1f43833d3..d84aa8e9c 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -26,7 +26,7 @@ */ import { parseDurableEvent } from "@executablemd/durable-streams"; -import { readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; +import { readDocumentExecution, readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { parseWorkspaceRootManifest, @@ -37,6 +37,8 @@ import { import { type ContentManifest, decodeContentManifest } from "../workspace/content-manifest.ts"; import { CommandError, + EXECUTION_PAGE_BYTES, + EXECUTION_PAGE_ENTRIES, JOURNAL_PAGE_BYTES, JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES, @@ -466,3 +468,88 @@ function piece( } return validatedBlob(storage, rootId, root.manifests, digest); } + +/** One page of document executions, anchored to the snapshot that began it. */ +export interface ExecutionsValue { + readonly runId: string; + readonly anchor: number | null; + readonly after: number | null; + readonly rows: readonly { readonly sequence: number; readonly record: Row }[]; + readonly done: boolean; +} + +/** + * Read one bounded page of the executions this run has begun. + * + * Anchored the way the journal is, and for the same reason: a caller assembling + * a list across several requests must see one snapshot rather than whatever the + * table held at each moment. The first page fixes the terminal sequence; every + * later page is constrained to it, so an execution begun while the read is in + * flight cannot appear halfway through the answer. + * + * The run identity travels with the page so the runner can refuse an answer + * from another run, and the sequence travels so it can prove adjacency. Neither + * becomes part of the semantic record. + */ +export function readExecutions( + storage: OwnerStorage, + runId: string, + anchor: number | null, + after: number | null, +): ExecutionsValue { + if (anchor === null) { + const last = byteRows( + storage, + "SELECT sequence FROM document_executions ORDER BY sequence DESC LIMIT 1", + )[0]; + if (last !== undefined) { + return corrupt("an empty execution snapshot was anchored against existing rows"); + } + return { runId, anchor, after, rows: [], done: true }; + } + + const found = byteRows( + storage, + `SELECT sequence, execution_id, started_at, stopped_at, stop_status, + stop_reason_kind, stop_reason_code, stop_reason_event_id + FROM document_executions + WHERE sequence > ? AND sequence <= ? ORDER BY sequence ASC LIMIT ?`, + after ?? 0, + anchor, + EXECUTION_PAGE_ENTRIES + 1, + ); + + const page: { sequence: number; record: Row }[] = []; + let encoded = 0; + for (const row of found.slice(0, EXECUTION_PAGE_ENTRIES)) { + const at = safeInteger(row["sequence"], "execution sequence"); + // Parsed here as well as on the runner: a row this owner cannot read is + // storage damage, and sending it would make the runner report damage it + // cannot attribute. + readDocumentExecution(row); + const bytes = new TextEncoder().encode(JSON.stringify(row)).length; + if (page.length > 0 && encoded + bytes > EXECUTION_PAGE_BYTES) { + break; + } + if (bytes > MAX_CONTENT_BYTES) { + throw new CommandError("too-large"); + } + page.push({ sequence: at, record: row }); + encoded += bytes; + } + + const done = found.length <= page.length; + if (done && page.at(-1)?.sequence !== anchor) { + return corrupt("an anchored execution snapshot is incomplete"); + } + return { runId, anchor, after, rows: page, done }; +} + +/** The terminal execution sequence right now, or `null` when there is none. */ +export function executionAnchor(storage: OwnerStorage): number | null { + const last = byteRows( + storage, + "SELECT sequence FROM document_executions ORDER BY sequence DESC LIMIT 1", + )[0]; + return last === undefined ? null : safeInteger(last["sequence"], "execution sequence"); +} diff --git a/packages/workflow/src/cloudflare/publish.ts b/packages/workflow/src/cloudflare/publish.ts index a1344465d..c8704718d 100644 --- a/packages/workflow/src/cloudflare/publish.ts +++ b/packages/workflow/src/cloudflare/publish.ts @@ -51,6 +51,7 @@ import { MAX_CONTENT_BYTES } from "./commands.ts"; import { sha256Hex } from "../workspace/sha256.ts"; import { CommandError, type CommitCommand, type ProposedMapping } from "./commands.ts"; import { validateRetainedRoot } from "./owner-reads.ts"; +import { readRetrieval } from "../sqlite/rows.ts"; import { bytesOf } from "./encoding.ts"; import { STAGING_TABLE } from "./private-schema.ts"; import type { OwnerStorage } from "./storage.ts"; @@ -678,3 +679,84 @@ function applyMapping(storage: OwnerStorage, mapping: ProposedMapping): void { throw new CommandError("mapping-conflict"); } } + +/** What the owner answers a performed retrieval replacement with. */ +export interface RetrievalValue { + readonly retrieval: { + readonly metadata: unknown; + readonly revision: number; + readonly updatedAt: string; + } | null; +} + +/** + * Replace or clear where this run's definition can be fetched from. + * + * Its own mutation rather than a degenerate commit. Nothing is appended to the + * journal, no root moves, and the revision is the owner's arithmetic over what + * is stored rather than a number the runner proposed — two handles that both + * read revision one before either wrote would otherwise both write two, and the + * second would silently lose the first. + * + * The expected root is revalidated here, inside the transaction that writes, so + * a replacement proposed against a frontier that has moved is refused on the + * same terms a commit is. + */ +export function applyRetrieval( + storage: OwnerStorage, + command: { expectedWorkspaceRootId: string; metadata: string | null }, + now: () => string, +): RetrievalValue { + const state = rows(storage, "SELECT current_root_id FROM workspace_state WHERE singleton_id = 1"); + const current = state[0]?.["current_root_id"]; + if (state.length !== 1 || typeof current !== "string") { + return corrupt("the Workspace has no single current root"); + } + if (current !== command.expectedWorkspaceRootId) { + throw new CommandError("stale-root"); + } + validateRetainedRoot(storage, current); + + if (command.metadata === null) { + // Clearing removes the row. The next replacement starts counting again, + // because a revision counts replacements since the metadata last existed. + storage.sql.exec("DELETE FROM definition_retrieval WHERE id = 1"); + return { retrieval: null }; + } + + const held = rows(storage, "SELECT revision FROM definition_retrieval WHERE id = 1")[0]; + const revision = held === undefined ? 1 : safeRevision(held["revision"]) + 1; + const updatedAt = now(); + storage.sql.exec( + `INSERT INTO definition_retrieval (id, metadata, revision, updated_at) VALUES (1, ?, ?, ?) + ON CONFLICT(id) DO UPDATE SET metadata = excluded.metadata, + revision = excluded.revision, updated_at = excluded.updated_at`, + command.metadata, + revision, + updatedAt, + ); + + const written = rows( + storage, + "SELECT metadata, revision, updated_at FROM definition_retrieval WHERE id = 1", + )[0]; + if (written === undefined) { + return corrupt("a retrieval replacement wrote no row"); + } + // Read back and parsed, so the answer describes what is actually stored. + const parsed = readRetrieval(written); + return { + retrieval: { + metadata: parsed.metadata, + revision: parsed.revision, + updatedAt: parsed.updatedAt, + }, + }; +} + +function safeRevision(value: unknown): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 1) { + return corrupt("a retained retrieval revision is not a positive whole number"); + } + return value; +} diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts new file mode 100644 index 000000000..ff6db23bb --- /dev/null +++ b/packages/workflow/src/remote/database.ts @@ -0,0 +1,331 @@ +/** + * One run's storage, when the run is owned somewhere else. + * + * The same handle the local host hands out, backed by a connection instead of a + * file. Everything the interface promises has to be true here for the same + * reasons it is true there — a snapshot is a snapshot, a transaction commits or + * it does not, and a closed handle is closed — and the differences are all + * beneath it: there is no connection to hold open across a callback, so the + * body runs on the runner and only what it enlisted crosses. + * + * Two mechanisms keep operations in order and they solve different problems. + * A *turn* serializes work so two operations do not interleave on one handle; + * unrelated work waits and then proceeds. A *marker* records that this scope is + * inside a transaction on this handle, so a nested transaction — or an ordinary + * operation called from inside the body — is refused immediately rather than + * waiting for a turn its own caller is holding and will not release. A queue + * alone would deadlock that case; a flag alone would mistake unrelated work for + * nested work. + * + * The handle is a lease. Closing it ends this handle and nothing else: the + * connection may be owned by an outer scope and shared with other handles, and + * a lease that closed it would end a run somebody else was still reading. + */ + +import { + createContext, + createSignal, + ensure, + Err, + Ok, + type Context, + type Operation, + type Result, + resource, +} from "effection"; +import type { DurableEvent, DurableStream, Json } from "@executablemd/durable-streams"; +import type { JournalEntry, WorkflowRunDatabase, WorkflowRunTransaction } from "../storage/api.ts"; +import { WorkflowDatabaseClosedError, WorkflowTransactionError } from "../storage/errors.ts"; +import type { + DefinitionRetrieval, + DocumentExecutionRecord, + WorkflowRunRecord, +} from "../storage/record.ts"; +import { createTransactionGate, type OwnerLink, transactRemotely } from "./collector.ts"; +import type { EnlistWorkspace } from "./collector.ts"; +import type { RemoteFrontierSnapshot } from "./read.ts"; + +/** What a remote handle needs to answer everything the interface asks. */ +export interface RemoteRunLink extends OwnerLink { + /** A fresh coherent frontier, for a read that must not use a snapshot. */ + frontierSnapshot(): Operation; + /** Replace or clear the retrieval metadata, and answer with the result. */ + replaceRetrieval( + expectedWorkspaceRootId: string, + metadata: string | null, + ): Operation>; + /** Every document execution, as one anchored snapshot. */ + readExecutions(): Operation>; +} + +/** + * Which handles this scope is inside a transaction on. + * + * Structural and inert, exactly like the local provider's: it can only ever + * cause an operation to be refused, never authorize one. A chain rather than a + * single handle, because transactions on *different* runs may nest and + * recording only the innermost would hide the outer one. + */ +interface OpenTransaction { + readonly handle: object; + readonly enclosing: OpenTransaction | undefined; +} + +const ActiveTransaction: Context = createContext< + OpenTransaction | undefined +>("executablemd.workflow.remote.transaction", undefined); + +function* holdsTransactionOn(handle: object): Operation { + let active = yield* ActiveTransaction.get(); + while (active !== undefined) { + if (active.handle === handle) { + return true; + } + active = active.enclosing; + } + return false; +} + +/** + * The route a Workspace coordinator reaches the active transaction through. + * + * Bound to one exact handle and one exact transaction object, and live only + * inside that transaction body's descendant scope. D3c installs a coordinator + * over it; nothing about a document execution or its provenance is decided + * here, and no placeholder for either is invented. + */ +export interface WorkspaceRoute { + readonly database: WorkflowRunDatabase; + readonly transaction: WorkflowRunTransaction; + readonly enlist: EnlistWorkspace; +} + +const ActiveRoute: Context = createContext( + "executablemd.workflow.remote.workspace-route", + undefined, +); + +/** + * The enlistment route for this exact database and transaction, if it is live. + * + * Answers nothing for a foreign database, a substituted or stale transaction + * object, or a scope outside the body — which is the whole point: a coordinator + * that has drifted from the transaction it belongs to must not be able to + * publish into it. + */ +export function* activeWorkspaceRoute( + database: WorkflowRunDatabase, + transaction: WorkflowRunTransaction, +): Operation { + const route = yield* ActiveRoute.get(); + if (route === undefined || route.database !== database || route.transaction !== transaction) { + return undefined; + } + return route; +} + +/** One handle's cooperative turn, so two operations never interleave on it. */ +interface Turns { + take(body: () => Operation): Operation; +} + +function createTurns(): Turns { + const waiting = createSignal(); + let held = false; + return { + *take(body: () => Operation): Operation { + while (held) { + // Someone else has the handle. Wait to be told it is free rather than + // polling, and check again, because several may be waiting and only one + // of them can take the turn that was just released. + const released = yield* waiting; + yield* released.next(); + } + held = true; + try { + return yield* body(); + } finally { + held = false; + waiting.send(); + } + }, + }; +} + +/** Open one scope-owned lease on a run whose storage is somewhere else. */ +export function useRemoteRunDatabase( + link: RemoteRunLink, + frontier: RemoteFrontierSnapshot, +): Operation { + return resource(function* (provide) { + let closed = false; + let record: WorkflowRunRecord = frontier.record; + let retrieval: DefinitionRetrieval | undefined = frontier.retrieval; + const turns = createTurns(); + const gate = createTransactionGate(); + + /** Whether this scope may reach the handle at all, and why not. */ + function* admit(): Operation> { + if (closed) { + return Err(new WorkflowDatabaseClosedError(record.runId)); + } + if (yield* holdsTransactionOn(handle)) { + return Err( + new WorkflowTransactionError( + "this scope is inside a transaction on the same workflow run database, and an " + + "operation outside that transaction cannot run until it commits. Use the " + + "transaction handed to the body, or move the operation outside it.", + ), + ); + } + return Ok(); + } + + /** One turn at the handle, for an ordinary operation. */ + function* turn(body: () => Operation>): Operation> { + const admitted = yield* admit(); + if (!admitted.ok) { + return admitted; + } + return yield* turns.take(body); + } + + /** The same, for a `DurableStream` member, which raises instead. */ + function* raising(result: Result): Operation { + if (!result.ok) { + throw result.error; + } + return result.value; + } + + const ordinary: DurableStream = { + *readAll(): Operation { + return yield* raising( + yield* turn(function* () { + const snapshot = yield* link.frontierSnapshot(); + return Ok(snapshot.entries.map((entry) => structuredClone(entry.event))); + }), + ); + }, + + *append(event: DurableEvent): Operation { + // One journal-only transaction through the same commit path a caller's + // transaction uses. A second insertion route would be a second thing to + // keep in agreement with the first. + yield* raising( + yield* transact(function* (transaction) { + yield* transaction.journal.append(event); + }), + ); + }, + }; + + function* transact( + body: (transaction: WorkflowRunTransaction) => Operation, + ): Operation> { + if (closed) { + return Err(new WorkflowDatabaseClosedError(record.runId)); + } + if (yield* holdsTransactionOn(handle)) { + return Err( + new WorkflowTransactionError( + "a transaction on this workflow run database is already open in this scope. " + + "Nesting one inside another would commit or roll back work the outer " + + "transaction has not finished deciding about.", + ), + ); + } + return yield* turns.take(function* (): Operation> { + return yield* transactRemotely(link, gate, function* (transaction, enlist) { + // The marker and the route are installed for the body's scope alone. + // Outside it neither exists, so a retained transaction object reaches + // nothing and an unrelated scope is not mistaken for a nested one. + yield* ActiveTransaction.set({ + handle, + enclosing: yield* ActiveTransaction.get(), + }); + yield* ActiveRoute.set({ database: handle, transaction, enlist }); + return yield* body(transaction); + }); + }); + } + + const handle: WorkflowRunDatabase = { + get record(): WorkflowRunRecord { + return record; + }, + + get retrieval(): DefinitionRetrieval | undefined { + return retrieval; + }, + + get journal(): DurableStream { + return ordinary; + }, + + transact, + + *readJournalEntries(): Operation> { + return yield* turn(function* () { + const snapshot = yield* link.frontierSnapshot(); + return Ok(snapshot.entries.map((entry) => Object.freeze({ ...entry }))); + }); + }, + + *replaceRetrievalMetadata(metadata: Json | undefined): Operation> { + const replaced = yield* turn(function* () { + const snapshot = yield* link.frontierSnapshot(); + return yield* link.replaceRetrieval( + snapshot.workspaceRootId, + metadata === undefined ? null : canonical(metadata), + ); + }); + if (!replaced.ok) { + return replaced; + } + // Only this handle, and only after its own successful replacement. The + // owner's revision and time are what is recorded; nothing is invented + // here. + retrieval = replaced.value; + return Ok(); + }, + + *readDocumentExecutions(): Operation> { + return yield* turn(() => link.readExecutions()); + }, + }; + + yield* ensure(() => { + closed = true; + }); + yield* provide(handle); + }); +} + +/** + * The canonical encoding of one retrieval metadata value. + * + * Sorted keys and no incidental whitespace, so two callers writing the same + * metadata write the same bytes and a comparison of what is stored means what + * it appears to mean. + */ +function canonical(value: Json): string { + return JSON.stringify(sorted(value)); +} + +function sorted(value: Json): Json { + if (Array.isArray(value)) { + return value.map(sorted); + } + if (value === null || typeof value !== "object") { + return value; + } + const members: Record = {}; + for (const key of Object.keys(value).sort()) { + const held = (value as Record)[key]; + if (held !== undefined) { + members[key] = sorted(held); + } + } + return members; +} diff --git a/packages/workflow/src/remote/records.ts b/packages/workflow/src/remote/records.ts index 1853f35a3..ab58b50ad 100644 --- a/packages/workflow/src/remote/records.ts +++ b/packages/workflow/src/remote/records.ts @@ -28,6 +28,7 @@ import { } from "../storage/members.ts"; import { type DefinitionRetrieval, + type DocumentExecutionRecord, parseRunId, parseWorkflowRunStatus, parseWorkflowStopReason, @@ -141,3 +142,38 @@ export function parseRemoteJournalEntry(value: unknown): JournalEntry { workspaceRootId: rootId(members.get("workspaceRootId"), "$.workspaceRootId"), }); } + +/** + * One document execution, read out of a value nothing has checked. + * + * The shared rules the local host holds its own rows to, applied to what + * arrived. A stopped execution has to carry its status, and a stop reason has + * to agree with the way the record spells one — an execution that stopped for a + * reason the shape does not admit is not a record this build can act on. + */ +export function parseRemoteExecution(value: unknown): DocumentExecutionRecord { + const found = parseMembers(value, "$", fail); + const executionId = parseStringMember(found, "executionId", "$", fail); + if (executionId === "") { + throw fail("expected a non-empty identity", "$.executionId"); + } + const record: DocumentExecutionRecord = { + executionId, + startedAt: instant(found.get("startedAt"), "$.startedAt"), + }; + if (!found.has("stoppedAt")) { + return Object.freeze(record); + } + const stopped: DocumentExecutionRecord = { + ...record, + stoppedAt: instant(found.get("stoppedAt"), "$.stoppedAt"), + stopStatus: parseWorkflowRunStatus(found.get("stopStatus"), "$.stopStatus", fail), + }; + if (!found.has("stopReason")) { + return Object.freeze(stopped); + } + return Object.freeze({ + ...stopped, + stopReason: parseWorkflowStopReason(found.get("stopReason"), "$.stopReason", fail), + }); +} diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index c7519c8b1..395ff462f 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -451,7 +451,9 @@ describe("the remote owner protocol", () => { expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 1, staged: 1 }); const before = await on(stub, (owner) => owner.authoritative()); const replaced = await on(stub, (owner) => owner.acquisitionId()); - await admit(stub); + // A real accepted connection for the replacement: the rest of this test + // reads across several object round trips. + const successor = await connect(stub); // A second acquisition, and the first one's scratch is gone rather than // inherited: it cannot be retried, adopted or read. expect(await on(stub, (owner) => owner.acquisitionId())).not.toBe(replaced); @@ -461,7 +463,7 @@ describe("the remote owner protocol", () => { // bytes writes a new row rather than finding the abandoned one. Nothing was // inherited; it was discarded and done afresh. expect( - await send(stub, "stage", { + await ask(successor, "stage", { command: "stage", kind: "blob", digest, @@ -469,7 +471,7 @@ describe("the remote owner protocol", () => { }), ).toMatchObject({ outcome: "performed", value: { digest } }); expect(await on(stub, (owner) => owner.scratch())).toEqual({ commands: 1, staged: 1 }); - expect(await send(stub, "frontier-new", { command: "frontier" })).toMatchObject({ + expect(await ask(successor, "frontier-new", { command: "frontier" })).toMatchObject({ outcome: "performed", value: { workspaceRootId: ROOT_ID }, }); From fdb2b4a5a71262231927bf246631c0125a87a7a0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 10:52:42 -0400 Subject: [PATCH 41/81] =?UTF-8?q?=F0=9F=A7=AE=20Anchor=20the=20execution?= =?UTF-8?q?=20list=20where=20the=20owner=20can=20choose=20the=20anchor=20(?= =?UTF-8?q?#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runner had no way to learn which snapshot it was reading. It sent the first page with no anchor, and the owner treated that as a claim the run had no executions — true only for an empty run, and storage damage for every other, so a run with any history could not read its own list at all. The owner chooses the anchor now. A first request carries none, the owner fixes the terminal row at that moment and answers with it, and every later page is held to what the first one chose. An empty run answers with an explicit empty anchor, which is a different answer from a terminal page and is read as one. The evidence is the case that exposed it: 129 executions, a first page that returns 128 rows and the anchor it selected, another execution begun while the read is in flight, and a final page that ends exactly at the anchor without it. Also proves the retrieval mutation on real storage: revisions counted by the owner, byte-identical metadata under two identities counted as two replacements, clearing removing the row so the next one starts again, one invocation retried across a lost answer and eviction applying once, the same identity with different content refused, and a replacement against a root the run has left refused without touching the row. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 8 +- .../workflow/src/cloudflare/owner-reads.ts | 23 +- packages/workflow/src/remote/database.ts | 34 +- .../tests/cloudflare/remote-publish.vitest.ts | 127 +++++ .../cloudflare/support/executor-object.ts | 17 + .../workflow/tests/remote-database.test.ts | 442 ++++++++++++++++++ 6 files changed, 624 insertions(+), 27 deletions(-) create mode 100644 packages/workflow/tests/remote-database.test.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 9937b1ab1..6fbacb65f 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -644,8 +644,10 @@ export function cloudflareRunLink( let done = false; while (!done) { const page: ExecutionPage = yield* askPage(connection, nextId(), anchor ?? null, after); - anchor ??= page.anchor; - if (page.anchor !== (anchor ?? null) || page.after !== after) { + // The first page chooses the anchor; every later one is held to it. + const expected = anchor === undefined ? page.anchor : anchor; + anchor = expected; + if (page.anchor !== expected || page.after !== after) { return Err( new WorkflowRecordMalformedError( "document executions", @@ -683,7 +685,7 @@ function* askPage( ): Operation { const answered = yield* connection.ask( id, - { command: "executions", anchor: anchor === 0 ? null : anchor, after }, + { command: "executions", anchor, after }, (value): ExecutionPage => { const found = members(value, ["runId", "anchor", "after", "rows", "done"]); const offered = found.get("rows"); diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index d84aa8e9c..5d5c16a79 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -497,15 +497,16 @@ export function readExecutions( anchor: number | null, after: number | null, ): ExecutionsValue { - if (anchor === null) { - const last = byteRows( - storage, - "SELECT sequence FROM document_executions ORDER BY sequence DESC LIMIT 1", - )[0]; - if (last !== undefined) { - return corrupt("an empty execution snapshot was anchored against existing rows"); + // The first request carries no anchor because the runner has nothing to + // anchor to yet. The owner chooses it — the terminal row at this moment — and + // answers with it, so every later page is held to the snapshot this one + // began. An empty run answers with an explicit empty anchor. + const selected = anchor ?? (after === null ? executionAnchor(storage) : null); + if (selected === null) { + if (after !== null) { + return corrupt("an empty execution snapshot names an earlier row"); } - return { runId, anchor, after, rows: [], done: true }; + return { runId, anchor: null, after, rows: [], done: true }; } const found = byteRows( @@ -515,7 +516,7 @@ export function readExecutions( FROM document_executions WHERE sequence > ? AND sequence <= ? ORDER BY sequence ASC LIMIT ?`, after ?? 0, - anchor, + selected, EXECUTION_PAGE_ENTRIES + 1, ); @@ -539,10 +540,10 @@ export function readExecutions( } const done = found.length <= page.length; - if (done && page.at(-1)?.sequence !== anchor) { + if (done && page.at(-1)?.sequence !== selected) { return corrupt("an anchored execution snapshot is incomplete"); } - return { runId, anchor, after, rows: page, done }; + return { runId, anchor: selected, after, rows: page, done }; } /** The terminal execution sequence right now, or `null` when there is none. */ diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index ff6db23bb..248bc25b5 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -124,6 +124,20 @@ export function* activeWorkspaceRoute( return route; } +/** + * What a `DurableStream` member does with a result. + * + * The interface splits these deliberately: a member returning `Result` answers + * with the failure, and a stream member raises it. Both describe the same + * condition. + */ +function* raising(result: Result): Operation { + if (!result.ok) { + throw result.error; + } + return result.value; +} + /** One handle's cooperative turn, so two operations never interleave on it. */ interface Turns { take(body: () => Operation): Operation; @@ -131,21 +145,21 @@ interface Turns { function createTurns(): Turns { const waiting = createSignal(); - let held = false; + const holder = { held: false }; return { *take(body: () => Operation): Operation { - while (held) { + while (holder.held) { // Someone else has the handle. Wait to be told it is free rather than // polling, and check again, because several may be waiting and only one // of them can take the turn that was just released. const released = yield* waiting; yield* released.next(); } - held = true; + holder.held = true; try { return yield* body(); } finally { - held = false; + holder.held = false; waiting.send(); } }, @@ -190,14 +204,6 @@ export function useRemoteRunDatabase( return yield* turns.take(body); } - /** The same, for a `DurableStream` member, which raises instead. */ - function* raising(result: Result): Operation { - if (!result.ok) { - throw result.error; - } - return result.value; - } - const ordinary: DurableStream = { *readAll(): Operation { return yield* raising( @@ -321,7 +327,9 @@ function sorted(value: Json): Json { return value; } const members: Record = {}; - for (const key of Object.keys(value).sort()) { + const names = Object.keys(value); + names.sort(); + for (const key of names) { const held = (value as Record)[key]; if (held !== undefined) { members[key] = sorted(held); diff --git a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts index d4616e2c1..e93197f6a 100644 --- a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts @@ -735,3 +735,130 @@ describe("publishing one proposal", () => { expect(await on(stub, (owner) => owner.published())).toEqual(before); }); }); + +describe("the run's own records", () => { + it("counts retrieval revisions authoritatively, and clearing starts again", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + + // The run is created with a retrieval row, so the first replacement is the + // next revision rather than the first. + const before = await on(stub, (owner) => owner.retrieval()); + expect(before).not.toBe(null); + + const first = await ask(socket, "r1", { + command: "retrieval", + expectedWorkspaceRootId: ROOT_ID, + metadata: '{"locator":"https://example.invalid/a.git"}', + }); + expect(first).toMatchObject({ outcome: "performed" }); + + // Byte-identical metadata under a different id is a second replacement. + const second = await ask(socket, "r2", { + command: "retrieval", + expectedWorkspaceRootId: ROOT_ID, + metadata: '{"locator":"https://example.invalid/a.git"}', + }); + expect(second).toMatchObject({ outcome: "performed" }); + const revisions = [first, second].map((answer) => + Number(record(record(answer["value"])["retrieval"])["revision"]), + ); + expect(revisions[1]).toBe((revisions[0] ?? 0) + 1); + + // Clearing removes the row; the next replacement counts from one. + expect( + await ask(socket, "r3", { + command: "retrieval", + expectedWorkspaceRootId: ROOT_ID, + metadata: null, + }), + ).toEqual({ id: "r3", outcome: "performed", value: { retrieval: null } }); + expect(await on(stub, (owner) => owner.retrieval())).toBe(null); + const restarted = await ask(socket, "r4", { + command: "retrieval", + expectedWorkspaceRootId: ROOT_ID, + metadata: '{"locator":"https://example.invalid/b.git"}', + }); + expect(Number(record(record(restarted["value"])["retrieval"])["revision"])).toBe(1); + }); + + it("applies one retrieval replacement once across a lost answer and eviction", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + const request = { + command: "retrieval", + expectedWorkspaceRootId: ROOT_ID, + metadata: '{"locator":"https://example.invalid/once.git"}', + }; + const performed = await ask(socket, "once", request); + expect(performed).toMatchObject({ outcome: "performed" }); + const stored = await on(stub, (owner) => owner.retrieval()); + + socket.close(1000, "lost"); + await new Promise((resolve) => setTimeout(resolve, 0)); + await evictDurableObject(stub); + + const replacement = await connect(stub); + // The same invocation asked again: the retained decision answers, and the + // revision does not move. + expect(await ask(replacement, "once", request)).toEqual(performed); + expect(await on(stub, (owner) => owner.retrieval())).toEqual(stored); + + // The same identity for different content is a conflict, not a retry. + expect( + await ask(replacement, "once", { ...request, metadata: '{"locator":"other"}' }), + ).toMatchObject({ outcome: "refused", refusal: "command:duplicate-conflict" }); + expect(await on(stub, (owner) => owner.retrieval())).toEqual(stored); + }); + + it("refuses a replacement proposed against a root the run has left", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + const before = await on(stub, (owner) => owner.retrieval()); + expect( + await ask(socket, "stale", { + command: "retrieval", + expectedWorkspaceRootId: `f${"0".repeat(63)}`, + metadata: '{"locator":"x"}', + }), + ).toMatchObject({ outcome: "refused", refusal: "command:stale-root" }); + expect(await on(stub, (owner) => owner.retrieval())).toEqual(before); + }); + + it("anchors a multipage execution snapshot and excludes a later one", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + for (let index = 0; index < 129; index += 1) { + await on(stub, (owner) => + owner.beginExecution( + `execution-${index}`, + `2026-09-04T00:00:${String(index % 60).padStart(2, "0")}.000Z`, + ), + ); + } + const socket = await connect(stub); + + // The first request carries no anchor; the owner chooses the terminal row + // at this moment and answers with it. + const anchored = record( + (await ask(socket, "x1", { command: "executions", anchor: null, after: null }))["value"], + ); + expect(anchored["anchor"]).toBe(129); + expect(Array.isArray(anchored["rows"]) && anchored["rows"]).toHaveLength(128); + expect(anchored["done"]).toBe(false); + + // A later execution begins while the read is in flight. + await on(stub, (owner) => owner.beginExecution("execution-later", "2026-09-04T01:00:00.000Z")); + + const second = record( + (await ask(socket, "x2", { command: "executions", anchor: 129, after: 128 }))["value"], + ); + expect(Array.isArray(second["rows"]) && second["rows"]).toHaveLength(1); + expect(second["done"]).toBe(true); + // The one begun after the anchor is not in the snapshot. + expect(record((second["rows"] as Record[])[0] ?? {})["sequence"]).toBe(129); + }); +}); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index e39683434..7ab9cb51d 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -353,6 +353,23 @@ export class ExecutorObject extends WorkflowOwnerObject { ); } + /** Begin one document execution, as a lifecycle transition would. */ + beginExecution(executionId: string, startedAt: string): void { + this.ctx.storage.sql.exec( + "INSERT INTO document_executions (execution_id, started_at) VALUES (?, ?)", + executionId, + startedAt, + ); + } + + /** What the retrieval row holds right now. */ + retrieval(): Record | null { + const row = this.ctx.storage.sql + .exec("SELECT metadata, revision, updated_at FROM definition_retrieval WHERE id = 1") + .toArray()[0]; + return row === undefined ? null : row; + } + damageRetainedBlob(): void { this.ctx.storage.sql.exec( "UPDATE vfs_blob_bytes SET bytes = ?", diff --git a/packages/workflow/tests/remote-database.test.ts b/packages/workflow/tests/remote-database.test.ts new file mode 100644 index 000000000..2b882055c --- /dev/null +++ b/packages/workflow/tests/remote-database.test.ts @@ -0,0 +1,442 @@ +/** + * Tier WRH — one run's storage, owned somewhere else. + * + * The interface is the same one the local host answers, so what is under test + * is conformance rather than mechanism: a snapshot stays a snapshot, a nested + * transaction refuses while unrelated work waits its turn, a closed handle is + * closed, and every read is a fresh anchored one rather than a cache. + * + * The owner is a deterministic fake. What it is standing in for — atomic + * application, authoritative revisions, anchored SQLite ordering — is proved on + * real workerd; what is proved here is the handle's own behaviour, which is + * arithmetic over what the owner said. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import { Err, Ok, type Operation, type Result, scoped, sleep, spawn } from "effection"; +import type { WorkflowRunDatabase, WorkflowRunTransaction } from "../src/storage/api.ts"; +import { WorkflowDatabaseClosedError, WorkflowTransactionError } from "../src/storage/errors.ts"; +import type { DefinitionRetrieval, DocumentExecutionRecord } from "../src/storage/record.ts"; +import type { CommitIntent, StartingFrontier } from "../src/remote/collector.ts"; +import type { CommitDecision } from "../src/remote/publication.ts"; +import { + activeWorkspaceRoute, + type RemoteRunLink, + useRemoteRunDatabase, +} from "../src/remote/database.ts"; +import type { RemoteFrontierSnapshot } from "../src/remote/read.ts"; + +const ROOT = "a".repeat(64); +const RUN_ID = "remote-run"; + +function event(name: string): DurableEvent { + return { + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }; +} + +function frontierOf(entries: { eventId: string; event: DurableEvent }[]): RemoteFrontierSnapshot { + return { + record: { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-04T00:00:00.000Z", + updatedAt: "2026-09-04T00:00:00.000Z", + }, + retrieval: undefined, + workspaceRootId: ROOT, + journalEventId: entries.at(-1)?.eventId ?? null, + entries: entries.map((entry) => ({ ...entry, workspaceRootId: ROOT })), + }; +} + +/** An owner that answers from what it has been told, and records what it was asked. */ +function owner( + options: { + retrieval?: (metadata: string | null) => Result; + executions?: () => Result; + commit?: (intent: CommitIntent) => Result; + } = {}, +) { + const retained: { eventId: string; event: DurableEvent }[] = []; + const commits: CommitIntent[] = []; + const retrievals: (string | null)[] = []; + let frontierReads = 0; + + const link: RemoteRunLink = { + *frontier(): Operation { + const snapshot = frontierOf(retained); + return { + workspaceRootId: snapshot.workspaceRootId, + journalEventId: snapshot.journalEventId, + events: snapshot.entries.map((entry) => entry.event), + }; + }, + // deno-lint-ignore require-yield + *frontierSnapshot(): Operation { + frontierReads += 1; + return frontierOf(retained); + }, + // deno-lint-ignore require-yield + *commit(intent: CommitIntent): Operation> { + commits.push(intent); + if (options.commit !== undefined) { + return options.commit(intent); + } + const ids = intent.events.map((_entry, index) => `event-${retained.length + index}`); + for (const [index, offered] of intent.events.entries()) { + retained.push({ eventId: ids[index] ?? "", event: offered }); + } + return Ok({ workspaceRootId: intent.expectedWorkspaceRootId, journalEventIds: ids }); + }, + // deno-lint-ignore require-yield + *replaceRetrieval( + _expected: string, + metadata: string | null, + ): Operation> { + retrievals.push(metadata); + return options.retrieval === undefined + ? Ok( + metadata === null + ? undefined + : { + metadata: JSON.parse(metadata) as Json, + revision: retrievals.filter((entry) => entry !== null).length, + updatedAt: "2026-09-04T00:00:01.000Z", + }, + ) + : options.retrieval(metadata); + }, + // deno-lint-ignore require-yield + *readExecutions(): Operation> { + return options.executions === undefined ? Ok([]) : options.executions(); + }, + }; + + return { + link, + commits, + retrievals, + retained, + get frontierReads(): number { + return frontierReads; + }, + /** An event the owner retained without this handle asking. */ + appendElsewhere(name: string): void { + retained.push({ eventId: `outside-${retained.length}`, event: event(name) }); + }, + }; +} + +function useDatabase(link: RemoteRunLink): Operation { + return useRemoteRunDatabase(link, frontierOf([])); +} + +function ok(result: Result): T { + if (!result.ok) { + throw result.error; + } + return result.value; +} + +describe("a run whose storage is somewhere else", () => { + it("initializes its snapshots from one frontier and does not refresh them", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + expect(database.record.runId).toBe(RUN_ID); + expect(database.retrieval).toBe(undefined); + + remote.appendElsewhere("written by somebody else"); + // A read consults the owner; the handle's own snapshots do not move. + expect(yield* database.journal.readAll()).toHaveLength(1); + expect(database.record.runId).toBe(RUN_ID); + expect(database.retrieval).toBe(undefined); + }); + }); + + it("reads a fresh journal every time and never serves a cache", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + expect(yield* database.journal.readAll()).toEqual([]); + remote.appendElsewhere("later"); + expect(yield* database.journal.readAll()).toHaveLength(1); + const entries = ok(yield* database.readJournalEntries()); + // The entry snapshot carries what the journal alone cannot: the owner's + // identity for the row and the root it was written against. + expect(entries[0]?.eventId).toBe("outside-0"); + expect(entries[0]?.workspaceRootId).toBe(ROOT); + expect(remote.frontierReads).toBe(3); + }); + }); + + it("appends through the one commit path, as a journal-only transaction", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + yield* database.journal.append(event("appended")); + expect(remote.commits).toHaveLength(1); + expect(remote.commits[0]?.publication).toBe(null); + expect(remote.commits[0]?.events).toHaveLength(1); + expect(yield* database.journal.readAll()).toHaveLength(1); + }); + }); + + it("shows a transaction its own writes, and commits them once", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + remote.appendElsewhere("already there"); + const outcome = ok( + yield* database.transact(function* (transaction) { + yield* transaction.journal.append(event("mine")); + // Read-your-writes: the admitted prefix, then this transaction's own. + const seen = yield* transaction.journal.readAll(); + expect(seen).toHaveLength(2); + return "body value"; + }), + ); + expect(outcome).toBe("body value"); + expect(remote.commits).toHaveLength(1); + }); + }); + + it("refuses a nested transaction and any same-handle operation inside a body", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + const refusals: unknown[] = []; + ok( + yield* database.transact(function* () { + const nested = yield* database.transact(function* () { + return "never"; + }); + refusals.push(nested.ok ? undefined : nested.error); + const read = yield* database.readJournalEntries(); + refusals.push(read.ok ? undefined : read.error); + try { + yield* database.journal.append(event("from inside")); + } catch (error) { + refusals.push(error); + } + return "done"; + }), + ); + expect(refusals).toHaveLength(3); + for (const refusal of refusals) { + expect(refusal).toBeInstanceOf(WorkflowTransactionError); + } + // None of them reached the owner. + expect(remote.commits).toHaveLength(1); + }); + }); + + it("lets unrelated work wait its turn rather than refusing it", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + const order: string[] = []; + const holding = yield* spawn(() => + database.transact(function* () { + order.push("transaction started"); + yield* sleep(5); + order.push("transaction finishing"); + return "held"; + }), + ); + yield* sleep(0); + // A different scope, not a descendant of the body. + const waiting = yield* spawn(function* () { + const entries = yield* database.readJournalEntries(); + order.push(entries.ok ? "read succeeded" : "read refused"); + }); + yield* holding; + yield* waiting; + expect(order).toEqual(["transaction started", "transaction finishing", "read succeeded"]); + }); + }); + + it("does not let another handle inherit this one's open transaction", function* () { + const remote = owner(); + yield* scoped(function* () { + const first = yield* useDatabase(remote.link); + const second = yield* useDatabase(remote.link); + const outcome = ok( + yield* first.transact(function* () { + // A transaction on one handle says nothing about another. + const other = yield* second.readJournalEntries(); + return other.ok ? "second read" : "second refused"; + }), + ); + expect(outcome).toBe("second read"); + }); + }); + + it("refuses every member once its scope has ended", function* () { + const remote = owner(); + let database: WorkflowRunDatabase | undefined; + yield* scoped(function* () { + database = yield* useDatabase(remote.link); + }); + if (database === undefined) { + throw new Error("expected a handle"); + } + const closed = database; + const results = [ + yield* closed.readJournalEntries(), + yield* closed.replaceRetrievalMetadata({ where: "later" }), + yield* closed.readDocumentExecutions(), + yield* closed.transact(function* () { + return "never"; + }), + ]; + for (const result of results) { + expect(result.ok).toBe(false); + if (!result.ok) { + expect(result.error).toBeInstanceOf(WorkflowDatabaseClosedError); + } + } + let raised: unknown; + try { + yield* closed.journal.append(event("after close")); + } catch (error) { + raised = error; + } + expect(raised).toBeInstanceOf(WorkflowDatabaseClosedError); + // Nothing reached the owner after the handle closed. + expect(remote.commits).toHaveLength(0); + expect(remote.retrievals).toHaveLength(0); + }); + + it("updates only the handle whose replacement succeeded", function* () { + const remote = owner(); + yield* scoped(function* () { + const first = yield* useDatabase(remote.link); + const second = yield* useDatabase(remote.link); + ok(yield* first.replaceRetrievalMetadata({ locator: "https://example.invalid/x.git" })); + expect(first.retrieval?.revision).toBe(1); + // Another handle's replacement is not this handle's snapshot. + expect(second.retrieval).toBe(undefined); + + // Two calls carrying identical metadata are two replacements. + ok(yield* first.replaceRetrievalMetadata({ locator: "https://example.invalid/x.git" })); + expect(first.retrieval?.revision).toBe(2); + + ok(yield* first.replaceRetrievalMetadata(undefined)); + expect(first.retrieval).toBe(undefined); + expect(remote.retrievals).toEqual([ + '{"locator":"https://example.invalid/x.git"}', + '{"locator":"https://example.invalid/x.git"}', + null, + ]); + }); + }); + + it("canonicalizes metadata before it is sent", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + ok(yield* database.replaceRetrievalMetadata({ b: 1, a: { d: 2, c: 3 } })); + // Sorted keys and no incidental whitespace, so two callers writing the + // same metadata write the same bytes. + expect(remote.retrievals[0]).toBe('{"a":{"c":3,"d":2},"b":1}'); + }); + }); + + it("leaves its snapshot alone when a replacement is refused", function* () { + const remote = owner({ + retrieval: () => Err(new WorkflowTransactionError("this run has moved")), + }); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + const refused = yield* database.replaceRetrievalMetadata({ locator: "x" }); + expect(refused.ok).toBe(false); + expect(database.retrieval).toBe(undefined); + }); + }); + + it("hands a Workspace route only to the exact database and transaction", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + const other = yield* useDatabase(remote.link); + let held: WorkflowRunTransaction | undefined; + ok( + yield* database.transact(function* (transaction) { + held = transaction; + expect(yield* activeWorkspaceRoute(database, transaction)).not.toBe(undefined); + // A foreign database, or a transaction object that is not this one. + expect(yield* activeWorkspaceRoute(other, transaction)).toBe(undefined); + expect(yield* activeWorkspaceRoute(database, { journal: transaction.journal })).toBe( + undefined, + ); + return "done"; + }), + ); + if (held === undefined) { + throw new Error("expected a transaction"); + } + // Outside the body the route is gone, so a retained object reaches nothing. + expect(yield* activeWorkspaceRoute(database, held)).toBe(undefined); + }); + }); + + it("sends no commit when the body fails, and answers with the refusal when the owner does", function* () { + const failing = owner({ commit: () => Err(new WorkflowTransactionError("owner refused")) }); + yield* scoped(function* () { + const database = yield* useDatabase(failing.link); + const refused = yield* database.transact(function* (transaction) { + yield* transaction.journal.append(event("attempted")); + return "never returned"; + }); + expect(refused.ok).toBe(false); + }); + + const raising = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(raising.link); + let caught: unknown; + try { + yield* database.transact(function* () { + throw new Error("the body failed"); + }); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(Error); + expect(raising.commits).toHaveLength(0); + }); + }); + + it("returns the executions the owner assembled, in order", function* () { + const executions: DocumentExecutionRecord[] = [ + { executionId: "one", startedAt: "2026-09-04T00:00:00.000Z" }, + { + executionId: "two", + startedAt: "2026-09-04T00:00:01.000Z", + stoppedAt: "2026-09-04T00:00:02.000Z", + stopStatus: "completed", + }, + ]; + const remote = owner({ executions: () => Ok(executions) }); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + expect(ok(yield* database.readDocumentExecutions())).toEqual(executions); + }); + }); +}); From 6473ff9dda020666f06b337ef9d9a5e44d8a406f Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 12:30:16 -0400 Subject: [PATCH 42/81] =?UTF-8?q?=F0=9F=A7=B7=20Prove=20the=20snapshot,=20?= =?UTF-8?q?and=20answer=20failures=20the=20way=20the=20interface=20says=20?= =?UTF-8?q?(#698)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three ways the remote database could accept or report something it should not. An execution page was believed on very little. The run identity was required to be present and never compared, so another run's retained history passed. Rows had only to increase, so a gap, a first row that was not the first, or a row past the anchor all passed — and a page could call itself terminal while stopping short of the anchor, which turned an incomplete history into a complete-looking answer. There is a state machine now: the first page fixes the snapshot, every later page continues from the exact cursor it was asked for, rows begin at the next sequence and stay adjacent and within the anchor, an empty snapshot is terminal and carries nothing, and terminal means exactly at the anchor. A record must be one of its two legal shapes, so a stop status without a stop, or a member the shape does not declare, is refused rather than ignored. The page's own byte bound is enforced on both sides, including a single row too large to page at all. The public boundary leaked in both directions. Members that return `Result` raised when the link raised; a body that failed propagated instead of returning the failed `Result` the same condition returns from the local host; and the publication link's own errors reached callers as themselves, so a private refusal class and its spelling crossed a provider-neutral interface. Each is translated once now, at the adapter, and the storage distinctions are kept: a store belonging to something else is not damage, a version this build does not implement is not damage, and a record it cannot read is not an unreachable owner. Those are different facts and a host acts on them differently. Retrieval accepted any well-formed answer. A performed reply naming different metadata would have installed where the definition is fetched from — so the answer is now required to be canonically the value that was asked for, and a contradiction fails closed with the snapshot untouched. Input is parsed by the rules a stored value is held to and bounded before anything is sent, because a value this build will not keep is not a request to make. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 156 +++++++++++++++--- .../workflow/src/cloudflare/owner-reads.ts | 6 +- packages/workflow/src/remote/database.ts | 121 ++++++++++++-- packages/workflow/src/remote/records.ts | 22 ++- .../workflow/tests/remote-database.test.ts | 52 +++++- packages/workflow/tests/remote-read.test.ts | 115 +++++++++++++ 6 files changed, 417 insertions(+), 55 deletions(-) diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 6fbacb65f..b942f8c2d 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -55,10 +55,18 @@ import { type WorkspaceRootManifest, } from "../workspace/root-manifest.ts"; import { decodeContentManifest } from "../workspace/content-manifest.ts"; -import { EXECUTION_PAGE_ENTRIES, JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES } from "./commands.ts"; +import { + EXECUTION_PAGE_BYTES, + EXECUTION_PAGE_ENTRIES, + JOURNAL_PAGE_ENTRIES, + MAX_CONTENT_BYTES, +} from "./commands.ts"; import type { RemoteRunLink } from "../remote/database.ts"; +import { SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; import { WorkflowDatabaseCorruptError, + WorkflowDatabaseFormatError, + WorkflowSchemaVersionError, WorkflowRecordMalformedError, WorkflowRequestError, WorkflowStorageError, @@ -589,14 +597,40 @@ export function cloudflareRunLink( connection: OwnerConnection, reads: RemoteReadLink, nextId: () => string, + expectedRunId: string, ): RemoteRunLink { const publication = cloudflareOwnerLink(connection, reads, nextId); return { - frontier: publication.frontier, - commit: publication.commit, + /** + * Both halves of the publication link, translated. + * + * The database returns these failures through a provider-neutral interface, + * so a private refusal or a transport error must not travel as itself. This + * is the one place that translation happens. + */ + *frontier(): Operation { + try { + return yield* publication.frontier(); + } catch (error) { + throw translate(error); + } + }, + + *commit(intent: CommitIntent): Operation> { + try { + const committed = yield* publication.commit(intent); + return committed.ok ? committed : Err(translate(committed.error)); + } catch (error) { + return Err(translate(error)); + } + }, *frontierSnapshot(): Operation { - return yield* reads.frontier(); + try { + return yield* reads.frontier(); + } catch (error) { + throw translate(error); + } }, *replaceRetrieval( @@ -638,30 +672,58 @@ export function cloudflareRunLink( *readExecutions(): Operation> { try { - const found: { sequence: number; record: DocumentExecutionRecord }[] = []; + const found: DocumentExecutionRecord[] = []; let anchor: number | null | undefined; let after: number | null = null; let done = false; while (!done) { - const page: ExecutionPage = yield* askPage(connection, nextId(), anchor ?? null, after); - // The first page chooses the anchor; every later one is held to it. + const page: ExecutionPage = yield* askPage( + connection, + nextId(), + expectedRunId, + anchor ?? null, + after, + ); + // The first page chooses the snapshot. Every later one is held to it, + // and to the cursor it was asked to continue from. const expected = anchor === undefined ? page.anchor : anchor; anchor = expected; if (page.anchor !== expected || page.after !== after) { - return Err( - new WorkflowRecordMalformedError( - "document executions", - "a page did not continue its anchored snapshot", - ), - ); + return Err(pageFailure("a page did not continue its anchored snapshot")); + } + if (page.anchor === null) { + // An empty snapshot is terminal and carries nothing. + if (page.rows.length > 0 || !page.done || after !== null) { + return Err(pageFailure("an empty snapshot carried rows or did not terminate")); + } + break; + } + if (page.rows.length === 0) { + // A page with nothing in it can only be the empty snapshot, which + // was handled above. Otherwise the read would never advance. + return Err(pageFailure("a page of an anchored snapshot carried no rows")); } + let previous: number = after ?? 0; for (const row of page.rows) { - found.push(row); + if (row.sequence !== previous + 1) { + // Exactly adjacent: a gap would be retained history omitted from + // a snapshot that claims to be complete. + return Err(pageFailure("a page skipped, repeated or reordered a row")); + } + if (row.sequence > page.anchor) { + return Err(pageFailure("a page carried a row outside its snapshot")); + } + previous = row.sequence; + found.push(row.record); + } + if (page.done !== (previous === page.anchor)) { + // Terminal exactly at the anchor, and only there. + return Err(pageFailure("a page disagreed with its terminal row")); } - after = page.rows.at(-1)?.sequence ?? after; + after = previous; done = page.done; } - return Ok(found.map((row) => row.record)); + return Ok(found); } catch (error) { return Err(translate(error)); } @@ -669,6 +731,11 @@ export function cloudflareRunLink( }; } +/** What a page that does not describe the snapshot it claims becomes. */ +function pageFailure(reason: string): WorkflowStorageError { + return new WorkflowRecordMalformedError("document executions", reason); +} + /** One execution page, with the private ordering the runner checks adjacency by. */ interface ExecutionPage { readonly anchor: number | null; @@ -680,6 +747,7 @@ interface ExecutionPage { function* askPage( connection: OwnerConnection, id: string, + expectedRunId: string, anchor: number | null, after: number | null, ): Operation { @@ -688,24 +756,28 @@ function* askPage( { command: "executions", anchor, after }, (value): ExecutionPage => { const found = members(value, ["runId", "anchor", "after", "rows", "done"]); + if (found.get("runId") !== expectedRunId) { + // Another run's retained history is not this run's, however well formed. + return fail("an execution page named another run"); + } const offered = found.get("rows"); if (!Array.isArray(offered) || offered.length > EXECUTION_PAGE_ENTRIES) { return fail("an execution page was not one bounded page"); } + if (new TextEncoder().encode(JSON.stringify(offered)).length > EXECUTION_PAGE_BYTES) { + // The page bound, not the message envelope. A page that ignored it + // would make the number of requests depend on how large one row is. + return fail("an execution page carried more than one page of rows"); + } if (typeof found.get("done") !== "boolean") { return fail("an execution page did not say whether it was terminal"); } - let previous = after; const rows = offered.map((entry) => { const item = members(entry, ["sequence", "record"]); const sequence = item.get("sequence"); if (typeof sequence !== "number" || !Number.isSafeInteger(sequence) || sequence < 1) { return fail("an execution row did not carry a position"); } - if (previous !== null && sequence <= previous) { - return fail("an execution page repeated or reordered a row"); - } - previous = sequence; return { sequence, record: parseRemoteExecution(item.get("record")) }; }); return { @@ -741,8 +813,18 @@ function nullableSequence(value: unknown): number | null { * line: they describe a protocol nobody above here is party to. */ function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { - if (refusal.startsWith("storage:")) { - return new WorkflowDatabaseCorruptError("the workflow run's remote storage", refusal.slice(8)); + // A host acts on these differently: storage belonging to something else may + // not be written, a version this build does not implement may not be + // migrated, and damage may not be repaired. Collapsing them would make all + // three look like the one that says "restore from a backup". + if (refusal === "storage:foreign") { + return new WorkflowDatabaseFormatError(REMOTE_STORE, "it belongs to something else"); + } + if (refusal === "storage:unsupported-version") { + return new WorkflowSchemaVersionError(REMOTE_STORE, 0, SCHEMA_VERSION); + } + if (refusal === "storage:corrupt") { + return new WorkflowDatabaseCorruptError(REMOTE_STORE, "its retained records do not agree"); } if (refusal === "command:stale-root" || refusal === "command:stale-journal") { return new WorkflowTransactionError( @@ -755,7 +837,22 @@ function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { return new WorkflowTransactionError("this run's owner refused the operation."); } -/** Any failure from the private protocol, as a provider-neutral one. */ +/** + * What a public error names instead of a path. + * + * A remote run has no file, and naming one would be an invitation to look for + * it. The store is named as what it is. + */ +const REMOTE_STORE = "this run's remote storage"; + +/** + * Any failure from the private protocol, as a provider-neutral one. + * + * Nothing private crosses: not a refusal class, not a refusal spelling, not the + * message a parser wrote about a value it refused. A record this build cannot + * read is a malformed record rather than an unreachable owner, because those + * are different facts and a caller acts on them differently. + */ function translate(error: unknown): WorkflowStorageError { if (error instanceof CloudflareOwnerRefusalError) { return storageFailure(error.refusal); @@ -763,5 +860,14 @@ function translate(error: unknown): WorkflowStorageError { if (error instanceof WorkflowStorageError) { return error; } - return new WorkflowTransactionError("this run's owner could not be reached."); + if (error instanceof RemoteRecordError) { + return new WorkflowRecordMalformedError( + "record this run's owner returned", + "it is not a record this build can read", + ); + } + if (error instanceof OwnerLinkError) { + return new WorkflowTransactionError("this run's owner could not be reached."); + } + return new WorkflowTransactionError("this run's owner could not answer the operation."); } diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index 5d5c16a79..c7ba43876 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -532,7 +532,11 @@ export function readExecutions( if (page.length > 0 && encoded + bytes > EXECUTION_PAGE_BYTES) { break; } - if (bytes > MAX_CONTENT_BYTES) { + if (bytes > EXECUTION_PAGE_BYTES) { + // One row larger than a whole page. The bound is the page's, not the + // message envelope's: a row that could never fit means this snapshot + // cannot be paged at all, and answering with it anyway would send + // something the runner is required to refuse. throw new CommandError("too-large"); } page.push({ sequence: at, record: row }); diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 248bc25b5..5bd7da037 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -35,7 +35,14 @@ import { } from "effection"; import type { DurableEvent, DurableStream, Json } from "@executablemd/durable-streams"; import type { JournalEntry, WorkflowRunDatabase, WorkflowRunTransaction } from "../storage/api.ts"; -import { WorkflowDatabaseClosedError, WorkflowTransactionError } from "../storage/errors.ts"; +import { + WorkflowDatabaseClosedError, + WorkflowRecordMalformedError, + WorkflowRequestError, + WorkflowStorageError, + WorkflowTransactionError, +} from "../storage/errors.ts"; +import { parseJsonValue } from "../storage/members.ts"; import type { DefinitionRetrieval, DocumentExecutionRecord, @@ -124,6 +131,18 @@ export function* activeWorkspaceRoute( return route; } +/** + * A failure this interface can return, whatever it arrived as. + * + * The adapter beneath has already translated what it knows about; anything else + * reaching here is the body's own error, which is carried as it is. A value + * that is not an error at all becomes one rather than travelling as a thrown + * string nobody can act on. + */ +function failure(error: unknown): Error { + return error instanceof Error ? error : new WorkflowTransactionError(String(error)); +} + /** * What a `DurableStream` member does with a result. * @@ -195,13 +214,25 @@ export function useRemoteRunDatabase( return Ok(); } - /** One turn at the handle, for an ordinary operation. */ + /** + * One turn at the handle, for an ordinary operation. + * + * A member that returns `Result` answers with the failure rather than + * raising it, so a link that raised is caught here. Cancellation is not a + * failure and is left to unwind as control flow. + */ function* turn(body: () => Operation>): Operation> { const admitted = yield* admit(); if (!admitted.ok) { return admitted; } - return yield* turns.take(body); + return yield* turns.take(function* (): Operation> { + try { + return yield* body(); + } catch (error) { + return Err(failure(error)); + } + }); } const ordinary: DurableStream = { @@ -242,17 +273,25 @@ export function useRemoteRunDatabase( ); } return yield* turns.take(function* (): Operation> { - return yield* transactRemotely(link, gate, function* (transaction, enlist) { - // The marker and the route are installed for the body's scope alone. - // Outside it neither exists, so a retained transaction object reaches - // nothing and an unrelated scope is not mistaken for a nested one. - yield* ActiveTransaction.set({ - handle, - enclosing: yield* ActiveTransaction.get(), + try { + return yield* transactRemotely(link, gate, function* (transaction, enlist) { + // The marker and the route are installed for the body's scope + // alone. Outside it neither exists, so a retained transaction + // object reaches nothing and an unrelated scope is not mistaken for + // a nested one. + yield* ActiveTransaction.set({ + handle, + enclosing: yield* ActiveTransaction.get(), + }); + yield* ActiveRoute.set({ database: handle, transaction, enlist }); + return yield* body(transaction); }); - yield* ActiveRoute.set({ database: handle, transaction, enlist }); - return yield* body(transaction); - }); + } catch (error) { + // A body that raised, or a resource of its that failed to tear down, + // is a failed transaction rather than a raised one: the interface + // answers with a `Result`, and nothing was committed. + return Err(failure(error)); + } }); } @@ -279,20 +318,48 @@ export function useRemoteRunDatabase( }, *replaceRetrievalMetadata(metadata: Json | undefined): Operation> { + let encoded: string | null; + try { + // Parsed by the same rules a stored value is held to, then encoded + // canonically. A value that is not JSON at all never becomes a + // request: refusing it here is what "no request" means. + encoded = + metadata === undefined + ? null + : canonical(parseJsonValue(metadata, "$", retrievalFailure)); + } catch (error) { + return Err(failure(error)); + } + if (encoded !== null && new TextEncoder().encode(encoded).length > MAX_RETRIEVAL_BYTES) { + return Err( + new WorkflowRequestError( + "this retrieval metadata is larger than one message may carry, so it was not sent.", + ), + ); + } + const replaced = yield* turn(function* () { const snapshot = yield* link.frontierSnapshot(); - return yield* link.replaceRetrieval( - snapshot.workspaceRootId, - metadata === undefined ? null : canonical(metadata), - ); + return yield* link.replaceRetrieval(snapshot.workspaceRootId, encoded); }); if (!replaced.ok) { return replaced; } + // The answer has to describe the replacement that was asked for. An + // owner that returned different metadata would otherwise install the + // location a later fetch of the definition would use. + const answered = replaced.value; + if (encoded === null) { + if (answered !== undefined) { + return Err(contradiction()); + } + } else if (answered === undefined || canonical(answered.metadata) !== encoded) { + return Err(contradiction()); + } // Only this handle, and only after its own successful replacement. The // owner's revision and time are what is recorded; nothing is invented // here. - retrieval = replaced.value; + retrieval = answered; return Ok(); }, @@ -308,6 +375,24 @@ export function useRemoteRunDatabase( }); } +/** The most bytes one canonical retrieval value may carry. */ +const MAX_RETRIEVAL_BYTES = 8 * 1024 * 1024; + +/** How a malformed retrieval value is reported, before anything is sent. */ +function retrievalFailure(reason: string, path: string): Error { + return new WorkflowRequestError( + `this retrieval metadata is not a JSON value storage can keep at ${path}: ${reason}.`, + ); +} + +/** An answer that does not describe the replacement it answered. */ +function contradiction(): WorkflowStorageError { + return new WorkflowRecordMalformedError( + "retrieval this run's owner returned", + "it does not describe the replacement that was asked for", + ); +} + /** * The canonical encoding of one retrieval metadata value. * diff --git a/packages/workflow/src/remote/records.ts b/packages/workflow/src/remote/records.ts index ab58b50ad..7897720d3 100644 --- a/packages/workflow/src/remote/records.ts +++ b/packages/workflow/src/remote/records.ts @@ -153,6 +153,22 @@ export function parseRemoteJournalEntry(value: unknown): JournalEntry { */ export function parseRemoteExecution(value: unknown): DocumentExecutionRecord { const found = parseMembers(value, "$", fail); + // One of exactly two legal shapes. A record carrying a stop status without + // having stopped, or an undeclared member, is a shape this build does not + // understand — and reading it leniently would make a history that means one + // thing here and another where it was written. + const active = ["executionId", "startedAt"]; + const stopped = [...active, "stoppedAt", "stopStatus"]; + const declared = found.has("stoppedAt") + ? found.has("stopReason") + ? [...stopped, "stopReason"] + : stopped + : active; + requireMemberNames(found, declared, "$", fail); + if (found.size !== declared.length) { + throw fail("expected exactly the members this shape declares", "$"); + } + const executionId = parseStringMember(found, "executionId", "$", fail); if (executionId === "") { throw fail("expected a non-empty identity", "$.executionId"); @@ -164,16 +180,16 @@ export function parseRemoteExecution(value: unknown): DocumentExecutionRecord { if (!found.has("stoppedAt")) { return Object.freeze(record); } - const stopped: DocumentExecutionRecord = { + const halted: DocumentExecutionRecord = { ...record, stoppedAt: instant(found.get("stoppedAt"), "$.stoppedAt"), stopStatus: parseWorkflowRunStatus(found.get("stopStatus"), "$.stopStatus", fail), }; if (!found.has("stopReason")) { - return Object.freeze(stopped); + return Object.freeze(halted); } return Object.freeze({ - ...stopped, + ...halted, stopReason: parseWorkflowStopReason(found.get("stopReason"), "$.stopReason", fail), }); } diff --git a/packages/workflow/tests/remote-database.test.ts b/packages/workflow/tests/remote-database.test.ts index 2b882055c..18bcef0bf 100644 --- a/packages/workflow/tests/remote-database.test.ts +++ b/packages/workflow/tests/remote-database.test.ts @@ -358,6 +358,38 @@ describe("a run whose storage is somewhere else", () => { }); }); + it("refuses metadata that is not a JSON value, without asking the owner", function* () { + const remote = owner(); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + const offered = { locator: () => "not json" } as unknown as Json; + const refused = yield* database.replaceRetrievalMetadata(offered); + expect(refused.ok).toBe(false); + // Nothing was sent: an inadmissible value is not a request. + expect(remote.retrievals).toHaveLength(0); + expect(database.retrieval).toBe(undefined); + }); + }); + + it("fails closed when the answer describes another replacement", function* () { + const remote = owner({ + retrieval: () => + Ok({ + metadata: { locator: "something else entirely" }, + revision: 1, + updatedAt: "2026-09-04T00:00:01.000Z", + }), + }); + yield* scoped(function* () { + const database = yield* useDatabase(remote.link); + const refused = yield* database.replaceRetrievalMetadata({ locator: "what was asked" }); + expect(refused.ok).toBe(false); + // The snapshot is what it was: an answer about another value installs + // nothing, because it would change where the definition is fetched from. + expect(database.retrieval).toBe(undefined); + }); + }); + it("leaves its snapshot alone when a replacement is refused", function* () { const remote = owner({ retrieval: () => Err(new WorkflowTransactionError("this run has moved")), @@ -410,16 +442,20 @@ describe("a run whose storage is somewhere else", () => { const raising = owner(); yield* scoped(function* () { const database = yield* useDatabase(raising.link); - let caught: unknown; - try { - yield* database.transact(function* () { - throw new Error("the body failed"); - }); - } catch (error) { - caught = error; + // A body that raised is a failed transaction, not a raised one: the + // interface answers with a `Result`, and the same condition returns + // `Err` from the local provider. + const failed = yield* database.transact(function* (transaction) { + yield* transaction.journal.append(event("attempted")); + throw new Error("the body failed"); + }); + expect(failed.ok).toBe(false); + if (!failed.ok) { + expect(String(failed.error)).toContain("the body failed"); } - expect(caught).toBeInstanceOf(Error); expect(raising.commits).toHaveLength(0); + // And the handle is still usable afterwards. + expect(ok(yield* database.readJournalEntries())).toEqual([]); }); }); diff --git a/packages/workflow/tests/remote-read.test.ts b/packages/workflow/tests/remote-read.test.ts index e0b0f4fbe..ab66205e5 100644 --- a/packages/workflow/tests/remote-read.test.ts +++ b/packages/workflow/tests/remote-read.test.ts @@ -20,8 +20,10 @@ import { scoped } from "effection"; import { cloudflareOwnerLink, cloudflareReadLink, + cloudflareRunLink, stageCloudflareContent, } from "../src/cloudflare/client.ts"; +import { WorkflowStorageError } from "../src/storage/errors.ts"; import { createTransactionGate, transactRemotely } from "../src/remote/collector.ts"; import { encodeBase64 } from "../src/cloudflare/encoding.ts"; import type { OwnerSocket, SocketListener } from "../src/remote/client.ts"; @@ -431,4 +433,117 @@ describe("semantic reads from a Cloudflare owner", () => { // rather than a success nothing performed. expect(committed).toMatchObject({ ok: false }); }); + it("assembles an execution snapshot only from pages that describe it", function* () { + const record = (id: string) => ({ executionId: id, startedAt: "2026-09-04T00:00:00.000Z" }); + const page = (rows: unknown[], overrides: Record = {}) => ({ + outcome: "performed", + value: { runId: RUN_ID, anchor: 2, after: null, rows, done: true, ...overrides }, + }); + const row = (sequence: number, id: string) => ({ sequence, record: record(id) }); + + // Each of these is a page that does not describe the snapshot it claims. + // The structural consequence is one: no partial history is returned. + const refused: Record = { + "another run's history": page([row(1, "a"), row(2, "b")], { runId: "somebody-else" }), + "a terminal page short of its anchor": page([row(1, "a")]), + "a first row that is not the first": page([row(2, "b")]), + "a gap between rows": page([row(1, "a"), row(3, "c")]), + "a repeated row": page([row(1, "a"), row(1, "a")]), + "a row beyond the anchor": page([row(1, "a"), row(2, "b"), row(3, "c")]), + "an empty page of a non-empty snapshot": page([], { done: false }), + "an empty snapshot that carries rows": page([row(1, "a")], { anchor: null }), + "a cursor it was not asked to continue from": page([row(1, "a"), row(2, "b")], { after: 7 }), + "a record with a member the shape does not declare": page([ + { sequence: 1, record: { ...record("a"), note: "extra" } }, + ]), + "a record that stopped without saying how": page([ + { sequence: 1, record: { ...record("a"), stopStatus: "completed" } }, + ]), + }; + + for (const [description, answer] of Object.entries(refused)) { + const transport = wire(() => answer as Record); + let outcome: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids(), RUN_ID), + ids(), + RUN_ID, + ); + outcome = yield* link.readExecutions(); + }); + expect([description, (outcome as { ok: boolean }).ok]).toEqual([description, false]); + if (!(outcome as { ok: boolean }).ok) { + const failed = outcome as { error: Error }; + // A provider-neutral failure, with nothing private in it. + expect([description, failed.error]).toEqual([ + description, + expect.any(WorkflowStorageError), + ]); + expect(String(failed.error)).not.toContain("command:"); + } + } + }); + + it("assembles an honest snapshot across pages, and an empty one", function* () { + const record = (id: string) => ({ executionId: id, startedAt: "2026-09-04T00:00:00.000Z" }); + const pages: Record> = { + null: { + outcome: "performed", + value: { + runId: RUN_ID, + anchor: 2, + after: null, + rows: [{ sequence: 1, record: record("first") }], + done: false, + }, + }, + "1": { + outcome: "performed", + value: { + runId: RUN_ID, + anchor: 2, + after: 1, + rows: [{ sequence: 2, record: record("second") }], + done: true, + }, + }, + }; + const transport = wire( + (request) => + pages[String(request["after"])] ?? { outcome: "refused", refusal: "storage:corrupt" }, + ); + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids(), RUN_ID), + ids(), + RUN_ID, + ); + const read = yield* link.readExecutions(); + expect(read.ok).toBe(true); + if (read.ok) { + expect(read.value.map((entry) => entry.executionId)).toEqual(["first", "second"]); + } + }); + + const empty = wire(() => ({ + outcome: "performed", + value: { runId: RUN_ID, anchor: null, after: null, rows: [], done: true }, + })); + yield* scoped(function* () { + const connection = yield* useOwnerConnection(empty.socket); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids(), RUN_ID), + ids(), + RUN_ID, + ); + const read = yield* link.readExecutions(); + expect(read.ok && read.value).toEqual([]); + }); + }); }); From e61793cacb8febac6d1cb64e49905e415b3140e0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 12:52:53 -0400 Subject: [PATCH 43/81] =?UTF-8?q?=F0=9F=90=9B=20Send=20the=20record=20the?= =?UTF-8?q?=20client=20parses,=20and=20carry=20the=20version=20the=20owner?= =?UTF-8?q?=20read?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner validated each execution row and then sent the physical row anyway, so the composed owner-to-client path could not return a single execution. Both halves passed against a hand-built counterpart, which is exactly how the disagreement survived; the new workerd regression composes the real pages through the real client and fails against the previous commit. The page bound is now one function both ends call, measuring the exact `rows` member as it crosses. The unsupported-version refusal carries the version the owner actually read instead of a placeholder. A retrieval answer describing another replacement is settled where the answer arrives, closing the channel. The message bound is enforced against the complete serialized request, correlation id included, before it is registered or sent. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 28 ++++- packages/workflow/src/cloudflare/commands.ts | 17 ++- .../workflow/src/cloudflare/owner-reads.ts | 34 +++--- packages/workflow/src/cloudflare/owner.ts | 7 ++ packages/workflow/src/remote/client.ts | 18 ++- packages/workflow/src/remote/database.ts | 19 ++- .../tests/cloudflare/remote-owner.vitest.ts | 112 +++++++++++++++++- .../cloudflare/support/executor-object.ts | 12 ++ packages/workflow/tests/remote-client.test.ts | 37 ++++++ packages/workflow/tests/remote-read.test.ts | 58 ++++++++- 10 files changed, 303 insertions(+), 39 deletions(-) diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index b942f8c2d..00d62e903 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -58,11 +58,13 @@ import { decodeContentManifest } from "../workspace/content-manifest.ts"; import { EXECUTION_PAGE_BYTES, EXECUTION_PAGE_ENTRIES, + executionPageBytes, JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES, } from "./commands.ts"; import type { RemoteRunLink } from "../remote/database.ts"; import { SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; +import { canonicalJson } from "../storage/record.ts"; import { WorkflowDatabaseCorruptError, WorkflowDatabaseFormatError, @@ -92,7 +94,7 @@ export type PrivateRefusal = | "command:stale-journal" | "command:mapping-conflict" | "storage:foreign" - | "storage:unsupported-version" + | `storage:unsupported-version-v${number}` | "storage:corrupt"; export class CloudflareOwnerRefusalError extends Error { @@ -168,11 +170,17 @@ function privateRefusal(value: string): PrivateRefusal { case "command:stale-journal": case "command:mapping-conflict": case "storage:foreign": - case "storage:unsupported-version": case "storage:corrupt": return value; - default: + default: { + // The one category that carries a value: the schema version the owner + // actually read, bounded and parsed rather than guessed. + const unsupported = /^storage:unsupported-version-v(\d{1,6})$/.exec(value); + if (unsupported !== null) { + return `storage:unsupported-version-v${Number(unsupported[1])}`; + } return fail("it named an unknown refusal category"); + } } } @@ -658,6 +666,13 @@ export function cloudflareRunLink( if (parsed === undefined || metadata === null) { return fail("a retrieval answer disagreed with the replacement it answered"); } + // Compared here, where the answer arrives. An owner that performed + // a different replacement than the one asked for is a channel the + // two sides disagree on, so it fails closed rather than handing + // back a value the caller would have to notice was wrong. + if (canonicalJson(parsed.metadata) !== metadata) { + return fail("a retrieval answer named metadata the request did not ask for"); + } return parsed; }, privateRefusal, @@ -764,7 +779,7 @@ function* askPage( if (!Array.isArray(offered) || offered.length > EXECUTION_PAGE_ENTRIES) { return fail("an execution page was not one bounded page"); } - if (new TextEncoder().encode(JSON.stringify(offered)).length > EXECUTION_PAGE_BYTES) { + if (executionPageBytes(offered) > EXECUTION_PAGE_BYTES) { // The page bound, not the message envelope. A page that ignored it // would make the number of requests depend on how large one row is. return fail("an execution page carried more than one page of rows"); @@ -820,8 +835,9 @@ function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { if (refusal === "storage:foreign") { return new WorkflowDatabaseFormatError(REMOTE_STORE, "it belongs to something else"); } - if (refusal === "storage:unsupported-version") { - return new WorkflowSchemaVersionError(REMOTE_STORE, 0, SCHEMA_VERSION); + const unsupported = /^storage:unsupported-version-v(\d{1,6})$/.exec(refusal); + if (unsupported !== null) { + return new WorkflowSchemaVersionError(REMOTE_STORE, Number(unsupported[1]), SCHEMA_VERSION); } if (refusal === "storage:corrupt") { return new WorkflowDatabaseCorruptError(REMOTE_STORE, "its retained records do not agree"); diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 55b2442f9..6fa306611 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -12,8 +12,10 @@ import { type WorktreeRecord, } from "../composition/records.ts"; import { type AgentSessionRecord, parseAgentSessionRecord } from "../storage/agent-session.ts"; +import { MAX_MESSAGE_BYTES } from "../remote/client.ts"; + +export { MAX_MESSAGE_BYTES }; -export const MAX_MESSAGE_BYTES = 8 * 1024 * 1024; export const MAX_CONTENT_BYTES = 1024 * 1024; export const MAX_STAGED_BYTES = 2 * 1024 * 1024; export const MAX_COMMANDS = 256; @@ -24,6 +26,19 @@ export const JOURNAL_PAGE_BYTES = 512 * 1024; export const EXECUTION_PAGE_ENTRIES = 128; /** The most serialized bytes of retained execution rows one page carries. */ export const EXECUTION_PAGE_BYTES = 512 * 1024; + +/** + * How both ends measure one execution page. + * + * One function rather than two similar sums: the owner decides what fits and + * the runner checks it, and if they measured different things an honest page + * near the bound would be sent by one and refused by the other. What is + * measured is the exact `rows` member as it crosses, wrappers and punctuation + * included, because that is what the bound is about. + */ +export function executionPageBytes(rows: readonly unknown[]): number { + return new TextEncoder().encode(JSON.stringify(rows)).length; +} /** The most content identities one proposal may name. */ export const MAX_PROPOSED_PIECES = 8192; /** The most retained mapping changes one proposal may carry. */ diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index c7ba43876..a76580432 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -27,6 +27,7 @@ import { parseDurableEvent } from "@executablemd/durable-streams"; import { readDocumentExecution, readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; +import type { DocumentExecutionRecord } from "../storage/record.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { parseWorkspaceRootManifest, @@ -39,6 +40,7 @@ import { CommandError, EXECUTION_PAGE_BYTES, EXECUTION_PAGE_ENTRIES, + executionPageBytes, JOURNAL_PAGE_BYTES, JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES, @@ -474,7 +476,7 @@ export interface ExecutionsValue { readonly runId: string; readonly anchor: number | null; readonly after: number | null; - readonly rows: readonly { readonly sequence: number; readonly record: Row }[]; + readonly rows: readonly { readonly sequence: number; readonly record: DocumentExecutionRecord }[]; readonly done: boolean; } @@ -520,27 +522,23 @@ export function readExecutions( EXECUTION_PAGE_ENTRIES + 1, ); - const page: { sequence: number; record: Row }[] = []; - let encoded = 0; + const page: { sequence: number; record: DocumentExecutionRecord }[] = []; for (const row of found.slice(0, EXECUTION_PAGE_ENTRIES)) { const at = safeInteger(row["sequence"], "execution sequence"); - // Parsed here as well as on the runner: a row this owner cannot read is - // storage damage, and sending it would make the runner report damage it - // cannot attribute. - readDocumentExecution(row); - const bytes = new TextEncoder().encode(JSON.stringify(row)).length; - if (page.length > 0 && encoded + bytes > EXECUTION_PAGE_BYTES) { + // The semantic record is what crosses, not the physical row. A row this + // owner cannot read is storage damage; sending its columns would make the + // runner responsible for a shape it has no business knowing. + const entry = { sequence: at, record: readDocumentExecution(row) }; + const grown = [...page, entry]; + if (executionPageBytes(grown) > EXECUTION_PAGE_BYTES) { + if (page.length === 0) { + // One record larger than a whole page: this snapshot cannot be paged, + // and answering with it would send what the runner must refuse. + throw new CommandError("too-large"); + } break; } - if (bytes > EXECUTION_PAGE_BYTES) { - // One row larger than a whole page. The bound is the page's, not the - // message envelope's: a row that could never fit means this snapshot - // cannot be paged at all, and answering with it anyway would send - // something the runner is required to refuse. - throw new CommandError("too-large"); - } - page.push({ sequence: at, record: row }); - encoded += bytes; + page.push(entry); } const done = found.length <= page.length; diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index c1dbf96d4..41d985f55 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -91,6 +91,13 @@ export function refusalOf(error: unknown): string { return `command:${error.refusal}`; } if (error instanceof WorkflowObjectStorageError) { + if (error.failure.kind === "unsupported-version") { + // The version travels in the category rather than beside it, because the + // answer envelope carries a refusal and nothing else. It is the one fact + // a host needs to decide whether this build may open the store, and a + // public error that guessed it would state something untrue. + return `storage:unsupported-version-v${error.failure.schemaVersion}`; + } return `storage:${error.failure.kind}`; } if (error instanceof WorkflowRecordMalformedError) { diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index 93aa599b0..ef4af4006 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -24,6 +24,15 @@ import { ensure, type Operation, resource, withResolvers } from "effection"; +/** + * The most bytes one message on this connection may carry. + * + * The bound is the whole message as it crosses, in both directions. Measuring + * one member of a request instead would let a request that clears the check + * still be too large once its correlation and framing are added. + */ +export const MAX_MESSAGE_BYTES = 8 * 1024 * 1024; + /** Why the connection itself could not carry a request. */ export type LinkRefusal = | "closed" @@ -309,6 +318,13 @@ export function useOwnerConnection(socket: OwnerSocket): Operation MAX_MESSAGE_BYTES) { + throw new OwnerLinkError("too-large"); + } const settle = withResolvers>(); waiting.set(id, { deliver(answer: RawAnswer): boolean { @@ -336,7 +352,7 @@ export function useOwnerConnection(socket: OwnerSocket): Operation MAX_RETRIEVAL_BYTES) { - return Err( - new WorkflowRequestError( - "this retrieval metadata is larger than one message may carry, so it was not sent.", - ), - ); - } - const replaced = yield* turn(function* () { const snapshot = yield* link.frontierSnapshot(); return yield* link.replaceRetrieval(snapshot.workspaceRootId, encoded); @@ -375,9 +375,6 @@ export function useRemoteRunDatabase( }); } -/** The most bytes one canonical retrieval value may carry. */ -const MAX_RETRIEVAL_BYTES = 8 * 1024 * 1024; - /** How a malformed retrieval value is reported, before anything is sent. */ function retrievalFailure(reason: string, path: string): Error { return new WorkflowRequestError( diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index 395ff462f..73c03f795 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -32,8 +32,16 @@ import { VALID_CLAIMS, } from "./support/executor-object.ts"; import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; +import { call, run } from "effection"; +import { + type OwnerSocket, + type SocketListener, + useOwnerConnection, +} from "../../src/remote/client.ts"; +import { cloudflareReadLink, cloudflareRunLink } from "../../src/cloudflare/client.ts"; let unique = 0; +const NEW_START = "2026-02-02T00:00:00.000Z"; const NOW = 1_800_000_000; let keys: TestKeys; @@ -118,6 +126,32 @@ function askFrame( }); } +/** + * The platform socket, as the runner's client needs it. + * + * A host binds its own socket to this interface; the runtime's event types are + * wider than the four members the client uses, so the binding is written out + * rather than asserted. + */ +function ownerSocket(socket: WebSocket): OwnerSocket { + const listeners = new Map(); + return { + send: (data) => socket.send(data), + close: () => socket.close(), + addEventListener(type, listener) { + const bound: EventListener = (event) => listener(event as { data?: unknown }); + listeners.set(listener, bound); + socket.addEventListener(type, bound); + }, + removeEventListener(type, listener) { + const bound = listeners.get(listener); + if (bound !== undefined) { + socket.removeEventListener(type, bound); + } + }, + }; +} + function record(value: unknown): Record { if (value === null || typeof value !== "object" || Array.isArray(value)) { throw new Error("expected an object answer"); @@ -157,7 +191,7 @@ describe("the remote owner protocol", () => { owner.initialize(); owner.rewriteMarker(0x584d4431, 2); }, - "storage:unsupported-version", + "storage:unsupported-version-v2", ], [ "damaged", @@ -268,6 +302,82 @@ describe("the remote owner protocol", () => { expect(second["done"]).toBe(true); }); + it("reads a whole retained history back through the runner's own client", async () => { + // The one test where the owner's answers and the runner's parser meet. Each + // half was already proven against a hand-built counterpart, which is + // exactly why a disagreement between them could survive: the owner may + // answer a shape no runner accepts and both halves still pass. This + // composes the real pages through the real client. + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + for (let index = 0; index < 129; index += 1) { + const id = `execution-${String(index).padStart(3, "0")}`; + await on(stub, (owner) => owner.beginExecution(id, `2026-01-01T00:00:0${index % 10}.000Z`)); + } + // Two stopped rows, so the optional members cross as well as the required + // ones. A record that only ever travelled in its shortest form would not + // prove the parser accepts the shape the owner actually builds. + await on(stub, (owner) => + owner.stopExecution("execution-001", "2026-01-01T01:00:00.000Z", "completed"), + ); + await on(stub, (owner) => + owner.stopExecution("execution-002", "2026-01-01T02:00:00.000Z", "failed", "it-stopped"), + ); + + const socket = await connect(stub); + let identifier = 0; + const outcome = await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + const ids = () => `read-${(identifier += 1)}`; + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids, RUN_ID), + ids, + RUN_ID, + ); + const first = yield* link.readExecutions(); + // Appended after the snapshot was anchored, and while the read is still + // paging: the anchor is what decides, not when the rows were written. + yield* call(() => on(stub, (owner) => owner.beginExecution("execution-later", NEW_START))); + return { first, second: yield* link.readExecutions() }; + }); + + if (!outcome.first.ok) { + throw outcome.first.error; + } + const records = outcome.first.value; + expect(records).toHaveLength(129); + expect(records.map((held) => held.executionId)).toEqual( + Array.from({ length: 129 }, (_, index) => `execution-${String(index).padStart(3, "0")}`), + ); + expect(records[0]).toEqual({ + executionId: "execution-000", + startedAt: "2026-01-01T00:00:00.000Z", + }); + expect(records[1]).toEqual({ + executionId: "execution-001", + startedAt: "2026-01-01T00:00:01.000Z", + stoppedAt: "2026-01-01T01:00:00.000Z", + stopStatus: "completed", + }); + expect(records[2]).toEqual({ + executionId: "execution-002", + startedAt: "2026-01-01T00:00:02.000Z", + stoppedAt: "2026-01-01T02:00:00.000Z", + stopStatus: "failed", + stopReason: { kind: "host", code: "it-stopped" }, + }); + // Nothing physical crossed: the runner never sees a column name. + expect(Object.keys(records[0])).toEqual(["executionId", "startedAt"]); + + if (!outcome.second.ok) { + throw outcome.second.error; + } + // The later row is outside the first anchored snapshot and inside the next. + expect(outcome.second.value).toHaveLength(130); + expect(outcome.second.value[129]?.executionId).toBe("execution-later"); + }); + it("returns only content referenced by one validated root", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 7ab9cb51d..7a6ef6512 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -362,6 +362,18 @@ export class ExecutorObject extends WorkflowOwnerObject { ); } + /** Stop one document execution, as the matching transition would. */ + stopExecution(executionId: string, stoppedAt: string, status: string, code?: string): void { + this.ctx.storage.sql.exec( + "UPDATE document_executions SET stopped_at = ?, stop_status = ?, stop_reason_kind = ?, stop_reason_code = ? WHERE execution_id = ?", + stoppedAt, + status, + code === undefined ? null : "host", + code ?? null, + executionId, + ); + } + /** What the retrieval row holds right now. */ retrieval(): Record | null { const row = this.ctx.storage.sql diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts index b9711a6de..74fa2b515 100644 --- a/packages/workflow/tests/remote-client.test.ts +++ b/packages/workflow/tests/remote-client.test.ts @@ -15,6 +15,7 @@ import { type OwnerSocket, OwnerLinkError, type SocketListener, + MAX_MESSAGE_BYTES, useOwnerConnection, } from "../src/remote/client.ts"; @@ -469,6 +470,42 @@ describe("a connection to a run's owner", () => { expect(wire.listening).toBe(0); }); + it("refuses a request larger than one message, before it is outstanding", function* () { + const wire = fakeSocket(); + let raised: unknown; + let reused: unknown; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + try { + // Under the bound on its own; over it once the correlation id and the + // framing around it are counted. Measuring one member instead would + // let exactly this request through. + yield* owner.ask( + "over", + { command: "retrieval", metadata: "m".repeat(MAX_MESSAGE_BYTES - 40) }, + readString, + ); + } catch (error) { + raised = error; + } + // The id never became outstanding, so it is still usable. A request that + // was registered and then refused would fail here as a duplicate. + try { + yield* spawn(function* () { + yield* owner.ask("over", { command: "frontier" }, readString); + }); + yield* sleep(0); + } catch (error) { + reused = error; + } + }); + expect(refusalOf(raised)).toBe("too-large"); + expect(reused).toBe(undefined); + // Exactly one message left: the small one. + expect(wire.sent).toEqual([{ id: "over", command: "frontier" }]); + }); + it("refuses to send a correlation id it would refuse to read", function* () { const wire = fakeSocket(); let raised: unknown; diff --git a/packages/workflow/tests/remote-read.test.ts b/packages/workflow/tests/remote-read.test.ts index ab66205e5..0616654b7 100644 --- a/packages/workflow/tests/remote-read.test.ts +++ b/packages/workflow/tests/remote-read.test.ts @@ -23,7 +23,8 @@ import { cloudflareRunLink, stageCloudflareContent, } from "../src/cloudflare/client.ts"; -import { WorkflowStorageError } from "../src/storage/errors.ts"; +import { WorkflowSchemaVersionError, WorkflowStorageError } from "../src/storage/errors.ts"; +import { SCHEMA_VERSION } from "../src/sqlite/workflow-schema.ts"; import { createTransactionGate, transactRemotely } from "../src/remote/collector.ts"; import { encodeBase64 } from "../src/cloudflare/encoding.ts"; import type { OwnerSocket, SocketListener } from "../src/remote/client.ts"; @@ -338,6 +339,61 @@ describe("semantic reads from a Cloudflare owner", () => { expect(transport.closes).toBe(1); }); + it("closes on a retrieval answer describing a replacement nobody asked for", function* () { + // The contradiction is settled where the answer arrives, not by a caller + // noticing afterwards. Two sides that disagree about which replacement was + // performed have no shared state left to continue from. + const transport = wire(() => ({ + outcome: "performed", + value: { + retrieval: { + metadata: { locator: "something else entirely" }, + revision: 1, + updatedAt: "2026-09-04T00:00:01.000Z", + }, + }, + })); + let outcome: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids(), RUN_ID), + ids(), + RUN_ID, + ); + outcome = yield* link.replaceRetrieval(ROOT_ID, '{"locator":"what was asked"}'); + }); + expect((outcome as { ok: boolean }).ok).toBe(false); + expect(transport.closes).toBe(1); + }); + + it("reports the schema version the owner actually read", function* () { + // A version this build cannot open is the one fact the refusal exists to + // carry. Reporting a placeholder would state something the owner never + // said, and a host deciding whether to upgrade would act on it. + const transport = wire(() => ({ + outcome: "refused", + refusal: "storage:unsupported-version-v7", + })); + let outcome: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids(), RUN_ID), + ids(), + RUN_ID, + ); + outcome = yield* link.readExecutions(); + }); + const failed = outcome as { ok: boolean; error: Error }; + expect(failed.ok).toBe(false); + expect(failed.error).toEqual(expect.any(WorkflowSchemaVersionError)); + const version = failed.error as WorkflowSchemaVersionError; + expect([version.stored, version.supported]).toEqual([7, SCHEMA_VERSION]); + }); + it("closes when content bytes disagree with the requested identity", function* () { const transport = wire(() => ({ outcome: "performed", From 0303ae20403a79503fc7b48e18e5ec5ee38510a3 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 13:24:33 -0400 Subject: [PATCH 44/81] =?UTF-8?q?=F0=9F=90=9B=20Report=20what=20actually?= =?UTF-8?q?=20failed,=20and=20close=20the=20version=20domain?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A parser failure was discarded in favour of the channel's, so a contradictory retrieval answer reached the public boundary as an unreachable owner. The request whose answer could not be read now keeps that failure and every other waiter still learns the channel ended, which lets the adapter report a malformed record. A commit answer this build cannot read is undecided for the same reason a lost connection is. An oversized request is classified once, at the adapter boundary, so the database returns a request failure rather than a transaction failure; the duplicate mapping that could never run is gone. Schema versions now have one domain, taken from the carrier they live in and used at marker recognition, refusal encoding, refusal parsing and the public error. Version zero is a partial initialization and so damage, and a value the carrier could never hold is damaged retained data — neither is a version this build is behind. The composed execution regression now writes its later row between the two page requests of one read, and both ends are held to the shared page bound. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 45 ++- packages/workflow/src/cloudflare/marker.ts | 13 +- .../workflow/src/cloudflare/recognition.ts | 4 +- packages/workflow/src/remote/client.ts | 43 ++- packages/workflow/src/remote/database.ts | 8 - .../workflow/src/sqlite/workflow-schema.ts | 22 ++ .../tests/cloudflare/owner-storage.vitest.ts | 24 +- .../tests/cloudflare/remote-owner.vitest.ts | 119 +++++++- packages/workflow/tests/remote-client.test.ts | 14 +- packages/workflow/tests/remote-read.test.ts | 267 ++++++++++++++++-- 10 files changed, 488 insertions(+), 71 deletions(-) diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 00d62e903..bc70835b9 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -63,7 +63,7 @@ import { MAX_CONTENT_BYTES, } from "./commands.ts"; import type { RemoteRunLink } from "../remote/database.ts"; -import { SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; +import { isSchemaVersion, SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; import { canonicalJson } from "../storage/record.ts"; import { WorkflowDatabaseCorruptError, @@ -175,9 +175,9 @@ function privateRefusal(value: string): PrivateRefusal { default: { // The one category that carries a value: the schema version the owner // actually read, bounded and parsed rather than guessed. - const unsupported = /^storage:unsupported-version-v(\d{1,6})$/.exec(value); - if (unsupported !== null) { - return `storage:unsupported-version-v${Number(unsupported[1])}`; + const unsupported = readUnsupportedVersion(value); + if (unsupported !== undefined) { + return `storage:unsupported-version-v${unsupported}`; } return fail("it named an unknown refusal category"); } @@ -305,6 +305,22 @@ function parseContent(value: unknown): RemoteContent { return { kind, digest, bytes }; } +/** + * The schema version an unsupported-version refusal names, if it names one. + * + * The grammar covers exactly the versions the owner can recognize as + * unsupported, so a same-release owner and client never disagree about whether + * a refusal is readable. Anything else is not this category. + */ +function readUnsupportedVersion(refusal: string): number | undefined { + const found = /^storage:unsupported-version-v(\d{1,10})$/.exec(refusal); + if (found === null) { + return undefined; + } + const version = Number(found[1]); + return isSchemaVersion(version) ? version : undefined; +} + export function cloudflareReadLink( connection: OwnerConnection, nextId: () => string, @@ -426,9 +442,10 @@ export function cloudflareOwnerLink( ? Err(new CloudflareOwnerRefusalError(answered.refusal)) : Ok(answered.decision); } catch (error) { - if (error instanceof OwnerLinkError) { - // The connection went while the answer was in flight. Whether the - // owner committed is exactly what cannot be known from here, so the + if (error instanceof OwnerLinkError || error instanceof RemoteRecordError) { + // The connection went while the answer was in flight, or the owner + // answered in a way this build cannot read. Whether the owner + // committed is exactly what cannot be known from either, so the // caller learns the outcome is undecided rather than being told it // failed — retrying this same id is what settles it. return Err(error); @@ -835,9 +852,9 @@ function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { if (refusal === "storage:foreign") { return new WorkflowDatabaseFormatError(REMOTE_STORE, "it belongs to something else"); } - const unsupported = /^storage:unsupported-version-v(\d{1,6})$/.exec(refusal); - if (unsupported !== null) { - return new WorkflowSchemaVersionError(REMOTE_STORE, Number(unsupported[1]), SCHEMA_VERSION); + const unsupported = readUnsupportedVersion(refusal); + if (unsupported !== undefined) { + return new WorkflowSchemaVersionError(REMOTE_STORE, unsupported, SCHEMA_VERSION); } if (refusal === "storage:corrupt") { return new WorkflowDatabaseCorruptError(REMOTE_STORE, "its retained records do not agree"); @@ -883,6 +900,14 @@ function translate(error: unknown): WorkflowStorageError { ); } if (error instanceof OwnerLinkError) { + if (error.refusal === "too-large") { + // The channel measured the whole request and never sent it. That is a + // request this caller cannot make, not an owner it could not reach, and + // the two lead a host to do different things. + return new WorkflowRequestError( + "this request is larger than one message may carry, so it was not sent.", + ); + } return new WorkflowTransactionError("this run's owner could not be reached."); } return new WorkflowTransactionError("this run's owner could not answer the operation."); diff --git a/packages/workflow/src/cloudflare/marker.ts b/packages/workflow/src/cloudflare/marker.ts index e111577dc..da15833c9 100644 --- a/packages/workflow/src/cloudflare/marker.ts +++ b/packages/workflow/src/cloudflare/marker.ts @@ -20,7 +20,7 @@ * refused rather than migrated. */ -import { APPLICATION_ID, SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; +import { APPLICATION_ID, isSchemaVersion, SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; /** The adapter-private table carrying this database's identity. */ export const MARKER_TABLE = "_xmd_workflow_schema"; @@ -43,6 +43,7 @@ export type MarkerFailure = | { readonly kind: "duplicated"; readonly rows: number } | { readonly kind: "malformed" } | { readonly kind: "foreign-application"; readonly applicationId: number } + | { readonly kind: "incomplete-version" } | { readonly kind: "unknown-version"; readonly schemaVersion: number }; /** @@ -76,6 +77,16 @@ export function readMarker(rows: readonly Record[]): SchemaMark if (applicationId !== APPLICATION_ID) { return { kind: "foreign-application", applicationId }; } + if (schemaVersion === 0) { + // The identity is this project's and the version says nothing was + // finished. That is a database left partly initialized, not an older one. + return { kind: "incomplete-version" }; + } + if (!isSchemaVersion(schemaVersion)) { + // Outside what the version carrier can hold, so no build wrote it. The row + // is damaged retained data rather than a version to report. + return { kind: "malformed" }; + } if (schemaVersion !== SCHEMA_VERSION) { return { kind: "unknown-version", schemaVersion }; } diff --git a/packages/workflow/src/cloudflare/recognition.ts b/packages/workflow/src/cloudflare/recognition.ts index e283e5b4d..9368f9e31 100644 --- a/packages/workflow/src/cloudflare/recognition.ts +++ b/packages/workflow/src/cloudflare/recognition.ts @@ -163,7 +163,9 @@ export function recognizeObject(storage: OwnerStorage): void { ? "its schema marker table holds no identity row" : marker.kind === "duplicated" ? "its schema marker table holds more than one identity row" - : "its schema marker row does not describe an identity", + : marker.kind === "incomplete-version" + ? "it carries the XMD application identity without a complete version-1 schema" + : "its schema marker row does not describe an identity", }); } diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index ef4af4006..d03d2e152 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -44,6 +44,17 @@ export type LinkRefusal = | "send-failed" | "socket-error"; +/** + * A parser's own failure, as something that can be settled and reported. + * + * A parser may throw anything. What travels back to the caller has to be an + * `Error`, and it has to stay the parser's failure rather than becoming the + * channel's, so the boundary that knows what the value meant can classify it. + */ +function unreadable(error: unknown): Error { + return error instanceof Error ? error : new OwnerLinkError("malformed-answer"); +} + export class OwnerLinkError extends Error { override name = "OwnerLinkError"; @@ -210,10 +221,13 @@ export function useOwnerConnection(socket: OwnerSocket): Operation(); @@ -252,11 +266,11 @@ export function useOwnerConnection(socket: OwnerSocket): Operation>(); waiting.set(id, { - deliver(answer: RawAnswer): boolean { + deliver(answer: RawAnswer): Error | undefined { if (answer.outcome === "refused") { let refusal: string; try { refusal = parseRefusal(answer.refusal); - } catch { - return false; + } catch (error) { + settle.reject(unreadable(error)); + return unreadable(error); } settle.resolve({ outcome: "refused", refusal }); - return true; + return undefined; } let value: T; try { value = parse(answer.value); - } catch { - return false; + } catch (error) { + // The request that asked learns why its own answer could not be + // read. Whoever else is waiting learns the channel ended, which + // is all that is true for them. + settle.reject(unreadable(error)); + return unreadable(error); } settle.resolve({ outcome: "performed", value }); - return true; + return undefined; }, fail(error: OwnerLinkError): void { settle.reject(error); diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 310e160a7..2607346f3 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -49,7 +49,6 @@ import type { WorkflowRunRecord, } from "../storage/record.ts"; import { createTransactionGate, type OwnerLink, transactRemotely } from "./collector.ts"; -import { OwnerLinkError } from "./client.ts"; import type { EnlistWorkspace } from "./collector.ts"; import type { RemoteFrontierSnapshot } from "./read.ts"; @@ -141,13 +140,6 @@ export function* activeWorkspaceRoute( * string nobody can act on. */ function failure(error: unknown): Error { - if (error instanceof OwnerLinkError && error.refusal === "too-large") { - // The channel measured the whole request and never sent it. To a caller - // that is not a lost connection, it is a request too large to make. - return new WorkflowRequestError( - "this request is larger than one message may carry, so it was not sent.", - ); - } return error instanceof Error ? error : new WorkflowTransactionError(String(error)); } diff --git a/packages/workflow/src/sqlite/workflow-schema.ts b/packages/workflow/src/sqlite/workflow-schema.ts index 1f9c8d0a3..df89b9e9e 100644 --- a/packages/workflow/src/sqlite/workflow-schema.ts +++ b/packages/workflow/src/sqlite/workflow-schema.ts @@ -33,6 +33,28 @@ export const APPLICATION_ID = 0x584d4431; /** The only schema version this build reads or writes. */ export const SCHEMA_VERSION = 1; +/** + * The largest value the schema version can be carried in. + * + * The logical carrier is SQLite's `user_version`, a signed 32-bit integer. Any + * host holding this schema has to represent the same versions, so the bound is + * the carrier's rather than one adapter's. + */ +export const MAX_SCHEMA_VERSION = 0x7fffffff; + +/** + * Whether a retained value could name a schema version at all. + * + * Version numbering starts at 1 and rises. Zero is a database carrying the XMD + * identity without a complete schema, which is a partial initialization and so + * damage; a negative or out-of-range value is retained data that no build of + * this project ever wrote. Neither is a version this build has not learned, so + * neither may travel as one. + */ +export function isSchemaVersion(value: number): boolean { + return Number.isInteger(value) && value >= 1 && value <= MAX_SCHEMA_VERSION; +} + const STATUSES = "'running', 'suspended', 'interrupted', 'completed', 'failed', 'cancelled'"; /** diff --git a/packages/workflow/tests/cloudflare/owner-storage.vitest.ts b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts index 82544a870..1a28ea72c 100644 --- a/packages/workflow/tests/cloudflare/owner-storage.vitest.ts +++ b/packages/workflow/tests/cloudflare/owner-storage.vitest.ts @@ -69,13 +69,35 @@ describe("recognizing an owner object", () => { expect(await on(stub, (o) => o.recognize())).toBe("refused:unsupported-version"); }); - it("refuses version zero the same way", async () => { + it("calls version zero a partial initialization rather than an old version", async () => { + // This project's identity with nothing finished under it. There has never + // been a version zero to be behind, so reporting one would send a host + // looking for a migration that cannot exist. const stub = owner(); await on(stub, (o) => o.initialize()); await on(stub, (o) => o.rewriteMarker(0x584d4431, 0)); + expect(await on(stub, (o) => o.recognize())).toBe("refused:corrupt"); + }); + + it("carries a version wider than the refusal's old grammar", async () => { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.rewriteMarker(0x584d4431, 1_000_000)); expect(await on(stub, (o) => o.recognize())).toBe("refused:unsupported-version"); }); + it("calls a version the carrier could never hold damaged retained data", async () => { + // Negative, and past the signed 32-bit carrier: no build of this project + // wrote either. A version this build has not learned and a row that cannot + // be a version are different facts. + for (const version of [-1, 0x80000000]) { + const stub = owner(); + await on(stub, (o) => o.initialize()); + await on(stub, (o) => o.rewriteMarker(0x584d4431, version)); + expect([version, await on(stub, (o) => o.recognize())]).toEqual([version, "refused:corrupt"]); + } + }); + it("refuses a shape that disagrees with what version 1 declares", async () => { const stub = owner(); await on(stub, (o) => o.initialize()); diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index 73c03f795..bded8699e 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -32,7 +32,7 @@ import { VALID_CLAIMS, } from "./support/executor-object.ts"; import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; -import { call, run } from "effection"; +import { run } from "effection"; import { type OwnerSocket, type SocketListener, @@ -133,23 +133,35 @@ function askFrame( * wider than the four members the client uses, so the binding is written out * rather than asserted. */ -function ownerSocket(socket: WebSocket): OwnerSocket { +function ownerSocket(socket: WebSocket, beforeSend?: (raw: string) => Promise | undefined) { const listeners = new Map(); - return { - send: (data) => socket.send(data), + const bound: OwnerSocket = { + send(data) { + // A frame may be held back before it reaches the owner, which is how a + // test puts a write between two pages of one read without reaching + // inside the client. + const waiting = beforeSend?.(data); + if (waiting === undefined) { + socket.send(data); + return; + } + void waiting.then(() => socket.send(data)); + }, close: () => socket.close(), addEventListener(type, listener) { - const bound: EventListener = (event) => listener(event as { data?: unknown }); - listeners.set(listener, bound); - socket.addEventListener(type, bound); + const forward: EventListener = (event) => listener(event as { data?: unknown }); + listeners.set(listener, forward); + socket.addEventListener(type, forward); }, removeEventListener(type, listener) { const bound = listeners.get(listener); - if (bound !== undefined) { - socket.removeEventListener(type, bound); + const found = listeners.get(listener); + if (found !== undefined) { + socket.removeEventListener(type, found); } }, }; + return bound; } function record(value: unknown): Record { @@ -326,8 +338,25 @@ describe("the remote owner protocol", () => { const socket = await connect(stub); let identifier = 0; + // 129 rows page at 128, so the read takes two requests. The later row is + // written between them: after the first page fixed the anchor, and before + // the owner is asked for the second. That is the moment the anchor exists + // to survive, and asserting it any later would prove nothing about paging. + let requests = 0; + let inserted = false; + const wire = ownerSocket(socket, (raw) => { + if (!raw.includes('"executions"')) { + return undefined; + } + requests += 1; + if (requests !== 2) { + return undefined; + } + inserted = true; + return on(stub, (owner) => owner.beginExecution("execution-later", NEW_START)); + }); const outcome = await run(function* () { - const connection = yield* useOwnerConnection(ownerSocket(socket)); + const connection = yield* useOwnerConnection(wire); const ids = () => `read-${(identifier += 1)}`; const link = cloudflareRunLink( connection, @@ -336,11 +365,10 @@ describe("the remote owner protocol", () => { RUN_ID, ); const first = yield* link.readExecutions(); - // Appended after the snapshot was anchored, and while the read is still - // paging: the anchor is what decides, not when the rows were written. - yield* call(() => on(stub, (owner) => owner.beginExecution("execution-later", NEW_START))); return { first, second: yield* link.readExecutions() }; }); + // The write really did land between the two page requests. + expect([requests >= 2, inserted]).toEqual([true, true]); if (!outcome.first.ok) { throw outcome.first.error; @@ -378,6 +406,71 @@ describe("the remote owner protocol", () => { expect(outcome.second.value[129]?.executionId).toBe("execution-later"); }); + it("ends a page on the byte bound, and refuses a record that can never fit", async () => { + // The entry bound is 128 rows; this one is reached by bytes first. Both + // ends measure the same serialized `rows` array, so what the owner decides + // fits is exactly what the runner accepts — and the whole history still + // arrives, in order, across however many pages that takes. + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const padding = "p".repeat(64 * 1024); + for (let index = 0; index < 20; index += 1) { + const id = `${String(index).padStart(2, "0")}-${padding}`; + await on(stub, (owner) => owner.beginExecution(id, "2026-01-01T00:00:00.000Z")); + } + const socket = await connect(stub); + let identifier = 0; + let requests = 0; + const wire = ownerSocket(socket, (raw) => { + if (raw.includes('"executions"')) { + requests += 1; + } + return undefined; + }); + const outcome = await run(function* () { + const connection = yield* useOwnerConnection(wire); + const ids = () => `page-${(identifier += 1)}`; + return yield* cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids, RUN_ID), + ids, + RUN_ID, + ).readExecutions(); + }); + if (!outcome.ok) { + throw outcome.error; + } + expect(outcome.value).toHaveLength(20); + expect(outcome.value.map((held) => held.executionId.slice(0, 2))).toEqual( + Array.from({ length: 20 }, (_, index) => String(index).padStart(2, "0")), + ); + // Well under 128 entries a page, so bytes ended these pages, not the count. + expect(requests).toBeGreaterThan(1); + + // One record larger than a whole page. There is no page that could carry + // it, so the owner refuses rather than answering with something the runner + // is required to reject. + const single = executor(); + await on(single, (owner) => owner.initialize()); + const huge = "h".repeat(600 * 1024); + await on(single, (owner) => owner.beginExecution(huge, "2026-01-01T00:00:00.000Z")); + const alone = await connect(single); + let count = 0; + const refused = await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(alone)); + const ids = () => `huge-${(count += 1)}`; + return yield* cloudflareRunLink( + connection, + cloudflareReadLink(connection, ids, RUN_ID), + ids, + RUN_ID, + ).readExecutions(); + }); + expect(refused.ok).toBe(false); + // Provider-neutral, with no private refusal spelling in it. + expect(String(refused.ok === false && refused.error)).not.toContain("command:"); + }); + it("returns only content referenced by one validated root", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts index 74fa2b515..6dcd3fb5f 100644 --- a/packages/workflow/tests/remote-client.test.ts +++ b/packages/workflow/tests/remote-client.test.ts @@ -272,7 +272,7 @@ describe("a connection to a run's owner", () => { expect(refusalOf(raised)).toBe("unknown-answer"); }); - it("fails every waiter when a success value cannot be parsed", function* () { + it("tells the asker why its own answer was unreadable, and everyone else the channel ended", function* () { const wire = fakeSocket(); const raised: unknown[] = []; yield* scoped(function* () { @@ -300,9 +300,15 @@ describe("a connection to a run's owner", () => { yield* second; }); expect(raised).toHaveLength(2); - for (const error of raised) { - expect(refusalOf(error)).toBe("malformed-answer"); - } + // The request whose answer failed keeps the parser's own failure: the + // boundary above it can only classify what a value meant if it still holds + // the failure that said so. Reporting an unreachable owner here would be + // untrue — the owner answered, and this build could not read it. + const asker = raised.find((error) => !(error instanceof OwnerLinkError)); + expect(String(asker)).toContain("expected a string"); + // Nothing else is true for the other waiter except that the channel ended. + const other = raised.filter((error) => error !== asker); + expect(other.map(refusalOf)).toEqual(["malformed-answer"]); expect(wire.closes).toBe(1); expect(wire.listening).toBe(0); }); diff --git a/packages/workflow/tests/remote-read.test.ts b/packages/workflow/tests/remote-read.test.ts index 0616654b7..d4166c05d 100644 --- a/packages/workflow/tests/remote-read.test.ts +++ b/packages/workflow/tests/remote-read.test.ts @@ -23,17 +23,28 @@ import { cloudflareRunLink, stageCloudflareContent, } from "../src/cloudflare/client.ts"; -import { WorkflowSchemaVersionError, WorkflowStorageError } from "../src/storage/errors.ts"; +import { + WorkflowRecordMalformedError, + WorkflowRequestError, + WorkflowSchemaVersionError, + WorkflowStorageError, +} from "../src/storage/errors.ts"; +import { useRemoteRunDatabase } from "../src/remote/database.ts"; +import { MAX_MESSAGE_BYTES } from "../src/remote/client.ts"; +import type { Result } from "effection"; +import type { DefinitionRetrieval } from "../src/storage/record.ts"; import { SCHEMA_VERSION } from "../src/sqlite/workflow-schema.ts"; import { createTransactionGate, transactRemotely } from "../src/remote/collector.ts"; import { encodeBase64 } from "../src/cloudflare/encoding.ts"; import type { OwnerSocket, SocketListener } from "../src/remote/client.ts"; import { OwnerLinkError, useOwnerConnection } from "../src/remote/client.ts"; +import { RemoteRecordError } from "../src/remote/records.ts"; import { EMPTY_WORKSPACE_MANIFEST, EMPTY_WORKSPACE_ROOT_ID, workspaceRootId, } from "../src/deno/workspace/manifest.ts"; +import { EXECUTION_PAGE_BYTES, executionPageBytes } from "../src/cloudflare/commands.ts"; import { WORKSPACE_ROOT_DOMAIN } from "../src/workspace/root-manifest.ts"; import { sha256Hex } from "../src/workspace/sha256.ts"; @@ -116,6 +127,16 @@ function wire(answer: (request: Record) => Record { + return { + record: runRecord(), + retrieval: null, + workspaceRootId: ROOT_ID, + journalEventId: null, + }; +} + function ids(): () => string { let id = 0; return () => `request-${(id += 1)}`; @@ -141,6 +162,21 @@ function failure(error: unknown): string { return error.refusal; } +/** + * The parser's own failure, having proved it is one. + * + * The request whose answer could not be read keeps that failure rather than + * the channel's, because only the boundary above it can say what the value was + * supposed to mean. Reading it as a string would let an unrelated error pass + * for the category a test expected. + */ +function unreadable(error: unknown): string { + if (!(error instanceof RemoteRecordError)) { + throw new Error(`expected a RemoteRecordError, received ${String(error)}`); + } + return "malformed-record"; +} + describe("semantic reads from a Cloudflare owner", () => { it("uses the standard SHA-256 identity rather than an adapter-local digest", function* () { // The published answers, including the two-block case the padding rule is @@ -318,7 +354,7 @@ describe("semantic reads from a Cloudflare owner", () => { raised = error; } }); - expect([description, failure(raised)]).toEqual([description, "malformed-answer"]); + expect([description, unreadable(raised)]).toEqual([description, "malformed-record"]); expect([description, transport.closes]).toEqual([description, 1]); expect([description, transport.listeners]).toEqual([description, 0]); } @@ -335,7 +371,7 @@ describe("semantic reads from a Cloudflare owner", () => { raised = error; } }); - expect(failure(raised)).toBe("malformed-answer"); + expect(unreadable(raised)).toBe("malformed-record"); expect(transport.closes).toBe(1); }); @@ -368,30 +404,175 @@ describe("semantic reads from a Cloudflare owner", () => { expect(transport.closes).toBe(1); }); - it("reports the schema version the owner actually read", function* () { - // A version this build cannot open is the one fact the refusal exists to - // carry. Reporting a placeholder would state something the owner never - // said, and a host deciding whether to upgrade would act on it. - const transport = wire(() => ({ - outcome: "refused", - refusal: "storage:unsupported-version-v7", - })); - let outcome: unknown; + it("returns a malformed record to the caller, and leaves nothing usable behind", function* () { + // The whole point of carrying the parser's failure: the public boundary + // says the owner returned a record this build cannot read, which is what + // happened, rather than that the owner could not be reached. + const mutations: Record[] = []; + const transport = wire((request) => { + if (request["command"] === "frontier") { + return { outcome: "performed", value: frontierValue() }; + } + mutations.push(request); + return { + outcome: "performed", + value: { + retrieval: { + metadata: { locator: "something else entirely" }, + revision: 1, + updatedAt: "2026-09-04T00:00:01.000Z", + }, + }, + }; + }); + let refused: Result | undefined; + let after: Result | undefined; + let held: DefinitionRetrieval | undefined; yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); + // One generator for both halves: two would mint the same correlation id + // and the connection would fail closed on the duplicate. + const next = ids(); const link = cloudflareRunLink( connection, - cloudflareReadLink(connection, ids(), RUN_ID), - ids(), + cloudflareReadLink(connection, next, RUN_ID), + next, + RUN_ID, + ); + const database = yield* useRemoteRunDatabase(link, yield* link.frontierSnapshot()); + refused = yield* database.replaceRetrievalMetadata({ locator: "what was asked" }); + held = database.retrieval; + expect(mutations).toHaveLength(1); + after = yield* database.replaceRetrievalMetadata({ locator: "later" }); + // The channel is gone, so the second call never reached the owner. + expect(mutations).toHaveLength(1); + }); + expect(refused?.ok).toBe(false); + expect(refused?.ok === false && refused.error).toEqual( + expect.any(WorkflowRecordMalformedError), + ); + // Nothing private crossed with it. + expect(String(refused?.ok === false && refused.error)).not.toContain("something else"); + // The snapshot is what the frontier established: an answer about another + // value installs nothing, because it decides where the definition is read. + expect(held).toEqual(undefined); + expect(after?.ok).toBe(false); + expect(after?.ok === false && after.error).toEqual(expect.any(WorkflowStorageError)); + expect(transport.closes).toBe(1); + }); + + it("returns a request failure when the whole request cannot be carried", function* () { + // Metadata that fits the bound on its own and does not once the command + // around it and its correlation id are counted. The public boundary has to + // say the request was too large, not that the owner was unreachable. + const mutations: Record[] = []; + const transport = wire((request) => { + if (request["command"] === "frontier") { + return { outcome: "performed", value: frontierValue() }; + } + mutations.push(request); + // An honest owner: it performed exactly the replacement it was asked for. + return { + outcome: "performed", + value: { + retrieval: { + metadata: JSON.parse(String(request["metadata"])), + revision: 1, + updatedAt: "2026-09-04T00:00:01.000Z", + }, + }, + }; + }); + let refused: Result | undefined; + let accepted: Result | undefined; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + // One generator for both halves: two would mint the same correlation id + // and the connection would fail closed on the duplicate. + const next = ids(); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, next, RUN_ID), + next, RUN_ID, ); - outcome = yield* link.readExecutions(); + const database = yield* useRemoteRunDatabase(link, yield* link.frontierSnapshot()); + refused = yield* database.replaceRetrievalMetadata({ + locator: "m".repeat(MAX_MESSAGE_BYTES - 64), + }); + // Never sent, so the owner has no idea this was asked. + expect(mutations).toEqual([]); + // The connection was not spent on it either: the next one goes through. + accepted = yield* database.replaceRetrievalMetadata({ locator: "small" }); + expect(mutations).toHaveLength(1); }); - const failed = outcome as { ok: boolean; error: Error }; - expect(failed.ok).toBe(false); - expect(failed.error).toEqual(expect.any(WorkflowSchemaVersionError)); - const version = failed.error as WorkflowSchemaVersionError; - expect([version.stored, version.supported]).toEqual([7, SCHEMA_VERSION]); + expect(refused?.ok).toBe(false); + expect(refused?.ok === false && refused.error).toEqual(expect.any(WorkflowRequestError)); + expect(String(refused?.ok === false && refused.error)).not.toContain("too-large"); + expect(accepted?.ok).toBe(true); + expect(transport.closes).toBe(1); + }); + + it("refuses a version spelling outside what a version can be", function* () { + // Zero is a partial initialization and anything past the carrier is + // damaged retained data. Neither is a version this build is behind, so a + // same-release owner never sends one and this client never reads one. + for (const refusal of [ + "storage:unsupported-version-v0", + "storage:unsupported-version-v99999999999", + ]) { + const transport = wire(() => ({ outcome: "refused", refusal })); + let outcome: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const next = ids(); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, next, RUN_ID), + next, + RUN_ID, + ); + outcome = yield* link.readExecutions(); + }); + const failed = outcome as { ok: boolean; error: Error }; + expect([refusal, failed.ok]).toEqual([refusal, false]); + expect([refusal, failed.error]).toEqual([refusal, expect.any(WorkflowStorageError)]); + // Not a version report, and nothing of the spelling crossed. + expect(failed.error).not.toEqual(expect.any(WorkflowSchemaVersionError)); + expect(String(failed.error)).not.toContain("storage:"); + expect([refusal, transport.closes]).toEqual([refusal, 1]); + } + }); + + it("reports the schema version the owner actually read", function* () { + // A version this build cannot open is the one fact the refusal exists to + // carry. Reporting a placeholder would state something the owner never + // said, and a host deciding whether to upgrade would act on it. + // Seven, and a value wider than the grammar this refusal once had: both + // are versions the owner can recognize, so both must arrive exactly. + for (const stored of [7, 1_000_000]) { + const transport = wire(() => ({ + outcome: "refused", + refusal: `storage:unsupported-version-v${stored}`, + })); + let outcome: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const next = ids(); + const link = cloudflareRunLink( + connection, + cloudflareReadLink(connection, next, RUN_ID), + next, + RUN_ID, + ); + outcome = yield* link.readExecutions(); + }); + const failed = outcome as { ok: boolean; error: Error }; + expect([stored, failed.ok]).toEqual([stored, false]); + expect([stored, failed.error]).toEqual([stored, expect.any(WorkflowSchemaVersionError)]); + const version = failed.error as WorkflowSchemaVersionError; + expect([version.stored, version.supported]).toEqual([stored, SCHEMA_VERSION]); + } }); it("closes when content bytes disagree with the requested identity", function* () { @@ -417,7 +598,7 @@ describe("semantic reads from a Cloudflare owner", () => { raised = error; } }); - expect(failure(raised)).toBe("malformed-answer"); + expect(unreadable(raised)).toBe("malformed-record"); expect(transport.closes).toBe(1); }); it("hands the collector one assembled frontier and no page mechanics", function* () { @@ -515,6 +696,12 @@ describe("semantic reads from a Cloudflare owner", () => { "a record that stopped without saying how": page([ { sequence: 1, record: { ...record("a"), stopStatus: "completed" } }, ]), + // Measured the same way the owner measures it, over the same wrappers. + // A page past the bound is refused whole: no prefix of it is returned. + "a page past the byte bound": page([ + row(1, "a"), + { sequence: 2, record: record("b".repeat(EXECUTION_PAGE_BYTES)) }, + ]), }; for (const [description, answer] of Object.entries(refused)) { @@ -543,6 +730,44 @@ describe("semantic reads from a Cloudflare owner", () => { } }); + it("accepts a page filled to the byte bound", function* () { + // The boundary itself, from the runner's side: one page whose serialized + // rows land at or just under the bound is honest and is assembled. If the + // two ends measured different things, this is the page they would + // disagree about. + const fill = (size: number) => ({ + sequence: 1, + record: { executionId: "e".repeat(size), startedAt: "2026-09-04T00:00:00.000Z" }, + }); + // The identity is ASCII, so one byte of it is one byte of the page and the + // largest that fits follows from the wrapper's own size. + const overhead = executionPageBytes([fill(0)]); + const largest = fill(EXECUTION_PAGE_BYTES - overhead); + expect(executionPageBytes([largest])).toBe(EXECUTION_PAGE_BYTES); + expect(executionPageBytes([fill(EXECUTION_PAGE_BYTES - overhead + 1)])).toBeGreaterThan( + EXECUTION_PAGE_BYTES, + ); + + const transport = wire(() => ({ + outcome: "performed", + value: { runId: RUN_ID, anchor: 1, after: null, rows: [largest], done: true }, + })); + let outcome: unknown; + yield* scoped(function* () { + const connection = yield* useOwnerConnection(transport.socket); + const next = ids(); + outcome = yield* cloudflareRunLink( + connection, + cloudflareReadLink(connection, next, RUN_ID), + next, + RUN_ID, + ).readExecutions(); + }); + const found = outcome as { ok: boolean; value: { executionId: string }[] }; + expect(found.ok).toBe(true); + expect(found.value.map((held) => held.executionId)).toEqual([largest.record.executionId]); + }); + it("assembles an honest snapshot across pages, and an empty one", function* () { const record = (id: string) => ({ executionId: id, startedAt: "2026-09-04T00:00:00.000Z" }); const pages: Record> = { From ddd9f0c02bc239a50cfad63bd742864e0a257621 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 14:33:20 -0400 Subject: [PATCH 45/81] =?UTF-8?q?=E2=9C=A8=20Run=20Workspace=20work=20on?= =?UTF-8?q?=20the=20runner=20against=20a=20run=20the=20owner=20holds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Deno coordinator opens a transaction, hands a mutation the authoritative filesystem and the retained metadata, and commits both together. This is that shape with the storage somewhere else: the Workspace is a real directory materialized from the exact admitted root, the metadata is a detached snapshot of that same admitted state, and the commit is one intent the owner performs atomically or not at all. Root, journal anchor and every retained mapping are admitted in one owner-side read, because they are one state. It is complete rather than paged: the mapping tables are insert-only, so a cursor over sorted names cannot be made safe by root and journal equality — a later insert can sort before it and never be seen. A count and byte ceiling refuses instead, and refuses whole. The attempt is created outside the transaction, because the collector seals it after the body tears down. Inside the exact callback the route proves it starts from the admitted state before the document runs, and drift refuses there. A documented Workspace failure journals against the unchanged root; everything else raises and publishes nothing. The Filesystem contract, the Repository/Worktree metadata shapes, the journaled-failure base class and Agent-session reconciliation move to neutral modules so both hosts implement one contract rather than two. The Deno adapter keeps its rows, its savepoint and its synchronous-SQLite exception. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 13 + packages/workflow/src/cloudflare/commands.ts | 12 + .../workflow/src/cloudflare/dispatcher.ts | 11 +- .../workflow/src/cloudflare/owner-reads.ts | 126 +++ .../src/deno/remote-workspace-files.ts | 181 +++++ .../src/deno/workspace/agent-sessions.ts | 103 +-- .../workflow/src/deno/workspace/errors.ts | 26 +- .../workflow/src/deno/workspace/filesystem.ts | 41 +- .../src/deno/workspace/repositories.ts | 24 +- packages/workflow/src/remote/collector.ts | 26 +- packages/workflow/src/remote/database.ts | 8 +- packages/workflow/src/remote/journal-route.ts | 94 +++ packages/workflow/src/remote/mappings.ts | 255 ++++++ packages/workflow/src/remote/read.ts | 3 + packages/workflow/src/remote/records.ts | 139 ++++ packages/workflow/src/remote/workspace.ts | 366 +++++++++ .../workflow/src/storage/agent-session.ts | 90 ++ packages/workflow/src/workspace/failure.ts | 28 + packages/workflow/src/workspace/filesystem.ts | 43 + packages/workflow/src/workspace/metadata.ts | 33 + .../tests/cloudflare/remote-publish.vitest.ts | 59 ++ .../cloudflare/support/executor-object.ts | 17 + .../tests/remote-interoperability.test.ts | 6 + .../tests/remote-materialization.test.ts | 6 + .../workflow/tests/remote-publication.test.ts | 6 + .../workflow/tests/remote-workspace.test.ts | 766 ++++++++++++++++++ 26 files changed, 2309 insertions(+), 173 deletions(-) create mode 100644 packages/workflow/src/deno/remote-workspace-files.ts create mode 100644 packages/workflow/src/remote/journal-route.ts create mode 100644 packages/workflow/src/remote/mappings.ts create mode 100644 packages/workflow/src/remote/workspace.ts create mode 100644 packages/workflow/src/workspace/failure.ts create mode 100644 packages/workflow/src/workspace/filesystem.ts create mode 100644 packages/workflow/src/workspace/metadata.ts create mode 100644 packages/workflow/tests/remote-workspace.test.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index bc70835b9..f5d55a800 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -36,7 +36,9 @@ import type { CommitDecision } from "../remote/publication.ts"; import { OwnerLinkError, type OwnerAnswer, type OwnerConnection } from "../remote/client.ts"; import { parseRemoteExecution, + parseRemoteInvocationSnapshot, parseRemoteJournalEntry, + type RemoteInvocationSnapshot, parseRemoteRetrieval, parseRemoteRunRecord, RemoteRecordError, @@ -327,6 +329,17 @@ export function cloudflareReadLink( expectedRunId: string, ): RemoteReadLink { return { + *invocationSnapshot(): Operation { + return answer( + yield* connection.ask( + nextId(), + { command: "mappings" }, + (value) => parseRemoteInvocationSnapshot(value), + privateRefusal, + ), + ); + }, + *frontier(): Operation { const header = answer( yield* connection.ask( diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 6fa306611..0cdd1f8d9 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -55,6 +55,7 @@ export type CommandName = | "commit" | "retrieval" | "executions" + | "mappings" | "settle"; export type CommandRefusal = @@ -198,6 +199,11 @@ export interface ExecutionsCommand extends CommandEnvelope { readonly after: number | null; } +/** One coherent admitted state, asked for exactly once per invocation. */ +export interface MappingsCommand extends CommandEnvelope { + readonly command: "mappings"; +} + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -213,6 +219,7 @@ export type RunnerCommand = | CommitCommand | RetrievalCommand | ExecutionsCommand + | MappingsCommand | SettleCommand; export type CommandResult = @@ -238,6 +245,7 @@ const MEMBERS: Record = { ], retrieval: [...ENVELOPE, "expectedWorkspaceRootId", "metadata"], executions: [...ENVELOPE, "anchor", "after"], + mappings: ENVELOPE, settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], }; @@ -367,6 +375,7 @@ export function parseCommand(raw: string): RunnerCommand { command !== "commit" && command !== "retrieval" && command !== "executions" && + command !== "mappings" && command !== "settle" ) { throw new CommandError("unknown-command"); @@ -426,6 +435,9 @@ export function parseCommand(raw: string): RunnerCommand { metadata, }; } + if (command === "mappings") { + return { id, command }; + } if (command === "executions") { const anchor = sequence(members, "anchor"); const after = sequence(members, "after"); diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 8ee8096f8..bf3345ec7 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -47,6 +47,7 @@ import { bytesOf, decodeBase64, sha256Hex } from "./encoding.ts"; import { readContent, readExecutions, + readInvocationSnapshot, readFrontier, readJournalPage, readRoot, @@ -139,7 +140,8 @@ function retainedDecision(command: RunnerCommand, result: CommandResult): string (command.command === "journal" || command.command === "root" || command.command === "content" || - command.command === "executions") + command.command === "executions" || + command.command === "mappings") ) { return JSON.stringify({ id: command.id, outcome: "reconstruct" }); } @@ -266,6 +268,13 @@ function perform( value: readExecutions(ctx.storage, runId, command.anchor, command.after), }; } + if (command.command === "mappings") { + return { + id: command.id, + outcome: "performed", + value: readInvocationSnapshot(ctx.storage, runId), + }; + } // `settle` is a later checkpoint's. It parses strictly and is declined, // because a placeholder that reported success is the one answer a runner // cannot recover from. diff --git a/packages/workflow/src/cloudflare/owner-reads.ts b/packages/workflow/src/cloudflare/owner-reads.ts index a76580432..d1ec640f5 100644 --- a/packages/workflow/src/cloudflare/owner-reads.ts +++ b/packages/workflow/src/cloudflare/owner-reads.ts @@ -28,6 +28,13 @@ import { parseDurableEvent } from "@executablemd/durable-streams"; import { readDocumentExecution, readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; import type { DocumentExecutionRecord } from "../storage/record.ts"; +import { + parseRepositoryRecord, + parseWorktreeRecord, + type RepositoryRecord, + type WorktreeRecord, +} from "../composition/records.ts"; +import { type AgentSessionRecord, parseAgentSessionRecord } from "../storage/agent-session.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { parseWorkspaceRootManifest, @@ -41,6 +48,8 @@ import { EXECUTION_PAGE_BYTES, EXECUTION_PAGE_ENTRIES, executionPageBytes, + MAX_LEDGER_BYTES, + MAX_MAPPINGS, JOURNAL_PAGE_BYTES, JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES, @@ -351,6 +360,114 @@ export function readFrontier(storage: OwnerStorage, runId: string): FrontierValu }; } +/** + * One coherent admission fact: the root, the journal anchor and every mapping. + * + * Read together, in one owner-side read, because they are one state. Taking the + * mappings from one request and the root from another would let an invocation + * begin against a Workspace whose retained Repository rows describe a different + * moment — and nothing later could notice, because each answer was true when it + * was given. + * + * Complete rather than paged. The mapping tables are insert-only, so a cursor + * over sorted names cannot be made safe by root and journal equality alone: a + * name inserted later can sort before the cursor and never be seen. A count and + * byte ceiling refuses instead, and refuses whole. + */ +export function readInvocationSnapshot( + storage: OwnerStorage, + runId: string, +): InvocationSnapshotValue { + const frontier = readFrontier(storage, runId); + const repositories = byteRows( + storage, + `SELECT name, locator, locator_fingerprint, requested_base, creation_commit, + primary_branch, object_format, checkout_path + FROM workspace_repositories ORDER BY name`, + ).map((row) => ({ + record: readRepositoryRecord(row), + locator: safeText(row, "locator"), + })); + const worktrees = byteRows( + storage, + `SELECT repository_name, name, requested_branch, requested_base, + creation_commit, checkout_path + FROM workspace_worktrees ORDER BY repository_name, name`, + ).map((row) => readWorktreeRecord(row)); + const agentSessions = byteRows( + storage, + `SELECT session_key, provider, agent_command, session_identity, policy, + assertion_kind, assertion_value, created_at + FROM agent_sessions ORDER BY session_key`, + ).map((row) => readAgentSessionRow(row)); + + const entries = repositories.length + worktrees.length + agentSessions.length; + if (entries > MAX_MAPPINGS) { + throw new CommandError("too-large"); + } + const snapshot = { + workspaceRootId: frontier.workspaceRootId, + journalEventId: frontier.journalEventId, + repositories, + worktrees, + agentSessions, + }; + // Measured over the complete semantic answer, before any of it is returned. A + // ceiling checked per record would let an aggregate no message can carry + // through one record at a time. + if (new TextEncoder().encode(JSON.stringify(snapshot)).length > MAX_LEDGER_BYTES) { + throw new CommandError("too-large"); + } + return snapshot; +} + +function readRepositoryRecord(row: Row): RepositoryRecord { + const parsed = parseRepositoryRecord({ + name: row["name"], + locatorFingerprint: row["locator_fingerprint"], + requestedBase: row["requested_base"] ?? null, + creationCommit: row["creation_commit"], + primaryBranch: row["primary_branch"], + objectFormat: row["object_format"], + checkoutPath: row["checkout_path"], + }); + if (parsed === undefined) { + return corrupt("a retained Repository row does not describe a Repository"); + } + return parsed; +} + +function readWorktreeRecord(row: Row): WorktreeRecord { + const parsed = parseWorktreeRecord({ + repositoryName: row["repository_name"], + name: row["name"], + requestedBranch: row["requested_branch"], + requestedBase: row["requested_base"] ?? null, + creationCommit: row["creation_commit"], + checkoutPath: row["checkout_path"], + }); + if (parsed === undefined) { + return corrupt("a retained Worktree row does not describe a Worktree"); + } + return parsed; +} + +function readAgentSessionRow(row: Row): AgentSessionRecord { + const parsed = parseAgentSessionRecord({ + sessionKey: row["session_key"], + provider: row["provider"], + agentCommand: row["agent_command"], + sessionIdentity: row["session_identity"], + policy: row["policy"], + assertion: { kind: row["assertion_kind"], value: row["assertion_value"] }, + createdAt: row["created_at"], + }); + if (parsed === undefined) { + return corrupt("a retained Agent session row does not describe a session"); + } + return parsed; +} + export function readJournalPage( storage: OwnerStorage, anchorEventId: string | null, @@ -472,6 +589,15 @@ function piece( } /** One page of document executions, anchored to the snapshot that began it. */ +/** The one admitted state a remote Workspace invocation begins from. */ +export interface InvocationSnapshotValue { + readonly workspaceRootId: string; + readonly journalEventId: string | null; + readonly repositories: readonly { readonly record: RepositoryRecord; readonly locator: string }[]; + readonly worktrees: readonly WorktreeRecord[]; + readonly agentSessions: readonly AgentSessionRecord[]; +} + export interface ExecutionsValue { readonly runId: string; readonly anchor: number | null; diff --git a/packages/workflow/src/deno/remote-workspace-files.ts b/packages/workflow/src/deno/remote-workspace-files.ts new file mode 100644 index 000000000..042c7c7ad --- /dev/null +++ b/packages/workflow/src/deno/remote-workspace-files.ts @@ -0,0 +1,181 @@ +/** + * The Workspace filesystem, over the attempt directory this invocation owns. + * + * The runner's Workspace is a real tree it materialized from the owner, so the + * operations are the runtime's own asynchronous primitives adapted with + * `until`. Nothing above this module names a runtime, and nothing in it decides + * anything about a Workspace: it moves bytes where it is told, and refuses to + * be told anywhere outside the attempt. + * + * `node:fs/promises` rather than a runtime global, for the same reason the + * materialization adapter uses it: the same code has to work wherever the + * runner runs. + */ + +import { + chmod, + link, + lstat, + mkdir, + readdir, + readFile, + readlink, + rename, + rm, + stat, + symlink, + writeFile, +} from "node:fs/promises"; +import { isAbsolute, relative, resolve } from "node:path"; +import { type Operation, until } from "effection"; +import type { + WorkspaceEntry, + WorkspaceFilesystem, + WorkspaceStat, +} from "../workspace/filesystem.ts"; +import { throwWorkspaceFilesystemFailure } from "./workspace/errors.ts"; +import type { HostPath } from "../remote/materialize.ts"; + +/** + * Where a logical path is allowed to land. + * + * The attempt root is the whole of what this invocation may touch. A logical + * path is resolved and then held to that root, so `..`, an absolute path and a + * path that merely starts with the root's name are each refused before any + * syscall — the host path is a place to work, never a durable identity. + */ +function within(at: HostPath, root: string, path: string): string { + const host = resolve(at(path)); + const inside = relative(root, host); + // The empty relative path is the Workspace root itself, which is a directory + // this invocation owns and may read. Only leaving the tree is refused. + if (inside.startsWith("..") || isAbsolute(inside)) { + throw new WorkspacePathError("this Workspace path is outside the tree this invocation owns."); + } + return host; +} + +/** A path no Workspace operation may reach, whatever it names. */ +export class WorkspacePathError extends Error { + override name = "WorkspacePathError"; +} + +function described(value: { + mode: number; + mtimeMs: number; + size: number; + isFile(): boolean; + isDirectory(): boolean; +}): WorkspaceStat { + const kind = value.isFile() ? "file" : value.isDirectory() ? "directory" : "symlink"; + // The retained mode is the permission bits; the type bits belong to the + // node's kind, which is reported beside it. + return { kind, mode: value.mode & 0o7777, mtime: Math.trunc(value.mtimeMs), size: value.size }; +} + +/** + * A runtime failure, named the way the shared classifier reads one. + * + * The classifier asks for a `WorkspaceFsError` carrying a documented code, + * because that is what the other host raises. Renaming here rather than + * widening the classifier keeps one list of documented conditions. + */ +function named(error: unknown): unknown { + const code = error instanceof Error ? Reflect.get(error, "code") : undefined; + if (error instanceof Error && typeof code === "string") { + const renamed = new Error(error.message, { cause: error }); + renamed.name = "WorkspaceFsError"; + Reflect.set(renamed, "code", code); + return renamed; + } + return error; +} + +export function createRemoteWorkspaceFilesystem( + at: HostPath, + authorize: () => void, +): WorkspaceFilesystem { + // The attempt's own root, taken from the same resolver every other path goes + // through. Passing it separately would let the two disagree. + const resolved = resolve(at("/")); + + function* run(path: string, body: (host: string) => Promise): Operation { + authorize(); + const host = within(at, resolved, path); + try { + return yield* until(body(host)); + } catch (error) { + // The same classification the Deno host applies: a documented filesystem + // condition is the effect's own outcome, and everything else is the run + // failing. + return throwWorkspaceFilesystemFailure(named(error)); + } + } + + return { + *readFile(path): Operation { + return yield* run(path, (host) => readFile(host)); + }, + + *readTextFile(path): Operation { + const bytes = yield* run(path, (host) => readFile(host)); + return new TextDecoder().decode(bytes); + }, + + *stat(path): Operation { + return described(yield* run(path, (host) => stat(host))); + }, + + *lstat(path): Operation { + return described(yield* run(path, (host) => lstat(host))); + }, + + *readlink(path): Operation { + return yield* run(path, (host) => readlink(host)); + }, + + *readdir(path): Operation { + const entries = yield* run(path, (host) => readdir(host, { withFileTypes: true })); + return entries.map((entry) => ({ + name: entry.name, + kind: entry.isFile() ? "file" : entry.isDirectory() ? "directory" : "symlink", + })); + }, + + *writeFile(path, content, mode): Operation { + const bytes = typeof content === "string" ? new TextEncoder().encode(content) : content; + yield* run(path, (host) => writeFile(host, bytes, mode === undefined ? {} : { mode })); + }, + + *mkdir(path, options = {}): Operation { + yield* run(path, (host) => mkdir(host, options).then(() => undefined)); + }, + + *remove(path, options = {}): Operation { + yield* run(path, (host) => rm(host, options)); + }, + + *rename(from, to): Operation { + // Both ends are held to the attempt: a rename is two paths, and checking + // one of them would let the other leave the tree. + const destination = within(at, resolved, to); + yield* run(from, (host) => rename(host, destination)); + }, + + *chmod(path, mode): Operation { + yield* run(path, (host) => chmod(host, mode)); + }, + + *symlink(target, path): Operation { + // The target is not resolved here. A symbolic link's target is retained + // exactly as written, and reading through one goes back through this + // interface, where it is held to the attempt like any other path. + yield* run(path, (host) => symlink(target, host)); + }, + + *link(existingPath, newPath): Operation { + const destination = within(at, resolved, newPath); + yield* run(existingPath, (host) => link(host, destination)); + }, + }; +} diff --git a/packages/workflow/src/deno/workspace/agent-sessions.ts b/packages/workflow/src/deno/workspace/agent-sessions.ts index 1142cc92b..e3a418153 100644 --- a/packages/workflow/src/deno/workspace/agent-sessions.ts +++ b/packages/workflow/src/deno/workspace/agent-sessions.ts @@ -39,11 +39,6 @@ import type { DatabaseSync } from "node:sqlite"; -/** A retained Agent session this host will not continue under. */ -export class WorkflowAgentSessionError extends Error { - override name = "WorkflowAgentSessionError"; -} - /** * The shape and the key derivation are the shared rule, not this adapter's. * @@ -53,13 +48,18 @@ export class WorkflowAgentSessionError extends Error { * statements, and the transaction they run in. */ import { - agentSessionKey, - type AgentSessionIdentity, type AgentSessionRecord, - type ProviderAssertion, + type AgentSessions, + WorkflowAgentSessionError, } from "../../storage/agent-session.ts"; -export { agentSessionKey, parseAgentSessionRecord } from "../../storage/agent-session.ts"; +export { + agentSessionKey, + parseAgentSessionRecord, + resolveAgentSession, + WorkflowAgentSessionError, +} from "../../storage/agent-session.ts"; +export type { AgentSessionResolution, AgentSessions } from "../../storage/agent-session.ts"; export type { AgentSessionIdentity, AgentSessionRecord, @@ -77,12 +77,6 @@ const INSERT = `INSERT INTO agent_sessions (session_key, provider, agent_command session_identity, policy, assertion_kind, assertion_value, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`; -/** Every retained mapping this run holds. Reading is not a transaction. */ -export interface AgentSessions { - read(sessionKey: string): AgentSessionRecord | undefined; - commit(record: AgentSessionRecord): void; -} - function text(value: unknown): string | undefined { return typeof value === "string" ? value : undefined; } @@ -162,82 +156,3 @@ export function createAgentSessions(database: DatabaseSync, authorize: () => voi }, }; } - -/** What a continuation may do with the session a key names. */ -export type AgentSessionResolution = - | { readonly kind: "create"; readonly sessionKey: string } - | { readonly kind: "reattach"; readonly record: AgentSessionRecord }; - -/** - * Decide what this attachment may do with the session this identity names. - * - * `asserted` is every canonical identity the provider currently asserts for that - * key — none, one, or more than one. It is deliberately not "does the provider - * hold this key": occupancy says something is there, not what conversation it - * is, and adopting one on that basis is how a run continues a session it cannot - * name. - */ -export function resolveAgentSession( - retained: AgentSessionRecord | undefined, - policy: string, - asserted: readonly ProviderAssertion[], - identity: AgentSessionIdentity, -): AgentSessionResolution { - const sessionKey = agentSessionKey(identity); - if (asserted.length > 1) { - throw new WorkflowAgentSessionError( - "the provider asserts more than one durable identity for this run's Agent session, so " + - "this host cannot tell which conversation it would be continuing. Start a new run " + - "rather than continuing this one.", - ); - } - const current = asserted[0]; - - if (retained === undefined) { - if (current === undefined) { - // Neither side holds anything: nothing was ever established here. - return { kind: "create", sessionKey }; - } - // The pre-commit window. An attempt was interrupted between the provider - // asserting an identity and this run recording it, and exactly one - // canonical assertion is what reconciles it — nothing else may. - return { - kind: "reattach", - record: { - sessionKey, - ...identity, - policy, - assertion: current, - createdAt: new Date().toISOString(), - }, - }; - } - - if ( - retained.provider !== identity.provider || - retained.agentCommand !== identity.agentCommand || - retained.sessionIdentity !== identity.sessionIdentity || - retained.policy !== policy - ) { - throw new WorkflowAgentSessionError( - "this run's Agent session was established under a different provider, agent or session " + - "policy than this host states, and a session created under one ceiling is not " + - "continued under another. Start a new run rather than continuing this one.", - ); - } - if (current === undefined) { - throw new WorkflowAgentSessionError( - "the provider asserts no durable identity for the Agent session this run retained, and " + - "this host does not reconstruct a conversation by replaying it into a new session. " + - "Start a new run rather than continuing this one.", - ); - } - if (current.kind !== retained.assertion.kind || current.value !== retained.assertion.value) { - throw new WorkflowAgentSessionError( - "the provider asserts a different durable identity than the Agent session this run " + - "retained, so it did not resume the conversation this run was having. This host does " + - "not continue under a replacement session.", - ); - } - return { kind: "reattach", record: retained }; -} diff --git a/packages/workflow/src/deno/workspace/errors.ts b/packages/workflow/src/deno/workspace/errors.ts index 571fd1fb0..7b65f09cb 100644 --- a/packages/workflow/src/deno/workspace/errors.ts +++ b/packages/workflow/src/deno/workspace/errors.ts @@ -13,31 +13,9 @@ const JOURNALABLE_CODES = new Set([ "ELOOP", ]); -/** - * A failure this effect publishes as its own durable outcome instead of raising. - * - * The distinction the effect layer needs is not "what went wrong" but "who - * this belongs to". A failure of this kind is part of what the effect *did*: it - * is written into the journal as the effect's result, the Workspace root stays - * where it was, and a replay reproduces it without performing anything. Every - * other failure is the run failing, and travels as an ordinary raise. - * - * It is a base class rather than a predicate over shapes so that being publishable - * is something a failure declares by construction. A module that wants its own - * refusal published extends this; nothing acquires the property by resembling - * something. - */ -export abstract class JournaledEffectFailure extends Error {} +export { isJournaledEffectFailure, JournaledEffectFailure } from "../../workspace/failure.ts"; -/** - * Whether this failure is the effect's outcome rather than the run's failure. - * - * Asked by the one place that has to choose between writing a result and - * letting a failure through. - */ -export function isJournaledEffectFailure(error: unknown): error is Error { - return error instanceof JournaledEffectFailure; -} +import { JournaledEffectFailure } from "../../workspace/failure.ts"; class JournalableWorkspaceFailure extends JournaledEffectFailure { override name = "WorkspaceFsError"; diff --git a/packages/workflow/src/deno/workspace/filesystem.ts b/packages/workflow/src/deno/workspace/filesystem.ts index 1ae4087a1..357adaf82 100644 --- a/packages/workflow/src/deno/workspace/filesystem.ts +++ b/packages/workflow/src/deno/workspace/filesystem.ts @@ -1,4 +1,9 @@ import { type Operation } from "effection"; +import type { + WorkspaceEntry, + WorkspaceFilesystem, + WorkspaceStat, +} from "../../workspace/filesystem.ts"; import { chmod as chmodPath } from "../../../vendor/cloudflare-computer-dofs/generated/fs/chmod.js"; import { link as linkFile } from "../../../vendor/cloudflare-computer-dofs/generated/fs/link.js"; import { mkdir as mkdirPath } from "../../../vendor/cloudflare-computer-dofs/generated/fs/mkdir.js"; @@ -17,33 +22,15 @@ import { writeFileSync } from "../../../vendor/cloudflare-computer-dofs/generate import type { RunConnection } from "../connections.ts"; import { throwWorkspaceFilesystemFailure } from "./errors.ts"; -export interface DenoWorkspaceEntry { - readonly name: string; - readonly kind: "file" | "directory" | "symlink"; -} - -export interface DenoWorkspaceStat { - readonly kind: "file" | "directory" | "symlink"; - readonly mode: number; - readonly mtime: number; - readonly size: number; -} - -export interface DenoWorkspaceFilesystem { - readFile(path: string): Operation; - readTextFile(path: string): Operation; - stat(path: string): Operation; - lstat(path: string): Operation; - readlink(path: string): Operation; - readdir(path: string): Operation; - writeFile(path: string, content: string | Uint8Array, mode?: number): Operation; - mkdir(path: string, options?: { recursive?: boolean; mode?: number }): Operation; - remove(path: string, options?: { recursive?: boolean; force?: boolean }): Operation; - rename(from: string, to: string): Operation; - chmod(path: string, mode: number): Operation; - symlink(target: string, path: string): Operation; - link(existingPath: string, newPath: string): Operation; -} +/** + * The names this host has always used, for the one shared contract. + * + * The interface moved rather than changed: this adapter is one implementation + * of it, and the runner's attempt-backed adapter is the other. + */ +export type DenoWorkspaceEntry = WorkspaceEntry; +export type DenoWorkspaceStat = WorkspaceStat; +export type DenoWorkspaceFilesystem = WorkspaceFilesystem; export function createDenoWorkspaceFilesystem( connection: RunConnection, diff --git a/packages/workflow/src/deno/workspace/repositories.ts b/packages/workflow/src/deno/workspace/repositories.ts index 82befb9a7..cb21e7326 100644 --- a/packages/workflow/src/deno/workspace/repositories.ts +++ b/packages/workflow/src/deno/workspace/repositories.ts @@ -29,12 +29,9 @@ import { type WorktreeRecord, } from "../../composition/records.ts"; import { reading } from "../reading.ts"; +import type { StoredRepository, WorkspaceMetadata } from "../../workspace/metadata.ts"; -/** A Repository row: its journal-safe record, and the locator only storage sees. */ -export interface StoredRepository { - readonly record: RepositoryRecord; - readonly locator: string; -} +export type { StoredRepository, WorkspaceMetadata } from "../../workspace/metadata.ts"; const REPOSITORY_COLUMNS = `name, locator, locator_fingerprint, requested_base, creation_commit, primary_branch, object_format, checkout_path`; @@ -195,23 +192,6 @@ export function insertWorktree(database: DatabaseSync, record: WorktreeRecord): ); } -/** - * The metadata one Workspace transaction may read and write. - * - * Handed to a mutation beside the filesystem, so retained Git identity and - * retained Git bytes move together inside one transaction. It is the provider's - * surface and not a document's: a component reaches it only by asking the - * composition provider to perform an effect. - */ -export interface WorkspaceMetadata { - readRepository(name: string): StoredRepository | undefined; - readRepositories(): StoredRepository[]; - insertRepository(stored: StoredRepository): void; - readWorktree(repositoryName: string, name: string): WorktreeRecord | undefined; - readWorktreesForRepository(repositoryName: string): WorktreeRecord[]; - insertWorktree(record: WorktreeRecord): void; -} - export function createWorkspaceMetadata( database: DatabaseSync, authorize: () => void, diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index f3f613ee8..6803ee1f2 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -153,7 +153,11 @@ export function requireNoOpenTransaction(gate: TransactionGate): void { export function transactRemotely( link: OwnerLink, gate: TransactionGate, - body: (transaction: WorkflowRunTransaction, enlist: EnlistWorkspace) => Operation, + body: ( + transaction: WorkflowRunTransaction, + enlist: EnlistWorkspace, + anchor: TransactionAnchor, + ) => Operation, ): Operation> { return call(function* (): Operation> { // Taken synchronously, before the first suspension. Checking and then @@ -238,7 +242,12 @@ export function transactRemotely( // finish" are one statement. `call()` alone would let a resource whose // teardown fails surface its failure after the commit had already gone // out, which is the one ordering that cannot be taken back. - outcome = yield* scoped(() => body({ journal }, enlist)); + outcome = yield* scoped(() => + body({ journal }, enlist, { + workspaceRootId: starting.workspaceRootId, + journalEventId: starting.journalEventId, + }), + ); } finally { // The handle is closed before the commit goes out, so a retained // transaction object refuses while the handle-level gate is still held. @@ -274,6 +283,19 @@ export function transactRemotely( }); } +/** + * Exactly where this transaction began, and nothing else about it. + * + * A coordinator has to prove that the state it admitted its invocation from is + * the state this transaction will commit against. It needs the two anchors for + * that and no more — the journal prefix is the body's to read through the + * transaction, not something a route hands out. + */ +export interface TransactionAnchor { + readonly workspaceRootId: string; + readonly journalEventId: string | null; +} + /** * How a Workspace operation designates its attempt for publication. * diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 2607346f3..74282936f 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -49,7 +49,7 @@ import type { WorkflowRunRecord, } from "../storage/record.ts"; import { createTransactionGate, type OwnerLink, transactRemotely } from "./collector.ts"; -import type { EnlistWorkspace } from "./collector.ts"; +import type { EnlistWorkspace, TransactionAnchor } from "./collector.ts"; import type { RemoteFrontierSnapshot } from "./read.ts"; /** What a remote handle needs to answer everything the interface asks. */ @@ -105,6 +105,8 @@ export interface WorkspaceRoute { readonly database: WorkflowRunDatabase; readonly transaction: WorkflowRunTransaction; readonly enlist: EnlistWorkspace; + /** Where this transaction began, so a coordinator can prove it has not drifted. */ + readonly anchor: TransactionAnchor; } const ActiveRoute: Context = createContext( @@ -274,7 +276,7 @@ export function useRemoteRunDatabase( } return yield* turns.take(function* (): Operation> { try { - return yield* transactRemotely(link, gate, function* (transaction, enlist) { + return yield* transactRemotely(link, gate, function* (transaction, enlist, anchor) { // The marker and the route are installed for the body's scope // alone. Outside it neither exists, so a retained transaction // object reaches nothing and an unrelated scope is not mistaken for @@ -283,7 +285,7 @@ export function useRemoteRunDatabase( handle, enclosing: yield* ActiveTransaction.get(), }); - yield* ActiveRoute.set({ database: handle, transaction, enlist }); + yield* ActiveRoute.set({ database: handle, transaction, enlist, anchor }); return yield* body(transaction); }); } catch (error) { diff --git a/packages/workflow/src/remote/journal-route.ts b/packages/workflow/src/remote/journal-route.ts new file mode 100644 index 000000000..52755e988 --- /dev/null +++ b/packages/workflow/src/remote/journal-route.ts @@ -0,0 +1,94 @@ +/** + * Where a Workspace effect's publication goes. + * + * A durable operation publishes its result into the run's journal. When a + * Workspace effect is the thing publishing, that append has to land in the + * exact transaction the effect ran inside, so the Files change and the row + * describing it commit together or not at all. The ordinary journal would + * append outside the transaction, which is the one ordering that cannot be + * taken back. + * + * So the route is installed for one transaction's descendant scope, keyed to + * one exact database, transaction and token. Outside that scope the wrapper + * falls through to the ordinary journal, and a retained token reaches nothing. + */ + +import { type Api, createApi } from "@effectionx/context-api"; +import type { DurableEvent, DurableStream } from "@executablemd/durable-streams"; +import { ensure, type Operation, scoped } from "effection"; +import type { WorkflowRunDatabase, WorkflowRunTransaction } from "../storage/api.ts"; + +interface JournalDestinationApi { + append(database: WorkflowRunDatabase, event: DurableEvent): Operation; +} + +const RemoteJournalDestination: Api = createApi( + "executablemd.workflow.remote.journal.destination", + { + // deno-lint-ignore require-yield + *append(): Operation { + return false; + }, + }, +); + +/** + * Run `publication` with this exact transaction as the journal's destination. + * + * The transaction object is the capability. Only code inside the live + * transaction body holds one, and the caller has already proved through + * `activeWorkspaceRoute()` that this is that transaction — so there is nothing + * further to look up, and no registry to outlive the run. + * + * Scoped, so the redirection ends with the operation that needed it rather than + * outliving the transaction it names. `live` closes with that scope: an append + * arriving afterwards falls through to the ordinary journal instead of reaching + * a transaction that has closed. + */ +export function withRemoteJournalRoute( + database: WorkflowRunDatabase, + transaction: WorkflowRunTransaction, + publication: Operation, +): Operation { + return scoped(function* () { + let live = true; + yield* ensure(() => { + live = false; + }); + yield* RemoteJournalDestination.around( + { + *append([candidate, event], next): Operation { + if (candidate !== database || !live) { + return yield* next(candidate, event); + } + yield* transaction.journal.append(event); + return true; + }, + }, + { at: "min" }, + ); + return yield* publication; + }); +} + +/** + * The run's journal, willing to be redirected into an open transaction. + * + * Reads always come from the ordinary journal: what a transaction has appended + * is read back through the transaction itself, and a reader outside it is + * asking about committed history. + */ +export function routeRemoteRunJournal( + database: WorkflowRunDatabase, + ordinary: DurableStream, +): DurableStream { + return { + readAll: () => ordinary.readAll(), + + *append(event: DurableEvent): Operation { + if (!(yield* RemoteJournalDestination.operations.append(database, event))) { + yield* ordinary.append(event); + } + }, + }; +} diff --git a/packages/workflow/src/remote/mappings.ts b/packages/workflow/src/remote/mappings.ts new file mode 100644 index 000000000..495c858e2 --- /dev/null +++ b/packages/workflow/src/remote/mappings.ts @@ -0,0 +1,255 @@ +/** + * Retained mappings, as one invocation on the runner sees them. + * + * The runner has no database. What it has is the coherent snapshot the owner + * admitted this invocation from, and whatever this invocation has staged since. + * That is enough to answer every question the shared composition rules ask, + * because those rules only ever read a mapping back and compare it — and this + * answers with the retained row when there is one, and with what this + * invocation staged when there is not. + * + * Read-your-writes without durability. A document that creates a Repository and + * then asks for it again is asking about its own work, and must see it; nothing + * about that makes it committed. Only the exact list handed through the live + * enlistment capability reaches an intent, and only the owner's transaction + * makes any of it authoritative. + * + * The reconciliation rules are not restated here. A same-name Repository is + * compared by the composition provider that already knows what compatible + * means, and an Agent session by `resolveAgentSession()`; this module decides + * only where a record comes from and what a new one stages. + */ + +import { + type AgentSessionRecord, + type AgentSessions, + WorkflowAgentSessionError, +} from "../storage/agent-session.ts"; +import type { WorktreeRecord } from "../composition/records.ts"; +import { parseCheckoutPath } from "../composition/records.ts"; +import { locatorFingerprintOf } from "../composition/locator.ts"; +import type { StoredRepository, WorkspaceMetadata } from "../workspace/metadata.ts"; +import type { RemoteInvocationSnapshot } from "./records.ts"; +import type { RetainedMapping } from "./publication.ts"; +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; + +/** The most mappings one invocation may stage before it is refused. */ +const MAX_STAGED_MAPPINGS = 256; + +/** The most serialized bytes one invocation may stage before it is refused. */ +const MAX_STAGED_BYTES = 256 * 1024; + +/** + * What an invocation may reach, and what it has decided to retain. + * + * `deltas()` is the whole of what may be enlisted. It is a fresh array each + * time, deterministically ordered, so the caller cannot reach back into what + * the view is still holding. + */ +export interface InvocationMappings { + readonly metadata: WorkspaceMetadata; + readonly agentSessions: AgentSessions; + deltas(): readonly RetainedMapping[]; +} + +function refuse(reason: string): never { + throw new WorkflowRecordMalformedError("this run's retained mappings", reason); +} + +/** + * A copy that shares nothing with what it was given. + * + * These records are handed to a document and staged for a commit, and both hold + * them for longer than the call. A structural clone is what makes "the delta is + * what was staged" true rather than a description of what nobody mutated. + */ +function detach(value: T): T { + return structuredClone(value) as T; +} + +export function createInvocationMappings( + snapshot: RemoteInvocationSnapshot, + live: () => void, +): InvocationMappings { + const repositories = new Map(); + const worktrees = new Map(); + const sessions = new Map(); + for (const stored of snapshot.repositories) { + repositories.set(stored.record.name, stored); + } + for (const record of snapshot.worktrees) { + worktrees.set(worktreeKey(record.repositoryName, record.name), record); + } + for (const record of snapshot.agentSessions) { + sessions.set(record.sessionKey, record); + } + + const staged: RetainedMapping[] = []; + function stage(mapping: RetainedMapping): void { + if (staged.length >= MAX_STAGED_MAPPINGS) { + refuse("this invocation stages more retained mappings than one commit may carry"); + } + const next = [...staged, mapping]; + if (new TextEncoder().encode(JSON.stringify(next)).length > MAX_STAGED_BYTES) { + refuse("this invocation stages more retained mapping bytes than one commit may carry"); + } + staged.push(mapping); + } + + return { + metadata: { + readRepository(name: string): StoredRepository | undefined { + live(); + const found = repositories.get(name); + return found === undefined ? undefined : detach(found); + }, + + readRepositories(): StoredRepository[] { + live(); + return [...repositories.values()] + .toSorted((left, right) => compare(left.record.name, right.record.name)) + .map(detach); + }, + + insertRepository(stored: StoredRepository): void { + live(); + if (locatorFingerprintOf(stored.locator) !== stored.record.locatorFingerprint) { + refuse("a Repository was retained with a fingerprint its locator does not produce"); + } + if (parseCheckoutPath(stored.record.checkoutPath) === undefined) { + refuse("a Repository was retained with a checkout path this build does not admit"); + } + const existing = repositories.get(stored.record.name); + if (existing !== undefined) { + // The same insert twice in one invocation is the same fact stated + // twice. Anything else is a conflict, and a conflict never replaces + // what is already there. + if (!sameRepository(existing, stored)) { + refuse("a Repository name was retained twice under different identities"); + } + // Either the owner already holds it, or this invocation staged it + // earlier. Both mean there is nothing new to retain: a snapshot row + // re-sent as a mutation would ask the owner to insert what it has. + return; + } + const admitted = detach(stored); + repositories.set(admitted.record.name, admitted); + stage({ kind: "repository", record: admitted.record, locator: admitted.locator }); + }, + + readWorktree(repositoryName: string, name: string): WorktreeRecord | undefined { + live(); + const found = worktrees.get(worktreeKey(repositoryName, name)); + return found === undefined ? undefined : detach(found); + }, + + readWorktreesForRepository(repositoryName: string): WorktreeRecord[] { + live(); + return [...worktrees.values()] + .filter((record) => record.repositoryName === repositoryName) + .toSorted((left, right) => compare(left.name, right.name)) + .map(detach); + }, + + insertWorktree(record: WorktreeRecord): void { + live(); + if (parseCheckoutPath(record.checkoutPath) === undefined) { + refuse("a Worktree was retained with a checkout path this build does not admit"); + } + if (!repositories.has(record.repositoryName)) { + refuse("a Worktree was retained for a Repository this run does not hold"); + } + const key = worktreeKey(record.repositoryName, record.name); + const existing = worktrees.get(key); + if (existing !== undefined) { + if (!sameWorktree(existing, record)) { + refuse("a Worktree name was retained twice under different identities"); + } + return; + } + const admitted = detach(record); + worktrees.set(key, admitted); + stage({ kind: "worktree", record: admitted }); + }, + }, + + agentSessions: { + read(sessionKey: string): AgentSessionRecord | undefined { + live(); + const found = sessions.get(sessionKey); + return found === undefined ? undefined : detach(found); + }, + + commit(record: AgentSessionRecord): void { + live(); + const existing = sessions.get(record.sessionKey); + if (existing !== undefined) { + if (!sameSession(existing, record)) { + throw new WorkflowAgentSessionError( + "this run already retains a different Agent session under this identity, and a " + + "session established under one ceiling is not continued under another.", + ); + } + return; + } + const admitted = detach(record); + sessions.set(admitted.sessionKey, admitted); + stage({ kind: "agent-session", record: admitted }); + }, + }, + + deltas(): readonly RetainedMapping[] { + // Parents before children, then by name: the owner applies them in + // dependency order, and a deterministic list is what makes one + // invocation's proposal the same proposal on a retry. + const order = { repository: 0, worktree: 1, "agent-session": 2 } as const; + return staged + .map((mapping, index) => ({ mapping, index })) + .toSorted((left, right) => { + const kinds = order[left.mapping.kind] - order[right.mapping.kind]; + return kinds !== 0 ? kinds : left.index - right.index; + }) + .map((entry) => detach(entry.mapping)); + }, + }; +} + +function worktreeKey(repositoryName: string, name: string): string { + return `${repositoryName}\u0000${name}`; +} + +function compare(left: string, right: string): number { + return left < right ? -1 : left > right ? 1 : 0; +} + +function sameRepository(left: StoredRepository, right: StoredRepository): boolean { + return ( + left.locator === right.locator && + left.record.locatorFingerprint === right.record.locatorFingerprint && + left.record.requestedBase === right.record.requestedBase && + left.record.creationCommit === right.record.creationCommit && + left.record.primaryBranch === right.record.primaryBranch && + left.record.objectFormat === right.record.objectFormat && + left.record.checkoutPath === right.record.checkoutPath + ); +} + +function sameWorktree(left: WorktreeRecord, right: WorktreeRecord): boolean { + return ( + left.requestedBranch === right.requestedBranch && + left.requestedBase === right.requestedBase && + left.creationCommit === right.creationCommit && + left.checkoutPath === right.checkoutPath + ); +} + +function sameSession(left: AgentSessionRecord, right: AgentSessionRecord): boolean { + return ( + left.provider === right.provider && + left.agentCommand === right.agentCommand && + left.sessionIdentity === right.sessionIdentity && + left.policy === right.policy && + left.assertion.kind === right.assertion.kind && + left.assertion.value === right.assertion.value + ); +} diff --git a/packages/workflow/src/remote/read.ts b/packages/workflow/src/remote/read.ts index 273bb02bd..cd2870968 100644 --- a/packages/workflow/src/remote/read.ts +++ b/packages/workflow/src/remote/read.ts @@ -19,6 +19,7 @@ */ import type { Operation } from "effection"; +import type { RemoteInvocationSnapshot } from "./records.ts"; import type { JournalEntry } from "../storage/api.ts"; import type { DefinitionRetrieval, WorkflowRunRecord } from "../storage/record.ts"; import type { WorkspaceRootManifest } from "../workspace/root-manifest.ts"; @@ -44,6 +45,8 @@ export type RemoteContentRequest = export interface RemoteReadLink { frontier(): Operation; + /** The one coherent admitted state a Workspace invocation begins from. */ + invocationSnapshot(): Operation; root(workspaceRootId: string): Operation; content(workspaceRootId: string, request: RemoteContentRequest): Operation; } diff --git a/packages/workflow/src/remote/records.ts b/packages/workflow/src/remote/records.ts index 7897720d3..3bb3383c8 100644 --- a/packages/workflow/src/remote/records.ts +++ b/packages/workflow/src/remote/records.ts @@ -35,6 +35,14 @@ import { type WorkflowRunRecord, } from "../storage/record.ts"; import { SHA256 } from "../workspace/root-manifest.ts"; +import { admitLocator, locatorFingerprintOf } from "../composition/locator.ts"; +import { + parseRepositoryRecord, + parseWorktreeRecord, + type WorktreeRecord, +} from "../composition/records.ts"; +import { type AgentSessionRecord, parseAgentSessionRecord } from "../storage/agent-session.ts"; +import type { StoredRepository } from "../workspace/metadata.ts"; export class RemoteRecordError extends Error { override name = "RemoteRecordError"; @@ -193,3 +201,134 @@ export function parseRemoteExecution(value: unknown): DocumentExecutionRecord { stopReason: parseWorkflowStopReason(found.get("stopReason"), "$.stopReason", fail), }); } + +/** + * One admitted invocation snapshot, as the runner is allowed to read it. + * + * The root and journal anchor travel with the mappings because they are one + * fact, and the runner holds the whole answer to that: before a document runs, + * the transaction it runs inside has to start from exactly this root and this + * anchor. + */ +export interface RemoteInvocationSnapshot { + readonly workspaceRootId: string; + readonly journalEventId: string | null; + readonly repositories: readonly StoredRepository[]; + readonly worktrees: readonly WorktreeRecord[]; + readonly agentSessions: readonly AgentSessionRecord[]; +} + +/** The most mapping entries one admitted snapshot may carry. */ +const MAX_SNAPSHOT_ENTRIES = 256; + +export function parseRemoteInvocationSnapshot(value: unknown): RemoteInvocationSnapshot { + const found = parseMembers(value, "$", fail); + requireMemberNames( + found, + ["workspaceRootId", "journalEventId", "repositories", "worktrees", "agentSessions"], + "$", + fail, + ); + const workspaceRootId = parseStringMember(found, "workspaceRootId", "$", fail); + if (!SHA256.test(workspaceRootId)) { + throw fail("expected a Workspace root identity", "$.workspaceRootId"); + } + const anchor = found.get("journalEventId"); + if (anchor !== null && (typeof anchor !== "string" || anchor === "")) { + throw fail("expected a journal event identity or an explicit empty anchor", "$.journalEventId"); + } + + const repositories = list(found.get("repositories"), "$.repositories").map((entry, index) => + parseStoredRepository(entry, `$.repositories[${index}]`), + ); + const worktrees = list(found.get("worktrees"), "$.worktrees").map((entry, index) => + admitted(parseWorktreeRecord(entry), `$.worktrees[${index}]`, "a Worktree"), + ); + const agentSessions = list(found.get("agentSessions"), "$.agentSessions").map((entry, index) => + admitted(parseAgentSessionRecord(entry), `$.agentSessions[${index}]`, "an Agent session"), + ); + + if (repositories.length + worktrees.length + agentSessions.length > MAX_SNAPSHOT_ENTRIES) { + throw fail("expected fewer retained mappings than one snapshot may carry", "$"); + } + requireOrdered( + repositories.map((stored) => stored.record.name), + "$.repositories", + ); + requireOrdered( + worktrees.map((record) => `${record.repositoryName} ${record.name}`), + "$.worktrees", + ); + requireOrdered( + agentSessions.map((record) => record.sessionKey), + "$.agentSessions", + ); + // Every Worktree names a Repository this snapshot also carries. A checkout + // whose Repository is missing is not a state this run was ever in. + const names = new Set(repositories.map((stored) => stored.record.name)); + for (const [index, record] of worktrees.entries()) { + if (!names.has(record.repositoryName)) { + throw fail( + "expected a Worktree whose Repository this snapshot holds", + `$.worktrees[${index}]`, + ); + } + } + return Object.freeze({ + workspaceRootId, + journalEventId: anchor === null ? null : anchor, + repositories: Object.freeze(repositories), + worktrees: Object.freeze(worktrees), + agentSessions: Object.freeze(agentSessions), + }); +} + +function list(value: unknown, path: string): unknown[] { + if (!Array.isArray(value)) { + throw fail("expected an array", path); + } + return value; +} + +function admitted(parsed: T | undefined, path: string, expectation: string): T { + if (parsed === undefined) { + throw fail(`expected ${expectation}`, path); + } + return parsed; +} + +/** + * Deterministic and without repeats, checked rather than assumed. + * + * The owner reads these in one order; a snapshot that arrived in another, or + * twice under one name, is not the state it claims to describe — and a mapping + * view built from it would answer differently depending on which copy it read. + */ +function requireOrdered(keys: readonly string[], path: string): void { + for (const [index, key] of keys.entries()) { + const previous = keys[index - 1]; + if (previous !== undefined && previous >= key) { + throw fail("expected retained mappings in one deterministic order, without repeats", path); + } + } +} + +function parseStoredRepository(value: unknown, path: string): StoredRepository { + const found = parseMembers(value, path, fail); + requireMemberNames(found, ["record", "locator"], path, fail); + const locator = parseStringMember(found, "locator", path, fail); + // Admitted by the same rule the local host admits one by, so a locator this + // build would refuse to use never becomes one it reconciles against. + if (admitLocator(locator) === undefined) { + throw fail("expected a Repository locator this build admits", `${path}.locator`); + } + const record = admitted( + parseRepositoryRecord(found.get("record")), + `${path}.record`, + "a Repository", + ); + if (locatorFingerprintOf(locator) !== record.locatorFingerprint) { + throw fail("expected a locator the record's fingerprint follows from", `${path}.locator`); + } + return Object.freeze({ record, locator }); +} diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts new file mode 100644 index 000000000..b261fdc24 --- /dev/null +++ b/packages/workflow/src/remote/workspace.ts @@ -0,0 +1,366 @@ +/** + * Running Workspace work on the runner, against a run the owner holds. + * + * The Deno coordinator opens a transaction, hands a mutation the authoritative + * filesystem and the retained metadata, and commits both together. This is the + * same shape with the storage somewhere else: the Workspace is a real directory + * this invocation materialized from the exact admitted root, the metadata is a + * detached snapshot of that same admitted state, and "commit" is one intent the + * owner performs atomically or not at all. + * + * The ordering is the whole of the correctness argument, so it is written out + * rather than implied: + * + * 1. The execution identity is claimed by one database before any effect is + * created, so a foreign or reused one cannot coordinate. + * 2. One coherent snapshot is admitted: root, journal anchor and mappings + * together, because they are one state. + * 3. The accepted tree and the disposable attempt are created *outside* the + * transaction, because the collector seals the attempt after the transaction + * body has torn down — an attempt scoped to the body would be gone by then. + * 4. Inside the exact transaction callback, the route proves it starts from the + * admitted state. Drift refuses here, before the document runs and before + * anything is sent. + * 5. The document runs once. A documented Workspace failure is the effect's own + * result and journals against the unchanged root; everything else is the run + * failing and publishes nothing. + * 6. Only a successful result enlists the attempt and its staged deltas, and + * the publication is routed into this exact transaction's journal. + * 7. The collector seals, sends, and transfers the tree only on the exact + * performed answer. + */ + +import { type Api, createApi } from "@effectionx/context-api"; +import { + createOwnedDurableWorkspaceOperation, + type WorkspaceCoordinationAuthority, + type WorkspaceCoordinationProvider, + withWorkspaceCoordinationProvider, +} from "../workspace/effect.ts"; +import { + type DurableEffect, + type EffectDescription, + type Json, + type JournalProvenance, + type Result as DurableResult, + serializeError, +} from "@executablemd/durable-streams"; +import { ensure, type Operation, scoped } from "effection"; +import type { WorkflowRunDatabase, WorkflowRunTransaction } from "../storage/api.ts"; +import { WorkflowTransactionError } from "../storage/errors.ts"; +import type { WorkspaceFilesystem } from "../workspace/filesystem.ts"; +import { isJournaledEffectFailure } from "../workspace/failure.ts"; +import type { WorkspaceMetadata } from "../workspace/metadata.ts"; +import type { AgentSessions } from "../storage/agent-session.ts"; +import { activeWorkspaceRoute, type WorkspaceRoute } from "./database.ts"; +import { createInvocationMappings } from "./mappings.ts"; +import { + type Attempt, + type Materialization, + useAttempt, + useMaterialization, +} from "./invocation.ts"; +import type { HostPath, RunnerFiles } from "./materialize.ts"; +import type { RemoteReadLink } from "./read.ts"; +import type { RemoteInvocationSnapshot } from "./records.ts"; +import { withRemoteJournalRoute } from "./journal-route.ts"; +import type { TemporaryTrees } from "./invocation.ts"; + +/** + * What a Workspace mutation is given. + * + * The same two things the Deno coordinator hands one, plus the Agent-session + * mappings, so one contract describes both hosts' work. + */ +export type RemoteWorkspaceMutation = ( + filesystem: WorkspaceFilesystem, + metadata: WorkspaceMetadata, + agentSessions: AgentSessions, +) => Operation; + +/** + * The host facts the coordinator cannot know. + * + * Where temporary trees come from, how bytes are written, and how a Workspace + * filesystem is built over a directory. Supplied by a runtime-named adapter, so + * nothing here names a host. + */ +export interface RemoteWorkspaceRuntime { + readonly files: RunnerFiles; + readonly trees: TemporaryTrees; + readonly reads: RemoteReadLink; + createFilesystem(at: HostPath, authorize: () => void): WorkspaceFilesystem; +} + +interface WorkspaceMutationApi { + run( + database: WorkflowRunDatabase, + mutate: RemoteWorkspaceMutation, + ): Operation; +} + +function unavailable(reason: string): never { + throw new WorkflowTransactionError(reason); +} + +const WorkspaceMutation: Api = createApi( + "executablemd.workflow.remote.workspace.effect.mutation", + { + // deno-lint-ignore require-yield + *run(): Operation { + return unavailable( + "the Workspace effect is not bound to an active remote WorkflowRun transaction.", + ); + }, + }, +); + +/** + * Which database claimed which execution identity. + * + * A `WeakMap` keyed by the identity object, so the claim is the object itself + * rather than anything written down. A second loaded copy of this module has + * its own map and its own identities, and neither can answer for the other's. + */ +const workspaceEffectOwners = (() => { + const owners = new WeakMap(); + return { + claim(identity: object, database: WorkflowRunDatabase): void { + owners.set(identity, database); + }, + get(identity: object): WorkflowRunDatabase | undefined { + return owners.get(identity); + }, + }; +})(); + +interface Registration { + open: boolean; + readonly runtime: RemoteWorkspaceRuntime; + readonly provenance: JournalProvenance; +} + +interface ProviderApi { + readonly provider: object | undefined; +} + +const RemoteWorkspaceProvider: Api = createApi( + "executablemd.workflow.remote.workspace.effect.provider", + { provider: undefined }, +); + +const registrations = (() => { + const held = new WeakMap(); + return { + register(runtime: RemoteWorkspaceRuntime, provenance: JournalProvenance) { + const selection = Object.freeze({}); + const registration: Registration = { open: true, runtime, provenance }; + held.set(selection, registration); + return { + selection, + close(): void { + registration.open = false; + held.delete(selection); + }, + }; + }, + get(selection: object): Registration | undefined { + const registration = held.get(selection); + return registration?.open === true ? registration : undefined; + }, + }; +})(); + +/** + * Install the runner's Workspace coordination for this scope. + * + * The provenance is the one the run's journal was established with. It is held + * here rather than compared structurally, because two journals can describe the + * same events and only one of them is this run's. + */ +export function* useRemoteWorkspaceEffects( + runtime: RemoteWorkspaceRuntime, + provenance: JournalProvenance, +): Operation { + const registration = registrations.register(runtime, provenance); + yield* ensure(registration.close); + yield* RemoteWorkspaceProvider.around({ provider: () => registration.selection }, { at: "min" }); +} + +export function withRemoteWorkspaceEffects( + database: WorkflowRunDatabase, + operation: Operation, +): Operation { + return scoped(function* () { + const selection = yield* RemoteWorkspaceProvider.operations.provider; + const registration = selection === undefined ? undefined : registrations.get(selection); + if (registration === undefined) { + return unavailable("no remote Workspace coordinator is installed for this run."); + } + return yield* withWorkspaceCoordinationProvider(coordinator(database, registration), operation); + }); +} + +export function createRemoteWorkspaceEffect( + database: WorkflowRunDatabase, + description: EffectDescription, + mutate: RemoteWorkspaceMutation, +): DurableEffect { + const execute = () => WorkspaceMutation.operations.run(database, mutate); + const executionIdentity = Object.freeze({}); + workspaceEffectOwners.claim(executionIdentity, database); + return createOwnedDurableWorkspaceOperation(description, execute, executionIdentity); +} + +function coordinator( + database: WorkflowRunDatabase, + registration: Registration, +): WorkspaceCoordinationProvider { + return { + *run(authority: WorkspaceCoordinationAuthority): Operation { + let transacted; + try { + if (workspaceEffectOwners.get(authority.executionIdentity) !== database) { + unavailable( + "the live Workspace effect is missing, foreign, completed, or stale for this " + + "WorkflowRun database.", + ); + } + if ( + authority.journalProvenance === undefined || + authority.journalProvenance !== registration.provenance + ) { + unavailable( + "the live Workspace journal does not have the provenance of the selected WorkflowRun.", + ); + } + transacted = yield* invoke(database, registration, authority); + } catch (error) { + throw yield* authority.activateFailure(error); + } + if (!transacted.ok) { + throw yield* authority.activateFailure(transacted.error); + } + return transacted.value; + }, + }; +} + +function* invoke( + database: WorkflowRunDatabase, + registration: Registration, + authority: WorkspaceCoordinationAuthority, +) { + const { runtime } = registration; + const reject = (reason: string): never => unavailable(reason); + const snapshot = yield* runtime.reads.invocationSnapshot(); + + // Outside the transaction, deliberately. The collector seals the attempt + // after the transaction body and everything it started have torn down, so an + // attempt owned by the body would already be gone when its proposal is taken. + const materialization: Materialization = yield* useMaterialization( + runtime.files, + runtime.trees, + runtime.reads, + snapshot.workspaceRootId, + reject, + ); + const attempt: Attempt = yield* useAttempt( + runtime.files, + runtime.trees, + runtime.reads, + materialization, + reject, + ); + + return yield* database.transact(function* (transaction) { + const route = yield* activeWorkspaceRoute(database, transaction); + if (route === undefined) { + unavailable( + "the live Workspace coordinator is not inside this WorkflowRun's active transaction.", + ); + } + // Before the document runs, and before anything is sent. If the run moved + // between the snapshot and this transaction, everything admitted describes + // a state this commit would not be against. + if ( + route.anchor.workspaceRootId !== snapshot.workspaceRootId || + route.anchor.journalEventId !== snapshot.journalEventId + ) { + unavailable( + "this Workspace invocation was admitted from a state this run has since moved past.", + ); + } + return yield* coordinateTransaction( + database, + transaction, + route, + runtime, + snapshot, + attempt, + authority, + ); + }); +} + +function* coordinateTransaction( + database: WorkflowRunDatabase, + transaction: WorkflowRunTransaction, + route: WorkspaceRoute, + runtime: RemoteWorkspaceRuntime, + snapshot: RemoteInvocationSnapshot, + attempt: Attempt, + authority: WorkspaceCoordinationAuthority, +): Operation { + return yield* scoped(function* () { + let live = true; + // The capabilities exist while this invocation does and no longer. A + // filesystem or mapping view captured for later is asking about a + // Workspace that has already been committed or discarded. + yield* ensure(() => { + live = false; + }); + const authorize = (): void => { + if (!live) { + unavailable("this Workspace capability is completed, cancelled, or stale."); + } + }; + const mappings = createInvocationMappings(snapshot, authorize); + const filesystem = runtime.createFilesystem(attempt.at, authorize); + + let result: DurableResult; + try { + const value = yield* scoped(function* () { + yield* WorkspaceMutation.around( + { + *run([candidate, mutate]): Operation { + if (candidate !== database) { + unavailable( + "the Workspace effect is not bound to an active remote WorkflowRun transaction.", + ); + } + return yield* mutate(filesystem, mappings.metadata, mappings.agentSessions); + }, + }, + { at: "min" }, + ); + return yield* authority.execute(); + }); + result = { status: "ok", value }; + // Only a successful result publishes a Workspace. The attempt is named + // rather than captured: the collector seals it after this body tears + // down, so what the owner decides is the tree as it finally is. + route.enlist(attempt, mappings.deltas()); + } catch (error) { + if (!isJournaledEffectFailure(error)) { + throw error; + } + // The effect's own outcome. Nothing is enlisted, so the commit carries + // only this row and the root stays exactly where it was. + result = { status: "err", error: serializeError(error) }; + } + + yield* withRemoteJournalRoute(database, transaction, authority.publish(result)); + return result; + }); +} diff --git a/packages/workflow/src/storage/agent-session.ts b/packages/workflow/src/storage/agent-session.ts index 2bba915aa..5fd371494 100644 --- a/packages/workflow/src/storage/agent-session.ts +++ b/packages/workflow/src/storage/agent-session.ts @@ -123,3 +123,93 @@ export function parseAgentSessionRecord(value: unknown): AgentSessionRecord | un createdAt, }); } + +/** A retained Agent session this host will not continue under. */ +export class WorkflowAgentSessionError extends Error { + override name = "WorkflowAgentSessionError"; +} + +/** Every retained mapping one run holds, as a coordinator may reach it. */ +export interface AgentSessions { + read(sessionKey: string): AgentSessionRecord | undefined; + commit(record: AgentSessionRecord): void; +} + +/** What a continuation may do with the session a key names. */ +export type AgentSessionResolution = + | { readonly kind: "create"; readonly sessionKey: string } + | { readonly kind: "reattach"; readonly record: AgentSessionRecord }; + +/** + * Decide what this attachment may do with the session this identity names. + * + * `asserted` is every canonical identity the provider currently asserts for that + * key — none, one, or more than one. It is deliberately not "does the provider + * hold this key": occupancy says something is there, not what conversation it + * is, and adopting one on that basis is how a run continues a session it cannot + * name. + */ +export function resolveAgentSession( + retained: AgentSessionRecord | undefined, + policy: string, + asserted: readonly ProviderAssertion[], + identity: AgentSessionIdentity, +): AgentSessionResolution { + const sessionKey = agentSessionKey(identity); + if (asserted.length > 1) { + throw new WorkflowAgentSessionError( + "the provider asserts more than one durable identity for this run's Agent session, so " + + "this host cannot tell which conversation it would be continuing. Start a new run " + + "rather than continuing this one.", + ); + } + const current = asserted[0]; + + if (retained === undefined) { + if (current === undefined) { + // Neither side holds anything: nothing was ever established here. + return { kind: "create", sessionKey }; + } + // The pre-commit window. An attempt was interrupted between the provider + // asserting an identity and this run recording it, and exactly one + // canonical assertion is what reconciles it — nothing else may. + return { + kind: "reattach", + record: { + sessionKey, + ...identity, + policy, + assertion: current, + createdAt: new Date().toISOString(), + }, + }; + } + + if ( + retained.provider !== identity.provider || + retained.agentCommand !== identity.agentCommand || + retained.sessionIdentity !== identity.sessionIdentity || + retained.policy !== policy + ) { + throw new WorkflowAgentSessionError( + "this run's Agent session was established under a different provider, agent or session " + + "policy than this host states, and a session created under one ceiling is not " + + "continued under another. Start a new run rather than continuing this one.", + ); + } + if (current === undefined) { + throw new WorkflowAgentSessionError( + "the provider asserts no durable identity for the Agent session this run retained, and " + + "this host does not reconstruct a conversation by replaying it into a new session. " + + "Start a new run rather than continuing this one.", + ); + } + if (current.kind !== retained.assertion.kind || current.value !== retained.assertion.value) { + throw new WorkflowAgentSessionError( + "the provider asserts a different durable identity than the Agent session this run " + + "retained, so it did not resume the conversation this run was having. This host does " + + "not continue under a replacement session.", + ); + } + return { kind: "reattach", record: retained }; +} diff --git a/packages/workflow/src/workspace/failure.ts b/packages/workflow/src/workspace/failure.ts new file mode 100644 index 000000000..d13f65df7 --- /dev/null +++ b/packages/workflow/src/workspace/failure.ts @@ -0,0 +1,28 @@ +/** + * A failure the Workspace effect publishes instead of raising. + * + * The distinction is not "what went wrong" but "who this belongs to". A failure + * of this kind is part of what the effect *did*: it is written into the journal + * as the effect's result, the Workspace root stays where it was, and a replay + * reproduces it without performing anything. Every other failure is the run + * failing, and travels as an ordinary raise. + * + * It is a base class rather than a predicate over shapes so that being + * publishable is something a failure declares by construction. A module that + * wants its own refusal published extends this; nothing acquires the property + * by resembling something. + * + * Shared because both coordinators have to make the same choice, and two + * classifiers would eventually disagree about which failures are the run's. + */ +export abstract class JournaledEffectFailure extends Error {} + +/** + * Whether this failure is the effect's outcome rather than the run's failure. + * + * Asked by the one place in each host that has to choose between writing a + * result and letting a failure through. + */ +export function isJournaledEffectFailure(error: unknown): error is Error { + return error instanceof JournaledEffectFailure; +} diff --git a/packages/workflow/src/workspace/filesystem.ts b/packages/workflow/src/workspace/filesystem.ts new file mode 100644 index 000000000..07bfc57bc --- /dev/null +++ b/packages/workflow/src/workspace/filesystem.ts @@ -0,0 +1,43 @@ +/** + * The Workspace filesystem, as an operation rather than a place. + * + * Both hosts run the same Workspace work and neither one's storage is the + * contract. The Deno host's Workspace is rows in the run's own SQLite database + * reached through DOFS; the runner's is a real directory it materialized from + * the owner. What a mutation is allowed to ask for is the same either way, so + * it is stated here and implemented twice. + * + * Every member is an `Operation`. That is not decoration: one implementation is + * synchronous by necessity and the other is asynchronous by necessity, and a + * caller written against either shape would only work against that one. + */ + +import type { Operation } from "effection"; + +export interface WorkspaceEntry { + readonly name: string; + readonly kind: "file" | "directory" | "symlink"; +} + +export interface WorkspaceStat { + readonly kind: "file" | "directory" | "symlink"; + readonly mode: number; + readonly mtime: number; + readonly size: number; +} + +export interface WorkspaceFilesystem { + readFile(path: string): Operation; + readTextFile(path: string): Operation; + stat(path: string): Operation; + lstat(path: string): Operation; + readlink(path: string): Operation; + readdir(path: string): Operation; + writeFile(path: string, content: string | Uint8Array, mode?: number): Operation; + mkdir(path: string, options?: { recursive?: boolean; mode?: number }): Operation; + remove(path: string, options?: { recursive?: boolean; force?: boolean }): Operation; + rename(from: string, to: string): Operation; + chmod(path: string, mode: number): Operation; + symlink(target: string, path: string): Operation; + link(existingPath: string, newPath: string): Operation; +} diff --git a/packages/workflow/src/workspace/metadata.ts b/packages/workflow/src/workspace/metadata.ts new file mode 100644 index 000000000..905ac9803 --- /dev/null +++ b/packages/workflow/src/workspace/metadata.ts @@ -0,0 +1,33 @@ +/** + * Retained Repository and Worktree identity, as a mutation may reach it. + * + * These rows are immutable creation identity. Insertion adds one and never + * mutates one; a reused name is answered by reading the row back and comparing + * it. Where the rows live is the host's business — SQLite rows inside the Deno + * transaction's savepoint, a detached invocation snapshot and staged deltas on + * the runner — and the rules that decide whether a reused name is the same + * repository are shared, so they are stated against this interface rather than + * against either store. + * + * The locator is retained beside the record rather than inside it. Deciding + * whether a reused name asks for the same repository needs the bytes; the + * journal and the document need only the fingerprint, and a URL that turned out + * to carry a credential is then one column rather than one history. + */ + +import type { RepositoryRecord, WorktreeRecord } from "../composition/records.ts"; + +/** A Repository row: its journal-safe record, and the locator only storage sees. */ +export interface StoredRepository { + readonly record: RepositoryRecord; + readonly locator: string; +} + +export interface WorkspaceMetadata { + readRepository(name: string): StoredRepository | undefined; + readRepositories(): StoredRepository[]; + insertRepository(stored: StoredRepository): void; + readWorktree(repositoryName: string, name: string): WorktreeRecord | undefined; + readWorktreesForRepository(repositoryName: string): WorktreeRecord[]; + insertWorktree(record: WorktreeRecord): void; +} diff --git a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts index e93197f6a..676880fd1 100644 --- a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts @@ -215,6 +215,65 @@ describe("publishing one proposal", () => { ).toMatchObject({ outcome: "performed" }); }); + it("admits root, journal anchor and every mapping as one state", async () => { + // The invocation snapshot D3c begins from. It is one read on the owner + // because it is one fact: mappings taken from one moment and a root from + // another would let an invocation start against a Workspace its retained + // rows do not describe, and nothing later could tell. + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + + const empty = record(record(await ask(socket, "empty", { command: "mappings" }))["value"]); + expect(empty).toEqual({ + workspaceRootId: ROOT_ID, + journalEventId: null, + repositories: [], + worktrees: [], + agentSessions: [], + }); + + await stageThrough(socket); + const published = await ask(socket, "publish", commit()); + expect(published).toEqual(expect.objectContaining({ outcome: "performed" })); + + // One commit moved the pointer, retained the mapping and wrote the row. + // The next snapshot observes all of it, and observes it together. + const after = record(record(await ask(socket, "after", { command: "mappings" }))["value"]); + expect(after["workspaceRootId"]).toBe(NEXT_ROOT_ID); + expect(typeof after["journalEventId"]).toBe("string"); + expect(after["repositories"]).toEqual([{ record: REPOSITORY.record, locator: LOCATOR }]); + expect(after["worktrees"]).toEqual([]); + expect(after["agentSessions"]).toEqual([]); + + // The same anchor the frontier reports, from the same owner state. + const frontier = record( + record(await ask(socket, "frontier", { command: "frontier" }))["value"], + ); + expect([frontier["workspaceRootId"], frontier["journalEventId"]]).toEqual([ + after["workspaceRootId"], + after["journalEventId"], + ]); + }); + + it("returns no partial snapshot when more is retained than one may carry", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const socket = await connect(stub); + // Under the ceiling the snapshot answers whole. + await on(stub, (owner) => owner.fillRepositories(0, 200)); + const held = record(record(await ask(socket, "under", { command: "mappings" }))["value"]); + expect(Array.isArray(held["repositories"]) && held["repositories"]).toHaveLength(200); + + // Over it, the answer is a refusal rather than as much as would fit. A + // partial snapshot would describe a run holding fewer Repositories than it + // does, and every reconciliation against it would be decided wrongly. + await on(stub, (owner) => owner.fillRepositories(200, 200)); + const answer = await ask(socket, "over", { command: "mappings" }); + expect(answer["outcome"]).toBe("refused"); + expect(answer["value"]).toBe(undefined); + }); + it("keeps the expected root current for a journal-only transaction", async () => { const stub = executor(); await on(stub, (owner) => owner.initialize()); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 7a6ef6512..adc2034ec 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -382,6 +382,23 @@ export class ExecutorObject extends WorkflowOwnerObject { return row === undefined ? null : row; } + /** Retain more Repository rows than one admitted snapshot may carry. */ + fillRepositories(from: number, count: number): void { + for (let index = from; index < from + count; index += 1) { + const name = `repo-${String(index).padStart(4, "0")}`; + this.ctx.storage.sql.exec( + `INSERT INTO workspace_repositories (name, locator, locator_fingerprint, requested_base, + creation_commit, primary_branch, object_format, checkout_path) + VALUES (?, ?, ?, NULL, ?, 'main', 'sha1', ?)`, + name, + `https://git.example.invalid/${name}.git`, + "a".repeat(64), + "9".repeat(40), + `/${name}`, + ); + } + } + damageRetainedBlob(): void { this.ctx.storage.sql.exec( "UPDATE vfs_blob_bytes SET bytes = ?", diff --git a/packages/workflow/tests/remote-interoperability.test.ts b/packages/workflow/tests/remote-interoperability.test.ts index abe7e7a66..6c1e48f9f 100644 --- a/packages/workflow/tests/remote-interoperability.test.ts +++ b/packages/workflow/tests/remote-interoperability.test.ts @@ -20,6 +20,7 @@ * to cross more than one chunk. */ +import type { RemoteInvocationSnapshot } from "../src/remote/records.ts"; import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { DatabaseSync } from "node:sqlite"; @@ -95,6 +96,11 @@ function servedBy( content: { manifests: Map; blobs: Map }, ): RemoteReadLink { return { + // Materialization never asks for this; a stub that answered would say this + // test proved something it did not. + *invocationSnapshot(): Operation { + throw new Error("this read link carries no invocation snapshot"); + }, // deno-lint-ignore require-yield *frontier(): Operation { throw new Error("this owner serves only a root and its content"); diff --git a/packages/workflow/tests/remote-materialization.test.ts b/packages/workflow/tests/remote-materialization.test.ts index ab68ec386..0ba0b8046 100644 --- a/packages/workflow/tests/remote-materialization.test.ts +++ b/packages/workflow/tests/remote-materialization.test.ts @@ -14,6 +14,7 @@ * map kept what it was given. */ +import type { RemoteInvocationSnapshot } from "../src/remote/records.ts"; import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { ensure, type Operation, resource, scoped, until } from "effection"; @@ -65,6 +66,11 @@ function servedBy(captured: { blobs: ReadonlyMap; }): RemoteReadLink { return { + // Materialization never asks for this; a stub that answered would say this + // test proved something it did not. + *invocationSnapshot(): Operation { + throw new Error("this read link carries no invocation snapshot"); + }, // deno-lint-ignore require-yield *frontier(): Operation { throw new Error("this owner serves only a root and its content"); diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index 72760e737..6b3464153 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -11,6 +11,7 @@ * that was supposed to be removed is not a claim a fake can settle. */ +import type { RemoteInvocationSnapshot } from "../src/remote/records.ts"; import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; import { serializeDurableEvent } from "@executablemd/durable-streams"; @@ -213,6 +214,11 @@ function readsOf(captured: { blobs: ReadonlyMap; }): RemoteReadLink { return { + // Materialization never asks for this; a stub that answered would say this + // test proved something it did not. + *invocationSnapshot(): Operation { + throw new Error("this read link carries no invocation snapshot"); + }, // deno-lint-ignore require-yield *frontier(): Operation { return { diff --git a/packages/workflow/tests/remote-workspace.test.ts b/packages/workflow/tests/remote-workspace.test.ts new file mode 100644 index 000000000..9e1a71403 --- /dev/null +++ b/packages/workflow/tests/remote-workspace.test.ts @@ -0,0 +1,766 @@ +/** + * Tier WRH — the runner's Workspace coordinator, end to end. + * + * What this is about is ordering and authority, not arithmetic. The Files are + * real: a documented failure has to leave a directory that recaptures to the + * root it started from, and a fake cannot settle that. The owner is a scripted + * connection, because what crosses it here is what the coordinator decided — + * whether the atomic commit is really atomic is proved on real workerd, where + * atomicity is real. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { DurableStream } from "@executablemd/durable-streams"; +import { + durableRun, + establishJournalProvenance, + InMemoryStream, + type DurableEvent, + type Json, + type JournalProvenance, + type Workflow, +} from "@executablemd/durable-streams"; +import { type Operation, scoped, sleep, spawn, until } from "effection"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; +import { encodeBase64 } from "../src/cloudflare/encoding.ts"; +import { createRemoteWorkspaceFilesystem } from "../src/deno/remote-workspace-files.ts"; +import { captureWorkspace, type CapturedWorkspace } from "../src/remote/materialize.ts"; +import type { RemoteContent, RemoteContentRequest, RemoteReadLink } from "../src/remote/read.ts"; +import type { RemoteFrontierSnapshot } from "../src/remote/read.ts"; +import type { RemoteInvocationSnapshot } from "../src/remote/records.ts"; +import { useRemoteRunDatabase } from "../src/remote/database.ts"; +import { cloudflareRunLink, cloudflareReadLink } from "../src/cloudflare/client.ts"; +import { type OwnerSocket, type SocketListener, useOwnerConnection } from "../src/remote/client.ts"; +import type { WorkspaceFilesystem } from "../src/workspace/filesystem.ts"; +import type { WorkspaceMetadata } from "../src/workspace/metadata.ts"; +import { + createRemoteWorkspaceEffect, + type RemoteWorkspaceMutation, + type RemoteWorkspaceRuntime, + useRemoteWorkspaceEffects, + withRemoteWorkspaceEffects, +} from "../src/remote/workspace.ts"; +import { routeRemoteRunJournal } from "../src/remote/journal-route.ts"; +import { createInvocationMappings } from "../src/remote/mappings.ts"; +import { JournaledEffectFailure } from "../src/workspace/failure.ts"; +import { parseWorkspaceRootManifest } from "../src/workspace/root-manifest.ts"; +import { locatorFingerprintOf } from "../src/composition/locator.ts"; +import { agentSessionKey, resolveAgentSession } from "../src/storage/agent-session.ts"; +import type { WorkflowRunDatabase } from "../src/storage/api.ts"; + +const RUN_ID = "remote-run"; +const LOCATOR = "https://git.example.invalid/octo/app.git"; + +function reject(reason: string): never { + throw new Error(reason); +} + +/** A refusal the effect publishes rather than raises, as a document's would be. */ +class DocumentedFailure extends JournaledEffectFailure { + override name = "DocumentedFailure"; +} + +function runRecord() { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-03T00:00:00.000Z", + updatedAt: "2026-09-03T00:00:00.000Z", + }; +} + +function repository(name = "app") { + return { + record: { + name, + locatorFingerprint: locatorFingerprintOf(LOCATOR), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1" as const, + checkoutPath: `/${name}`, + }, + locator: LOCATOR, + }; +} + +function emptySnapshot(workspaceRootId: string): RemoteInvocationSnapshot { + return { + workspaceRootId, + journalEventId: null, + repositories: [], + worktrees: [], + agentSessions: [], + }; +} + +/** A connection whose answers a test writes, and which records what it was sent. */ +function wire(answer: (request: Record) => Record) { + const sent: Record[] = []; + const listeners = new Map>(); + const socket: OwnerSocket = { + send(data: string): void { + const request = JSON.parse(data) as Record; + sent.push(request); + const response = answer(request); + if (response["outcome"] === "lost") { + // The connection went while the answer was in flight. + for (const listener of listeners.get("close") ?? []) { + listener({}); + } + return; + } + for (const listener of listeners.get("message") ?? []) { + listener({ data: JSON.stringify({ id: request["id"], ...response }) }); + } + }, + close(): void {}, + addEventListener(type, listener): void { + const found = listeners.get(type) ?? new Set(); + found.add(listener); + listeners.set(type, found); + }, + removeEventListener(type, listener): void { + listeners.get(type)?.delete(listener); + }, + }; + return { socket, sent }; +} + +/** The owner's answers for one starting tree, and what it was asked to commit. */ +function ownerOf( + captured: CapturedWorkspace, + snapshot: () => RemoteInvocationSnapshot | { refused: string }, +) { + const commits: Record[] = []; + let refusal: string | undefined; + let lost = false; + return { + commits, + refuse(reason: string): void { + refusal = reason; + }, + lose(): void { + lost = true; + }, + get lost(): boolean { + return lost; + }, + answer(request: Record): Record { + const command = request["command"]; + if (command === "mappings") { + const value = snapshot(); + return "refused" in value + ? { outcome: "performed", value } + : { outcome: "performed", value }; + } + if (command === "frontier") { + return { + outcome: "performed", + value: { + record: runRecord(), + retrieval: null, + workspaceRootId: captured.root.rootId, + journalEventId: null, + }, + }; + } + if (command === "root") { + return { + outcome: "performed", + value: { + workspaceRootId: captured.root.rootId, + manifest: captured.root.manifest, + }, + }; + } + if (command === "content") { + const digest = String(request["digest"]); + const bytes = + request["kind"] === "manifest" + ? captured.contents.get(digest)?.manifestBytes + : captured.blobs.get(digest); + if (bytes === undefined) { + throw new Error("asked for content this owner does not hold"); + } + return { + outcome: "performed", + value: { + kind: request["kind"], + digest, + size: bytes.length, + bytes: encodeBase64(bytes), + }, + }; + } + if (command === "stage") { + const encoded = String(request["bytes"] ?? ""); + const padding = encoded.endsWith("==") ? 2 : encoded.endsWith("=") ? 1 : 0; + return { + outcome: "performed", + value: { + kind: request["kind"], + digest: request["digest"], + size: (encoded.length / 4) * 3 - padding, + }, + }; + } + commits.push(request); + if (lost) { + return { outcome: "lost" }; + } + if (refusal !== undefined) { + return { outcome: "refused", refusal }; + } + const publication = request["publication"]; + const events = Array.isArray(request["events"]) ? request["events"] : []; + return { + outcome: "performed", + value: { + workspaceRootId: + publication === null || publication === undefined + ? request["expectedWorkspaceRootId"] + : (publication as Record)["proposedWorkspaceRootId"], + journalEventIds: events.map((_entry, index) => `event-${index}`), + }, + }; + }, + }; +} + +/** A small starting tree, captured so a scripted owner can serve it. */ +function* startingTree(): Operation { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); + const root = yield* trees.create("source"); + yield* until(writeFile(`${root}/README.md`, "starting\n", { mode: 0o644 })); + yield* until(mkdir(`${root}/docs`, { mode: 0o755 })); + return yield* captureWorkspace( + files, + (logical) => (logical === "/" ? root : `${root}${logical}`), + reject, + ); +} + +interface Harness { + readonly database: WorkflowRunDatabase; + readonly runtime: RemoteWorkspaceRuntime; + readonly commits: Record[]; + refuse(reason: string): void; + lose(): void; + readonly sent: Record[]; + readonly captured: CapturedWorkspace; +} + +/** + * Everything one remote invocation needs, wired the way a host would wire it. + * + * Deliberately the production pieces: the real client over a scripted socket, + * the real database handle, the real coordinator and the real native adapters. + * A test that assembled a simpler stand-in would prove that the stand-in works. + */ +function* harness( + snapshot: (rootId: string) => RemoteInvocationSnapshot | { refused: string } = emptySnapshot, +): Operation { + const captured = yield* startingTree(); + const owner = ownerOf(captured, () => snapshot(captured.root.rootId)); + const transport = wire((request) => owner.answer(request)); + const connection = yield* useOwnerConnection(transport.socket); + let identifier = 0; + const next = () => `request-${(identifier += 1)}`; + const reads = cloudflareReadLink(connection, next, RUN_ID); + const link = cloudflareRunLink(connection, reads, next, RUN_ID); + const database = yield* useRemoteRunDatabase(link, yield* link.frontierSnapshot()); + const runtime: RemoteWorkspaceRuntime = { + files: runnerFiles(), + trees: yield* useRunnerTrees(), + reads, + createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), + }; + return { + database, + runtime, + commits: owner.commits, + refuse: owner.refuse, + lose: owner.lose, + sent: transport.sent, + captured, + }; +} + +/** + * One invocation, with its own journal and its own provenance. + * + * Separate per call because that is what a host does: a run's journal is + * established once per live session, and an invocation that reused another + * one's would be publishing into a journal it does not belong to. It also lets + * a later invocation observe what an earlier one left — which is the only way + * to see the accepted Workspace, since the coordinator owns its trees and + * removes them when it is done. + */ +function* invocation( + held: Harness, + name: string, + mutate: RemoteWorkspaceMutation, +): Operation<{ raised: unknown; events: DurableEvent[] }> { + return yield* scoped(function* () { + const stream = new InMemoryStream(); + const routed = routeRemoteRunJournal(held.database, stream); + yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); + const effect = createRemoteWorkspaceEffect(held.database, { type: "workspace", name }, mutate); + function* workflow(): Workflow { + yield effect; + } + const raised = yield* trapped( + withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: routed })), + ); + return { raised, events: stream.snapshot() }; + }); +} + +function yielded(events: readonly DurableEvent[]): DurableEvent[] { + return events.filter((event) => event.type === "yield"); +} + +describe("the runner's Workspace coordinator", () => { + it("commits Files, one mapping and the filtered result as one intent", function* () { + yield* scoped(function* () { + const held = yield* harness(); + const { raised, events } = yield* invocation( + held, + "write", + function* (filesystem, metadata): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + yield* filesystem.mkdir("/app", { mode: 0o755 }); + metadata.insertRepository(repository()); + // Read-your-writes: its own insert, before anything is committed. + return metadata.readRepository("app")?.record.checkoutPath ?? "missing"; + }, + ); + expect(raised).toBe(undefined); + + // Exactly one intent, carrying all three things together. + expect(held.commits).toHaveLength(1); + const intent = held.commits[0] ?? {}; + expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); + const mappings = intent["mappings"]; + expect(Array.isArray(mappings) && mappings).toHaveLength(1); + expect((mappings as Record[])[0]?.["kind"]).toBe("repository"); + expect(intent["publication"]).not.toBe(null); + // The result travelled in this same intent rather than through the + // ordinary journal, so nothing was written before the owner agreed. + expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); + expect(yielded(events)).toHaveLength(0); + }); + }); + + it("journals a documented failure against the unchanged root, and keeps nothing", function* () { + yield* scoped(function* () { + const held = yield* harness(); + const { raised } = yield* invocation( + held, + "refuse", + function* (filesystem, metadata): Operation { + yield* filesystem.writeFile("/SCRATCH.md", "discarded\n", 0o644); + yield* filesystem.remove("/README.md"); + metadata.insertRepository(repository()); + throw new DocumentedFailure("this Workspace effect refused"); + }, + ); + expect(String(raised)).toContain("this Workspace effect refused"); + + // One commit, and it proposes nothing about the Workspace. + expect(held.commits).toHaveLength(1); + const intent = held.commits[0] ?? {}; + expect(intent["publication"]).toBe(null); + expect(intent["mappings"]).toEqual([]); + expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); + expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); + + // A later invocation materializes the same root it always had: the + // discarded attempt left nothing, and nothing was promoted. + const observed = yield* invocation(held, "observe", function* (filesystem): Operation { + const names = yield* filesystem.readdir("/"); + return names.map((entry) => entry.name).toSorted(); + }); + expect(observed.raised).toBe(undefined); + const seen = held.commits[1] ?? {}; + expect(seen["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); + }); + }); + + it("prevents the document from running when the admitted root is unreachable", function* () { + yield* scoped(function* () { + const held = yield* harness(() => emptySnapshot("f".repeat(64))); + let executed = 0; + // deno-lint-ignore require-yield + yield* invocation(held, "unreachable", function* (): Operation { + executed += 1; + return "ran"; + }); + // Materialization is before the transaction, so this never reaches the + // anchor check — and it must still leave the run exactly as it was. + expect(executed).toBe(0); + expect(held.commits).toEqual([]); + }); + }); + + it("refuses before the document runs when the run moved since admission", function* () { + yield* scoped(function* () { + // The root still materializes, so the invocation gets all the way to the + // transaction; the journal anchor is what has moved. Nothing later could + // notice on its own — both answers were true when they were given. + const held = yield* harness((rootId) => ({ + ...emptySnapshot(rootId), + journalEventId: "event-from-another-moment", + })); + let executed = 0; + // deno-lint-ignore require-yield + const { raised } = yield* invocation(held, "drifted", function* (): Operation { + executed += 1; + return "ran"; + }); + expect(String(raised)).toContain("moved past"); + expect(executed).toBe(0); + expect(held.commits).toEqual([]); + }); + }); + + it("leaves the accepted Workspace alone when the owner refuses the commit", function* () { + yield* scoped(function* () { + const held = yield* harness(); + held.refuse("command:stale-root"); + const { raised } = yield* invocation( + held, + "refused", + function* (filesystem): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + return "ran"; + }, + ); + expect(raised).not.toBe(undefined); + // The owner said no, so nothing is promoted and nothing private crossed. + expect(String(raised)).not.toContain("command:"); + expect(held.commits).toHaveLength(1); + }); + }); + + it("refuses an execution identity another database claimed", function* () { + yield* scoped(function* () { + const held = yield* harness(); + const other = yield* harness(); + yield* scoped(function* () { + const stream = new InMemoryStream(); + const routed = routeRemoteRunJournal(held.database, stream); + yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); + // Created against one handle, coordinated under another. + const effect = createRemoteWorkspaceEffect( + other.database, + { type: "workspace", name: "foreign" }, + // deno-lint-ignore require-yield + function* (): Operation { + return "ran"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + const raised = yield* trapped( + withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: routed })), + ); + expect(String(raised)).toContain("foreign"); + }); + expect(held.commits).toEqual([]); + expect(other.commits).toEqual([]); + }); + }); + + it("refuses a journal whose provenance is not this run's", function* () { + yield* scoped(function* () { + const held = yield* harness(); + yield* scoped(function* () { + const stream = new InMemoryStream(); + const routed = routeRemoteRunJournal(held.database, stream); + yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); + const effect = createRemoteWorkspaceEffect( + held.database, + { type: "workspace", name: "provenance" }, + // deno-lint-ignore require-yield + function* (): Operation { + return "ran"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + // A different stream: structurally identical, and not this run's journal. + const foreign = new InMemoryStream(); + establishJournalProvenance(foreign); + const raised = yield* trapped( + withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: foreign })), + ); + expect(String(raised)).toContain("provenance"); + }); + expect(held.commits).toEqual([]); + }); + }); + + it("refuses a Files capability kept past the invocation that owned it", function* () { + yield* scoped(function* () { + const held = yield* harness(); + let escaped: WorkspaceFilesystem | undefined; + let metadata: WorkspaceMetadata | undefined; + // deno-lint-ignore require-yield + yield* invocation(held, "captured", function* (filesystem, held): Operation { + escaped = filesystem; + metadata = held; + return "ran"; + }); + const wrote = yield* trapped(escaped?.writeFile("/LATE.md", "too late") ?? sleep(0)); + expect(String(wrote)).toContain("stale"); + expect(() => metadata?.insertRepository(repository())).toThrow(); + }); + }); + + it("authorizes only paths beneath the attempt this invocation owns", function* () { + yield* scoped(function* () { + const held = yield* harness(); + const refused: string[] = []; + // deno-lint-ignore require-yield + const { raised } = yield* invocation(held, "escape", function* (filesystem): Operation { + return yield* (function* (): Operation { + for (const path of ["/../escaped", "/docs/../../escaped"]) { + const failure = yield* trapped(filesystem.writeFile(path, "outside")); + refused.push(String(failure)); + } + // Both ends of a rename: checking one would let the other leave. + refused.push(String(yield* trapped(filesystem.rename("/README.md", "/../moved")))); + // The Workspace root itself is a directory this invocation owns. + const entries = yield* filesystem.readdir("/"); + return entries.map((entry) => entry.name).toSorted(); + })(); + }); + expect(raised).toBe(undefined); + expect(refused).toHaveLength(3); + for (const failure of refused) { + expect(failure).toContain("outside the tree this invocation owns"); + } + }); + }); + + it("claims nothing when the answer to its commit is lost", function* () { + yield* scoped(function* () { + const held = yield* harness(); + held.lose(); + const { raised } = yield* invocation(held, "lost", function* (filesystem): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + return "ran"; + }); + // Whether the owner committed is exactly what cannot be known from here. + // What must not happen is claiming it did. + expect(raised).not.toBe(undefined); + expect(String(raised)).not.toContain("command:"); + expect(held.commits).toHaveLength(1); + }); + }); + + it("sends nothing and keeps nothing when the invocation is cancelled", function* () { + yield* scoped(function* () { + const held = yield* harness(); + const stream = new InMemoryStream(); + const routed = routeRemoteRunJournal(held.database, stream); + yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); + const effect = createRemoteWorkspaceEffect( + held.database, + { type: "workspace", name: "cancelled" }, + function* (filesystem): Operation { + yield* filesystem.writeFile("/SLOW.md", "in progress\n", 0o644); + yield* sleep(10_000); + return "never"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + const task = yield* spawn(() => + withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: routed })), + ); + yield* sleep(0); + yield* task.halt(); + // Cancellation is control flow: nothing was claimed, and nothing was sent. + expect(held.commits).toEqual([]); + expect(yielded(stream.snapshot())).toHaveLength(0); + }); + }); +}); + +describe("what one invocation retains", () => { + function view(snapshot: RemoteInvocationSnapshot) { + return createInvocationMappings(snapshot, () => {}); + } + + it("reconciles a compatible same-name Repository without staging it again", function* () { + const mappings = view({ ...emptySnapshot("a".repeat(64)), repositories: [repository()] }); + // The retained row is what a same-name read answers with. + expect(mappings.metadata.readRepository("app")?.locator).toBe(LOCATOR); + mappings.metadata.insertRepository(repository()); + expect(mappings.deltas()).toEqual([]); + yield* sleep(0); + }); + + it("refuses a same-name Repository that is not the same Repository", function* () { + const mappings = view({ ...emptySnapshot("a".repeat(64)), repositories: [repository()] }); + const conflicting = repository(); + expect(() => + mappings.metadata.insertRepository({ + ...conflicting, + record: { ...conflicting.record, creationCommit: "1".repeat(40) }, + }), + ).toThrow(); + // A conflict never replaces what is already there. + expect(mappings.metadata.readRepository("app")?.record.creationCommit).toBe("9".repeat(40)); + expect(mappings.deltas()).toEqual([]); + yield* sleep(0); + }); + + it("shows an invocation its own inserts and stages each exactly once", function* () { + const mappings = view(emptySnapshot("a".repeat(64))); + mappings.metadata.insertRepository(repository()); + mappings.metadata.insertRepository(repository()); + expect(mappings.metadata.readRepository("app")?.record.name).toBe("app"); + mappings.metadata.insertWorktree({ + repositoryName: "app", + name: "feature", + requestedBranch: "feature", + requestedBase: null, + creationCommit: "9".repeat(40), + checkoutPath: "/app-feature", + }); + const deltas = mappings.deltas(); + // Parents before children, whatever order they were staged in. + expect(deltas.map((delta) => delta.kind)).toEqual(["repository", "worktree"]); + yield* sleep(0); + }); + + it("keeps a Repository locator out of every value the record carries", function* () { + const mappings = view(emptySnapshot("a".repeat(64))); + mappings.metadata.insertRepository(repository()); + const [delta] = mappings.deltas(); + expect(delta?.kind).toBe("repository"); + if (delta?.kind === "repository") { + expect(delta.locator).toBe(LOCATOR); + // The record names the fingerprint and never the bytes. + expect(JSON.stringify(delta.record)).not.toContain("git.example.invalid"); + } + yield* sleep(0); + }); + + it("refuses a Worktree with no Repository and a path this build does not admit", function* () { + const mappings = view(emptySnapshot("a".repeat(64))); + expect(() => + mappings.metadata.insertWorktree({ + repositoryName: "missing", + name: "feature", + requestedBranch: "feature", + requestedBase: null, + creationCommit: "9".repeat(40), + checkoutPath: "/app-feature", + }), + ).toThrow(); + mappings.metadata.insertRepository(repository()); + expect(() => + mappings.metadata.insertWorktree({ + repositoryName: "app", + name: "feature", + requestedBranch: "feature", + requestedBase: null, + creationCommit: "9".repeat(40), + checkoutPath: "not-a-workspace-path", + }), + ).toThrow(); + yield* sleep(0); + }); + + it("resolves an Agent session by the shared rules and stages it once", function* () { + const identity = { + provider: "claude", + agentCommand: "claude", + sessionIdentity: "expansion-1", + }; + const mappings = view(emptySnapshot("a".repeat(64))); + const sessionKey = agentSessionKey(identity); + // Nothing retained and nothing asserted: this is a new conversation. + expect(resolveAgentSession(undefined, "reattach", [], identity).kind).toBe("create"); + + // The pre-commit window: exactly one canonical assertion reconciles it. + const reconciled = resolveAgentSession( + mappings.agentSessions.read(sessionKey), + "reattach", + [{ kind: "session-id", value: "abc" }], + identity, + ); + expect(reconciled.kind).toBe("reattach"); + if (reconciled.kind === "reattach") { + mappings.agentSessions.commit(reconciled.record); + mappings.agentSessions.commit(reconciled.record); + } + expect(mappings.deltas().map((delta) => delta.kind)).toEqual(["agent-session"]); + + // A retained mapping the provider now contradicts refuses rather than + // starting a replacement conversation. + const retained = mappings.agentSessions.read(sessionKey); + expect(retained).not.toBe(undefined); + expect(() => + resolveAgentSession(retained, "reattach", [{ kind: "session-id", value: "other" }], identity), + ).toThrow(); + expect(() => resolveAgentSession(retained, "reattach", [], identity)).toThrow(); + yield* sleep(0); + }); + + it("refuses more mappings, and more mapping bytes, than one commit may carry", function* () { + const byCount = view(emptySnapshot("a".repeat(64))); + expect(() => { + for (let index = 0; index < 512; index += 1) { + byCount.metadata.insertRepository(repository(`app-${String(index).padStart(4, "0")}`)); + } + }).toThrow(); + + // Few enough to pass the count, large enough that no message could carry + // them. Bounding one without the other would leave the other reachable. + const byBytes = view(emptySnapshot("a".repeat(64))); + expect(() => { + for (let index = 0; index < 64; index += 1) { + const wide = repository(`wide-${String(index).padStart(4, "0")}`); + byBytes.metadata.insertRepository({ + ...wide, + record: { + ...wide.record, + creationCommit: "9".repeat(40), + primaryBranch: "b".repeat(8192), + }, + }); + } + }).toThrow(); + yield* sleep(0); + }); +}); + +function* trapped(operation: Operation): Operation { + try { + yield* operation; + return undefined; + } catch (error) { + return error; + } +} From 7c0bf30a71c733a55af9fec5b475b7e8a03b28df Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 15:57:33 -0400 Subject: [PATCH 46/81] =?UTF-8?q?=F0=9F=90=9B=20Resolve=20what=20a=20Works?= =?UTF-8?q?pace=20path=20lands=20on,=20and=20bind=20one=20run=20as=20one?= =?UTF-8?q?=20thing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lexical admission compared the spelling of a path with the attempt root and then handed it to a syscall that follows symbolic links. An in-tree link read and overwrote a file outside the attempt, and `..notes.md` was refused for beginning with two dots. Every operation now resolves before it acts, on the rule `packages/runtime/host-files.ts` states: only a complete `..` segment leaves, the existing prefix is walked so a path that does not exist yet is still judged, an operation about a link does not follow it, and both ends of a rename or hardlink are admitted at the time of use. A link's target is a Workspace path, so an absolute one names the Workspace root rather than the machine's — resolution is done here rather than by the kernel. The runtime, database, owner link, journal and provenance were supplied separately and could be recombined: pair run B's handle with run A's link and journal, and if both began at the same root and anchor, one effect journalled in A and published its Workspace in B. `useRemoteRun()` now constructs them together and hands back one opaque binding, so there is nothing to recombine and nothing structural to forge. Wiring the production coordinator to a real Durable Object found a third defect the scripted owner could not: a sealed proposal listed its content manifests before its blobs, while the owner requires one sequence ordered by kind and digest. Two owners agreeing about a proposal is what that test is for. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- .../src/deno/remote-workspace-files.ts | 258 ++++++++++++--- packages/workflow/src/remote/invocation.ts | 19 +- packages/workflow/src/remote/workspace.ts | 194 ++++++++---- .../cloudflare/remote-workspace.vitest.ts | 273 ++++++++++++++++ .../tests/cloudflare/support/worker-files.ts | 293 ++++++++++++++++++ .../tests/remote-workspace-files.test.ts | 181 +++++++++++ .../workflow/tests/remote-workspace.test.ts | 193 ++++++++---- packages/workflow/tsconfig.cloudflare.json | 2 +- 8 files changed, 1246 insertions(+), 167 deletions(-) create mode 100644 packages/workflow/tests/cloudflare/remote-workspace.vitest.ts create mode 100644 packages/workflow/tests/cloudflare/support/worker-files.ts create mode 100644 packages/workflow/tests/remote-workspace-files.test.ts diff --git a/packages/workflow/src/deno/remote-workspace-files.ts b/packages/workflow/src/deno/remote-workspace-files.ts index 042c7c7ad..af9a727b5 100644 --- a/packages/workflow/src/deno/remote-workspace-files.ts +++ b/packages/workflow/src/deno/remote-workspace-files.ts @@ -1,15 +1,41 @@ /** * The Workspace filesystem, over the attempt directory this invocation owns. * - * The runner's Workspace is a real tree it materialized from the owner, so the - * operations are the runtime's own asynchronous primitives adapted with + * The runner's Workspace is a real directory it materialized from the owner, so + * the operations are the runtime's own asynchronous primitives adapted with * `until`. Nothing above this module names a runtime, and nothing in it decides * anything about a Workspace: it moves bytes where it is told, and refuses to * be told anywhere outside the attempt. * - * `node:fs/promises` rather than a runtime global, for the same reason the - * materialization adapter uses it: the same code has to work wherever the - * runner runs. + * ## Why lexical admission is not containment here + * + * The Deno host's Workspace is rows in a database, so a path there has no + * outside to reach and admission is arithmetic. This one is a real directory on + * a host that has an outside, and a symbolic link is a path the kernel follows + * on its own. Comparing the *spelling* of a path with the attempt root admits + * `/link` while the syscall that follows reads whatever `/link` points at. + * + * So every operation resolves before it acts, on the same terms + * `packages/runtime/host-files.ts` states for the host provider: a complete + * `..` segment leaves and `..notes.md` does not; the existing prefix is walked + * so a path that does not exist yet can still be judged by its deepest + * existing ancestor; an operation that acts on a link does not follow it, and + * one whose contract follows a link follows only where that link lands inside + * the attempt. + * + * ## A link's target is a Workspace path, not a host path + * + * A retained symbolic link carries its target as text, and that text is + * interpreted in the Workspace it belongs to. An absolute target names the + * logical Workspace root — the root of the tree this invocation owns — not the + * runner host's root. Letting the kernel interpret `/etc/passwd` would turn a + * retained Workspace entry into authority over the machine, so resolution is + * done here, one segment at a time, and the host is asked only about paths that + * are already known to be inside. + * + * The stable-host-namespace limitation the host provider documents applies here + * too: another process can replace a directory between the moment this resolves + * a path and the moment it uses one. That window is not what this closes. */ import { @@ -26,7 +52,7 @@ import { symlink, writeFile, } from "node:fs/promises"; -import { isAbsolute, relative, resolve } from "node:path"; +import type { Stats } from "node:fs"; import { type Operation, until } from "effection"; import type { WorkspaceEntry, @@ -36,28 +62,131 @@ import type { import { throwWorkspaceFilesystemFailure } from "./workspace/errors.ts"; import type { HostPath } from "../remote/materialize.ts"; +/** A path no Workspace operation may reach, whatever it names. */ +export class WorkspacePathError extends Error { + override name = "WorkspacePathError"; + + constructor() { + // No path, no target and no host directory: what a document may learn is + // that it asked for somewhere it does not own. + super("this Workspace path is outside the tree this invocation owns."); + } +} + +/** How many links one resolution will follow before calling it a loop. */ +const MAX_LINKS = 32; + /** - * Where a logical path is allowed to land. + * The logical segments this path names, or `undefined` if it leaves the root. * - * The attempt root is the whole of what this invocation may touch. A logical - * path is resolved and then held to that root, so `..`, an absolute path and a - * path that merely starts with the root's name are each refused before any - * syscall — the host path is a place to work, never a durable identity. + * Pure arithmetic on POSIX segments, decided before anything touches the host. + * `.` and an empty segment are nothing; a complete `..` pops, and popping past + * the root is the escape. A segment that merely begins with two dots is an + * ordinary name and stays. */ -function within(at: HostPath, root: string, path: string): string { - const host = resolve(at(path)); - const inside = relative(root, host); - // The empty relative path is the Workspace root itself, which is a directory - // this invocation owns and may read. Only leaving the tree is refused. - if (inside.startsWith("..") || isAbsolute(inside)) { - throw new WorkspacePathError("this Workspace path is outside the tree this invocation owns."); +function segmentsOf(base: readonly string[], path: string): string[] | undefined { + const segments = path.startsWith("/") ? [] : [...base]; + for (const segment of path.split("/")) { + if (segment === "" || segment === ".") { + continue; + } + if (segment === "..") { + if (segments.length === 0) { + return undefined; + } + segments.pop(); + continue; + } + segments.push(segment); } - return host; + return segments; } -/** A path no Workspace operation may reach, whatever it names. */ -export class WorkspacePathError extends Error { - override name = "WorkspacePathError"; +/** What one operation may act on: where it is, and whether a link was left alone. */ +interface Resolved { + readonly segments: readonly string[]; +} + +/** + * Walk the path, following the links inside it, and refuse the ones that leave. + * + * `followFinal` is the difference between an operation about a file and an + * operation about a link. A read follows the last link to the file it names, + * because replacing or reporting the link would surprise a caller that asked + * for the file; `lstat`, `readlink`, a removal and a rename act on the entry + * the caller named, so the last segment is left exactly as written. + * + * A path that does not exist is not an error: the walk stops at the deepest + * existing ancestor and keeps the rest, which is what lets a write name a file + * it is about to create and still be judged. + */ +function* resolve(root: string, path: string, followFinal: boolean): Operation { + if (path === "" || path.includes("\u0000")) { + throw new WorkspacePathError(); + } + const admitted = segmentsOf([], path); + if (admitted === undefined) { + throw new WorkspacePathError(); + } + let segments: string[] = admitted; + + for (let followed = 0; ; followed += 1) { + if (followed > MAX_LINKS) { + throw new WorkspacePathError(); + } + const crossing = yield* firstLink(root, segments, followFinal); + if (crossing === undefined) { + return { segments }; + } + // The target is read in the Workspace this link belongs to: absolute means + // the Workspace root, and relative means beside the link. + const next = segmentsOf(segments.slice(0, crossing.depth - 1), crossing.target); + if (next === undefined) { + throw new WorkspacePathError(); + } + segments = [...next, ...segments.slice(crossing.depth)]; + } +} + +/** + * The shallowest segment of this path that is a symbolic link, if any. + * + * Shallowest rather than any, because substituting a link's target changes + * every segment beneath it — resolving a deeper one first would resolve it + * against a prefix that is about to be replaced. + */ +function* firstLink( + root: string, + segments: readonly string[], + followFinal: boolean, +): Operation<{ depth: number; target: string } | undefined> { + const last = followFinal ? segments.length : segments.length - 1; + for (let depth = 1; depth <= last; depth += 1) { + const host = hostPath(root, segments.slice(0, depth)); + const entry: Stats | undefined = yield* describing(host); + if (entry === undefined) { + // Nothing here, so nothing below it exists either. What the caller named + // is judged by the ancestor that does exist, which this walk has passed. + return undefined; + } + if (entry.isSymbolicLink()) { + return { depth, target: yield* until(readlink(host)) }; + } + } + return undefined; +} + +/** What this entry is, or nothing when there is no entry here. */ +function* describing(host: string): Operation { + try { + return yield* until(lstat(host)); + } catch { + return undefined; + } +} + +function hostPath(root: string, segments: readonly string[]): string { + return segments.length === 0 ? root : `${root}/${segments.join("/")}`; } function described(value: { @@ -78,12 +207,14 @@ function described(value: { * * The classifier asks for a `WorkspaceFsError` carrying a documented code, * because that is what the other host raises. Renaming here rather than - * widening the classifier keeps one list of documented conditions. + * widening the classifier keeps one list of documented conditions. The + * message and the host path inside it are dropped: what reaches a document is + * the condition, never where this invocation happened to put its tree. */ function named(error: unknown): unknown { const code = error instanceof Error ? Reflect.get(error, "code") : undefined; if (error instanceof Error && typeof code === "string") { - const renamed = new Error(error.message, { cause: error }); + const renamed = new Error(`the Workspace operation failed (${code})`); renamed.name = "WorkspaceFsError"; Reflect.set(renamed, "code", code); return renamed; @@ -96,14 +227,21 @@ export function createRemoteWorkspaceFilesystem( authorize: () => void, ): WorkspaceFilesystem { // The attempt's own root, taken from the same resolver every other path goes - // through. Passing it separately would let the two disagree. - const resolved = resolve(at("/")); + // through. Every host path this module builds is this root plus segments it + // has already admitted, so no authored text reaches a syscall unexamined. + const root = at("/"); - function* run(path: string, body: (host: string) => Promise): Operation { + function* run( + path: string, + followFinal: boolean, + body: (host: string) => Promise, + ): Operation { authorize(); - const host = within(at, resolved, path); + // Resolved immediately before the operation it authorizes, never cached: a + // path admitted once is not a capability to use later. + const resolved = yield* resolve(root, path, followFinal); try { - return yield* until(body(host)); + return yield* until(body(hostPath(root, resolved.segments))); } catch (error) { // The same classification the Deno host applies: a documented filesystem // condition is the effect's own outcome, and everything else is the run @@ -112,30 +250,52 @@ export function createRemoteWorkspaceFilesystem( } } + /** Both ends of a two-path operation, each admitted at the time of use. */ + function* pair( + from: string, + to: string, + followFrom: boolean, + body: (source: string, destination: string) => Promise, + ): Operation { + authorize(); + const source = yield* resolve(root, from, followFrom); + const destination = yield* resolve(root, to, false); + try { + return yield* until( + body(hostPath(root, source.segments), hostPath(root, destination.segments)), + ); + } catch (error) { + return throwWorkspaceFilesystemFailure(named(error)); + } + } + return { *readFile(path): Operation { - return yield* run(path, (host) => readFile(host)); + return yield* run(path, true, (host) => readFile(host)); }, *readTextFile(path): Operation { - const bytes = yield* run(path, (host) => readFile(host)); + const bytes = yield* run(path, true, (host) => readFile(host)); return new TextDecoder().decode(bytes); }, *stat(path): Operation { - return described(yield* run(path, (host) => stat(host))); + return described(yield* run(path, true, (host) => stat(host))); }, *lstat(path): Operation { - return described(yield* run(path, (host) => lstat(host))); + // About the entry, so the last segment stays what it is. + return described(yield* run(path, false, (host) => lstat(host))); }, *readlink(path): Operation { - return yield* run(path, (host) => readlink(host)); + // The retained target, exactly as it was written. It is a Workspace path, + // and reading it back is not resolving it. + return yield* run(path, false, (host) => readlink(host)); }, *readdir(path): Operation { - const entries = yield* run(path, (host) => readdir(host, { withFileTypes: true })); + const entries = yield* run(path, true, (host) => readdir(host, { withFileTypes: true })); return entries.map((entry) => ({ name: entry.name, kind: entry.isFile() ? "file" : entry.isDirectory() ? "directory" : "symlink", @@ -144,38 +304,38 @@ export function createRemoteWorkspaceFilesystem( *writeFile(path, content, mode): Operation { const bytes = typeof content === "string" ? new TextEncoder().encode(content) : content; - yield* run(path, (host) => writeFile(host, bytes, mode === undefined ? {} : { mode })); + // Follows an internal link to the file it names: replacing the link would + // be the surprising outcome, and an outward one never got this far. + yield* run(path, true, (host) => writeFile(host, bytes, mode === undefined ? {} : { mode })); }, *mkdir(path, options = {}): Operation { - yield* run(path, (host) => mkdir(host, options).then(() => undefined)); + yield* run(path, true, (host) => mkdir(host, options).then(() => undefined)); }, *remove(path, options = {}): Operation { - yield* run(path, (host) => rm(host, options)); + // A removal takes the entry the caller named. Following a final link + // would remove something never mentioned. + yield* run(path, false, (host) => rm(host, options)); }, *rename(from, to): Operation { - // Both ends are held to the attempt: a rename is two paths, and checking - // one of them would let the other leave the tree. - const destination = within(at, resolved, to); - yield* run(from, (host) => rename(host, destination)); + yield* pair(from, to, false, (source, destination) => rename(source, destination)); }, *chmod(path, mode): Operation { - yield* run(path, (host) => chmod(host, mode)); + yield* run(path, true, (host) => chmod(host, mode)); }, *symlink(target, path): Operation { - // The target is not resolved here. A symbolic link's target is retained - // exactly as written, and reading through one goes back through this - // interface, where it is held to the attempt like any other path. - yield* run(path, (host) => symlink(target, host)); + // The target is not resolved: a link's target is retained text, and it is + // interpreted when the link is walked. What is admitted here is where the + // link itself is created. + yield* run(path, false, (host) => symlink(target, host)); }, *link(existingPath, newPath): Operation { - const destination = within(at, resolved, newPath); - yield* run(existingPath, (host) => link(host, destination)); + yield* pair(existingPath, newPath, true, (source, destination) => link(source, destination)); }, }; } diff --git a/packages/workflow/src/remote/invocation.ts b/packages/workflow/src/remote/invocation.ts index 795d2b4c8..923df77c7 100644 --- a/packages/workflow/src/remote/invocation.ts +++ b/packages/workflow/src/remote/invocation.ts @@ -221,9 +221,17 @@ export function useAttempt( }); } -/** The exact closure a captured root names, in canonical order. */ +/** + * The exact closure a captured root names, in canonical order. + * + * Ordered by `kind:digest`, which is the order the owner reads it in: it holds + * the inventory to a strictly increasing sequence, because a proposal that + * named its pieces in another order is not the proposal whose identity the + * runner computed. Concatenating one kind after the other would produce a list + * this owner refuses, and only a real owner would say so. + */ function inventoryOf(captured: CapturedWorkspace): ProposedContent[] { - return [ + const inventory: ProposedContent[] = [ ...captured.root.manifests.map((digest) => ({ kind: "manifest" as const, digest, @@ -235,6 +243,13 @@ function inventoryOf(captured: CapturedWorkspace): ProposedContent[] { size: captured.blobs.get(digest)?.length ?? 0, })), ]; + return inventory.toSorted((left, right) => + orderingOf(left) < orderingOf(right) ? -1 : orderingOf(left) > orderingOf(right) ? 1 : 0, + ); +} + +function orderingOf(piece: ProposedContent): string { + return `${piece.kind}:${piece.digest}`; } /** Every piece the capture can supply, by identity. */ diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts index b261fdc24..ce11d180c 100644 --- a/packages/workflow/src/remote/workspace.ts +++ b/packages/workflow/src/remote/workspace.ts @@ -64,6 +64,11 @@ import type { HostPath, RunnerFiles } from "./materialize.ts"; import type { RemoteReadLink } from "./read.ts"; import type { RemoteInvocationSnapshot } from "./records.ts"; import { withRemoteJournalRoute } from "./journal-route.ts"; +import { resource } from "effection"; +import { establishJournalProvenance, type DurableStream } from "@executablemd/durable-streams"; +import { useRemoteRunDatabase, type RemoteRunLink } from "./database.ts"; +import { routeRemoteRunJournal } from "./journal-route.ts"; + import type { TemporaryTrees } from "./invocation.ts"; /** @@ -123,23 +128,113 @@ const WorkspaceMutation: Api = createApi { - const owners = new WeakMap(); + const owners = new WeakMap(); return { - claim(identity: object, database: WorkflowRunDatabase): void { - owners.set(identity, database); + claim(identity: object, run: object): void { + owners.set(identity, run); }, - get(identity: object): WorkflowRunDatabase | undefined { + get(identity: object): object | undefined { return owners.get(identity); }, }; })(); -interface Registration { - open: boolean; +/** + * One remote run, as one thing. + * + * The pieces a Workspace invocation needs — the database handle, the owner link + * its reads and commits go through, the runtime adapters that materialize from + * that owner, the routed journal and the provenance taken over it — describe + * one run only when they came from the same one. Supplied separately they can + * be recombined: pair run B's database with run A's link and journal, and if + * both happen to start at the same root and anchor, one effect journals in A + * and publishes its Workspace in B. + * + * So they are not supplied separately. This constructs them together and hands + * back one opaque value. There is nothing to recombine, and nothing structural + * to forge: the coordinator compares the object it was given, not the run id, + * root or anchor inside it. + */ +export interface RemoteRun { + /** The run's storage handle, for work that is not a Workspace effect. */ + readonly database: WorkflowRunDatabase; + /** + * The run's journal, routed so a Workspace publication lands in its + * transaction. This exact stream is the one provenance was taken over. + */ + readonly journal: DurableStream; +} + +/** What only this module may read off a binding. */ +interface BoundRun extends RemoteRun { readonly runtime: RemoteWorkspaceRuntime; readonly provenance: JournalProvenance; } +/** + * The private view of a binding, keyed by the binding itself. + * + * A `WeakSet` would answer "did this module make it"; this answers "and here is + * what it was made from", without putting either on the value a host holds. A + * second loaded copy of this module has its own map and cannot answer for one + * of these, which is the loaded-copy contract. + */ +const bindings = (() => { + const held = new WeakMap(); + return { + bind(run: BoundRun): RemoteRun { + const handle: RemoteRun = Object.freeze({ database: run.database, journal: run.journal }); + held.set(handle, run); + return handle; + }, + of(run: RemoteRun | undefined): BoundRun | undefined { + return run === undefined ? undefined : held.get(run); + }, + }; +})(); + +/** What a host supplies to open one remote run. */ +export interface RemoteRunOptions { + /** The owner link this run's database, reads and commits all go through. */ + readonly link: RemoteRunLink; + readonly reads: RemoteReadLink; + readonly files: RunnerFiles; + readonly trees: TemporaryTrees; + createFilesystem(at: HostPath, authorize: () => void): WorkspaceFilesystem; + /** The run's ordinary journal, which this routes and takes provenance over. */ + readonly journal: DurableStream; +} + +/** + * Open one remote run: its database, its routed journal and its provenance. + * + * The database is created here from the same link the runtime reads through, so + * "this runtime belongs to this handle" is true by construction rather than by + * a check that could be passed with another handle. + */ +export function useRemoteRun(options: RemoteRunOptions): Operation { + return resource(function* (provide) { + const database = yield* useRemoteRunDatabase( + options.link, + yield* options.link.frontierSnapshot(), + ); + const journal = routeRemoteRunJournal(database, options.journal); + yield* provide( + bindings.bind({ + database, + journal, + provenance: establishJournalProvenance(journal), + runtime: { + files: options.files, + trees: options.trees, + reads: options.reads, + createFilesystem: options.createFilesystem, + }, + }), + ); + }); +} + interface ProviderApi { readonly provider: object | undefined; } @@ -149,12 +244,17 @@ const RemoteWorkspaceProvider: Api = createApi( { provider: undefined }, ); +interface Registration { + open: boolean; + readonly run: BoundRun; +} + const registrations = (() => { const held = new WeakMap(); return { - register(runtime: RemoteWorkspaceRuntime, provenance: JournalProvenance) { + register(run: BoundRun) { const selection = Object.freeze({}); - const registration: Registration = { open: true, runtime, provenance }; + const registration: Registration = { open: true, run }; held.set(selection, registration); return { selection, @@ -171,70 +271,72 @@ const registrations = (() => { }; })(); -/** - * Install the runner's Workspace coordination for this scope. - * - * The provenance is the one the run's journal was established with. It is held - * here rather than compared structurally, because two journals can describe the - * same events and only one of them is this run's. - */ -export function* useRemoteWorkspaceEffects( - runtime: RemoteWorkspaceRuntime, - provenance: JournalProvenance, -): Operation { - const registration = registrations.register(runtime, provenance); +/** Install the runner's Workspace coordination for this run, in this scope. */ +export function* useRemoteWorkspaceEffects(run: RemoteRun): Operation { + const bound = bindings.of(run); + if (bound === undefined) { + unavailable("this is not a remote run this build opened."); + } + const registration = registrations.register(bound); yield* ensure(registration.close); yield* RemoteWorkspaceProvider.around({ provider: () => registration.selection }, { at: "min" }); } export function withRemoteWorkspaceEffects( - database: WorkflowRunDatabase, + run: RemoteRun, operation: Operation, ): Operation { return scoped(function* () { const selection = yield* RemoteWorkspaceProvider.operations.provider; const registration = selection === undefined ? undefined : registrations.get(selection); - if (registration === undefined) { + // The exact binding, not one that describes the same run. Two handles on + // two owners can hold identical records; only one of them is this one. + if (registration === undefined || registration.run !== bindings.of(run)) { return unavailable("no remote Workspace coordinator is installed for this run."); } - return yield* withWorkspaceCoordinationProvider(coordinator(database, registration), operation); + return yield* withWorkspaceCoordinationProvider(coordinator(registration.run), operation); }); } export function createRemoteWorkspaceEffect( - database: WorkflowRunDatabase, + run: RemoteRun, description: EffectDescription, mutate: RemoteWorkspaceMutation, ): DurableEffect { - const execute = () => WorkspaceMutation.operations.run(database, mutate); + const bound = bindings.of(run); + if (bound === undefined) { + unavailable("this is not a remote run this build opened."); + } + const execute = () => WorkspaceMutation.operations.run(bound.database, mutate); const executionIdentity = Object.freeze({}); - workspaceEffectOwners.claim(executionIdentity, database); + // Claimed for the binding rather than for a database, so an effect cannot be + // created against one run and coordinated by another that holds it. + workspaceEffectOwners.claim(executionIdentity, bound); return createOwnedDurableWorkspaceOperation(description, execute, executionIdentity); } -function coordinator( - database: WorkflowRunDatabase, - registration: Registration, -): WorkspaceCoordinationProvider { +function coordinator(run: BoundRun): WorkspaceCoordinationProvider { return { *run(authority: WorkspaceCoordinationAuthority): Operation { let transacted; try { - if (workspaceEffectOwners.get(authority.executionIdentity) !== database) { + // Both against the same binding, so there is no pair of checks that a + // recombination could satisfy one at a time. + if (workspaceEffectOwners.get(authority.executionIdentity) !== run) { unavailable( "the live Workspace effect is missing, foreign, completed, or stale for this " + - "WorkflowRun database.", + "remote run.", ); } if ( authority.journalProvenance === undefined || - authority.journalProvenance !== registration.provenance + authority.journalProvenance !== run.provenance ) { unavailable( - "the live Workspace journal does not have the provenance of the selected WorkflowRun.", + "the live Workspace journal does not have the provenance of the selected remote run.", ); } - transacted = yield* invoke(database, registration, authority); + transacted = yield* invoke(run, authority); } catch (error) { throw yield* authority.activateFailure(error); } @@ -246,12 +348,8 @@ function coordinator( }; } -function* invoke( - database: WorkflowRunDatabase, - registration: Registration, - authority: WorkspaceCoordinationAuthority, -) { - const { runtime } = registration; +function* invoke(run: BoundRun, authority: WorkspaceCoordinationAuthority) { + const { runtime, database } = run; const reject = (reason: string): never => unavailable(reason); const snapshot = yield* runtime.reads.invocationSnapshot(); @@ -291,27 +389,19 @@ function* invoke( "this Workspace invocation was admitted from a state this run has since moved past.", ); } - return yield* coordinateTransaction( - database, - transaction, - route, - runtime, - snapshot, - attempt, - authority, - ); + return yield* coordinateTransaction(run, transaction, route, snapshot, attempt, authority); }); } function* coordinateTransaction( - database: WorkflowRunDatabase, + run: BoundRun, transaction: WorkflowRunTransaction, route: WorkspaceRoute, - runtime: RemoteWorkspaceRuntime, snapshot: RemoteInvocationSnapshot, attempt: Attempt, authority: WorkspaceCoordinationAuthority, ): Operation { + const { database, runtime } = run; return yield* scoped(function* () { let live = true; // The capabilities exist while this invocation does and no longer. A diff --git a/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts b/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts new file mode 100644 index 000000000..9ca022cd7 --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts @@ -0,0 +1,273 @@ +/** + * The runner's Workspace coordinator, against a real owner. + * + * Everything on the owner's side is real here: a real Durable Object, its own + * SQLite storage, a real accepted Hibernation WebSocket, and the production + * client, database handle, run binding and coordinator on the other end of it. + * What runs is `createRemoteWorkspaceEffect()` through + * `withRemoteWorkspaceEffects()`, so the admission read, the attempt, the + * anchor check, the mapping staging, the enlistment, the journal route and + * D3a's atomic commit are all the production path. + * + * The one stand-in is the runner's host filesystem: workerd has none, and the + * vendored DOFS cannot set a modification time, so it cannot reproduce a + * retained mtime — which is the thing materialization refuses a host for. The + * native adapter it stands in for is proved against real files in + * `packages/workflow/tests/remote-workspace-files.test.ts`. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { InMemoryStream, type Workflow, type Json } from "@executablemd/durable-streams"; +import { run, type Operation } from "effection"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; +import { createWorkerFiles } from "./support/worker-files.ts"; +import { cloudflareReadLink, cloudflareRunLink } from "../../src/cloudflare/client.ts"; +import { + type OwnerSocket, + type SocketListener, + useOwnerConnection, +} from "../../src/remote/client.ts"; +import { + createRemoteWorkspaceEffect, + type RemoteRun, + useRemoteRun, + useRemoteWorkspaceEffects, + withRemoteWorkspaceEffects, +} from "../../src/remote/workspace.ts"; +import { durableRun } from "@executablemd/durable-streams"; +import { locatorFingerprintOf } from "../../src/composition/locator.ts"; +import { useMaterialization } from "../../src/remote/invocation.ts"; +import { JournaledEffectFailure } from "../../src/workspace/failure.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +const LOCATOR = "https://git.example.invalid/octo/app.git"; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`coordinated-${unique}-${Math.random()}`)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise { + return runInDurableObject(stub, body); +} + +async function connect(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + const response = await stub.fetch("https://owner.invalid/executor", { + headers: { + authorization: `Bearer ${token}`, + upgrade: "websocket", + "x-release": POLICY.release, + "x-run-id": RUN_ID, + }, + }); + const socket = response.webSocket; + if (socket === null) { + throw new Error(`expected an executor WebSocket, received ${response.status}`); + } + socket.accept(); + return socket; +} + +/** The platform socket, bound to the four members the client uses. */ +function ownerSocket(socket: WebSocket): OwnerSocket { + const listeners = new Map(); + return { + send: (data) => socket.send(data), + close: () => socket.close(), + addEventListener(type, listener) { + const forward: EventListener = (event) => listener(event as { data?: unknown }); + listeners.set(listener, forward); + socket.addEventListener(type, forward); + }, + removeEventListener(type, listener) { + const found = listeners.get(listener); + if (found !== undefined) { + socket.removeEventListener(type, found); + } + }, + }; +} + +function repository() { + return { + record: { + name: "app", + locatorFingerprint: locatorFingerprintOf(LOCATOR), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1" as const, + checkoutPath: "/app", + }, + locator: LOCATOR, + }; +} + +/** + * Open one production run binding over a real accepted socket. + * + * Everything the coordinator will use comes from here, together: the client, + * the handle, the runtime and the routed journal. + */ +function* opened(socket: WebSocket): Operation { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + let identifier = 0; + const next = () => `coordinated-${(identifier += 1)}`; + const reads = cloudflareReadLink(connection, next, RUN_ID); + const host = createWorkerFiles(); + return yield* useRemoteRun({ + link: cloudflareRunLink(connection, reads, next, RUN_ID), + reads, + files: host.files, + trees: host.trees, + createFilesystem: (at) => host.workspace(at("/")), + journal: new InMemoryStream(), + }); +} + +describe("the coordinator against a real owner", () => { + it("publishes Files, one mapping and the filtered result as one commit", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const before = await on(stub, (owner) => owner.published()); + const socket = await connect(stub); + + const outcome = await run(function* () { + const opening = yield* opened(socket); + yield* useRemoteWorkspaceEffects(opening); + const effect = createRemoteWorkspaceEffect( + opening, + { type: "workspace", name: "write" }, + function* (filesystem, metadata): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + // The checkout the Repository record names has to be in the Workspace + // this proposal publishes; the owner refuses a mapping to a place the + // root does not contain. + yield* filesystem.mkdir("/app", { mode: 0o755 }); + metadata.insertRepository(repository()); + return "published"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + yield* withRemoteWorkspaceEffects(opening, durableRun(workflow, { stream: opening.journal })); + return yield* opening.journal.readAll(); + }); + // The result travelled inside the commit, so the ordinary journal never + // saw it. + expect(outcome.filter((event) => event.type === "yield")).toHaveLength(0); + + const after = await on(stub, (owner) => owner.published()); + // Content, root, references, mapping, pointer and the journal row moved + // together, and the pointer is no longer where it started. + expect(after["currentRootId"]).not.toBe(before["currentRootId"]); + expect(after["roots"]).toBe(2); + expect(after["repositories"]).toEqual([{ name: "app", checkout_path: "/app" }]); + expect(after["events"]).toEqual([ + expect.objectContaining({ workspace_root_id: after["currentRootId"] }), + ]); + + // A fresh admitted invocation, through the production read link: the owner + // answers with the new root, the new anchor and the retained mapping + // together, and that root materializes to the bytes the effect wrote. + const second = await connect(stub); + const observed = await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(second)); + let identifier = 0; + const next = () => `observe-${(identifier += 1)}`; + const reads = cloudflareReadLink(connection, next, RUN_ID); + const snapshot = yield* reads.invocationSnapshot(); + const host = createWorkerFiles(); + const materialization = yield* useMaterialization( + host.files, + host.trees, + reads, + snapshot.workspaceRootId, + (reason) => { + throw new Error(reason); + }, + ); + const workspace = host.workspace(materialization.at("/")); + return { + workspaceRootId: snapshot.workspaceRootId, + journalEventId: snapshot.journalEventId, + repositories: snapshot.repositories.map((stored) => stored.record.name), + notes: yield* workspace.readTextFile("/NOTES.md"), + }; + }); + expect(observed.workspaceRootId).toBe(after["currentRootId"]); + expect(typeof observed.journalEventId).toBe("string"); + expect(observed.repositories).toEqual(["app"]); + expect(observed.notes).toBe("written by the effect\n"); + }); + + it("commits only the filtered failed result, and moves nothing else", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + const before = await on(stub, (owner) => owner.published()); + const socket = await connect(stub); + + await run(function* () { + const opening = yield* opened(socket); + yield* useRemoteWorkspaceEffects(opening); + const effect = createRemoteWorkspaceEffect( + opening, + { type: "workspace", name: "refuse" }, + function* (filesystem, metadata): Operation { + yield* filesystem.writeFile("/SCRATCH.md", "discarded\n", 0o644); + yield* filesystem.mkdir("/app", { mode: 0o755 }); + metadata.insertRepository(repository()); + throw new DocumentedFailure("this Workspace effect refused"); + }, + ); + function* workflow(): Workflow { + yield effect; + } + try { + yield* withRemoteWorkspaceEffects( + opening, + durableRun(workflow, { stream: opening.journal }), + ); + } catch { + // The documented failure is the run's outcome; what it left behind is + // the claim being made. + } + }); + + const after = await on(stub, (owner) => owner.published()); + // The pointer did not move, no second root was retained, and the mapping + // the effect staged never became one. + expect(after["currentRootId"]).toBe(before["currentRootId"]); + expect(after["roots"]).toBe(before["roots"]); + expect(after["repositories"]).toEqual([]); + // One row, and it names the root the run is still on. + expect(after["events"]).toEqual([ + expect.objectContaining({ workspace_root_id: before["currentRootId"] }), + ]); + }); +}); + +/** A refusal the effect publishes rather than raises, as a document's would be. */ +class DocumentedFailure extends JournaledEffectFailure { + override name = "DocumentedFailure"; +} diff --git a/packages/workflow/tests/cloudflare/support/worker-files.ts b/packages/workflow/tests/cloudflare/support/worker-files.ts new file mode 100644 index 000000000..c694c34fa --- /dev/null +++ b/packages/workflow/tests/cloudflare/support/worker-files.ts @@ -0,0 +1,293 @@ +/** + * A filesystem for the runner half, inside the worker. + * + * The owner in these tests is real: a real Durable Object, real SQLite, a real + * accepted WebSocket, and the production client and coordinator on the other + * end of it. The runner's *host filesystem* cannot be. workerd has no native + * filesystem, and the vendored DOFS cannot set a modification time — so it + * cannot reproduce a retained mtime, which is exactly what materialization + * refuses a host for. + * + * So this stands in for the one thing the runtime cannot provide, and nothing + * else. It is not a model of the owner, and it is not a model of the + * coordinator: it stores modes, whole-second times, symbolic links and hardlink + * identity the way a filesystem does, and the production + * `materializeWorkspaceRoot`, `captureWorkspace` and coordinator run against it + * unchanged. The native adapter this stands in for is proved against real files + * in `packages/workflow/tests/remote-workspace-files.test.ts`. + */ + +import { type Operation } from "effection"; +import type { RunnerFiles, RunnerNode } from "../../../src/remote/materialize.ts"; +import type { TemporaryTrees } from "../../../src/remote/invocation.ts"; +import type { + WorkspaceEntry, + WorkspaceFilesystem, + WorkspaceStat, +} from "../../../src/workspace/filesystem.ts"; + +/** One file's bytes, shared by every path hardlinked to it. */ +interface Content { + bytes: Uint8Array; + readonly identity: string; +} + +interface Node { + kind: "directory" | "file" | "symlink"; + mode: number; + mtime: number; + content?: Content; + target?: string; +} + +function failure(code: string): Error { + const error = new Error(`the Workspace operation failed (${code})`); + error.name = "WorkspaceFsError"; + Reflect.set(error, "code", code); + return error; +} + +function parentOf(path: string): string { + const at = path.lastIndexOf("/"); + return at <= 0 ? "/" : path.slice(0, at); +} + +/** One tree, addressed by absolute path. */ +export function createWorkerFiles(): { + files: RunnerFiles; + trees: TemporaryTrees; + workspace(root: string): WorkspaceFilesystem; +} { + // The tree's own root exists from the start, the way a filesystem's does. + const nodes = new Map([["/", { kind: "directory", mode: 0o755, mtime: 0 }]]); + let identities = 0; + let roots = 0; + let clock = 1_700_000_000; + + function node(path: string): Node { + const found = nodes.get(path); + if (found === undefined) { + throw failure("ENOENT"); + } + return found; + } + + function requireParent(path: string): void { + const parent = nodes.get(parentOf(path)); + if (parent === undefined || parent.kind !== "directory") { + throw failure("ENOENT"); + } + } + + function children(path: string): string[] { + const prefix = path === "/" ? "/" : `${path}/`; + return [...nodes.keys()].filter( + (candidate) => + candidate !== path && + candidate.startsWith(prefix) && + !candidate.slice(prefix.length).includes("/"), + ); + } + + function describe(path: string, name: string): RunnerNode { + const held = node(path); + return { + name, + kind: held.kind, + mode: held.mode, + mtime: held.mtime, + size: held.content?.bytes.length ?? held.target?.length ?? 0, + identity: held.content?.identity, + target: held.target, + }; + } + + function nameOf(path: string): string { + return path.slice(path.lastIndexOf("/") + 1); + } + + const files: RunnerFiles = { + // deno-lint-ignore require-yield + *makeDirectory(path, mode): Operation { + if (nodes.has(path)) { + throw failure("EEXIST"); + } + if (path !== "/") { + requireParent(path); + } + nodes.set(path, { kind: "directory", mode, mtime: (clock += 1) }); + }, + + // deno-lint-ignore require-yield + *writeFile(path, bytes, mode): Operation { + requireParent(path); + identities += 1; + nodes.set(path, { + kind: "file", + mode, + mtime: (clock += 1), + content: { bytes: new Uint8Array(bytes), identity: `content-${identities}` }, + }); + }, + + // deno-lint-ignore require-yield + *makeSymlink(target, path): Operation { + requireParent(path); + nodes.set(path, { kind: "symlink", mode: 0o777, mtime: (clock += 1), target }); + }, + + // deno-lint-ignore require-yield + *makeHardlink(existing, path): Operation { + requireParent(path); + const source = node(existing); + if (source.content === undefined) { + throw failure("EPERM"); + } + // The same content, so both paths are one file and capture sees it. + nodes.set(path, { + kind: "file", + mode: source.mode, + mtime: source.mtime, + content: source.content, + }); + }, + + // deno-lint-ignore require-yield + *setMode(path, mode): Operation { + node(path).mode = mode; + }, + + // deno-lint-ignore require-yield + *setModifiedAt(path, mtime): Operation { + node(path).mtime = mtime; + }, + + setLinkModifiedAt: function* (path, mtime): Operation { + node(path).mtime = mtime; + }, + + setLinkMode: function* (path, mode): Operation { + node(path).mode = mode; + }, + + // deno-lint-ignore require-yield + *readFile(path): Operation { + const held = node(path); + if (held.content === undefined) { + throw failure("EISDIR"); + } + return new Uint8Array(held.content.bytes); + }, + + // deno-lint-ignore require-yield + *list(path): Operation { + const held = node(path); + if (held.kind !== "directory") { + throw failure("ENOTDIR"); + } + return children(path).map((child) => describe(child, nameOf(child))); + }, + + // deno-lint-ignore require-yield + *describe(path): Operation { + return describe(path, nameOf(path)); + }, + }; + + const trees: TemporaryTrees = { + *create(purpose): Operation { + roots += 1; + const root = `/${purpose}-${roots}`; + yield* files.makeDirectory(root, 0o755); + return root; + }, + + // deno-lint-ignore require-yield + *remove(path): Operation { + for (const candidate of [...nodes.keys()]) { + if (candidate === path || candidate.startsWith(`${path}/`)) { + nodes.delete(candidate); + } + } + }, + }; + + /** + * The Workspace filesystem over one tree in it. + * + * Containment is the native adapter's subject and is proved there; what this + * needs to be is a filesystem the coordinator can really change, so a commit + * carries bytes a document actually wrote. + */ + function workspace(root: string): WorkspaceFilesystem { + const at = (logical: string) => (logical === "/" ? root : `${root}${logical}`); + function stat(path: string): WorkspaceStat { + const held = node(path); + return { + kind: held.kind, + mode: held.mode, + mtime: held.mtime, + size: held.content?.bytes.length ?? 0, + }; + } + return { + *readFile(path): Operation { + return yield* files.readFile(at(path)); + }, + *readTextFile(path): Operation { + return new TextDecoder().decode(yield* files.readFile(at(path))); + }, + // deno-lint-ignore require-yield + *stat(path): Operation { + return stat(at(path)); + }, + // deno-lint-ignore require-yield + *lstat(path): Operation { + return stat(at(path)); + }, + // deno-lint-ignore require-yield + *readlink(path): Operation { + const target = node(at(path)).target; + if (target === undefined) { + throw failure("EINVAL"); + } + return target; + }, + // deno-lint-ignore require-yield + *readdir(path): Operation { + return children(at(path)).map((child) => ({ + name: nameOf(child), + kind: node(child).kind, + })); + }, + *writeFile(path, content, mode): Operation { + const bytes = typeof content === "string" ? new TextEncoder().encode(content) : content; + yield* files.writeFile(at(path), bytes, mode ?? 0o644); + }, + *mkdir(path, options = {}): Operation { + yield* files.makeDirectory(at(path), options.mode ?? 0o755); + }, + *remove(path): Operation { + yield* trees.remove(at(path)); + }, + // deno-lint-ignore require-yield + *rename(from, to): Operation { + const held = node(at(from)); + nodes.delete(at(from)); + nodes.set(at(to), held); + }, + // deno-lint-ignore require-yield + *chmod(path, mode): Operation { + node(at(path)).mode = mode; + }, + *symlink(target, path): Operation { + yield* files.makeSymlink(target, at(path)); + }, + *link(existing, path): Operation { + yield* files.makeHardlink(at(existing), at(path)); + }, + }; + } + + return { files, trees, workspace }; +} diff --git a/packages/workflow/tests/remote-workspace-files.test.ts b/packages/workflow/tests/remote-workspace-files.test.ts new file mode 100644 index 000000000..221280586 --- /dev/null +++ b/packages/workflow/tests/remote-workspace-files.test.ts @@ -0,0 +1,181 @@ +/** + * Tier WRH — what the runner's Workspace filesystem will act on. + * + * The attempt is a real directory on a host that has an outside, and a symbolic + * link is a path the kernel follows on its own. So these use real temporary + * files and the production adapter: a fake filesystem would follow whatever the + * fake decided to follow, which is the one thing under test. + * + * The rule is the host provider's, stated in `packages/runtime/host-files.ts`: + * a complete `..` segment leaves and `..notes.md` does not; an operation about + * a link does not follow it; and a link's target is a Workspace path, so an + * absolute one names the Workspace root rather than the machine's. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { type Operation, until } from "effection"; +import { mkdir, readFile, symlink, writeFile } from "node:fs/promises"; +import { useRunnerTrees } from "../src/deno/remote-files.ts"; +import { createRemoteWorkspaceFilesystem } from "../src/deno/remote-workspace-files.ts"; +import type { WorkspaceFilesystem } from "../src/workspace/filesystem.ts"; + +const SECRET = "the file outside\n"; + +interface Scene { + readonly files: WorkspaceFilesystem; + readonly attempt: string; + readonly outside: string; +} + +/** + * An attempt directory, and a separate directory it must never reach. + * + * Both are real, and the outside one holds a file whose bytes are recognizable: + * an escape that succeeded would return exactly them. + */ +function* scene(): Operation { + const trees = yield* useRunnerTrees(); + const attempt = yield* trees.create("attempt"); + const outside = yield* trees.create("outside"); + yield* until(writeFile(`${outside}/secret.txt`, SECRET, { mode: 0o644 })); + yield* until(mkdir(`${attempt}/docs`, { mode: 0o755 })); + yield* until(writeFile(`${attempt}/docs/inside.txt`, "inside\n", { mode: 0o644 })); + const files = createRemoteWorkspaceFilesystem( + (logical) => (logical === "/" ? attempt : `${attempt}${logical}`), + () => {}, + ); + return { files, attempt, outside }; +} + +/** What an operation refused with, having proved it refused at all. */ +function* refusal(operation: Operation): Operation { + try { + yield* operation; + return "it was allowed"; + } catch (error) { + return String(error); + } +} + +describe("the runner's Workspace filesystem", () => { + it("follows a link inside the attempt, and leaves it a link", function* () { + const { files, attempt } = yield* scene(); + yield* until(symlink("docs/inside.txt", `${attempt}/here`)); + + expect(yield* files.readTextFile("/here")).toBe("inside\n"); + // The contract of each: one is about the file, the other about the entry. + expect((yield* files.stat("/here")).kind).toBe("file"); + expect((yield* files.lstat("/here")).kind).toBe("symlink"); + expect(yield* files.readlink("/here")).toBe("docs/inside.txt"); + + yield* files.writeFile("/here", "through the link\n", 0o644); + // The file the link names changed; the link is still a link. + expect(yield* until(readFile(`${attempt}/docs/inside.txt`, "utf8"))).toBe("through the link\n"); + expect((yield* files.lstat("/here")).kind).toBe("symlink"); + }); + + it("reads a Workspace-absolute link target against the attempt, not the host", function* () { + const { files, attempt } = yield* scene(); + // The target is a Workspace path. Interpreted by the kernel it would be a + // machine path; interpreted here it is this attempt's own `/docs`. + yield* until(symlink("/docs/inside.txt", `${attempt}/logical`)); + expect(yield* files.readTextFile("/logical")).toBe("inside\n"); + }); + + it("exposes nothing through a final link that leaves the attempt", function* () { + const { files, attempt, outside } = yield* scene(); + yield* until(symlink(`${outside}/secret.txt`, `${attempt}/escape`)); + yield* until(symlink("../../../../etc/hosts", `${attempt}/relative`)); + + for (const path of ["/escape", "/relative"]) { + // The host-absolute target is a Workspace path here, so it names nothing; + // the relative one climbs out of the tree and is refused. Neither is a + // way to the bytes, which is the claim. + expect([path, yield* refusal(files.readTextFile(path))]).not.toEqual([ + path, + "it was allowed", + ]); + expect(yield* refusal(files.readTextFile(path))).not.toContain("the file outside"); + yield* refusal(files.writeFile(path, "overwritten\n")); + yield* refusal(files.chmod(path, 0o600)); + } + // Neither the outside file nor the link it went through changed. + expect(yield* until(readFile(`${outside}/secret.txt`, "utf8"))).toBe(SECRET); + expect(yield* files.readlink("/escape")).toBe(`${outside}/secret.txt`); + }); + + it("reaches nothing through an ancestor link that leaves the attempt", function* () { + const { files, attempt, outside } = yield* scene(); + yield* until(symlink(outside, `${attempt}/door`)); + yield* until(symlink("../..", `${attempt}/up`)); + + for (const path of ["/door/secret.txt", "/up/anything"]) { + expect(yield* refusal(files.readTextFile(path))).not.toContain("the file outside"); + yield* refusal(files.writeFile(path, "created\n")); + yield* refusal(files.mkdir(path, { recursive: true })); + yield* refusal(files.remove(path)); + yield* refusal(files.chmod(path, 0o600)); + yield* refusal(files.rename("/docs/inside.txt", path)); + yield* refusal(files.link("/docs/inside.txt", path)); + } + expect(yield* until(readFile(`${outside}/secret.txt`, "utf8"))).toBe(SECRET); + // And the file that was there to move is still where it was. + expect(yield* files.readTextFile("/docs/inside.txt")).toBe("inside\n"); + }); + + it("contains both ends of a rename and a hardlink", function* () { + const { files, attempt, outside } = yield* scene(); + yield* until(symlink(outside, `${attempt}/door`)); + + expect(yield* refusal(files.rename("/docs/inside.txt", "/../moved"))).toContain( + "outside the tree", + ); + expect(yield* refusal(files.link("/docs/inside.txt", "/../linked"))).toContain( + "outside the tree", + ); + yield* refusal(files.rename("/door/secret.txt", "/taken")); + yield* refusal(files.link("/door/secret.txt", "/taken")); + // Nothing arrived, and nothing left. + expect(yield* refusal(files.readTextFile("/taken"))).not.toContain("the file outside"); + expect(yield* files.readTextFile("/docs/inside.txt")).toBe("inside\n"); + }); + + it("does not turn a dangling outward link into a way to write outside", function* () { + const { files, attempt, outside } = yield* scene(); + yield* until(symlink(`${outside}/absent.txt`, `${attempt}/dangling`)); + yield* refusal(files.writeFile("/dangling", "created outside\n")); + // Nothing was created where the link pointed. + expect(yield* refusal(until(readFile(`${outside}/absent.txt`, "utf8")))).toContain("ENOENT"); + // The link is still exactly what it was. + expect(yield* files.readlink("/dangling")).toBe(`${outside}/absent.txt`); + }); + + it("admits an ordinary name beginning with two dots, and refuses a whole segment", function* () { + const { files } = yield* scene(); + yield* files.writeFile("/..notes.md", "two dots is a name\n", 0o644); + expect(yield* files.readTextFile("/..notes.md")).toBe("two dots is a name\n"); + expect(yield* files.readTextFile("/docs/../..notes.md")).toBe("two dots is a name\n"); + + for (const path of ["/..", "/../escaped", "/docs/../../escaped", ""]) { + expect([path, yield* refusal(files.writeFile(path, "no"))]).toEqual([ + path, + expect.stringContaining("outside the tree this invocation owns"), + ]); + } + }); + + it("says nothing about the host in what it refuses with", function* () { + const { files, attempt, outside } = yield* scene(); + yield* until(symlink(`${outside}/secret.txt`, `${attempt}/escape`)); + const reported = [ + yield* refusal(files.readTextFile("/escape")), + yield* refusal(files.readTextFile("/../escaped")), + yield* refusal(files.readTextFile("/docs/absent.txt")), + ].join("\n"); + // Not where this invocation put its tree, and not where a link pointed. + expect(reported).not.toContain(attempt); + expect(reported).not.toContain(outside); + expect(reported).not.toContain("secret.txt"); + }); +}); diff --git a/packages/workflow/tests/remote-workspace.test.ts b/packages/workflow/tests/remote-workspace.test.ts index 9e1a71403..a78db7a5a 100644 --- a/packages/workflow/tests/remote-workspace.test.ts +++ b/packages/workflow/tests/remote-workspace.test.ts @@ -37,7 +37,9 @@ import type { WorkspaceFilesystem } from "../src/workspace/filesystem.ts"; import type { WorkspaceMetadata } from "../src/workspace/metadata.ts"; import { createRemoteWorkspaceEffect, + type RemoteRun, type RemoteWorkspaceMutation, + useRemoteRun, type RemoteWorkspaceRuntime, useRemoteWorkspaceEffects, withRemoteWorkspaceEffects, @@ -254,8 +256,7 @@ function* startingTree(): Operation { } interface Harness { - readonly database: WorkflowRunDatabase; - readonly runtime: RemoteWorkspaceRuntime; + readonly run: RemoteRun; readonly commits: Record[]; refuse(reason: string): void; lose(): void; @@ -272,25 +273,27 @@ interface Harness { */ function* harness( snapshot: (rootId: string) => RemoteInvocationSnapshot | { refused: string } = emptySnapshot, + shared?: CapturedWorkspace, ): Operation { - const captured = yield* startingTree(); + const captured = shared ?? (yield* startingTree()); const owner = ownerOf(captured, () => snapshot(captured.root.rootId)); const transport = wire((request) => owner.answer(request)); const connection = yield* useOwnerConnection(transport.socket); let identifier = 0; const next = () => `request-${(identifier += 1)}`; const reads = cloudflareReadLink(connection, next, RUN_ID); - const link = cloudflareRunLink(connection, reads, next, RUN_ID); - const database = yield* useRemoteRunDatabase(link, yield* link.frontierSnapshot()); - const runtime: RemoteWorkspaceRuntime = { + // The production constructor: the handle, the routed journal and the + // provenance are made together from this one link. + const run = yield* useRemoteRun({ + link: cloudflareRunLink(connection, reads, next, RUN_ID), + reads, files: runnerFiles(), trees: yield* useRunnerTrees(), - reads, createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), - }; + journal: new InMemoryStream(), + }); return { - database, - runtime, + run, commits: owner.commits, refuse: owner.refuse, lose: owner.lose, @@ -315,20 +318,24 @@ function* invocation( mutate: RemoteWorkspaceMutation, ): Operation<{ raised: unknown; events: DurableEvent[] }> { return yield* scoped(function* () { - const stream = new InMemoryStream(); - const routed = routeRemoteRunJournal(held.database, stream); - yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); - const effect = createRemoteWorkspaceEffect(held.database, { type: "workspace", name }, mutate); + yield* useRemoteWorkspaceEffects(held.run); + const effect = createRemoteWorkspaceEffect(held.run, { type: "workspace", name }, mutate); function* workflow(): Workflow { yield effect; } const raised = yield* trapped( - withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: routed })), + withRemoteWorkspaceEffects(held.run, durableRun(workflow, { stream: held.run.journal })), ); - return { raised, events: stream.snapshot() }; + return { raised, events: yield* held.run.journal.readAll() }; }); } +/** A mutation that touches nothing: these tests are about who may run one. */ +// deno-lint-ignore require-yield +function* own(): Operation { + return "ran"; +} + function yielded(events: readonly DurableEvent[]): DurableEvent[] { return events.filter((event) => event.type === "yield"); } @@ -388,15 +395,11 @@ describe("the runner's Workspace coordinator", () => { expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); - // A later invocation materializes the same root it always had: the - // discarded attempt left nothing, and nothing was promoted. - const observed = yield* invocation(held, "observe", function* (filesystem): Operation { - const names = yield* filesystem.readdir("/"); - return names.map((entry) => entry.name).toSorted(); - }); - expect(observed.raised).toBe(undefined); - const seen = held.commits[1] ?? {}; - expect(seen["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); + // That the owner still holds the starting root after this is a claim + // about storage, and it is made against real owner storage in + // `remote-workspace.vitest.ts`. What is settled here is that nothing was + // proposed: no publication, no mapping, and the root this commit expected + // is the one the invocation was admitted from. }); }); @@ -456,66 +459,132 @@ describe("the runner's Workspace coordinator", () => { }); }); - it("refuses an execution identity another database claimed", function* () { + it("cannot pair one run's handle with another run's link, journal or provenance", function* () { yield* scoped(function* () { - const held = yield* harness(); - const other = yield* harness(); + // Two owners, deliberately begun from the same root and the same empty + // journal. Every structural value they hold is equal; only the objects + // differ, and only the objects decide. + const tree = yield* startingTree(); + const a = yield* harness(emptySnapshot, tree); + const b = yield* harness(emptySnapshot, tree); + expect(a.run.database.record.runId).toBe(b.run.database.record.runId); + + // An effect made against A, coordinated under B. yield* scoped(function* () { - const stream = new InMemoryStream(); - const routed = routeRemoteRunJournal(held.database, stream); - yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); - // Created against one handle, coordinated under another. - const effect = createRemoteWorkspaceEffect( - other.database, - { type: "workspace", name: "foreign" }, - // deno-lint-ignore require-yield - function* (): Operation { - return "ran"; - }, - ); + yield* useRemoteWorkspaceEffects(b.run); + const effect = createRemoteWorkspaceEffect(a.run, { type: "workspace", name: "a" }, own); function* workflow(): Workflow { yield effect; } const raised = yield* trapped( - withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: routed })), + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: b.run.journal })), ); expect(String(raised)).toContain("foreign"); }); - expect(held.commits).toEqual([]); - expect(other.commits).toEqual([]); + + // A's coordinator installed, B's binding asked to use it. + yield* scoped(function* () { + yield* useRemoteWorkspaceEffects(a.run); + const effect = createRemoteWorkspaceEffect(b.run, { type: "workspace", name: "b" }, own); + function* workflow(): Workflow { + yield effect; + } + const raised = yield* trapped( + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: b.run.journal })), + ); + expect(String(raised)).toContain("no remote Workspace coordinator is installed"); + }); + + // B throughout, running over A's journal. The provenance is A's. + yield* scoped(function* () { + yield* useRemoteWorkspaceEffects(b.run); + const effect = createRemoteWorkspaceEffect(b.run, { type: "workspace", name: "c" }, own); + function* workflow(): Workflow { + yield effect; + } + const raised = yield* trapped( + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: a.run.journal })), + ); + expect(String(raised)).toContain("provenance"); + }); + + // A value shaped like a binding is not one. + const forged = { database: b.run.database, journal: b.run.journal }; + expect( + String(yield* trapped(useRemoteWorkspaceEffects(forged as unknown as RemoteRun))), + ).toContain("not a remote run this build opened"); + + // Neither owner was asked for anything, and neither journal moved. + expect([a.commits, b.commits]).toEqual([[], []]); + expect(yielded(yield* a.run.journal.readAll())).toEqual([]); + expect(yielded(yield* b.run.journal.readAll())).toEqual([]); }); }); - it("refuses a journal whose provenance is not this run's", function* () { + it("refuses before the split, not after: the other run's journal stays empty", function* () { yield* scoped(function* () { - const held = yield* harness(); + // The discriminator. Before this correction, B's transaction would enlist + // the Workspace while the publication appended through A's journal, and a + // refusal from B would leave A holding an event for a commit that never + // happened. The refusal has to come first. + const tree = yield* startingTree(); + const a = yield* harness(emptySnapshot, tree); + const b = yield* harness(emptySnapshot, tree); + b.refuse("command:stale-root"); + yield* scoped(function* () { - const stream = new InMemoryStream(); - const routed = routeRemoteRunJournal(held.database, stream); - yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); + yield* useRemoteWorkspaceEffects(b.run); const effect = createRemoteWorkspaceEffect( - held.database, - { type: "workspace", name: "provenance" }, - // deno-lint-ignore require-yield - function* (): Operation { + b.run, + { type: "workspace", name: "split" }, + function* (filesystem): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); return "ran"; }, ); function* workflow(): Workflow { yield effect; } - // A different stream: structurally identical, and not this run's journal. - const foreign = new InMemoryStream(); - establishJournalProvenance(foreign); const raised = yield* trapped( - withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: foreign })), + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: a.run.journal })), ); expect(String(raised)).toContain("provenance"); }); - expect(held.commits).toEqual([]); + + // No commit reached either owner, and A holds no event for work that + // happened somewhere else. + expect([a.commits, b.commits]).toEqual([[], []]); + expect(yielded(yield* a.run.journal.readAll())).toEqual([]); }); }); + it("refuses a binding whose scope has closed", function* () { + let retained: RemoteRun | undefined; + yield* scoped(function* () { + retained = (yield* harness()).run; + }); + if (retained === undefined) { + throw new Error("expected a binding"); + } + // The value outlived the scope that opened it; what it names did not. + const held = retained; + expect((yield* held.database.replaceRetrievalMetadata({ a: 1 })).ok).toBe(false); + const raised = yield* trapped( + scoped(function* () { + yield* useRemoteWorkspaceEffects(held); + const effect = createRemoteWorkspaceEffect(held, { type: "workspace", name: "late" }, own); + function* workflow(): Workflow { + yield effect; + } + return yield* withRemoteWorkspaceEffects( + held, + durableRun(workflow, { stream: held.journal }), + ); + }), + ); + expect(raised).not.toBe(undefined); + }); + it("refuses a Files capability kept past the invocation that owned it", function* () { yield* scoped(function* () { const held = yield* harness(); @@ -578,11 +647,9 @@ describe("the runner's Workspace coordinator", () => { it("sends nothing and keeps nothing when the invocation is cancelled", function* () { yield* scoped(function* () { const held = yield* harness(); - const stream = new InMemoryStream(); - const routed = routeRemoteRunJournal(held.database, stream); - yield* useRemoteWorkspaceEffects(held.runtime, establishJournalProvenance(routed)); + yield* useRemoteWorkspaceEffects(held.run); const effect = createRemoteWorkspaceEffect( - held.database, + held.run, { type: "workspace", name: "cancelled" }, function* (filesystem): Operation { yield* filesystem.writeFile("/SLOW.md", "in progress\n", 0o644); @@ -594,13 +661,13 @@ describe("the runner's Workspace coordinator", () => { yield effect; } const task = yield* spawn(() => - withRemoteWorkspaceEffects(held.database, durableRun(workflow, { stream: routed })), + withRemoteWorkspaceEffects(held.run, durableRun(workflow, { stream: held.run.journal })), ); yield* sleep(0); yield* task.halt(); // Cancellation is control flow: nothing was claimed, and nothing was sent. expect(held.commits).toEqual([]); - expect(yielded(stream.snapshot())).toHaveLength(0); + expect(yielded(yield* held.run.journal.readAll())).toHaveLength(0); }); }); }); diff --git a/packages/workflow/tsconfig.cloudflare.json b/packages/workflow/tsconfig.cloudflare.json index d78979841..ad86bd059 100644 --- a/packages/workflow/tsconfig.cloudflare.json +++ b/packages/workflow/tsconfig.cloudflare.json @@ -3,7 +3,7 @@ "target": "ES2022", "module": "ESNext", "moduleResolution": "bundler", - "lib": ["ES2022"], + "lib": ["ES2023"], "types": ["@cloudflare/workers-types", "@cloudflare/vitest-plugin/types"], "strict": true, "allowImportingTsExtensions": true, From 5b04ad217f79eff5704efabed4ef1b6f1ff724ec Mon Sep 17 00:00:00 2001 From: Min Kim Date: Fri, 4 Sep 2026 21:27:13 -0400 Subject: [PATCH 47/81] =?UTF-8?q?=F0=9F=90=9B=20Make=20one=20owner=20link?= =?UTF-8?q?=20one=20value,=20so=20a=20run=20cannot=20be=20opened=20from=20?= =?UTF-8?q?two?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `useRemoteRun()` took the database/commit link and the Workspace read link as two unrelated members. A caller could pass B's link and A's reads and get a legitimate, module-created binding: the invocation would be admitted from A's retained mappings and content, and if both owners began at the same root and anchor the start-anchor check passed and the result, the Files proposal and the typed deltas committed to B. There is one member now. `cloudflareRunLink()` builds its own read link from the same connection and returns one value satisfying both halves, so the reads an invocation is admitted from and the commits it publishes are the same authority by construction rather than by a check. The runtime's read view is derived from that object — `frontier` names two different reads on the two contracts, and materialization wants the coherent one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SP9dEis51dcwBnwdaXZQAg --- packages/workflow/src/cloudflare/client.ts | 15 ++- packages/workflow/src/remote/database.ts | 21 +++- packages/workflow/src/remote/workspace.ts | 27 ++++- .../tests/cloudflare/remote-owner.vitest.ts | 21 +--- .../cloudflare/remote-workspace.vitest.ts | 4 +- packages/workflow/tests/remote-read.test.ts | 63 ++---------- .../workflow/tests/remote-workspace.test.ts | 99 ++++++++++++++++++- 7 files changed, 161 insertions(+), 89 deletions(-) diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index f5d55a800..0ae1b4092 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -64,7 +64,7 @@ import { JOURNAL_PAGE_ENTRIES, MAX_CONTENT_BYTES, } from "./commands.ts"; -import type { RemoteRunLink } from "../remote/database.ts"; +import type { RemoteRunLink, RemoteWorkspaceLink } from "../remote/database.ts"; import { isSchemaVersion, SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; import { canonicalJson } from "../storage/record.ts"; import { @@ -631,14 +631,23 @@ export function* stageCloudflareContent( * it becomes a semantic value, and every failure crosses as a provider-neutral * storage error rather than as a private refusal. */ +/** + * One run's whole owner link, from one connection. + * + * The read link is made here rather than accepted, so the reads a Workspace + * invocation is admitted from and the commits it publishes cannot be two + * different owners. A caller holding this holds one authority. + */ export function cloudflareRunLink( connection: OwnerConnection, - reads: RemoteReadLink, nextId: () => string, expectedRunId: string, -): RemoteRunLink { +): RemoteWorkspaceLink { + const reads = cloudflareReadLink(connection, nextId, expectedRunId); const publication = cloudflareOwnerLink(connection, reads, nextId); return { + ...reads, + /** * Both halves of the publication link, translated. * diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 74282936f..4967fb3d2 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -50,7 +50,9 @@ import type { } from "../storage/record.ts"; import { createTransactionGate, type OwnerLink, transactRemotely } from "./collector.ts"; import type { EnlistWorkspace, TransactionAnchor } from "./collector.ts"; -import type { RemoteFrontierSnapshot } from "./read.ts"; +import type { RemoteContent, RemoteContentRequest, RemoteFrontierSnapshot } from "./read.ts"; +import type { RemoteInvocationSnapshot } from "./records.ts"; +import type { WorkspaceRootManifest } from "../workspace/root-manifest.ts"; /** What a remote handle needs to answer everything the interface asks. */ export interface RemoteRunLink extends OwnerLink { @@ -65,6 +67,23 @@ export interface RemoteRunLink extends OwnerLink { readExecutions(): Operation>; } +/** + * Everything one remote run is reached through, as one value. + * + * The Workspace reads and the commits are the same authority, so they are the + * same object. Carried as two — a link and a read link a caller supplies + * separately — they can be taken from two owners: an invocation would then + * execute against one run's retained mappings and content and commit the + * result to another, and if the two began at the same root and anchor nothing + * downstream could notice. There is no such pair to make. + */ +export interface RemoteWorkspaceLink extends RemoteRunLink { + /** The one coherent admitted state a Workspace invocation begins from. */ + invocationSnapshot(): Operation; + root(workspaceRootId: string): Operation; + content(workspaceRootId: string, request: RemoteContentRequest): Operation; +} + /** * Which handles this scope is inside a transaction on. * diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts index ce11d180c..14f477e54 100644 --- a/packages/workflow/src/remote/workspace.ts +++ b/packages/workflow/src/remote/workspace.ts @@ -66,7 +66,7 @@ import type { RemoteInvocationSnapshot } from "./records.ts"; import { withRemoteJournalRoute } from "./journal-route.ts"; import { resource } from "effection"; import { establishJournalProvenance, type DurableStream } from "@executablemd/durable-streams"; -import { useRemoteRunDatabase, type RemoteRunLink } from "./database.ts"; +import { useRemoteRunDatabase, type RemoteWorkspaceLink } from "./database.ts"; import { routeRemoteRunJournal } from "./journal-route.ts"; import type { TemporaryTrees } from "./invocation.ts"; @@ -195,9 +195,13 @@ const bindings = (() => { /** What a host supplies to open one remote run. */ export interface RemoteRunOptions { - /** The owner link this run's database, reads and commits all go through. */ - readonly link: RemoteRunLink; - readonly reads: RemoteReadLink; + /** + * The one owner link this run's database, reads and commits go through. + * + * Deliberately one member. A separate read link could be another owner's, + * and an invocation admitted from one run would commit to the other. + */ + readonly link: RemoteWorkspaceLink; readonly files: RunnerFiles; readonly trees: TemporaryTrees; createFilesystem(at: HostPath, authorize: () => void): WorkspaceFilesystem; @@ -227,7 +231,10 @@ export function useRemoteRun(options: RemoteRunOptions): Operation { runtime: { files: options.files, trees: options.trees, - reads: options.reads, + // A view of the same object the database and the commits came from, + // not a second link: `frontier` names two different reads on the two + // contracts, and materialization wants the coherent one. + reads: readsOf(options.link), createFilesystem: options.createFilesystem, }, }), @@ -235,6 +242,16 @@ export function useRemoteRun(options: RemoteRunOptions): Operation { }); } +/** The read half of one owner link, presented the way materialization reads it. */ +function readsOf(link: RemoteWorkspaceLink): RemoteReadLink { + return { + frontier: () => link.frontierSnapshot(), + root: (workspaceRootId) => link.root(workspaceRootId), + content: (workspaceRootId, request) => link.content(workspaceRootId, request), + invocationSnapshot: () => link.invocationSnapshot(), + }; +} + interface ProviderApi { readonly provider: object | undefined; } diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index bded8699e..1138f2bab 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -358,12 +358,7 @@ describe("the remote owner protocol", () => { const outcome = await run(function* () { const connection = yield* useOwnerConnection(wire); const ids = () => `read-${(identifier += 1)}`; - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, ids, RUN_ID), - ids, - RUN_ID, - ); + const link = cloudflareRunLink(connection, ids, RUN_ID); const first = yield* link.readExecutions(); return { first, second: yield* link.readExecutions() }; }); @@ -430,12 +425,7 @@ describe("the remote owner protocol", () => { const outcome = await run(function* () { const connection = yield* useOwnerConnection(wire); const ids = () => `page-${(identifier += 1)}`; - return yield* cloudflareRunLink( - connection, - cloudflareReadLink(connection, ids, RUN_ID), - ids, - RUN_ID, - ).readExecutions(); + return yield* cloudflareRunLink(connection, ids, RUN_ID).readExecutions(); }); if (!outcome.ok) { throw outcome.error; @@ -459,12 +449,7 @@ describe("the remote owner protocol", () => { const refused = await run(function* () { const connection = yield* useOwnerConnection(ownerSocket(alone)); const ids = () => `huge-${(count += 1)}`; - return yield* cloudflareRunLink( - connection, - cloudflareReadLink(connection, ids, RUN_ID), - ids, - RUN_ID, - ).readExecutions(); + return yield* cloudflareRunLink(connection, ids, RUN_ID).readExecutions(); }); expect(refused.ok).toBe(false); // Provider-neutral, with no private refusal spelling in it. diff --git a/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts b/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts index 9ca022cd7..433108066 100644 --- a/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts @@ -132,11 +132,9 @@ function* opened(socket: WebSocket): Operation { const connection = yield* useOwnerConnection(ownerSocket(socket)); let identifier = 0; const next = () => `coordinated-${(identifier += 1)}`; - const reads = cloudflareReadLink(connection, next, RUN_ID); const host = createWorkerFiles(); return yield* useRemoteRun({ - link: cloudflareRunLink(connection, reads, next, RUN_ID), - reads, + link: cloudflareRunLink(connection, next, RUN_ID), files: host.files, trees: host.trees, createFilesystem: (at) => host.workspace(at("/")), diff --git a/packages/workflow/tests/remote-read.test.ts b/packages/workflow/tests/remote-read.test.ts index d4166c05d..bff7e4342 100644 --- a/packages/workflow/tests/remote-read.test.ts +++ b/packages/workflow/tests/remote-read.test.ts @@ -392,12 +392,7 @@ describe("semantic reads from a Cloudflare owner", () => { let outcome: unknown; yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, ids(), RUN_ID), - ids(), - RUN_ID, - ); + const link = cloudflareRunLink(connection, ids(), RUN_ID); outcome = yield* link.replaceRetrieval(ROOT_ID, '{"locator":"what was asked"}'); }); expect((outcome as { ok: boolean }).ok).toBe(false); @@ -433,12 +428,7 @@ describe("semantic reads from a Cloudflare owner", () => { // One generator for both halves: two would mint the same correlation id // and the connection would fail closed on the duplicate. const next = ids(); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, next, RUN_ID), - next, - RUN_ID, - ); + const link = cloudflareRunLink(connection, next, RUN_ID); const database = yield* useRemoteRunDatabase(link, yield* link.frontierSnapshot()); refused = yield* database.replaceRetrievalMetadata({ locator: "what was asked" }); held = database.retrieval; @@ -490,12 +480,7 @@ describe("semantic reads from a Cloudflare owner", () => { // One generator for both halves: two would mint the same correlation id // and the connection would fail closed on the duplicate. const next = ids(); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, next, RUN_ID), - next, - RUN_ID, - ); + const link = cloudflareRunLink(connection, next, RUN_ID); const database = yield* useRemoteRunDatabase(link, yield* link.frontierSnapshot()); refused = yield* database.replaceRetrievalMetadata({ locator: "m".repeat(MAX_MESSAGE_BYTES - 64), @@ -526,12 +511,7 @@ describe("semantic reads from a Cloudflare owner", () => { yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); const next = ids(); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, next, RUN_ID), - next, - RUN_ID, - ); + const link = cloudflareRunLink(connection, next, RUN_ID); outcome = yield* link.readExecutions(); }); const failed = outcome as { ok: boolean; error: Error }; @@ -559,12 +539,7 @@ describe("semantic reads from a Cloudflare owner", () => { yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); const next = ids(); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, next, RUN_ID), - next, - RUN_ID, - ); + const link = cloudflareRunLink(connection, next, RUN_ID); outcome = yield* link.readExecutions(); }); const failed = outcome as { ok: boolean; error: Error }; @@ -709,12 +684,7 @@ describe("semantic reads from a Cloudflare owner", () => { let outcome: unknown; yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, ids(), RUN_ID), - ids(), - RUN_ID, - ); + const link = cloudflareRunLink(connection, ids(), RUN_ID); outcome = yield* link.readExecutions(); }); expect([description, (outcome as { ok: boolean }).ok]).toEqual([description, false]); @@ -756,12 +726,7 @@ describe("semantic reads from a Cloudflare owner", () => { yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); const next = ids(); - outcome = yield* cloudflareRunLink( - connection, - cloudflareReadLink(connection, next, RUN_ID), - next, - RUN_ID, - ).readExecutions(); + outcome = yield* cloudflareRunLink(connection, next, RUN_ID).readExecutions(); }); const found = outcome as { ok: boolean; value: { executionId: string }[] }; expect(found.ok).toBe(true); @@ -798,12 +763,7 @@ describe("semantic reads from a Cloudflare owner", () => { ); yield* scoped(function* () { const connection = yield* useOwnerConnection(transport.socket); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, ids(), RUN_ID), - ids(), - RUN_ID, - ); + const link = cloudflareRunLink(connection, ids(), RUN_ID); const read = yield* link.readExecutions(); expect(read.ok).toBe(true); if (read.ok) { @@ -817,12 +777,7 @@ describe("semantic reads from a Cloudflare owner", () => { })); yield* scoped(function* () { const connection = yield* useOwnerConnection(empty.socket); - const link = cloudflareRunLink( - connection, - cloudflareReadLink(connection, ids(), RUN_ID), - ids(), - RUN_ID, - ); + const link = cloudflareRunLink(connection, ids(), RUN_ID); const read = yield* link.readExecutions(); expect(read.ok && read.value).toEqual([]); }); diff --git a/packages/workflow/tests/remote-workspace.test.ts b/packages/workflow/tests/remote-workspace.test.ts index a78db7a5a..341dcb4b6 100644 --- a/packages/workflow/tests/remote-workspace.test.ts +++ b/packages/workflow/tests/remote-workspace.test.ts @@ -38,6 +38,7 @@ import type { WorkspaceMetadata } from "../src/workspace/metadata.ts"; import { createRemoteWorkspaceEffect, type RemoteRun, + type RemoteRunOptions, type RemoteWorkspaceMutation, useRemoteRun, type RemoteWorkspaceRuntime, @@ -255,6 +256,40 @@ function* startingTree(): Operation { ); } +/** + * The constructor takes one link and no separate read input. + * + * A type-level assertion rather than a runtime one, because that is where the + * property lives: adding a `reads` member back to `RemoteRunOptions` — the + * shape this correction removed — stops this file compiling. + */ +type NoSeparateReads = "reads" extends keyof RemoteRunOptions ? never : true; +const ONE_LINK: NoSeparateReads = true; + +/** + * One scripted owner and a live connection to it, with nothing built on top. + * + * The harness below opens a binding; this stops short of that, so a test can + * hold two owners and ask what each one was actually sent. + */ +function* owner(captured: CapturedWorkspace) { + const scripted = ownerOf(captured, () => emptySnapshot(captured.root.rootId)); + const requests: Record[] = []; + const transport = wire((request) => { + requests.push(request); + return scripted.answer(request); + }); + const connection = yield* useOwnerConnection(transport.socket); + let identifier = 0; + return { + connection, + requests, + next: () => `owner-${(identifier += 1)}`, + commits: scripted.commits, + journal: new InMemoryStream(), + }; +} + interface Harness { readonly run: RemoteRun; readonly commits: Record[]; @@ -281,12 +316,10 @@ function* harness( const connection = yield* useOwnerConnection(transport.socket); let identifier = 0; const next = () => `request-${(identifier += 1)}`; - const reads = cloudflareReadLink(connection, next, RUN_ID); - // The production constructor: the handle, the routed journal and the - // provenance are made together from this one link. + // The production constructor: one link, and the handle, the routed journal + // and the provenance made together from it. const run = yield* useRemoteRun({ - link: cloudflareRunLink(connection, reads, next, RUN_ID), - reads, + link: cloudflareRunLink(connection, next, RUN_ID), files: runnerFiles(), trees: yield* useRunnerTrees(), createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), @@ -521,6 +554,62 @@ describe("the runner's Workspace coordinator", () => { }); }); + it("cannot be opened from one owner's reads and another owner's commits", function* () { + yield* scoped(function* () { + // The construction the correction closes. Two owners, deliberately begun + // from one captured tree, so their root, anchor and run record are equal + // and only the objects differ. Before this, `useRemoteRun` took the + // database/commit link and the Workspace read link separately, and this + // combination produced a legitimate binding: the invocation would be + // admitted from A's mappings and content and commit its result to B. + const tree = yield* startingTree(); + const a = yield* owner(tree); + const b = yield* owner(tree); + + // Each link is built from one connection and carries its own reads, so + // reading through one reaches that owner and no other. + const linkA = cloudflareRunLink(a.connection, a.next, RUN_ID); + const linkB = cloudflareRunLink(b.connection, b.next, RUN_ID); + yield* linkA.invocationSnapshot(); + expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); + expect(b.requests).toEqual([]); + + expect(ONE_LINK).toBe(true); + const options: RemoteRunOptions = { + link: linkB, + files: runnerFiles(), + trees: yield* useRunnerTrees(), + createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), + journal: new InMemoryStream(), + }; + + let executed = 0; + const run = yield* useRemoteRun(options); + yield* useRemoteWorkspaceEffects(run); + const effect = createRemoteWorkspaceEffect( + run, + { type: "workspace", name: "one-owner" }, + function* (filesystem): Operation { + executed += 1; + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + return "ran"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + yield* withRemoteWorkspaceEffects(run, durableRun(workflow, { stream: run.journal })); + + // Everything the invocation read and everything it committed went to B. + // A answered the one snapshot this test asked it for directly, and + // nothing else: no root, no content, no staging, no commit. + expect(executed).toBe(1); + expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); + expect(b.commits).toHaveLength(1); + expect(yielded(yield* a.journal.readAll())).toEqual([]); + }); + }); + it("refuses before the split, not after: the other run's journal stays empty", function* () { yield* scoped(function* () { // The discriminator. Before this correction, B's transaction would enlist From 74b8a280f82d779d90ce41d6f009196f8bc9fab8 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Sat, 5 Sep 2026 19:26:39 -0400 Subject: [PATCH 48/81] =?UTF-8?q?=E2=9C=A8=20Find=20a=20run=20on=20its=20o?= =?UTF-8?q?wner,=20or=20create=20it=20exactly=20once?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first D4 slice: `WorkflowRunStorageApi` over an admitted owner link. The Durable Object is the run's storage, so "where is this run" was already answered by routing. What is left is what the local provider asks of a file — is there a run here, and is it this run — asked in one `open` command so the answer cannot be assembled from two moments. A lookup makes nothing; a creation initializes the schema, the run record, the canonical empty Workspace and the pointer that selects it in one transaction, and repeating a compatible request returns the run that is already there. Opening lives on the link rather than beside it. An opener supplied separately could have been admitted for another owner, and a create authorized by one would then return a handle that reads and commits through the other; matching run ids would make that look correct. One value, so there is nothing to pair. Four conditions stay four conditions. Nothing stored here is absent; storage this build cannot use is foreign; a record that will not parse is damage; and a record that parses and names another run is that other run, reported as a mismatch rather than as corruption — an operator told "damaged" goes looking for a backup that will not help. The distinction is made at the open boundary alone: inside an open run a record that changed identity underneath it is a different fact, and those reads are unchanged. A conflicting creation answers with the exact immutable fields that differ and none of their values — a closed list, strictly parsed against the fields this build reads and the order it reads them in. What travels is the parsed request, never the object it came from: a getter could answer one identity while it is validated and another while it is serialized. Idempotence needs no retained decision. A repeat finds the run the first call created and compares immutable identity, which is the same answer; there is no pristine store left to fill twice. The creation-request parser and the canonical empty manifest move to shared modules. Both providers admit the same request and start from the same Workspace, and a second definition of either would be the more permissive one deciding what a run is. --- packages/workflow/src/cloudflare/client.ts | 178 +++++++-- packages/workflow/src/cloudflare/commands.ts | 49 ++- .../workflow/src/cloudflare/dispatcher.ts | 19 + .../workflow/src/cloudflare/owner-open.ts | 152 ++++++++ packages/workflow/src/cloudflare/owner.ts | 5 + packages/workflow/src/deno/provider.ts | 89 +---- .../workflow/src/deno/workspace/manifest.ts | 5 +- packages/workflow/src/remote/database.ts | 13 + packages/workflow/src/remote/storage.ts | 75 ++++ .../workflow/src/storage/create-request.ts | 112 ++++++ .../workflow/src/workspace/root-manifest.ts | 11 + .../tests/cloudflare/remote-storage.vitest.ts | 367 ++++++++++++++++++ .../cloudflare/support/executor-object.ts | 37 ++ .../workflow/tests/remote-storage.test.ts | 220 +++++++++++ 14 files changed, 1219 insertions(+), 113 deletions(-) create mode 100644 packages/workflow/src/cloudflare/owner-open.ts create mode 100644 packages/workflow/src/remote/storage.ts create mode 100644 packages/workflow/src/storage/create-request.ts create mode 100644 packages/workflow/tests/cloudflare/remote-storage.vitest.ts create mode 100644 packages/workflow/tests/remote-storage.test.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 0ae1b4092..218a1972e 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -65,6 +65,12 @@ import { MAX_CONTENT_BYTES, } from "./commands.ts"; import type { RemoteRunLink, RemoteWorkspaceLink } from "../remote/database.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import { + WorkflowRunConflictError, + WorkflowRunIdMismatchError, + WorkflowRunNotFoundError, +} from "../storage/errors.ts"; import { isSchemaVersion, SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; import { canonicalJson } from "../storage/record.ts"; import { @@ -95,6 +101,8 @@ export type PrivateRefusal = | "command:stale-root" | "command:stale-journal" | "command:mapping-conflict" + | "command:absent" + | "command:wrong-run" | "storage:foreign" | `storage:unsupported-version-v${number}` | "storage:corrupt"; @@ -171,6 +179,8 @@ function privateRefusal(value: string): PrivateRefusal { case "command:stale-root": case "command:stale-journal": case "command:mapping-conflict": + case "command:absent": + case "command:wrong-run": case "storage:foreign": case "storage:corrupt": return value; @@ -327,7 +337,33 @@ export function cloudflareReadLink( connection: OwnerConnection, nextId: () => string, expectedRunId: string, -): RemoteReadLink { +): AnchoringReadLink { + function* anchored(header: FrontierHeader): Operation { + const entries: JournalEntry[] = []; + const seen = new Set(); + let afterEventId: string | null = null; + let done = header.journalEventId === null; + while (!done) { + const page: JournalPage = answer( + yield* connection.ask( + nextId(), + { command: "journal", anchorEventId: header.journalEventId, afterEventId }, + (value) => + parseAnchoredJournalPage(value, header.journalEventId ?? "", afterEventId, seen), + privateRefusal, + ), + ); + for (const item of page.entries) { + const entry = item.entry; + seen.add(entry.eventId); + entries.push(entry); + afterEventId = entry.eventId; + } + done = page.done; + } + return { ...header, entries }; + } + return { *invocationSnapshot(): Operation { return answer( @@ -355,29 +391,18 @@ export function cloudflareReadLink( privateRefusal, ), ); - const entries: JournalEntry[] = []; - const seen = new Set(); - let afterEventId: string | null = null; - let done = header.journalEventId === null; - while (!done) { - const page: JournalPage = answer( - yield* connection.ask( - nextId(), - { command: "journal", anchorEventId: header.journalEventId, afterEventId }, - (value) => - parseAnchoredJournalPage(value, header.journalEventId ?? "", afterEventId, seen), - privateRefusal, - ), - ); - for (const item of page.entries) { - const entry = item.entry; - seen.add(entry.eventId); - entries.push(entry); - afterEventId = entry.eventId; - } - done = page.done; - } - return { ...header, entries }; + return yield* anchored(header); + }, + + /** + * One header's complete journal, read as pages anchored to it. + * + * Separate so the command that opened a run can finish the same coherent + * frontier from the header it already has, rather than asking for the + * header again and assembling a handle from two observations. + */ + *anchored(header: FrontierHeader): Operation { + return yield* anchored(header); }, *root(workspaceRootId: string): Operation { const read = answer( @@ -638,6 +663,66 @@ export function* stageCloudflareContent( * invocation is admitted from and the commits it publishes cannot be two * different owners. A caller holding this holds one authority. */ +/** The read link, plus the paging an open answer finishes its frontier with. */ +interface AnchoringReadLink extends RemoteReadLink { + anchored(header: FrontierHeader): Operation; +} + +/** + * What an owner answers when asked to open a run. + * + * A conflict is an answer rather than a refusal because it carries something: + * the exact immutable fields that differ. The values behind them stay on the + * owner — what differs is enough for a caller to act, and what it differs to + * is the run's own content. + */ +type Opened = + | { readonly kind: "open"; readonly header: FrontierHeader } + | { readonly kind: "conflict"; readonly fields: readonly string[] }; + +/** The immutable fields a creation can differ in, in the order they are read. */ +const CONFLICT_FIELDS: readonly string[] = ["run id", "definition", "base", "props"]; + +function parseOpened(value: unknown, expectedRunId: string, runId: string): Opened { + const found = members(value, ["conflict", "frontier"]); + const conflict = found.get("conflict"); + if (conflict !== null) { + if (found.get("frontier") !== null) { + return fail("an open answer both opened a run and refused one"); + } + return { kind: "conflict", fields: parseConflictFields(conflict) }; + } + const parsed = parseFrontier(found.get("frontier")); + if (parsed.record.runId !== expectedRunId || parsed.record.runId !== runId) { + return fail("an open answer named another run"); + } + return { kind: "open", header: parsed }; +} + +/** + * The differing fields, held to the closed set and the canonical order. + * + * Strict because it decides what a public error says. An unknown name, a + * repeat, an empty list or a reordering is an answer this build cannot read, + * and reading it leniently would put text in an error that nothing produced. + */ +function parseConflictFields(value: unknown): readonly string[] { + if (!Array.isArray(value) || value.length === 0) { + return fail("an open answer named no differing field"); + } + let previous = -1; + const fields: string[] = []; + for (const entry of value) { + const at = typeof entry === "string" ? CONFLICT_FIELDS.indexOf(entry) : -1; + if (at < 0 || at <= previous) { + return fail("an open answer named a differing field this build does not read"); + } + previous = at; + fields.push(CONFLICT_FIELDS[at] ?? ""); + } + return Object.freeze(fields); +} + export function cloudflareRunLink( connection: OwnerConnection, nextId: () => string, @@ -648,6 +733,51 @@ export function cloudflareRunLink( return { ...reads, + /** + * Find this run, or create it exactly once. + * + * The answer is one coherent frontier: the header this command returns, + * and the journal anchored to it. Nothing here reads the frontier a second + * time, so what a handle is built from is one owner observation. + */ + *open( + runId: string, + creation: CreateWorkflowRunRequest | null, + ): Operation> { + try { + const opened = yield* connection.ask( + nextId(), + { command: "open", runId, creation }, + (value) => parseOpened(value, expectedRunId, runId), + privateRefusal, + ); + if (opened.outcome === "refused") { + // Parsed rather than asserted: the answer's refusal is a string + // until this build reads it as one of its own categories. + const refusal = privateRefusal(opened.refusal); + // The one category this command adds. Nothing is stored here, which + // is a different fact from storage this build cannot use. + if (refusal === "command:absent") { + return Err(new WorkflowRunNotFoundError(runId)); + } + if (refusal === "command:wrong-run") { + // Intact storage that belongs to another run. The retained id is + // the other run's business and does not travel. + return Err(new WorkflowRunIdMismatchError(runId, REMOTE_STORE)); + } + return Err(storageFailure(refusal)); + } + if (opened.value.kind === "conflict") { + // The exact fields the owner found differing, and none of their + // values. A run wearing this id is not this run. + return Err(new WorkflowRunConflictError(runId, opened.value.fields)); + } + return Ok(yield* reads.anchored(opened.value.header)); + } catch (error) { + return Err(translate(error)); + } + }, + /** * Both halves of the publication link, translated. * diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 0cdd1f8d9..8f97b8dae 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -12,6 +12,11 @@ import { type WorktreeRecord, } from "../composition/records.ts"; import { type AgentSessionRecord, parseAgentSessionRecord } from "../storage/agent-session.ts"; +import { parseCreateRequest } from "../storage/create-request.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; + +/** The most characters a public run id may carry. */ +const MAX_RUN_ID = 128; import { MAX_MESSAGE_BYTES } from "../remote/client.ts"; export { MAX_MESSAGE_BYTES }; @@ -56,6 +61,7 @@ export type CommandName = | "retrieval" | "executions" | "mappings" + | "open" | "settle"; export type CommandRefusal = @@ -73,7 +79,17 @@ export type CommandRefusal = | "stale-journal" // A retained mapping already exists and describes something else. Creation // identity is immutable, so this is refused rather than rewritten. - | "mapping-conflict"; + | "mapping-conflict" + /** No run is stored here at all. A lookup found nothing, and made nothing. */ + | "absent" + /** + * A run is stored here, and it is not the run this request addresses. + * + * A retained record that parses and names another run. Distinct from damage: + * the storage is intact and this is simply not its run, and a caller that + * conflated them would go looking for a backup. + */ + | "wrong-run"; export class CommandError extends Error { override name = "CommandError"; @@ -204,6 +220,19 @@ export interface MappingsCommand extends CommandEnvelope { readonly command: "mappings"; } +/** + * Find this run, or create it exactly once. + * + * `creation` absent is a lookup and makes nothing. Present, it is the run's + * complete immutable identity, and repeating it is how a caller addresses the + * same run again rather than a second attempt at making one. + */ +export interface OpenCommand extends CommandEnvelope { + readonly command: "open"; + readonly runId: string; + readonly creation: CreateWorkflowRunRequest | null; +} + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -220,6 +249,7 @@ export type RunnerCommand = | RetrievalCommand | ExecutionsCommand | MappingsCommand + | OpenCommand | SettleCommand; export type CommandResult = @@ -246,6 +276,7 @@ const MEMBERS: Record = { retrieval: [...ENVELOPE, "expectedWorkspaceRootId", "metadata"], executions: [...ENVELOPE, "anchor", "after"], mappings: ENVELOPE, + open: [...ENVELOPE, "runId", "creation"], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], }; @@ -376,6 +407,7 @@ export function parseCommand(raw: string): RunnerCommand { command !== "retrieval" && command !== "executions" && command !== "mappings" && + command !== "open" && command !== "settle" ) { throw new CommandError("unknown-command"); @@ -438,6 +470,21 @@ export function parseCommand(raw: string): RunnerCommand { if (command === "mappings") { return { id, command }; } + if (command === "open") { + const runId = text(members, "runId", MAX_RUN_ID); + const offered = members.get("creation"); + if (offered === null) { + return { id, command, runId, creation: null }; + } + // Parsed through the shared request parser, so what the owner will retain + // is what this build calls a creation request rather than an object that + // resembles one. + const creation = parseCreateRequest(offered); + if (!creation.ok || creation.value.runId !== runId) { + throw new CommandError("malformed-member"); + } + return { id, command, runId, creation: creation.value }; + } if (command === "executions") { const anchor = sequence(members, "anchor"); const after = sequence(members, "after"); diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index bf3345ec7..90fc0ef9e 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -56,6 +56,7 @@ import type { OwnerTransactions } from "./owner-transaction.ts"; import { COMMAND_TABLE, MUTATION_TABLE, STAGING_TABLE } from "./private-schema.ts"; import { applyCommit, applyRetrieval } from "./publish.ts"; import { recognizeObject } from "./recognition.ts"; +import { openRun } from "./owner-open.ts"; function requestFingerprint(command: RunnerCommand): string { // The command name is part of the fingerprint, so one textual id used for a @@ -289,12 +290,30 @@ export function dispatchCommand( command: RunnerCommand, ): CommandResult { const held = requireAcquisition(ctx, socket, runId); + if (command.command === "open") { + // Outside the dispatcher's transaction, because creating owns one of its + // own: initialization writes the schema, the run and the starting + // Workspace together, and nesting that inside another transaction would + // be a second one on the same storage. + // + // It needs no retained decision either. A repeat finds the run the first + // call created and compares immutable identity, which is the same answer; + // there is no pristine store left to fill twice. + return { + id: command.id, + outcome: "performed", + value: openRun(ctx.storage, transactions, command.runId, command.creation, ownerTime), + }; + } const fingerprint = requestFingerprint(command); return transactions.run(ctx.storage, () => { const inside = requireAcquisition(ctx, socket, runId); if (inside.acquisitionId !== held.acquisitionId) { throw new CommandError("duplicate-conflict"); } + // Every command that reaches here is asked of a run that already exists, + // so the store is held to this build's schema before it is read. `open` is + // the one that may find nothing, and it answered above. recognizeObject(ctx.storage); // A mutation's decision is looked for by the run, not by the connection. diff --git a/packages/workflow/src/cloudflare/owner-open.ts b/packages/workflow/src/cloudflare/owner-open.ts new file mode 100644 index 000000000..627cd0552 --- /dev/null +++ b/packages/workflow/src/cloudflare/owner-open.ts @@ -0,0 +1,152 @@ +/** + * Opening one run on its owner: finding it, or creating it exactly once. + * + * The Durable Object *is* the run's storage, so "where is this run" has already + * been answered by the time a command arrives — routing chose the object. What + * is left is the same question the local provider asks of a file: is there a + * run here, is it this run, and if there is none may this request start one. + * + * Creation is lookup-or-create, and the identity that decides is the run's + * immutable identity alone: its id, its definition, its base and its normalized + * props. Status, timestamps, executions, roots, mappings, answers, retrieval + * and journal history are things a run *has*, not things it *is*, and a + * creation that differed only in those would be the same run asked for twice. + * + * Everything happens inside one owner transaction. A creation that fails leaves + * either no run at all or the whole run that was already committed: there is no + * state in which an empty candidate exists for inspection to mistake for a run. + */ + +import { canonicalJson } from "../storage/record.ts"; +import { definitionToJson } from "../storage/definition.ts"; +import { conflictingFields } from "../storage/compatibility.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import { readRunRecord } from "../sqlite/rows.ts"; +import { EMPTY_WORKSPACE_MANIFEST, WORKSPACE_ROOT_DOMAIN } from "../workspace/root-manifest.ts"; +import { WORKSPACE_ROOT_FORMAT } from "../workspace/root-manifest.ts"; +import { sha256Hex } from "./encoding.ts"; +import { CommandError } from "./commands.ts"; +import { declaredObjects, initializeObject, recognizeObject } from "./recognition.ts"; +import type { OwnerStorage } from "./storage.ts"; +import type { OwnerTransactions } from "./owner-transaction.ts"; +import { readFrontier, type FrontierValue } from "./owner-reads.ts"; + +const INSERT_RUN = `INSERT INTO workflow_run + (id, run_id, definition, base, props, status, created_at, updated_at) + VALUES (1, ?, ?, ?, ?, 'running', ?, ?)`; + +/** + * What opening answered: the run, or which immutable fields say it is another. + * + * A conflict is an answer rather than a refusal because it carries something a + * refusal category cannot. Exactly one member is present. + */ +export interface OpenedValue { + readonly conflict: readonly string[] | null; + readonly frontier: FrontierValue | null; +} + +/** The root every run starts from, by the identity its bytes produce. */ +export function emptyWorkspaceRootId(): string { + return sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${EMPTY_WORKSPACE_MANIFEST}`); +} + +/** + * Refuse a store that holds a different run, before anything else is read. + * + * The record is parsed by the reader the rest of this build uses, so a row + * that cannot be read is damage and says so; one that reads and names another + * run is this owner answering about somebody else's. + */ +function requireRetainedRun(storage: OwnerStorage, runId: string): void { + const rows = storage.sql + .exec( + `SELECT run_id, definition, base, props, status, + stop_reason_kind, stop_reason_code, stop_reason_event_id, + created_at, updated_at FROM workflow_run`, + ) + .toArray(); + const row = rows[0]; + if (rows.length !== 1 || row === undefined) { + // Not a run at all. The frontier read reports what is wrong with it. + return; + } + if (readRunRecord(row).runId !== runId) { + throw new CommandError("wrong-run"); + } +} + +/** Whether this object holds nothing at all yet. */ +function pristine(storage: OwnerStorage): boolean { + return declaredObjects(storage).length === 0; +} + +/** + * The run this owner holds, or why it holds none this request may use. + * + * `creation` absent is a lookup: it creates nothing, and pristine storage is + * an absent run rather than a foreign one — nothing was ever written here, and + * saying "foreign" would send a host looking for someone else's data. + */ +export function openRun( + storage: OwnerStorage, + transactions: OwnerTransactions, + runId: string, + creation: CreateWorkflowRunRequest | null, + now: () => string, +): OpenedValue { + if (pristine(storage)) { + if (creation === null) { + throw new CommandError("absent"); + } + const stamp = now(); + initializeObject(storage, transactions, () => { + storage.sql.exec( + INSERT_RUN, + creation.runId, + canonicalJson(definitionToJson(creation.definition)), + creation.base, + canonicalJson(creation.props), + stamp, + stamp, + ); + // The Workspace every run starts from, and the pointer that selects it. + // Written here rather than by a later command: a run whose current root + // named nothing would be a run no execution could begin against. + const rootId = emptyWorkspaceRootId(); + storage.sql.exec( + "INSERT INTO workspace_roots (root_id, format_version, manifest) VALUES (?, ?, ?)", + rootId, + WORKSPACE_ROOT_FORMAT, + EMPTY_WORKSPACE_MANIFEST, + ); + storage.sql.exec( + "INSERT INTO workspace_state (singleton_id, current_root_id) VALUES (1, ?)", + rootId, + ); + }); + return { conflict: null, frontier: readFrontier(storage, creation.runId) }; + } + + // Not pristine, so it is held to the schema this build writes before any of + // it is read. A foreign, damaged or newer store refuses as itself. + recognizeObject(storage); + // Asked before the frontier, and only here. An intact store holding another + // run is not damage — the records parse, the references hold, and it is + // simply not this run. Every read *inside* an open run still treats a + // mismatch as damage, because by then the run has already been addressed and + // a record that changed identity underneath it is a different fact. + requireRetainedRun(storage, runId); + const frontier = readFrontier(storage, runId); + if (creation === null) { + return { conflict: null, frontier }; + } + const differing = conflictingFields(frontier.record, creation); + if (differing.length > 0) { + // The same id wearing a different identity. Nothing is written, the run + // that is here stays exactly as it was, and what travels back is which + // fields differ — never what they differ to, which is the run's content. + return { conflict: differing, frontier: null }; + } + return { conflict: null, frontier }; +} diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index 41d985f55..f4b3d6018 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -271,6 +271,11 @@ const ANSWERED: readonly string[] = [ "command:stale-root", "command:stale-journal", "command:mapping-conflict", + // Both are answers about this run rather than faults in the protocol: there + // is nothing stored here, or something is and it is not this run. A caller + // acts on either and goes on using the connection. + "command:absent", + "command:wrong-run", ]; function fatal(answer: CommandResult): boolean { diff --git a/packages/workflow/src/deno/provider.ts b/packages/workflow/src/deno/provider.ts index cdf10fe29..e3d89037b 100644 --- a/packages/workflow/src/deno/provider.ts +++ b/packages/workflow/src/deno/provider.ts @@ -55,7 +55,9 @@ import { parseMembers, requireMemberNames, } from "../storage/members.ts"; -import { canonicalJson, parseRunId, type WorkflowRunRecord } from "../storage/record.ts"; +import { canonicalJson, type WorkflowRunRecord } from "../storage/record.ts"; +import { type CheckedRequest, checkRunId, parseCreateRequest } from "../storage/create-request.ts"; +export { checkRunId } from "../storage/create-request.ts"; import { openWorkflowRunDatabase, readRunRow } from "./database.ts"; import { type RunConnection, @@ -162,20 +164,12 @@ export function authorizedRoot(root: string): string { return root; } -/** A request whose every member has been checked rather than believed. */ -interface CheckedRequest { - readonly runId: string; - readonly definition: WorkflowDefinition; - readonly base: string; - readonly props: JsonObject; -} - function* createWorkflowRun( root: string, connections: WorkflowRunConnections, request: CreateWorkflowRunRequest, ): Operation> { - const checked = checkRequest(request); + const checked = parseCreateRequest(request); if (!checked.ok) { return checked; } @@ -368,78 +362,3 @@ const REQUEST_MEMBERS = ["runId", "definition", "base", "props"]; * without types, or one that read the id out of a file, can hand over anything * at all, and hashing that would fail somewhere far less legible. */ -export function checkRunId(runId: unknown): Result { - try { - return Ok(parseRunId(runId, "$", runIdFailure)); - } catch (error) { - if (error instanceof WorkflowRequestError) { - return Err(error); - } - throw error; - } -} - -function runIdFailure(reason: string): Error { - return new WorkflowRequestError(`${reason}.`); -} - -/** - * The whole request, parsed as a closed shape before any member is read. - * - * The type describes what a caller meant. What arrives is whatever the - * language allows, and reading `.runId` off `null` fails as a `TypeError` - * rather than as an answer about the request. - */ -function checkRequest(offered: CreateWorkflowRunRequest): Result { - let members: Members; - try { - members = parseMembers(offered, "$", requestFailure); - requireMemberNames(members, REQUEST_MEMBERS, "$", requestFailure); - } catch (error) { - if (error instanceof WorkflowRequestError) { - return Err(error); - } - throw error; - } - - const runId = checkRunId(members.get("runId")); - if (!runId.ok) { - return runId; - } - - const base = members.get("base"); - if (typeof base !== "string" || base === "") { - return Err( - new WorkflowRequestError("a base is required: it is what the run's starting state is."), - ); - } - - const definition = parseWorkflowDefinition(members.get("definition")); - if (!definition.ok) { - return definition; - } - - let props: JsonObject; - try { - props = parseJsonObject(members.get("props"), "$", propsFailure); - } catch (error) { - if (error instanceof WorkflowRequestError) { - return Err(error); - } - throw error; - } - - return Ok({ runId: runId.value, definition: definition.value, base, props }); -} - -function requestFailure(reason: string, path: string): Error { - return new WorkflowRequestError( - `the request does not describe a workflow run: ${reason} at ${path}`, - ); -} - -function propsFailure(reason: string, path: string): Error { - return new WorkflowRequestError( - `the normalized props are not a JSON value: ${reason} at ${path}`, - ); -} diff --git a/packages/workflow/src/deno/workspace/manifest.ts b/packages/workflow/src/deno/workspace/manifest.ts index e57194887..d729f6f17 100644 --- a/packages/workflow/src/deno/workspace/manifest.ts +++ b/packages/workflow/src/deno/workspace/manifest.ts @@ -6,6 +6,7 @@ import { SHA256, validateCanonicalWorkspacePath, validateWorkspaceRootEntries, + EMPTY_WORKSPACE_MANIFEST, WORKSPACE_ROOT_DOMAIN, WORKSPACE_ROOT_FORMAT, type WorkspaceRejection, @@ -18,6 +19,7 @@ export { hasUnpairedSurrogate, parentFirst, parentPath, + EMPTY_WORKSPACE_MANIFEST, WORKSPACE_ROOT_DOMAIN, WORKSPACE_ROOT_FORMAT, } from "../../workspace/root-manifest.ts"; @@ -46,9 +48,6 @@ export interface StoredWorkspaceRoot { readonly blobHashes: readonly string[]; } -export const EMPTY_WORKSPACE_MANIFEST = - '{"format":1,"entries":[{"path":"/","kind":"directory","mode":493,"mtime":0}]}'; - export const EMPTY_WORKSPACE_ROOT = workspaceRoot(EMPTY_WORKSPACE_MANIFEST, [], []); export const EMPTY_WORKSPACE_ROOT_ID = EMPTY_WORKSPACE_ROOT.rootId; diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 4967fb3d2..07a0a4acf 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -52,6 +52,7 @@ import { createTransactionGate, type OwnerLink, transactRemotely } from "./colle import type { EnlistWorkspace, TransactionAnchor } from "./collector.ts"; import type { RemoteContent, RemoteContentRequest, RemoteFrontierSnapshot } from "./read.ts"; import type { RemoteInvocationSnapshot } from "./records.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; import type { WorkspaceRootManifest } from "../workspace/root-manifest.ts"; /** What a remote handle needs to answer everything the interface asks. */ @@ -78,6 +79,18 @@ export interface RemoteRunLink extends OwnerLink { * downstream could notice. There is no such pair to make. */ export interface RemoteWorkspaceLink extends RemoteRunLink { + /** + * Find this run on its owner, or create it exactly once. + * + * On the link rather than beside it. An opener supplied separately could + * have been admitted for another owner, and a create authorized by one owner + * would then return a handle that reads and commits through the other. + * Opening and operating are the same authority, so they are the same object. + */ + open( + runId: string, + creation: CreateWorkflowRunRequest | null, + ): Operation>; /** The one coherent admitted state a Workspace invocation begins from. */ invocationSnapshot(): Operation; root(workspaceRootId: string): Operation; diff --git a/packages/workflow/src/remote/storage.ts b/packages/workflow/src/remote/storage.ts new file mode 100644 index 000000000..6c9a68f02 --- /dev/null +++ b/packages/workflow/src/remote/storage.ts @@ -0,0 +1,75 @@ +/** + * Finding and creating a run whose storage is somewhere else. + * + * The local provider answers "where is this run" with a path and then opens a + * file. Here the question is already answered before a command is sent: the + * connection was admitted for one run, so the owner on the other end of it *is* + * that run's storage. What is left is the same pair of questions the local + * provider asks — is there a run here, and is it this run — asked of the owner + * in one command so the answer cannot be assembled from two moments. + * + * `lookup()` creates nothing. `create()` is lookup-or-create, and repeating a + * byte-compatible creation returns the run that is already there rather than + * making a second one. Neither hands back the link, the connection or the + * acquisition: what a caller receives is the same scope-owned + * `WorkflowRunDatabase` the local provider returns. + * + * One argument, deliberately. Opening a run and operating on it are the same + * authority; an opener admitted for one owner paired with another owner's link + * would authorize a create against the first and return a database backed by + * the second, and matching run ids would make that look correct. + */ + +import { Ok, type Operation, type Result } from "effection"; +import type { WorkflowRunDatabase } from "../storage/api.ts"; +import { WorkflowRunStorage } from "../storage/api.ts"; +import { checkRunId, parseCreateRequest } from "../storage/create-request.ts"; +import { useRemoteRunDatabase, type RemoteWorkspaceLink } from "./database.ts"; + +/** + * Install `WorkflowRunStorage` over one admitted owner. + * + * The link is the one the connection created, so every run this provider can + * open is the run that connection was admitted for. There is no registry and + * nothing to route: a request naming another run is refused by the owner, + * which is the only thing that knows what it holds. + */ +export function useRemoteRunStorage(link: RemoteWorkspaceLink): Operation { + return WorkflowRunStorage.around( + { + *create([request]): Operation> { + // Checked here as well as on the owner. A request this build would not + // send is not one it asks an owner to refuse. + const checked = parseCreateRequest(request); + if (!checked.ok) { + return checked; + } + // The parsed value, never the object it came from. A getter could + // answer one identity while this validates and another while the + // request is serialized, and the owner reparsing the second one would + // retain a run nothing here admitted. + const opened = yield* link.open(checked.value.runId, checked.value); + if (!opened.ok) { + return opened; + } + // Built from that exact answer. A second frontier read would assemble + // the handle from two owner observations, and could fail outside the + // `Result` this interface promises. + return Ok(yield* useRemoteRunDatabase(link, opened.value)); + }, + + *lookup([runId]): Operation> { + const checked = checkRunId(runId); + if (!checked.ok) { + return checked; + } + const opened = yield* link.open(checked.value, null); + if (!opened.ok) { + return opened; + } + return Ok(yield* useRemoteRunDatabase(link, opened.value)); + }, + }, + { at: "min" }, + ); +} diff --git a/packages/workflow/src/storage/create-request.ts b/packages/workflow/src/storage/create-request.ts new file mode 100644 index 000000000..21f03fe76 --- /dev/null +++ b/packages/workflow/src/storage/create-request.ts @@ -0,0 +1,112 @@ +/** + * One creation request, parsed as a closed shape before any member is read. + * + * The type describes what a caller meant. What arrives is whatever the language + * allows, and reading `.runId` off `null` fails as a `TypeError` rather than as + * an answer about the request. + * + * Shared because both providers admit the same request: the local one from a + * caller in its own process, the remote one from a command that crossed a + * connection. Two parsers would be two definitions of what a run *is*, and the + * more permissive one would decide. + */ + +import { Err, Ok, type Result } from "effection"; +import { + type JsonObject, + type Members, + parseJsonObject, + parseMembers, + requireMemberNames, +} from "./members.ts"; +import { parseWorkflowDefinition } from "./definition.ts"; +import { WorkflowRequestError } from "./errors.ts"; +import { parseRunId } from "./record.ts"; +import type { CreateWorkflowRunRequest } from "./api.ts"; +import type { WorkflowDefinition } from "./definition.ts"; + +const REQUEST_MEMBERS: readonly string[] = ["runId", "definition", "base", "props"]; + +/** A request whose every member has been checked rather than believed. */ +export interface CheckedRequest { + readonly runId: string; + readonly definition: WorkflowDefinition; + readonly base: string; + readonly props: JsonObject; +} + +export function checkRunId(runId: unknown): Result { + try { + return Ok(parseRunId(runId, "$", runIdFailure)); + } catch (error) { + if (error instanceof WorkflowRequestError) { + return Err(error); + } + throw error; + } +} + +function runIdFailure(reason: string): Error { + return new WorkflowRequestError(`${reason}.`); +} + +/** + * The whole request, parsed as a closed shape before any member is read. + * + * The type describes what a caller meant. What arrives is whatever the + * language allows, and reading `.runId` off `null` fails as a `TypeError` + * rather than as an answer about the request. + */ +export function parseCreateRequest(offered: unknown): Result { + let members: Members; + try { + members = parseMembers(offered, "$", requestFailure); + requireMemberNames(members, REQUEST_MEMBERS, "$", requestFailure); + } catch (error) { + if (error instanceof WorkflowRequestError) { + return Err(error); + } + throw error; + } + + const runId = checkRunId(members.get("runId")); + if (!runId.ok) { + return runId; + } + + const base = members.get("base"); + if (typeof base !== "string" || base === "") { + return Err( + new WorkflowRequestError("a base is required: it is what the run's starting state is."), + ); + } + + const definition = parseWorkflowDefinition(members.get("definition")); + if (!definition.ok) { + return definition; + } + + let props: JsonObject; + try { + props = parseJsonObject(members.get("props"), "$", propsFailure); + } catch (error) { + if (error instanceof WorkflowRequestError) { + return Err(error); + } + throw error; + } + + return Ok({ runId: runId.value, definition: definition.value, base, props }); +} + +function requestFailure(reason: string, path: string): Error { + return new WorkflowRequestError( + `the request does not describe a workflow run: ${reason} at ${path}`, + ); +} + +function propsFailure(reason: string, path: string): Error { + return new WorkflowRequestError( + `the normalized props are not a JSON value: ${reason} at ${path}`, + ); +} diff --git a/packages/workflow/src/workspace/root-manifest.ts b/packages/workflow/src/workspace/root-manifest.ts index 90090d74f..13374a090 100644 --- a/packages/workflow/src/workspace/root-manifest.ts +++ b/packages/workflow/src/workspace/root-manifest.ts @@ -354,3 +354,14 @@ export function validateCanonicalWorkspacePath(value: string, reject: WorkspaceR } } } + +/** + * The Workspace every run starts from. + * + * One directory and nothing in it. Shared rather than written twice: the root + * identity is the hash of these exact bytes, so two spellings of "empty" would + * be two different starting Workspaces, and a run created by one host would not + * be recognized by the other. + */ +export const EMPTY_WORKSPACE_MANIFEST = + '{"format":1,"entries":[{"path":"/","kind":"directory","mode":493,"mtime":0}]}'; diff --git a/packages/workflow/tests/cloudflare/remote-storage.vitest.ts b/packages/workflow/tests/cloudflare/remote-storage.vitest.ts new file mode 100644 index 000000000..30f916d32 --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-storage.vitest.ts @@ -0,0 +1,367 @@ +/** + * Finding and creating a run on its own owner. + * + * Against a real Durable Object, because what is being claimed is that a + * creation either commits whole or leaves the object exactly as it was — and + * "whole" here means the schema, the run record, the starting Workspace and the + * pointer that selects it, written in one transaction the runtime either + * applies or does not. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { call, run, type Operation } from "effection"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; +import { cloudflareRunLink } from "../../src/cloudflare/client.ts"; +import { + type OwnerSocket, + type SocketListener, + useOwnerConnection, +} from "../../src/remote/client.ts"; +import type { CreateWorkflowRunRequest, WorkflowRunDatabase } from "../../src/storage/api.ts"; +import { WorkflowRunStorage } from "../../src/storage/api.ts"; +import { useRemoteRunStorage } from "../../src/remote/storage.ts"; +import type { RemoteWorkspaceLink } from "../../src/remote/database.ts"; +import { + WorkflowDatabaseCorruptError, + WorkflowRunConflictError, + WorkflowRunIdMismatchError, + WorkflowRunNotFoundError, +} from "../../src/storage/errors.ts"; +import type { Result } from "effection"; + +let unique = 0; +const NOW = 1_800_000_000; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`storage-${unique}-${Math.random()}`)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise { + return runInDurableObject(stub, body); +} + +async function connect(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + const response = await stub.fetch("https://owner.invalid/executor", { + headers: { + authorization: `Bearer ${token}`, + upgrade: "websocket", + "x-release": POLICY.release, + "x-run-id": RUN_ID, + }, + }); + const socket = response.webSocket; + if (socket === null) { + throw new Error(`expected an executor WebSocket, received ${response.status}`); + } + socket.accept(); + return socket; +} + +function ownerSocket(socket: WebSocket): OwnerSocket { + const listeners = new Map(); + return { + send: (data) => socket.send(data), + close: () => socket.close(), + addEventListener(type, listener) { + // The event's data is read out and handed over as the small shape the + // listener declares, rather than the runtime event being renamed into + // it: a message carries text, and every other kind carries nothing. + const forward: EventListener = (event) => { + const data: unknown = Reflect.get(event, "data"); + listener(typeof data === "string" ? { data } : {}); + }; + listeners.set(listener, forward); + socket.addEventListener(type, forward); + }, + removeEventListener(type, listener) { + const found = listeners.get(listener); + if (found !== undefined) { + socket.removeEventListener(type, found); + } + }, + }; +} + +function creation(overrides: Partial = {}): CreateWorkflowRunRequest { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + ...overrides, + }; +} + +/** + * The storage provider, installed the way a host installs one. + * + * Through `WorkflowRunStorage.operations` rather than by calling the adapter + * directly: what is under test is the provider a caller reaches, including + * that every failure comes back inside `Result` rather than escaping. + * + * One connection for the whole body. The connection owns the socket and closes + * it when its scope ends, so a second one over the same socket would be asking + * through a channel the first already ended. + */ +function installed(socket: WebSocket, body: () => Operation): Promise { + return run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + let identifier = 0; + yield* useRemoteRunStorage( + cloudflareRunLink(connection, () => `open-${(identifier += 1)}`, RUN_ID), + ); + return yield* body(); + }); +} + +function create(request: CreateWorkflowRunRequest): Operation> { + return WorkflowRunStorage.operations.create(request); +} + +function lookup(runId: string): Operation> { + return WorkflowRunStorage.operations.lookup(runId); +} + +/** + * The provider takes one value, and that value is the link. + * + * A type-level assertion because that is where the property lives: giving + * `useRemoteRunStorage` a second parameter — the opener/link pair this + * correction removed — stops this file compiling. A runtime check could not + * say anything about a shape that no longer exists. + */ +type OneCapability = + Parameters extends [RemoteWorkspaceLink] ? true : never; +const ONE_CAPABILITY: OneCapability = true; + +/** The error a refused result carries, having proved it refused at all. */ +function failure(result: Result): Error { + if (result.ok) { + throw new Error("expected a refused result"); + } + return result.error; +} + +describe("opening a run on its owner", () => { + it("creates once, and answers the same run for a compatible repeat", async () => { + const stub = executor(); + const socket = await connect(stub); + + const outcome = await installed(socket, function* () { + const first = yield* create(creation()); + // The same request again is the same run, not a second one. + const again = yield* create(creation()); + return { first: first.ok, again: again.ok }; + }); + expect(outcome).toEqual({ first: true, again: true }); + + const after = await on(stub, (owner) => owner.published()); + // The schema, the run, the starting Workspace and its pointer, together. + expect(after["roots"]).toBe(1); + expect(after["events"]).toEqual([]); + expect(await on(stub, (owner) => owner.runRow())).not.toBe(null); + }); + + it("names the exact immutable fields that differ, and none of their values", async () => { + const stub = executor(); + const socket = await connect(stub); + + const refusals = await installed(socket, function* () { + yield* create(creation()); + const reported: Error[] = []; + for (const differing of [ + creation({ base: "other-base-entirely" }), + creation({ props: { secret: "do-not-echo-me" } }), + creation({ + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "1".repeat(40), + rootDocumentPath: "README.md", + }, + }), + // Two at once: the list is what differs, in the order they are read. + creation({ base: "other-base-entirely", props: { secret: "do-not-echo-me" } }), + ]) { + reported.push(failure(yield* create(differing))); + } + return reported.map((error) => ({ + conflict: error instanceof WorkflowRunConflictError, + fields: error instanceof WorkflowRunConflictError ? error.fields : undefined, + text: String(error), + })); + }); + + expect(refusals.map((entry) => entry.conflict)).toEqual([true, true, true, true]); + expect(refusals.map((entry) => entry.fields)).toEqual([ + ["base"], + ["props"], + ["definition"], + ["base", "props"], + ]); + for (const entry of refusals) { + // What differs, never what it differs to, and nothing of the protocol. + expect(entry.text).not.toContain("do-not-echo-me"); + expect(entry.text).not.toContain("other-base-entirely"); + expect(entry.text).not.toContain("command:"); + } + // One run, exactly as the compatible creation left it. + expect((await on(stub, (owner) => owner.published()))["roots"]).toBe(1); + }); + + it("looks up without creating, and says so when there is nothing", async () => { + const stub = executor(); + const socket = await connect(stub); + + const seen = await installed(socket, function* () { + const absent = failure(yield* lookup(RUN_ID)); + // Asked before anything else, so what it reports is about an owner that + // has never held a run. + const pristine = yield* call(() => on(stub, (owner) => owner.objectCount())); + yield* create(creation()); + const found = yield* lookup(RUN_ID); + return { + absent: { missing: absent instanceof WorkflowRunNotFoundError, text: String(absent) }, + pristine, + found: found.ok ? found.value.record.status : "refused", + }; + }); + + expect(seen.absent.missing).toBe(true); + // Nothing of the private protocol crossed with it. + expect(seen.absent.text).not.toContain("command:"); + // A lookup that missed created nothing. + expect(seen.pristine).toBe(0); + expect(seen.found).toBe("running"); + }); + + it("says a run stored here is another run, without saying which", async () => { + const stub = executor(); + const socket = await connect(stub); + await installed(socket, () => create(creation())); + // Intact storage, holding somebody else's run. + const other = "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + await on(stub, (owner) => owner.retainAnotherRun(other)); + const before = await on(stub, (owner) => owner.runRow()); + + const again = await connect(stub); + const seen = await installed(again, function* () { + const looked = failure(yield* lookup(RUN_ID)); + const created = failure(yield* create(creation())); + return [looked, created].map((error) => ({ + mismatch: error instanceof WorkflowRunIdMismatchError, + damaged: error instanceof WorkflowDatabaseCorruptError, + text: String(error), + })); + }); + + for (const entry of seen) { + // Not this run, and not damage. + expect([entry.mismatch, entry.damaged]).toEqual([true, false]); + // The retained id belongs to that other run and does not travel. + expect(entry.text).not.toContain(other); + expect(entry.text).not.toContain("command:"); + } + // Nothing was written on the way to either answer. + expect(await on(stub, (owner) => owner.runRow())).toEqual(before); + }); + + it("initializes nothing into storage that is not this build's", async () => { + const stub = executor(); + await on(stub, (owner) => owner.holdForeignObject()); + const socket = await connect(stub); + const refused = await installed(socket, function* () { + const outcome = yield* create(creation()); + return outcome.ok ? "it was allowed" : String(outcome.error); + }); + // A storage condition, inside `Result`, with nothing private in it. + expect(refused).not.toBe("it was allowed"); + expect(refused).not.toContain("storage:"); + // Foreign, and left exactly as it was found. + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("refuses every operation once the provider scope has closed", async () => { + const stub = executor(); + const socket = await connect(stub); + const held = await installed(socket, function* () { + const opened = yield* create(creation()); + return opened.ok ? opened.value : undefined; + }); + if (held === undefined) { + throw new Error("expected a database"); + } + // The handle outlived the scope that opened it; what it names did not, and + // it does not reconnect. + const late = await run(() => held.replaceRetrievalMetadata({ a: 1 })); + expect(late.ok).toBe(false); + }); + + it("cannot open through one owner and answer with the other", async () => { + expect(ONE_CAPABILITY).toBe(true); + // Two owners that look alike: same public run id, same compatible + // creation, identical retained state. Only the objects differ. + const first = executor(); + const second = executor(); + const socketA = await connect(first); + const socketB = await connect(second); + await installed(socketA, () => create(creation())); + await installed(socketB, () => create(creation())); + + // There is one value to install, and it came from one connection. A + // provider built on A's link reads and commits through A, whatever B + // holds — there is no second argument to give it B's. + const again = await connect(first); + const seen = await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(again)); + let identifier = 0; + yield* useRemoteRunStorage( + cloudflareRunLink(connection, () => `pair-${(identifier += 1)}`, RUN_ID), + ); + const opened = yield* lookup(RUN_ID); + if (!opened.ok) { + return "refused"; + } + const written = yield* opened.value.transact(function* (transaction) { + yield* transaction.journal.append({ + type: "yield", + coroutineId: "root", + description: { type: "test", name: "written" }, + result: { status: "ok", value: "written" }, + }); + }); + return written.ok ? "appended" : "refused"; + }); + expect(seen).toBe("appended"); + + // The append landed on the owner whose link was installed, and nowhere else. + expect((await on(first, (owner) => owner.published()))["events"]).toHaveLength(1); + expect((await on(second, (owner) => owner.published()))["events"]).toEqual([]); + }); +}); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index adc2034ec..c0ef42d3c 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -307,6 +307,43 @@ export class ExecutorObject extends WorkflowOwnerObject { } /** Everything a reader could observe about the published frontier. */ + /** The one run row, as it is stored, or nothing when there is none. */ + runRow(): Record | null { + return this.ctx.storage.sql.exec("SELECT * FROM workflow_run").toArray()[0] ?? null; + } + + /** How many objects this storage declares at all. */ + objectCount(): number { + return this.ctx.storage.sql + .exec("SELECT name FROM sqlite_master WHERE name NOT LIKE '_cf_%'") + .toArray().length; + } + + /** Whether this build's schema is here. */ + hasWorkflowSchema(): boolean { + return ( + this.ctx.storage.sql + .exec("SELECT name FROM sqlite_master WHERE name = 'workflow_run'") + .toArray().length > 0 + ); + } + + /** + * Retain a different run here, intact. + * + * The record still parses and every reference still holds; it simply names + * another run. That is a different condition from damage, and a store that + * conflated them would send an operator looking for a backup. + */ + retainAnotherRun(runId: string): void { + this.ctx.storage.sql.exec("UPDATE workflow_run SET run_id = ? WHERE id = 1", runId); + } + + /** Put something here that is not a workflow run, and never was one. */ + holdForeignObject(): void { + this.ctx.storage.sql.exec("CREATE TABLE somebody_else (id INTEGER PRIMARY KEY)"); + } + published(): Record { const state = this.ctx.storage.sql .exec("SELECT current_root_id FROM workspace_state WHERE singleton_id = 1") diff --git a/packages/workflow/tests/remote-storage.test.ts b/packages/workflow/tests/remote-storage.test.ts new file mode 100644 index 000000000..04972abb3 --- /dev/null +++ b/packages/workflow/tests/remote-storage.test.ts @@ -0,0 +1,220 @@ +/** + * Tier WRH — what the remote storage provider answers when the owner does not. + * + * The owner's own behavior is proved against a real Durable Object in + * `tests/cloudflare/remote-storage.vitest.ts`. These are the cases a real owner + * cannot produce: an answer this build cannot read, and a connection that ends + * before it answers. Both go through the production + * `cloudflareRunLink().open()` and the installed `WorkflowRunStorage`, because + * what is claimed is that neither escapes the provider's `Result`. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { type Operation, type Result, scoped } from "effection"; +import { cloudflareRunLink } from "../src/cloudflare/client.ts"; +import { type OwnerSocket, type SocketListener, useOwnerConnection } from "../src/remote/client.ts"; +import { useRemoteRunStorage } from "../src/remote/storage.ts"; +import type { CreateWorkflowRunRequest, WorkflowRunDatabase } from "../src/storage/api.ts"; +import { WorkflowRunStorage } from "../src/storage/api.ts"; +import { WorkflowRecordMalformedError, WorkflowStorageError } from "../src/storage/errors.ts"; + +const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + +function creation(): CreateWorkflowRunRequest { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + }; +} + +function runRecord(): Record { + return { + runId: RUN_ID, + definition: creation().definition, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-05T00:00:00.000Z", + updatedAt: "2026-09-05T00:00:00.000Z", + }; +} + +/** + * One request, read rather than believed. + * + * `JSON.parse` answers `unknown`, and this owner has to reflect a correlation + * id back off whatever it was sent. Checking that it is an object before + * reading one is what makes the reflection honest instead of a promise about + * what the client sends. + */ +function decoded(data: string): Record { + const value: unknown = JSON.parse(data); + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new Error("expected the client to send one JSON object"); + } + return Object.fromEntries(Object.entries(value)); +} + +/** An owner whose answers a test writes, and which can simply stop answering. */ +function wire(answer: (request: Record) => Record | "lost") { + const sent: Record[] = []; + const listeners = new Map>(); + const socket: OwnerSocket = { + send(data: string): void { + const request = decoded(data); + sent.push(request); + const response = answer(request); + if (response === "lost") { + for (const listener of listeners.get("close") ?? []) { + listener({}); + } + return; + } + for (const listener of listeners.get("message") ?? []) { + listener({ data: JSON.stringify({ id: request["id"], ...response }) }); + } + }, + close(): void {}, + addEventListener(type, listener): void { + const found = listeners.get(type) ?? new Set(); + found.add(listener); + listeners.set(type, found); + }, + removeEventListener(type, listener): void { + listeners.get(type)?.delete(listener); + }, + }; + return { socket, sent }; +} + +/** The installed provider, over one scripted owner. */ +function* installed( + answer: (request: Record) => Record | "lost", + body: () => Operation, +): Operation { + return yield* scoped(function* () { + const transport = wire(answer); + const connection = yield* useOwnerConnection(transport.socket); + let identifier = 0; + yield* useRemoteRunStorage( + cloudflareRunLink(connection, () => `open-${(identifier += 1)}`, RUN_ID), + ); + return yield* body(); + }); +} + +function refused(result: Result): Error { + if (result.ok) { + throw new Error("expected a refused result"); + } + return result.error; +} + +describe("remote storage, when the owner answers badly", () => { + it("returns a malformed record for every open answer it cannot read", function* () { + // Each of these is a shape the owner could never build. What matters is + // that reading one is a refusal rather than a value, and that the refusal + // says the record was unreadable rather than repeating what was in it. + const answers: Record> = { + "an answer that both opened and refused": { + conflict: ["base"], + frontier: { + record: runRecord(), + retrieval: null, + workspaceRootId: "a".repeat(64), + journalEventId: null, + }, + }, + "a differing field this build does not read": { conflict: ["everything"], frontier: null }, + "no differing field at all": { conflict: [], frontier: null }, + "differing fields out of order": { conflict: ["props", "base"], frontier: null }, + "one differing field twice": { conflict: ["base", "base"], frontier: null }, + "a differing field that is not text": { conflict: [7], frontier: null }, + "a frontier naming another run": { + conflict: null, + frontier: { + record: { ...runRecord(), runId: "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa" }, + retrieval: null, + workspaceRootId: "a".repeat(64), + journalEventId: null, + }, + }, + "an answer carrying neither": { conflict: null, frontier: null }, + }; + + for (const [description, value] of Object.entries(answers)) { + const outcome = yield* installed( + () => ({ outcome: "performed", value }), + () => WorkflowRunStorage.operations.create(creation()), + ); + const error = refused(outcome); + expect([description, error]).toEqual([description, expect.any(WorkflowRecordMalformedError)]); + // Nothing of the answer, and nothing of the protocol. + expect([description, String(error)]).not.toContain("everything"); + expect(String(error)).not.toContain("command:"); + expect(String(error)).not.toContain("6dktgrv"); + } + }); + + it("returns a provider failure when the connection ends before its answer", function* () { + const outcome = yield* installed( + () => "lost", + () => WorkflowRunStorage.operations.create(creation()), + ); + const error = refused(outcome); + // Inside the `Result` this interface promises, provider-neutral, and + // carrying no transport vocabulary. + expect(error).toEqual(expect.any(WorkflowStorageError)); + expect(String(error)).not.toContain("OwnerLinkError"); + expect(String(error)).not.toContain("command:"); + + const looked = yield* installed( + () => "lost", + () => WorkflowRunStorage.operations.lookup(RUN_ID), + ); + expect(refused(looked)).toEqual(expect.any(WorkflowStorageError)); + }); + + it("sends the request it parsed, not the object it was handed", function* () { + // A request that answers one identity while it is validated and another + // when it is read again. Only the parsed value may reach the owner. + let reads = 0; + const unstable = { + get runId(): string { + return RUN_ID; + }, + definition: creation().definition, + get base(): string { + reads += 1; + return reads > 1 ? "a-later-base" : "main"; + }, + props: {}, + }; + + let observed: unknown; + yield* installed( + (request) => { + if (request["command"] === "open") { + observed = request["creation"]; + } + return { outcome: "performed", value: { conflict: ["base"], frontier: null } }; + }, + () => WorkflowRunStorage.operations.create(unstable), + ); + + const sent = observed === null || typeof observed !== "object" ? {} : { ...observed }; + // The getter ran while the request was parsed, and what travelled is that + // reading. A later reading never reached the owner. + expect(reads).toBe(1); + expect(Reflect.get(sent, "base")).toBe("main"); + }); +}); From 37bd50fae6d7dd21e2d255bbfde5a0eb2c058ea1 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Sat, 5 Sep 2026 20:06:18 -0400 Subject: [PATCH 49/81] =?UTF-8?q?=E2=9C=A8=20Decide=20a=20run's=20lifecycl?= =?UTF-8?q?e=20once,=20and=20keep=20it=20on=20the=20owner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both hosts store a run's lifecycle in the same schema and have to reach the same conclusions about it: whether a caller may continue, what a dead executor's unfinished execution became, and whether beginning publishes `running` or leaves an outcome that already won alone. Those decisions move to `lifecycle/policy.ts` as functions over values, and the Deno adapter now reaches them there instead of holding its own copy — ninety lines of it. Where the rows live stays each host's business; what they mean does not. The owner implements begin, settle and cancel over those same decisions. Each is one transaction: recovery closes whatever the previous executor left, on the run's own retained evidence, then admission decides whether this caller may continue, then an admitted caller's execution is inserted. Splitting them would publish a recovery that a refusal then had to take back. Beginning a run that does not exist creates it in that same commit, so a reader observes the whole begun run or no run at all. An unfinished execution is stale because the connection that owned it is gone, never because time passed. A retained root Close restores the outcome the document recorded; without one the executor was interrupted. A replay closes only its own envelope and leaves the terminal outcome exactly as it is. Refusals that carry a reason travel as closed answers rather than refusal spellings: which immutable fields differ, or which retained condition applies. All three commands are retried through the existing decision ledger, so a commit whose answer was lost is observed rather than applied twice. --- packages/workflow/src/cloudflare/client.ts | 2 + packages/workflow/src/cloudflare/commands.ts | 65 ++- .../workflow/src/cloudflare/dispatcher.ts | 56 ++- .../src/cloudflare/owner-lifecycle.ts | 385 ++++++++++++++++++ .../workflow/src/cloudflare/owner-open.ts | 87 ++-- .../workflow/src/cloudflare/recognition.ts | 39 +- packages/workflow/src/deno/transitions.ts | 101 +---- packages/workflow/src/lifecycle/policy.ts | 131 ++++++ .../tests/cloudflare/remote-owner.vitest.ts | 21 +- 9 files changed, 743 insertions(+), 144 deletions(-) create mode 100644 packages/workflow/src/cloudflare/owner-lifecycle.ts create mode 100644 packages/workflow/src/lifecycle/policy.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 218a1972e..0ba58c632 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -103,6 +103,7 @@ export type PrivateRefusal = | "command:mapping-conflict" | "command:absent" | "command:wrong-run" + | "command:corrupt-journal" | "storage:foreign" | `storage:unsupported-version-v${number}` | "storage:corrupt"; @@ -181,6 +182,7 @@ function privateRefusal(value: string): PrivateRefusal { case "command:mapping-conflict": case "command:absent": case "command:wrong-run": + case "command:corrupt-journal": case "storage:foreign": case "storage:corrupt": return value; diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 8f97b8dae..fbc054038 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -62,6 +62,8 @@ export type CommandName = | "executions" | "mappings" | "open" + | "begin" + | "cancel" | "settle"; export type CommandRefusal = @@ -89,7 +91,9 @@ export type CommandRefusal = * the storage is intact and this is simply not its run, and a caller that * conflated them would go looking for a backup. */ - | "wrong-run"; + | "wrong-run" + /** Retained journal history this owner cannot read. */ + | "corrupt-journal"; export class CommandError extends Error { override name = "CommandError"; @@ -233,6 +237,28 @@ export interface OpenCommand extends CommandEnvelope { readonly creation: CreateWorkflowRunRequest | null; } +/** Begin one document execution under the live acquisition. */ +export interface BeginCommand extends CommandEnvelope { + readonly command: "begin"; + readonly runId: string; + readonly action: "start" | "resume"; + readonly creation: CreateWorkflowRunRequest | null; + /** + * The execution's identity, minted by the runner. + * + * Minted there rather than here so the command is the same bytes on a retry: + * an owner that invented one would begin a second execution for a request it + * had already answered. + */ + readonly executionId: string; +} + +/** Make one run terminal, following what it retains. */ +export interface CancelCommand extends CommandEnvelope { + readonly command: "cancel"; + readonly runId: string; +} + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -250,6 +276,8 @@ export type RunnerCommand = | ExecutionsCommand | MappingsCommand | OpenCommand + | BeginCommand + | CancelCommand | SettleCommand; export type CommandResult = @@ -277,6 +305,8 @@ const MEMBERS: Record = { executions: [...ENVELOPE, "anchor", "after"], mappings: ENVELOPE, open: [...ENVELOPE, "runId", "creation"], + begin: [...ENVELOPE, "runId", "action", "creation", "executionId"], + cancel: [...ENVELOPE, "runId"], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], }; @@ -408,6 +438,8 @@ export function parseCommand(raw: string): RunnerCommand { command !== "executions" && command !== "mappings" && command !== "open" && + command !== "begin" && + command !== "cancel" && command !== "settle" ) { throw new CommandError("unknown-command"); @@ -485,6 +517,37 @@ export function parseCommand(raw: string): RunnerCommand { } return { id, command, runId, creation: creation.value }; } + if (command === "begin") { + const runId = text(members, "runId", MAX_RUN_ID); + const action = members.get("action"); + if (action !== "start" && action !== "resume") { + throw new CommandError("malformed-member"); + } + const offered = members.get("creation"); + let creation: CreateWorkflowRunRequest | null = null; + if (offered !== null) { + const parsed = parseCreateRequest(offered); + if (!parsed.ok || parsed.value.runId !== runId) { + throw new CommandError("malformed-member"); + } + creation = parsed.value; + } + // A start that creates carries its creation; a resume never does. + if (action === "resume" && creation !== null) { + throw new CommandError("malformed-member"); + } + return { + id, + command, + runId, + action, + creation, + executionId: text(members, "executionId", MAX_RUN_ID), + }; + } + if (command === "cancel") { + return { id, command, runId: text(members, "runId", MAX_RUN_ID) }; + } if (command === "executions") { const anchor = sequence(members, "anchor"); const after = sequence(members, "after"); diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 90fc0ef9e..27ea40521 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -52,11 +52,12 @@ import { readJournalPage, readRoot, } from "./owner-reads.ts"; -import type { OwnerTransactions } from "./owner-transaction.ts"; +import type { OwnerTransaction, OwnerTransactions } from "./owner-transaction.ts"; import { COMMAND_TABLE, MUTATION_TABLE, STAGING_TABLE } from "./private-schema.ts"; import { applyCommit, applyRetrieval } from "./publish.ts"; import { recognizeObject } from "./recognition.ts"; import { openRun } from "./owner-open.ts"; +import { beginRun, cancelRunOnOwner, settleRun } from "./owner-lifecycle.ts"; function requestFingerprint(command: RunnerCommand): string { // The command name is part of the fingerprint, so one textual id used for a @@ -73,7 +74,16 @@ function requestFingerprint(command: RunnerCommand): string { * the next connection can find it. */ function mutating(command: RunnerCommand): boolean { - return command.command === "commit" || command.command === "retrieval"; + return ( + command.command === "commit" || + command.command === "retrieval" || + // Each of these changes the run's own lifecycle, and each can commit + // before its answer is observed. A retry has to find the first decision + // rather than apply the transition again. + command.command === "begin" || + command.command === "cancel" || + command.command === "settle" + ); } function integer(value: unknown): number { @@ -214,6 +224,7 @@ function perform( runId: string, acquisitionId: string, command: RunnerCommand, + transaction: OwnerTransaction, ): CommandResult { if (command.command === "frontier") { return { id: command.id, outcome: "performed", value: readFrontier(ctx.storage, runId) }; @@ -269,6 +280,41 @@ function perform( value: readExecutions(ctx.storage, runId, command.anchor, command.after), }; } + if (command.command === "begin") { + return { + id: command.id, + outcome: "performed", + value: beginRun( + ctx.storage, + transaction, + command.runId, + command.action, + command.creation, + command.executionId, + ownerTime, + ), + }; + } + if (command.command === "cancel") { + return { + id: command.id, + outcome: "performed", + value: cancelRunOnOwner(ctx.storage, command.runId, ownerTime), + }; + } + if (command.command === "settle") { + return { + id: command.id, + outcome: "performed", + value: settleRun( + ctx.storage, + runId, + command.completion, + command.expectedWorkspaceRootId, + ownerTime, + ), + }; + } if (command.command === "mappings") { return { id: command.id, @@ -306,7 +352,7 @@ export function dispatchCommand( }; } const fingerprint = requestFingerprint(command); - return transactions.run(ctx.storage, () => { + return transactions.run(ctx.storage, (transaction) => { const inside = requireAcquisition(ctx, socket, runId); if (inside.acquisitionId !== held.acquisitionId) { throw new CommandError("duplicate-conflict"); @@ -355,7 +401,7 @@ export function dispatchCommand( } const decision = storedDecision(previous.response, command.id); return decision === "reconstruct" - ? perform(ctx, runId, held.acquisitionId, command) + ? perform(ctx, runId, held.acquisitionId, command, transaction) : decision; } const usage = ctx.storage.sql @@ -371,7 +417,7 @@ export function dispatchCommand( ) { throw new CommandError("capacity"); } - const result = perform(ctx, runId, held.acquisitionId, command); + const result = perform(ctx, runId, held.acquisitionId, command, transaction); const encoded = retainedDecision(command, result); const responseBytes = new TextEncoder().encode(encoded).length; if (integer(usage?.["bytes"]) + responseBytes > MAX_LEDGER_BYTES) { diff --git a/packages/workflow/src/cloudflare/owner-lifecycle.ts b/packages/workflow/src/cloudflare/owner-lifecycle.ts new file mode 100644 index 000000000..2040c28ac --- /dev/null +++ b/packages/workflow/src/cloudflare/owner-lifecycle.ts @@ -0,0 +1,385 @@ +/** + * The run's lifecycle, as its owner keeps it. + * + * Beginning, settling and cancelling are the three moments a run's own state + * changes, and each is one owner transaction: what a dead executor's unfinished + * execution became, whether this caller may continue, and the execution this + * caller began all commit together or not at all. Splitting them would leave a + * recovery published that a refusal then had to take back, or a window where + * this executor's own execution looks like somebody else's leftovers. + * + * The decisions are not made here. `lifecycle/policy.ts` says what an + * unfinished execution becomes, whether an action is admitted and what + * beginning does; this module is where those conclusions meet rows. Both hosts + * reach them through the same functions, so a run means the same thing + * wherever it is stored. + */ + +import { + admissionRefusal, + beginDecision, + closingOutcome, + rootOutcome, + terminal, +} from "../lifecycle/policy.ts"; +import { conflictingFields } from "../storage/compatibility.ts"; +import { canonicalJson } from "../storage/record.ts"; +import { definitionToJson } from "../storage/definition.ts"; +import type { + DocumentExecutionCompletion, + DocumentExecutionRecord, + WorkflowRunRecord, + WorkflowRunStatus, +} from "../storage/record.ts"; +import type { JournalEntry } from "../storage/api.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import { parseDurableEvent } from "@executablemd/durable-streams"; +import { readDocumentExecution, readRunRecord, type Row } from "../sqlite/rows.ts"; +import { CommandError } from "./commands.ts"; +import type { OwnerStorage } from "./storage.ts"; +import type { OwnerTransaction } from "./owner-transaction.ts"; +import { establishRun } from "./owner-open.ts"; +import { readFrontier, type FrontierValue } from "./owner-reads.ts"; + +const RUN_COLUMNS = `run_id, definition, base, props, status, + stop_reason_kind, stop_reason_code, stop_reason_event_id, created_at, updated_at`; + +const EXECUTION_COLUMNS = `execution_id, started_at, stopped_at, stop_status, + stop_reason_kind, stop_reason_code, stop_reason_event_id`; + +/** + * Why a lifecycle transition would not proceed. + * + * A closed set of categories rather than refusal spellings, because each is a + * fact about the run that a caller acts on, and each maps to an error the + * provider-neutral vocabulary already has. What the run holds — its props, its + * definition, its history — never travels with one. + */ +export type LifecycleRefusal = + /** Resume reached a run that failed. */ + | "resume-failed" + /** Resume or start reached a run that was cancelled. */ + | "cancelled" + /** Cancellation reached a run whose outcome already won. */ + | "terminal"; + +/** What one begin committed, as the runner is allowed to know it. */ +export interface BegunValue { + readonly frontier: FrontierValue; + readonly execution: DocumentExecutionRecord; + readonly replay: boolean; + /** What recovery closed on the way in, when it closed anything. */ + readonly recovered: DocumentExecutionRecord | null; +} + +/** + * One lifecycle answer: what it did, or why it would not. + * + * Exactly one member is present. A refusal is an answer rather than a protocol + * refusal because it carries which condition applied, and because the recovery + * it may have committed on the way in stands either way. + */ +export interface LifecycleValue { + readonly conflict: readonly string[] | null; + readonly refusal: LifecycleRefusal | null; + readonly value: T | null; +} + +function rows(storage: OwnerStorage, sql: string, ...bindings: unknown[]): Row[] { + return storage.sql.exec(sql, ...bindings).toArray(); +} + +function storedRun(storage: OwnerStorage): WorkflowRunRecord { + const row = rows(storage, `SELECT ${RUN_COLUMNS} FROM workflow_run`)[0]; + if (row === undefined) { + throw new CommandError("absent"); + } + return readRunRecord(row); +} + +function unfinished(storage: OwnerStorage): DocumentExecutionRecord[] { + return rows( + storage, + `SELECT ${EXECUTION_COLUMNS} FROM document_executions + WHERE stopped_at IS NULL ORDER BY sequence`, + ).map((row) => readDocumentExecution(row)); +} + +function journalEntries(storage: OwnerStorage): JournalEntry[] { + return rows( + storage, + "SELECT event_id, record, workspace_root_id FROM journal_events ORDER BY sequence", + ).map((row) => { + const parsed = parseDurableEvent(String(row["record"])); + if (!parsed.ok) { + // Retained history this owner cannot read. Deciding an outcome from a + // journal it cannot parse would be deciding from nothing. + throw new CommandError("corrupt-journal"); + } + return { + eventId: String(row["event_id"]), + event: parsed.value, + workspaceRootId: String(row["workspace_root_id"]), + }; + }); +} + +function readExecution(storage: OwnerStorage, executionId: string): DocumentExecutionRecord { + const row = rows( + storage, + `SELECT ${EXECUTION_COLUMNS} FROM document_executions WHERE execution_id = ?`, + executionId, + )[0]; + if (row === undefined) { + throw new CommandError("malformed-member"); + } + return readDocumentExecution(row); +} + +function finish( + storage: OwnerStorage, + completion: { + executionId: string; + status: WorkflowRunStatus; + reason: DocumentExecutionCompletion["reason"]; + }, + now: () => string, +): void { + const reason = completion.reason; + storage.sql.exec( + `UPDATE document_executions + SET stopped_at = ?, stop_status = ?, stop_reason_kind = ?, + stop_reason_code = ?, stop_reason_event_id = ? + WHERE execution_id = ? AND stopped_at IS NULL`, + now(), + completion.status, + reason?.kind ?? null, + reason?.kind === "host" ? reason.code : null, + reason?.kind === "journal" ? reason.eventId : null, + completion.executionId, + ); +} + +function publish( + storage: OwnerStorage, + status: WorkflowRunStatus, + reason: DocumentExecutionCompletion["reason"], + now: () => string, +): void { + storage.sql.exec( + `UPDATE workflow_run + SET status = ?, stop_reason_kind = ?, stop_reason_code = ?, + stop_reason_event_id = ?, updated_at = ? + WHERE id = 1`, + status, + reason?.kind ?? null, + reason?.kind === "host" ? reason.code : null, + reason?.kind === "journal" ? reason.eventId : null, + now(), + ); +} + +function insertExecution( + storage: OwnerStorage, + executionId: string, + now: () => string, +): DocumentExecutionRecord { + storage.sql.exec( + "INSERT INTO document_executions (execution_id, started_at) VALUES (?, ?)", + executionId, + now(), + ); + return readExecution(storage, executionId); +} + +/** + * Close whatever the previous executor left, on the run's own evidence. + * + * Reached only when this caller holds the acquisition and has begun nothing of + * its own, so an unfinished execution is proven stale by the connection that + * owned it being gone — never by elapsed time. + */ +function reconcile( + storage: OwnerStorage, + stored: WorkflowRunRecord, + now: () => string, +): { status: WorkflowRunStatus; recovered: DocumentExecutionRecord | null } { + const open = unfinished(storage); + if (open.length === 0) { + return { status: stored.status, recovered: null }; + } + const closing = closingOutcome(stored.status, rootOutcome(journalEntries(storage))); + let last: DocumentExecutionRecord | null = null; + for (const execution of open) { + finish( + storage, + { executionId: execution.executionId, status: closing.status, reason: closing.reason }, + now, + ); + last = readExecution(storage, execution.executionId); + } + if (!closing.publishes) { + return { status: stored.status, recovered: last }; + } + publish(storage, closing.status, closing.reason, now); + return { status: closing.status, recovered: last }; +} + +/** + * Begin one document execution, in one transaction. + * + * Recovery decides what the previous executor's execution became, admission + * decides whether this caller may continue, and an admitted caller's execution + * is inserted — all or none. + */ +export function beginRun( + storage: OwnerStorage, + transaction: OwnerTransaction, + runId: string, + action: "start" | "resume", + creation: CreateWorkflowRunRequest | null, + executionId: string, + now: () => string, +): LifecycleValue { + if (action === "start" && creation !== null) { + // Creating and beginning are one commit — this runs inside the caller's + // transaction — so a reader observes the whole begun run or no run at all, + // never an initialized candidate with no execution. + const opened = establishRun(storage, transaction, runId, creation, now); + if (opened !== null) { + return { conflict: opened, refusal: null, value: null }; + } + } + + return (() => { + const stored = storedRun(storage); + if (stored.runId !== runId) { + throw new CommandError("wrong-run"); + } + if (creation !== null) { + const differing = conflictingFields(stored, creation); + if (differing.length > 0) { + return { conflict: differing, refusal: null, value: null }; + } + } + + const recovered = reconcile(storage, stored, now); + // The recovery above stays committed whatever this decides: what the + // previous executor's execution became is not undone by this caller being + // told it may not continue. + if (admissionRefusal(action, recovered.status) !== undefined) { + return { + conflict: null, + refusal: recovered.status === "cancelled" ? "cancelled" : "resume-failed", + value: null, + }; + } + + const decision = beginDecision(recovered.status); + const execution = insertExecution(storage, executionId, now); + if (decision.kind === "running") { + publish(storage, "running", undefined, now); + } + return { + conflict: null, + refusal: null, + value: { + frontier: readFrontier(storage, runId), + execution, + replay: decision.kind === "replay", + recovered: recovered.recovered, + }, + }; + })(); +} + +/** + * Finish the execution this acquisition began, and publish what it decided. + * + * The expected root is checked in the same transaction: a settlement built + * against a Workspace the run has moved off describes an execution of + * something else. + */ +export function settleRun( + storage: OwnerStorage, + runId: string, + completion: DocumentExecutionCompletion, + expectedWorkspaceRootId: string, + now: () => string, +): FrontierValue { + return (() => { + const stored = storedRun(storage); + if (stored.runId !== runId) { + throw new CommandError("wrong-run"); + } + const current = rows( + storage, + "SELECT current_root_id FROM workspace_state WHERE singleton_id = 1", + )[0]; + if (current === undefined || String(current["current_root_id"]) !== expectedWorkspaceRootId) { + throw new CommandError("stale-root"); + } + const execution = readExecution(storage, completion.executionId); + if (execution.stoppedAt !== undefined) { + // Already settled. Settling it again would replace an outcome that won. + throw new CommandError("duplicate-conflict"); + } + finish( + storage, + { + executionId: completion.executionId, + status: completion.status, + reason: completion.reason, + }, + now, + ); + // A replay closes only its own envelope: the terminal outcome it observed + // is not made mutable again. + if (!terminal(stored.status)) { + publish(storage, completion.status, completion.reason, now); + } + return readFrontier(storage, runId); + })(); +} + +/** + * Make one run terminal, following what it retains. + * + * Takes no execution of its own. A stale execution is reconciled first, on the + * same evidence a begin would use, so a document that finished before its + * executor disappeared keeps the outcome it recorded. + */ +export function cancelRunOnOwner( + storage: OwnerStorage, + runId: string, + now: () => string, +): LifecycleValue { + return (() => { + const stored = storedRun(storage); + if (stored.runId !== runId) { + throw new CommandError("wrong-run"); + } + if (stored.status === "cancelled") { + // Already what the caller asked for. Saying so twice is the same answer. + return { conflict: null, refusal: null, value: readFrontier(storage, runId) }; + } + if (terminal(stored.status)) { + return { conflict: null, refusal: "terminal", value: null }; + } + const recovered = reconcile(storage, stored, now); + if (terminal(recovered.status) || recovered.status === "cancelled") { + // The document finished before its executor disappeared. Restoring what + // it recorded is not cancelling it. + return { conflict: null, refusal: null, value: readFrontier(storage, runId) }; + } + for (const execution of unfinished(storage)) { + finish( + storage, + { executionId: execution.executionId, status: "cancelled", reason: undefined }, + now, + ); + } + publish(storage, "cancelled", undefined, now); + return { conflict: null, refusal: null, value: readFrontier(storage, runId) }; + })(); +} diff --git a/packages/workflow/src/cloudflare/owner-open.ts b/packages/workflow/src/cloudflare/owner-open.ts index 627cd0552..b3f799fc6 100644 --- a/packages/workflow/src/cloudflare/owner-open.ts +++ b/packages/workflow/src/cloudflare/owner-open.ts @@ -26,9 +26,14 @@ import { EMPTY_WORKSPACE_MANIFEST, WORKSPACE_ROOT_DOMAIN } from "../workspace/ro import { WORKSPACE_ROOT_FORMAT } from "../workspace/root-manifest.ts"; import { sha256Hex } from "./encoding.ts"; import { CommandError } from "./commands.ts"; -import { declaredObjects, initializeObject, recognizeObject } from "./recognition.ts"; +import { + declaredObjects, + initializeInside, + initializeObject, + recognizeObject, +} from "./recognition.ts"; import type { OwnerStorage } from "./storage.ts"; -import type { OwnerTransactions } from "./owner-transaction.ts"; +import type { OwnerTransaction, OwnerTransactions } from "./owner-transaction.ts"; import { readFrontier, type FrontierValue } from "./owner-reads.ts"; const INSERT_RUN = `INSERT INTO workflow_run @@ -46,6 +51,32 @@ export interface OpenedValue { readonly frontier: FrontierValue | null; } +/** + * Create this run, or confirm the one that is here is it. + * + * Runs inside a transaction the caller already opened, so beginning a run can + * create it and record its first execution in one commit. Answers with the + * differing immutable fields when the id wears another identity, and `null` + * when the run is this one. + */ +export function establishRun( + storage: OwnerStorage, + transaction: OwnerTransaction, + runId: string, + creation: CreateWorkflowRunRequest, + now: () => string, +): readonly string[] | null { + if (pristine(storage)) { + const stamp = now(); + initializeInside(storage, transaction, () => insertRun(storage, creation, stamp)); + return null; + } + recognizeObject(storage); + requireRetainedRun(storage, runId); + const differing = conflictingFields(readFrontier(storage, runId).record, creation); + return differing.length > 0 ? differing : null; +} + /** The root every run starts from, by the identity its bytes produce. */ export function emptyWorkspaceRootId(): string { return sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${EMPTY_WORKSPACE_MANIFEST}`); @@ -76,6 +107,32 @@ function requireRetainedRun(storage: OwnerStorage, runId: string): void { } } +/** The run row, the Workspace it starts from, and the pointer that selects it. */ +function insertRun(storage: OwnerStorage, creation: CreateWorkflowRunRequest, stamp: string): void { + storage.sql.exec( + INSERT_RUN, + creation.runId, + canonicalJson(definitionToJson(creation.definition)), + creation.base, + canonicalJson(creation.props), + stamp, + stamp, + ); + // Written with the run rather than by a later command: a run whose current + // root named nothing would be a run no execution could begin against. + const rootId = emptyWorkspaceRootId(); + storage.sql.exec( + "INSERT INTO workspace_roots (root_id, format_version, manifest) VALUES (?, ?, ?)", + rootId, + WORKSPACE_ROOT_FORMAT, + EMPTY_WORKSPACE_MANIFEST, + ); + storage.sql.exec( + "INSERT INTO workspace_state (singleton_id, current_root_id) VALUES (1, ?)", + rootId, + ); +} + /** Whether this object holds nothing at all yet. */ function pristine(storage: OwnerStorage): boolean { return declaredObjects(storage).length === 0; @@ -100,31 +157,7 @@ export function openRun( throw new CommandError("absent"); } const stamp = now(); - initializeObject(storage, transactions, () => { - storage.sql.exec( - INSERT_RUN, - creation.runId, - canonicalJson(definitionToJson(creation.definition)), - creation.base, - canonicalJson(creation.props), - stamp, - stamp, - ); - // The Workspace every run starts from, and the pointer that selects it. - // Written here rather than by a later command: a run whose current root - // named nothing would be a run no execution could begin against. - const rootId = emptyWorkspaceRootId(); - storage.sql.exec( - "INSERT INTO workspace_roots (root_id, format_version, manifest) VALUES (?, ?, ?)", - rootId, - WORKSPACE_ROOT_FORMAT, - EMPTY_WORKSPACE_MANIFEST, - ); - storage.sql.exec( - "INSERT INTO workspace_state (singleton_id, current_root_id) VALUES (1, ?)", - rootId, - ); - }); + initializeObject(storage, transactions, () => insertRun(storage, creation, stamp)); return { conflict: null, frontier: readFrontier(storage, creation.runId) }; } diff --git a/packages/workflow/src/cloudflare/recognition.ts b/packages/workflow/src/cloudflare/recognition.ts index 9368f9e31..19e68ab42 100644 --- a/packages/workflow/src/cloudflare/recognition.ts +++ b/packages/workflow/src/cloudflare/recognition.ts @@ -30,7 +30,7 @@ import { PRIVATE_OBJECT_NAMES, privateStructureFailure, } from "./private-schema.ts"; -import type { OwnerTransactions } from "./owner-transaction.ts"; +import type { OwnerTransaction, OwnerTransactions } from "./owner-transaction.ts"; import type { OwnerStorage } from "./storage.ts"; /** Why storage could not be read as a version-1 workflow run. */ @@ -103,20 +103,35 @@ export function initializeObject( detail: "it already holds objects and carries no workflow schema marker", }); } - transactions.run(storage, ({ dofs }) => { - storage.sql.exec(SCHEMA_SQL); - initializeDofsSchema(dofs, () => 0); - initializePrivateSchema(storage); - initializeRun(); - storage.sql.exec(MARKER_SQL); - storage.sql.exec( - `INSERT INTO ${MARKER_TABLE} (id, application_id, schema_version) VALUES (1, ?, ?)`, - APPLICATION_ID, - SCHEMA_VERSION, - ); + transactions.run(storage, (transaction) => { + initializeInside(storage, transaction, initializeRun); }); } +/** + * The same initialization, inside a transaction the caller already opened. + * + * Beginning a run creates it and records its first execution together, and + * those are one commit; opening a second transaction for the schema would make + * them two, with a window in between holding a run nothing had begun. + */ +export function initializeInside( + storage: OwnerStorage, + transaction: OwnerTransaction, + initializeRun: () => void, +): void { + storage.sql.exec(SCHEMA_SQL); + initializeDofsSchema(transaction.dofs, () => 0); + initializePrivateSchema(storage); + initializeRun(); + storage.sql.exec(MARKER_SQL); + storage.sql.exec( + `INSERT INTO ${MARKER_TABLE} (id, application_id, schema_version) VALUES (1, ?, ?)`, + APPLICATION_ID, + SCHEMA_VERSION, + ); +} + /** * Refuse anything that is not a version-1 workflow run. * diff --git a/packages/workflow/src/deno/transitions.ts b/packages/workflow/src/deno/transitions.ts index 27808c7a7..7e11739c7 100644 --- a/packages/workflow/src/deno/transitions.ts +++ b/packages/workflow/src/deno/transitions.ts @@ -36,6 +36,14 @@ import type { } from "../lifecycle/execution.ts"; import type { WorkflowRunDatabase } from "../storage/api.ts"; import { conflictingFields } from "../storage/compatibility.ts"; +import { + admissionRefusal, + type Closing, + closingOutcome, + INTERRUPTED, + rootOutcome, + terminal, +} from "../lifecycle/policy.ts"; import { definitionToJson } from "../storage/definition.ts"; import { WorkflowDocumentExecutionError, @@ -604,39 +612,6 @@ function firstExecution( return { execution: insertExecution(database, path), replay: false }; } -/** - * Why this action may not continue from this status, or nothing when it may. - * - * Answered rather than raised, and asked outside the transaction that recovered - * the run: refusing is this caller's outcome, not a reason to undo what the - * previous workflow executor's execution was found to have become. - */ -function admissionRefusal( - action: "start" | "resume", - status: WorkflowRunStatus | undefined, -): Error | undefined { - if (status === undefined) { - return undefined; - } - if (action === "resume" && (status === "failed" || status === "cancelled")) { - return new WorkflowRequestError( - `workflow run ${status}: a run that ${ - status === "failed" ? "failed" : "was cancelled" - } is not resumed. The run is left exactly as it is.`, - ); - } - if (status === "cancelled") { - return new WorkflowRequestError( - "workflow run cancelled: a cancelled run reports its retained state and is not advanced.", - ); - } - return undefined; -} - -function terminal(status: WorkflowRunStatus): boolean { - return status === "completed" || status === "failed"; -} - interface Reconciled { readonly status: WorkflowRunStatus; readonly of: { recovered?: DocumentExecutionRecord }; @@ -656,7 +631,7 @@ function reconcile(database: DatabaseSync, path: string, stored: WorkflowRunReco return { status: stored.status }; } - const closing = closingOutcome(database, stored); + const closing = closingOutcome(stored.status, rootOutcome(readJournalEntries(database))); let last: DocumentExecutionRecord | undefined; for (const execution of unfinished) { @@ -675,62 +650,6 @@ function reconcile(database: DatabaseSync, path: string, stored: WorkflowRunReco return { status: closing.status, ...closed }; } -interface Closing { - readonly status: WorkflowRunStatus; - readonly reason: DocumentExecutionCompletion["reason"]; - readonly publishes: boolean; -} - -/** - * What the previous workflow executor's execution became, on the evidence the run holds. - * - * A retained root Close proves the canonical outcome won before anything could - * interrupt it, so it is restored. Failing that, the execution was interrupted: - * the workflow executor went away without recording an outcome, and that is what happened. - */ -function closingOutcome(database: DatabaseSync, stored: WorkflowRunRecord): Closing { - // A replay whose terminal state was preserved closes only its own execution, - // and the authoritative outcome stays exactly as it was. - if (terminal(stored.status)) { - return { status: "interrupted", reason: interrupted, publishes: false }; - } - - const canonical = rootOutcome(database); - if (canonical !== undefined) { - return { status: canonical.status, reason: canonical.reason, publishes: true }; - } - - return { status: "interrupted", reason: interrupted, publishes: true }; -} - -const interrupted = { kind: "host", code: "executor-interrupted" } as const; - -/** - * The canonical outcome the root recorded, when it recorded one. - * - * A root Close is what proves the document itself finished. Its result decides - * the run's terminal status, and its own event identity is the reason — the - * journal already filtered it, so nothing new is retained to say why. - */ -function rootOutcome( - database: DatabaseSync, -): { status: WorkflowRunStatus; reason: DocumentExecutionCompletion["reason"] } | undefined { - for (const entry of readJournalEntries(database)) { - const { event } = entry; - if (event.type !== "close" || event.coroutineId !== "root") { - continue; - } - if (event.result.status === "ok") { - return { status: "completed", reason: undefined }; - } - return { - status: event.result.status === "cancelled" ? "cancelled" : "failed", - reason: { kind: "journal", eventId: entry.eventId }, - }; - } - return undefined; -} - function insertExecution(database: DatabaseSync, path: string): DocumentExecutionRecord { const executionId = randomUUID(); database.prepare(INSERT_EXECUTION).run(executionId, new Date().toISOString()); @@ -916,7 +835,7 @@ export function* cancelRun( ); } - const canonical = rootOutcome(database); + const canonical = rootOutcome(readJournalEntries(database)); if (canonical !== undefined) { // The document finished before its workflow executor disappeared. Restoring what it // recorded is not cancelling it. diff --git a/packages/workflow/src/lifecycle/policy.ts b/packages/workflow/src/lifecycle/policy.ts new file mode 100644 index 000000000..48d7204a7 --- /dev/null +++ b/packages/workflow/src/lifecycle/policy.ts @@ -0,0 +1,131 @@ +/** + * What a lifecycle transition decides, apart from where the rows live. + * + * Both hosts store a run's lifecycle in the same schema and must reach the same + * conclusions about it: whether a caller may continue, what a dead executor's + * unfinished execution became, and whether beginning publishes `running` or + * leaves an outcome that already won alone. Where those rows are read from is + * the host's business — a local SQLite file, or a Durable Object on the other + * end of a connection — but the conclusions are not, and two copies of them + * would eventually disagree about what a run is. + * + * So the decisions live here, as functions over values. Nothing in this module + * reads or writes anything. + */ + +import type { DurableEvent } from "@executablemd/durable-streams"; +import { WorkflowRequestError } from "../storage/errors.ts"; +import type { DocumentExecutionCompletion, WorkflowRunStatus } from "../storage/record.ts"; +import type { JournalEntry } from "../storage/api.ts"; + +/** An outcome that already won. A run in one of these is not made mutable again. */ +export function terminal(status: WorkflowRunStatus): boolean { + return status === "completed" || status === "failed"; +} + +/** What a run says when its executor went without saying anything. */ +export const INTERRUPTED: DocumentExecutionCompletion["reason"] = Object.freeze({ + kind: "host", + code: "executor-interrupted", +}); + +/** + * The canonical outcome the root recorded, when it recorded one. + * + * A root Close is what proves the document itself finished. Its result decides + * the run's terminal status, and its own event identity is the reason. + */ +export function rootOutcome( + entries: readonly JournalEntry[], +): { status: WorkflowRunStatus; reason: DocumentExecutionCompletion["reason"] } | undefined { + for (const entry of entries) { + const event: DurableEvent = entry.event; + if (event.type !== "close" || event.coroutineId !== "root") { + continue; + } + if (event.result.status === "ok") { + return { status: "completed", reason: undefined }; + } + return { + status: event.result.status === "cancelled" ? "cancelled" : "failed", + reason: { kind: "journal", eventId: entry.eventId }, + }; + } + return undefined; +} + +/** What closing a dead executor's execution makes of it, and of the run. */ +export interface Closing { + readonly status: WorkflowRunStatus; + readonly reason: DocumentExecutionCompletion["reason"]; + /** Whether the run's own status follows the execution's, or stays as it is. */ + readonly publishes: boolean; +} + +/** + * What an unfinished execution becomes, on the evidence the run itself holds. + * + * A replay whose terminal state was preserved closes only its own execution: + * the authoritative outcome stays exactly as it was. Otherwise a retained root + * Close proves the canonical outcome won, and without one the executor was + * interrupted. + */ +export function closingOutcome( + storedStatus: WorkflowRunStatus, + canonical: + | { status: WorkflowRunStatus; reason: DocumentExecutionCompletion["reason"] } + | undefined, +): Closing { + if (terminal(storedStatus)) { + return { status: "interrupted", reason: INTERRUPTED, publishes: false }; + } + if (canonical !== undefined) { + return { status: canonical.status, reason: canonical.reason, publishes: true }; + } + return { status: "interrupted", reason: INTERRUPTED, publishes: true }; +} + +/** + * Why this caller may not continue, or nothing when it may. + * + * A run nobody has begun admits either action. A failed or cancelled run is not + * resumed, and a cancelled run is not advanced at all — both report what is + * retained rather than moving it. + */ +export function admissionRefusal( + action: "start" | "resume", + status: WorkflowRunStatus | undefined, +): Error | undefined { + if (status === undefined) { + return undefined; + } + if (action === "resume" && (status === "failed" || status === "cancelled")) { + return new WorkflowRequestError( + `workflow run ${status}: a run that ${ + status === "failed" ? "failed" : "was cancelled" + } is not resumed. The run is left exactly as it is.`, + ); + } + if (status === "cancelled") { + return new WorkflowRequestError( + "workflow run cancelled: a cancelled run reports its retained state and is not advanced.", + ); + } + return undefined; +} + +/** What beginning does, given what recovery left behind. */ +export type BeginDecision = + /** Nothing was there: this begin creates the run and its first execution. */ + | { readonly kind: "create" } + /** An outcome already won: record the execution and leave the status alone. */ + | { readonly kind: "replay" } + /** An ordinary continuation: record the execution and publish `running`. */ + | { readonly kind: "running" }; + +export function beginDecision(recoveredStatus: WorkflowRunStatus | undefined): BeginDecision { + if (recoveredStatus === undefined) { + return { kind: "create" }; + } + return terminal(recoveredStatus) ? { kind: "replay" } : { kind: "running" }; +} diff --git a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts index 1138f2bab..88f88f02c 100644 --- a/packages/workflow/tests/cloudflare/remote-owner.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner.vitest.ts @@ -768,16 +768,21 @@ describe("the remote owner protocol", () => { expect(first).toMatchObject({ outcome: "performed" }); }); - it("refuses a settlement rather than reporting placeholder success", async () => { + it("refuses a settlement naming an execution this run never began", async () => { + // Settlement is implemented now, so what it refuses is what it should: + // a completion addressed to an execution that is not here. The earlier + // shape of this test asserted the command was declined outright, which was + // true only while the transition was somebody else's checkpoint. const stub = executor(); await on(stub, (owner) => owner.initialize()); await admit(stub); - expect( - await send(stub, "settle", { - command: "settle", - completion: { executionId: "execution", status: "completed" }, - expectedWorkspaceRootId: ROOT_ID, - }), - ).toEqual({ id: "settle", outcome: "refused", refusal: "command:unavailable" }); + const answer = await send(stub, "settle", { + command: "settle", + completion: { executionId: "execution", status: "completed" }, + expectedWorkspaceRootId: ROOT_ID, + }); + expect(answer["outcome"]).toBe("refused"); + // And nothing was published: the run is exactly where it was. + expect((await on(stub, (owner) => owner.published()))["events"]).toEqual([]); }); }); From 81b9c6783b0d52cec799531b023792dd8069cf0d Mon Sep 17 00:00:00 2001 From: Min Kim Date: Sun, 6 Sep 2026 15:30:32 -0300 Subject: [PATCH 50/81] =?UTF-8?q?=E2=9C=A8=20Read=20a=20run's=20owner=20wi?= =?UTF-8?q?thout=20taking=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every admitted WebSocket is an executor acquisition: whoever holds it is the executor, and nobody else may advance the run. That is the right shape for mutation and the wrong shape for reading. Inspecting a run, replaying its history and selecting a fork's source are questions about committed state, and asking one must not make the run unrunnable for as long as the answer takes. So reads arrive as ordinary authenticated requests, and admission travels outside the body. The release fingerprint decides whether this build will talk to the caller at all, so it is compared before anything is decoded; the token is verified before the run is named; and only then is the operation parsed. Retained values are read as the types they are declared to be rather than converted. `String(null)` is "null" and `Number(null)` is zero, and once a damaged row has become one of those, nothing downstream can tell the store held the wrong thing — a client checking shapes sees a well-formed answer. So a row that is not what it claims is damage, reported as damage. A fork's source is read whole and proved whole. The prefix ends at the checkpoint without the run record and root import the fork writes for itself; every Workspace root it names comes with it, and the runner derives each root's content references from that root's own canonical manifest rather than believing the ones it was sent. Those references are an array, not a set: a destination compares its own derivation element for element when it retains the root, so a right set in the wrong order is refused here rather than later. Sizes are proved against the bytes, not only against each other: a blob's actual length must equal the size stored beside it, and the root's file entry must equal what its content manifest decodes to. Two coordinated wrong sizes agree with one another, so only the bytes settle it. What crosses that seam is the record the source retained, byte for byte. Each one is parsed before it is accepted, but the string is what is carried: two spellings can parse to the same event, and a destination inserting a spelling rebuilt from a parse would retain a history it never inherited. Public history still projects parsed events, and raw records cross only this private seam. A fork is given the heads and the checkouts it will write against, or it is given nothing. All three head roots — the checkpoint's, the run record's and the root import's — are Workspace root identities that must be among the roots that arrived. Every checkout must sit in a directory the checkpoint's own Workspace holds, no two in the same directory, no name or identity twice, and no Worktree of a Repository that did not come with it. A checkout is identified by a tuple rather than by a joined string, because a retained name may hold any character: `("a:b", "c")` and `("a", "b:c")` are two Worktrees a separator would spell as one. The selection is paged in sections, each bounded by count and by bytes, and every page carries an anchor over everything a destination will copy that a content identity does not already imply — the inherited rows with their bytes and root associations, the head roots, each root's record and reference arrays, each manifest's and blob's retained metadata including its watermark, and every selected checkout. Retained mappings and watermarks both move while content stands still, and either changes the anchor, so a sequence refuses rather than joining two committed states. Continuing is a position in that anchored selection, not a name out of it. A name would have to be written inside the request asking for the next page, and a retained name is text that JSON escapes — once for the name's own encoding and again for the request carrying it — so a member small enough for a page could name a page too large to ask for, and a selection could be started and never finished. A position is the same size whatever it points at, and it says nothing outside the anchor that pins what it counts into. Bounds are measured on the finished encoding of what crosses, and each one belongs to the shape it bounds. A fork-source page is bounded by bytes, so its answer's ceiling follows from what a page may hold. Public history and inspection are paged by count instead, and one retained record may be as large as the transaction that wrote it, so a fork source's arithmetic says nothing about them and they keep the capacity they have. Which ceiling an answer is held to follows from the operation this build asked, decided before the request was sent, so an answer never chooses the limit it is measured against. A section is a sequence, so it is checked as one. Each page says where in the selection it begins, which must be where the sequence has reached, and the position it ends on, which must be the last row it actually carried, so no page continues past rows it never sent. The size is pinned by the first page, the sections sorted by identities this side can derive must arrive in that order, and a section ends only when what arrived is the size that was declared. The journal's own order is derivable from nothing here, so each inherited row says where it stands in the prefix and must stand exactly one further on than the row before it: two rows exchanged inside a page carry each other's positions. Reads are held to the run they name. `inspect(runId)` and `history(runId)` refuse another run before the transport is reached. `list()` reports the owner it is bound to: nothing when that owner holds no run, one snapshot when it does, and a failure when the owner is damaged rather than a healthy subset. Forkability stays where it already lives, in public history and CLI preflight; this reader establishes source integrity and copies. --- packages/workflow/src/cloudflare/client.ts | 6 +- packages/workflow/src/cloudflare/commands.ts | 4 +- packages/workflow/src/cloudflare/owner.ts | 33 + .../workflow/src/cloudflare/read-client.ts | 1041 +++++++++++++++ .../workflow/src/cloudflare/read-plane.ts | 866 ++++++++++++ packages/workflow/src/cloudflare/retained.ts | 90 ++ packages/workflow/src/fork.ts | 21 +- packages/workflow/src/journal-events.ts | 39 + packages/workflow/src/remote/inspection.ts | 109 ++ packages/workflow/src/remote/read.ts | 150 ++- .../cloudflare/remote-read-plane.vitest.ts | 726 ++++++++++ .../cloudflare/support/executor-object.ts | 128 ++ .../workflow/tests/remote-inspection.test.ts | 1183 +++++++++++++++++ 13 files changed, 4373 insertions(+), 23 deletions(-) create mode 100644 packages/workflow/src/cloudflare/read-client.ts create mode 100644 packages/workflow/src/cloudflare/read-plane.ts create mode 100644 packages/workflow/src/cloudflare/retained.ts create mode 100644 packages/workflow/src/journal-events.ts create mode 100644 packages/workflow/src/remote/inspection.ts create mode 100644 packages/workflow/tests/cloudflare/remote-read-plane.vitest.ts create mode 100644 packages/workflow/tests/remote-inspection.test.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 0ba58c632..b1f8e0634 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -104,6 +104,7 @@ export type PrivateRefusal = | "command:absent" | "command:wrong-run" | "command:corrupt-journal" + | "command:not-forkable" | "storage:foreign" | `storage:unsupported-version-v${number}` | "storage:corrupt"; @@ -163,7 +164,7 @@ function nullableIdentity(value: unknown): string | null { return value; } -function privateRefusal(value: string): PrivateRefusal { +export function privateRefusal(value: string): PrivateRefusal { switch (value) { case "acquisition:already-running": case "acquisition:not-acquired": @@ -183,6 +184,7 @@ function privateRefusal(value: string): PrivateRefusal { case "command:absent": case "command:wrong-run": case "command:corrupt-journal": + case "command:not-forkable": case "storage:foreign": case "storage:corrupt": return value; @@ -998,7 +1000,7 @@ function nullableSequence(value: unknown): number | null { * condition. Command names, refusal spellings, rows and cursors stay below this * line: they describe a protocol nobody above here is party to. */ -function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { +export function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { // A host acts on these differently: storage belonging to something else may // not be written, a version this build does not implement may not be // migrated, and damage may not be repaired. Collapsing them would make all diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index fbc054038..719bb79cb 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -93,7 +93,9 @@ export type CommandRefusal = */ | "wrong-run" /** Retained journal history this owner cannot read. */ - | "corrupt-journal"; + | "corrupt-journal" + /** The selected prefix is not one a fork could inherit. */ + | "not-forkable"; export class CommandError extends Error { override name = "CommandError"; diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index f4b3d6018..dbc0f2952 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -34,6 +34,12 @@ import { import { admitToken, type AdmissionPolicy, AdmissionError } from "./admission.ts"; import { TokenError, type TokenVerification } from "./token.ts"; import { CommandError, type CommandResult, parseCommand, type RunnerCommand } from "./commands.ts"; +import { + answerRead, + parseReadOperation, + type ReadAdmission, + type ReadAnswer, +} from "./read-plane.ts"; import { dispatchCommand } from "./dispatcher.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { discardPriorAcquisitions, PRIVATE_OBJECT_NAMES } from "./private-schema.ts"; @@ -166,6 +172,33 @@ export abstract class WorkflowOwnerObject extends DurableObject { }); } + /** + * Answer one ordinary read, taking nothing. + * + * The same order admission uses — the build is compared before any token + * work, and the token is verified before the run is touched — and then it + * stops. No socket is accepted, no acquisition is minted or compared, and + * nothing is written, so this can be answered while an executor is live and + * a refusal at any step leaves the acquisition set and the run untouched. + */ + *read(admission: ReadAdmission, body: string): Operation { + const { policy, verification } = this.configuration(); + try { + // The order is the contract, and the body takes no part in it. A build + // this owner will not talk to is refused before its request is decoded, + // and an unauthenticated one before the run is named. + requireSameRelease(policy.release, admission.release); + yield* admitToken(policy, verification, admission.token); + const runId = admitRunId(admission.runId); + return { + outcome: "performed", + value: answerRead(this.owned, runId, parseReadOperation(body)), + }; + } catch (error) { + return { outcome: "refused", refusal: refusalOf(error) }; + } + } + /** * Handle one message from an admitted connection. * diff --git a/packages/workflow/src/cloudflare/read-client.ts b/packages/workflow/src/cloudflare/read-client.ts new file mode 100644 index 000000000..7a4f9d3f8 --- /dev/null +++ b/packages/workflow/src/cloudflare/read-client.ts @@ -0,0 +1,1041 @@ +/** + * Reading a run's owner from the runner, over the no-acquisition plane. + * + * Every answer is parsed before it is believed, and a paged answer is held to + * the anchor its first page chose: a missing, repeated, reordered, wrong-run or + * changed-anchor page fails the whole read rather than producing a shorter + * history nobody asked for. Nothing partial is ever published. + * + * The transport is narrow on purpose. It sends one request and returns one + * response, and knows nothing about runs, anchors or authority — a host wires + * it to an ordinary HTTP request, and a test wires it to the object directly. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import { parseMembers, requireMemberNames } from "../storage/members.ts"; +import type { WorkflowStorageError } from "../storage/errors.ts"; +import { WorkflowRecordMalformedError, WorkflowRunNotFoundError } from "../storage/errors.ts"; +import { type DurableEvent, parseDurableEvent } from "@executablemd/durable-streams"; +import type { + DefinitionRetrieval, + DocumentExecutionRecord, + WorkflowRunRecord, +} from "../storage/record.ts"; +import { + parseRemoteExecution, + parseRemoteRetrieval, + parseRemoteRunRecord, + RemoteRecordError, +} from "../remote/records.ts"; +import { READ_PAGE_BYTES, READ_PAGE_ENTRIES, READ_REQUEST_ENVELOPE } from "./read-plane.ts"; +import { decodeBase64, sha256Hex } from "./encoding.ts"; +import { + compareUtf8, + parseWorkspaceRootManifest, + SHA256, + WORKSPACE_ROOT_DOMAIN, + WORKSPACE_ROOT_FORMAT, +} from "../workspace/root-manifest.ts"; +import { decodeContentManifest } from "../workspace/content-manifest.ts"; +import type { + RemoteBlob, + RemoteCheckout, + RemoteForkSource, + RemoteManifest, + RemoteReadPlane, + RemoteStoredRoot, + RetainedHistory, + RetainedInspection, + RetainedProvenance, + RetainedRow, +} from "../remote/read.ts"; +import { type PrivateRefusal, privateRefusal, storageFailure } from "./client.ts"; + +/** + * One request out, one response back. + * + * The admission travels beside the body rather than inside it, because the + * owner decides on the release before it decodes anything. + */ +export interface ReadTransport { + send(admission: ReadAdmission, body: string): Operation; +} + +/** What a request carries outside its body. */ +export interface ReadAdmission { + readonly release: string; + readonly token: string; + readonly runId: string; +} + +/** The most pages one answer may take before it is refused as unbounded. */ +const MAX_PAGES = 4096; + +/** + * The most serialized bytes one fork-source answer may carry. + * + * Measured over the finished UTF-8 encoding, and derived from what a page may + * hold rather than picked: one page of rows, the checkpoint the answer names — + * an event id, bounded by the journal row that carries it being a member — and + * the fixed envelope of anchors, roots, positions and counts around them. The + * rows are nested values rather than strings inside a string, so what a page + * measured is what an answer carries. A smaller number would make a retained + * selection this build accepts impossible to read back. + */ +export const FORK_SOURCE_ANSWER_BYTES = 2 * READ_PAGE_BYTES + READ_REQUEST_ENVELOPE; + +/** + * The most serialized bytes one public answer may carry. + * + * Public history and inspection are paged by count rather than by bytes, and + * one retained record may be as large as the transaction that wrote it, so + * what bounds these is not what bounds a fork source. This is the capacity + * these operations already had, kept as it was: a fork source's own arithmetic + * is a fact about fork-source pages, and adopting it here would make retained + * history that this plane could read unreadable. + */ +export const PUBLIC_ANSWER_BYTES = 1638400; + +/** + * Which ceiling one answer is held to, decided by what was asked. + * + * The operation is this build's own, chosen before the request is sent, so an + * answer never selects the bound it is measured against. + */ +function answerBytes(operation: unknown): number { + return operation === "fork-source" ? FORK_SOURCE_ANSWER_BYTES : PUBLIC_ANSWER_BYTES; +} + +function fail(reason: string): never { + throw new RemoteRecordError(`the owner returned a malformed read answer: ${reason}`); +} + +function failure(reason: string, path: string): Error { + return new RemoteRecordError(`the owner returned a malformed read answer at ${path}: ${reason}`); +} + +function members(value: unknown, names: readonly string[]): Map { + const found = parseMembers(value, "$", failure); + requireMemberNames(found, names, "$", failure); + return found; +} + +function text(value: unknown, what: string): string { + if (typeof value !== "string" || value === "") { + return fail(`it did not name ${what}`); + } + return value; +} + +function list(value: unknown, what: string): unknown[] { + if (!Array.isArray(value)) { + return fail(`it did not carry ${what}`); + } + return value; +} + +/** + * Open one read plane over this transport. + * + * `expectedRunId` is what every answer is held to. An owner that answered about + * another run is not this run's owner, whatever the answer says. + */ +export function cloudflareReadPlane( + transport: ReadTransport, + release: string, + token: () => Operation, + expectedRunId: string, +): RemoteReadPlane { + function* ask(read: Record): Operation { + const admission = { release, token: yield* token(), runId: expectedRunId }; + const raw = yield* transport.send(admission, JSON.stringify(read)); + if (new TextEncoder().encode(raw).length > answerBytes(read["operation"])) { + // An answer larger than one may be is not one this build reads, and + // reading it far enough to find out would be reading it. + return fail("it exceeded the bytes one answer may carry"); + } + let decoded: unknown; + try { + decoded = JSON.parse(raw); + } catch { + // Invalid JSON becomes the same bounded malformed-owner failure that + // every unreadable answer does; no parser diagnostic escapes. + return fail("it was not one JSON object"); + } + const answer = members(decoded, ["outcome", "value", "refusal"]); + if (answer.get("outcome") === "refused") { + // Parsed into this build's closed vocabulary before it becomes an error, + // so no refusal spelling reaches a caller. + const refusal = privateRefusal(text(answer.get("refusal"), "a refusal")); + // Nothing stored here is a different fact from storage this build cannot + // use, and a caller listing an owner acts on the difference. + throw refusal === "command:absent" + ? new WorkflowRunNotFoundError(expectedRunId) + : storageFailure(refusal); + } + if (answer.get("outcome") !== "performed") { + return fail("it named no outcome this build reads"); + } + return answer.get("value"); + } + + return { + runId: expectedRunId, + + *inspect(): Operation> { + try { + return Ok(parseInspection(yield* ask({ operation: "inspect" }), expectedRunId)); + } catch (error) { + return Err(translateRead(error)); + } + }, + + *history(): Operation> { + try { + return Ok(yield* pages(ask, (anchor, after) => ({ operation: "history", anchor, after }))); + } catch (error) { + return Err(translateRead(error)); + } + }, + + *forkSource(checkpointEventId: string): Operation> { + try { + return Ok(yield* collectForkSource(ask, expectedRunId, checkpointEventId)); + } catch (error) { + return Err(translateRead(error)); + } + }, + }; +} + +function parseEvent(record: string): DurableEvent { + const parsed = parseDurableEvent(record); + if (!parsed.ok) { + return fail("it carried a retained event this build cannot read"); + } + return parsed.value; +} + +/** + * Walk one anchored page sequence to its end, or refuse the whole answer. + * + * The first page chooses the anchor and every later page is held to it and to + * the cursor it was asked to continue from. A page that skips, repeats, + * reorders or changes the anchor is a page of some other snapshot, and there is + * no partial answer to give. + */ +function* pages( + ask: (read: Record) => Operation, + request: (anchor: string | null, after: string | null) => Record, +): Operation { + const entries: RetainedRow[] = []; + const seen = new Set(); + let anchor: string | null | undefined; + let after: string | null = null; + let retainedRoots: ReadonlySet = new Set(); + let inherited: ReadonlyMap = new Map(); + + for (let page = 0; ; page += 1) { + if (page > MAX_PAGES) { + return fail("it did not terminate its anchored answer"); + } + const found = members(yield* ask(request(anchor ?? null, after)), [ + "anchor", + "after", + "rows", + "done", + "retainedRoots", + "provenance", + ]); + const offered = found.get("anchor"); + const expected = anchor === undefined ? offered : anchor; + if (offered !== expected || found.get("after") !== after) { + return fail("a page did not continue its anchored snapshot"); + } + anchor = offered === null ? null : text(offered, "an anchor"); + const rows = list(found.get("rows"), "rows"); + if (rows.length > READ_PAGE_ENTRIES) { + return fail("a page carried more rows than one may"); + } + if (anchor === null) { + if (rows.length > 0 || found.get("done") !== true || after !== null) { + return fail("an empty snapshot carried rows or did not terminate"); + } + return { entries, retainedRoots, inherited }; + } + if (rows.length === 0) { + return fail("a page of an anchored snapshot carried no rows"); + } + for (const row of rows) { + const entry = members(row, ["eventId", "record", "workspaceRootId"]); + const eventId = text(entry.get("eventId"), "an event"); + if (seen.has(eventId)) { + return fail("a page repeated an event"); + } + seen.add(eventId); + entries.push({ + eventId, + event: parseEvent(text(entry.get("record"), "a retained record")), + workspaceRootId: text(entry.get("workspaceRootId"), "a Workspace root"), + }); + after = eventId; + } + if (found.get("done") !== true) { + continue; + } + if (after !== anchor) { + return fail("a page terminated short of its anchor"); + } + retainedRoots = new Set( + list(found.get("retainedRoots"), "retained roots").map((root) => text(root, "a root")), + ); + inherited = new Map( + list(found.get("provenance"), "provenance").map((row) => { + const entry = members(row, ["eventId", "sourceRunId", "sourceEventId"]); + return [ + text(entry.get("eventId"), "an event"), + Object.freeze({ + sourceRunId: text(entry.get("sourceRunId"), "a source run"), + sourceEventId: text(entry.get("sourceEventId"), "a source event"), + }), + ]; + }), + ); + return { entries, retainedRoots, inherited }; + } +} + +function parseInspection(value: unknown, expectedRunId: string): RetainedInspection { + const found = members(value, [ + "record", + "executions", + "retrieval", + "journalFrontier", + "currentWorkspaceRootId", + "lineage", + ]); + const record = parseRemoteRunRecord(found.get("record")); + if (record.runId !== expectedRunId) { + return fail("it described another run"); + } + const executions = list(found.get("executions"), "executions").map((entry) => + parseRemoteExecution(entry), + ); + const frontier = found.get("journalFrontier"); + const lineage = found.get("lineage"); + const retrieval = parseRemoteRetrieval(found.get("retrieval")); + + return Object.freeze({ + record, + executions: Object.freeze(executions), + ...(retrieval === undefined ? {} : { retrieval }), + ...(frontier === null + ? {} + : { + journalFrontier: Object.freeze({ + eventId: text( + members(frontier, ["eventId", "workspaceRootId"]).get("eventId"), + "an event", + ), + workspaceRootId: text( + members(frontier, ["eventId", "workspaceRootId"]).get("workspaceRootId"), + "a Workspace root", + ), + }), + }), + currentWorkspaceRootId: text(found.get("currentWorkspaceRootId"), "a Workspace root"), + ...(lineage === null + ? {} + : { + lineage: Object.freeze(parseLineage(lineage)), + }), + }); +} + +function parseLineage(value: unknown) { + const found = members(value, ["sourceRunId", "checkpointEventId", "checkpointWorkspaceRootId"]); + return { + sourceRunId: text(found.get("sourceRunId"), "a source run"), + checkpointEventId: text(found.get("checkpointEventId"), "a checkpoint"), + checkpointWorkspaceRootId: text(found.get("checkpointWorkspaceRootId"), "a Workspace root"), + }; +} + +/** Any failure from the read plane, as a provider-neutral one. */ +function translateRead(error: unknown): WorkflowStorageError { + if (error instanceof WorkflowRecordMalformedError || error instanceof WorkflowRunNotFoundError) { + return error; + } + if (error instanceof RemoteRecordError) { + return new WorkflowRecordMalformedError( + "record this run's owner returned", + "it is not a record this build can read", + ); + } + return storageFailure(readRefusal(error)); +} + +function readRefusal(error: unknown): PrivateRefusal { + return error instanceof Error && "refusal" in error + ? privateRefusal(String(Reflect.get(error, "refusal"))) + : "command:unavailable"; +} + +/** + * Read every section of one fork source, and hold them all to one selection. + * + * Each section is its own anchored sequence; the anchor covers the whole + * selection rather than the checkpoint alone, so a section that arrived from a + * different selection is refused rather than mixed in. The head members are + * checked to agree across every page of every section for the same reason. + */ +function* collectForkSource( + ask: (read: Record) => Operation, + sourceRunId: string, + checkpointEventId: string, +): Operation { + let head: Map | undefined; + let anchor: string | undefined; + + /** + * Read one section to its end, holding every page to the same selection. + * + * The cursor is not taken on trust: it must be the identity of the last row + * the page actually carried, so a page cannot advance past rows it did not + * send, and each page must begin where the sequence has reached. Identities + * are unique within a section, and the sorted sections are required to + * arrive in the order they are sorted in, so a sequence cannot repeat, skip + * or reorder the members a destination will retain. The declared total is + * pinned by the first page and must describe what finally arrived. + */ + function* section(name: string, order: "sorted" | "journal"): Operation { + const carried: unknown[] = []; + const seen = new Set(); + let after: number | null = null; + let declared: number | undefined; + let previous: readonly string[] | undefined; + + for (let page = 0; ; page += 1) { + if (page > MAX_PAGES) { + return fail("it did not terminate a fork-source section"); + } + const found = members( + yield* ask({ + operation: "fork-source", + checkpointEventId, + section: name, + anchor: anchor ?? null, + after, + }), + [ + "anchor", + "after", + "section", + "checkpointEventId", + "checkpointWorkspaceRootId", + "runRecordWorkspaceRootId", + "rootImportWorkspaceRootId", + "rows", + "from", + "cursor", + "done", + "total", + ], + ); + const offered = text(found.get("anchor"), "a selection anchor"); + if (anchor === undefined) { + anchor = offered; + head = found; + } + if ( + offered !== anchor || + found.get("section") !== name || + found.get("after") !== after || + found.get("checkpointEventId") !== checkpointEventId || + found.get("checkpointWorkspaceRootId") !== head?.get("checkpointWorkspaceRootId") || + found.get("runRecordWorkspaceRootId") !== head?.get("runRecordWorkspaceRootId") || + found.get("rootImportWorkspaceRootId") !== head?.get("rootImportWorkspaceRootId") + ) { + return fail("a page did not continue its selection"); + } + const total = count(found.get("total"), "a section total"); + if (declared === undefined) { + declared = total; + } + if (total !== declared) { + // The section changed size under the sequence, so its pages describe + // two different answers. + return fail("a page redeclared the size of its section"); + } + + const rows = list(found.get("rows"), "rows"); + if (rows.length > READ_PAGE_ENTRIES) { + return fail("a page carried more rows than one may"); + } + // Where the owner says this page begins has to be where the sequence + // has got to. A page beginning anywhere else skipped members or sent + // some of them twice, whatever its cursor says. + if (count(found.get("from"), "a section position") !== carried.length) { + return fail("a page did not begin where the sequence had reached"); + } + for (const row of rows) { + const identity = identityOf(name, row); + if (seen.has(identity)) { + return fail("a section repeated a member"); + } + if (order === "journal") { + // The one order nothing here can derive, so the owner states it per + // member and it must advance by one from where the section stood. + // Two rows swapped inside a page carry each other's positions. + if (positionOf(row) !== carried.length) { + return fail("a member did not stand where the section had reached"); + } + } + if (order === "sorted") { + const sorts = orderOf(name, row); + if (previous !== undefined && !precedes(previous, sorts)) { + return fail("a section carried its members out of order"); + } + previous = sorts; + } + seen.add(identity); + carried.push(row); + } + + // Derived from what arrived, never believed: the page ended on the last + // row it actually carried, so that is the only position it may name. + const ended: number | null = rows.length === 0 ? after : carried.length - 1; + const cursor = found.get("cursor"); + if (found.get("done") === true) { + if (carried.length !== declared) { + return fail("a section terminated short of what it declared"); + } + // A terminal page still describes where it ended. + if (cursor !== ended) { + return fail("a terminal page did not name the member it ended on"); + } + return carried; + } + if (rows.length === 0) { + return fail("a page of an unfinished section carried no rows"); + } + if (cursor !== ended) { + return fail("a page did not advance to the member it ended on"); + } + after = ended; + } + } + + const inherited = (yield* section("inherited", "journal")).map((row) => { + const found = members(row, ["eventId", "record", "workspaceRootId", "position"]); + const record = text(found.get("record"), "a retained record"); + // Parsed to prove it is a record this build can read, and then kept + // exactly: a destination inserts these bytes, and a spelling reconstructed + // from the parse would be a different history. + parseEvent(record); + return { + eventId: text(found.get("eventId"), "an event"), + record, + workspaceRootId: digest(found.get("workspaceRootId"), "a Workspace root"), + }; + }); + // These four arrive sorted by identities this side can derive, so a + // reordering is a sequence this build did not produce. The inherited rows + // arrive in the source's own journal order, which nothing here can derive: + // what holds them is that each page begins where the sequence reached, that + // no identity arrives twice, and that the whole selection is anchored. + const roots = (yield* section("roots", "sorted")).map((row) => parseStoredRoot(row)); + const manifests = (yield* section("manifests", "sorted")).map((row) => parseManifest(row)); + const blobs = (yield* section("blobs", "sorted")).map((row) => parseBlob(row)); + const checkouts = (yield* section("checkouts", "sorted")).map((row) => parseCheckout(row)); + + validateClosure(inherited, roots, manifests, blobs); + + const checkpointWorkspaceRootId = digest( + head?.get("checkpointWorkspaceRootId"), + "a checkpoint Workspace root", + ); + const runRecordWorkspaceRootId = digest( + head?.get("runRecordWorkspaceRootId"), + "a Workspace root", + ); + const rootImportWorkspaceRootId = digest( + head?.get("rootImportWorkspaceRootId"), + "a Workspace root", + ); + // All three heads, not just the checkpoint. A destination writes its own run + // record and root import against these, and it cannot write against a root + // it was not given. + const carried = new Set(roots.map((root) => root.rootId)); + for (const rootId of [ + checkpointWorkspaceRootId, + runRecordWorkspaceRootId, + rootImportWorkspaceRootId, + ]) { + if (!carried.has(rootId)) { + return fail("a head names a Workspace root the selection did not carry"); + } + } + validateCheckouts(checkouts, roots, checkpointWorkspaceRootId); + return Object.freeze({ + sourceRunId, + checkpointEventId, + checkpointWorkspaceRootId, + runRecordWorkspaceRootId, + rootImportWorkspaceRootId, + inherited: Object.freeze(inherited), + roots: Object.freeze(roots), + manifests: Object.freeze(manifests), + blobs: Object.freeze(blobs), + checkouts: Object.freeze(checkouts), + }); +} + +/** + * Prove the transported bytes really are the Workspace this selection names. + * + * A digest proves its own bytes, so a manifest or blob that arrived under the + * wrong name is caught where it is parsed. What is left is the shape of the + * closure: every root the prefix names is here, every root's own manifest + * derives exactly the references it was sent with, every content manifest a + * root needs is here and decodes to the size it claims, every chunk those + * manifests name is a blob that is here, and nothing arrived that the selection + * does not need. Anything less would let a destination retain a history whose + * Workspace cannot be restored. + */ +function validateClosure( + inherited: readonly { readonly workspaceRootId: string }[], + roots: readonly RemoteStoredRoot[], + manifests: readonly RemoteManifest[], + blobs: readonly RemoteBlob[], +): void { + const held = new Set(roots.map((root) => root.rootId)); + for (const row of inherited) { + if (!held.has(row.workspaceRootId)) { + return fail("a selected row names a Workspace root the selection did not carry"); + } + } + + const manifestsByHash = new Map(manifests.map((manifest) => [manifest.hash, manifest])); + const blobsByHash = new Map(blobs.map((blob) => [blob.hash, blob])); + if (manifestsByHash.size !== manifests.length || blobsByHash.size !== blobs.length) { + return fail("the selection carried one piece twice"); + } + + // Derived from the roots themselves rather than believed: a reference set the + // owner sent is only correct if the root's own manifest produces it. + for (const blob of blobs) { + // The bytes themselves, against the size the destination will persist + // beside them. The digest proves which bytes these are and the chunk + // comparison proves the manifest agrees, but two coordinated wrong sizes + // satisfy both — only the actual length settles it. + if (blob.content.byteLength !== blob.size) { + return fail("a selected blob disagreed with its recorded size"); + } + } + + const neededManifests = new Set(); + const neededBlobs = new Set(); + for (const root of roots) { + const parsed = parseWorkspaceRootManifest(root.manifest, (reason) => + fail(`a selected root is not a canonical Workspace root: ${reason}`), + ); + const declared = new Set(); + const sizes = new Map(); + for (const entry of parsed.entries) { + if (entry.kind === "file") { + declared.add(entry.manifest); + const already = sizes.get(entry.manifest); + if (already !== undefined && already !== entry.size) { + // Two entries sharing one content identity must describe the same + // bytes, so they cannot claim different sizes. + return fail("a selected root gave one content two sizes"); + } + sizes.set(entry.manifest, entry.size); + } + } + if (!sameOrder(canonical(declared), root.manifestHashes)) { + return fail("a selected root disagreed with the content references it carried"); + } + const rootBlobs = new Set(); + for (const hash of declared) { + neededManifests.add(hash); + const manifest = manifestsByHash.get(hash); + if (manifest === undefined) { + return fail("a selected root names a manifest the selection did not carry"); + } + const content = decodeContentManifest(manifest.encoded, (reason) => + fail(`a selected manifest is not canonically encoded: ${reason}`), + ); + if (content.size !== manifest.size) { + return fail("a selected manifest disagreed with the size it describes"); + } + // And with the file the root says it holds. A destination restores that + // entry from these bytes, so a root claiming another length describes a + // Workspace this content cannot produce. + if (sizes.get(hash) !== content.size) { + return fail("a selected root disagreed with the content it names"); + } + for (const chunk of content.chunks) { + rootBlobs.add(chunk.hash); + neededBlobs.add(chunk.hash); + const blob = blobsByHash.get(chunk.hash); + if (blob === undefined) { + return fail("a selected manifest names a blob the selection did not carry"); + } + if (blob.size !== chunk.size) { + return fail("a selected chunk disagreed with the blob it names"); + } + } + } + if (!sameOrder(canonical(rootBlobs), root.blobHashes)) { + return fail("a selected root disagreed with the blob references it carried"); + } + } + + // Nothing beyond the closure: an extra piece is data the selection does not + // account for, and a destination retaining it would hold content no root of + // its own refers to. + if (!sameSet(neededManifests, new Set(manifestsByHash.keys()))) { + return fail("the selection carried a manifest no selected root requires"); + } + if (!sameSet(neededBlobs, new Set(blobsByHash.keys()))) { + return fail("the selection carried a blob no selected manifest requires"); + } +} + +/** + * Prove the checkouts are a graph a destination can retain. + * + * Each one names a directory the checkpoint's Workspace actually holds, so a + * fork does not inherit a checkout with nowhere to live; each Worktree names a + * Repository that came with it; and nothing is named twice, because the + * retained schema keys these by name and by path. + */ +function validateCheckouts( + checkouts: readonly RemoteCheckout[], + roots: readonly RemoteStoredRoot[], + checkpointWorkspaceRootId: string, +): void { + const checkpoint = roots.find((root) => root.rootId === checkpointWorkspaceRootId); + if (checkpoint === undefined) { + return fail("the checkpoint names a Workspace root the selection did not carry"); + } + const directories = new Set(); + for (const entry of parseWorkspaceRootManifest(checkpoint.manifest, (reason) => + fail(`a selected root is not a canonical Workspace root: ${reason}`), + ).entries) { + if (entry.kind === "directory") { + directories.add(entry.path); + } + } + + const repositories = new Set(); + const worktrees = new Set(); + const paths = new Set(); + for (const checkout of checkouts) { + if (!directories.has(checkout.checkoutPath)) { + return fail("a selected checkout names a directory the checkpoint Workspace does not hold"); + } + if (paths.has(checkout.checkoutPath)) { + return fail("two selected checkouts name one directory"); + } + paths.add(checkout.checkoutPath); + if (checkout.kind === "repository") { + if (repositories.has(checkout.name)) { + return fail("one Repository name was selected twice"); + } + repositories.add(checkout.name); + } + } + for (const checkout of checkouts) { + if (checkout.kind !== "worktree") { + continue; + } + const identity = checkoutKey(["worktree", checkout.repositoryName, checkout.name]); + if (worktrees.has(identity)) { + return fail("one Worktree identity was selected twice"); + } + worktrees.add(identity); + if (!repositories.has(checkout.repositoryName)) { + // A checkout belonging to nothing. The destination would retain a + // Worktree of a Repository it does not have. + return fail("a selected Worktree names a Repository the selection did not carry"); + } + } +} + +/** + * A root's references, in the one order a root is retained with. + * + * The same derivation the Workspace root implementation makes: the identities + * a root's own manifest produces, deduplicated, ordered by their UTF-8 bytes. + */ +function canonical(references: ReadonlySet): string[] { + return [...references].sort(compareUtf8); +} + +/** + * Element for element, not member for member. + * + * A destination compares a root's reference arrays exactly when it retains + * them, so an array that is the right set in the wrong order — or with one + * identity twice — is a source it would refuse later. It is refused here + * instead, where the whole selection can still be rejected. + */ +function sameOrder(left: readonly string[], right: readonly string[]): boolean { + return left.length === right.length && left.every((value, at) => value === right[at]); +} + +function sameSet(left: ReadonlySet, right: ReadonlySet): boolean { + if (left.size !== right.size) { + return false; + } + for (const value of left) { + if (!right.has(value)) { + return false; + } + } + return true; +} + +/** + * What names one member of a section. + * + * The same key the owner pages by, derived here from the row itself so a + * cursor is checked against what arrived rather than taken from beside it. + */ +function identityOf(section: string, row: unknown): string { + const found = parseMembers(row, "$", failure); + if (section === "inherited") { + return text(found.get("eventId"), "an event"); + } + if (section === "roots") { + return text(found.get("rootId"), "a Workspace root"); + } + if (section === "manifests" || section === "blobs") { + return text(found.get("hash"), "a content identity"); + } + if (found.get("kind") === "repository") { + return checkoutKey(["repository", text(found.get("name"), "a name")]); + } + return checkoutKey([ + "worktree", + text(found.get("repositoryName"), "a Repository"), + text(found.get("name"), "a name"), + ]); +} + +/** + * One checkout's identity, as a key nothing else can spell. + * + * The owner pages by this exact string. Repository and Worktree names are + * retained text and may hold any character, so `("a:b", "c")` and + * `("a", "b:c")` are two retained Worktrees that a separator would join into + * one key: the client would refuse a valid source as a duplicate, or continue + * from the wrong member. A JSON array separates the parts it escapes. + */ +function checkoutKey(parts: readonly string[]): string { + return JSON.stringify(parts); +} + +/** Where one member of a journal-ordered section stands in its selection. */ +function positionOf(row: unknown): number { + return count(parseMembers(row, "$", failure).get("position"), "a position"); +} + +/** + * How the owner sorted this section, as the parts it sorted by. + * + * Compared part by part rather than as one string: the checkouts are sorted by + * a Repository name and then a Worktree name, and joining those into one string + * would order two names differently than sorting them separately does. + */ +function orderOf(section: string, row: unknown): readonly string[] { + const found = parseMembers(row, "$", failure); + if (section === "roots") { + return [text(found.get("rootId"), "a Workspace root")]; + } + if (section === "manifests" || section === "blobs") { + return [text(found.get("hash"), "a content identity")]; + } + if (found.get("kind") === "repository") { + return ["0", text(found.get("name"), "a name")]; + } + return [ + "1", + text(found.get("repositoryName"), "a Repository"), + text(found.get("name"), "a name"), + ]; +} + +/** Whether one member sorts strictly before another, by UTF-8 bytes. */ +function precedes(previous: readonly string[], next: readonly string[]): boolean { + for (let at = 0; at < Math.max(previous.length, next.length); at += 1) { + const left = previous[at]; + const right = next[at]; + if (left === undefined || right === undefined) { + return right !== undefined; + } + const order = compareUtf8(left, right); + if (order !== 0) { + return order < 0; + } + } + return false; +} + +function parseStoredRoot(value: unknown): RemoteStoredRoot { + const found = members(value, [ + "rootId", + "formatVersion", + "manifest", + "manifestHashes", + "blobHashes", + ]); + const manifest = text(found.get("manifest"), "a root manifest"); + const rootId = digest(found.get("rootId"), "a Workspace root"); + if (found.get("formatVersion") !== WORKSPACE_ROOT_FORMAT) { + return fail("a selected root is not the Workspace root format this build writes"); + } + // The identity is the hash of the bytes, so a root that did not produce it + // is not the root it says it is. + if (sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${manifest}`) !== rootId) { + return fail("a selected root disagreed with its own identity"); + } + return Object.freeze({ + rootId, + formatVersion: count(found.get("formatVersion"), "a format version"), + manifest, + manifestHashes: Object.freeze( + list(found.get("manifestHashes"), "manifest hashes").map((hash) => + digest(hash, "a manifest"), + ), + ), + blobHashes: Object.freeze( + list(found.get("blobHashes"), "blob hashes").map((hash) => digest(hash, "a blob")), + ), + }); +} + +function parseManifest(value: unknown): RemoteManifest { + const found = members(value, ["hash", "size", "lastSeen", "encoded"]); + const encoded = decodeBase64(text(found.get("encoded"), "encoded bytes")); + const hash = digest(found.get("hash"), "a manifest"); + // Content-addressed, so the bytes decide the name here exactly as they do + // for a blob. A manifest under the wrong name is not the manifest a root + // referred to. + if (sha256Hex(encoded) !== hash) { + return fail("a selected manifest disagreed with its own identity"); + } + return Object.freeze({ + hash, + size: count(found.get("size"), "a size"), + lastSeen: count(found.get("lastSeen"), "a timestamp"), + encoded, + }); +} + +function parseBlob(value: unknown): RemoteBlob { + const found = members(value, ["hash", "size", "lastSeen", "content"]); + const content = decodeBase64(text(found.get("content"), "content bytes")); + const hash = digest(found.get("hash"), "a blob"); + // Content-addressed, so the bytes decide the name. A piece whose bytes do + // not hash to it is not the piece the root referred to. + if (sha256Hex(content) !== hash) { + return fail("a selected blob disagreed with its own identity"); + } + return Object.freeze({ + hash, + size: count(found.get("size"), "a size"), + lastSeen: count(found.get("lastSeen"), "a timestamp"), + content, + }); +} + +function parseCheckout(value: unknown): RemoteCheckout { + const probe = parseMembers(value, "$", failure); + if (probe.get("kind") === "repository") { + const found = members(value, [ + "kind", + "name", + "locator", + "locatorFingerprint", + "requestedBase", + "creationCommit", + "primaryBranch", + "objectFormat", + "checkoutPath", + ]); + return Object.freeze({ + kind: "repository", + name: text(found.get("name"), "a name"), + locator: text(found.get("locator"), "a locator"), + locatorFingerprint: digest(found.get("locatorFingerprint"), "a fingerprint"), + requestedBase: optional(found.get("requestedBase")), + creationCommit: text(found.get("creationCommit"), "a commit"), + primaryBranch: text(found.get("primaryBranch"), "a branch"), + objectFormat: objectFormat(found.get("objectFormat")), + checkoutPath: workspacePath(found.get("checkoutPath")), + }); + } + const found = members(value, [ + "kind", + "repositoryName", + "name", + "requestedBranch", + "requestedBase", + "creationCommit", + "checkoutPath", + ]); + if (found.get("kind") !== "worktree") { + return fail("a selected checkout named no kind this build reads"); + } + return Object.freeze({ + kind: "worktree", + repositoryName: text(found.get("repositoryName"), "a Repository"), + name: text(found.get("name"), "a name"), + requestedBranch: text(found.get("requestedBranch"), "a branch"), + requestedBase: optional(found.get("requestedBase")), + creationCommit: text(found.get("creationCommit"), "a commit"), + checkoutPath: workspacePath(found.get("checkoutPath")), + }); +} + +/** One of the object formats this build writes. */ +function objectFormat(value: unknown): "sha1" | "sha256" { + const candidate = text(value, "an object format"); + if (candidate !== "sha1" && candidate !== "sha256") { + return fail("it did not name an object format"); + } + return candidate; +} + +/** A Workspace-relative path, which is absolute within the Workspace. */ +function workspacePath(value: unknown): string { + const candidate = text(value, "a checkout path"); + if (!candidate.startsWith("/")) { + return fail("it did not name a Workspace path"); + } + return candidate; +} + +/** A content identity, held to the shape every digest in this build has. */ +function digest(value: unknown, what: string): string { + const candidate = text(value, `${what} identity`); + if (!SHA256.test(candidate)) { + return fail(`it did not name ${what}`); + } + return candidate; +} + +function count(value: unknown, what: string): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + return fail(`it did not name ${what}`); + } + return value; +} + +function optional(value: unknown): string | null { + return value === null ? null : text(value, "text"); +} diff --git a/packages/workflow/src/cloudflare/read-plane.ts b/packages/workflow/src/cloudflare/read-plane.ts new file mode 100644 index 000000000..bd36b2feb --- /dev/null +++ b/packages/workflow/src/cloudflare/read-plane.ts @@ -0,0 +1,866 @@ +/** + * Reading a run's owner without taking it. + * + * The executor plane is one admitted WebSocket and one acquisition: whoever + * holds it is *the* executor, and nobody else may advance the run. That is the + * right shape for mutation and the wrong shape for reading. Inspecting a run, + * listing what an owner holds, replaying its history, and selecting a fork's + * source are all questions about committed state, and asking one must not make + * the run unrunnable for as long as the answer takes. + * + * So this is an ordinary request plane. It accepts no socket, mints no + * acquisition, writes nothing, and can be answered while an executor is live. + * Admission shares the executor plane's order — release before token, token + * before the run is touched — but stops before the acquisition the executor + * plane takes last. + * + * What crosses is closed and private to this release. Public history is + * projected on the runner from the retained rows this returns, so there is one + * interpretation of a journal rather than two. + */ + +import { parseMembers, requireMemberNames } from "../storage/members.ts"; +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; +import { readDocumentExecution, readRetrieval, readRunRecord, type Row } from "../sqlite/rows.ts"; +import { CommandError } from "./commands.ts"; +import type { OwnerStorage } from "./storage.ts"; +import { recognizeObject } from "./recognition.ts"; +import { isRootImportEvent, isRunRecordEvent } from "../journal-events.ts"; +import { parseDurableEvent } from "@executablemd/durable-streams"; +import { compareUtf8, parseWorkspaceRootManifest } from "../workspace/root-manifest.ts"; +import { encodeBase64, sha256Hex } from "./encoding.ts"; +import { + retainedBytes, + retainedCount, + retainedDigest, + retainedNullableText, + retainedObjectFormat, + retainedPath, + retainedText, +} from "./retained.ts"; + +/** What a read answered, or why it would not. */ +export type ReadAnswer = + | { readonly outcome: "performed"; readonly value: unknown } + | { readonly outcome: "refused"; readonly refusal: string }; + +/** The most rows one page of a read answer may carry. */ +export const READ_PAGE_ENTRIES = 128; + +/** The most serialized bytes one page of a read answer may carry. */ +export const READ_PAGE_BYTES = 512 * 1024; + +/** The most characters a public run id may carry. */ +const MAX_RUN_ID = 128; + +/** + * What a request carries besides the checkpoint it names. + * + * An operation, a section, an anchor, an ordinal and the punctuation around + * them: all fixed shapes, so one generous constant covers every request this + * build makes. + */ +export const READ_REQUEST_ENVELOPE = 4096; + +/** + * The most serialized bytes one request may carry. + * + * Measured over the finished UTF-8 encoding, because that is what crosses. A + * request names a checkpoint, and a checkpoint event id is retained text: what + * bounds it is that its own journal row is a member, and no page carries a + * member larger than a page. Everything else in a request is fixed. + */ +export const READ_REQUEST_BYTES = READ_PAGE_BYTES + READ_REQUEST_ENVELOPE; + +/** + * What a reader may ask for. + * + * Closed, and none of it names a table, a row, a root or a hash: a caller + * chooses a run, an operation and — where an answer is paged — the anchor it is + * continuing. Everything else is the owner's to decide. + */ +export type ReadOperation = + | { readonly operation: "inspect" } + | { readonly operation: "history"; readonly anchor: string | null; readonly after: string | null } + | { + readonly operation: "fork-source"; + readonly checkpointEventId: string; + /** Which part of the selection this page continues. */ + readonly section: ForkSourceSection; + readonly anchor: string | null; + /** + * Where in the anchored selection to continue, as a position rather than + * a name. + * + * A name would have to be spelled inside this request, and a retained + * name is text that JSON escapes — twice, once for the name's own + * encoding and once for the request carrying it — so a member the owner + * will put in a page could name a page nobody can ask for. A position is + * the same size whatever it points at, and it means nothing outside the + * anchor that pins the selection. + */ + readonly after: number | null; + }; + +/** + * The parts a fork's source is read in. + * + * Separate sections rather than one stream because they are different kinds of + * thing and different sizes: rows, root manifests, encoded manifests and blob + * content each need their own bound, and content needs chunking that rows do + * not. + */ +export type ForkSourceSection = "inherited" | "roots" | "manifests" | "blobs" | "checkouts"; + +/** + * What a read request carries outside its body. + * + * Separate from the operation on purpose. The release fingerprint decides + * whether this build will talk to the caller at all, and deciding that after + * decoding the body would mean a mismatched release had already been allowed to + * drive a parser. A host reads these from the request's own metadata, exactly + * as the executor plane reads them from its upgrade headers. + */ +export interface ReadAdmission { + readonly release: string | null; + readonly token: string | null; + readonly runId: string | null; +} + +function failure(reason: string, path: string): Error { + return new WorkflowRecordMalformedError(`read request at ${path}`, reason); +} + +function text(value: unknown, path: string, maximum = MAX_RUN_ID): string { + if (typeof value !== "string" || value === "" || value.length > maximum) { + throw failure("expected bounded non-empty text", path); + } + return value; +} + +function nullableText(value: unknown, path: string): string | null { + return value === null ? null : text(value, path); +} + +/** A position in a selection: whole, not negative, and one this build can hold. */ +function ordinal(value: unknown, path: string): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + throw failure("expected a position in the selection", path); + } + return value; +} + +/** + * One read operation, parsed only after admission has passed. + * + * Strict about membership on purpose: a body carrying more than this release + * declares is not a request this build understands, and reading it leniently + * would accept a shape a later version wrote. + */ +export function parseReadOperation(raw: string): ReadOperation { + if (new TextEncoder().encode(raw).length > READ_REQUEST_BYTES) { + throw failure("expected a bounded request", "$"); + } + let decoded: unknown; + try { + decoded = JSON.parse(raw); + } catch { + throw failure("expected one JSON object", "$"); + } + const read = parseMembers(decoded, "$", failure); + const operation = read.get("operation"); + + if (operation === "inspect") { + requireMemberNames(read, ["operation"], "$", failure); + return { operation }; + } + if (operation === "history" || operation === "fork-source") { + const names = + operation === "history" + ? ["operation", "anchor", "after"] + : ["operation", "checkpointEventId", "section", "anchor", "after"]; + requireMemberNames(read, names, "$", failure); + const anchor = nullableText(read.get("anchor"), "$.anchor"); + const after = + read.get("after") === null + ? null + : operation === "history" + ? text(read.get("after"), "$.after") + : ordinal(read.get("after"), "$.after"); + if (anchor === null && after !== null) { + // Nothing to continue from: a cursor without a snapshot names no page. + throw failure("expected a cursor only inside an anchored snapshot", "$"); + } + if (operation === "history") { + if (typeof after === "number") { + throw failure("expected a cursor this operation pages by", "$.after"); + } + return { operation, anchor, after }; + } + if (typeof after === "string") { + throw failure("expected a cursor this operation pages by", "$.after"); + } + const section = read.get("section"); + if ( + section !== "inherited" && + section !== "roots" && + section !== "manifests" && + section !== "blobs" && + section !== "checkouts" + ) { + throw failure("expected a section this build answers", "$.section"); + } + return { + operation, + checkpointEventId: text(read.get("checkpointEventId"), "$.checkpointEventId"), + section, + anchor, + after, + }; + } + throw failure("expected an operation this build answers", "$.operation"); +} + +function rows(storage: OwnerStorage, sql: string, ...bindings: unknown[]): Row[] { + return storage.sql.exec(sql, ...bindings).toArray(); +} + +function only(storage: OwnerStorage, sql: string): Row | undefined { + return rows(storage, sql)[0]; +} + +/** The run this owner holds, or a refusal that it holds another or none. */ +function retained(storage: OwnerStorage, runId: string): Row { + recognizeObject(storage); + const row = only( + storage, + `SELECT run_id, definition, base, props, status, stop_reason_kind, stop_reason_code, + stop_reason_event_id, created_at, updated_at FROM workflow_run`, + ); + if (row === undefined) { + throw new CommandError("absent"); + } + if (readRunRecord(row).runId !== runId) { + throw new CommandError("wrong-run"); + } + return row; +} + +/** + * One coherent inspection, from one committed reading. + * + * Every member describes the same moment: the record, its executions, the + * frontier, the current root and the lineage are read together so a caller + * cannot be handed a run whose parts came from different commits. + */ +export function readInspection(storage: OwnerStorage, runId: string): Record { + const row = retained(storage, runId); + const executions = rows( + storage, + `SELECT execution_id, started_at, stopped_at, stop_status, stop_reason_kind, + stop_reason_code, stop_reason_event_id FROM document_executions ORDER BY sequence`, + ).map((entry) => readDocumentExecution(entry)); + const retrieval = only( + storage, + "SELECT metadata, revision, updated_at FROM definition_retrieval WHERE id = 1", + ); + const frontier = only( + storage, + "SELECT event_id, workspace_root_id FROM journal_events ORDER BY sequence DESC LIMIT 1", + ); + const state = only(storage, "SELECT current_root_id FROM workspace_state WHERE singleton_id = 1"); + if (state === undefined) { + throw new WorkflowRecordMalformedError("workflow owner storage", "it selects no Workspace"); + } + const lineage = only( + storage, + `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id + FROM workflow_fork_lineage WHERE id = 1`, + ); + + return { + record: readRunRecord(row), + executions, + retrieval: retrieval === undefined ? null : readRetrieval(retrieval), + journalFrontier: + frontier === undefined + ? null + : { + eventId: retainedText(frontier, "event_id"), + workspaceRootId: retainedDigest(frontier, "workspace_root_id"), + }, + currentWorkspaceRootId: retainedDigest(state, "current_root_id"), + lineage: + lineage === undefined + ? null + : { + sourceRunId: retainedText(lineage, "source_run_id"), + checkpointEventId: retainedText(lineage, "checkpoint_event_id"), + checkpointWorkspaceRootId: retainedDigest(lineage, "checkpoint_workspace_root_id"), + }, + }; +} + +/** + * One anchored page of retained journal rows. + * + * The anchor is the terminal event the first page chose, and every later page + * is held to it: a run that appended after the snapshot began does not grow the + * answer, because the anchor decides what the answer is about. + */ +export function readHistoryPage( + storage: OwnerStorage, + runId: string, + anchor: string | null, + after: string | null, +): Record { + retained(storage, runId); + const terminal = only( + storage, + "SELECT event_id FROM journal_events ORDER BY sequence DESC LIMIT 1", + ); + const selected = anchor ?? (terminal === undefined ? null : retainedText(terminal, "event_id")); + if (selected === null) { + // An empty history is terminal and carries nothing. + return { anchor: null, after: null, rows: [], done: true, retainedRoots: [], provenance: [] }; + } + const sequenceOf = (eventId: string): number => { + const row = rows(storage, "SELECT sequence FROM journal_events WHERE event_id = ?", eventId)[0]; + if (row === undefined) { + // An anchor or cursor this run does not hold. Answering from the rest + // would be answering about a different snapshot. + throw new CommandError("stale-journal"); + } + return retainedCount(row, "sequence"); + }; + const anchorAt = sequenceOf(selected); + const afterAt = after === null ? 0 : sequenceOf(after); + const page = rows( + storage, + `SELECT event_id, record, workspace_root_id FROM journal_events + WHERE sequence > ? AND sequence <= ? ORDER BY sequence LIMIT ?`, + afterAt, + anchorAt, + READ_PAGE_ENTRIES, + ).map((row) => ({ + eventId: retainedText(row, "event_id"), + record: retainedText(row, "record"), + workspaceRootId: retainedDigest(row, "workspace_root_id"), + })); + const last = page.at(-1); + const done = last === undefined || last.eventId === selected; + return { + anchor: selected, + after, + rows: page, + done, + // Sent with the terminal page alone, because they describe the whole + // snapshot rather than one page of it. + retainedRoots: done + ? rows(storage, "SELECT root_id FROM workspace_roots ORDER BY root_id").map((row) => + retainedDigest(row, "root_id"), + ) + : [], + provenance: done + ? rows( + storage, + `SELECT event_id, source_run_id, source_event_id + FROM journal_event_provenance ORDER BY event_id`, + ).map((row) => ({ + eventId: retainedText(row, "event_id"), + sourceRunId: retainedText(row, "source_run_id"), + sourceEventId: retainedText(row, "source_event_id"), + })) + : [], + }; +} + +/** + * The selection one checkpoint names, as values rather than pages. + * + * Computed whole every time a page is asked for, from one committed reading. + * That is what makes paging safe without a retained session: there is nothing + * to keep alive between pages, and a page is answered from the same selection + * the first one was — or the anchor no longer matches and the whole read fails. + */ +interface ForkSelection { + readonly anchor: string; + readonly checkpointWorkspaceRootId: string; + readonly runRecordWorkspaceRootId: string; + readonly rootImportWorkspaceRootId: string; + readonly inherited: readonly { eventId: string; record: string; workspaceRootId: string }[]; + readonly rootIds: readonly string[]; + readonly manifestHashes: readonly string[]; + readonly blobHashes: readonly string[]; + readonly checkoutPaths: ReadonlySet; +} + +/** + * Select the prefix this checkpoint names, and everything it depends on. + * + * The rules are the shared ones: the source's own run record and root import + * are what a fork writes for itself and are excluded; every root the prefix + * touches is needed, and so is the content those roots close over; and only a + * checkout whose directory exists in the checkpoint's own Workspace is + * inherited, so a Repository the source created afterwards does not arrive in a + * fork with nowhere to put it. + */ +function selectForkSource(storage: OwnerStorage, checkpointEventId: string): ForkSelection { + const all = rows( + storage, + "SELECT event_id, record, workspace_root_id FROM journal_events ORDER BY sequence", + ).map((row) => ({ + eventId: retainedText(row, "event_id"), + record: retainedText(row, "record"), + workspaceRootId: retainedDigest(row, "workspace_root_id"), + })); + const at = all.findIndex((row) => row.eventId === checkpointEventId); + if (at === -1) { + throw new CommandError("stale-journal"); + } + const prefix = all.slice(0, at + 1); + const checkpoint = prefix[at]; + if (checkpoint === undefined) { + throw new CommandError("stale-journal"); + } + + const classify = (row: { record: string }) => { + const parsed = parseDurableEvent(row.record); + if (!parsed.ok) { + throw new CommandError("corrupt-journal"); + } + return parsed.value; + }; + const record = prefix.find((row) => isRunRecordEvent(classify(row))); + if (record === undefined) { + // Nothing to inherit: the source recorded no run of its own before here. + throw new CommandError("not-forkable"); + } + const rootImport = prefix.find((row) => isRootImportEvent(classify(row))); + const inherited = prefix.filter((row) => row !== record && row !== rootImport); + + const rootIds = new Set(prefix.map((row) => row.workspaceRootId)); + rootIds.add(checkpoint.workspaceRootId); + const ordered = [...rootIds].sort(); + + const manifestHashes = new Set(); + const blobHashes = new Set(); + for (const rootId of ordered) { + for (const hash of referenced( + storage, + "workspace_root_manifest_refs", + "manifest_hash", + rootId, + )) { + manifestHashes.add(hash); + } + for (const hash of referenced(storage, "workspace_root_blob_refs", "blob_hash", rootId)) { + blobHashes.add(hash); + } + } + + const checkoutPaths = checkpointDirectories(storage, checkpoint.workspaceRootId); + const selection = { + checkpointEventId, + checkpointWorkspaceRootId: checkpoint.workspaceRootId, + runRecordWorkspaceRootId: record.workspaceRootId, + rootImportWorkspaceRootId: rootImport?.workspaceRootId ?? record.workspaceRootId, + inherited, + rootIds: ordered, + manifestHashes: [...manifestHashes].sort(), + blobHashes: [...blobHashes].sort(), + checkoutPaths, + }; + return { ...selection, anchor: selectionAnchor(storage, selection) }; +} + +function referenced( + storage: OwnerStorage, + table: string, + column: string, + rootId: string, +): string[] { + const sql = + table === "workspace_root_manifest_refs" + ? "SELECT lower(hex(manifest_hash)) AS hash FROM workspace_root_manifest_refs WHERE root_id = ?" + : "SELECT lower(hex(blob_hash)) AS hash FROM workspace_root_blob_refs WHERE root_id = ?"; + return rows(storage, sql, rootId).map((row) => retainedDigest(row, "hash")); +} + +/** + * An identity for the whole selection, not for the checkpoint alone. + * + * The checkpoint says which prefix; it says nothing about which roots, content + * or checkouts came with it, and those travel in separate requests. Everything + * a destination will copy that is not already named by a digest goes in here: + * the inherited rows with their bytes and root associations, the three head + * roots, each root's canonical record and its exact ordered reference sets, and + * every selected checkout in full. + * + * Retained mappings are appendable, which is the case this exists for. A + * qualifying Repository added between the earlier sections and the checkouts + * section changes this value, so the sequence refuses instead of joining two + * committed states. + * + * A content-addressed identity stands for its bytes, because the parse on the + * other side proves the bytes hash to it. Nothing else is stood in for. + */ +function selectionAnchor( + storage: OwnerStorage, + selection: { + checkpointEventId: string; + checkpointWorkspaceRootId: string; + runRecordWorkspaceRootId: string; + rootImportWorkspaceRootId: string; + inherited: readonly { eventId: string; record: string; workspaceRootId: string }[]; + rootIds: readonly string[]; + manifestHashes: readonly string[]; + blobHashes: readonly string[]; + checkoutPaths: ReadonlySet; + }, +): string { + return sha256Hex( + JSON.stringify({ + checkpointEventId: selection.checkpointEventId, + checkpointWorkspaceRootId: selection.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: selection.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: selection.rootImportWorkspaceRootId, + inherited: selection.inherited.map((row) => [row.eventId, row.record, row.workspaceRootId]), + roots: selection.rootIds.map((rootId) => readStoredRoot(storage, rootId)), + // The metadata, not only the identities. A digest stands for its bytes + // and for the size derived from them, but not for a watermark: that is + // copied into destination storage and can change while the content does + // not. + manifests: selection.manifestHashes.map((hash) => readStoredManifest(storage, hash)), + blobs: selection.blobHashes.map((hash) => blobMetadata(storage, hash)), + checkouts: [ + ...readCheckoutRepositories(storage, selection.checkoutPaths), + ...readCheckoutWorktrees(storage, selection.checkoutPaths), + ].map((entry) => [entry.cursor, entry.value]), + }), + ); +} + +/** The directories the checkpoint's own Workspace held. */ +function checkpointDirectories(storage: OwnerStorage, rootId: string): ReadonlySet { + const row = rows(storage, "SELECT manifest FROM workspace_roots WHERE root_id = ?", rootId)[0]; + if (row === undefined) { + throw new CommandError("stale-root"); + } + const parsed = parseWorkspaceRootManifest(retainedText(row, "manifest"), (reason) => { + throw new WorkflowRecordMalformedError("a retained Workspace root", reason); + }); + const directories = new Set(); + for (const entry of parsed.entries) { + if (entry.kind === "directory") { + directories.add(entry.path); + } + } + return directories; +} + +/** + * One page of one section of a fork's source. + * + * Sectioned because the parts are different kinds and sizes: rows, root + * manifests, encoded manifests and blob content each carry their own bound, and + * a page ends on whichever of count or bytes it reaches first. + */ +export function readForkSourcePage( + storage: OwnerStorage, + runId: string, + checkpointEventId: string, + section: ForkSourceSection, + anchor: string | null, + after: number | null, +): Record { + retained(storage, runId); + const selection = selectForkSource(storage, checkpointEventId); + if (anchor !== null && anchor !== selection.anchor) { + // The selection moved, so this page belongs to a snapshot that no longer + // exists. There is no partial answer to give. + throw new CommandError("stale-journal"); + } + + const head = { + anchor: selection.anchor, + after, + section, + checkpointEventId, + checkpointWorkspaceRootId: selection.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: selection.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: selection.rootImportWorkspaceRootId, + }; + + if (section === "inherited") { + return { + ...head, + ...page( + selection.inherited, + after, + // The journal's own order is not derivable from the rows, so each one + // says where it stands in the selected prefix. + (row, at) => ({ ...row, position: at }), + ), + }; + } + if (section === "roots") { + return { + ...head, + ...page(selection.rootIds, after, (rootId) => readStoredRoot(storage, rootId)), + }; + } + if (section === "manifests") { + return { + ...head, + ...page(selection.manifestHashes, after, (hash) => readStoredManifest(storage, hash)), + }; + } + if (section === "blobs") { + return { + ...head, + ...page(selection.blobHashes, after, (hash) => readStoredBlob(storage, hash)), + }; + } + const checkouts = [ + ...readCheckoutRepositories(storage, selection.checkoutPaths), + ...readCheckoutWorktrees(storage, selection.checkoutPaths), + ]; + return { + ...head, + ...page(checkouts, after, (entry) => entry.value), + }; +} + +/** + * One page out of an ordered selection, bounded by count and by bytes. + * + * The cursor is the position of the last member carried, which is meaningful + * only inside the anchor that pins this selection: the selection cannot have + * moved under a position without the anchor changing first. The page also says + * where it begins, which is what makes a whole sequence checkable against the + * size the selection declared. + */ +function page( + members: readonly T[], + after: number | null, + valueOf: (member: T, at: number) => V, +): Record { + if (after !== null && after >= members.length) { + // A position this selection does not hold. Under one anchor that can only + // be a cursor from somewhere else. + throw new CommandError("stale-journal"); + } + const from = after === null ? 0 : after + 1; + const carried: V[] = []; + let bytes = 0; + let at = from; + for (; at < members.length && carried.length < READ_PAGE_ENTRIES; at += 1) { + const member = members[at]; + if (member === undefined) { + break; + } + const value = valueOf(member, at); + const size = new TextEncoder().encode(JSON.stringify(value)).length; + if (carried.length > 0 && bytes + size > READ_PAGE_BYTES) { + break; + } + if (size > READ_PAGE_BYTES) { + // One member larger than a whole page. There is no page that could + // carry it, so the read refuses rather than answering with something the + // runner must reject. + throw new CommandError("too-large"); + } + carried.push(value); + bytes += size; + } + return { + rows: carried, + // Where this page begins in the selection, so a reader can tell that a + // sequence covered every member exactly once rather than trusting that it + // did. + from, + // Where it ends, which is where the next one continues from. + cursor: carried.length === 0 ? after : at - 1, + done: at >= members.length, + total: members.length, + }; +} + +function readStoredRoot(storage: OwnerStorage, rootId: string): Record { + const row = rows( + storage, + "SELECT root_id, format_version, manifest FROM workspace_roots WHERE root_id = ?", + rootId, + )[0]; + if (row === undefined) { + throw new CommandError("stale-root"); + } + return { + rootId: retainedDigest(row, "root_id"), + formatVersion: retainedCount(row, "format_version"), + manifest: retainedText(row, "manifest"), + // In the one order a root is retained with, which is what a destination + // compares its own derivation against element for element. + manifestHashes: referenced( + storage, + "workspace_root_manifest_refs", + "manifest_hash", + rootId, + ).sort(compareUtf8), + blobHashes: referenced(storage, "workspace_root_blob_refs", "blob_hash", rootId).sort( + compareUtf8, + ), + }; +} + +function readStoredManifest(storage: OwnerStorage, hash: string): Record { + const row = rows( + storage, + "SELECT size, encoded, last_seen FROM vfs_manifests WHERE lower(hex(hash)) = ?", + hash, + )[0]; + if (row === undefined) { + throw new CommandError("stale-root"); + } + return { + hash, + size: retainedCount(row, "size"), + lastSeen: retainedCount(row, "last_seen"), + encoded: encodeBase64(retainedBytes(row, "encoded")), + }; +} + +/** + * One blob's retained metadata, without its bytes. + * + * The anchor needs what a destination will copy beside the content; the + * content itself is already named by the digest, and hashing megabytes into an + * anchor recomputed on every page would cost what it does not prove. + */ +function blobMetadata(storage: OwnerStorage, hash: string): Record { + const row = rows( + storage, + "SELECT size, last_seen FROM vfs_blobs WHERE lower(hex(hash)) = ?", + hash, + )[0]; + if (row === undefined) { + throw new CommandError("stale-root"); + } + return { + hash, + size: retainedCount(row, "size"), + lastSeen: retainedCount(row, "last_seen"), + }; +} + +function readStoredBlob(storage: OwnerStorage, hash: string): Record { + const row = rows( + storage, + `SELECT b.size AS size, b.last_seen AS last_seen, x.bytes AS bytes + FROM vfs_blobs AS b JOIN vfs_blob_bytes AS x ON x.hash = b.hash + WHERE lower(hex(b.hash)) = ?`, + hash, + )[0]; + if (row === undefined) { + throw new CommandError("stale-root"); + } + return { + hash, + size: retainedCount(row, "size"), + lastSeen: retainedCount(row, "last_seen"), + content: encodeBase64(retainedBytes(row, "bytes")), + }; +} + +/** + * One checkout's identity, as a key nothing else can spell. + * + * A Repository name and a Worktree name are retained as text and may hold any + * character, so joining them with a separator is not an identity: `("a:b", "c")` + * and `("a", "b:c")` are two retained Worktrees that would join to one string. + * A JSON array of the parts escapes what it must and separates what it must, + * so distinct tuples spell distinct keys. + */ +function checkoutKey(parts: readonly string[]): string { + return JSON.stringify(parts); +} + +/** Only the Repositories whose checkout the checkpoint's Workspace holds. */ +function readCheckoutRepositories( + storage: OwnerStorage, + checkoutPaths: ReadonlySet, +): { cursor: string; value: Record }[] { + return rows( + storage, + `SELECT name, locator, locator_fingerprint, requested_base, creation_commit, + primary_branch, object_format, checkout_path + FROM workspace_repositories ORDER BY name`, + ) + .filter((row) => checkoutPaths.has(retainedPath(row, "checkout_path"))) + .map((row) => ({ + cursor: checkoutKey(["repository", retainedText(row, "name")]), + value: { + kind: "repository", + name: retainedText(row, "name"), + locator: retainedText(row, "locator"), + locatorFingerprint: retainedDigest(row, "locator_fingerprint"), + requestedBase: retainedNullableText(row, "requested_base"), + creationCommit: retainedText(row, "creation_commit"), + primaryBranch: retainedText(row, "primary_branch"), + objectFormat: retainedObjectFormat(row, "object_format"), + checkoutPath: retainedPath(row, "checkout_path"), + }, + })); +} + +function readCheckoutWorktrees( + storage: OwnerStorage, + checkoutPaths: ReadonlySet, +): { cursor: string; value: Record }[] { + return rows( + storage, + `SELECT repository_name, name, requested_branch, requested_base, + creation_commit, checkout_path + FROM workspace_worktrees ORDER BY repository_name, name`, + ) + .filter((row) => checkoutPaths.has(retainedPath(row, "checkout_path"))) + .map((row) => ({ + cursor: checkoutKey([ + "worktree", + retainedText(row, "repository_name"), + retainedText(row, "name"), + ]), + value: { + kind: "worktree", + repositoryName: retainedText(row, "repository_name"), + name: retainedText(row, "name"), + requestedBranch: retainedText(row, "requested_branch"), + requestedBase: retainedNullableText(row, "requested_base"), + creationCommit: retainedText(row, "creation_commit"), + checkoutPath: retainedPath(row, "checkout_path"), + }, + })); +} + +/** Answer one admitted read from one committed reading of this owner. */ +export function answerRead( + storage: OwnerStorage, + runId: string, + read: ReadOperation, +): Record { + if (read.operation === "inspect") { + return readInspection(storage, runId); + } + if (read.operation === "history") { + return readHistoryPage(storage, runId, read.anchor, read.after); + } + return readForkSourcePage( + storage, + runId, + read.checkpointEventId, + read.section, + read.anchor, + read.after, + ); +} diff --git a/packages/workflow/src/cloudflare/retained.ts b/packages/workflow/src/cloudflare/retained.ts new file mode 100644 index 000000000..69245920c --- /dev/null +++ b/packages/workflow/src/cloudflare/retained.ts @@ -0,0 +1,90 @@ +/** + * Reading a retained SQLite value, rather than converting one. + * + * `String(null)` is `"null"` and `Number(null)` is `0`. Both are plausible + * values, and once a damaged row has been converted into one, nothing further + * down can tell that the store held the wrong type — a client checking shapes + * sees a well-formed answer. So a retained member is read as the type it is + * declared to be, and a row that is not that is damage, reported as damage. + * + * These are the same rules the accepted owner reads already hold their rows to; + * they live here so the read plane holds its rows to them too rather than + * keeping a second, weaker set. + */ + +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; +import { SHA256 } from "../workspace/root-manifest.ts"; +import { bytesOf } from "./encoding.ts"; + +/** One row as the runtime hands it over. */ +export type RetainedRow = Record; + +export function damaged(reason: string): never { + throw new WorkflowRecordMalformedError("workflow owner storage", reason); +} + +/** Non-empty text, or damage. */ +export function retainedText(row: RetainedRow, column: string): string { + const value = row[column]; + if (typeof value !== "string" || value === "") { + return damaged(`expected ${column} to be non-empty text`); + } + return value; +} + +/** Non-empty text or a real null, and nothing else. */ +export function retainedNullableText(row: RetainedRow, column: string): string | null { + const value = row[column]; + if (value === null) { + return null; + } + if (typeof value !== "string" || value === "") { + return damaged(`expected ${column} to be non-empty text or null`); + } + return value; +} + +/** A nonnegative whole number, or damage. */ +export function retainedCount(row: RetainedRow, column: string): number { + const value = row[column]; + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + return damaged(`expected ${column} to be a nonnegative whole number`); + } + return value; +} + +/** A lowercase hex content identity, or damage. */ +export function retainedDigest(row: RetainedRow, column: string): string { + const value = retainedText(row, column); + if (!SHA256.test(value)) { + return damaged(`expected ${column} to be a content identity`); + } + return value; +} + +/** A stored byte sequence, copied so the caller holds no cursor memory. */ +export function retainedBytes(row: RetainedRow, column: string): Uint8Array { + try { + return bytesOf(row[column]); + } catch { + return damaged(`expected ${column} to be a byte sequence`); + } +} + +/** One of the object formats this build writes, or damage. */ +export function retainedObjectFormat(row: RetainedRow, column: string): "sha1" | "sha256" { + const value = retainedText(row, column); + if (value !== "sha1" && value !== "sha256") { + return damaged(`expected ${column} to name an object format`); + } + return value; +} + +/** A Workspace-relative path, which is absolute within the Workspace. */ +export function retainedPath(row: RetainedRow, column: string): string { + const value = retainedText(row, column); + if (!value.startsWith("/")) { + return damaged(`expected ${column} to be a Workspace path`); + } + return value; +} diff --git a/packages/workflow/src/fork.ts b/packages/workflow/src/fork.ts index a564fc2e4..423296b72 100644 --- a/packages/workflow/src/fork.ts +++ b/packages/workflow/src/fork.ts @@ -38,6 +38,8 @@ import { Err, Ok, type Result } from "effection"; import { describeWorkflowRun, WORKFLOW_RUN, type WorkflowRun } from "./journal.ts"; import type { Forkability } from "./lifecycle/forkability.ts"; import { WorkflowRequestError } from "./storage/errors.ts"; +import { isRootImportEvent, isRunRecordEvent } from "./journal-events.ts"; +export { isRootImportEvent, isRunRecordEvent } from "./journal-events.ts"; /** The coroutine a run's own record and canonical outcome belong to. */ const ROOT_COROUTINE = "root"; @@ -156,25 +158,6 @@ export function forkJournal( ]); } -/** Whether this event is the root coroutine's import of the root document. */ -export function isRootImportEvent(event: DurableEvent): boolean { - return ( - event.type === "yield" && - event.description.type === IMPORT_COMPONENT && - event.description.name === ROOT_DOCUMENT - ); -} - -/** Whether this event is the root coroutine's own `workflow_run` record. */ -export function isRunRecordEvent(event: DurableEvent): boolean { - return ( - event.type === "yield" && - event.coroutineId === ROOT_COROUTINE && - event.description.type === WORKFLOW_RUN && - event.description.name === WORKFLOW_RUN - ); -} - /** Whether this event is the root's Close — the run's canonical outcome. */ function isRootOutcome(event: DurableEvent): boolean { return event.type === "close" && event.coroutineId === ROOT_COROUTINE; diff --git a/packages/workflow/src/journal-events.ts b/packages/workflow/src/journal-events.ts new file mode 100644 index 000000000..73f66aa21 --- /dev/null +++ b/packages/workflow/src/journal-events.ts @@ -0,0 +1,39 @@ +/** + * The two events a fork writes for itself, recognized by shape. + * + * A fork inherits a prefix of its source's journal, but not the source's own + * identity and not the import of the document the source was run from — it has + * its own of each. Telling those two rows apart from everything else is a + * property of the event, so it is stated here rather than wherever a prefix + * happens to be selected. + * + * Separate from `fork.ts` because both hosts need it and the owner cannot + * reach that module: selecting a fork's source on a Durable Object must not + * drag in forkability classification and everything it imports. + */ + +import type { DurableEvent } from "@executablemd/durable-streams"; +import { WORKFLOW_RUN } from "./journal.ts"; + +const ROOT_COROUTINE = "root"; +const IMPORT_COMPONENT = "import_component"; +const ROOT_DOCUMENT = "__root__"; + +/** Whether this event is the import of the run's root document. */ +export function isRootImportEvent(event: DurableEvent): boolean { + return ( + event.type === "yield" && + event.description.type === IMPORT_COMPONENT && + event.description.name === ROOT_DOCUMENT + ); +} + +/** Whether this event is the root coroutine's own `workflow_run` record. */ +export function isRunRecordEvent(event: DurableEvent): boolean { + return ( + event.type === "yield" && + event.coroutineId === ROOT_COROUTINE && + event.description.type === WORKFLOW_RUN && + event.description.name === WORKFLOW_RUN + ); +} diff --git a/packages/workflow/src/remote/inspection.ts b/packages/workflow/src/remote/inspection.ts new file mode 100644 index 000000000..45c5361d8 --- /dev/null +++ b/packages/workflow/src/remote/inspection.ts @@ -0,0 +1,109 @@ +/** + * What a run says about itself, when its storage is somewhere else. + * + * Reading is not advancing, so none of this takes the run's executor + * acquisition: an inspection can be answered while an executor is live, and + * asking one never makes a run unrunnable. Nothing here recovers a stale + * execution, attaches a Workspace, materializes a root, imports a document, + * contacts a provider or appends anything. + * + * The projection is done here rather than by the adapter. The owner returns + * retained rows; what a history *means* — its authored source, its cumulative + * forkability, its inherited provenance — is provider-neutral, and computing it + * on the runner is what keeps one interpretation of a journal rather than one + * per adapter. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import { checkRunId } from "../storage/create-request.ts"; +import { WorkflowRunIdMismatchError, WorkflowRunNotFoundError } from "../storage/errors.ts"; +import { WorkflowLifecycle } from "../lifecycle/api.ts"; +import type { WorkflowLifecycleSnapshot } from "../lifecycle/api.ts"; +import { projectHistory, type WorkflowHistoryEntry } from "../lifecycle/history.ts"; +import type { RemoteReadPlane, RetainedInspection } from "./read.ts"; + +/** + * Install the read-only lifecycle operations over one owner's read plane. + * + * The plane is bound to one admitted owner, so the domain of `list()` is that + * owner: zero snapshots when it holds no run, one when it does. That is a + * complete answer to "every run this is bound to", and it claims nothing about + * a namespace — there is no registry here and nothing to enumerate. + */ +export function useRemoteLifecycleReads(plane: RemoteReadPlane): Operation { + return WorkflowLifecycle.around( + { + *inspect([runId]): Operation> { + const addressed = addressing(plane, runId); + if (addressed !== undefined) { + return addressed; + } + const read = yield* plane.inspect(); + return read.ok ? Ok(snapshotOf(read.value)) : read; + }, + + *list(): Operation> { + const read = yield* plane.inspect(); + if (read.ok) { + return Ok(Object.freeze([snapshotOf(read.value)])); + } + // An owner holding no run at all lists nothing. Everything else — a + // foreign, incompatible, damaged or unreadable owner — fails the whole + // request, exactly as the local provider does: a shorter list is an + // answer to a question nobody asked. + return read.error instanceof WorkflowRunNotFoundError ? Ok(Object.freeze([])) : read; + }, + + *history([runId]): Operation> { + const addressed = addressing(plane, runId); + if (addressed !== undefined) { + return addressed; + } + const read = yield* plane.history(); + if (!read.ok) { + return read; + } + return Ok( + projectHistory(read.value.entries, { + retainedRoots: read.value.retainedRoots, + inherited: read.value.inherited, + }), + ); + }, + }, + { at: "min" }, + ); +} + +/** + * Why this request is not for this plane's run, or nothing when it is. + * + * Refused here, before the transport is reached, so a request naming another + * run neither travels nor comes back with the bound run's answer. The bound id + * is compared rather than trusted: it can only ever cause a refusal. + */ +function addressing(plane: RemoteReadPlane, runId: string): Result | undefined { + const checked = checkRunId(runId); + if (!checked.ok) { + return checked; + } + if (checked.value !== plane.runId) { + return Err(new WorkflowRunIdMismatchError(checked.value, REMOTE_RUN)); + } + return undefined; +} + +/** How a remote run's storage is named in a public error. */ +const REMOTE_RUN = "this run's remote storage"; + +/** The retained reading, as the public snapshot it projects to. */ +function snapshotOf(read: RetainedInspection): WorkflowLifecycleSnapshot { + return Object.freeze({ + record: read.record, + executions: read.executions, + ...(read.retrieval === undefined ? {} : { retrieval: read.retrieval }), + ...(read.journalFrontier === undefined ? {} : { journalFrontier: read.journalFrontier }), + currentWorkspaceRootId: read.currentWorkspaceRootId, + ...(read.lineage === undefined ? {} : { lineage: read.lineage }), + }); +} diff --git a/packages/workflow/src/remote/read.ts b/packages/workflow/src/remote/read.ts index cd2870968..dad7756e7 100644 --- a/packages/workflow/src/remote/read.ts +++ b/packages/workflow/src/remote/read.ts @@ -20,8 +20,14 @@ import type { Operation } from "effection"; import type { RemoteInvocationSnapshot } from "./records.ts"; +import type { Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; import type { JournalEntry } from "../storage/api.ts"; -import type { DefinitionRetrieval, WorkflowRunRecord } from "../storage/record.ts"; +import type { + DefinitionRetrieval, + DocumentExecutionRecord, + WorkflowRunRecord, +} from "../storage/record.ts"; import type { WorkspaceRootManifest } from "../workspace/root-manifest.ts"; import type { StartingFrontier } from "./collector.ts"; @@ -58,3 +64,145 @@ export function startingFrontier(snapshot: RemoteFrontierSnapshot): StartingFron events: snapshot.entries.map((entry) => structuredClone(entry.event)), }; } + +/** Where one inherited row came from. Rows a run wrote itself have none. */ +export interface RetainedProvenance { + readonly sourceRunId: string; + readonly sourceEventId: string; +} + +/** One retained journal row, parsed but not yet projected. */ +export interface RetainedRow { + readonly eventId: string; + readonly event: DurableEvent; + readonly workspaceRootId: string; +} + +/** Everything one anchored history sequence produced, once it terminated. */ +export interface RetainedHistory { + readonly entries: readonly RetainedRow[]; + readonly retainedRoots: ReadonlySet; + readonly inherited: ReadonlyMap; +} + +/** One run's committed state, as the owner reported it. */ +export interface RetainedInspection { + readonly record: WorkflowRunRecord; + readonly executions: readonly DocumentExecutionRecord[]; + readonly retrieval?: DefinitionRetrieval; + readonly journalFrontier?: { readonly eventId: string; readonly workspaceRootId: string }; + readonly currentWorkspaceRootId: string; + readonly lineage?: { + readonly sourceRunId: string; + readonly checkpointEventId: string; + readonly checkpointWorkspaceRootId: string; + }; +} + +/** + * What the runner may ask this owner for, without taking it. + * + * Both answer with parsed retained values rather than public projections: the + * projection is provider-neutral and belongs on the runner, so there is one + * meaning of a history rather than one per adapter. + */ +export interface RemoteReadPlane { + /** + * The one run this plane was opened for. + * + * Descriptive, and compared rather than trusted: it lets a provider refuse a + * request for another run before it reaches the owner. It authorizes + * nothing — the plane can only ever answer about the run it was built with. + */ + readonly runId: string; + inspect(): Operation>; + history(): Operation>; + /** + * Everything a fork must copy out of this run at one checkpoint. + * + * Private: a destination transition calls it through a narrow internal + * accessor. It is copy data and not authority — the destination still needs + * its own live acquisition and its own atomic commit. + */ + forkSource(checkpointEventId: string): Operation>; +} + +/** + * One inherited row, as a fork must copy it. + * + * The exact retained record string, not only the event it parses to. A record + * is validated before it is accepted, but two different spellings can parse to + * one event, and a destination inserting a reconstructed spelling would retain + * history that is not the history it inherited. A destination writes this + * string into its journal as it stands, with nothing re-encoded. Public + * history projects the parsed event and never carries it. + */ +export interface RemoteForkRow { + readonly eventId: string; + /** The retained record, byte for byte. */ + readonly record: string; + readonly workspaceRootId: string; +} + +/** One DOFS manifest, as content a fork must hold for itself. */ +export interface RemoteManifest { + readonly hash: string; + readonly size: number; + readonly lastSeen: number; + readonly encoded: Uint8Array; +} + +/** One DOFS blob and its bytes. */ +export interface RemoteBlob { + readonly hash: string; + readonly size: number; + readonly lastSeen: number; + readonly content: Uint8Array; +} + +/** One immutable Workspace root the selected prefix requires. */ +export interface RemoteStoredRoot { + readonly rootId: string; + readonly formatVersion: number; + readonly manifest: string; + readonly manifestHashes: readonly string[]; + readonly blobHashes: readonly string[]; +} + +/** One checkout the checkpoint's Workspace holds, as a fork inherits it. */ +export type RemoteCheckout = + | { + readonly kind: "repository"; + readonly name: string; + readonly locator: string; + readonly locatorFingerprint: string; + readonly requestedBase: string | null; + readonly creationCommit: string; + readonly primaryBranch: string; + readonly objectFormat: string; + readonly checkoutPath: string; + } + | { + readonly kind: "worktree"; + readonly repositoryName: string; + readonly name: string; + readonly requestedBranch: string; + readonly requestedBase: string | null; + readonly creationCommit: string; + readonly checkoutPath: string; + }; + +/** Everything one checkpoint hands a fork, read in one committed selection. */ +export interface RemoteForkSource { + readonly sourceRunId: string; + readonly checkpointEventId: string; + readonly checkpointWorkspaceRootId: string; + readonly runRecordWorkspaceRootId: string; + readonly rootImportWorkspaceRootId: string; + /** The prefix without the two rows the fork writes for itself. */ + readonly inherited: readonly RemoteForkRow[]; + readonly roots: readonly RemoteStoredRoot[]; + readonly manifests: readonly RemoteManifest[]; + readonly blobs: readonly RemoteBlob[]; + readonly checkouts: readonly RemoteCheckout[]; +} diff --git a/packages/workflow/tests/cloudflare/remote-read-plane.vitest.ts b/packages/workflow/tests/cloudflare/remote-read-plane.vitest.ts new file mode 100644 index 000000000..724905899 --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-read-plane.vitest.ts @@ -0,0 +1,726 @@ +/** + * Reading a run's owner without taking it. + * + * The claim under test is one a fake cannot make: that an ordinary + * authenticated request answers from a real Durable Object's committed SQLite + * state while an executor WebSocket is live, and that asking takes no + * acquisition. Only the runtime's own acquisition set can settle that. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { run, type Operation, until } from "effection"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; +import { + cloudflareReadPlane, + FORK_SOURCE_ANSWER_BYTES, + type ReadTransport, +} from "../../src/cloudflare/read-client.ts"; +import { + READ_PAGE_BYTES, + READ_PAGE_ENTRIES, + READ_REQUEST_BYTES, +} from "../../src/cloudflare/read-plane.ts"; +import { cloudflareRunLink } from "../../src/cloudflare/client.ts"; +import { + useOwnerConnection, + type OwnerSocket, + type SocketListener, +} from "../../src/remote/client.ts"; +import { useRemoteRunStorage } from "../../src/remote/storage.ts"; +import { WorkflowRunStorage } from "../../src/storage/api.ts"; +import type { CreateWorkflowRunRequest } from "../../src/storage/api.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`read-${unique}-${Math.random()}`)); +} + +function on(stub: ReturnType, body: (o: ExecutorObject) => T): Promise { + return runInDurableObject(stub, body); +} + +async function token(): Promise { + return await signToken(keys, { ...VALID_CLAIMS, iat: NOW - 10, nbf: NOW - 10, exp: NOW + 600 }); +} + +async function connect(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const response = await stub.fetch("https://owner.invalid/executor", { + headers: { + authorization: `Bearer ${await token()}`, + upgrade: "websocket", + "x-release": POLICY.release, + "x-run-id": RUN_ID, + }, + }); + const socket = response.webSocket; + if (socket === null) { + throw new Error(`expected an executor WebSocket, received ${response.status}`); + } + socket.accept(); + return socket; +} + +function ownerSocket(socket: WebSocket): OwnerSocket { + const listeners = new Map(); + return { + send: (data) => socket.send(data), + close: () => socket.close(), + addEventListener(type, listener) { + const forward: EventListener = (event) => { + const data: unknown = Reflect.get(event, "data"); + listener(typeof data === "string" ? { data } : {}); + }; + listeners.set(listener, forward); + socket.addEventListener(type, forward); + }, + removeEventListener(type, listener) { + const found = listeners.get(listener); + if (found !== undefined) { + socket.removeEventListener(type, found); + } + }, + }; +} + +/** One ordinary read request, answered by the object itself. */ +async function ask( + stub: ReturnType, + admission: { release: string | null; token: string | null; runId: string | null }, + body: string, +): Promise { + return await on(stub, (owner) => owner.readRequest(admission, body)); +} + +/** The read plane's transport: an ordinary request, and no socket at all. */ +function transportTo(stub: ReturnType): ReadTransport { + return { + *send(admission, body: string): Operation { + // Through the object itself, which is what makes "reading took no + // acquisition" observable rather than asserted. + return yield* until(ask(stub, admission, body)); + }, + }; +} + +function creation(): CreateWorkflowRunRequest { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + }; +} + +describe("reading a run's owner without taking it", () => { + it("answers while an executor is live, and takes no acquisition", async () => { + const stub = executor(); + const socket = await connect(stub); + const bearer = await token(); + const held = (state: string) => + on(stub, (owner) => ({ + state, + holders: owner.holders(), + acquisition: owner.acquisitionId(), + run: owner.runRow(), + })); + + // Everything inside one scope, because the connection *is* the + // acquisition: reading after it closed would prove nothing about + // coexisting with a live executor. + const seen = await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + let identifier = 0; + yield* useRemoteRunStorage( + cloudflareRunLink(connection, () => `open-${(identifier += 1)}`, RUN_ID), + ); + yield* WorkflowRunStorage.operations.create(creation()); + + const before = yield* until(held("before")); + const answered = JSON.parse( + yield* until( + ask( + stub, + { release: POLICY.release, token: bearer, runId: RUN_ID }, + JSON.stringify({ operation: "inspect" }), + ), + ), + ); + return { before, answered, after: yield* until(held("after")) }; + }); + + expect(seen.before.holders).toBe(1); + expect(seen.answered["outcome"]).toBe("performed"); + // The exact holder before and after, no second holder, and no retained row + // changed by reading. + expect(seen.after).toEqual({ ...seen.before, state: "after" }); + }); + + it("refuses a wrong release before it verifies anything", async () => { + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + // A body this build could not parse at all. If the release were compared + // after parsing, this would refuse as malformed instead. + const answered = JSON.parse( + await ask( + stub, + { release: "some-other-release", token: "not-a-token", runId: RUN_ID }, + "{ this is not JSON at all", + ), + ); + expect(answered["outcome"]).toBe("refused"); + expect(String(answered["refusal"])).toContain("release:"); + // Nothing was created, and nobody holds anything. + expect(await on(stub, (owner) => owner.objectCount())).toBe(0); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + + it("refuses an unauthenticated read without touching the run", async () => { + const stub = executor(); + const socket = await connect(stub); + await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + let identifier = 0; + yield* useRemoteRunStorage( + cloudflareRunLink(connection, () => `open-${(identifier += 1)}`, RUN_ID), + ); + return yield* WorkflowRunStorage.operations.create(creation()); + }); + const before = await on(stub, (owner) => owner.runRow()); + + // Again with a body nothing could parse: authentication decides first. + const answered = JSON.parse( + await ask( + stub, + { release: POLICY.release, token: "not-a-token", runId: RUN_ID }, + "{ this is not JSON at all", + ), + ); + expect(answered["outcome"]).toBe("refused"); + // Not the release, and not a parse of the body either: authentication is + // what stopped it. + expect(String(answered["refusal"])).toContain("token:"); + // Whether an executor happened to be live is the coexistence test's claim; + // this one is that a refused read leaves the run exactly as it was. + expect(await on(stub, (owner) => owner.runRow())).toEqual(before); + }); + + it("answers one coherent inspection from one committed reading", async () => { + const stub = executor(); + const socket = await connect(stub); + await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + let identifier = 0; + yield* useRemoteRunStorage( + cloudflareRunLink(connection, () => `open-${(identifier += 1)}`, RUN_ID), + ); + return yield* WorkflowRunStorage.operations.create(creation()); + }); + + const bearer = await token(); + const answered = JSON.parse( + await run(() => + transportTo(stub).send( + { release: POLICY.release, token: bearer, runId: RUN_ID }, + JSON.stringify({ operation: "inspect" }), + ), + ), + ); + expect(answered["outcome"]).toBe("performed"); + const value = answered["value"]; + // The run, its executions, the frontier, the current root and the lineage + // all describe the same committed moment. + expect(value["record"]["runId"]).toBe(RUN_ID); + expect(value["record"]["status"]).toBe("running"); + expect(value["executions"]).toEqual([]); + expect(value["journalFrontier"]).toBe(null); + expect(value["lineage"]).toBe(null); + expect(typeof value["currentWorkspaceRootId"]).toBe("string"); + // It agrees with what the object actually selects. + expect(value["currentWorkspaceRootId"]).toBe( + (await on(stub, (owner) => owner.published()))["currentRootId"], + ); + }); + + it("refuses a read addressed to another run, and writes nothing", async () => { + const stub = executor(); + const socket = await connect(stub); + await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + let identifier = 0; + yield* useRemoteRunStorage( + cloudflareRunLink(connection, () => `open-${(identifier += 1)}`, RUN_ID), + ); + return yield* WorkflowRunStorage.operations.create(creation()); + }); + const before = await on(stub, (owner) => owner.runRow()); + + const bearer = await token(); + const answered = JSON.parse( + await run(() => + transportTo(stub).send( + { + release: POLICY.release, + token: bearer, + runId: "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa", + }, + JSON.stringify({ operation: "inspect" }), + ), + ), + ); + expect(answered["outcome"]).toBe("refused"); + expect(await on(stub, (owner) => owner.runRow())).toEqual(before); + }); + + it("hands a fork the whole selection, and nothing the source keeps", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + const socket = await connect(stub); + const bearer = await token(); + + const seen = await run(function* () { + // The executor stays live for the whole read, so this is also the proof + // that selecting a source takes nothing from it. + const connection = yield* useOwnerConnection(ownerSocket(socket)); + yield* connection.ask("keep-alive", { command: "frontier" }, (value) => value); + const before = yield* until( + on(stub, (owner) => ({ holders: owner.holders(), run: owner.runRow() })), + ); + + const plane = cloudflareReadPlane( + transportTo(stub), + POLICY.release, + // deno-lint-ignore require-yield + function* () { + return bearer; + }, + RUN_ID, + ); + const source = yield* plane.forkSource("event-work"); + // Appended after the checkpoint was selected, while the read is done. + yield* until(on(stub, (owner) => owner.appendJournal("event-later", "later"))); + const again = yield* plane.forkSource("event-work"); + return { + before, + source, + again, + after: yield* until( + on(stub, (owner) => ({ holders: owner.holders(), run: owner.runRow() })), + ), + }; + }); + + expect(seen.source.ok).toBe(true); + if (!seen.source.ok) { + throw seen.source.error; + } + const source = seen.source.value; + // The prefix without the two rows the fork writes for itself. + expect(source.inherited.map((row) => row.eventId)).toEqual(["event-work"]); + expect(source.checkpointEventId).toBe("event-work"); + expect(source.checkpointWorkspaceRootId).toBe(source.runRecordWorkspaceRootId); + // Everything the destination must own independently came with it: the + // roots the prefix names, and the content those roots close over. + expect(source.roots.map((root) => root.rootId)).toEqual([source.checkpointWorkspaceRootId]); + expect(source.manifests.length).toBeGreaterThan(0); + expect(source.blobs.length).toBeGreaterThan(0); + for (const root of source.roots) { + for (const hash of root.manifestHashes) { + expect(source.manifests.some((manifest) => manifest.hash === hash)).toBe(true); + } + for (const hash of root.blobHashes) { + expect(source.blobs.some((blob) => blob.hash === hash)).toBe(true); + } + } + + // An append after the checkpoint does not enter the selection, however + // long the sequence took. + expect(seen.again.ok).toBe(true); + if (seen.again.ok) { + expect(seen.again.value.inherited.map((row) => row.eventId)).toEqual(["event-work"]); + } + // The executor still holds the run, and nothing about it changed. + expect(seen.before.holders).toBe(1); + expect(seen.after).toEqual(seen.before); + }); + + it("carries a multi-page selection in one order, with the records as retained", async () => { + const stub = executor(); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + // More rows than one page may carry, so the selection is a sequence the + // client has to hold together rather than a single answer. + await on(stub, (owner) => owner.fillJournal(200)); + await on(stub, (owner) => owner.retainQualifyingRepository("alpha")); + await on(stub, (owner) => owner.retainQualifyingWorktree("alpha", "topic")); + const retained = await on(stub, (owner) => owner.journalRecords()); + const socket = await connect(stub); + const bearer = await token(); + + const seen = await run(function* () { + const connection = yield* useOwnerConnection(ownerSocket(socket)); + yield* connection.ask("keep-alive", { command: "frontier" }, (value) => value); + const before = yield* until( + on(stub, (owner) => ({ holders: owner.holders(), run: owner.runRow() })), + ); + const plane = cloudflareReadPlane( + transportTo(stub), + POLICY.release, + // deno-lint-ignore require-yield + function* () { + return bearer; + }, + RUN_ID, + ); + const source = yield* plane.forkSource("event-0199"); + return { + before, + source, + after: yield* until( + on(stub, (owner) => ({ holders: owner.holders(), run: owner.runRow() })), + ), + }; + }); + + expect(seen.source.ok).toBe(true); + if (!seen.source.ok) { + throw seen.source.error; + } + const source = seen.source.value; + // The prefix in journal order, without the two rows a fork writes for + // itself, and every record exactly as this owner retained it. + const inherited = retained.filter( + (row) => row.eventId !== "event-run" && row.eventId !== "event-import", + ); + expect(source.inherited.length).toBeGreaterThan(READ_PAGE_ENTRIES); + expect(source.inherited.map((row) => row.eventId)).toEqual(inherited.map((row) => row.eventId)); + expect(source.inherited.map((row) => row.record)).toEqual(inherited.map((row) => row.record)); + + // All three heads a destination writes against came with the selection. + const carried = source.roots.map((root) => root.rootId); + expect(carried).toContain(source.checkpointWorkspaceRootId); + expect(carried).toContain(source.runRecordWorkspaceRootId); + expect(carried).toContain(source.rootImportWorkspaceRootId); + + // The checkout graph, whole: the Repository and the Worktree that names + // it, each in a directory the checkpoint's Workspace holds. + expect(source.checkouts.map((checkout) => checkout.checkoutPath)).toEqual(["/", "/work"]); + const worktrees = source.checkouts.filter((checkout) => checkout.kind === "worktree"); + expect(worktrees.map((checkout) => checkout.repositoryName)).toEqual(["alpha"]); + expect( + source.checkouts.filter((checkout) => checkout.kind === "repository").map((one) => one.name), + ).toEqual(["alpha"]); + + // The executor held the run for the whole sequence, and nothing moved. + expect(seen.before.holders).toBe(1); + expect(seen.after).toEqual(seen.before); + }); + + it("continues a page of names JSON has to escape, and asks for the next in bytes", async () => { + // The shape a cursor made of names cannot survive. A retained name is text + // the schema bounds only by being non-empty, and a backslash is one + // character that JSON writes as two — twice over, if a cursor spelled out + // of names is then carried inside a request. These rows are near the + // largest a page admits, so nothing but a position could ask for the next + // one. + const escaping = (count: number, tail: string) => `${"\\".repeat(count)}"«${tail}»`; + const repositoryName = escaping(150_000, "repository"); + const first = escaping(80_000, "a"); + const second = escaping(80_000, "b"); + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + await on(stub, (owner) => owner.retainRepositoryAt(repositoryName, "/checkouts")); + await on(stub, (owner) => owner.retainWorktreeAt(repositoryName, first, "/work")); + await on(stub, (owner) => owner.retainWorktreeAt(repositoryName, second, "/")); + const bearer = await token(); + + const bytes = (text: string) => new TextEncoder().encode(text).length; + const asked: { section: string; request: number; answer: number }[] = []; + const measuring: ReadTransport = { + *send(admission, body: string): Operation { + const answer = yield* until(ask(stub, admission, body)); + const value: unknown = JSON.parse(body); + const section = + value !== null && typeof value === "object" ? String(Reflect.get(value, "section")) : ""; + asked.push({ section, request: bytes(body), answer: bytes(answer) }); + return answer; + }, + }; + + const outcome = await run(function* () { + const plane = cloudflareReadPlane( + measuring, + POLICY.release, + // deno-lint-ignore require-yield + function* () { + return bearer; + }, + RUN_ID, + ); + return yield* plane.forkSource("event-work"); + }); + + expect(outcome.ok).toBe(true); + if (!outcome.ok) { + throw outcome.error; + } + // Three checkouts, each near the largest member a page admits, so the + // section took three pages and two of them were continuations. + const pages = asked.filter((one) => one.section === "checkouts"); + expect(pages.length).toBe(3); + for (const checkout of outcome.value.checkouts) { + expect(bytes(JSON.stringify(checkout))).toBeLessThanOrEqual(READ_PAGE_BYTES); + } + expect(bytes(JSON.stringify(outcome.value.checkouts[1]))).toBeGreaterThan(400_000); + + // Every request the sequence needed fits the parser's bound, because a + // position is the same size whatever it points at. A cursor spelled out of + // these names would have been larger than the member itself. + for (const page of asked) { + expect(page.request).toBeLessThanOrEqual(READ_REQUEST_BYTES); + expect(page.answer).toBeLessThanOrEqual(FORK_SOURCE_ANSWER_BYTES); + } + expect(Math.max(...pages.map((one) => one.request))).toBeLessThan(1024); + expect(Math.max(...pages.map((one) => one.answer))).toBeGreaterThan(400_000); + + // The owner resolved each continuation to exactly one further member, and + // the composite identities came back distinct and whole. + expect(outcome.value.checkouts.map((one) => one.checkoutPath)).toEqual([ + "/checkouts", + "/work", + "/", + ]); + expect( + outcome.value.checkouts + .filter((one) => one.kind === "worktree") + .map((one) => [one.repositoryName, one.name]), + ).toEqual([ + [repositoryName, first], + [repositoryName, second], + ]); + }); + + it("refuses a continuation that names no member of the anchored selection", async () => { + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + const bearer = await token(); + const admission = { release: POLICY.release, token: bearer, runId: RUN_ID }; + const read = async (after: unknown, overrides: Record = {}) => + JSON.parse( + await ask( + stub, + admission, + JSON.stringify({ + operation: "fork-source", + checkpointEventId: "event-work", + section: "roots", + anchor: null, + after: null, + ...overrides, + ...(after === undefined ? {} : { after }), + }), + ), + ); + + const opening = await read(undefined); + expect(opening["outcome"]).toBe("performed"); + const anchor: unknown = opening["value"]["anchor"]; + const total: unknown = opening["value"]["total"]; + expect(total).toBe(1); + + // A position past the end of the section it names. + expect(await read(1, { anchor })).toEqual({ + outcome: "refused", + refusal: "command:stale-journal", + }); + // A position that is not one: a name, a fraction, a negative. + for (const malformed of ["0", 0.5, -1]) { + expect((await read(malformed, { anchor }))["outcome"]).toBe("refused"); + } + // A position with nothing pinning the selection it counts into. + expect((await read(0))["outcome"]).toBe("refused"); + // A section this build does not answer, and a checkpoint this run does not + // hold, are refused whether or not a position accompanies them. + expect((await read(0, { anchor, section: "elsewhere" }))["outcome"]).toBe("refused"); + expect((await read(0, { anchor, checkpointEventId: "event-nowhere" }))["outcome"]).toBe( + "refused", + ); + expect((await read(0, { anchor: "b".repeat(64) }))["outcome"]).toBe("refused"); + }); + + it("refuses a checkpoint this run does not hold, and a prefix with no run", async () => { + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + await on(stub, (owner) => owner.initialize()); + const bearer = await token(); + const plane = () => + cloudflareReadPlane( + transportTo(stub), + POLICY.release, + // deno-lint-ignore require-yield + function* () { + return bearer; + }, + RUN_ID, + ); + + // Nothing retained at all: no checkpoint to select. + const missing = await run(() => plane().forkSource("event-nowhere")); + expect(missing.ok).toBe(false); + expect(String(missing.ok === false && missing.error)).not.toContain("command:"); + + // A prefix that records no run of its own is not one a fork could inherit. + await on(stub, (owner) => owner.appendJournal("event-alone", "alone")); + const unforkable = await run(() => plane().forkSource("event-alone")); + expect(unforkable.ok).toBe(false); + expect(String(unforkable.ok === false && unforkable.error)).not.toContain("command:"); + // Neither refusal wrote anything. + expect((await on(stub, (owner) => owner.published()))["events"]).toHaveLength(1); + }); + + it("changes the selection anchor when a qualifying checkout is added", async () => { + // The case the anchor exists for. Retained mappings are appendable, and a + // qualifying one added between sections would otherwise let a client join + // earlier pages to checkouts from a different committed state. + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + const bearer = await token(); + const anchorNow = async (): Promise => { + const answered = JSON.parse( + await ask( + stub, + { release: POLICY.release, token: bearer, runId: RUN_ID }, + JSON.stringify({ + operation: "fork-source", + checkpointEventId: "event-work", + section: "inherited", + anchor: null, + after: null, + }), + ), + ); + expect(answered["outcome"]).toBe("performed"); + return answered["value"]["anchor"]; + }; + + const before = await anchorNow(); + expect(typeof before).toBe("string"); + // The same selection, read twice, is the same selection. + expect(await anchorNow()).toBe(before); + + await on(stub, (owner) => owner.retainQualifyingRepository("added-between")); + // A checkout the destination would copy changed, so the selection did. + expect(await anchorNow()).not.toBe(before); + }); + + it("refuses a fork-source page from a selection that has moved", async () => { + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + const bearer = await token(); + + const answered = JSON.parse( + await ask( + stub, + { release: POLICY.release, token: bearer, runId: RUN_ID }, + JSON.stringify({ + operation: "fork-source", + checkpointEventId: "event-work", + // An anchor from a selection this owner never produced. + section: "roots", + anchor: "c".repeat(64), + after: null, + }), + ), + ); + expect(answered["outcome"]).toBe("refused"); + // And nothing was written on the way to refusing. + expect((await on(stub, (owner) => owner.published()))["events"]).toHaveLength(3); + }); + + it("changes the anchor when copied metadata moves and content does not", async () => { + // A watermark is copied into destination storage but is not implied by any + // content identity, so a digest cannot stand for it. + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + const bearer = await token(); + const anchorNow = async (): Promise => { + const answered = JSON.parse( + await ask( + stub, + { release: POLICY.release, token: bearer, runId: RUN_ID }, + JSON.stringify({ + operation: "fork-source", + checkpointEventId: "event-work", + section: "blobs", + anchor: null, + after: null, + }), + ), + ); + expect(answered["outcome"]).toBe("performed"); + return answered["value"]["anchor"]; + }; + + const before = await anchorNow(); + expect(typeof before).toBe("string"); + await on(stub, (owner) => owner.touchBlobWatermark()); + // Identical content, different metadata: a different selection. + expect(await anchorNow()).not.toBe(before); + }); + + it("refuses a retained value of the wrong type rather than converting it", async () => { + const stub = executor(); + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + await on(stub, (owner) => owner.initialize()); + await on(stub, (owner) => owner.appendForkableHistory()); + await on(stub, (owner) => owner.damageRetainedWatermark()); + const bearer = await token(); + + const answered = JSON.parse( + await ask( + stub, + { release: POLICY.release, token: bearer, runId: RUN_ID }, + JSON.stringify({ + operation: "fork-source", + checkpointEventId: "event-work", + section: "blobs", + anchor: null, + after: null, + }), + ), + ); + // Damage, not a zero. A reader that coerced would have answered with one. + expect(answered["outcome"]).toBe("refused"); + expect(String(answered["refusal"])).toContain("storage:corrupt"); + // And nothing of the retained value crossed with it. + expect(JSON.stringify(answered)).not.toContain("not a number"); + }); +}); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index c0ef42d3c..8798f0c74 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -8,6 +8,7 @@ */ import { run } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; import { serializeDurableEvent } from "@executablemd/durable-streams"; import { acquisitionHolders } from "../../../src/cloudflare/acquisition.ts"; import { WorkflowOwnerObject } from "../../../src/cloudflare/owner.ts"; @@ -66,6 +67,10 @@ export const ROOT_MANIFEST = JSON.stringify({ manifest: MANIFEST_ID, hardlink: null, }, + // Two directories a checkout can live in, so a retained Repository and a + // retained Worktree can qualify without sharing one. + { path: "/checkouts", kind: "directory", mode: 493, mtime: 0 }, + { path: "/work", kind: "directory", mode: 493, mtime: 0 }, ], }); export const ROOT_ID = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${ROOT_MANIFEST}`); @@ -419,6 +424,90 @@ export class ExecutorObject extends WorkflowOwnerObject { return row === undefined ? null : row; } + /** + * Retain one Repository whose checkout the starting Workspace holds. + * + * `/` is a directory in `ROOT_MANIFEST`, so this one qualifies for a fork + * source selected at that root — which is what makes it usable for proving + * that adding a qualifying mapping changes the selection's anchor. + */ + retainQualifyingRepository(name: string): void { + this.retainRepositoryAt(name, "/"); + } + + /** Retain one Repository checked out in a directory this test names. */ + retainRepositoryAt(name: string, checkoutPath: string): void { + this.ctx.storage.sql.exec( + `INSERT INTO workspace_repositories (name, locator, locator_fingerprint, requested_base, + creation_commit, primary_branch, object_format, checkout_path) + VALUES (?, ?, ?, NULL, ?, 'main', 'sha1', ?)`, + name, + "https://git.example.invalid/one.git", + "b".repeat(64), + "9".repeat(40), + checkoutPath, + ); + } + + /** + * Retain one Worktree of a retained Repository, in its own directory. + * + * `/work` is a directory in `ROOT_MANIFEST` and no Repository holds it, so + * the pair is a checkout graph a fork could restore as it stands. + */ + retainQualifyingWorktree(repositoryName: string, name: string): void { + this.retainWorktreeAt(repositoryName, name, "/work"); + } + + /** Retain one Worktree checked out in a directory this test names. */ + retainWorktreeAt(repositoryName: string, name: string, checkoutPath: string): void { + this.ctx.storage.sql.exec( + `INSERT INTO workspace_worktrees (repository_name, name, requested_branch, requested_base, + creation_commit, checkout_path) + VALUES (?, ?, 'topic', NULL, ?, ?)`, + repositoryName, + name, + "9".repeat(40), + checkoutPath, + ); + } + + /** Append enough journal events that one section cannot be read in one page. */ + fillJournal(count: number): void { + for (let index = 0; index < count; index += 1) { + this.appendJournal(`event-${String(index).padStart(4, "0")}`, `step ${index}`); + } + } + + /** Every retained journal record, in the order the journal holds them. */ + journalRecords(): { eventId: string; record: string }[] { + return this.ctx.storage.sql + .exec("SELECT event_id, record FROM journal_events ORDER BY sequence") + .toArray() + .map((row) => ({ + eventId: String(row["event_id"]), + record: String(row["record"]), + })); + } + + /** Move one retained blob's watermark, leaving its content identical. */ + touchBlobWatermark(): void { + this.ctx.storage.sql.exec("UPDATE vfs_blobs SET last_seen = last_seen + 1"); + } + + /** + * Give one retained row a value of the wrong SQLite type. + * + * `STRICT` tables refuse most of these, so the damaged column is one the + * schema declares loosely enough to hold it — which is exactly the case a + * reader that coerced would turn into a plausible value. + */ + damageRetainedWatermark(): void { + // INTEGER affinity converts what it can; text that names no number stays + // text, which is the value a reader that coerced would turn into zero. + this.ctx.storage.sql.exec("UPDATE vfs_blobs SET last_seen = 'not a number'"); + } + /** Retain more Repository rows than one admitted snapshot may carry. */ fillRepositories(from: number, count: number): void { for (let index = from; index < from + count; index += 1) { @@ -596,6 +685,45 @@ export class ExecutorObject extends WorkflowOwnerObject { } } + /** + * Answer one ordinary read, the way a host's request route would. + * + * Deliberately not the WebSocket path: no socket, no upgrade, no + * acquisition. What a test proves through this is that reading takes nothing. + */ + async readRequest( + admission: { release: string | null; token: string | null; runId: string | null }, + body: string, + ): Promise { + return JSON.stringify(await run(() => this.read(admission, body))); + } + + /** + * Append the two rows a fork writes for itself, and one ordinary row. + * + * The run record and the root import are what a destination replaces, so a + * fork-source selection must exclude exactly them and keep the rest. + */ + appendForkableHistory(): void { + const write = (eventId: string, type: string, name: string) => { + const record: DurableEvent = { + type: "yield", + coroutineId: "root", + description: { type, name }, + result: { status: "ok", value: name }, + }; + this.ctx.storage.sql.exec( + "INSERT INTO journal_events (event_id, record, workspace_root_id) VALUES (?, ?, ?)", + eventId, + serializeDurableEvent(record), + ROOT_ID, + ); + }; + write("event-run", "workflow_run", "workflow_run"); + write("event-import", "import_component", "__root__"); + this.appendJournal("event-work", "work"); + } + /** The correlation the live acquisition is partitioned by. */ acquisitionId(): string { const held = acquisitionHolders(this.ctx)[0]; diff --git a/packages/workflow/tests/remote-inspection.test.ts b/packages/workflow/tests/remote-inspection.test.ts new file mode 100644 index 000000000..43bccb1c8 --- /dev/null +++ b/packages/workflow/tests/remote-inspection.test.ts @@ -0,0 +1,1183 @@ +/** + * Tier WRH — what a remote inspection projects to, and what it refuses. + * + * The owner-side facts — that reading takes no acquisition, and that one + * answer comes from one committed reading — are proved against a real Durable + * Object in `tests/cloudflare/remote-read-plane.vitest.ts`. These are the other + * half: that the provider-neutral values a caller receives are the shared + * projection, and that an anchored sequence which does not hold together + * publishes nothing at all. + */ + +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { type Operation, scoped } from "effection"; +import { + cloudflareReadPlane, + FORK_SOURCE_ANSWER_BYTES, + PUBLIC_ANSWER_BYTES, + type ReadTransport, +} from "../src/cloudflare/read-client.ts"; +import type { RemoteReadPlane } from "../src/remote/read.ts"; +import { sha256Hex } from "../src/workspace/sha256.ts"; +import { compareUtf8, WORKSPACE_ROOT_DOMAIN } from "../src/workspace/root-manifest.ts"; +import { useRemoteLifecycleReads } from "../src/remote/inspection.ts"; +import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; +import { WorkflowRecordMalformedError, WorkflowRunIdMismatchError } from "../src/storage/errors.ts"; + +const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +const ROOT = "a".repeat(64); + +function runRecord(): Record { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-05T00:00:00.000Z", + updatedAt: "2026-09-05T00:00:00.000Z", + }; +} + +function inspection(overrides: Record = {}): Record { + return { + record: runRecord(), + executions: [], + retrieval: null, + journalFrontier: null, + currentWorkspaceRootId: ROOT, + lineage: null, + ...overrides, + }; +} + +function event(name: string): string { + return JSON.stringify({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }); +} + +/** An owner whose answers a test writes, recording what it was asked. */ +function plane(answer: (read: Record) => Record) { + const asked: Record[] = []; + const transport: ReadTransport = { + // deno-lint-ignore require-yield + *send(_admission, body: string): Operation { + const value: unknown = JSON.parse(body); + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new Error("expected one read operation"); + } + const operation = Object.fromEntries(Object.entries(value)); + asked.push(operation); + return JSON.stringify(answer(operation)); + }, + }; + return { asked, transport }; +} + +/** The bytes as the private protocol carries them. */ +function base64(bytes: Uint8Array): string { + let text = ""; + for (const byte of bytes) { + text += String.fromCharCode(byte); + } + return btoa(text); +} + +/** + * Where a real owner says a page ended. + * + * A cursor is the position of the last member the page carried, inside the + * anchored selection it belongs to — not a name, and not a value beside the + * rows, so a scripted owner has to derive it the way a real one does. + */ +function cursorOf(from: number, rows: readonly unknown[], after: number | null): number | null { + return rows.length === 0 ? after : from + rows.length - 1; +} + +/** Install the read operations over one scripted owner, for one body. */ +function* installed( + answer: (read: Record) => Record, + body: (opened: RemoteReadPlane) => Operation, +): Operation { + return yield* scoped(function* () { + const held = plane(answer); + const opened = cloudflareReadPlane( + held.transport, + "release-1", + // deno-lint-ignore require-yield + function* () { + return "token"; + }, + RUN_ID, + ); + yield* useRemoteLifecycleReads(opened); + // The plane itself is handed to the body: a fork source is private, and no + // public lifecycle operation returns one. + return yield* body(opened); + }); +} + +describe("a remote run's inspection", () => { + it("refuses a request for another run before it reaches the owner", function* () { + const other = "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + const held = plane(() => ({ outcome: "performed", value: inspection() })); + const seen = yield* scoped(function* () { + yield* useRemoteLifecycleReads( + // deno-lint-ignore require-yield + cloudflareReadPlane( + held.transport, + "release-1", + function* () { + return "token"; + }, + RUN_ID, + ), + ); + return { + inspected: yield* WorkflowLifecycle.operations.inspect(other), + history: yield* WorkflowLifecycle.operations.history(other), + }; + }); + + expect([seen.inspected.ok, seen.history.ok]).toEqual([false, false]); + // Refused here, so the owner was never asked and nothing of the bound run + // came back. + expect(held.asked).toEqual([]); + for (const outcome of [seen.inspected, seen.history]) { + expect(outcome.ok === false && outcome.error).toEqual(expect.any(WorkflowRunIdMismatchError)); + expect(String(outcome.ok === false && outcome.error)).not.toContain(RUN_ID); + } + }); + + it("lists nothing for an owner holding no run, and fails on a damaged one", function* () { + const empty = yield* installed( + () => ({ outcome: "refused", refusal: "command:absent" }), + () => WorkflowLifecycle.operations.list(), + ); + // Pristine storage lists nothing rather than failing: there is no run, and + // that is a complete answer. + expect(empty.ok && empty.value).toEqual([]); + + const damaged = yield* installed( + () => ({ outcome: "refused", refusal: "storage:corrupt" }), + () => WorkflowLifecycle.operations.list(), + ); + // Anything else fails whole rather than reporting a healthy subset. + expect(damaged.ok).toBe(false); + expect(String(damaged.ok === false && damaged.error)).not.toContain("storage:"); + }); + + it("answers one frozen snapshot, and a bound owner's list of one", function* () { + const seen = yield* installed( + () => ({ outcome: "performed", value: inspection() }), + function* () { + return { + inspected: yield* WorkflowLifecycle.operations.inspect(RUN_ID), + listed: yield* WorkflowLifecycle.operations.list(), + }; + }, + ); + + expect(seen.inspected.ok).toBe(true); + if (seen.inspected.ok) { + expect(seen.inspected.value.record.runId).toBe(RUN_ID); + expect(seen.inspected.value.currentWorkspaceRootId).toBe(ROOT); + // Nothing callable, and nothing a caller can change. + expect(Object.isFrozen(seen.inspected.value)).toBe(true); + expect(seen.inspected.value.journalFrontier).toBe(undefined); + expect(seen.inspected.value.lineage).toBe(undefined); + } + // The plane is bound to one owner, so its whole visible domain is that + // owner: one coherent snapshot, and no enumeration of anything else. + expect(seen.listed.ok && seen.listed.value).toHaveLength(1); + }); + + it("projects history through the shared projection, across anchored pages", function* () { + const history = yield* installed( + (read) => { + if (read["operation"] !== "history") { + return { outcome: "performed", value: inspection() }; + } + // Two pages, anchored to the terminal event the first one chose. + if (read["after"] === null) { + return { + outcome: "performed", + value: { + anchor: "event-2", + after: null, + rows: [{ eventId: "event-1", record: event("first"), workspaceRootId: ROOT }], + done: false, + retainedRoots: [], + provenance: [], + }, + }; + } + return { + outcome: "performed", + value: { + anchor: "event-2", + after: "event-1", + rows: [{ eventId: "event-2", record: event("second"), workspaceRootId: ROOT }], + done: true, + retainedRoots: [ROOT], + provenance: [ + { eventId: "event-1", sourceRunId: "somewhere", sourceEventId: "event-9" }, + ], + }, + }; + }, + () => WorkflowLifecycle.operations.history(RUN_ID), + ); + + expect(history.ok).toBe(true); + if (history.ok) { + expect(history.value.map((entry) => entry.eventId)).toEqual(["event-1", "event-2"]); + // The shared projection's own members, not a second interpretation. + expect(history.value[0]?.forkability).not.toBe(undefined); + expect(history.value[0]?.inherited).toEqual({ + sourceRunId: "somewhere", + sourceEventId: "event-9", + }); + expect(history.value[1]?.inherited).toBe(undefined); + // No retained record bytes reach the public answer. + expect(JSON.stringify(history.value)).not.toContain('\\"type\\":\\"yield'); + } + }); + + it("publishes nothing when an anchored sequence does not hold together", function* () { + const first = { + anchor: "event-2", + after: null, + rows: [{ eventId: "event-1", record: event("first"), workspaceRootId: ROOT }], + done: false, + retainedRoots: [], + provenance: [], + }; + // Each of these is a second page that belongs to some other snapshot. + const broken: Record> = { + "a changed anchor": { ...first, anchor: "event-9", after: "event-1" }, + "a cursor it was not asked to continue": { ...first, after: "event-7" }, + "a repeated event": { ...first, after: "event-1" }, + "a page that terminates short of its anchor": { + ...first, + after: "event-1", + rows: [{ eventId: "event-3", record: event("third"), workspaceRootId: ROOT }], + done: true, + }, + "an empty page of a non-empty snapshot": { ...first, after: "event-1", rows: [] }, + "a member this build does not declare": { ...first, after: "event-1", extra: true }, + }; + + for (const [description, second] of Object.entries(broken)) { + const outcome = yield* installed( + (read) => ({ + outcome: "performed", + value: read["after"] === null ? first : second, + }), + () => WorkflowLifecycle.operations.history(RUN_ID), + ); + expect([description, outcome.ok]).toEqual([description, false]); + if (!outcome.ok) { + expect([description, outcome.error]).toEqual([ + description, + expect.any(WorkflowRecordMalformedError), + ]); + // Nothing of the page, and nothing of the protocol. + expect(String(outcome.error)).not.toContain("event-"); + } + } + }); + + it("refuses an inspection that describes another run", function* () { + const outcome = yield* installed( + () => ({ + outcome: "performed", + value: inspection({ + record: { ...runRecord(), runId: "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa" }, + }), + }), + () => WorkflowLifecycle.operations.inspect(RUN_ID), + ); + expect(outcome.ok).toBe(false); + expect(outcome.ok === false && outcome.error).toEqual(expect.any(WorkflowRecordMalformedError)); + expect(String(outcome.ok === false && outcome.error)).not.toContain("6dktgrv"); + }); + + it("hands back a refusal as a storage failure, with nothing private in it", function* () { + const outcome = yield* installed( + () => ({ outcome: "refused", refusal: "command:absent" }), + () => WorkflowLifecycle.operations.inspect(RUN_ID), + ); + expect(outcome.ok).toBe(false); + expect(String(outcome.ok === false && outcome.error)).not.toContain("command:"); + }); + + it("refuses a fork source whose transported closure does not hold", function* () { + // One representative of each distinct structural failure, driven through + // the fork-source client itself rather than inferred from the history + // pager. A destination built from any of these could not restore the + // Workspace it was given. + const BLOB = new TextEncoder().encode("file bytes"); + const blobHash = sha256Hex(BLOB); + const contentManifest = JSON.stringify({ + version: 1, + chunks: [{ hash: blobHash, size: BLOB.length }], + }); + const encoded = new TextEncoder().encode(contentManifest); + const manifestHash = sha256Hex(encoded); + const rootManifest = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/file.txt", + kind: "file", + mode: 420, + mtime: 0, + size: BLOB.length, + manifest: manifestHash, + hardlink: null, + }, + ], + }); + const rootId = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${rootManifest}`); + + const sound = { + inherited: [ + { eventId: "event-work", record: event("work"), workspaceRootId: rootId, position: 0 }, + ], + roots: [ + { + rootId, + formatVersion: 1, + manifest: rootManifest, + manifestHashes: [manifestHash], + blobHashes: [blobHash], + }, + ], + manifests: [{ hash: manifestHash, size: BLOB.length, lastSeen: 0, encoded: base64(encoded) }], + blobs: [{ hash: blobHash, size: BLOB.length, lastSeen: 0, content: base64(BLOB) }], + checkouts: [], + }; + + const damaged: Record typeof sound> = { + "a manifest that is not its own digest": (held) => ({ + ...held, + manifests: [{ ...held.manifests[0], hash: "d".repeat(64) }], + }), + "a manifest describing another size": (held) => ({ + ...held, + manifests: [{ ...held.manifests[0], size: 999 }], + }), + "a root whose references do not follow from its manifest": (held) => ({ + ...held, + roots: [{ ...held.roots[0], manifestHashes: [] }], + }), + "a blob that is not its own digest": (held) => ({ + ...held, + blobs: [{ ...held.blobs[0], hash: "e".repeat(64) }], + }), + "a blob disagreeing with the chunk that names it": (held) => ({ + ...held, + blobs: [{ ...held.blobs[0], size: 999 }], + }), + // Two wrong sizes that agree with each other: the blob row and the + // chunk both say eleven while the bytes are ten. Every one-dimensional + // comparison passes, so only the byte length catches it. + "coordinated sizes that disagree with the bytes": (held) => { + const wide = JSON.stringify({ + version: 1, + chunks: [{ hash: blobHash, size: BLOB.length + 1 }], + }); + const wideBytes = new TextEncoder().encode(wide); + const wideHash = sha256Hex(wideBytes); + const wideRoot = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/file.txt", + kind: "file", + mode: 420, + mtime: 0, + size: BLOB.length + 1, + manifest: wideHash, + hardlink: null, + }, + ], + }); + const wideRootId = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${wideRoot}`); + return { + inherited: [ + { + eventId: "event-work", + record: event("work"), + workspaceRootId: wideRootId, + position: 0, + }, + ], + roots: [ + { + rootId: wideRootId, + formatVersion: 1, + manifest: wideRoot, + manifestHashes: [wideHash], + blobHashes: [blobHash], + }, + ], + manifests: [ + { hash: wideHash, size: BLOB.length + 1, lastSeen: 0, encoded: base64(wideBytes) }, + ], + blobs: [{ hash: blobHash, size: BLOB.length + 1, lastSeen: 0, content: base64(BLOB) }], + checkouts: [], + }; + }, + // The root claims a file length the content it names does not produce. + "a root file size the content does not produce": (held) => { + const wrongRoot = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/file.txt", + kind: "file", + mode: 420, + mtime: 0, + size: BLOB.length + 5, + manifest: manifestHash, + hardlink: null, + }, + ], + }); + const wrongRootId = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${wrongRoot}`); + return { + ...held, + inherited: [ + { + eventId: "event-work", + record: event("work"), + workspaceRootId: wrongRootId, + position: 0, + }, + ], + roots: [{ ...held.roots[0], rootId: wrongRootId, manifest: wrongRoot }], + }; + }, + // A manifest that is sound in itself — its own digest, its own size — + // and simply nothing the selection asked for. + "a manifest nothing selected requires": (held) => { + const spare = new TextEncoder().encode( + JSON.stringify({ + version: 1, + chunks: [ + { hash: blobHash, size: BLOB.length }, + { hash: blobHash, size: BLOB.length }, + ], + }), + ); + return { + ...held, + // In the section's own order, so what is refused is the extra + // manifest and not the sequence that carried it. + manifests: [ + ...held.manifests, + { + hash: sha256Hex(spare), + size: BLOB.length * 2, + lastSeen: 0, + encoded: base64(spare), + }, + ].toSorted((left, right) => (left.hash < right.hash ? -1 : 1)), + }; + }, + "a row naming a root the selection did not carry": (held) => ({ + ...held, + inherited: [{ ...held.inherited[0], workspaceRootId: "a".repeat(64) }], + }), + }; + + for (const [description, damage] of Object.entries(damaged)) { + const held = damage(sound); + const outcome = yield* installed( + (read) => { + const section = String(read["section"]); + const found = Reflect.get(held, section); + const rows = Array.isArray(found) ? found : []; + const rootOf = held.roots[0]; + return { + outcome: "performed", + value: { + anchor: "anchor-1", + after: null, + section, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: rootOf?.rootId ?? rootId, + runRecordWorkspaceRootId: rootOf?.rootId ?? rootId, + rootImportWorkspaceRootId: rootOf?.rootId ?? rootId, + rows, + from: 0, + cursor: cursorOf(0, rows, null), + done: true, + total: rows.length, + }, + }; + }, + (opened) => opened.forkSource("event-work"), + ); + expect([description, outcome.ok]).toEqual([description, false]); + if (!outcome.ok) { + // Nothing of the bytes, the hashes or the protocol. + expect(String(outcome.error)).not.toContain(blobHash); + expect(String(outcome.error)).not.toContain("command:"); + } + } + }); + + it("accepts a fork source whose closure holds, and refuses a moved anchor", function* () { + const BLOB = new TextEncoder().encode("file bytes"); + const blobHash = sha256Hex(BLOB); + const encoded = new TextEncoder().encode( + JSON.stringify({ version: 1, chunks: [{ hash: blobHash, size: BLOB.length }] }), + ); + const manifestHash = sha256Hex(encoded); + const rootManifest = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/file.txt", + kind: "file", + mode: 420, + mtime: 0, + size: BLOB.length, + manifest: manifestHash, + hardlink: null, + }, + ], + }); + const rootId = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${rootManifest}`); + // One record spelled the way its source retained it, which is not the + // spelling re-encoding the parsed event would produce. + const spelled = JSON.stringify(JSON.parse(event("work")), null, 2); + const sections: Record = { + inherited: [{ eventId: "event-work", record: spelled, workspaceRootId: rootId, position: 0 }], + roots: [ + { + rootId, + formatVersion: 1, + manifest: rootManifest, + manifestHashes: [manifestHash], + blobHashes: [blobHash], + }, + ], + manifests: [{ hash: manifestHash, size: BLOB.length, lastSeen: 0, encoded: base64(encoded) }], + blobs: [{ hash: blobHash, size: BLOB.length, lastSeen: 0, content: base64(BLOB) }], + checkouts: [], + }; + const answer = (anchorFor: (section: string) => string) => (read: Record) => { + const section = String(read["section"]); + const rows = sections[section] ?? []; + return { + outcome: "performed", + value: { + anchor: anchorFor(section), + after: null, + section, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: rootId, + runRecordWorkspaceRootId: rootId, + rootImportWorkspaceRootId: rootId, + rows, + from: 0, + cursor: cursorOf(0, rows, null), + done: true, + total: rows.length, + }, + }; + }; + + const whole = yield* installed( + answer(() => "anchor-1"), + (opened) => opened.forkSource("event-work"), + ); + expect(whole.ok).toBe(true); + if (whole.ok) { + expect(whole.value.inherited.map((row) => row.eventId)).toEqual(["event-work"]); + // Byte for byte: a destination retains these bytes, and a spelling + // rebuilt from the parse would be a history it never inherited. + expect(whole.value.inherited[0]?.record).toBe(spelled); + expect(spelled).not.toBe(event("work")); + expect(whole.value.roots.map((root) => root.rootId)).toEqual([rootId]); + expect(whole.value.blobs[0]?.content).toEqual(BLOB); + } + + // The checkouts section arrives from a selection that has moved on. + const moved = yield* installed( + answer((section) => (section === "checkouts" ? "anchor-2" : "anchor-1")), + (opened) => opened.forkSource("event-work"), + ); + expect(moved.ok).toBe(false); + expect(moved.ok === false && moved.error).toEqual(expect.any(WorkflowRecordMalformedError)); + }); + + it("refuses a fork-source sequence that does not describe one selection", function* () { + // A selection is one answer carried over several pages. These are the ways + // a sequence can stop being that answer: a page that describes a different + // selection, a page that advances past what it carried, and a graph of + // checkouts a destination could not retain. + const BLOB = new TextEncoder().encode("file bytes"); + const blobHash = sha256Hex(BLOB); + const encoded = new TextEncoder().encode( + JSON.stringify({ version: 1, chunks: [{ hash: blobHash, size: BLOB.length }] }), + ); + const manifestHash = sha256Hex(encoded); + const directory = (path: string) => ({ path, kind: "directory", mode: 493, mtime: 0 }); + const rootManifest = JSON.stringify({ + format: 1, + // Canonical order: the root, then every path by its UTF-8 bytes. Seven + // directories, because a checkout identity that collides under a + // separator still needs a place of its own. + entries: [ + directory("/"), + { + path: "/file.txt", + kind: "file", + mode: 420, + mtime: 0, + size: BLOB.length, + manifest: manifestHash, + hardlink: null, + }, + ...["/five", "/four", "/one", "/seven", "/six", "/three", "/two"].map((path) => + directory(path), + ), + ], + }); + const rootId = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${rootManifest}`); + const repository = (name: string, checkoutPath: string) => ({ + kind: "repository", + name, + locator: `https://git.example.invalid/${name}.git`, + locatorFingerprint: "b".repeat(64), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath, + }); + const worktree = (repositoryName: string, name: string, checkoutPath: string) => ({ + kind: "worktree", + repositoryName, + name, + requestedBranch: "topic", + requestedBase: null, + creationCommit: "9".repeat(40), + checkoutPath, + }); + const alpha = repository("alpha", "/one"); + const beta = repository("beta", "/two"); + const ALPHA_KEY = JSON.stringify(["repository", "alpha"]); + const BETA_KEY = JSON.stringify(["repository", "beta"]); + // A prefix of three, so a permutation of two of them is a thing a page can + // carry and a sequence can be asked to accept. + const journal = ["a", "b", "c"].map((name, position) => ({ + eventId: `event-${name}`, + record: event(name), + workspaceRootId: rootId, + position, + })); + + interface Page { + readonly after: number | null; + readonly from: number; + readonly rows: readonly unknown[]; + readonly cursor: number | null; + readonly done: boolean; + readonly total: number; + } + interface Plan { + readonly heads: { checkpoint: string; runRecord: string; rootImport: string }; + readonly sections: Record; + } + + /** One page carrying a whole section, as an owner with little to say sends. */ + const whole = (section: string, rows: readonly unknown[]): Page[] => [ + { + after: null, + from: 0, + rows, + cursor: cursorOf(0, rows, null), + done: true, + total: rows.length, + }, + ]; + const sound: Plan = { + heads: { checkpoint: rootId, runRecord: rootId, rootImport: rootId }, + sections: { + inherited: whole("inherited", journal), + roots: whole("roots", [ + { + rootId, + formatVersion: 1, + manifest: rootManifest, + manifestHashes: [manifestHash], + blobHashes: [blobHash], + }, + ]), + manifests: whole("manifests", [ + { hash: manifestHash, size: BLOB.length, lastSeen: 0, encoded: base64(encoded) }, + ]), + blobs: whole("blobs", [ + { hash: blobHash, size: BLOB.length, lastSeen: 0, content: base64(BLOB) }, + ]), + checkouts: whole("checkouts", [alpha, beta]), + }, + }; + const checkouts = (pages: readonly Page[]): Plan => ({ + ...sound, + sections: { ...sound.sections, checkouts: pages }, + }); + const inherited = (pages: readonly Page[]): Plan => ({ + ...sound, + sections: { ...sound.sections, inherited: pages }, + }); + + const broken: Record = { + "a head naming a Workspace root the selection did not carry": { + ...sound, + heads: { ...sound.heads, runRecord: "c".repeat(64) }, + }, + "a head that is no Workspace root identity at all": { + ...sound, + heads: { ...sound.heads, rootImport: "the root import" }, + }, + "a checkout in a directory the checkpoint Workspace does not hold": checkouts( + whole("checkouts", [repository("alpha", "/nowhere")]), + ), + "a Worktree of a Repository the selection did not carry": checkouts( + whole("checkouts", [alpha, worktree("gamma", "topic", "/two")]), + ), + "two checkouts in one directory": checkouts( + whole("checkouts", [alpha, repository("beta", "/one")]), + ), + "a page that redeclares the size of its section": checkouts([ + { after: null, from: 0, rows: [alpha], cursor: 0, done: false, total: 2 }, + { + after: 0, + from: 1, + rows: [beta], + cursor: 1, + done: true, + total: 3, + }, + ]), + "a cursor naming a member the page did not carry": checkouts([ + { after: null, from: 0, rows: [alpha], cursor: 1, done: false, total: 2 }, + { + after: 1, + from: 1, + rows: [beta], + cursor: 1, + done: true, + total: 2, + }, + ]), + "a page repeating what an earlier page carried": checkouts([ + { after: null, from: 0, rows: [alpha], cursor: 0, done: false, total: 2 }, + { + after: 0, + from: 1, + rows: [alpha], + cursor: 0, + done: true, + total: 2, + }, + ]), + "a section arriving out of the order it is sorted in": checkouts([ + { after: null, from: 0, rows: [beta], cursor: 0, done: false, total: 2 }, + { after: 0, from: 1, rows: [alpha], cursor: 1, done: true, total: 2 }, + ]), + "a section ending short of what it declared": checkouts([ + { + after: null, + from: 0, + rows: [alpha, beta], + cursor: 1, + done: true, + total: 3, + }, + ]), + "an unfinished page carrying nothing to continue from": checkouts([ + { after: null, from: 0, rows: [], cursor: null, done: false, total: 2 }, + ]), + "a page beginning past where the sequence had reached": checkouts([ + { after: null, from: 0, rows: [alpha], cursor: 0, done: false, total: 3 }, + { + after: 0, + from: 2, + rows: [beta], + cursor: 1, + done: true, + total: 3, + }, + ]), + "one Repository selected twice in one page": checkouts([ + { + after: null, + from: 0, + rows: [alpha, alpha], + cursor: 0, + done: true, + total: 2, + }, + ]), + "a locator fingerprint that is no digest": checkouts( + whole("checkouts", [{ ...alpha, locatorFingerprint: "the fingerprint" }]), + ), + "an object format this build does not write": checkouts( + whole("checkouts", [{ ...alpha, objectFormat: "sha3" }]), + ), + "a checkout path that is no Workspace path": checkouts( + whole("checkouts", [{ ...alpha, checkoutPath: "one" }]), + ), + // Two rows exchanged inside one page. Their event ids are still unique, + // the page still begins where the sequence reached, and the cursor still + // names the row the page ended on — only the positions travelling with + // the rows say the journal never held them this way. + "two inherited rows exchanged inside one page": inherited( + whole("inherited", [journal[1], journal[0], journal[2]]), + ), + "two inherited rows exchanged across a page boundary": inherited([ + { + after: null, + from: 0, + rows: [journal[0], journal[2]], + cursor: 1, + done: false, + total: 3, + }, + { after: 1, from: 2, rows: [journal[1]], cursor: 2, done: true, total: 3 }, + ]), + "a root import naming a Workspace root the selection did not carry": { + ...sound, + heads: { ...sound.heads, rootImport: "d".repeat(64) }, + }, + }; + + const answering = (plan: Plan) => (read: Record) => { + const section = String(read["section"]); + const pages = plan.sections[section] ?? []; + const asked = read["after"]; + const after = typeof asked === "number" ? asked : null; + const page = pages.find((candidate) => candidate.after === after); + if (page === undefined) { + // A page nothing scripted: the client asked to continue from somewhere + // this owner never sent it. + return { outcome: "refused", refusal: "command:absent" }; + } + return { + outcome: "performed", + value: { + anchor: "anchor-1", + after, + section, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: plan.heads.checkpoint, + runRecordWorkspaceRootId: plan.heads.runRecord, + rootImportWorkspaceRootId: plan.heads.rootImport, + rows: page.rows, + from: page.from, + cursor: page.cursor, + done: page.done, + total: page.total, + }, + }; + }; + + // The same prefix over two pages, with positions that continue across the + // boundary: accepted, in the source's order, with the records untouched. + const paged = yield* installed( + answering( + inherited([ + { + after: null, + from: 0, + rows: [journal[0], journal[1]], + cursor: 1, + done: false, + total: 3, + }, + { after: 1, from: 2, rows: [journal[2]], cursor: 2, done: true, total: 3 }, + ]), + ), + (opened) => opened.forkSource("event-work"), + ); + expect([paged.ok, paged.ok === false && String(paged.error)]).toEqual([true, false]); + if (paged.ok) { + expect(paged.value.inherited.map((row) => row.eventId)).toEqual([ + "event-a", + "event-b", + "event-c", + ]); + expect(paged.value.inherited.map((row) => row.record)).toEqual( + journal.map((row) => row.record), + ); + } + + // Names the retained schema accepts and a separator does not survive. + // `("a:b", "c")` and `("a", "b:c")` join to one string under a colon; + // `("a/b", "c")` and `("a", "b/c")` join to one string under a slash. All + // four are distinct Worktrees of Repositories that came with them. + const colliding: Plan = checkouts( + whole("checkouts", [ + repository("a", "/three"), + repository("a/b", "/five"), + repository("a:b", "/one"), + worktree("a", "b/c", "/seven"), + worktree("a", "b:c", "/four"), + worktree("a/b", "c", "/six"), + worktree("a:b", "c", "/two"), + ]), + ); + const distinct = yield* installed(answering(colliding), (opened) => + opened.forkSource("event-work"), + ); + expect([distinct.ok, distinct.ok === false && String(distinct.error)]).toEqual([true, false]); + if (distinct.ok) { + expect( + distinct.value.checkouts + .filter((one) => one.kind === "worktree") + .map((one) => [one.repositoryName, one.name]), + ).toEqual([ + ["a", "b/c"], + ["a", "b:c"], + ["a/b", "c"], + ["a:b", "c"], + ]); + } + + // The same scripting, undamaged, is accepted: every refusal below is the + // damage and not the shape of the script. + const held = yield* installed(answering(sound), (opened) => opened.forkSource("event-work")); + expect(held.ok).toBe(true); + if (held.ok) { + expect(held.value.checkouts.map((checkout) => checkout.name)).toEqual(["alpha", "beta"]); + } + + for (const [description, plan] of Object.entries(broken)) { + const outcome = yield* installed(answering(plan), (opened) => + opened.forkSource("event-work"), + ); + expect([description, outcome.ok]).toEqual([description, false]); + if (!outcome.ok) { + expect(String(outcome.error)).not.toContain("repository:"); + expect(String(outcome.error)).not.toContain("command:"); + } + } + }); + + it("holds a root to the canonical reference arrays a destination will derive", function* () { + // Two files, so a root's references are an array with an order rather than + // a single value. A destination compares its own derivation element for + // element when it retains the root, so the set being right is not enough. + const files = ["first bytes", "second bytes"].map((text) => { + const bytes = new TextEncoder().encode(text); + const blobHash = sha256Hex(bytes); + const encoded = new TextEncoder().encode( + JSON.stringify({ version: 1, chunks: [{ hash: blobHash, size: bytes.length }] }), + ); + return { bytes, blobHash, encoded, manifestHash: sha256Hex(encoded) }; + }); + const rootManifest = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + ...files.map((file, index) => ({ + path: `/file-${index}.txt`, + kind: "file", + mode: 420, + mtime: 0, + size: file.bytes.length, + manifest: file.manifestHash, + hardlink: null, + })), + ], + }); + const rootId = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${rootManifest}`); + const manifestHashes = files.map((file) => file.manifestHash).toSorted(compareUtf8); + const blobHashes = files.map((file) => file.blobHash).toSorted(compareUtf8); + const sections = (root: Record): Record => ({ + inherited: [ + { eventId: "event-work", record: event("work"), workspaceRootId: rootId, position: 0 }, + ], + roots: [root], + manifests: files + .map((file) => ({ + hash: file.manifestHash, + size: file.bytes.length, + lastSeen: 0, + encoded: base64(file.encoded), + })) + .toSorted((left, right) => compareUtf8(left.hash, right.hash)), + blobs: files + .map((file) => ({ + hash: file.blobHash, + size: file.bytes.length, + lastSeen: 0, + content: base64(file.bytes), + })) + .toSorted((left, right) => compareUtf8(left.hash, right.hash)), + checkouts: [], + }); + const answering = (root: Record) => (read: Record) => { + const section = String(read["section"]); + const rows = sections(root)[section] ?? []; + return { + outcome: "performed", + value: { + anchor: "anchor-1", + after: null, + section, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: rootId, + runRecordWorkspaceRootId: rootId, + rootImportWorkspaceRootId: rootId, + rows, + from: 0, + cursor: cursorOf(0, rows, null), + done: true, + total: rows.length, + }, + }; + }; + const canonical = { + rootId, + formatVersion: 1, + manifest: rootManifest, + manifestHashes, + blobHashes, + }; + + const held = yield* installed(answering(canonical), (opened) => + opened.forkSource("event-work"), + ); + expect([held.ok, held.ok === false && String(held.error)]).toEqual([true, false]); + if (held.ok) { + expect(held.value.roots[0]?.manifestHashes).toEqual(manifestHashes); + expect(held.value.roots[0]?.blobHashes).toEqual(blobHashes); + } + + const damaged: Record> = { + "content references in an order a root is never retained with": { + ...canonical, + manifestHashes: manifestHashes.toReversed(), + }, + "blob references in an order a root is never retained with": { + ...canonical, + blobHashes: blobHashes.toReversed(), + }, + "one content reference carried twice": { + ...canonical, + manifestHashes: [manifestHashes[0], ...manifestHashes], + }, + "one blob reference carried twice": { + ...canonical, + blobHashes: [blobHashes[0], ...blobHashes], + }, + }; + for (const [description, root] of Object.entries(damaged)) { + const outcome = yield* installed(answering(root), (opened) => + opened.forkSource("event-work"), + ); + expect([description, outcome.ok]).toEqual([description, false]); + } + }); + + it("reads a history answer larger than a fork source's own ceiling", function* () { + // A retained record may be as large as the transaction that wrote it, and + // history is paged by count rather than by bytes, so a single valid row can + // carry more than any fork-source page ever will. What a fork source's + // arithmetic bounds is fork-source pages. + const wide = "w".repeat(1_200_000); + // The canonical retained spelling, terminating newline included: what a + // journal holds is what `serializeDurableEvent()` wrote, and the owner + // accepts a record only when it round-trips to exactly that. + const record = serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name: "wide" }, + result: { status: "ok", value: wide }, + }); + const answer = (rows: unknown[]) => ({ + anchor: "event-wide", + after: null, + rows, + done: true, + retainedRoots: [ROOT], + provenance: [], + }); + const page = answer([{ eventId: "event-wide", record, workspaceRootId: ROOT }]); + const bytes = (value: unknown) => + new TextEncoder().encode(JSON.stringify({ outcome: "performed", value })).length; + + // Between the two ceilings: too large for a fork-source answer, and well + // within what public history has always carried. + expect(new TextEncoder().encode(record).length).toBeGreaterThan(1_200_000); + expect(bytes(page)).toBeGreaterThan(FORK_SOURCE_ANSWER_BYTES); + expect(bytes(page)).toBeLessThan(PUBLIC_ANSWER_BYTES); + + const history = yield* installed( + (read) => ({ + outcome: "performed", + value: read["operation"] === "history" ? page : inspection(), + }), + () => WorkflowLifecycle.operations.history(RUN_ID), + ); + expect([history.ok, history.ok === false && String(history.error)]).toEqual([true, false]); + if (history.ok) { + expect(history.value.map((entry) => entry.eventId)).toEqual(["event-wide"]); + expect(history.value[0]?.workspaceRootId).toBe(ROOT); + const held = history.value[0]?.event; + expect(held?.type).toBe("yield"); + expect(held?.type === "yield" && held.result).toEqual({ status: "ok", value: wide }); + } + + // The same bytes as a fork-source answer are refused before they are read, + // so what admits the history answer is the operation asked and not a + // ceiling raised for everyone. + const forked = yield* installed( + () => ({ outcome: "performed", value: page }), + (opened) => opened.forkSource("event-wide"), + ); + expect(forked.ok).toBe(false); + expect(forked.ok === false && forked.error).toEqual(expect.any(WorkflowRecordMalformedError)); + + // And a history answer past its own ceiling still fails closed. + const huge = answer([ + { eventId: "event-wide", record, workspaceRootId: ROOT }, + { eventId: "event-wider", record, workspaceRootId: ROOT }, + ]); + expect(bytes(huge)).toBeGreaterThan(PUBLIC_ANSWER_BYTES); + const refused = yield* installed( + (read) => ({ + outcome: "performed", + value: read["operation"] === "history" ? huge : inspection(), + }), + () => WorkflowLifecycle.operations.history(RUN_ID), + ); + expect(refused.ok).toBe(false); + expect(refused.ok === false && refused.error).toEqual(expect.any(WorkflowRecordMalformedError)); + }); +}); From 6dcf90612fc900a424abd617290b94d03afcaae0 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Mon, 7 Sep 2026 14:58:23 -0300 Subject: [PATCH 51/81] =?UTF-8?q?=E2=9C=A8=20Move=20a=20run's=20lifecycle,?= =?UTF-8?q?=20and=20fork=20one,=20from=20its=20owner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An admitted connection is an executor acquisition. What was missing was everything that acquisition is for: beginning an execution, finishing it, cancelling a run, and making a fork of one — each a decision the run's owner takes in one transaction, under a lock the runner cannot fabricate. The lock is an object. It is issued beside one exact connection, kept in the provider's own closure, and recognized by identity: a copy of it, another provider's, one for a different run, or one whose scope has ended authorizes nothing, and none of them reaches a read plane, an owner command or a database. Nothing expires. The acquisition ends when its connection does, which is why there is no lease here, no heartbeat and no elapsed time at all. One acquisition begins one execution, and both sides know which. The runner's hold refuses a second begin before a message is composed; the owner retains the association, because an evicted object keeps its sockets and forgets everything that was not written down, and a settlement is checked against what the owner retained rather than against what a runner remembers. A live executor cannot finish an earlier executor's work by naming its id. Beginning reaches storage that holds nothing. A start that carries its creation makes the schema, the run, its Workspace, its first execution and the decision it just took together, or makes none of them; recognizing the store first would have refused it for being empty, which is the one thing a starting run always is. What an executor left behind is closed on the next acquisition's way in, from the run's own retained evidence and in the same transaction as whatever that acquisition decides next. A fork is copied, not referenced. Its source is read through the plane that takes no acquisition, so the source keeps running while it is read; the parts cross as bounded offers that are scratch until one command adopts them; and that command writes the destination's content, its roots, its checkouts, the prefix it inherited with the records exactly as the source retained them, its lineage and its first execution in a single transaction. A transfer that does not add up — a part missing, a head root nobody sent, a root whose references its own manifest does not produce — refuses with the destination still holding nothing. Afterwards the destination needs the source for nothing at all. Staging a fork asks the same source and assembles the same thing somewhere local, taking no destination acquisition and creating nothing a host would discover. It belongs to the scope that asked for it. Retries are answered rather than repeated. A mutation's decision is retained beside the mutation it describes, so a connection that died between the commit and the answer learns what already happened instead of doing it again; the same identity carrying different content is refused rather than confused for it. --- packages/workflow/src/cloudflare/client.ts | 25 +- packages/workflow/src/cloudflare/commands.ts | 182 ++++- .../workflow/src/cloudflare/dispatcher.ts | 168 ++++- .../src/cloudflare/executor-connection.ts | 61 ++ .../workflow/src/cloudflare/lifecycle-link.ts | 279 ++++++++ .../workflow/src/cloudflare/owner-fork.ts | 649 ++++++++++++++++++ .../src/cloudflare/owner-lifecycle.ts | 22 + .../workflow/src/cloudflare/owner-open.ts | 12 +- packages/workflow/src/cloudflare/owner.ts | 9 +- .../workflow/src/cloudflare/private-schema.ts | 103 ++- .../workflow/src/cloudflare/read-client.ts | 1 + .../workflow/src/cloudflare/recognition.ts | 12 + .../workflow/src/remote/lifecycle-link.ts | 121 ++++ packages/workflow/src/remote/lifecycle.ts | 490 +++++++++++++ packages/workflow/src/remote/read.ts | 8 + .../tests/cloudflare/remote-fork.vitest.ts | 314 +++++++++ .../cloudflare/remote-lifecycle.vitest.ts | 271 ++++++++ .../cloudflare/support/executor-object.ts | 72 +- packages/workflow/tests/remote-fork.test.ts | 220 ++++++ .../workflow/tests/remote-lifecycle.test.ts | 207 ++++++ .../workflow/tests/remote-recovery.test.ts | 109 +++ .../tests/support/remote-lifecycle-host.ts | 252 +++++++ 22 files changed, 3537 insertions(+), 50 deletions(-) create mode 100644 packages/workflow/src/cloudflare/executor-connection.ts create mode 100644 packages/workflow/src/cloudflare/lifecycle-link.ts create mode 100644 packages/workflow/src/cloudflare/owner-fork.ts create mode 100644 packages/workflow/src/remote/lifecycle-link.ts create mode 100644 packages/workflow/src/remote/lifecycle.ts create mode 100644 packages/workflow/tests/cloudflare/remote-fork.vitest.ts create mode 100644 packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts create mode 100644 packages/workflow/tests/remote-fork.test.ts create mode 100644 packages/workflow/tests/remote-lifecycle.test.ts create mode 100644 packages/workflow/tests/remote-recovery.test.ts create mode 100644 packages/workflow/tests/support/remote-lifecycle-host.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index b1f8e0634..3ad72bc8c 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -34,6 +34,7 @@ import type { DefinitionRetrieval, WorkflowRunRecord } from "../storage/record.t import type { CommitIntent, OwnerLink, StartingFrontier } from "../remote/collector.ts"; import type { CommitDecision } from "../remote/publication.ts"; import { OwnerLinkError, type OwnerAnswer, type OwnerConnection } from "../remote/client.ts"; +export type { OwnerConnection }; import { parseRemoteExecution, parseRemoteInvocationSnapshot, @@ -105,6 +106,7 @@ export type PrivateRefusal = | "command:wrong-run" | "command:corrupt-journal" | "command:not-forkable" + | "command:wrong-execution" | "storage:foreign" | `storage:unsupported-version-v${number}` | "storage:corrupt"; @@ -185,6 +187,7 @@ export function privateRefusal(value: string): PrivateRefusal { case "command:wrong-run": case "command:corrupt-journal": case "command:not-forkable": + case "command:wrong-execution": case "storage:foreign": case "storage:corrupt": return value; @@ -342,6 +345,13 @@ export function cloudflareReadLink( nextId: () => string, expectedRunId: string, ): AnchoringReadLink { + const parseHeader = (value: unknown): FrontierHeader => { + const header = parseFrontier(value); + if (header.record.runId !== expectedRunId) { + return fail("an answer named another run"); + } + return header; + }; function* anchored(header: FrontierHeader): Operation { const entries: JournalEntry[] = []; const seen = new Set(); @@ -369,6 +379,8 @@ export function cloudflareReadLink( } return { + parseHeader, + *invocationSnapshot(): Operation { return answer( yield* connection.ask( @@ -668,8 +680,10 @@ export function* stageCloudflareContent( * different owners. A caller holding this holds one authority. */ /** The read link, plus the paging an open answer finishes its frontier with. */ -interface AnchoringReadLink extends RemoteReadLink { +export interface AnchoringReadLink extends RemoteReadLink { anchored(header: FrontierHeader): Operation; + /** One owner answer read as a frontier header, before its journal is walked. */ + parseHeader(value: unknown): FrontierHeader; } /** @@ -1020,6 +1034,13 @@ export function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { "this run has moved since the operation read it, so the change was not applied.", ); } + if (refusal === "command:wrong-execution") { + // A request about somebody else's work rather than a failure of this one: + // the caller named an execution its own acquisition never began. + return new WorkflowRequestError( + "this executor lock did not begin the document execution it is settling.", + ); + } if (refusal === "command:capacity") { return new WorkflowRequestError("this run's owner cannot accept more work on this connection."); } @@ -1042,7 +1063,7 @@ const REMOTE_STORE = "this run's remote storage"; * read is a malformed record rather than an unreachable owner, because those * are different facts and a caller acts on them differently. */ -function translate(error: unknown): WorkflowStorageError { +export function translate(error: unknown): WorkflowStorageError { if (error instanceof CloudflareOwnerRefusalError) { return storageFailure(error.refusal); } diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 719bb79cb..2470bdd71 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -2,7 +2,11 @@ import { type DocumentExecutionCompletion, parseDocumentExecutionCompletion, } from "../storage/record.ts"; -import { parseDurableEvent, serializeDurableEvent } from "@executablemd/durable-streams"; +import { + type DurableEvent, + parseDurableEvent, + serializeDurableEvent, +} from "@executablemd/durable-streams"; import { SHA256 } from "../workspace/root-manifest.ts"; import { admitLocator, locatorFingerprintOf } from "../composition/locator.ts"; import { @@ -64,7 +68,9 @@ export type CommandName = | "open" | "begin" | "cancel" - | "settle"; + | "settle" + | "fork-stage" + | "fork"; export type CommandRefusal = | "not-an-object" @@ -95,7 +101,15 @@ export type CommandRefusal = /** Retained journal history this owner cannot read. */ | "corrupt-journal" /** The selected prefix is not one a fork could inherit. */ - | "not-forkable"; + | "not-forkable" + /** + * This acquisition did not begin the execution it is asking about. + * + * The run is intact and the execution may well exist; it belongs to a + * different acquisition, and a live executor does not get to finish an + * earlier executor's work by naming its id. + */ + | "wrong-execution"; export class CommandError extends Error { override name = "CommandError"; @@ -261,6 +275,59 @@ export interface CancelCommand extends CommandEnvelope { readonly runId: string; } +/** The sections a fork's parts arrive in, each in its own order. */ +export type ForkSection = "inherited" | "roots" | "checkouts"; + +/** One part of a fork, as the runner offers it. */ +export interface ForkPart { + readonly section: ForkSection; + readonly position: number; + readonly part: Record; +} + +/** What the final command says the staged selection should add up to. */ +export interface ForkCounts { + readonly inherited: number; + readonly roots: number; + readonly checkouts: number; +} + +/** Which committed checkpoint of which run this fork continues. */ +export interface ForkOrigin { + readonly sourceRunId: string; + readonly checkpointEventId: string; + readonly checkpointWorkspaceRootId: string; + readonly runRecordWorkspaceRootId: string; + readonly rootImportWorkspaceRootId: string; + /** The source selection's own anchor, kept with the lineage's evidence. */ + readonly anchor: string; +} + +/** + * Offer one part of a fork's source, before any of it is a run. + * + * Scratch belonging to this connection. A part says where it stands in its + * section so the final command can tell a complete transfer from a partial one. + */ +export interface ForkStageCommand extends CommandEnvelope { + readonly command: "fork-stage"; + readonly section: ForkSection; + readonly position: number; + readonly part: Record; +} + +/** Commit the offered parts as one destination run and its first execution. */ +export interface ForkCommand extends CommandEnvelope { + readonly command: "fork"; + readonly runId: string; + readonly creation: CreateWorkflowRunRequest; + readonly origin: ForkOrigin; + readonly counts: ForkCounts; + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; + readonly executionId: string; +} + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -280,7 +347,9 @@ export type RunnerCommand = | OpenCommand | BeginCommand | CancelCommand - | SettleCommand; + | SettleCommand + | ForkStageCommand + | ForkCommand; export type CommandResult = | { readonly id: string; readonly outcome: "performed"; readonly value: unknown } @@ -310,6 +379,17 @@ const MEMBERS: Record = { begin: [...ENVELOPE, "runId", "action", "creation", "executionId"], cancel: [...ENVELOPE, "runId"], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], + "fork-stage": [...ENVELOPE, "section", "position", "part"], + fork: [ + ...ENVELOPE, + "runId", + "creation", + "origin", + "counts", + "runRecord", + "rootImport", + "executionId", + ], }; function object(value: unknown): Map { @@ -442,7 +522,9 @@ export function parseCommand(raw: string): RunnerCommand { command !== "open" && command !== "begin" && command !== "cancel" && - command !== "settle" + command !== "settle" && + command !== "fork-stage" && + command !== "fork" ) { throw new CommandError("unknown-command"); } @@ -550,6 +632,41 @@ export function parseCommand(raw: string): RunnerCommand { if (command === "cancel") { return { id, command, runId: text(members, "runId", MAX_RUN_ID) }; } + if (command === "fork-stage") { + const section = members.get("section"); + if (section !== "inherited" && section !== "roots" && section !== "checkouts") { + throw new CommandError("malformed-member"); + } + const part = members.get("part"); + if (part === null || typeof part !== "object" || Array.isArray(part)) { + throw new CommandError("malformed-member"); + } + return { + id, + command, + section, + position: whole(members.get("position")), + part: Object.fromEntries(Object.entries(part)), + }; + } + if (command === "fork") { + const runId = text(members, "runId", MAX_RUN_ID); + const creation = parseCreateRequest(members.get("creation")); + if (!creation.ok || creation.value.runId !== runId) { + throw new CommandError("malformed-member"); + } + return { + id, + command, + runId, + creation: creation.value, + origin: origin(members.get("origin")), + counts: counts(members.get("counts")), + runRecord: forkEvent(members.get("runRecord")), + rootImport: forkEvent(members.get("rootImport")), + executionId: text(members, "executionId", MAX_RUN_ID), + }; + } if (command === "executions") { const anchor = sequence(members, "anchor"); const after = sequence(members, "after"); @@ -585,6 +702,61 @@ export function parseCommand(raw: string): RunnerCommand { }; } +/** A whole count, as a member rather than a column. */ +function whole(value: unknown): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + throw new CommandError("malformed-member"); + } + return value; +} + +/** Where a fork came from: one source, one checkpoint, three head roots. */ +function origin(value: unknown): ForkOrigin { + const found = object(value); + closed(found, [ + "sourceRunId", + "checkpointEventId", + "checkpointWorkspaceRootId", + "runRecordWorkspaceRootId", + "rootImportWorkspaceRootId", + "anchor", + ]); + return { + sourceRunId: text(found, "sourceRunId", MAX_RUN_ID), + checkpointEventId: text(found, "checkpointEventId", MAX_ID), + checkpointWorkspaceRootId: digest(found, "checkpointWorkspaceRootId"), + runRecordWorkspaceRootId: digest(found, "runRecordWorkspaceRootId"), + rootImportWorkspaceRootId: digest(found, "rootImportWorkspaceRootId"), + anchor: digest(found, "anchor"), + }; +} + +/** How many parts each section should have arrived in. */ +function counts(value: unknown): ForkCounts { + const found = object(value); + closed(found, ["inherited", "roots", "checkouts"]); + return { + inherited: whole(found.get("inherited")), + roots: whole(found.get("roots")), + checkouts: whole(found.get("checkouts")), + }; +} + +/** One of the two records a fork writes for itself. */ +function forkEvent(value: unknown): DurableEvent { + if (typeof value !== "string" || value === "") { + throw new CommandError("malformed-member"); + } + if (new TextEncoder().encode(value).length > MAX_MESSAGE_BYTES) { + throw new CommandError("too-large"); + } + const parsed = parseDurableEvent(value); + if (!parsed.ok) { + throw new CommandError("malformed-member"); + } + return parsed.value; +} + /** * The Workspace half of a proposal, or its absence. * diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 27ea40521..c42ba84bb 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -53,11 +53,17 @@ import { readRoot, } from "./owner-reads.ts"; import type { OwnerTransaction, OwnerTransactions } from "./owner-transaction.ts"; -import { COMMAND_TABLE, MUTATION_TABLE, STAGING_TABLE } from "./private-schema.ts"; +import { + COMMAND_TABLE, + initializePrivateSchema, + MUTATION_TABLE, + STAGING_TABLE, +} from "./private-schema.ts"; import { applyCommit, applyRetrieval } from "./publish.ts"; -import { recognizeObject } from "./recognition.ts"; +import { holdsNoRun, recognizeObject } from "./recognition.ts"; import { openRun } from "./owner-open.ts"; import { beginRun, cancelRunOnOwner, settleRun } from "./owner-lifecycle.ts"; +import { commitFork, discardForkParts, stageForkPart } from "./owner-fork.ts"; function requestFingerprint(command: RunnerCommand): string { // The command name is part of the fingerprint, so one textual id used for a @@ -73,6 +79,34 @@ function requestFingerprint(command: RunnerCommand): string { * A read can be asked again; a mutation cannot, so its answer is retained where * the next connection can find it. */ +/** + * Whether this command may find nothing and make the run anyway. + * + * A starting `begin` carries the run's whole immutable identity, and a + * committed fork carries the destination's. Both create the schema, the run and + * their first execution in one transaction, so both have to be allowed to + * arrive at a store that holds nothing. + */ +function initializes(command: RunnerCommand): boolean { + return ( + command.command === "fork" || + (command.command === "begin" && command.action === "start" && command.creation !== null) + ); +} + +/** + * Whether this command only offers scratch. + * + * A fork's parts and its content are offered to a destination that does not + * exist yet — that is the whole point of offering them — so these have to reach + * a store holding no run. They write nothing a reader can see: the scratch + * tables are this adapter's own, and the command that adopts them is what makes + * a run. + */ +function offersScratch(command: RunnerCommand): boolean { + return command.command === "stage" || command.command === "fork-stage"; +} + function mutating(command: RunnerCommand): boolean { return ( command.command === "commit" || @@ -82,7 +116,10 @@ function mutating(command: RunnerCommand): boolean { // rather than apply the transition again. command.command === "begin" || command.command === "cancel" || - command.command === "settle" + command.command === "settle" || + // A committed fork creates a destination run. Its decision has to outlive + // the connection for the same reason a begin's does. + command.command === "fork" ); } @@ -93,6 +130,21 @@ function integer(value: unknown): number { return value; } +/** What this acquisition has already spent of its own ledger. */ +function ledgerUsage( + storage: AcquisitionContext["storage"], + acquisitionId: string, +): { commands: number; bytes: number } { + const row = storage.sql + .exec( + `SELECT count(*) AS commands, coalesce(sum(response_bytes), 0) AS bytes + FROM ${COMMAND_TABLE} WHERE acquisition_id = ?`, + acquisitionId, + ) + .toArray()[0]; + return { commands: integer(row?.["commands"]), bytes: integer(row?.["bytes"]) }; +} + function storedDecision(value: unknown, id: string): CommandResult | "reconstruct" { if (typeof value !== "string") { throw new Error("private protocol storage holds a malformed result"); @@ -287,6 +339,7 @@ function perform( value: beginRun( ctx.storage, transaction, + acquisitionId, command.runId, command.action, command.creation, @@ -295,6 +348,41 @@ function perform( ), }; } + if (command.command === "fork-stage") { + return { + id: command.id, + outcome: "performed", + value: stageForkPart(ctx.storage, acquisitionId, { + section: command.section, + position: command.position, + part: command.part, + }), + }; + } + if (command.command === "fork") { + const forked = commitFork( + ctx.storage, + transaction, + acquisitionId, + { + runId: command.runId, + creation: command.creation, + origin: command.origin, + counts: command.counts, + runRecord: command.runRecord, + rootImport: command.rootImport, + executionId: command.executionId, + }, + mintEventId, + ownerTime, + ); + if (forked.value !== null) { + // Adopted, so the parts are no longer anything. What they described is + // the run now. + discardForkParts(ctx.storage, acquisitionId); + } + return { id: command.id, outcome: "performed", value: forked }; + } if (command.command === "cancel") { return { id: command.id, @@ -308,6 +396,7 @@ function perform( outcome: "performed", value: settleRun( ctx.storage, + acquisitionId, runId, command.completion, command.expectedWorkspaceRootId, @@ -357,16 +446,30 @@ export function dispatchCommand( if (inside.acquisitionId !== held.acquisitionId) { throw new CommandError("duplicate-conflict"); } - // Every command that reaches here is asked of a run that already exists, - // so the store is held to this build's schema before it is read. `open` is - // the one that may find nothing, and it answered above. - recognizeObject(ctx.storage); + // Almost every command that reaches here is asked of a run that already + // exists, so the store is held to this build's schema before it is read. + // The exceptions are the two that create one: a starting `begin` and a + // committed `fork` reach pristine storage on purpose and initialize it + // inside this same transaction. Recognizing first would refuse them for + // holding nothing at all, and there would be no way to start a remote run. + const empty = holdsNoRun(ctx.storage); + const creating = initializes(command) && empty; + const offering = offersScratch(command) && empty; + if (offering) { + // The scratch this command needs, and nothing else: no schema, no run, no + // marker. What is here after it is still a store holding no run. + initializePrivateSchema(ctx.storage); + } + if (!creating && !offering) { + recognizeObject(ctx.storage); + } // A mutation's decision is looked for by the run, not by the connection. // The case this exists for is the one where the connection that asked is // gone: the owner committed, the answer never arrived, and the runner - // reconnected to ask the same question again. - if (mutating(command)) { + // reconnected to ask the same question again. Pristine storage retains no + // decision, and its private substrate does not exist yet to be asked. + if (mutating(command) && !creating) { const decided = ctx.storage.sql .exec( `SELECT request_fingerprint, response FROM ${MUTATION_TABLE} WHERE command_id = ?`, @@ -387,14 +490,17 @@ export function dispatchCommand( } } - const previous = ctx.storage.sql - .exec( - `SELECT request_fingerprint, response FROM ${COMMAND_TABLE} - WHERE acquisition_id = ? AND command_id = ?`, - held.acquisitionId, - command.id, - ) - .toArray()[0]; + const previous = + creating && !offering + ? undefined + : ctx.storage.sql + .exec( + `SELECT request_fingerprint, response FROM ${COMMAND_TABLE} + WHERE acquisition_id = ? AND command_id = ?`, + held.acquisitionId, + command.id, + ) + .toArray()[0]; if (previous !== undefined) { if (previous.request_fingerprint !== fingerprint) { throw new CommandError("duplicate-conflict"); @@ -404,23 +510,19 @@ export function dispatchCommand( ? perform(ctx, runId, held.acquisitionId, command, transaction) : decision; } - const usage = ctx.storage.sql - .exec( - `SELECT count(*) AS commands, coalesce(sum(response_bytes), 0) AS bytes - FROM ${COMMAND_TABLE} WHERE acquisition_id = ?`, - held.acquisitionId, - ) - .toArray()[0]; - if ( - integer(usage?.["commands"]) >= MAX_COMMANDS || - integer(usage?.["bytes"]) >= MAX_LEDGER_BYTES - ) { + // A store with no run has spent nothing this ledger knows about, and until + // the scratch exists there is nothing to ask. + const usage = + creating && !offering + ? { commands: 0, bytes: 0 } + : ledgerUsage(ctx.storage, held.acquisitionId); + if (usage.commands >= MAX_COMMANDS || usage.bytes >= MAX_LEDGER_BYTES) { throw new CommandError("capacity"); } const result = perform(ctx, runId, held.acquisitionId, command, transaction); const encoded = retainedDecision(command, result); const responseBytes = new TextEncoder().encode(encoded).length; - if (integer(usage?.["bytes"]) + responseBytes > MAX_LEDGER_BYTES) { + if (usage.bytes + responseBytes > MAX_LEDGER_BYTES) { throw new CommandError("capacity"); } ctx.storage.sql.exec( @@ -436,10 +538,10 @@ export function dispatchCommand( if (mutating(command)) { // Recorded in this same transaction as the mutation it describes, so a // crash cannot leave one without the other. - const mutations = ctx.storage.sql - .exec(`SELECT count(*) AS decided FROM ${MUTATION_TABLE}`) - .toArray()[0]; - if (integer(mutations?.["decided"]) >= MAX_COMMANDS) { + const mutations = creating + ? undefined + : ctx.storage.sql.exec(`SELECT count(*) AS decided FROM ${MUTATION_TABLE}`).toArray()[0]; + if (mutations !== undefined && integer(mutations["decided"]) >= MAX_COMMANDS) { throw new CommandError("capacity"); } ctx.storage.sql.exec( diff --git a/packages/workflow/src/cloudflare/executor-connection.ts b/packages/workflow/src/cloudflare/executor-connection.ts new file mode 100644 index 000000000..937379590 --- /dev/null +++ b/packages/workflow/src/cloudflare/executor-connection.ts @@ -0,0 +1,61 @@ +/** + * One admitted executor connection, assembled for this owner. + * + * The provider asks its host for an acquisition; this is what a Cloudflare host + * gives it. Both halves come from the same socket — the link that reads and + * commits, and the lifecycle commands that move the run — because they are the + * same authority. Reaching the socket is the host's business and stays behind + * the `open` it is handed, so nothing here knows about tokens, releases or + * upgrade headers. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import type { RemoteExecutorConnection } from "../remote/lifecycle-link.ts"; +import { useOwnerConnection, type OwnerSocket } from "../remote/client.ts"; +import { cloudflareLifecycleLink } from "./lifecycle-link.ts"; +import { cloudflareReadLink, cloudflareRunLink, translate } from "./client.ts"; + +/** + * How a host reaches one run's executor socket. + * + * Answering `already-running` is a fact about the run: another live executor + * holds it, and that is not an error to translate but an outcome to report. + */ +export interface ExecutorAdmission { + open(runId: string): Operation>; + /** Fresh correlation identities for this connection's commands. */ + ids(): () => string; +} + +/** + * Admit one connection and build both halves over it. + * + * The connection is a resource of the calling scope, so it closes exactly when + * that scope ends — which is what makes the acquisition's lifetime the + * connection's lifetime rather than a duration. + */ +export function* useExecutorConnection( + admission: ExecutorAdmission, + runId: string, +): Operation> { + const socket = yield* admission.open(runId); + if (!socket.ok) { + return socket; + } + if (socket.value === "already-running") { + return Ok("already-running"); + } + try { + const connection = yield* useOwnerConnection(socket.value); + const nextId = admission.ids(); + const reads = cloudflareReadLink(connection, nextId, runId); + return Ok({ + link: cloudflareRunLink(connection, nextId, runId), + lifecycle: cloudflareLifecycleLink(connection, reads, nextId), + }); + } catch (error) { + // Whatever went wrong reaching or building the connection, a caller learns + // it as a storage failure rather than as this adapter's own vocabulary. + return Err(translate(error)); + } +} diff --git a/packages/workflow/src/cloudflare/lifecycle-link.ts b/packages/workflow/src/cloudflare/lifecycle-link.ts new file mode 100644 index 000000000..c9d72a8da --- /dev/null +++ b/packages/workflow/src/cloudflare/lifecycle-link.ts @@ -0,0 +1,279 @@ +/** + * The lifecycle commands, spelled for this owner. + * + * The same connection that reads a run and commits to it is the one that + * begins, settles, cancels and forks it: the authority is the socket, and + * splitting the lifecycle onto a second link would be a second authority. So + * this is built from the same connection as the Workspace half and composed + * with it, never paired from somewhere else. + * + * Everything an owner answers is parsed before it becomes a value. A begin that + * says it began an execution has to say which one, against a frontier that + * describes this run; a refusal has to be one of the three conditions this + * build knows; and anything else is an answer this build cannot read, reported + * as damage rather than guessed at. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import type { + DocumentExecutionCompletion, + DocumentExecutionRecord, + WorkflowRunRecord, +} from "../storage/record.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import type { RemoteFrontierSnapshot } from "../remote/read.ts"; +import type { + RemoteBegun, + RemoteForkCommit, + RemoteForkPart, + RemoteLifecycleAnswer, + RemoteLifecycleLink, +} from "../remote/lifecycle-link.ts"; +import { WorkflowRecordMalformedError, WorkflowRunConflictError } from "../storage/errors.ts"; +import { parseRemoteExecution } from "../remote/records.ts"; +import type { AnchoringReadLink, OwnerConnection } from "./client.ts"; +import { privateRefusal, storageFailure, translate } from "./client.ts"; + +const REFUSALS = new Set(["cancelled", "resume-failed", "terminal"]); + +function fail(reason: string): never { + throw new WorkflowRecordMalformedError("lifecycle answer this run's owner returned", reason); +} + +function members(value: unknown, names: readonly string[]): Map { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + return fail("it was not one object"); + } + const found = new Map(Object.entries(value)); + if (found.size !== names.length || names.some((name) => !found.has(name))) { + return fail("it did not carry the members this build reads"); + } + return found; +} + +/** The three fields a lifecycle answer is one of, and never two of. */ +function answered( + value: unknown, + runId: string, + read: (value: unknown) => T, +): RemoteLifecycleAnswer { + const found = members(value, ["conflict", "refusal", "value"]); + const conflict = found.get("conflict"); + const refusal = found.get("refusal"); + const held = found.get("value"); + const present = [conflict, refusal, held].filter((member) => member !== null).length; + if (present !== 1) { + return fail("it did not answer exactly one way"); + } + if (conflict !== null) { + if (!Array.isArray(conflict) || conflict.length === 0) { + return fail("it named no differing field"); + } + throw new WorkflowRunConflictError( + runId, + conflict.map((field) => (typeof field === "string" ? field : fail("it named no field"))), + ); + } + if (refusal !== null) { + if (typeof refusal !== "string" || !REFUSALS.has(refusal)) { + return fail("it named no condition this build reads"); + } + if (refusal !== "cancelled" && refusal !== "resume-failed" && refusal !== "terminal") { + return fail("it named no condition this build reads"); + } + return { kind: "refused", refusal }; + } + return { kind: "performed", value: read(held) }; +} + +function execution(value: unknown): DocumentExecutionRecord { + return parseRemoteExecution(value); +} + +/** + * One lifecycle link over an admitted connection. + * + * `reads` is the same anchoring read link the Workspace half uses, so a + * frontier this returns is assembled exactly the way every other frontier is. + */ +export function cloudflareLifecycleLink( + connection: OwnerConnection, + reads: AnchoringReadLink, + nextId: () => string, +): RemoteLifecycleLink { + function* frontierOf(value: unknown): Operation { + return yield* reads.anchored(reads.parseHeader(value)); + } + + function* begun(value: unknown): Operation { + const found = members(value, ["frontier", "execution", "replay", "recovered"]); + if (typeof found.get("replay") !== "boolean") { + return fail("it did not say whether the run replayed"); + } + const recovered = found.get("recovered"); + return { + frontier: yield* frontierOf(found.get("frontier")), + execution: execution(found.get("execution")), + replay: found.get("replay") === true, + recovered: recovered === null ? null : execution(recovered), + }; + } + + return { + *begin(request: { + readonly runId: string; + readonly action: "start" | "resume"; + readonly creation: CreateWorkflowRunRequest | null; + readonly executionId: string; + }): Operation>> { + try { + const offered = yield* connection.ask( + nextId(), + { + command: "begin", + runId: request.runId, + action: request.action, + creation: request.creation, + executionId: request.executionId, + }, + (value: unknown) => value, + privateRefusal, + ); + if (offered.outcome === "refused") { + return Err(storageFailure(privateRefusal(offered.refusal))); + } + // Parsed outside the answer callback because assembling a frontier is + // more owner reads, and those belong to this operation rather than to + // the one message that carried the header. + const decided = answered(offered.value, request.runId, (value) => value); + if (decided.kind === "refused") { + return Ok(decided); + } + return Ok({ kind: "performed", value: yield* begun(decided.value) }); + } catch (error) { + return Err(translate(error)); + } + }, + + *settle( + completion: DocumentExecutionCompletion, + expectedWorkspaceRootId: string, + ): Operation> { + try { + const offered = yield* connection.ask( + nextId(), + { command: "settle", completion, expectedWorkspaceRootId }, + (value: unknown) => value, + privateRefusal, + ); + if (offered.outcome === "refused") { + return Err(storageFailure(privateRefusal(offered.refusal))); + } + return Ok(yield* frontierOf(offered.value)); + } catch (error) { + return Err(translate(error)); + } + }, + + *cancel(runId: string): Operation>> { + try { + const offered = yield* connection.ask( + nextId(), + { command: "cancel", runId }, + (value: unknown) => value, + privateRefusal, + ); + if (offered.outcome === "refused") { + return Err(storageFailure(privateRefusal(offered.refusal))); + } + const decided = answered(offered.value, runId, (value) => value); + if (decided.kind === "refused") { + return Ok(decided); + } + const frontier = yield* frontierOf(decided.value); + return Ok({ kind: "performed", value: frontier.record }); + } catch (error) { + return Err(translate(error)); + } + }, + + *stageForkPart(part: RemoteForkPart): Operation> { + try { + const offered = yield* connection.ask( + nextId(), + { + command: "fork-stage", + section: part.section, + position: part.position, + part: part.part, + }, + (value: unknown) => members(value, ["staged"]).get("staged"), + privateRefusal, + ); + if (offered.outcome === "refused") { + return Err(storageFailure(privateRefusal(offered.refusal))); + } + return Ok(undefined); + } catch (error) { + return Err(translate(error)); + } + }, + + *commitFork(commit: RemoteForkCommit): Operation> { + try { + const offered = yield* connection.ask( + nextId(), + { + command: "fork", + runId: commit.runId, + creation: commit.creation, + origin: commit.origin, + counts: commit.counts, + runRecord: record(commit.runRecord), + rootImport: record(commit.rootImport), + executionId: commit.executionId, + }, + (value: unknown) => value, + privateRefusal, + ); + if (offered.outcome === "refused") { + return Err(storageFailure(privateRefusal(offered.refusal))); + } + const found = members(offered.value, ["conflict", "value"]); + const conflict = found.get("conflict"); + if (conflict !== null) { + if (!Array.isArray(conflict) || conflict.length === 0) { + return Err( + new WorkflowRecordMalformedError( + "lifecycle answer this run's owner returned", + "it named no differing field", + ), + ); + } + return Err( + new WorkflowRunConflictError( + commit.runId, + conflict.map((field) => (typeof field === "string" ? field : "definition")), + ), + ); + } + const held = members(found.get("value"), ["frontier", "execution"]); + return Ok({ + frontier: yield* frontierOf(held.get("frontier")), + execution: execution(held.get("execution")), + replay: false, + recovered: null, + }); + } catch (error) { + return Err(translate(error)); + } + }, + }; +} + +/** One head record, in the canonical spelling a journal retains. */ +function record(event: DurableEvent): string { + return serializeDurableEvent(event); +} diff --git a/packages/workflow/src/cloudflare/owner-fork.ts b/packages/workflow/src/cloudflare/owner-fork.ts new file mode 100644 index 000000000..c3b7375d3 --- /dev/null +++ b/packages/workflow/src/cloudflare/owner-fork.ts @@ -0,0 +1,649 @@ +/** + * Planting one source's committed prefix in a destination this owner holds. + * + * A fork arrives in two stages because a source is larger than one message may + * be. First the runner offers its parts — the roots, the inherited rows, the + * checkouts, and the content those roots name — and each part is scratch that + * belongs to the offering connection and describes nothing. Then one command + * commits them: the destination's schema, its immutable identity, the whole + * copied prefix, its Workspace closure, its lineage and its first execution all + * appear together, or the destination holds nothing at all. + * + * Nothing here trusts the parts for being staged. A staged root is held to its + * own manifest, its reference arrays are derived rather than believed, every + * piece of content it names must have been offered, and every inherited row + * must belong to a root that came with it. Staging is a way to cross, not a way + * to be believed. + */ + +import { + type DurableEvent, + parseDurableEvent, + serializeDurableEvent, +} from "@executablemd/durable-streams"; +import { + compareUtf8, + parseWorkspaceRootManifest, + WORKSPACE_ROOT_DOMAIN, + WORKSPACE_ROOT_FORMAT, +} from "../workspace/root-manifest.ts"; +import { decodeContentManifest } from "../workspace/content-manifest.ts"; +import { sha256Hex } from "../workspace/sha256.ts"; +import { bytesOf } from "./encoding.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import { + CommandError, + type ForkCounts, + type ForkOrigin, + type ForkPart, + type ForkSection, +} from "./commands.ts"; +import type { OwnerStorage } from "./storage.ts"; +import type { OwnerTransaction } from "./owner-transaction.ts"; +import { establishRun } from "./owner-open.ts"; +import { readFrontier, validateRetainedRoot, type FrontierValue } from "./owner-reads.ts"; +import { FORK_TABLE, holdExecution, STAGING_TABLE } from "./private-schema.ts"; +import { readDocumentExecution, readRunRecord, type Row } from "../sqlite/rows.ts"; +import type { DocumentExecutionRecord } from "../storage/record.ts"; + +/** What one committed fork produced. */ +export interface ForkedValue { + readonly frontier: FrontierValue; + readonly execution: DocumentExecutionRecord; +} + +/** A fork answer: the destination, or which immutable fields say it is another. */ +export interface ForkValue { + readonly conflict: readonly string[] | null; + readonly value: ForkedValue | null; +} + +const MAX_PART_BYTES = 256 * 1024; +const MAX_PARTS = 8192; + +function rows(storage: OwnerStorage, sql: string, ...bindings: unknown[]): Row[] { + return storage.sql.exec(sql, ...bindings).toArray(); +} + +function text(value: unknown): string { + if (typeof value !== "string" || value === "") { + throw new CommandError("malformed-member"); + } + return value; +} + +function digest(value: unknown): string { + const candidate = text(value); + if (!/^[0-9a-f]{64}$/.test(candidate)) { + throw new CommandError("malformed-member"); + } + return candidate; +} + +function count(value: unknown): number { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + throw new CommandError("malformed-member"); + } + return value; +} + +function list(value: unknown): unknown[] { + if (!Array.isArray(value)) { + throw new CommandError("malformed-member"); + } + return value; +} + +function members(value: unknown, names: readonly string[]): Map { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new CommandError("malformed-member"); + } + const found = new Map(Object.entries(value)); + if (found.size !== names.length || names.some((name) => !found.has(name))) { + throw new CommandError("unknown-member"); + } + return found; +} + +/** + * Keep one offered part, bounded and in its place. + * + * Offering the same position twice is a conflict rather than a replacement: a + * transfer that rewrote its own members would be one nobody could describe. + */ +export function stageForkPart( + storage: OwnerStorage, + acquisitionId: string, + part: ForkPart, +): { staged: number } { + const encoded = JSON.stringify(part.part); + const size = new TextEncoder().encode(encoded).length; + if (size > MAX_PART_BYTES) { + throw new CommandError("too-large"); + } + const held = rows( + storage, + `SELECT count(*) AS parts FROM ${FORK_TABLE} WHERE acquisition_id = ?`, + acquisitionId, + )[0]; + if (count(held?.["parts"]) >= MAX_PARTS) { + throw new CommandError("capacity"); + } + const already = rows( + storage, + `SELECT part FROM ${FORK_TABLE} + WHERE acquisition_id = ? AND section = ? AND position = ?`, + acquisitionId, + part.section, + part.position, + )[0]; + if (already !== undefined) { + if (already["part"] !== encoded) { + throw new CommandError("duplicate-conflict"); + } + return { staged: count(held?.["parts"]) }; + } + storage.sql.exec( + `INSERT INTO ${FORK_TABLE} (acquisition_id, section, position, part, part_bytes) + VALUES (?, ?, ?, ?, ?)`, + acquisitionId, + part.section, + part.position, + encoded, + size, + ); + return { staged: count(held?.["parts"]) + 1 }; +} + +/** Every part of one section, in the order it was offered, with no gaps. */ +function section( + storage: OwnerStorage, + acquisitionId: string, + name: ForkSection, + expected: number, +): Record[] { + const held = rows( + storage, + `SELECT position, part FROM ${FORK_TABLE} + WHERE acquisition_id = ? AND section = ? ORDER BY position`, + acquisitionId, + name, + ); + if (held.length !== expected) { + throw new CommandError("malformed-member"); + } + return held.map((row, at) => { + if (count(row["position"]) !== at) { + // A gap, so the section is not the one the counts describe. + throw new CommandError("malformed-member"); + } + const parsed: unknown = JSON.parse(text(row["part"])); + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new CommandError("malformed-member"); + } + return Object.fromEntries(Object.entries(parsed)); + }); +} + +/** Content this acquisition offered, by the identity its bytes produce. */ +function staged( + storage: OwnerStorage, + acquisitionId: string, + kind: "manifest" | "blob", + hash: string, +): Uint8Array { + const row = rows( + storage, + `SELECT bytes FROM ${STAGING_TABLE} WHERE acquisition_id = ? AND kind = ? AND digest = ?`, + acquisitionId, + kind, + hash, + )[0]; + if (row === undefined) { + // Named by a root but never offered: the transfer is not the closure it + // claims to be. + throw new CommandError("malformed-member"); + } + const bytes = bytesOf(row["bytes"]); + if (sha256Hex(bytes) !== hash) { + throw new CommandError("malformed-member"); + } + return bytes; +} + +interface StagedRoot { + readonly rootId: string; + readonly manifest: string; + readonly manifestHashes: readonly string[]; + readonly blobHashes: readonly string[]; +} + +function parseRoot(part: Record): StagedRoot { + const found = members(part, [ + "rootId", + "formatVersion", + "manifest", + "manifestHashes", + "blobHashes", + ]); + if (found.get("formatVersion") !== WORKSPACE_ROOT_FORMAT) { + throw new CommandError("malformed-member"); + } + const manifest = text(found.get("manifest")); + const rootId = digest(found.get("rootId")); + if (sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${manifest}`) !== rootId) { + throw new CommandError("malformed-member"); + } + return { + rootId, + manifest, + manifestHashes: list(found.get("manifestHashes")).map((value) => digest(value)), + blobHashes: list(found.get("blobHashes")).map((value) => digest(value)), + }; +} + +function sameOrder(left: readonly string[], right: readonly string[]): boolean { + return left.length === right.length && left.every((value, at) => value === right[at]); +} + +/** + * Retain one root's content and the root itself, proving the closure as it goes. + * + * The reference arrays are derived from the root's own manifest and compared + * element for element, which is the same comparison a destination makes when it + * reads the root back. A root whose arrays are the right set in the wrong order + * is refused here rather than becoming unreadable later. + */ +function retainRoot(storage: OwnerStorage, acquisitionId: string, root: StagedRoot): void { + const parsed = parseWorkspaceRootManifest(root.manifest, () => { + throw new CommandError("malformed-member"); + }); + const named = new Set(); + const sizes = new Map(); + for (const entry of parsed.entries) { + if (entry.kind === "file") { + named.add(entry.manifest); + const already = sizes.get(entry.manifest); + if (already !== undefined && already !== entry.size) { + throw new CommandError("malformed-member"); + } + sizes.set(entry.manifest, entry.size); + } + } + const blobs = new Map(); + const manifests = new Map(); + for (const hash of named) { + const bytes = staged(storage, acquisitionId, "manifest", hash); + const content = decodeContentManifest(bytes, () => { + throw new CommandError("malformed-member"); + }); + if (sizes.get(hash) !== content.size) { + throw new CommandError("malformed-member"); + } + manifests.set(hash, bytes); + for (const chunk of content.chunks) { + const seen = blobs.get(chunk.hash); + if (seen !== undefined && seen !== chunk.size) { + throw new CommandError("malformed-member"); + } + blobs.set(chunk.hash, chunk.size); + } + } + if ( + !sameOrder([...named].toSorted(compareUtf8), root.manifestHashes) || + !sameOrder([...blobs.keys()].toSorted(compareUtf8), root.blobHashes) + ) { + throw new CommandError("malformed-member"); + } + + for (const [hash, size] of blobs) { + const bytes = staged(storage, acquisitionId, "blob", hash); + if (bytes.length !== size) { + throw new CommandError("malformed-member"); + } + const key = hexBytes(hash); + storage.sql.exec( + "INSERT INTO vfs_blobs (hash, size, last_seen) VALUES (?, ?, 0) ON CONFLICT(hash) DO NOTHING", + key, + size, + ); + storage.sql.exec( + "INSERT INTO vfs_blob_bytes (hash, bytes) VALUES (?, ?) ON CONFLICT(hash) DO NOTHING", + key, + bytes, + ); + } + for (const [hash, bytes] of manifests) { + storage.sql.exec( + `INSERT INTO vfs_manifests (hash, size, encoded, last_seen) VALUES (?, ?, ?, 0) + ON CONFLICT(hash) DO NOTHING`, + hexBytes(hash), + decodeContentManifest(bytes, () => { + throw new CommandError("malformed-member"); + }).size, + bytes, + ); + } + + const existing = rows( + storage, + "SELECT manifest FROM workspace_roots WHERE root_id = ?", + root.rootId, + )[0]; + if (existing === undefined) { + storage.sql.exec( + "INSERT INTO workspace_roots (root_id, format_version, manifest) VALUES (?, ?, ?)", + root.rootId, + WORKSPACE_ROOT_FORMAT, + root.manifest, + ); + for (const hash of root.manifestHashes) { + storage.sql.exec( + "INSERT INTO workspace_root_manifest_refs (root_id, manifest_hash) VALUES (?, ?)", + root.rootId, + hexBytes(hash), + ); + } + for (const hash of root.blobHashes) { + storage.sql.exec( + "INSERT INTO workspace_root_blob_refs (root_id, blob_hash) VALUES (?, ?)", + root.rootId, + hexBytes(hash), + ); + } + } else if (existing["manifest"] !== root.manifest) { + throw new CommandError("duplicate-conflict"); + } + validateRetainedRoot(storage, root.rootId); +} + +function hexBytes(digestHex: string): Uint8Array { + const bytes = new Uint8Array(32); + for (let at = 0; at < 32; at += 1) { + bytes[at] = Number.parseInt(digestHex.slice(at * 2, at * 2 + 2), 16); + } + return bytes; +} + +/** One inherited row, exactly as the source retained it. */ +function writeInherited( + storage: OwnerStorage, + part: Record, + sourceRunId: string, + carried: ReadonlySet, +): void { + const found = members(part, ["eventId", "record", "workspaceRootId"]); + const eventId = text(found.get("eventId")); + const record = text(found.get("record")); + const rootId = digest(found.get("workspaceRootId")); + if (!carried.has(rootId)) { + throw new CommandError("malformed-member"); + } + const parsed = parseDurableEvent(record); + if (!parsed.ok) { + throw new CommandError("corrupt-journal"); + } + storage.sql.exec( + "INSERT INTO journal_events (event_id, record, workspace_root_id) VALUES (?, ?, ?)", + eventId, + record, + rootId, + ); + storage.sql.exec( + `INSERT INTO journal_event_provenance (event_id, source_run_id, source_event_id) + VALUES (?, ?, ?)`, + eventId, + sourceRunId, + eventId, + ); +} + +/** One inherited checkout, in the directory the checkpoint's Workspace holds. */ +function writeCheckout( + storage: OwnerStorage, + part: Record, + directories: ReadonlySet, + repositories: Set, +): void { + const kind = part["kind"]; + if (kind === "repository") { + const found = members(part, [ + "kind", + "name", + "locator", + "locatorFingerprint", + "requestedBase", + "creationCommit", + "primaryBranch", + "objectFormat", + "checkoutPath", + ]); + const path = text(found.get("checkoutPath")); + if (!directories.has(path)) { + throw new CommandError("malformed-member"); + } + const format = found.get("objectFormat"); + if (format !== "sha1" && format !== "sha256") { + throw new CommandError("malformed-member"); + } + const name = text(found.get("name")); + storage.sql.exec( + `INSERT INTO workspace_repositories (name, locator, locator_fingerprint, requested_base, + creation_commit, primary_branch, object_format, checkout_path) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + name, + text(found.get("locator")), + digest(found.get("locatorFingerprint")), + found.get("requestedBase") === null ? null : text(found.get("requestedBase")), + text(found.get("creationCommit")), + text(found.get("primaryBranch")), + format, + path, + ); + repositories.add(name); + return; + } + const found = members(part, [ + "kind", + "repositoryName", + "name", + "requestedBranch", + "requestedBase", + "creationCommit", + "checkoutPath", + ]); + if (kind !== "worktree") { + throw new CommandError("malformed-member"); + } + const path = text(found.get("checkoutPath")); + if (!directories.has(path)) { + throw new CommandError("malformed-member"); + } + const repository = text(found.get("repositoryName")); + if (!repositories.has(repository)) { + // A Worktree of a Repository that did not come with it belongs to nothing. + throw new CommandError("malformed-member"); + } + storage.sql.exec( + `INSERT INTO workspace_worktrees (repository_name, name, requested_branch, requested_base, + creation_commit, checkout_path) + VALUES (?, ?, ?, ?, ?, ?)`, + repository, + text(found.get("name")), + text(found.get("requestedBranch")), + found.get("requestedBase") === null ? null : text(found.get("requestedBase")), + text(found.get("creationCommit")), + path, + ); +} + +/** + * Commit one fork: the destination and everything it inherited, together. + * + * Runs inside the caller's transaction. What it writes is what the schema's own + * references require, in that order: content, then the roots that name it, then + * the Workspace pointer those roots are restored into, then the checkouts, then + * the journal rows that name the roots, then the lineage, and last the fork's + * own first execution. A failure anywhere rolls the whole thing back, so the + * destination is either absent or complete. + */ +export function commitFork( + storage: OwnerStorage, + transaction: OwnerTransaction, + acquisitionId: string, + input: { + readonly runId: string; + readonly creation: CreateWorkflowRunRequest; + readonly origin: ForkOrigin; + readonly counts: ForkCounts; + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; + readonly executionId: string; + }, + mintEventId: () => string, + now: () => string, +): ForkValue { + // Whether this destination already holds a run decides what committing + // means: making one, or confirming the one that is here is this fork. + const fresh = + rows(storage, "SELECT name FROM sqlite_schema WHERE type = 'table' AND name = 'workflow_run'") + .length === 0; + const conflict = establishRun(storage, transaction, input.runId, input.creation, now); + if (conflict !== null) { + return { conflict, value: null }; + } + if (!fresh) { + // A destination that already exists is the same fork only when it came + // from the same place. Its identity was compared above; this is its + // lineage. + const lineage = rows( + storage, + `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id + FROM workflow_fork_lineage WHERE id = 1`, + )[0]; + if ( + lineage === undefined || + lineage["source_run_id"] !== input.origin.sourceRunId || + lineage["checkpoint_event_id"] !== input.origin.checkpointEventId || + lineage["checkpoint_workspace_root_id"] !== input.origin.checkpointWorkspaceRootId + ) { + return { conflict: ["lineage"], value: null }; + } + return { conflict: null, value: begunOn(storage, acquisitionId, input, now) }; + } + + const roots = section(storage, acquisitionId, "roots", input.counts.roots).map((part) => + parseRoot(part), + ); + const carried = new Set(roots.map((root) => root.rootId)); + for (const rootId of [ + input.origin.checkpointWorkspaceRootId, + input.origin.runRecordWorkspaceRootId, + input.origin.rootImportWorkspaceRootId, + ]) { + if (!carried.has(rootId)) { + throw new CommandError("malformed-member"); + } + } + for (const root of roots) { + retainRoot(storage, acquisitionId, root); + } + + const checkpoint = roots.find((root) => root.rootId === input.origin.checkpointWorkspaceRootId); + if (checkpoint === undefined) { + throw new CommandError("malformed-member"); + } + const directories = new Set(); + for (const entry of parseWorkspaceRootManifest(checkpoint.manifest, () => { + throw new CommandError("malformed-member"); + }).entries) { + if (entry.kind === "directory") { + directories.add(entry.path); + } + } + // The fork's live Workspace is the checkpoint's, named before a single + // journal row names a root. + storage.sql.exec( + "UPDATE workspace_state SET current_root_id = ? WHERE singleton_id = 1", + input.origin.checkpointWorkspaceRootId, + ); + + const repositories = new Set(); + for (const part of section(storage, acquisitionId, "checkouts", input.counts.checkouts)) { + writeCheckout(storage, part, directories, repositories); + } + + // The two records the fork writes for itself stand where the source's stood, + // against the same roots, under identities of this run's own. + storage.sql.exec( + "INSERT INTO journal_events (event_id, record, workspace_root_id) VALUES (?, ?, ?)", + mintEventId(), + serializeDurableEvent(input.runRecord), + input.origin.runRecordWorkspaceRootId, + ); + storage.sql.exec( + "INSERT INTO journal_events (event_id, record, workspace_root_id) VALUES (?, ?, ?)", + mintEventId(), + serializeDurableEvent(input.rootImport), + input.origin.rootImportWorkspaceRootId, + ); + for (const part of section(storage, acquisitionId, "inherited", input.counts.inherited)) { + writeInherited(storage, part, input.origin.sourceRunId, carried); + } + + storage.sql.exec( + `INSERT INTO workflow_fork_lineage + (id, source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, created_at) + VALUES (1, ?, ?, ?, ?)`, + input.origin.sourceRunId, + input.origin.checkpointEventId, + input.origin.checkpointWorkspaceRootId, + now(), + ); + + return { conflict: null, value: begunOn(storage, acquisitionId, input, now) }; +} + +/** The fork's first execution, begun by the acquisition that committed it. */ +function begunOn( + storage: OwnerStorage, + acquisitionId: string, + input: { readonly runId: string; readonly executionId: string }, + now: () => string, +): ForkedValue { + storage.sql.exec( + "INSERT INTO document_executions (execution_id, started_at) VALUES (?, ?)", + input.executionId, + now(), + ); + holdExecution(storage, acquisitionId, input.executionId); + storage.sql.exec( + "UPDATE workflow_run SET status = 'running', updated_at = ? WHERE id = 1", + now(), + ); + const row = rows( + storage, + `SELECT execution_id, started_at, stopped_at, stop_status, + stop_reason_kind, stop_reason_code, stop_reason_event_id + FROM document_executions WHERE execution_id = ?`, + input.executionId, + )[0]; + if (row === undefined) { + throw new CommandError("malformed-member"); + } + return { + frontier: readFrontier(storage, input.runId), + execution: readDocumentExecution(row), + }; +} + +/** What a destination already holds, when a retry finds one. */ +export function retainedFork(storage: OwnerStorage, runId: string): boolean { + const row = rows(storage, "SELECT run_id FROM workflow_run")[0]; + return row !== undefined && readRunRecord(row).runId === runId; +} + +/** Discard this acquisition's offered parts once they have been adopted. */ +export function discardForkParts(storage: OwnerStorage, acquisitionId: string): void { + storage.sql.exec(`DELETE FROM ${FORK_TABLE} WHERE acquisition_id = ?`, acquisitionId); +} diff --git a/packages/workflow/src/cloudflare/owner-lifecycle.ts b/packages/workflow/src/cloudflare/owner-lifecycle.ts index 2040c28ac..3d1332531 100644 --- a/packages/workflow/src/cloudflare/owner-lifecycle.ts +++ b/packages/workflow/src/cloudflare/owner-lifecycle.ts @@ -40,6 +40,7 @@ import type { OwnerStorage } from "./storage.ts"; import type { OwnerTransaction } from "./owner-transaction.ts"; import { establishRun } from "./owner-open.ts"; import { readFrontier, type FrontierValue } from "./owner-reads.ts"; +import { heldExecution, holdExecution, releaseExecution } from "./private-schema.ts"; const RUN_COLUMNS = `run_id, definition, base, props, status, stop_reason_kind, stop_reason_code, stop_reason_event_id, created_at, updated_at`; @@ -235,6 +236,7 @@ function reconcile( export function beginRun( storage: OwnerStorage, transaction: OwnerTransaction, + acquisitionId: string, runId: string, action: "start" | "resume", creation: CreateWorkflowRunRequest | null, @@ -252,6 +254,13 @@ export function beginRun( } return (() => { + if (heldExecution(storage, acquisitionId) !== undefined) { + // One acquisition begins one execution. Its own unfinished execution is + // not somebody else's leftovers, so this is refused rather than + // recovered. Asked once the store is known to exist, because a store + // that holds nothing holds no acquisition either. + throw new CommandError("duplicate-conflict"); + } const stored = storedRun(storage); if (stored.runId !== runId) { throw new CommandError("wrong-run"); @@ -277,6 +286,10 @@ export function beginRun( const decision = beginDecision(recovered.status); const execution = insertExecution(storage, executionId, now); + // Recorded here rather than only in the runner's hold: the owner is what a + // settlement is checked against, and an evicted object keeps its sockets + // but forgets everything that was not written down. + holdExecution(storage, acquisitionId, executionId); if (decision.kind === "running") { publish(storage, "running", undefined, now); } @@ -302,6 +315,7 @@ export function beginRun( */ export function settleRun( storage: OwnerStorage, + acquisitionId: string, runId: string, completion: DocumentExecutionCompletion, expectedWorkspaceRootId: string, @@ -319,6 +333,11 @@ export function settleRun( if (current === undefined || String(current["current_root_id"]) !== expectedWorkspaceRootId) { throw new CommandError("stale-root"); } + if (heldExecution(storage, acquisitionId) !== completion.executionId) { + // Either this acquisition began nothing, or it began something else. + // Naming an execution is not the same as having begun it. + throw new CommandError("wrong-execution"); + } const execution = readExecution(storage, completion.executionId); if (execution.stoppedAt !== undefined) { // Already settled. Settling it again would replace an outcome that won. @@ -338,6 +357,9 @@ export function settleRun( if (!terminal(stored.status)) { publish(storage, completion.status, completion.reason, now); } + // Finished, so this acquisition holds no execution any more. It does not + // get another: what it may do next is read, and let go. + releaseExecution(storage, acquisitionId); return readFrontier(storage, runId); })(); } diff --git a/packages/workflow/src/cloudflare/owner-open.ts b/packages/workflow/src/cloudflare/owner-open.ts index b3f799fc6..c8641f6df 100644 --- a/packages/workflow/src/cloudflare/owner-open.ts +++ b/packages/workflow/src/cloudflare/owner-open.ts @@ -26,12 +26,7 @@ import { EMPTY_WORKSPACE_MANIFEST, WORKSPACE_ROOT_DOMAIN } from "../workspace/ro import { WORKSPACE_ROOT_FORMAT } from "../workspace/root-manifest.ts"; import { sha256Hex } from "./encoding.ts"; import { CommandError } from "./commands.ts"; -import { - declaredObjects, - initializeInside, - initializeObject, - recognizeObject, -} from "./recognition.ts"; +import { holdsNoRun, initializeInside, initializeObject, recognizeObject } from "./recognition.ts"; import type { OwnerStorage } from "./storage.ts"; import type { OwnerTransaction, OwnerTransactions } from "./owner-transaction.ts"; import { readFrontier, type FrontierValue } from "./owner-reads.ts"; @@ -135,7 +130,10 @@ function insertRun(storage: OwnerStorage, creation: CreateWorkflowRunRequest, st /** Whether this object holds nothing at all yet. */ function pristine(storage: OwnerStorage): boolean { - return declaredObjects(storage).length === 0; + // Scratch this adapter wrote is not a run. A destination that was offered a + // fork's parts before the fork was committed holds exactly that and nothing + // else, and creating its run has to be able to proceed. + return holdsNoRun(storage); } /** diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index dbc0f2952..b17902e26 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -45,6 +45,7 @@ import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { discardPriorAcquisitions, PRIVATE_OBJECT_NAMES } from "./private-schema.ts"; import { declaredObjects, + holdsNoRun, initializeObject, isPristine, recognizeObject, @@ -164,7 +165,13 @@ export abstract class WorkflowOwnerObject extends DurableObject { return acquireExecutor(this.ctx, socket, runId, acquisitionId, () => { const names = new Set(declaredObjects(this.owned).map((object) => object.name)); if (PRIVATE_OBJECT_NAMES.every((name) => names.has(name))) { - recognizeObject(this.owned); + // A store that holds nothing but this adapter's scratch holds no run + // to recognize — a fork was offered parts here and never committed. + // The scratch still goes, because it belonged to a connection that is + // gone. + if (!holdsNoRun(this.owned)) { + recognizeObject(this.owned); + } this.transactions.run(this.owned, () => { discardPriorAcquisitions(this.owned, acquisitionId); }); diff --git a/packages/workflow/src/cloudflare/private-schema.ts b/packages/workflow/src/cloudflare/private-schema.ts index ee3b6ae39..2855634b6 100644 --- a/packages/workflow/src/cloudflare/private-schema.ts +++ b/packages/workflow/src/cloudflare/private-schema.ts @@ -48,6 +48,32 @@ export const COMMAND_TABLE = "_xmd_executor_commands"; export const MUTATION_TABLE = "_xmd_run_mutations"; export const STAGING_TABLE = "_xmd_executor_staging"; +/** + * Which execution each acquisition began, as the owner knows it. + * + * The runner remembers this too, in the hold it issued, but a runner's memory + * is not authority: a settlement arrives naming an execution, and what decides + * whether this caller may finish it is what the owner retained when that + * execution began. Kept here rather than in a field because an evicted Durable + * Object forgets fields and keeps its sockets, so the association has to be + * where the next message can still find it. + * + * One row per acquisition: one acquisition begins one execution. The row is + * this connection's, so a replacement acquisition discards it and cannot adopt + * the execution it named. + */ +export const HOLD_TABLE = "_xmd_executor_holds"; + +/** + * The parts of a fork one acquisition has offered, before any of it is a run. + * + * A fork source is larger than one message may be, so it crosses in bounded + * parts that name where they belong in the selection, and the final command + * commits them together. Like staged content these are scratch: nothing reads + * them, nothing inherits them, and a replacement acquisition throws them away. + */ +export const FORK_TABLE = "_xmd_executor_fork_parts"; + const COMMAND_SQL = `CREATE TABLE ${COMMAND_TABLE} ( acquisition_id TEXT NOT NULL, command_id TEXT NOT NULL, @@ -70,6 +96,22 @@ const STAGING_SQL = `CREATE TABLE ${STAGING_TABLE} ( PRIMARY KEY (acquisition_id, kind, digest) ) STRICT, WITHOUT ROWID`; +const HOLD_SQL = `CREATE TABLE ${HOLD_TABLE} ( + acquisition_id TEXT PRIMARY KEY, + execution_id TEXT NOT NULL +) STRICT, WITHOUT ROWID`; + +const FORK_SQL = `CREATE TABLE ${FORK_TABLE} ( + acquisition_id TEXT NOT NULL, + section TEXT NOT NULL CHECK ( + section IN ('inherited', 'roots', 'manifests', 'blobs', 'checkouts') + ), + position INTEGER NOT NULL CHECK (position >= 0), + part TEXT NOT NULL CHECK (json_valid(part)), + part_bytes INTEGER NOT NULL CHECK (part_bytes > 0), + PRIMARY KEY (acquisition_id, section, position) +) STRICT, WITHOUT ROWID`; + const MUTATION_SQL = `CREATE TABLE ${MUTATION_TABLE} ( command_id TEXT PRIMARY KEY, request_fingerprint TEXT NOT NULL CHECK ( @@ -83,12 +125,32 @@ const PRIVATE_OBJECTS = new Map([ [COMMAND_TABLE, { type: "table", sql: COMMAND_SQL }], [STAGING_TABLE, { type: "table", sql: STAGING_SQL }], [MUTATION_TABLE, { type: "table", sql: MUTATION_SQL }], + [HOLD_TABLE, { type: "table", sql: HOLD_SQL }], + [FORK_TABLE, { type: "table", sql: FORK_SQL }], ]); export const PRIVATE_OBJECT_NAMES: readonly string[] = Object.freeze([...PRIVATE_OBJECTS.keys()]); export function initializePrivateSchema(storage: OwnerStorage): void { - storage.sql.exec(`${COMMAND_SQL};\n\n${STAGING_SQL};\n\n${MUTATION_SQL};`); + if (privateSchemaPresent(storage)) { + // Already here, because a transfer was offered before the run it belongs + // to existed. The scratch is this adapter's own and is not rewritten. + return; + } + storage.sql.exec( + `${COMMAND_SQL};\n\n${STAGING_SQL};\n\n${MUTATION_SQL};\n\n${HOLD_SQL};\n\n${FORK_SQL};`, + ); +} + +/** Whether this adapter's own scratch tables are already declared here. */ +export function privateSchemaPresent(storage: OwnerStorage): boolean { + const names = new Set( + storage.sql + .exec("SELECT name FROM sqlite_schema WHERE type = 'table'") + .toArray() + .map((row) => String(row["name"])), + ); + return PRIVATE_OBJECT_NAMES.every((name) => names.has(name)); } export function privateStructureFailure( @@ -118,4 +180,43 @@ export function privateStructureFailure( export function discardPriorAcquisitions(storage: OwnerStorage, acquisitionId: string): void { storage.sql.exec(`DELETE FROM ${COMMAND_TABLE} WHERE acquisition_id <> ?`, acquisitionId); storage.sql.exec(`DELETE FROM ${STAGING_TABLE} WHERE acquisition_id <> ?`, acquisitionId); + // A previous connection's fork parts describe a transfer nobody is going to + // finish, and its execution association belonged to a connection that can no + // longer settle anything. Neither is inherited: what an earlier executor left + // unfinished is decided by recovery, from what the run itself retains. + storage.sql.exec(`DELETE FROM ${FORK_TABLE} WHERE acquisition_id <> ?`, acquisitionId); + storage.sql.exec(`DELETE FROM ${HOLD_TABLE} WHERE acquisition_id <> ?`, acquisitionId); +} + +/** Which execution this acquisition began, when it has begun one. */ +export function heldExecution(storage: OwnerStorage, acquisitionId: string): string | undefined { + const row = storage.sql + .exec(`SELECT execution_id FROM ${HOLD_TABLE} WHERE acquisition_id = ?`, acquisitionId) + .toArray()[0]; + const held = row?.["execution_id"]; + return typeof held === "string" ? held : undefined; +} + +/** + * Record that this acquisition began this execution. + * + * Refuses a second one. An acquisition begins one execution, and the owner is + * where that is decided: a runner that lost track of its own hold cannot talk + * this store into holding two. + */ +export function holdExecution( + storage: OwnerStorage, + acquisitionId: string, + executionId: string, +): void { + storage.sql.exec( + `INSERT INTO ${HOLD_TABLE} (acquisition_id, execution_id) VALUES (?, ?)`, + acquisitionId, + executionId, + ); +} + +/** Let go of the execution this acquisition began, once it is finished. */ +export function releaseExecution(storage: OwnerStorage, acquisitionId: string): void { + storage.sql.exec(`DELETE FROM ${HOLD_TABLE} WHERE acquisition_id = ?`, acquisitionId); } diff --git a/packages/workflow/src/cloudflare/read-client.ts b/packages/workflow/src/cloudflare/read-client.ts index 7a4f9d3f8..07c49f198 100644 --- a/packages/workflow/src/cloudflare/read-client.ts +++ b/packages/workflow/src/cloudflare/read-client.ts @@ -579,6 +579,7 @@ function* collectForkSource( validateCheckouts(checkouts, roots, checkpointWorkspaceRootId); return Object.freeze({ sourceRunId, + anchor: anchor ?? fail("it answered no page of the selection"), checkpointEventId, checkpointWorkspaceRootId, runRecordWorkspaceRootId, diff --git a/packages/workflow/src/cloudflare/recognition.ts b/packages/workflow/src/cloudflare/recognition.ts index 19e68ab42..87b6fa2a8 100644 --- a/packages/workflow/src/cloudflare/recognition.ts +++ b/packages/workflow/src/cloudflare/recognition.ts @@ -80,6 +80,18 @@ export function isPristine(objects: readonly SchemaObject[]): boolean { return objects.length === 0; } +/** + * Whether this store holds a run yet, ignoring this adapter's own scratch. + * + * A destination that has been offered a fork's parts is not pristine any more — + * the private tables are there to hold them — but it holds no run, and the + * command that makes one has to be able to say so. Anything else declared here + * belongs to something else and is not covered by this. + */ +export function holdsNoRun(storage: OwnerStorage): boolean { + return declaredObjects(storage).every((object) => PRIVATE_OBJECT_NAMES.includes(object.name)); +} + function markerRows(storage: OwnerStorage): Record[] { return storage.sql.exec(`SELECT application_id, schema_version FROM ${MARKER_TABLE}`).toArray(); } diff --git a/packages/workflow/src/remote/lifecycle-link.ts b/packages/workflow/src/remote/lifecycle-link.ts new file mode 100644 index 000000000..dc67b157a --- /dev/null +++ b/packages/workflow/src/remote/lifecycle-link.ts @@ -0,0 +1,121 @@ +/** + * The lifecycle half of one run's connection. + * + * Beginning, settling, cancelling and forking are the moments a run's own state + * changes, and they travel on the same connection as everything else that + * carries authority. This is that half of the link, stated where nothing knows + * which host answers: a provider composes these with the Workspace half it + * already has, and the adapter underneath decides how a command is spelled. + * + * Every answer here is already this build's own vocabulary. A refusal spelling, + * a command name, a socket or a row never reaches a caller through one of + * these; what reaches a caller is a `Result` of values the shared record + * parsers produced. + */ + +import type { Operation, Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import type { + DocumentExecutionCompletion, + DocumentExecutionRecord, + WorkflowRunRecord, +} from "../storage/record.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import type { RemoteFrontierSnapshot } from "./read.ts"; +import type { RemoteWorkspaceLink } from "./database.ts"; + +/** What one begin committed, as the runner is allowed to know it. */ +export interface RemoteBegun { + readonly frontier: RemoteFrontierSnapshot; + readonly execution: DocumentExecutionRecord; + readonly replay: boolean; + /** What stale recovery closed on the way in, when it closed anything. */ + readonly recovered: DocumentExecutionRecord | null; +} + +/** + * A lifecycle answer that may decline for a reason about the run. + * + * `refused` carries which condition applied — a cancelled run, a failed one, a + * terminal one — because a caller acts on the difference. What the run holds + * never travels with it. + */ +export type RemoteLifecycleAnswer = + | { readonly kind: "performed"; readonly value: T } + | { readonly kind: "refused"; readonly refusal: "cancelled" | "resume-failed" | "terminal" }; + +/** Which committed checkpoint of which run a fork continues. */ +export interface RemoteForkOrigin { + readonly sourceRunId: string; + readonly checkpointEventId: string; + readonly checkpointWorkspaceRootId: string; + readonly runRecordWorkspaceRootId: string; + readonly rootImportWorkspaceRootId: string; + readonly anchor: string; +} + +/** How many parts of each section a committed fork should find staged. */ +export interface RemoteForkCounts { + readonly inherited: number; + readonly roots: number; + readonly checkouts: number; +} + +/** One part of a fork's source, offered before any of it is a run. */ +export interface RemoteForkPart { + readonly section: "inherited" | "roots" | "checkouts"; + readonly position: number; + readonly part: Record; +} + +/** Everything one committed fork is decided from. */ +export interface RemoteForkCommit { + readonly runId: string; + readonly creation: CreateWorkflowRunRequest; + readonly origin: RemoteForkOrigin; + readonly counts: RemoteForkCounts; + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; + readonly executionId: string; +} + +/** + * One admitted executor connection, as the host arranges it. + * + * The two halves are the same authority and arrive together: a link that reads + * and commits, and the lifecycle commands that move the run. A host that + * returned them from different owners would be handing out an acquisition of + * one run that mutates another, so they are one value. + */ +export interface RemoteExecutorConnection { + readonly link: RemoteWorkspaceLink; + readonly lifecycle: RemoteLifecycleLink; +} + +/** + * The lifecycle commands one admitted connection may perform. + * + * Each is one owner transaction, and each is retried by identity rather than + * repeated: the same command id and the same content is the same request, and + * an owner that already decided it answers with what it decided. + */ +export interface RemoteLifecycleLink { + /** Begin one document execution under this acquisition. */ + begin(request: { + readonly runId: string; + readonly action: "start" | "resume"; + readonly creation: CreateWorkflowRunRequest | null; + readonly executionId: string; + }): Operation>>; + /** Finish the execution this acquisition began. */ + settle( + completion: DocumentExecutionCompletion, + expectedWorkspaceRootId: string, + ): Operation>; + /** Make this run terminal, following what it retains. */ + cancel(runId: string): Operation>>; + /** Offer one part of a fork's source to this acquisition's scratch. */ + stageForkPart(part: RemoteForkPart): Operation>; + /** Commit the offered parts as one destination run and its first execution. */ + commitFork(commit: RemoteForkCommit): Operation>; +} diff --git a/packages/workflow/src/remote/lifecycle.ts b/packages/workflow/src/remote/lifecycle.ts new file mode 100644 index 000000000..52e7d1a6f --- /dev/null +++ b/packages/workflow/src/remote/lifecycle.ts @@ -0,0 +1,490 @@ +/** + * The remote provider's half of the executor lifecycle. + * + * Taking the lock is opening a connection: an admitted socket *is* the + * acquisition, so the lock this hands back is an object issued beside one exact + * connection, held in this provider's own closure, and recognized by identity. + * A run id, a copy of the object, another provider's lock or a lock whose + * connection has closed authorizes nothing, and nothing about it is checked by + * comparing fields. + * + * ## The connection is the lifetime + * + * The connection belongs to the scope that asked for it. When that scope ends — + * normally, by cancellation, or because the socket failed — the hold is retired + * and the connection closes once. Nothing expires: there is no lease, no + * heartbeat and no elapsed time anywhere in this file. An acquisition ends when + * its connection does. + * + * ## One acquisition begins one execution + * + * The hold remembers which execution this acquisition began, and so does the + * owner. Both are needed: the runner's copy refuses a second begin before a + * message is sent, and the owner's copy is what a settlement is actually + * checked against, because a runner that lost track of its own hold must not be + * able to finish an execution it never began. + */ + +import { Err, ensure, Ok, type Operation, type Result, scoped } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { + type ExecutorAcquisition, + type ExecutorLock, + WorkflowLifecycle, +} from "../lifecycle/api.ts"; +import type { + WorkflowBeginRequest, + WorkflowExecutionBegun, + WorkflowExecutionTransitions, + WorkflowForkRequest, + WorkflowRunCreation, +} from "../lifecycle/execution.ts"; +import type { WorkflowRunDatabase } from "../storage/api.ts"; +import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import type { DocumentExecutionCompletion, WorkflowRunRecord } from "../storage/record.ts"; +import { + WorkflowRequestError, + WorkflowRunNotFoundError, + type WorkflowStorageError, +} from "../storage/errors.ts"; +import { useRemoteRunDatabase } from "./database.ts"; +import type { RemoteForkSource, RemoteReadPlane } from "./read.ts"; +import { forkRunRecordEvent } from "../fork.ts"; +import type { + RemoteBegun, + RemoteExecutorConnection, + RemoteForkPart, + RemoteLifecycleLink, +} from "./lifecycle-link.ts"; + +export type { RemoteExecutorConnection }; + +/** + * What this provider needs from its host, and nothing more. + * + * Narrow on purpose: reaching an owner, reading a source without acquiring it, + * and assembling a candidate somewhere local are host arrangements. None of + * them is a registry, none enumerates anything, and none is public API. + */ +export interface RemoteLifecycleHost { + /** + * Admit one executor connection for this run, owned by the calling scope. + * + * Answers `already-running` when the owner refuses because another live + * executor holds the run — a fact about the run, not a failure of this call. + */ + admit(runId: string): Operation>; + /** The no-acquisition read plane for one run, for copy and inspection only. */ + source(runId: string): Operation>; + /** + * Assemble one fork candidate in runner-local disposable storage. + * + * Takes no acquisition and creates nothing a host would discover. What it + * returns belongs to the calling scope and goes when that scope ends. + */ + stage( + request: WorkflowForkRequest, + source: RemoteForkSource, + head: { readonly runRecord: DurableEvent; readonly rootImport: DurableEvent }, + ): Operation>; + /** Fresh identities for this provider's own commands and executions. */ + readonly ids: { readonly execution: () => string }; +} + +/** What one issued lock is allowed to do, and what it has already done. */ +interface Hold { + readonly runId: string; + readonly connection: RemoteExecutorConnection; + /** Which execution this acquisition began, once it has begun one. */ + execution: string | undefined; + /** Whether the connection this lock was issued beside is still open. */ + live: boolean; +} + +/** + * Install the remote executor lifecycle, and hand back its transitions. + * + * The transitions are returned rather than installed: they hand out an open + * database, which is a transport, and a capability like that belongs to the + * executor that already holds the lock rather than to a contextual surface + * anything in the process can reach. + */ +export function* useRemoteLifecycle( + host: RemoteLifecycleHost, +): Operation { + // Keyed by the object itself: two locks are the same lock when they are the + // same object, and nothing about their fields is consulted. + const held = new Map(); + + function hold(lock: ExecutorLock): Hold | undefined { + // Fabricated, copied, foreign, released and closed locks all answer + // `undefined` here — before a read plane, an owner command or a database is + // touched. + const found = held.get(lock); + return found === undefined || !found.live ? undefined : found; + } + + function* acquireExecutor(runId: string): Operation> { + if (runId === "") { + return Err(new WorkflowRequestError("a workflow run id cannot be empty.")); + } + const admitted = yield* host.admit(runId); + if (!admitted.ok) { + return admitted; + } + if (admitted.value === "already-running") { + return Ok({ kind: "already-running" }); + } + const connection = admitted.value; + const lock: ExecutorLock = Object.freeze({ runId }); + const record: Hold = { + runId, + connection, + execution: undefined, + live: true, + }; + held.set(lock, record); + // Registered before it is returned, and retired when the scope that asked + // for it ends — the same scope that owns the connection underneath. + yield* ensure(function* () { + record.live = false; + held.delete(lock); + }); + return Ok({ kind: "acquired", lock }); + } + + function* cancel(runId: string): Operation> { + if (runId === "") { + return Err(new WorkflowRequestError("a workflow run id cannot be empty.")); + } + // Cancellation takes an acquisition of its own for exactly this operation + // and gives it back. It never reads the no-acquisition plane to decide + // whether it may proceed: that plane holds no authority. + return yield* scoped(function* () { + const admitted = yield* host.admit(runId); + if (!admitted.ok) { + return admitted; + } + if (admitted.value === "already-running") { + return Err( + new WorkflowRequestError( + "a live workflow executor holds this run, so it cannot be cancelled from here.", + ), + ); + } + const answered = yield* admitted.value.lifecycle.cancel(runId); + if (!answered.ok) { + return answered; + } + if (answered.value.kind === "refused") { + return Err( + new WorkflowRequestError( + answered.value.refusal === "terminal" + ? "this run already reached a terminal outcome, so it cannot be cancelled." + : "this run cannot be cancelled from the state it is in.", + ), + ); + } + return Ok(answered.value.value); + }); + } + + yield* WorkflowLifecycle.around({ + *acquireExecutor([runId]): Operation> { + return yield* acquireExecutor(runId); + }, + *cancel([runId]): Operation> { + return yield* cancel(runId); + }, + }); + + return transitions(host, hold); +} + +/** What an unrecognized lock answers, wherever one is offered. */ +function unauthorized(): WorkflowRequestError { + return new WorkflowRequestError( + "this executor lock was not issued by this provider, or its acquisition has ended.", + ); +} + +function transitions( + host: RemoteLifecycleHost, + hold: (lock: ExecutorLock) => Hold | undefined, +): WorkflowExecutionTransitions { + return { + *begin( + lock: ExecutorLock, + request: WorkflowBeginRequest, + ): Operation> { + const held = hold(lock); + if (held === undefined) { + return Err(unauthorized()); + } + if (request.runId !== held.runId) { + return Err( + new WorkflowRequestError("this executor lock was issued for a different workflow run."), + ); + } + if (held.execution !== undefined) { + return Err( + new WorkflowRequestError( + "this executor lock has already begun a document execution. One acquisition begins one.", + ), + ); + } + if (request.action === "resume" && request.creation !== undefined) { + return Err(new WorkflowRequestError("a resume does not carry a creation.")); + } + // Minted once, outside anything that could retry: the owner recognizes a + // repeat by this identity, and a fresh one would be a second execution. + const executionId = host.ids.execution(); + const answered = yield* held.connection.lifecycle.begin({ + runId: request.runId, + action: request.action, + creation: creationOf(request.runId, request.creation), + executionId, + }); + if (!answered.ok) { + return answered; + } + if (answered.value.kind === "refused") { + return Err(refusalError(answered.value.refusal, request.runId)); + } + held.execution = answered.value.value.execution.executionId; + return Ok(yield* begun(held, answered.value.value)); + }, + + *settle( + lock: ExecutorLock, + completion: DocumentExecutionCompletion, + ): Operation> { + const held = hold(lock); + if (held === undefined) { + return Err(unauthorized()); + } + if (held.execution !== completion.executionId) { + // Nothing is sent. An execution this acquisition did not begin is not + // this acquisition's to finish, and asking would be asking about + // somebody else's work. + return Err( + new WorkflowRequestError( + "this executor lock did not begin the document execution it is settling.", + ), + ); + } + // The root the owner is held to comes from the same connection-owned + // frontier the execution ran against, after the host has torn down. + const frontier = yield* held.connection.link.frontierSnapshot(); + const answered = yield* held.connection.lifecycle.settle( + completion, + frontier.workspaceRootId, + ); + if (!answered.ok) { + return answered; + } + held.execution = undefined; + return Ok(answered.value.record); + }, + + *fork( + lock: ExecutorLock, + request: WorkflowForkRequest, + ): Operation> { + const held = hold(lock); + if (held === undefined) { + return Err(unauthorized()); + } + if (request.runId !== held.runId) { + return Err( + new WorkflowRequestError("this executor lock was issued for a different workflow run."), + ); + } + if (held.execution !== undefined) { + return Err( + new WorkflowRequestError( + "this executor lock has already begun a document execution. One acquisition begins one.", + ), + ); + } + const source = yield* readSource(host, request); + if (!source.ok) { + return source; + } + const staged = yield* offer(held.connection.lifecycle, source.value); + if (!staged.ok) { + return staged; + } + const executionId = host.ids.execution(); + const answered = yield* held.connection.lifecycle.commitFork({ + runId: request.runId, + creation: creationRequest(request.runId, request.creation), + origin: { + sourceRunId: source.value.sourceRunId, + checkpointEventId: source.value.checkpointEventId, + checkpointWorkspaceRootId: source.value.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: source.value.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: source.value.rootImportWorkspaceRootId, + anchor: source.value.anchor, + }, + counts: { + inherited: source.value.inherited.length, + roots: source.value.roots.length, + checkouts: source.value.checkouts.length, + }, + runRecord: headOf(request).runRecord, + rootImport: request.rootImport, + executionId, + }); + if (!answered.ok) { + return answered; + } + held.execution = answered.value.execution.executionId; + return Ok(yield* begun(held, answered.value)); + }, + + *stageFork(request: WorkflowForkRequest): Operation> { + const source = yield* readSource(host, request); + if (!source.ok) { + return source; + } + // No destination acquisition, no destination owner, nothing a host would + // discover: the candidate is assembled locally and belongs to the scope + // that asked for it. + return yield* host.stage(request, source.value, headOf(request)); + }, + }; +} + +function* begun(held: Hold, answer: RemoteBegun): Operation { + const database = yield* useRemoteRunDatabase(held.connection.link, answer.frontier); + return { + database, + record: answer.frontier.record, + execution: answer.execution, + replay: answer.replay, + ...(answer.recovered === null ? {} : { recovered: answer.recovered }), + }; +} + +/** + * The two records a fork writes for itself. + * + * Its own run record, because the fork is its own run and the source's record + * describes the source; and the root import its own definition produced, + * because a fork that inherited the source's would run the source's document. + */ +function headOf(request: WorkflowForkRequest): { + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; +} { + return { + runRecord: forkRunRecordEvent({ + runId: request.runId, + base: request.creation.base, + pinnedCommit: request.creation.definition.objectId, + }), + rootImport: request.rootImport, + }; +} + +/** Read one source through the accepted no-acquisition plane. */ +function* readSource( + host: RemoteLifecycleHost, + request: WorkflowForkRequest, +): Operation> { + if (request.selection.sourceRunId === "" || request.selection.checkpointEventId === "") { + return Err(new WorkflowRequestError("a fork names one source run and one checkpoint.")); + } + const plane = yield* host.source(request.selection.sourceRunId); + if (!plane.ok) { + return plane; + } + if (plane.value.runId !== request.selection.sourceRunId) { + return Err(new WorkflowRunNotFoundError(request.selection.sourceRunId)); + } + return yield* plane.value.forkSource(request.selection.checkpointEventId); +} + +/** + * Offer the whole snapshot as bounded parts, in the order it will be read back. + * + * Content crosses through the staging the publication path already uses, and + * the rest — the roots, the inherited rows, the checkouts — crosses as parts + * that name where they belong. Nothing here is a run: the final command decides + * whether these add up to one. + */ +function* offer(lifecycle: RemoteLifecycleLink, source: RemoteForkSource): Operation> { + const parts: RemoteForkPart[] = []; + source.roots.forEach((root, position) => { + parts.push({ + section: "roots", + position, + part: { + rootId: root.rootId, + formatVersion: root.formatVersion, + manifest: root.manifest, + manifestHashes: [...root.manifestHashes], + blobHashes: [...root.blobHashes], + }, + }); + }); + source.inherited.forEach((row, position) => { + parts.push({ + section: "inherited", + position, + part: { eventId: row.eventId, record: row.record, workspaceRootId: row.workspaceRootId }, + }); + }); + source.checkouts.forEach((checkout, position) => { + parts.push({ section: "checkouts", position, part: { ...checkout } }); + }); + for (const part of parts) { + const staged = yield* lifecycle.stageForkPart(part); + if (!staged.ok) { + return staged; + } + } + return Ok(undefined); +} + +function creationOf( + runId: string, + creation: WorkflowBeginRequest["creation"], +): CreateWorkflowRunRequest | null { + return creation === undefined ? null : creationRequest(runId, creation); +} + +/** + * One creation, as the request storage retains. + * + * The definition and the props travel as the caller built them; what this adds + * is the run they belong to. Normalizing them is the owner's, through the same + * shared parser every host uses, so a remote run's identity is computed exactly + * where a local one's is. + */ +function creationRequest(runId: string, creation: WorkflowRunCreation): CreateWorkflowRunRequest { + return { + runId, + definition: creation.definition, + base: creation.base, + props: creation.props, + }; +} + +function refusalError( + refusal: "cancelled" | "resume-failed" | "terminal", + runId: string, +): WorkflowStorageError { + if (refusal === "cancelled") { + return new WorkflowRequestError(`workflow run ${JSON.stringify(runId)} was cancelled.`); + } + if (refusal === "resume-failed") { + return new WorkflowRequestError( + `workflow run ${JSON.stringify(runId)} failed, so it cannot be resumed.`, + ); + } + return new WorkflowRequestError( + `workflow run ${JSON.stringify(runId)} already reached a terminal outcome.`, + ); +} diff --git a/packages/workflow/src/remote/read.ts b/packages/workflow/src/remote/read.ts index dad7756e7..6d2fee703 100644 --- a/packages/workflow/src/remote/read.ts +++ b/packages/workflow/src/remote/read.ts @@ -195,6 +195,14 @@ export type RemoteCheckout = /** Everything one checkpoint hands a fork, read in one committed selection. */ export interface RemoteForkSource { readonly sourceRunId: string; + /** + * The selection this snapshot was read under. + * + * Every page of it agreed on this, so it identifies the committed state the + * whole snapshot describes. A destination keeps it with what it copied, which + * is what lets a retry say whether it is the same transfer. + */ + readonly anchor: string; readonly checkpointEventId: string; readonly checkpointWorkspaceRootId: string; readonly runRecordWorkspaceRootId: string; diff --git a/packages/workflow/tests/cloudflare/remote-fork.vitest.ts b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts new file mode 100644 index 000000000..bcabd30dd --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts @@ -0,0 +1,314 @@ +/** + * Committing a fork on a real destination owner. + * + * Staging is scratch and the commit is one transaction: what this suite settles + * is that a destination is either absent or whole, that the parts it was built + * from stop being anything the moment they are adopted, and that a transfer + * which does not add up refuses without leaving a half-run behind. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { + BLOB_ID, + DOFS_MANIFEST, + FILE_BYTES, + MANIFEST_ID, + POLICY, + ROOT_ID, + ROOT_MANIFEST, + RUN_ID, + VALID_CLAIMS, +} from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +const SOURCE_RUN_ID = "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`fork-${unique}-${Math.random()}`)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise { + return runInDurableObject(stub, body); +} + +async function connected(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + expect(await on(stub, (owner) => owner.admitConnection({ token }))).toBe("admitted"); +} + +async function ask( + stub: ReturnType, + command: Record, +): Promise> { + const answered = await on(stub, (owner) => owner.sendLatest(JSON.stringify(command))); + if (answered === null || typeof answered !== "object") { + throw new Error("expected one command answer"); + } + return Object.fromEntries(Object.entries(answered)); +} + +function base64(bytes: Uint8Array): string { + let text = ""; + for (const byte of bytes) { + text += String.fromCharCode(byte); + } + return btoa(text); +} + +function event(name: string): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }); +} + +const HEAD = serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "workflow_run", name: "workflow_run", base: "main" }, + result: { + status: "ok", + value: { runId: RUN_ID, base: "main", pinnedCommit: "0".repeat(40) }, + }, +}); + +const IMPORT = serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path: "README.md", content: "# fork" } }, +}); + +/** Offer everything one small source is made of, as the runner would. */ +async function offer(stub: ReturnType, id: () => string): Promise { + expect( + ( + await ask(stub, { + id: id(), + command: "stage", + kind: "manifest", + digest: MANIFEST_ID, + bytes: base64(new TextEncoder().encode(DOFS_MANIFEST)), + }) + )["outcome"], + ).toBe("performed"); + expect( + ( + await ask(stub, { + id: id(), + command: "stage", + kind: "blob", + digest: BLOB_ID, + bytes: base64(FILE_BYTES), + }) + )["outcome"], + ).toBe("performed"); + expect( + ( + await ask(stub, { + id: id(), + command: "fork-stage", + section: "roots", + position: 0, + part: { + rootId: ROOT_ID, + formatVersion: 1, + manifest: ROOT_MANIFEST, + manifestHashes: [MANIFEST_ID], + blobHashes: [BLOB_ID], + }, + }) + )["outcome"], + ).toBe("performed"); + expect( + ( + await ask(stub, { + id: id(), + command: "fork-stage", + section: "inherited", + position: 0, + part: { eventId: "event-work", record: event("work"), workspaceRootId: ROOT_ID }, + }) + )["outcome"], + ).toBe("performed"); +} + +/** The event ids one published snapshot reports, in journal order. */ +function journalOf(published: Record): string[] { + const events = published["events"]; + return Array.isArray(events) + ? events.map((row) => String(Reflect.get(Object(row), "event_id"))) + : []; +} + +function commit(overrides: Record = {}): Record { + return { + id: "fork-commit", + command: "fork", + runId: RUN_ID, + creation: { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + }, + origin: { + sourceRunId: SOURCE_RUN_ID, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: ROOT_ID, + runRecordWorkspaceRootId: ROOT_ID, + rootImportWorkspaceRootId: ROOT_ID, + anchor: "f".repeat(64), + }, + counts: { inherited: 1, roots: 1, checkouts: 0 }, + runRecord: HEAD, + rootImport: IMPORT, + executionId: "execution-1", + ...overrides, + }; +} + +describe("committing a fork on its destination owner", () => { + it("makes the destination whole on pristine storage, in one commit", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + const id = () => `command-${(minted += 1)}`; + await offer(stub, id); + + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + const forked = await ask(stub, commit()); + expect(forked["outcome"]).toBe("performed"); + + const state = await on(stub, (owner) => ({ + run: owner.runRow(), + executions: owner.executionRows(), + workspace: owner.published(), + held: owner.heldExecutions(), + parts: owner.forkParts(), + })); + // The prefix it inherited, its own two head records, its Workspace, its + // lineage and its first execution all arrived together. + expect(state.run?.["status"]).toBe("running"); + expect(state.workspace["currentRootId"]).toBe(ROOT_ID); + expect(journalOf(state.workspace)).toContain("event-work"); + expect(journalOf(state.workspace)).toHaveLength(3); + expect(state.executions).toHaveLength(1); + expect(state.held).toHaveLength(1); + // The parts stopped being anything the moment they were adopted. + expect(state.parts).toEqual([]); + }); + + it("copies the content, so the destination needs no source afterwards", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + + const workspace = await on(stub, (owner) => owner.published()); + // The blob and the reference are the destination's own rows now. + expect(Number(workspace["blobs"])).toBeGreaterThan(0); + expect(Number(workspace["blobRefs"])).toBeGreaterThan(0); + expect(Number(workspace["roots"])).toBeGreaterThan(0); + }); + + it("refuses a transfer that does not add up, and leaves nothing behind", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + + // One more inherited row than was ever offered. + const refused = await ask(stub, commit({ counts: { inherited: 2, roots: 1, checkouts: 0 } })); + + expect(refused["outcome"]).toBe("refused"); + // No half-run: the destination still holds no schema at all. + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("refuses a head root the transfer never carried", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + + const refused = await ask( + stub, + commit({ + origin: { + sourceRunId: SOURCE_RUN_ID, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: ROOT_ID, + runRecordWorkspaceRootId: "b".repeat(64), + rootImportWorkspaceRootId: ROOT_ID, + anchor: "f".repeat(64), + }, + }), + ); + + expect(refused["outcome"]).toBe("refused"); + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("answers a repeated commit with the destination it already made", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + + const first = await ask(stub, commit()); + const again = await ask(stub, commit()); + + expect(again).toEqual(first); + // One run, one execution, one journal: the retry found the decision. + const state = await on(stub, (owner) => ({ + executions: owner.executionRows(), + workspace: owner.published(), + })); + expect(state.executions).toHaveLength(1); + expect(journalOf(state.workspace)).toHaveLength(3); + }); + + it("keeps one connection's offered parts to itself", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + expect(await on(stub, (owner) => owner.forkParts())).toHaveLength(2); + + // A replacement acquisition inherits nothing its predecessor offered. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + expect(await on(stub, (owner) => owner.forkParts())).toEqual([]); + }); +}); diff --git a/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts new file mode 100644 index 000000000..3ed40c2dd --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts @@ -0,0 +1,271 @@ +/** + * A run's lifecycle on its real owner. + * + * The facts here are the ones only a Durable Object can settle: that a starting + * begin reaches pristine storage and makes the whole run in one transaction, + * that the acquisition which began an execution is the only one that can finish + * it, that the association outlives eviction because it was written down, and + * that a second live executor advances nothing at all. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`lifecycle-${unique}-${Math.random()}`)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise { + return runInDurableObject(stub, body); +} + +/** One owner with its keys configured and one admitted executor connection. */ +async function connected(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + const admitted = await on(stub, (owner) => owner.admitConnection({ token })); + expect(admitted).toBe("admitted"); +} + +function creation(runId = RUN_ID): Record { + return { + runId, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + }; +} + +/** Send one command as the connection admitted most recently. */ +async function ask( + stub: ReturnType, + command: Record, +): Promise> { + const answered = await on(stub, (owner) => owner.sendLatest(JSON.stringify(command))); + if (answered === null || typeof answered !== "object") { + throw new Error("expected one command answer"); + } + return Object.fromEntries(Object.entries(answered)); +} + +async function started( + stub: ReturnType, + id: string, + executionId: string, +): Promise> { + return await ask(stub, { + id, + command: "begin", + runId: RUN_ID, + action: "start", + creation: creation(), + executionId, + }); +} + +describe("a run's lifecycle on its owner", () => { + it("makes the whole run on pristine storage, in one begin", async () => { + const stub = executor(); + await connected(stub); + + const before = await on(stub, (owner) => owner.objectCount()); + const answered = await started(stub, "command-1", "execution-1"); + + expect(answered["outcome"]).toBe("performed"); + // Everything a run owns appeared together: the schema, the marker, the run + // record, its starting Workspace, the first execution and `running`. + expect(before).toBe(0); + const state = await on(stub, (owner) => ({ + run: owner.runRow(), + executions: owner.executionRows(), + schema: owner.hasWorkflowSchema(), + })); + expect(state.schema).toBe(true); + expect(state.run?.["status"]).toBe("running"); + expect(state.executions).toHaveLength(1); + expect(state.executions[0]?.["execution_id"]).toBe("execution-1"); + }); + + it("refuses a second live executor, and that executor advances nothing", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + const before = await on(stub, (owner) => owner.runRow()); + + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + const second = await on(stub, (owner) => owner.admitConnection({ token })); + + expect(second).toBe("acquisition:already-running"); + expect(await on(stub, (owner) => owner.runRow())).toEqual(before); + expect(await on(stub, (owner) => owner.holders())).toBe(1); + }); + + it("lets only the acquisition that began an execution settle it", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + const root = await on(stub, (owner) => owner.currentRootId()); + + const foreign = await ask(stub, { + id: "command-2", + command: "settle", + completion: { executionId: "execution-elsewhere", status: "completed" }, + expectedWorkspaceRootId: root, + }); + expect(foreign).toEqual({ + id: "command-2", + outcome: "refused", + refusal: "command:wrong-execution", + }); + + const settled = await ask(stub, { + id: "command-3", + command: "settle", + completion: { executionId: "execution-1", status: "completed" }, + expectedWorkspaceRootId: root, + }); + expect(settled["outcome"]).toBe("performed"); + expect((await on(stub, (owner) => owner.runRow()))?.["status"]).toBe("completed"); + }); + + it("holds the association where an evicted object can still find it", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + + // Nothing in memory survives eviction; what the settlement is checked + // against has to be retained, so this proves it was. + const held = await on(stub, (owner) => owner.heldExecutions()); + expect(held).toHaveLength(1); + expect(held[0]?.["execution_id"]).toBe("execution-1"); + }); + + it("begins one execution per acquisition", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + + const again = await ask(stub, { + id: "command-2", + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + executionId: "execution-2", + }); + + expect(again).toEqual({ + id: "command-2", + outcome: "refused", + refusal: "command:duplicate-conflict", + }); + expect(await on(stub, (owner) => owner.executionRows())).toHaveLength(1); + }); + + it("answers a repeated command with the decision it already made", async () => { + const stub = executor(); + await connected(stub); + const first = await started(stub, "command-1", "execution-1"); + const again = await started(stub, "command-1", "execution-1"); + + expect(again).toEqual(first); + // One execution, not two: the retry found the decision rather than + // applying it a second time. + expect(await on(stub, (owner) => owner.executionRows())).toHaveLength(1); + }); + + it("refuses a repeat that carries different content", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + + const changed = await ask(stub, { + id: "command-1", + command: "begin", + runId: RUN_ID, + action: "start", + creation: creation(), + executionId: "execution-2", + }); + + expect(changed["refusal"]).toBe("command:duplicate-conflict"); + }); + + it("closes what a lost executor left, before the next one begins", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + + // The connection is gone; nothing about time says so, and nothing needs to. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const resumed = await ask(stub, { + id: "command-2", + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + executionId: "execution-2", + }); + + expect(resumed["outcome"]).toBe("performed"); + const executions = await on(stub, (owner) => owner.executionRows()); + expect(executions).toHaveLength(2); + // The stale one was finished as interrupted; the replacement is open. + expect(executions[0]?.["stop_status"]).toBe("interrupted"); + expect(executions[1]?.["stopped_at"]).toBe(null); + }); + + it("cancels a run without beginning anything, and stays cancelled", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + const cancelled = await ask(stub, { id: "command-2", command: "cancel", runId: RUN_ID }); + expect(cancelled["outcome"]).toBe("performed"); + + const state = await on(stub, (owner) => ({ + run: owner.runRow(), + executions: owner.executionRows(), + })); + expect(state.run?.["status"]).toBe("cancelled"); + // Cancelling begins nothing. The one execution is the one the lost + // executor left, closed as interrupted by the recovery that ran first — + // cancelling the run does not rewrite what that execution became. + expect(state.executions).toHaveLength(1); + expect(state.executions[0]?.["stop_status"]).toBe("interrupted"); + }); +}); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 8798f0c74..697b6c84c 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -18,7 +18,12 @@ import type { TokenVerification, VerificationKey } from "../../../src/cloudflare import { refusalOf } from "../../../src/cloudflare/owner.ts"; import { sha256Hex } from "../../../src/workspace/sha256.ts"; import { WORKSPACE_ROOT_DOMAIN } from "../../../src/workspace/root-manifest.ts"; -import { COMMAND_TABLE, STAGING_TABLE } from "../../../src/cloudflare/private-schema.ts"; +import { + COMMAND_TABLE, + FORK_TABLE, + HOLD_TABLE, + STAGING_TABLE, +} from "../../../src/cloudflare/private-schema.ts"; import { MARKER_TABLE } from "../../../src/cloudflare/marker.ts"; /** The identities this owner is configured to admit. */ @@ -148,6 +153,8 @@ export class ExecutorObject extends WorkflowOwnerObject { * an admission request can name. */ #keys: VerificationKey[] = []; + /** Client ends of the admitted pairs, held so they are not collected. */ + readonly #clients: WebSocket[] = []; #now = 1_800_000_000; #skew = 0; @@ -317,6 +324,48 @@ export class ExecutorObject extends WorkflowOwnerObject { return this.ctx.storage.sql.exec("SELECT * FROM workflow_run").toArray()[0] ?? null; } + /** Every document execution, in the order the run recorded them. */ + executionRows(): Record[] { + return this.ctx.storage.sql + .exec("SELECT * FROM document_executions ORDER BY sequence") + .toArray(); + } + + /** The Workspace root this run currently stands on. */ + currentRootId(): string { + const row = this.ctx.storage.sql + .exec("SELECT current_root_id FROM workspace_state WHERE singleton_id = 1") + .toArray()[0]; + return String(row?.["current_root_id"] ?? ""); + } + + /** + * Which execution each acquisition began, as the owner retained it. + * + * Read from storage rather than from a field, which is the point: an evicted + * object keeps its sockets and forgets everything else. + */ + heldExecutions(): Record[] { + return this.ctx.storage.sql.exec(`SELECT * FROM ${HOLD_TABLE}`).toArray(); + } + + /** Every fork part this owner is holding for anyone. */ + forkParts(): Record[] { + return this.ctx.storage.sql + .exec( + `SELECT acquisition_id, section, position FROM ${FORK_TABLE} ORDER BY section, position`, + ) + .toArray(); + } + + /** Close every admitted connection, as a lost executor leaves them. */ + dropConnections(): void { + for (const socket of this.ctx.getWebSockets("executor")) { + socket.close(1000, "gone"); + this.webSocketClose(socket); + } + } + /** How many objects this storage declares at all. */ objectCount(): number { return this.ctx.storage.sql @@ -651,6 +700,11 @@ export class ExecutorObject extends WorkflowOwnerObject { async admitConnection(request: Partial): Promise { const pair = new WebSocketPair(); const server = pair[1]; + // The client end is kept for as long as this object lives. Nothing reads + // it, but a pair whose other end is collected closes the end the owner + // accepted, and a test that then sends has no connection through no fault + // of the code under test. + this.#clients.push(pair[0]); const presented: AdmissionRequest = { runId: "runId" in request ? request.runId : RUN_ID, release: "release" in request ? request.release : POLICY.release, @@ -744,6 +798,22 @@ export class ExecutorObject extends WorkflowOwnerObject { return this.onRunnerMessage(socket, RUN_ID, raw); } + /** + * Send as the connection admitted most recently. + * + * A replacement acquisition is a different socket, and the earlier one may + * still be listed; addressing by position would send as the connection that + * is gone. + */ + sendLatest(raw: string): unknown { + const sockets = this.ctx.getWebSockets("executor"); + const socket = sockets[sockets.length - 1]; + if (socket === undefined) { + return { id: "", outcome: "refused", refusal: "no-such-connection" }; + } + return this.onRunnerMessage(socket, RUN_ID, raw); + } + /** Send as a socket this object never admitted. */ sendAsStranger(raw: string): unknown { const pair = new WebSocketPair(); diff --git a/packages/workflow/tests/remote-fork.test.ts b/packages/workflow/tests/remote-fork.test.ts new file mode 100644 index 000000000..aedb4bdd7 --- /dev/null +++ b/packages/workflow/tests/remote-fork.test.ts @@ -0,0 +1,220 @@ +/** + * Tier WRH — how a remote fork crosses from a source snapshot to a destination. + * + * That the destination commits whole or not at all, and that its copied prefix + * outlives the source, are owner facts and are proved against a real Durable + * Object in `tests/cloudflare/remote-fork.vitest.ts`. These are the runner's + * half: that the source is read through the no-acquisition plane, that every + * member of the snapshot is offered before anything is committed, and that what + * the final command claims is what was actually offered. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { type Operation, scoped } from "effection"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; +import type { ExecutorLock } from "../src/lifecycle/api.ts"; +import type { WorkflowForkRequest } from "../src/lifecycle/execution.ts"; +import type { WorkflowExecutionTransitions } from "../src/lifecycle/execution.ts"; +import { useRemoteLifecycle } from "../src/remote/lifecycle.ts"; +import type { RemoteForkSource } from "../src/remote/read.ts"; +import type { RemoteForkCommit, RemoteForkPart } from "../src/remote/lifecycle-link.ts"; +import { installedHost, RUN_ID, ROOT, type Script } from "./support/remote-lifecycle-host.ts"; + +const SOURCE_RUN_ID = "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +const DESTINATION = "7ektgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + +function event(name: string): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }); +} + +function source(): RemoteForkSource { + return { + sourceRunId: SOURCE_RUN_ID, + anchor: "f".repeat(64), + checkpointEventId: "event-work", + checkpointWorkspaceRootId: ROOT, + runRecordWorkspaceRootId: ROOT, + rootImportWorkspaceRootId: ROOT, + inherited: [ + { eventId: "event-a", record: event("a"), workspaceRootId: ROOT }, + { eventId: "event-b", record: event("b"), workspaceRootId: ROOT }, + ], + roots: [ + { + rootId: ROOT, + formatVersion: 1, + manifest: "{}", + manifestHashes: ["b".repeat(64)], + blobHashes: ["c".repeat(64)], + }, + ], + manifests: [{ hash: "b".repeat(64), size: 3, lastSeen: 0, encoded: new Uint8Array([1, 2, 3]) }], + blobs: [{ hash: "c".repeat(64), size: 3, lastSeen: 0, content: new Uint8Array([1, 2, 3]) }], + checkouts: [ + { + kind: "repository", + name: "alpha", + locator: "https://git.example.invalid/alpha.git", + locatorFingerprint: "d".repeat(64), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/", + }, + ], + }; +} + +function request(runId = DESTINATION): WorkflowForkRequest { + return { + runId, + selection: { sourceRunId: SOURCE_RUN_ID, checkpointEventId: "event-work" }, + creation: { + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + }, + rootImport: { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path: "README.md", content: "# fork" } }, + }, + }; +} + +function* installed( + script: Script, + body: (transitions: WorkflowExecutionTransitions) => Operation, +): Operation { + return yield* scoped(function* () { + const transitions = yield* useRemoteLifecycle(installedHost(script)); + return yield* body(transitions); + }); +} + +function* acquired(runId: string): Operation { + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(runId); + if (!taken.ok) { + throw taken.error; + } + if (taken.value.kind !== "acquired") { + throw new Error("expected the executor lock to be acquired"); + } + return taken.value.lock; +} + +describe("a remote fork's destination", () => { + it("offers the whole snapshot before it commits any of it", function* () { + const asked: string[] = []; + const staged: RemoteForkPart[] = []; + const commits: RemoteForkCommit[] = []; + const outcome = yield* installed( + { asked, staged, commits, source: source() }, + function* (transitions) { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }, + ); + + expect([outcome.ok, outcome.ok === false && String(outcome.error)]).toEqual([true, false]); + // Everything was offered, and the commit came last. + expect(asked.at(-1)).toBe("fork"); + expect(asked.filter((command) => command === "fork-stage")).toHaveLength(4); + expect(staged.map((part) => `${part.section}:${part.position}`)).toEqual([ + "roots:0", + "inherited:0", + "inherited:1", + "checkouts:0", + ]); + // What the final command claims is what was offered, and where it came + // from is the selection that was read. + expect(commits[0]?.counts).toEqual({ inherited: 2, roots: 1, checkouts: 1 }); + expect(commits[0]?.origin).toEqual({ + sourceRunId: SOURCE_RUN_ID, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: ROOT, + runRecordWorkspaceRootId: ROOT, + rootImportWorkspaceRootId: ROOT, + anchor: "f".repeat(64), + }); + }); + + it("carries the inherited records exactly as the source retained them", function* () { + const staged: RemoteForkPart[] = []; + yield* installed({ staged, source: source() }, function* (transitions) { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + + const inherited = staged.filter((part) => part.section === "inherited"); + expect(inherited.map((part) => part.part["record"])).toEqual([event("a"), event("b")]); + expect(inherited.map((part) => part.part["eventId"])).toEqual(["event-a", "event-b"]); + }); + + it("writes the fork's own run record rather than the source's", function* () { + const commits: RemoteForkCommit[] = []; + yield* installed({ commits, source: source() }, function* (transitions) { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + + const head = commits[0]; + // Its own identity, and the root import its own definition produced. + expect(JSON.stringify(head?.runRecord)).toContain(DESTINATION); + expect(JSON.stringify(head?.runRecord)).not.toContain(SOURCE_RUN_ID); + expect(head?.rootImport).toEqual(request().rootImport); + }); + + it("refuses a fork under a lock issued for another run, and reads nothing", function* () { + const asked: string[] = []; + const outcome = yield* installed({ asked, source: source() }, function* (transitions) { + const lock = yield* acquired(RUN_ID); + return yield* transitions.fork(lock, request()); + }); + + expect(outcome.ok).toBe(false); + // Not one part offered, and no source read: the lock was wrong before any + // of that could matter. + expect(asked).toEqual([]); + }); + + it("refuses a staged fork whose source it cannot read, and takes no acquisition", function* () { + const opened: string[] = []; + const outcome = yield* installed({ opened }, function* (transitions) { + return yield* transitions.stageFork(request()); + }); + + expect(outcome.ok).toBe(false); + // Staging takes no destination acquisition at all, failure or not. + expect(opened).toEqual([]); + }); + + it("stages a candidate without acquiring or committing anything", function* () { + const asked: string[] = []; + const opened: string[] = []; + const outcome = yield* installed({ asked, opened, source: source() }, function* (transitions) { + return yield* transitions.stageFork(request()); + }); + + // This scripted host stages nothing, which is the point: what is proved + // here is that nothing was acquired and nothing was committed on the way. + expect(outcome.ok).toBe(false); + expect(opened).toEqual([]); + expect(asked).toEqual([]); + }); +}); diff --git a/packages/workflow/tests/remote-lifecycle.test.ts b/packages/workflow/tests/remote-lifecycle.test.ts new file mode 100644 index 000000000..7eaf31e02 --- /dev/null +++ b/packages/workflow/tests/remote-lifecycle.test.ts @@ -0,0 +1,207 @@ +/** + * Tier WRH — what the remote executor lifecycle authorizes, and what it refuses. + * + * The owner-side facts — admission contention, pristine initialization, the + * acquisition/execution association surviving hibernation, and one transaction + * per transition — are proved against a real Durable Object in + * `tests/cloudflare/remote-lifecycle.vitest.ts`. These are the other half: that + * the lock is an object rather than a description, that its lifetime is its + * connection's, that one acquisition begins one execution, and that a caller + * never sees a private refusal. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { type Operation, type Result, scoped } from "effection"; +import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; +import type { ExecutorLock } from "../src/lifecycle/api.ts"; +import type { + WorkflowBeginRequest, + WorkflowExecutionTransitions, +} from "../src/lifecycle/execution.ts"; +import { useRemoteLifecycle } from "../src/remote/lifecycle.ts"; +import { WorkflowRequestError } from "../src/storage/errors.ts"; +import { installedHost, RUN_ID, type Script } from "./support/remote-lifecycle-host.ts"; + +function* acquired(runId = RUN_ID): Operation { + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(runId); + if (!taken.ok) { + throw taken.error; + } + if (taken.value.kind !== "acquired") { + throw new Error("expected the executor lock to be acquired"); + } + return taken.value.lock; +} + +function* installed( + script: Script, + body: (transitions: WorkflowExecutionTransitions) => Operation, +): Operation { + return yield* scoped(function* () { + const transitions = yield* useRemoteLifecycle(installedHost(script)); + return yield* body(transitions); + }); +} + +describe("a remote run's executor lifecycle", () => { + it("hands back a lock nothing else can be mistaken for", function* () { + const outcomes = yield* installed({}, function* (transitions) { + const lock = yield* acquired(); + const request: WorkflowBeginRequest = { runId: RUN_ID, action: "resume" }; + return { + // The same run, the same shape, a different object. + copied: yield* transitions.begin({ runId: lock.runId }, request), + frozen: yield* transitions.begin(Object.freeze({ runId: RUN_ID }), request), + // Another provider's lock: this one was never issued here at all. + foreign: yield* transitions.begin(Object.freeze({ runId: RUN_ID }), request), + held: yield* transitions.begin(lock, request), + }; + }); + + for (const [name, outcome] of Object.entries(outcomes)) { + if (name === "held") { + expect([name, outcome.ok]).toEqual([name, true]); + continue; + } + expect([name, outcome.ok]).toEqual([name, false]); + expect(outcome.ok === false && outcome.error).toEqual(expect.any(WorkflowRequestError)); + } + }); + + it("refuses a lock whose acquisition has ended, and takes nothing while it does", function* () { + const asked: string[] = []; + const outcome = yield* installed({ asked }, function* (transitions) { + // The lock outlives the scope that acquired it; its authority does not. + const escaped = yield* scoped(function* () { + return yield* acquired(); + }); + const after = asked.length; + const refused = yield* transitions.begin(escaped, { runId: RUN_ID, action: "resume" }); + return { refused, before: after, sent: asked.length }; + }); + + expect(outcome.refused.ok).toBe(false); + // Nothing was asked of the owner: a released lock is refused before a + // command is composed, let alone sent. + expect(outcome.sent).toBe(outcome.before); + }); + + it("reports a live executor rather than failing, and advances nothing", function* () { + const asked: string[] = []; + const taken = yield* installed({ asked, admit: "already-running" }, function* () { + return yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + }); + + expect(taken.ok).toBe(true); + expect(taken.ok && taken.value.kind).toBe("already-running"); + expect(asked).toEqual([]); + }); + + it("begins one execution per acquisition, and settles only that one", function* () { + const asked: string[] = []; + const outcome = yield* installed({ asked }, function* (transitions) { + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + if (!begun.ok) { + throw begun.error; + } + const again = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + const foreign = yield* transitions.settle(lock, { + executionId: "execution-elsewhere", + status: "completed", + }); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + return { again, foreign, settled, asked: [...asked] }; + }); + + expect(outcome.again.ok).toBe(false); + expect(outcome.foreign.ok).toBe(false); + expect(outcome.settled.ok).toBe(true); + // The second begin and the foreign settlement never reached the owner. + expect(outcome.asked.filter((command) => command === "begin")).toHaveLength(1); + expect(outcome.asked.filter((command) => command === "settle")).toHaveLength(1); + }); + + it("refuses a begin addressed to another run before the transport", function* () { + const asked: string[] = []; + const outcome = yield* installed({ asked }, function* (transitions) { + const lock = yield* acquired(); + return yield* transitions.begin(lock, { runId: "another-run", action: "resume" }); + }); + + expect(outcome.ok).toBe(false); + expect(asked).toEqual([]); + }); + + it("carries a refusal about the run as its own condition", function* () { + const conditions: readonly ("cancelled" | "resume-failed")[] = ["cancelled", "resume-failed"]; + for (const refusal of conditions) { + const outcome = yield* installed({ begin: refusal }, function* (transitions) { + const lock = yield* acquired(); + return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + }); + expect([refusal, outcome.ok]).toEqual([refusal, false]); + if (!outcome.ok) { + // The condition, and nothing about how it was spelled underneath. + expect(String(outcome.error)).not.toContain("command:"); + expect(String(outcome.error)).toContain(RUN_ID); + } + } + }); + + it("takes its own acquisition to cancel, and gives it back", function* () { + const opened: string[] = []; + const closed: string[] = []; + const outcome = yield* installed({ opened, closed }, function* () { + return yield* WorkflowLifecycle.operations.cancel(RUN_ID); + }); + + expect(outcome.ok).toBe(true); + expect(opened).toEqual([RUN_ID]); + // The acquisition cancellation took for itself is not still held. + expect(closed).toEqual([RUN_ID]); + }); + + it("refuses to cancel a run a live executor holds", function* () { + const asked: string[] = []; + const outcome = yield* installed({ asked, admit: "already-running" }, function* () { + return yield* WorkflowLifecycle.operations.cancel(RUN_ID); + }); + + expect(outcome.ok).toBe(false); + // Nothing was asked of the owner, and the no-acquisition plane was not + // consulted to decide who holds the run. + expect(asked).toEqual([]); + }); + + it("composes with the reads already installed rather than replacing them", function* () { + const outcome = yield* scoped(function* () { + let asked = false; + yield* WorkflowLifecycle.around({ + // deno-lint-ignore require-yield + *inspect(): Operation> { + asked = true; + throw new WorkflowRequestError("the installed read answered"); + }, + }); + yield* useRemoteLifecycle(installedHost({})); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + try { + yield* WorkflowLifecycle.operations.inspect(RUN_ID); + } catch { + // The installed read answered by raising; what matters is that it was + // the one that answered. + } + return { taken, asked }; + }); + + expect(outcome.taken.ok).toBe(true); + // The lifecycle provider added its two operations over the read provider's, + // rather than installing an object that answers only its own. + expect(outcome.asked).toBe(true); + }); +}); diff --git a/packages/workflow/tests/remote-recovery.test.ts b/packages/workflow/tests/remote-recovery.test.ts new file mode 100644 index 000000000..bced3b824 --- /dev/null +++ b/packages/workflow/tests/remote-recovery.test.ts @@ -0,0 +1,109 @@ +/** + * Tier WRH — what a remote run reports about the executor that came before it. + * + * What recovery *decides* — that a retained root `Close` restores the outcome + * it recorded, and that its absence leaves `interrupted` — is the shared + * policy's, proved on a real owner in + * `tests/cloudflare/remote-lifecycle.vitest.ts`. This is what a caller learns: + * that the execution recovery closed is reported beside the one that was begun, + * and that a run which kept a terminal outcome says so rather than looking + * freshly running. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { type Operation, scoped } from "effection"; +import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; +import type { ExecutorLock } from "../src/lifecycle/api.ts"; +import type { WorkflowExecutionTransitions } from "../src/lifecycle/execution.ts"; +import { useRemoteLifecycle } from "../src/remote/lifecycle.ts"; +import { installedHost, RUN_ID, type Script } from "./support/remote-lifecycle-host.ts"; + +function* installed( + script: Script, + body: (transitions: WorkflowExecutionTransitions) => Operation, +): Operation { + return yield* scoped(function* () { + const transitions = yield* useRemoteLifecycle(installedHost(script)); + return yield* body(transitions); + }); +} + +function* acquired(): Operation { + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + if (!taken.ok) { + throw taken.error; + } + if (taken.value.kind !== "acquired") { + throw new Error("expected the executor lock to be acquired"); + } + return taken.value.lock; +} + +describe("what a remote run reports about its previous executor", () => { + it("reports the execution recovery closed, beside the one it began", function* () { + const outcome = yield* installed({ recovered: "execution-before" }, function* (transitions) { + const lock = yield* acquired(); + return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + }); + + expect(outcome.ok).toBe(true); + if (outcome.ok) { + expect(outcome.value.recovered?.executionId).toBe("execution-before"); + // The one this acquisition began is its own, and a different execution. + expect(outcome.value.execution.executionId).not.toBe("execution-before"); + } + }); + + it("says nothing about recovery when there was none", function* () { + const outcome = yield* installed({}, function* (transitions) { + const lock = yield* acquired(); + return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + }); + + expect(outcome.ok).toBe(true); + // Absent rather than null: a caller reads "nothing was recovered" from the + // member not being there at all. + expect(outcome.ok && "recovered" in outcome.value).toBe(false); + }); + + it("carries a replay as a replay rather than as a fresh run", function* () { + const outcome = yield* installed({ replay: true }, function* (transitions) { + const lock = yield* acquired(); + return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + }); + + expect(outcome.ok).toBe(true); + expect(outcome.ok && outcome.value.replay).toBe(true); + }); + + it("settles the execution it began after a recovery, and only that one", function* () { + const asked: string[] = []; + const outcome = yield* installed( + { asked, recovered: "execution-before" }, + function* (transitions) { + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + if (!begun.ok) { + throw begun.error; + } + return { + // The recovered execution belonged to an acquisition that is gone. + stale: yield* transitions.settle(lock, { + executionId: "execution-before", + status: "completed", + }), + own: yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }), + asked: [...asked], + }; + }, + ); + + expect(outcome.stale.ok).toBe(false); + expect(outcome.own.ok).toBe(true); + expect(outcome.asked.filter((command) => command === "settle")).toHaveLength(1); + }); +}); diff --git a/packages/workflow/tests/support/remote-lifecycle-host.ts b/packages/workflow/tests/support/remote-lifecycle-host.ts new file mode 100644 index 000000000..935af0939 --- /dev/null +++ b/packages/workflow/tests/support/remote-lifecycle-host.ts @@ -0,0 +1,252 @@ +/** + * A scripted host for the remote lifecycle provider. + * + * Stands in for reaching an owner, not for the owner: what it proves is what + * the provider decides before and after a command — which lock authorizes, + * which run is addressed, what reaches the transport at all. Whether an owner + * transaction is atomic, whether admission contends, and whether an association + * survives hibernation are owner facts and are proved against a real Durable + * Object instead. + */ + +import { ensure, Err, Ok, type Operation, type Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import type { + RemoteBegun, + RemoteExecutorConnection, + RemoteForkCommit, + RemoteForkPart, + RemoteLifecycleAnswer, + RemoteLifecycleLink, +} from "../../src/remote/lifecycle-link.ts"; +import type { RemoteLifecycleHost } from "../../src/remote/lifecycle.ts"; +import type { + RemoteForkSource, + RemoteFrontierSnapshot, + RemoteReadPlane, +} from "../../src/remote/read.ts"; +import type { WorkflowRunDatabase } from "../../src/storage/api.ts"; +import type { WorkflowForkRequest } from "../../src/lifecycle/execution.ts"; +import type { + DocumentExecutionCompletion, + DocumentExecutionRecord, + WorkflowRunRecord, +} from "../../src/storage/record.ts"; +import { WorkflowRequestError } from "../../src/storage/errors.ts"; +import type { RemoteWorkspaceLink } from "../../src/remote/database.ts"; + +export const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +export const ROOT = "a".repeat(64); + +/** What this host does when it is asked, and what it records while it does. */ +export interface Script { + /** Every command the provider actually sent. */ + readonly asked?: string[]; + /** Every run an acquisition was opened for. */ + readonly opened?: string[]; + /** Every run an acquisition was given back for. */ + readonly closed?: string[]; + /** Answer admission with a live executor rather than a connection. */ + readonly admit?: "already-running"; + /** Answer a begin with one of the run's own conditions. */ + readonly begin?: "cancelled" | "resume-failed"; + /** Report that recovery closed this execution on the way in. */ + readonly recovered?: string; + /** Answer a begin with a run that kept its terminal state. */ + readonly replay?: boolean; + /** Answer a fork commit with a conflict. */ + readonly forkConflict?: readonly string[]; + /** The source snapshot a fork reads, when a test supplies one. */ + readonly source?: RemoteForkSource; + /** Every fork part the provider offered, in the order it offered them. */ + readonly staged?: RemoteForkPart[]; + /** Every fork commit the provider asked for. */ + readonly commits?: RemoteForkCommit[]; +} + +export function record(): WorkflowRunRecord { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-01-01T00:00:00.000Z", + updatedAt: "2026-01-01T00:00:00.000Z", + }; +} + +export function frontier(): RemoteFrontierSnapshot { + return { + record: record(), + retrieval: undefined, + workspaceRootId: ROOT, + journalEventId: null, + entries: [], + }; +} + +function execution(executionId: string): DocumentExecutionRecord { + return { executionId, startedAt: "2026-01-01T00:00:00.000Z" }; +} + +function begun(executionId: string, script: Script = {}): RemoteBegun { + return { + frontier: frontier(), + execution: execution(executionId), + replay: script.replay === true, + recovered: script.recovered === undefined ? null : execution(script.recovered), + }; +} + +/** A link that answers the one read the provider makes of it. */ +function link(): RemoteWorkspaceLink { + const unsupported = () => { + throw new WorkflowRequestError("this scripted link answers no such operation"); + }; + return { + // deno-lint-ignore require-yield + *frontierSnapshot(): Operation { + return frontier(); + }, + // deno-lint-ignore require-yield + *frontier(): Operation { + return unsupported(); + }, + // deno-lint-ignore require-yield + *commit(): Operation { + return unsupported(); + }, + // deno-lint-ignore require-yield + *replaceRetrieval(): Operation { + return unsupported(); + }, + // deno-lint-ignore require-yield + *readExecutions(): Operation { + return unsupported(); + }, + // deno-lint-ignore require-yield + *open(): Operation { + return unsupported(); + }, + // deno-lint-ignore require-yield + *invocationSnapshot(): Operation { + return unsupported(); + }, + // deno-lint-ignore require-yield + *root(): Operation { + return unsupported(); + }, + // deno-lint-ignore require-yield + *content(): Operation { + return unsupported(); + }, + }; +} + +function lifecycle(script: Script): RemoteLifecycleLink { + let minted = 0; + return { + // deno-lint-ignore require-yield + *begin(request: { + readonly executionId: string; + }): Operation>> { + script.asked?.push("begin"); + if (script.begin !== undefined) { + return Ok({ kind: "refused", refusal: script.begin }); + } + minted += 1; + return Ok({ kind: "performed", value: begun(request.executionId, script) }); + }, + // deno-lint-ignore require-yield + *settle(_completion: DocumentExecutionCompletion): Operation> { + script.asked?.push("settle"); + return Ok(frontier()); + }, + // deno-lint-ignore require-yield + *cancel(): Operation>> { + script.asked?.push("cancel"); + return Ok({ kind: "performed", value: record() }); + }, + // deno-lint-ignore require-yield + *stageForkPart(part: RemoteForkPart): Operation> { + script.asked?.push("fork-stage"); + script.staged?.push(part); + return Ok(undefined); + }, + // deno-lint-ignore require-yield + *commitFork(commit: RemoteForkCommit): Operation> { + script.asked?.push("fork"); + script.commits?.push(commit); + if (script.forkConflict !== undefined) { + return Err(new WorkflowRequestError("this destination is another run")); + } + void minted; + return Ok(begun(commit.executionId)); + }, + }; +} + +/** The provider's host, scripted. */ +export function installedHost(script: Script): RemoteLifecycleHost { + let executions = 0; + return { + *admit(runId: string): Operation> { + if (script.admit === "already-running") { + return Ok("already-running"); + } + script.opened?.push(runId); + const connection: RemoteExecutorConnection = { link: link(), lifecycle: lifecycle(script) }; + yield* ensureClosed(script, runId); + return Ok(connection); + }, + // deno-lint-ignore require-yield + *source(runId: string): Operation> { + const held = script.source; + if (held === undefined) { + return Err(new WorkflowRequestError("this scripted host holds no source")); + } + return Ok({ + runId, + // deno-lint-ignore require-yield + *inspect(): Operation> { + throw new WorkflowRequestError("this scripted plane answers no inspection"); + }, + // deno-lint-ignore require-yield + *history(): Operation> { + throw new WorkflowRequestError("this scripted plane answers no history"); + }, + // deno-lint-ignore require-yield + *forkSource(): Operation> { + return Ok(held); + }, + }); + }, + // deno-lint-ignore require-yield + *stage( + _request: WorkflowForkRequest, + _source: RemoteForkSource, + _head: { readonly runRecord: DurableEvent; readonly rootImport: DurableEvent }, + ): Operation> { + return Err(new WorkflowRequestError("this scripted host stages nothing")); + }, + ids: { + execution: () => { + executions += 1; + return `execution-${executions}`; + }, + }, + }; +} + +function* ensureClosed(script: Script, runId: string): Operation { + yield* ensure(function* () { + script.closed?.push(runId); + }); +} From 9df6094504997ebc720b6fef0cf60d6e43aff8a6 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Mon, 7 Sep 2026 17:00:02 -0300 Subject: [PATCH 52/81] =?UTF-8?q?=F0=9F=90=9B=20Prove=20a=20fork's=20copy,?= =?UTF-8?q?=20and=20stage=20one=20where=20it=20can=20be=20replayed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fork is admitted by replaying it, and a replay needs the fork's own Workspace — a document resolves its paths through the run's filesystem, and a candidate without one produces effects of a different kind and diverges for a reason that has nothing to do with the candidate. Staging one was a seam with nothing behind it. Now it is this runner's own disk: the snapshot the no-acquisition plane returned, handed to the same assembly kernel a local fork already used, at a path that belongs to the scope that asked and goes when that scope ends. Nothing acquires anything, nothing reaches an owner, and nothing discovers it. A destination believed the anchor it was handed. It does not any more. The selection a source commits to is now said once — the checkpoint and its three head roots, the inherited rows with their exact retained bytes, each root's record and its ordered references, each piece of content's metadata including the watermark copied beside it, and every checkout in the owner's own order — and both sides hash that same shape. The source builds it from rows it reads; the destination rebuilds it from the parts it was offered and compares, before it writes anything at all. A changed watermark, a changed byte, a reordered member or a well-formed digest from nowhere leaves the destination holding nothing. Watermarks now travel. A digest stands for bytes, and for the size those bytes have; it does not stand for a watermark, which is copied into destination storage and moves while the content stands still. So the metadata crosses beside the content and is retained as the source retained it, rather than starting again at zero. The two records a fork writes for itself are held to what they must be. The run record is the canonical event this destination's own identity, base and pinned commit imply, not any parseable event that happens to arrive; the root import is a root import, against a root the transfer carried. A destination that already holds this fork is not forked again. It is a run being taken up, so it goes through the same recovery and admission a resume does: what the previous executor left is closed on the run's own evidence, and exactly one replacement execution begins. Repeating the same command is still the same command, and answers with the decision it already made. Where a run's definition can be fetched from is written with the run that carries it. It travels beside the creation rather than inside it, because a run is not a different run for having been fetched from somewhere else, and it is written at revision one inside the transaction that makes the run — or not at all, when nothing was carried. Taking an existing run up again neither compares it nor clears it. An invocation keeps its identity. A begin, a settlement, a cancellation and a committed fork each mint one command id and one execution id before their first send, kept by the run and the question asked, so a replacement connection after a lost answer asks the same question rather than a second one. The owner records which execution a decision began; a replacement acquisition adopts it when nobody live holds it, and is refused when somebody does. Knowing a command id inherits nothing else: not the old connection, not its scratch, not its authority. The lifecycle provider installs nearest the work, as every provider here does. --- packages/workflow/src/cloudflare/commands.ts | 56 +++- .../workflow/src/cloudflare/dispatcher.ts | 31 +- .../workflow/src/cloudflare/fork-anchor.ts | 121 +++++++ .../workflow/src/cloudflare/lifecycle-link.ts | 55 ++-- .../workflow/src/cloudflare/owner-fork.ts | 271 +++++++++++++++- .../src/cloudflare/owner-lifecycle.ts | 3 +- .../workflow/src/cloudflare/owner-open.ts | 16 +- .../workflow/src/cloudflare/private-schema.ts | 53 +++- .../workflow/src/cloudflare/read-plane.ts | 111 +++++-- packages/workflow/src/deno/remote-host.ts | 67 ++++ packages/workflow/src/deno/remote-staging.ts | 132 ++++++++ packages/workflow/src/fork.ts | 23 +- packages/workflow/src/journal-events.ts | 26 +- .../workflow/src/remote/lifecycle-link.ts | 38 ++- packages/workflow/src/remote/lifecycle.ts | 179 +++++++++-- .../workflow/src/sqlite/workflow-schema.ts | 4 + .../tests/cloudflare/remote-fork.vitest.ts | 231 ++++++++++++-- .../cloudflare/remote-lifecycle.vitest.ts | 130 ++++++++ .../cloudflare/support/executor-object.ts | 10 + packages/workflow/tests/remote-fork.test.ts | 13 +- .../workflow/tests/remote-lifecycle.test.ts | 91 ++++++ .../workflow/tests/remote-staged-fork.test.ts | 296 ++++++++++++++++++ .../tests/support/remote-lifecycle-host.ts | 58 +++- 23 files changed, 1865 insertions(+), 150 deletions(-) create mode 100644 packages/workflow/src/cloudflare/fork-anchor.ts create mode 100644 packages/workflow/src/deno/remote-host.ts create mode 100644 packages/workflow/src/deno/remote-staging.ts create mode 100644 packages/workflow/tests/remote-staged-fork.test.ts diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 2470bdd71..b9aa71886 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -18,6 +18,8 @@ import { import { type AgentSessionRecord, parseAgentSessionRecord } from "../storage/agent-session.ts"; import { parseCreateRequest } from "../storage/create-request.ts"; import type { CreateWorkflowRunRequest } from "../storage/api.ts"; +import { parseJsonValue } from "../storage/members.ts"; +import { canonicalJson } from "../storage/record.ts"; /** The most characters a public run id may carry. */ const MAX_RUN_ID = 128; @@ -259,6 +261,15 @@ export interface BeginCommand extends CommandEnvelope { readonly runId: string; readonly action: "start" | "resume"; readonly creation: CreateWorkflowRunRequest | null; + /** + * Where this run's definition can be fetched from again, when it is being + * created. + * + * Replaceable state rather than identity, so it travels beside the creation + * instead of inside it: a run is not a different run for having been fetched + * from somewhere else. + */ + readonly retrieval: string | null; /** * The execution's identity, minted by the runner. * @@ -276,7 +287,7 @@ export interface CancelCommand extends CommandEnvelope { } /** The sections a fork's parts arrive in, each in its own order. */ -export type ForkSection = "inherited" | "roots" | "checkouts"; +export type ForkSection = "inherited" | "roots" | "manifests" | "blobs" | "checkouts"; /** One part of a fork, as the runner offers it. */ export interface ForkPart { @@ -289,6 +300,8 @@ export interface ForkPart { export interface ForkCounts { readonly inherited: number; readonly roots: number; + readonly manifests: number; + readonly blobs: number; readonly checkouts: number; } @@ -321,6 +334,7 @@ export interface ForkCommand extends CommandEnvelope { readonly command: "fork"; readonly runId: string; readonly creation: CreateWorkflowRunRequest; + readonly retrieval: string | null; readonly origin: ForkOrigin; readonly counts: ForkCounts; readonly runRecord: DurableEvent; @@ -376,7 +390,7 @@ const MEMBERS: Record = { executions: [...ENVELOPE, "anchor", "after"], mappings: ENVELOPE, open: [...ENVELOPE, "runId", "creation"], - begin: [...ENVELOPE, "runId", "action", "creation", "executionId"], + begin: [...ENVELOPE, "runId", "action", "creation", "retrieval", "executionId"], cancel: [...ENVELOPE, "runId"], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], "fork-stage": [...ENVELOPE, "section", "position", "part"], @@ -384,6 +398,7 @@ const MEMBERS: Record = { ...ENVELOPE, "runId", "creation", + "retrieval", "origin", "counts", "runRecord", @@ -626,6 +641,7 @@ export function parseCommand(raw: string): RunnerCommand { runId, action, creation, + retrieval: retrieval(members.get("retrieval"), creation), executionId: text(members, "executionId", MAX_RUN_ID), }; } @@ -634,7 +650,13 @@ export function parseCommand(raw: string): RunnerCommand { } if (command === "fork-stage") { const section = members.get("section"); - if (section !== "inherited" && section !== "roots" && section !== "checkouts") { + if ( + section !== "inherited" && + section !== "roots" && + section !== "manifests" && + section !== "blobs" && + section !== "checkouts" + ) { throw new CommandError("malformed-member"); } const part = members.get("part"); @@ -660,6 +682,7 @@ export function parseCommand(raw: string): RunnerCommand { command, runId, creation: creation.value, + retrieval: retrieval(members.get("retrieval"), creation.value), origin: origin(members.get("origin")), counts: counts(members.get("counts")), runRecord: forkEvent(members.get("runRecord")), @@ -702,6 +725,29 @@ export function parseCommand(raw: string): RunnerCommand { }; } +/** + * The retrieval metadata a creation carries, canonically encoded. + * + * Only a creating request may carry one: a resume is not creating anything for + * it to belong to. The value is held to the same JSON rules every retained + * record is, and to the same bound one message is. + */ +function retrieval(value: unknown, creation: CreateWorkflowRunRequest | null): string | null { + if (value === null) { + return null; + } + if (creation === null) { + throw new CommandError("malformed-member"); + } + const encoded = canonicalJson( + parseJsonValue(value, "$.retrieval", () => new CommandError("malformed-member")), + ); + if (new TextEncoder().encode(encoded).length > MAX_MESSAGE_BYTES) { + throw new CommandError("too-large"); + } + return encoded; +} + /** A whole count, as a member rather than a column. */ function whole(value: unknown): number { if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { @@ -734,10 +780,12 @@ function origin(value: unknown): ForkOrigin { /** How many parts each section should have arrived in. */ function counts(value: unknown): ForkCounts { const found = object(value); - closed(found, ["inherited", "roots", "checkouts"]); + closed(found, ["inherited", "roots", "manifests", "blobs", "checkouts"]); return { inherited: whole(found.get("inherited")), roots: whole(found.get("roots")), + manifests: whole(found.get("manifests")), + blobs: whole(found.get("blobs")), checkouts: whole(found.get("checkouts")), }; } diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index c42ba84bb..576501cbc 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -54,6 +54,7 @@ import { } from "./owner-reads.ts"; import type { OwnerTransaction, OwnerTransactions } from "./owner-transaction.ts"; import { + adoptExecution, COMMAND_TABLE, initializePrivateSchema, MUTATION_TABLE, @@ -130,6 +131,22 @@ function integer(value: unknown): number { return value; } +/** + * Which execution one performed mutation began, if it began one. + * + * Read from the command rather than from the answer: what the runner asked to + * begin is what the owner began, and a decision that refused began nothing. + */ +function begunExecution(command: RunnerCommand, result: CommandResult): string | null { + if (result.outcome !== "performed") { + return null; + } + if (command.command === "begin" || command.command === "fork") { + return command.executionId; + } + return null; +} + /** What this acquisition has already spent of its own ledger. */ function ledgerUsage( storage: AcquisitionContext["storage"], @@ -343,6 +360,7 @@ function perform( command.runId, command.action, command.creation, + command.retrieval, command.executionId, ownerTime, ), @@ -367,6 +385,7 @@ function perform( { runId: command.runId, creation: command.creation, + retrieval: command.retrieval, origin: command.origin, counts: command.counts, runRecord: command.runRecord, @@ -486,6 +505,10 @@ export function dispatchCommand( // would mean applying it again. throw new Error("private protocol storage holds a malformed result"); } + // The answer was lost, not the fact. If this decision began an + // execution and nobody live holds it, it becomes this acquisition's — + // otherwise the caller would be handed a run it could not settle. + adoptExecution(ctx.storage, held.acquisitionId, command.id); return decision; } } @@ -546,12 +569,16 @@ export function dispatchCommand( } ctx.storage.sql.exec( `INSERT INTO ${MUTATION_TABLE} - (command_id, request_fingerprint, response, response_bytes) - VALUES (?, ?, ?, ?)`, + (command_id, request_fingerprint, response, response_bytes, execution_id) + VALUES (?, ?, ?, ?, ?)`, command.id, fingerprint, encoded, responseBytes, + // Which execution this decision began, when it began one, so a + // replacement acquisition re-observing it can adopt the run rather + // than being told about an execution it may not touch. + begunExecution(command, result), ); } return result; diff --git a/packages/workflow/src/cloudflare/fork-anchor.ts b/packages/workflow/src/cloudflare/fork-anchor.ts new file mode 100644 index 000000000..10be3288f --- /dev/null +++ b/packages/workflow/src/cloudflare/fork-anchor.ts @@ -0,0 +1,121 @@ +/** + * What a fork's source selection is, said once. + * + * A source owner computes this over rows it reads out of its own storage; a + * destination owner computes it over the parts it was offered. They have to + * agree exactly, or the anchor proves nothing — so the ordered logical value + * and the digest over it live here, in one place, and both sides build the same + * shape rather than each spelling their own. + * + * What goes in is everything a destination copies that a content identity does + * not already imply: the checkpoint and the three head roots, the inherited + * rows with their exact retained bytes and their root associations, each root's + * record and its ordered reference arrays, each manifest's and blob's retained + * metadata *including its watermark* — a digest stands for bytes and for the + * size derived from them, never for a watermark, which is copied and can move + * while the content stands still — and every selected checkout in the owner's + * own order. + */ + +import { sha256Hex } from "../workspace/sha256.ts"; + +/** One inherited row, as both sides describe it. */ +export interface AnchorRow { + readonly eventId: string; + readonly record: string; + readonly workspaceRootId: string; +} + +/** One Workspace root's retained record and its ordered references. */ +export interface AnchorRoot { + readonly rootId: string; + readonly formatVersion: number; + readonly manifest: string; + readonly manifestHashes: readonly string[]; + readonly blobHashes: readonly string[]; +} + +/** One content manifest's retained metadata and bytes. */ +export interface AnchorManifest { + readonly hash: string; + readonly size: number; + readonly lastSeen: number; + /** The encoded manifest, base64 as the private protocol carries it. */ + readonly encoded: string; +} + +/** One blob's retained metadata, without its bytes. */ +export interface AnchorBlob { + readonly hash: string; + readonly size: number; + readonly lastSeen: number; +} + +/** One checkout, by the key it is paged under and the record it is. */ +export interface AnchorCheckout { + readonly key: string; + readonly value: Record; +} + +/** The whole selection, in the order it is hashed. */ +export interface ForkSelection { + readonly checkpointEventId: string; + readonly checkpointWorkspaceRootId: string; + readonly runRecordWorkspaceRootId: string; + readonly rootImportWorkspaceRootId: string; + readonly inherited: readonly AnchorRow[]; + readonly roots: readonly AnchorRoot[]; + readonly manifests: readonly AnchorManifest[]; + readonly blobs: readonly AnchorBlob[]; + readonly checkouts: readonly AnchorCheckout[]; +} + +/** + * The identity of one committed selection. + * + * Ordered throughout: the members are hashed in the order the owner selected + * them, so a reordering is a different selection rather than the same one + * described differently. + */ +export function forkSelectionAnchor(selection: ForkSelection): string { + return sha256Hex( + JSON.stringify({ + checkpointEventId: selection.checkpointEventId, + checkpointWorkspaceRootId: selection.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: selection.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: selection.rootImportWorkspaceRootId, + inherited: selection.inherited.map((row) => [row.eventId, row.record, row.workspaceRootId]), + roots: selection.roots.map((root) => ({ + rootId: root.rootId, + formatVersion: root.formatVersion, + manifest: root.manifest, + manifestHashes: [...root.manifestHashes], + blobHashes: [...root.blobHashes], + })), + manifests: selection.manifests.map((manifest) => ({ + hash: manifest.hash, + size: manifest.size, + lastSeen: manifest.lastSeen, + encoded: manifest.encoded, + })), + blobs: selection.blobs.map((blob) => ({ + hash: blob.hash, + size: blob.size, + lastSeen: blob.lastSeen, + })), + checkouts: selection.checkouts.map((checkout) => [checkout.key, checkout.value]), + }), + ); +} + +/** + * One checkout's identity, as a key nothing else can spell. + * + * A Repository name and a Worktree name are retained text and may hold any + * character, so joining them with a separator is not an identity: `("a:b", "c")` + * and `("a", "b:c")` are two retained Worktrees that would join to one string. + * A JSON array of the parts escapes what it must and separates what it must. + */ +export function checkoutKey(parts: readonly string[]): string { + return JSON.stringify(parts); +} diff --git a/packages/workflow/src/cloudflare/lifecycle-link.ts b/packages/workflow/src/cloudflare/lifecycle-link.ts index c9d72a8da..936372338 100644 --- a/packages/workflow/src/cloudflare/lifecycle-link.ts +++ b/packages/workflow/src/cloudflare/lifecycle-link.ts @@ -25,6 +25,7 @@ import type { import type { CreateWorkflowRunRequest } from "../storage/api.ts"; import type { RemoteFrontierSnapshot } from "../remote/read.ts"; import type { + RemoteBeginCommand, RemoteBegun, RemoteForkCommit, RemoteForkPart, @@ -122,20 +123,16 @@ export function cloudflareLifecycleLink( } return { - *begin(request: { - readonly runId: string; - readonly action: "start" | "resume"; - readonly creation: CreateWorkflowRunRequest | null; - readonly executionId: string; - }): Operation>> { + *begin(request: RemoteBeginCommand): Operation>> { try { const offered = yield* connection.ask( - nextId(), + request.commandId, { command: "begin", runId: request.runId, action: request.action, creation: request.creation, + retrieval: request.retrieval ?? null, executionId: request.executionId, }, (value: unknown) => value, @@ -158,12 +155,13 @@ export function cloudflareLifecycleLink( }, *settle( + commandId: string, completion: DocumentExecutionCompletion, expectedWorkspaceRootId: string, ): Operation> { try { const offered = yield* connection.ask( - nextId(), + commandId, { command: "settle", completion, expectedWorkspaceRootId }, (value: unknown) => value, privateRefusal, @@ -177,10 +175,13 @@ export function cloudflareLifecycleLink( } }, - *cancel(runId: string): Operation>> { + *cancel( + commandId: string, + runId: string, + ): Operation>> { try { const offered = yield* connection.ask( - nextId(), + commandId, { command: "cancel", runId }, (value: unknown) => value, privateRefusal, @@ -199,10 +200,10 @@ export function cloudflareLifecycleLink( } }, - *stageForkPart(part: RemoteForkPart): Operation> { + *stageForkPart(commandId: string, part: RemoteForkPart): Operation> { try { const offered = yield* connection.ask( - nextId(), + commandId, { command: "fork-stage", section: part.section, @@ -221,14 +222,15 @@ export function cloudflareLifecycleLink( } }, - *commitFork(commit: RemoteForkCommit): Operation> { + *commitFork(commit: RemoteForkCommit): Operation>> { try { const offered = yield* connection.ask( - nextId(), + commit.commandId, { command: "fork", runId: commit.runId, creation: commit.creation, + retrieval: commit.retrieval ?? null, origin: commit.origin, counts: commit.counts, runRecord: record(commit.runRecord), @@ -241,7 +243,19 @@ export function cloudflareLifecycleLink( if (offered.outcome === "refused") { return Err(storageFailure(privateRefusal(offered.refusal))); } - const found = members(offered.value, ["conflict", "value"]); + const found = members(offered.value, ["conflict", "refusal", "value"]); + const refusal = found.get("refusal"); + if (refusal !== null) { + if (refusal !== "cancelled" && refusal !== "resume-failed" && refusal !== "terminal") { + return Err( + new WorkflowRecordMalformedError( + "lifecycle answer this run's owner returned", + "it named no condition this build reads", + ), + ); + } + return Ok({ kind: "refused", refusal }); + } const conflict = found.get("conflict"); if (conflict !== null) { if (!Array.isArray(conflict) || conflict.length === 0) { @@ -261,10 +275,13 @@ export function cloudflareLifecycleLink( } const held = members(found.get("value"), ["frontier", "execution"]); return Ok({ - frontier: yield* frontierOf(held.get("frontier")), - execution: execution(held.get("execution")), - replay: false, - recovered: null, + kind: "performed", + value: { + frontier: yield* frontierOf(held.get("frontier")), + execution: execution(held.get("execution")), + replay: false, + recovered: null, + }, }); } catch (error) { return Err(translate(error)); diff --git a/packages/workflow/src/cloudflare/owner-fork.ts b/packages/workflow/src/cloudflare/owner-fork.ts index c3b7375d3..98029a32e 100644 --- a/packages/workflow/src/cloudflare/owner-fork.ts +++ b/packages/workflow/src/cloudflare/owner-fork.ts @@ -41,8 +41,12 @@ import { import type { OwnerStorage } from "./storage.ts"; import type { OwnerTransaction } from "./owner-transaction.ts"; import { establishRun } from "./owner-open.ts"; +import { beginRun, type LifecycleRefusal } from "./owner-lifecycle.ts"; import { readFrontier, validateRetainedRoot, type FrontierValue } from "./owner-reads.ts"; import { FORK_TABLE, holdExecution, STAGING_TABLE } from "./private-schema.ts"; +import { checkoutKey, forkSelectionAnchor } from "./fork-anchor.ts"; +import { forkRunRecordEvent, isRootImportEvent } from "../journal-events.ts"; +import { encodeBase64 } from "./encoding.ts"; import { readDocumentExecution, readRunRecord, type Row } from "../sqlite/rows.ts"; import type { DocumentExecutionRecord } from "../storage/record.ts"; @@ -55,6 +59,8 @@ export interface ForkedValue { /** A fork answer: the destination, or which immutable fields say it is another. */ export interface ForkValue { readonly conflict: readonly string[] | null; + /** Why the lifecycle would not continue, when it would not. */ + readonly refusal: LifecycleRefusal | null; readonly value: ForkedValue | null; } @@ -213,6 +219,7 @@ function staged( interface StagedRoot { readonly rootId: string; + readonly formatVersion: number; readonly manifest: string; readonly manifestHashes: readonly string[]; readonly blobHashes: readonly string[]; @@ -236,6 +243,7 @@ function parseRoot(part: Record): StagedRoot { } return { rootId, + formatVersion: WORKSPACE_ROOT_FORMAT, manifest, manifestHashes: list(found.get("manifestHashes")).map((value) => digest(value)), blobHashes: list(found.get("blobHashes")).map((value) => digest(value)), @@ -254,7 +262,12 @@ function sameOrder(left: readonly string[], right: readonly string[]): boolean { * reads the root back. A root whose arrays are the right set in the wrong order * is refused here rather than becoming unreadable later. */ -function retainRoot(storage: OwnerStorage, acquisitionId: string, root: StagedRoot): void { +function retainRoot( + storage: OwnerStorage, + acquisitionId: string, + root: StagedRoot, + watermarks: ReadonlyMap, +): void { const parsed = parseWorkspaceRootManifest(root.manifest, () => { throw new CommandError("malformed-member"); }); @@ -303,9 +316,12 @@ function retainRoot(storage: OwnerStorage, acquisitionId: string, root: StagedRo } const key = hexBytes(hash); storage.sql.exec( - "INSERT INTO vfs_blobs (hash, size, last_seen) VALUES (?, ?, 0) ON CONFLICT(hash) DO NOTHING", + "INSERT INTO vfs_blobs (hash, size, last_seen) VALUES (?, ?, ?) ON CONFLICT(hash) DO NOTHING", key, size, + // The watermark the source retained beside these bytes. A digest stands + // for the bytes and for their size; it does not stand for this. + watermarkOf(watermarks, hash), ); storage.sql.exec( "INSERT INTO vfs_blob_bytes (hash, bytes) VALUES (?, ?) ON CONFLICT(hash) DO NOTHING", @@ -315,13 +331,14 @@ function retainRoot(storage: OwnerStorage, acquisitionId: string, root: StagedRo } for (const [hash, bytes] of manifests) { storage.sql.exec( - `INSERT INTO vfs_manifests (hash, size, encoded, last_seen) VALUES (?, ?, ?, 0) + `INSERT INTO vfs_manifests (hash, size, encoded, last_seen) VALUES (?, ?, ?, ?) ON CONFLICT(hash) DO NOTHING`, hexBytes(hash), decodeContentManifest(bytes, () => { throw new CommandError("malformed-member"); }).size, bytes, + watermarkOf(watermarks, hash), ); } @@ -357,6 +374,17 @@ function retainRoot(storage: OwnerStorage, acquisitionId: string, root: StagedRo validateRetainedRoot(storage, root.rootId); } +/** What the source retained beside one piece of content. */ +function watermarkOf(watermarks: ReadonlyMap, hash: string): number { + const found = watermarks.get(hash); + if (found === undefined) { + // Every piece a root names was described by the transfer, or the anchor + // would not have matched. Reaching here would mean it did. + throw new CommandError("malformed-member"); + } + return found; +} + function hexBytes(digestHex: string): Uint8Array { const bytes = new Uint8Array(32); for (let at = 0; at < 32; at += 1) { @@ -477,6 +505,162 @@ function writeCheckout( ); } +/** + * Rebuild the source selection out of the parts that were offered. + * + * Every value is validated as it is read — roots against their own manifests, + * content against its digests, rows against the roots they name — and then + * described in the shared logical shape and hashed by the shared rule. The + * digest that comes out is comparable with the one the source computed only + * because neither side spells the selection for itself. + */ +function reconstruct( + storage: OwnerStorage, + acquisitionId: string, + input: { readonly origin: ForkOrigin; readonly counts: ForkCounts }, +): { anchor: string; rootIds: Set; watermarks: Map } { + const roots = section(storage, acquisitionId, "roots", input.counts.roots).map((part) => + parseRoot(part), + ); + const rootIds = new Set(roots.map((root) => root.rootId)); + if (rootIds.size !== roots.length) { + throw new CommandError("malformed-member"); + } + for (const rootId of [ + input.origin.checkpointWorkspaceRootId, + input.origin.runRecordWorkspaceRootId, + input.origin.rootImportWorkspaceRootId, + ]) { + if (!rootIds.has(rootId)) { + throw new CommandError("malformed-member"); + } + } + + const manifests = section(storage, acquisitionId, "manifests", input.counts.manifests).map( + (part) => parseContentMetadata(part), + ); + const blobs = section(storage, acquisitionId, "blobs", input.counts.blobs).map((part) => + parseContentMetadata(part), + ); + const inherited = section(storage, acquisitionId, "inherited", input.counts.inherited).map( + (part) => parseInherited(part, rootIds), + ); + const checkouts = section(storage, acquisitionId, "checkouts", input.counts.checkouts).map( + (part) => parseCheckoutPart(part), + ); + + const watermarks = new Map(); + for (const piece of [...manifests, ...blobs]) { + watermarks.set(piece.hash, piece.lastSeen); + } + + return { + rootIds, + watermarks, + anchor: forkSelectionAnchor({ + checkpointEventId: input.origin.checkpointEventId, + checkpointWorkspaceRootId: input.origin.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: input.origin.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: input.origin.rootImportWorkspaceRootId, + inherited, + roots, + manifests: manifests.map((piece) => ({ + hash: piece.hash, + size: piece.size, + lastSeen: piece.lastSeen, + // The bytes the source hashed, read back out of what was staged rather + // than taken on the word of the part that described them. + encoded: encodeBase64(staged(storage, acquisitionId, "manifest", piece.hash)), + })), + blobs, + checkouts, + }), + }; +} + +/** One manifest's or blob's retained metadata, as a part carries it. */ +function parseContentMetadata(part: Record): { + hash: string; + size: number; + lastSeen: number; +} { + const found = members(part, ["hash", "size", "lastSeen"]); + return { + hash: digest(found.get("hash")), + size: count(found.get("size")), + lastSeen: count(found.get("lastSeen")), + }; +} + +/** One inherited row, as a part carries it. */ +function parseInherited( + part: Record, + carried: ReadonlySet, +): { eventId: string; record: string; workspaceRootId: string } { + const found = members(part, ["eventId", "record", "workspaceRootId"]); + const rootId = digest(found.get("workspaceRootId")); + if (!carried.has(rootId)) { + throw new CommandError("malformed-member"); + } + const record = text(found.get("record")); + if (!parseDurableEvent(record).ok) { + throw new CommandError("corrupt-journal"); + } + return { eventId: text(found.get("eventId")), record, workspaceRootId: rootId }; +} + +/** One checkout, as the selection describes it: its key and its record. */ +function parseCheckoutPart(part: Record): { + key: string; + value: Record; +} { + const kind = part["kind"]; + if (kind === "repository") { + return { key: checkoutKey(["repository", text(part["name"])]), value: { ...part } }; + } + if (kind !== "worktree") { + throw new CommandError("malformed-member"); + } + return { + key: checkoutKey(["worktree", text(part["repositoryName"]), text(part["name"])]), + value: { ...part }, + }; +} + +/** + * The two records this fork writes for itself, held to what they must be. + * + * A generic parseable event is not enough where the command promises one + * specific role: the run record is the canonical event this destination's own + * identity, base and pinned commit imply, and the root import is a root import + * written against a root the transfer carried. + */ +function requireHeadRecords( + input: { + readonly runId: string; + readonly creation: CreateWorkflowRunRequest; + readonly origin: ForkOrigin; + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; + }, + carried: ReadonlySet, +): void { + const expected = forkRunRecordEvent({ + runId: input.runId, + base: input.creation.base, + pinnedCommit: input.creation.definition.objectId, + }); + if (serializeDurableEvent(input.runRecord) !== serializeDurableEvent(expected)) { + throw new CommandError("malformed-member"); + } + if (!isRootImportEvent(input.rootImport)) { + throw new CommandError("malformed-member"); + } + if (!carried.has(input.origin.rootImportWorkspaceRootId)) { + throw new CommandError("malformed-member"); + } +} + /** * Commit one fork: the destination and everything it inherited, together. * @@ -494,6 +678,7 @@ export function commitFork( input: { readonly runId: string; readonly creation: CreateWorkflowRunRequest; + readonly retrieval: string | null; readonly origin: ForkOrigin; readonly counts: ForkCounts; readonly runRecord: DurableEvent; @@ -508,9 +693,33 @@ export function commitFork( const fresh = rows(storage, "SELECT name FROM sqlite_schema WHERE type = 'table' AND name = 'workflow_run'") .length === 0; - const conflict = establishRun(storage, transaction, input.runId, input.creation, now); + + // Before anything semantic: prove that what was offered is the selection the + // source anchored. Everything below is derived from the staged parts, + // described in the shared shape and hashed by the shared rule, then compared. + // A transfer that is not that selection stops here, with the destination + // holding no run. + const staged = fresh ? reconstruct(storage, acquisitionId, input) : undefined; + const watermarks = staged?.watermarks ?? new Map(); + if (staged !== undefined) { + if (staged.anchor !== input.origin.anchor) { + throw new CommandError("stale-journal"); + } + // The two records this fork writes for itself are the fork's own, not any + // parseable event. + requireHeadRecords(input, staged.rootIds); + } + + const conflict = establishRun( + storage, + transaction, + input.runId, + input.creation, + now, + input.retrieval ?? undefined, + ); if (conflict !== null) { - return { conflict, value: null }; + return { conflict, refusal: null, value: null }; } if (!fresh) { // A destination that already exists is the same fork only when it came @@ -518,7 +727,7 @@ export function commitFork( // lineage. const lineage = rows( storage, - `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id + `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, selection_anchor FROM workflow_fork_lineage WHERE id = 1`, )[0]; if ( @@ -527,9 +736,43 @@ export function commitFork( lineage["checkpoint_event_id"] !== input.origin.checkpointEventId || lineage["checkpoint_workspace_root_id"] !== input.origin.checkpointWorkspaceRootId ) { - return { conflict: ["lineage"], value: null }; + return { conflict: ["lineage"], refusal: null, value: null }; + } + if (lineage["selection_anchor"] !== input.origin.anchor) { + // The same source and checkpoint, copied out of a different committed + // state. That is a different fork wearing this one's identity. + return { conflict: ["lineage"], refusal: null, value: null }; + } + // The fork is already here, so this is not making one: it is taking a run + // up again. It goes through the same recovery and admission a resume does, + // reconciling what the previous executor left before beginning exactly one + // replacement, rather than inserting a second open execution beside it. + const resumed = beginRun( + storage, + transaction, + acquisitionId, + input.runId, + "resume", + null, + null, + input.executionId, + now, + ); + if (resumed.conflict !== null) { + return { conflict: resumed.conflict, refusal: null, value: null }; + } + if (resumed.refusal !== null) { + return { conflict: null, refusal: resumed.refusal, value: null }; + } + const begun = resumed.value; + if (begun === null) { + throw new CommandError("malformed-member"); } - return { conflict: null, value: begunOn(storage, acquisitionId, input, now) }; + return { + conflict: null, + refusal: null, + value: { frontier: begun.frontier, execution: begun.execution }, + }; } const roots = section(storage, acquisitionId, "roots", input.counts.roots).map((part) => @@ -546,7 +789,7 @@ export function commitFork( } } for (const root of roots) { - retainRoot(storage, acquisitionId, root); + retainRoot(storage, acquisitionId, root, watermarks); } const checkpoint = roots.find((root) => root.rootId === input.origin.checkpointWorkspaceRootId); @@ -593,15 +836,19 @@ export function commitFork( storage.sql.exec( `INSERT INTO workflow_fork_lineage - (id, source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, created_at) - VALUES (1, ?, ?, ?, ?)`, + (id, source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, + selection_anchor, created_at) + VALUES (1, ?, ?, ?, ?, ?)`, input.origin.sourceRunId, input.origin.checkpointEventId, input.origin.checkpointWorkspaceRootId, + // The copy identity, retained with the lineage: a later fork of the same + // checkpoint is checkable here without reading the source at all. + input.origin.anchor, now(), ); - return { conflict: null, value: begunOn(storage, acquisitionId, input, now) }; + return { conflict: null, refusal: null, value: begunOn(storage, acquisitionId, input, now) }; } /** The fork's first execution, begun by the acquisition that committed it. */ diff --git a/packages/workflow/src/cloudflare/owner-lifecycle.ts b/packages/workflow/src/cloudflare/owner-lifecycle.ts index 3d1332531..82e18985e 100644 --- a/packages/workflow/src/cloudflare/owner-lifecycle.ts +++ b/packages/workflow/src/cloudflare/owner-lifecycle.ts @@ -240,6 +240,7 @@ export function beginRun( runId: string, action: "start" | "resume", creation: CreateWorkflowRunRequest | null, + retrieval: string | null, executionId: string, now: () => string, ): LifecycleValue { @@ -247,7 +248,7 @@ export function beginRun( // Creating and beginning are one commit — this runs inside the caller's // transaction — so a reader observes the whole begun run or no run at all, // never an initialized candidate with no execution. - const opened = establishRun(storage, transaction, runId, creation, now); + const opened = establishRun(storage, transaction, runId, creation, now, retrieval ?? undefined); if (opened !== null) { return { conflict: opened, refusal: null, value: null }; } diff --git a/packages/workflow/src/cloudflare/owner-open.ts b/packages/workflow/src/cloudflare/owner-open.ts index c8641f6df..611877b52 100644 --- a/packages/workflow/src/cloudflare/owner-open.ts +++ b/packages/workflow/src/cloudflare/owner-open.ts @@ -60,10 +60,24 @@ export function establishRun( runId: string, creation: CreateWorkflowRunRequest, now: () => string, + retrieval?: string, ): readonly string[] | null { if (pristine(storage)) { const stamp = now(); - initializeInside(storage, transaction, () => insertRun(storage, creation, stamp)); + initializeInside(storage, transaction, () => { + insertRun(storage, creation, stamp); + if (retrieval !== undefined) { + // Written with the run rather than by a later command: where a + // definition can be fetched from is part of what this caller created, + // and a run that had to be told twice could be told once and crash. + storage.sql.exec( + `INSERT INTO definition_retrieval (id, metadata, revision, updated_at) + VALUES (1, ?, 1, ?)`, + retrieval, + stamp, + ); + } + }); return null; } recognizeObject(storage); diff --git a/packages/workflow/src/cloudflare/private-schema.ts b/packages/workflow/src/cloudflare/private-schema.ts index 2855634b6..7e24a1017 100644 --- a/packages/workflow/src/cloudflare/private-schema.ts +++ b/packages/workflow/src/cloudflare/private-schema.ts @@ -118,7 +118,8 @@ const MUTATION_SQL = `CREATE TABLE ${MUTATION_TABLE} ( length(request_fingerprint) = 64 AND request_fingerprint NOT GLOB '*[^0-9a-f]*' ), response TEXT NOT NULL CHECK (json_valid(response)), - response_bytes INTEGER NOT NULL CHECK (response_bytes >= 0) + response_bytes INTEGER NOT NULL CHECK (response_bytes >= 0), + execution_id TEXT ) STRICT, WITHOUT ROWID`; const PRIVATE_OBJECTS = new Map([ @@ -188,6 +189,56 @@ export function discardPriorAcquisitions(storage: OwnerStorage, acquisitionId: s storage.sql.exec(`DELETE FROM ${HOLD_TABLE} WHERE acquisition_id <> ?`, acquisitionId); } +/** + * Adopt the execution a retained decision began, when nobody else holds it. + * + * The case this exists for: a mutation committed, its answer was lost, the + * connection that asked died, and a replacement acquisition asked the same + * question again. Re-observing the decision is not enough — the execution it + * began has to become this acquisition's, or the caller would be handed a run + * it cannot settle. The old acquisition is already gone by the time this runs, + * because its scratch and its hold went with it. + */ +export function adoptExecution( + storage: OwnerStorage, + acquisitionId: string, + commandId: string, +): void { + const decided = storage.sql + .exec(`SELECT execution_id FROM ${MUTATION_TABLE} WHERE command_id = ?`, commandId) + .toArray()[0]; + const executionId = decided?.["execution_id"]; + if (typeof executionId !== "string") { + return; + } + const open = storage.sql + .exec( + "SELECT execution_id FROM document_executions WHERE execution_id = ? AND stopped_at IS NULL", + executionId, + ) + .toArray()[0]; + if (open === undefined) { + // Finished since. There is nothing to authorize, and the decision stands + // as the answer it always was. + return; + } + const holder = storage.sql + .exec(`SELECT acquisition_id FROM ${HOLD_TABLE} WHERE execution_id = ?`, executionId) + .toArray()[0]; + if (holder !== undefined) { + if (holder["acquisition_id"] !== acquisitionId) { + // Somebody live began it. Two acquisitions cannot hold one execution. + throw new Error("private protocol storage holds a conflicting execution hold"); + } + return; + } + storage.sql.exec( + `INSERT INTO ${HOLD_TABLE} (acquisition_id, execution_id) VALUES (?, ?)`, + acquisitionId, + executionId, + ); +} + /** Which execution this acquisition began, when it has begun one. */ export function heldExecution(storage: OwnerStorage, acquisitionId: string): string | undefined { const row = storage.sql diff --git a/packages/workflow/src/cloudflare/read-plane.ts b/packages/workflow/src/cloudflare/read-plane.ts index bd36b2feb..bca51f760 100644 --- a/packages/workflow/src/cloudflare/read-plane.ts +++ b/packages/workflow/src/cloudflare/read-plane.ts @@ -28,6 +28,13 @@ import { recognizeObject } from "./recognition.ts"; import { isRootImportEvent, isRunRecordEvent } from "../journal-events.ts"; import { parseDurableEvent } from "@executablemd/durable-streams"; import { compareUtf8, parseWorkspaceRootManifest } from "../workspace/root-manifest.ts"; +import { + type AnchorBlob, + type AnchorManifest, + type AnchorRoot, + checkoutKey, + forkSelectionAnchor, +} from "./fork-anchor.ts"; import { encodeBase64, sha256Hex } from "./encoding.ts"; import { retainedBytes, @@ -519,26 +526,86 @@ function selectionAnchor( checkoutPaths: ReadonlySet; }, ): string { - return sha256Hex( - JSON.stringify({ - checkpointEventId: selection.checkpointEventId, - checkpointWorkspaceRootId: selection.checkpointWorkspaceRootId, - runRecordWorkspaceRootId: selection.runRecordWorkspaceRootId, - rootImportWorkspaceRootId: selection.rootImportWorkspaceRootId, - inherited: selection.inherited.map((row) => [row.eventId, row.record, row.workspaceRootId]), - roots: selection.rootIds.map((rootId) => readStoredRoot(storage, rootId)), - // The metadata, not only the identities. A digest stands for its bytes - // and for the size derived from them, but not for a watermark: that is - // copied into destination storage and can change while the content does - // not. - manifests: selection.manifestHashes.map((hash) => readStoredManifest(storage, hash)), - blobs: selection.blobHashes.map((hash) => blobMetadata(storage, hash)), - checkouts: [ - ...readCheckoutRepositories(storage, selection.checkoutPaths), - ...readCheckoutWorktrees(storage, selection.checkoutPaths), - ].map((entry) => [entry.cursor, entry.value]), - }), - ); + // Built here from what this owner retains, and hashed by the shared rule a + // destination will hash its own copy with. The two sides read from different + // places on purpose; what they must not do is describe the selection + // differently. + return forkSelectionAnchor({ + checkpointEventId: selection.checkpointEventId, + checkpointWorkspaceRootId: selection.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: selection.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: selection.rootImportWorkspaceRootId, + inherited: selection.inherited, + roots: selection.rootIds.map((rootId) => anchorRoot(storage, rootId)), + manifests: selection.manifestHashes.map((hash) => anchorManifest(storage, hash)), + blobs: selection.blobHashes.map((hash) => anchorBlob(storage, hash)), + checkouts: [ + ...readCheckoutRepositories(storage, selection.checkoutPaths), + ...readCheckoutWorktrees(storage, selection.checkoutPaths), + ].map((entry) => ({ key: entry.cursor, value: entry.value })), + }); +} + +/** One root, as the shared selection describes it. */ +function anchorRoot(storage: OwnerStorage, rootId: string): AnchorRoot { + const read = readStoredRoot(storage, rootId); + return { + rootId: retainedTextOf(read, "rootId"), + formatVersion: retainedNumberOf(read, "formatVersion"), + manifest: retainedTextOf(read, "manifest"), + manifestHashes: retainedListOf(read, "manifestHashes"), + blobHashes: retainedListOf(read, "blobHashes"), + }; +} + +/** One content manifest, as the shared selection describes it. */ +function anchorManifest(storage: OwnerStorage, hash: string): AnchorManifest { + const read = readStoredManifest(storage, hash); + return { + hash, + size: retainedNumberOf(read, "size"), + lastSeen: retainedNumberOf(read, "lastSeen"), + encoded: retainedTextOf(read, "encoded"), + }; +} + +/** One blob's metadata, as the shared selection describes it. */ +function anchorBlob(storage: OwnerStorage, hash: string): AnchorBlob { + const read = blobMetadata(storage, hash); + return { + hash, + size: retainedNumberOf(read, "size"), + lastSeen: retainedNumberOf(read, "lastSeen"), + }; +} + +function retainedTextOf(value: Record, name: string): string { + const found = value[name]; + if (typeof found !== "string") { + throw new CommandError("corrupt-journal"); + } + return found; +} + +function retainedNumberOf(value: Record, name: string): number { + const found = value[name]; + if (typeof found !== "number") { + throw new CommandError("corrupt-journal"); + } + return found; +} + +function retainedListOf(value: Record, name: string): string[] { + const found = value[name]; + if (!Array.isArray(found)) { + throw new CommandError("corrupt-journal"); + } + return found.map((entry) => { + if (typeof entry !== "string") { + throw new CommandError("corrupt-journal"); + } + return entry; + }); } /** The directories the checkpoint's own Workspace held. */ @@ -782,10 +849,6 @@ function readStoredBlob(storage: OwnerStorage, hash: string): Record>; + /** The no-acquisition read plane for one run. */ + source(runId: string): Operation>; +} + +/** + * Install the remote executor lifecycle for a Deno runner. + * + * The connections it stages through belong to this scope, so a candidate's + * database handles and files go when the scope that asked for them ends. + */ +export function* installRemoteWorkflowLifecycle( + options: RemoteWorkflowLifecycleOptions, + connections?: WorkflowRunConnections, +): Operation { + const root = authorizedRoot(options.root); + const held = connections ?? (yield* useWorkflowRunConnections()); + return yield* useRemoteLifecycle({ + admit: options.admit, + source: options.source, + *stage( + request: WorkflowForkRequest, + source: RemoteForkSource, + head: { readonly runRecord: DurableEvent; readonly rootImport: DurableEvent }, + ): Operation> { + return yield* stageRemoteFork(held, root, request, source, head); + }, + ids: { + // Opaque and minted here, never taken from a caller: an identity a + // document could choose is one two runs could share. + execution: () => randomUUID(), + command: () => randomUUID(), + }, + }); +} diff --git a/packages/workflow/src/deno/remote-staging.ts b/packages/workflow/src/deno/remote-staging.ts new file mode 100644 index 000000000..c1fefe664 --- /dev/null +++ b/packages/workflow/src/deno/remote-staging.ts @@ -0,0 +1,132 @@ +/** + * Assembling a remote fork's candidate on this runner's own disk. + * + * A fork is admitted by replaying it, and a replay needs the fork's own + * Workspace: a `` resolves through the run's filesystem, and a candidate + * without one produces effects of a different kind and diverges for a reason + * that has nothing to do with the candidate. So the snapshot read from a remote + * source is assembled here, in full, at a staging path — and thrown away when + * the scope that asked for it ends. + * + * This is the same assembly a committed fork gets. It is the local host's own + * `stageFork()` kernel, handed the same snapshot shape it always takes; nothing + * here writes a second fork writer, and nothing re-reads the source. What this + * module is, is the translation: a `RemoteForkSource` says its digests in hex + * because that is what crossed a wire, and the local snapshot says them in + * bytes because that is what SQLite holds. + */ + +import { type Operation, type Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import type { WorkflowForkRequest } from "../lifecycle/execution.ts"; +import type { WorkflowRunDatabase } from "../storage/api.ts"; +import type { RemoteForkSource } from "../remote/read.ts"; +import type { ForkSourceSnapshot } from "./fork-source.ts"; +import type { WorkflowRunConnections } from "./connections.ts"; +import { stageFork } from "./transitions.ts"; +import { workflowForkStaging } from "./path.ts"; + +/** One digest, as the store holds it rather than as a wire spells it. */ +function bytesOfHex(hex: string): Uint8Array { + const bytes = new Uint8Array(hex.length / 2); + for (let at = 0; at < bytes.length; at += 1) { + bytes[at] = Number.parseInt(hex.slice(at * 2, at * 2 + 2), 16); + } + return bytes; +} + +/** + * The same snapshot, said the way this host says it. + * + * Nothing is recomputed and nothing is dropped: the retained record strings + * cross byte for byte, the roots keep their canonical reference arrays, the + * content keeps the watermarks copied beside it, and the checkouts keep every + * member the destination will retain. + */ +export function localForkSnapshot(source: RemoteForkSource): ForkSourceSnapshot { + return { + sourceRunId: source.sourceRunId, + checkpointEventId: source.checkpointEventId, + checkpointWorkspaceRootId: source.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: source.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: source.rootImportWorkspaceRootId, + inherited: source.inherited.map((row) => ({ + eventId: row.eventId, + record: row.record, + workspaceRootId: row.workspaceRootId, + })), + roots: source.roots.map((root) => ({ + rootId: root.rootId, + formatVersion: root.formatVersion, + manifest: root.manifest, + manifestHashes: [...root.manifestHashes], + blobHashes: [...root.blobHashes], + })), + manifests: source.manifests.map((manifest) => ({ + hash: bytesOfHex(manifest.hash), + size: manifest.size, + encoded: manifest.encoded, + lastSeen: manifest.lastSeen, + })), + blobs: source.blobs.map((blob) => ({ + hash: bytesOfHex(blob.hash), + size: blob.size, + lastSeen: blob.lastSeen, + content: blob.content, + })), + repositories: source.checkouts.flatMap((checkout) => + checkout.kind === "repository" + ? [ + { + name: checkout.name, + locator: checkout.locator, + locatorFingerprint: checkout.locatorFingerprint, + requestedBase: checkout.requestedBase, + creationCommit: checkout.creationCommit, + primaryBranch: checkout.primaryBranch, + objectFormat: checkout.objectFormat, + checkoutPath: checkout.checkoutPath, + }, + ] + : [], + ), + worktrees: source.checkouts.flatMap((checkout) => + checkout.kind === "worktree" + ? [ + { + repositoryName: checkout.repositoryName, + name: checkout.name, + requestedBranch: checkout.requestedBranch, + requestedBase: checkout.requestedBase, + creationCommit: checkout.creationCommit, + checkoutPath: checkout.checkoutPath, + }, + ] + : [], + ), + }; +} + +/** + * Build one remote fork's candidate locally, owned by the calling scope. + * + * The staging file is scratch: a leftover from an attempt that did not finish + * is replaced rather than continued, and whatever happens — success, failure or + * cancellation — it goes when the scope ends. Nothing about it is a run: no + * lock is taken anywhere, no owner is contacted, and no host discovers it. + */ +export function stageRemoteFork( + connections: WorkflowRunConnections, + root: string, + request: WorkflowForkRequest, + source: RemoteForkSource, + head: { readonly runRecord: DurableEvent; readonly rootImport: DurableEvent }, +): Operation> { + return stageFork( + connections, + workflowForkStaging(root, request.runId), + request, + localForkSnapshot(source), + head, + ); +} diff --git a/packages/workflow/src/fork.ts b/packages/workflow/src/fork.ts index 423296b72..d00c394d7 100644 --- a/packages/workflow/src/fork.ts +++ b/packages/workflow/src/fork.ts @@ -39,7 +39,9 @@ import { describeWorkflowRun, WORKFLOW_RUN, type WorkflowRun } from "./journal.t import type { Forkability } from "./lifecycle/forkability.ts"; import { WorkflowRequestError } from "./storage/errors.ts"; import { isRootImportEvent, isRunRecordEvent } from "./journal-events.ts"; -export { isRootImportEvent, isRunRecordEvent } from "./journal-events.ts"; +import { forkRunRecordEvent } from "./journal-events.ts"; + +export { forkRunRecordEvent, isRootImportEvent, isRunRecordEvent } from "./journal-events.ts"; /** The coroutine a run's own record and canonical outcome belong to. */ const ROOT_COROUTINE = "root"; @@ -121,25 +123,6 @@ export function selectForkPrefix( }); } -/** - * The record the fork writes at position zero, exactly as its own execution - * would have written it. - * - * Composed here rather than in a host, so the value a fork is admitted with and - * the value its first execution replays are the same shape by construction. - */ -export function forkRunRecordEvent(run: WorkflowRun): DurableEvent { - return { - type: "yield", - coroutineId: ROOT_COROUTINE, - description: describeWorkflowRun(run.base), - result: { - status: "ok", - value: { runId: run.runId, base: run.base, pinnedCommit: run.pinnedCommit }, - }, - }; -} - /** * The fork's logical journal: its own two head records, then what it inherited. * diff --git a/packages/workflow/src/journal-events.ts b/packages/workflow/src/journal-events.ts index 73f66aa21..c9da3d2d1 100644 --- a/packages/workflow/src/journal-events.ts +++ b/packages/workflow/src/journal-events.ts @@ -13,7 +13,7 @@ */ import type { DurableEvent } from "@executablemd/durable-streams"; -import { WORKFLOW_RUN } from "./journal.ts"; +import { describeWorkflowRun, WORKFLOW_RUN } from "./journal.ts"; const ROOT_COROUTINE = "root"; const IMPORT_COMPONENT = "import_component"; @@ -37,3 +37,27 @@ export function isRunRecordEvent(event: DurableEvent): boolean { event.description.name === WORKFLOW_RUN ); } + +/** + * The record a fork writes at position zero, exactly as its own execution would + * have written it. + * + * Composed here rather than in a host, so the value a fork is admitted with, + * the value its destination owner validates, and the value its first execution + * replays are the same shape by construction. + */ +export function forkRunRecordEvent(run: { + readonly runId: string; + readonly base: string; + readonly pinnedCommit: string; +}): DurableEvent { + return { + type: "yield", + coroutineId: ROOT_COROUTINE, + description: describeWorkflowRun(run.base), + result: { + status: "ok", + value: { runId: run.runId, base: run.base, pinnedCommit: run.pinnedCommit }, + }, + }; +} diff --git a/packages/workflow/src/remote/lifecycle-link.ts b/packages/workflow/src/remote/lifecycle-link.ts index dc67b157a..8fdbde4bf 100644 --- a/packages/workflow/src/remote/lifecycle-link.ts +++ b/packages/workflow/src/remote/lifecycle-link.ts @@ -14,7 +14,7 @@ */ import type { Operation, Result } from "effection"; -import type { DurableEvent } from "@executablemd/durable-streams"; +import type { DurableEvent, Json } from "@executablemd/durable-streams"; import type { DocumentExecutionCompletion, DocumentExecutionRecord, @@ -58,20 +58,37 @@ export interface RemoteForkOrigin { export interface RemoteForkCounts { readonly inherited: number; readonly roots: number; + readonly manifests: number; + readonly blobs: number; readonly checkouts: number; } /** One part of a fork's source, offered before any of it is a run. */ export interface RemoteForkPart { - readonly section: "inherited" | "roots" | "checkouts"; + readonly section: "inherited" | "roots" | "manifests" | "blobs" | "checkouts"; readonly position: number; readonly part: Record; } +/** One begin, as the provider addresses it and may address it again. */ +export interface RemoteBeginCommand { + /** The identity this logical invocation keeps, retry after retry. */ + readonly commandId: string; + readonly runId: string; + readonly action: "start" | "resume"; + readonly creation: CreateWorkflowRunRequest | null; + /** Where the definition can be fetched from, when this begin creates. */ + readonly retrieval: Json | undefined; + readonly executionId: string; +} + /** Everything one committed fork is decided from. */ export interface RemoteForkCommit { + /** The identity this logical invocation keeps, retry after retry. */ + readonly commandId: string; readonly runId: string; readonly creation: CreateWorkflowRunRequest; + readonly retrieval: Json | undefined; readonly origin: RemoteForkOrigin; readonly counts: RemoteForkCounts; readonly runRecord: DurableEvent; @@ -101,21 +118,20 @@ export interface RemoteExecutorConnection { */ export interface RemoteLifecycleLink { /** Begin one document execution under this acquisition. */ - begin(request: { - readonly runId: string; - readonly action: "start" | "resume"; - readonly creation: CreateWorkflowRunRequest | null; - readonly executionId: string; - }): Operation>>; + begin(request: RemoteBeginCommand): Operation>>; /** Finish the execution this acquisition began. */ settle( + commandId: string, completion: DocumentExecutionCompletion, expectedWorkspaceRootId: string, ): Operation>; /** Make this run terminal, following what it retains. */ - cancel(runId: string): Operation>>; + cancel( + commandId: string, + runId: string, + ): Operation>>; /** Offer one part of a fork's source to this acquisition's scratch. */ - stageForkPart(part: RemoteForkPart): Operation>; + stageForkPart(commandId: string, part: RemoteForkPart): Operation>; /** Commit the offered parts as one destination run and its first execution. */ - commitFork(commit: RemoteForkCommit): Operation>; + commitFork(commit: RemoteForkCommit): Operation>>; } diff --git a/packages/workflow/src/remote/lifecycle.ts b/packages/workflow/src/remote/lifecycle.ts index 52e7d1a6f..0e380c022 100644 --- a/packages/workflow/src/remote/lifecycle.ts +++ b/packages/workflow/src/remote/lifecycle.ts @@ -48,6 +48,8 @@ import { type WorkflowStorageError, } from "../storage/errors.ts"; import { useRemoteRunDatabase } from "./database.ts"; +import { canonicalJson } from "../storage/record.ts"; +import { definitionToJson } from "../storage/definition.ts"; import type { RemoteForkSource, RemoteReadPlane } from "./read.ts"; import { forkRunRecordEvent } from "../fork.ts"; import type { @@ -88,7 +90,23 @@ export interface RemoteLifecycleHost { head: { readonly runRecord: DurableEvent; readonly rootImport: DurableEvent }, ): Operation>; /** Fresh identities for this provider's own commands and executions. */ - readonly ids: { readonly execution: () => string }; + readonly ids: { readonly execution: () => string; readonly command: () => string }; +} + +/** + * One logical lifecycle invocation, as it is addressed and re-addressed. + * + * Minted once and kept by the run it belongs to rather than by the connection + * that first sent it. A connection that died between an owner's commit and its + * answer is exactly the case this exists for: the replacement acquisition asks + * the same question, with the same identity and the same bytes, and the owner + * answers with what it already decided instead of deciding again. + */ +interface Invocation { + readonly commandId: string; + readonly executionId: string; + /** What this invocation asked, so a different question gets a new identity. */ + readonly shape: string; } /** What one issued lock is allowed to do, and what it has already done. */ @@ -115,6 +133,29 @@ export function* useRemoteLifecycle( // Keyed by the object itself: two locks are the same lock when they are the // same object, and nothing about their fields is consulted. const held = new Map(); + // Keyed by the run and the question asked, so the same question after a lost + // answer is the same invocation. Not authority: what it carries is an + // identity, and the owner decides what that identity already means. + const invocations = new Map(); + + function invocation(runId: string, shape: string): Invocation { + const key = `${runId}\u0000${shape}`; + const found = invocations.get(key); + if (found !== undefined) { + return found; + } + const minted: Invocation = { + commandId: host.ids.command(), + executionId: host.ids.execution(), + shape, + }; + invocations.set(key, minted); + return minted; + } + + function settled(runId: string, shape: string): void { + invocations.delete(`${runId}\u0000${shape}`); + } function hold(lock: ExecutorLock): Hold | undefined { // Fabricated, copied, foreign, released and closed locks all answer @@ -172,10 +213,13 @@ export function* useRemoteLifecycle( ), ); } - const answered = yield* admitted.value.lifecycle.cancel(runId); + const shape = questionOf(["cancel"]); + const addressed = invocation(runId, shape); + const answered = yield* admitted.value.lifecycle.cancel(addressed.commandId, runId); if (!answered.ok) { return answered; } + settled(runId, shape); if (answered.value.kind === "refused") { return Err( new WorkflowRequestError( @@ -189,16 +233,22 @@ export function* useRemoteLifecycle( }); } - yield* WorkflowLifecycle.around({ - *acquireExecutor([runId]): Operation> { - return yield* acquireExecutor(runId); - }, - *cancel([runId]): Operation> { - return yield* cancel(runId); + yield* WorkflowLifecycle.around( + { + *acquireExecutor([runId]): Operation> { + return yield* acquireExecutor(runId); + }, + *cancel([runId]): Operation> { + return yield* cancel(runId); + }, }, - }); + // Nearest wins, the same way storage and the read provider install: a + // scope that installed this one answers with it, not with whatever an + // enclosing scope happened to install first. + { at: "min" }, + ); - return transitions(host, hold); + return transitions(host, hold, invocation, settled); } /** What an unrecognized lock answers, wherever one is offered. */ @@ -211,6 +261,8 @@ function unauthorized(): WorkflowRequestError { function transitions( host: RemoteLifecycleHost, hold: (lock: ExecutorLock) => Hold | undefined, + invocation: (runId: string, shape: string) => Invocation, + settled: (runId: string, shape: string) => void, ): WorkflowExecutionTransitions { return { *begin( @@ -236,21 +288,38 @@ function transitions( if (request.action === "resume" && request.creation !== undefined) { return Err(new WorkflowRequestError("a resume does not carry a creation.")); } - // Minted once, outside anything that could retry: the owner recognizes a - // repeat by this identity, and a fresh one would be a second execution. - const executionId = host.ids.execution(); + // Minted once for this question, outside anything that could retry: the + // owner recognizes a repeat by this identity, and a fresh one would be a + // second execution. A later acquisition asking the same question finds + // the same identity and re-observes the decision. + const creation = creationOf(request.runId, request.creation); + const shape = questionOf([ + "begin", + request.action, + creation === null ? null : creationShape(creation), + request.creation?.retrieval === undefined + ? null + : canonicalJson(request.creation.retrieval), + ]); + const addressed = invocation(request.runId, shape); const answered = yield* held.connection.lifecycle.begin({ + commandId: addressed.commandId, runId: request.runId, action: request.action, - creation: creationOf(request.runId, request.creation), - executionId, + creation, + retrieval: request.creation?.retrieval, + executionId: addressed.executionId, }); if (!answered.ok) { + // The answer is lost, not the question. The identity stays, so the + // next acquisition asks the same one. return answered; } if (answered.value.kind === "refused") { + settled(request.runId, shape); return Err(refusalError(answered.value.refusal, request.runId)); } + settled(request.runId, shape); held.execution = answered.value.value.execution.executionId; return Ok(yield* begun(held, answered.value.value)); }, @@ -276,13 +345,22 @@ function transitions( // The root the owner is held to comes from the same connection-owned // frontier the execution ran against, after the host has torn down. const frontier = yield* held.connection.link.frontierSnapshot(); + const shape = questionOf([ + "settle", + completion.executionId, + completion.status, + frontier.workspaceRootId, + ]); + const addressed = invocation(held.runId, shape); const answered = yield* held.connection.lifecycle.settle( + addressed.commandId, completion, frontier.workspaceRootId, ); if (!answered.ok) { return answered; } + settled(held.runId, shape); held.execution = undefined; return Ok(answered.value.record); }, @@ -311,13 +389,21 @@ function transitions( if (!source.ok) { return source; } - const staged = yield* offer(held.connection.lifecycle, source.value); + const staged = yield* offer(host, held.connection.lifecycle, source.value); if (!staged.ok) { return staged; } - const executionId = host.ids.execution(); + const shape = questionOf([ + "fork", + request.runId, + source.value.anchor, + source.value.checkpointEventId, + ]); + const addressed = invocation(request.runId, shape); const answered = yield* held.connection.lifecycle.commitFork({ + commandId: addressed.commandId, runId: request.runId, + retrieval: request.creation.retrieval, creation: creationRequest(request.runId, request.creation), origin: { sourceRunId: source.value.sourceRunId, @@ -330,17 +416,24 @@ function transitions( counts: { inherited: source.value.inherited.length, roots: source.value.roots.length, + manifests: source.value.manifests.length, + blobs: source.value.blobs.length, checkouts: source.value.checkouts.length, }, runRecord: headOf(request).runRecord, rootImport: request.rootImport, - executionId, + executionId: addressed.executionId, }); if (!answered.ok) { return answered; } - held.execution = answered.value.execution.executionId; - return Ok(yield* begun(held, answered.value)); + if (answered.value.kind === "refused") { + settled(request.runId, shape); + return Err(refusalError(answered.value.refusal, request.runId)); + } + settled(request.runId, shape); + held.execution = answered.value.value.execution.executionId; + return Ok(yield* begun(held, answered.value.value)); }, *stageFork(request: WorkflowForkRequest): Operation> { @@ -388,6 +481,27 @@ function headOf(request: WorkflowForkRequest): { }; } +/** One creation, as text: the fields a run's identity is compared by. */ +function creationShape(creation: CreateWorkflowRunRequest): string { + return canonicalJson({ + runId: creation.runId, + definition: definitionToJson(creation.definition), + base: creation.base, + props: creation.props, + }); +} + +/** + * What one invocation is asking, as text two calls can be compared by. + * + * Canonical, so the same question always spells the same way, and a different + * one never spells like it. This decides only whether a retry is the same + * logical invocation; what that identity already means is the owner's to say. + */ +function questionOf(parts: readonly (string | null)[]): string { + return canonicalJson([...parts]); +} + /** Read one source through the accepted no-acquisition plane. */ function* readSource( host: RemoteLifecycleHost, @@ -414,7 +528,11 @@ function* readSource( * that name where they belong. Nothing here is a run: the final command decides * whether these add up to one. */ -function* offer(lifecycle: RemoteLifecycleLink, source: RemoteForkSource): Operation> { +function* offer( + host: RemoteLifecycleHost, + lifecycle: RemoteLifecycleLink, + source: RemoteForkSource, +): Operation> { const parts: RemoteForkPart[] = []; source.roots.forEach((root, position) => { parts.push({ @@ -429,6 +547,23 @@ function* offer(lifecycle: RemoteLifecycleLink, source: RemoteForkSource): Opera }, }); }); + source.manifests.forEach((manifest, position) => { + // The metadata, not the bytes: the bytes cross through the content staging + // the publication path already uses, and what a digest cannot stand for is + // the watermark copied beside it. + parts.push({ + section: "manifests", + position, + part: { hash: manifest.hash, size: manifest.size, lastSeen: manifest.lastSeen }, + }); + }); + source.blobs.forEach((blob, position) => { + parts.push({ + section: "blobs", + position, + part: { hash: blob.hash, size: blob.size, lastSeen: blob.lastSeen }, + }); + }); source.inherited.forEach((row, position) => { parts.push({ section: "inherited", @@ -440,7 +575,7 @@ function* offer(lifecycle: RemoteLifecycleLink, source: RemoteForkSource): Opera parts.push({ section: "checkouts", position, part: { ...checkout } }); }); for (const part of parts) { - const staged = yield* lifecycle.stageForkPart(part); + const staged = yield* lifecycle.stageForkPart(host.ids.command(), part); if (!staged.ok) { return staged; } diff --git a/packages/workflow/src/sqlite/workflow-schema.ts b/packages/workflow/src/sqlite/workflow-schema.ts index df89b9e9e..563a5e686 100644 --- a/packages/workflow/src/sqlite/workflow-schema.ts +++ b/packages/workflow/src/sqlite/workflow-schema.ts @@ -458,6 +458,10 @@ export const OBJECTS: ReadonlyMap = new Map([ checkpoint_event_id TEXT NOT NULL CHECK (length(checkpoint_event_id) > 0), checkpoint_workspace_root_id TEXT NOT NULL REFERENCES workspace_roots(root_id) ON DELETE RESTRICT, + selection_anchor TEXT CHECK ( + selection_anchor IS NULL + OR (length(selection_anchor) = 64 AND selection_anchor NOT GLOB '*[^0-9a-f]*') + ), created_at TEXT NOT NULL ) STRICT`, }, diff --git a/packages/workflow/tests/cloudflare/remote-fork.vitest.ts b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts index bcabd30dd..a4a4819ae 100644 --- a/packages/workflow/tests/cloudflare/remote-fork.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts @@ -23,6 +23,8 @@ import { VALID_CLAIMS, } from "./support/executor-object.ts"; import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; +import { forkSelectionAnchor } from "../../src/cloudflare/fork-anchor.ts"; +import { forkRunRecordEvent } from "../../src/journal-events.ts"; let unique = 0; const NOW = 1_800_000_000; @@ -84,15 +86,11 @@ function event(name: string): string { }); } -const HEAD = serializeDurableEvent({ - type: "yield", - coroutineId: "root", - description: { type: "workflow_run", name: "workflow_run", base: "main" }, - result: { - status: "ok", - value: { runId: RUN_ID, base: "main", pinnedCommit: "0".repeat(40) }, - }, -}); +// The canonical record this destination's own identity implies. Composed by +// the shared helper, because the owner holds the head to exactly that. +const HEAD = serializeDurableEvent( + forkRunRecordEvent({ runId: RUN_ID, base: "main", pinnedCommit: "0".repeat(40) }), +); const IMPORT = serializeDurableEvent({ type: "yield", @@ -101,6 +99,37 @@ const IMPORT = serializeDurableEvent({ result: { status: "ok", value: { kind: "repository", path: "README.md", content: "# fork" } }, }); +const ROOT_PART = { + rootId: ROOT_ID, + formatVersion: 1, + manifest: ROOT_MANIFEST, + manifestHashes: [MANIFEST_ID], + blobHashes: [BLOB_ID], +}; +const MANIFEST_PART = { hash: MANIFEST_ID, size: FILE_BYTES.length, lastSeen: 7 }; +const BLOB_PART = { hash: BLOB_ID, size: FILE_BYTES.length, lastSeen: 9 }; +const INHERITED_PART = { + eventId: "event-work", + record: event("work"), + workspaceRootId: ROOT_ID, +}; + +/** The anchor this selection has, computed the way the source computes it. */ +function anchorOf(overrides: Record = {}): string { + return forkSelectionAnchor({ + checkpointEventId: "event-work", + checkpointWorkspaceRootId: ROOT_ID, + runRecordWorkspaceRootId: ROOT_ID, + rootImportWorkspaceRootId: ROOT_ID, + inherited: [INHERITED_PART], + roots: [ROOT_PART], + manifests: [{ ...MANIFEST_PART, encoded: base64(new TextEncoder().encode(DOFS_MANIFEST)) }], + blobs: [BLOB_PART], + checkouts: [], + ...overrides, + }); +} + /** Offer everything one small source is made of, as the runner would. */ async function offer(stub: ReturnType, id: () => string): Promise { expect( @@ -132,13 +161,7 @@ async function offer(stub: ReturnType, id: () => string): Promi command: "fork-stage", section: "roots", position: 0, - part: { - rootId: ROOT_ID, - formatVersion: 1, - manifest: ROOT_MANIFEST, - manifestHashes: [MANIFEST_ID], - blobHashes: [BLOB_ID], - }, + part: ROOT_PART, }) )["outcome"], ).toBe("performed"); @@ -149,10 +172,28 @@ async function offer(stub: ReturnType, id: () => string): Promi command: "fork-stage", section: "inherited", position: 0, - part: { eventId: "event-work", record: event("work"), workspaceRootId: ROOT_ID }, + part: INHERITED_PART, }) )["outcome"], ).toBe("performed"); + // The metadata a digest cannot stand for, offered beside the content. + const metadata: { section: string; part: Record }[] = [ + { section: "manifests", part: MANIFEST_PART }, + { section: "blobs", part: BLOB_PART }, + ]; + for (const offered of metadata) { + expect( + ( + await ask(stub, { + id: id(), + command: "fork-stage", + section: offered.section, + position: 0, + part: offered.part, + }) + )["outcome"], + ).toBe("performed"); + } } /** The event ids one published snapshot reports, in journal order. */ @@ -186,9 +227,10 @@ function commit(overrides: Record = {}): Record { checkpointWorkspaceRootId: ROOT_ID, runRecordWorkspaceRootId: "b".repeat(64), rootImportWorkspaceRootId: ROOT_ID, - anchor: "f".repeat(64), + anchor: anchorOf({ runRecordWorkspaceRootId: "b".repeat(64) }), }, }), ); @@ -303,7 +345,7 @@ describe("committing a fork on its destination owner", () => { await connected(stub); let minted = 0; await offer(stub, () => `command-${(minted += 1)}`); - expect(await on(stub, (owner) => owner.forkParts())).toHaveLength(2); + expect(await on(stub, (owner) => owner.forkParts())).toHaveLength(4); // A replacement acquisition inherits nothing its predecessor offered. await on(stub, (owner) => owner.dropConnections()); @@ -311,4 +353,151 @@ describe("committing a fork on its destination owner", () => { expect(await on(stub, (owner) => owner.forkParts())).toEqual([]); }); + + it("refuses an anchor that is not this selection's, before it creates anything", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + + // Well-formed, and not the digest this selection produces. + const refused = await ask( + stub, + commit({ + origin: { + sourceRunId: SOURCE_RUN_ID, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: ROOT_ID, + runRecordWorkspaceRootId: ROOT_ID, + rootImportWorkspaceRootId: ROOT_ID, + anchor: "f".repeat(64), + }, + }), + ); + + expect(refused["outcome"]).toBe("refused"); + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("refuses a changed watermark, which no content digest stands for", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + const id = () => `command-${(minted += 1)}`; + // Everything as before, except the blob's copied watermark. + await ask(stub, { + id: id(), + command: "stage", + kind: "manifest", + digest: MANIFEST_ID, + bytes: base64(new TextEncoder().encode(DOFS_MANIFEST)), + }); + await ask(stub, { + id: id(), + command: "stage", + kind: "blob", + digest: BLOB_ID, + bytes: base64(FILE_BYTES), + }); + await ask(stub, { + id: id(), + command: "fork-stage", + section: "roots", + position: 0, + part: ROOT_PART, + }); + await ask(stub, { + id: id(), + command: "fork-stage", + section: "inherited", + position: 0, + part: INHERITED_PART, + }); + await ask(stub, { + id: id(), + command: "fork-stage", + section: "manifests", + position: 0, + part: MANIFEST_PART, + }); + await ask(stub, { + id: id(), + command: "fork-stage", + section: "blobs", + position: 0, + part: { ...BLOB_PART, lastSeen: BLOB_PART.lastSeen + 1 }, + }); + + const refused = await ask(stub, commit()); + expect(refused["outcome"]).toBe("refused"); + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("refuses a head record that is not the one this fork's identity implies", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + + const refused = await ask( + stub, + commit({ + runRecord: serializeDurableEvent( + forkRunRecordEvent({ runId: RUN_ID, base: "other", pinnedCommit: "0".repeat(40) }), + ), + }), + ); + + expect(refused["outcome"]).toBe("refused"); + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("retains the watermarks the source copied rather than starting them again", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + + const watermarks = await on(stub, (owner) => owner.contentWatermarks()); + expect(watermarks.manifests).toEqual([MANIFEST_PART.lastSeen]); + expect(watermarks.blobs).toEqual([BLOB_PART.lastSeen]); + }); + + it("recovers the previous executor's work before a later fork begins again", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + expect((await ask(stub, commit()))["outcome"]).toBe("performed"); + + // The executor that committed the fork is gone with its execution open. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + await offer(stub, () => `later-${(minted += 1)}`); + const again = await ask(stub, commit({ id: "fork-again", executionId: "execution-2" })); + + expect(again["outcome"]).toBe("performed"); + const executions = await on(stub, (owner) => owner.executionRows()); + // The first was closed by recovery, and exactly one replacement began. + expect(executions).toHaveLength(2); + expect(executions[0]?.["stop_status"]).toBe("interrupted"); + expect(executions[1]?.["stopped_at"]).toBe(null); + expect(executions.filter((row) => row["stopped_at"] === null)).toHaveLength(1); + }); + + it("writes the retrieval its creation carried, with the run", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit({ retrieval: { kind: "git", remote: "origin" } })); + + const retrieval = await on(stub, (owner) => owner.retrieval()); + expect(retrieval?.["revision"]).toBe(1); + expect(JSON.parse(String(retrieval?.["metadata"]))).toEqual({ + kind: "git", + remote: "origin", + }); + }); }); diff --git a/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts index 3ed40c2dd..16819d78a 100644 --- a/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts @@ -85,6 +85,7 @@ async function started( runId: RUN_ID, action: "start", creation: creation(), + retrieval: null, executionId, }); } @@ -182,6 +183,7 @@ describe("a run's lifecycle on its owner", () => { runId: RUN_ID, action: "resume", creation: null, + retrieval: null, executionId: "execution-2", }); @@ -216,6 +218,7 @@ describe("a run's lifecycle on its owner", () => { runId: RUN_ID, action: "start", creation: creation(), + retrieval: null, executionId: "execution-2", }); @@ -236,6 +239,7 @@ describe("a run's lifecycle on its owner", () => { runId: RUN_ID, action: "resume", creation: null, + retrieval: null, executionId: "execution-2", }); @@ -268,4 +272,130 @@ describe("a run's lifecycle on its owner", () => { expect(state.executions).toHaveLength(1); expect(state.executions[0]?.["stop_status"]).toBe("interrupted"); }); + + it("writes the retrieval a start carried, with the run it creates", async () => { + const stub = executor(); + await connected(stub); + + const answered = await ask(stub, { + id: "command-1", + command: "begin", + runId: RUN_ID, + action: "start", + creation: creation(), + retrieval: { kind: "git", remote: "origin" }, + executionId: "execution-1", + }); + + expect(answered["outcome"]).toBe("performed"); + const retrieval = await on(stub, (owner) => owner.retrieval()); + // Revision one, written in the transaction that made the run. + expect(retrieval?.["revision"]).toBe(1); + expect(JSON.parse(String(retrieval?.["metadata"]))).toEqual({ + kind: "git", + remote: "origin", + }); + }); + + it("writes no retrieval row when a start carries none", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + + expect(await on(stub, (owner) => owner.retrieval())).toBe(null); + }); + + it("refuses a retrieval nothing is being created for, and one too large", async () => { + const stub = executor(); + await connected(stub); + + // A resume creates nothing, so there is nothing for a retrieval to belong + // to. Carrying one is a request this build does not answer. + const orphan = await ask(stub, { + id: "command-1", + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + retrieval: { kind: "git" }, + executionId: "execution-1", + }); + // Refused while the command was still being read, so the answer names no + // command at all. + expect(orphan).toEqual({ id: "", outcome: "refused", refusal: "command:malformed-member" }); + + const huge = await ask(stub, { + id: "command-2", + command: "begin", + runId: RUN_ID, + action: "start", + creation: creation(), + retrieval: { remote: "x".repeat(2 * 1024 * 1024) }, + executionId: "execution-1", + }); + expect(huge["outcome"]).toBe("refused"); + + // Neither one made anything. + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("leaves an existing run's retrieval alone when it is taken up again", async () => { + const stub = executor(); + await connected(stub); + await ask(stub, { + id: "command-1", + command: "begin", + runId: RUN_ID, + action: "start", + creation: creation(), + retrieval: { kind: "git", remote: "origin" }, + executionId: "execution-1", + }); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + await ask(stub, { + id: "command-2", + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + retrieval: null, + executionId: "execution-2", + }); + + const retrieval = await on(stub, (owner) => owner.retrieval()); + // Replaceable state, not identity: a resume neither compares it nor + // clears it. + expect(retrieval?.["revision"]).toBe(1); + }); + + it("hands a replacement acquisition the execution a lost answer began", async () => { + const stub = executor(); + await connected(stub); + const first = await started(stub, "command-1", "execution-1"); + expect(first["outcome"]).toBe("performed"); + + // The answer never arrived and the connection died. The replacement asks + // the same question, with the same identity. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const again = await started(stub, "command-1", "execution-1"); + + expect(again).toEqual(first); + // One execution, and it is this acquisition's to settle now. + expect(await on(stub, (owner) => owner.executionRows())).toHaveLength(1); + const held = await on(stub, (owner) => owner.heldExecutions()); + expect(held).toHaveLength(1); + expect(held[0]?.["execution_id"]).toBe("execution-1"); + + const root = await on(stub, (owner) => owner.currentRootId()); + const settled = await ask(stub, { + id: "command-3", + command: "settle", + completion: { executionId: "execution-1", status: "completed" }, + expectedWorkspaceRootId: root, + }); + expect(settled["outcome"]).toBe("performed"); + }); }); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 697b6c84c..c580c9620 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -349,6 +349,16 @@ export class ExecutorObject extends WorkflowOwnerObject { return this.ctx.storage.sql.exec(`SELECT * FROM ${HOLD_TABLE}`).toArray(); } + /** The watermarks retained beside this run's copied content. */ + contentWatermarks(): { manifests: number[]; blobs: number[] } { + const read = (table: string) => + this.ctx.storage.sql + .exec(`SELECT last_seen FROM ${table} ORDER BY last_seen`) + .toArray() + .map((row) => Number(row["last_seen"])); + return { manifests: read("vfs_manifests"), blobs: read("vfs_blobs") }; + } + /** Every fork part this owner is holding for anyone. */ forkParts(): Record[] { return this.ctx.storage.sql diff --git a/packages/workflow/tests/remote-fork.test.ts b/packages/workflow/tests/remote-fork.test.ts index aedb4bdd7..38b52c88f 100644 --- a/packages/workflow/tests/remote-fork.test.ts +++ b/packages/workflow/tests/remote-fork.test.ts @@ -134,16 +134,25 @@ describe("a remote fork's destination", () => { expect([outcome.ok, outcome.ok === false && String(outcome.error)]).toEqual([true, false]); // Everything was offered, and the commit came last. expect(asked.at(-1)).toBe("fork"); - expect(asked.filter((command) => command === "fork-stage")).toHaveLength(4); + expect(asked.filter((command) => command === "fork-stage")).toHaveLength(6); expect(staged.map((part) => `${part.section}:${part.position}`)).toEqual([ "roots:0", + // The metadata a digest cannot stand for travels beside the content. + "manifests:0", + "blobs:0", "inherited:0", "inherited:1", "checkouts:0", ]); // What the final command claims is what was offered, and where it came // from is the selection that was read. - expect(commits[0]?.counts).toEqual({ inherited: 2, roots: 1, checkouts: 1 }); + expect(commits[0]?.counts).toEqual({ + inherited: 2, + roots: 1, + manifests: 1, + blobs: 1, + checkouts: 1, + }); expect(commits[0]?.origin).toEqual({ sourceRunId: SOURCE_RUN_ID, checkpointEventId: "event-work", diff --git a/packages/workflow/tests/remote-lifecycle.test.ts b/packages/workflow/tests/remote-lifecycle.test.ts index 7eaf31e02..1ad0a4598 100644 --- a/packages/workflow/tests/remote-lifecycle.test.ts +++ b/packages/workflow/tests/remote-lifecycle.test.ts @@ -204,4 +204,95 @@ describe("a remote run's executor lifecycle", () => { // rather than installing an object that answers only its own. expect(outcome.asked).toBe(true); }); + + it("answers from the nearest provider, over one installed further out", function* () { + const answered: string[] = []; + const outcome = yield* scoped(function* () { + // A provider in an enclosing scope, installed the way every provider in + // this repository installs. Nothing should reach it. + yield* WorkflowLifecycle.around( + { + // deno-lint-ignore require-yield + *acquireExecutor(): Operation> { + answered.push("outer-acquire"); + throw new WorkflowRequestError("the outer provider answered acquireExecutor"); + }, + // deno-lint-ignore require-yield + *cancel(): Operation> { + answered.push("outer-cancel"); + throw new WorkflowRequestError("the outer provider answered cancel"); + }, + // deno-lint-ignore require-yield + *inspect(): Operation> { + answered.push("outer-inspect"); + throw new WorkflowRequestError("the outer provider answered inspect"); + }, + }, + { at: "min" }, + ); + return yield* scoped(function* () { + // The read provider, then the lifecycle provider, both nearer the work. + yield* WorkflowLifecycle.around( + { + // deno-lint-ignore require-yield + *inspect(): Operation> { + answered.push("inner-inspect"); + throw new WorkflowRequestError("the inner read provider answered inspect"); + }, + }, + { at: "min" }, + ); + yield* useRemoteLifecycle(installedHost({})); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + const cancelled = yield* WorkflowLifecycle.operations.cancel(RUN_ID); + try { + yield* WorkflowLifecycle.operations.inspect(RUN_ID); + } catch { + // The read provider answers by raising; which one raised is what is + // being observed. + } + return { taken, cancelled }; + }); + }); + + expect(outcome.taken.ok).toBe(true); + expect(outcome.cancelled.ok).toBe(true); + // The nearest provider answered its own operations, the read provider + // installed beside it still answered its own, and the outer one answered + // nothing at all. + expect(answered).toEqual(["inner-inspect"]); + }); + + it("asks the same question after a lost answer, and gets one decision", function* () { + const commands: string[] = []; + const loseAnswer = new Set(); + const committed = new Map(); + const script: Script = { commands, loseAnswer, committed: committed as never }; + const outcome = yield* scoped(function* () { + const transitions = yield* useRemoteLifecycle(installedHost(script)); + const first = yield* scoped(function* () { + const lock = yield* acquired(); + // The owner will commit and the answer will be lost. + loseAnswer.add("command-1"); + return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + }); + // The connection is gone with its answer. A replacement acquisition asks + // the same question. + const second = yield* scoped(function* () { + const lock = yield* acquired(); + return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + }); + return { first, second }; + }); + + expect(outcome.first.ok).toBe(false); + expect(outcome.second.ok).toBe(true); + // The same command identity both times, so the owner answered with the + // decision it had already made rather than making a second one. + expect(commands).toEqual(["command-1", "command-1"]); + if (outcome.second.ok) { + // And the execution the caller is handed is the one that was begun. + expect(outcome.second.value.execution.executionId).toBe("execution-1"); + } + }); }); diff --git a/packages/workflow/tests/remote-staged-fork.test.ts b/packages/workflow/tests/remote-staged-fork.test.ts new file mode 100644 index 000000000..d21a98990 --- /dev/null +++ b/packages/workflow/tests/remote-staged-fork.test.ts @@ -0,0 +1,296 @@ +/** + * Tier WRH — the candidate a remote fork is admitted by, assembled locally. + * + * A staged fork is not a run: nothing acquires anything, no owner is contacted, + * and no host discovers it. What it is, is the fork's own database, built from + * the snapshot the accepted no-acquisition plane returned and thrown away with + * the scope that asked for it. These are the observations that distinguish a + * real candidate from an interface: its identity, its inherited history in + * order with the records byte for byte, its Workspace root, its checkouts and + * its lineage, read back out of the database it handed over. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { call, Err, Ok, type Operation, type Result, scoped } from "effection"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import { stat } from "node:fs/promises"; +import { useStorageRoot } from "./support/storage.ts"; +import { useWorkflowRunConnections } from "../src/deno/connections.ts"; +import { stageRemoteFork } from "../src/deno/remote-staging.ts"; +import { workflowForkStaging } from "../src/deno/path.ts"; +import { installRemoteWorkflowLifecycle } from "../src/deno/remote-host.ts"; +import { WorkflowRequestError } from "../src/storage/errors.ts"; +import type { WorkflowForkRequest } from "../src/lifecycle/execution.ts"; +import type { RemoteForkSource } from "../src/remote/read.ts"; +import { forkRunRecordEvent } from "../src/journal-events.ts"; +import { sha256Hex } from "../src/workspace/sha256.ts"; +import { WORKSPACE_ROOT_DOMAIN } from "../src/workspace/root-manifest.ts"; +import type { WorkflowRunDatabase } from "../src/storage/api.ts"; + +const SOURCE_RUN_ID = "6dktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +const DESTINATION = "7ektgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + +const FILE = new TextEncoder().encode("inherited by the fork"); +const BLOB = sha256Hex(FILE); +const CONTENT = new TextEncoder().encode( + JSON.stringify({ version: 1, chunks: [{ hash: BLOB, size: FILE.length }] }), +); +const MANIFEST = sha256Hex(CONTENT); +const ROOT_MANIFEST = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/NOTES.md", + kind: "file", + mode: 420, + mtime: 0, + size: FILE.length, + manifest: MANIFEST, + hardlink: null, + }, + ], +}); +const ROOT = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${ROOT_MANIFEST}`); + +function event(name: string): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "test", name }, + result: { status: "ok", value: name }, + }); +} + +/** One small committed source, as the accepted read plane returns it. */ +function source(): RemoteForkSource { + return { + sourceRunId: SOURCE_RUN_ID, + anchor: "a".repeat(64), + checkpointEventId: "event-second", + checkpointWorkspaceRootId: ROOT, + runRecordWorkspaceRootId: ROOT, + rootImportWorkspaceRootId: ROOT, + inherited: [ + { eventId: "event-first", record: event("first"), workspaceRootId: ROOT }, + { eventId: "event-second", record: event("second"), workspaceRootId: ROOT }, + ], + roots: [ + { + rootId: ROOT, + formatVersion: 1, + manifest: ROOT_MANIFEST, + manifestHashes: [MANIFEST], + blobHashes: [BLOB], + }, + ], + manifests: [{ hash: MANIFEST, size: FILE.length, lastSeen: 4, encoded: CONTENT }], + blobs: [{ hash: BLOB, size: FILE.length, lastSeen: 6, content: FILE }], + checkouts: [ + { + kind: "repository", + name: "alpha", + locator: "https://git.example.invalid/alpha.git", + locatorFingerprint: "b".repeat(64), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/", + }, + ], + }; +} + +function request(): WorkflowForkRequest { + return { + runId: DESTINATION, + selection: { sourceRunId: SOURCE_RUN_ID, checkpointEventId: "event-second" }, + creation: { + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + retrieval: { kind: "git", remote: "origin" }, + }, + rootImport: { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path: "README.md", content: "# fork" } }, + }, + }; +} + +/** Stage one candidate and observe it, inside a scope that then ends. */ +function* staged( + root: string, + body: (database: WorkflowRunDatabase) => Operation, +): Operation { + return yield* scoped(function* () { + const connections = yield* useWorkflowRunConnections(); + const built = yield* stageRemoteFork(connections, root, request(), source(), { + runRecord: forkRunRecordEvent({ + runId: DESTINATION, + base: "main", + pinnedCommit: "0".repeat(40), + }), + rootImport: request().rootImport, + }); + if (!built.ok) { + throw built.error; + } + return yield* body(built.value); + }); +} + +function* exists(path: string): Operation { + return yield* call(async function () { + try { + await stat(path); + return true; + } catch { + return false; + } + }); +} + +describe("a remote fork's staged candidate", () => { + it("is a real database holding the fork's own identity and history", function* () { + const root = yield* useStorageRoot(); + const seen = yield* staged(root, function* (database) { + const history = yield* database.readJournalEntries(); + if (!history.ok) { + throw history.error; + } + return { + record: database.record, + events: history.value.map((entry) => entry.eventId), + events2: history.value.map((entry) => serializeDurableEvent(entry.event)), + }; + }); + + // Its own identity, not the source's. + expect(seen.record.runId).toBe(DESTINATION); + expect(seen.record.base).toBe("main"); + // Its own two head records, then the prefix it inherited, in order. + expect(seen.events).toHaveLength(4); + expect(seen.events.slice(2)).toEqual(["event-first", "event-second"]); + // The events the source retained, in the source's order. + expect(seen.events2.slice(2)).toEqual([event("first"), event("second")]); + }); + + it("restores the checkpoint's Workspace as the fork's own", function* () { + const root = yield* useStorageRoot(); + const seen = yield* staged(root, function* (database) { + const history = yield* database.readJournalEntries(); + if (!history.ok) { + throw history.error; + } + // Every retained row names the Workspace root it was written against; + // the fork's own rows name the checkpoint's. + return { + workspaceRootId: history.value.at(-1)?.workspaceRootId ?? "", + }; + }); + + expect(seen.workspaceRootId).toBe(ROOT); + }); + + it("keeps the retrieval its creation carried", function* () { + const root = yield* useStorageRoot(); + const held = yield* staged(root, function* (database) { + return database.retrieval; + }); + + expect(held?.metadata).toEqual({ kind: "git", remote: "origin" }); + }); + + it("takes no lock, and nothing discovers it", function* () { + const root = yield* useStorageRoot(); + const path = workflowForkStaging(root, DESTINATION); + const during = yield* staged(root, function* () { + return yield* exists(path); + }); + + // It was there while its scope was open, and the run's own path never was: + // staging assembles a candidate, not a run. + expect(during).toBe(true); + expect(yield* exists(path)).toBe(false); + }); + + it("replaces what an interrupted attempt left rather than continuing it", function* () { + const root = yield* useStorageRoot(); + const first = yield* staged(root, function* (database) { + const history = yield* database.readJournalEntries(); + return history.ok ? history.value.length : -1; + }); + // A second candidate at the same path is built from scratch, so it holds + // exactly what one assembly holds rather than two. + const second = yield* staged(root, function* (database) { + const history = yield* database.readJournalEntries(); + return history.ok ? history.value.length : -1; + }); + + expect(first).toBe(4); + expect(second).toBe(4); + }); + + it("is what the provider's own stageFork() returns, through the real host", function* () { + const root = yield* useStorageRoot(); + const opened: string[] = []; + const seen = yield* scoped(function* () { + const transitions = yield* installRemoteWorkflowLifecycle({ + root, + // Reaching an owner is the one thing a staged fork must never do, so + // this records any attempt and refuses. + *admit(runId: string) { + opened.push(runId); + return Err(new WorkflowRequestError("a staged fork admits nothing")); + }, + // deno-lint-ignore require-yield + *source(runId: string) { + return Ok({ + runId, + // deno-lint-ignore require-yield + *inspect(): Operation> { + throw new WorkflowRequestError("a staged fork inspects nothing"); + }, + // deno-lint-ignore require-yield + *history(): Operation> { + throw new WorkflowRequestError("a staged fork reads no history"); + }, + // deno-lint-ignore require-yield + *forkSource(): Operation> { + return Ok(source()); + }, + }); + }, + }); + const built = yield* transitions.stageFork(request()); + if (!built.ok) { + throw built.error; + } + const history = yield* built.value.readJournalEntries(); + return { + runId: built.value.record.runId, + events: history.ok ? history.value.map((entry) => entry.eventId) : [], + retrieval: built.value.retrieval?.metadata, + }; + }); + + // A real candidate came back from the production seam, with the same + // assembly the kernel produces. + expect(seen.runId).toBe(DESTINATION); + expect(seen.events.slice(2)).toEqual(["event-first", "event-second"]); + expect(seen.retrieval).toEqual({ kind: "git", remote: "origin" }); + // And nothing was acquired on the way. + expect(opened).toEqual([]); + }); +}); diff --git a/packages/workflow/tests/support/remote-lifecycle-host.ts b/packages/workflow/tests/support/remote-lifecycle-host.ts index 935af0939..5eabed31f 100644 --- a/packages/workflow/tests/support/remote-lifecycle-host.ts +++ b/packages/workflow/tests/support/remote-lifecycle-host.ts @@ -12,6 +12,7 @@ import { ensure, Err, Ok, type Operation, type Result } from "effection"; import type { DurableEvent } from "@executablemd/durable-streams"; import type { + RemoteBeginCommand, RemoteBegun, RemoteExecutorConnection, RemoteForkCommit, @@ -32,7 +33,7 @@ import type { DocumentExecutionRecord, WorkflowRunRecord, } from "../../src/storage/record.ts"; -import { WorkflowRequestError } from "../../src/storage/errors.ts"; +import { WorkflowRequestError, WorkflowTransactionError } from "../../src/storage/errors.ts"; import type { RemoteWorkspaceLink } from "../../src/remote/database.ts"; export const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; @@ -62,6 +63,14 @@ export interface Script { readonly staged?: RemoteForkPart[]; /** Every fork commit the provider asked for. */ readonly commits?: RemoteForkCommit[]; + /** Every command identity the provider addressed, in order. */ + readonly commands?: string[]; + /** Every retrieval value a begin carried. */ + readonly retrievals?: (unknown | null)[]; + /** Command identities whose first answer is lost after the owner commits. */ + readonly loseAnswer?: Set; + /** What an owner decided for a command identity, once it has decided. */ + readonly committed?: Map; } export function record(): WorkflowRunRecord { @@ -154,10 +163,21 @@ function lifecycle(script: Script): RemoteLifecycleLink { let minted = 0; return { // deno-lint-ignore require-yield - *begin(request: { - readonly executionId: string; - }): Operation>> { + *begin(request: RemoteBeginCommand): Operation>> { script.asked?.push("begin"); + script.commands?.push(request.commandId); + script.retrievals?.push(request.retrieval ?? null); + if (script.loseAnswer?.has(request.commandId) === true) { + // The owner committed and the answer never arrived. + script.loseAnswer.delete(request.commandId); + script.committed?.set(request.commandId, begun(request.executionId, script)); + return Err(new WorkflowTransactionError("the connection ended before it answered.")); + } + const already = script.committed?.get(request.commandId); + if (already !== undefined) { + // The same question again: the decision it already made. + return Ok({ kind: "performed", value: already }); + } if (script.begin !== undefined) { return Ok({ kind: "refused", refusal: script.begin }); } @@ -165,30 +185,45 @@ function lifecycle(script: Script): RemoteLifecycleLink { return Ok({ kind: "performed", value: begun(request.executionId, script) }); }, // deno-lint-ignore require-yield - *settle(_completion: DocumentExecutionCompletion): Operation> { + *settle( + commandId: string, + _completion: DocumentExecutionCompletion, + ): Operation> { script.asked?.push("settle"); + script.commands?.push(commandId); return Ok(frontier()); }, // deno-lint-ignore require-yield - *cancel(): Operation>> { + *cancel(commandId: string): Operation>> { script.asked?.push("cancel"); + script.commands?.push(commandId); return Ok({ kind: "performed", value: record() }); }, // deno-lint-ignore require-yield - *stageForkPart(part: RemoteForkPart): Operation> { + *stageForkPart(_commandId: string, part: RemoteForkPart): Operation> { script.asked?.push("fork-stage"); script.staged?.push(part); return Ok(undefined); }, // deno-lint-ignore require-yield - *commitFork(commit: RemoteForkCommit): Operation> { + *commitFork(commit: RemoteForkCommit): Operation>> { script.asked?.push("fork"); script.commits?.push(commit); if (script.forkConflict !== undefined) { return Err(new WorkflowRequestError("this destination is another run")); } void minted; - return Ok(begun(commit.executionId)); + script.commands?.push(commit.commandId); + if (script.loseAnswer?.has(commit.commandId) === true) { + script.loseAnswer.delete(commit.commandId); + script.committed?.set(commit.commandId, begun(commit.executionId, script)); + return Err(new WorkflowTransactionError("the connection ended before it answered.")); + } + const already = script.committed?.get(commit.commandId); + if (already !== undefined) { + return Ok({ kind: "performed", value: already }); + } + return Ok({ kind: "performed", value: begun(commit.executionId, script) }); }, }; } @@ -196,6 +231,7 @@ function lifecycle(script: Script): RemoteLifecycleLink { /** The provider's host, scripted. */ export function installedHost(script: Script): RemoteLifecycleHost { let executions = 0; + let commands = 0; return { *admit(runId: string): Operation> { if (script.admit === "already-running") { @@ -241,6 +277,10 @@ export function installedHost(script: Script): RemoteLifecycleHost { executions += 1; return `execution-${executions}`; }, + command: () => { + commands += 1; + return `command-${commands}`; + }, }, }; } From 79a1e6e4dbdaaee24fc7beaff4370ac3da7c0f05 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Mon, 7 Sep 2026 17:53:08 -0300 Subject: [PATCH 53/81] =?UTF-8?q?=F0=9F=90=9B=20Own=20a=20lifecycle=20call?= =?UTF-8?q?,=20and=20continue=20a=20fork=20from=20what=20it=20retains?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two calls that look alike were treated as one. A provider that keeps its identities by what a request says cannot tell a retry from a second attempt, so two identical begins both passed the check and both sent, and the second was handed the first one's execution. A call is now owned rather than described: the acquisition is taken for the duration of a transition, so a second one refuses while the first is still waiting for its answer as well as after it returns. Only one thing may reuse an identity, and only for one reason. A call whose answer never arrived might already have committed, so its identity and the exact bytes it sent are kept and asked again. Anything the owner actually decided — a value, a refusal, a conflict — finishes that question, and a corrected request afterwards is a new call with new identities rather than one poisoned by the last. A committed fork is independent of the run it was copied from, and now behaves that way. Taking one up asks the destination first: whether it already holds this fork, and whether a decision was already made here. Only when the destination has nothing is the source read and staged. So a fork whose answer was lost can be re-observed when the source is gone, and one that is already here continues without the source existing at all. What makes a destination this fork is what it retains. Its immutable identity, its lineage, and both records it wrote for itself: a request carrying a different root import is a different fork however much else agrees, and it is refused without beginning anything. A retained decision is not authority by itself. The execution it began has to still be open, unheld and unrecovered before a replacement acquisition may take it up; if the run moved on — recovered by a later executor, settled, or held by somebody live — the decision is history, and answering with it would hand back a database nobody may settle. So that case refuses and the caller recovers the ordinary way. What the lifecycle decided is what the caller hears. A fork that continued a terminal destination says it replayed, and the execution recovery closed on the way in is reported beside the one that began, rather than both being replaced by defaults on the way out. A digest identifies bytes, not a role. The same bytes can be one manifest's encoding and another manifest's chunk, and the two tables retain their own watermarks, so the metadata a transfer carries is kept in two namespaces and neither answers for the other. --- packages/workflow/src/cloudflare/commands.ts | 42 ++- .../workflow/src/cloudflare/dispatcher.ts | 52 +++- .../workflow/src/cloudflare/lifecycle-link.ts | 130 ++++++--- .../workflow/src/cloudflare/owner-fork.ts | 186 ++++++++++++- .../workflow/src/cloudflare/private-schema.ts | 20 +- .../workflow/src/remote/lifecycle-link.ts | 19 ++ packages/workflow/src/remote/lifecycle.ts | 255 +++++++++++++----- .../tests/cloudflare/remote-fork.vitest.ts | 240 +++++++++++++++++ .../cloudflare/remote-lifecycle.vitest.ts | 46 ++++ packages/workflow/tests/remote-fork.test.ts | 83 ++++++ .../workflow/tests/remote-lifecycle.test.ts | 52 +++- .../workflow/tests/remote-staged-fork.test.ts | 15 +- .../tests/support/remote-lifecycle-host.ts | 26 ++ 13 files changed, 1019 insertions(+), 147 deletions(-) diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index b9aa71886..fe56e26dd 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -72,7 +72,8 @@ export type CommandName = | "cancel" | "settle" | "fork-stage" - | "fork"; + | "fork" + | "fork-continue"; export type CommandRefusal = | "not-an-object" @@ -342,6 +343,22 @@ export interface ForkCommand extends CommandEnvelope { readonly executionId: string; } +/** + * Take up a destination that already holds this fork. + * + * No origin and no counts: a committed fork is independent of the run it was + * copied from, so continuing one asks only what the destination itself + * retains. + */ +export interface ForkContinueCommand extends CommandEnvelope { + readonly command: "fork-continue"; + readonly runId: string; + readonly creation: CreateWorkflowRunRequest; + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; + readonly executionId: string; +} + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -363,7 +380,8 @@ export type RunnerCommand = | CancelCommand | SettleCommand | ForkStageCommand - | ForkCommand; + | ForkCommand + | ForkContinueCommand; export type CommandResult = | { readonly id: string; readonly outcome: "performed"; readonly value: unknown } @@ -394,6 +412,7 @@ const MEMBERS: Record = { cancel: [...ENVELOPE, "runId"], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], "fork-stage": [...ENVELOPE, "section", "position", "part"], + "fork-continue": [...ENVELOPE, "runId", "creation", "runRecord", "rootImport", "executionId"], fork: [ ...ENVELOPE, "runId", @@ -539,7 +558,8 @@ export function parseCommand(raw: string): RunnerCommand { command !== "cancel" && command !== "settle" && command !== "fork-stage" && - command !== "fork" + command !== "fork" && + command !== "fork-continue" ) { throw new CommandError("unknown-command"); } @@ -671,6 +691,22 @@ export function parseCommand(raw: string): RunnerCommand { part: Object.fromEntries(Object.entries(part)), }; } + if (command === "fork-continue") { + const runId = text(members, "runId", MAX_RUN_ID); + const creation = parseCreateRequest(members.get("creation")); + if (!creation.ok || creation.value.runId !== runId) { + throw new CommandError("malformed-member"); + } + return { + id, + command, + runId, + creation: creation.value, + runRecord: forkEvent(members.get("runRecord")), + rootImport: forkEvent(members.get("rootImport")), + executionId: text(members, "executionId", MAX_RUN_ID), + }; + } if (command === "fork") { const runId = text(members, "runId", MAX_RUN_ID); const creation = parseCreateRequest(members.get("creation")); diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 576501cbc..aaebabf8a 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -64,7 +64,7 @@ import { applyCommit, applyRetrieval } from "./publish.ts"; import { holdsNoRun, recognizeObject } from "./recognition.ts"; import { openRun } from "./owner-open.ts"; import { beginRun, cancelRunOnOwner, settleRun } from "./owner-lifecycle.ts"; -import { commitFork, discardForkParts, stageForkPart } from "./owner-fork.ts"; +import { commitFork, continueFork, discardForkParts, stageForkPart } from "./owner-fork.ts"; function requestFingerprint(command: RunnerCommand): string { // The command name is part of the fingerprint, so one textual id used for a @@ -120,7 +120,8 @@ function mutating(command: RunnerCommand): boolean { command.command === "settle" || // A committed fork creates a destination run. Its decision has to outlive // the connection for the same reason a begin's does. - command.command === "fork" + command.command === "fork" || + command.command === "fork-continue" ); } @@ -402,6 +403,25 @@ function perform( } return { id: command.id, outcome: "performed", value: forked }; } + if (command.command === "fork-continue") { + return { + id: command.id, + outcome: "performed", + value: continueFork( + ctx.storage, + transaction, + acquisitionId, + { + runId: command.runId, + creation: command.creation, + runRecord: command.runRecord, + rootImport: command.rootImport, + executionId: command.executionId, + }, + ownerTime, + ), + }; + } if (command.command === "cancel") { return { id: command.id, @@ -474,12 +494,15 @@ export function dispatchCommand( const empty = holdsNoRun(ctx.storage); const creating = initializes(command) && empty; const offering = offersScratch(command) && empty; + // Asking whether a destination already holds a fork is a question a store + // with no run can answer: the answer is that it does not. + const asking = command.command === "fork-continue" && empty; if (offering) { // The scratch this command needs, and nothing else: no schema, no run, no // marker. What is here after it is still a store holding no run. initializePrivateSchema(ctx.storage); } - if (!creating && !offering) { + if (!creating && !offering && !asking) { recognizeObject(ctx.storage); } @@ -488,7 +511,7 @@ export function dispatchCommand( // gone: the owner committed, the answer never arrived, and the runner // reconnected to ask the same question again. Pristine storage retains no // decision, and its private substrate does not exist yet to be asked. - if (mutating(command) && !creating) { + if (mutating(command) && !creating && !asking) { const decided = ctx.storage.sql .exec( `SELECT request_fingerprint, response FROM ${MUTATION_TABLE} WHERE command_id = ?`, @@ -506,15 +529,19 @@ export function dispatchCommand( throw new Error("private protocol storage holds a malformed result"); } // The answer was lost, not the fact. If this decision began an - // execution and nobody live holds it, it becomes this acquisition's — - // otherwise the caller would be handed a run it could not settle. - adoptExecution(ctx.storage, held.acquisitionId, command.id); + // execution and nobody live holds it, it becomes this acquisition's. + // If the run moved past it — recovered, settled, or held by somebody + // live — the decision is history rather than authority, and handing it + // back would hand back a database nobody may settle. + if (adoptExecution(ctx.storage, held.acquisitionId, command.id) === "stale") { + throw new CommandError("stale-journal"); + } return decision; } } const previous = - creating && !offering + (creating || asking) && !offering ? undefined : ctx.storage.sql .exec( @@ -536,7 +563,7 @@ export function dispatchCommand( // A store with no run has spent nothing this ledger knows about, and until // the scratch exists there is nothing to ask. const usage = - creating && !offering + (creating || asking) && !offering ? { commands: 0, bytes: 0 } : ledgerUsage(ctx.storage, held.acquisitionId); if (usage.commands >= MAX_COMMANDS || usage.bytes >= MAX_LEDGER_BYTES) { @@ -561,9 +588,10 @@ export function dispatchCommand( if (mutating(command)) { // Recorded in this same transaction as the mutation it describes, so a // crash cannot leave one without the other. - const mutations = creating - ? undefined - : ctx.storage.sql.exec(`SELECT count(*) AS decided FROM ${MUTATION_TABLE}`).toArray()[0]; + const mutations = + creating || asking + ? undefined + : ctx.storage.sql.exec(`SELECT count(*) AS decided FROM ${MUTATION_TABLE}`).toArray()[0]; if (mutations !== undefined && integer(mutations["decided"]) >= MAX_COMMANDS) { throw new CommandError("capacity"); } diff --git a/packages/workflow/src/cloudflare/lifecycle-link.ts b/packages/workflow/src/cloudflare/lifecycle-link.ts index 936372338..6fdc726c8 100644 --- a/packages/workflow/src/cloudflare/lifecycle-link.ts +++ b/packages/workflow/src/cloudflare/lifecycle-link.ts @@ -28,6 +28,7 @@ import type { RemoteBeginCommand, RemoteBegun, RemoteForkCommit, + RemoteForkContinuation, RemoteForkPart, RemoteLifecycleAnswer, RemoteLifecycleLink, @@ -222,6 +223,38 @@ export function cloudflareLifecycleLink( } }, + *continueFork( + continuation: RemoteForkContinuation, + ): Operation | "absent">> { + try { + const offered = yield* connection.ask( + continuation.commandId, + { + command: "fork-continue", + runId: continuation.runId, + creation: continuation.creation, + runRecord: record(continuation.runRecord), + rootImport: record(continuation.rootImport), + executionId: continuation.executionId, + }, + (value: unknown) => value, + privateRefusal, + ); + if (offered.outcome === "refused") { + const refusal = privateRefusal(offered.refusal); + if (refusal === "command:absent") { + // Nothing there to continue. Not a failure: the caller's next move + // is the source copy it has not needed until now. + return Ok("absent"); + } + return Err(storageFailure(refusal)); + } + return yield* forked(offered.value, continuation.runId); + } catch (error) { + return Err(translate(error)); + } + }, + *commitFork(commit: RemoteForkCommit): Operation>> { try { const offered = yield* connection.ask( @@ -243,51 +276,68 @@ export function cloudflareLifecycleLink( if (offered.outcome === "refused") { return Err(storageFailure(privateRefusal(offered.refusal))); } - const found = members(offered.value, ["conflict", "refusal", "value"]); - const refusal = found.get("refusal"); - if (refusal !== null) { - if (refusal !== "cancelled" && refusal !== "resume-failed" && refusal !== "terminal") { - return Err( - new WorkflowRecordMalformedError( - "lifecycle answer this run's owner returned", - "it named no condition this build reads", - ), - ); - } - return Ok({ kind: "refused", refusal }); - } - const conflict = found.get("conflict"); - if (conflict !== null) { - if (!Array.isArray(conflict) || conflict.length === 0) { - return Err( - new WorkflowRecordMalformedError( - "lifecycle answer this run's owner returned", - "it named no differing field", - ), - ); - } - return Err( - new WorkflowRunConflictError( - commit.runId, - conflict.map((field) => (typeof field === "string" ? field : "definition")), - ), - ); - } - const held = members(found.get("value"), ["frontier", "execution"]); - return Ok({ - kind: "performed", - value: { - frontier: yield* frontierOf(held.get("frontier")), - execution: execution(held.get("execution")), - replay: false, - recovered: null, - }, - }); + return yield* forked(offered.value, commit.runId); } catch (error) { return Err(translate(error)); } }, }; + + /** One fork answer: a conflict, a condition, or the destination it made. */ + function* forked( + value: unknown, + runId: string, + ): Operation>> { + const found = members(value, ["conflict", "refusal", "value"]); + const refusal = found.get("refusal"); + if (refusal !== null) { + if (refusal !== "cancelled" && refusal !== "resume-failed" && refusal !== "terminal") { + return Err( + new WorkflowRecordMalformedError( + "lifecycle answer this run's owner returned", + "it named no condition this build reads", + ), + ); + } + return Ok({ kind: "refused", refusal }); + } + const conflict = found.get("conflict"); + if (conflict !== null) { + if (!Array.isArray(conflict) || conflict.length === 0) { + return Err( + new WorkflowRecordMalformedError( + "lifecycle answer this run's owner returned", + "it named no differing field", + ), + ); + } + return Err( + new WorkflowRunConflictError( + runId, + conflict.map((field) => (typeof field === "string" ? field : "definition")), + ), + ); + } + const held = members(found.get("value"), ["frontier", "execution", "replay", "recovered"]); + if (typeof held.get("replay") !== "boolean") { + return Err( + new WorkflowRecordMalformedError( + "lifecycle answer this run's owner returned", + "it did not say whether the destination replayed", + ), + ); + } + const recovered = held.get("recovered"); + return Ok({ + kind: "performed", + value: { + frontier: yield* frontierOf(held.get("frontier")), + execution: execution(held.get("execution")), + replay: held.get("replay") === true, + recovered: recovered === null ? null : execution(recovered), + }, + }); + } } /** One head record, in the canonical spelling a journal retains. */ diff --git a/packages/workflow/src/cloudflare/owner-fork.ts b/packages/workflow/src/cloudflare/owner-fork.ts index 98029a32e..dd8bd0330 100644 --- a/packages/workflow/src/cloudflare/owner-fork.ts +++ b/packages/workflow/src/cloudflare/owner-fork.ts @@ -42,6 +42,8 @@ import type { OwnerStorage } from "./storage.ts"; import type { OwnerTransaction } from "./owner-transaction.ts"; import { establishRun } from "./owner-open.ts"; import { beginRun, type LifecycleRefusal } from "./owner-lifecycle.ts"; +import { conflictingFields } from "../storage/compatibility.ts"; +import { recognizeObject as recognize } from "./recognition.ts"; import { readFrontier, validateRetainedRoot, type FrontierValue } from "./owner-reads.ts"; import { FORK_TABLE, holdExecution, STAGING_TABLE } from "./private-schema.ts"; import { checkoutKey, forkSelectionAnchor } from "./fork-anchor.ts"; @@ -54,6 +56,10 @@ import type { DocumentExecutionRecord } from "../storage/record.ts"; export interface ForkedValue { readonly frontier: FrontierValue; readonly execution: DocumentExecutionRecord; + /** Whether the destination kept a terminal state instead of running. */ + readonly replay: boolean; + /** What stale recovery closed on the way in, when it closed anything. */ + readonly recovered: DocumentExecutionRecord | null; } /** A fork answer: the destination, or which immutable fields say it is another. */ @@ -64,6 +70,9 @@ export interface ForkValue { readonly value: ForkedValue | null; } +const RUN_COLUMNS = `run_id, definition, base, props, status, + stop_reason_kind, stop_reason_code, stop_reason_event_id, created_at, updated_at`; + const MAX_PART_BYTES = 256 * 1024; const MAX_PARTS = 8192; @@ -266,7 +275,7 @@ function retainRoot( storage: OwnerStorage, acquisitionId: string, root: StagedRoot, - watermarks: ReadonlyMap, + watermarks: ContentWatermarks, ): void { const parsed = parseWorkspaceRootManifest(root.manifest, () => { throw new CommandError("malformed-member"); @@ -321,7 +330,7 @@ function retainRoot( size, // The watermark the source retained beside these bytes. A digest stands // for the bytes and for their size; it does not stand for this. - watermarkOf(watermarks, hash), + watermarkOf(watermarks.blob, hash), ); storage.sql.exec( "INSERT INTO vfs_blob_bytes (hash, bytes) VALUES (?, ?) ON CONFLICT(hash) DO NOTHING", @@ -338,7 +347,7 @@ function retainRoot( throw new CommandError("malformed-member"); }).size, bytes, - watermarkOf(watermarks, hash), + watermarkOf(watermarks.manifest, hash), ); } @@ -518,7 +527,7 @@ function reconstruct( storage: OwnerStorage, acquisitionId: string, input: { readonly origin: ForkOrigin; readonly counts: ForkCounts }, -): { anchor: string; rootIds: Set; watermarks: Map } { +): { anchor: string; rootIds: Set; watermarks: ContentWatermarks } { const roots = section(storage, acquisitionId, "roots", input.counts.roots).map((part) => parseRoot(part), ); @@ -549,10 +558,13 @@ function reconstruct( (part) => parseCheckoutPart(part), ); - const watermarks = new Map(); - for (const piece of [...manifests, ...blobs]) { - watermarks.set(piece.hash, piece.lastSeen); - } + // Two namespaces, never one. A digest identifies bytes, and the same bytes + // can be one manifest's encoding and another manifest's chunk: the two tables + // retain their own watermarks, so one role must not answer for the other. + const watermarks = { + manifest: metadataOf(manifests), + blob: metadataOf(blobs), + }; return { rootIds, @@ -578,6 +590,28 @@ function reconstruct( }; } +/** The watermarks one transfer carried, kept by the role each belongs to. */ +export interface ContentWatermarks { + readonly manifest: ReadonlyMap; + readonly blob: ReadonlyMap; +} + +/** One section's metadata, refusing a digest it names twice. */ +function metadataOf( + pieces: readonly { hash: string; size: number; lastSeen: number }[], +): Map { + const held = new Map(); + for (const piece of pieces) { + if (held.has(piece.hash)) { + // The same identity described twice in one role. Which description is + // the transfer's is not a question this can answer. + throw new CommandError("malformed-member"); + } + held.set(piece.hash, piece.lastSeen); + } + return held; +} + /** One manifest's or blob's retained metadata, as a part carries it. */ function parseContentMetadata(part: Record): { hash: string; @@ -661,6 +695,124 @@ function requireHeadRecords( } } +/** + * Continue a destination that already holds this fork, without its source. + * + * A committed fork is independent: it holds its own prefix, its own content and + * its own Workspace, and nothing about continuing it needs the run it was + * copied from. So this asks only what the destination itself retains — its + * immutable identity, its lineage, and the two head records it wrote for + * itself — and then takes the run up through the same recovery and admission a + * resume uses. + * + * Absent when nothing is here. Making a fork needs a source; this is for when + * one was already made. + */ +export function continueFork( + storage: OwnerStorage, + transaction: OwnerTransaction, + acquisitionId: string, + input: { + readonly runId: string; + readonly creation: CreateWorkflowRunRequest; + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; + readonly executionId: string; + }, + now: () => string, +): ForkValue { + if ( + rows(storage, "SELECT name FROM sqlite_schema WHERE type = 'table' AND name = 'workflow_run'") + .length === 0 + ) { + // Nothing here to continue. Copying a source is a different command. + throw new CommandError("absent"); + } + recognize(storage); + const stored = rows(storage, `SELECT ${RUN_COLUMNS} FROM workflow_run`)[0]; + if (stored === undefined) { + throw new CommandError("absent"); + } + const record = readRunRecord(stored); + if (record.runId !== input.runId) { + throw new CommandError("wrong-run"); + } + const differing = conflictingFields(record, input.creation); + if (differing.length > 0) { + return { conflict: differing, refusal: null, value: null }; + } + + const lineage = rows( + storage, + `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, selection_anchor + FROM workflow_fork_lineage WHERE id = 1`, + )[0]; + if (lineage === undefined) { + // A run, and not a fork. Continuing it as one would claim a lineage it + // does not have. + return { conflict: ["lineage"], refusal: null, value: null }; + } + + // The two records this fork wrote for itself, as it retains them. A request + // carrying a different root import is a different fork, whatever else agrees. + const heads = rows( + storage, + "SELECT record, workspace_root_id FROM journal_events ORDER BY sequence LIMIT 2", + ); + const expected = serializeDurableEvent( + forkRunRecordEvent({ + runId: input.runId, + base: input.creation.base, + pinnedCommit: input.creation.definition.objectId, + }), + ); + const roots = new Set( + rows(storage, "SELECT root_id FROM workspace_roots").map((row) => String(row["root_id"])), + ); + if ( + heads.length !== 2 || + serializeDurableEvent(input.runRecord) !== expected || + heads[0]?.["record"] !== expected || + heads[1]?.["record"] !== serializeDurableEvent(input.rootImport) || + !roots.has(String(heads[0]?.["workspace_root_id"])) || + !roots.has(String(heads[1]?.["workspace_root_id"])) + ) { + return { conflict: ["lineage"], refusal: null, value: null }; + } + + const resumed = beginRun( + storage, + transaction, + acquisitionId, + input.runId, + "resume", + null, + null, + input.executionId, + now, + ); + if (resumed.conflict !== null) { + return { conflict: resumed.conflict, refusal: null, value: null }; + } + if (resumed.refusal !== null) { + return { conflict: null, refusal: resumed.refusal, value: null }; + } + const begun = resumed.value; + if (begun === null) { + throw new CommandError("malformed-member"); + } + return { + conflict: null, + refusal: null, + value: { + frontier: begun.frontier, + execution: begun.execution, + replay: begun.replay, + recovered: begun.recovered, + }, + }; +} + /** * Commit one fork: the destination and everything it inherited, together. * @@ -700,7 +852,10 @@ export function commitFork( // A transfer that is not that selection stops here, with the destination // holding no run. const staged = fresh ? reconstruct(storage, acquisitionId, input) : undefined; - const watermarks = staged?.watermarks ?? new Map(); + const watermarks: ContentWatermarks = staged?.watermarks ?? { + manifest: new Map(), + blob: new Map(), + }; if (staged !== undefined) { if (staged.anchor !== input.origin.anchor) { throw new CommandError("stale-journal"); @@ -771,7 +926,14 @@ export function commitFork( return { conflict: null, refusal: null, - value: { frontier: begun.frontier, execution: begun.execution }, + // Exactly what the shared policy decided, including a terminal run that + // replays and whatever recovery closed on the way in. + value: { + frontier: begun.frontier, + execution: begun.execution, + replay: begun.replay, + recovered: begun.recovered, + }, }; } @@ -881,6 +1043,10 @@ function begunOn( return { frontier: readFrontier(storage, input.runId), execution: readDocumentExecution(row), + // A fork this transaction just created runs; there was nothing here to + // replay and nothing to recover. + replay: false, + recovered: null, }; } diff --git a/packages/workflow/src/cloudflare/private-schema.ts b/packages/workflow/src/cloudflare/private-schema.ts index 7e24a1017..5a458aba7 100644 --- a/packages/workflow/src/cloudflare/private-schema.ts +++ b/packages/workflow/src/cloudflare/private-schema.ts @@ -203,13 +203,15 @@ export function adoptExecution( storage: OwnerStorage, acquisitionId: string, commandId: string, -): void { +): "adopted" | "nothing-to-adopt" | "stale" { const decided = storage.sql .exec(`SELECT execution_id FROM ${MUTATION_TABLE} WHERE command_id = ?`, commandId) .toArray()[0]; const executionId = decided?.["execution_id"]; if (typeof executionId !== "string") { - return; + // The decision began nothing, so there is nothing to hold. Answering it + // again is answering a question, not granting authority. + return "nothing-to-adopt"; } const open = storage.sql .exec( @@ -218,25 +220,27 @@ export function adoptExecution( ) .toArray()[0]; if (open === undefined) { - // Finished since. There is nothing to authorize, and the decision stands - // as the answer it always was. - return; + // Finished since — recovered by a later executor, or settled. The run has + // moved past this decision, and handing it back as current authority would + // hand back a database nobody may settle. + return "stale"; } const holder = storage.sql .exec(`SELECT acquisition_id FROM ${HOLD_TABLE} WHERE execution_id = ?`, executionId) .toArray()[0]; if (holder !== undefined) { if (holder["acquisition_id"] !== acquisitionId) { - // Somebody live began it. Two acquisitions cannot hold one execution. - throw new Error("private protocol storage holds a conflicting execution hold"); + // Somebody live holds it. Two acquisitions cannot hold one execution. + return "stale"; } - return; + return "adopted"; } storage.sql.exec( `INSERT INTO ${HOLD_TABLE} (acquisition_id, execution_id) VALUES (?, ?)`, acquisitionId, executionId, ); + return "adopted"; } /** Which execution this acquisition began, when it has begun one. */ diff --git a/packages/workflow/src/remote/lifecycle-link.ts b/packages/workflow/src/remote/lifecycle-link.ts index 8fdbde4bf..de24aa7f5 100644 --- a/packages/workflow/src/remote/lifecycle-link.ts +++ b/packages/workflow/src/remote/lifecycle-link.ts @@ -83,6 +83,16 @@ export interface RemoteBeginCommand { } /** Everything one committed fork is decided from. */ +/** Taking up a destination that already holds this fork, without its source. */ +export interface RemoteForkContinuation { + readonly commandId: string; + readonly runId: string; + readonly creation: CreateWorkflowRunRequest; + readonly runRecord: DurableEvent; + readonly rootImport: DurableEvent; + readonly executionId: string; +} + export interface RemoteForkCommit { /** The identity this logical invocation keeps, retry after retry. */ readonly commandId: string; @@ -134,4 +144,13 @@ export interface RemoteLifecycleLink { stageForkPart(commandId: string, part: RemoteForkPart): Operation>; /** Commit the offered parts as one destination run and its first execution. */ commitFork(commit: RemoteForkCommit): Operation>>; + /** + * Continue a destination that already holds this fork. + * + * `absent` rather than a failure when the destination holds no run: that is + * the answer that sends a caller to the source it has not needed yet. + */ + continueFork( + continuation: RemoteForkContinuation, + ): Operation | "absent">>; } diff --git a/packages/workflow/src/remote/lifecycle.ts b/packages/workflow/src/remote/lifecycle.ts index 0e380c022..8b941f77e 100644 --- a/packages/workflow/src/remote/lifecycle.ts +++ b/packages/workflow/src/remote/lifecycle.ts @@ -46,15 +46,18 @@ import { WorkflowRequestError, WorkflowRunNotFoundError, type WorkflowStorageError, + WorkflowTransactionError, } from "../storage/errors.ts"; import { useRemoteRunDatabase } from "./database.ts"; import { canonicalJson } from "../storage/record.ts"; import { definitionToJson } from "../storage/definition.ts"; import type { RemoteForkSource, RemoteReadPlane } from "./read.ts"; import { forkRunRecordEvent } from "../fork.ts"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; import type { RemoteBegun, RemoteExecutorConnection, + RemoteForkCommit, RemoteForkPart, RemoteLifecycleLink, } from "./lifecycle-link.ts"; @@ -94,19 +97,28 @@ export interface RemoteLifecycleHost { } /** - * One logical lifecycle invocation, as it is addressed and re-addressed. + * One logical lifecycle call, as this provider owns it. * - * Minted once and kept by the run it belongs to rather than by the connection - * that first sent it. A connection that died between an owner's commit and its - * answer is exactly the case this exists for: the replacement acquisition asks - * the same question, with the same identity and the same bytes, and the owner - * answers with what it already decided instead of deciding again. + * A call is not the same call as another because their arguments compare equal: + * two identical `begin()` calls are two calls, and the second must be refused + * rather than handed the first one's execution. What may reuse an identity is + * one narrow thing — the continuation of a call whose answer was lost after the + * owner may already have committed — and that is what this records. + * + * `in flight` is held by the acquisition, so a second transition under one live + * lock is refused before anything is sent. `ambiguous` outlives the connection + * it was sent on, because the whole point is that a replacement acquisition + * asks the same question. A definitive answer — a decision, a conflict, an + * owner refusal — retires the record: it has been answered, and a later + * corrected request is a new call with new identities. */ -interface Invocation { +interface Ambiguous { readonly commandId: string; readonly executionId: string; - /** What this invocation asked, so a different question gets a new identity. */ - readonly shape: string; + /** The complete canonical request this identity belongs to. */ + readonly question: string; + /** The exact command that was sent, resent verbatim on continuation. */ + readonly command: RemoteForkCommit | undefined; } /** What one issued lock is allowed to do, and what it has already done. */ @@ -115,6 +127,14 @@ interface Hold { readonly connection: RemoteExecutorConnection; /** Which execution this acquisition began, once it has begun one. */ execution: string | undefined; + /** + * Whether a transition is in flight on this acquisition. + * + * One acquisition begins one execution, and it has to be refused while the + * first call is still waiting as well as after it returns — otherwise two + * identical calls both pass the check and both send. + */ + busy: boolean; /** Whether the connection this lock was issued beside is still open. */ live: boolean; } @@ -136,25 +156,38 @@ export function* useRemoteLifecycle( // Keyed by the run and the question asked, so the same question after a lost // answer is the same invocation. Not authority: what it carries is an // identity, and the owner decides what that identity already means. - const invocations = new Map(); + const ambiguous = new Map(); - function invocation(runId: string, shape: string): Invocation { - const key = `${runId}\u0000${shape}`; - const found = invocations.get(key); - if (found !== undefined) { + /** + * The identity this call carries. + * + * A fresh identity, unless this is the continuation of a call whose answer + * was lost and whose question is exactly this one. + */ + function identify(runId: string, question: string): Ambiguous { + const found = ambiguous.get(runId); + if (found !== undefined && found.question === question) { return found; } - const minted: Invocation = { + return { commandId: host.ids.command(), executionId: host.ids.execution(), - shape, + question, + command: undefined, }; - invocations.set(key, minted); - return minted; } - function settled(runId: string, shape: string): void { - invocations.delete(`${runId}\u0000${shape}`); + /** Remember a call whose answer never arrived, with what it sent. */ + function unanswered(runId: string, held: Ambiguous, command?: RemoteForkCommit): void { + ambiguous.set(runId, { ...held, command: command ?? held.command }); + } + + /** Retire a call that was answered, whatever the answer was. */ + function answeredNow(runId: string, held: Ambiguous): void { + const found = ambiguous.get(runId); + if (found?.commandId === held.commandId) { + ambiguous.delete(runId); + } } function hold(lock: ExecutorLock): Hold | undefined { @@ -182,6 +215,7 @@ export function* useRemoteLifecycle( runId, connection, execution: undefined, + busy: false, live: true, }; held.set(lock, record); @@ -213,13 +247,13 @@ export function* useRemoteLifecycle( ), ); } - const shape = questionOf(["cancel"]); - const addressed = invocation(runId, shape); + const question = questionOf(["cancel"]); + const addressed = identify(runId, question); const answered = yield* admitted.value.lifecycle.cancel(addressed.commandId, runId); if (!answered.ok) { return answered; } - settled(runId, shape); + answeredNow(runId, addressed); if (answered.value.kind === "refused") { return Err( new WorkflowRequestError( @@ -248,7 +282,35 @@ export function* useRemoteLifecycle( { at: "min" }, ); - return transitions(host, hold, invocation, settled); + return transitions(host, hold, identify, unanswered, answeredNow); +} + +/** + * Take this acquisition for one transition, or say why it cannot be taken. + * + * One acquisition begins one execution. Two identical calls are two calls, so + * the second is refused while the first is still in flight as well as after it + * has returned. + */ +function engage(held: Hold): WorkflowRequestError | undefined { + if (held.execution !== undefined || held.busy) { + return new WorkflowRequestError( + "this executor lock has already begun a document execution. One acquisition begins one.", + ); + } + held.busy = true; + return undefined; +} + +/** + * Whether a failure left the owner's decision unknown. + * + * A refusal or a conflict is an answer: the owner decided, and the question is + * finished. A transport that ended without answering is not, and the same + * question has to be asked again rather than replaced by a new one. + */ +function lost(error: WorkflowStorageError): boolean { + return error instanceof WorkflowTransactionError; } /** What an unrecognized lock answers, wherever one is offered. */ @@ -261,8 +323,9 @@ function unauthorized(): WorkflowRequestError { function transitions( host: RemoteLifecycleHost, hold: (lock: ExecutorLock) => Hold | undefined, - invocation: (runId: string, shape: string) => Invocation, - settled: (runId: string, shape: string) => void, + identify: (runId: string, question: string) => Ambiguous, + unanswered: (runId: string, held: Ambiguous, command?: RemoteForkCommit) => void, + answeredNow: (runId: string, held: Ambiguous) => void, ): WorkflowExecutionTransitions { return { *begin( @@ -278,14 +341,12 @@ function transitions( new WorkflowRequestError("this executor lock was issued for a different workflow run."), ); } - if (held.execution !== undefined) { - return Err( - new WorkflowRequestError( - "this executor lock has already begun a document execution. One acquisition begins one.", - ), - ); + const busy = engage(held); + if (busy !== undefined) { + return Err(busy); } if (request.action === "resume" && request.creation !== undefined) { + held.busy = false; return Err(new WorkflowRequestError("a resume does not carry a creation.")); } // Minted once for this question, outside anything that could retry: the @@ -293,7 +354,7 @@ function transitions( // second execution. A later acquisition asking the same question finds // the same identity and re-observes the decision. const creation = creationOf(request.runId, request.creation); - const shape = questionOf([ + const question = questionOf([ "begin", request.action, creation === null ? null : creationShape(creation), @@ -301,7 +362,7 @@ function transitions( ? null : canonicalJson(request.creation.retrieval), ]); - const addressed = invocation(request.runId, shape); + const addressed = identify(request.runId, question); const answered = yield* held.connection.lifecycle.begin({ commandId: addressed.commandId, runId: request.runId, @@ -310,16 +371,23 @@ function transitions( retrieval: request.creation?.retrieval, executionId: addressed.executionId, }); + held.busy = false; if (!answered.ok) { - // The answer is lost, not the question. The identity stays, so the - // next acquisition asks the same one. + // Which kind of failure decides whether the question survives it. An + // answer that was lost leaves the identity standing, so a replacement + // acquisition asks the same one; anything the owner actually decided + // retires it. + if (lost(answered.error)) { + unanswered(request.runId, addressed); + } else { + answeredNow(request.runId, addressed); + } return answered; } + answeredNow(request.runId, addressed); if (answered.value.kind === "refused") { - settled(request.runId, shape); return Err(refusalError(answered.value.refusal, request.runId)); } - settled(request.runId, shape); held.execution = answered.value.value.execution.executionId; return Ok(yield* begun(held, answered.value.value)); }, @@ -345,22 +413,30 @@ function transitions( // The root the owner is held to comes from the same connection-owned // frontier the execution ran against, after the host has torn down. const frontier = yield* held.connection.link.frontierSnapshot(); - const shape = questionOf([ + const question = questionOf([ "settle", completion.executionId, completion.status, + // The whole completion, stop reason included: a settlement that named + // a different reason is a different settlement. + canonicalJson(completion.reason ?? null), frontier.workspaceRootId, ]); - const addressed = invocation(held.runId, shape); + const addressed = identify(held.runId, question); const answered = yield* held.connection.lifecycle.settle( addressed.commandId, completion, frontier.workspaceRootId, ); if (!answered.ok) { + if (lost(answered.error)) { + unanswered(held.runId, addressed); + } else { + answeredNow(held.runId, addressed); + } return answered; } - settled(held.runId, shape); + answeredNow(held.runId, addressed); held.execution = undefined; return Ok(answered.value.record); }, @@ -378,33 +454,83 @@ function transitions( new WorkflowRequestError("this executor lock was issued for a different workflow run."), ); } - if (held.execution !== undefined) { - return Err( - new WorkflowRequestError( - "this executor lock has already begun a document execution. One acquisition begins one.", - ), - ); + const busy = engage(held); + if (busy !== undefined) { + return Err(busy); } + const head = headOf(request); + const creation = creationRequest(request.runId, request.creation); + const question = questionOf([ + "fork", + request.runId, + creationShape(creation), + request.creation.retrieval === undefined ? null : canonicalJson(request.creation.retrieval), + request.selection.sourceRunId, + request.selection.checkpointEventId, + serializeDurableEvent(head.runRecord), + serializeDurableEvent(request.rootImport), + ]); + const addressed = identify(request.runId, question); + + // Before the source: a destination that already holds this fork can be + // continued from what it retains, and a decision this owner already made + // can be re-observed. Either way the source is not needed, and it may not + // be there any more. + const retained = + addressed.command === undefined + ? undefined + : yield* held.connection.lifecycle.commitFork(addressed.command); + if (retained !== undefined && retained.ok && retained.value.kind === "performed") { + held.busy = false; + answeredNow(request.runId, addressed); + held.execution = retained.value.value.execution.executionId; + return Ok(yield* begun(held, retained.value.value)); + } + const continued = yield* held.connection.lifecycle.continueFork({ + commandId: addressed.commandId, + runId: request.runId, + creation, + runRecord: head.runRecord, + rootImport: request.rootImport, + executionId: addressed.executionId, + }); + if (!continued.ok) { + held.busy = false; + if (lost(continued.error)) { + unanswered(request.runId, addressed); + } else { + answeredNow(request.runId, addressed); + } + return continued; + } + if (continued.value !== "absent") { + held.busy = false; + answeredNow(request.runId, addressed); + if (continued.value.kind === "refused") { + return Err(refusalError(continued.value.refusal, request.runId)); + } + held.execution = continued.value.value.execution.executionId; + return Ok(yield* begun(held, continued.value.value)); + } + + // Nothing there. Making a fork needs the whole source, staged under this + // acquisition and committed in one transaction. const source = yield* readSource(host, request); if (!source.ok) { + held.busy = false; + answeredNow(request.runId, addressed); return source; } const staged = yield* offer(host, held.connection.lifecycle, source.value); if (!staged.ok) { + held.busy = false; return staged; } - const shape = questionOf([ - "fork", - request.runId, - source.value.anchor, - source.value.checkpointEventId, - ]); - const addressed = invocation(request.runId, shape); - const answered = yield* held.connection.lifecycle.commitFork({ + const command: RemoteForkCommit = { commandId: addressed.commandId, runId: request.runId, retrieval: request.creation.retrieval, - creation: creationRequest(request.runId, request.creation), + creation, origin: { sourceRunId: source.value.sourceRunId, checkpointEventId: source.value.checkpointEventId, @@ -420,18 +546,27 @@ function transitions( blobs: source.value.blobs.length, checkouts: source.value.checkouts.length, }, - runRecord: headOf(request).runRecord, + runRecord: head.runRecord, rootImport: request.rootImport, executionId: addressed.executionId, - }); + }; + const answered = yield* held.connection.lifecycle.commitFork(command); + held.busy = false; if (!answered.ok) { + if (lost(answered.error)) { + // Kept with the exact bytes it was sent with, so a replacement + // acquisition resends this command rather than reading the source + // again. + unanswered(request.runId, addressed, command); + } else { + answeredNow(request.runId, addressed); + } return answered; } + answeredNow(request.runId, addressed); if (answered.value.kind === "refused") { - settled(request.runId, shape); return Err(refusalError(answered.value.refusal, request.runId)); } - settled(request.runId, shape); held.execution = answered.value.value.execution.executionId; return Ok(yield* begun(held, answered.value.value)); }, diff --git a/packages/workflow/tests/cloudflare/remote-fork.vitest.ts b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts index a4a4819ae..cf530b20a 100644 --- a/packages/workflow/tests/cloudflare/remote-fork.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts @@ -24,6 +24,8 @@ import { } from "./support/executor-object.ts"; import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; import { forkSelectionAnchor } from "../../src/cloudflare/fork-anchor.ts"; +import { sha256Hex } from "../../src/workspace/sha256.ts"; +import { WORKSPACE_ROOT_DOMAIN } from "../../src/workspace/root-manifest.ts"; import { forkRunRecordEvent } from "../../src/journal-events.ts"; let unique = 0; @@ -501,3 +503,241 @@ describe("committing a fork on its destination owner", () => { }); }); }); + +/** Take up a destination that already holds this fork, naming no source. */ +function continuation(overrides: Record = {}): Record { + return { + id: "fork-continue", + command: "fork-continue", + runId: RUN_ID, + creation: commit()["creation"], + runRecord: HEAD, + rootImport: IMPORT, + executionId: "execution-2", + ...overrides, + }; +} + +describe("continuing a fork the destination already holds", () => { + it("answers absent when there is nothing here, and needs no source", async () => { + const stub = executor(); + await connected(stub); + + expect(await ask(stub, continuation())).toEqual({ + id: "fork-continue", + outcome: "refused", + refusal: "command:absent", + }); + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); + + it("recovers and begins one replacement, without reading any source", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + const continued = await ask(stub, continuation()); + + expect(continued["outcome"]).toBe("performed"); + const value = Object(Object(continued["value"])["value"]); + expect(value["replay"]).toBe(false); + // The lost executor's execution was closed on the way in and surfaced. + expect(Object(value["recovered"])["stopStatus"]).toBe("interrupted"); + const executions = await on(stub, (owner) => owner.executionRows()); + expect(executions.filter((row) => row["stopped_at"] === null)).toHaveLength(1); + }); + + it("reports a terminal destination as a replay, and leaves it terminal", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + const root = await on(stub, (owner) => owner.currentRootId()); + await ask(stub, { + id: "settle-1", + command: "settle", + completion: { executionId: "execution-1", status: "completed" }, + expectedWorkspaceRootId: root, + }); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + const continued = await ask(stub, continuation()); + + expect(continued["outcome"]).toBe("performed"); + const value = Object(Object(continued["value"])["value"]); + expect(value["replay"]).toBe(true); + // The outcome that won is not made mutable again. + expect((await on(stub, (owner) => owner.runRow()))?.["status"]).toBe("completed"); + }); + + it("refuses a continuation whose root import is not the one it retains", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + const refused = await ask( + stub, + continuation({ + rootImport: serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "README.md", content: "# elsewhere" }, + }, + }), + }), + ); + + expect(refused["outcome"]).toBe("performed"); + expect(Object(refused["value"])["conflict"]).toEqual(["lineage"]); + // No replacement execution began. + expect(await on(stub, (owner) => owner.executionRows())).toHaveLength(1); + }); + + it("refuses a continuation whose run record is not the one this fork implies", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + const refused = await ask( + stub, + continuation({ + runRecord: serializeDurableEvent( + forkRunRecordEvent({ runId: RUN_ID, base: "other", pinnedCommit: "0".repeat(40) }), + ), + }), + ); + + expect(Object(refused["value"])["conflict"]).toEqual(["lineage"]); + expect(await on(stub, (owner) => owner.executionRows())).toHaveLength(1); + }); +}); + +describe("copying content whose digest is valid in both roles", () => { + it("keeps the manifest and blob watermarks apart", async () => { + // A Workspace whose file is the bytes of one content manifest, and whose + // content manifest for that file names those same bytes as its chunk. The + // digest is then both a manifest identity and a blob identity, with its + // own watermark in each table. + const inner = new TextEncoder().encode(DOFS_MANIFEST); + const shared = MANIFEST_ID; + const outer = JSON.stringify({ + version: 1, + chunks: [{ hash: shared, size: inner.length }], + }); + const outerBytes = new TextEncoder().encode(outer); + const outerId = sha256Hex(outerBytes); + const manifest = JSON.stringify({ + format: 1, + entries: [ + { path: "/", kind: "directory", mode: 493, mtime: 0 }, + { + path: "/MANIFEST.json", + kind: "file", + mode: 420, + mtime: 0, + size: inner.length, + manifest: outerId, + hardlink: null, + }, + ], + }); + const rootId = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${manifest}`); + const root = { + rootId, + formatVersion: 1, + manifest, + manifestHashes: [outerId], + blobHashes: [shared], + }; + const manifestPart = { hash: outerId, size: inner.length, lastSeen: 11 }; + // The same digest, in the other role, with its own watermark. + const blobPart = { hash: shared, size: inner.length, lastSeen: 23 }; + const inheritedPart = { + eventId: "event-work", + record: event("work"), + workspaceRootId: rootId, + }; + + const stub = executor(); + await connected(stub); + let minted = 0; + const id = () => `command-${(minted += 1)}`; + await ask(stub, { + id: id(), + command: "stage", + kind: "manifest", + digest: outerId, + bytes: base64(outerBytes), + }); + await ask(stub, { + id: id(), + command: "stage", + kind: "blob", + digest: shared, + bytes: base64(inner), + }); + for (const part of [ + { section: "roots", body: root }, + { section: "inherited", body: inheritedPart }, + { section: "manifests", body: manifestPart }, + { section: "blobs", body: blobPart }, + ]) { + expect( + ( + await ask(stub, { + id: id(), + command: "fork-stage", + section: part.section, + position: 0, + part: part.body, + }) + )["outcome"], + ).toBe("performed"); + } + + const forked = await ask(stub, { + ...commit(), + origin: { + sourceRunId: SOURCE_RUN_ID, + checkpointEventId: "event-work", + checkpointWorkspaceRootId: rootId, + runRecordWorkspaceRootId: rootId, + rootImportWorkspaceRootId: rootId, + anchor: forkSelectionAnchor({ + checkpointEventId: "event-work", + checkpointWorkspaceRootId: rootId, + runRecordWorkspaceRootId: rootId, + rootImportWorkspaceRootId: rootId, + inherited: [inheritedPart], + roots: [root], + manifests: [{ ...manifestPart, encoded: base64(outerBytes) }], + blobs: [blobPart], + checkouts: [], + }), + }, + }); + + expect(forked["outcome"]).toBe("performed"); + const watermarks = await on(stub, (owner) => owner.contentWatermarks()); + // Two roles, two watermarks, neither overwriting the other. + expect(watermarks.manifests).toEqual([11]); + expect(watermarks.blobs).toEqual([23]); + }); +}); diff --git a/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts index 16819d78a..3dda371f5 100644 --- a/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts @@ -398,4 +398,50 @@ describe("a run's lifecycle on its owner", () => { }); expect(settled["outcome"]).toBe("performed"); }); + + it("refuses a retained decision the run has already moved past", async () => { + const stub = executor(); + await connected(stub); + // One executor begins and loses its answer. + expect((await started(stub, "command-1", "execution-1"))["outcome"]).toBe("performed"); + + // Another takes the run, recovers that execution and begins its own. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const recovered = await ask(stub, { + id: "command-2", + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + retrieval: null, + executionId: "execution-2", + }); + expect(recovered["outcome"]).toBe("performed"); + + // The first executor retries its retained command under a later + // acquisition. Its execution is closed, so it is history rather than + // authority. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const retried = await started(stub, "command-1", "execution-1"); + + expect(retried).toEqual({ + id: "command-1", + outcome: "refused", + refusal: "command:stale-journal", + }); + // And it cannot settle what it did not keep. + const root = await on(stub, (owner) => owner.currentRootId()); + const settled = await ask(stub, { + id: "command-4", + command: "settle", + completion: { executionId: "execution-1", status: "completed" }, + expectedWorkspaceRootId: root, + }); + expect(settled["refusal"]).toBe("command:wrong-execution"); + // Exactly one execution is open, and it is the recovering executor's. + const executions = await on(stub, (owner) => owner.executionRows()); + expect(executions.filter((row) => row["stopped_at"] === null)).toHaveLength(1); + }); }); diff --git a/packages/workflow/tests/remote-fork.test.ts b/packages/workflow/tests/remote-fork.test.ts index 38b52c88f..e7786d5a8 100644 --- a/packages/workflow/tests/remote-fork.test.ts +++ b/packages/workflow/tests/remote-fork.test.ts @@ -226,4 +226,87 @@ describe("a remote fork's destination", () => { expect(opened).toEqual([]); expect(asked).toEqual([]); }); + + it("continues a destination that already holds this fork, without its source", function* () { + const asked: string[] = []; + const sourced: string[] = []; + const outcome = yield* installed( + // No source at all: this host would fail if one were asked for. + { asked, sourced, continues: true }, + function* (transitions) { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }, + ); + + expect([outcome.ok, outcome.ok === false && String(outcome.error)]).toEqual([true, false]); + // The destination answered from what it retains. Nothing was read from the + // source, and nothing was staged. + expect(sourced).toEqual([]); + expect(asked).toEqual(["fork-continue"]); + }); + + it("asks the source only when the destination holds no fork yet", function* () { + const asked: string[] = []; + const sourced: string[] = []; + const outcome = yield* installed({ asked, sourced, source: source() }, function* (transitions) { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + + expect(outcome.ok).toBe(true); + // Absent, so the source was read and staged, and the commit came last. + expect(sourced).toEqual([SOURCE_RUN_ID]); + expect(asked[0]).toBe("fork-continue"); + expect(asked.at(-1)).toBe("fork"); + }); + + it("stays absent when the destination is pristine and the source cannot be read", function* () { + const asked: string[] = []; + const outcome = yield* installed({ asked }, function* (transitions) { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + + expect(outcome.ok).toBe(false); + // It asked the destination, learned there was nothing, and stopped when + // the source it needed was unavailable. Nothing was committed. + expect(asked).toEqual(["fork-continue"]); + }); + + it("resends the command it already sent when its answer was lost", function* () { + const asked: string[] = []; + const sourced: string[] = []; + const commands: string[] = []; + const loseAnswer = new Set(["command-1"]); + const committed = new Map(); + const script: Script = { + asked, + sourced, + commands, + loseAnswer, + committed, + source: source(), + }; + const outcome = yield* installed(script, function* (transitions) { + const first = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + // The source is gone by the time the retry happens. + const retried = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + return { first, retried, sourced: [...sourced], asked: [...asked] }; + }); + + expect(outcome.first.ok).toBe(false); + expect(outcome.retried.ok).toBe(true); + // The retry resent the exact command before anything else, so the source + // was read once — for the first attempt — and not again. + expect(outcome.sourced).toEqual([SOURCE_RUN_ID]); + expect(outcome.asked.at(-1)).toBe("fork"); + expect(commands.filter((id) => id === "command-1").length).toBeGreaterThan(1); + }); }); diff --git a/packages/workflow/tests/remote-lifecycle.test.ts b/packages/workflow/tests/remote-lifecycle.test.ts index 1ad0a4598..ed8314911 100644 --- a/packages/workflow/tests/remote-lifecycle.test.ts +++ b/packages/workflow/tests/remote-lifecycle.test.ts @@ -12,7 +12,7 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { type Operation, type Result, scoped } from "effection"; +import { type Operation, type Result, scoped, spawn, withResolvers } from "effection"; import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; import type { ExecutorLock } from "../src/lifecycle/api.ts"; import type { @@ -295,4 +295,54 @@ describe("a remote run's executor lifecycle", () => { expect(outcome.second.value.execution.executionId).toBe("execution-1"); } }); + + it("refuses a second call while the first is still waiting for its answer", function* () { + const asked: string[] = []; + const entered = withResolvers(); + const release = withResolvers(); + const script: Script = { + asked, + gate: { + *wait(): Operation { + entered.resolve(); + yield* release.operation; + }, + }, + }; + const outcome = yield* installed(script, function* (transitions) { + const lock = yield* acquired(); + const request: WorkflowBeginRequest = { runId: RUN_ID, action: "resume" }; + const first = yield* spawn(() => transitions.begin(lock, request)); + // The first call has sent and is waiting for its answer. The second is a + // different call, however equal its arguments look. + yield* entered.operation; + const second = yield* transitions.begin(lock, request); + release.resolve(); + return { first: yield* first, second }; + }); + + expect(outcome.first.ok).toBe(true); + expect(outcome.second.ok).toBe(false); + // One owner mutation, not two. + expect(asked.filter((command) => command === "begin")).toHaveLength(1); + }); + + it("gives a corrected request a new identity after a definitive answer", function* () { + const commands: string[] = []; + const outcome = yield* installed({ commands, begin: "cancelled" }, function* (transitions) { + const lock = yield* acquired(); + const refused = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + return { refused, commands: [...commands] }; + }); + // The owner answered, so that question is finished. What it is not is a + // claim on the next one's identity. + expect(outcome.refused.ok).toBe(false); + expect(outcome.commands).toEqual(["command-1"]); + + const second = yield* installed({ commands: [] }, function* (transitions) { + const lock = yield* acquired(); + return yield* transitions.begin(lock, { runId: RUN_ID, action: "start" }); + }); + expect(second.ok).toBe(true); + }); }); diff --git a/packages/workflow/tests/remote-staged-fork.test.ts b/packages/workflow/tests/remote-staged-fork.test.ts index d21a98990..10e1d8860 100644 --- a/packages/workflow/tests/remote-staged-fork.test.ts +++ b/packages/workflow/tests/remote-staged-fork.test.ts @@ -12,9 +12,9 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { call, Err, Ok, type Operation, type Result, scoped } from "effection"; +import { Err, Ok, type Operation, type Result, scoped } from "effection"; import { serializeDurableEvent } from "@executablemd/durable-streams"; -import { stat } from "node:fs/promises"; +import { exists } from "@effectionx/fs"; import { useStorageRoot } from "./support/storage.ts"; import { useWorkflowRunConnections } from "../src/deno/connections.ts"; import { stageRemoteFork } from "../src/deno/remote-staging.ts"; @@ -150,17 +150,6 @@ function* staged( }); } -function* exists(path: string): Operation { - return yield* call(async function () { - try { - await stat(path); - return true; - } catch { - return false; - } - }); -} - describe("a remote fork's staged candidate", () => { it("is a real database holding the fork's own identity and history", function* () { const root = yield* useStorageRoot(); diff --git a/packages/workflow/tests/support/remote-lifecycle-host.ts b/packages/workflow/tests/support/remote-lifecycle-host.ts index 5eabed31f..d3ab2b3fc 100644 --- a/packages/workflow/tests/support/remote-lifecycle-host.ts +++ b/packages/workflow/tests/support/remote-lifecycle-host.ts @@ -13,6 +13,7 @@ import { ensure, Err, Ok, type Operation, type Result } from "effection"; import type { DurableEvent } from "@executablemd/durable-streams"; import type { RemoteBeginCommand, + RemoteForkContinuation, RemoteBegun, RemoteExecutorConnection, RemoteForkCommit, @@ -67,6 +68,12 @@ export interface Script { readonly commands?: string[]; /** Every retrieval value a begin carried. */ readonly retrievals?: (unknown | null)[]; + /** Held open until a test releases it, so a call can be caught in flight. */ + readonly gate?: { wait(): Operation }; + /** Every run whose source was resolved. */ + readonly sourced?: string[]; + /** Whether a destination already holds this fork, so no source is needed. */ + readonly continues?: boolean; /** Command identities whose first answer is lost after the owner commits. */ readonly loseAnswer?: Set; /** What an owner decided for a command identity, once it has decided. */ @@ -166,6 +173,10 @@ function lifecycle(script: Script): RemoteLifecycleLink { *begin(request: RemoteBeginCommand): Operation>> { script.asked?.push("begin"); script.commands?.push(request.commandId); + if (script.gate !== undefined) { + // Caught in flight: the call has been sent and has not been answered. + yield* script.gate.wait(); + } script.retrievals?.push(request.retrieval ?? null); if (script.loseAnswer?.has(request.commandId) === true) { // The owner committed and the answer never arrived. @@ -206,6 +217,20 @@ function lifecycle(script: Script): RemoteLifecycleLink { return Ok(undefined); }, // deno-lint-ignore require-yield + // deno-lint-ignore require-yield + *continueFork( + continuation: RemoteForkContinuation, + ): Operation | "absent">> { + script.asked?.push("fork-continue"); + script.commands?.push(continuation.commandId); + const held = script.continues; + if (held === undefined) { + // Nothing there to continue, which sends the caller to the source. + return Ok("absent"); + } + return Ok({ kind: "performed", value: begun(continuation.executionId, script) }); + }, + *commitFork(commit: RemoteForkCommit): Operation>> { script.asked?.push("fork"); script.commits?.push(commit); @@ -244,6 +269,7 @@ export function installedHost(script: Script): RemoteLifecycleHost { }, // deno-lint-ignore require-yield *source(runId: string): Operation> { + script.sourced?.push(runId); const held = script.source; if (held === undefined) { return Err(new WorkflowRequestError("this scripted host holds no source")); From 570ef6389361e016cf2570a82442cc0289c026ed Mon Sep 17 00:00:00 2001 From: Min Kim Date: Mon, 7 Sep 2026 18:44:32 -0300 Subject: [PATCH 54/81] =?UTF-8?q?=F0=9F=90=9B=20Prove=20a=20fork=20continu?= =?UTF-8?q?ation,=20and=20end=20a=20call=20safely?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Continuing a fork asked the destination to take a request's word for what fork it was. The command named no source and no checkpoint, and the lineage row was only checked for existing — so a request naming somewhere else entirely could take up this destination as though it were the fork it asked for. It now names the source and the checkpoint it claims, and the destination compares them with what it retains. The two head records are held to their exact associations rather than to membership. Proving that a head names some root this store happens to hold proves nothing: every retained root satisfies it. So the roots those two rows were committed against are retained with the lineage when the fork is made, and a head reassociated to another perfectly valid root is refused as the different fork it is. A decision that granted an execution is only re-observable once that execution is this acquisition's. A continuation records which execution it began — and records nothing when it began nothing, because a conflict and a refusal are answers too — so a replacement acquisition either adopts the exact open execution atomically or is told the run has moved past it. What it can no longer be handed is a database paired with an execution it may not settle. An exact command that was resent is now heard out. Every outcome is classified before anything else is sent: a second silence keeps the claim and sends nothing further, a refusal or conflict ends the question, and one closed answer — the destination holds nothing and was offered nothing — is the only one that sends the same logical fork back to its source to stage again. A lost cancellation is kept the same way, so the cancellation that committed is the one the next attempt hears about. A call that is interrupted lets go on purpose. Every transition registers how it ends before it can be interrupted, and where it was interrupted decides what that means: nothing sent, so the guard lifts and the acquisition is free; a command sent and unanswered, so the acquisition is retired instead — because the one thing that must not follow an unknown decision is a fresh mutation racing it. --- packages/workflow/src/cloudflare/client.ts | 2 + packages/workflow/src/cloudflare/commands.ts | 45 ++++++- .../workflow/src/cloudflare/dispatcher.ts | 22 +++- .../workflow/src/cloudflare/lifecycle-link.ts | 27 +++- .../workflow/src/cloudflare/owner-fork.ts | 72 ++++++++-- .../workflow/src/remote/lifecycle-link.ts | 18 ++- packages/workflow/src/remote/lifecycle.ts | 124 +++++++++++++++--- .../workflow/src/sqlite/workflow-schema.ts | 2 + .../tests/cloudflare/remote-fork.vitest.ts | 104 +++++++++++++++ .../cloudflare/support/executor-object.ts | 23 ++++ packages/workflow/tests/remote-fork.test.ts | 91 +++++++++++++ .../workflow/tests/remote-lifecycle.test.ts | 91 +++++++++++++ .../tests/support/remote-lifecycle-host.ts | 30 ++++- 13 files changed, 606 insertions(+), 45 deletions(-) diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 3ad72bc8c..8261c8f48 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -107,6 +107,7 @@ export type PrivateRefusal = | "command:corrupt-journal" | "command:not-forkable" | "command:wrong-execution" + | "command:needs-transfer" | "storage:foreign" | `storage:unsupported-version-v${number}` | "storage:corrupt"; @@ -188,6 +189,7 @@ export function privateRefusal(value: string): PrivateRefusal { case "command:corrupt-journal": case "command:not-forkable": case "command:wrong-execution": + case "command:needs-transfer": case "storage:foreign": case "storage:corrupt": return value; diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index fe56e26dd..20b783ee7 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -112,7 +112,15 @@ export type CommandRefusal = * different acquisition, and a live executor does not get to finish an * earlier executor's work by naming its id. */ - | "wrong-execution"; + | "wrong-execution" + /** + * A fork was asked to commit a transfer this connection never offered. + * + * Distinct from a malformed request: the request is well formed, the + * destination holds nothing, and the parts it names are not here — so the + * caller's next move is to copy the source again rather than to give up. + */ + | "needs-transfer"; export class CommandError extends Error { override name = "CommandError"; @@ -354,11 +362,25 @@ export interface ForkContinueCommand extends CommandEnvelope { readonly command: "fork-continue"; readonly runId: string; readonly creation: CreateWorkflowRunRequest; + /** Which fork this claims to be, as the destination retains it. */ + readonly origin: ForkContinuationOrigin; readonly runRecord: DurableEvent; readonly rootImport: DurableEvent; readonly executionId: string; } +/** + * The identity a continuation claims, compared against retained state. + * + * No anchor and no counts: those describe a copy in flight. What a destination + * that already holds the fork can be held to is where it came from and what it + * wrote for itself. + */ +export interface ForkContinuationOrigin { + readonly sourceRunId: string; + readonly checkpointEventId: string; +} + export interface SettleCommand extends CommandEnvelope { readonly command: "settle"; readonly completion: DocumentExecutionCompletion; @@ -412,7 +434,15 @@ const MEMBERS: Record = { cancel: [...ENVELOPE, "runId"], settle: [...ENVELOPE, "completion", "expectedWorkspaceRootId"], "fork-stage": [...ENVELOPE, "section", "position", "part"], - "fork-continue": [...ENVELOPE, "runId", "creation", "runRecord", "rootImport", "executionId"], + "fork-continue": [ + ...ENVELOPE, + "runId", + "creation", + "origin", + "runRecord", + "rootImport", + "executionId", + ], fork: [ ...ENVELOPE, "runId", @@ -702,6 +732,7 @@ export function parseCommand(raw: string): RunnerCommand { command, runId, creation: creation.value, + origin: continuationOrigin(members.get("origin")), runRecord: forkEvent(members.get("runRecord")), rootImport: forkEvent(members.get("rootImport")), executionId: text(members, "executionId", MAX_RUN_ID), @@ -813,6 +844,16 @@ function origin(value: unknown): ForkOrigin { }; } +/** Which fork a continuation claims to be continuing. */ +function continuationOrigin(value: unknown): ForkContinuationOrigin { + const found = object(value); + closed(found, ["sourceRunId", "checkpointEventId"]); + return { + sourceRunId: text(found, "sourceRunId", MAX_RUN_ID), + checkpointEventId: text(found, "checkpointEventId", MAX_ID), + }; +} + /** How many parts each section should have arrived in. */ function counts(value: unknown): ForkCounts { const found = object(value); diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index aaebabf8a..ac7b61517 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -142,10 +142,21 @@ function begunExecution(command: RunnerCommand, result: CommandResult): string | if (result.outcome !== "performed") { return null; } - if (command.command === "begin" || command.command === "fork") { - return command.executionId; + if ( + command.command !== "begin" && + command.command !== "fork" && + command.command !== "fork-continue" + ) { + return null; + } + // Performed is not the same as begun: a conflict and a lifecycle refusal are + // both answers this command performed, and neither began anything. What + // decides is whether the answer carries a value. + const value = result.value; + if (value === null || typeof value !== "object" || Array.isArray(value)) { + return null; } - return null; + return Reflect.get(value, "value") === null ? null : command.executionId; } /** What this acquisition has already spent of its own ledger. */ @@ -414,6 +425,7 @@ function perform( { runId: command.runId, creation: command.creation, + origin: command.origin, runRecord: command.runRecord, rootImport: command.rootImport, executionId: command.executionId, @@ -533,6 +545,10 @@ export function dispatchCommand( // If the run moved past it — recovered, settled, or held by somebody // live — the decision is history rather than authority, and handing it // back would hand back a database nobody may settle. + // A decision that granted execution authority is only re-observable + // once that exact execution is this acquisition's. Anything else — it + // was recovered, settled, or somebody live holds it — is history, and + // returning it would hand back a database nobody may settle. if (adoptExecution(ctx.storage, held.acquisitionId, command.id) === "stale") { throw new CommandError("stale-journal"); } diff --git a/packages/workflow/src/cloudflare/lifecycle-link.ts b/packages/workflow/src/cloudflare/lifecycle-link.ts index 6fdc726c8..9067d16de 100644 --- a/packages/workflow/src/cloudflare/lifecycle-link.ts +++ b/packages/workflow/src/cloudflare/lifecycle-link.ts @@ -233,6 +233,7 @@ export function cloudflareLifecycleLink( command: "fork-continue", runId: continuation.runId, creation: continuation.creation, + origin: continuation.origin, runRecord: record(continuation.runRecord), rootImport: record(continuation.rootImport), executionId: continuation.executionId, @@ -245,17 +246,21 @@ export function cloudflareLifecycleLink( if (refusal === "command:absent") { // Nothing there to continue. Not a failure: the caller's next move // is the source copy it has not needed until now. - return Ok("absent"); + return Ok | "absent">("absent"); } return Err(storageFailure(refusal)); } - return yield* forked(offered.value, continuation.runId); + const decided = yield* forked(offered.value, continuation.runId); + return decided.ok + ? Ok | "absent">(decided.value) + : decided; } catch (error) { return Err(translate(error)); } }, - - *commitFork(commit: RemoteForkCommit): Operation>> { + *commitFork( + commit: RemoteForkCommit, + ): Operation | "needs-transfer">> { try { const offered = yield* connection.ask( commit.commandId, @@ -274,9 +279,19 @@ export function cloudflareLifecycleLink( privateRefusal, ); if (offered.outcome === "refused") { - return Err(storageFailure(privateRefusal(offered.refusal))); + const refusal = privateRefusal(offered.refusal); + if (refusal === "command:needs-transfer") { + // The destination is empty and this connection offered it nothing. + // A closed outcome, not a failure: the caller copies the source. + const outcome: RemoteLifecycleAnswer | "needs-transfer" = "needs-transfer"; + return Ok(outcome); + } + return Err(storageFailure(refusal)); } - return yield* forked(offered.value, commit.runId); + const decided = yield* forked(offered.value, commit.runId); + return decided.ok + ? Ok | "needs-transfer">(decided.value) + : decided; } catch (error) { return Err(translate(error)); } diff --git a/packages/workflow/src/cloudflare/owner-fork.ts b/packages/workflow/src/cloudflare/owner-fork.ts index dd8bd0330..7a838df05 100644 --- a/packages/workflow/src/cloudflare/owner-fork.ts +++ b/packages/workflow/src/cloudflare/owner-fork.ts @@ -33,6 +33,7 @@ import { bytesOf } from "./encoding.ts"; import type { CreateWorkflowRunRequest } from "../storage/api.ts"; import { CommandError, + type ForkContinuationOrigin, type ForkCounts, type ForkOrigin, type ForkPart, @@ -170,6 +171,30 @@ export function stageForkPart( return { staged: count(held?.["parts"]) + 1 }; } +/** Whether this transfer describes anything at all. */ +function expectsTransfer(counts: ForkCounts): boolean { + return counts.inherited + counts.roots + counts.manifests + counts.blobs + counts.checkouts > 0; +} + +/** Whether this acquisition has offered any part of a fork. */ +function offered(storage: OwnerStorage, acquisitionId: string): boolean { + // A store with no scratch table at all has been offered nothing, which is + // the same answer: that table is created by the command that offers a part. + if ( + rows(storage, "SELECT name FROM sqlite_schema WHERE type = 'table' AND name = ?", FORK_TABLE) + .length === 0 + ) { + return false; + } + return ( + rows( + storage, + `SELECT 1 AS held FROM ${FORK_TABLE} WHERE acquisition_id = ? LIMIT 1`, + acquisitionId, + ).length > 0 + ); +} + /** Every part of one section, in the order it was offered, with no gaps. */ function section( storage: OwnerStorage, @@ -715,6 +740,7 @@ export function continueFork( input: { readonly runId: string; readonly creation: CreateWorkflowRunRequest; + readonly origin: ForkContinuationOrigin; readonly runRecord: DurableEvent; readonly rootImport: DurableEvent; readonly executionId: string; @@ -744,12 +770,17 @@ export function continueFork( const lineage = rows( storage, - `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, selection_anchor + `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, + selection_anchor, run_record_root_id, root_import_root_id FROM workflow_fork_lineage WHERE id = 1`, )[0]; - if (lineage === undefined) { - // A run, and not a fork. Continuing it as one would claim a lineage it - // does not have. + if ( + lineage === undefined || + lineage["source_run_id"] !== input.origin.sourceRunId || + lineage["checkpoint_event_id"] !== input.origin.checkpointEventId + ) { + // Not a fork at all, or a fork of somewhere else. Either way this request + // is not describing the run that is here. return { conflict: ["lineage"], refusal: null, value: null }; } @@ -766,16 +797,18 @@ export function continueFork( pinnedCommit: input.creation.definition.objectId, }), ); - const roots = new Set( - rows(storage, "SELECT root_id FROM workspace_roots").map((row) => String(row["root_id"])), - ); if ( heads.length !== 2 || serializeDurableEvent(input.runRecord) !== expected || heads[0]?.["record"] !== expected || heads[1]?.["record"] !== serializeDurableEvent(input.rootImport) || - !roots.has(String(heads[0]?.["workspace_root_id"])) || - !roots.has(String(heads[1]?.["workspace_root_id"])) + // The exact roots these rows were committed against, as this fork's own + // lineage retained them — not merely roots this store happens to hold. A + // head reassociated to another valid root is a different fork's head. + lineage["run_record_root_id"] === null || + lineage["root_import_root_id"] === null || + heads[0]?.["workspace_root_id"] !== lineage["run_record_root_id"] || + heads[1]?.["workspace_root_id"] !== lineage["root_import_root_id"] ) { return { conflict: ["lineage"], refusal: null, value: null }; } @@ -851,6 +884,13 @@ export function commitFork( // described in the shared shape and hashed by the shared rule, then compared. // A transfer that is not that selection stops here, with the destination // holding no run. + if (fresh && expectsTransfer(input.counts) && !offered(storage, acquisitionId)) { + // Nothing here, and nothing offered: the commit this command names never + // happened, and the staging it would have used went with the connection + // that made it. Said as its own answer, because it is the one failure a + // caller responds to by copying the source again. + throw new CommandError("needs-transfer"); + } const staged = fresh ? reconstruct(storage, acquisitionId, input) : undefined; const watermarks: ContentWatermarks = staged?.watermarks ?? { manifest: new Map(), @@ -882,7 +922,8 @@ export function commitFork( // lineage. const lineage = rows( storage, - `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, selection_anchor + `SELECT source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, + selection_anchor, run_record_root_id, root_import_root_id FROM workflow_fork_lineage WHERE id = 1`, )[0]; if ( @@ -999,14 +1040,17 @@ export function commitFork( storage.sql.exec( `INSERT INTO workflow_fork_lineage (id, source_run_id, checkpoint_event_id, checkpoint_workspace_root_id, - selection_anchor, created_at) - VALUES (1, ?, ?, ?, ?, ?)`, + selection_anchor, run_record_root_id, root_import_root_id, created_at) + VALUES (1, ?, ?, ?, ?, ?, ?, ?)`, input.origin.sourceRunId, input.origin.checkpointEventId, input.origin.checkpointWorkspaceRootId, - // The copy identity, retained with the lineage: a later fork of the same - // checkpoint is checkable here without reading the source at all. + // The copy identity and the two head associations, retained with the + // lineage: continuing this fork later is then checkable here, without + // reading the source at all. input.origin.anchor, + input.origin.runRecordWorkspaceRootId, + input.origin.rootImportWorkspaceRootId, now(), ); diff --git a/packages/workflow/src/remote/lifecycle-link.ts b/packages/workflow/src/remote/lifecycle-link.ts index de24aa7f5..70a3ba253 100644 --- a/packages/workflow/src/remote/lifecycle-link.ts +++ b/packages/workflow/src/remote/lifecycle-link.ts @@ -82,12 +82,19 @@ export interface RemoteBeginCommand { readonly executionId: string; } +/** Which fork a continuation claims, as the destination retains it. */ +export interface RemoteContinuationOrigin { + readonly sourceRunId: string; + readonly checkpointEventId: string; +} + /** Everything one committed fork is decided from. */ /** Taking up a destination that already holds this fork, without its source. */ export interface RemoteForkContinuation { readonly commandId: string; readonly runId: string; readonly creation: CreateWorkflowRunRequest; + readonly origin: RemoteContinuationOrigin; readonly runRecord: DurableEvent; readonly rootImport: DurableEvent; readonly executionId: string; @@ -143,7 +150,16 @@ export interface RemoteLifecycleLink { /** Offer one part of a fork's source to this acquisition's scratch. */ stageForkPart(commandId: string, part: RemoteForkPart): Operation>; /** Commit the offered parts as one destination run and its first execution. */ - commitFork(commit: RemoteForkCommit): Operation>>; + /** + * Commit the offered parts as one destination run and its first execution. + * + * `needs-transfer` is the one failure a caller answers by copying the source + * again: the destination holds nothing and the parts this command names were + * never offered on this connection. + */ + commitFork( + commit: RemoteForkCommit, + ): Operation | "needs-transfer">>; /** * Continue a destination that already holds this fork. * diff --git a/packages/workflow/src/remote/lifecycle.ts b/packages/workflow/src/remote/lifecycle.ts index 8b941f77e..3e4043d63 100644 --- a/packages/workflow/src/remote/lifecycle.ts +++ b/packages/workflow/src/remote/lifecycle.ts @@ -251,6 +251,14 @@ export function* useRemoteLifecycle( const addressed = identify(runId, question); const answered = yield* admitted.value.lifecycle.cancel(addressed.commandId, runId); if (!answered.ok) { + // The same distinction the other mutations make. A cancellation may + // have committed before its answer was lost, and the next attempt has + // to ask that question rather than a new one. + if (lost(answered.error)) { + unanswered(runId, addressed); + } else { + answeredNow(runId, addressed); + } return answered; } answeredNow(runId, addressed); @@ -285,6 +293,23 @@ export function* useRemoteLifecycle( return transitions(host, hold, identify, unanswered, answeredNow); } +/** + * One transition's grip on its acquisition, ended however the transition ends. + * + * `sent` is the line between two very different cancellations. Before a command + * has gone out, nothing can have happened, so the guard simply lifts. After one + * has gone out and no answer came back, the owner's decision is unknown — and + * the acquisition is retired rather than freed, because a fresh mutation racing + * an unknown decision is the one thing that must not happen. Whoever wants to + * continue takes a new acquisition and asks the same question again. + */ +interface Grip { + /** Say that a command has gone out and its answer is not yet known. */ + sending(): void; + /** Say that this transition reached an answer, whatever the answer was. */ + done(): void; +} + /** * Take this acquisition for one transition, or say why it cannot be taken. * @@ -302,6 +327,40 @@ function engage(held: Hold): WorkflowRequestError | undefined { return undefined; } +/** + * Register how this transition lets go, before it can be interrupted. + * + * `ensure` runs on every exit — a return, a raise, or cancellation while an + * owner answer is still outstanding — so no path can leave the guard held or + * leave a sent command unaccounted for. + */ +function* gripped(held: Hold): Operation { + let sent = false; + let finished = false; + yield* ensure(function* () { + if (finished) { + return; + } + if (sent) { + // A command went out and nothing came back. This acquisition is retired + // rather than released: its authority ends with it, and nothing new can + // be sent on it while the first outcome is unknown. + held.live = false; + return; + } + held.busy = false; + }); + return { + sending() { + sent = true; + }, + done() { + finished = true; + held.busy = false; + }, + }; +} + /** * Whether a failure left the owner's decision unknown. * @@ -345,8 +404,9 @@ function transitions( if (busy !== undefined) { return Err(busy); } + const grip = yield* gripped(held); if (request.action === "resume" && request.creation !== undefined) { - held.busy = false; + grip.done(); return Err(new WorkflowRequestError("a resume does not carry a creation.")); } // Minted once for this question, outside anything that could retry: the @@ -363,6 +423,7 @@ function transitions( : canonicalJson(request.creation.retrieval), ]); const addressed = identify(request.runId, question); + grip.sending(); const answered = yield* held.connection.lifecycle.begin({ commandId: addressed.commandId, runId: request.runId, @@ -371,7 +432,7 @@ function transitions( retrieval: request.creation?.retrieval, executionId: addressed.executionId, }); - held.busy = false; + grip.done(); if (!answered.ok) { // Which kind of failure decides whether the question survives it. An // answer that was lost leaves the identity standing, so a replacement @@ -458,6 +519,7 @@ function transitions( if (busy !== undefined) { return Err(busy); } + const grip = yield* gripped(held); const head = headOf(request); const creation = creationRequest(request.runId, request.creation); const question = questionOf([ @@ -476,26 +538,51 @@ function transitions( // continued from what it retains, and a decision this owner already made // can be re-observed. Either way the source is not needed, and it may not // be there any more. - const retained = - addressed.command === undefined - ? undefined - : yield* held.connection.lifecycle.commitFork(addressed.command); - if (retained !== undefined && retained.ok && retained.value.kind === "performed") { - held.busy = false; - answeredNow(request.runId, addressed); - held.execution = retained.value.value.execution.executionId; - return Ok(yield* begun(held, retained.value.value)); + if (addressed.command !== undefined) { + // An exact command was sent once and never answered. It is resent + // verbatim, and every outcome is classified here: nothing else is + // tried until this one's is known. + grip.sending(); + const retained = yield* held.connection.lifecycle.commitFork(addressed.command); + grip.done(); + if (!retained.ok) { + if (lost(retained.error)) { + // Ambiguous again. The claim stands, and no other command is sent. + unanswered(request.runId, addressed, addressed.command); + } else { + answeredNow(request.runId, addressed); + } + return retained; + } + if (retained.value !== "needs-transfer") { + answeredNow(request.runId, addressed); + if (retained.value.kind === "refused") { + return Err(refusalError(retained.value.refusal, request.runId)); + } + held.execution = retained.value.value.execution.executionId; + return Ok(yield* begun(held, retained.value.value)); + } + // The one outcome that sends this same logical fork back to its + // source: the destination holds nothing, and the parts this command + // names went with the connection that offered them. } + grip.sending(); const continued = yield* held.connection.lifecycle.continueFork({ commandId: addressed.commandId, runId: request.runId, creation, + // What this request names, which the destination proves against what + // it retained. Nothing here was read from the source. + origin: { + sourceRunId: request.selection.sourceRunId, + checkpointEventId: request.selection.checkpointEventId, + }, runRecord: head.runRecord, rootImport: request.rootImport, executionId: addressed.executionId, }); + grip.done(); if (!continued.ok) { - held.busy = false; if (lost(continued.error)) { unanswered(request.runId, addressed); } else { @@ -504,7 +591,6 @@ function transitions( return continued; } if (continued.value !== "absent") { - held.busy = false; answeredNow(request.runId, addressed); if (continued.value.kind === "refused") { return Err(refusalError(continued.value.refusal, request.runId)); @@ -515,15 +601,15 @@ function transitions( // Nothing there. Making a fork needs the whole source, staged under this // acquisition and committed in one transaction. + // Nothing has been mutated yet, so the guard is simply held across the + // source read and the staging: a cancellation here leaves no unknown. const source = yield* readSource(host, request); if (!source.ok) { - held.busy = false; answeredNow(request.runId, addressed); return source; } const staged = yield* offer(host, held.connection.lifecycle, source.value); if (!staged.ok) { - held.busy = false; return staged; } const command: RemoteForkCommit = { @@ -550,8 +636,9 @@ function transitions( rootImport: request.rootImport, executionId: addressed.executionId, }; + grip.sending(); const answered = yield* held.connection.lifecycle.commitFork(command); - held.busy = false; + grip.done(); if (!answered.ok) { if (lost(answered.error)) { // Kept with the exact bytes it was sent with, so a replacement @@ -564,6 +651,11 @@ function transitions( return answered; } answeredNow(request.runId, addressed); + if (answered.value === "needs-transfer") { + // Offered and still not there. Repeating the same transfer would meet + // the same answer. + return Err(new WorkflowRequestError("this fork's transfer did not reach its destination.")); + } if (answered.value.kind === "refused") { return Err(refusalError(answered.value.refusal, request.runId)); } diff --git a/packages/workflow/src/sqlite/workflow-schema.ts b/packages/workflow/src/sqlite/workflow-schema.ts index 563a5e686..82a9d61c5 100644 --- a/packages/workflow/src/sqlite/workflow-schema.ts +++ b/packages/workflow/src/sqlite/workflow-schema.ts @@ -462,6 +462,8 @@ export const OBJECTS: ReadonlyMap = new Map([ selection_anchor IS NULL OR (length(selection_anchor) = 64 AND selection_anchor NOT GLOB '*[^0-9a-f]*') ), + run_record_root_id TEXT REFERENCES workspace_roots(root_id) ON DELETE RESTRICT, + root_import_root_id TEXT REFERENCES workspace_roots(root_id) ON DELETE RESTRICT, created_at TEXT NOT NULL ) STRICT`, }, diff --git a/packages/workflow/tests/cloudflare/remote-fork.vitest.ts b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts index cf530b20a..bcbf9cf43 100644 --- a/packages/workflow/tests/cloudflare/remote-fork.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-fork.vitest.ts @@ -511,6 +511,7 @@ function continuation(overrides: Record = {}): Record { expect(watermarks.manifests).toEqual([11]); expect(watermarks.blobs).toEqual([23]); }); + it("refuses a continuation naming another source or another checkpoint", async () => { + for (const changed of [ + { sourceRunId: "8fktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa" }, + { checkpointEventId: "event-elsewhere" }, + ]) { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + const refused = await ask( + stub, + continuation({ + origin: { + sourceRunId: SOURCE_RUN_ID, + checkpointEventId: "event-work", + ...changed, + }, + }), + ); + + expect(Object(refused["value"])["conflict"]).toEqual(["lineage"]); + // No recovery and no replacement: the first execution is still open. + const executions = await on(stub, (owner) => owner.executionRows()); + expect(executions).toHaveLength(1); + expect(executions[0]?.["stopped_at"]).toBe(null); + } + }); + + it("refuses a head reassociated to another root this store retains", async () => { + for (const head of ["run_record", "root_import"]) { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + // Another valid retained root, and the head now names it. Membership is + // not identity: this is not the association the fork committed. + await on(stub, (owner) => owner.reassociateHead(head)); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + + const refused = await ask(stub, continuation()); + + expect([head, Object(refused["value"])["conflict"]]).toEqual([head, ["lineage"]]); + expect(await on(stub, (owner) => owner.executionRows())).toHaveLength(1); + } + }); + + it("adopts the execution a lost continuation began, and settles once", async () => { + const stub = executor(); + await connected(stub); + let minted = 0; + await offer(stub, () => `command-${(minted += 1)}`); + await ask(stub, commit()); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const first = await ask(stub, continuation()); + expect(first["outcome"]).toBe("performed"); + + // Its answer never arrived and the connection died. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const again = await ask(stub, continuation()); + + expect(again).toEqual(first); + // The decision came back only because its execution became this + // acquisition's, so this acquisition can settle it. + const held = await on(stub, (owner) => owner.heldExecutions()); + expect(held.map((row) => row["execution_id"])).toEqual(["execution-2"]); + const root = await on(stub, (owner) => owner.currentRootId()); + const settled = await ask(stub, { + id: "settle-2", + command: "settle", + completion: { executionId: "execution-2", status: "completed" }, + expectedWorkspaceRootId: root, + }); + expect(settled["outcome"]).toBe("performed"); + + // And once settled, the same continuation is history rather than authority. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + expect((await ask(stub, continuation()))["refusal"]).toBe("command:stale-journal"); + }); + + it("says a transfer it never received is one it needs", async () => { + const stub = executor(); + await connected(stub); + + // The destination is empty and this connection offered nothing: the final + // command's own answer, not a malformed request. + const refused = await ask(stub, commit()); + + expect(refused).toEqual({ + id: "fork-commit", + outcome: "refused", + refusal: "command:needs-transfer", + }); + expect(await on(stub, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); }); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index c580c9620..5b7bb57bf 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -368,6 +368,29 @@ export class ExecutorObject extends WorkflowOwnerObject { .toArray(); } + /** + * Point one head row at a different root this store already retains. + * + * A root that is here and valid, and not the one the fork committed against. + * What this makes is a destination whose head association is no longer its + * own — which a continuation has to notice. + */ + reassociateHead(head: string): void { + const other = sha256Hex(`${WORKSPACE_ROOT_DOMAIN}${NEXT_ROOT_MANIFEST}`); + this.ctx.storage.sql.exec( + "INSERT INTO workspace_roots (root_id, format_version, manifest) VALUES (?, 1, ?) " + + "ON CONFLICT(root_id) DO NOTHING", + other, + NEXT_ROOT_MANIFEST, + ); + const sequence = head === "run_record" ? 1 : 2; + this.ctx.storage.sql.exec( + "UPDATE journal_events SET workspace_root_id = ? WHERE sequence = ?", + other, + sequence, + ); + } + /** Close every admitted connection, as a lost executor leaves them. */ dropConnections(): void { for (const socket of this.ctx.getWebSockets("executor")) { diff --git a/packages/workflow/tests/remote-fork.test.ts b/packages/workflow/tests/remote-fork.test.ts index e7786d5a8..ae3608a35 100644 --- a/packages/workflow/tests/remote-fork.test.ts +++ b/packages/workflow/tests/remote-fork.test.ts @@ -309,4 +309,95 @@ describe("a remote fork's destination", () => { expect(outcome.asked.at(-1)).toBe("fork"); expect(commands.filter((id) => id === "command-1").length).toBeGreaterThan(1); }); + + it("sends only the command it already sent when its answer is lost twice", function* () { + const asked: string[] = []; + const sourced: string[] = []; + const script: Script = { + asked, + sourced, + loseAnswer: new Set(["command-1"]), + committed: new Map(), + source: source(), + }; + const outcome = yield* installed(script, function* (transitions) { + const first = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + // Lost again on the resend. + script.loseAnswer?.add("command-1"); + const asking = asked.length; + const second = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + return { first, second, sent: asked.slice(asking), sourced: [...sourced] }; + }); + + expect(outcome.first.ok).toBe(false); + expect(outcome.second.ok).toBe(false); + // The second attempt sent the finalized command and nothing else: no + // continuation, and no second source read. + expect(outcome.sent).toEqual(["fork"]); + expect(outcome.sourced).toEqual([SOURCE_RUN_ID]); + }); + + it("retires an invocation the owner definitively refused", function* () { + const asked: string[] = []; + const script: Script = { + asked, + forkRefuses: true, + loseAnswer: new Set(["command-1"]), + committed: new Map(), + source: source(), + }; + const outcome = yield* installed(script, function* (transitions) { + const first = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + const asking = asked.length; + const second = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + return { first, second, sent: asked.slice(asking) }; + }); + + expect(outcome.second.ok).toBe(false); + // The resend was answered — with a conflict — so nothing followed it. + expect(outcome.sent).toEqual(["fork"]); + }); + + it("restages the same snapshot when the destination says it needs one", function* () { + const asked: string[] = []; + const script: Script = { + asked, + loseAnswer: new Set(["command-1"]), + needsTransfer: new Set(["command-1"]), + committed: new Map(), + source: source(), + }; + const outcome = yield* installed(script, function* (transitions) { + const first = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + const asking = asked.length; + const second = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + return { first, second, sent: asked.slice(asking) }; + }); + + expect(outcome.first.ok).toBe(false); + expect(outcome.second.ok).toBe(true); + // The resend was told its transfer is not there, so the same logical fork + // staged the snapshot again and committed once. + expect(outcome.sent[0]).toBe("fork"); + expect(outcome.sent.filter((command) => command === "fork-stage")).toHaveLength(6); + expect(outcome.sent.at(-1)).toBe("fork"); + }); }); diff --git a/packages/workflow/tests/remote-lifecycle.test.ts b/packages/workflow/tests/remote-lifecycle.test.ts index ed8314911..9549acca0 100644 --- a/packages/workflow/tests/remote-lifecycle.test.ts +++ b/packages/workflow/tests/remote-lifecycle.test.ts @@ -345,4 +345,95 @@ describe("a remote run's executor lifecycle", () => { }); expect(second.ok).toBe(true); }); + + it("asks the same cancellation again when its answer was lost", function* () { + const commands: string[] = []; + const outcome = yield* installed( + { commands, loseAnswer: new Set(["command-1"]) }, + function* () { + const first = yield* WorkflowLifecycle.operations.cancel(RUN_ID); + // The connection that asked is gone; the question is not. + const second = yield* WorkflowLifecycle.operations.cancel(RUN_ID); + return { first, second, commands: [...commands] }; + }, + ); + + expect(outcome.first.ok).toBe(false); + expect(outcome.second.ok).toBe(true); + // One identity, asked twice, so the owner answers with what it decided. + expect(outcome.commands).toEqual(["command-1", "command-1"]); + }); + + it("gives a later cancellation a fresh identity once one was answered", function* () { + const commands: string[] = []; + const outcome = yield* installed({ commands }, function* () { + const first = yield* WorkflowLifecycle.operations.cancel(RUN_ID); + const second = yield* WorkflowLifecycle.operations.cancel(RUN_ID); + return { first, second, commands: [...commands] }; + }); + + expect(outcome.first.ok).toBe(true); + expect(outcome.second.ok).toBe(true); + // Answered, so the claim was retired and the next call is its own. + expect(outcome.commands).toEqual(["command-1", "command-2"]); + }); + + it("frees the acquisition when a call is cancelled before it sends", function* () { + const asked: string[] = []; + const outcome = yield* installed({ asked }, function* (transitions) { + const lock = yield* acquired(); + // Cancelled while it is still deciding it may not proceed: nothing was + // sent, so the acquisition is free to try again. + const refused = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "resume", + creation: { + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + }, + }); + const after = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + return { refused, after, asked: [...asked] }; + }); + + expect(outcome.refused.ok).toBe(false); + // The guard lifted, so the corrected call went through. + expect(outcome.after.ok).toBe(true); + expect(outcome.asked).toEqual(["begin"]); + }); + + it("retires an acquisition cancelled after its command went out", function* () { + const asked: string[] = []; + const entered = withResolvers(); + const script: Script = { + asked, + gate: { + *wait(): Operation { + entered.resolve(); + // Never resolved: this call is cancelled while it waits. + yield* withResolvers().operation; + }, + }, + }; + const outcome = yield* installed(script, function* (transitions) { + const lock = yield* acquired(); + const sent = yield* spawn(() => transitions.begin(lock, { runId: RUN_ID, action: "resume" })); + yield* entered.operation; + // The call is interrupted with its answer outstanding. + yield* sent.halt(); + // The lock cannot start anything else: whether the owner committed is + // unknown, and a fresh mutation must not race that. + return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + }); + + expect(outcome.ok).toBe(false); + expect(asked.filter((command) => command === "begin")).toHaveLength(1); + }); }); diff --git a/packages/workflow/tests/support/remote-lifecycle-host.ts b/packages/workflow/tests/support/remote-lifecycle-host.ts index d3ab2b3fc..6bad34af9 100644 --- a/packages/workflow/tests/support/remote-lifecycle-host.ts +++ b/packages/workflow/tests/support/remote-lifecycle-host.ts @@ -34,7 +34,11 @@ import type { DocumentExecutionRecord, WorkflowRunRecord, } from "../../src/storage/record.ts"; -import { WorkflowRequestError, WorkflowTransactionError } from "../../src/storage/errors.ts"; +import { + WorkflowRequestError, + WorkflowRunConflictError, + WorkflowTransactionError, +} from "../../src/storage/errors.ts"; import type { RemoteWorkspaceLink } from "../../src/remote/database.ts"; export const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; @@ -68,6 +72,10 @@ export interface Script { readonly commands?: string[]; /** Every retrieval value a begin carried. */ readonly retrievals?: (unknown | null)[]; + /** Answer every fork commit with a definitive conflict. */ + readonly forkRefuses?: boolean; + /** Answer the first fork commit by saying its transfer is not here. */ + readonly needsTransfer?: Set; /** Held open until a test releases it, so a call can be caught in flight. */ readonly gate?: { wait(): Operation }; /** Every run whose source was resolved. */ @@ -208,6 +216,10 @@ function lifecycle(script: Script): RemoteLifecycleLink { *cancel(commandId: string): Operation>> { script.asked?.push("cancel"); script.commands?.push(commandId); + if (script.loseAnswer?.has(commandId) === true) { + script.loseAnswer.delete(commandId); + return Err(new WorkflowTransactionError("the connection ended before it answered.")); + } return Ok({ kind: "performed", value: record() }); }, // deno-lint-ignore require-yield @@ -231,7 +243,9 @@ function lifecycle(script: Script): RemoteLifecycleLink { return Ok({ kind: "performed", value: begun(continuation.executionId, script) }); }, - *commitFork(commit: RemoteForkCommit): Operation>> { + *commitFork( + commit: RemoteForkCommit, + ): Operation | "needs-transfer">> { script.asked?.push("fork"); script.commits?.push(commit); if (script.forkConflict !== undefined) { @@ -241,9 +255,19 @@ function lifecycle(script: Script): RemoteLifecycleLink { script.commands?.push(commit.commandId); if (script.loseAnswer?.has(commit.commandId) === true) { script.loseAnswer.delete(commit.commandId); - script.committed?.set(commit.commandId, begun(commit.executionId, script)); + // The mutation committed unless this scenario says it never did. + if (script.needsTransfer?.has(commit.commandId) !== true) { + script.committed?.set(commit.commandId, begun(commit.executionId, script)); + } return Err(new WorkflowTransactionError("the connection ended before it answered.")); } + if (script.needsTransfer?.has(commit.commandId) === true) { + script.needsTransfer.delete(commit.commandId); + return Ok | "needs-transfer">("needs-transfer"); + } + if (script.forkRefuses === true) { + return Err(new WorkflowRunConflictError(commit.runId, ["definition"])); + } const already = script.committed?.get(commit.commandId); if (already !== undefined) { return Ok({ kind: "performed", value: already }); From 15cc2111b52ccae3c741e09c2dc1590c7c3e6509 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Mon, 7 Sep 2026 19:12:55 -0300 Subject: [PATCH 55/81] =?UTF-8?q?=F0=9F=90=9B=20Name=20each=20question=20a?= =?UTF-8?q?=20call=20asks,=20and=20give=20up=20the=20socket=20with=20the?= =?UTF-8?q?=20lock?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fork asks its destination several things under one call: whether it already holds this fork, an offer for every member of the snapshot, and the commit. All of them went out under the call's single identity, and an owner keys a retained decision by the identity it was asked under. So the commit met the continuation's own fingerprint and was refused as a repeat of something it is not, and a fork that had staged everything could never be committed at all. Each question now carries its own name, derived from the call and from which question it is — a part's section and position included — so two of them never collide, and a retry still spells every one of them exactly the way the first attempt did. The staged parts stop being minted fresh per attempt for the same reason: a part the destination already holds has to be recognizable as that part rather than arriving as a new one. An acquisition retired while an outcome is unknown now ends its connection. Retiring the lock and leaving the socket open left the owner holding a live executor nobody was allowed to use: this runner would send nothing more on it, and no one else could take the run while it was held, so the run was unreachable by anybody until the enclosing scope happened to end. The connection is the acquisition, so it goes when the acquisition does — through the same teardown that scope exit reaches, once, with whoever was still waiting told the connection closed. --- .../src/cloudflare/executor-connection.ts | 8 +++ packages/workflow/src/remote/client.ts | 12 ++++ .../workflow/src/remote/lifecycle-link.ts | 9 +++ packages/workflow/src/remote/lifecycle.ts | 43 ++++++++++++--- packages/workflow/tests/remote-client.test.ts | 37 +++++++++++++ packages/workflow/tests/remote-fork.test.ts | 55 ++++++++++++++++--- .../workflow/tests/remote-lifecycle.test.ts | 18 ++++-- .../tests/support/remote-lifecycle-host.ts | 18 +++++- 8 files changed, 174 insertions(+), 26 deletions(-) diff --git a/packages/workflow/src/cloudflare/executor-connection.ts b/packages/workflow/src/cloudflare/executor-connection.ts index 937379590..468f5f496 100644 --- a/packages/workflow/src/cloudflare/executor-connection.ts +++ b/packages/workflow/src/cloudflare/executor-connection.ts @@ -52,6 +52,14 @@ export function* useExecutorConnection( return Ok({ link: cloudflareRunLink(connection, nextId, runId), lifecycle: cloudflareLifecycleLink(connection, reads, nextId), + // deno-lint-ignore require-yield + *close(): Operation { + // The socket is the acquisition. Ending the connection is how this + // runner stops being the run's executor before its scope ends, and it + // is the same teardown scope exit would reach, so the scope ending + // afterwards finds nothing left to do. + connection.close(); + }, }); } catch (error) { // Whatever went wrong reaching or building the connection, a caller learns diff --git a/packages/workflow/src/remote/client.ts b/packages/workflow/src/remote/client.ts index d03d2e152..d597962e6 100644 --- a/packages/workflow/src/remote/client.ts +++ b/packages/workflow/src/remote/client.ts @@ -110,6 +110,15 @@ export interface OwnerConnection { parse: AnswerParser, parseRefusal?: (refusal: string) => string, ): Operation>; + /** + * End this connection now, ahead of the scope that owns it. + * + * The same teardown scope exit reaches, and it still runs once: whoever is + * waiting is told the connection closed, the listeners come off, and the + * socket goes. A caller that has given up on an acquisition uses this to + * stop being the run's executor without waiting for its scope to end. + */ + close(): void; } /** The most bytes one answer may carry. */ @@ -315,6 +324,9 @@ export function useOwnerConnection(socket: OwnerSocket): Operation( id: string, command: Record, diff --git a/packages/workflow/src/remote/lifecycle-link.ts b/packages/workflow/src/remote/lifecycle-link.ts index 70a3ba253..eaf4b4682 100644 --- a/packages/workflow/src/remote/lifecycle-link.ts +++ b/packages/workflow/src/remote/lifecycle-link.ts @@ -124,6 +124,15 @@ export interface RemoteForkCommit { export interface RemoteExecutorConnection { readonly link: RemoteWorkspaceLink; readonly lifecycle: RemoteLifecycleLink; + /** + * End this connection now, before the scope that owns it ends. + * + * An acquisition retired while a command's outcome is unknown must stop + * being the owner's live executor: a lock the runner has given up on while + * its socket still holds the run would leave the run unreachable by anybody, + * including whoever wants to ask the same question again. + */ + close(): Operation; } /** diff --git a/packages/workflow/src/remote/lifecycle.ts b/packages/workflow/src/remote/lifecycle.ts index 3e4043d63..9c35d1ddc 100644 --- a/packages/workflow/src/remote/lifecycle.ts +++ b/packages/workflow/src/remote/lifecycle.ts @@ -249,7 +249,10 @@ export function* useRemoteLifecycle( } const question = questionOf(["cancel"]); const addressed = identify(runId, question); - const answered = yield* admitted.value.lifecycle.cancel(addressed.commandId, runId); + const answered = yield* admitted.value.lifecycle.cancel( + commandOf(addressed, "cancel"), + runId, + ); if (!answered.ok) { // The same distinction the other mutations make. A cancellation may // have committed before its answer was lost, and the next attempt has @@ -344,8 +347,11 @@ function* gripped(held: Hold): Operation { if (sent) { // A command went out and nothing came back. This acquisition is retired // rather than released: its authority ends with it, and nothing new can - // be sent on it while the first outcome is unknown. + // be sent on it while the first outcome is unknown. The connection goes + // with it, because a lock nobody may use whose socket still holds the + // run would leave the run unreachable by anyone at all. held.live = false; + yield* held.connection.close(); return; } held.busy = false; @@ -425,7 +431,7 @@ function transitions( const addressed = identify(request.runId, question); grip.sending(); const answered = yield* held.connection.lifecycle.begin({ - commandId: addressed.commandId, + commandId: commandOf(addressed, "begin"), runId: request.runId, action: request.action, creation, @@ -485,7 +491,7 @@ function transitions( ]); const addressed = identify(held.runId, question); const answered = yield* held.connection.lifecycle.settle( - addressed.commandId, + commandOf(addressed, "settle"), completion, frontier.workspaceRootId, ); @@ -568,7 +574,7 @@ function transitions( } grip.sending(); const continued = yield* held.connection.lifecycle.continueFork({ - commandId: addressed.commandId, + commandId: commandOf(addressed, "continue"), runId: request.runId, creation, // What this request names, which the destination proves against what @@ -608,12 +614,12 @@ function transitions( answeredNow(request.runId, addressed); return source; } - const staged = yield* offer(host, held.connection.lifecycle, source.value); + const staged = yield* offer(addressed, held.connection.lifecycle, source.value); if (!staged.ok) { return staged; } const command: RemoteForkCommit = { - commandId: addressed.commandId, + commandId: commandOf(addressed, "commit"), runId: request.runId, retrieval: request.creation.retrieval, creation, @@ -718,6 +724,20 @@ function creationShape(creation: CreateWorkflowRunRequest): string { }); } +/** + * One internal command's own identity within a logical call. + * + * A fork asks the destination more than one question — whether it already + * holds this fork, and then to commit the transfer — and the owner keys a + * retained decision by the identity it was asked under. One identity naming + * two different requests would meet its own earlier fingerprint and be refused + * as a repeat of something else, so each kind gets its own, derived from the + * call so that a retry spells it the same way. + */ +function commandOf(invocation: Ambiguous, kind: string): string { + return `${invocation.commandId}:${kind}`; +} + /** * What one invocation is asking, as text two calls can be compared by. * @@ -756,7 +776,7 @@ function* readSource( * whether these add up to one. */ function* offer( - host: RemoteLifecycleHost, + invocation: Ambiguous, lifecycle: RemoteLifecycleLink, source: RemoteForkSource, ): Operation> { @@ -802,7 +822,12 @@ function* offer( parts.push({ section: "checkouts", position, part: { ...checkout } }); }); for (const part of parts) { - const staged = yield* lifecycle.stageForkPart(host.ids.command(), part); + // Named by the call and the place in it, so restaging the same logical + // fork offers the same parts under the same identities. + const staged = yield* lifecycle.stageForkPart( + commandOf(invocation, `part:${part.section}:${part.position}`), + part, + ); if (!staged.ok) { return staged; } diff --git a/packages/workflow/tests/remote-client.test.ts b/packages/workflow/tests/remote-client.test.ts index 6dcd3fb5f..a9f74f92d 100644 --- a/packages/workflow/tests/remote-client.test.ts +++ b/packages/workflow/tests/remote-client.test.ts @@ -252,6 +252,43 @@ describe("a connection to a run's owner", () => { expect(wire.listening).toBe(0); }); + it("ends on request, once, ahead of the scope that owns it", function* () { + const wire = fakeSocket(); + const raised: unknown[] = []; + yield* scoped(function* () { + const owner = yield* useOwnerConnection(wire.socket); + yield* sleep(0); + const asking = yield* spawn(function* () { + try { + yield* owner.ask("a1", { command: "frontier" }, readString); + } catch (error) { + raised.push(error); + } + }); + yield* sleep(0); + + // A caller that has given up on this acquisition ends the connection + // rather than waiting for its scope: the socket is the acquisition, and + // one still open still holds the run. + owner.close(); + yield* asking; + expect(raised.map(refusalOf)).toEqual(["closed"]); + expect(wire.closes).toBe(1); + expect(wire.listening).toBe(0); + + // Nothing new goes out on it afterwards. + try { + yield* owner.ask("a2", { command: "frontier" }, readString); + } catch (error) { + raised.push(error); + } + expect(raised.map(refusalOf)).toEqual(["closed", "closed"]); + }); + // And the scope ending finds the teardown already done rather than + // closing a second time. + expect(wire.closes).toBe(1); + }); + it("fails closed on an answer naming a request nobody made", function* () { const wire = fakeSocket(); let raised: unknown; diff --git a/packages/workflow/tests/remote-fork.test.ts b/packages/workflow/tests/remote-fork.test.ts index ae3608a35..8096bb4d7 100644 --- a/packages/workflow/tests/remote-fork.test.ts +++ b/packages/workflow/tests/remote-fork.test.ts @@ -163,6 +163,28 @@ describe("a remote fork's destination", () => { }); }); + it("names every question of one fork distinctly, under the one identity", function* () { + const commands: string[] = []; + const parts: string[] = []; + yield* installed({ commands, parts, source: source() }, function* (transitions) { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + + // One fork asks the destination eight questions: whether it already holds + // this fork, six offers of the snapshot, and the commit. + const asked = [...commands, ...parts]; + expect(asked).toHaveLength(8); + // Each is its own command. An owner keys a retained decision by the + // identity it was asked under, so two different requests sharing one + // identity would meet each other's fingerprint and be refused as repeats + // of something they are not. + expect(new Set(asked).size).toBe(asked.length); + // And all of them belong to the one call, so a retry spells each of them + // exactly the way the first attempt did. + expect(new Set(asked.map((command) => command.split(":")[0])).size).toBe(1); + }); + it("carries the inherited records exactly as the source retained them", function* () { const staged: RemoteForkPart[] = []; yield* installed({ staged, source: source() }, function* (transitions) { @@ -278,7 +300,7 @@ describe("a remote fork's destination", () => { const asked: string[] = []; const sourced: string[] = []; const commands: string[] = []; - const loseAnswer = new Set(["command-1"]); + const loseAnswer = new Set(["command-1:commit"]); const committed = new Map(); const script: Script = { asked, @@ -307,7 +329,7 @@ describe("a remote fork's destination", () => { // was read once — for the first attempt — and not again. expect(outcome.sourced).toEqual([SOURCE_RUN_ID]); expect(outcome.asked.at(-1)).toBe("fork"); - expect(commands.filter((id) => id === "command-1").length).toBeGreaterThan(1); + expect(commands.filter((id) => id === "command-1:commit").length).toBeGreaterThan(1); }); it("sends only the command it already sent when its answer is lost twice", function* () { @@ -316,7 +338,7 @@ describe("a remote fork's destination", () => { const script: Script = { asked, sourced, - loseAnswer: new Set(["command-1"]), + loseAnswer: new Set(["command-1:commit"]), committed: new Map(), source: source(), }; @@ -326,7 +348,7 @@ describe("a remote fork's destination", () => { return yield* transitions.fork(lock, request()); }); // Lost again on the resend. - script.loseAnswer?.add("command-1"); + script.loseAnswer?.add("command-1:commit"); const asking = asked.length; const second = yield* scoped(function* () { const lock = yield* acquired(DESTINATION); @@ -348,7 +370,7 @@ describe("a remote fork's destination", () => { const script: Script = { asked, forkRefuses: true, - loseAnswer: new Set(["command-1"]), + loseAnswer: new Set(["command-1:commit"]), committed: new Map(), source: source(), }; @@ -372,10 +394,12 @@ describe("a remote fork's destination", () => { it("restages the same snapshot when the destination says it needs one", function* () { const asked: string[] = []; + const parts: string[] = []; const script: Script = { asked, - loseAnswer: new Set(["command-1"]), - needsTransfer: new Set(["command-1"]), + parts, + loseAnswer: new Set(["command-1:commit"]), + needsTransfer: new Set(["command-1:commit"]), committed: new Map(), source: source(), }; @@ -385,11 +409,18 @@ describe("a remote fork's destination", () => { return yield* transitions.fork(lock, request()); }); const asking = asked.length; + const staging = parts.length; const second = yield* scoped(function* () { const lock = yield* acquired(DESTINATION); return yield* transitions.fork(lock, request()); }); - return { first, second, sent: asked.slice(asking) }; + return { + first, + second, + sent: asked.slice(asking), + offered: parts.slice(0, staging), + reoffered: parts.slice(staging), + }; }); expect(outcome.first.ok).toBe(false); @@ -397,6 +428,14 @@ describe("a remote fork's destination", () => { // The resend was told its transfer is not there, so the same logical fork // staged the snapshot again and committed once. expect(outcome.sent[0]).toBe("fork"); + // Each part of one offer is its own command, so no part meets another + // part's fingerprint at the owner. + expect(new Set(outcome.offered).size).toBe(outcome.offered.length); + // And the parts are named by the call and their place in it, so the same + // logical fork restages under exactly the identities it used before: a + // part the owner already holds is recognized as that part, not as a new + // one. + expect(outcome.reoffered).toEqual(outcome.offered); expect(outcome.sent.filter((command) => command === "fork-stage")).toHaveLength(6); expect(outcome.sent.at(-1)).toBe("fork"); }); diff --git a/packages/workflow/tests/remote-lifecycle.test.ts b/packages/workflow/tests/remote-lifecycle.test.ts index 9549acca0..7445321e1 100644 --- a/packages/workflow/tests/remote-lifecycle.test.ts +++ b/packages/workflow/tests/remote-lifecycle.test.ts @@ -273,7 +273,7 @@ describe("a remote run's executor lifecycle", () => { const first = yield* scoped(function* () { const lock = yield* acquired(); // The owner will commit and the answer will be lost. - loseAnswer.add("command-1"); + loseAnswer.add("command-1:begin"); return yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); }); // The connection is gone with its answer. A replacement acquisition asks @@ -289,7 +289,7 @@ describe("a remote run's executor lifecycle", () => { expect(outcome.second.ok).toBe(true); // The same command identity both times, so the owner answered with the // decision it had already made rather than making a second one. - expect(commands).toEqual(["command-1", "command-1"]); + expect(commands).toEqual(["command-1:begin", "command-1:begin"]); if (outcome.second.ok) { // And the execution the caller is handed is the one that was begun. expect(outcome.second.value.execution.executionId).toBe("execution-1"); @@ -337,7 +337,7 @@ describe("a remote run's executor lifecycle", () => { // The owner answered, so that question is finished. What it is not is a // claim on the next one's identity. expect(outcome.refused.ok).toBe(false); - expect(outcome.commands).toEqual(["command-1"]); + expect(outcome.commands).toEqual(["command-1:begin"]); const second = yield* installed({ commands: [] }, function* (transitions) { const lock = yield* acquired(); @@ -349,7 +349,7 @@ describe("a remote run's executor lifecycle", () => { it("asks the same cancellation again when its answer was lost", function* () { const commands: string[] = []; const outcome = yield* installed( - { commands, loseAnswer: new Set(["command-1"]) }, + { commands, loseAnswer: new Set(["command-1:cancel"]) }, function* () { const first = yield* WorkflowLifecycle.operations.cancel(RUN_ID); // The connection that asked is gone; the question is not. @@ -361,7 +361,7 @@ describe("a remote run's executor lifecycle", () => { expect(outcome.first.ok).toBe(false); expect(outcome.second.ok).toBe(true); // One identity, asked twice, so the owner answers with what it decided. - expect(outcome.commands).toEqual(["command-1", "command-1"]); + expect(outcome.commands).toEqual(["command-1:cancel", "command-1:cancel"]); }); it("gives a later cancellation a fresh identity once one was answered", function* () { @@ -375,7 +375,7 @@ describe("a remote run's executor lifecycle", () => { expect(outcome.first.ok).toBe(true); expect(outcome.second.ok).toBe(true); // Answered, so the claim was retired and the next call is its own. - expect(outcome.commands).toEqual(["command-1", "command-2"]); + expect(outcome.commands).toEqual(["command-1:cancel", "command-2:cancel"]); }); it("frees the acquisition when a call is cancelled before it sends", function* () { @@ -411,9 +411,11 @@ describe("a remote run's executor lifecycle", () => { it("retires an acquisition cancelled after its command went out", function* () { const asked: string[] = []; + const retired: string[] = []; const entered = withResolvers(); const script: Script = { asked, + retired, gate: { *wait(): Operation { entered.resolve(); @@ -435,5 +437,9 @@ describe("a remote run's executor lifecycle", () => { expect(outcome.ok).toBe(false); expect(asked.filter((command) => command === "begin")).toHaveLength(1); + // The connection went with the lock, before its scope ended. A socket + // still holding the run for a lock nobody may use would leave the run + // unreachable by anybody. + expect(retired).toEqual([RUN_ID]); }); }); diff --git a/packages/workflow/tests/support/remote-lifecycle-host.ts b/packages/workflow/tests/support/remote-lifecycle-host.ts index 6bad34af9..6320d0251 100644 --- a/packages/workflow/tests/support/remote-lifecycle-host.ts +++ b/packages/workflow/tests/support/remote-lifecycle-host.ts @@ -50,8 +50,10 @@ export interface Script { readonly asked?: string[]; /** Every run an acquisition was opened for. */ readonly opened?: string[]; - /** Every run an acquisition was given back for. */ + /** Every run an acquisition was given back for at scope exit. */ readonly closed?: string[]; + /** Every run whose connection was ended early, before its scope did. */ + readonly retired?: string[]; /** Answer admission with a live executor rather than a connection. */ readonly admit?: "already-running"; /** Answer a begin with one of the run's own conditions. */ @@ -70,6 +72,8 @@ export interface Script { readonly commits?: RemoteForkCommit[]; /** Every command identity the provider addressed, in order. */ readonly commands?: string[]; + /** Every command identity one staged fork part was offered under. */ + readonly parts?: string[]; /** Every retrieval value a begin carried. */ readonly retrievals?: (unknown | null)[]; /** Answer every fork commit with a definitive conflict. */ @@ -223,9 +227,10 @@ function lifecycle(script: Script): RemoteLifecycleLink { return Ok({ kind: "performed", value: record() }); }, // deno-lint-ignore require-yield - *stageForkPart(_commandId: string, part: RemoteForkPart): Operation> { + *stageForkPart(commandId: string, part: RemoteForkPart): Operation> { script.asked?.push("fork-stage"); script.staged?.push(part); + script.parts?.push(commandId); return Ok(undefined); }, // deno-lint-ignore require-yield @@ -287,7 +292,14 @@ export function installedHost(script: Script): RemoteLifecycleHost { return Ok("already-running"); } script.opened?.push(runId); - const connection: RemoteExecutorConnection = { link: link(), lifecycle: lifecycle(script) }; + const connection: RemoteExecutorConnection = { + link: link(), + lifecycle: lifecycle(script), + // deno-lint-ignore require-yield + *close(): Operation { + script.retired?.push(runId); + }, + }; yield* ensureClosed(script, runId); return Ok(connection); }, From 6635d2c13770b630a5948e97dbc9b05d377d8ab3 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Mon, 7 Sep 2026 20:48:14 -0300 Subject: [PATCH 56/81] =?UTF-8?q?=F0=9F=90=9B=20Give=20a=20second=20transf?= =?UTF-8?q?er=20its=20own=20name,=20and=20let=20a=20cancelled=20call=20kee?= =?UTF-8?q?p=20its=20question?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A destination that answers `needs-transfer` has answered. It says the run is not there and this connection offered it nothing, and it says so under the exact name it was asked by. The fork then staged the whole snapshot again and sent its final command under that same answered name, which the client refuses before it reaches an owner, because a settled correlation may not carry a second question. On the one recovery path this answer exists for, no fork could commit at all. So what follows a `needs-transfer` is a second transfer, and it carries an identity of its own: a new command name for the commit, new names for every offered part, and a new execution to begin, because the answered decision began nothing to inherit. An invocation now says which of the two phases it is in, so a retry resumes the phase it is in rather than reaching back for a name that has been answered — and it no longer asks whether the destination already holds this fork, having just been told it holds nothing. A cancelled call keeps the question it was asking. A transition registers what to retain before each command goes out and clears it when the answer comes back, so an interruption with an owner decision outstanding retains that exact identity — and the exact bytes, when a fork's final command is what was outstanding — before it retires the acquisition and closes the connection. What follows is what follows a lost answer: a replacement acquisition asks the same question under the same name and adopts the decision, or is told the run has moved past it. Nothing is retained for an interruption that mutated nothing: reading a source and offering scratch parts stay free of ambiguity, and the acquisition they were interrupted on stays usable. That distinction is now the grip's, rather than a single answered-once flag: one transition sends several commands, and a preliminary answer in the middle of a fork is not the end of the fork. The flag made it one, so a cancellation during the source read or the staging that followed left the acquisition guarded forever and a cancellation at the final commit retained nothing at all. A retained decision is re-observable on the terms the decision itself sets. An answer that carries a begun value grants execution authority, and that is returned only when the exact execution becomes this acquisition's — a ledger row that no longer names one is a disagreement, not authority. An answer that began nothing adopts nothing and invents no hold to go with it. --- .../workflow/src/cloudflare/dispatcher.ts | 28 +- .../workflow/src/cloudflare/private-schema.ts | 15 + packages/workflow/src/remote/lifecycle.ts | 584 +++++++++++------- .../cloudflare/executor-acquisition.vitest.ts | 35 ++ .../cloudflare/remote-lifecycle.vitest.ts | 68 ++ .../cloudflare/support/executor-object.ts | 22 + packages/workflow/tests/remote-fork.test.ts | 283 ++++++++- .../workflow/tests/remote-lifecycle.test.ts | 73 +++ .../tests/support/remote-lifecycle-host.ts | 105 +++- 9 files changed, 953 insertions(+), 260 deletions(-) diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index ac7b61517..899264770 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -55,6 +55,7 @@ import { import type { OwnerTransaction, OwnerTransactions } from "./owner-transaction.ts"; import { adoptExecution, + recordedExecution, COMMAND_TABLE, initializePrivateSchema, MUTATION_TABLE, @@ -540,16 +541,23 @@ export function dispatchCommand( // would mean applying it again. throw new Error("private protocol storage holds a malformed result"); } - // The answer was lost, not the fact. If this decision began an - // execution and nobody live holds it, it becomes this acquisition's. - // If the run moved past it — recovered, settled, or held by somebody - // live — the decision is history rather than authority, and handing it - // back would hand back a database nobody may settle. - // A decision that granted execution authority is only re-observable - // once that exact execution is this acquisition's. Anything else — it - // was recovered, settled, or somebody live holds it — is history, and - // returning it would hand back a database nobody may settle. - if (adoptExecution(ctx.storage, held.acquisitionId, command.id) === "stale") { + // The answer was lost, not the fact. What the answer *is* decides + // what re-observing it means, and the retained answer is the thing + // that says so: an answer carrying a begun value grants execution + // authority, and a conflict or a lifecycle refusal grants none. + if (begunExecution(command, decision) !== null) { + // Authority is only re-observable once that exact execution is this + // acquisition's. Anything else — it was recovered, settled, held by + // somebody live, or the ledger never recorded it at all — is history + // rather than authority, and returning it would hand back a database + // nobody may settle. + if (adoptExecution(ctx.storage, held.acquisitionId, command.id) !== "adopted") { + throw new CommandError("stale-journal"); + } + } else if (recordedExecution(ctx.storage, command.id) !== undefined) { + // The ledger says this decision began an execution and the decision + // itself grants none. They cannot both be right, and neither is + // authority to hand back. throw new CommandError("stale-journal"); } return decision; diff --git a/packages/workflow/src/cloudflare/private-schema.ts b/packages/workflow/src/cloudflare/private-schema.ts index 5a458aba7..112d69fa9 100644 --- a/packages/workflow/src/cloudflare/private-schema.ts +++ b/packages/workflow/src/cloudflare/private-schema.ts @@ -243,6 +243,21 @@ export function adoptExecution( return "adopted"; } +/** + * Which execution one retained decision began, as the ledger recorded it. + * + * Read without adopting anything. The mutation row and the answer it retains + * have to agree about whether a decision granted execution authority, and + * establishing that is not the same act as taking the authority. + */ +export function recordedExecution(storage: OwnerStorage, commandId: string): string | undefined { + const row = storage.sql + .exec(`SELECT execution_id FROM ${MUTATION_TABLE} WHERE command_id = ?`, commandId) + .toArray()[0]; + const recorded = row?.["execution_id"]; + return typeof recorded === "string" ? recorded : undefined; +} + /** Which execution this acquisition began, when it has begun one. */ export function heldExecution(storage: OwnerStorage, acquisitionId: string): string | undefined { const row = storage.sql diff --git a/packages/workflow/src/remote/lifecycle.ts b/packages/workflow/src/remote/lifecycle.ts index 9c35d1ddc..a43e51155 100644 --- a/packages/workflow/src/remote/lifecycle.ts +++ b/packages/workflow/src/remote/lifecycle.ts @@ -117,6 +117,18 @@ interface Ambiguous { readonly executionId: string; /** The complete canonical request this identity belongs to. */ readonly question: string; + /** + * Which phase of its call this identity belongs to. + * + * A fork can be told, definitively, that the destination holds nothing and + * was offered nothing — `needs-transfer`. That answer finishes the identity + * it was given: the owner has answered it, and the same name may never carry + * a question again. What follows is a second transfer of the same logical + * fork under an identity of its own, and this says which of the two an + * invocation is, so a retry resumes the phase it is in rather than reaching + * back for a name that has already been answered. + */ + readonly phase: "initial" | "retransfer"; /** The exact command that was sent, resent verbatim on continuation. */ readonly command: RemoteForkCommit | undefined; } @@ -173,6 +185,7 @@ export function* useRemoteLifecycle( commandId: host.ids.command(), executionId: host.ids.execution(), question, + phase: "initial", command: undefined, }; } @@ -249,10 +262,23 @@ export function* useRemoteLifecycle( } const question = questionOf(["cancel"]); const addressed = identify(runId, question); + // The same line the transitions draw, on a call that has no acquisition + // of its own to retire: if this is interrupted after the command went + // out, the question is retained so a later attempt asks that one rather + // than a new one. The connection ends with this scope either way. + let outstanding = false; + // deno-lint-ignore require-yield + yield* ensure(function* () { + if (outstanding) { + unanswered(runId, addressed); + } + }); + outstanding = true; const answered = yield* admitted.value.lifecycle.cancel( commandOf(addressed, "cancel"), runId, ); + outstanding = false; if (!answered.ok) { // The same distinction the other mutations make. A cancellation may // have committed before its answer was lost, and the next attempt has @@ -299,17 +325,34 @@ export function* useRemoteLifecycle( /** * One transition's grip on its acquisition, ended however the transition ends. * - * `sent` is the line between two very different cancellations. Before a command - * has gone out, nothing can have happened, so the guard simply lifts. After one - * has gone out and no answer came back, the owner's decision is unknown — and - * the acquisition is retired rather than freed, because a fresh mutation racing - * an unknown decision is the one thing that must not happen. Whoever wants to - * continue takes a new acquisition and asks the same question again. + * An outstanding command is the line between two very different cancellations. + * Before one has gone out, nothing can have happened, so the guard simply + * lifts. After one has gone out and no answer came back, the owner's decision + * is unknown — so the exact question is retained first, and then the + * acquisition is retired rather than freed, because a fresh mutation racing an + * unknown decision is the one thing that must not happen. Whoever wants to + * continue takes a new acquisition and asks the retained question again. + * + * A transition sends more than one command, and only the last of them is in + * flight at a time. So `sending` and `answered` bracket one command each, + * while `done` is what says the whole public transition has returned. A + * preliminary answer in the middle of a fork is not the end of the fork, and + * treating it as one is how a cancellation during the source read used to slip + * past cleanup entirely. */ interface Grip { - /** Say that a command has gone out and its answer is not yet known. */ - sending(): void; - /** Say that this transition reached an answer, whatever the answer was. */ + /** + * One command is going out now. + * + * `retain` is what to remember if this call is interrupted before the answer + * arrives: the exact identity, and the exact bytes when there are any. It + * runs only on that path, because a command that was answered is not + * ambiguous however the answer read. + */ + sending(retain: () => void): void; + /** Say that the command in flight was answered, whatever the answer was. */ + answered(): void; + /** Say that this public transition has returned. */ done(): void; } @@ -337,36 +380,65 @@ function engage(held: Hold): WorkflowRequestError | undefined { * owner answer is still outstanding — so no path can leave the guard held or * leave a sent command unaccounted for. */ -function* gripped(held: Hold): Operation { - let sent = false; +function* gripped(held: Hold, guard: boolean): Operation { + let outstanding: (() => void) | undefined; let finished = false; + const release = () => { + if (guard) { + held.busy = false; + } + }; yield* ensure(function* () { if (finished) { return; } - if (sent) { - // A command went out and nothing came back. This acquisition is retired - // rather than released: its authority ends with it, and nothing new can - // be sent on it while the first outcome is unknown. The connection goes - // with it, because a lock nobody may use whose socket still holds the - // run would leave the run unreachable by anyone at all. + const retain = outstanding; + if (retain !== undefined) { + // A command went out and nothing came back. What it was asking is + // retained first — a cancellation unwinds past the branches that would + // have retained it, and an identity nobody kept is one no replacement + // can ask under. Then this acquisition is retired rather than released: + // its authority ends with it, and nothing new can be sent on it while + // the first outcome is unknown. The connection goes with it, because a + // lock nobody may use whose socket still holds the run would leave the + // run unreachable by anyone at all. + retain(); held.live = false; yield* held.connection.close(); return; } - held.busy = false; + release(); }); return { - sending() { - sent = true; + sending(retain: () => void) { + outstanding = retain; + }, + answered() { + outstanding = undefined; }, done() { finished = true; - held.busy = false; + release(); }, }; } +/** + * Run one public transition holding this acquisition. + * + * The grip is taken before the body starts and given back when the body + * returns, however many commands the body sent on the way. A body that is + * interrupted never reaches the release, which is exactly the point: what + * happens then is the cleanup `gripped` registered, decided by whether a + * command was outstanding at the moment of the interruption. + */ +function* holding(held: Hold, guard: boolean, body: (grip: Grip) => Operation): Operation { + const grip = yield* gripped(held, guard); + const outcome = yield* body(grip); + grip.done(); + return outcome; +} + /** * Whether a failure left the owner's decision unknown. * @@ -410,53 +482,53 @@ function transitions( if (busy !== undefined) { return Err(busy); } - const grip = yield* gripped(held); - if (request.action === "resume" && request.creation !== undefined) { - grip.done(); - return Err(new WorkflowRequestError("a resume does not carry a creation.")); - } - // Minted once for this question, outside anything that could retry: the - // owner recognizes a repeat by this identity, and a fresh one would be a - // second execution. A later acquisition asking the same question finds - // the same identity and re-observes the decision. - const creation = creationOf(request.runId, request.creation); - const question = questionOf([ - "begin", - request.action, - creation === null ? null : creationShape(creation), - request.creation?.retrieval === undefined - ? null - : canonicalJson(request.creation.retrieval), - ]); - const addressed = identify(request.runId, question); - grip.sending(); - const answered = yield* held.connection.lifecycle.begin({ - commandId: commandOf(addressed, "begin"), - runId: request.runId, - action: request.action, - creation, - retrieval: request.creation?.retrieval, - executionId: addressed.executionId, - }); - grip.done(); - if (!answered.ok) { - // Which kind of failure decides whether the question survives it. An - // answer that was lost leaves the identity standing, so a replacement - // acquisition asks the same one; anything the owner actually decided - // retires it. - if (lost(answered.error)) { - unanswered(request.runId, addressed); - } else { - answeredNow(request.runId, addressed); + return yield* holding(held, true, function* (grip) { + if (request.action === "resume" && request.creation !== undefined) { + return Err(new WorkflowRequestError("a resume does not carry a creation.")); } - return answered; - } - answeredNow(request.runId, addressed); - if (answered.value.kind === "refused") { - return Err(refusalError(answered.value.refusal, request.runId)); - } - held.execution = answered.value.value.execution.executionId; - return Ok(yield* begun(held, answered.value.value)); + // Minted once for this question, outside anything that could retry: the + // owner recognizes a repeat by this identity, and a fresh one would be a + // second execution. A later acquisition asking the same question finds + // the same identity and re-observes the decision. + const creation = creationOf(request.runId, request.creation); + const question = questionOf([ + "begin", + request.action, + creation === null ? null : creationShape(creation), + request.creation?.retrieval === undefined + ? null + : canonicalJson(request.creation.retrieval), + ]); + const addressed = identify(request.runId, question); + grip.sending(() => unanswered(request.runId, addressed)); + const answered = yield* held.connection.lifecycle.begin({ + commandId: commandOf(addressed, "begin"), + runId: request.runId, + action: request.action, + creation, + retrieval: request.creation?.retrieval, + executionId: addressed.executionId, + }); + grip.answered(); + if (!answered.ok) { + // Which kind of failure decides whether the question survives it. An + // answer that was lost leaves the identity standing, so a replacement + // acquisition asks the same one; anything the owner actually decided + // retires it. + if (lost(answered.error)) { + unanswered(request.runId, addressed); + } else { + answeredNow(request.runId, addressed); + } + return answered; + } + answeredNow(request.runId, addressed); + if (answered.value.kind === "refused") { + return Err(refusalError(answered.value.refusal, request.runId)); + } + held.execution = answered.value.value.execution.executionId; + return Ok(yield* begun(held, answered.value.value)); + }); }, *settle( @@ -477,35 +549,43 @@ function transitions( ), ); } - // The root the owner is held to comes from the same connection-owned - // frontier the execution ran against, after the host has torn down. - const frontier = yield* held.connection.link.frontierSnapshot(); - const question = questionOf([ - "settle", - completion.executionId, - completion.status, - // The whole completion, stop reason included: a settlement that named - // a different reason is a different settlement. - canonicalJson(completion.reason ?? null), - frontier.workspaceRootId, - ]); - const addressed = identify(held.runId, question); - const answered = yield* held.connection.lifecycle.settle( - commandOf(addressed, "settle"), - completion, - frontier.workspaceRootId, - ); - if (!answered.ok) { - if (lost(answered.error)) { - unanswered(held.runId, addressed); - } else { - answeredNow(held.runId, addressed); + // A settlement runs on an acquisition that already holds its execution, + // so it takes no one-execution guard — but it sends a mutation, and an + // interrupted mutation is retained and retires its acquisition exactly + // the way begin's is. + return yield* holding(held, false, function* (grip) { + // The root the owner is held to comes from the same connection-owned + // frontier the execution ran against, after the host has torn down. + const frontier = yield* held.connection.link.frontierSnapshot(); + const question = questionOf([ + "settle", + completion.executionId, + completion.status, + // The whole completion, stop reason included: a settlement that named + // a different reason is a different settlement. + canonicalJson(completion.reason ?? null), + frontier.workspaceRootId, + ]); + const addressed = identify(held.runId, question); + grip.sending(() => unanswered(held.runId, addressed)); + const answered = yield* held.connection.lifecycle.settle( + commandOf(addressed, "settle"), + completion, + frontier.workspaceRootId, + ); + grip.answered(); + if (!answered.ok) { + if (lost(answered.error)) { + unanswered(held.runId, addressed); + } else { + answeredNow(held.runId, addressed); + } + return answered; } - return answered; - } - answeredNow(held.runId, addressed); - held.execution = undefined; - return Ok(answered.value.record); + answeredNow(held.runId, addressed); + held.execution = undefined; + return Ok(answered.value.record); + }); }, *fork( @@ -525,148 +605,174 @@ function transitions( if (busy !== undefined) { return Err(busy); } - const grip = yield* gripped(held); - const head = headOf(request); - const creation = creationRequest(request.runId, request.creation); - const question = questionOf([ - "fork", - request.runId, - creationShape(creation), - request.creation.retrieval === undefined ? null : canonicalJson(request.creation.retrieval), - request.selection.sourceRunId, - request.selection.checkpointEventId, - serializeDurableEvent(head.runRecord), - serializeDurableEvent(request.rootImport), - ]); - const addressed = identify(request.runId, question); - - // Before the source: a destination that already holds this fork can be - // continued from what it retains, and a decision this owner already made - // can be re-observed. Either way the source is not needed, and it may not - // be there any more. - if (addressed.command !== undefined) { - // An exact command was sent once and never answered. It is resent - // verbatim, and every outcome is classified here: nothing else is - // tried until this one's is known. - grip.sending(); - const retained = yield* held.connection.lifecycle.commitFork(addressed.command); - grip.done(); - if (!retained.ok) { - if (lost(retained.error)) { - // Ambiguous again. The claim stands, and no other command is sent. - unanswered(request.runId, addressed, addressed.command); - } else { - answeredNow(request.runId, addressed); + return yield* holding(held, true, function* (grip) { + const head = headOf(request); + const creation = creationRequest(request.runId, request.creation); + const question = questionOf([ + "fork", + request.runId, + creationShape(creation), + request.creation.retrieval === undefined + ? null + : canonicalJson(request.creation.retrieval), + request.selection.sourceRunId, + request.selection.checkpointEventId, + serializeDurableEvent(head.runRecord), + serializeDurableEvent(request.rootImport), + ]); + let addressed = identify(request.runId, question); + + // Before the source: a destination that already holds this fork can be + // continued from what it retains, and a decision this owner already made + // can be re-observed. Either way the source is not needed, and it may not + // be there any more. + const outstanding = addressed.command; + if (outstanding !== undefined) { + // An exact command was sent once and never answered. It is resent + // verbatim, and every outcome is classified here: nothing else is + // tried until this one's is known. + const resent = addressed; + grip.sending(() => unanswered(request.runId, resent, outstanding)); + const retained = yield* held.connection.lifecycle.commitFork(outstanding); + grip.answered(); + if (!retained.ok) { + if (lost(retained.error)) { + // Ambiguous again. The claim stands, and no other command is sent. + unanswered(request.runId, resent, outstanding); + } else { + answeredNow(request.runId, resent); + } + return retained; + } + if (retained.value !== "needs-transfer") { + answeredNow(request.runId, resent); + if (retained.value.kind === "refused") { + return Err(refusalError(retained.value.refusal, request.runId)); + } + held.execution = retained.value.value.execution.executionId; + return Ok(yield* begun(held, retained.value.value)); } - return retained; + // The one outcome that sends this same logical fork back to its + // source: the destination holds nothing, and the parts this command + // names went with the connection that offered them. + answeredNow(request.runId, resent); + if (resent.phase !== "initial") { + // This was already the second transfer. Offering the same snapshot a + // third time would meet the same answer. + return Err( + new WorkflowRequestError("this fork's transfer did not reach its destination."), + ); + } + // That answer finished the identity it was asked under, and a name the + // owner has answered may never carry another question. The rest of + // this call is a second transfer, under an identity of its own. + addressed = retransfer(host, question); } - if (retained.value !== "needs-transfer") { - answeredNow(request.runId, addressed); - if (retained.value.kind === "refused") { - return Err(refusalError(retained.value.refusal, request.runId)); + if (addressed.phase === "initial") { + // Whether the destination already holds this fork. A second transfer + // never asks: the destination answered that question by saying it + // holds nothing and was offered nothing. + const asking = addressed; + grip.sending(() => unanswered(request.runId, asking)); + const continued = yield* held.connection.lifecycle.continueFork({ + commandId: commandOf(asking, "continue"), + runId: request.runId, + creation, + // What this request names, which the destination proves against what + // it retained. Nothing here was read from the source. + origin: { + sourceRunId: request.selection.sourceRunId, + checkpointEventId: request.selection.checkpointEventId, + }, + runRecord: head.runRecord, + rootImport: request.rootImport, + executionId: asking.executionId, + }); + grip.answered(); + if (!continued.ok) { + if (lost(continued.error)) { + unanswered(request.runId, asking); + } else { + answeredNow(request.runId, asking); + } + return continued; + } + if (continued.value !== "absent") { + answeredNow(request.runId, asking); + if (continued.value.kind === "refused") { + return Err(refusalError(continued.value.refusal, request.runId)); + } + held.execution = continued.value.value.execution.executionId; + return Ok(yield* begun(held, continued.value.value)); } - held.execution = retained.value.value.execution.executionId; - return Ok(yield* begun(held, retained.value.value)); } - // The one outcome that sends this same logical fork back to its - // source: the destination holds nothing, and the parts this command - // names went with the connection that offered them. - } - grip.sending(); - const continued = yield* held.connection.lifecycle.continueFork({ - commandId: commandOf(addressed, "continue"), - runId: request.runId, - creation, - // What this request names, which the destination proves against what - // it retained. Nothing here was read from the source. - origin: { - sourceRunId: request.selection.sourceRunId, - checkpointEventId: request.selection.checkpointEventId, - }, - runRecord: head.runRecord, - rootImport: request.rootImport, - executionId: addressed.executionId, - }); - grip.done(); - if (!continued.ok) { - if (lost(continued.error)) { - unanswered(request.runId, addressed); - } else { + + // Nothing there. Making a fork needs the whole source, staged under this + // acquisition and committed in one transaction. + // Nothing has been mutated yet, so the guard is simply held across the + // source read and the staging: a cancellation here retains no question + // and leaves the acquisition usable. + const source = yield* readSource(host, request); + if (!source.ok) { answeredNow(request.runId, addressed); + return source; } - return continued; - } - if (continued.value !== "absent") { - answeredNow(request.runId, addressed); - if (continued.value.kind === "refused") { - return Err(refusalError(continued.value.refusal, request.runId)); + const staged = yield* offer(addressed, held.connection.lifecycle, source.value); + if (!staged.ok) { + return staged; } - held.execution = continued.value.value.execution.executionId; - return Ok(yield* begun(held, continued.value.value)); - } - - // Nothing there. Making a fork needs the whole source, staged under this - // acquisition and committed in one transaction. - // Nothing has been mutated yet, so the guard is simply held across the - // source read and the staging: a cancellation here leaves no unknown. - const source = yield* readSource(host, request); - if (!source.ok) { - answeredNow(request.runId, addressed); - return source; - } - const staged = yield* offer(addressed, held.connection.lifecycle, source.value); - if (!staged.ok) { - return staged; - } - const command: RemoteForkCommit = { - commandId: commandOf(addressed, "commit"), - runId: request.runId, - retrieval: request.creation.retrieval, - creation, - origin: { - sourceRunId: source.value.sourceRunId, - checkpointEventId: source.value.checkpointEventId, - checkpointWorkspaceRootId: source.value.checkpointWorkspaceRootId, - runRecordWorkspaceRootId: source.value.runRecordWorkspaceRootId, - rootImportWorkspaceRootId: source.value.rootImportWorkspaceRootId, - anchor: source.value.anchor, - }, - counts: { - inherited: source.value.inherited.length, - roots: source.value.roots.length, - manifests: source.value.manifests.length, - blobs: source.value.blobs.length, - checkouts: source.value.checkouts.length, - }, - runRecord: head.runRecord, - rootImport: request.rootImport, - executionId: addressed.executionId, - }; - grip.sending(); - const answered = yield* held.connection.lifecycle.commitFork(command); - grip.done(); - if (!answered.ok) { - if (lost(answered.error)) { - // Kept with the exact bytes it was sent with, so a replacement - // acquisition resends this command rather than reading the source - // again. - unanswered(request.runId, addressed, command); - } else { - answeredNow(request.runId, addressed); + const command: RemoteForkCommit = { + commandId: commandOf(addressed, "commit"), + runId: request.runId, + retrieval: request.creation.retrieval, + creation, + origin: { + sourceRunId: source.value.sourceRunId, + checkpointEventId: source.value.checkpointEventId, + checkpointWorkspaceRootId: source.value.checkpointWorkspaceRootId, + runRecordWorkspaceRootId: source.value.runRecordWorkspaceRootId, + rootImportWorkspaceRootId: source.value.rootImportWorkspaceRootId, + anchor: source.value.anchor, + }, + counts: { + inherited: source.value.inherited.length, + roots: source.value.roots.length, + manifests: source.value.manifests.length, + blobs: source.value.blobs.length, + checkouts: source.value.checkouts.length, + }, + runRecord: head.runRecord, + rootImport: request.rootImport, + executionId: addressed.executionId, + }; + const committing = addressed; + // Kept with the exact bytes it was sent with, so a replacement + // acquisition resends this command rather than reading the source again. + grip.sending(() => unanswered(request.runId, committing, command)); + const answered = yield* held.connection.lifecycle.commitFork(command); + grip.answered(); + if (!answered.ok) { + if (lost(answered.error)) { + unanswered(request.runId, committing, command); + } else { + answeredNow(request.runId, committing); + } + return answered; } - return answered; - } - answeredNow(request.runId, addressed); - if (answered.value === "needs-transfer") { - // Offered and still not there. Repeating the same transfer would meet - // the same answer. - return Err(new WorkflowRequestError("this fork's transfer did not reach its destination.")); - } - if (answered.value.kind === "refused") { - return Err(refusalError(answered.value.refusal, request.runId)); - } - held.execution = answered.value.value.execution.executionId; - return Ok(yield* begun(held, answered.value.value)); + answeredNow(request.runId, committing); + if (answered.value === "needs-transfer") { + // Offered and still not there. Repeating the same transfer would meet + // the same answer. + return Err( + new WorkflowRequestError("this fork's transfer did not reach its destination."), + ); + } + if (answered.value.kind === "refused") { + return Err(refusalError(answered.value.refusal, request.runId)); + } + held.execution = answered.value.value.execution.executionId; + return Ok(yield* begun(held, answered.value.value)); + }); }, *stageFork(request: WorkflowForkRequest): Operation> { @@ -738,6 +844,26 @@ function commandOf(invocation: Ambiguous, kind: string): string { return `${invocation.commandId}:${kind}`; } +/** + * The identity the second transfer of one fork is carried under. + * + * `needs-transfer` is an answer: the destination holds no run and was offered + * no parts, and the command that asked has been answered by name. Every + * command of what follows — the offers and the commit — is a question that + * name has never carried, so it takes a whole new identity rather than a + * variation on the answered one. The execution identity is new for the same + * reason: the answered decision began nothing, so there is nothing to inherit. + */ +function retransfer(host: RemoteLifecycleHost, question: string): Ambiguous { + return { + commandId: host.ids.command(), + executionId: host.ids.execution(), + question, + phase: "retransfer", + command: undefined, + }; +} + /** * What one invocation is asking, as text two calls can be compared by. * diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts index 86c14bfe2..23305f039 100644 --- a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -314,6 +314,41 @@ describe("holding an acquisition", () => { expect(await admitted(stub)).toBe("admitted"); }); + it("owns nothing once the runner closes its own end", async () => { + const stub = executor(); + await on(stub, (o) => o.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, claims()); + // Admitted the way a runner is really admitted: an upgrade, answered with + // the runner's own end of the socket, on the other side of the object. + const upgraded = await stub.fetch("https://executor.invalid/", { + headers: { + Upgrade: "websocket", + "x-run-id": RUN_ID, + "x-release": POLICY.release, + authorization: `Bearer ${token}`, + }, + }); + const runner = upgraded.webSocket; + expect(upgraded.status).toBe(101); + expect(runner).not.toBe(null); + runner?.accept(); + expect(await on(stub, (o) => o.holders())).toBe(1); + + // The runner closes its socket and tells the owner nothing else — all a + // client that has given up an acquisition can do. The release has to come + // from the platform delivering that close. + runner?.close(1000, "done"); + let holders = await on(stub, (o) => o.holders()); + for (let attempt = 0; holders !== 0 && attempt < 200; attempt += 1) { + await new Promise((resolve) => setTimeout(resolve, 5)); + holders = await on(stub, (o) => o.holders()); + } + + expect(holders).toBe(0); + // And the replacement admission a retired acquisition depends on succeeds. + expect(await admitted(stub)).toBe("admitted"); + }); + it("proves the acquisition before it reads a command", async () => { const stub = executor(); // Nothing is admitted, so even a well-formed command is refused for diff --git a/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts index 3dda371f5..d5ffa30b6 100644 --- a/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-lifecycle.vitest.ts @@ -399,6 +399,74 @@ describe("a run's lifecycle on its owner", () => { expect(settled["outcome"]).toBe("performed"); }); + it("refuses a retained decision its ledger row no longer says began anything", async () => { + const stub = executor(); + await connected(stub); + expect((await started(stub, "command-1", "execution-1"))["outcome"]).toBe("performed"); + + // The retained answer grants execution authority and the row beside it + // names no execution to grant. They cannot both be right. + await on(stub, (owner) => owner.forgetRecordedExecution("command-1")); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const retried = await started(stub, "command-1", "execution-1"); + + expect(retried).toEqual({ + id: "command-1", + outcome: "refused", + refusal: "command:stale-journal", + }); + // Nothing was handed back and nothing was taken: no database authority, + // and no hold invented to go with it. + expect(await on(stub, (owner) => owner.heldExecutions())).toEqual([]); + }); + + it("replays a decision that began nothing without granting anything to hold", async () => { + const stub = executor(); + await connected(stub); + await started(stub, "command-1", "execution-1"); + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + expect( + (await ask(stub, { id: "command-2", command: "cancel", runId: RUN_ID }))["outcome"], + ).toBe("performed"); + + // A begin the run's own state refuses. It is an answer the owner + // performed, and it began nothing. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const refused = await ask(stub, { + id: "command-3", + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + retrieval: null, + executionId: "execution-3", + }); + expect(refused["outcome"]).toBe("performed"); + expect( + await on(stub, (owner) => owner.mutationRow("command-3")?.["execution_id"] ?? null), + ).toBe(null); + + // Asked again after the answer was lost, it comes back unchanged — and + // still without a hold, because there is nothing to hold. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const again = await ask(stub, { + id: "command-3", + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + retrieval: null, + executionId: "execution-3", + }); + + expect(again).toEqual(refused); + expect(await on(stub, (owner) => owner.heldExecutions())).toEqual([]); + }); + it("refuses a retained decision the run has already moved past", async () => { const stub = executor(); await connected(stub); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 5b7bb57bf..12a914766 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -22,6 +22,7 @@ import { COMMAND_TABLE, FORK_TABLE, HOLD_TABLE, + MUTATION_TABLE, STAGING_TABLE, } from "../../../src/cloudflare/private-schema.ts"; import { MARKER_TABLE } from "../../../src/cloudflare/marker.ts"; @@ -349,6 +350,27 @@ export class ExecutorObject extends WorkflowOwnerObject { return this.ctx.storage.sql.exec(`SELECT * FROM ${HOLD_TABLE}`).toArray(); } + /** + * Forget which execution one retained decision began. + * + * The ledger row and the answer it retains are written together, so this is + * a store they cannot both be right about: an answer that granted execution + * authority beside a row that names no execution to grant. + */ + forgetRecordedExecution(commandId: string): void { + this.ctx.storage.sql.exec( + `UPDATE ${MUTATION_TABLE} SET execution_id = NULL WHERE command_id = ?`, + commandId, + ); + } + + /** What the mutation ledger recorded for one decision. */ + mutationRow(commandId: string): Record | undefined { + return this.ctx.storage.sql + .exec(`SELECT * FROM ${MUTATION_TABLE} WHERE command_id = ?`, commandId) + .toArray()[0]; + } + /** The watermarks retained beside this run's copied content. */ contentWatermarks(): { manifests: number[]; blobs: number[] } { const read = (table: string) => diff --git a/packages/workflow/tests/remote-fork.test.ts b/packages/workflow/tests/remote-fork.test.ts index 8096bb4d7..258f80ffa 100644 --- a/packages/workflow/tests/remote-fork.test.ts +++ b/packages/workflow/tests/remote-fork.test.ts @@ -11,7 +11,7 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { type Operation, scoped } from "effection"; +import { type Operation, scoped, spawn, withResolvers } from "effection"; import { serializeDurableEvent } from "@executablemd/durable-streams"; import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; import type { ExecutorLock } from "../src/lifecycle/api.ts"; @@ -392,12 +392,18 @@ describe("a remote fork's destination", () => { expect(outcome.sent).toEqual(["fork"]); }); - it("restages the same snapshot when the destination says it needs one", function* () { + it("restages under a second transfer's own identity when the destination needs one", function* () { const asked: string[] = []; const parts: string[] = []; + const commits: RemoteForkCommit[] = []; + const reused: string[] = []; + const decided: string[] = []; const script: Script = { asked, parts, + commits, + reused, + decided, loseAnswer: new Set(["command-1:commit"]), needsTransfer: new Set(["command-1:commit"]), committed: new Map(), @@ -424,19 +430,270 @@ describe("a remote fork's destination", () => { }); expect(outcome.first.ok).toBe(false); - expect(outcome.second.ok).toBe(true); + expect([ + outcome.second.ok, + outcome.second.ok === false && String(outcome.second.error), + ]).toEqual([true, false]); // The resend was told its transfer is not there, so the same logical fork - // staged the snapshot again and committed once. - expect(outcome.sent[0]).toBe("fork"); - // Each part of one offer is its own command, so no part meets another - // part's fingerprint at the owner. + // staged the snapshot again and committed. It never asked whether the + // destination already holds the fork: the destination just said it holds + // nothing and was offered nothing. + expect(outcome.sent).toEqual([ + "fork", + "fork-stage", + "fork-stage", + "fork-stage", + "fork-stage", + "fork-stage", + "fork-stage", + "fork", + ]); + // `needs-transfer` is an answer, so the identity it answered is finished. + // Every command of the second transfer carries a name that has never been + // answered — which is exactly what the link enforces. + expect(commits.map((commit) => commit.commandId)).toEqual([ + "command-1:commit", + "command-1:commit", + "command-2:commit", + ]); expect(new Set(outcome.offered).size).toBe(outcome.offered.length); - // And the parts are named by the call and their place in it, so the same - // logical fork restages under exactly the identities it used before: a - // part the owner already holds is recognized as that part, not as a new - // one. - expect(outcome.reoffered).toEqual(outcome.offered); + expect(new Set(outcome.reoffered).size).toBe(outcome.reoffered.length); + for (const offered of outcome.reoffered) { + expect(outcome.offered).not.toContain(offered); + } + expect(reused).toEqual([]); + // And exactly one destination execution was ever begun. + expect(decided).toHaveLength(1); + expect(outcome.second.ok && outcome.second.value.execution.executionId).toBe(decided[0]); + }); + + it("keeps the fork it was committing when it is cancelled, and commits it once", function* () { + const asked: string[] = []; + const commits: RemoteForkCommit[] = []; + const decided: string[] = []; + const retired: string[] = []; + const reused: string[] = []; + const sourced: string[] = []; + const entered = withResolvers(); + let held = 0; + const script: Script = { + asked, + commits, + decided, + retired, + reused, + sourced, + committed: new Map(), + source: source(), + commitGate: { + *wait(): Operation { + held += 1; + if (held > 1) { + return; + } + entered.resolve(); + yield* withResolvers().operation; + }, + }, + }; + const outcome = yield* installed(script, function* (transitions) { + yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + const sent = yield* spawn(() => transitions.fork(lock, request())); + yield* entered.operation; + // Interrupted with the destination's decision made and its answer in + // flight — the one moment a fork is genuinely ambiguous. + yield* sent.halt(); + }); + const asking = asked.length; + const reads = sourced.length; + const second = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + return { second, sent: asked.slice(asking), reads: sourced.length - reads }; + }); + + // The interrupted acquisition gave up its connection rather than holding a + // run nobody could reach. + expect(retired).toEqual([DESTINATION]); + // The replacement resent that exact command and nothing else, without + // reading the source again. + expect(outcome.sent).toEqual(["fork"]); + expect(outcome.reads).toBe(0); + expect(commits.map((commit) => commit.commandId)).toEqual([ + "command-1:commit", + "command-1:commit", + ]); + // One fork was committed, and the replacement was handed that one. + expect(decided).toHaveLength(1); + expect([ + outcome.second.ok, + outcome.second.ok === false && String(outcome.second.error), + ]).toEqual([true, false]); + expect(outcome.second.ok && outcome.second.value.execution.executionId).toBe(decided[0]); + expect(reused).toEqual([]); + }); + + it("keeps no question when it is cancelled reading the source, and stays usable", function* () { + const asked: string[] = []; + const commands: string[] = []; + const retired: string[] = []; + const reused: string[] = []; + const entered = withResolvers(); + let held = 0; + const script: Script = { + asked, + commands, + retired, + reused, + committed: new Map(), + source: source(), + sourceGate: { + *wait(): Operation { + held += 1; + if (held > 1) { + return; + } + entered.resolve(); + yield* withResolvers().operation; + }, + }, + }; + const outcome = yield* installed(script, function* (transitions) { + return yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + const sent = yield* spawn(() => transitions.fork(lock, request())); + yield* entered.operation; + // The continuation was answered and nothing has been mutated. Reading + // a source is not a mutation however it is interrupted. + yield* sent.halt(); + const asking = asked.length; + // The same acquisition, which was released rather than retired. + const again = yield* transitions.fork(lock, request()); + return { again, sent: asked.slice(asking) }; + }); + }); + + // Nothing was retired, because nothing was outstanding. + expect(retired).toEqual([]); + expect([outcome.again.ok, outcome.again.ok === false && String(outcome.again.error)]).toEqual([ + true, + false, + ]); + // The second call is a new question under a new identity — no ambiguity + // was retained for a mutation that never happened — and it asks the + // destination the whole thing again. + expect(outcome.sent[0]).toBe("fork-continue"); + expect(commands).toEqual(["command-1:continue", "command-2:continue", "command-2:commit"]); + expect(reused).toEqual([]); + }); + + it("keeps no question when it is cancelled offering the snapshot", function* () { + const asked: string[] = []; + const commands: string[] = []; + const parts: string[] = []; + const retired: string[] = []; + const reused: string[] = []; + const entered = withResolvers(); + let held = 0; + const script: Script = { + asked, + commands, + parts, + retired, + reused, + committed: new Map(), + source: source(), + stageGate: { + *wait(): Operation { + held += 1; + if (held > 1) { + return; + } + entered.resolve(); + yield* withResolvers().operation; + }, + }, + }; + const outcome = yield* installed(script, function* (transitions) { + return yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + const sent = yield* spawn(() => transitions.fork(lock, request())); + yield* entered.operation; + // Offered parts are scratch until a final command claims them. An + // offer interrupted halfway has mutated nothing. + yield* sent.halt(); + const asking = asked.length; + const again = yield* transitions.fork(lock, request()); + return { again, sent: asked.slice(asking) }; + }); + }); + + expect(retired).toEqual([]); + expect([outcome.again.ok, outcome.again.ok === false && String(outcome.again.error)]).toEqual([ + true, + false, + ]); + // A whole second offer, under a second identity: no part and no commit + // reaches for a name the first attempt already used. expect(outcome.sent.filter((command) => command === "fork-stage")).toHaveLength(6); - expect(outcome.sent.at(-1)).toBe("fork"); + expect(commands.at(-1)).toBe("command-2:commit"); + expect(new Set(parts).size).toBe(parts.length); + expect(reused).toEqual([]); + }); + + it("resends only the second transfer's own command when its answer is lost", function* () { + const asked: string[] = []; + const sourced: string[] = []; + const reused: string[] = []; + const decided: string[] = []; + const commits: RemoteForkCommit[] = []; + const script: Script = { + asked, + sourced, + reused, + decided, + commits, + // The first transfer's commit is lost and never happened; the second + // transfer's commit is lost after the owner made it. + loseAnswer: new Set(["command-1:commit", "command-2:commit"]), + needsTransfer: new Set(["command-1:commit"]), + committed: new Map(), + source: source(), + }; + const outcome = yield* installed(script, function* (transitions) { + const attempts = []; + for (let attempt = 0; attempt < 3; attempt += 1) { + const asking = asked.length; + const reads = sourced.length; + const attempted = yield* scoped(function* () { + const lock = yield* acquired(DESTINATION); + return yield* transitions.fork(lock, request()); + }); + attempts.push({ + outcome: attempted, + sent: asked.slice(asking), + reads: sourced.length - reads, + }); + } + return attempts; + }); + + expect(outcome.map((attempt) => attempt.outcome.ok)).toEqual([false, false, true]); + // The third attempt resent one command — the second transfer's own final + // command, verbatim — and read no source to build it. + expect(outcome[2]?.sent).toEqual(["fork"]); + expect(outcome[2]?.reads).toBe(0); + expect(commits.map((commit) => commit.commandId)).toEqual([ + "command-1:commit", + "command-1:commit", + "command-2:commit", + "command-2:commit", + ]); + // A retry reaches for the identity the second transfer already used rather + // than for the answered one, and never for a fresh one. + expect(reused).toEqual([]); + expect(decided).toHaveLength(1); }); }); diff --git a/packages/workflow/tests/remote-lifecycle.test.ts b/packages/workflow/tests/remote-lifecycle.test.ts index 7445321e1..0325f1d3c 100644 --- a/packages/workflow/tests/remote-lifecycle.test.ts +++ b/packages/workflow/tests/remote-lifecycle.test.ts @@ -409,6 +409,79 @@ describe("a remote run's executor lifecycle", () => { expect(outcome.asked).toEqual(["begin"]); }); + it("keeps the question a cancelled begin was asking, and lets a replacement finish it", function* () { + const asked: string[] = []; + const commands: string[] = []; + const decided: string[] = []; + const retired: string[] = []; + const reused: string[] = []; + const entered = withResolvers(); + let held = 0; + const script: Script = { + asked, + commands, + decided, + retired, + reused, + committed: new Map(), + gate: { + *wait(): Operation { + held += 1; + if (held > 1) { + // Only the first answer is caught in flight. The replacement's is + // delivered, which is the whole point of taking one. + return; + } + entered.resolve(); + yield* withResolvers().operation; + }, + }, + }; + const outcome = yield* installed(script, function* (transitions) { + yield* scoped(function* () { + const lock = yield* acquired(); + const sent = yield* spawn(() => + transitions.begin(lock, { runId: RUN_ID, action: "resume" }), + ); + yield* entered.operation; + // Interrupted with the owner's decision made and its answer in flight. + yield* sent.halt(); + }); + // A replacement acquisition, asking the same question. + return yield* scoped(function* () { + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + if (!begun.ok) { + return { begun, settled: undefined, again: undefined }; + } + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + // The same execution cannot be settled twice through this lock. + const again = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + return { begun, settled, again }; + }); + }); + + // The interrupted acquisition gave up its connection, and the replacement + // asked under the exact identity the interrupted call was asking under. + expect(retired).toEqual([RUN_ID]); + expect(commands.slice(0, 2)).toEqual(["command-1:begin", "command-1:begin"]); + expect(outcome.begun.ok).toBe(true); + // One execution was ever begun, and the replacement adopted that one. + expect(decided).toEqual(["execution-1"]); + expect(outcome.begun.ok && outcome.begun.value.execution.executionId).toBe("execution-1"); + // It settles exactly once through the lock that adopted it. + expect(outcome.settled?.ok).toBe(true); + expect(outcome.again?.ok).toBe(false); + expect(asked.filter((command) => command === "settle")).toHaveLength(1); + expect(reused).toEqual([]); + }); + it("retires an acquisition cancelled after its command went out", function* () { const asked: string[] = []; const retired: string[] = []; diff --git a/packages/workflow/tests/support/remote-lifecycle-host.ts b/packages/workflow/tests/support/remote-lifecycle-host.ts index 6320d0251..0b279cc46 100644 --- a/packages/workflow/tests/support/remote-lifecycle-host.ts +++ b/packages/workflow/tests/support/remote-lifecycle-host.ts @@ -80,8 +80,22 @@ export interface Script { readonly forkRefuses?: boolean; /** Answer the first fork commit by saying its transfer is not here. */ readonly needsTransfer?: Set; - /** Held open until a test releases it, so a call can be caught in flight. */ + /** + * Held open until a test releases it, so a begin can be caught in flight. + * + * The owner has already decided by the time this is reached: what is caught + * is the answer on its way back, which is the only moment an interrupted + * mutation is genuinely ambiguous. + */ readonly gate?: { wait(): Operation }; + /** The same, for a fork's final mutation: decided, and answer in flight. */ + readonly commitGate?: { wait(): Operation }; + /** The same, for reading the source — where nothing has been mutated yet. */ + readonly sourceGate?: { wait(): Operation }; + /** The same, for offering a staged part. */ + readonly stageGate?: { wait(): Operation }; + /** Every execution this owner actually began, as opposed to re-answered. */ + readonly decided?: string[]; /** Every run whose source was resolved. */ readonly sourced?: string[]; /** Whether a destination already holds this fork, so no source is needed. */ @@ -90,6 +104,15 @@ export interface Script { readonly loseAnswer?: Set; /** What an owner decided for a command identity, once it has decided. */ readonly committed?: Map; + /** + * Every identity this host was asked to carry twice on one connection. + * + * The real client refuses that before the owner sees it, so a provider that + * reuses an answered name gets a channel failure instead of a decision. This + * host holds the same rule, and records every violation so a test can say + * that none happened rather than only that the call came out right. + */ + readonly reused?: string[]; } export function record(): WorkflowRunRecord { @@ -180,19 +203,36 @@ function link(): RemoteWorkspaceLink { function lifecycle(script: Script): RemoteLifecycleLink { let minted = 0; + // One connection's own correlation ids. The real client keeps exactly this + // set — the ids it is waiting on and the ids it has already settled — and + // refuses to send either again, so an identity that has been answered can + // never carry another question on this connection. + const spoken = new Set(); + function reused(commandId: string): Result | undefined { + if (!spoken.has(commandId)) { + spoken.add(commandId); + return undefined; + } + script.reused?.push(commandId); + // What `OwnerConnection.ask()` raises for a duplicate id, as the adapter + // translates it: the command never reaches an owner, and the caller is + // told only that the channel could not carry it. + return Err(new WorkflowTransactionError("this run's owner could not be reached.")); + } return { // deno-lint-ignore require-yield *begin(request: RemoteBeginCommand): Operation>> { + const again = reused>(request.commandId); + if (again !== undefined) { + return again; + } script.asked?.push("begin"); script.commands?.push(request.commandId); - if (script.gate !== undefined) { - // Caught in flight: the call has been sent and has not been answered. - yield* script.gate.wait(); - } script.retrievals?.push(request.retrieval ?? null); if (script.loseAnswer?.has(request.commandId) === true) { // The owner committed and the answer never arrived. script.loseAnswer.delete(request.commandId); + script.decided?.push(request.executionId); script.committed?.set(request.commandId, begun(request.executionId, script)); return Err(new WorkflowTransactionError("the connection ended before it answered.")); } @@ -205,19 +245,36 @@ function lifecycle(script: Script): RemoteLifecycleLink { return Ok({ kind: "refused", refusal: script.begin }); } minted += 1; - return Ok({ kind: "performed", value: begun(request.executionId, script) }); + const decided = begun(request.executionId, script); + script.decided?.push(request.executionId); + // Decided, and retained under the identity it was asked by, before the + // answer starts back. A gate here catches the one moment that matters: + // the owner has committed and the caller does not know it yet. + script.committed?.set(request.commandId, decided); + if (script.gate !== undefined) { + yield* script.gate.wait(); + } + return Ok({ kind: "performed", value: decided }); }, // deno-lint-ignore require-yield *settle( commandId: string, _completion: DocumentExecutionCompletion, ): Operation> { + const again = reused(commandId); + if (again !== undefined) { + return again; + } script.asked?.push("settle"); script.commands?.push(commandId); return Ok(frontier()); }, // deno-lint-ignore require-yield *cancel(commandId: string): Operation>> { + const again = reused>(commandId); + if (again !== undefined) { + return again; + } script.asked?.push("cancel"); script.commands?.push(commandId); if (script.loseAnswer?.has(commandId) === true) { @@ -226,11 +283,17 @@ function lifecycle(script: Script): RemoteLifecycleLink { } return Ok({ kind: "performed", value: record() }); }, - // deno-lint-ignore require-yield *stageForkPart(commandId: string, part: RemoteForkPart): Operation> { + const again = reused(commandId); + if (again !== undefined) { + return again; + } script.asked?.push("fork-stage"); script.staged?.push(part); script.parts?.push(commandId); + if (script.stageGate !== undefined) { + yield* script.stageGate.wait(); + } return Ok(undefined); }, // deno-lint-ignore require-yield @@ -238,6 +301,10 @@ function lifecycle(script: Script): RemoteLifecycleLink { *continueFork( continuation: RemoteForkContinuation, ): Operation | "absent">> { + const again = reused | "absent">(continuation.commandId); + if (again !== undefined) { + return again; + } script.asked?.push("fork-continue"); script.commands?.push(continuation.commandId); const held = script.continues; @@ -251,6 +318,10 @@ function lifecycle(script: Script): RemoteLifecycleLink { *commitFork( commit: RemoteForkCommit, ): Operation | "needs-transfer">> { + const again = reused | "needs-transfer">(commit.commandId); + if (again !== undefined) { + return again; + } script.asked?.push("fork"); script.commits?.push(commit); if (script.forkConflict !== undefined) { @@ -258,10 +329,26 @@ function lifecycle(script: Script): RemoteLifecycleLink { } void minted; script.commands?.push(commit.commandId); + if (script.loseAnswer?.has(commit.commandId) !== true) { + // Everything below either re-answers a decision this owner already + // made or makes one now. A gate belongs after that, for the same + // reason a begin's does. + const already = script.committed?.get(commit.commandId); + const answering = + script.needsTransfer?.has(commit.commandId) === true || script.forkRefuses === true; + if (already === undefined && !answering) { + script.decided?.push(commit.executionId); + script.committed?.set(commit.commandId, begun(commit.executionId, script)); + } + if (script.commitGate !== undefined) { + yield* script.commitGate.wait(); + } + } if (script.loseAnswer?.has(commit.commandId) === true) { script.loseAnswer.delete(commit.commandId); // The mutation committed unless this scenario says it never did. if (script.needsTransfer?.has(commit.commandId) !== true) { + script.decided?.push(commit.executionId); script.committed?.set(commit.commandId, begun(commit.executionId, script)); } return Err(new WorkflowTransactionError("the connection ended before it answered.")); @@ -320,8 +407,10 @@ export function installedHost(script: Script): RemoteLifecycleHost { *history(): Operation> { throw new WorkflowRequestError("this scripted plane answers no history"); }, - // deno-lint-ignore require-yield *forkSource(): Operation> { + if (script.sourceGate !== undefined) { + yield* script.sourceGate.wait(); + } return Ok(held); }, }); From 79b83f8bc18278e4c7826b8e484859bc622ad815 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Tue, 8 Sep 2026 11:49:28 -0300 Subject: [PATCH 57/81] =?UTF-8?q?=E2=9C=A8=20Answer=20a=20run=20somebody?= =?UTF-8?q?=20else=20owns,=20and=20spend=20the=20answer=20once?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run that is waiting has no executor, and handing it a value must not start one. So delivery is a third plane on the owner beside the executor socket and the read plane: it accepts no socket, mints no acquisition, moves no lifecycle, and can be answered while another executor is live. What separates it from the read plane is that it writes exactly one row — the value, correlated to the wait it answers — and nothing else. No journal event, no execution, no status, no root, no mapping. The value is judged before it is retained, and it is judged on the runner. A wait retained a response schema, and judging against it means compiling that schema with the secret scanner beside it; both are the document runtime's, and a run's owner is not a document runtime. So the owner says what the run is waiting at, the runner judges the offered value against exactly that, and the owner writes only after reading the same facts again inside the transaction that writes. The fingerprint is what makes that safe: the runner recomputes it from the description it was given and refuses an owner that names another, and the owner refuses a retention whose fingerprint is not what its retained request spells now. A value judged against one request can never be retained against a different one. Spending it is one commit. The execution that reaches the wait — the one standing at it, which is not something a row can decide — reads what the owner retained through its own acquisition, publishes the answer into the transaction it is already inside, and enlists the consumption beside it. The owner receives one proposal carrying both: it reads the row it retained, requires it pending and delivered against this request, requires exactly one appended event to be that wait's answer carrying exactly the value it holds, and then spends the row and appends the events. A runner cannot publish one value and spend the row for another, cannot spend without publishing, and cannot publish without spending; any disagreement refuses the whole commit, which leaves the answer pending and appends nothing. The value reaches the document only after that commit returns. An offered publication proves nothing — the transaction it was offered inside may not have committed — so what ends the wait is the owner having committed, and a later execution replays the recorded event rather than reading delivery state again. Three pieces moved to make this possible without a host reaching into another. The wait's two effect names and its identity now live in a leaf module, and the fingerprint is composed from the canonicalization and the SHA-256 this package already carries rather than from `node:crypto`, so a Durable Object can recognize a request and an answer without loading a document runtime. And the check for whether an execution is standing at its own wait is shared, because every host that ends a wait asks exactly that question and none of them may answer it differently. --- packages/workflow/src/cloudflare/client.ts | 123 ++++ packages/workflow/src/cloudflare/commands.ts | 90 ++- .../src/cloudflare/delivery-client.ts | 214 ++++++ .../workflow/src/cloudflare/delivery-plane.ts | 202 ++++++ .../workflow/src/cloudflare/dispatcher.ts | 23 +- .../workflow/src/cloudflare/owner-answers.ts | 332 +++++++++ packages/workflow/src/cloudflare/owner.ts | 38 + packages/workflow/src/cloudflare/publish.ts | 9 + packages/workflow/src/deno/delivery.ts | 98 +-- packages/workflow/src/deno/suspension.ts | 110 +-- packages/workflow/src/remote/answer-link.ts | 72 ++ packages/workflow/src/remote/answers.ts | 138 ++++ packages/workflow/src/remote/collector.ts | 70 +- packages/workflow/src/remote/database.ts | 47 +- packages/workflow/src/remote/delivery.ts | 302 ++++++++ packages/workflow/src/remote/workspace.ts | 9 +- packages/workflow/src/suspension/answer.ts | 5 +- packages/workflow/src/suspension/api.ts | 6 +- packages/workflow/src/suspension/delivery.ts | 99 ++- packages/workflow/src/suspension/effects.ts | 35 + .../workflow/src/suspension/fingerprint.ts | 18 + packages/workflow/src/suspension/position.ts | 110 +++ packages/workflow/src/suspension/suspend.ts | 26 +- .../cloudflare/executor-acquisition.vitest.ts | 1 + .../cloudflare/remote-delivery.vitest.ts | 660 ++++++++++++++++++ .../tests/cloudflare/remote-publish.vitest.ts | 1 + .../tests/cloudflare/settle-parser.vitest.ts | 1 + .../cloudflare/support/executor-object.ts | 20 + .../workflow/tests/remote-database.test.ts | 7 + .../workflow/tests/remote-delivery.test.ts | 560 +++++++++++++++ .../workflow/tests/remote-publication.test.ts | 4 + .../tests/support/remote-lifecycle-host.ts | 7 + 32 files changed, 3182 insertions(+), 255 deletions(-) create mode 100644 packages/workflow/src/cloudflare/delivery-client.ts create mode 100644 packages/workflow/src/cloudflare/delivery-plane.ts create mode 100644 packages/workflow/src/cloudflare/owner-answers.ts create mode 100644 packages/workflow/src/remote/answer-link.ts create mode 100644 packages/workflow/src/remote/answers.ts create mode 100644 packages/workflow/src/remote/delivery.ts create mode 100644 packages/workflow/src/suspension/effects.ts create mode 100644 packages/workflow/src/suspension/fingerprint.ts create mode 100644 packages/workflow/src/suspension/position.ts create mode 100644 packages/workflow/tests/cloudflare/remote-delivery.vitest.ts create mode 100644 packages/workflow/tests/remote-delivery.test.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 8261c8f48..2b1491ac5 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -66,6 +66,7 @@ import { MAX_CONTENT_BYTES, } from "./commands.ts"; import type { RemoteRunLink, RemoteWorkspaceLink } from "../remote/database.ts"; +import type { RemoteRetainedAnswer } from "../remote/answer-link.ts"; import type { CreateWorkflowRunRequest } from "../storage/api.ts"; import { WorkflowRunConflictError, @@ -74,6 +75,7 @@ import { } from "../storage/errors.ts"; import { isSchemaVersion, SCHEMA_VERSION } from "../sqlite/workflow-schema.ts"; import { canonicalJson } from "../storage/record.ts"; +import { parseJsonValue } from "../storage/members.ts"; import { WorkflowDatabaseCorruptError, WorkflowDatabaseFormatError, @@ -108,6 +110,9 @@ export type PrivateRefusal = | "command:not-forkable" | "command:wrong-execution" | "command:needs-transfer" + | "command:not-suspended" + | "command:wrong-suspension" + | "command:answer-unavailable" | "storage:foreign" | `storage:unsupported-version-v${number}` | "storage:corrupt"; @@ -190,6 +195,9 @@ export function privateRefusal(value: string): PrivateRefusal { case "command:not-forkable": case "command:wrong-execution": case "command:needs-transfer": + case "command:not-suspended": + case "command:wrong-suspension": + case "command:answer-unavailable": case "storage:foreign": case "storage:corrupt": return value; @@ -632,6 +640,17 @@ function commitRequest(intent: CommitIntent): Record { // Exactly what the serializer produces, in the order the transaction // appended them. The owner parses each one and requires these same bytes. events: intent.events.map((event) => serializeDurableEvent(event)), + // The retained answer this proposal spends, when it spends one. It names + // the wait and carries no value: the owner holds the value, and checks the + // event above against it before it spends anything. + answer: + intent.answer === null + ? null + : { + suspensionId: intent.answer.suspensionId, + requestEventId: intent.answer.requestEventId, + requestFingerprint: intent.answer.requestFingerprint, + }, }; } @@ -743,6 +762,72 @@ function parseConflictFields(value: unknown): readonly string[] { return Object.freeze(fields); } +/** + * One retained answer, as this build reads an owner's account of it. + * + * The value arrives as the canonical text the owner retained and is parsed + * here: what a later commit spends is compared against those bytes, so a value + * this build could not read back the same way is not one it may publish. + */ +function parseRetainedAnswer( + value: unknown, + suspensionId: string, +): RemoteRetainedAnswer | undefined { + if (value === null) { + return undefined; + } + const found = members(value, [ + "suspensionId", + "requestEventId", + "requestFingerprint", + "answer", + "state", + ]); + const named = found.get("suspensionId"); + if (named !== suspensionId) { + return fail("a retained answer named a different wait"); + } + const state = found.get("state"); + if (state !== "pending" && state !== "consumed") { + return fail("a retained answer named a state this build does not read"); + } + const encoded = found.get("answer"); + if (typeof encoded !== "string" || encoded === "") { + return fail("a retained answer carried no value"); + } + let decoded: unknown; + try { + decoded = JSON.parse(encoded); + } catch { + return fail("a retained answer carried a value this build cannot read"); + } + const answer = parseJsonValue( + decoded, + "$", + () => new RemoteRecordError("a retained answer carried a value this build cannot read"), + ); + if (canonicalJson(answer) !== encoded) { + return fail("a retained answer was not canonically encoded"); + } + return Object.freeze({ + suspensionId, + requestEventId: text(found.get("requestEventId"), "a retained answer named no request event"), + requestFingerprint: text( + found.get("requestFingerprint"), + "a retained answer named no request fingerprint", + ), + answer, + state, + }); +} + +function text(value: unknown, reason: string): string { + if (typeof value !== "string" || value === "") { + return fail(reason); + } + return value; +} + export function cloudflareRunLink( connection: OwnerConnection, nextId: () => string, @@ -874,6 +959,28 @@ export function cloudflareRunLink( } }, + /** + * What this run retains for one wait, on this acquisition's authority. + * + * Answered as the owner retains it, canonical text and all, so the value a + * caller publishes is the value the owner will compare its commit against. + */ + *pendingAnswer(suspensionId: string): Operation> { + try { + const answered = yield* connection.ask( + nextId(), + { command: "answer", suspensionId }, + (value) => parseRetainedAnswer(value, suspensionId), + privateRefusal, + ); + return answered.outcome === "refused" + ? Err(storageFailure(privateRefusal(answered.refusal))) + : Ok(answered.value); + } catch (error) { + return Err(translate(error)); + } + }, + *readExecutions(): Operation> { try { const found: DocumentExecutionRecord[] = []; @@ -1046,6 +1153,22 @@ export function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { if (refusal === "command:capacity") { return new WorkflowRequestError("this run's owner cannot accept more work on this connection."); } + if (refusal === "command:not-suspended") { + return new WorkflowRequestError( + "this workflow run is not waiting for an answer, and only a suspended run is.", + ); + } + if (refusal === "command:wrong-suspension") { + return new WorkflowRequestError( + "this workflow run is not waiting at that suspension. A run waits at one at a time.", + ); + } + if (refusal === "command:answer-unavailable") { + // Nothing retained, already published, or delivered against a different + // request. All three are facts about the wait rather than failures to + // reach the run. + return new WorkflowRequestError("there is no delivered answer this wait may be ended with."); + } return new WorkflowTransactionError("this run's owner refused the operation."); } diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index 20b783ee7..a5049cdd7 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -73,7 +73,8 @@ export type CommandName = | "settle" | "fork-stage" | "fork" - | "fork-continue"; + | "fork-continue" + | "answer"; export type CommandRefusal = | "not-an-object" @@ -120,7 +121,30 @@ export type CommandRefusal = * destination holds nothing, and the parts it names are not here — so the * caller's next move is to copy the source again rather than to give up. */ - | "needs-transfer"; + | "needs-transfer" + /** + * A value was offered to a run that is not waiting for one. + * + * The run is intact and this owner holds it; it is running, finished, + * cancelled, or stopped for something other than a durable wait. A caller + * acts on that rather than retrying. + */ + | "not-suspended" + /** + * A value was offered to a wait this run is not standing at. + * + * A run waits at one suspension at a time. The identifier names another one, + * or names a request published elsewhere in this run's history. + */ + | "wrong-suspension" + /** + * There is no retained answer this commit may spend. + * + * Nothing was delivered, or it was already published, or it was delivered + * against a different request, or the event this commit appends is not the + * one that answer would become. The commit is refused whole. + */ + | "answer-unavailable"; export class CommandError extends Error { override name = "CommandError"; @@ -190,6 +214,35 @@ export interface CommitCommand extends CommandEnvelope { readonly mappings: readonly ProposedMapping[]; /** Exactly what `serializeDurableEvent` produced, terminating newline included. */ readonly events: readonly string[]; + /** + * The retained answer this proposal spends, when it spends one. + * + * `null` for every ordinary commit. It names a wait, the event its request + * was published as and the fingerprint it was delivered against, and carries + * no value: the owner holds the value already, and one arriving here would be + * the runner saying what it is owed. + */ + readonly answer: ProposedAnswerConsumption | null; +} + +/** Which retained answer one proposal spends. */ +export interface ProposedAnswerConsumption { + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; +} + +/** + * What one run retains for a wait, asked for by the acquisition that may spend + * it. + * + * A read rather than a mutation, and on the executor plane rather than the + * delivery plane, because it is read in order to be published: only the + * executor publishes. + */ +export interface AnswerCommand extends CommandEnvelope { + readonly command: "answer"; + readonly suspensionId: string; } /** The Workspace half of a proposal, when there is one. */ @@ -403,7 +456,8 @@ export type RunnerCommand = | SettleCommand | ForkStageCommand | ForkCommand - | ForkContinueCommand; + | ForkContinueCommand + | AnswerCommand; export type CommandResult = | { readonly id: string; readonly outcome: "performed"; readonly value: unknown } @@ -425,7 +479,9 @@ const MEMBERS: Record = { "publication", "mappings", "events", + "answer", ], + answer: [...ENVELOPE, "suspensionId"], retrieval: [...ENVELOPE, "expectedWorkspaceRootId", "metadata"], executions: [...ENVELOPE, "anchor", "after"], mappings: ENVELOPE, @@ -589,7 +645,8 @@ export function parseCommand(raw: string): RunnerCommand { command !== "settle" && command !== "fork-stage" && command !== "fork" && - command !== "fork-continue" + command !== "fork-continue" && + command !== "answer" ) { throw new CommandError("unknown-command"); } @@ -651,6 +708,9 @@ export function parseCommand(raw: string): RunnerCommand { if (command === "mappings") { return { id, command }; } + if (command === "answer") { + return { id, command, suspensionId: text(members, "suspensionId", MAX_ID) }; + } if (command === "open") { const runId = text(members, "runId", MAX_RUN_ID); const offered = members.get("creation"); @@ -789,6 +849,28 @@ export function parseCommand(raw: string): RunnerCommand { publication: publication(members.get("publication")), mappings: mappings(members.get("mappings")), events: eventRecords(members.get("events")), + answer: answerConsumption(members.get("answer")), + }; +} + +/** + * The retained answer a proposal spends, or its absence. + * + * `null` is every ordinary commit. Everything else names one wait, the exact + * journal event its request was published as, and the fingerprint the value was + * delivered against — and nothing else, because a value here would be a runner + * telling the owner what it retained. + */ +function answerConsumption(value: unknown): ProposedAnswerConsumption | null { + if (value === null) { + return null; + } + const members = object(value); + closed(members, ["suspensionId", "requestEventId", "requestFingerprint"]); + return { + suspensionId: text(members, "suspensionId", MAX_ID), + requestEventId: text(members, "requestEventId", MAX_ID), + requestFingerprint: digest(members, "requestFingerprint"), }; } diff --git a/packages/workflow/src/cloudflare/delivery-client.ts b/packages/workflow/src/cloudflare/delivery-client.ts new file mode 100644 index 000000000..12d199bed --- /dev/null +++ b/packages/workflow/src/cloudflare/delivery-client.ts @@ -0,0 +1,214 @@ +/** + * Answering a run on a Cloudflare owner, from wherever the value came from. + * + * The transport is narrow on purpose: one request out, one response back, and + * it knows nothing about runs, waits or authority. A host wires it to an + * ordinary HTTPS request; a test wires it to the object directly. Delivery + * never opens a socket, so there is no connection here to hold and nothing that + * could be mistaken for an acquisition. + * + * Every answer is parsed before it is believed. A wait this build cannot read + * back the way it was described is not a wait to judge a value against, and a + * retention naming a different run or wait is an owner disagreeing with the + * question rather than an accepted delivery. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import type { Json } from "@executablemd/durable-streams"; +import type { + RemoteAnswerRetained, + RemoteAnswerRetention, + RemoteDeliveryLink, + RemoteRetainedWaitRecord, +} from "../remote/answer-link.ts"; +import { RemoteRecordError } from "../remote/records.ts"; +import { parseJsonValue, parseMembers, requireMemberNames } from "../storage/members.ts"; +import { WorkflowRunNotFoundError } from "../storage/errors.ts"; +import { canonicalJson } from "../storage/record.ts"; +import { privateRefusal, storageFailure } from "./client.ts"; +import { DELIVERY_REQUEST_BYTES } from "./delivery-plane.ts"; + +/** + * One request out, one response back. + * + * The admission travels beside the body rather than inside it, because the + * owner decides on the release before it decodes anything. + */ +export interface DeliveryTransport { + send(admission: DeliveryAdmission, body: string): Operation; +} + +/** What a request carries outside its body. */ +export interface DeliveryAdmission { + readonly release: string; + readonly token: string; + readonly runId: string; +} + +/** How a host supplies one delivery's admission. */ +export interface DeliveryAdmissionSource { + /** The build this deployment agreed to talk to. */ + readonly release: string; + /** A short-lived token, minted per delivery rather than retained. */ + token(runId: string): Operation; +} + +/** + * The most serialized bytes one delivery answer may carry. + * + * A `wait` answer carries a retained request and its response schema, which are + * journal values, so what bounds it is what bounds a request carrying one. + */ +const ANSWER_BYTES = DELIVERY_REQUEST_BYTES + 4096; + +function fail(reason: string): never { + throw new RemoteRecordError(`the owner returned a malformed delivery answer: ${reason}`); +} + +/** Reach one Cloudflare owner's delivery plane. */ +export function cloudflareDeliveryLink( + transport: DeliveryTransport, + admission: DeliveryAdmissionSource, +): RemoteDeliveryLink { + function* ask(runId: string, body: Record): Operation> { + const encoded = JSON.stringify(body); + if (new TextEncoder().encode(encoded).length > DELIVERY_REQUEST_BYTES) { + return Err(storageFailure("command:too-large")); + } + let raw: string; + try { + const token = yield* admission.token(runId); + raw = yield* transport.send({ release: admission.release, token, runId }, encoded); + } catch { + // Whatever the transport raised, the owner was not reached and nothing + // was decided. What went wrong underneath is the host's to log; a public + // error carrying it would carry an endpoint or a token with it. + return Err(storageFailure("command:unavailable")); + } + if (new TextEncoder().encode(raw).length > ANSWER_BYTES) { + return Err(storageFailure("command:too-large")); + } + let decoded: unknown; + try { + decoded = JSON.parse(raw); + } catch { + return Err(storageFailure("command:malformed-member")); + } + const answered = parseMembers(decoded, "$", (reason) => new RemoteRecordError(reason)); + const outcome = answered.get("outcome"); + if (outcome === "refused") { + const refusal = answered.get("refusal"); + if (typeof refusal !== "string") { + return Err(storageFailure("command:malformed-member")); + } + const named = privateRefusal(refusal); + // The one category that is a fact about the run rather than a failure. + if (named === "command:absent") { + return Err(new WorkflowRunNotFoundError(runId)); + } + return Err(storageFailure(named)); + } + if (outcome !== "performed") { + return Err(storageFailure("command:malformed-member")); + } + return Ok(answered.get("value")); + } + + return { + *wait(runId: string, suspensionId: string): Operation> { + const answered = yield* ask(runId, { operation: "wait", suspensionId }); + if (!answered.ok) { + return answered; + } + try { + return Ok(parseWait(answered.value)); + } catch (error) { + return Err( + error instanceof RemoteRecordError ? error : storageFailure("command:malformed-member"), + ); + } + }, + + *retain(retention: RemoteAnswerRetention): Operation> { + const answered = yield* ask(retention.runId, { + operation: "retain", + suspensionId: retention.suspensionId, + requestEventId: retention.requestEventId, + requestFingerprint: retention.requestFingerprint, + // Canonically encoded here, once, so the bytes the owner retains are + // the bytes a later commit is compared against. + answer: canonicalJson(retention.answer), + }); + if (!answered.ok) { + return answered; + } + try { + const retained = parseRetained(answered.value); + if ( + retained.runId !== retention.runId || + retained.suspensionId !== retention.suspensionId + ) { + fail("a retention named a different run or wait"); + } + return Ok(retained); + } catch (error) { + return Err( + error instanceof RemoteRecordError ? error : storageFailure("command:malformed-member"), + ); + } + }, + }; +} + +function parseWait(value: unknown): RemoteRetainedWaitRecord { + const found = parseMembers(value, "$", (reason) => new RemoteRecordError(reason)); + requireMemberNames( + found, + ["runId", "suspensionId", "requestEventId", "request", "responseSchema", "requestFingerprint"], + "$", + (reason) => new RemoteRecordError(reason), + ); + return Object.freeze({ + runId: text(found.get("runId"), "a wait named no run"), + suspensionId: text(found.get("suspensionId"), "a wait named no suspension"), + requestEventId: text(found.get("requestEventId"), "a wait named no request event"), + request: json(found.get("request"), "a wait carried a request this build cannot read"), + responseSchema: json( + found.get("responseSchema"), + "a wait carried a response schema this build cannot read", + ), + requestFingerprint: digest(found.get("requestFingerprint")), + }); +} + +function parseRetained(value: unknown): RemoteAnswerRetained { + const found = parseMembers(value, "$", (reason) => new RemoteRecordError(reason)); + requireMemberNames( + found, + ["runId", "suspensionId"], + "$", + (reason) => new RemoteRecordError(reason), + ); + return Object.freeze({ + runId: text(found.get("runId"), "a retention named no run"), + suspensionId: text(found.get("suspensionId"), "a retention named no wait"), + }); +} + +function text(value: unknown, reason: string): string { + if (typeof value !== "string" || value === "") { + return fail(reason); + } + return value; +} + +function digest(value: unknown): string { + if (typeof value !== "string" || !/^[0-9a-f]{64}$/.test(value)) { + return fail("a wait named no request fingerprint"); + } + return value; +} + +function json(value: unknown, reason: string): Json { + return parseJsonValue(value, "$", () => new RemoteRecordError(reason)); +} diff --git a/packages/workflow/src/cloudflare/delivery-plane.ts b/packages/workflow/src/cloudflare/delivery-plane.ts new file mode 100644 index 000000000..191911d24 --- /dev/null +++ b/packages/workflow/src/cloudflare/delivery-plane.ts @@ -0,0 +1,202 @@ +/** + * Answering a run's owner without taking the run. + * + * A third plane, and it is a third plane for the same reason the read plane is + * a second one: what it does cannot be done over the executor socket. A run + * that is waiting has no executor, and it must be answerable while another + * executor is live — so this accepts no socket, mints no acquisition, and + * cannot move a lifecycle. What separates it from the read plane is that it + * writes exactly one row, in one transaction, and nothing else. + * + * It writes no journal event, no execution row, no status, no root, no mapping + * and no acquisition state. What a delivery leaves behind is a pending answer + * correlated to the wait it answers, which the next acquired execution to reach + * that wait spends. + * + * What crosses is closed and private to this release. The value is judged + * against the wait's response schema on the runner, where the schema compiler + * lives; what this decides is identity — which run, which wait, which request — + * and it decides it again inside the transaction that writes. + */ + +import { parseMembers, requireMemberNames } from "../storage/members.ts"; +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; +import { canonicalJson } from "../storage/record.ts"; +import { CommandError } from "./commands.ts"; +import { READ_PAGE_BYTES, READ_REQUEST_ENVELOPE } from "./read-plane.ts"; +import { fingerprintOf, readRetainedWait, retainAnswer } from "./owner-answers.ts"; +import type { OwnerStorage } from "./storage.ts"; + +/** What a delivery answered, or why it would not. */ +export type DeliveryAnswer = + | { readonly outcome: "performed"; readonly value: unknown } + | { readonly outcome: "refused"; readonly refusal: string }; + +/** The most characters a public run id may carry. */ +const MAX_RUN_ID = 128; + +/** The most characters one wait's identifier may carry. */ +const MAX_SUSPENSION_ID = 256; + +/** The most characters one request fingerprint may carry. */ +const FINGERPRINT = /^[0-9a-f]{64}$/; + +/** + * The most serialized bytes one delivery request may carry. + * + * Derived rather than picked. A retained answer is a value this owner will hand + * back through the read plane once it is an event, so what bounds it is what + * one page of that plane may carry; the envelope around it — a wait, an event + * id, a fingerprint and the punctuation between them — is the same fixed + * envelope a read request carries. + */ +export const DELIVERY_REQUEST_BYTES = READ_PAGE_BYTES + READ_REQUEST_ENVELOPE; + +/** What a caller may ask this plane for. */ +export type DeliveryOperation = + | { readonly operation: "wait"; readonly suspensionId: string } + | { + readonly operation: "retain"; + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; + /** Canonical JSON, already encoded by the runner that judged it. */ + readonly answer: string; + }; + +function failure(reason: string, path: string): Error { + return new WorkflowRecordMalformedError("workflow delivery request", `${reason} at ${path}`); +} + +/** + * The whole request, parsed as a closed shape before any member is read. + * + * Nothing here reaches storage. A request that is not one of the two shapes + * this plane implements is refused as malformed, before a run is recognized, + * before a wait is read, and before anything could be written. + */ +export function parseDeliveryOperation(raw: string): DeliveryOperation { + if (new TextEncoder().encode(raw).length > DELIVERY_REQUEST_BYTES) { + throw failure("expected a bounded request", "$"); + } + let decoded: unknown; + try { + decoded = JSON.parse(raw); + } catch { + throw failure("expected one JSON object", "$"); + } + const read = parseMembers(decoded, "$", failure); + const operation = read.get("operation"); + + if (operation === "wait") { + requireMemberNames(read, ["operation", "suspensionId"], "$", failure); + return { operation, suspensionId: identifier(read.get("suspensionId"), "$.suspensionId") }; + } + if (operation === "retain") { + requireMemberNames( + read, + ["operation", "suspensionId", "requestEventId", "requestFingerprint", "answer"], + "$", + failure, + ); + const fingerprint = read.get("requestFingerprint"); + if (typeof fingerprint !== "string" || !FINGERPRINT.test(fingerprint)) { + throw failure("expected a sha-256 digest", "$.requestFingerprint"); + } + const answer = read.get("answer"); + if (typeof answer !== "string" || answer === "") { + throw failure("expected the canonical encoding of one value", "$.answer"); + } + // Held to the same rules the journal holds a value to, and to the exact + // canonical spelling: two encodings of one value would be two answers, and + // the row is compared as text when a lost response is delivered again. + let value: unknown; + try { + value = JSON.parse(answer); + } catch { + throw failure("expected the canonical encoding of one value", "$.answer"); + } + if (canonicalJson(readValue(value)) !== answer) { + throw failure("expected the canonical encoding of one value", "$.answer"); + } + return { + operation, + suspensionId: identifier(read.get("suspensionId"), "$.suspensionId"), + requestEventId: identifier(read.get("requestEventId"), "$.requestEventId"), + requestFingerprint: fingerprint, + answer, + }; + } + throw failure("expected an operation this owner implements", "$.operation"); +} + +/** + * Answer one delivery request. + * + * `wait` reads and writes nothing. `retain` runs inside the caller's own owner + * transaction, which is where every fact it depends on is read again. + */ +export function answerDelivery( + storage: OwnerStorage, + runId: string, + request: DeliveryOperation, + now: string, +): Record { + if (request.operation === "wait") { + const waiting = readRetainedWait(storage, runId, request.suspensionId); + return { + runId: waiting.runId, + suspensionId: waiting.suspensionId, + requestEventId: waiting.requestEventId, + request: waiting.request, + responseSchema: waiting.responseSchema, + requestFingerprint: fingerprintOf(waiting), + }; + } + return retainAnswer( + storage, + runId, + { + suspensionId: request.suspensionId, + requestEventId: request.requestEventId, + requestFingerprint: request.requestFingerprint, + answer: request.answer, + }, + now, + ); +} + +function identifier(value: unknown, path: string): string { + if (typeof value !== "string" || value === "") { + throw failure("expected a non-empty identifier", path); + } + const limit = path === "$.suspensionId" ? MAX_SUSPENSION_ID : MAX_RUN_ID; + if (value.length > limit) { + throw failure("expected a bounded identifier", path); + } + return value; +} + +/** One offered value, held to the JSON rules a retained value is held to. */ +function readValue(value: unknown): Parameters[0] { + if (value === null || typeof value === "string" || typeof value === "boolean") { + return value; + } + if (typeof value === "number") { + if (!Number.isFinite(value)) { + throw new CommandError("malformed-member"); + } + return value; + } + if (Array.isArray(value)) { + return value.map((entry) => readValue(entry)); + } + if (typeof value === "object") { + const held: Record[0]> = {}; + for (const [name, member] of Object.entries(value)) { + held[name] = readValue(member); + } + return held; + } + throw new CommandError("malformed-member"); +} diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 899264770..347c9662a 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -44,6 +44,7 @@ import { type RunnerCommand, } from "./commands.ts"; import { bytesOf, decodeBase64, sha256Hex } from "./encoding.ts"; +import { readRetainedAnswer } from "./owner-answers.ts"; import { readContent, readExecutions, @@ -345,7 +346,27 @@ function perform( return { id: command.id, outcome: "performed", - value: applyCommit(ctx.storage, acquisitionId, command, mintEventId), + value: applyCommit(ctx.storage, acquisitionId, command, mintEventId, ownerTime()), + }; + } + if (command.command === "answer") { + const retained = readRetainedAnswer(ctx.storage, command.suspensionId); + return { + id: command.id, + outcome: "performed", + // The value travels as the canonical text this owner retained, so a + // runner comparing what it publishes with what was delivered compares the + // same bytes this owner will. + value: + retained === undefined + ? null + : { + suspensionId: retained.suspensionId, + requestEventId: retained.requestEventId, + requestFingerprint: retained.requestFingerprint, + answer: retained.answer, + state: retained.state, + }, }; } if (command.command === "retrieval") { diff --git a/packages/workflow/src/cloudflare/owner-answers.ts b/packages/workflow/src/cloudflare/owner-answers.ts new file mode 100644 index 000000000..d58f7a62f --- /dev/null +++ b/packages/workflow/src/cloudflare/owner-answers.ts @@ -0,0 +1,332 @@ +/** + * What one owner retains for the answers delivered to its durable waits. + * + * A retained answer is a row rather than a journal event, for the same reason + * it is a row on a local host: the value arrives while nothing is running, and + * it becomes history only when an execution reaches the wait it answers and + * publishes it. Until then this is the whole of what the run holds. + * + * The row is written by the delivery plane, which takes no acquisition, and + * spent by a commit, which requires the exact one. Both are decided here rather + * than believed: the wait is read from the run's own account of why it stopped, + * and the consumption is checked against the event the commit is appending. + */ + +import { parseDurableEvent } from "@executablemd/durable-streams"; +import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import { readRunRecord, type Row } from "../sqlite/rows.ts"; +import { WorkflowRecordMalformedError } from "../storage/errors.ts"; +import { parseJsonValue } from "../storage/members.ts"; +import { canonicalJson } from "../storage/record.ts"; +import { SUSPENSION_ANSWER, SUSPENSION_REQUEST } from "../suspension/effects.ts"; +import { CommandError } from "./commands.ts"; +import { sha256Hex } from "./encoding.ts"; +import { declaredObjects, holdsNoRun, isPristine, recognizeObject } from "./recognition.ts"; +import { retainedText } from "./retained.ts"; +import type { OwnerStorage } from "./storage.ts"; + +/** + * What one wait retains, as this owner reads it. + * + * The request and its response schema travel as the retained description held + * them — unparsed here, because judging a value against a schema is the + * document runtime's work and this is not the document runtime. What this owner + * decides is identity: which run, which wait, which event, and the fingerprint + * a later retention is held to. + */ +export interface OwnerRetainedWait { + readonly runId: string; + readonly suspensionId: string; + readonly requestEventId: string; + readonly request: unknown; + readonly responseSchema: unknown; +} + +/** One answer this owner retains, whatever state it is in. */ +export interface OwnerRetainedAnswer { + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; + readonly answer: string; + readonly state: "pending" | "consumed"; +} + +/** + * The wait this run is standing at, or a refusal naming why it is not at one. + * + * Recognition first, so a store that is not this build's run refuses as itself. + * Then the run's own account of why it stopped: a status of `suspended` whose + * stop reason names a retained request event, and that event being the wait + * being asked about. Anything else — a completed run, a cancelled one, a run + * stopped for another reason, another wait's request — is a run this value does + * not answer. + */ +export function readRetainedWait( + storage: OwnerStorage, + runId: string, + suspensionId: string, +): OwnerRetainedWait { + const record = readRunRecord(retainedRun(storage, runId)); + if (record.status !== "suspended") { + throw new CommandError("not-suspended"); + } + const reason = record.stopReason; + if (reason === undefined || reason.kind !== "journal") { + throw new CommandError("not-suspended"); + } + const row = storage.sql + .exec("SELECT event_id, record FROM journal_events WHERE event_id = ?", reason.eventId) + .toArray()[0]; + if (row === undefined) { + throw new CommandError("corrupt-journal"); + } + const event = readEvent(row); + if ( + event.type !== "yield" || + event.description.type !== SUSPENSION_REQUEST || + event.description.name !== suspensionId + ) { + throw new CommandError("wrong-suspension"); + } + return { + runId: record.runId, + suspensionId, + requestEventId: retainedText(row, "event_id"), + request: event.description.request, + responseSchema: event.description.responseSchema, + }; +} + +/** What this run retains for one wait, if it retains anything. */ +export function readRetainedAnswer( + storage: OwnerStorage, + suspensionId: string, +): OwnerRetainedAnswer | undefined { + const row = storage.sql + .exec( + `SELECT suspension_id, request_event_id, request_fingerprint, answer, state + FROM workflow_suspension_answers WHERE suspension_id = ?`, + suspensionId, + ) + .toArray()[0]; + return row === undefined ? undefined : parseRetainedAnswer(row); +} + +/** + * Retain one delivered answer, inside the caller's open transaction. + * + * Every fact the value was judged against is read again here, under the write + * this transaction is about to make: a run that moved on — resumed, cancelled, + * answered by somebody else, or waiting at a request that changed — refuses, + * and the transaction it refuses inside wrote nothing. + * + * A compatible repeat is not a second write. The same value, against the same + * wait, the same request event and the same fingerprint, re-observes the row + * that is already there; anything else about it disagreeing is a conflict. + */ +export function retainAnswer( + storage: OwnerStorage, + runId: string, + offered: { + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; + readonly answer: string; + }, + now: string, +): { readonly runId: string; readonly suspensionId: string } { + const waiting = readRetainedWait(storage, runId, offered.suspensionId); + if (waiting.requestEventId !== offered.requestEventId) { + // The value was judged against a request published somewhere else in this + // run's history, so it is not an answer to the wait this run is at. + throw new CommandError("wrong-suspension"); + } + if (fingerprintOf(waiting) !== offered.requestFingerprint) { + throw new CommandError("stale-journal"); + } + + const already = readRetainedAnswer(storage, offered.suspensionId); + if (already !== undefined) { + if ( + already.state === "pending" && + already.requestEventId === offered.requestEventId && + already.requestFingerprint === offered.requestFingerprint && + already.answer === offered.answer + ) { + // The same delivery again, after its answer was lost. One decision, one + // row, and nothing written a second time. + return { runId: waiting.runId, suspensionId: offered.suspensionId }; + } + throw new CommandError("duplicate-conflict"); + } + + storage.sql.exec( + `INSERT INTO workflow_suspension_answers + (suspension_id, request_event_id, request_fingerprint, answer, state, created_at) + VALUES (?, ?, ?, ?, 'pending', ?)`, + offered.suspensionId, + offered.requestEventId, + offered.requestFingerprint, + offered.answer, + now, + ); + return { runId: waiting.runId, suspensionId: offered.suspensionId }; +} + +/** + * Spend one retained answer, inside the commit that publishes it. + * + * The events this commit appends are what decides it. A consumption is admitted + * only when exactly one of them is the answer Yield for this wait carrying + * exactly the value this owner retained — so a runner cannot publish one value + * and spend the row for another, and cannot spend a row without publishing at + * all. A row that is gone, already spent, or delivered against a different + * request refuses, and the whole commit goes with it. + */ +export function consumeRetainedAnswer( + storage: OwnerStorage, + consumption: { + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; + }, + events: readonly string[], + now: string, +): void { + const spending = readRetainedAnswer(storage, consumption.suspensionId); + if (spending === undefined || spending.state !== "pending") { + throw new CommandError("answer-unavailable"); + } + if ( + spending.requestEventId !== consumption.requestEventId || + spending.requestFingerprint !== consumption.requestFingerprint + ) { + throw new CommandError("answer-unavailable"); + } + + const published = events.filter((record) => + isAnswerFor(record, consumption.suspensionId, spending.answer), + ); + if (published.length !== 1) { + throw new CommandError("answer-unavailable"); + } + + storage.sql.exec( + `UPDATE workflow_suspension_answers SET state = 'consumed', consumed_at = ? + WHERE suspension_id = ? AND state = 'pending'`, + now, + consumption.suspensionId, + ); + // Read back rather than counted: what matters is that the row this commit + // spends is spent, and a statement's own report of how many rows it touched + // is not the same statement as the one that says what the row now is. + if (readRetainedAnswer(storage, consumption.suspensionId)?.state !== "consumed") { + throw new CommandError("answer-unavailable"); + } +} + +/** + * Whether one appended record is this wait's answer, carrying this value. + * + * Parsed rather than matched as text: what a serialization spells is not what + * it means, and the value is compared canonically so two encodings of one JSON + * value are one answer. + */ +function isAnswerFor(record: string, suspensionId: string, answer: string): boolean { + const parsed = parseDurableEvent(record); + if (!parsed.ok) { + return false; + } + const event = parsed.value; + if ( + event.type !== "yield" || + event.description.type !== SUSPENSION_ANSWER || + event.description.name !== suspensionId + ) { + return false; + } + if (event.result === undefined || event.result.status !== "ok") { + return false; + } + try { + return canonicalJson(retained(event.result.value)) === answer; + } catch { + return false; + } +} + +/** + * One retained value, held to the JSON rules storage holds every value to. + * + * The retained description and a published result are journal data: what they + * hold is whatever was written, and canonicalizing something that is not JSON + * would name a value nothing could store. + */ +function retained(value: unknown): Json { + return parseJsonValue(value, "$", () => new CommandError("malformed-member")); +} + +/** The run this owner holds, or a refusal that it holds another or none. */ +function retainedRun(storage: OwnerStorage, runId: string): Row { + if (isPristine(declaredObjects(storage)) || holdsNoRun(storage)) { + // Nothing is stored here. Delivery names a run to answer, and answering a + // run that does not exist is a fact about the run rather than damage — and + // recognizing an empty store would report it as somebody else's. + throw new CommandError("absent"); + } + recognizeObject(storage); + const row = storage.sql + .exec( + `SELECT run_id, definition, base, props, status, stop_reason_kind, stop_reason_code, + stop_reason_event_id, created_at, updated_at FROM workflow_run`, + ) + .toArray()[0]; + if (row === undefined) { + throw new CommandError("absent"); + } + if (readRunRecord(row).runId !== runId) { + throw new CommandError("wrong-run"); + } + return row; +} + +function readEvent(row: Row): DurableEvent { + const parsed = parseDurableEvent(retainedText(row, "record")); + if (!parsed.ok) { + throw new CommandError("corrupt-journal"); + } + return parsed.value; +} + +function parseRetainedAnswer(row: Row): OwnerRetainedAnswer { + const state = retainedText(row, "state"); + if (state !== "pending" && state !== "consumed") { + throw new WorkflowRecordMalformedError( + "workflow_suspension_answers.state", + "expected pending or consumed", + ); + } + return { + suspensionId: retainedText(row, "suspension_id"), + requestEventId: retainedText(row, "request_event_id"), + requestFingerprint: retainedText(row, "request_fingerprint"), + answer: retainedText(row, "answer"), + state, + }; +} + +/** + * The fingerprint of the request this wait retained. + * + * Computed from the retained description exactly as the shared contract + * computes it, so an owner and a runner that read the same wait derive the same + * name for it. + */ +export function fingerprintOf(waiting: OwnerRetainedWait): string { + return sha256Hex( + canonicalJson({ + request: retained(waiting.request), + responseSchema: retained(waiting.responseSchema), + }), + ); +} diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index b17902e26..92f84d801 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -40,6 +40,7 @@ import { type ReadAdmission, type ReadAnswer, } from "./read-plane.ts"; +import { answerDelivery, type DeliveryAnswer, parseDeliveryOperation } from "./delivery-plane.ts"; import { dispatchCommand } from "./dispatcher.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { discardPriorAcquisitions, PRIVATE_OBJECT_NAMES } from "./private-schema.ts"; @@ -206,6 +207,43 @@ export abstract class WorkflowOwnerObject extends DurableObject { } } + /** + * Answer one typed delivery, taking nothing. + * + * The same order the other two planes use — the build is compared before any + * token work, and the token is verified before the run is named — and then it + * writes exactly one row inside one transaction. No socket is accepted and no + * acquisition is minted or compared, so a run with a live executor can still + * be answered and a refusal at any step leaves the run untouched. + */ + *deliver(admission: ReadAdmission, body: string): Operation { + const { policy, verification } = this.configuration(); + try { + requireSameRelease(policy.release, admission.release); + yield* admitToken(policy, verification, admission.token); + const runId = admitRunId(admission.runId); + const request = parseDeliveryOperation(body); + const now = new Date().toISOString(); + if (request.operation === "wait") { + return { + outcome: "performed", + value: answerDelivery(this.owned, runId, request, now), + }; + } + // One transaction, entered here rather than inside the answer, so every + // fact the retention depends on is read under the write it is about to + // make and a refusal rolls back having written nothing. + return { + outcome: "performed", + value: this.transactions.run(this.owned, () => + answerDelivery(this.owned, runId, request, now), + ), + }; + } catch (error) { + return { outcome: "refused", refusal: refusalOf(error) }; + } + } + /** * Handle one message from an admitted connection. * diff --git a/packages/workflow/src/cloudflare/publish.ts b/packages/workflow/src/cloudflare/publish.ts index c8704718d..9844105cc 100644 --- a/packages/workflow/src/cloudflare/publish.ts +++ b/packages/workflow/src/cloudflare/publish.ts @@ -51,6 +51,7 @@ import { MAX_CONTENT_BYTES } from "./commands.ts"; import { sha256Hex } from "../workspace/sha256.ts"; import { CommandError, type CommitCommand, type ProposedMapping } from "./commands.ts"; import { validateRetainedRoot } from "./owner-reads.ts"; +import { consumeRetainedAnswer } from "./owner-answers.ts"; import { readRetrieval } from "../sqlite/rows.ts"; import { bytesOf } from "./encoding.ts"; import { STAGING_TABLE } from "./private-schema.ts"; @@ -220,6 +221,7 @@ export function applyCommit( acquisitionId: string, command: CommitCommand, mintEventId: () => string, + at: string, ): CommitValue { const now = frontier(storage); if (now.rootId !== command.expectedWorkspaceRootId) { @@ -252,6 +254,13 @@ export function applyCommit( applyMapping(storage, mapping); } + // Before the events are written, and inside the same transaction that writes + // them: the retained answer this proposal spends has to be spendable, and the + // event it is spent for has to be the one this proposal is appending. + if (command.answer !== null) { + consumeRetainedAnswer(storage, command.answer, command.events, at); + } + const journalEventIds: string[] = []; for (const record of command.events) { const eventId = mintEventId(); diff --git a/packages/workflow/src/deno/delivery.ts b/packages/workflow/src/deno/delivery.ts index 6c3e345b9..70f640ad5 100644 --- a/packages/workflow/src/deno/delivery.ts +++ b/packages/workflow/src/deno/delivery.ts @@ -41,6 +41,7 @@ import { } from "../suspension/api.ts"; import { SUSPENSION_ANSWER } from "../suspension/answer.ts"; import { + parseAnswerDelivery, type WorkflowAnswerDelivery, WorkflowAnswerDeliveryError, type WorkflowAnswerRetention, @@ -48,7 +49,6 @@ import { } from "../suspension/delivery.ts"; import { SUSPENSION_REQUEST } from "../suspension/suspend.ts"; import { - WorkflowRequestError, WorkflowRunIdMismatchError, WorkflowRunNotFoundError, WorkflowStorageError, @@ -61,7 +61,7 @@ import { readRunRow } from "./database.ts"; import { useHostConnections } from "./host-connections.ts"; import { readJournalEntries } from "./journal.ts"; import { workflowRunPath } from "./path.ts"; -import { authorizedRoot, checkRunId } from "./provider.ts"; +import { authorizedRoot } from "./provider.ts"; import { readTransaction } from "./reading.ts"; import { translateSqliteError, verifySchema } from "./schema.ts"; @@ -92,14 +92,6 @@ export function* installWorkflowInputDelivery( ); } -/** A delivery whose every member has been checked rather than believed. */ -interface CheckedDelivery { - readonly runId: string; - readonly suspensionId: string; - readonly value: Json; - readonly secretDetection: boolean; -} - /** The wait a run is at, read from what it retained. */ interface RetainedWait { readonly record: WorkflowRunRecord; @@ -113,7 +105,7 @@ function* deliverAnswer( connections: WorkflowRunConnections, request: WorkflowAnswerDelivery, ): Operation> { - const checked = checkDelivery(request); + const checked = parseAnswerDelivery(request); if (!checked.ok) { return checked; } @@ -429,90 +421,6 @@ function rollback(database: DatabaseSync): void { } } -const DELIVERY_MEMBERS = ["runId", "suspensionId", "value", "secretDetection"]; - -/** - * The whole request, parsed as a closed shape before any member is read. - * - * The type describes what a caller meant; what arrives is whatever the language - * allows. A suspension id is opaque and every character of it is part of it, so - * the only thing asked of it is that it is a non-empty string this run could - * have derived. - */ -function checkDelivery(offered: WorkflowAnswerDelivery): Result { - if (typeof offered !== "object" || offered === null || Array.isArray(offered)) { - return Err(new WorkflowRequestError("a delivery takes an object describing one answer.")); - } - const names = new Set(Object.keys(offered)); - const missing = DELIVERY_MEMBERS.filter((name) => !names.has(name)); - if (missing.length > 0) { - return Err(new WorkflowRequestError(`the delivery is missing ${missing.join(", ")}.`)); - } - - const runId = checkRunId(Reflect.get(offered, "runId")); - if (!runId.ok) { - return runId; - } - - const suspensionId = Reflect.get(offered, "suspensionId"); - if (typeof suspensionId !== "string" || suspensionId === "") { - return Err( - new WorkflowRequestError( - "a delivery names the wait it answers, and a suspension id is a non-empty string.", - ), - ); - } - - const secretDetection = Reflect.get(offered, "secretDetection"); - if (typeof secretDetection !== "boolean") { - return Err( - new WorkflowRequestError("a delivery says whether it crosses the secret gate, as a boolean."), - ); - } - - const value = retainableJson(Reflect.get(offered, "value")); - if (value === undefined) { - return Err( - new WorkflowRequestError( - "an answer is retained in this run's storage, so it must be JSON this run can store.", - ), - ); - } - - return Ok({ runId: runId.value, suspensionId, value, secretDetection }); -} - -/** The value, if every part of it is JSON this run can retain. */ -function retainableJson(value: unknown): Json | undefined { - let encoded: string | undefined; - try { - encoded = JSON.stringify(value); - } catch { - return undefined; - } - if (encoded === undefined) { - return undefined; - } - const parsed: unknown = JSON.parse(encoded); - return isJson(parsed) ? parsed : undefined; -} - -function isJson(value: unknown): value is Json { - if (value === null || typeof value === "string" || typeof value === "number") { - return true; - } - if (typeof value === "boolean") { - return true; - } - if (Array.isArray(value)) { - return value.every(isJson); - } - if (typeof value === "object") { - return Object.values(value).every(isJson); - } - return false; -} - /** * Report a storage refusal as itself, and let anything else propagate. * diff --git a/packages/workflow/src/deno/suspension.ts b/packages/workflow/src/deno/suspension.ts index a7b32c751..8b8cc5a69 100644 --- a/packages/workflow/src/deno/suspension.ts +++ b/packages/workflow/src/deno/suspension.ts @@ -41,32 +41,19 @@ * not. */ +import { call, type Operation, race, scoped, spawn, suspend, withResolvers } from "effection"; import { - call, - ensure, - type Operation, - race, - scoped, - spawn, - suspend, - withResolvers, -} from "effection"; -import { canonicalFingerprint } from "@executablemd/core"; -import type { EffectDescription } from "@executablemd/durable-streams"; -import { - parseSuspensionRequest, suspensionRequestFingerprint, WorkflowSuspension, type WorkflowSuspensionRequest, } from "../suspension/api.ts"; -import { durablePosition } from "@executablemd/durable-streams"; import type { Json } from "@executablemd/durable-streams"; -import { SUSPENSION_REQUEST, suspensionId } from "../suspension/suspend.ts"; import { type SuspensionAnswerAuthority, type SuspensionAnswerProvider, useSuspensionAnswerProvider, } from "../suspension/answer.ts"; +import { atOwnRequest } from "../suspension/position.ts"; import type { WorkflowRunDatabase } from "../storage/api.ts"; import { WorkflowRequestError, WorkflowTransactionError } from "../storage/errors.ts"; import { consumeRetainedAnswer, readRetainedAnswer } from "./answers.ts"; @@ -110,99 +97,6 @@ export interface SuspensionController { entered(error: unknown): boolean; } -/** - * Whether this execution is, right now, at the wait it says it is. - * - * Authority is the *current* execution reaching its own request, not the - * existence of a matching row. Retained history alone cannot decide this: on a - * resume the request from the previous execution is already in the journal, so - * a caller that ran before replay reached it could present its identifier and be - * believed. What separates the real wait from that is where the execution is. - * - * `suspendFor()` publishes its request and then enters, so by the time it gets - * here the coroutine has settled exactly one more durable yield than it had when - * the request was made — the request's own. The identifier is therefore the one - * this run derives for the position immediately behind this one, and a caller - * standing anywhere else derives a different identifier and is refused. - * - * The journal is then read to confirm that the yield at that exact position is - * this request, describing what is being presented. That is publication - * evidence, and it is checked at one position rather than searched for. - */ -function* atOwnRequest( - database: WorkflowRunDatabase, - suspension: string, - request: WorkflowSuspensionRequest, -): Operation { - const position = yield* durablePosition(); - if (position.index === 0) { - return NOT_AT_A_WAIT; - } - const published = { - coroutineId: position.coroutineId, - index: position.index - 1, - }; - if (suspensionId(database.record.runId, published) !== suspension) { - return NOT_AT_A_WAIT; - } - - const entries = yield* database.readJournalEntries(); - if (!entries.ok) { - return NOT_AT_A_WAIT; - } - - const counts = new Map(); - let found: EffectDescription | undefined; - for (const entry of entries.value) { - if (entry.event.type !== "yield") { - continue; - } - const coroutineId = entry.event.coroutineId; - const index = counts.get(coroutineId) ?? 0; - counts.set(coroutineId, index + 1); - if (coroutineId === published.coroutineId && index === published.index) { - found = entry.event.description; - } - } - if (found === undefined || found.type !== SUSPENSION_REQUEST || found.name !== suspension) { - return NOT_AT_A_WAIT; - } - // Parsed, not merely read. A retained description is journal data, and this - // one is reached through a public durable operation any document can publish, - // so what it holds is a claim about a request rather than a request. Comparing - // raw fields would let a row that could never have come from `suspendFor()` — - // a `responseSchema` that is an array, say — admit a wait whose schema nothing - // could later validate an answer against. - let retained: WorkflowSuspensionRequest; - try { - retained = parseSuspensionRequest({ - request: found.request, - responseSchema: found.responseSchema, - }); - } catch (error) { - return ( - "the request retained at this position is not one a durable wait can be entered " + - `for: ${error instanceof Error ? error.message : String(error)}` - ); - } - - const same = - canonicalFingerprint({ - request: request.request, - responseSchema: request.responseSchema, - }) === - canonicalFingerprint({ - request: retained.request, - responseSchema: retained.responseSchema, - }); - return same ? undefined : NOT_AT_A_WAIT; -} - -const NOT_AT_A_WAIT = - "this execution is not at that durable wait. A wait is entered by the execution that has " + - "just published its request, at the position that request was made — not by presenting an " + - "identifier a run retains somewhere else."; - export function createSuspensionController( options: SuspensionControllerOptions, ): SuspensionController { diff --git a/packages/workflow/src/remote/answer-link.ts b/packages/workflow/src/remote/answer-link.ts new file mode 100644 index 000000000..0738c0929 --- /dev/null +++ b/packages/workflow/src/remote/answer-link.ts @@ -0,0 +1,72 @@ +/** + * What crosses a connection when a durable wait is answered. + * + * Types only, and deliberately a leaf. An adapter implements these, and an + * adapter is the one place that must not reach a document runtime: judging a + * value against a response schema needs a schema compiler and a secret scanner, + * and neither belongs in a run's owner. So the shapes live here, where nothing + * an adapter cannot load lives, and the judging lives beside the contract that + * needs it. + * + * Nothing here is a parsed request. What a wait retained is journal data until + * something walks it, and walking it is what `remote/delivery.ts` does before a + * value is judged against it. + */ + +import type { Operation, Result } from "effection"; +import type { Json } from "@executablemd/durable-streams"; + +/** One delivered answer, as a run's owner retains it. */ +export interface RemoteRetainedAnswer { + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; + readonly answer: Json; + readonly state: "pending" | "consumed"; +} + +/** + * The wait a run is standing at, as its owner retains it. + * + * `request` and `responseSchema` are exactly what the retained description + * held. The fingerprint is the owner's own, over that description, and is what + * a retention is later held to — so a runner that derives a different one from + * the same bytes has found a disagreement rather than a wait. + */ +export interface RemoteRetainedWaitRecord { + readonly runId: string; + readonly suspensionId: string; + readonly requestEventId: string; + readonly request: Json; + readonly responseSchema: Json; + readonly requestFingerprint: string; +} + +/** One judged value, offered to the owner for retention. */ +export interface RemoteAnswerRetention { + readonly runId: string; + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; + readonly answer: Json; +} + +/** What one accepted delivery left behind. */ +export interface RemoteAnswerRetained { + readonly runId: string; + readonly suspensionId: string; +} + +/** + * Reaching a run's owner to answer it, and nothing else. + * + * Two operations, both about one wait. There is no acquisition to take, no + * lifecycle to move and no journal to append: an implementation offering any of + * those would be an executor plane wearing this one's name. + */ +export interface RemoteDeliveryLink { + /** What this run is waiting at, or why it is not waiting at that. */ + wait(runId: string, suspensionId: string): Operation>; + /** Retain one judged value, or say why this run will not. */ + retain(retention: RemoteAnswerRetention): Operation>; +} diff --git a/packages/workflow/src/remote/answers.ts b/packages/workflow/src/remote/answers.ts new file mode 100644 index 000000000..34413b5ab --- /dev/null +++ b/packages/workflow/src/remote/answers.ts @@ -0,0 +1,138 @@ +/** + * Ending a durable wait on a run whose owner is somewhere else. + * + * The delivered value is retained on the owner, and the event that answers the + * wait is published on the owner, and the two have to happen together. Locally + * that is one SQLite transaction. Here it is one commit: the execution reads + * what the owner retains, publishes the answer into the transaction it is + * already inside, and enlists the consumption beside it — so the owner receives + * one proposal that appends the event and spends the row, and applies both or + * neither. + * + * Nothing about that is a claim the runner gets to make. The owner holds the + * value, checks the event against what it holds, and refuses a consumption + * whose row is gone, spent, or delivered against a different request. What the + * runner decides is only where its execution is standing. + */ + +import { type Operation, scoped } from "effection"; +import type { Json, JournalProvenance } from "@executablemd/durable-streams"; +import { atOwnRequest } from "../suspension/position.ts"; +import { suspensionRequestFingerprint } from "../suspension/api.ts"; +import { + type SuspensionAnswerAuthority, + type SuspensionAnswerProvider, + useSuspensionAnswerProvider, +} from "../suspension/answer.ts"; +import type { WorkflowRunDatabase } from "../storage/api.ts"; +import { WorkflowRequestError, WorkflowTransactionError } from "../storage/errors.ts"; +import type { RemoteRetainedAnswer } from "./answer-link.ts"; +import { activeWorkspaceRoute, type RemoteRunLink } from "./database.ts"; +import { withRemoteJournalRoute } from "./journal-route.ts"; + +/** The one acquired run an answer may be claimed against. */ +export interface RemoteAnsweredRun { + /** The acquisition this run is reached through. */ + readonly link: RemoteRunLink; + /** The handle the execution reads and transacts through. */ + readonly database: WorkflowRunDatabase; + /** The witness taken over the exact journal this run publishes into. */ + readonly provenance: JournalProvenance; +} + +/** + * Install this run's answer provider for the current scope. + * + * Installed beside one acquired execution's run and closed with it. There is no + * name to reach it by and no value to copy: registration is keyed by an opaque + * selection this module hands the coordinator, so a fabricated one resolves to + * nothing and a closed one stops resolving. + */ +export function* installRemoteSuspensionAnswers(run: RemoteAnsweredRun): Operation { + yield* useSuspensionAnswerProvider(remoteAnswerProvider(run)); +} + +function remoteAnswerProvider(run: RemoteAnsweredRun): SuspensionAnswerProvider { + const { link, database } = run; + return { + *claim(authority: SuspensionAnswerAuthority): Operation { + // Where the execution stands, before what the owner retains. An execution + // that is not at this wait has nothing to claim, and asking the owner + // about a wait this execution is not standing at would be asking it to + // decide something only position can decide. + const refused = yield* atOwnRequest(database, authority.suspensionId, authority.request); + if (refused !== undefined) { + return undefined; + } + + const pending = yield* link.pendingAnswer(authority.suspensionId); + if (!pending.ok) { + throw pending.error; + } + if (pending.value === undefined || pending.value.state !== "pending") { + // Nothing retained, or an answer this run already published. Either way + // this wait is not ended by a delivery, and the wait itself follows. + return undefined; + } + const retained = pending.value; + if (retained.requestFingerprint !== suspensionRequestFingerprint(authority.request)) { + // Retained against a different request. It is not an answer to the wait + // this execution reached, whatever it is an answer to. + return undefined; + } + + const claimed = yield* database.transact(function* (transaction) { + const route = yield* activeWorkspaceRoute(database, transaction); + if (route === undefined) { + throw new WorkflowTransactionError( + "the answer claim is not bound to this active workflow run transaction.", + ); + } + // The provenance this coordinator was given has to be the journal this + // transaction commits against. A publication routed anywhere else would + // append its answer outside the transaction that spends the row. + if ( + authority.journalProvenance === undefined || + authority.journalProvenance !== run.provenance + ) { + throw new WorkflowRequestError( + "the live journal this answer would be published into does not have the " + + "provenance of the selected remote run.", + ); + } + + yield* withRemoteJournalRoute(database, transaction, authority.publish(retained.answer)); + // Named, not asserted: the owner reads what it retained, holds the + // event this transaction appends to it, and spends the row in the same + // transaction — or spends nothing and appends nothing. + route.consume({ + suspensionId: retained.suspensionId, + requestEventId: retained.requestEventId, + requestFingerprint: retained.requestFingerprint, + }); + return retained.answer; + }); + if (!claimed.ok) { + throw claimed.error; + } + // Only now. What proves the wait ended is the owner having committed the + // transaction that appended the answer and spent the row — not that a + // publication was offered inside it. + return claimed.value; + }, + }; +} + +/** Read one run's retained answer in a scope of its own. */ +export function readRemoteAnswer( + link: RemoteRunLink, + suspensionId: string, +): Operation { + return scoped(function* () { + const pending = yield* link.pendingAnswer(suspensionId); + if (!pending.ok) { + throw pending.error; + } + return pending.value; + }); +} diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index 6803ee1f2..e531ca2e7 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -33,6 +33,7 @@ import type { WorkflowRunTransaction } from "../storage/api.ts"; export type CollectorRefusal = | "nested-transaction" | "publication-already-enlisted" + | "answer-already-enlisted" | "too-many-mappings" | "transaction-closed" | "operation-inside-body" @@ -71,6 +72,27 @@ export interface CommitIntent { readonly mappings: readonly RetainedMapping[]; /** The sealed bytes for the pieces this proposal may have to supply. */ readonly bytes: ReadonlyMap; + /** + * The retained answer this transaction is spending, when it is spending one. + * + * `null` for every ordinary transaction, which is nearly all of them. It is + * here rather than beside the commit because consuming a delivered answer and + * publishing the event that answers the wait are one act: an answer consumed + * without its event is lost, and an event published without the consumption + * could be delivered twice. + * + * It names the wait and the request it answers and carries no value. The + * owner holds the value already; a value travelling here would be the runner + * saying what it is owed rather than spending what it was given. + */ + readonly answer: AnswerConsumption | null; +} + +/** Which retained answer one transaction spends. */ +export interface AnswerConsumption { + readonly suspensionId: string; + readonly requestEventId: string; + readonly requestFingerprint: string; } /** What the collector needs from the connection. */ @@ -157,6 +179,7 @@ export function transactRemotely( transaction: WorkflowRunTransaction, enlist: EnlistWorkspace, anchor: TransactionAnchor, + consume: EnlistAnswer, ) => Operation, ): Operation> { return call(function* (): Operation> { @@ -234,6 +257,29 @@ export function transactRemotely( enlisted = { attempt, mappings: Object.freeze(mappings.map(detachMapping)) }; }; + let consumption: AnswerConsumption | undefined; + /** + * How an answer claim spends the retained value inside this transaction. + * + * Private for the same reason `enlist` is: it is handed to the body, so + * work that never received it cannot spend an answer, and one transaction + * spends at most one — a second would be a second wait ending inside a + * unit of work that describes one. + */ + const consume: EnlistAnswer = (offered: AnswerConsumption): void => { + if (!live) { + throw new RemoteTransactionError("transaction-closed"); + } + if (consumption !== undefined) { + throw new RemoteTransactionError("answer-already-enlisted"); + } + consumption = Object.freeze({ + suspensionId: offered.suspensionId, + requestEventId: offered.requestEventId, + requestFingerprint: offered.requestFingerprint, + }); + }; + let outcome: T; try { // A scope of its own, closed here. Everything the body started — @@ -243,10 +289,15 @@ export function transactRemotely( // teardown fails surface its failure after the commit had already gone // out, which is the one ordering that cannot be taken back. outcome = yield* scoped(() => - body({ journal }, enlist, { - workspaceRootId: starting.workspaceRootId, - journalEventId: starting.journalEventId, - }), + body( + { journal }, + enlist, + { + workspaceRootId: starting.workspaceRootId, + journalEventId: starting.journalEventId, + }, + consume, + ), ); } finally { // The handle is closed before the commit goes out, so a retained @@ -266,6 +317,7 @@ export function transactRemotely( publication: sealed?.publication ?? null, mappings: sealed?.mappings ?? [], bytes: sealed?.bytes ?? new Map(), + answer: consumption ?? null, }); if (!committed.ok) { return committed; @@ -311,6 +363,16 @@ export type EnlistWorkspace = ( mappings?: readonly RetainedMapping[], ) => void; +/** + * How an answer claim designates the retained value this transaction spends. + * + * It names the retained row rather than handing over a value, because what is + * being asked for is a consumption the owner performs: the owner reads what it + * retained, checks the event this transaction is appending against it, and + * marks the row spent in the same transaction that appends the event. + */ +export type EnlistAnswer = (consumption: AnswerConsumption) => void; + /** * A copy nobody else holds a reference into. * diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 07a0a4acf..33a402d6c 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -49,8 +49,9 @@ import type { WorkflowRunRecord, } from "../storage/record.ts"; import { createTransactionGate, type OwnerLink, transactRemotely } from "./collector.ts"; -import type { EnlistWorkspace, TransactionAnchor } from "./collector.ts"; +import type { EnlistAnswer, EnlistWorkspace, TransactionAnchor } from "./collector.ts"; import type { RemoteContent, RemoteContentRequest, RemoteFrontierSnapshot } from "./read.ts"; +import type { RemoteRetainedAnswer } from "./answer-link.ts"; import type { RemoteInvocationSnapshot } from "./records.ts"; import type { CreateWorkflowRunRequest } from "../storage/api.ts"; import type { WorkspaceRootManifest } from "../workspace/root-manifest.ts"; @@ -59,6 +60,13 @@ import type { WorkspaceRootManifest } from "../workspace/root-manifest.ts"; export interface RemoteRunLink extends OwnerLink { /** A fresh coherent frontier, for a read that must not use a snapshot. */ frontierSnapshot(): Operation; + /** + * What this run retains for one wait, if it retains anything. + * + * On the acquisition's own authority, because it is read to be spent: an + * execution asks what it may publish, and only the executor may publish. + */ + pendingAnswer(suspensionId: string): Operation>; /** Replace or clear the retrieval metadata, and answer with the result. */ replaceRetrieval( expectedWorkspaceRootId: string, @@ -139,6 +147,15 @@ export interface WorkspaceRoute { readonly enlist: EnlistWorkspace; /** Where this transaction began, so a coordinator can prove it has not drifted. */ readonly anchor: TransactionAnchor; + /** + * How this transaction spends a retained answer. + * + * On the same route, and reachable on the same terms: an answer claim that + * cannot prove it holds this exact database and this exact live transaction + * cannot spend anything, which is what keeps a wait from being ended outside + * the transaction that publishes its answer. + */ + readonly consume: EnlistAnswer; } const ActiveRoute: Context = createContext( @@ -308,18 +325,22 @@ export function useRemoteRunDatabase( } return yield* turns.take(function* (): Operation> { try { - return yield* transactRemotely(link, gate, function* (transaction, enlist, anchor) { - // The marker and the route are installed for the body's scope - // alone. Outside it neither exists, so a retained transaction - // object reaches nothing and an unrelated scope is not mistaken for - // a nested one. - yield* ActiveTransaction.set({ - handle, - enclosing: yield* ActiveTransaction.get(), - }); - yield* ActiveRoute.set({ database: handle, transaction, enlist, anchor }); - return yield* body(transaction); - }); + return yield* transactRemotely( + link, + gate, + function* (transaction, enlist, anchor, consume) { + // The marker and the route are installed for the body's scope + // alone. Outside it neither exists, so a retained transaction + // object reaches nothing and an unrelated scope is not mistaken for + // a nested one. + yield* ActiveTransaction.set({ + handle, + enclosing: yield* ActiveTransaction.get(), + }); + yield* ActiveRoute.set({ database: handle, transaction, enlist, anchor, consume }); + return yield* body(transaction); + }, + ); } catch (error) { // A body that raised, or a resource of its that failed to tear down, // is a failed transaction rather than a raised one: the interface diff --git a/packages/workflow/src/remote/delivery.ts b/packages/workflow/src/remote/delivery.ts new file mode 100644 index 000000000..c8790de0d --- /dev/null +++ b/packages/workflow/src/remote/delivery.ts @@ -0,0 +1,302 @@ +/** + * Answering a durable wait on a run whose owner is somewhere else. + * + * Delivery is not execution, and being remote does not change that: nothing + * here takes an acquisition, opens a socket, begins an execution, appends a + * journal event or moves a run's status. What it does is retain one typed value + * against the exact wait the run is standing at, so the next execution that + * reaches that wait finds it. + * + * ## Why the value is judged here rather than at the owner + * + * The wait retained a response schema, and judging a value against it means + * compiling that schema — the same compilation `` uses. That compiler + * and the secret scanner beside it are the document runtime's, and the document + * runtime is not what a run's owner is. So the shape of this exchange follows + * the shape of the authority: the owner says what the run is waiting at, this + * judges the offered value against exactly that, and the owner then re-reads + * the same facts inside its own transaction before it writes anything. + * + * The fingerprint is what makes that safe. A value is judged against one + * request, and the retention names the fingerprint of the request it was judged + * against. An owner whose retained request has changed in between refuses, + * rather than retaining a value that was judged against a schema this run no + * longer waits on. Nothing is retained on the strength of a claim made here. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import { + createSecretScanner, + type Json, + prepareElicitation, + SecretDetectedError, + type SecretFinding, + validateParsed, +} from "@executablemd/core"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { SUSPENSION_ANSWER } from "../suspension/effects.ts"; +import { + parseSuspensionRequest, + suspensionRequestFingerprint, + type WorkflowSuspensionRequest, +} from "../suspension/api.ts"; +import type { RemoteDeliveryLink, RemoteRetainedWaitRecord } from "./answer-link.ts"; +import { + parseAnswerDelivery, + type WorkflowAnswerDelivery, + WorkflowAnswerDeliveryError, + type WorkflowAnswerRetention, + WorkflowInputDelivery, +} from "../suspension/delivery.ts"; +import { canonicalJson } from "../storage/record.ts"; + +/** + * The wait one run is standing at, once its retained description is walked. + * + * Three identities travel together because a value answers all three: the wait, + * the exact journal event its request was published as, and a fingerprint of + * that request with its response schema. The fingerprint is what the retention + * is held to — a run whose retained request changed is a run this value was + * judged for and is no longer an answer to. + */ +export interface RemoteRetainedWait { + readonly runId: string; + readonly suspensionId: string; + readonly requestEventId: string; + readonly request: WorkflowSuspensionRequest; + readonly requestFingerprint: string; +} + +/** + * Install typed answer delivery over one remote link, for the current scope. + * + * `{ at: "min" }` for the reason every provider here uses it: middleware at the + * default position runs outermost, so an enclosing scope's installation would + * be selected ahead of the one installed nearer the run. + */ +export function* installRemoteInputDelivery(link: RemoteDeliveryLink): Operation { + yield* WorkflowInputDelivery.around( + { + *deliver([request]) { + return yield* deliverRemotely(link, request); + }, + }, + { at: "min" }, + ); +} + +function* deliverRemotely( + link: RemoteDeliveryLink, + request: WorkflowAnswerDelivery, +): Operation> { + const checked = parseAnswerDelivery(request); + if (!checked.ok) { + return checked; + } + const { runId, suspensionId, value, secretDetection } = checked.value; + + const answered = yield* link.wait(runId, suspensionId); + if (!answered.ok) { + return answered; + } + const waiting = walkRetainedWait(answered.value, runId, suspensionId); + if (!waiting.ok) { + return waiting; + } + + const judged = yield* judgeAnswer(waiting.value, suspensionId, value); + if (!judged.ok) { + return judged; + } + + if (secretDetection) { + const scanned = yield* scanDelivery(waiting.value, suspensionId, value); + if (!scanned.ok) { + return scanned; + } + } + + return yield* link.retain({ + runId, + suspensionId, + requestEventId: waiting.value.requestEventId, + // What the value was judged against, named so the owner can refuse a run + // that moved on while this was being judged. + requestFingerprint: waiting.value.requestFingerprint, + answer: value, + }); +} + +/** + * The wait this owner described, as a request rather than as a claim about one. + * + * The retained description is journal data reached through a public durable + * operation, so a schema nothing could validate against must not become the + * schema a value is judged by. The fingerprint is recomputed rather than + * believed: an owner that names one and retains another would have this value + * judged against a request it will not hold the retention to. + */ +function walkRetainedWait( + record: RemoteRetainedWaitRecord, + runId: string, + suspensionId: string, +): Result { + // The owner answered about a wait, and this is the one that was asked about. + // An answer describing something else is an owner disagreeing with the + // question, not a wait to judge a value against. + if (record.runId !== runId || record.suspensionId !== suspensionId) { + return Err( + new WorkflowAnswerDeliveryError( + "this run's owner answered about a different wait, so the value was not judged.", + ), + ); + } + let request: WorkflowSuspensionRequest; + try { + request = parseSuspensionRequest({ + request: record.request, + responseSchema: record.responseSchema, + }); + } catch (error) { + return Err( + new WorkflowAnswerDeliveryError( + `the request retained for ${suspensionId} is not one a durable wait can be answered ` + + `for: ${error instanceof Error ? error.message : String(error)}`, + ), + ); + } + if (suspensionRequestFingerprint(request) !== record.requestFingerprint) { + return Err( + new WorkflowAnswerDeliveryError( + `this run's owner names a different request for ${suspensionId} than the one it ` + + "returned, so the value was not judged.", + ), + ); + } + return Ok({ + runId: record.runId, + suspensionId: record.suspensionId, + requestEventId: record.requestEventId, + request, + requestFingerprint: record.requestFingerprint, + }); +} + +/** + * The value, judged by the schema the wait retained. + * + * The same compilation `` uses, so what a document may receive here is + * exactly what it may receive there. The refusal names where the value went + * wrong and never what it held: a diagnostic that quoted a rejected value would + * publish it in a place nothing filters. + */ +function* judgeAnswer( + waiting: RemoteRetainedWait, + suspensionId: string, + value: Json, +): Operation> { + let issues; + try { + const prepared = yield* prepareElicitation(waiting.request.responseSchema, "workflow answer"); + issues = validateParsed(prepared.validate, value); + } catch (error) { + return Err( + new WorkflowAnswerDeliveryError( + `the response schema retained for ${suspensionId} cannot judge an answer: ` + + (error instanceof Error ? error.message : String(error)), + ), + ); + } + if (issues.length === 0) { + return Ok(); + } + const described = issues + .map( + (issue) => `${issue.instancePath === "" ? "the value" : issue.instancePath} ${issue.message}`, + ) + .join("; "); + return Err( + new WorkflowAnswerDeliveryError( + `the value offered to ${suspensionId} does not satisfy the response schema that wait ` + + `retained: ${described}.`, + ), + ); +} + +/** + * Cross the same gate a durable event crosses, before anything is retained. + * + * Both framings, exactly as the local host scans them: the retained row this + * value becomes, and the durable event a later execution would publish from it. + * A credential that reached retained state has already leaked, and it has + * leaked into somebody else's storage — which is a reason to scan here rather + * than a reason not to. + */ +function* scanDelivery( + waiting: RemoteRetainedWait, + suspensionId: string, + value: Json, +): Operation> { + const scanner = createSecretScanner(); + const findings: SecretFinding[] = []; + for (const content of [ + canonicalJson({ + suspensionId, + requestEventId: waiting.requestEventId, + requestFingerprint: waiting.requestFingerprint, + answer: value, + }), + serializeDurableEvent(answerEvent(suspensionId, value)), + ]) { + try { + findings.push(...(yield* scanner.scan(content))); + } catch (error) { + return Err( + new WorkflowAnswerDeliveryError( + "secret detection could not scan this answer, so it was not retained: " + + (error instanceof Error ? error.message : String(error)), + ), + ); + } + } + if (findings.length === 0) { + return Ok(); + } + return Err(new WorkflowAnswerDeliveryError(describeDetection(findings))); +} + +/** + * The event a resume would publish, for the scanner to read. + * + * Which coroutine reaches the wait is not known until an execution does, and + * nothing delivered travels in that field — so it is left empty here. + */ +function answerEvent(suspensionId: string, value: Json): DurableEvent { + return { + type: "yield", + coroutineId: "", + description: { type: SUSPENSION_ANSWER, name: suspensionId, suspensionId }, + result: { status: "ok", value }, + }; +} + +/** + * What was found, without what was matched. + * + * The rule and the position say enough to fix the data flow. The fingerprints + * are keyed to a scanner that existed for this call alone, so reporting them + * would say nothing, and the matched text is exactly what must not travel. + */ +function describeDetection(findings: readonly SecretFinding[]): string { + const detected = new SecretDetectedError(findings); + const where = findings + .map((finding) => `${finding.ruleId} (${finding.messageId})`) + .filter((description, index, all) => all.indexOf(description) === index) + .join(", "); + return ( + `${detected.name}: this answer was not retained because secret detection matched it: ` + + `${where}. Neither the value nor the match is recorded. Disable detection for this ` + + "delivery with --no-secret-detection only when the value is known not to be a credential." + ); +} diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts index 14f477e54..16074ab31 100644 --- a/packages/workflow/src/remote/workspace.ts +++ b/packages/workflow/src/remote/workspace.ts @@ -67,6 +67,7 @@ import { withRemoteJournalRoute } from "./journal-route.ts"; import { resource } from "effection"; import { establishJournalProvenance, type DurableStream } from "@executablemd/durable-streams"; import { useRemoteRunDatabase, type RemoteWorkspaceLink } from "./database.ts"; +import { installRemoteSuspensionAnswers } from "./answers.ts"; import { routeRemoteRunJournal } from "./journal-route.ts"; import type { TemporaryTrees } from "./invocation.ts"; @@ -223,11 +224,17 @@ export function useRemoteRun(options: RemoteRunOptions): Operation { yield* options.link.frontierSnapshot(), ); const journal = routeRemoteRunJournal(database, options.journal); + const provenance = establishJournalProvenance(journal); + // Installed here because here is where a run's three halves exist at once: + // the acquisition it is reached through, the handle its execution transacts + // on, and the witness over the exact journal an answer would be published + // into. It closes with this scope, so a run that is over answers nothing. + yield* installRemoteSuspensionAnswers({ link: options.link, database, provenance }); yield* provide( bindings.bind({ database, journal, - provenance: establishJournalProvenance(journal), + provenance, runtime: { files: options.files, trees: options.trees, diff --git a/packages/workflow/src/suspension/answer.ts b/packages/workflow/src/suspension/answer.ts index e62ebefc2..b388212b4 100644 --- a/packages/workflow/src/suspension/answer.ts +++ b/packages/workflow/src/suspension/answer.ts @@ -44,9 +44,8 @@ import { type Result, } from "@executablemd/durable-streams"; import { WorkflowSuspensionRequestError, type WorkflowSuspensionRequest } from "./api.ts"; - -/** The effect type one delivered answer is retained under. */ -export const SUSPENSION_ANSWER = "suspension_answer"; +import { SUSPENSION_ANSWER } from "./effects.ts"; +export { SUSPENSION_ANSWER } from "./effects.ts"; /** What a live answer publication is given, and what it may do with it. */ export interface SuspensionAnswerAuthority { diff --git a/packages/workflow/src/suspension/api.ts b/packages/workflow/src/suspension/api.ts index 8151a652a..71b45ca20 100644 --- a/packages/workflow/src/suspension/api.ts +++ b/packages/workflow/src/suspension/api.ts @@ -44,7 +44,9 @@ import { type Api, createApi } from "@effectionx/context-api"; import type { Operation } from "effection"; -import { canonicalFingerprint, type Json, type JsonObject } from "@executablemd/core"; +import type { Json } from "@executablemd/durable-streams"; +import type { JsonObject } from "../storage/members.ts"; +import { fingerprintOfValue } from "./fingerprint.ts"; import { WorkflowStorageError } from "../storage/errors.ts"; /** What one durable wait is for, and what may end it. */ @@ -148,7 +150,7 @@ export function parseSuspensionRequest(value: unknown): WorkflowSuspensionReques * answer to the same question. */ export function suspensionRequestFingerprint(request: WorkflowSuspensionRequest): string { - return canonicalFingerprint({ + return fingerprintOfValue({ request: request.request, responseSchema: request.responseSchema, }); diff --git a/packages/workflow/src/suspension/delivery.ts b/packages/workflow/src/suspension/delivery.ts index 60849cd34..f983a775e 100644 --- a/packages/workflow/src/suspension/delivery.ts +++ b/packages/workflow/src/suspension/delivery.ts @@ -30,9 +30,10 @@ */ import { type Api, createApi } from "@effectionx/context-api"; -import type { Operation, Result } from "effection"; +import { Err, Ok, type Operation, type Result } from "effection"; import type { Json } from "@executablemd/core"; -import { WorkflowStorageError } from "../storage/errors.ts"; +import { checkRunId } from "../storage/create-request.ts"; +import { WorkflowRequestError, WorkflowStorageError } from "../storage/errors.ts"; /** One typed value offered to one retained durable wait. */ export interface WorkflowAnswerDelivery { @@ -94,3 +95,97 @@ export const WorkflowInputDelivery: Api = throw new WorkflowInputDeliveryProviderError(); }, }); + +/** A delivery whose every member has been checked rather than believed. */ +export interface CheckedAnswerDelivery { + readonly runId: string; + readonly suspensionId: string; + readonly value: Json; + readonly secretDetection: boolean; +} + +const DELIVERY_MEMBERS = ["runId", "suspensionId", "value", "secretDetection"]; + +/** + * The whole request, parsed as a closed shape before any member is read. + * + * The type describes what a caller meant; what arrives is whatever the language + * allows. A suspension id is opaque and every character of it is part of it, so + * the only thing asked of it is that it is a non-empty string this run could + * have derived. + */ +export function parseAnswerDelivery( + offered: WorkflowAnswerDelivery, +): Result { + if (typeof offered !== "object" || offered === null || Array.isArray(offered)) { + return Err(new WorkflowRequestError("a delivery takes an object describing one answer.")); + } + const names = new Set(Object.keys(offered)); + const missing = DELIVERY_MEMBERS.filter((name) => !names.has(name)); + if (missing.length > 0) { + return Err(new WorkflowRequestError(`the delivery is missing ${missing.join(", ")}.`)); + } + + const runId = checkRunId(Reflect.get(offered, "runId")); + if (!runId.ok) { + return runId; + } + + const suspensionId = Reflect.get(offered, "suspensionId"); + if (typeof suspensionId !== "string" || suspensionId === "") { + return Err( + new WorkflowRequestError( + "a delivery names the wait it answers, and a suspension id is a non-empty string.", + ), + ); + } + + const secretDetection = Reflect.get(offered, "secretDetection"); + if (typeof secretDetection !== "boolean") { + return Err( + new WorkflowRequestError("a delivery says whether it crosses the secret gate, as a boolean."), + ); + } + + const value = retainableJson(Reflect.get(offered, "value")); + if (value === undefined) { + return Err( + new WorkflowRequestError( + "an answer is retained in this run's storage, so it must be JSON this run can store.", + ), + ); + } + + return Ok({ runId: runId.value, suspensionId, value, secretDetection }); +} + +/** The value, if every part of it is JSON this run can retain. */ +function retainableJson(value: unknown): Json | undefined { + let encoded: string | undefined; + try { + encoded = JSON.stringify(value); + } catch { + return undefined; + } + if (encoded === undefined) { + return undefined; + } + const parsed: unknown = JSON.parse(encoded); + return isJson(parsed) ? parsed : undefined; +} + +function isJson(value: unknown): value is Json { + if (value === null || typeof value === "string" || typeof value === "number") { + return true; + } + if (typeof value === "boolean") { + return true; + } + if (Array.isArray(value)) { + return value.every(isJson); + } + if (typeof value === "object") { + return Object.values(value).every(isJson); + } + return false; +} diff --git a/packages/workflow/src/suspension/effects.ts b/packages/workflow/src/suspension/effects.ts new file mode 100644 index 000000000..0ebce1da9 --- /dev/null +++ b/packages/workflow/src/suspension/effects.ts @@ -0,0 +1,35 @@ +/** + * What names a durable wait: its two effect types, and its identity. + * + * Their own module because they are names rather than behaviour, and the + * modules that implement the behaviour reach a whole document runtime. A run's + * owner has to recognize both effect types — a request is how it knows what a + * run is waiting at, and an answer is what ends it — and an owner is not a + * document runtime. + */ + +import type { DurablePosition } from "@executablemd/durable-streams"; +import { fingerprintOfValue } from "./fingerprint.ts"; + +/** The effect type one durable wait's request is retained under. */ +export const SUSPENSION_REQUEST = "suspension_request"; + +/** The effect type one delivered answer is retained under. */ +export const SUSPENSION_ANSWER = "suspension_answer"; + +/** + * The opaque name one wait has, in this run, at this position. + * + * A digest rather than the three values joined, because the parts are a run + * identifier a caller chose and a coroutine identifier with its own separators; + * joined, two different triples could spell one string. It is opaque on + * purpose: #300 will correlate an answer to it, and a correlation key that + * revealed the position it came from would invite guessing a neighbouring wait. + */ +export function suspensionId(runId: string, position: DurablePosition): string { + return fingerprintOfValue({ + runId, + coroutineId: position.coroutineId, + index: position.index, + }).slice(0, 32); +} diff --git a/packages/workflow/src/suspension/fingerprint.ts b/packages/workflow/src/suspension/fingerprint.ts new file mode 100644 index 000000000..af1309458 --- /dev/null +++ b/packages/workflow/src/suspension/fingerprint.ts @@ -0,0 +1,18 @@ +/** + * A stable name for one value, computed the same way on every host. + * + * `canonicalFingerprint` says exactly this and reaches `node:crypto` to say it, + * which is a host builtin a run's owner cannot load. The two halves it is made + * of are both here already — the canonicalization the shared record module + * uses, and the SHA-256 the Workspace identities are computed with — so this + * composes them and produces the same digest for the same value. + */ + +import type { Json } from "@executablemd/durable-streams"; +import { canonicalJson } from "../storage/record.ts"; +import { sha256Hex } from "../workspace/sha256.ts"; + +/** The SHA-256 of a canonicalized value, as hex. */ +export function fingerprintOfValue(value: Json): string { + return sha256Hex(canonicalJson(value)); +} diff --git a/packages/workflow/src/suspension/position.ts b/packages/workflow/src/suspension/position.ts new file mode 100644 index 000000000..7ba951099 --- /dev/null +++ b/packages/workflow/src/suspension/position.ts @@ -0,0 +1,110 @@ +/** + * Whether an execution is standing at the wait it says it is. + * + * Shared rather than any one host's, because every host that ends a wait asks + * exactly this question and none of them may answer it differently. What + * decides is position: a run's storage is reached the same way through the + * neutral handle wherever it lives, and standing somewhere is not something a + * caller can claim. + */ + +import type { Operation } from "effection"; +import { fingerprintOfValue } from "./fingerprint.ts"; +import { durablePosition } from "@executablemd/durable-streams"; +import type { EffectDescription } from "@executablemd/durable-streams"; +import type { WorkflowRunDatabase } from "../storage/api.ts"; +import { parseSuspensionRequest, type WorkflowSuspensionRequest } from "./api.ts"; +import { SUSPENSION_REQUEST, suspensionId } from "./effects.ts"; + +/** + * Whether this execution is, right now, at the wait it says it is. + * + * Authority is the *current* execution reaching its own request, not the + * existence of a matching row. Retained history alone cannot decide this: on a + * resume the request from the previous execution is already in the journal, so + * a caller that ran before replay reached it could present its identifier and be + * believed. What separates the real wait from that is where the execution is. + * + * `suspendFor()` publishes its request and then enters, so by the time it gets + * here the coroutine has settled exactly one more durable yield than it had when + * the request was made — the request's own. The identifier is therefore the one + * this run derives for the position immediately behind this one, and a caller + * standing anywhere else derives a different identifier and is refused. + * + * The journal is then read to confirm that the yield at that exact position is + * this request, describing what is being presented. That is publication + * evidence, and it is checked at one position rather than searched for. + */ +export function* atOwnRequest( + database: WorkflowRunDatabase, + suspension: string, + request: WorkflowSuspensionRequest, +): Operation { + const position = yield* durablePosition(); + if (position.index === 0) { + return NOT_AT_A_WAIT; + } + const published = { + coroutineId: position.coroutineId, + index: position.index - 1, + }; + if (suspensionId(database.record.runId, published) !== suspension) { + return NOT_AT_A_WAIT; + } + + const entries = yield* database.readJournalEntries(); + if (!entries.ok) { + return NOT_AT_A_WAIT; + } + + const counts = new Map(); + let found: EffectDescription | undefined; + for (const entry of entries.value) { + if (entry.event.type !== "yield") { + continue; + } + const coroutineId = entry.event.coroutineId; + const index = counts.get(coroutineId) ?? 0; + counts.set(coroutineId, index + 1); + if (coroutineId === published.coroutineId && index === published.index) { + found = entry.event.description; + } + } + if (found === undefined || found.type !== SUSPENSION_REQUEST || found.name !== suspension) { + return NOT_AT_A_WAIT; + } + // Parsed, not merely read. A retained description is journal data, and this + // one is reached through a public durable operation any document can publish, + // so what it holds is a claim about a request rather than a request. Comparing + // raw fields would let a row that could never have come from `suspendFor()` — + // a `responseSchema` that is an array, say — admit a wait whose schema nothing + // could later validate an answer against. + let retained: WorkflowSuspensionRequest; + try { + retained = parseSuspensionRequest({ + request: found.request, + responseSchema: found.responseSchema, + }); + } catch (error) { + return ( + "the request retained at this position is not one a durable wait can be entered " + + `for: ${error instanceof Error ? error.message : String(error)}` + ); + } + + const same = + fingerprintOfValue({ + request: request.request, + responseSchema: request.responseSchema, + }) === + fingerprintOfValue({ + request: retained.request, + responseSchema: retained.responseSchema, + }); + return same ? undefined : NOT_AT_A_WAIT; +} + +export const NOT_AT_A_WAIT = + "this execution is not at that durable wait. A wait is entered by the execution that has " + + "just published its request, at the position that request was made — not by presenting an " + + "identifier a run retains somewhere else."; diff --git a/packages/workflow/src/suspension/suspend.ts b/packages/workflow/src/suspension/suspend.ts index 4c0f46e96..d41330c3b 100644 --- a/packages/workflow/src/suspension/suspend.ts +++ b/packages/workflow/src/suspension/suspend.ts @@ -44,9 +44,9 @@ */ import type { Operation } from "effection"; -import { canonicalFingerprint, type Json } from "@executablemd/core"; +import type { Json } from "@executablemd/durable-streams"; import { createDurableOperation, durablePosition } from "@executablemd/durable-streams"; -import type { DurablePosition, EffectDescription, Workflow } from "@executablemd/durable-streams"; +import type { EffectDescription, Workflow } from "@executablemd/durable-streams"; import { getWorkflowRun } from "../run.ts"; import { parseJsonValue } from "../storage/members.ts"; import { suspensionAnswerEffect } from "./answer.ts"; @@ -56,26 +56,8 @@ import { WorkflowSuspensionRequestError, type WorkflowSuspensionRequest, } from "./api.ts"; - -/** The effect type one durable wait's request is retained under. */ -export const SUSPENSION_REQUEST = "suspension_request"; - -/** - * The opaque name one wait has, in this run, at this position. - * - * A digest rather than the three values joined, because the parts are a run - * identifier a caller chose and a coroutine identifier with its own separators; - * joined, two different triples could spell one string. It is opaque on - * purpose: #300 will correlate an answer to it, and a correlation key that - * revealed the position it came from would invite guessing a neighbouring wait. - */ -export function suspensionId(runId: string, position: DurablePosition): string { - return canonicalFingerprint({ - runId, - coroutineId: position.coroutineId, - index: position.index, - }).slice(0, 32); -} +import { SUSPENSION_REQUEST, suspensionId } from "./effects.ts"; +export { SUSPENSION_REQUEST, suspensionId } from "./effects.ts"; function describeSuspension(id: string, request: WorkflowSuspensionRequest): EffectDescription { return { diff --git a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts index 23305f039..90f979768 100644 --- a/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts +++ b/packages/workflow/tests/cloudflare/executor-acquisition.vitest.ts @@ -397,6 +397,7 @@ describe("reading a runner command", () => { }, mappings: [], events: [serializedEvent("read whole")], + answer: null, }); // The shape is read — an unknown member or a malformed root would refuse // differently — and then declined on its merits: this run's frontier is not diff --git a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts new file mode 100644 index 000000000..7a8923acb --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts @@ -0,0 +1,660 @@ +/** + * Answering a run on its real owner. + * + * The facts here are the ones only a Durable Object can settle: that a delivery + * is one row written in one transaction and a refusal writes nothing at all, + * that answering takes no acquisition and leaves a live executor's socket, + * execution and lifecycle exactly where they were, and that spending the answer + * and publishing the event that answers the wait are one transaction — so a + * commit either leaves the row pending with no event, or consumed with exactly + * one. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import type { Json } from "@executablemd/durable-streams"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; +import { run, until, type Operation } from "effection"; +import { sha256Hex } from "../../src/workspace/sha256.ts"; +import { + cloudflareDeliveryLink, + type DeliveryAdmission, +} from "../../src/cloudflare/delivery-client.ts"; +import { canonicalJson } from "../../src/storage/record.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +let keys: TestKeys; + +const SUSPENSION = "wait-1"; +const REQUEST = { kind: "approval", release: "1.4" }; +const SCHEMA = { + type: "object", + properties: { approved: { type: "boolean" } }, + required: ["approved"], + additionalProperties: false, +}; +const ANSWER = { approved: true }; +const FINGERPRINT = sha256Hex(canonicalJson({ request: REQUEST, responseSchema: SCHEMA })); + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`delivery-${unique}-${Math.random()}`)); +} + +async function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T | Promise, +): Promise { + return await runInDurableObject(stub, body); +} + +async function token(): Promise { + return await signToken(keys, { ...VALID_CLAIMS, iat: NOW - 10, nbf: NOW - 10, exp: NOW + 600 }); +} + +/** One owner with its keys configured and one admitted executor connection. */ +async function connected(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const presented = await token(); + const admitted = await on(stub, (owner) => owner.admitConnection({ token: presented })); + expect(admitted).toBe("admitted"); +} + +function creation(): Record { + return { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + }; +} + +/** Send one command as the connection admitted most recently. */ +async function ask( + stub: ReturnType, + command: Record, +): Promise> { + const answered = await on(stub, (owner) => owner.sendLatest(JSON.stringify(command))); + if (answered === null || typeof answered !== "object") { + throw new Error("expected one command answer"); + } + return Object.fromEntries(Object.entries(answered)); +} + +/** Ask the delivery plane, with a correct admission unless one is supplied. */ +async function deliver( + stub: ReturnType, + body: Record | string, + admission: Partial<{ release: string | null; token: string | null; runId: string | null }> = {}, +): Promise> { + const presented = "token" in admission ? (admission.token ?? null) : await token(); + const named = { + release: "release" in admission ? (admission.release ?? null) : POLICY.release, + token: presented, + runId: "runId" in admission ? (admission.runId ?? null) : RUN_ID, + }; + const encoded = typeof body === "string" ? body : JSON.stringify(body); + const answered = await on(stub, (owner) => owner.deliverRequest(named, encoded)); + return JSON.parse(answered); +} + +function requestEvent(suspensionId = SUSPENSION): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { + type: "suspension_request", + name: suspensionId, + suspensionId, + request: REQUEST, + responseSchema: SCHEMA, + }, + result: { status: "ok", value: null }, + }); +} + +function answerEvent(suspensionId = SUSPENSION, value: Json = ANSWER): string { + return serializeDurableEvent({ + type: "yield", + coroutineId: "root", + description: { type: "suspension_answer", name: suspensionId, suspensionId }, + result: { status: "ok", value }, + }); +} + +/** What the owner's current frontier is, for a proposal to be built against. */ +async function frontier( + stub: ReturnType, +): Promise<{ rootId: string; eventId: string | null }> { + return await on(stub, (owner) => ({ + rootId: owner.currentRootId(), + eventId: owner.journalRecords().at(-1)?.eventId ?? null, + })); +} + +/** + * One run left suspended at one wait, with a live executor connection. + * + * The whole of what delivery reads: a run whose status is `suspended` and whose + * stop reason names the retained request event it stopped on. + */ +async function suspended( + stub: ReturnType, + suspensionId = SUSPENSION, +): Promise { + await connected(stub); + const begun = await ask(stub, { + id: "begin-1", + command: "begin", + runId: RUN_ID, + action: "start", + creation: creation(), + retrieval: null, + executionId: "execution-1", + }); + expect(begun["outcome"]).toBe("performed"); + + const at = await frontier(stub); + const committed = await ask(stub, { + id: "commit-1", + command: "commit", + expectedWorkspaceRootId: at.rootId, + expectedJournalEventId: at.eventId, + publication: null, + mappings: [], + events: [requestEvent(suspensionId)], + answer: null, + }); + expect(committed["outcome"]).toBe("performed"); + const decided = committed["value"]; + const minted = + decided !== null && typeof decided === "object" + ? Reflect.get(decided, "journalEventIds") + : undefined; + const eventId = Array.isArray(minted) ? String(minted[0]) : ""; + + const settled = await ask(stub, { + id: "settle-1", + command: "settle", + completion: { + executionId: "execution-1", + status: "suspended", + reason: { kind: "journal", eventId }, + }, + expectedWorkspaceRootId: at.rootId, + }); + expect(settled["outcome"]).toBe("performed"); + return eventId; +} + +function retention( + eventId: string, + overrides: Record = {}, +): Record { + return { + operation: "retain", + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + answer: canonicalJson(ANSWER), + ...overrides, + }; +} + +describe("delivering an answer to a run's owner", () => { + it("answers what the run is waiting at, and retains one value for it", async () => { + const stub = executor(); + const eventId = await suspended(stub); + + const waiting = await deliver(stub, { operation: "wait", suspensionId: SUSPENSION }); + expect(waiting["outcome"]).toBe("performed"); + expect(waiting["value"]).toEqual({ + runId: RUN_ID, + suspensionId: SUSPENSION, + requestEventId: eventId, + request: REQUEST, + responseSchema: SCHEMA, + requestFingerprint: FINGERPRINT, + }); + + const retained = await deliver(stub, retention(eventId)); + expect(retained).toEqual({ + outcome: "performed", + value: { runId: RUN_ID, suspensionId: SUSPENSION }, + }); + + const rows = await on(stub, (owner) => owner.retainedAnswers()); + expect(rows).toHaveLength(1); + expect(rows[0]?.["suspension_id"]).toBe(SUSPENSION); + expect(rows[0]?.["request_event_id"]).toBe(eventId); + expect(rows[0]?.["request_fingerprint"]).toBe(FINGERPRINT); + expect(rows[0]?.["answer"]).toBe(canonicalJson(ANSWER)); + expect(rows[0]?.["state"]).toBe("pending"); + expect(rows[0]?.["consumed_at"]).toBe(null); + }); + + it("takes nothing, and leaves a live executor exactly where it was", async () => { + const stub = executor(); + const eventId = await suspended(stub); + const before = await on(stub, (owner) => ({ + run: owner.runRow(), + executions: owner.executionRows(), + journal: owner.journalRecords(), + root: owner.currentRootId(), + holders: owner.holders(), + acquisition: owner.acquisitionId(), + })); + + expect(await deliver(stub, retention(eventId))).toMatchObject({ outcome: "performed" }); + + const after = await on(stub, (owner) => ({ + run: owner.runRow(), + executions: owner.executionRows(), + journal: owner.journalRecords(), + root: owner.currentRootId(), + holders: owner.holders(), + acquisition: owner.acquisitionId(), + })); + // The whole run, unchanged: no execution, no event, no status, no root — + // and the acquisition that was live is the same acquisition. + expect(after).toEqual(before); + // The executor can still act, which is what "took nothing" means. + expect((await ask(stub, { id: "frontier-1", command: "frontier" }))["outcome"]).toBe( + "performed", + ); + }); + + it("re-observes one decision when its answer was lost, and refuses a different one", async () => { + const stub = executor(); + const eventId = await suspended(stub); + const first = await deliver(stub, retention(eventId)); + + // The same delivery again, after its answer never arrived. + const again = await deliver(stub, retention(eventId)); + expect(again).toEqual(first); + expect(await on(stub, (owner) => owner.retainedAnswers())).toHaveLength(1); + + // A different value under the same wait is a second answer, not a retry. + const conflicting = await deliver( + stub, + retention(eventId, { answer: canonicalJson({ approved: false }) }), + ); + expect(conflicting).toEqual({ + outcome: "refused", + refusal: "command:duplicate-conflict", + }); + const rows = await on(stub, (owner) => owner.retainedAnswers()); + expect(rows).toHaveLength(1); + expect(rows[0]?.["answer"]).toBe(canonicalJson(ANSWER)); + }); + + it("refuses everything it is not, and writes nothing on the way", async () => { + const stub = executor(); + const eventId = await suspended(stub); + const before = await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + })); + + const refusals = { + wrongWait: await deliver(stub, { operation: "wait", suspensionId: "wait-elsewhere" }), + wrongRequest: await deliver(stub, retention("event-elsewhere")), + wrongFingerprint: await deliver( + stub, + retention(eventId, { requestFingerprint: "b".repeat(64) }), + ), + unknownOperation: await deliver(stub, { operation: "publish" }), + malformed: await deliver(stub, "{"), + // An unauthenticated request is refused before the run is named, so a + // body naming nothing this owner holds still refuses for the token. + unauthenticatedFirst: await deliver(stub, "{", { token: "not a token" }), + unknownMember: await deliver(stub, { ...retention(eventId), extra: 1 }), + uncanonical: await deliver(stub, retention(eventId, { answer: '{"approved":true,"a":1}' })), + oversized: await deliver( + stub, + retention(eventId, { answer: canonicalJson("x".repeat(2_000_000)) }), + ), + // The token is deliberately unusable. If the release were checked after + // it, the refusal would name the token rather than the build. + badRelease: await deliver(stub, retention(eventId), { + release: "other-build", + token: "not a token", + }), + badToken: await deliver(stub, retention(eventId), { token: "not a token" }), + wrongRun: await deliver(stub, retention(eventId), { runId: "9".repeat(52) }), + }; + + for (const [named, answered] of Object.entries(refusals)) { + expect([named, answered["outcome"]]).toEqual([named, "refused"]); + } + expect(refusals.wrongWait["refusal"]).toBe("command:wrong-suspension"); + expect(refusals.wrongRequest["refusal"]).toBe("command:wrong-suspension"); + expect(refusals.wrongFingerprint["refusal"]).toBe("command:stale-journal"); + expect(refusals.badRelease["refusal"]).toBe("release:release-mismatch"); + expect(String(refusals.unauthenticatedFirst["refusal"]).startsWith("token:")).toBe(true); + expect(refusals.wrongRun["refusal"]).toBe("command:wrong-run"); + // Byte for byte, row for row: a refusal is not a small write. + expect( + await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + })), + ).toEqual(before); + }); + + it("refuses a run that is not waiting, and one that is not here", async () => { + const running = executor(); + await connected(running); + expect( + ( + await ask(running, { + id: "begin-1", + command: "begin", + runId: RUN_ID, + action: "start", + creation: creation(), + retrieval: null, + executionId: "execution-1", + }) + )["outcome"], + ).toBe("performed"); + expect( + (await deliver(running, { operation: "wait", suspensionId: SUSPENSION }))["refusal"], + ).toBe("command:not-suspended"); + + const pristine = executor(); + await on(pristine, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + expect( + (await deliver(pristine, { operation: "wait", suspensionId: SUSPENSION }))["refusal"], + ).toBe("command:absent"); + // Naming an absent run creates nothing. + expect(await on(pristine, (owner) => owner.hasWorkflowSchema())).toBe(false); + }); +}); + +describe("reaching the delivery plane through the production client", () => { + it("asks what the run is waiting at and retains a value through the real link", async () => { + const stub = executor(); + const eventId = await suspended(stub); + const presented = await token(); + const sent: string[] = []; + const link = cloudflareDeliveryLink( + { + *send(admission: DeliveryAdmission, body: string): Operation { + sent.push(body); + return yield* until( + on(stub, (owner) => + owner.deliverRequest( + { release: admission.release, token: admission.token, runId: admission.runId }, + body, + ), + ), + ); + }, + }, + { + release: POLICY.release, + // deno-lint-ignore require-yield + *token(): Operation { + return presented; + }, + }, + ); + + const outcome = await run(function* () { + const waiting = yield* link.wait(RUN_ID, SUSPENSION); + if (!waiting.ok) { + throw waiting.error; + } + const retained = yield* link.retain({ + runId: RUN_ID, + suspensionId: SUSPENSION, + requestEventId: waiting.value.requestEventId, + requestFingerprint: waiting.value.requestFingerprint, + answer: ANSWER, + }); + if (!retained.ok) { + throw retained.error; + } + return { waiting: waiting.value, retained: retained.value }; + }); + + // The client read the owner's own account of the wait, and the owner + // retained the value under exactly the identities that account named. + expect(outcome.waiting).toEqual({ + runId: RUN_ID, + suspensionId: SUSPENSION, + requestEventId: eventId, + request: REQUEST, + responseSchema: SCHEMA, + requestFingerprint: FINGERPRINT, + }); + expect(outcome.retained).toEqual({ runId: RUN_ID, suspensionId: SUSPENSION }); + const rows = await on(stub, (owner) => owner.retainedAnswers()); + expect(rows).toHaveLength(1); + expect(rows[0]?.["answer"]).toBe(canonicalJson(ANSWER)); + // Two requests, and neither of them opened a socket or named a command. + expect(sent.map((body) => JSON.parse(body)["operation"])).toEqual(["wait", "retain"]); + expect(await on(stub, (owner) => owner.holders())).toBe(1); + }); + + it("reports an owner refusal without naming how the owner was reached", async () => { + const stub = executor(); + await suspended(stub); + const presented = await token(); + const link = cloudflareDeliveryLink( + { + *send(admission: DeliveryAdmission, body: string): Operation { + return yield* until( + on(stub, (owner) => + owner.deliverRequest( + { release: admission.release, token: admission.token, runId: admission.runId }, + body, + ), + ), + ); + }, + }, + { + release: POLICY.release, + // deno-lint-ignore require-yield + *token(): Operation { + return presented; + }, + }, + ); + + const refused = await run(function* () { + return yield* link.wait(RUN_ID, "wait-elsewhere"); + }); + + expect(refused.ok).toBe(false); + const message = refused.ok === false ? refused.error.message : ""; + expect(message).not.toContain(presented); + expect(message).not.toContain("deliverRequest"); + }); +}); + +describe("spending a retained answer", () => { + it("reads the retained answer back on the acquisition that may spend it", async () => { + const stub = executor(); + const eventId = await suspended(stub); + await deliver(stub, retention(eventId)); + + const answered = await ask(stub, { + id: "answer-1", + command: "answer", + suspensionId: SUSPENSION, + }); + + expect(answered["outcome"]).toBe("performed"); + expect(answered["value"]).toEqual({ + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + answer: canonicalJson(ANSWER), + state: "pending", + }); + // A wait nothing was delivered to answers with nothing rather than failing. + expect( + (await ask(stub, { id: "answer-2", command: "answer", suspensionId: "wait-2" }))["value"], + ).toBe(null); + }); + + it("consumes the row and appends its event in one transaction", async () => { + const stub = executor(); + const eventId = await suspended(stub); + await deliver(stub, retention(eventId)); + const at = await frontier(stub); + + const committed = await ask(stub, { + id: "commit-answer", + command: "commit", + expectedWorkspaceRootId: at.rootId, + expectedJournalEventId: at.eventId, + publication: null, + mappings: [], + events: [answerEvent()], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }); + + expect(committed["outcome"]).toBe("performed"); + const rows = await on(stub, (owner) => owner.retainedAnswers()); + expect(rows[0]?.["state"]).toBe("consumed"); + expect(rows[0]?.["consumed_at"]).not.toBe(null); + const journal = await on(stub, (owner) => owner.journalRecords()); + expect(journal.filter((entry) => entry.record === answerEvent())).toHaveLength(1); + }); + + it("spends nothing and appends nothing when the two do not agree", async () => { + const stub = executor(); + const eventId = await suspended(stub); + await deliver(stub, retention(eventId)); + const at = await frontier(stub); + const before = await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + })); + + const proposals = { + // The event carries a different value from the one that was delivered. + otherValue: { + events: [answerEvent(SUSPENSION, { approved: false })], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }, + // The consumption names a wait, and the event answers another one. + otherWait: { + events: [answerEvent("wait-2")], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }, + // Nothing is published at all, and the row is asked to be spent anyway. + noEvent: { + events: [], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }, + // The delivery this names is not the one the owner retained. + otherRequest: { + events: [answerEvent()], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: "c".repeat(64), + }, + }, + }; + + let attempt = 0; + for (const [named, proposal] of Object.entries(proposals)) { + attempt += 1; + const refused = await ask(stub, { + id: `spend-${attempt}`, + command: "commit", + expectedWorkspaceRootId: at.rootId, + expectedJournalEventId: at.eventId, + publication: null, + mappings: [], + ...proposal, + }); + expect([named, refused["outcome"], refused["refusal"]]).toEqual([ + named, + "refused", + "command:answer-unavailable", + ]); + } + + // Neither half happened: the row is still pending and no event was kept. + expect( + await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + })), + ).toEqual(before); + }); + + it("refuses to spend an answer a second time", async () => { + const stub = executor(); + const eventId = await suspended(stub); + await deliver(stub, retention(eventId)); + const at = await frontier(stub); + const spend = (id: string, expected: string | null) => ({ + id, + command: "commit", + expectedWorkspaceRootId: at.rootId, + expectedJournalEventId: expected, + publication: null, + mappings: [], + events: [answerEvent()], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }); + + expect((await ask(stub, spend("spend-1", at.eventId)))["outcome"]).toBe("performed"); + const after = await frontier(stub); + const again = await ask(stub, spend("spend-2", after.eventId)); + + expect(again).toEqual({ + id: "spend-2", + outcome: "refused", + refusal: "command:answer-unavailable", + }); + // One answer event, and the row spent once. + const journal = await on(stub, (owner) => owner.journalRecords()); + expect(journal.filter((entry) => entry.record === answerEvent())).toHaveLength(1); + expect((await on(stub, (owner) => owner.retainedAnswers()))[0]?.["state"]).toBe("consumed"); + }); +}); diff --git a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts index 676880fd1..c7f874190 100644 --- a/packages/workflow/tests/cloudflare/remote-publish.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-publish.vitest.ts @@ -174,6 +174,7 @@ function commit(overrides: Record = {}): Record { publication: { proposedWorkspaceRootId: ROOT, proposedManifest: "{}", content: [] }, mappings: [], events: [], + answer: null, ...overrides, }); } diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index 12a914766..d75abf7bf 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -807,6 +807,26 @@ export class ExecutorObject extends WorkflowOwnerObject { return JSON.stringify(await run(() => this.read(admission, body))); } + /** + * Answer one typed delivery, the way a host's request route would. + * + * The same shape the read route has and the same absence of a socket: what a + * test proves through this is that answering a wait takes no acquisition. + */ + async deliverRequest( + admission: { release: string | null; token: string | null; runId: string | null }, + body: string, + ): Promise { + return JSON.stringify(await run(() => this.deliver(admission, body))); + } + + /** Every answer this run retains, as rows. */ + retainedAnswers(): Record[] { + return this.ctx.storage.sql + .exec("SELECT * FROM workflow_suspension_answers ORDER BY suspension_id") + .toArray(); + } + /** * Append the two rows a fork writes for itself, and one ordinary row. * diff --git a/packages/workflow/tests/remote-database.test.ts b/packages/workflow/tests/remote-database.test.ts index 18bcef0bf..42820b40a 100644 --- a/packages/workflow/tests/remote-database.test.ts +++ b/packages/workflow/tests/remote-database.test.ts @@ -27,6 +27,7 @@ import { useRemoteRunDatabase, } from "../src/remote/database.ts"; import type { RemoteFrontierSnapshot } from "../src/remote/read.ts"; +import type { RemoteRetainedAnswer } from "../src/remote/answer-link.ts"; const ROOT = "a".repeat(64); const RUN_ID = "remote-run"; @@ -87,6 +88,12 @@ function owner( }; }, // deno-lint-ignore require-yield + *pendingAnswer(): Operation> { + // Nothing is delivered to these runs. A handle that answered otherwise + // would be claiming for a wait no test here reaches. + return Ok(undefined); + }, + // deno-lint-ignore require-yield *frontierSnapshot(): Operation { frontierReads += 1; return frontierOf(retained); diff --git a/packages/workflow/tests/remote-delivery.test.ts b/packages/workflow/tests/remote-delivery.test.ts new file mode 100644 index 000000000..cea492394 --- /dev/null +++ b/packages/workflow/tests/remote-delivery.test.ts @@ -0,0 +1,560 @@ +/** + * Tier WAD — answering a durable wait on a run somebody else owns. + * + * Two halves that must not be confused, and neither is observable alone. + * **Delivery** happens while nothing is running: it asks the owner what the run + * is waiting at, judges the offered value against exactly that, crosses the + * secret gate, and asks the owner to retain it. **The claim** is what spends + * it: an execution standing at that same wait publishes one answer event and + * enlists the consumption in the same transaction, and the value reaches the + * document only once the owner has committed both. + * + * The owner-side facts — that the retention is one row written in one + * transaction, that a refusal writes nothing, and that consuming and appending + * commit together — are proved against a real Durable Object in + * `tests/cloudflare/remote-delivery.vitest.ts`. These are the runner's half: + * what is judged before anything is sent, what is sent, and what a claim + * enlists. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { call, Err, Ok, type Operation, race, type Result, scoped } from "effection"; +import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import { serializeDurableEvent } from "@executablemd/durable-streams"; +import { collect, inlineSource, registerComponents } from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import { retainedWorkflowInstallation } from "../src/run.ts"; +import { installRemoteInputDelivery } from "../src/remote/delivery.ts"; +import { installRemoteSuspensionAnswers } from "../src/remote/answers.ts"; +import type { + RemoteAnswerRetained, + RemoteAnswerRetention, + RemoteDeliveryLink, + RemoteRetainedAnswer, + RemoteRetainedWaitRecord, +} from "../src/remote/answer-link.ts"; +import type { CommitIntent, StartingFrontier } from "../src/remote/collector.ts"; +import type { CommitDecision } from "../src/remote/publication.ts"; +import { useRemoteRunDatabase, type RemoteRunLink } from "../src/remote/database.ts"; +import { routeRemoteRunJournal } from "../src/remote/journal-route.ts"; +import type { RemoteFrontierSnapshot } from "../src/remote/read.ts"; +import { WorkflowInputDelivery, type WorkflowAnswerRetention } from "../src/suspension/delivery.ts"; +import { SUSPENSION_ANSWER, SUSPENSION_REQUEST } from "../src/suspension/effects.ts"; +import { suspensionRequestFingerprint } from "../src/suspension/api.ts"; +import { suspendFor } from "../src/suspension/suspend.ts"; +import { createSuspensionController } from "../src/deno/suspension.ts"; +import type { SuspensionNotice } from "../src/deno/suspension.ts"; +import type { DefinitionRetrieval, DocumentExecutionRecord } from "../src/storage/record.ts"; +import { establishJournalProvenance } from "@executablemd/durable-streams"; + +const RUN_ID = "release-1.4"; +const SUSPENSION = "wait-1"; +const REQUEST_EVENT = "event-request"; +const SCHEMA = { + type: "object", + properties: { approved: { type: "boolean" }, note: { type: "string" } }, + required: ["approved"], + additionalProperties: false, +}; +const REQUEST = { kind: "approval", release: "1.4" }; +const SECOND = { kind: "approval", release: "1.5" }; +const ANSWER: Json = { approved: true }; +const ROOT = "a".repeat(64); + +/** + * A synthetic credential, assembled at run time. + * + * Written out as a literal it would be rejected by push protection, and joining + * the parts leaves the runtime value identical — so what the scanner sees here + * is exactly what it would see in a delivered answer. + */ +const CANARY = `ghp_${"abcdefghijklmnopqrstuvwxyz0123456789".slice(0, 36)}`; + +const FINGERPRINT = suspensionRequestFingerprint({ request: REQUEST, responseSchema: SCHEMA }); + +/** What one retained event describes, for events that describe anything. */ +function describedType(event: DurableEvent): string { + return event.type === "yield" ? event.description.type : event.type; +} + +function waitRecord(overrides: Partial = {}): RemoteRetainedWaitRecord { + return { + runId: RUN_ID, + suspensionId: SUSPENSION, + requestEventId: REQUEST_EVENT, + request: REQUEST, + responseSchema: SCHEMA, + requestFingerprint: FINGERPRINT, + ...overrides, + }; +} + +/** A scripted delivery plane, which records everything it was asked. */ +function scriptedLink( + script: { + wait?: Result; + retain?: Result; + } = {}, +) { + const asked: string[] = []; + const retained: RemoteAnswerRetention[] = []; + const link: RemoteDeliveryLink = { + // deno-lint-ignore require-yield + *wait(): Operation> { + asked.push("wait"); + return script.wait ?? Ok(waitRecord()); + }, + // deno-lint-ignore require-yield + *retain(retention: RemoteAnswerRetention): Operation> { + asked.push("retain"); + retained.push(retention); + return script.retain ?? Ok({ runId: retention.runId, suspensionId: retention.suspensionId }); + }, + }; + return { link, asked, retained }; +} + +function delivered( + link: RemoteDeliveryLink, + request: { runId?: string; suspensionId?: string; value?: Json; secretDetection?: boolean } = {}, +): Operation> { + return scoped(function* () { + yield* installRemoteInputDelivery(link); + return yield* WorkflowInputDelivery.operations.deliver({ + runId: request.runId ?? RUN_ID, + suspensionId: request.suspensionId ?? SUSPENSION, + value: request.value ?? ANSWER, + secretDetection: request.secretDetection ?? true, + }); + }); +} + +describe("delivering one typed value to a remote run", () => { + it("judges the value against the wait the owner names, then retains it", function* () { + const scripted = scriptedLink(); + const outcome = yield* delivered(scripted.link); + + expect([outcome.ok, outcome.ok === false && String(outcome.error)]).toEqual([true, false]); + expect(outcome.ok && outcome.value).toEqual({ runId: RUN_ID, suspensionId: SUSPENSION }); + // The owner was asked what the run is waiting at before anything was sent + // for retention, and the retention names what the value was judged against. + expect(scripted.asked).toEqual(["wait", "retain"]); + expect(scripted.retained[0]).toEqual({ + runId: RUN_ID, + suspensionId: SUSPENSION, + requestEventId: REQUEST_EVENT, + requestFingerprint: FINGERPRINT, + answer: ANSWER, + }); + }); + + it("refuses a value the retained schema does not admit, and retains nothing", function* () { + const scripted = scriptedLink(); + const outcome = yield* delivered(scripted.link, { value: { approved: "yes" } }); + + expect(outcome.ok).toBe(false); + expect(outcome.ok === false && outcome.error.message).toContain( + "does not satisfy the response", + ); + // Nothing crossed. A run must not hold a value it could never be given. + expect(scripted.asked).toEqual(["wait"]); + }); + + it("refuses a credential without repeating it, and retains it when told to", function* () { + const scanned = scriptedLink(); + const refused = yield* delivered(scanned.link, { + value: { approved: true, note: CANARY }, + }); + + expect(refused.ok).toBe(false); + const message = refused.ok === false ? refused.error.message : ""; + expect(message).toContain("secret detection matched it"); + // Neither the value nor the match travels with the refusal. + expect(message).not.toContain(CANARY); + expect(scanned.asked).toEqual(["wait"]); + + // The explicit opt-out is the only way past the gate, and it retains the + // value the caller offered. + const opted = scriptedLink(); + const retained = yield* delivered(opted.link, { + value: { approved: true, note: CANARY }, + secretDetection: false, + }); + expect(retained.ok).toBe(true); + expect(opted.retained[0]?.answer).toEqual({ approved: true, note: CANARY }); + }); + + it("refuses a request that is not one, before the owner is reached", function* () { + const scripted = scriptedLink(); + const outcomes = yield* scoped(function* () { + yield* installRemoteInputDelivery(scripted.link); + return { + empty: yield* WorkflowInputDelivery.operations.deliver({ + runId: "", + suspensionId: SUSPENSION, + value: ANSWER, + secretDetection: true, + }), + unnamed: yield* WorkflowInputDelivery.operations.deliver({ + runId: RUN_ID, + suspensionId: "", + value: ANSWER, + secretDetection: true, + }), + }; + }); + + expect(outcomes.empty.ok).toBe(false); + expect(outcomes.unnamed.ok).toBe(false); + expect(scripted.asked).toEqual([]); + }); + + it("refuses an owner that answers about another wait", function* () { + const scripted = scriptedLink({ wait: Ok(waitRecord({ suspensionId: "wait-2" })) }); + const outcome = yield* delivered(scripted.link); + + expect(outcome.ok).toBe(false); + expect(outcome.ok === false && outcome.error.message).toContain("a different wait"); + expect(scripted.asked).toEqual(["wait"]); + }); + + it("refuses an owner whose fingerprint is not the request it returned", function* () { + const scripted = scriptedLink({ + wait: Ok(waitRecord({ requestFingerprint: "b".repeat(64) })), + }); + const outcome = yield* delivered(scripted.link); + + expect(outcome.ok).toBe(false); + // The value would have been judged against one request and retained against + // another. Nothing is sent. + expect(outcome.ok === false && outcome.error.message).toContain("a different request"); + expect(scripted.asked).toEqual(["wait"]); + }); + + it("refuses a retained request no wait could be answered for", function* () { + const scripted = scriptedLink({ wait: Ok(waitRecord({ responseSchema: [] })) }); + const outcome = yield* delivered(scripted.link); + + expect(outcome.ok).toBe(false); + expect(scripted.asked).toEqual(["wait"]); + }); + + it("reports what the owner refused, and nothing about how it was reached", function* () { + const scripted = scriptedLink({ retain: Err(new Error("this run is not waiting")) }); + const outcome = yield* delivered(scripted.link); + + expect(outcome.ok).toBe(false); + expect(scripted.asked).toEqual(["wait", "retain"]); + }); +}); + +/** + * One owner that retains what it is told, as the real protocol would. + * + * It keeps the journal the transaction commits, answers the frontier from it, + * and records the intents it received — which is what a claim has to be judged + * by, because what a claim does is propose one. + */ +function scriptedOwner(script: { pending?: RemoteRetainedAnswer | undefined } = {}) { + const events: { eventId: string; event: DurableEvent }[] = []; + const intents: CommitIntent[] = []; + const asked: string[] = []; + let minted = 0; + let pending = script.pending; + let refuse = false; + + const snapshot = (): RemoteFrontierSnapshot => ({ + record: { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-01-01T00:00:00.000Z", + updatedAt: "2026-01-01T00:00:00.000Z", + }, + retrieval: undefined, + workspaceRootId: ROOT, + journalEventId: events.at(-1)?.eventId ?? null, + entries: events.map((entry) => ({ + eventId: entry.eventId, + event: entry.event, + workspaceRootId: ROOT, + })), + }); + + const link: RemoteRunLink = { + // deno-lint-ignore require-yield + *frontier(): Operation { + const now = snapshot(); + return { + workspaceRootId: now.workspaceRootId, + journalEventId: now.journalEventId, + events: now.entries.map((entry) => entry.event), + }; + }, + // deno-lint-ignore require-yield + *frontierSnapshot(): Operation { + return snapshot(); + }, + // deno-lint-ignore require-yield + *pendingAnswer(suspensionId: string): Operation> { + asked.push(suspensionId); + return Ok(pending?.suspensionId === suspensionId ? pending : undefined); + }, + // deno-lint-ignore require-yield + *commit(intent: CommitIntent): Operation> { + intents.push(intent); + if (refuse) { + return Err(new Error("this owner refused the proposal")); + } + // What the owner does with a consumption: it spends the row, in the same + // transaction as the events, or neither. + if (intent.answer !== null) { + if (pending === undefined || pending.state !== "pending") { + return Err(new Error("there is no retained answer to spend")); + } + pending = { ...pending, state: "consumed" }; + } + const ids: string[] = []; + for (const event of intent.events) { + minted += 1; + const eventId = `event-${minted}`; + events.push({ eventId, event }); + ids.push(eventId); + } + return Ok({ workspaceRootId: ROOT, journalEventIds: ids }); + }, + // deno-lint-ignore require-yield + *replaceRetrieval(): Operation> { + return Ok(undefined); + }, + // deno-lint-ignore require-yield + *readExecutions(): Operation> { + return Ok([]); + }, + }; + + return { + link, + intents, + events, + asked, + waits(): string[] { + return events.flatMap((entry) => + describedType(entry.event) === SUSPENSION_REQUEST && entry.event.type === "yield" + ? [String(entry.event.description.name ?? "")] + : [], + ); + }, + get pending(): RemoteRetainedAnswer | undefined { + return pending; + }, + deliver(answer: Json, suspensionId: string, fingerprint = FINGERPRINT): void { + const request = events.find( + (entry) => + describedType(entry.event) === SUSPENSION_REQUEST && + entry.event.type === "yield" && + entry.event.description.name === suspensionId, + ); + pending = { + suspensionId, + requestEventId: request?.eventId ?? "", + requestFingerprint: fingerprint, + answer, + state: "pending", + }; + }, + refuseNext(): void { + refuse = true; + }, + }; +} + +interface Reached { + readonly notice: SuspensionNotice | undefined; + readonly returned: unknown; + readonly thrown: unknown; +} + +/** + * Run the waiting document once against a remote run, and settle what it did. + * + * The production pieces: the real remote handle over the scripted owner, the + * real routed journal and provenance, the real answer provider installed beside + * them, and the real suspension controller standing in for the executor. + */ +function reach(owner: ReturnType): Operation { + return scoped(function* () { + const database = yield* useRemoteRunDatabase(owner.link, yield* owner.link.frontierSnapshot()); + // The run's own journal underneath, so an append outside a transaction is + // a journal-only commit to the owner rather than something kept locally. + const stream = routeRemoteRunJournal(database, database.journal); + const provenance = establishJournalProvenance(stream); + yield* installRemoteSuspensionAnswers({ link: owner.link, database, provenance }); + + const suspension = createSuspensionController({ database }); + let thrown: unknown; + let returned: unknown; + let notice: SuspensionNotice | undefined; + + yield* registerComponents([ + { + name: "Probe", + origin: "tier-wad", + props: { type: "object", properties: {}, additionalProperties: false }, + // Two waits, so an answered one is behind a run that is still going. + // A document with one wait completes the moment it is answered, and a + // completed run replays nothing. + *fn() { + returned = yield* suspendFor({ request: REQUEST, responseSchema: SCHEMA }); + yield* suspendFor({ request: SECOND, responseSchema: SCHEMA }); + return ""; + }, + }, + ]); + + yield* race([ + call(function* (): Operation { + try { + yield* suspension.own( + call(function* (): Operation { + yield* collect( + yield* executeInstalled({ ...inlineSource("\n"), stream }, [ + retainedWorkflowInstallation({ + runId: RUN_ID, + base: "main", + pinnedCommit: "9fceb02d0ae598e95dc970b74767f19372d61af8", + }), + ]), + ); + }), + ); + } catch (error) { + thrown = error; + } + }), + call(function* (): Operation { + notice = yield* suspension.notice; + }), + ]); + + return { notice, returned, thrown }; + }); +} + +describe("a remote run's answer claim", () => { + it("waits when nothing is retained, and publishes no answer", function* () { + const owner = scriptedOwner(); + const reached = yield* reach(owner); + + expect(reached.notice?.suspensionId).toBeDefined(); + expect(reached.returned).toBe(undefined); + // One request event and nothing else. A wait nobody answered is a wait. + const published = owner.events.map((entry) => describedType(entry.event)); + expect(published).toContain(SUSPENSION_REQUEST); + expect(published).not.toContain(SUSPENSION_ANSWER); + expect(owner.intents.every((intent) => intent.answer === null)).toBe(true); + }); + + it("spends the retained answer and publishes exactly one answer event", function* () { + const owner = scriptedOwner(); + // One execution reaches the wait and stops; the value arrives afterwards. + yield* reach(owner); + const first = owner.waits()[0] ?? ""; + owner.deliver(ANSWER, first); + + const resumed = yield* reach(owner); + + // The document received the delivered value, and it received it from the + // wait rather than from a handler — then went on to its next wait. + expect(resumed.returned).toEqual(ANSWER); + expect(resumed.notice?.suspensionId).toBeDefined(); + expect(resumed.notice?.suspensionId).not.toBe(first); + // Exactly one answer event, behind the request it answers. + const answers = owner.events.filter( + (entry) => describedType(entry.event) === SUSPENSION_ANSWER, + ); + expect(answers).toHaveLength(1); + const answered = answers[0]?.event; + expect(answered !== undefined && answered.type === "yield" && answered.description.name).toBe( + first, + ); + expect(answered !== undefined && answered.type === "yield" && answered.result).toEqual({ + status: "ok", + value: ANSWER, + }); + const positions = owner.events.map((entry) => describedType(entry.event)); + expect(positions.indexOf(SUSPENSION_ANSWER)).toBeGreaterThan( + positions.indexOf(SUSPENSION_REQUEST), + ); + + // And the proposal that published it is the one that spent the row. + const spending = owner.intents.filter((intent) => intent.answer !== null); + expect(spending).toHaveLength(1); + expect(spending[0]?.answer?.suspensionId).toBe(first); + expect(spending[0]?.answer?.requestFingerprint).toBe(FINGERPRINT); + expect(spending[0]?.events).toHaveLength(1); + expect(spending[0]?.events.map((event) => serializeDurableEvent(event))).toEqual( + answers.map((entry) => serializeDurableEvent(entry.event)), + ); + expect(owner.pending?.state).toBe("consumed"); + }); + + it("leaves the answer pending when the owner refuses the proposal", function* () { + const owner = scriptedOwner(); + yield* reach(owner); + owner.deliver(ANSWER, owner.waits()[0] ?? ""); + owner.refuseNext(); + + const resumed = yield* reach(owner); + + // Nothing was returned to the document, nothing was published, and the + // answer is still there to be spent by a later execution. + expect(resumed.returned).toBe(undefined); + expect(owner.pending?.state).toBe("pending"); + expect( + owner.events.filter((entry) => describedType(entry.event) === SUSPENSION_ANSWER), + ).toHaveLength(0); + }); + + it("does not claim an answer delivered against a different request", function* () { + const owner = scriptedOwner(); + yield* reach(owner); + owner.deliver(ANSWER, owner.waits()[0] ?? "", "c".repeat(64)); + + const resumed = yield* reach(owner); + + // The wait is reached and stays a wait: this value answers something else. + expect(resumed.returned).toBe(undefined); + expect(resumed.notice?.suspensionId).toBe(owner.waits()[0]); + expect(owner.pending?.state).toBe("pending"); + expect(owner.intents.every((intent) => intent.answer === null)).toBe(true); + }); + + it("replays a published answer without reading or spending anything again", function* () { + const owner = scriptedOwner(); + yield* reach(owner); + const first = owner.waits()[0] ?? ""; + owner.deliver(ANSWER, first); + yield* reach(owner); + const published = owner.events.length; + const asked = owner.asked.length; + + const replayed = yield* reach(owner); + + // The answer came back from the journal. Nothing was published, nothing + // was spent, and the retained state was never asked about for that wait. + expect(replayed.returned).toEqual(ANSWER); + expect(owner.events).toHaveLength(published); + expect(owner.intents.filter((intent) => intent.answer !== null)).toHaveLength(1); + expect(owner.asked.slice(asked)).not.toContain(first); + }); +}); diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index 6b3464153..8deed2789 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -635,6 +635,7 @@ describe("what the production runner publishes", () => { publication: null, mappings: [], bytes: new Map(), + answer: null, }; const committed = yield* link.commit(intent); expect([attempt, committed.ok]).toEqual([attempt, true]); @@ -660,6 +661,7 @@ describe("what the production runner publishes", () => { publication: null, mappings: [], bytes: new Map(), + answer: null, }; yield* link.commit(intent); yield* link.commit({ ...intent, events: [event("later")] }); @@ -821,6 +823,7 @@ describe("what the production runner publishes", () => { publication: null, mappings: [], bytes: new Map(), + answer: null, }); expect(committed.ok).toBe(false); }); @@ -840,6 +843,7 @@ describe("what the production runner publishes", () => { publication: null, mappings: [], bytes: new Map(), + answer: null, }); // One identity per event, or the two sides disagree about what history // this commit created. diff --git a/packages/workflow/tests/support/remote-lifecycle-host.ts b/packages/workflow/tests/support/remote-lifecycle-host.ts index 0b279cc46..543809183 100644 --- a/packages/workflow/tests/support/remote-lifecycle-host.ts +++ b/packages/workflow/tests/support/remote-lifecycle-host.ts @@ -40,6 +40,7 @@ import { WorkflowTransactionError, } from "../../src/storage/errors.ts"; import type { RemoteWorkspaceLink } from "../../src/remote/database.ts"; +import type { RemoteRetainedAnswer } from "../../src/remote/answer-link.ts"; export const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; export const ROOT = "a".repeat(64); @@ -167,6 +168,12 @@ function link(): RemoteWorkspaceLink { return frontier(); }, // deno-lint-ignore require-yield + *pendingAnswer(): Operation> { + // These are lifecycle scripts. Nothing is delivered to their runs, and a + // link that answered otherwise would end a wait no script reaches. + return Ok(undefined); + }, + // deno-lint-ignore require-yield *frontier(): Operation { return unsupported(); }, From fcee26527abd07d923af181c5702345958c6d686 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Tue, 8 Sep 2026 13:06:38 -0300 Subject: [PATCH 58/81] =?UTF-8?q?=F0=9F=90=9B=20Judge=20a=20delivered=20an?= =?UTF-8?q?swer=20where=20it=20is=20written,=20and=20let=20one=20wait=20en?= =?UTF-8?q?d=20once?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three places took a caller's word for something only the owner can decide. The delivery plane accepted a value beside identities the caller supplied, and retained it without ever asking whether the value satisfied the schema the wait published. A caller with delivery admission could read the wait, then offer any canonical JSON against it and leave durable state holding an answer the wait does not admit — one a later execution would publish and hand to authored control flow. The lower operation that made that possible is gone. There is one mutating delivery operation now, it carries the value and the gate decision and nothing else, and the owner resolves the wait, judges the value against the schema that wait retained, and applies the selected gate, all inside the transaction that writes. Judging it there needed a judgment that runs there. The compiler the document path uses builds validators with `new Function`, which a Worker refuses, so the judgment is written in the language itself and both sides run that exact module: the runner before it offers anything, and the owner before it writes. It is closed rather than lenient — every keyword it applies is listed, and a schema using anything else is refused rather than judged with that constraint quietly skipped, so a value it accepts is one every constraint its wait stated was actually checked against. A parity table holds it to the compiler's own verdicts across every schema it admits. The credential gate is the same shape and is honestly narrower: the full scanner needs a Node runtime the owner does not have, so the runner still runs it and the owner applies a floor over the same two framings. What the caller may no longer do is omit the choice. Reading the value took only a socket. Any admitted connection that knew a suspension identifier was handed the retained answer — including one whose execution had already settled, and one that had begun nothing — which discloses document input and lets a publication be built before anything owns the right to publish it. A claim now names the journal event the request was published as, and the owner requires the asking acquisition to hold an execution the run has not moved past. A settled, replaced or never-begun acquisition is told nothing. Publishing an answer was not tied to spending one. A commit could append a `suspension_answer` with no consumption at all, or append a second answer event beside the one its consumption authorized, and the owner would write both. The two are one act, so they are checked as one: an answer event with no consumption is refused, a consumption with anything other than exactly one answer event is refused, and a consumption from an acquisition holding no open execution is refused. Every check happens inside the owner's transaction before any durable mutation, so a refusal takes the whole proposal with it. --- packages/workflow/src/cloudflare/client.ts | 35 +- packages/workflow/src/cloudflare/commands.ts | 48 +- .../src/cloudflare/delivery-client.ts | 8 +- .../workflow/src/cloudflare/delivery-plane.ts | 50 +- .../workflow/src/cloudflare/dispatcher.ts | 10 +- .../workflow/src/cloudflare/owner-answers.ts | 244 ++++++++- packages/workflow/src/cloudflare/publish.ts | 10 +- packages/workflow/src/remote/answer-link.ts | 15 +- packages/workflow/src/remote/answers.ts | 37 +- packages/workflow/src/remote/database.ts | 10 +- packages/workflow/src/remote/delivery.ts | 81 ++- .../workflow/src/suspension/credentials.ts | 125 +++++ packages/workflow/src/suspension/judgment.ts | 514 ++++++++++++++++++ .../cloudflare/remote-delivery.vitest.ts | 276 ++++++++-- .../workflow/tests/remote-delivery.test.ts | 7 +- .../tests/suspension-judgment.test.ts | 281 ++++++++++ 16 files changed, 1609 insertions(+), 142 deletions(-) create mode 100644 packages/workflow/src/suspension/credentials.ts create mode 100644 packages/workflow/src/suspension/judgment.ts create mode 100644 packages/workflow/tests/suspension-judgment.test.ts diff --git a/packages/workflow/src/cloudflare/client.ts b/packages/workflow/src/cloudflare/client.ts index 2b1491ac5..2a408b9a5 100644 --- a/packages/workflow/src/cloudflare/client.ts +++ b/packages/workflow/src/cloudflare/client.ts @@ -113,6 +113,10 @@ export type PrivateRefusal = | "command:not-suspended" | "command:wrong-suspension" | "command:answer-unavailable" + | "command:answer-rejected" + | "command:unjudgeable-schema" + | "command:credential-detected" + | "command:answer-unauthorized" | "storage:foreign" | `storage:unsupported-version-v${number}` | "storage:corrupt"; @@ -198,6 +202,10 @@ export function privateRefusal(value: string): PrivateRefusal { case "command:not-suspended": case "command:wrong-suspension": case "command:answer-unavailable": + case "command:answer-rejected": + case "command:unjudgeable-schema": + case "command:credential-detected": + case "command:answer-unauthorized": case "storage:foreign": case "storage:corrupt": return value; @@ -965,11 +973,14 @@ export function cloudflareRunLink( * Answered as the owner retains it, canonical text and all, so the value a * caller publishes is the value the owner will compare its commit against. */ - *pendingAnswer(suspensionId: string): Operation> { + *pendingAnswer( + suspensionId: string, + requestEventId: string, + ): Operation> { try { const answered = yield* connection.ask( nextId(), - { command: "answer", suspensionId }, + { command: "answer", suspensionId, requestEventId }, (value) => parseRetainedAnswer(value, suspensionId), privateRefusal, ); @@ -1163,6 +1174,26 @@ export function storageFailure(refusal: PrivateRefusal): WorkflowStorageError { "this workflow run is not waiting at that suspension. A run waits at one at a time.", ); } + if (refusal === "command:answer-rejected") { + return new WorkflowRequestError( + "the value offered to this wait does not satisfy the response schema that wait retained.", + ); + } + if (refusal === "command:unjudgeable-schema") { + return new WorkflowRequestError( + "the response schema this wait retained is not one an answer can be judged against, so " + + "no value can be delivered to it.", + ); + } + if (refusal === "command:credential-detected") { + // What was matched never travels: a diagnostic quoting it would publish + // exactly what the gate exists to keep out of retained state. + return new WorkflowRequestError( + "this answer was not retained because credential detection matched it. Neither the value " + + "nor the match is recorded. Deliver with secret detection disabled only when the value " + + "is known not to be a credential.", + ); + } if (refusal === "command:answer-unavailable") { // Nothing retained, already published, or delivered against a different // request. All three are facts about the wait rather than failures to diff --git a/packages/workflow/src/cloudflare/commands.ts b/packages/workflow/src/cloudflare/commands.ts index a5049cdd7..2371fc6ea 100644 --- a/packages/workflow/src/cloudflare/commands.ts +++ b/packages/workflow/src/cloudflare/commands.ts @@ -144,7 +144,36 @@ export type CommandRefusal = * against a different request, or the event this commit appends is not the * one that answer would become. The commit is refused whole. */ - | "answer-unavailable"; + | "answer-unavailable" + /** + * The offered value is not one this wait's schema admits. + * + * The value itself never travels back with the refusal, and neither does + * what was wrong with it beyond this category: a diagnostic quoting either + * would publish, where nothing filters it, what the judgment refused. + */ + | "answer-rejected" + /** + * The wait retained a schema this build cannot judge an answer against. + * + * Distinct from a rejected value: nothing is wrong with what was offered, + * and this owner will not retain a value it could not check. + */ + | "unjudgeable-schema" + /** + * The offered value crossed the credential gate and did not pass it. + * + * What was matched is never reported, and neither is the value. + */ + | "credential-detected" + /** + * A proposal's answer events are not the ones its consumption authorizes. + * + * One retained answer ends one wait with one event. A proposal appending an + * answer event that no consumption authorizes is forging history, and one + * appending more than one is ending more waits than it spends. + */ + | "answer-unauthorized"; export class CommandError extends Error { override name = "CommandError"; @@ -243,6 +272,14 @@ export interface ProposedAnswerConsumption { export interface AnswerCommand extends CommandEnvelope { readonly command: "answer"; readonly suspensionId: string; + /** + * The exact journal event this claim says the wait's request was published as. + * + * Named because a suspension identifier is derivable and this is not: the + * owner compares it with what the run is actually standing at, so a caller + * that guessed an identifier is asking about a wait rather than claiming one. + */ + readonly requestEventId: string; } /** The Workspace half of a proposal, when there is one. */ @@ -481,7 +518,7 @@ const MEMBERS: Record = { "events", "answer", ], - answer: [...ENVELOPE, "suspensionId"], + answer: [...ENVELOPE, "suspensionId", "requestEventId"], retrieval: [...ENVELOPE, "expectedWorkspaceRootId", "metadata"], executions: [...ENVELOPE, "anchor", "after"], mappings: ENVELOPE, @@ -709,7 +746,12 @@ export function parseCommand(raw: string): RunnerCommand { return { id, command }; } if (command === "answer") { - return { id, command, suspensionId: text(members, "suspensionId", MAX_ID) }; + return { + id, + command, + suspensionId: text(members, "suspensionId", MAX_ID), + requestEventId: text(members, "requestEventId", MAX_ID), + }; } if (command === "open") { const runId = text(members, "runId", MAX_RUN_ID); diff --git a/packages/workflow/src/cloudflare/delivery-client.ts b/packages/workflow/src/cloudflare/delivery-client.ts index 12d199bed..7af96f430 100644 --- a/packages/workflow/src/cloudflare/delivery-client.ts +++ b/packages/workflow/src/cloudflare/delivery-client.ts @@ -131,13 +131,13 @@ export function cloudflareDeliveryLink( *retain(retention: RemoteAnswerRetention): Operation> { const answered = yield* ask(retention.runId, { - operation: "retain", + operation: "deliver", suspensionId: retention.suspensionId, - requestEventId: retention.requestEventId, - requestFingerprint: retention.requestFingerprint, // Canonically encoded here, once, so the bytes the owner retains are - // the bytes a later commit is compared against. + // the bytes a later commit is compared against. Everything the owner + // decides about this value, it decides from the value. answer: canonicalJson(retention.answer), + secretDetection: retention.secretDetection, }); if (!answered.ok) { return answered; diff --git a/packages/workflow/src/cloudflare/delivery-plane.ts b/packages/workflow/src/cloudflare/delivery-plane.ts index 191911d24..1b1e57b4a 100644 --- a/packages/workflow/src/cloudflare/delivery-plane.ts +++ b/packages/workflow/src/cloudflare/delivery-plane.ts @@ -13,10 +13,13 @@ * correlated to the wait it answers, which the next acquired execution to reach * that wait spends. * - * What crosses is closed and private to this release. The value is judged - * against the wait's response schema on the runner, where the schema compiler - * lives; what this decides is identity — which run, which wait, which request — - * and it decides it again inside the transaction that writes. + * What crosses is closed and private to this release, and there is exactly one + * operation that writes. It carries a value and a gate decision and nothing + * else: no request identity, no fingerprint, no claim that anything was + * checked. The owner resolves the wait itself, judges the value against the + * schema that wait retained, applies the gate the request selected, and only + * then writes — inside one transaction, having read every one of those facts + * again. There is no lower operation to select instead. */ import { parseMembers, requireMemberNames } from "../storage/members.ts"; @@ -32,15 +35,9 @@ export type DeliveryAnswer = | { readonly outcome: "performed"; readonly value: unknown } | { readonly outcome: "refused"; readonly refusal: string }; -/** The most characters a public run id may carry. */ -const MAX_RUN_ID = 128; - /** The most characters one wait's identifier may carry. */ const MAX_SUSPENSION_ID = 256; -/** The most characters one request fingerprint may carry. */ -const FINGERPRINT = /^[0-9a-f]{64}$/; - /** * The most serialized bytes one delivery request may carry. * @@ -56,12 +53,18 @@ export const DELIVERY_REQUEST_BYTES = READ_PAGE_BYTES + READ_REQUEST_ENVELOPE; export type DeliveryOperation = | { readonly operation: "wait"; readonly suspensionId: string } | { - readonly operation: "retain"; + readonly operation: "deliver"; readonly suspensionId: string; - readonly requestEventId: string; - readonly requestFingerprint: string; - /** Canonical JSON, already encoded by the runner that judged it. */ + /** The canonical encoding of the value being offered. */ readonly answer: string; + /** + * Whether this value crosses the credential gate before it is retained. + * + * Required, with no default, because the choice is the caller's and + * omitting it must not be a way of making it. `false` is the documented + * opt-out and is the only way past the gate. + */ + readonly secretDetection: boolean; }; function failure(reason: string, path: string): Error { @@ -92,16 +95,16 @@ export function parseDeliveryOperation(raw: string): DeliveryOperation { requireMemberNames(read, ["operation", "suspensionId"], "$", failure); return { operation, suspensionId: identifier(read.get("suspensionId"), "$.suspensionId") }; } - if (operation === "retain") { + if (operation === "deliver") { requireMemberNames( read, - ["operation", "suspensionId", "requestEventId", "requestFingerprint", "answer"], + ["operation", "suspensionId", "answer", "secretDetection"], "$", failure, ); - const fingerprint = read.get("requestFingerprint"); - if (typeof fingerprint !== "string" || !FINGERPRINT.test(fingerprint)) { - throw failure("expected a sha-256 digest", "$.requestFingerprint"); + const secretDetection = read.get("secretDetection"); + if (typeof secretDetection !== "boolean") { + throw failure("expected a secret-gate decision", "$.secretDetection"); } const answer = read.get("answer"); if (typeof answer !== "string" || answer === "") { @@ -122,9 +125,8 @@ export function parseDeliveryOperation(raw: string): DeliveryOperation { return { operation, suspensionId: identifier(read.get("suspensionId"), "$.suspensionId"), - requestEventId: identifier(read.get("requestEventId"), "$.requestEventId"), - requestFingerprint: fingerprint, answer, + secretDetection, }; } throw failure("expected an operation this owner implements", "$.operation"); @@ -158,9 +160,8 @@ export function answerDelivery( runId, { suspensionId: request.suspensionId, - requestEventId: request.requestEventId, - requestFingerprint: request.requestFingerprint, answer: request.answer, + secretDetection: request.secretDetection, }, now, ); @@ -170,8 +171,7 @@ function identifier(value: unknown, path: string): string { if (typeof value !== "string" || value === "") { throw failure("expected a non-empty identifier", path); } - const limit = path === "$.suspensionId" ? MAX_SUSPENSION_ID : MAX_RUN_ID; - if (value.length > limit) { + if (value.length > MAX_SUSPENSION_ID) { throw failure("expected a bounded identifier", path); } return value; diff --git a/packages/workflow/src/cloudflare/dispatcher.ts b/packages/workflow/src/cloudflare/dispatcher.ts index 347c9662a..3d8d90d14 100644 --- a/packages/workflow/src/cloudflare/dispatcher.ts +++ b/packages/workflow/src/cloudflare/dispatcher.ts @@ -44,7 +44,7 @@ import { type RunnerCommand, } from "./commands.ts"; import { bytesOf, decodeBase64, sha256Hex } from "./encoding.ts"; -import { readRetainedAnswer } from "./owner-answers.ts"; +import { readClaimableAnswer } from "./owner-answers.ts"; import { readContent, readExecutions, @@ -350,7 +350,13 @@ function perform( }; } if (command.command === "answer") { - const retained = readRetainedAnswer(ctx.storage, command.suspensionId); + // Read on the authority that ends a wait, not on the authority that opened + // a socket: this acquisition's own open execution, and the wait this run is + // actually standing at. + const retained = readClaimableAnswer(ctx.storage, runId, acquisitionId, { + suspensionId: command.suspensionId, + requestEventId: command.requestEventId, + }); return { id: command.id, outcome: "performed", diff --git a/packages/workflow/src/cloudflare/owner-answers.ts b/packages/workflow/src/cloudflare/owner-answers.ts index d58f7a62f..af05099b9 100644 --- a/packages/workflow/src/cloudflare/owner-answers.ts +++ b/packages/workflow/src/cloudflare/owner-answers.ts @@ -12,15 +12,18 @@ * and the consumption is checked against the event the commit is appending. */ -import { parseDurableEvent } from "@executablemd/durable-streams"; +import { parseDurableEvent, serializeDurableEvent } from "@executablemd/durable-streams"; import type { DurableEvent, Json } from "@executablemd/durable-streams"; import { readRunRecord, type Row } from "../sqlite/rows.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { parseJsonValue } from "../storage/members.ts"; import { canonicalJson } from "../storage/record.ts"; +import { sightCredentials } from "../suspension/credentials.ts"; import { SUSPENSION_ANSWER, SUSPENSION_REQUEST } from "../suspension/effects.ts"; +import { judgeAgainstSchema, requireJudgeableSchema } from "../suspension/judgment.ts"; import { CommandError } from "./commands.ts"; import { sha256Hex } from "./encoding.ts"; +import { heldExecution } from "./private-schema.ts"; import { declaredObjects, holdsNoRun, isPristine, recognizeObject } from "./recognition.ts"; import { retainedText } from "./retained.ts"; import type { OwnerStorage } from "./storage.ts"; @@ -97,6 +100,84 @@ export function readRetainedWait( }; } +/** + * The retained answer this acquisition may spend, if there is one. + * + * Reading retained input is part of ending a wait, so it takes the same + * authority ending one does: the acquisition asking has to hold an open + * execution, and the wait asked about has to be the one this run is standing + * at, with the request this claim names. A socket that has begun nothing, or + * whose execution has been settled, recovered or replaced, is told nothing — + * not because the row is missing, but because reading it is not its to do. + */ +export function readClaimableAnswer( + storage: OwnerStorage, + runId: string, + acquisitionId: string, + claim: { readonly suspensionId: string; readonly requestEventId: string }, +): OwnerRetainedAnswer | undefined { + requireOpenExecution(storage, acquisitionId); + // The request this claim names, read from the run's own history. Not the + // stop reason: a run being resumed is running, and the wait it is replaying + // toward is a published event rather than the reason it last stopped. + requirePublishedRequest(storage, runId, claim); + return readRetainedAnswer(storage, claim.suspensionId); +} + +/** + * The exact journal event this run published one wait's request as. + * + * A suspension identifier is derivable and the event it was published as is + * not, so a claim names both and this requires them to describe one retained + * event of this run's. A caller that guessed an identifier is asking about a + * wait rather than claiming one. + */ +function requirePublishedRequest( + storage: OwnerStorage, + runId: string, + claim: { readonly suspensionId: string; readonly requestEventId: string }, +): void { + readRunRecord(retainedRun(storage, runId)); + const row = storage.sql + .exec("SELECT event_id, record FROM journal_events WHERE event_id = ?", claim.requestEventId) + .toArray()[0]; + if (row === undefined) { + throw new CommandError("wrong-suspension"); + } + const event = readEvent(row); + if ( + event.type !== "yield" || + event.description.type !== SUSPENSION_REQUEST || + event.description.name !== claim.suspensionId + ) { + throw new CommandError("wrong-suspension"); + } +} + +/** + * The execution this acquisition holds, or a refusal that it holds none. + * + * A socket is not an execution. What may read retained input and what may + * publish an answer is the acquisition that began an execution the run has not + * moved past — settled, recovered, or taken over by somebody else all end it. + */ +export function requireOpenExecution(storage: OwnerStorage, acquisitionId: string): string { + const held = heldExecution(storage, acquisitionId); + if (held === undefined) { + throw new CommandError("wrong-execution"); + } + const open = storage.sql + .exec( + "SELECT execution_id FROM document_executions WHERE execution_id = ? AND stopped_at IS NULL", + held, + ) + .toArray()[0]; + if (open === undefined) { + throw new CommandError("wrong-execution"); + } + return held; +} + /** What this run retains for one wait, if it retains anything. */ export function readRetainedAnswer( storage: OwnerStorage, @@ -115,42 +196,41 @@ export function readRetainedAnswer( /** * Retain one delivered answer, inside the caller's open transaction. * - * Every fact the value was judged against is read again here, under the write - * this transaction is about to make: a run that moved on — resumed, cancelled, - * answered by somebody else, or waiting at a request that changed — refuses, - * and the transaction it refuses inside wrote nothing. + * This is the mutation boundary, so this is where the value is judged. The wait + * is resolved from what the run itself retained, the schema that wait published + * is the schema the value is judged by, and the gate the request selected is + * applied — all of it here, under the write, with nothing taken from the + * caller but the value and the choice. A run that moved on refuses, and the + * transaction it refuses inside wrote nothing. * - * A compatible repeat is not a second write. The same value, against the same - * wait, the same request event and the same fingerprint, re-observes the row - * that is already there; anything else about it disagreeing is a conflict. + * A compatible repeat is not a second write. The same value against the same + * wait, still answering the same retained request, re-observes the row that is + * already there; anything else disagreeing is a conflict. */ export function retainAnswer( storage: OwnerStorage, runId: string, offered: { readonly suspensionId: string; - readonly requestEventId: string; - readonly requestFingerprint: string; readonly answer: string; + readonly secretDetection: boolean; }, now: string, ): { readonly runId: string; readonly suspensionId: string } { const waiting = readRetainedWait(storage, runId, offered.suspensionId); - if (waiting.requestEventId !== offered.requestEventId) { - // The value was judged against a request published somewhere else in this - // run's history, so it is not an answer to the wait this run is at. - throw new CommandError("wrong-suspension"); - } - if (fingerprintOf(waiting) !== offered.requestFingerprint) { - throw new CommandError("stale-journal"); + const fingerprint = fingerprintOf(waiting); + + judgeOffered(waiting, offered.answer); + if (offered.secretDetection) { + gateOffered(waiting, fingerprint, offered.answer); } const already = readRetainedAnswer(storage, offered.suspensionId); if (already !== undefined) { if ( already.state === "pending" && - already.requestEventId === offered.requestEventId && - already.requestFingerprint === offered.requestFingerprint && + already.requestEventId === waiting.requestEventId && + already.requestFingerprint === fingerprint && already.answer === offered.answer ) { // The same delivery again, after its answer was lost. One decision, one @@ -165,26 +245,124 @@ export function retainAnswer( (suspension_id, request_event_id, request_fingerprint, answer, state, created_at) VALUES (?, ?, ?, ?, 'pending', ?)`, offered.suspensionId, - offered.requestEventId, - offered.requestFingerprint, + waiting.requestEventId, + fingerprint, offered.answer, now, ); return { runId: waiting.runId, suspensionId: offered.suspensionId }; } +/** + * Judge the offered value against the schema this wait retained. + * + * The schema is walked before the value is looked at: one it cannot judge is + * refused rather than partly applied, so a value this accepts is one every + * constraint its wait published was actually checked against. A refusal names + * where the value went wrong and never what it held. + */ +function judgeOffered(waiting: OwnerRetainedWait, answer: string): void { + let value: Json; + let schema: Json; + try { + value = retained(JSON.parse(answer)); + schema = retained(waiting.responseSchema); + } catch { + throw new CommandError("malformed-member"); + } + try { + requireJudgeableSchema(schema); + } catch { + // The wait published a schema this build cannot judge an answer against. + // Retaining a value it could not check would be retaining an unjudged one. + throw new CommandError("unjudgeable-schema"); + } + if (judgeAgainstSchema(schema, value).length > 0) { + throw new CommandError("answer-rejected"); + } +} + +/** + * Cross the credential gate, in both framings the value will be stored in. + * + * The retained row and the durable event a later execution would publish, the + * same two the local host scans. What is refused is the kind that was seen; + * neither the value nor the match is recorded or reported. + */ +function gateOffered(waiting: OwnerRetainedWait, fingerprint: string, answer: string): void { + let value: Json; + try { + value = retained(JSON.parse(answer)); + } catch { + throw new CommandError("malformed-member"); + } + const framings = [ + canonicalJson({ + suspensionId: waiting.suspensionId, + requestEventId: waiting.requestEventId, + requestFingerprint: fingerprint, + answer: value, + }), + serializeDurableEvent({ + type: "yield", + coroutineId: "", + description: { + type: SUSPENSION_ANSWER, + name: waiting.suspensionId, + suspensionId: waiting.suspensionId, + }, + result: { status: "ok", value }, + }), + ]; + for (const framing of framings) { + if (sightCredentials(framing).length > 0) { + throw new CommandError("credential-detected"); + } + } +} + +/** + * Hold one proposal's answer events to the consumption that authorizes them. + * + * The two are one act and this is where that is enforced, before anything is + * written. A proposal appending an answer event without a consumption is + * forging durable history: nothing authorized that value, and no retained state + * moves with it. A proposal appending more than one answer event is ending more + * than one wait inside a unit of work that describes ending one. Both refuse + * whole. + * + * Ordinary journal events are not looked at. What is counted is exactly the + * events that claim to end a wait. + */ +export function requireAnswerEventsAuthorized( + events: readonly string[], + consumption: { readonly suspensionId: string } | null, +): void { + const claiming = events.filter((record) => isAnswerEvent(record)); + if (consumption === null) { + if (claiming.length > 0) { + throw new CommandError("answer-unauthorized"); + } + return; + } + if (claiming.length !== 1) { + throw new CommandError("answer-unauthorized"); + } +} + /** * Spend one retained answer, inside the commit that publishes it. * * The events this commit appends are what decides it. A consumption is admitted - * only when exactly one of them is the answer Yield for this wait carrying - * exactly the value this owner retained — so a runner cannot publish one value - * and spend the row for another, and cannot spend a row without publishing at - * all. A row that is gone, already spent, or delivered against a different - * request refuses, and the whole commit goes with it. + * only when the one answer event it carries is this wait's, carrying exactly + * the value this owner retained — so a runner cannot publish one value and + * spend the row for another, and cannot spend a row without publishing at all. + * A row that is gone, already spent, or delivered against a different request + * refuses, and the whole commit goes with it. */ export function consumeRetainedAnswer( storage: OwnerStorage, + acquisitionId: string, consumption: { readonly suspensionId: string; readonly requestEventId: string; @@ -193,6 +371,10 @@ export function consumeRetainedAnswer( events: readonly string[], now: string, ): void { + // Publishing an answer is ending a wait, which is the execution's to do. A + // socket that has begun nothing, or whose execution the run has moved past, + // spends nothing however well formed its proposal is. + requireOpenExecution(storage, acquisitionId); const spending = readRetainedAnswer(storage, consumption.suspensionId); if (spending === undefined || spending.state !== "pending") { throw new CommandError("answer-unavailable"); @@ -225,6 +407,16 @@ export function consumeRetainedAnswer( } } +/** Whether one appended record claims to end a wait at all. */ +function isAnswerEvent(record: string): boolean { + const parsed = parseDurableEvent(record); + if (!parsed.ok) { + return false; + } + const event = parsed.value; + return event.type === "yield" && event.description.type === SUSPENSION_ANSWER; +} + /** * Whether one appended record is this wait's answer, carrying this value. * diff --git a/packages/workflow/src/cloudflare/publish.ts b/packages/workflow/src/cloudflare/publish.ts index 9844105cc..a74ca3167 100644 --- a/packages/workflow/src/cloudflare/publish.ts +++ b/packages/workflow/src/cloudflare/publish.ts @@ -51,7 +51,7 @@ import { MAX_CONTENT_BYTES } from "./commands.ts"; import { sha256Hex } from "../workspace/sha256.ts"; import { CommandError, type CommitCommand, type ProposedMapping } from "./commands.ts"; import { validateRetainedRoot } from "./owner-reads.ts"; -import { consumeRetainedAnswer } from "./owner-answers.ts"; +import { consumeRetainedAnswer, requireAnswerEventsAuthorized } from "./owner-answers.ts"; import { readRetrieval } from "../sqlite/rows.ts"; import { bytesOf } from "./encoding.ts"; import { STAGING_TABLE } from "./private-schema.ts"; @@ -255,10 +255,12 @@ export function applyCommit( } // Before the events are written, and inside the same transaction that writes - // them: the retained answer this proposal spends has to be spendable, and the - // event it is spent for has to be the one this proposal is appending. + // them. An answer event is authorized by a consumption or by nothing, so the + // two are checked together: the events this proposal appends must carry + // exactly the answer its consumption spends, and none if it spends none. + requireAnswerEventsAuthorized(command.events, command.answer); if (command.answer !== null) { - consumeRetainedAnswer(storage, command.answer, command.events, at); + consumeRetainedAnswer(storage, acquisitionId, command.answer, command.events, at); } const journalEventIds: string[] = []; diff --git a/packages/workflow/src/remote/answer-link.ts b/packages/workflow/src/remote/answer-link.ts index 0738c0929..6ed2d76fd 100644 --- a/packages/workflow/src/remote/answer-link.ts +++ b/packages/workflow/src/remote/answer-link.ts @@ -42,13 +42,22 @@ export interface RemoteRetainedWaitRecord { readonly requestFingerprint: string; } -/** One judged value, offered to the owner for retention. */ +/** + * One value offered to a wait, for its owner to judge and retain. + * + * It carries the value and the gate decision and nothing else. There is + * deliberately no request identity and no claim that anything was checked: the + * owner resolves the wait it is answering, judges the value against the schema + * that wait retained, and applies the selected gate, all where the write + * happens. A member saying "already validated" would be a caller deciding what + * it is allowed to store. + */ export interface RemoteAnswerRetention { readonly runId: string; readonly suspensionId: string; - readonly requestEventId: string; - readonly requestFingerprint: string; readonly answer: Json; + /** Whether the owner applies the credential gate before retaining. */ + readonly secretDetection: boolean; } /** What one accepted delivery left behind. */ diff --git a/packages/workflow/src/remote/answers.ts b/packages/workflow/src/remote/answers.ts index 34413b5ab..51b3c4439 100644 --- a/packages/workflow/src/remote/answers.ts +++ b/packages/workflow/src/remote/answers.ts @@ -19,6 +19,7 @@ import { type Operation, scoped } from "effection"; import type { Json, JournalProvenance } from "@executablemd/durable-streams"; import { atOwnRequest } from "../suspension/position.ts"; import { suspensionRequestFingerprint } from "../suspension/api.ts"; +import { SUSPENSION_REQUEST } from "../suspension/effects.ts"; import { type SuspensionAnswerAuthority, type SuspensionAnswerProvider, @@ -65,7 +66,15 @@ function remoteAnswerProvider(run: RemoteAnsweredRun): SuspensionAnswerProvider return undefined; } - const pending = yield* link.pendingAnswer(authority.suspensionId); + // The event this run published the request as, read from the run's own + // journal rather than derived. The owner compares it with what the run is + // standing at, so a claim naming the wrong one is asking about a wait + // rather than claiming this one. + const requestEventId = yield* publishedRequest(database, authority.suspensionId); + if (requestEventId === undefined) { + return undefined; + } + const pending = yield* link.pendingAnswer(authority.suspensionId, requestEventId); if (!pending.ok) { throw pending.error; } @@ -123,13 +132,37 @@ function remoteAnswerProvider(run: RemoteAnsweredRun): SuspensionAnswerProvider }; } +/** + * The journal event this run published one wait's request as. + * + * Read from the run's own history, so what the claim names is what the run + * retains rather than something derived from the identifier it was given. + */ +function* publishedRequest( + database: WorkflowRunDatabase, + suspensionId: string, +): Operation { + const entries = yield* database.readJournalEntries(); + if (!entries.ok) { + return undefined; + } + const found = entries.value.find( + (entry) => + entry.event.type === "yield" && + entry.event.description.type === SUSPENSION_REQUEST && + entry.event.description.name === suspensionId, + ); + return found?.eventId; +} + /** Read one run's retained answer in a scope of its own. */ export function readRemoteAnswer( link: RemoteRunLink, suspensionId: string, + requestEventId: string, ): Operation { return scoped(function* () { - const pending = yield* link.pendingAnswer(suspensionId); + const pending = yield* link.pendingAnswer(suspensionId, requestEventId); if (!pending.ok) { throw pending.error; } diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 33a402d6c..2245390fc 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -64,9 +64,15 @@ export interface RemoteRunLink extends OwnerLink { * What this run retains for one wait, if it retains anything. * * On the acquisition's own authority, because it is read to be spent: an - * execution asks what it may publish, and only the executor may publish. + * execution asks what it may publish, only the executor may publish, and the + * owner requires that acquisition to hold an open execution before it says + * anything. The request event is named because a wait's identifier is + * derivable and the event it was published as is not. */ - pendingAnswer(suspensionId: string): Operation>; + pendingAnswer( + suspensionId: string, + requestEventId: string, + ): Operation>; /** Replace or clear the retrieval metadata, and answer with the result. */ replaceRetrieval( expectedWorkspaceRootId: string, diff --git a/packages/workflow/src/remote/delivery.ts b/packages/workflow/src/remote/delivery.ts index c8790de0d..f638dc0d5 100644 --- a/packages/workflow/src/remote/delivery.ts +++ b/packages/workflow/src/remote/delivery.ts @@ -7,21 +7,24 @@ * against the exact wait the run is standing at, so the next execution that * reaches that wait finds it. * - * ## Why the value is judged here rather than at the owner + * ## What is judged here, and what the owner judges anyway * - * The wait retained a response schema, and judging a value against it means - * compiling that schema — the same compilation `` uses. That compiler - * and the secret scanner beside it are the document runtime's, and the document - * runtime is not what a run's owner is. So the shape of this exchange follows - * the shape of the authority: the owner says what the run is waiting at, this - * judges the offered value against exactly that, and the owner then re-reads - * the same facts inside its own transaction before it writes anything. + * The owner is the authority: it resolves the wait, judges the value against + * the schema that wait retained, and applies the selected credential gate, + * inside the transaction that writes. Nothing this module reports is taken on + * trust there, and no member of the retention says a value was checked. * - * The fingerprint is what makes that safe. A value is judged against one - * request, and the retention names the fingerprint of the request it was judged - * against. An owner whose retained request has changed in between refuses, - * rather than retaining a value that was judged against a schema this run no - * longer waits on. Nothing is retained on the strength of a claim made here. + * What happens here is the document runtime's half, and it happens first + * because it is better at it. The schema compiler `` uses gives a + * document-shaped diagnostic naming where a value went wrong; the full secret + * scanner catches far more than the owner's floor. A value refused here never + * reaches the owner, and a value the owner refuses was refused for a reason + * this had no way to see. + * + * Both judgments must agree before anything is sent. The shared judgment is the + * one the owner will run, so running it here as well turns a disagreement + * between the two into a refusal on this side rather than a surprise on the + * other. */ import { Err, Ok, type Operation, type Result } from "effection"; @@ -42,6 +45,11 @@ import { type WorkflowSuspensionRequest, } from "../suspension/api.ts"; import type { RemoteDeliveryLink, RemoteRetainedWaitRecord } from "./answer-link.ts"; +import { + describeJudgment, + judgeAgainstSchema, + requireJudgeableSchema, +} from "../suspension/judgment.ts"; import { parseAnswerDelivery, type WorkflowAnswerDelivery, @@ -110,6 +118,14 @@ function* deliverRemotely( return judged; } + // The judgment the owner will make, made here too. A schema the owner cannot + // judge is refused before a value is offered against it, and a disagreement + // between the compiler and the shared judgment stops here. + const shared = judgeShared(waiting.value, suspensionId, value); + if (!shared.ok) { + return shared; + } + if (secretDetection) { const scanned = yield* scanDelivery(waiting.value, suspensionId, value); if (!scanned.ok) { @@ -120,11 +136,11 @@ function* deliverRemotely( return yield* link.retain({ runId, suspensionId, - requestEventId: waiting.value.requestEventId, - // What the value was judged against, named so the owner can refuse a run - // that moved on while this was being judged. - requestFingerprint: waiting.value.requestFingerprint, answer: value, + // The choice travels; the judgment does not. The owner applies the gate + // this names, and a value that reached here without the choice being made + // could not have been offered at all. + secretDetection, }); } @@ -224,6 +240,37 @@ function* judgeAnswer( ); } +/** + * The same judgment the owner makes, made before anything is offered. + * + * The owner refuses a schema it cannot judge rather than retaining a value it + * could not check, so a wait whose schema is outside the judged subset is + * reported here — where a caller can be told what happened — instead of + * arriving as a bare refusal from somewhere else. + */ +function judgeShared(waiting: RemoteRetainedWait, suspensionId: string, value: Json): Result { + try { + requireJudgeableSchema(waiting.request.responseSchema); + } catch (error) { + return Err( + new WorkflowAnswerDeliveryError( + `the response schema retained for ${suspensionId} is not one an answer can be judged ` + + `against: ${error instanceof Error ? error.message : String(error)}`, + ), + ); + } + const issues = judgeAgainstSchema(waiting.request.responseSchema, value); + if (issues.length === 0) { + return Ok(); + } + return Err( + new WorkflowAnswerDeliveryError( + `the value offered to ${suspensionId} does not satisfy the response schema that wait ` + + `retained: ${describeJudgment(issues)}.`, + ), + ); +} + /** * Cross the same gate a durable event crosses, before anything is retained. * diff --git a/packages/workflow/src/suspension/credentials.ts b/packages/workflow/src/suspension/credentials.ts new file mode 100644 index 000000000..f162b1d9d --- /dev/null +++ b/packages/workflow/src/suspension/credentials.ts @@ -0,0 +1,125 @@ +/** + * The credential gate a run's owner can apply for itself. + * + * Secret detection belongs where the write happens, for the same reason schema + * judgment does: a caller that decides for itself decides nothing. The full + * scanner is `@secretlint`'s and needs a Node runtime, which a run's owner does + * not have — so the owner applies this, and the runner applies the full scanner + * as well before it offers anything. + * + * This is deliberately a floor and not a replacement. It matches the shapes + * that are unambiguous on sight — issued token formats, a bearer credential, + * and a credential-named field carrying an opaque value — and it matches them + * with the same conservatism the repository's own rules use: a placeholder is + * not a credential, and neither is a short or obviously descriptive value. + * + * What it must never do is report what it matched. A diagnostic quoting the + * value would publish, in a place nothing filters, exactly what the gate exists + * to keep out of retained state. + */ + +/** What was recognized, named by kind and never by content. */ +export interface CredentialSighting { + /** The kind of credential this looks like, as a stable name. */ + readonly kind: string; +} + +/** Issued token formats that are unmistakable on sight. */ +const ISSUED: readonly { readonly kind: string; readonly pattern: RegExp }[] = [ + { kind: "github-token", pattern: /\bgh[pousr]_[A-Za-z0-9]{16,}\b/ }, + { kind: "github-pat", pattern: /\bgithub_pat_[A-Za-z0-9_]{20,}\b/ }, + { kind: "openai-key", pattern: /\bsk-[A-Za-z0-9_-]{20,}\b/ }, + { kind: "aws-access-key-id", pattern: /\b(?:AKIA|ASIA)[0-9A-Z]{16}\b/ }, + { kind: "slack-token", pattern: /\bxox[baprs]-[A-Za-z0-9-]{10,}\b/ }, + { kind: "google-api-key", pattern: /\bAIza[0-9A-Za-z_-]{35}\b/ }, + { kind: "stripe-key", pattern: /\b(?:sk|rk)_live_[0-9A-Za-z]{16,}\b/ }, + { kind: "npm-token", pattern: /\bnpm_[A-Za-z0-9]{36}\b/ }, + { kind: "private-key-block", pattern: /-----BEGIN (?:[A-Z ]+ )?PRIVATE KEY-----/ }, + { + kind: "json-web-token", + pattern: /\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b/, + }, +]; + +/** A bearer credential, whatever issued it. */ +const BEARER = /\bBearer\s+[A-Za-z0-9._~+/-]{20,}={0,2}/i; + +/** + * Field names that carry a credential, as word parts. + * + * The same field arrives under every convention a producer happens to use — + * `apiKey` from JSON, `api_key` from an environment, `api-key` from a header — + * so the names are matched by their parts rather than by one spelling. + */ +const FIELDS = [ + ["api", "key"], + ["access", "token"], + ["refresh", "token"], + ["client", "secret"], + ["session", "token"], + ["secret", "key"], + ["private", "key"], + ["auth", "token"], + ["authorization"], + ["password"], + ["passwd"], + ["apikey"], +] + .map((parts) => parts.join("[-_ ]?")) + .join("|"); + +/** + * A credential-named field carrying an opaque value. + * + * The value has to look like a secret rather than like prose: long enough to be + * one, and drawn from the alphabet issued credentials use. + */ +const NAMED_FIELD = new RegExp(`"?(?:${FIELDS})"?\\s*[:=]\\s*"?([A-Za-z0-9._~+/-]{12,})"?`, "i"); + +/** Words that mark a value as a stand-in rather than a credential. */ +const PLACEHOLDERS = [ + "example", + "placeholder", + "redacted", + "dummy", + "sample", + "changeme", + "todo", + "fixme", + "your", + "xxxx", +]; + +/** + * Everything this recognizes in one piece of content, by kind. + * + * Deduplicated, because a kind seen twice is the same finding about the same + * delivery, and ordered so two runs describe one value the same way. + */ +export function sightCredentials(content: string): CredentialSighting[] { + const found = new Set(); + for (const { kind, pattern } of ISSUED) { + if (pattern.test(content)) { + found.add(kind); + } + } + if (BEARER.test(content)) { + found.add("bearer-credential"); + } + const named = NAMED_FIELD.exec(content); + if (named !== null && !placeholder(named[1] ?? "")) { + found.add("credential-field"); + } + return [...found].toSorted().map((kind) => ({ kind })); +} + +/** Whether a matched value reads as a stand-in rather than as a credential. */ +function placeholder(value: string): boolean { + const lowered = value.toLowerCase(); + return PLACEHOLDERS.some((word) => lowered.includes(word)); +} + +/** What a refusal says, which is the kinds and never the content. */ +export function describeCredentials(sightings: readonly CredentialSighting[]): string { + return sightings.map((sighting) => sighting.kind).join(", "); +} diff --git a/packages/workflow/src/suspension/judgment.ts b/packages/workflow/src/suspension/judgment.ts new file mode 100644 index 000000000..09212ecc2 --- /dev/null +++ b/packages/workflow/src/suspension/judgment.ts @@ -0,0 +1,514 @@ +/** + * Judging one answer against the schema its wait retained, anywhere. + * + * A delivered value must satisfy the response schema the wait published before + * it enters durable state, and the place that decides that has to be the place + * that writes — otherwise a caller with delivery admission decides for itself. + * A run's owner is where the write happens, and a run's owner cannot compile a + * schema: `ajv` builds validators with `new Function`, and a Cloudflare Worker + * refuses code generation from strings. So the judgment lives here, in the + * language itself, and both the runner that offers a value and the owner that + * retains it run this exact module. + * + * ## Closed, not lenient + * + * Every keyword this understands is listed. A schema using anything else is not + * judged leniently — it is refused, and the delivery with it. That is the whole + * safety property: an unimplemented constraint can never be silently skipped, + * so a value this accepts is a value every constraint its schema states was + * actually checked against. + * + * `format` is the one exception and it matches the compiler the document path + * uses, which is configured with `validateFormats: false`: a format annotation + * constrains nothing on either side. + * + * ## What it is not + * + * Not a JSON Schema implementation, and not a second dialect. It is one + * predicate over a bounded subset, held to the compiler's own verdicts by a + * parity table, and it refuses everything outside that subset rather than + * guessing. + */ + +import type { Json } from "@executablemd/durable-streams"; +import { canonicalJson } from "../storage/record.ts"; + +/** Why one value is not an answer to one wait. */ +export interface JudgmentIssue { + /** Where in the value, as a JSON pointer fragment. */ + readonly at: string; + /** What is wrong with it, in the schema's own vocabulary. */ + readonly reason: string; +} + +/** A schema this module will not judge a value against. */ +export class UnjudgeableSchemaError extends Error { + override name = "UnjudgeableSchemaError"; +} + +/** Keywords that say nothing about whether a value is admitted. */ +const ANNOTATIONS = new Set([ + "$schema", + "$comment", + "title", + "description", + "examples", + "default", + "readOnly", + "writeOnly", + "deprecated", + // Configured off in the document path's compiler, so it constrains nothing + // there either. Accepting it and applying nothing is agreement, not leniency. + "format", + // Inert without `$ref`, which is refused, so a definition nothing can reach + // constrains nothing. + "definitions", + "$defs", +]); + +/** Keywords this understands, beyond the ones that say nothing. */ +const SUPPORTED = new Set([ + "type", + "enum", + "const", + "properties", + "required", + "additionalProperties", + "minProperties", + "maxProperties", + "propertyNames", + "items", + "additionalItems", + "minItems", + "maxItems", + "uniqueItems", + "minLength", + "maxLength", + "pattern", + "minimum", + "maximum", + "exclusiveMinimum", + "exclusiveMaximum", + "multipleOf", + "allOf", + "anyOf", + "oneOf", + "not", +]); + +const TYPES = new Set(["null", "boolean", "object", "array", "number", "integer", "string"]); + +/** + * Whether this schema is one this module can judge a value against. + * + * Walked whole, before any value is looked at, so a schema carrying a keyword + * this does not implement is refused rather than partly applied. Raises with + * what it could not read; the caller decides what to say about it. + */ +export function requireJudgeableSchema(schema: Json, at = "#"): void { + if (typeof schema === "boolean") { + return; + } + if (schema === null || typeof schema !== "object" || Array.isArray(schema)) { + throw new UnjudgeableSchemaError(`the schema at ${at} is not a schema object`); + } + for (const [keyword, value] of Object.entries(schema)) { + if (ANNOTATIONS.has(keyword)) { + continue; + } + if (!SUPPORTED.has(keyword)) { + throw new UnjudgeableSchemaError(`the schema at ${at} uses ${keyword}, which is not judged`); + } + walkKeyword(keyword, value, at); + } +} + +function walkKeyword(keyword: string, value: Json, at: string): void { + if (keyword === "type") { + const named = Array.isArray(value) ? value : [value]; + for (const name of named) { + if (typeof name !== "string" || !TYPES.has(name)) { + throw new UnjudgeableSchemaError(`the schema at ${at} names a type this does not know`); + } + } + return; + } + if (keyword === "properties" || keyword === "propertyNames") { + if (keyword === "propertyNames") { + requireJudgeableSchema(value, `${at}/propertyNames`); + return; + } + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares malformed properties`); + } + for (const [name, member] of Object.entries(value)) { + requireJudgeableSchema(member, `${at}/properties/${name}`); + } + return; + } + if (keyword === "additionalProperties" || keyword === "additionalItems" || keyword === "not") { + requireJudgeableSchema(value, `${at}/${keyword}`); + return; + } + if (keyword === "items") { + if (Array.isArray(value)) { + value.forEach((member, index) => requireJudgeableSchema(member, `${at}/items/${index}`)); + return; + } + requireJudgeableSchema(value, `${at}/items`); + return; + } + if (keyword === "allOf" || keyword === "anyOf" || keyword === "oneOf") { + if (!Array.isArray(value) || value.length === 0) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed ${keyword}`); + } + value.forEach((member, index) => requireJudgeableSchema(member, `${at}/${keyword}/${index}`)); + return; + } + if (keyword === "required") { + if (!Array.isArray(value) || value.some((name) => typeof name !== "string")) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed required`); + } + return; + } + if (keyword === "enum") { + if (!Array.isArray(value) || value.length === 0) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed enum`); + } + return; + } + if (keyword === "pattern") { + if (typeof value !== "string" || !compiled(value)) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a pattern this cannot read`); + } + return; + } + if ( + keyword === "minProperties" || + keyword === "maxProperties" || + keyword === "minItems" || + keyword === "maxItems" || + keyword === "minLength" || + keyword === "maxLength" + ) { + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed ${keyword}`); + } + return; + } + if ( + keyword === "minimum" || + keyword === "maximum" || + keyword === "exclusiveMinimum" || + keyword === "exclusiveMaximum" || + keyword === "multipleOf" + ) { + if (typeof value !== "number" || !Number.isFinite(value)) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed ${keyword}`); + } + if (keyword === "multipleOf" && value <= 0) { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed multipleOf`); + } + return; + } + if (keyword === "uniqueItems") { + if (typeof value !== "boolean") { + throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed uniqueItems`); + } + return; + } + // `const` admits any JSON value, and nothing about it can be malformed. +} + +function compiled(pattern: string): boolean { + try { + expression(pattern); + return true; + } catch { + return false; + } +} + +function expression(pattern: string): RegExp { + try { + return new RegExp(pattern, "u"); + } catch { + return new RegExp(pattern); + } +} + +/** + * Judge one value against one schema, and say what is wrong with it. + * + * The schema must already have been admitted by `requireJudgeableSchema`. An + * empty result is the value satisfying every constraint the schema states. + */ +export function judgeAgainstSchema(schema: Json, value: Json, at = ""): JudgmentIssue[] { + const issues: JudgmentIssue[] = []; + judge(schema, value, at === "" ? "" : at, issues); + return issues; +} + +function judge(schema: Json, value: Json, at: string, issues: JudgmentIssue[]): void { + if (typeof schema === "boolean") { + if (!schema) { + issues.push({ at, reason: "is not admitted here" }); + } + return; + } + if (schema === null || typeof schema !== "object" || Array.isArray(schema)) { + issues.push({ at, reason: "has no schema to be judged by" }); + return; + } + + const named = schema["type"]; + if (named !== undefined) { + const types = Array.isArray(named) ? named : [named]; + if (!types.some((type) => typeof type === "string" && isType(type, value))) { + issues.push({ at, reason: `must be ${types.join(" or ")}` }); + return; + } + } + + const constant = schema["const"]; + if (constant !== undefined && !same(constant, value)) { + issues.push({ at, reason: "must equal the constant this schema declares" }); + } + + const admitted = schema["enum"]; + if (Array.isArray(admitted) && !admitted.some((entry) => same(entry, value))) { + issues.push({ at, reason: "must be one of the values this schema enumerates" }); + } + + judgeCombinators(schema, value, at, issues); + + if (typeof value === "string") { + judgeString(schema, value, at, issues); + } + if (typeof value === "number") { + judgeNumber(schema, value, at, issues); + } + if (Array.isArray(value)) { + judgeArray(schema, value, at, issues); + } else if (value !== null && typeof value === "object") { + judgeObject(schema, value, at, issues); + } +} + +function judgeCombinators( + schema: Record, + value: Json, + at: string, + issues: JudgmentIssue[], +): void { + const all = schema["allOf"]; + if (Array.isArray(all)) { + for (const member of all) { + judge(member, value, at, issues); + } + } + const any = schema["anyOf"]; + if ( + Array.isArray(any) && + !any.some((member) => judgeAgainstSchema(member, value, at).length === 0) + ) { + issues.push({ at, reason: "must satisfy one of the alternatives this schema allows" }); + } + const one = schema["oneOf"]; + if (Array.isArray(one)) { + const matched = one.filter((member) => judgeAgainstSchema(member, value, at).length === 0); + if (matched.length !== 1) { + issues.push({ + at, + reason: "must satisfy exactly one of the alternatives this schema allows", + }); + } + } + const refused = schema["not"]; + if (refused !== undefined && judgeAgainstSchema(refused, value, at).length === 0) { + issues.push({ at, reason: "must not be what this schema excludes" }); + } +} + +function judgeString( + schema: Record, + value: string, + at: string, + issues: JudgmentIssue[], +): void { + const length = [...value].length; + const least = schema["minLength"]; + if (typeof least === "number" && length < least) { + issues.push({ at, reason: `must be at least ${least} characters` }); + } + const most = schema["maxLength"]; + if (typeof most === "number" && length > most) { + issues.push({ at, reason: `must be at most ${most} characters` }); + } + const pattern = schema["pattern"]; + if (typeof pattern === "string" && !expression(pattern).test(value)) { + issues.push({ at, reason: "must match the pattern this schema declares" }); + } +} + +function judgeNumber( + schema: Record, + value: number, + at: string, + issues: JudgmentIssue[], +): void { + const least = schema["minimum"]; + if (typeof least === "number" && value < least) { + issues.push({ at, reason: `must be at least ${least}` }); + } + const most = schema["maximum"]; + if (typeof most === "number" && value > most) { + issues.push({ at, reason: `must be at most ${most}` }); + } + const above = schema["exclusiveMinimum"]; + if (typeof above === "number" && value <= above) { + issues.push({ at, reason: `must be greater than ${above}` }); + } + const below = schema["exclusiveMaximum"]; + if (typeof below === "number" && value >= below) { + issues.push({ at, reason: `must be less than ${below}` }); + } + const multiple = schema["multipleOf"]; + if (typeof multiple === "number") { + const quotient = value / multiple; + if (!Number.isFinite(quotient) || Math.abs(quotient - Math.round(quotient)) > 1e-9) { + issues.push({ at, reason: `must be a multiple of ${multiple}` }); + } + } +} + +function judgeArray( + schema: Record, + value: readonly Json[], + at: string, + issues: JudgmentIssue[], +): void { + const least = schema["minItems"]; + if (typeof least === "number" && value.length < least) { + issues.push({ at, reason: `must have at least ${least} items` }); + } + const most = schema["maxItems"]; + if (typeof most === "number" && value.length > most) { + issues.push({ at, reason: `must have at most ${most} items` }); + } + if (schema["uniqueItems"] === true) { + const seen = new Set(value.map((entry) => canonicalJson(entry))); + if (seen.size !== value.length) { + issues.push({ at, reason: "must not repeat an item" }); + } + } + + const items = schema["items"]; + if (Array.isArray(items)) { + value.forEach((entry, index) => { + const member = items[index]; + if (member !== undefined) { + judge(member, entry, `${at}/${index}`, issues); + return; + } + const extra = schema["additionalItems"]; + if (extra !== undefined) { + judge(extra, entry, `${at}/${index}`, issues); + } + }); + return; + } + if (items !== undefined) { + value.forEach((entry, index) => judge(items, entry, `${at}/${index}`, issues)); + } +} + +function judgeObject( + schema: Record, + value: Record, + at: string, + issues: JudgmentIssue[], +): void { + const names = Object.keys(value); + const least = schema["minProperties"]; + if (typeof least === "number" && names.length < least) { + issues.push({ at, reason: `must have at least ${least} properties` }); + } + const most = schema["maxProperties"]; + if (typeof most === "number" && names.length > most) { + issues.push({ at, reason: `must have at most ${most} properties` }); + } + + const required = schema["required"]; + if (Array.isArray(required)) { + for (const name of required) { + if (typeof name === "string" && !Object.hasOwn(value, name)) { + issues.push({ at, reason: `must have the property ${JSON.stringify(name)}` }); + } + } + } + + const properties = schema["properties"]; + const declared = + properties !== null && typeof properties === "object" && !Array.isArray(properties) + ? properties + : undefined; + + const nameSchema = schema["propertyNames"]; + if (nameSchema !== undefined) { + for (const name of names) { + judge(nameSchema, name, `${at}/${name}`, issues); + } + } + + for (const name of names) { + const member = declared?.[name]; + if (member !== undefined) { + judge(member, value[name] ?? null, `${at}/${name}`, issues); + continue; + } + const extra = schema["additionalProperties"]; + if (extra === undefined) { + continue; + } + judge(extra, value[name] ?? null, `${at}/${name}`, issues); + } +} + +function isType(type: string, value: Json): boolean { + if (type === "null") { + return value === null; + } + if (type === "boolean") { + return typeof value === "boolean"; + } + if (type === "string") { + return typeof value === "string"; + } + if (type === "number") { + return typeof value === "number"; + } + if (type === "integer") { + return typeof value === "number" && Number.isInteger(value); + } + if (type === "array") { + return Array.isArray(value); + } + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +/** Whether two retained values are the same value, by canonical encoding. */ +function same(left: Json, right: Json): boolean { + try { + return canonicalJson(left) === canonicalJson(right); + } catch { + return false; + } +} + +/** What a refusal says about a value, without repeating the value. */ +export function describeJudgment(issues: readonly JudgmentIssue[]): string { + return issues + .map((issue) => `${issue.at === "" ? "the value" : issue.at} ${issue.reason}`) + .join("; "); +} diff --git a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts index 7a8923acb..94c228321 100644 --- a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts @@ -33,13 +33,22 @@ const SUSPENSION = "wait-1"; const REQUEST = { kind: "approval", release: "1.4" }; const SCHEMA = { type: "object", - properties: { approved: { type: "boolean" } }, + properties: { approved: { type: "boolean" }, note: { type: "string" } }, required: ["approved"], additionalProperties: false, }; const ANSWER = { approved: true }; const FINGERPRINT = sha256Hex(canonicalJson({ request: REQUEST, responseSchema: SCHEMA })); +/** + * A synthetic credential, assembled at run time. + * + * Written out as a literal it would be rejected by push protection, and joining + * the parts leaves the runtime value identical — so what the gate sees here is + * exactly what it would see in a delivered answer. + */ +const CANARY = `ghp_${"abcdefghijklmnopqrstuvwxyz0123456789".slice(0, 36)}`; + beforeAll(async () => { keys = await generateKeys(); }); @@ -201,20 +210,47 @@ async function suspended( return eventId; } -function retention( - eventId: string, - overrides: Record = {}, -): Record { +/** + * One delivery, as it crosses. + * + * It carries the value and the gate decision and nothing else — there is no + * lower operation that takes a value without them, which is the point. + */ +function delivery(overrides: Record = {}): Record { return { - operation: "retain", + operation: "deliver", suspensionId: SUSPENSION, - requestEventId: eventId, - requestFingerprint: FINGERPRINT, answer: canonicalJson(ANSWER), + secretDetection: true, ...overrides, }; } +/** + * Resume the suspended run on a fresh acquisition, and hold the execution open. + * + * What a claim needs: an acquisition that began an execution the run has not + * moved past. A settled execution and a bare admitted socket are both proved + * elsewhere in this file to obtain nothing. + */ +async function resumed( + stub: ReturnType, + executionId = "execution-2", +): Promise { + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const begun = await ask(stub, { + id: `begin-${executionId}`, + command: "begin", + runId: RUN_ID, + action: "resume", + creation: null, + retrieval: null, + executionId, + }); + expect(begun["outcome"]).toBe("performed"); +} + describe("delivering an answer to a run's owner", () => { it("answers what the run is waiting at, and retains one value for it", async () => { const stub = executor(); @@ -231,7 +267,7 @@ describe("delivering an answer to a run's owner", () => { requestFingerprint: FINGERPRINT, }); - const retained = await deliver(stub, retention(eventId)); + const retained = await deliver(stub, delivery()); expect(retained).toEqual({ outcome: "performed", value: { runId: RUN_ID, suspensionId: SUSPENSION }, @@ -249,7 +285,7 @@ describe("delivering an answer to a run's owner", () => { it("takes nothing, and leaves a live executor exactly where it was", async () => { const stub = executor(); - const eventId = await suspended(stub); + await suspended(stub); const before = await on(stub, (owner) => ({ run: owner.runRow(), executions: owner.executionRows(), @@ -259,7 +295,7 @@ describe("delivering an answer to a run's owner", () => { acquisition: owner.acquisitionId(), })); - expect(await deliver(stub, retention(eventId))).toMatchObject({ outcome: "performed" }); + expect(await deliver(stub, delivery())).toMatchObject({ outcome: "performed" }); const after = await on(stub, (owner) => ({ run: owner.runRow(), @@ -280,18 +316,18 @@ describe("delivering an answer to a run's owner", () => { it("re-observes one decision when its answer was lost, and refuses a different one", async () => { const stub = executor(); - const eventId = await suspended(stub); - const first = await deliver(stub, retention(eventId)); + await suspended(stub); + const first = await deliver(stub, delivery()); // The same delivery again, after its answer never arrived. - const again = await deliver(stub, retention(eventId)); + const again = await deliver(stub, delivery()); expect(again).toEqual(first); expect(await on(stub, (owner) => owner.retainedAnswers())).toHaveLength(1); // A different value under the same wait is a second answer, not a retry. const conflicting = await deliver( stub, - retention(eventId, { answer: canonicalJson({ approved: false }) }), + delivery({ answer: canonicalJson({ approved: false }) }), ); expect(conflicting).toEqual({ outcome: "refused", @@ -304,7 +340,7 @@ describe("delivering an answer to a run's owner", () => { it("refuses everything it is not, and writes nothing on the way", async () => { const stub = executor(); - const eventId = await suspended(stub); + await suspended(stub); const before = await on(stub, (owner) => ({ answers: owner.retainedAnswers(), journal: owner.journalRecords(), @@ -313,30 +349,38 @@ describe("delivering an answer to a run's owner", () => { const refusals = { wrongWait: await deliver(stub, { operation: "wait", suspensionId: "wait-elsewhere" }), - wrongRequest: await deliver(stub, retention("event-elsewhere")), - wrongFingerprint: await deliver( + wrongRequest: await deliver(stub, delivery({ suspensionId: "wait-elsewhere" })), + // A value the retained schema does not admit, offered on exactly the + // authenticated surface a valid one is offered on. + rejectedValue: await deliver(stub, delivery({ answer: canonicalJson({ approved: "yes" }) })), + // The same surface, with content the credential gate matches. + credential: await deliver( stub, - retention(eventId, { requestFingerprint: "b".repeat(64) }), + delivery({ answer: canonicalJson({ approved: true, note: CANARY }) }), ), + // The gate decision is a required member, so omitting it is not a way of + // making it. + ungated: await deliver(stub, { + operation: "deliver", + suspensionId: SUSPENSION, + answer: canonicalJson(ANSWER), + }), unknownOperation: await deliver(stub, { operation: "publish" }), malformed: await deliver(stub, "{"), // An unauthenticated request is refused before the run is named, so a // body naming nothing this owner holds still refuses for the token. unauthenticatedFirst: await deliver(stub, "{", { token: "not a token" }), - unknownMember: await deliver(stub, { ...retention(eventId), extra: 1 }), - uncanonical: await deliver(stub, retention(eventId, { answer: '{"approved":true,"a":1}' })), - oversized: await deliver( - stub, - retention(eventId, { answer: canonicalJson("x".repeat(2_000_000)) }), - ), + unknownMember: await deliver(stub, { ...delivery(), extra: 1 }), + uncanonical: await deliver(stub, delivery({ answer: '{"approved":true,"a":1}' })), + oversized: await deliver(stub, delivery({ answer: canonicalJson("x".repeat(2_000_000)) })), // The token is deliberately unusable. If the release were checked after // it, the refusal would name the token rather than the build. - badRelease: await deliver(stub, retention(eventId), { + badRelease: await deliver(stub, delivery(), { release: "other-build", token: "not a token", }), - badToken: await deliver(stub, retention(eventId), { token: "not a token" }), - wrongRun: await deliver(stub, retention(eventId), { runId: "9".repeat(52) }), + badToken: await deliver(stub, delivery(), { token: "not a token" }), + wrongRun: await deliver(stub, delivery(), { runId: "9".repeat(52) }), }; for (const [named, answered] of Object.entries(refusals)) { @@ -344,7 +388,9 @@ describe("delivering an answer to a run's owner", () => { } expect(refusals.wrongWait["refusal"]).toBe("command:wrong-suspension"); expect(refusals.wrongRequest["refusal"]).toBe("command:wrong-suspension"); - expect(refusals.wrongFingerprint["refusal"]).toBe("command:stale-journal"); + expect(refusals.rejectedValue["refusal"]).toBe("command:answer-rejected"); + expect(refusals.credential["refusal"]).toBe("command:credential-detected"); + expect(refusals.ungated["refusal"]).toBe("storage:corrupt"); expect(refusals.badRelease["refusal"]).toBe("release:release-mismatch"); expect(String(refusals.unauthenticatedFirst["refusal"]).startsWith("token:")).toBe(true); expect(refusals.wrongRun["refusal"]).toBe("command:wrong-run"); @@ -425,9 +471,8 @@ describe("reaching the delivery plane through the production client", () => { const retained = yield* link.retain({ runId: RUN_ID, suspensionId: SUSPENSION, - requestEventId: waiting.value.requestEventId, - requestFingerprint: waiting.value.requestFingerprint, answer: ANSWER, + secretDetection: true, }); if (!retained.ok) { throw retained.error; @@ -450,7 +495,7 @@ describe("reaching the delivery plane through the production client", () => { expect(rows).toHaveLength(1); expect(rows[0]?.["answer"]).toBe(canonicalJson(ANSWER)); // Two requests, and neither of them opened a socket or named a command. - expect(sent.map((body) => JSON.parse(body)["operation"])).toEqual(["wait", "retain"]); + expect(sent.map((body) => JSON.parse(body)["operation"])).toEqual(["wait", "deliver"]); expect(await on(stub, (owner) => owner.holders())).toBe(1); }); @@ -492,35 +537,94 @@ describe("reaching the delivery plane through the production client", () => { }); describe("spending a retained answer", () => { - it("reads the retained answer back on the acquisition that may spend it", async () => { + it("releases the value only to an acquisition holding the open execution", async () => { const stub = executor(); const eventId = await suspended(stub); - await deliver(stub, retention(eventId)); - - const answered = await ask(stub, { - id: "answer-1", + await deliver(stub, delivery()); + const claim = (id: string) => ({ + id, command: "answer", suspensionId: SUSPENSION, + requestEventId: eventId, + }); + + // The socket that suspended the run is still admitted and its execution is + // settled. It holds no execution, so it is told nothing. + const settled = await ask(stub, claim("answer-settled")); + expect(settled).toEqual({ + id: "answer-settled", + outcome: "refused", + refusal: "command:wrong-execution", + }); + + // A replacement acquisition that has begun nothing is in the same position. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + const admitted = await ask(stub, claim("answer-admitted")); + expect(admitted).toEqual({ + id: "answer-admitted", + outcome: "refused", + refusal: "command:wrong-execution", }); - expect(answered["outcome"]).toBe("performed"); - expect(answered["value"]).toEqual({ + // The execution that resumed the run is the one that may read it. + await resumed(stub); + const claiming = await ask(stub, claim("answer-open")); + expect(claiming["outcome"]).toBe("performed"); + expect(claiming["value"]).toEqual({ suspensionId: SUSPENSION, requestEventId: eventId, requestFingerprint: FINGERPRINT, answer: canonicalJson(ANSWER), state: "pending", }); - // A wait nothing was delivered to answers with nothing rather than failing. + + // And it may read only the wait this run is standing at, named by the + // event that run published its request as. + expect( + ( + await ask(stub, { + id: "answer-elsewhere", + command: "answer", + suspensionId: "wait-elsewhere", + requestEventId: eventId, + }) + )["refusal"], + ).toBe("command:wrong-suspension"); expect( - (await ask(stub, { id: "answer-2", command: "answer", suspensionId: "wait-2" }))["value"], - ).toBe(null); + ( + await ask(stub, { + id: "answer-wrong-event", + command: "answer", + suspensionId: SUSPENSION, + requestEventId: "event-elsewhere", + }) + )["refusal"], + ).toBe("command:wrong-suspension"); + }); + + it("says nothing about a wait nothing was delivered to", async () => { + const stub = executor(); + const eventId = await suspended(stub); + await resumed(stub); + + // The wait exists and is the one this run is standing at; no value is + // retained for it. That is nothing rather than a refusal. + const answered = await ask(stub, { + id: "answer-1", + command: "answer", + suspensionId: SUSPENSION, + requestEventId: eventId, + }); + + expect(answered).toEqual({ id: "answer-1", outcome: "performed", value: null }); }); it("consumes the row and appends its event in one transaction", async () => { const stub = executor(); const eventId = await suspended(stub); - await deliver(stub, retention(eventId)); + await deliver(stub, delivery()); + await resumed(stub); const at = await frontier(stub); const committed = await ask(stub, { @@ -549,7 +653,8 @@ describe("spending a retained answer", () => { it("spends nothing and appends nothing when the two do not agree", async () => { const stub = executor(); const eventId = await suspended(stub); - await deliver(stub, retention(eventId)); + await deliver(stub, delivery()); + await resumed(stub); const at = await frontier(stub); const before = await on(stub, (owner) => ({ answers: owner.retainedAnswers(), @@ -593,6 +698,29 @@ describe("spending a retained answer", () => { requestFingerprint: "c".repeat(64), }, }, + // One matching answer event, and a second one for the same wait carrying + // another value. One retained answer ends one wait. + twoForOneWait: { + events: [answerEvent(), answerEvent(SUSPENSION, { approved: false })], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }, + // One matching answer event, and a second one for another wait entirely. + twoForTwoWaits: { + events: [answerEvent(), answerEvent("wait-2")], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }, + // An answer event no consumption authorizes at all. + unauthorized: { events: [answerEvent()], answer: null }, + // The same, for a wait nothing was ever delivered to. + unauthorizedElsewhere: { events: [answerEvent("wait-2")], answer: null }, }; let attempt = 0; @@ -607,11 +735,17 @@ describe("spending a retained answer", () => { mappings: [], ...proposal, }); - expect([named, refused["outcome"], refused["refusal"]]).toEqual([ - named, - "refused", - "command:answer-unavailable", - ]); + const expected = + named === "twoForOneWait" || + named === "twoForTwoWaits" || + named === "unauthorized" || + named === "unauthorizedElsewhere" || + // A consumption with no answer event at all is the same violation seen + // from the other side: nothing it spends would end anything. + named === "noEvent" + ? "command:answer-unauthorized" + : "command:answer-unavailable"; + expect([named, refused["outcome"], refused["refusal"]]).toEqual([named, "refused", expected]); } // Neither half happened: the row is still pending and no event was kept. @@ -623,10 +757,52 @@ describe("spending a retained answer", () => { ).toEqual(before); }); + it("refuses a consumption from an acquisition that holds no open execution", async () => { + const stub = executor(); + const eventId = await suspended(stub); + await deliver(stub, delivery()); + const at = await frontier(stub); + const before = await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + })); + const spend = (id: string) => ({ + id, + command: "commit", + expectedWorkspaceRootId: at.rootId, + expectedJournalEventId: at.eventId, + publication: null, + mappings: [], + events: [answerEvent()], + answer: { + suspensionId: SUSPENSION, + requestEventId: eventId, + requestFingerprint: FINGERPRINT, + }, + }); + + // The acquisition that suspended the run: still admitted, execution + // settled. A socket is not an execution. + expect((await ask(stub, spend("spend-settled")))["refusal"]).toBe("command:wrong-execution"); + + // A replacement acquisition that has begun nothing. + await on(stub, (owner) => owner.dropConnections()); + await connected(stub); + expect((await ask(stub, spend("spend-admitted")))["refusal"]).toBe("command:wrong-execution"); + + expect( + await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + })), + ).toEqual(before); + }); + it("refuses to spend an answer a second time", async () => { const stub = executor(); const eventId = await suspended(stub); - await deliver(stub, retention(eventId)); + await deliver(stub, delivery()); + await resumed(stub); const at = await frontier(stub); const spend = (id: string, expected: string | null) => ({ id, diff --git a/packages/workflow/tests/remote-delivery.test.ts b/packages/workflow/tests/remote-delivery.test.ts index cea492394..3aa5aab00 100644 --- a/packages/workflow/tests/remote-delivery.test.ts +++ b/packages/workflow/tests/remote-delivery.test.ts @@ -140,12 +140,13 @@ describe("delivering one typed value to a remote run", () => { // The owner was asked what the run is waiting at before anything was sent // for retention, and the retention names what the value was judged against. expect(scripted.asked).toEqual(["wait", "retain"]); + // The retention carries the value and the gate decision and nothing else. + // There is no member saying the value was checked: the owner judges it. expect(scripted.retained[0]).toEqual({ runId: RUN_ID, suspensionId: SUSPENSION, - requestEventId: REQUEST_EVENT, - requestFingerprint: FINGERPRINT, answer: ANSWER, + secretDetection: true, }); }); @@ -183,6 +184,8 @@ describe("delivering one typed value to a remote run", () => { }); expect(retained.ok).toBe(true); expect(opted.retained[0]?.answer).toEqual({ approved: true, note: CANARY }); + // And the choice travels with it, because the owner applies the gate. + expect(opted.retained[0]?.secretDetection).toBe(false); }); it("refuses a request that is not one, before the owner is reached", function* () { diff --git a/packages/workflow/tests/suspension-judgment.test.ts b/packages/workflow/tests/suspension-judgment.test.ts new file mode 100644 index 000000000..2001b9ac8 --- /dev/null +++ b/packages/workflow/tests/suspension-judgment.test.ts @@ -0,0 +1,281 @@ +/** + * Tier WAD — what a run's owner can decide about a delivered value by itself. + * + * A delivered value has to be judged where it is written, and a run's owner + * cannot compile a schema or load a scanner. So both decisions are written in + * the language itself, and what has to be true of them is different in each + * case. + * + * For the schema judgment the claim is parity: on every schema it admits it + * reaches the same verdict as the compiler the document path uses, and it + * admits nothing it cannot judge — a schema using an unimplemented keyword is + * refused outright rather than judged with that constraint quietly skipped. + * + * For the credential gate the claim is narrower and is stated as narrowly: it + * is a floor the owner applies at the write, not a replacement for the scanner + * the runner runs first. What it must do is match the shapes it names, leave + * stand-ins alone, and never report what it matched. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Operation } from "effection"; +import type { Json } from "@executablemd/durable-streams"; +import { prepareElicitation, validateParsed } from "@executablemd/core"; +import { + judgeAgainstSchema, + requireJudgeableSchema, + UnjudgeableSchemaError, +} from "../src/suspension/judgment.ts"; +import { describeCredentials, sightCredentials } from "../src/suspension/credentials.ts"; + +/** One schema, and the values a caller might offer it. */ +interface Case { + readonly name: string; + readonly schema: Json; + readonly values: readonly Json[]; +} + +const CASES: readonly Case[] = [ + { + name: "an approval object", + schema: { + type: "object", + properties: { approved: { type: "boolean" }, note: { type: "string" } }, + required: ["approved"], + additionalProperties: false, + }, + values: [ + { approved: true }, + { approved: true, note: "shipped" }, + { approved: "yes" }, + { note: "shipped" }, + { approved: true, extra: 1 }, + [], + "approved", + null, + ], + }, + { + name: "a bounded string", + schema: { type: "string", minLength: 2, maxLength: 5, pattern: "^[a-z]+$" }, + values: ["ab", "abcde", "a", "abcdef", "AB", "ab1", 12, null], + }, + { + name: "a bounded number", + schema: { type: "number", minimum: 0, exclusiveMaximum: 10, multipleOf: 0.5 }, + values: [0, 9.5, 10, -1, 0.25, 3, true], + }, + { + name: "an integer", + schema: { type: "integer", minimum: 1 }, + values: [1, 0, 1.5, 2, "1"], + }, + { + name: "a list of unique items", + schema: { + type: "array", + items: { type: "string" }, + minItems: 1, + maxItems: 3, + uniqueItems: true, + }, + values: [["a"], ["a", "b"], [], ["a", "a"], ["a", "b", "c", "d"], [1], "a"], + }, + { + name: "a tuple with extra items", + schema: { + type: "array", + items: [{ type: "string" }, { type: "number" }], + additionalItems: false, + minItems: 2, + maxItems: 2, + }, + values: [["a", 1], ["a"], ["a", 1, true], [1, "a"]], + }, + { + name: "an enumeration", + schema: { enum: ["yes", "no", null] }, + values: ["yes", "no", null, "maybe", 1], + }, + { + name: "a constant", + schema: { const: { kind: "approval" } }, + values: [{ kind: "approval" }, { kind: "approval", extra: 1 }, "approval"], + }, + { + name: "alternatives", + schema: { anyOf: [{ type: "string" }, { type: "number", minimum: 0 }] }, + values: ["a", 1, -1, true, null], + }, + { + name: "exactly one alternative", + schema: { + oneOf: [ + { type: "number", minimum: 0 }, + { type: "number", maximum: 10 }, + ], + }, + values: [-1, 11, 5, "a"], + }, + { + name: "everything at once", + schema: { + type: "object", + allOf: [ + { type: "object" }, + { type: "object", properties: { id: { type: "string" } }, required: ["id"] }, + ], + properties: { id: { type: "string", minLength: 1 } }, + }, + values: [{ id: "a" }, { id: "" }, {}, "a"], + }, + { + name: "an exclusion", + schema: { not: { type: "string" } }, + values: ["a", 1, null, {}], + }, + { + name: "a nullable member", + schema: { + type: "object", + properties: { note: { type: ["string", "null"] } }, + additionalProperties: true, + }, + values: [{ note: "a" }, { note: null }, { note: 1 }, { other: 1 }], + }, + { + name: "bounded property counts", + schema: { type: "object", minProperties: 1, maxProperties: 2 }, + values: [{}, { a: 1 }, { a: 1, b: 2 }, { a: 1, b: 2, c: 3 }], + }, + { + name: "constrained property names", + schema: { type: "object", propertyNames: { type: "string", pattern: "^[a-z]+$" } }, + values: [{ ok: 1 }, { NotOk: 1 }, {}], + }, + { + name: "a nested object", + schema: { + type: "object", + properties: { + who: { + type: "object", + properties: { name: { type: "string" } }, + required: ["name"], + additionalProperties: false, + }, + }, + required: ["who"], + }, + values: [{ who: { name: "a" } }, { who: {} }, { who: { name: 1 } }, { who: "a" }, {}], + }, +]; + +/** What the compiler the document path uses says about one value. */ +function* compiled(schema: Json, value: Json): Operation { + const prepared = yield* prepareElicitation(schema, "parity"); + return validateParsed(prepared.validate, value).length === 0; +} + +describe("judging an answer without a compiler", () => { + it("reaches the compiler's verdict on every schema it admits", function* () { + const disagreed: string[] = []; + let judged = 0; + for (const example of CASES) { + requireJudgeableSchema(example.schema); + for (const value of example.values) { + judged += 1; + const expected = yield* compiled(example.schema, value); + const found = judgeAgainstSchema(example.schema, value).length === 0; + if (expected !== found) { + disagreed.push( + `${example.name}: ${JSON.stringify(value)} — compiler ${expected}, judgment ${found}`, + ); + } + } + } + + expect(disagreed).toEqual([]); + // The table is the evidence, so its size is part of the claim. + expect(judged).toBeGreaterThan(70); + }); + + // deno-lint-ignore require-yield + it("refuses a schema whose constraints it cannot apply", function* () { + const unjudgeable: Json[] = [ + { $ref: "#/definitions/other" }, + { if: { type: "string" }, else: { minLength: 1 } }, + { type: "object", patternProperties: { "^a": { type: "string" } } }, + { type: "object", dependencies: { a: ["b"] } }, + { type: "array", contains: { type: "string" } }, + { type: "object", unevaluatedProperties: false }, + { type: "wrong" }, + { type: "object", properties: { a: { $ref: "#" } } }, + { anyOf: [] }, + { pattern: "([" }, + { minLength: -1 }, + { multipleOf: 0 }, + ]; + + for (const schema of unjudgeable) { + let refused: unknown; + try { + requireJudgeableSchema(schema); + } catch (error) { + refused = error; + } + expect([JSON.stringify(schema), refused instanceof UnjudgeableSchemaError]).toEqual([ + JSON.stringify(schema), + true, + ]); + } + + // And an admitted schema stays admitted, so the refusal is about the + // keyword rather than about being cautious. + expect(() => requireJudgeableSchema({ type: "object", title: "fine" })).not.toThrow(); + }); +}); + +describe("the credential gate a run's owner applies at the write", () => { + // deno-lint-ignore require-yield + it("matches issued credentials, and reports only what kind", function* () { + const canary = `ghp_${"abcdefghijklmnopqrstuvwxyz0123456789".slice(0, 36)}`; + const sighted = sightCredentials(`{"note":"${canary}"}`); + + expect(sighted.map((sighting) => sighting.kind)).toEqual(["github-token"]); + // The value never travels with the finding, in any field of it. + expect(JSON.stringify(sighted)).not.toContain(canary); + expect(describeCredentials(sighted)).toBe("github-token"); + expect(describeCredentials(sighted)).not.toContain(canary); + }); + + // deno-lint-ignore require-yield + it("matches a bearer credential and a credential-named field", function* () { + expect( + sightCredentials("Authorization: Bearer abcdefghijklmnopqrstuvwxyz012345").map( + (sighting) => sighting.kind, + ), + ).toContain("bearer-credential"); + expect( + sightCredentials('{"apiKey":"AbCdEf0123456789xyz"}').map((sighting) => sighting.kind), + ).toEqual(["credential-field"]); + expect( + sightCredentials('{"api_key":"AbCdEf0123456789xyz"}').map((sighting) => sighting.kind), + ).toEqual(["credential-field"]); + }); + + // deno-lint-ignore require-yield + it("leaves ordinary answers and stand-ins alone", function* () { + for (const content of [ + '{"approved":true}', + '{"note":"shipped the release"}', + '{"apiKey":"your-api-key-here"}', + '{"password":"example-value"}', + '{"note":"short"}', + "", + ]) { + expect([content, sightCredentials(content)]).toEqual([content, []]); + } + }); +}); From e87db6b3f19cd8e29cd77b4ce7bd9469d6b641d1 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Tue, 8 Sep 2026 14:32:48 -0300 Subject: [PATCH 59/81] =?UTF-8?q?=F0=9F=90=9B=20Run=20the=20real=20credent?= =?UTF-8?q?ial=20gate=20at=20the=20owner,=20and=20stop=20claiming=20the=20?= =?UTF-8?q?schema=20check=20is=20the=20compiler?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate a delivered answer crosses is the one durable journal persistence is written through. The owner was applying a summary of it instead — a pattern list, documented as a floor — and the difference was not theoretical: a credential-named field carrying an opaque value passes the summary and is refused by the configured scanner, so an authenticated caller reaching the owner directly could retain a value the journal itself would not accept. The owner now runs that scanner. It is the recommended preset and this repository's own credential rule, over the same two framings the local host scans, and it needed nothing but a resolution path that does not drag a terminal renderer into a Worker — so `@executablemd/core/secrets` publishes the gate the way `@executablemd/core/canonicalize` publishes the ordering. It runs before the transaction, because it is asynchronous and a Durable Object transaction cannot wait, and what it read is pinned by a request identity the transaction requires to still be the retained one before it writes. A scanner that cannot run at all is a refusal. The explicit opt-out is still the only way past it, and the pattern list is gone rather than kept beside the real thing. The schema check is a different story and this commit tells it plainly rather than papering over it. The settled contract names the local host's compiler, and that compiler builds validators with `new Function`: it performs four code constructions to compile one schema, where the scanner performs none. A deployed Worker does not generate code from strings, and the reason it works under test is that the test pool proxies `globalThis.Function` into an unsafe evaluation binding. So the owner cannot be that compiler without a capability its deployment would have to grant, and this commit does not pretend otherwise: every claim of equivalence is removed from the module, its tests and its comments, and the disagreement is asserted instead — `{ multipleOf: 0.1 }` accepts `0.3` at the owner and refuses it at the compiler. What the owner runs is named for what it is, an additional check that refuses early and refuses more, and the gap it leaves is reported rather than described as closed. One gate is now proved to be one gate: a regression runs the scanner the journal is written through and the gate the owner applies over the same content and requires the same verdict, so a rule or configuration change cannot split them again without saying so. --- packages/core/deno.json | 3 +- packages/core/package.json | 3 +- packages/core/secrets.ts | 20 +++ .../workflow/src/cloudflare/delivery-plane.ts | 67 +++++++--- .../workflow/src/cloudflare/owner-answers.ts | 51 ++++--- .../workflow/src/cloudflare/owner-gate.ts | 48 +++++++ packages/workflow/src/cloudflare/owner.ts | 24 +++- .../workflow/src/suspension/credentials.ts | 125 ------------------ packages/workflow/src/suspension/judgment.ts | 49 +++---- .../cloudflare/remote-delivery.vitest.ts | 69 +++++++++- packages/workflow/tests/delivery-gate.test.ts | 93 +++++++++++++ .../tests/suspension-judgment.test.ts | 90 +++++-------- packages/workflow/tsconfig.cloudflare.json | 2 +- 13 files changed, 391 insertions(+), 253 deletions(-) create mode 100644 packages/core/secrets.ts create mode 100644 packages/workflow/src/cloudflare/owner-gate.ts delete mode 100644 packages/workflow/src/suspension/credentials.ts create mode 100644 packages/workflow/tests/delivery-gate.test.ts diff --git a/packages/core/deno.json b/packages/core/deno.json index 1b62eeae6..7d167db77 100644 --- a/packages/core/deno.json +++ b/packages/core/deno.json @@ -6,7 +6,8 @@ "./canonicalize": "./canonicalize.ts", "./component-name": "./component-name.ts", "./document-target": "./document-target.ts", - "./host": "./host.ts" + "./host": "./host.ts", + "./secrets": "./secrets.ts" }, "imports": { "@effectionx/context-api": "npm:@effectionx/context-api@0.6.0", diff --git a/packages/core/package.json b/packages/core/package.json index f787a04cb..f5da749c6 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -8,7 +8,8 @@ "./canonicalize": "./canonicalize.ts", "./component-name": "./component-name.ts", "./document-target": "./document-target.ts", - "./host": "./host.ts" + "./host": "./host.ts", + "./secrets": "./secrets.ts" }, "dependencies": { "@effectionx/context-api": "0.6.0", diff --git a/packages/core/secrets.ts b/packages/core/secrets.ts new file mode 100644 index 000000000..6c3fe29d7 --- /dev/null +++ b/packages/core/secrets.ts @@ -0,0 +1,20 @@ +/** + * @module + * + * The configured secret gate, for runtimes that cannot load the package root. + * + * `createSecretScanner` is already public from the package root. This subpath + * exists so a consumer can select the gate without loading the root barrel, + * which reaches a terminal renderer and the rest of the host surface — a + * Cloudflare Worker resolving that graph fails to load it at all. The gate + * itself is the recommended Secretlint preset and this repository's own + * credential rule, and it generates no code, so it runs wherever the run's + * owner does. + * + * Same scanner, same rules, same findings. Narrower resolution path. + */ + +export { createSecretScanner } from "./src/secrets/scanner.ts"; +export type { SecretScanner } from "./src/secrets/scanner.ts"; +export { SecretDetectedError } from "./src/secrets/findings.ts"; +export type { SecretFinding } from "./src/secrets/findings.ts"; diff --git a/packages/workflow/src/cloudflare/delivery-plane.ts b/packages/workflow/src/cloudflare/delivery-plane.ts index 1b1e57b4a..81f59a84b 100644 --- a/packages/workflow/src/cloudflare/delivery-plane.ts +++ b/packages/workflow/src/cloudflare/delivery-plane.ts @@ -27,7 +27,7 @@ import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { canonicalJson } from "../storage/record.ts"; import { CommandError } from "./commands.ts"; import { READ_PAGE_BYTES, READ_REQUEST_ENVELOPE } from "./read-plane.ts"; -import { fingerprintOf, readRetainedWait, retainAnswer } from "./owner-answers.ts"; +import { answerFramings, fingerprintOf, readRetainedWait, retainAnswer } from "./owner-answers.ts"; import type { OwnerStorage } from "./storage.ts"; /** What a delivery answered, or why it would not. */ @@ -132,36 +132,67 @@ export function parseDeliveryOperation(raw: string): DeliveryOperation { throw failure("expected an operation this owner implements", "$.operation"); } +/** What one wait retains, and the framings a value offered to it would take. */ +export interface DeliverySubject { + readonly requestFingerprint: string; + /** The retained row and the durable event, as the gate will read them. */ + readonly framings: readonly string[]; +} + +/** Answer one `wait` read, which writes nothing. */ +export function answerRetainedWait( + storage: OwnerStorage, + runId: string, + suspensionId: string, +): Record { + const waiting = readRetainedWait(storage, runId, suspensionId); + return { + runId: waiting.runId, + suspensionId: waiting.suspensionId, + requestEventId: waiting.requestEventId, + request: waiting.request, + responseSchema: waiting.responseSchema, + requestFingerprint: fingerprintOf(waiting), + }; +} + /** - * Answer one delivery request. + * What the credential gate reads, before the transaction that writes. * - * `wait` reads and writes nothing. `retain` runs inside the caller's own owner - * transaction, which is where every fact it depends on is read again. + * The gate is asynchronous and a Durable Object transaction cannot wait, so it + * runs here — over the framings this exact value would be stored in, built from + * the wait as it stands now. The identity those framings were built under + * travels into the transaction, which requires it to still be the one retained + * before it writes anything. */ -export function answerDelivery( +export function deliverySubject( storage: OwnerStorage, runId: string, - request: DeliveryOperation, + request: { readonly suspensionId: string; readonly answer: string }, +): DeliverySubject { + const waiting = readRetainedWait(storage, runId, request.suspensionId); + const fingerprint = fingerprintOf(waiting); + return { + requestFingerprint: fingerprint, + framings: answerFramings(waiting, fingerprint, request.answer), + }; +} + +/** Retain one delivered answer, inside the caller's own owner transaction. */ +export function retainDeliveredAnswer( + storage: OwnerStorage, + runId: string, + request: { readonly suspensionId: string; readonly answer: string }, + gatedFingerprint: string, now: string, ): Record { - if (request.operation === "wait") { - const waiting = readRetainedWait(storage, runId, request.suspensionId); - return { - runId: waiting.runId, - suspensionId: waiting.suspensionId, - requestEventId: waiting.requestEventId, - request: waiting.request, - responseSchema: waiting.responseSchema, - requestFingerprint: fingerprintOf(waiting), - }; - } return retainAnswer( storage, runId, { suspensionId: request.suspensionId, answer: request.answer, - secretDetection: request.secretDetection, + gatedFingerprint, }, now, ); diff --git a/packages/workflow/src/cloudflare/owner-answers.ts b/packages/workflow/src/cloudflare/owner-answers.ts index af05099b9..a0657108f 100644 --- a/packages/workflow/src/cloudflare/owner-answers.ts +++ b/packages/workflow/src/cloudflare/owner-answers.ts @@ -18,7 +18,6 @@ import { readRunRecord, type Row } from "../sqlite/rows.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { parseJsonValue } from "../storage/members.ts"; import { canonicalJson } from "../storage/record.ts"; -import { sightCredentials } from "../suspension/credentials.ts"; import { SUSPENSION_ANSWER, SUSPENSION_REQUEST } from "../suspension/effects.ts"; import { judgeAgainstSchema, requireJudgeableSchema } from "../suspension/judgment.ts"; import { CommandError } from "./commands.ts"; @@ -213,7 +212,16 @@ export function retainAnswer( offered: { readonly suspensionId: string; readonly answer: string; - readonly secretDetection: boolean; + /** + * The fingerprint the credential gate read this wait's request under. + * + * The gate runs before this transaction, because the scanner is + * asynchronous and a Durable Object transaction cannot wait. What makes + * that sound is this: the value is the same bytes either way, and the + * framings the gate read are recomputed from a request identity this + * requires to still be the one retained. + */ + readonly gatedFingerprint: string; }, now: string, ): { readonly runId: string; readonly suspensionId: string } { @@ -221,8 +229,11 @@ export function retainAnswer( const fingerprint = fingerprintOf(waiting); judgeOffered(waiting, offered.answer); - if (offered.secretDetection) { - gateOffered(waiting, fingerprint, offered.answer); + if (fingerprint !== offered.gatedFingerprint) { + // The framings the gate read named this wait's request as it was a moment + // ago. A request that changed since is one this value was neither judged + // for nor scanned against. + throw new CommandError("stale-journal"); } const already = readRetainedAnswer(storage, offered.suspensionId); @@ -254,12 +265,14 @@ export function retainAnswer( } /** - * Judge the offered value against the schema this wait retained. + * Check the offered value against the schema this wait retained. * - * The schema is walked before the value is looked at: one it cannot judge is - * refused rather than partly applied, so a value this accepts is one every - * constraint its wait published was actually checked against. A refusal names - * where the value went wrong and never what it held. + * The additional check, not the settled one. The semantics the contract names + * are the local host's compiler, which a Worker cannot run; what this refuses + * is a subset of what that compiler refuses, on the schemas it admits, and a + * schema whose keywords it does not implement is refused outright rather than + * judged with that constraint quietly skipped. A refusal names where the value + * went wrong and never what it held. */ function judgeOffered(waiting: OwnerRetainedWait, answer: string): void { let value: Json; @@ -283,20 +296,23 @@ function judgeOffered(waiting: OwnerRetainedWait, answer: string): void { } /** - * Cross the credential gate, in both framings the value will be stored in. + * The two framings this value will be stored in, for the gate to read. * - * The retained row and the durable event a later execution would publish, the - * same two the local host scans. What is refused is the kind that was seen; - * neither the value nor the match is recorded or reported. + * The retained row and the durable event a later execution would publish — + * exactly the two the local host scans, in the framing each will have. */ -function gateOffered(waiting: OwnerRetainedWait, fingerprint: string, answer: string): void { +export function answerFramings( + waiting: OwnerRetainedWait, + fingerprint: string, + answer: string, +): string[] { let value: Json; try { value = retained(JSON.parse(answer)); } catch { throw new CommandError("malformed-member"); } - const framings = [ + return [ canonicalJson({ suspensionId: waiting.suspensionId, requestEventId: waiting.requestEventId, @@ -314,11 +330,6 @@ function gateOffered(waiting: OwnerRetainedWait, fingerprint: string, answer: st result: { status: "ok", value }, }), ]; - for (const framing of framings) { - if (sightCredentials(framing).length > 0) { - throw new CommandError("credential-detected"); - } - } } /** diff --git a/packages/workflow/src/cloudflare/owner-gate.ts b/packages/workflow/src/cloudflare/owner-gate.ts new file mode 100644 index 000000000..b22eba82f --- /dev/null +++ b/packages/workflow/src/cloudflare/owner-gate.ts @@ -0,0 +1,48 @@ +/** + * The credential gate a delivered answer crosses, at the owner. + * + * The same gate durable journal persistence is written through — the scanner + * the local host runs, not a summary of it. It runs here because the settled + * contract is that the retained row and the event it may become cross that + * exact gate before either exists, and an authenticated caller reaching this + * owner directly must not be able to decide which checks happened. + * + * It runs outside the transaction because it cannot run inside one: the scanner + * is asynchronous and a Durable Object transaction has to finish before it can + * commit. That is sound rather than convenient — the framings it reads are + * built from the value in the request, which does not change, and from a wait + * identity the transaction requires to still be the one retained before it + * writes anything. + * + * A scanner that could not run at all is a refusal. What must not happen is a + * value entering retained state because the gate failed to say no. + */ + +import type { Operation } from "effection"; +import { createSecretScanner } from "@executablemd/core/secrets"; +import type { SecretFinding } from "@executablemd/core/secrets"; +import { CommandError } from "./commands.ts"; + +/** + * Read every framing, and refuse the value if any of them is a credential. + * + * The scanner is created for this delivery and reclaimed with it, so its + * fingerprints mean nothing outside this call. Nothing about what was matched + * travels: the refusal is one category, and a diagnostic quoting the value or + * the match would publish exactly what the gate exists to keep out. + */ +export function* crossSecretGate(framings: readonly string[]): Operation { + const scanner = createSecretScanner(); + for (const framing of framings) { + let found: SecretFinding[]; + try { + found = yield* scanner.scan(framing); + } catch { + // A gate that could not read this value has not passed it. + throw new CommandError("credential-detected"); + } + if (found.length > 0) { + throw new CommandError("credential-detected"); + } + } +} diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index 92f84d801..758a3f1cc 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -40,7 +40,14 @@ import { type ReadAdmission, type ReadAnswer, } from "./read-plane.ts"; -import { answerDelivery, type DeliveryAnswer, parseDeliveryOperation } from "./delivery-plane.ts"; +import { + answerRetainedWait, + type DeliveryAnswer, + deliverySubject, + parseDeliveryOperation, + retainDeliveredAnswer, +} from "./delivery-plane.ts"; +import { crossSecretGate } from "./owner-gate.ts"; import { dispatchCommand } from "./dispatcher.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { discardPriorAcquisitions, PRIVATE_OBJECT_NAMES } from "./private-schema.ts"; @@ -227,16 +234,27 @@ export abstract class WorkflowOwnerObject extends DurableObject { if (request.operation === "wait") { return { outcome: "performed", - value: answerDelivery(this.owned, runId, request, now), + value: answerRetainedWait(this.owned, runId, request.suspensionId), }; } + + // The gate durable journal persistence is written through, over the two + // framings this value would be stored in. It runs before the transaction + // because it is asynchronous and a Durable Object transaction cannot + // wait; what it read is pinned by the request identity carried into the + // write, which the transaction requires to still be the retained one. + const subject = deliverySubject(this.owned, runId, request); + if (request.secretDetection) { + yield* crossSecretGate(subject.framings); + } + // One transaction, entered here rather than inside the answer, so every // fact the retention depends on is read under the write it is about to // make and a refusal rolls back having written nothing. return { outcome: "performed", value: this.transactions.run(this.owned, () => - answerDelivery(this.owned, runId, request, now), + retainDeliveredAnswer(this.owned, runId, request, subject.requestFingerprint, now), ), }; } catch (error) { diff --git a/packages/workflow/src/suspension/credentials.ts b/packages/workflow/src/suspension/credentials.ts deleted file mode 100644 index f162b1d9d..000000000 --- a/packages/workflow/src/suspension/credentials.ts +++ /dev/null @@ -1,125 +0,0 @@ -/** - * The credential gate a run's owner can apply for itself. - * - * Secret detection belongs where the write happens, for the same reason schema - * judgment does: a caller that decides for itself decides nothing. The full - * scanner is `@secretlint`'s and needs a Node runtime, which a run's owner does - * not have — so the owner applies this, and the runner applies the full scanner - * as well before it offers anything. - * - * This is deliberately a floor and not a replacement. It matches the shapes - * that are unambiguous on sight — issued token formats, a bearer credential, - * and a credential-named field carrying an opaque value — and it matches them - * with the same conservatism the repository's own rules use: a placeholder is - * not a credential, and neither is a short or obviously descriptive value. - * - * What it must never do is report what it matched. A diagnostic quoting the - * value would publish, in a place nothing filters, exactly what the gate exists - * to keep out of retained state. - */ - -/** What was recognized, named by kind and never by content. */ -export interface CredentialSighting { - /** The kind of credential this looks like, as a stable name. */ - readonly kind: string; -} - -/** Issued token formats that are unmistakable on sight. */ -const ISSUED: readonly { readonly kind: string; readonly pattern: RegExp }[] = [ - { kind: "github-token", pattern: /\bgh[pousr]_[A-Za-z0-9]{16,}\b/ }, - { kind: "github-pat", pattern: /\bgithub_pat_[A-Za-z0-9_]{20,}\b/ }, - { kind: "openai-key", pattern: /\bsk-[A-Za-z0-9_-]{20,}\b/ }, - { kind: "aws-access-key-id", pattern: /\b(?:AKIA|ASIA)[0-9A-Z]{16}\b/ }, - { kind: "slack-token", pattern: /\bxox[baprs]-[A-Za-z0-9-]{10,}\b/ }, - { kind: "google-api-key", pattern: /\bAIza[0-9A-Za-z_-]{35}\b/ }, - { kind: "stripe-key", pattern: /\b(?:sk|rk)_live_[0-9A-Za-z]{16,}\b/ }, - { kind: "npm-token", pattern: /\bnpm_[A-Za-z0-9]{36}\b/ }, - { kind: "private-key-block", pattern: /-----BEGIN (?:[A-Z ]+ )?PRIVATE KEY-----/ }, - { - kind: "json-web-token", - pattern: /\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b/, - }, -]; - -/** A bearer credential, whatever issued it. */ -const BEARER = /\bBearer\s+[A-Za-z0-9._~+/-]{20,}={0,2}/i; - -/** - * Field names that carry a credential, as word parts. - * - * The same field arrives under every convention a producer happens to use — - * `apiKey` from JSON, `api_key` from an environment, `api-key` from a header — - * so the names are matched by their parts rather than by one spelling. - */ -const FIELDS = [ - ["api", "key"], - ["access", "token"], - ["refresh", "token"], - ["client", "secret"], - ["session", "token"], - ["secret", "key"], - ["private", "key"], - ["auth", "token"], - ["authorization"], - ["password"], - ["passwd"], - ["apikey"], -] - .map((parts) => parts.join("[-_ ]?")) - .join("|"); - -/** - * A credential-named field carrying an opaque value. - * - * The value has to look like a secret rather than like prose: long enough to be - * one, and drawn from the alphabet issued credentials use. - */ -const NAMED_FIELD = new RegExp(`"?(?:${FIELDS})"?\\s*[:=]\\s*"?([A-Za-z0-9._~+/-]{12,})"?`, "i"); - -/** Words that mark a value as a stand-in rather than a credential. */ -const PLACEHOLDERS = [ - "example", - "placeholder", - "redacted", - "dummy", - "sample", - "changeme", - "todo", - "fixme", - "your", - "xxxx", -]; - -/** - * Everything this recognizes in one piece of content, by kind. - * - * Deduplicated, because a kind seen twice is the same finding about the same - * delivery, and ordered so two runs describe one value the same way. - */ -export function sightCredentials(content: string): CredentialSighting[] { - const found = new Set(); - for (const { kind, pattern } of ISSUED) { - if (pattern.test(content)) { - found.add(kind); - } - } - if (BEARER.test(content)) { - found.add("bearer-credential"); - } - const named = NAMED_FIELD.exec(content); - if (named !== null && !placeholder(named[1] ?? "")) { - found.add("credential-field"); - } - return [...found].toSorted().map((kind) => ({ kind })); -} - -/** Whether a matched value reads as a stand-in rather than as a credential. */ -function placeholder(value: string): boolean { - const lowered = value.toLowerCase(); - return PLACEHOLDERS.some((word) => lowered.includes(word)); -} - -/** What a refusal says, which is the kinds and never the content. */ -export function describeCredentials(sightings: readonly CredentialSighting[]): string { - return sightings.map((sighting) => sighting.kind).join(", "); -} diff --git a/packages/workflow/src/suspension/judgment.ts b/packages/workflow/src/suspension/judgment.ts index 09212ecc2..1b4a58f29 100644 --- a/packages/workflow/src/suspension/judgment.ts +++ b/packages/workflow/src/suspension/judgment.ts @@ -1,33 +1,36 @@ /** - * Judging one answer against the schema its wait retained, anywhere. + * The additional schema check a run's owner can make for itself. * - * A delivered value must satisfy the response schema the wait published before - * it enters durable state, and the place that decides that has to be the place - * that writes — otherwise a caller with delivery admission decides for itself. - * A run's owner is where the write happens, and a run's owner cannot compile a - * schema: `ajv` builds validators with `new Function`, and a Cloudflare Worker - * refuses code generation from strings. So the judgment lives here, in the - * language itself, and both the runner that offers a value and the owner that - * retains it run this exact module. + * The settled contract is that a delivered value satisfies its wait's response + * schema under the semantics the local host uses: `prepareElicitation` compiles + * the schema and `validateParsed` judges the value. That compiler builds + * validators with `new Function`, and a deployed Cloudflare Worker does not + * generate code from strings — so a run's owner cannot run it, and this is what + * the owner runs instead. * - * ## Closed, not lenient + * ## This is not that compiler, and it is not equivalent to it * - * Every keyword this understands is listed. A schema using anything else is not - * judged leniently — it is refused, and the delivery with it. That is the whole - * safety property: an unimplemented constraint can never be silently skipped, - * so a value this accepts is a value every constraint its schema states was - * actually checked against. + * Nothing here claims equivalence and nothing should be read as claiming it. A + * table of agreements is not a proof, and there is a supported schema the two + * disagree about: `{ type: "number", multipleOf: 0.1 }` accepts `0.3` here and + * refuses it there, because the two use different arithmetic for the same + * keyword. The disagreement is asserted in `tests/suspension-judgment.test.ts` + * so it cannot be mistaken for parity. * - * `format` is the one exception and it matches the compiler the document path - * uses, which is configured with `validateFormats: false`: a format annotation - * constrains nothing on either side. + * What this is, is an additional check the owner makes before it writes: on the + * schemas it admits it refuses values the compiler would refuse, and it refuses + * outright any schema whose keywords it does not implement rather than judging + * with that constraint quietly skipped. Making the owner's judgment *the* + * settled semantics needs the compiler to run there, which needs a code + * generation capability a Worker only has if its deployment grants one. * - * ## What it is not + * ## Closed, not lenient * - * Not a JSON Schema implementation, and not a second dialect. It is one - * predicate over a bounded subset, held to the compiler's own verdicts by a - * parity table, and it refuses everything outside that subset rather than - * guessing. + * Every keyword this understands is listed. A schema using anything else is + * refused, and the delivery with it. `format` is the one exception and it + * matches the compiler the document path uses, which is configured with + * `validateFormats: false`: a format annotation constrains nothing on either + * side. */ import type { Json } from "@executablemd/durable-streams"; diff --git a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts index 94c228321..413a1ede1 100644 --- a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts @@ -33,7 +33,13 @@ const SUSPENSION = "wait-1"; const REQUEST = { kind: "approval", release: "1.4" }; const SCHEMA = { type: "object", - properties: { approved: { type: "boolean" }, note: { type: "string" } }, + properties: { + approved: { type: "boolean" }, + note: { type: "string" }, + // Admitted by the schema on purpose: what refuses a credential here has to + // be the gate rather than the shape of the value. + password: { type: "string" }, + }, required: ["approved"], additionalProperties: false, }; @@ -49,6 +55,15 @@ const FINGERPRINT = sha256Hex(canonicalJson({ request: REQUEST, responseSchema: */ const CANARY = `ghp_${"abcdefghijklmnopqrstuvwxyz0123456789".slice(0, 36)}`; +/** + * A safe canary the repository's own credential rule matches. + * + * Not an issued token and not a real secret: a credential-named field carrying + * an opaque-looking value. It is here because it is exactly the shape a weaker + * detector lets through, so it is what proves the owner runs the real gate. + */ +const SAFE_CANARY = "example-Purple7Elephant"; + beforeAll(async () => { keys = await generateKeys(); }); @@ -358,6 +373,13 @@ describe("delivering an answer to a run's owner", () => { stub, delivery({ answer: canonicalJson({ approved: true, note: CANARY }) }), ), + // The shape a weaker detector lets through and the configured gate does + // not. This is the whole difference between running the real gate and + // summarizing it. + safeCanary: await deliver( + stub, + delivery({ answer: canonicalJson({ approved: true, password: SAFE_CANARY }) }), + ), // The gate decision is a required member, so omitting it is not a way of // making it. ungated: await deliver(stub, { @@ -390,6 +412,7 @@ describe("delivering an answer to a run's owner", () => { expect(refusals.wrongRequest["refusal"]).toBe("command:wrong-suspension"); expect(refusals.rejectedValue["refusal"]).toBe("command:answer-rejected"); expect(refusals.credential["refusal"]).toBe("command:credential-detected"); + expect(refusals.safeCanary["refusal"]).toBe("command:credential-detected"); expect(refusals.ungated["refusal"]).toBe("storage:corrupt"); expect(refusals.badRelease["refusal"]).toBe("release:release-mismatch"); expect(String(refusals.unauthenticatedFirst["refusal"]).startsWith("token:")).toBe(true); @@ -404,6 +427,50 @@ describe("delivering an answer to a run's owner", () => { ).toEqual(before); }); + it("runs the configured gate, and lets only the explicit opt-out past it", async () => { + const stub = executor(); + await suspended(stub); + const carrying = canonicalJson({ approved: true, password: SAFE_CANARY }); + const before = await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + executions: owner.executionRows(), + root: owner.currentRootId(), + acquisition: owner.acquisitionId(), + private: owner.scratch(), + })); + + // The same detection the durable journal is written through refuses it, + // and nothing about the run moves. + const gated = await deliver(stub, delivery({ answer: carrying })); + expect(gated["refusal"]).toBe("command:credential-detected"); + // The refusal names a category and nothing else: not the value, not the + // rule, not what was matched. + expect(JSON.stringify(gated)).not.toContain(SAFE_CANARY); + expect( + await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + executions: owner.executionRows(), + root: owner.currentRootId(), + acquisition: owner.acquisitionId(), + private: owner.scratch(), + })), + ).toEqual(before); + + // The documented opt-out is the only path that retains it. + const opted = await deliver(stub, delivery({ answer: carrying, secretDetection: false })); + expect(opted).toEqual({ + outcome: "performed", + value: { runId: RUN_ID, suspensionId: SUSPENSION }, + }); + const rows = await on(stub, (owner) => owner.retainedAnswers()); + expect(rows).toHaveLength(1); + expect(rows[0]?.["answer"]).toBe(carrying); + }); + it("refuses a run that is not waiting, and one that is not here", async () => { const running = executor(); await connected(running); diff --git a/packages/workflow/tests/delivery-gate.test.ts b/packages/workflow/tests/delivery-gate.test.ts new file mode 100644 index 000000000..14d98503d --- /dev/null +++ b/packages/workflow/tests/delivery-gate.test.ts @@ -0,0 +1,93 @@ +/** + * Tier WAD — one credential gate, at both boundaries that write a value. + * + * A delivered answer becomes retained state and then a journal event, and the + * settled contract is that it crosses the same gate durable journal persistence + * crosses before either exists. There are two places that write it — the local + * host's own delivery and a run's owner — and the risk this suite exists for is + * that they drift apart: one of them summarizing the gate, or the configuration + * changing under one and not the other. + * + * So this runs the scanner the journal is written through and the gate the + * owner applies over the same content and requires the same verdict. It is not + * a test of what the rules match; `packages/core` owns that. It is a test that + * there is one gate. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Operation } from "effection"; +import { createSecretScanner } from "@executablemd/core/secrets"; +import { crossSecretGate } from "../src/cloudflare/owner-gate.ts"; +import { CommandError } from "../src/cloudflare/commands.ts"; + +/** + * A safe canary the repository's own credential rule matches. + * + * Not an issued token and not a real secret: a credential-named field carrying + * an opaque-looking value. It is here because it is exactly the shape a weaker + * detector lets through, so it is what a split gate would disagree about. + */ +const SAFE_CANARY = '{"password":"example-Purple7Elephant"}'; + +/** An issued token, assembled at run time so no literal is committed. */ +const ISSUED = `{"note":"ghp_${"abcdefghijklmnopqrstuvwxyz0123456789".slice(0, 36)}"}`; + +/** Whether the scanner the journal is written through refuses this content. */ +function* scanned(content: string): Operation { + return (yield* createSecretScanner().scan(content)).length > 0; +} + +/** Whether the gate a run's owner applies refuses this content. */ +function* gated(content: string): Operation { + try { + yield* crossSecretGate([content]); + return false; + } catch (error) { + if (error instanceof CommandError && error.refusal === "credential-detected") { + return true; + } + throw error; + } +} + +describe("the credential gate a delivered answer crosses", () => { + it("reaches the same verdict at the owner as at the journal", function* () { + const contents = [ + SAFE_CANARY, + ISSUED, + '{"approved":true}', + '{"note":"shipped the release"}', + '{"apiKey":"your-api-key-here"}', + "", + ]; + + const verdicts: { content: string; journal: boolean; owner: boolean }[] = []; + for (const content of contents) { + verdicts.push({ content, journal: yield* scanned(content), owner: yield* gated(content) }); + } + + // One gate: every disagreement here is a value one boundary would write and + // the other would refuse. + expect(verdicts.filter((verdict) => verdict.journal !== verdict.owner)).toEqual([]); + // And the safe canary is one the gate actually refuses, so the agreement + // above is not two detectors both saying nothing. + expect(verdicts.find((verdict) => verdict.content === SAFE_CANARY)?.owner).toBe(true); + expect(verdicts.find((verdict) => verdict.content === ISSUED)?.owner).toBe(true); + expect(verdicts.find((verdict) => verdict.content === '{"approved":true}')?.owner).toBe(false); + }); + + it("refuses every framing it is given, and reports no content", function* () { + let refused: unknown; + try { + yield* crossSecretGate(['{"approved":true}', SAFE_CANARY]); + } catch (error) { + refused = error; + } + + expect(refused instanceof CommandError).toBe(true); + // The refusal is one category. Neither the value nor the match travels. + expect(String(refused)).not.toContain("Purple7Elephant"); + expect(String(refused)).not.toContain("password"); + }); +}); diff --git a/packages/workflow/tests/suspension-judgment.test.ts b/packages/workflow/tests/suspension-judgment.test.ts index 2001b9ac8..bb4017c3b 100644 --- a/packages/workflow/tests/suspension-judgment.test.ts +++ b/packages/workflow/tests/suspension-judgment.test.ts @@ -1,20 +1,22 @@ /** - * Tier WAD — what a run's owner can decide about a delivered value by itself. + * Tier WAD — the owner's additional schema check, and what it is not. * - * A delivered value has to be judged where it is written, and a run's owner - * cannot compile a schema or load a scanner. So both decisions are written in - * the language itself, and what has to be true of them is different in each - * case. + * The settled contract is that a delivered value satisfies its wait's response + * schema under the semantics the local host uses — `prepareElicitation` and + * `validateParsed`, which compile with `new Function`. A deployed Worker does + * not generate code, so a run's owner cannot run that compiler, and this module + * is what it runs instead. * - * For the schema judgment the claim is parity: on every schema it admits it - * reaches the same verdict as the compiler the document path uses, and it - * admits nothing it cannot judge — a schema using an unimplemented keyword is - * refused outright rather than judged with that constraint quietly skipped. + * It is **not** equivalent to that compiler and this file does not claim it is. + * A table of agreements is not a proof of equivalence, and there is at least + * one supported schema they disagree about: `{ type: "number", multipleOf: 0.1 }` + * accepts `0.3` here and refuses it there. That disagreement is asserted below + * rather than avoided, so nothing reads this table as parity. * - * For the credential gate the claim is narrower and is stated as narrowly: it - * is a floor the owner applies at the write, not a replacement for the scanner - * the runner runs first. What it must do is match the shapes it names, leave - * stand-ins alone, and never report what it matched. + * What it is for is refusing early and refusing more: the values it rejects are + * values the compiler rejects too, on the schemas it admits, and a schema whose + * keywords it does not implement is refused outright rather than judged with + * that constraint quietly skipped. */ import { describe, it } from "@executablemd/test-support/bdd"; @@ -27,7 +29,6 @@ import { requireJudgeableSchema, UnjudgeableSchemaError, } from "../src/suspension/judgment.ts"; -import { describeCredentials, sightCredentials } from "../src/suspension/credentials.ts"; /** One schema, and the values a caller might offer it. */ interface Case { @@ -178,8 +179,8 @@ function* compiled(schema: Json, value: Json): Operation { return validateParsed(prepared.validate, value).length === 0; } -describe("judging an answer without a compiler", () => { - it("reaches the compiler's verdict on every schema it admits", function* () { +describe("the owner's additional schema check", () => { + it("agrees with the compiler across this table, which is not equivalence", function* () { const disagreed: string[] = []; let judged = 0; for (const example of CASES) { @@ -197,10 +198,22 @@ describe("judging an answer without a compiler", () => { } expect(disagreed).toEqual([]); - // The table is the evidence, so its size is part of the claim. + // The table's size is part of what it is worth, and its worth is bounded: + // agreement on these cases says nothing about the cases not in it. expect(judged).toBeGreaterThan(70); }); + it("is not equivalent to the compiler, and here is where they differ", function* () { + // A supported schema an ordinary wait can retain, and a value the two + // reach opposite verdicts on. Asserted rather than avoided: this is the + // exact gap that keeps the owner from being the settled authority. + const schema: Json = { type: "number", multipleOf: 0.1 }; + requireJudgeableSchema(schema); + + expect(yield* compiled(schema, 0.3)).toBe(false); + expect(judgeAgainstSchema(schema, 0.3).length === 0).toBe(true); + }); + // deno-lint-ignore require-yield it("refuses a schema whose constraints it cannot apply", function* () { const unjudgeable: Json[] = [ @@ -236,46 +249,3 @@ describe("judging an answer without a compiler", () => { expect(() => requireJudgeableSchema({ type: "object", title: "fine" })).not.toThrow(); }); }); - -describe("the credential gate a run's owner applies at the write", () => { - // deno-lint-ignore require-yield - it("matches issued credentials, and reports only what kind", function* () { - const canary = `ghp_${"abcdefghijklmnopqrstuvwxyz0123456789".slice(0, 36)}`; - const sighted = sightCredentials(`{"note":"${canary}"}`); - - expect(sighted.map((sighting) => sighting.kind)).toEqual(["github-token"]); - // The value never travels with the finding, in any field of it. - expect(JSON.stringify(sighted)).not.toContain(canary); - expect(describeCredentials(sighted)).toBe("github-token"); - expect(describeCredentials(sighted)).not.toContain(canary); - }); - - // deno-lint-ignore require-yield - it("matches a bearer credential and a credential-named field", function* () { - expect( - sightCredentials("Authorization: Bearer abcdefghijklmnopqrstuvwxyz012345").map( - (sighting) => sighting.kind, - ), - ).toContain("bearer-credential"); - expect( - sightCredentials('{"apiKey":"AbCdEf0123456789xyz"}').map((sighting) => sighting.kind), - ).toEqual(["credential-field"]); - expect( - sightCredentials('{"api_key":"AbCdEf0123456789xyz"}').map((sighting) => sighting.kind), - ).toEqual(["credential-field"]); - }); - - // deno-lint-ignore require-yield - it("leaves ordinary answers and stand-ins alone", function* () { - for (const content of [ - '{"approved":true}', - '{"note":"shipped the release"}', - '{"apiKey":"your-api-key-here"}', - '{"password":"example-value"}', - '{"note":"short"}', - "", - ]) { - expect([content, sightCredentials(content)]).toEqual([content, []]); - } - }); -}); diff --git a/packages/workflow/tsconfig.cloudflare.json b/packages/workflow/tsconfig.cloudflare.json index ad86bd059..10bc4647a 100644 --- a/packages/workflow/tsconfig.cloudflare.json +++ b/packages/workflow/tsconfig.cloudflare.json @@ -4,7 +4,7 @@ "module": "ESNext", "moduleResolution": "bundler", "lib": ["ES2023"], - "types": ["@cloudflare/workers-types", "@cloudflare/vitest-plugin/types"], + "types": ["@cloudflare/workers-types", "@cloudflare/vitest-plugin/types", "node"], "strict": true, "allowImportingTsExtensions": true, "allowJs": true, From 4a2ae75e1c418593d0b0e1c5377f1b7851a7d3cb Mon Sep 17 00:00:00 2001 From: Min Kim Date: Tue, 8 Sep 2026 17:28:26 -0300 Subject: [PATCH 60/81] =?UTF-8?q?=E2=9C=A8=20Judge=20an=20elicitation=20re?= =?UTF-8?q?sponse=20with=20one=20implementation,=20everywhere=20it=20is=20?= =?UTF-8?q?judged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A response schema was decided by two different things. The document runtime compiled it; a run's owner could not, because a Worker refuses code generation during a request and a schema learned from a retained wait cannot be compiled before the request that carries it. So the owner ran something else, and the two disagreed — `{ multipleOf: 0.1 }` refused `0.3` on one side and accepted it on the other. A value's verdict depended on where it happened to be judged. Now there is one judgment and it generates no code, so the boundary that writes a value is the boundary that judges it: `` and `xmd prompt` judging a provider's answer, local delivery and artifact delivery judging a delivered one, the remote client before it offers anything, and the owner inside the transaction that retains it. The settled draft-07 verdict is the same at all of them, `0.3` included. Preparation still refuses everything it refused, and one thing it had stopped refusing. `$async`, `__proto__` as a declared name, a reference that leaves the supplied schema, and a keyword draft-07 does not define are all decided in one walk, before a question is rendered or a provider is contacted — and the schema is then validated against the draft-07 meta-schema, which is what the compiler did and what the owner's own subset never could. `format` annotates and constrains nothing, as it did. The public schema surface comes back with it. The subset the owner had been using refused `if`/`then`/`else`, `patternProperties`, `contains` and `dependencies` as unjudgeable; those are draft-07 keywords a document may write, and they are judged again. Props, returns, `` and `` keep the compiler they had: they never run inside an owner, and nothing about them changes. Two assertions moved with the implementation, because failures now read in its words rather than the previous compiler's. Neither was weakened: both still require the issue to name the member that failed and what was expected there. And `enum` and `const` failures are described rather than quoted, because the old text listed the schema's own allowed values and an issue travels further than the document that declared them. --- architecture.md | 11 + deno.lock | 6 + .../cli/tests/workflow-suspension.test.ts | 4 +- packages/core/deno.json | 2 + packages/core/elicitation.ts | 17 + packages/core/package.json | 2 + packages/core/src/draft-07-meta-schema.ts | 107 ++++ packages/core/src/elicit.ts | 105 +--- packages/core/src/elicitation-schema.ts | Bin 0 -> 13708 bytes packages/core/tests/elicit-component.test.ts | 5 +- .../workflow/src/cloudflare/owner-answers.ts | 27 +- .../workflow/src/deno/artifact/records.ts | 4 +- packages/workflow/src/deno/delivery.ts | 3 +- packages/workflow/src/remote/delivery.ts | 47 +- packages/workflow/src/suspension/judgment.ts | 517 ------------------ .../cloudflare/remote-delivery.vitest.ts | 126 ++++- .../workflow/tests/remote-delivery.test.ts | 135 ++++- .../tests/suspension-judgment.test.ts | 251 --------- pnpm-lock.yaml | 8 + specs/workflow-workspace-spec.md | 13 +- 20 files changed, 465 insertions(+), 925 deletions(-) create mode 100644 packages/core/elicitation.ts create mode 100644 packages/core/src/draft-07-meta-schema.ts create mode 100644 packages/core/src/elicitation-schema.ts delete mode 100644 packages/workflow/src/suspension/judgment.ts delete mode 100644 packages/workflow/tests/suspension-judgment.test.ts diff --git a/architecture.md b/architecture.md index 5bf9d7dbb..c47208a65 100644 --- a/architecture.md +++ b/architecture.md @@ -1030,6 +1030,17 @@ installed nearer — may answer the question or refuse it, and in both cases the run it affected never suspended; continuation of a suspended run is retained delivery state and nothing else. +A response schema is judged by one implementation wherever it is judged. The +document runtime judges what a provider returns, a local host judges a delivered +answer, and a run whose storage lives somewhere else judges it there — inside +the transaction that writes it, because a boundary that accepts somebody's +report of a judgment has not made one. That is possible only because the +judgment generates no code: a runtime that refuses code generation during a +request cannot compile a schema it learns from a retained wait, so a validator +that compiles could never be the one every boundary runs. It is draft-07, the +schema is admitted against the draft-07 meta-schema before anything is asked, +and `format` annotates without constraining. + The retained request at the execution's exact current durable position is what authorizes entry. The controller derives the identifier for the position immediately behind the caller's own, requires the presented one to equal it, diff --git a/deno.lock b/deno.lock index c89620201..12113f719 100644 --- a/deno.lock +++ b/deno.lock @@ -41,6 +41,7 @@ "npm:@agentclientprotocol/sdk@1.3.0": "1.3.0_zod@4.4.3", "npm:@babel/core@^7.28.0": "7.29.7", "npm:@babel/preset-react@^7.27.1": "7.29.7_@babel+core@7.29.7", + "npm:@cfworker/json-schema@^4.1.1": "4.1.1", "npm:@cloudflare/vitest-plugin@1.1.3": "1.1.3_@vitest+runner@4.1.11_@vitest+snapshot@4.1.11_vitest@4.1.11__@opentelemetry+api@1.9.1__@types+node@24.13.3__vite@7.3.6___@types+node@24.13.3___tsx@4.23.1_@cloudflare+workers-types@5.20260901.1", "npm:@cloudflare/workers-types@^5.20260831.1": "5.20260901.1", "npm:@durable-streams/client@~0.2.2": "0.2.6", @@ -470,6 +471,9 @@ "@babel/helper-validator-identifier" ] }, + "@cfworker/json-schema@4.1.1": { + "integrity": "sha512-gAmrUZSGtKc3AiBL71iNWxDsyUC5uMaKKGdvzYsBoTW/xi42JQHl7eKV2OYzCUqvc+D2RCcf7EXY2iCyFIk6og==" + }, "@clack/core@1.4.3": { "integrity": "sha512-/kr3UWNtdJfxZtPgDqUOmG2pvwlmcLGheex5yiZKdwbzZJxhV+HMNR9QNmyY5cGwTNV6LrR7Jtp+KjhUAP1qBQ==", "dependencies": [ @@ -4599,6 +4603,7 @@ }, "packages/core": { "dependencies": [ + "npm:@cfworker/json-schema@^4.1.1", "npm:@effectionx/context-api@0.6.0", "npm:@secretlint/core@13.0.4", "npm:@secretlint/profiler@13.0.4", @@ -4618,6 +4623,7 @@ ], "packageJson": { "dependencies": [ + "npm:@cfworker/json-schema@^4.1.1", "npm:@effectionx/context-api@0.6.0", "npm:@effectionx/converge@0.1.4", "npm:@effectionx/fetch@0.2.1", diff --git a/packages/cli/tests/workflow-suspension.test.ts b/packages/cli/tests/workflow-suspension.test.ts index a635d3a97..7723bea85 100644 --- a/packages/cli/tests/workflow-suspension.test.ts +++ b/packages/cli/tests/workflow-suspension.test.ts @@ -1273,7 +1273,9 @@ describe("Tier CKX — a checkpoint a document asked for", () => { ); expect(invalid.exitCode).not.toBe(0); expect(invalid.written.err.join(" ")).toContain("does not satisfy the response schema"); - expect(invalid.written.err.join(" ")).toContain("/proceed must be boolean"); + expect(invalid.written.err.join(" ")).toContain( + '/proceed Instance type "string" is invalid. Expected "boolean"', + ); expect(yield* storageDigest(path)).toEqual(before); const accepted = yield* manage( diff --git a/packages/core/deno.json b/packages/core/deno.json index 7d167db77..950254932 100644 --- a/packages/core/deno.json +++ b/packages/core/deno.json @@ -7,6 +7,7 @@ "./component-name": "./component-name.ts", "./document-target": "./document-target.ts", "./host": "./host.ts", + "./elicitation": "./elicitation.ts", "./secrets": "./secrets.ts" }, "imports": { @@ -15,6 +16,7 @@ "@secretlint/secretlint-rule-preset-recommend": "npm:@secretlint/secretlint-rule-preset-recommend@13.0.4", "@secretlint/types": "npm:@secretlint/types@13.0.4", "acorn": "npm:acorn@^8.16.0", + "@cfworker/json-schema": "npm:@cfworker/json-schema@^4.1.1", "ajv": "npm:ajv@^8.17.1", "gray-matter": "npm:gray-matter@^4.0.3", "magic-string": "npm:magic-string@^0.30.21", diff --git a/packages/core/elicitation.ts b/packages/core/elicitation.ts new file mode 100644 index 000000000..a26820a10 --- /dev/null +++ b/packages/core/elicitation.ts @@ -0,0 +1,17 @@ +/** + * @module + * + * The elicitation response judgment, for runtimes that cannot load the root. + * + * `prepareResponseValidator` is already public from the package root. This + * subpath exists so a consumer can select it without loading the root barrel, + * which reaches a terminal renderer, `node:crypto` and the rest of the host + * surface — a Cloudflare Worker resolving that graph fails to load it at all. + * + * The judgment itself generates no code, so it runs wherever a run's owner + * does, and a response schema receives one verdict whichever boundary asks. + */ + +export { prepareResponseValidator, ResponseSchemaError } from "./src/elicitation-schema.ts"; +export type { ResponseValidator } from "./src/elicitation-schema.ts"; +export type { NormalizedIssue } from "./src/validate.ts"; diff --git a/packages/core/package.json b/packages/core/package.json index f5da749c6..59b4b394d 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -9,9 +9,11 @@ "./component-name": "./component-name.ts", "./document-target": "./document-target.ts", "./host": "./host.ts", + "./elicitation": "./elicitation.ts", "./secrets": "./secrets.ts" }, "dependencies": { + "@cfworker/json-schema": "^4.1.1", "@effectionx/context-api": "0.6.0", "@effectionx/converge": "0.1.4", "@effectionx/fetch": "0.2.1", diff --git a/packages/core/src/draft-07-meta-schema.ts b/packages/core/src/draft-07-meta-schema.ts new file mode 100644 index 000000000..f1b53e80e --- /dev/null +++ b/packages/core/src/draft-07-meta-schema.ts @@ -0,0 +1,107 @@ +/** + * The draft-07 meta-schema, as published. + * + * A response schema is admitted by validating it against this, the way the + * compiler this replaced admitted one with `validateSchema: true`. Carrying it + * here rather than fetching it is the only way a run's owner can admit a schema + * at all: it resolves no references and reaches no network. + * + * Transcribed from . Its own `$id` and + * `$schema` are kept so a schema that declares `"$schema": "…draft-07/schema#"` + * is describing this exact document. + */ + +import type { Json } from "./types.ts"; + +export const DRAFT_07_META_SCHEMA: Json = { + $schema: "http://json-schema.org/draft-07/schema#", + $id: "http://json-schema.org/draft-07/schema#", + title: "Core schema meta-schema", + definitions: { + schemaArray: { type: "array", minItems: 1, items: { $ref: "#" } }, + nonNegativeInteger: { type: "integer", minimum: 0 }, + nonNegativeIntegerDefault0: { + allOf: [{ $ref: "#/definitions/nonNegativeInteger" }, { default: 0 }], + }, + simpleTypes: { + enum: ["array", "boolean", "integer", "null", "number", "object", "string"], + }, + stringArray: { + type: "array", + items: { type: "string" }, + uniqueItems: true, + default: [], + }, + }, + type: ["object", "boolean"], + properties: { + $id: { type: "string", format: "uri-reference" }, + $schema: { type: "string", format: "uri" }, + $ref: { type: "string", format: "uri-reference" }, + $comment: { type: "string" }, + title: { type: "string" }, + description: { type: "string" }, + default: true, + readOnly: { type: "boolean", default: false }, + writeOnly: { type: "boolean", default: false }, + examples: { type: "array", items: true }, + multipleOf: { type: "number", exclusiveMinimum: 0 }, + maximum: { type: "number" }, + exclusiveMaximum: { type: "number" }, + minimum: { type: "number" }, + exclusiveMinimum: { type: "number" }, + maxLength: { $ref: "#/definitions/nonNegativeInteger" }, + minLength: { $ref: "#/definitions/nonNegativeIntegerDefault0" }, + pattern: { type: "string", format: "regex" }, + additionalItems: { $ref: "#" }, + items: { anyOf: [{ $ref: "#" }, { $ref: "#/definitions/schemaArray" }], default: true }, + maxItems: { $ref: "#/definitions/nonNegativeInteger" }, + minItems: { $ref: "#/definitions/nonNegativeIntegerDefault0" }, + uniqueItems: { type: "boolean", default: false }, + contains: { $ref: "#" }, + maxProperties: { $ref: "#/definitions/nonNegativeInteger" }, + minProperties: { $ref: "#/definitions/nonNegativeIntegerDefault0" }, + required: { $ref: "#/definitions/stringArray" }, + additionalProperties: { $ref: "#" }, + definitions: { type: "object", additionalProperties: { $ref: "#" }, default: {} }, + properties: { type: "object", additionalProperties: { $ref: "#" }, default: {} }, + patternProperties: { + type: "object", + additionalProperties: { $ref: "#" }, + propertyNames: { format: "regex" }, + default: {}, + }, + dependencies: { + type: "object", + additionalProperties: { + anyOf: [{ $ref: "#" }, { $ref: "#/definitions/stringArray" }], + }, + }, + propertyNames: { $ref: "#" }, + const: true, + enum: { type: "array", items: true }, + type: { + anyOf: [ + { $ref: "#/definitions/simpleTypes" }, + { + type: "array", + items: { $ref: "#/definitions/simpleTypes" }, + minItems: 1, + uniqueItems: true, + }, + ], + }, + format: { type: "string" }, + contentMediaType: { type: "string" }, + contentEncoding: { type: "string" }, + if: { $ref: "#" }, + // deno-lint-ignore no-thenable + then: { $ref: "#" }, + else: { $ref: "#" }, + allOf: { $ref: "#/definitions/schemaArray" }, + anyOf: { $ref: "#/definitions/schemaArray" }, + oneOf: { $ref: "#/definitions/schemaArray" }, + not: { $ref: "#" }, + }, + default: true, +}; diff --git a/packages/core/src/elicit.ts b/packages/core/src/elicit.ts index 9c1223e64..92d3349c5 100644 --- a/packages/core/src/elicit.ts +++ b/packages/core/src/elicit.ts @@ -19,20 +19,13 @@ * retry belongs in visible Markdown control flow. */ -import type { ValidateFunction } from "ajv"; import type { Operation } from "effection"; import { Elicitation } from "./elicitation-api.ts"; import type { ElicitationRequest } from "./elicitation-api.ts"; -import { - ParseSchemaError, - compileParseSchema, - readParseSchema, - validateParsed, -} from "./components/parse-schema.ts"; +import { prepareResponseValidator } from "./elicitation-schema.ts"; +import type { ResponseValidator } from "./elicitation-schema.ts"; import { parseJson } from "./json.ts"; -import { walkSchema } from "./schema-walk.ts"; -import type { NameKind } from "./schema-walk.ts"; import { SchemaValidationError } from "./validate.ts"; import type { NormalizedIssue } from "./validate.ts"; import type { Json, JsonObject } from "./types.ts"; @@ -48,30 +41,37 @@ export class ElicitValidationError extends SchemaValidationError { } } -/** A compiled question. Nothing has been asked yet. */ +/** A prepared question. Nothing has been asked yet. */ export interface PreparedElicitation { /** Normalized draft-07, as the provider will receive it. */ schema: JsonObject; - validate: ValidateFunction; + /** + * What judges a response against this schema. + * + * The repository's own contract rather than a validator library's type: the + * same judgment runs at every boundary that decides a response, including a + * run's owner, and none of them may depend on which library is underneath. + */ + validator: ResponseValidator; label: string; } /** - * Normalize and compile a question's schema. + * Normalize and admit a question's schema. * * Synchronous and effect-free: it either produces a question that can be asked - * or throws, and a caller that has not yet begun anything can still stop. + * or throws, and a caller that has not yet begun anything can still stop. The + * judgment it prepares is the one every boundary makes — a document's provider + * answer here, a workflow answer delivered locally, and a workflow answer + * retained by a run's owner somewhere else. */ +// deno-lint-ignore require-yield export function* prepareElicitation( schema: Json, label: string = DEFAULT_LABEL, ): Operation { - const declaration = readParseSchema(label, schema); - - refuseUnsupportedNames(label, declaration); - refuseExternalReferences(label, declaration); - - return { schema: declaration, validate: yield* compileParseSchema(label, declaration), label }; + const validator = prepareResponseValidator(label, schema); + return { schema: validator.schema, validator, label }; } /** Ask the configured provider, and judge what it returns. */ @@ -86,7 +86,7 @@ export function* runPreparedElicitation( // back is `unknown` until this boundary has walked it. const response = parseJson(answer); - const issues = validateParsed(prepared.validate, response); + const issues = prepared.validator.judge(response); if (issues.length > 0) { throw new ElicitValidationError(prepared.label, issues); } @@ -104,68 +104,3 @@ export function* elicit(request: { request.message, ); } - -/** - * Refuse `__proto__` where a schema declares it as a name. - * - * Two reasons, and either alone would be enough. A validated response binds - * into the evaluation environment, and a schema is how a document says it - * expects that name — so the safest moment to say the name is unsupported is - * before anyone is asked for a value carrying it. - * - * The other is that the underlying validator loses it. Ajv builds its internal - * tables from schema keys by assignment, so `properties: { "__proto__": … }` - * compiles and then never applies: a response carrying that key is judged as - * though the property had never been declared, and under - * `additionalProperties: false` it is rejected outright. `dependencies` compiles - * and never applies; `required` is refused by strict mode. None of those is a - * failure a document could see or work around. - * - * The same string as *data* — a `const`, an `enum` member, a title, a default — - * is untouched, because nothing reads it as a key. - */ -function refuseUnsupportedNames(label: string, schema: JsonObject): void { - walkSchema(schema, { - subschema() {}, - declaredName(name: string, kind: NameKind, path: string) { - if (name !== "__proto__") { - return; - } - throw new ParseSchemaError( - `<${label} /> schema declares "__proto__" as a ${kind} at ${path}, which is not ` + - "supported: the underlying validator loses that name, so the rule would " + - "silently not apply. Rename it, or carry the value under a different key.", - ); - }, - }); -} - -/** - * Refuse a reference that leaves the document. - * - * Ajv reports an unreachable external reference and a mistyped local pointer - * with the same `can't resolve reference` message, so the two are told apart - * here — by the shape of the reference itself — rather than by reading an error - * string. A local pointer that does not resolve is still Ajv's to report, and - * `compileParseSchema` names it. - * - * `$ref` is read only at real schema positions. An object carrying `$ref` inside - * a `const` or an `enum` member is a JSON value the author wants matched, not a - * reference, and Ajv never resolves it — so neither does this. - */ -function refuseExternalReferences(label: string, schema: JsonObject): void { - walkSchema(schema, { - subschema(subschema: JsonObject, path: string) { - const reference = subschema["$ref"]; - if (typeof reference !== "string" || reference.startsWith("#")) { - return; - } - throw new ParseSchemaError( - `<${label} /> schema references "${reference}" at ${path}, which is outside the ` + - "supplied schema. Only references contained within it resolve; external file " + - "and HTTP(S) references are deferred to #192.", - ); - }, - declaredName() {}, - }); -} diff --git a/packages/core/src/elicitation-schema.ts b/packages/core/src/elicitation-schema.ts new file mode 100644 index 0000000000000000000000000000000000000000..8a65e787787b78b7209f7ebd0eb441de94ca23d1 GIT binary patch literal 13708 zcmb_j?RFcd}{AW)nOB=;YAFtzG?OD>^ABrM0F%4ptgBfIcMi5N% zoIFIHFi(_&UZsGgE&fjIlL1WtFZPQQiMfW zgu<1zTQ_ZRo60XjblnQmbx@&b7QIZ5Cmlyuz6}M%mEDD#|K?IudNJ3zA~e7-?>24o z4su&e+4XH5&)od|y5QsWrptplLG@OX}^u>05tO1^4Ra=dkFK&JYVt@biEF z=YPQv5sJ&&yB3Q014 zRo1u3AJ;ThZ>g%2Om`dSY$rP^shtWH9FBhW&#RgpL(wCJ( zUiu9~TFh)qNJ9qaAQa$3yb$xmPTWmofc!SpNIp6bt52#8Nr@4QrtT2hp}2s0)bJ_X zI-U<>IG=ZaxO(-yyHdbtByY}$7NLYyTjv*xuz?uH`}&y*nxFW%s~2-5{oA)l`L22U zb`EnuSOO4NGwtyJ285#Z!h!+u`3~U#(G=xe>^GaLgm_ggMe|+Q0W1nvw0_w={qifo zlm$@{mcWK*T=)m80-B}qYhcJzTdrlPxpL&~IoIR?SCSW=c~5)7bjUV9P|j$~#L2D^ zg_br<4Do~EDUif%VKFzi)Cs>6+3^BytYUzz#N%m7id*=$<>41;qAhRmlrgR0 z+%<$fiCYCuRkM~ShVPVz;-D^rJO$V-eNwE;4i21&`a@zP0Ro`HLuWCrha-04W#2^s zY$z;fuc&;BjFQ*`95$Pst`_sH;o^1k4jL0(skNDXC=yi8ax%KR5H=D3>j7b--x1U% zVpEtE#zgrltI1yO1EXY_KxmFGJG%1x(W7#`Y1+=+yFX{tdQbqI{PSYTq7&NlpJP*7 zKun&cSGwIM;JsIUx9Ph-)dYP@gwGUk@4or*C)S? z^AwOzM*%tO;-JF41J=aw2dH;8e6;#C$f5ds@0se~Gs>#+FQNE0MnK>2MvB9|x9;86 zS9b?%oSm!3p7OW7;b{`&^1qhIj6wP*-q7Xne?M3WWk#Ll(fJdy7L|q20roDM9#ltx z5HZ8T1Psa{{G%_4j0y1)7;D8rEQI$G3Lr5tx*v@}>2fdI7NWxY4$v8$e!G9{^{_i=@Cs(>hIC}fY!Bq7*bp3EUkD4674$tXa5Kw2wd0g;5wm@mxLA?0qs zRgBj}$tn*DD(TELCNa%I*FY#>5R+1DNQ~qFP)Jx(-r>)gN@`k~EAVP)m*AY)9YdDV zx(tCKh%)B^W8-Y6bLyu7;@*>?$rx zEk*XSi4?0&kv=^Ff#OatMjA?*y?*m-8Li zaY2vvHPEvP+yP+nT=^IZc!LQj(E(>9i=keTzJgo8>NF^6N)K`^;Eh5B`km>O(p+p^ z82YFm2H7%ChE8Nq9k%W&bko=H+2o`wCMZDopv$)}SgCzN4W`2Stt-1K=tnJ{8}j+| zVCnnH268d45GwkU@ciK`7DPM5{9cJ_Ku76=HKdgI{gtV(_V3Gezt$ha`vp9V^7!5O z=^7G_3>iO#ls|_0wp-aGWj+47@hID^t;83F=;NzzyRbH`uv1ANA!_sk<{o{6kHH;B z4}=?n-q^JM{SR%k32ldT$eJ8J;&!7%)aZagox^dUgL|TE6^Fx$upx!57kN&5arZqi z)zpwM(FsGHtj+qN&y>CCXD$gL@m(lNf1O9z)5{vWDkO~#EzO6BT{A9oNq<91CKrS0 zr;h-;^bwT_Bj2D1BwYWIMT(FpRZ4*I0NoYCiI(ohFYXlU4b25+K=OAZT)zUhUbRi# z05!a*?DO2|HD0D>qpgc~L+uOsgkk49UZ?UJ& zdm8RiVn>{fU9;JP@nv7jJ~@@JRPi#HPb+^Ds#H0_c4b460rZ3MQd-5qwr9}o?sLJM zD|uN(CQ&pw@t*s!Kh?V8G(vYBB0__ zFv3u3Sg?<#CV79;cSiW9V`Way+~ljtneG8ZwFfw&!=`pBwKOq|4*uQ>3MNCTY@sPr zQN*Dcb8;_MesJe6QfUAf>o!ghKiALkt`2P+_YWq+*~DyvbROF>?E94qeXZ4fW*o>} zxGQNnn4+gdWao4Z>Io8r#E{I}qqXb!&nl)S#JHG)ErIYPR?ppjq#lvKRrw5D45{4KbkDB7Q8FF1uH3 zoH9xygSq^G|0>Wd%NETzu#JyJ#_KOZ?9yoIwYul{?wq~({5ck{9f-pH^{@JPmQv{l z?xmk&%w&QX#AY^ag(rwck+3&ZaOTr_|G&@rlY3dhgFCsG*}u-Y{Wa8^Jj|H^up;&- z*=2)i7!>GUFtjANU*%2m>o><3@yKuN+X{Jttwqdi?KjgYKc2ehFXUM^#UE|3xHA<5 zgr#^&-(Hp#n2^F?mLh&}?u$IxjN~|BC2%hNz$tAoU6~%cFT|VJNhJRzP28`3l}F8h zlE~Jd0P)lL$vyS?VEn`R+|Uu`G@D}UagM}PJVee}jti<-?!vrmn4+x>#>}Qig@#MY zDD9@@x6m!4Kn;9XAWI1PCVkk{DW(k5VK+@gK(+7DEdqX_$EA%Rm8bImUiC;k9;#TA ze+(lTnzcNdQPNAtqlaw$r36sO5(POD0uqhu zZ6ha4V9yxss1 zmFL|W$HM-OqC~&syHR&9{Upi2yf-6D0g@fgfOr2kJ5j(JC-gOpF7)2=fSGJ-OQ%zi zDMvafeP3dw8$Gn7X-B&}tO9)qy$XN;lb$uP8@!|G0c&6zSty_yn+QhqkfH#&oXC6+ zht3{9r);$?4G!rjYcJeA1tlm}gdEN`=9DEY=B`GMRcu+5=%eUSB!PuVt}RQPdSE!k zVY|fy4|tVrOfj(>+_CJl%dB=`g635Tr9e6^#E3u@P@gj!e0|=A?R}aiEi4TR=mnKHMgcPLVd4MCDQL4#L+-sGjVFFIPQiy8vu{<+Q3YPr&!X##~WNcF#C)a*GHOM9ggjo(cJWK-u;f& z|Fx%OU9H5m%^j(|r+JgQh-#l41%Gp_rP#g;kBLoX^-KdNTOalIz>z~=(q zvqs~+y6|(Xf3($*YWJ~=B0QXx>m%YIBMz*isP8jrFXWEla1eSq+>Id(AZQbx@)Zbe z@q%Ka2Brb25M6q-PL(+5@fR@MCXDFYa;HkUl-`d^N6N>q z*|1FQ6*@rj;q{tyE5_*knq?eGOcFuzHyqvZu$1?)R41N$V6o~?R2IG0q6f^v_GBfv zATyi_Q~*;{tEYR{Z*r>vyL~U$Z1F!%w<}syAx+6LD{utkAqhf}G{&ytlw?Q~nJtJXtnE+4t-1cw1V-bnMS>@| zh=wCQIrvvW=(zD153i1h#nANtD#(#2(NPapxo#+CR>Ev22&y?C2tD252GZf>Bdj$x z+)a_8CZVk9;wV*x_XkMK(a(d5+<5%{`ud0I)hS*!c7e%-qvI=YXhGKFfBdIki)l;` zEQ7kLMBx>B){1i%0e5V8K^%DcDC8rX92srrz|0G!Q&_2OcT^l?vrtO)z;DNi)ZX=} zRBY2#6}$rzXn%=!tw1~u??b6tgYeX!c5aK?U~=(}zUms>?BBg#Q9Wn8}ed-#X%?QPVNyCkmHwcw+7lw0mZz$G%8+(o(07S1I>K z{Q`G<#36E58LlwpjymE)KsodUyrHKlIK<*z6ycm{Q};34fj0y|c@Vr0FNfZMc)RN| zw)M!`LsDpRSl29-QeH)nT~R`TGk1puB$~zeffwTb7DmW!O11#`j%;Sybh#FM<;;T~ zpP42{nVPAM6d7h@;q`z8>iuv$S%~!ej*%1r?#LaX5pnnhTtG@KO(3OW-!XDBCbUEw zp%JCDW?7&nat(EcZLZjlBnc#BXq9SRle?=}*MZ!$!|-N1S{v?D`dD{i2E|^ZVZ&^Kl;lK1aVFTG- zz+H(1YMPN~NWtLsz(gk4+=e*(M_Md2UAh)c#iXlZJ$H0;a9AZRrg@tFN=S}YtvPO? zrE(@Q1ECTI6qEtPmcX9!CoaT;NTUFzHtkKoW`vS*f0e8k446rwp;R~gn*$7}^{+*= zTcb6Sr_LykyqxyubhUVzHgMB~vg4b-hxa|wC81X^qePc-XAsIv)3nKGY}yvey?Wa6 z(9KW>o2Do{iKbC0#dg4qYtY(LQR vpqZ78p^E3$i@JE!yy9Ol{r@$u4tKpe`)RImoGrJe#1O>UQA3IYxG(=78(FVK literal 0 HcmV?d00001 diff --git a/packages/core/tests/elicit-component.test.ts b/packages/core/tests/elicit-component.test.ts index 74e8c8557..dacafb5c3 100644 --- a/packages/core/tests/elicit-component.test.ts +++ b/packages/core/tests/elicit-component.test.ts @@ -370,7 +370,10 @@ describe("Elicit: judging the answer", () => { const result = yield* run(workspace, document("Approve?"), constant({ decision: 7 })); expect(result.failure?.message).toContain(""); - expect(result.failure?.message).toContain('"/decision" must be string'); + // The validator's own words. What matters to a document is that the issue + // names where the value went wrong and what was expected there. + expect(result.failure?.message).toContain('"/decision" Instance type "number" is invalid'); + expect(result.failure?.message).toContain('Expected "string"'); }); /** diff --git a/packages/workflow/src/cloudflare/owner-answers.ts b/packages/workflow/src/cloudflare/owner-answers.ts index a0657108f..ec9e48bd4 100644 --- a/packages/workflow/src/cloudflare/owner-answers.ts +++ b/packages/workflow/src/cloudflare/owner-answers.ts @@ -19,7 +19,8 @@ import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { parseJsonValue } from "../storage/members.ts"; import { canonicalJson } from "../storage/record.ts"; import { SUSPENSION_ANSWER, SUSPENSION_REQUEST } from "../suspension/effects.ts"; -import { judgeAgainstSchema, requireJudgeableSchema } from "../suspension/judgment.ts"; +import { prepareResponseValidator } from "@executablemd/core/elicitation"; +import type { ResponseValidator } from "@executablemd/core/elicitation"; import { CommandError } from "./commands.ts"; import { sha256Hex } from "./encoding.ts"; import { heldExecution } from "./private-schema.ts"; @@ -265,14 +266,17 @@ export function retainAnswer( } /** - * Check the offered value against the schema this wait retained. + * Judge the offered value against the schema this wait retained. * - * The additional check, not the settled one. The semantics the contract names - * are the local host's compiler, which a Worker cannot run; what this refuses - * is a subset of what that compiler refuses, on the schemas it admits, and a - * schema whose keywords it does not implement is refused outright rather than - * judged with that constraint quietly skipped. A refusal names where the value - * went wrong and never what it held. + * The same judgment every other boundary makes. `prepareResponseValidator` is + * what `` prepares, what local delivery judges with and what the remote + * client judges with, and it generates no code — so the verdict here is the + * verdict there, for the same schema and the same value, rather than an + * approximation of it. + * + * A schema that cannot be admitted at all is its own refusal: retaining a value + * against a schema nothing could judge would be retaining an unjudged one. A + * rejected value's refusal names neither the value nor what was wrong with it. */ function judgeOffered(waiting: OwnerRetainedWait, answer: string): void { let value: Json; @@ -283,14 +287,13 @@ function judgeOffered(waiting: OwnerRetainedWait, answer: string): void { } catch { throw new CommandError("malformed-member"); } + let validator: ResponseValidator; try { - requireJudgeableSchema(schema); + validator = prepareResponseValidator("workflow answer", schema); } catch { - // The wait published a schema this build cannot judge an answer against. - // Retaining a value it could not check would be retaining an unjudged one. throw new CommandError("unjudgeable-schema"); } - if (judgeAgainstSchema(schema, value).length > 0) { + if (validator.judge(value).length > 0) { throw new CommandError("answer-rejected"); } } diff --git a/packages/workflow/src/deno/artifact/records.ts b/packages/workflow/src/deno/artifact/records.ts index 4752e1620..af2f1e6ae 100644 --- a/packages/workflow/src/deno/artifact/records.ts +++ b/packages/workflow/src/deno/artifact/records.ts @@ -28,7 +28,7 @@ import { createHash } from "node:crypto"; import { Buffer } from "node:buffer"; import type { Operation } from "effection"; -import { prepareElicitation, validateParsed } from "@executablemd/core"; +import { prepareElicitation } from "@executablemd/core"; import { AGENT_PROMPT, parsePromptRecord } from "@executablemd/core/host"; import type { PromptRecord } from "@executablemd/core/host"; import { parseDurableEvent } from "@executablemd/durable-streams"; @@ -1695,7 +1695,7 @@ function* judgeRetainedAnswer( let issues; try { const prepared = yield* prepareElicitation(wait.responseSchema, "artifact answer"); - issues = validateParsed(prepared.validate, answer); + issues = prepared.validator.judge(answer); } catch { reject("a retained response schema cannot judge the answer stored against it"); } diff --git a/packages/workflow/src/deno/delivery.ts b/packages/workflow/src/deno/delivery.ts index 70f640ad5..6511d7b74 100644 --- a/packages/workflow/src/deno/delivery.ts +++ b/packages/workflow/src/deno/delivery.ts @@ -30,7 +30,6 @@ import { prepareElicitation, SecretDetectedError, type SecretFinding, - validateParsed, } from "@executablemd/core"; import { serializeDurableEvent } from "@executablemd/durable-streams"; import type { DurableEvent } from "@executablemd/durable-streams"; @@ -254,7 +253,7 @@ function* judgeAnswer( let issues; try { const prepared = yield* prepareElicitation(waiting.request.responseSchema, "workflow answer"); - issues = validateParsed(prepared.validate, value); + issues = prepared.validator.judge(value); } catch (error) { return Err( new WorkflowAnswerDeliveryError( diff --git a/packages/workflow/src/remote/delivery.ts b/packages/workflow/src/remote/delivery.ts index f638dc0d5..664ece350 100644 --- a/packages/workflow/src/remote/delivery.ts +++ b/packages/workflow/src/remote/delivery.ts @@ -34,7 +34,6 @@ import { prepareElicitation, SecretDetectedError, type SecretFinding, - validateParsed, } from "@executablemd/core"; import { serializeDurableEvent } from "@executablemd/durable-streams"; import type { DurableEvent } from "@executablemd/durable-streams"; @@ -45,11 +44,6 @@ import { type WorkflowSuspensionRequest, } from "../suspension/api.ts"; import type { RemoteDeliveryLink, RemoteRetainedWaitRecord } from "./answer-link.ts"; -import { - describeJudgment, - judgeAgainstSchema, - requireJudgeableSchema, -} from "../suspension/judgment.ts"; import { parseAnswerDelivery, type WorkflowAnswerDelivery, @@ -118,14 +112,6 @@ function* deliverRemotely( return judged; } - // The judgment the owner will make, made here too. A schema the owner cannot - // judge is refused before a value is offered against it, and a disagreement - // between the compiler and the shared judgment stops here. - const shared = judgeShared(waiting.value, suspensionId, value); - if (!shared.ok) { - return shared; - } - if (secretDetection) { const scanned = yield* scanDelivery(waiting.value, suspensionId, value); if (!scanned.ok) { @@ -215,7 +201,7 @@ function* judgeAnswer( let issues; try { const prepared = yield* prepareElicitation(waiting.request.responseSchema, "workflow answer"); - issues = validateParsed(prepared.validate, value); + issues = prepared.validator.judge(value); } catch (error) { return Err( new WorkflowAnswerDeliveryError( @@ -240,37 +226,6 @@ function* judgeAnswer( ); } -/** - * The same judgment the owner makes, made before anything is offered. - * - * The owner refuses a schema it cannot judge rather than retaining a value it - * could not check, so a wait whose schema is outside the judged subset is - * reported here — where a caller can be told what happened — instead of - * arriving as a bare refusal from somewhere else. - */ -function judgeShared(waiting: RemoteRetainedWait, suspensionId: string, value: Json): Result { - try { - requireJudgeableSchema(waiting.request.responseSchema); - } catch (error) { - return Err( - new WorkflowAnswerDeliveryError( - `the response schema retained for ${suspensionId} is not one an answer can be judged ` + - `against: ${error instanceof Error ? error.message : String(error)}`, - ), - ); - } - const issues = judgeAgainstSchema(waiting.request.responseSchema, value); - if (issues.length === 0) { - return Ok(); - } - return Err( - new WorkflowAnswerDeliveryError( - `the value offered to ${suspensionId} does not satisfy the response schema that wait ` + - `retained: ${describeJudgment(issues)}.`, - ), - ); -} - /** * Cross the same gate a durable event crosses, before anything is retained. * diff --git a/packages/workflow/src/suspension/judgment.ts b/packages/workflow/src/suspension/judgment.ts deleted file mode 100644 index 1b4a58f29..000000000 --- a/packages/workflow/src/suspension/judgment.ts +++ /dev/null @@ -1,517 +0,0 @@ -/** - * The additional schema check a run's owner can make for itself. - * - * The settled contract is that a delivered value satisfies its wait's response - * schema under the semantics the local host uses: `prepareElicitation` compiles - * the schema and `validateParsed` judges the value. That compiler builds - * validators with `new Function`, and a deployed Cloudflare Worker does not - * generate code from strings — so a run's owner cannot run it, and this is what - * the owner runs instead. - * - * ## This is not that compiler, and it is not equivalent to it - * - * Nothing here claims equivalence and nothing should be read as claiming it. A - * table of agreements is not a proof, and there is a supported schema the two - * disagree about: `{ type: "number", multipleOf: 0.1 }` accepts `0.3` here and - * refuses it there, because the two use different arithmetic for the same - * keyword. The disagreement is asserted in `tests/suspension-judgment.test.ts` - * so it cannot be mistaken for parity. - * - * What this is, is an additional check the owner makes before it writes: on the - * schemas it admits it refuses values the compiler would refuse, and it refuses - * outright any schema whose keywords it does not implement rather than judging - * with that constraint quietly skipped. Making the owner's judgment *the* - * settled semantics needs the compiler to run there, which needs a code - * generation capability a Worker only has if its deployment grants one. - * - * ## Closed, not lenient - * - * Every keyword this understands is listed. A schema using anything else is - * refused, and the delivery with it. `format` is the one exception and it - * matches the compiler the document path uses, which is configured with - * `validateFormats: false`: a format annotation constrains nothing on either - * side. - */ - -import type { Json } from "@executablemd/durable-streams"; -import { canonicalJson } from "../storage/record.ts"; - -/** Why one value is not an answer to one wait. */ -export interface JudgmentIssue { - /** Where in the value, as a JSON pointer fragment. */ - readonly at: string; - /** What is wrong with it, in the schema's own vocabulary. */ - readonly reason: string; -} - -/** A schema this module will not judge a value against. */ -export class UnjudgeableSchemaError extends Error { - override name = "UnjudgeableSchemaError"; -} - -/** Keywords that say nothing about whether a value is admitted. */ -const ANNOTATIONS = new Set([ - "$schema", - "$comment", - "title", - "description", - "examples", - "default", - "readOnly", - "writeOnly", - "deprecated", - // Configured off in the document path's compiler, so it constrains nothing - // there either. Accepting it and applying nothing is agreement, not leniency. - "format", - // Inert without `$ref`, which is refused, so a definition nothing can reach - // constrains nothing. - "definitions", - "$defs", -]); - -/** Keywords this understands, beyond the ones that say nothing. */ -const SUPPORTED = new Set([ - "type", - "enum", - "const", - "properties", - "required", - "additionalProperties", - "minProperties", - "maxProperties", - "propertyNames", - "items", - "additionalItems", - "minItems", - "maxItems", - "uniqueItems", - "minLength", - "maxLength", - "pattern", - "minimum", - "maximum", - "exclusiveMinimum", - "exclusiveMaximum", - "multipleOf", - "allOf", - "anyOf", - "oneOf", - "not", -]); - -const TYPES = new Set(["null", "boolean", "object", "array", "number", "integer", "string"]); - -/** - * Whether this schema is one this module can judge a value against. - * - * Walked whole, before any value is looked at, so a schema carrying a keyword - * this does not implement is refused rather than partly applied. Raises with - * what it could not read; the caller decides what to say about it. - */ -export function requireJudgeableSchema(schema: Json, at = "#"): void { - if (typeof schema === "boolean") { - return; - } - if (schema === null || typeof schema !== "object" || Array.isArray(schema)) { - throw new UnjudgeableSchemaError(`the schema at ${at} is not a schema object`); - } - for (const [keyword, value] of Object.entries(schema)) { - if (ANNOTATIONS.has(keyword)) { - continue; - } - if (!SUPPORTED.has(keyword)) { - throw new UnjudgeableSchemaError(`the schema at ${at} uses ${keyword}, which is not judged`); - } - walkKeyword(keyword, value, at); - } -} - -function walkKeyword(keyword: string, value: Json, at: string): void { - if (keyword === "type") { - const named = Array.isArray(value) ? value : [value]; - for (const name of named) { - if (typeof name !== "string" || !TYPES.has(name)) { - throw new UnjudgeableSchemaError(`the schema at ${at} names a type this does not know`); - } - } - return; - } - if (keyword === "properties" || keyword === "propertyNames") { - if (keyword === "propertyNames") { - requireJudgeableSchema(value, `${at}/propertyNames`); - return; - } - if (value === null || typeof value !== "object" || Array.isArray(value)) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares malformed properties`); - } - for (const [name, member] of Object.entries(value)) { - requireJudgeableSchema(member, `${at}/properties/${name}`); - } - return; - } - if (keyword === "additionalProperties" || keyword === "additionalItems" || keyword === "not") { - requireJudgeableSchema(value, `${at}/${keyword}`); - return; - } - if (keyword === "items") { - if (Array.isArray(value)) { - value.forEach((member, index) => requireJudgeableSchema(member, `${at}/items/${index}`)); - return; - } - requireJudgeableSchema(value, `${at}/items`); - return; - } - if (keyword === "allOf" || keyword === "anyOf" || keyword === "oneOf") { - if (!Array.isArray(value) || value.length === 0) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed ${keyword}`); - } - value.forEach((member, index) => requireJudgeableSchema(member, `${at}/${keyword}/${index}`)); - return; - } - if (keyword === "required") { - if (!Array.isArray(value) || value.some((name) => typeof name !== "string")) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed required`); - } - return; - } - if (keyword === "enum") { - if (!Array.isArray(value) || value.length === 0) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed enum`); - } - return; - } - if (keyword === "pattern") { - if (typeof value !== "string" || !compiled(value)) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a pattern this cannot read`); - } - return; - } - if ( - keyword === "minProperties" || - keyword === "maxProperties" || - keyword === "minItems" || - keyword === "maxItems" || - keyword === "minLength" || - keyword === "maxLength" - ) { - if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed ${keyword}`); - } - return; - } - if ( - keyword === "minimum" || - keyword === "maximum" || - keyword === "exclusiveMinimum" || - keyword === "exclusiveMaximum" || - keyword === "multipleOf" - ) { - if (typeof value !== "number" || !Number.isFinite(value)) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed ${keyword}`); - } - if (keyword === "multipleOf" && value <= 0) { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed multipleOf`); - } - return; - } - if (keyword === "uniqueItems") { - if (typeof value !== "boolean") { - throw new UnjudgeableSchemaError(`the schema at ${at} declares a malformed uniqueItems`); - } - return; - } - // `const` admits any JSON value, and nothing about it can be malformed. -} - -function compiled(pattern: string): boolean { - try { - expression(pattern); - return true; - } catch { - return false; - } -} - -function expression(pattern: string): RegExp { - try { - return new RegExp(pattern, "u"); - } catch { - return new RegExp(pattern); - } -} - -/** - * Judge one value against one schema, and say what is wrong with it. - * - * The schema must already have been admitted by `requireJudgeableSchema`. An - * empty result is the value satisfying every constraint the schema states. - */ -export function judgeAgainstSchema(schema: Json, value: Json, at = ""): JudgmentIssue[] { - const issues: JudgmentIssue[] = []; - judge(schema, value, at === "" ? "" : at, issues); - return issues; -} - -function judge(schema: Json, value: Json, at: string, issues: JudgmentIssue[]): void { - if (typeof schema === "boolean") { - if (!schema) { - issues.push({ at, reason: "is not admitted here" }); - } - return; - } - if (schema === null || typeof schema !== "object" || Array.isArray(schema)) { - issues.push({ at, reason: "has no schema to be judged by" }); - return; - } - - const named = schema["type"]; - if (named !== undefined) { - const types = Array.isArray(named) ? named : [named]; - if (!types.some((type) => typeof type === "string" && isType(type, value))) { - issues.push({ at, reason: `must be ${types.join(" or ")}` }); - return; - } - } - - const constant = schema["const"]; - if (constant !== undefined && !same(constant, value)) { - issues.push({ at, reason: "must equal the constant this schema declares" }); - } - - const admitted = schema["enum"]; - if (Array.isArray(admitted) && !admitted.some((entry) => same(entry, value))) { - issues.push({ at, reason: "must be one of the values this schema enumerates" }); - } - - judgeCombinators(schema, value, at, issues); - - if (typeof value === "string") { - judgeString(schema, value, at, issues); - } - if (typeof value === "number") { - judgeNumber(schema, value, at, issues); - } - if (Array.isArray(value)) { - judgeArray(schema, value, at, issues); - } else if (value !== null && typeof value === "object") { - judgeObject(schema, value, at, issues); - } -} - -function judgeCombinators( - schema: Record, - value: Json, - at: string, - issues: JudgmentIssue[], -): void { - const all = schema["allOf"]; - if (Array.isArray(all)) { - for (const member of all) { - judge(member, value, at, issues); - } - } - const any = schema["anyOf"]; - if ( - Array.isArray(any) && - !any.some((member) => judgeAgainstSchema(member, value, at).length === 0) - ) { - issues.push({ at, reason: "must satisfy one of the alternatives this schema allows" }); - } - const one = schema["oneOf"]; - if (Array.isArray(one)) { - const matched = one.filter((member) => judgeAgainstSchema(member, value, at).length === 0); - if (matched.length !== 1) { - issues.push({ - at, - reason: "must satisfy exactly one of the alternatives this schema allows", - }); - } - } - const refused = schema["not"]; - if (refused !== undefined && judgeAgainstSchema(refused, value, at).length === 0) { - issues.push({ at, reason: "must not be what this schema excludes" }); - } -} - -function judgeString( - schema: Record, - value: string, - at: string, - issues: JudgmentIssue[], -): void { - const length = [...value].length; - const least = schema["minLength"]; - if (typeof least === "number" && length < least) { - issues.push({ at, reason: `must be at least ${least} characters` }); - } - const most = schema["maxLength"]; - if (typeof most === "number" && length > most) { - issues.push({ at, reason: `must be at most ${most} characters` }); - } - const pattern = schema["pattern"]; - if (typeof pattern === "string" && !expression(pattern).test(value)) { - issues.push({ at, reason: "must match the pattern this schema declares" }); - } -} - -function judgeNumber( - schema: Record, - value: number, - at: string, - issues: JudgmentIssue[], -): void { - const least = schema["minimum"]; - if (typeof least === "number" && value < least) { - issues.push({ at, reason: `must be at least ${least}` }); - } - const most = schema["maximum"]; - if (typeof most === "number" && value > most) { - issues.push({ at, reason: `must be at most ${most}` }); - } - const above = schema["exclusiveMinimum"]; - if (typeof above === "number" && value <= above) { - issues.push({ at, reason: `must be greater than ${above}` }); - } - const below = schema["exclusiveMaximum"]; - if (typeof below === "number" && value >= below) { - issues.push({ at, reason: `must be less than ${below}` }); - } - const multiple = schema["multipleOf"]; - if (typeof multiple === "number") { - const quotient = value / multiple; - if (!Number.isFinite(quotient) || Math.abs(quotient - Math.round(quotient)) > 1e-9) { - issues.push({ at, reason: `must be a multiple of ${multiple}` }); - } - } -} - -function judgeArray( - schema: Record, - value: readonly Json[], - at: string, - issues: JudgmentIssue[], -): void { - const least = schema["minItems"]; - if (typeof least === "number" && value.length < least) { - issues.push({ at, reason: `must have at least ${least} items` }); - } - const most = schema["maxItems"]; - if (typeof most === "number" && value.length > most) { - issues.push({ at, reason: `must have at most ${most} items` }); - } - if (schema["uniqueItems"] === true) { - const seen = new Set(value.map((entry) => canonicalJson(entry))); - if (seen.size !== value.length) { - issues.push({ at, reason: "must not repeat an item" }); - } - } - - const items = schema["items"]; - if (Array.isArray(items)) { - value.forEach((entry, index) => { - const member = items[index]; - if (member !== undefined) { - judge(member, entry, `${at}/${index}`, issues); - return; - } - const extra = schema["additionalItems"]; - if (extra !== undefined) { - judge(extra, entry, `${at}/${index}`, issues); - } - }); - return; - } - if (items !== undefined) { - value.forEach((entry, index) => judge(items, entry, `${at}/${index}`, issues)); - } -} - -function judgeObject( - schema: Record, - value: Record, - at: string, - issues: JudgmentIssue[], -): void { - const names = Object.keys(value); - const least = schema["minProperties"]; - if (typeof least === "number" && names.length < least) { - issues.push({ at, reason: `must have at least ${least} properties` }); - } - const most = schema["maxProperties"]; - if (typeof most === "number" && names.length > most) { - issues.push({ at, reason: `must have at most ${most} properties` }); - } - - const required = schema["required"]; - if (Array.isArray(required)) { - for (const name of required) { - if (typeof name === "string" && !Object.hasOwn(value, name)) { - issues.push({ at, reason: `must have the property ${JSON.stringify(name)}` }); - } - } - } - - const properties = schema["properties"]; - const declared = - properties !== null && typeof properties === "object" && !Array.isArray(properties) - ? properties - : undefined; - - const nameSchema = schema["propertyNames"]; - if (nameSchema !== undefined) { - for (const name of names) { - judge(nameSchema, name, `${at}/${name}`, issues); - } - } - - for (const name of names) { - const member = declared?.[name]; - if (member !== undefined) { - judge(member, value[name] ?? null, `${at}/${name}`, issues); - continue; - } - const extra = schema["additionalProperties"]; - if (extra === undefined) { - continue; - } - judge(extra, value[name] ?? null, `${at}/${name}`, issues); - } -} - -function isType(type: string, value: Json): boolean { - if (type === "null") { - return value === null; - } - if (type === "boolean") { - return typeof value === "boolean"; - } - if (type === "string") { - return typeof value === "string"; - } - if (type === "number") { - return typeof value === "number"; - } - if (type === "integer") { - return typeof value === "number" && Number.isInteger(value); - } - if (type === "array") { - return Array.isArray(value); - } - return value !== null && typeof value === "object" && !Array.isArray(value); -} - -/** Whether two retained values are the same value, by canonical encoding. */ -function same(left: Json, right: Json): boolean { - try { - return canonicalJson(left) === canonicalJson(right); - } catch { - return false; - } -} - -/** What a refusal says about a value, without repeating the value. */ -export function describeJudgment(issues: readonly JudgmentIssue[]): string { - return issues - .map((issue) => `${issue.at === "" ? "the value" : issue.at} ${issue.reason}`) - .join("; "); -} diff --git a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts index 413a1ede1..2fa51d601 100644 --- a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts @@ -136,7 +136,7 @@ async function deliver( return JSON.parse(answered); } -function requestEvent(suspensionId = SUSPENSION): string { +function requestEvent(suspensionId = SUSPENSION, responseSchema: Json = SCHEMA): string { return serializeDurableEvent({ type: "yield", coroutineId: "root", @@ -145,7 +145,7 @@ function requestEvent(suspensionId = SUSPENSION): string { name: suspensionId, suspensionId, request: REQUEST, - responseSchema: SCHEMA, + responseSchema, }, result: { status: "ok", value: null }, }); @@ -179,6 +179,7 @@ async function frontier( async function suspended( stub: ReturnType, suspensionId = SUSPENSION, + responseSchema: Json = SCHEMA, ): Promise { await connected(stub); const begun = await ask(stub, { @@ -200,7 +201,7 @@ async function suspended( expectedJournalEventId: at.eventId, publication: null, mappings: [], - events: [requestEvent(suspensionId)], + events: [requestEvent(suspensionId, responseSchema)], answer: null, }); expect(committed["outcome"]).toBe("performed"); @@ -603,6 +604,125 @@ describe("reaching the delivery plane through the production client", () => { }); }); +describe("judging a delivered value at the owner", () => { + it("reaches the settled draft-07 verdict, including the arithmetic one", async () => { + const stub = executor(); + // A wait whose schema the compiler this replaced and the shared validator + // disagreed about. Draft-07 says 0.3 is a multiple of 0.1, and that is the + // verdict every boundary now reaches. + await suspended(stub, SUSPENSION, { type: "number", multipleOf: 0.1 }); + + const accepted = await deliver( + stub, + delivery({ answer: canonicalJson(0.3), secretDetection: false }), + ); + expect(accepted).toEqual({ + outcome: "performed", + value: { runId: RUN_ID, suspensionId: SUSPENSION }, + }); + expect((await on(stub, (owner) => owner.retainedAnswers()))[0]?.["answer"]).toBe("0.3"); + }); + + it("refuses a value the retained schema does not admit, and writes nothing", async () => { + const stub = executor(); + await suspended(stub, SUSPENSION, { type: "number", multipleOf: 0.1 }); + const before = await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + executions: owner.executionRows(), + private: owner.scratch(), + })); + + const refused = await deliver( + stub, + delivery({ answer: canonicalJson("not a number"), secretDetection: false }), + ); + + expect(refused["refusal"]).toBe("command:answer-rejected"); + expect( + await on(stub, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + executions: owner.executionRows(), + private: owner.scratch(), + })), + ).toEqual(before); + }); + + it("resolves a self-contained reference, and refuses one that leaves", async () => { + const contained = executor(); + await suspended(contained, SUSPENSION, { + definitions: { decision: { type: "string", enum: ["approve", "reject"] } }, + type: "object", + properties: { decision: { $ref: "#/definitions/decision" } }, + required: ["decision"], + }); + + expect( + ( + await deliver( + contained, + delivery({ answer: canonicalJson({ decision: "approve" }), secretDetection: false }), + ) + )["outcome"], + ).toBe("performed"); + + const leaving = executor(); + await suspended(leaving, SUSPENSION, { + type: "object", + properties: { decision: { $ref: "other.json#/x" } }, + }); + const refused = await deliver( + leaving, + delivery({ answer: canonicalJson({ decision: "approve" }), secretDetection: false }), + ); + + // A schema no answer can be judged against is refused rather than retained + // on weaker terms. + expect(refused["refusal"]).toBe("command:unjudgeable-schema"); + expect(await on(leaving, (owner) => owner.retainedAnswers())).toEqual([]); + }); + + it("judges without generating code, on the path a delivery actually takes", async () => { + const stub = executor(); + await suspended(stub, SUSPENSION, { type: "number", multipleOf: 0.1 }); + const presented = await token(); + const named = { release: POLICY.release, token: presented, runId: RUN_ID }; + const body = JSON.stringify(delivery({ answer: canonicalJson(0.3), secretDetection: false })); + + // Counted inside the object, around the delivery itself: what the owner + // does when it judges a value is what is being measured, not what a helper + // does somewhere else. A Worker refuses code generation during a request, + // and the pool only hides that because it proxies `Function` into an + // unsafe evaluation binding. + const generated = await on(stub, async (owner) => { + const original = globalThis.Function; + let made = 0; + globalThis.Function = new Proxy(original, { + construct(target, args, newTarget) { + made += 1; + return Reflect.construct(target, args, newTarget); + }, + apply(target, thisArg, args) { + made += 1; + return Reflect.apply(target, thisArg, args); + }, + }); + try { + const answered = await owner.deliverRequest(named, body); + return { made, answered }; + } finally { + globalThis.Function = original; + } + }); + + expect(JSON.parse(generated.answered)["outcome"]).toBe("performed"); + expect(generated.made).toBe(0); + }); +}); + describe("spending a retained answer", () => { it("releases the value only to an acquisition holding the open execution", async () => { const stub = executor(); diff --git a/packages/workflow/tests/remote-delivery.test.ts b/packages/workflow/tests/remote-delivery.test.ts index 3aa5aab00..54cb386fd 100644 --- a/packages/workflow/tests/remote-delivery.test.ts +++ b/packages/workflow/tests/remote-delivery.test.ts @@ -22,7 +22,8 @@ import { expect } from "@executablemd/test-support/expect"; import { call, Err, Ok, type Operation, race, type Result, scoped } from "effection"; import type { DurableEvent, Json } from "@executablemd/durable-streams"; import { serializeDurableEvent } from "@executablemd/durable-streams"; -import { collect, inlineSource, registerComponents } from "@executablemd/core"; +import { collect, inlineSource, prepareElicitation, registerComponents } from "@executablemd/core"; +import type { JsonObject } from "@executablemd/core"; import { executeInstalled } from "@executablemd/core/host"; import { retainedWorkflowInstallation } from "../src/run.ts"; import { installRemoteInputDelivery } from "../src/remote/delivery.ts"; @@ -252,6 +253,138 @@ describe("delivering one typed value to a remote run", () => { }); }); +describe("one judgment, wherever a response is judged", () => { + /** + * The cases the boundaries have to agree about. + * + * `multipleOf` is here because it is where the compiler this replaced and the + * settled draft-07 judgment disagreed: `0.3` is a multiple of `0.1`, and the + * value is now accepted everywhere rather than accepted at one boundary and + * refused at another. + */ + const cases: { name: string; schema: JsonObject; value: Json; admitted: boolean }[] = [ + { name: "a valid answer", schema: SCHEMA, value: ANSWER, admitted: true }, + { + name: "an answer the schema does not admit", + schema: SCHEMA, + value: { approved: "yes" }, + admitted: false, + }, + { + name: "a non-representable step", + schema: { type: "number", multipleOf: 0.1 }, + value: 0.3, + admitted: true, + }, + { + name: "a self-contained reference", + schema: { + definitions: { flag: { type: "boolean" } }, + type: "object", + properties: { approved: { $ref: "#/definitions/flag" } }, + required: ["approved"], + }, + value: { approved: true }, + admitted: true, + }, + { + name: "a self-contained reference the value fails", + schema: { + definitions: { flag: { type: "boolean" } }, + type: "object", + properties: { approved: { $ref: "#/definitions/flag" } }, + required: ["approved"], + }, + value: { approved: "yes" }, + admitted: false, + }, + ]; + + it("reaches the same verdict through the document path and remote delivery", function* () { + const verdicts: { name: string; document: boolean; remote: boolean }[] = []; + for (const example of cases) { + // What `` judges a provider's answer with, and what the local + // host judges a delivered answer with: one prepared validator. + const prepared = yield* prepareElicitation(example.schema, "workflow answer"); + const document = prepared.validator.judge(example.value).length === 0; + + // The remote delivery boundary, whole: a scripted owner returns the + // retained wait and the production installer judges the value. + const scripted = scriptedLink({ + wait: Ok( + waitRecord({ + request: REQUEST, + responseSchema: example.schema, + requestFingerprint: suspensionRequestFingerprint({ + request: REQUEST, + responseSchema: example.schema, + }), + }), + ), + }); + const outcome = yield* delivered(scripted.link, { + value: example.value, + secretDetection: false, + }); + verdicts.push({ name: example.name, document, remote: outcome.ok }); + } + + expect(verdicts.filter((verdict) => verdict.document !== verdict.remote)).toEqual([]); + expect(verdicts).toEqual( + cases.map((example) => ({ + name: example.name, + document: example.admitted, + remote: example.admitted, + })), + ); + }); + + /** One schema written as JSON, so every declared name survives. */ + function parsedSchema(text: string): JsonObject { + const parsed: unknown = JSON.parse(text); + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new Error("the fixture schema is not an object"); + } + const held: JsonObject = {}; + for (const name of Object.getOwnPropertyNames(parsed)) { + held[name] = Reflect.get(parsed, name); + } + return held; + } + + it("refuses a schema no answer can be judged against, before anything is sent", function* () { + const unusable: JsonObject[] = [ + { type: "object", properties: { decision: { $ref: "other.json#/x" } } }, + // Parsed rather than written as a literal: an object literal takes + // `__proto__` as the prototype and the key never exists. + parsedSchema('{"type":"object","properties":{"__proto__":{"type":"string"}}}'), + { type: "not-a-type" }, + { type: "object", nope: 1 }, + { $async: true, type: "object" }, + ]; + + for (const schema of unusable) { + const scripted = scriptedLink({ + wait: Ok( + waitRecord({ + responseSchema: schema, + requestFingerprint: suspensionRequestFingerprint({ + request: REQUEST, + responseSchema: schema, + }), + }), + ), + }); + const outcome = yield* delivered(scripted.link, { secretDetection: false }); + expect([JSON.stringify(schema), outcome.ok, scripted.asked]).toEqual([ + JSON.stringify(schema), + false, + ["wait"], + ]); + } + }); +}); + /** * One owner that retains what it is told, as the real protocol would. * diff --git a/packages/workflow/tests/suspension-judgment.test.ts b/packages/workflow/tests/suspension-judgment.test.ts deleted file mode 100644 index bb4017c3b..000000000 --- a/packages/workflow/tests/suspension-judgment.test.ts +++ /dev/null @@ -1,251 +0,0 @@ -/** - * Tier WAD — the owner's additional schema check, and what it is not. - * - * The settled contract is that a delivered value satisfies its wait's response - * schema under the semantics the local host uses — `prepareElicitation` and - * `validateParsed`, which compile with `new Function`. A deployed Worker does - * not generate code, so a run's owner cannot run that compiler, and this module - * is what it runs instead. - * - * It is **not** equivalent to that compiler and this file does not claim it is. - * A table of agreements is not a proof of equivalence, and there is at least - * one supported schema they disagree about: `{ type: "number", multipleOf: 0.1 }` - * accepts `0.3` here and refuses it there. That disagreement is asserted below - * rather than avoided, so nothing reads this table as parity. - * - * What it is for is refusing early and refusing more: the values it rejects are - * values the compiler rejects too, on the schemas it admits, and a schema whose - * keywords it does not implement is refused outright rather than judged with - * that constraint quietly skipped. - */ - -import { describe, it } from "@executablemd/test-support/bdd"; -import { expect } from "@executablemd/test-support/expect"; -import type { Operation } from "effection"; -import type { Json } from "@executablemd/durable-streams"; -import { prepareElicitation, validateParsed } from "@executablemd/core"; -import { - judgeAgainstSchema, - requireJudgeableSchema, - UnjudgeableSchemaError, -} from "../src/suspension/judgment.ts"; - -/** One schema, and the values a caller might offer it. */ -interface Case { - readonly name: string; - readonly schema: Json; - readonly values: readonly Json[]; -} - -const CASES: readonly Case[] = [ - { - name: "an approval object", - schema: { - type: "object", - properties: { approved: { type: "boolean" }, note: { type: "string" } }, - required: ["approved"], - additionalProperties: false, - }, - values: [ - { approved: true }, - { approved: true, note: "shipped" }, - { approved: "yes" }, - { note: "shipped" }, - { approved: true, extra: 1 }, - [], - "approved", - null, - ], - }, - { - name: "a bounded string", - schema: { type: "string", minLength: 2, maxLength: 5, pattern: "^[a-z]+$" }, - values: ["ab", "abcde", "a", "abcdef", "AB", "ab1", 12, null], - }, - { - name: "a bounded number", - schema: { type: "number", minimum: 0, exclusiveMaximum: 10, multipleOf: 0.5 }, - values: [0, 9.5, 10, -1, 0.25, 3, true], - }, - { - name: "an integer", - schema: { type: "integer", minimum: 1 }, - values: [1, 0, 1.5, 2, "1"], - }, - { - name: "a list of unique items", - schema: { - type: "array", - items: { type: "string" }, - minItems: 1, - maxItems: 3, - uniqueItems: true, - }, - values: [["a"], ["a", "b"], [], ["a", "a"], ["a", "b", "c", "d"], [1], "a"], - }, - { - name: "a tuple with extra items", - schema: { - type: "array", - items: [{ type: "string" }, { type: "number" }], - additionalItems: false, - minItems: 2, - maxItems: 2, - }, - values: [["a", 1], ["a"], ["a", 1, true], [1, "a"]], - }, - { - name: "an enumeration", - schema: { enum: ["yes", "no", null] }, - values: ["yes", "no", null, "maybe", 1], - }, - { - name: "a constant", - schema: { const: { kind: "approval" } }, - values: [{ kind: "approval" }, { kind: "approval", extra: 1 }, "approval"], - }, - { - name: "alternatives", - schema: { anyOf: [{ type: "string" }, { type: "number", minimum: 0 }] }, - values: ["a", 1, -1, true, null], - }, - { - name: "exactly one alternative", - schema: { - oneOf: [ - { type: "number", minimum: 0 }, - { type: "number", maximum: 10 }, - ], - }, - values: [-1, 11, 5, "a"], - }, - { - name: "everything at once", - schema: { - type: "object", - allOf: [ - { type: "object" }, - { type: "object", properties: { id: { type: "string" } }, required: ["id"] }, - ], - properties: { id: { type: "string", minLength: 1 } }, - }, - values: [{ id: "a" }, { id: "" }, {}, "a"], - }, - { - name: "an exclusion", - schema: { not: { type: "string" } }, - values: ["a", 1, null, {}], - }, - { - name: "a nullable member", - schema: { - type: "object", - properties: { note: { type: ["string", "null"] } }, - additionalProperties: true, - }, - values: [{ note: "a" }, { note: null }, { note: 1 }, { other: 1 }], - }, - { - name: "bounded property counts", - schema: { type: "object", minProperties: 1, maxProperties: 2 }, - values: [{}, { a: 1 }, { a: 1, b: 2 }, { a: 1, b: 2, c: 3 }], - }, - { - name: "constrained property names", - schema: { type: "object", propertyNames: { type: "string", pattern: "^[a-z]+$" } }, - values: [{ ok: 1 }, { NotOk: 1 }, {}], - }, - { - name: "a nested object", - schema: { - type: "object", - properties: { - who: { - type: "object", - properties: { name: { type: "string" } }, - required: ["name"], - additionalProperties: false, - }, - }, - required: ["who"], - }, - values: [{ who: { name: "a" } }, { who: {} }, { who: { name: 1 } }, { who: "a" }, {}], - }, -]; - -/** What the compiler the document path uses says about one value. */ -function* compiled(schema: Json, value: Json): Operation { - const prepared = yield* prepareElicitation(schema, "parity"); - return validateParsed(prepared.validate, value).length === 0; -} - -describe("the owner's additional schema check", () => { - it("agrees with the compiler across this table, which is not equivalence", function* () { - const disagreed: string[] = []; - let judged = 0; - for (const example of CASES) { - requireJudgeableSchema(example.schema); - for (const value of example.values) { - judged += 1; - const expected = yield* compiled(example.schema, value); - const found = judgeAgainstSchema(example.schema, value).length === 0; - if (expected !== found) { - disagreed.push( - `${example.name}: ${JSON.stringify(value)} — compiler ${expected}, judgment ${found}`, - ); - } - } - } - - expect(disagreed).toEqual([]); - // The table's size is part of what it is worth, and its worth is bounded: - // agreement on these cases says nothing about the cases not in it. - expect(judged).toBeGreaterThan(70); - }); - - it("is not equivalent to the compiler, and here is where they differ", function* () { - // A supported schema an ordinary wait can retain, and a value the two - // reach opposite verdicts on. Asserted rather than avoided: this is the - // exact gap that keeps the owner from being the settled authority. - const schema: Json = { type: "number", multipleOf: 0.1 }; - requireJudgeableSchema(schema); - - expect(yield* compiled(schema, 0.3)).toBe(false); - expect(judgeAgainstSchema(schema, 0.3).length === 0).toBe(true); - }); - - // deno-lint-ignore require-yield - it("refuses a schema whose constraints it cannot apply", function* () { - const unjudgeable: Json[] = [ - { $ref: "#/definitions/other" }, - { if: { type: "string" }, else: { minLength: 1 } }, - { type: "object", patternProperties: { "^a": { type: "string" } } }, - { type: "object", dependencies: { a: ["b"] } }, - { type: "array", contains: { type: "string" } }, - { type: "object", unevaluatedProperties: false }, - { type: "wrong" }, - { type: "object", properties: { a: { $ref: "#" } } }, - { anyOf: [] }, - { pattern: "([" }, - { minLength: -1 }, - { multipleOf: 0 }, - ]; - - for (const schema of unjudgeable) { - let refused: unknown; - try { - requireJudgeableSchema(schema); - } catch (error) { - refused = error; - } - expect([JSON.stringify(schema), refused instanceof UnjudgeableSchemaError]).toEqual([ - JSON.stringify(schema), - true, - ]); - } - - // And an admitted schema stays admitted, so the refusal is about the - // keyword rather than about being cautious. - expect(() => requireJudgeableSchema({ type: "object", title: "fine" })).not.toThrow(); - }); -}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index d4709f0e7..b9909494c 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -216,6 +216,9 @@ importers: packages/core: dependencies: + '@cfworker/json-schema': + specifier: ^4.1.1 + version: 4.1.1 '@effectionx/context-api': specifier: 0.6.0 version: 0.6.0(effection@4.1.0) @@ -509,6 +512,9 @@ packages: resolution: {integrity: sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==} engines: {node: '>=6.9.0'} + '@cfworker/json-schema@4.1.1': + resolution: {integrity: sha512-gAmrUZSGtKc3AiBL71iNWxDsyUC5uMaKKGdvzYsBoTW/xi42JQHl7eKV2OYzCUqvc+D2RCcf7EXY2iCyFIk6og==} + '@clack/core@1.4.3': resolution: {integrity: sha512-/kr3UWNtdJfxZtPgDqUOmG2pvwlmcLGheex5yiZKdwbzZJxhV+HMNR9QNmyY5cGwTNV6LrR7Jtp+KjhUAP1qBQ==} engines: {node: '>= 20.12.0'} @@ -3177,6 +3183,8 @@ snapshots: '@babel/helper-validator-identifier@7.28.5': {} + '@cfworker/json-schema@4.1.1': {} + '@clack/core@1.4.3': dependencies: fast-wrap-ansi: 0.2.2 diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 78b24cfc5..6dad7b28e 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -336,10 +336,15 @@ workflow answer: () A value is accepted only when the run is `suspended`, its stop reason names a retained `suspension_request` event carrying the supplied suspension ID, and the -value satisfies the response schema that request retained. Secret detection -applies to the retained state and the answer event it would become, on the same -terms as durable journal persistence, and is on unless `--no-secret-detection` -disables it for that delivery. Duplicate, consumed, wrong-run, wrong-request, +value satisfies the response schema that request retained. That schema is +judged as draft-07 by one shared implementation which generates no code, so a +value receives one verdict whether the run's storage is a local file or an +object somewhere else — and the boundary that writes the value is the boundary +that judges it, rather than accepting a judgment somebody else reports having +made. Secret detection applies to the retained state and the answer event it +would become, on the same terms as durable journal persistence — the same +configured scanner, at whichever boundary writes — and is on unless +`--no-secret-detection` disables it for that delivery. Duplicate, consumed, wrong-run, wrong-request, late, invalid, cancelled-run and missing-run delivery is refused, and a refusal leaves the run's storage unchanged. A refusal retains neither the rejected value nor a secret match in its diagnostic. From d235030869675f19e0444ddef9b1c2ac719c316a Mon Sep 17 00:00:00 2001 From: Min Kim Date: Tue, 8 Sep 2026 18:42:54 -0300 Subject: [PATCH 61/81] =?UTF-8?q?=F0=9F=90=9B=20Tell=20a=20schema=20from?= =?UTF-8?q?=20the=20data=20inside=20it,=20and=20say=20what=20failed=20with?= =?UTF-8?q?out=20saying=20the=20value?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four things the shared judgment got wrong. It deleted every key named `format`, everywhere, treating each object it met as a schema. A `format` is an annotation and belongs nowhere in a decision, but a literal under a `const` is data a document wants matched exactly, and a property a document declares as `format` is a name rather than a keyword. So the exact literal stopped matching, and the declared property was deleted and then refused as one the schema never declared. The transform now walks the same positions the schema walker already knew about: keywords are keywords, data is carried across by value, declared names stay names, and the authored declaration is never touched — a provider still sees the `format` that describes what to answer. A reference whose target did not exist passed preparation and failed later, and only if some value happened to reach that branch. Everything that makes a schema unusable has to fail before a question is rendered and before a provider is contacted, so every reference is now resolved when the schema is prepared, in the same lookup the validator will use, at every schema position — including branches nothing samples. A value was judged by what the language answered for it rather than by what it held: `{}` satisfied a schema requiring `toString`. What the validator is given is now a private projection whose objects hold only their own names, so an inherited name is absent, and absence is an ordinary reported failure rather than something thrown from inside a dependency. What the caller keeps, binds, retains and journals is untouched. And a failure said too much and too little at once. Independent rules were being dropped — a `minProperties` failure disappeared because a member failed too — where only the wrapper of another failure should go. Locations came back as URI fragments, so a member named `🐲` was reported as `/%F0%9F%90%B2` instead of a JSON pointer. Messages were the dependency's, and quoted the rejected value and the schema's own thresholds into something that is printed, bound into the evaluation environment and carried across a journal. Each is fixed: independent failures survive, pointers are raw, and every message is this repository's own, naming the rule and never the value — except `required`, which names the member that is missing, because the location cannot. The deduplication key had a NUL byte in it, which is why Git showed the file as binary. It is a serialized tuple now. --- .../cli/tests/workflow-suspension.test.ts | 2 +- packages/core/src/draft-07-meta-schema.ts | 2 +- packages/core/src/elicitation-schema.ts | Bin 13708 -> 19503 bytes packages/core/src/schema-walk.ts | 115 ++++++ packages/core/tests/elicit-component.test.ts | 11 +- .../core/tests/elicitation-schema.test.ts | 327 ++++++++++++++++++ .../cloudflare/remote-delivery.vitest.ts | 97 ++++++ .../tests/workflow-suspension-answer.test.ts | 119 ++++++- 8 files changed, 665 insertions(+), 8 deletions(-) create mode 100644 packages/core/tests/elicitation-schema.test.ts diff --git a/packages/cli/tests/workflow-suspension.test.ts b/packages/cli/tests/workflow-suspension.test.ts index 7723bea85..f496eb07d 100644 --- a/packages/cli/tests/workflow-suspension.test.ts +++ b/packages/cli/tests/workflow-suspension.test.ts @@ -1274,7 +1274,7 @@ describe("Tier CKX — a checkpoint a document asked for", () => { expect(invalid.exitCode).not.toBe(0); expect(invalid.written.err.join(" ")).toContain("does not satisfy the response schema"); expect(invalid.written.err.join(" ")).toContain( - '/proceed Instance type "string" is invalid. Expected "boolean"', + "/proceed must be of the type this schema declares", ); expect(yield* storageDigest(path)).toEqual(before); diff --git a/packages/core/src/draft-07-meta-schema.ts b/packages/core/src/draft-07-meta-schema.ts index f1b53e80e..b8cf8ee07 100644 --- a/packages/core/src/draft-07-meta-schema.ts +++ b/packages/core/src/draft-07-meta-schema.ts @@ -95,7 +95,7 @@ export const DRAFT_07_META_SCHEMA: Json = { contentMediaType: { type: "string" }, contentEncoding: { type: "string" }, if: { $ref: "#" }, - // deno-lint-ignore no-thenable + // oxlint-disable-next-line unicorn/no-thenable then: { $ref: "#" }, else: { $ref: "#" }, allOf: { $ref: "#/definitions/schemaArray" }, diff --git a/packages/core/src/elicitation-schema.ts b/packages/core/src/elicitation-schema.ts index 8a65e787787b78b7209f7ebd0eb441de94ca23d1..eee99b53c7b6de9fc89887c139576029537a2478 100644 GIT binary patch literal 19503 zcmb_k+j1L6l6}`#lws{GsHH%8UK~r02BC{Lh7mIc8dX_YS(zsWS~oMv$u*BOT_ z<04m?trH*Ffs1)6Lf4@oZzH&uPLDykc7wYlLYdZVi*S@^c7y5)*|hwr*NyC`Bfx%ee6TIU&L z!3lo;```Wtf@r|F?7gkQNVcH3Nm3+mQ|B$@ledRpYLRs~fGm&=tmY!Z{3d{5jA~-B zG8s3atlG=GeCy|NMo|@6)2h(9-J8oTsNh3lxDUJAFU2MRn3dc^oh*?e1aheP=QQl+_!XhidBKAds3!GG5U}$z3 zvt_7Oq0Mnpp{X$MIxF&J)?$mYXXD4&OLXw7D6vWpALUSRPT1KAbXx^hVIJ3YD!NAy z0Vhr&Jfd1}FRJBMbeS}$?)l=9a^pIiDleguu2aEw%TSrHtH#gk^UuT2JhDw4H<^jQj&dwi2GP*0+{zh~Da;j1;qG4!=J0sA z867Ax;AU4@UIf@>wxLRkm~9I_q(2;}0?rT#vm1{9)E? z%f%c=Z>#h3Im7{C4j`eV+2a8O2u90=DFgKL8tMSjBo#u_Z8k*?^2(ZP^Htme zES6zeXRG${lP>{NrbNrQ0yccbjo;f9(9FlLfgz7=yJn?!l!Mcrb59xIO6($|-BX?r z9jpxylq?!6aj`RKp_vU0L(akQ6i5#cFb>*^XD6J5!*iM@9cWX^1| z30E<0L;==4!Zz8Kpf(YiLaY!b+*e*qMz#-(l5GN^IXZ87lt&L9WQPJL%d@*Zx$}7npr~B7xiR-K%fD zKRbW&<@pcK&%QZ7{q~Q~fA}W&?Aej)>EnT(&f2DD;Whw*n)oNMci4Y4x7O1k`=fcr z`j1q;BLDBWe9<&O#D0dW=V%s12ei064C`#8TLoHX#nsNXXGeOrkNMk(g0Bm!|BoCt z)Gzpv8M;IM|GmwTW}~Y-IC{kRMrlFGfYOVqLr|oU3GJc_16n0zf0gCLP{&4xZS_si zn)ptX2H{TAgjW`8>2{yjHAuyGEkL#j`t9~XFkI*saEsG*Y?|zCJT||45I8{KET$h1 z0GIfqQqKs7;c1xcSqX<8!rgd(JP80mLmPygkQptDR^1!t1)>^@fEuyC48f`xJ`J8BY$Cb_ zmI1A3z9K`SDF-0fgd}ASe@;QC8n7J&rp9`O=rkQiZ@82$RiM}F1ohar#Rnde-sDB$ zctar&B!Z|&RRZE%kiE|F07(5gVpjm$C2$VB8u~Vf!)>o;EwPWCy&0wCb7a}%G2cxO{|x)*au@b)~$fbg@rNpT9|U8*)BflXI$@R?zx znFf7|U};kUS{RKYsn8@s{U0={KXD$X1H#e~9k%5h=5I@c<;!E9XjcMfi^yC7izgpR zRlp1yplAmU!zBjK#y}TQ2&7JdGN9=Yh6~KYr6EvHftUuwNOGw9@H7VQvQ9!%SWp>n z!f9-$uW@FRhxu{>SA`F{efxr)+9!BxGOXX)ye*=BR5!a}@1D2r_W2|UoaXSn%fydcTKpoCH)O1=|_y5z&`@){3ARQM!ta) za9n>yCq+mUFD1aZgYJsqM9XlIEv^*n4b25+kPd7IxPFQ#{Iagf3b4URakWC)`*>AH z01|M+Vt5E_*)i+SLf1%sOq!DQ!_g7-qegtf+$h_VY3p|8pCcWd&l%5(7&K`R3fgA}%sv3D2ss;#75@~9b=Y&d* zkq|TyZou=OY*AMT!a9Tw9b~lBRH+P_jN&=#Oj($VV41AJf|Qmc0Stv($h!8D{z{Th zIv?K?+zeuvrmL@67BH8D;6vS*mC@i_QuI3=2>&olo9;r7{vbSk7AhoEn%xb4(3318 zDn3EA(IEs7yi ziL5-{Y+?O3;uBpLRm&u3oeJTABHqJWQRKUD^ehDm_rhmbx~C<_;o+@#=D4)=?&kEf zAt*2SXef_mtKyoN^2!ta#+6gKKeVl>fHVU2XxYqZC~R3pW_~QNG9pRm1t2|eGo^1L zbPbAt`1mn`8mIEgL_<3!0K8Ew!DJD2AMN6+q;|Un7zwx&Tn`O!fXJFa zA_ngxd(n}66{er*k2CnQlbdp?>|p;E863Dl*{=0>TIe1Ar&7H_9c)DVX}HBE18XK> zs$*VTU6;{Nv&VQ!B`P)=u>GJ^zEoWQQ>z=ah^M>g~M_hUQBpF9g~2L2Te;G^FwZ0_@jrJV1~G zrMT)sqO~gSQ>f%a@?y1NcffjCAlWI7MSw@rbpQh}6hI~MP%dedr%mGlAY3Ecd(rYR zwg3vh%fO=9GDDrVCPzmPlBs+MrlC0F_y!MDGD9_|Eq|&@JA44^yRP!3C~sdtO7^C{ zF1gd*e~wl}!K=OWsLL(W&9K+yr%+k*%daO7@qB`0n`RLLt%~74% W=B4g<(o=Qo z@u{>Y{sFS*17AD`A^LMY-RuHK*0o4!frN~-?F`{x-m-A)mkrQ{o*dNzrY`DXzzpP| zY=PtmqQ8d=U<@pwKNOtfnS4Sg`(5~W8(Co;VHq&qzS)}o*(zTx=8l`uxivH<&mI}QvalYh6DEoHR-Gylmu?m zx4TJzS9y3#wYu}XfznNoHxK_}f+cMf6bvC!%AwZ|Gs*pQNOF{_FyiwkS0%j70Xr7v}_PV^mNUL`D5b$fHRgFoau7 z+jpNeb{!WH<{-dqHzff6Rfb%ES`TT!iIf$u3=tcb8q+BJsot{m4#j1~=r4W+fWmE? z=?j()xkkB z60ZH46i1=JAPC?9QMcp+?kA3pt?)tn;-`e*y4^mUq(n6mL?Y;s46U3N?xCQlspG3g zCLBT*mEJ0^aVL(0d0!Oy`iTI!j+T~(@D#HVY3WQL50gS(1TMp`zv|-|VHZ(DdD7_V6c{Kbhj`$T{FG5_5qA#-GIn?&EkZI9(|WZ}kb4>2 z2%s*9LN%`tc_W1j)@xMLNNm;BSVme6_VuJ#^$OMGRvS13*(AQx>jv@|$(z(lNHLbz zIch$vaG|JvNWAq&4$XkS4L%XDpU;kib72&s$zmYY>6TlXh|W1!fyL80zY z$>v=vB?jzpvCq8}Iw#nYO^ph1y0lyG4nV?LT0fPw@eVy>r46_|Sxh zbAV7bPY#_b-p@oZi#scsC(I2d_G&VmkYvh?!|p_U5;X~2q8N*E7pL3rqZM5nWlg(x z*&%=$Fw4Uvpf@n@C(z&3=tYcE0vkP*H0a|DrJ+CQ$^&}+w$h-F^VeTA4Jc*7AN$EP zZQQ(nlMtg`BsmQeG*G=pGKq;m^Wh;4P8AZ|JGVfw{UcM&SNNjn!9lXmQ?VaLOOB7# z?yG#NKnzR%t0*0XyyyBD*K)sgkAQSX<{zCuxFQ>pfKNs%4SgR;xYo1z4jn*>DX;>5 z_c5bAN{FV$f+KmEZpM7>&NdBH^iukRC-ebQ+76nQ`k>VI-QwNjZNya%1PQ$CAVoMm z!TqP<2k{XywxnJKIzs~9{);)&kq)y-M1?)PT=eNZ#-g8WaQQeou#r+q0)gpZmIswm zIl5cfEuK1c4&ap(L%bnVV)^ilG9F>V$;xLqKVA}I3}LXF6}sCyi+Fxt-u&GUPcY3x?y8%hD~v9yC7~!J+ z3Mgm7QMXcu;7y9J&OH1kq9BX^Vw{BX*5LuzW0SQ-?DHNO<}z8TT-IGZKg#r>wkYx* z?*o>`Vyf$F2bT<6g`ob~>;C6^eQ<0J-PeZ=bCk^+ZE z2w+j1M&gA6!_E*1Z!~#9-l_cvLQW!ov1G)=46#!!PHGg+kX##$fPD@S5D^V!Lka4& z5sDVleye$bU0g~!H>mCQ`7py zQ#Nuidtjx5d*DqHFK$v{MyeZVOda0Mpq|d5aCS1(C`jYoY|}ND(_5hls)O3aF_cJp z;Z9bXy!emKn240TDAKUralC&AM&0tK@gSZH{{XY!;k^@NmN+U|@FRah9_p*bQa-Xu z#c@KmrK+Ctd?!aSD0!*I?ABPt9vvwA3rB>=l1HOS=zZ^;zyRX zg2!mFejUX_V(j}?bdXPVoKIR_k(ZF62!5aXe#y%Da+0y_;+m=Fv#;X&B#BqHcZH5y z4K;#Pcur7mUPU>xMR>77hKX}}ZG`ud^wkc=lLPCsSSUqIgUDyxG0{8bVY$tm^zad) z1+!h*x}Sv9$;{8|85ysl2Er^Dwr3hS4CG<{$2V=udRfmOA3)`MwYw*(o`w5A{ram+ z3|b(1-ysP9_a-0EeC|tno|kwBZyr!+=Q~*XqRbq39DK+uK4BTQ!>>)$j!JJl7$1O2 zL#BI;lLdV&;a_(oZ?cpj2w*bA-#;THoX#KK3YEK~d19I{djidHy&JHmp*b2dZuC7b z3oYOhfVBNZf_%gh?(NPrkuWD^??e_BE{SjZ>D3FoOq0o;;k%`Gqphxhy&#NyBtyWi zals>9`K1Om^jUOByb$2Kjxq@11HD=Jsa#@|ynw(5Iz%1_W$ho)RoKg@PS3Z*Yjbt; zVgA=A7JvNpi+L#G3cd$rDHag*l}^bY*n~*Yfg2CGEGSU^83-UAKp=!ehXC4+`BHu$m|O-h>K7_F2R8pL~wO_t%4;@P2NbesO@i zEgw#Eq~YEp$Q%1qHa7*EOm?hyXO!xwM>y?oC+tv+)U1%6|L`pQ{>c+ehwb|>jU4-vL;iAb_xHd3?;HwXKe|Y$+j8@# zu@Lp2Eji*OyuL4v3Ekg>G#}s~@SOcvT4~ZZNcc8qhHGx3ohHq=cV|IM`ZXCgYpNCM7#6MexPa02=6zRT*bpyK|MuYs}ITrXkuGOQ0T zalw`G9rUM;1zY; z1ZqdRVJBa!WStgXyS-fDrfG{LhD$nv3}3dE7C6X<_tD+kOG>no`#i%XIMN1Y&iBLt zJS-Jhe2iR>)d#~A0MaA&VLy^%O`X#eLC}oMau__pTi@uH6PYVtxL4{hjY!1#~kNxYueXotF4l)_-dcz4hQ5H<)pp+H9f+v@=s6Vg| zcNzY8cJ^|5dVmA8hDJR#@C+NG;HvQX7ytTu5smSIZID-`C;=1r!jpS#i_iHUBH-G&`PpYa-dD;A>|?b<_$J5GiHEN+l$N5yUG>{^{oP)d%-r?&!8<5qnD4B zt)4gqtIOlP2DG3Fdtd|?t)s?EyoD3_=9RvW-z`-Oc)#3zeC+rUwTK5 zEbBsvtf6dXt??vn;k8XA2Ripg^+dN% zxIOFJCg%2is)KT$23enZyOtLO*^o0D^&2#ES2yiI&@C(tjmF&GcE;3fm-Kn}EYQ`c z7$b#j3uIE|UljE|?%*^8K-ow>hL_#b0C{^|bSUeAN%Tgc+3voMNp?u|SaXFp$x#Q! zGrm*K_u%)n1rsAUPb(zpAEZcuF3nIK{y3EUZBUX(sevmk=@!U|{F@ImyeA;? zV~~V}`x3AEHGltVhwd`R%ZB)uJ>+fS{>Tjhclut@amvOcdE?wtQt6Q!_{<9k4K7%^ zA;+4AH}ug#!`fp1-hOH(LKq?KvM+@AHx1h`r+$S%$1g7bM{hOMe-A-!ML1H592Rt^ zK1`dg>GfctYnX=jZ5^gmCL6oe zkZ;2#%s!K{j3QO9KI|Iz>URt5rRh^mcvSmd3G=g-%*>>iH7X=)W|bEV{8VN9pl-i^ ztV(G|`um3VCm@QN1b6=odLizfwXrD#ls)J_yHRdsQy&(b40#V`OFI~Qg|zyiLh)A` WVP&HJB4oS0wMf&W36s^`gZ~5Lr+LNz delta 3226 zcmai1OK%)S5XKTBVIc}352GNG%GwEQ$E*Pp2OJ!T0HatD4+Rkj0d4P0t#|P3^rmOV zt5sq)9Jp~nY7ShuB2Iv04*UWRAc6P=k#d87fP{drx_cjvkjNoBn(3~tuJ2LhZ@-xO zbo0Bbdv>nxnVs7CdhhAI_M-#02Pqe{rP3r;g*Gen_VDpsg{Uxnx`AIBub@@RxHn!_ zQ~Bkn%^g+iB}H48#hNx9Rphmpqf{AHBs!a$TcGOM>H@V?n)94Q-lj+uQJZS^g+s5} zpHH0J*PzqxBsr~S@2oEGn_F-%{CfN6p?4mSbe0!{Ln7T_WHQRT9-g>{hths|@fM>7o80ed6O(G{ zEvBSEG*yJChFjXO+sks8uG0m?R56>Q)KeKE=;mT#6jAk00Anm0L8WzK;b zqx0PS@j96h=jbD;cd{e*HD->Lddf2>)w$578F8 zD+NLz&`5%afr|*$9dC!5o7_&IBSnpw2B<0j_9LROAmTSv(Ow}z`#>Gpq$=Xt;s-N} zu#2j_@HD?9X-=Ndg`;#BNuk=hOk=o+fj-@-4#aU{P%jKe8+-fdPi(b1JqZ3J*Jelp zIv;NnO^ebv9(hDh3r4bJGJy`~x!Qn0ua~;u$p}pqZPJ}vpSyP|O?3}^aTNm}TNN72 zL&DVHtn4(rnnUbT_|(D@ff{5P? zJt`)c`zX3A2MRv+Ozl2~dIf4oz{zSNb!S(VR+&ZOQsd~g(mr@{X-&#ioeNFKa8a6! zs+|&s)nr1gV1*#3&Ugsc?#fqPuc=r%`^dmm`2S-GD!YP&7|togpwNCi?tq3t;wLzZ zg-vkrM56;tGKP94DWv4QHIPOE>rUBt*`tR+f8Og+Ux$K%BW4?bEDym0Q&9U>nWmH% zx(l6UVl%o2wxGW$Kq)Q;UrQjXO$p_)7}nn7G{8Gyi2vq@DS8Ywb#B4NXfqb#wfd}6 zB7AqdQdyiI^-d@7KuJ+m=#mYn4Xs~D+(DllN$CR2rQ07Y5R6!=Gcg$;>8{aR=xeYU z(7Y|qjfWDJQ$o;jokme%8dxV38p}mXmky$meHdaCJ`a@>`Y)1-+5^aOlCer61%LEa zZuC^T>4%wy1ap4AndzRlwK!?D*a+fI=AyWwEz7VA%OxCcEox(Ih2wkmyK6&n&+07g zhtfzB8r{n^FZi$=i1;*_7~#GB=*;wem-jEk2>2^7G6H*w$#`w$9A zAG)Ub!QxogyH8)C^&DfN|8e3@T2#AWjOy2u-2H|$?~=Z!ZAW!$H3-D?W!@Oy%)j#U zBCht51oc;t(qtl_n>VTGcNxy^KgeKG)y)q%D(LIt^95?s*7xo!i F`WK^&IH3Rl diff --git a/packages/core/src/schema-walk.ts b/packages/core/src/schema-walk.ts index 8006d5343..ce347c7e6 100644 --- a/packages/core/src/schema-walk.ts +++ b/packages/core/src/schema-walk.ts @@ -26,6 +26,16 @@ export type NameKind = | "required property" | "property dependency"; +/** + * One schema position, rewritten. + * + * Returns what should stand at this position. Everything the walker knows about + * where schemas are and where data is applies, so a transform never reaches a + * `const`, an `enum` member, a `default` or an `examples` entry, and never + * touches a name a schema declares. + */ +export type SchemaTransform = (schema: JsonObject, path: string) => JsonObject; + export interface SchemaVisitor { /** One subschema, at a real schema position. */ subschema(schema: JsonObject, path: string): void; @@ -155,3 +165,108 @@ export function walkSchema(root: JsonObject, visitor: SchemaVisitor): void { } } } + +/** + * Rewrite every real schema position, leaving everything else exactly as it is. + * + * The same position table `walkSchema` visits, used to build a copy rather than + * to inspect one. Data keywords are carried across by value, declared names are + * carried across as names, and nothing in the input is mutated — a caller can + * hand this an authored schema and keep using it afterwards. + * + * Every object it builds has a null prototype, so a key named `__proto__` + * arriving in data is an ordinary member rather than an assignment that + * rewrites the object it was copied into. + */ +export function mapSchema(root: JsonObject, transform: SchemaTransform): JsonObject { + return rewrite(root, "#"); + + function rewrite(schema: JsonObject, path: string): JsonObject { + const mapped = transform(schema, path); + const copy = record(); + for (const [keyword, value] of Object.entries(mapped)) { + copy[keyword] = rewriteKeyword(keyword, value, path); + } + return copy; + } + + function rewriteKeyword(keyword: string, value: Json, path: string): Json { + if (SUBSCHEMA.includes(keyword)) { + return isJsonObject(value) ? rewrite(value, `${path}/${keyword}`) : detach(value); + } + if (SUBSCHEMA_LIST.includes(keyword)) { + return Array.isArray(value) + ? value.map((entry, index) => + isJsonObject(entry) ? rewrite(entry, `${path}/${keyword}/${index}`) : detach(entry), + ) + : detach(value); + } + if (SUBSCHEMA_MAP.some(([name]) => name === keyword)) { + return rewriteMap(value, `${path}/${keyword}`); + } + if (keyword === "items") { + if (isJsonObject(value)) { + return rewrite(value, `${path}/items`); + } + return Array.isArray(value) + ? value.map((entry, index) => + isJsonObject(entry) ? rewrite(entry, `${path}/items/${index}`) : detach(entry), + ) + : detach(value); + } + if (keyword === "dependencies") { + return rewriteDependencies(value, `${path}/dependencies`); + } + // Everything else is data: `const`, `enum`, `default`, `examples`, every + // scalar constraint, and every keyword this draft does not define. + return detach(value); + } + + function rewriteMap(value: Json, path: string): Json { + if (!isJsonObject(value)) { + return detach(value); + } + const copy = record(); + for (const [name, entry] of Object.entries(value)) { + copy[name] = isJsonObject(entry) ? rewrite(entry, `${path}/${name}`) : detach(entry); + } + return copy; + } + + function rewriteDependencies(value: Json, path: string): Json { + if (!isJsonObject(value)) { + return detach(value); + } + const copy = record(); + for (const [name, entry] of Object.entries(value)) { + copy[name] = isJsonObject(entry) ? rewrite(entry, `${path}/${name}`) : detach(entry); + } + return copy; + } +} + +/** + * One JSON value, copied all the way down, with every object own-keyed. + * + * Data, not schema. A copy so nothing downstream can change what the author + * wrote, and null-prototyped so a member named `__proto__`, `toString` or + * `constructor` is a member rather than something the language answers for. + */ +export function detach(value: Json): Json { + if (Array.isArray(value)) { + return value.map((entry) => detach(entry)); + } + if (value === null || typeof value !== "object") { + return value; + } + const copy = record(); + for (const name of Object.getOwnPropertyNames(value)) { + copy[name] = detach(Reflect.get(value, name)); + } + return copy; +} + +/** One object with no prototype, so every name it answers is a name it holds. */ +function record(): JsonObject { + return Object.create(null); +} diff --git a/packages/core/tests/elicit-component.test.ts b/packages/core/tests/elicit-component.test.ts index dacafb5c3..4f6de330d 100644 --- a/packages/core/tests/elicit-component.test.ts +++ b/packages/core/tests/elicit-component.test.ts @@ -370,10 +370,13 @@ describe("Elicit: judging the answer", () => { const result = yield* run(workspace, document("Approve?"), constant({ decision: 7 })); expect(result.failure?.message).toContain(""); - // The validator's own words. What matters to a document is that the issue - // names where the value went wrong and what was expected there. - expect(result.failure?.message).toContain('"/decision" Instance type "number" is invalid'); - expect(result.failure?.message).toContain('Expected "string"'); + // Where the value went wrong and which rule it broke — and not the value + // itself, nor the type the schema declared, because an issue is printed, + // bound and journaled, and neither belongs in all three. + expect(result.failure?.message).toContain( + '"/decision" must be of the type this schema declares', + ); + expect(result.failure?.message).not.toContain("7"); }); /** diff --git a/packages/core/tests/elicitation-schema.test.ts b/packages/core/tests/elicitation-schema.test.ts new file mode 100644 index 000000000..4469b8a4b --- /dev/null +++ b/packages/core/tests/elicitation-schema.test.ts @@ -0,0 +1,327 @@ +/** + * The one judgment an elicitation response is held to. + * + * Everything here is about the difference between a schema and the data inside + * one, and about what a failure is allowed to say. A schema carries values — + * under `const`, `enum`, `default`, `examples` — and declares names, and a + * transform that treated either as a schema would change what a document + * asked for. A failure carries a location and a rule, and one that carried the + * rejected value would publish it into every place an issue travels: the + * evaluation environment, a printed error, a journal. + * + * The judgment is the same object at every boundary, so these are the claims + * every boundary inherits. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Json, JsonObject } from "../src/types.ts"; +import { prepareResponseValidator, ResponseSchemaError } from "../src/elicitation-schema.ts"; + +/** One schema written as JSON, so every declared name survives the parse. */ +function schemaOf(text: string): JsonObject { + const parsed: unknown = JSON.parse(text); + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new Error("the fixture schema is not an object"); + } + const held: JsonObject = {}; + for (const name of Object.getOwnPropertyNames(parsed)) { + // Re-parsed rather than asserted: what a fixture holds is JSON, and this + // walks it as such. + held[name] = JSON.parse(JSON.stringify(Reflect.get(parsed, name) ?? null)); + } + return held; +} + +/** Whether one value satisfies one schema, through the production preparation. */ +function admits(schema: Json, value: Json): boolean { + return prepareResponseValidator("probe", schema).judge(value).length === 0; +} + +/** What the failures say, as location and rule. */ +function issuesOf(schema: Json, value: Json): { at: string; keyword: string; message: string }[] { + return prepareResponseValidator("probe", schema) + .judge(value) + .map((issue) => ({ at: issue.instancePath, keyword: issue.keyword, message: issue.message })); +} + +describe("what a schema means, and what is data inside it", () => { + // deno-lint-ignore require-yield + it("omits `format` where it is a keyword, at every schema position", function* () { + // A format annotates and never constrains, at the root, beneath a + // combinator, and through a local reference. + expect(admits({ type: "string", format: "email" }, "not an email")).toBe(true); + expect( + admits({ anyOf: [{ type: "string", format: "email" }, { type: "number" }] }, "not an email"), + ).toBe(true); + expect( + admits( + { + definitions: { mail: { type: "string", format: "email" } }, + $ref: "#/definitions/mail", + }, + "not an email", + ), + ).toBe(true); + // And the declaration a provider is shown keeps it, because saying "this is + // an email" is the point of writing it. + expect(prepareResponseValidator("probe", { type: "string", format: "email" }).schema).toEqual({ + type: "string", + format: "email", + }); + }); + + // deno-lint-ignore require-yield + it("keeps a literal that happens to carry `format`, exactly", function* () { + const constant = { const: { format: "email", x: 1 } }; + expect(admits(constant, { format: "email", x: 1 })).toBe(true); + expect(admits(constant, { x: 1 })).toBe(false); + expect(admits(constant, { format: "other", x: 1 })).toBe(false); + + const enumerated = { enum: [{ format: "email" }, { format: "uri" }] }; + expect(admits(enumerated, { format: "uri" })).toBe(true); + expect(admits(enumerated, {})).toBe(false); + }); + + // deno-lint-ignore require-yield + it("keeps a declared name that happens to be `format`", function* () { + const declared = schemaOf( + '{"type":"object","properties":{"format":{"type":"string","format":"email"}},' + + '"required":["format"],"additionalProperties":false}', + ); + + // The property exists and its own nested annotation constrains nothing, so + // an ordinary string is admitted and the name is not treated as additional. + expect(admits(declared, { format: "not-email" })).toBe(true); + expect(admits(declared, { format: 1 })).toBe(false); + expect(admits(declared, {})).toBe(false); + + // The same through a definition reached by reference. + const referenced = schemaOf( + '{"definitions":{"format":{"type":"string","format":"email"}},' + + '"type":"object","properties":{"a":{"$ref":"#/definitions/format"}}}', + ); + expect(admits(referenced, { a: "not-email" })).toBe(true); + expect(admits(referenced, { a: 1 })).toBe(false); + }); + + // deno-lint-ignore require-yield + it("leaves the authored schema and the judged value alone", function* () { + const schema = { type: "object", properties: { a: { type: "string", format: "email" } } }; + const before = JSON.stringify(schema); + const value = { a: "x" }; + const valueBefore = JSON.stringify(value); + + expect(admits(schema, value)).toBe(true); + + expect(JSON.stringify(schema)).toBe(before); + expect(JSON.stringify(value)).toBe(valueBefore); + }); +}); + +describe("what preparation refuses, before anything is asked", () => { + // deno-lint-ignore require-yield + it("refuses a reference whose target the schema does not define", function* () { + const dangling: Json[] = [ + { $ref: "#/definitions/missing" }, + { type: "object", properties: { a: { $ref: "#/definitions/missing" } } }, + // Inside a branch no sampled value would visit. + { anyOf: [{ type: "string" }, { $ref: "#/definitions/missing" }] }, + ]; + for (const schema of dangling) { + let refused: unknown; + try { + prepareResponseValidator("probe", schema); + } catch (error) { + refused = error; + } + expect([JSON.stringify(schema), refused instanceof ResponseSchemaError]).toEqual([ + JSON.stringify(schema), + true, + ]); + expect(String(refused)).toContain("does not define"); + } + }); + + // deno-lint-ignore require-yield + it("resolves a reference whose pointer token is escaped", function* () { + const schema = schemaOf( + '{"definitions":{"a/b":{"type":"string"},"c~d":{"type":"number"}},' + + '"type":"object","properties":{"x":{"$ref":"#/definitions/a~1b"},' + + '"y":{"$ref":"#/definitions/c~0d"}}}', + ); + + expect(admits(schema, { x: "s", y: 1 })).toBe(true); + expect(admits(schema, { x: 1, y: 1 })).toBe(false); + }); + + // deno-lint-ignore require-yield + it("refuses what it always refused, and says so boundedly", function* () { + const unusable: { schema: Json; says: string }[] = [ + { schema: { $ref: "other.json#/x" }, says: "#192" }, + { schema: schemaOf('{"properties":{"__proto__":{"type":"string"}}}'), says: "__proto__" }, + { schema: { type: "not-a-type" }, says: "not a valid draft-07" }, + { schema: { type: "object", nope: 1 }, says: "draft-07 does not" }, + { schema: { $async: true, type: "object" }, says: "asynchronous" }, + { schema: "not json at all", says: "not JSON" }, + { schema: [1, 2], says: "must be a JSON Schema object" }, + ]; + + for (const { schema, says } of unusable) { + let refused: unknown; + try { + prepareResponseValidator("probe", schema); + } catch (error) { + refused = error; + } + expect([says, refused instanceof ResponseSchemaError]).toEqual([says, true]); + expect(String(refused)).toContain(says); + expect(String(refused).length).toBeLessThan(600); + } + }); +}); + +describe("a value is judged by what it holds, not what its prototype answers", () => { + // deno-lint-ignore require-yield + it("treats an inherited name as absent", function* () { + for (const name of ["toString", "constructor", "valueOf"]) { + const schema = schemaOf( + `{"type":"object","properties":{"${name}":{"type":"string"}},` + + `"required":["${name}"],"additionalProperties":false}`, + ); + + // `{}` inherits the name and holds none, so the required member is + // missing — and it fails as an ordinary issue rather than by raising. + const missing = issuesOf(schema, {}); + expect([name, missing.map((issue) => issue.keyword)]).toEqual([name, ["required"]]); + expect(missing[0]?.message).toContain(name); + + // Holding it is what admits it. + expect(admits(schema, JSON.parse(`{"${name}":"held"}`))).toBe(true); + expect(admits(schema, JSON.parse(`{"${name}":1}`))).toBe(false); + } + }); + + // deno-lint-ignore require-yield + it("judges an ordinary property the same way", function* () { + const schema = { + type: "object", + properties: { note: { type: "string" } }, + required: ["note"], + additionalProperties: false, + }; + + expect(admits(schema, { note: "x" })).toBe(true); + expect(issuesOf(schema, {}).map((issue) => issue.keyword)).toEqual(["required"]); + expect(issuesOf(schema, { note: 1 }).map((issue) => issue.keyword)).toEqual(["type"]); + }); +}); + +describe("what a failure reports", () => { + // deno-lint-ignore require-yield + it("keeps every independent failure, and drops only wrappers", function* () { + const issues = issuesOf( + { type: "object", minProperties: 2, properties: { a: { type: "string" } } }, + { a: 1 }, + ); + + // Two rules failed at two depths, and both survive. `properties` failing + // because `/a` failed is the wrapper, and it does not. + expect(issues.map((issue) => [issue.at, issue.keyword])).toEqual([ + ["", "minProperties"], + ["/a", "type"], + ]); + }); + + // deno-lint-ignore require-yield + it("reports a raw JSON pointer, whatever the member is named", function* () { + const named = (name: string) => + schemaOf(`{"type":"object","properties":${JSON.stringify({ [name]: { type: "string" } })}}`); + + for (const [name, pointer] of [ + ["🐲", "/🐲"], + ["a/b", "/a~1b"], + ["c~d", "/c~0d"], + ["100%", "/100%"], + ["", "/"], + ["a b", "/a b"], + ]) { + const issues = issuesOf(named(name), JSON.parse(JSON.stringify({ [name]: 1 }))); + expect([name, issues.map((issue) => issue.at)]).toEqual([name, [pointer]]); + } + }); + + // deno-lint-ignore require-yield + it("says which rule failed without repeating the value or the schema", function* () { + const cases: { schema: Json; value: Json; keyword: string; absent: string[] }[] = [ + { + schema: { type: "number", minimum: 100 }, + value: 42, + keyword: "minimum", + absent: ["42", "100"], + }, + { + schema: { type: "number", multipleOf: 0.1 }, + value: 0.31, + keyword: "multipleOf", + absent: ["0.31", "0.1"], + }, + { + schema: { type: "string", maxLength: 3 }, + value: "hunter2secret", + keyword: "maxLength", + absent: ["hunter2secret", "3"], + }, + { + schema: { type: "string", pattern: "^[a-z]+$" }, + value: "hunter2secret", + keyword: "pattern", + absent: ["hunter2secret", "[a-z]"], + }, + { + schema: { enum: ["approve", "reject"] }, + value: "hunter2secret", + keyword: "enum", + absent: ["hunter2secret", "approve", "reject"], + }, + { + schema: { const: "approve" }, + value: "hunter2secret", + keyword: "const", + absent: ["hunter2secret", "approve"], + }, + { + schema: { type: "object", maxProperties: 1 }, + value: { a: 1, secret: "hunter2secret" }, + keyword: "maxProperties", + absent: ["hunter2secret", "1"], + }, + { schema: { type: "string" }, value: 42, keyword: "type", absent: ["42", "string"] }, + ]; + + for (const { schema, value, keyword, absent } of cases) { + const issues = issuesOf(schema, value); + expect([keyword, issues.map((issue) => issue.keyword)]).toEqual([keyword, [keyword]]); + const reported = JSON.stringify(issues); + for (const leaked of absent) { + expect([keyword, leaked, reported.includes(leaked)]).toEqual([keyword, leaked, false]); + } + } + }); + + // deno-lint-ignore require-yield + it("carries no library object in what it reports", function* () { + const issues = prepareResponseValidator("probe", { type: "string" }).judge(1); + + expect(issues).toHaveLength(1); + expect(Object.keys(issues[0] ?? {}).toSorted()).toEqual([ + "instancePath", + "keyword", + "message", + "params", + "schemaPath", + ]); + expect(issues[0]?.params).toEqual({}); + expect(issues[0]?.schemaPath).toBe("#/type"); + }); +}); diff --git a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts index 2fa51d601..beb07135d 100644 --- a/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-delivery.vitest.ts @@ -685,6 +685,103 @@ describe("judging a delivered value at the owner", () => { expect(await on(leaving, (owner) => owner.retainedAnswers())).toEqual([]); }); + it("keeps a schema's data and declared names, and refuses a dangling reference", async () => { + // A literal that happens to carry a `format` key. Only the exact value is + // the value, and the key is data rather than an annotation. + const literal = executor(); + await suspended(literal, SUSPENSION, { const: { format: "email", x: 1 } }); + expect( + ( + await deliver( + literal, + delivery({ + answer: canonicalJson({ format: "email", x: 1 }), + secretDetection: false, + }), + ) + )["outcome"], + ).toBe("performed"); + expect((await on(literal, (owner) => owner.retainedAnswers()))[0]?.["answer"]).toBe( + canonicalJson({ format: "email", x: 1 }), + ); + + const altered = executor(); + await suspended(altered, SUSPENSION, { const: { format: "email", x: 1 } }); + const before = await on(altered, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + executions: owner.executionRows(), + private: owner.scratch(), + })); + expect( + ( + await deliver( + altered, + delivery({ answer: canonicalJson({ x: 1 }), secretDetection: false }), + ) + )["refusal"], + ).toBe("command:answer-rejected"); + expect( + await on(altered, (owner) => ({ + answers: owner.retainedAnswers(), + journal: owner.journalRecords(), + run: owner.runRow(), + executions: owner.executionRows(), + private: owner.scratch(), + })), + ).toEqual(before); + + // A property whose authored name is `format`. It is declared, so it is not + // an additional property, and the annotation beneath it constrains nothing. + const named = executor(); + await suspended(named, SUSPENSION, { + type: "object", + properties: { format: { type: "string", format: "email" } }, + required: ["format"], + additionalProperties: false, + }); + expect( + ( + await deliver( + named, + delivery({ answer: canonicalJson({ format: "not-email" }), secretDetection: false }), + ) + )["outcome"], + ).toBe("performed"); + + // A required member the value inherits rather than holds. + const inherited = executor(); + await suspended(inherited, SUSPENSION, { + type: "object", + properties: { toString: { type: "string" } }, + required: ["toString"], + additionalProperties: false, + }); + expect( + (await deliver(inherited, delivery({ answer: canonicalJson({}), secretDetection: false })))[ + "refusal" + ], + ).toBe("command:answer-rejected"); + + // A reference the schema does not define is unusable, so nothing is judged + // against it and nothing is retained. + const dangling = executor(); + await suspended(dangling, SUSPENSION, { + type: "object", + properties: { a: { $ref: "#/definitions/missing" } }, + }); + expect( + ( + await deliver( + dangling, + delivery({ answer: canonicalJson({ a: 1 }), secretDetection: false }), + ) + )["refusal"], + ).toBe("command:unjudgeable-schema"); + expect(await on(dangling, (owner) => owner.retainedAnswers())).toEqual([]); + }); + it("judges without generating code, on the path a delivery actually takes", async () => { const stub = executor(); await suspended(stub, SUSPENSION, { type: "number", multipleOf: 0.1 }); diff --git a/packages/workflow/tests/workflow-suspension-answer.test.ts b/packages/workflow/tests/workflow-suspension-answer.test.ts index 7adb4d014..1de23679f 100644 --- a/packages/workflow/tests/workflow-suspension-answer.test.ts +++ b/packages/workflow/tests/workflow-suspension-answer.test.ts @@ -22,6 +22,7 @@ import { expect } from "@executablemd/test-support/expect"; import { call, type Operation, race, scoped } from "effection"; import { DatabaseSync } from "node:sqlite"; import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import type { JsonObject } from "@executablemd/core"; import { collect, inlineSource, registerComponents } from "@executablemd/core"; import { executeInstalled } from "@executablemd/core/host"; import type { Result } from "effection"; @@ -190,9 +191,14 @@ function waiting(): Operation { return suspendFor({ request: REQUEST, responseSchema: SCHEMA }); } +/** The same, for a suite that needs a wait to retain a different schema. */ +function waitingFor(responseSchema: JsonObject): () => Operation { + return () => suspendFor({ request: REQUEST, responseSchema }); +} + /** One run left suspended at its wait, and the identity of that wait. */ -function* suspendedRun(root: string): Operation { - const attempted = yield* attempt(root, "start", waiting); +function* suspendedRun(root: string, body: () => Operation = waiting): Operation { + const attempted = yield* attempt(root, "start", body); const id = attempted.notice?.suspensionId; if (id === undefined) { throw new Error("the fixture document did not reach a durable wait"); @@ -543,3 +549,112 @@ function attemptEvents(root: string): Operation { } }); } + +/** + * Tier WAD — the schema semantics the local boundary judges by. + * + * The same judgment a run's owner makes, reached the way a person reaches it: + * a real run in a real file, and the production delivery provider. What is + * being proved is not the validator — `packages/core` owns that — but that this + * boundary is the one that uses it, on the cases where a careless adapter would + * differ. + */ +describe("what the local delivery boundary admits", () => { + it("keeps a literal that carries `format`, and refuses an altered one", function* () { + const root = yield* useStorageRoot(); + const id = yield* suspendedRun(root, waitingFor({ const: { format: "email", x: 1 } })); + + const exact = yield* deliver(root, { + suspensionId: id, + value: { format: "email", x: 1 }, + secretDetection: false, + }); + expect([exact.ok, exact.ok === false && String(exact.error)]).toEqual([true, false]); + expect(answerRows(root)[0]?.["answer"]).toBe(JSON.stringify({ format: "email", x: 1 })); + + // A second run, because the first now holds an answer. + const other = yield* useStorageRoot(); + const second = yield* suspendedRun(other, waitingFor({ const: { format: "email", x: 1 } })); + const altered = yield* deliver(other, { + suspensionId: second, + value: { x: 1 }, + secretDetection: false, + }); + expect(altered.ok).toBe(false); + expect(answerRows(other)).toEqual([]); + }); + + it("keeps a declared property named `format`", function* () { + const root = yield* useStorageRoot(); + const id = yield* suspendedRun( + root, + waitingFor({ + type: "object", + properties: { format: { type: "string", format: "email" } }, + required: ["format"], + additionalProperties: false, + }), + ); + + // Declared, so not an additional property; annotated, so not constrained. + const outcome = yield* deliver(root, { + suspensionId: id, + value: { format: "not-email" }, + secretDetection: false, + }); + + expect([outcome.ok, outcome.ok === false && String(outcome.error)]).toEqual([true, false]); + }); + + it("treats an inherited name as a member the value does not hold", function* () { + const root = yield* useStorageRoot(); + const id = yield* suspendedRun( + root, + waitingFor({ + type: "object", + properties: { toString: { type: "string" } }, + required: ["toString"], + additionalProperties: false, + }), + ); + + const missing = yield* deliver(root, { suspensionId: id, value: {}, secretDetection: false }); + expect(missing.ok).toBe(false); + expect(missing.ok === false && missing.error.message).toContain("toString"); + expect(answerRows(root)).toEqual([]); + + const held = yield* deliver(root, { + suspensionId: id, + value: JSON.parse('{"toString":"held"}'), + secretDetection: false, + }); + expect([held.ok, held.ok === false && String(held.error)]).toEqual([true, false]); + }); + + it("refuses a wait whose schema references what it does not define", function* () { + const root = yield* useStorageRoot(); + const id = yield* suspendedRun( + root, + waitingFor({ type: "object", properties: { a: { $ref: "#/definitions/missing" } } }), + ); + + const outcome = yield* deliver(root, { + suspensionId: id, + value: { a: 1 }, + secretDetection: false, + }); + + expect(outcome.ok).toBe(false); + expect(answerRows(root)).toEqual([]); + }); + + it("admits the value draft-07 admits, non-representable steps included", function* () { + const root = yield* useStorageRoot(); + const id = yield* suspendedRun(root, waitingFor({ type: "number", multipleOf: 0.1 })); + + const outcome = yield* deliver(root, { suspensionId: id, value: 0.3, secretDetection: false }); + + expect([outcome.ok, outcome.ok === false && String(outcome.error)]).toEqual([true, false]); + expect(answerRows(root)[0]?.["answer"]).toBe("0.3"); + }); +}); From ae7d2b6de4bea303b538852e0f7b5ee89303ad76 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Tue, 8 Sep 2026 20:41:30 -0300 Subject: [PATCH 62/81] =?UTF-8?q?=E2=9C=A8=20Replay=20a=20completed=20run?= =?UTF-8?q?=20from=20what=20its=20owner=20holds,=20and=20reach=20nothing?= =?UTF-8?q?=20else?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A completed run asked to run again answers from the terminal its journal records: it imports nothing, performs nothing and appends nothing. But canonical execution still has to be given a root document, and locally that value came out of Git because a checkout was there. A run whose durable owner is somewhere else has no checkout at all, and fetching one for a replay that imports nothing is live retrieval performed for a document nobody reads. So it comes from the run. `retainedReplay()` reads the root the run recorded — the retained `__root__` import, or the binding core writes into a terminal created before any import — and holds its path to the definition the run record retains, which is the half of the comparison that never came from the journal. The bundle is the same judgment made narrower: a completed replay is handed no component source and no authority to resolve a name, only the admission that holds every recorded import to the name, canonical path and object id the definition declares. A declared member the history never imported is neither read nor fetched. The shared CLI decides that from the frontier its owner answers with, right after admission, and therefore ahead of every live-only construction below it: the retained definition, this host's own adapter, the suspension controller, the answer provider, the `` declaration and `host.attach()`. A lifecycle row claiming completion over a history that records no result, a history continuing past the result it records, a missing or malformed root import, and a retained root naming another document each refuse closed. --- packages/cli/src/workflow.ts | 185 ++-- packages/cli/tests/workflow-cli.test.ts | 74 +- .../cli/tests/workflow-installation.test.ts | 26 +- packages/cli/tests/workflow-replay.test.ts | 849 ++++++++++++++++++ .../cli/tests/workflow-suspension.test.ts | 13 +- packages/core/host.ts | 12 + packages/workflow/mod.ts | 2 + packages/workflow/src/bundle.ts | 93 +- packages/workflow/src/replay.ts | 306 +++++++ .../tests/cloudflare/remote-replay.vitest.ts | 382 ++++++++ packages/workflow/tests/replay-inputs.test.ts | 406 +++++++++ scripts/runtime-test-exclusions.ts | 6 + 12 files changed, 2277 insertions(+), 77 deletions(-) create mode 100644 packages/cli/tests/workflow-replay.test.ts create mode 100644 packages/workflow/src/replay.ts create mode 100644 packages/workflow/tests/cloudflare/remote-replay.vitest.ts create mode 100644 packages/workflow/tests/replay-inputs.test.ts diff --git a/packages/cli/src/workflow.ts b/packages/cli/src/workflow.ts index ebed19b44..36d5172a6 100644 --- a/packages/cli/src/workflow.ts +++ b/packages/cli/src/workflow.ts @@ -74,7 +74,9 @@ import type { PropsSchema } from "@executablemd/core"; import type { RootDocumentSource } from "@executablemd/core"; import { definitionComponents, + retainedReplay, retainedWorkflowInstallation, + replaysRetainedResult, workflowBundleInstallation, WORKFLOW_RUN_STATUSES, WorkflowLifecycle, @@ -82,7 +84,9 @@ import { import type { ExecutionInstallation } from "@executablemd/core/host"; import type { ExecutorLock, + RetainedReplay, WorkflowRunDatabase, + WorkflowRunRecord, WorkflowRunStatus, WorkflowStopReason, } from "@executablemd/workflow"; @@ -91,7 +95,11 @@ import type { WorkflowExecutionTransitions, WorkflowRunCreation, } from "@executablemd/workflow"; -import type { SuspensionControllerOptions, SuspensionNotice } from "@executablemd/workflow/deno"; +import type { + SuspensionController, + SuspensionControllerOptions, + SuspensionNotice, +} from "@executablemd/workflow/deno"; import { SUSPENSION_REQUEST } from "@executablemd/workflow"; import { describeError } from "./props.ts"; import { preflightFork } from "./workflow-fork.ts"; @@ -421,10 +429,9 @@ function reportStatus(status: WorkflowRunStatus): void { report(`workflow status: ${status}`); } -/** Whether this journal already holds the root's terminal event. */ -function* isCompleted(stream: DurableStream): Operation { - const events = yield* stream.readAll(); - return events.some((event) => event.type === "close" && event.coroutineId === "root"); +/** Whether one retained event is the root's terminal. */ +function isRootClose(event: DurableEvent): boolean { + return event.type === "close" && event.coroutineId === "root"; } /** @@ -985,13 +992,36 @@ export function runWorkflow( const { database, record, execution, replay } = begun.value; reportRun(record.runId); - // Only now, and only because execution or replay was admitted. - const source = yield* documentSource(start, database, reconstructed.value); - if (!source.ok) { - report(source.error.message); + // The frontier this run's owner answered with, read once and decided from. + // Everything below asks it the same two questions: whether a document + // result is already recorded, and — for a resume of a run whose retained + // state ended — what that result is a result of. + const frontier = yield* database.readJournalEntries(); + if (!frontier.ok) { + report(frontier.error.message); + return { exitCode: 1 }; + } + const completed = frontier.value.some((entry) => isRootClose(entry.event)); + + // A resume of a run that already ended replays what its own history holds. + // Decided here, from the admitted frontier, and therefore ahead of every + // live-only construction below it: the retained definition, this host's own + // adapter, the suspension controller, the answer provider, the `` + // declaration and `host.attach()`. Each of those is work for an execution + // that is going to import nothing, perform nothing and append nothing. + const replayed = replay && start === undefined; + const prepared = replayed + ? retainedReplay(record, frontier.value) + : yield* liveDocument(record, start, database, reconstructed.value); + if (!prepared.ok) { + report(prepared.error.message); return { exitCode: 1 }; } + // Nothing beyond this point loads for a completed replay: no controller, no + // answer provider, and no import of the adapter either one comes from. + const support = replayed ? undefined : yield* liveSupport(database); + // Interruption is the outcome nothing else publishes. Registered before the // execution starts, so a scope torn down by Ctrl-C settles the run rather // than leaving a record with no end and a status of `running`. The executor @@ -1001,19 +1031,6 @@ export function runWorkflow( // "this invocation is durably settled" are different facts, and collapsing // them is how a post-execution storage refusal would be republished as an // interruption. Teardown speaks only while the phase is still `running`. - // Imported where it is used rather than at the top of this module. This - // file is on the ordinary `xmd run` path too, and the Deno workflow adapter - // reaches `node:sqlite` — which Node greets with an experimental warning on - // standard error the moment it loads. A run that opens no workflow storage - // should not be announcing that it might have. - // `evaluationComponents` comes through the same import, and for the same - // reason: `` is the workflow host's component, and a run that - // opens no workflow storage must not load the adapter that reaches - // `node:sqlite` — which Bun does not have at all. - const { createSuspensionController, evaluationComponents } = yield* until( - import("@executablemd/workflow/deno"), - ); - const suspension = createSuspensionController({ database }); const phase: LifecyclePhase = { state: "running" }; yield* ensure(function* () { if (phase.state !== "running") { @@ -1040,35 +1057,23 @@ export function runWorkflow( reportStatus("interrupted"); }); - const completed = yield* isCompleted(database.journal); const documentExecution: WorkflowExecution = { - root: retainedSource(record.definition.rootDocumentPath, source.value.source), + root: prepared.value.root, props: record.props, stream: database.journal, // The run already exists: the begin transition created or found it before - // anything executed, so this installation records exactly that value, - // allocates nothing and never consults Git. Service denial is installed - // beside it, through the same host-service slot `xmd run` fills with a + // anything executed, so these installations record exactly that value, + // allocate nothing and never consult Git. Service denial is installed + // beside them, through the same host-service slot `xmd run` fills with a // real adapter. installations: [ - retainedWorkflowInstallation({ - runId: record.runId, - base: record.base, - pinnedCommit: record.definition.objectId, - }), - // The bundle this run is a run of, when it is a run of one. Both start - // and resume install it, and a completed replay installs it too: the - // retained history is held to the same components before its recorded - // output is accepted. - ...(source.value.components.length === 0 - ? [] - : [workflowBundleInstallation(source.value.components)]), + ...prepared.value.installations, // `` names durable work after its own invocation, so this run // declares it to the execution and canonical execution builds it from // the claimant it minted for this attachment. Declared where the // Workspace is attached — a completed replay restores its retained // output and expands nothing, so it needs no component of its own. - ...(completed || replay ? [] : [{ components: evaluationComponents(database) }]), + ...(support === undefined || completed || replay ? [] : [support.declaration]), ], around(operation: Operation): Operation { // A completed run replays its retained output and result. Attaching a @@ -1078,10 +1083,10 @@ export function runWorkflow( // // The suspension controller owns the scope *inside* the attachment, so // halting a suspended execution tears the Workspace down with it and - // nothing survives the settlement. - return completed || replay - ? suspension.own(operation) - : host.attach(database, suspension.own(operation)); + // nothing survives the settlement. A completed replay has no controller + // to own anything: it reaches no wait, so there is none to construct. + const owned = support === undefined ? operation : support.controller.own(operation); + return completed || replay ? owned : host.attach(database, owned); }, }; @@ -1097,10 +1102,17 @@ export function runWorkflow( // swallowed by a halt. An execution whose teardown failed did not reach a // durable wait, and `suspended` is never claimed for one. const attempted = yield* attempt(documentExecution, execute); - const waiting = suspension.reported() && !attempted.ok && suspension.entered(attempted.error); - const settlement: Settlement = waiting - ? { kind: "suspension", notice: yield* suspension.notice } - : { kind: "document", result: attempted }; + const notice = + support !== undefined && + support.controller.reported() && + !attempted.ok && + support.controller.entered(attempted.error) + ? yield* support.controller.notice + : undefined; + const settlement: Settlement = + notice === undefined + ? { kind: "document", result: attempted } + : { kind: "suspension", notice }; // The document is over, whatever storage does next — so teardown must not // relabel this run interrupted, even if what follows refuses. @@ -1343,12 +1355,40 @@ function* inheritedProps( } /** - * The document this run executes. + * The document a live or partial execution runs, and what it is held to. * * A `start` already established it from Git to read what the pinned document * declares. A resume fetches what the run retained, and only once the run has - * been admitted — a run that ended is not one to fetch a definition for. + * been admitted — a run that ended is not one to fetch a definition for, which + * is why a completed replay never arrives here at all. */ +function* liveDocument( + record: WorkflowRunRecord, + start: WorkflowStart | undefined, + database: WorkflowRunDatabase, + reconstructed: RetainedSources | undefined, +): Operation> { + const sources = yield* documentSource(start, database, reconstructed); + if (!sources.ok) { + return sources; + } + return Ok({ + root: retainedSource(record.definition.rootDocumentPath, sources.value.source), + installations: [ + retainedWorkflowInstallation({ + runId: record.runId, + base: record.base, + pinnedCommit: record.definition.objectId, + }), + // The bundle this run is a run of, when it is a run of one — the sources + // to import from and the admission every retained import is held to. + ...(sources.value.components.length === 0 + ? [] + : [workflowBundleInstallation(sources.value.components)]), + ], + }); +} + function* documentSource( start: WorkflowStart | undefined, database: WorkflowRunDatabase, @@ -1363,6 +1403,36 @@ function* documentSource( return yield* loadRetainedDefinition(database.record.definition, database.retrieval?.metadata); } +/** What a live or partial execution needs from this host's own adapter. */ +interface LiveSupport { + readonly controller: SuspensionController; + /** The declaration `` reaches this run's durable work through. */ + readonly declaration: ExecutionInstallation; +} + +/** + * The controller a live or partial execution waits through, and the component + * declaration that goes with it. + * + * Imported where it is used rather than at the top of this module, for two + * reasons that point the same way. This file is on the ordinary `xmd run` path + * too, and the Deno workflow adapter reaches `node:sqlite` — which Node greets + * with an experimental warning on standard error the moment it loads, and which + * Bun does not have at all; a run that opens no workflow storage should not be + * announcing that it might have. And a completed replay reaches no wait and + * expands no ``, so it needs neither of these — which is what lets a + * host whose runs live somewhere else replay one without this adapter existing. + */ +function* liveSupport(database: WorkflowRunDatabase): Operation { + const { createSuspensionController, evaluationComponents } = yield* until( + import("@executablemd/workflow/deno"), + ); + return { + controller: createSuspensionController({ database }), + declaration: { components: evaluationComponents(database) }, + }; +} + /** * The pinned sources a resumed run closed over a bundle needs before it begins. * @@ -1374,6 +1444,11 @@ function* documentSource( * A run this host cannot inspect answers with nothing too. What that run is, * and whether this action may advance it, is the begin transition's to decide, * and answering it here would report a different refusal for the same fact. + * + * So does a run whose retained state already ended, and for a reason of its + * own: that run replays what its history holds and imports nothing, so a bundle + * fetched for it would be live retrieval performed for components nobody + * resolves. */ function* reconstructedSources( request: WorkflowRequest, @@ -1386,6 +1461,16 @@ function* reconstructedSources( if (!snapshot.ok) { return Ok(undefined); } + // A run whose retained state already ended replays from its own history, and + // the sources that replay is held to come out of that history rather than out + // of a checkout. Reconstructing a bundle from Git for it would be live + // retrieval performed for components nothing is going to import. Whether the + // history actually describes a completed run is decided after admission, from + // the frontier the owner answers with, and a run that claims to have ended + // without one refuses there rather than falling back to this. + if (replaysRetainedResult(snapshot.value.record.status)) { + return Ok(undefined); + } const { definition } = snapshot.value.record; if (definitionComponents(definition).length === 0) { return Ok(undefined); diff --git a/packages/cli/tests/workflow-cli.test.ts b/packages/cli/tests/workflow-cli.test.ts index 422469f38..5835fd620 100644 --- a/packages/cli/tests/workflow-cli.test.ts +++ b/packages/cli/tests/workflow-cli.test.ts @@ -585,6 +585,17 @@ const LOOP_ROOT = [ "", ].join("\n"); +/** A checkpoint that waits, so one bundled run can be resumed while still live. */ +const WAITING_CHECKPOINT = [ + "checkpoint reached.", + "", + '', + "Proceed with the change?", + "", + "", +].join("\n"); + const LOOP_FILES: Record = { "flows/loop.md": LOOP_ROOT, "flows/InstructionFiles.md": "instruction files listed.\n", @@ -763,29 +774,72 @@ describe("Tier WFC — a workflow closed over a component bundle", () => { }); it("WFC19: a resume whose pinned components are unreachable is refused whole", function* () { + // A run that has not ended is the case this is about. It continues by + // importing the components its definition pins, so it reconstructs them + // from the repository — and a repository that is gone refuses the resume + // rather than continuing under whatever is there now. A run that already + // ended imports nothing and asks the repository nothing, which is WFC20. + yield* useFixture( + { ...LOOP_FILES, "flows/UserCheckpoint.md": WAITING_CHECKPOINT }, + function* (fixture) { + const started = yield* xmd(fixture, [ + "workflow", + "start", + "--id=loop-4", + "flows/loop.md", + ]).join(); + expect(started.code).toBe(2); + expect(reportedStatus(started.stderr)).toBe("suspended"); + + const before = yield* xmd(fixture, ["workflow", "history", "loop-4", "--json"]).join(); + + // The repository this run retains is no longer a repository. + yield* rm(join(fixture.repository, ".git"), { recursive: true, force: true }); + + const resumed = yield* xmd(fixture, ["workflow", "resume", "loop-4"]).join(); + + expect(resumed.code).toBe(1); + expect(reportedStatus(resumed.stderr)).toBeUndefined(); + + // Its lifecycle records are exactly what they were: the refusal happened + // before an execution was recorded. + yield* git(fixture.repository, ["init", "-q", "--initial-branch=main", "."]); + const after = yield* xmd(fixture, ["workflow", "history", "loop-4", "--json"]).join(); + expect(after.stdout).toBe(before.stdout); + }, + ); + }); + + it("WFC20: a completed bundled run replays with no repository at all", function* () { yield* useFixture(LOOP_FILES, function* (fixture) { const started = yield* xmd(fixture, [ "workflow", "start", - "--id=loop-4", + "--id=loop-5", "flows/loop.md", ]).join(); expect(started.code).toBe(0); + const before = yield* xmd(fixture, ["workflow", "history", "loop-5", "--json"]).join(); - const before = yield* xmd(fixture, ["workflow", "history", "loop-4", "--json"]).join(); - - // The repository this run retains is no longer a repository. + // Not a stale checkout and not a rewritten object: no repository. A + // completed replay restores what the run retained, so there is nothing + // here for it to read and nothing it asks for. yield* rm(join(fixture.repository, ".git"), { recursive: true, force: true }); + for (const name of Object.keys(LOOP_FILES)) { + yield* rm(join(fixture.repository, name), { force: true }); + } - const resumed = yield* xmd(fixture, ["workflow", "resume", "loop-4"]).join(); + const resumed = yield* xmd(fixture, ["workflow", "resume", "loop-5"]).join(); - expect(resumed.code).toBe(1); - expect(reportedStatus(resumed.stderr)).toBeUndefined(); + expect(resumed.code).toBe(0); + expect(reportedStatus(resumed.stderr)).toBe("completed"); + // Every stage the run recorded, in the order it recorded them. + expect(resumed.stdout).toContain("discovered."); + expect(resumed.stdout).toContain("instruction files listed."); + expect(resumed.stdout).toContain("implemented."); - // Its lifecycle records are exactly what they were: the refusal happened - // before an execution was recorded. yield* git(fixture.repository, ["init", "-q", "--initial-branch=main", "."]); - const after = yield* xmd(fixture, ["workflow", "history", "loop-4", "--json"]).join(); + const after = yield* xmd(fixture, ["workflow", "history", "loop-5", "--json"]).join(); expect(after.stdout).toBe(before.stdout); }); }); diff --git a/packages/cli/tests/workflow-installation.test.ts b/packages/cli/tests/workflow-installation.test.ts index f2a71bf0d..f1091a3fe 100644 --- a/packages/cli/tests/workflow-installation.test.ts +++ b/packages/cli/tests/workflow-installation.test.ts @@ -30,6 +30,7 @@ import type { WorkflowExecutionTransitions } from "@executablemd/workflow"; import { Git, WorkflowLifecycle, WorkflowRunStorage } from "@executablemd/workflow"; import type { WorkflowRunDatabase, WorkflowRunStatus } from "@executablemd/workflow"; import type { Json } from "@executablemd/core"; +import type { DurableEvent } from "@executablemd/durable-streams"; import { runWorkflow } from "../src/workflow.ts"; import type { WorkflowExecution, WorkflowHost, WorkflowRequest } from "../src/workflow.ts"; @@ -161,14 +162,31 @@ function refusingHost(root: string, refuse: "settle" | "none", attempted: string }; } +/** + * The root import a document execution records before anything else. + * + * A completed replay is held to it: the retained selection is what says which + * document the recorded result is a result of, and a history that closes the + * root without one describes a run that never imported anything. + */ +function rootImport(path: string, content: string): DurableEvent { + return { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path, content } }, + }; +} + /** Record a root terminal, so the next pass over this journal is a replay. */ -function* closeRoot(root: string, runId: string): Operation { +function* closeRoot(root: string, runId: string, contents: string): Operation { yield* scoped(function* () { yield* useWorkflowRunHost({ root }); const found = yield* WorkflowRunStorage.operations.lookup(runId); if (!found.ok) { throw found.error; } + yield* found.value.journal.append(rootImport("workflow.md", contents)); yield* found.value.journal.append({ type: "close", coroutineId: "root", @@ -326,7 +344,9 @@ describe("Tier WFI — what a run hands to canonical core", () => { recordingHost(root, attached), function* (execution): Operation> { executions += 1; - // Close the root, so the next pass is a completed replay. + // Close the root, so the next pass is a completed replay — behind the + // import that says which document the result is a result of. + yield* execution.stream.append(rootImport("workflow.md", created.contents)); yield* execution.stream.append({ type: "close", coroutineId: "root", @@ -423,7 +443,7 @@ describe("Tier WFI — what a run hands to canonical core", () => { const created = yield* startedRun(root); yield* useGit(created.repository, created.objectId, created.contents); if (admitted === "completed") { - yield* closeRoot(root, created.runId); + yield* closeRoot(root, created.runId, created.contents); } yield* endRun(root, created.runId, admitted); yield* runWorkflow( diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts new file mode 100644 index 000000000..ea4e4abba --- /dev/null +++ b/packages/cli/tests/workflow-replay.test.ts @@ -0,0 +1,849 @@ +/** + * Tier WRH12 — what a completed run reaches when it is asked to run again. + * + * The rule is easy to state and easy to get wrong in one direction: a completed + * replay may read the run's own storage, because that is where the result is, + * and may reach nothing else. So this drives `runWorkflow()` — the same + * orchestration the shared CLI drives — with canonical core underneath it, and + * makes every other boundary fail if it is entered: the Git capability throws + * on every question, and the host's `attach()` throws when it is called at all. + * + * A completed replay under those conditions is not "a run that happened to + * work". It is a run that could not have consulted a checkout, could not have + * opened a Workspace, and produced the retained bytes anyway. + * + * The local Deno host is the oracle here rather than the subject. What the + * remote owner does with the same reads is proved against a real Durable Object + * in `packages/workflow/tests/cloudflare/remote-replay.vitest.ts`; what the + * shared decision does with retained values is proved over values in + * `packages/workflow/tests/replay-inputs.test.ts`. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { call, ensure, Err, Ok, resource, scoped } from "effection"; +import type { Operation, Result } from "effection"; +import { rm, writeTextFile } from "@effectionx/fs"; +import { exec } from "@effectionx/process"; +import { mkdtemp } from "node:fs/promises"; +import { until } from "effection"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { DatabaseSync } from "node:sqlite"; +import type { Json } from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import { + useWorkflowInputDelivery, + useWorkflowLifecycle, + useWorkflowRunHost, + withWorkflowWorkspace, + workflowRunPath, +} from "@executablemd/workflow/deno"; +import type { WorkflowExecutionTransitions } from "@executablemd/workflow"; +import { Git, WorkflowLifecycle } from "@executablemd/workflow"; +import type { + WorkflowHistoryEntry, + WorkflowRunDatabase, + WorkflowRunStatus, +} from "@executablemd/workflow"; +import { establishDefinition } from "../src/workflow-definition.ts"; +import { runWorkflow } from "../src/workflow.ts"; +import { runWorkflowManagement } from "../src/workflow-management.ts"; +import type { + WorkflowExecution, + WorkflowHost, + WorkflowRequest, + WorkflowStart, +} from "../src/workflow.ts"; + +const REQUEST: WorkflowRequest = { + action: "start", + target: "workflow.md", + id: undefined, + verbose: false, + raw: false, + secretDetection: false, +}; + +const CHECKPOINT_SCHEMA = + '{"type":"object","properties":{"proceed":{"type":"boolean"}},"required":["proceed"]}'; + +/** A document with no wait and no effect: the smallest completed run. */ +const PLAIN = "# Retained\n\nthe run recorded this line.\n"; + +/** A document that fails after the root import, so its terminal carries an error. */ +const FAILING = "# Retained\n\npartial line.\n\n\n"; + +/** A document that waits, so a run can be observed while it has not ended. */ +const WAITING = [ + "# Retained", + "", + "before the wait.", + "", + ``, + "Proceed with the change?", + "", + "", + "decision: {decision.proceed}", + "", +].join("\n"); + +/** The same wait, inside a root closed over a component it must reconstruct. */ +const BUNDLED_WAITING = [ + "---", + "workflow:", + " components:", + " Stage: ./Stage.md", + "---", + "", + "# Retained", + "", + "", + "", + ``, + "Proceed with the change?", + "", + "", + "decision: {decision.proceed}", + "", +].join("\n"); + +/** A root closed over two components, one of which it never invokes. */ +const BUNDLED = [ + "---", + "workflow:", + " components:", + " Stage: ./Stage.md", + " Unused: ./Unused.md", + "---", + "", + "# Retained", + "", + "", + "", +].join("\n"); + +interface Fixture { + readonly repository: string; + readonly runs: string; +} + +function* git(repository: string, args: string[]): Operation { + const result = yield* exec("git", { arguments: args, cwd: repository }).expect(); + if (result.code !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${result.stderr}`); + } + return result.stdout; +} + +/** One committed definition and one empty run store, both this case's own. */ +function useFixture(source: string, components: Record = {}): Operation { + return resource(function* (provide) { + const repository = yield* until(mkdtemp(join(tmpdir(), "xmd-wrp-repo-"))); + const runs = yield* until(mkdtemp(join(tmpdir(), "xmd-wrp-runs-"))); + yield* ensure(function* () { + yield* rm(repository, { recursive: true, force: true }); + yield* rm(runs, { recursive: true, force: true }); + }); + yield* git(repository, ["init", "--quiet"]); + yield* git(repository, ["config", "user.email", "wrp@example.test"]); + yield* git(repository, ["config", "user.name", "WRP"]); + yield* writeTextFile(join(repository, "workflow.md"), source); + yield* git(repository, ["add", "workflow.md"]); + for (const [name, content] of Object.entries(components)) { + yield* writeTextFile(join(repository, `${name}.md`), content); + yield* git(repository, ["add", `${name}.md`]); + } + yield* git(repository, ["-c", "commit.gpgsign=false", "commit", "--quiet", "-m", "definition"]); + yield* provide({ repository, runs }); + }); +} + +/** The Git capability, answered from the fixture repository itself. */ +function useRepositoryGit(repository: string): Operation { + return Git.around( + { + // deno-lint-ignore require-yield + *repositoryRoot(): Operation { + return repository; + }, + *revParse([revision]): Operation { + return (yield* git(repository, [ + "rev-parse", + "--verify", + "--end-of-options", + revision, + ])).trim(); + }, + *readObject([commit, path]): Operation { + return yield* git(repository, ["cat-file", "blob", `${commit}:${path}`]); + }, + // deno-lint-ignore require-yield + *objectFormat(): Operation<"sha1" | "sha256"> { + return "sha1"; + }, + }, + { at: "min" }, + ); +} + +/** + * A Git capability that answers nothing and records being asked. + * + * The point of the recording is that the assertion can be about the question + * rather than about the answer: a replay that reached here would be refused, + * and `asked` says which question it reached with. + */ +function useRefusingGit(asked: string[]): Operation { + const refuse = (question: string): never => { + asked.push(question); + throw new Error(`PLANTED-GIT-REACHED: ${question}`); + }; + return Git.around( + { + // deno-lint-ignore require-yield + *repositoryRoot(): Operation { + return refuse("repositoryRoot"); + }, + // deno-lint-ignore require-yield + *revParse(): Operation { + return refuse("revParse"); + }, + // deno-lint-ignore require-yield + *readObject(): Operation { + return refuse("readObject"); + }, + // deno-lint-ignore require-yield + *objectFormat(): Operation<"sha1" | "sha256"> { + return refuse("objectFormat"); + }, + }, + { at: "min" }, + ); +} + +/** The production local host, recording each attachment it opens. */ +function liveHost(runs: string, attached: string[]): WorkflowHost { + return { + useRunHost(): Operation { + return useWorkflowRunHost({ root: runs }); + }, + useLifecycle(): Operation { + return useWorkflowLifecycle({ root: runs }); + }, + useDelivery(): Operation { + return useWorkflowInputDelivery({ root: runs }); + }, + attach(database: WorkflowRunDatabase, operation: Operation): Operation { + attached.push(database.record.runId); + return withWorkflowWorkspace(database, operation); + }, + }; +} + +/** The same host, with the one boundary a completed replay must never enter. */ +function replayHost(runs: string, attached: string[]): WorkflowHost { + const live = liveHost(runs, attached); + return { + useRunHost: live.useRunHost, + useLifecycle: live.useLifecycle, + useDelivery: live.useDelivery, + attach(): Operation { + attached.push("attach"); + throw new Error("PLANTED-ATTACHMENT-REACHED"); + }, + }; +} + +interface Invocation { + readonly exitCode: number; + readonly out: string[]; + readonly err: string[]; +} + +/** One `runWorkflow()` invocation, with what it reported on each stream. */ +function invoke( + request: WorkflowRequest, + start: WorkflowStart | undefined, + host: WorkflowHost, + execute: (execution: WorkflowExecution) => Operation>, +): Operation { + return scoped(function* () { + const out: string[] = []; + const err: string[] = []; + const log = console.log; + const error = console.error; + yield* ensure(() => { + console.log = log; + console.error = error; + }); + console.log = (...parts: unknown[]) => out.push(parts.map((part) => String(part)).join(" ")); + console.error = (...parts: unknown[]) => err.push(parts.map((part) => String(part)).join(" ")); + const outcome = yield* runWorkflow(request, start, host, execute); + return { exitCode: outcome.exitCode, out, err }; + }); +} + +/** What one document execution was given, what it rendered, and how it ended. */ +interface Rendered { + root: unknown; + output: string; + result: Result | undefined; +} + +/** The pinned document, executed as this run's root through canonical core. */ +function pinnedBody(seen: Rendered[]): (execution: WorkflowExecution) => Operation> { + return function* (execution): Operation> { + return yield* execution.around( + call(function* (): Operation> { + const running = yield* executeInstalled( + { ...execution.root, stream: execution.stream, props: execution.props }, + execution.installations, + ); + // The close value of the output stream is the complete or partial + // rendered text, so a failed execution still reports what it rendered. + const subscription = yield* running.output; + let next = yield* subscription.next(); + while (!next.done) { + next = yield* subscription.next(); + } + const result = yield* running; + seen.push({ root: { ...execution.root }, output: next.value, result }); + return result.ok ? Ok(undefined) : Err(new Error(String(result.error.message))); + }), + ); + }; +} + +/** What `xmd workflow start` establishes, through the command's own module. */ +function* startFor(fixture: Fixture): Operation { + const established = yield* establishDefinition(join(fixture.repository, "workflow.md")); + if (!established.ok) { + throw established.error; + } + return { established: established.value, props: {}, propsSchema: {} }; +} + +/** The run id one invocation reported, or the empty string when it reported none. */ +function runIdOf(invocation: Invocation): string { + const line = invocation.err.find((entry) => entry.startsWith("workflow run: ")); + return line === undefined ? "" : line.slice("workflow run: ".length).trim(); +} + +/** What one invocation published as this run's status, if anything. */ +function statusOf(invocation: Invocation): string | undefined { + const line = invocation.err.find((entry) => entry.startsWith("workflow status: ")); + return line === undefined ? undefined : line.slice("workflow status: ".length).trim(); +} + +/** Everything about a run a replay must not move, read through the host itself. */ +interface Retained { + readonly status: WorkflowRunStatus; + readonly executions: number; + readonly currentWorkspaceRootId: string; + readonly journal: string; +} + +function* retained(runs: string, runId: string): Operation { + return yield* scoped(function* () { + yield* useWorkflowLifecycle({ root: runs }); + const snapshot = yield* WorkflowLifecycle.operations.inspect(runId); + if (!snapshot.ok) { + throw snapshot.error; + } + const history = yield* WorkflowLifecycle.operations.history(runId); + if (!history.ok) { + throw history.error; + } + return { + status: snapshot.value.record.status, + executions: snapshot.value.executions.length, + currentWorkspaceRootId: snapshot.value.currentWorkspaceRootId, + // Identity and content of every retained row, in order: a length would + // not notice one rewritten under a new id. + journal: JSON.stringify( + history.value.map((entry: WorkflowHistoryEntry) => [ + entry.eventId, + entry.workspaceRootId, + entry.event, + ]), + ), + }; + }); +} + +/** The retained answers this run holds, read the way something outside XMD would. */ +function answers(runs: string, runId: string): { suspensionId: string; state: string }[] { + const database = new DatabaseSync(workflowRunPath(runs, runId), { readOnly: true }); + try { + return database + .prepare("SELECT suspension_id, state FROM workflow_suspension_answers ORDER BY rowid") + .all() + .map((row) => ({ + suspensionId: String(row["suspension_id"]), + state: String(row["state"]), + })); + } finally { + database.close(); + } +} + +/** How many `suspension_answer` events this run's history holds. */ +function* acceptedAnswers(runs: string, runId: string): Operation { + return yield* scoped(function* () { + yield* useWorkflowLifecycle({ root: runs }); + const history = yield* WorkflowLifecycle.operations.history(runId); + if (!history.ok) { + throw history.error; + } + return history.value.filter( + (entry) => + entry.event.type === "yield" && entry.event.description.type === "suspension_answer", + ).length; + }); +} + +describe("what a completed run reaches when it is asked to run again", () => { + it("WRP1: replays the retained result with no repository and no Workspace", function* () { + const asked: string[] = []; + const attached: string[] = []; + const live: Rendered[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(PLAIN); + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody(live), + ); + }); + expect(started.exitCode).toBe(0); + const runId = runIdOf(started); + const before = yield* retained(fixture.runs, runId); + + // From here the repository answers nothing and the host attaches + // nothing. Either one being reached is a planted failure. + const resumed = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { started, resumed, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.resumed.exitCode).toBe(0); + expect(statusOf(outcome.resumed)).toBe("completed"); + // Nothing was asked of the repository, and the only attachment is the live + // run's own. + expect(asked).toEqual([]); + expect(attached).toHaveLength(1); + + // The document canonical execution was handed is the one the run recorded, + // reported by the path its definition names. Not a placeholder, not the + // working tree, and not an empty source. + expect(replayed[0]?.root).toEqual({ path: "workflow.md", source: PLAIN, retained: true }); + expect(replayed[0]?.root).toEqual(live[0]?.root); + + // Byte for byte, and the same result. + expect(replayed).toHaveLength(1); + expect(replayed[0]?.output).toBe(live[0]?.output); + expect(replayed[0]?.result?.ok).toBe(true); + expect(replayed[0]?.result?.ok === true && replayed[0]?.result.value).toEqual( + live[0]?.result?.ok === true ? live[0]?.result.value : undefined, + ); + + // The run is exactly where it was, apart from the one execution envelope + // the lifecycle records for the invocation that replayed it. + expect(outcome.after.status).toBe("completed"); + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + expect(outcome.after.executions).toBe(outcome.before.executions + 1); + }); + + it("WRP2: replays a retained failure, with the output it had rendered", function* () { + const asked: string[] = []; + const attached: string[] = []; + const live: Rendered[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(FAILING); + + // The document fails and its settlement never lands, so the run is left + // holding a terminal nothing published. The next acquisition recovers it + // from that terminal, which is how a failed document becomes a run whose + // retained state ends without ever being resumable through Git. + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + refusingSettlement(fixture.runs), + pinnedBody(live), + ); + }); + const runId = runIdOf(started); + expect(runId).not.toBe(""); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("running"); + + const resumed = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { resumed, before, after: yield* retained(fixture.runs, runId) }; + }); + + // The same failure the live execution produced, and the partial output it + // had rendered before it failed. + expect(replayed).toHaveLength(1); + expect(replayed[0]?.result?.ok).toBe(false); + expect(replayed[0]?.output).toBe(live[0]?.output); + expect(replayed[0]?.output).toContain("partial line."); + expect(outcome.resumed.exitCode).toBe(1); + + // No provider was reached and no row was added to the history. + expect(asked).toEqual([]); + expect(attached).toEqual([]); + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + // The run follows the outcome it actually had: the terminal it replayed + // says the document failed, and that is what the invocation publishes. + expect(statusOf(outcome.resumed)).toBe("failed"); + }); + + it("WRP3: replays a bundled run without reading one component", function* () { + const asked: string[] = []; + const attached: string[] = []; + const live: Rendered[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + // Declared, committed, and never invoked by the root. A replay may not + // fetch it, and its absence from the history is not a refusal. + Unused: "never imported.\n", + }); + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody(live), + ); + }); + expect(started.exitCode).toBe(0); + const runId = runIdOf(started); + const before = yield* retained(fixture.runs, runId); + + const resumed = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { resumed, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.resumed.exitCode).toBe(0); + expect(statusOf(outcome.resumed)).toBe("completed"); + expect(asked).toEqual([]); + expect(attached).toHaveLength(1); + expect(replayed[0]?.root).toEqual({ path: "workflow.md", source: BUNDLED, retained: true }); + expect(replayed[0]?.output).toBe(live[0]?.output); + expect(replayed[0]?.output).toContain("staged."); + expect(replayed[0]?.output).not.toContain("never imported."); + expect(outcome.after.journal).toBe(outcome.before.journal); + }); + + it("WRP4: refuses retained state that describes no completed run", function* () { + const asked: string[] = []; + const attached: string[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(PLAIN); + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + const runId = runIdOf(started); + + // A lifecycle row that says the run ended, over a history that records no + // result: the two cannot both be right, and neither is a replay. + yield* emptyJournal(fixture.runs, runId); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("completed"); + + const refused = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + return { refused, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.refused.exitCode).toBe(1); + // Refused before an attachment, a native operation or a definition read. + expect(asked).toEqual([]); + expect(attached).toHaveLength(1); + expect(outcome.refused.err.join(" ")).toContain("records no document result"); + // No status was published for a run whose status did not change, and the + // journal and Workspace frontier are exactly what they were. + expect(statusOf(outcome.refused)).toBeUndefined(); + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.status).toBe("completed"); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + }); + + it("WRP5: a run that has not ended still reconstructs and attaches", function* () { + const asked: string[] = []; + const attached: string[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED_WAITING, { Stage: "staged.\n" }); + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + expect(started.exitCode).toBe(2); + const runId = runIdOf(started); + expect(statusOf(started)).toBe("suspended"); + const before = yield* retained(fixture.runs, runId); + + // The same repository refusal a completed replay is indifferent to. A + // suspended run is not: it continues by importing, so it reads the + // definition and refuses whole when it cannot. + const refused = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + liveHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + return { refused, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.refused.exitCode).toBe(1); + // The repository was asked, which is the whole distinction. + expect(asked.length).toBeGreaterThan(0); + expect(statusOf(outcome.refused)).toBeUndefined(); + // Refused before it was admitted: the run is left exactly as it was, down + // to its status and the executions it had. + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.status).toBe("suspended"); + expect(outcome.after.executions).toBe(outcome.before.executions); + // One attachment, and it is the live start's. + expect(attached).toHaveLength(1); + }); + + it("WRP6: replays a recorded answer without consuming or appending another", function* () { + const asked: string[] = []; + const attached: string[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(WAITING); + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + expect(started.exitCode).toBe(2); + const runId = runIdOf(started); + const suspensionId = String( + started.err.find((line) => line.startsWith("workflow suspension: ")), + ) + .slice("workflow suspension: ".length) + .trim(); + + const delivered = yield* scoped(function* () { + const out: string[] = []; + const log = console.log; + yield* ensure(() => { + console.log = log; + }); + console.log = (...parts: unknown[]) => out.push(parts.map(String).join(" ")); + return yield* runWorkflowManagement( + { + action: "answer", + runId, + suspensionId, + value: { proceed: true }, + secretDetection: true, + }, + liveHost(fixture.runs, attached), + ); + }); + expect(delivered.exitCode).toBe(0); + + const finished = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + liveHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + expect(finished.exitCode).toBe(0); + + const before = yield* retained(fixture.runs, runId); + const answersBefore = answers(fixture.runs, runId); + const acceptedBefore = yield* acceptedAnswers(fixture.runs, runId); + + const replay = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { + replay, + before, + answersBefore, + acceptedBefore, + after: yield* retained(fixture.runs, runId), + answersAfter: answers(fixture.runs, runId), + acceptedAfter: yield* acceptedAnswers(fixture.runs, runId), + }; + }); + + expect(outcome.replay.exitCode).toBe(0); + expect(statusOf(outcome.replay)).toBe("completed"); + // The delivered value reached the document through the retained event. + expect(replayed[0]?.output).toContain("decision: true"); + expect(asked).toEqual([]); + + // One answer, still spent, and one accepted event — before and after. + expect(outcome.answersBefore).toEqual([ + { suspensionId: outcome.answersBefore[0]?.suspensionId ?? "", state: "consumed" }, + ]); + expect(outcome.answersAfter).toEqual(outcome.answersBefore); + expect(outcome.acceptedBefore).toBe(1); + expect(outcome.acceptedAfter).toBe(1); + expect(outcome.after.journal).toBe(outcome.before.journal); + }); + + it("WRP7: releases the run it replayed, and closed only its own envelope", function* () { + const attached: string[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(PLAIN); + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + const runId = runIdOf(started); + + yield* scoped(function* () { + yield* useRefusingGit([]); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + + // The acquisition ended with the invocation, so the next one takes it. + const second = yield* scoped(function* () { + yield* useWorkflowLifecycle({ root: fixture.runs }); + return yield* WorkflowLifecycle.operations.acquireExecutor(runId); + }); + return { second, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.second.ok).toBe(true); + expect(outcome.second.ok === true && outcome.second.value.kind).toBe("acquired"); + // Two envelopes: the run's own execution and the replay's. Both are closed, + // and the run is still the completed run it was. + expect(outcome.after.executions).toBe(2); + expect(outcome.after.status).toBe("completed"); + }); +}); + +/** The production host with the settlement its storage refuses. */ +function refusingSettlement(runs: string): WorkflowHost { + const live = liveHost(runs, []); + return { + *useRunHost(): Operation { + const transitions = yield* live.useRunHost(); + return { + begin: transitions.begin, + fork: transitions.fork, + stageFork: transitions.stageFork, + // deno-lint-ignore require-yield + *settle(): Operation> { + return Err(new Error("PLANTED-STORAGE-REFUSAL")); + }, + }; + }, + useLifecycle: live.useLifecycle, + useDelivery: live.useDelivery, + attach: live.attach, + }; +} + +/** + * Take every retained event out of one run's journal, leaving its lifecycle row + * saying the run ended. + * + * Damage rather than a scenario: what is under test is that the two halves are + * required to agree, and a run cannot be brought into that state by asking the + * lifecycle for it. + */ +function* emptyJournal(runs: string, runId: string): Operation { + const database = new DatabaseSync(workflowRunPath(runs, runId)); + try { + database.prepare("DELETE FROM journal_events").run(); + } finally { + database.close(); + } + yield* until(Promise.resolve(undefined)); +} diff --git a/packages/cli/tests/workflow-suspension.test.ts b/packages/cli/tests/workflow-suspension.test.ts index f496eb07d..7374d5980 100644 --- a/packages/cli/tests/workflow-suspension.test.ts +++ b/packages/cli/tests/workflow-suspension.test.ts @@ -41,7 +41,13 @@ import { until } from "effection"; import { join } from "node:path"; import { tmpdir } from "node:os"; import { DatabaseSync } from "node:sqlite"; -import { collect, execute, inlineSource, registerComponents } from "@executablemd/core"; +import { + collect, + execute, + inlineSource, + registerComponents, + retainedSource, +} from "@executablemd/core"; import { executeInstalled } from "@executablemd/core/host"; import { durableCall, InMemoryStream } from "@executablemd/durable-streams"; import { @@ -403,7 +409,10 @@ function body(): (execution: WorkflowExecution) => Operation> { try { yield* collect( yield* executeInstalled( - { ...inlineSource("\n"), stream: execution.stream }, + // Reported by the path this run's definition names, as the shared + // CLI reports it: a completed replay is held to that agreement, + // and an inline identity would be a document the run is not of. + { ...retainedSource("workflow.md", "\n"), stream: execution.stream }, execution.installations, ), ); diff --git a/packages/core/host.ts b/packages/core/host.ts index 0850b8202..0e351f690 100644 --- a/packages/core/host.ts +++ b/packages/core/host.ts @@ -70,6 +70,18 @@ export { executeInstalled } from "./src/execute.ts"; export type { ExecutionInstallation, JournalAdmission } from "./src/execute.ts"; export type { DurablePreparation } from "./src/document-request.ts"; +/** + * Text a host holds, as a root document reported by the path it came from. + * + * The same function the package root publishes, reached here because a host + * that supplies a root is often a host that cannot resolve the root barrel: it + * pulls the terminal renderer and the rest of the reader-facing surface, and a + * durable owner running inside a Worker has neither. This entrypoint already + * resolves canonical execution and nothing beyond it. + */ +export { retainedSource } from "./src/root-source.ts"; +export type { RetainedRootDocument } from "./src/root-source.ts"; + /** * What a trusted host declares to an execution when one of its components names * durable work after its own invocation — see `src/invocation-identity.ts`. diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index 18a0a049c..dfd04c6bb 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -62,6 +62,8 @@ export { export type { GitApi, GitObjectFormat } from "./src/git.ts"; export { getWorkflowRun, retainedWorkflowInstallation, workflowInstallation } from "./src/run.ts"; export { workflowBundleInstallation, WorkflowBundleHistoryError } from "./src/bundle.ts"; +export { replaysRetainedResult, retainedReplay, WorkflowReplayHistoryError } from "./src/replay.ts"; +export type { RetainedReplay } from "./src/replay.ts"; export type { WorkflowRun } from "./src/run.ts"; export { useWorkflowServiceDenial, WorkflowServiceDeniedError } from "./src/service-denial.ts"; diff --git a/packages/workflow/src/bundle.ts b/packages/workflow/src/bundle.ts index 9f91ad99b..4af79cf71 100644 --- a/packages/workflow/src/bundle.ts +++ b/packages/workflow/src/bundle.ts @@ -24,6 +24,11 @@ * own exact-origin check, which this neither repeats nor relaxes. And the root * import stays the root import — a repository selection under `__root__`, * already held to the run's exact root source by core. + * + * A completed replay takes the second half without the first, through + * `workflowBundleReplayInstallation()`: it imports nothing, so it is handed no + * source to import from, and its records are held to the name, path and object + * id the immutable definition declares. */ import type { DurableEvent, Json, Yield } from "@executablemd/durable-streams"; @@ -32,6 +37,7 @@ import type { JournalAdmission, WorkflowBundleComponent, } from "@executablemd/core/host"; +import type { WorkflowComponentEntry } from "./storage/definition.ts"; /** The root's own import, which is not a bundle member and is admitted elsewhere. */ const ROOT = "__root__"; @@ -62,6 +68,24 @@ const REFUSALS = { "A retained component import recorded a repository file, which a workflow run resolves none of.", } as const; +/** + * One component a run's definition declares, as an admission holds a retained + * import to it. + * + * `content` is the exact pinned source, and it is present exactly when this run + * may also import the component. A live or partial execution holds it, so a + * retained record that says something else about the same object is refused + * against the bytes themselves. A completed replay holds none — it imports + * nothing and is given nothing to import — and its records are held to the name, + * path and object id the immutable definition retains, which is the half of + * this comparison that never came from the journal. + */ +interface DeclaredComponent { + readonly path: string; + readonly sourceHash: string; + readonly content?: string; +} + /** * Read one journal-controlled value, or answer that reading it refused. * @@ -130,13 +154,14 @@ function importedValue(event: DurableEvent): { value: unknown } | undefined | ty * * Every branch is a decision about what the record *is*, taken before anything * is replayed from it. A declared name must have been recorded as a bundled - * component, with this bundle's exact path, hash, and source; an undeclared + * component, with the exact path and object id the definition declares — and + * with this run's exact source too, where this run holds one; an undeclared * name must not claim to be one; and a repository selection is admitted only * for the root, which core holds to the run's own root source. */ function admitImport( event: DurableEvent, - components: ReadonlyMap, + components: ReadonlyMap, ): void { const name = importedName(event); if (name === undefined || name === ROOT) { @@ -172,13 +197,18 @@ function admitImport( } } const read = (member: string) => reading(() => (record as Record)[member]); - if ( - read("path") !== declared.path || - read("sourceHash") !== declared.sourceHash || - read("content") !== declared.content - ) { + if (read("path") !== declared.path || read("sourceHash") !== declared.sourceHash) { throw new WorkflowBundleHistoryError(REFUSALS.mismatched); } + // The source only when this run holds one. A replay was given none, so + // comparing the record with itself is what there would be to do, and the + // object id above is the definition's rather than the history's. + if (declared.content !== undefined && read("content") !== declared.content) { + throw new WorkflowBundleHistoryError(REFUSALS.mismatched); + } + if (typeof read("content") !== "string") { + throw new WorkflowBundleHistoryError(REFUSALS.unreadable); + } return; } @@ -192,7 +222,7 @@ function admitImport( } } -function admits(components: ReadonlyMap): JournalAdmission { +function admits(components: ReadonlyMap): JournalAdmission { // deno-lint-ignore require-yield return function* (retained: readonly DurableEvent[]) { for (const event of retained) { @@ -222,11 +252,18 @@ export function workflowBundleInstallation( // Copied entry by entry at construction, so the authority this installation // carries is closed over these values rather than over an array the caller // still holds and could rewrite between installation and import. - const index = new Map( - components.map((component) => [ + const bundled = components.map((component) => + Object.freeze({ + name: component.name, + path: component.path, + sourceHash: component.sourceHash, + content: component.content, + }), + ); + const index = new Map( + bundled.map((component) => [ component.name, Object.freeze({ - name: component.name, path: component.path, sourceHash: component.sourceHash, content: component.content, @@ -235,6 +272,38 @@ export function workflowBundleInstallation( ); return { admissions: [admits(index)], - bundle: { components: Object.freeze([...index.values()]) }, + bundle: { components: Object.freeze(bundled) }, }; } + +/** + * Hold a completed run's retained component imports to the bundle its + * definition declares, and grant no authority to import one. + * + * The other half of `workflowBundleInstallation()`, for the execution that + * reuses a terminal instead of running. There is no execution view here because + * there is nothing to resolve: a completed replay answers from its recorded + * root Close before any name is looked up, so a source read for it would be a + * fetch performed for a component nobody imports. What remains is the + * admission, and it is exactly as strict — every retained import is held to the + * declared name, canonical path and object id, and a member the history never + * imported is neither read nor granted anything by being declared. + * + * ```ts + * yield* executeInstalled(options, [ + * retainedWorkflowInstallation(run), + * workflowBundleReplayInstallation(definitionComponents(definition)), + * ]); + * ``` + */ +export function workflowBundleReplayInstallation( + declared: readonly WorkflowComponentEntry[], +): ExecutionInstallation { + const index = new Map( + declared.map((entry) => [ + entry.name, + Object.freeze({ path: entry.path, sourceHash: entry.sourceHash }), + ]), + ); + return { admissions: [admits(index)] }; +} diff --git a/packages/workflow/src/replay.ts b/packages/workflow/src/replay.ts new file mode 100644 index 000000000..014b73866 --- /dev/null +++ b/packages/workflow/src/replay.ts @@ -0,0 +1,306 @@ +/** + * What a completed run replays on, taken from the history its owner already + * holds. + * + * A completed run asked to run again does not run: canonical execution reads + * the terminal its journal recorded and answers with it, importing nothing, + * performing nothing and appending nothing. It still has to be *given* a root + * document, because that value is what the retained history is held to — the + * document a recorded import must have selected, and the one a terminal written + * before any import is bound to. + * + * Locally that value came out of Git, because a checkout was there. A run whose + * durable owner is somewhere else has no checkout at all, and fetching one for + * a replay that imports nothing would be live retrieval performed for a document + * nobody is going to read. So it comes from the run: the retained root import + * holds the exact selection canonical execution made — the document's path, its + * text and the target it resolved to — and a run that failed before importing + * anything holds the same three in the binding core writes into its terminal. + * + * Either way the path is held to the definition the run record retains, and + * that is the load-bearing half. Journal data decides nothing about which + * document a run is a run of; it supplies only the bytes the run already + * recorded for the document its immutable definition names. + * + * The bundle is the same judgment made narrower. A completed replay imports no + * component, so it is handed no component source and no authority to resolve + * one. What it is handed is the admission that holds every component import the + * history recorded to the exact name, canonical path and object id the + * definition declares — so a member the definition declares and this history + * never imported is neither read nor fetched, and grants nothing by existing. + * + * Nothing here reads or writes anything. It is a decision over values, like + * `lifecycle/policy.ts`, so both hosts reach it the same way. + */ + +import { Err, Ok, type Result } from "effection"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { retainedSource } from "@executablemd/core/host"; +import type { ExecutionInstallation, RetainedRootDocument } from "@executablemd/core/host"; +import { workflowBundleReplayInstallation } from "./bundle.ts"; +import { retainedWorkflowInstallation } from "./run.ts"; +import { terminal } from "./lifecycle/policy.ts"; +import type { JournalEntry } from "./storage/api.ts"; +import { definitionComponents } from "./storage/definition.ts"; +import type { WorkflowRunRecord, WorkflowRunStatus } from "./storage/record.ts"; + +/** The root import a run's own entry is recorded under. */ +const ROOT = "__root__"; + +const IMPORT_COMPONENT = "import_component"; + +/** Where core writes the document a terminal it created before importing was about. */ +const ROOT_BINDING = "root_binding"; + +/** What a completed run hands canonical execution, and nothing else. */ +export interface RetainedReplay { + /** The root document, exactly as this run's own history recorded it. */ + readonly root: RetainedRootDocument; + /** The run contract and the bundle admission this replay is held to. */ + readonly installations: readonly ExecutionInstallation[]; +} + +/** + * Retained state that describes no completed run. + * + * Fixed diagnostics throughout, and deliberately so: what is being refused is + * journal and lifecycle data, and a refusal that quoted a path, a source or a + * recorded value would publish exactly what it exists to reject. + */ +export class WorkflowReplayHistoryError extends Error { + override name = "WorkflowReplayHistoryError"; +} + +const REFUSALS = Object.freeze({ + live: "this run's retained state is not terminal, so it replays nothing.", + absent: + "this run is retained as ended and its history records no document result, so there is " + + "nothing for a replay to restore. The run is left exactly as it is.", + mixed: + "this run's history continues past the document result it records, so the two describe " + + "different moments of the run. The run is left exactly as it is.", + ambiguous: + "this run's history records more than one root document import, so no single one " + + "describes what it ran. The run is left exactly as it is.", + malformed: + "this run's retained root document cannot be read by this version. The run is left " + + "exactly as it is.", + document: + "this run's retained root document is not the document its definition names. The run is " + + "left exactly as it is.", +}); + +function refuse(reason: string): Result { + return Err(new WorkflowReplayHistoryError(reason)); +} + +/** + * Read one journal-controlled value, or answer that reading it refused. + * + * As narrow as the reads a retained import is parsed through, and for the same + * reason: a throwing accessor and a proxy trap both mean this record does not + * describe a replay, and nothing wider should be swallowed. + */ +function reading(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +/** A recorded value that is an ordinary object, or nothing. */ +function recorded(value: unknown): Record | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + const names = reading(() => Object.keys(value)); + if (names === undefined) { + return undefined; + } + const held: Record = {}; + for (const name of names) { + const member = reading(() => Reflect.get(value, name)); + if (member === undefined) { + return undefined; + } + held[name] = member; + } + return held; +} + +/** Whether a retained event is recognizably a settled component import. */ +function importedName(event: DurableEvent): string | undefined { + if (reading(() => event.type) !== "yield") { + return undefined; + } + const description = reading(() => (event.type === "yield" ? event.description : undefined)); + if (description === undefined || reading(() => description.type) !== IMPORT_COMPONENT) { + return undefined; + } + const name = reading(() => description.name); + return typeof name === "string" ? name : undefined; +} + +/** Whether a retained event is this document execution's own terminal. */ +function isRootClose(event: DurableEvent): boolean { + return reading(() => event.type === "close" && event.coroutineId === "root") === true; +} + +/** The document one retained selection names, as a root source is built from it. */ +interface RetainedRoot { + readonly path: string; + readonly content: string; + readonly target: string | undefined; +} + +/** + * The selection a settled root import recorded. + * + * Only the shapes canonical execution writes for a root: the whole document, + * one exact target, and a selector the document offered no single target for. + * A settlement that failed recorded no selection at all, and is refused by the + * caller rather than guessed at. + */ +function rootSelection(event: DurableEvent): RetainedRoot | undefined { + const settlement = recorded(reading(() => (event.type === "yield" ? event.result : undefined))); + if (settlement === undefined || settlement["status"] !== "ok") { + return undefined; + } + const selection = recorded(settlement["value"]); + if (selection === undefined) { + return undefined; + } + const path = selection["path"]; + const content = selection["content"]; + const kind = selection["kind"]; + const members = Object.keys(selection).length; + if (typeof path !== "string" || typeof content !== "string") { + return undefined; + } + + if (kind === "repository") { + const target = selection["target"]; + if (target === undefined) { + return members === 3 ? { path, content, target: undefined } : undefined; + } + return members === 4 && typeof target === "string" ? { path, content, target } : undefined; + } + + if (kind === "target-failure") { + // The selector as the run was asked for it. Handing it back is what makes + // the replayed request the same request: canonical execution resolves it + // against the recorded document again and finds the same failure. + const failure = recorded(selection["failure"]); + const selector = failure?.["selector"]; + return members === 4 && typeof selector === "string" + ? { path, content, target: selector } + : undefined; + } + + return undefined; +} + +/** + * The document a terminal written before any import was about. + * + * A run can fail before it imports anything, and canonical core records which + * document that failure was about inside the terminal itself. That binding is + * the only account such a history has of its own root, so it is what a replay + * of one is built from. + */ +function boundRoot(close: DurableEvent): RetainedRoot | undefined { + const settlement = recorded(reading(() => close.result)); + if (settlement === undefined || settlement["status"] !== "ok") { + return undefined; + } + const result = recorded(settlement["value"]); + const binding = recorded(result?.[ROOT_BINDING]); + if (binding === undefined || Object.keys(binding).length !== 3) { + return undefined; + } + const path = binding["path"]; + const source = binding["source"]; + const target = binding["target"]; + if (typeof path !== "string" || typeof source !== "string") { + return undefined; + } + if (target !== null && typeof target !== "string") { + return undefined; + } + return { path, content: source, target: target ?? undefined }; +} + +/** + * Whether a run in this state replays what it retained rather than continuing. + * + * The one place a caller needs the question answered before it has a frontier + * to read. A resume that is going to replay must not fetch a definition on the + * way in, and the retained status is what says so while the run is still being + * admitted. It is not the decision — `retainedReplay()` is, from the history + * itself — it is what keeps live retrieval from happening ahead of one. + */ +export function replaysRetainedResult(status: WorkflowRunStatus): boolean { + return terminal(status); +} + +/** + * What this run's owner already holds, as the inputs one canonical replay runs + * on — or why the state it holds describes no completed run. + * + * Every refusal here happens before an attachment, a provider, a materialized + * root or a native operation: the caller has read one coherent frontier and has + * done nothing else with it. + */ +export function retainedReplay( + record: WorkflowRunRecord, + entries: readonly JournalEntry[], +): Result { + if (!terminal(record.status)) { + return refuse(REFUSALS.live); + } + + const closes = entries.filter((entry) => isRootClose(entry.event)); + const close = closes[0]; + if (close === undefined) { + return refuse(REFUSALS.absent); + } + // The terminal is the frontier, and there is one of it. A history holding a + // second result, or continuing past the one it stands behind, is one whose + // lifecycle row and journal describe different moments of the run — and + // reconciling those is not a replay's to do. + const last = entries[entries.length - 1]; + if (closes.length !== 1 || last === undefined || !isRootClose(last.event)) { + return refuse(REFUSALS.mixed); + } + + const imports = entries.filter((entry) => importedName(entry.event) === ROOT); + if (imports.length > 1) { + return refuse(REFUSALS.ambiguous); + } + + const imported = imports[0]; + const retained = imported === undefined ? boundRoot(close.event) : rootSelection(imported.event); + if (retained === undefined) { + return refuse(REFUSALS.malformed); + } + if (retained.path !== record.definition.rootDocumentPath) { + return refuse(REFUSALS.document); + } + + const { path, content, target } = retained; + return Ok({ + root: + target === undefined + ? retainedSource(path, content) + : retainedSource(path, content, { target }), + installations: [ + retainedWorkflowInstallation({ + runId: record.runId, + base: record.base, + pinnedCommit: record.definition.objectId, + }), + workflowBundleReplayInstallation(definitionComponents(record.definition)), + ], + }); +} diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts new file mode 100644 index 000000000..bd8e2db5a --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -0,0 +1,382 @@ +/** + * Tier WRH12 — a completed run replayed through its real durable owner. + * + * The rule this file exists for has two halves that pull in opposite + * directions. A completed replay must reach *no* external-effect provider; and + * it must reach the run's durable owner, because that is where the retained + * result is and an ephemeral runner holds no copy of it. Reading its own + * history is not attaching a provider, and proving that distinction needs the + * real thing: a real Durable Object, its own SQLite storage, a real accepted + * Hibernation WebSocket, and the production executor connection over it. + * + * What runs on this side is the production decision — `retainedReplay()` over + * the frontier the owner answered with — handed to canonical + * `executeInstalled()`. The shared CLI assembles exactly these two around the + * same values; it is not itself importable here, because it resolves a terminal + * renderer a Worker has no use for. That orchestration is proved in + * `packages/cli/tests/workflow-replay.test.ts` against the local host. + * + * Nothing in this file imports `@executablemd/workflow/deno`. It could not: the + * adapter behind that specifier reaches `node:sqlite`, which workerd does not + * have. A completed replay that runs here is a completed replay that needed + * none of it. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { run, until, type Operation, type Result, Ok, scoped } from "effection"; +import { executeInstalled, retainedSource } from "@executablemd/core/host"; +import type { ExecutionInstallation, RetainedRootDocument } from "@executablemd/core/host"; +import type { Json } from "@executablemd/durable-streams"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; +import { type OwnerSocket, type SocketListener } from "../../src/remote/client.ts"; +import { useExecutorConnection } from "../../src/cloudflare/executor-connection.ts"; +import { useRemoteLifecycle } from "../../src/remote/lifecycle.ts"; +import type { RemoteLifecycleHost } from "../../src/remote/lifecycle.ts"; +import type { RemoteExecutorConnection } from "../../src/remote/lifecycle-link.ts"; +import type { RemoteReadPlane } from "../../src/remote/read.ts"; +import type { WorkflowRunDatabase } from "../../src/storage/api.ts"; +import { WorkflowLifecycle } from "../../src/lifecycle/api.ts"; +import type { + WorkflowExecutionTransitions, + WorkflowRunCreation, +} from "../../src/lifecycle/execution.ts"; +import { retainedReplay } from "../../src/replay.ts"; +import { retainedWorkflowInstallation } from "../../src/run.ts"; + +let unique = 0; +const NOW = 1_800_000_000; +const COMMIT = "0".repeat(40); +const DOCUMENT = "# Remote\n\nthe owner recorded this line.\n"; +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +function executor() { + unique += 1; + return env.EXECUTOR.get(env.EXECUTOR.idFromName(`replay-${unique}-${Math.random()}`)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise { + return runInDurableObject(stub, body); +} + +/** One real accepted executor WebSocket, admitted the way a runner's is. */ +async function connect(stub: ReturnType): Promise { + await on(stub, (owner) => owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW)); + const token = await signToken(keys, { + ...VALID_CLAIMS, + iat: NOW - 10, + nbf: NOW - 10, + exp: NOW + 600, + }); + const response = await stub.fetch("https://owner.invalid/executor", { + headers: { + authorization: `Bearer ${token}`, + upgrade: "websocket", + "x-release": POLICY.release, + "x-run-id": RUN_ID, + }, + }); + const socket = response.webSocket; + if (socket === null) { + throw new Error(`expected an executor WebSocket, received ${response.status}`); + } + socket.accept(); + return socket; +} + +/** The platform socket, bound to the four members the client uses. */ +function ownerSocket(socket: WebSocket): OwnerSocket { + const listeners = new Map(); + return { + send: (data) => socket.send(data), + close: () => socket.close(), + addEventListener(type, listener) { + const forward: EventListener = (event) => { + const data: unknown = Reflect.get(event, "data"); + listener(typeof data === "string" ? { data } : {}); + }; + listeners.set(listener, forward); + socket.addEventListener(type, forward); + }, + removeEventListener(type, listener) { + const found = listeners.get(listener); + if (found !== undefined) { + socket.removeEventListener(type, found); + } + }, + }; +} + +/** + * The provider's host, reaching this exact owner. + * + * Two of its four members refuse. The read plane and fork staging are the + * planes a replay has no business on, so entering either is a planted failure + * rather than something an assertion has to notice afterwards. + */ +function lifecycleHost(stub: ReturnType): RemoteLifecycleHost { + let executions = 0; + let commands = 0; + const next = () => `command-${(commands += 1)}`; + return { + *admit(runId: string): Operation> { + return yield* useExecutorConnection( + { + *open(): Operation> { + return Ok(ownerSocket(yield* until(connect(stub)))); + }, + ids: () => next, + }, + runId, + ); + }, + // deno-lint-ignore require-yield + *source(): Operation> { + throw new Error("PLANTED-READ-PLANE-REACHED"); + }, + // deno-lint-ignore require-yield + *stage(): Operation> { + throw new Error("PLANTED-STAGING-REACHED"); + }, + ids: { execution: () => `execution-${(executions += 1)}`, command: next }, + }; +} + +const CREATION: WorkflowRunCreation = { + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: COMMIT, + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, +}; + +/** What one document execution rendered, and how it ended. */ +interface Rendered { + readonly output: string; + readonly result: Result; +} + +/** Take the acquisition this provider issues, or say why there is none. */ +function* acquired(): Operation<{ runId: string }> { + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + if (!taken.ok) { + throw taken.error; + } + if (taken.value.kind !== "acquired") { + throw new Error("expected this connection to be the run's executor"); + } + return taken.value.lock; +} + +/** One canonical execution over the owner-backed journal, rendered in full. */ +function* canonical( + database: WorkflowRunDatabase, + root: RetainedRootDocument, + installations: readonly ExecutionInstallation[], +): Operation { + const running = yield* executeInstalled( + { ...root, stream: database.journal, componentDirs: [] }, + installations, + ); + const subscription = yield* running.output; + let next = yield* subscription.next(); + while (!next.done) { + next = yield* subscription.next(); + } + return { output: next.value, result: yield* running }; +} + +/** Everything about this owner a replay must leave alone. */ +function ownerState(stub: ReturnType) { + return on(stub, (owner) => ({ + run: owner.runRow(), + executions: owner.executionRows().length, + currentRootId: owner.currentRootId(), + journal: owner.journalRecords(), + published: owner.published(), + answers: owner.retainedAnswers(), + })); +} + +describe("a completed run replayed through its own owner", () => { + it("restores the retained result, and moves nothing but its own envelope", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + // The run, made and completed the way a runner makes one: one admitted + // connection, one begin, canonical execution over the owner's journal, one + // settlement. + const live = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + const rendered = yield* canonical( + begun.value.database, + retainedSource("README.md", DOCUMENT), + [ + retainedWorkflowInstallation({ + runId: RUN_ID, + base: CREATION.base, + pinnedCommit: COMMIT, + }), + ], + ); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + if (!settled.ok) { + throw settled.error; + } + return rendered; + }); + }); + + expect(live.result.ok).toBe(true); + const before = await ownerState(stub); + expect(before.run?.["status"]).toBe("completed"); + expect(before.journal.length).toBeGreaterThan(0); + // The acquisition ended with its scope, so nothing holds the run. + expect(await on(stub, (owner) => owner.holders())).toBe(0); + + // A second connection, a resume, and the replay: everything it is held to + // comes from the frontier this owner answers with. + const replayed = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + if (!begun.ok) { + throw begun.error; + } + // The run kept the outcome that already won. + expect(begun.value.replay).toBe(true); + const database = begun.value.database; + const frontier = yield* database.readJournalEntries(); + if (!frontier.ok) { + throw frontier.error; + } + const prepared = retainedReplay(begun.value.record, frontier.value); + if (!prepared.ok) { + throw prepared.error; + } + // The document canonical execution is handed comes out of the owner's + // own history, reported by the path the run record names. + expect(prepared.value.root).toEqual({ + path: "README.md", + source: DOCUMENT, + retained: true, + }); + const rendered = yield* canonical(database, prepared.value.root, [ + ...prepared.value.installations, + ]); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + if (!settled.ok) { + throw settled.error; + } + return rendered; + }); + }); + + // Byte for byte, and the same result — reconstructed from the owner's own + // history, with no definition, checkout, Workspace or provider anywhere. + expect(replayed.output).toBe(live.output); + expect(replayed.result.ok).toBe(true); + expect(replayed.result.ok === true && replayed.result.value).toEqual( + live.result.ok === true ? live.result.value : undefined, + ); + + const after = await ownerState(stub); + // Every retained row, under the same identity, in the same order. + expect(after.journal).toEqual(before.journal); + expect(after.currentRootId).toBe(before.currentRootId); + expect(after.published).toEqual(before.published); + expect(after.answers).toEqual(before.answers); + expect(after.run?.["status"]).toBe("completed"); + // The one durable change: the lifecycle envelope this invocation recorded. + expect(after.executions).toBe(before.executions + 1); + // And it is closed, so the run is not left looking live. + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + + it("hands the run to the next connection when the replay's own ends", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + await run(function* () { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + yield* canonical(begun.value.database, retainedSource("README.md", DOCUMENT), [ + retainedWorkflowInstallation({ + runId: RUN_ID, + base: CREATION.base, + pinnedCommit: COMMIT, + }), + ]); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + if (!settled.ok) { + throw settled.error; + } + }); + }); + + const outcome = await run(function* () { + const first = yield* scoped(function* () { + yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + return lock.runId; + }); + // The first acquisition is over. A replacement takes the run, which is + // what "the connection is the acquisition" means when the replay ends. + const second = yield* scoped(function* () { + yield* useRemoteLifecycle(host); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + return taken.ok ? taken.value.kind : "refused"; + }); + return { first, second }; + }); + + expect(outcome.first).toBe(RUN_ID); + expect(outcome.second).toBe("acquired"); + // Nothing the first connection did outlives it, and the run is still the + // completed run both acquisitions found. + expect((await on(stub, (owner) => owner.runRow()))?.["status"]).toBe("completed"); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); +}); diff --git a/packages/workflow/tests/replay-inputs.test.ts b/packages/workflow/tests/replay-inputs.test.ts new file mode 100644 index 000000000..a041728a0 --- /dev/null +++ b/packages/workflow/tests/replay-inputs.test.ts @@ -0,0 +1,406 @@ +/** + * Tier WRH12 — what a completed run replays on, and where it comes from. + * + * A completed replay imports nothing and performs nothing, so everything it is + * held to has to come from what the run already retains. Two halves of that are + * settled here, over values, because both hosts reach the same function: which + * root document the recorded result was a result of, and which component + * imports the history is allowed to contain. + * + * The load-bearing word throughout is *definition*. Journal data supplies the + * bytes; the immutable run record supplies the identity those bytes have to + * agree with. A history that names another document, another path or another + * object id is refused before anything is replayed from it — and every refusal + * says so without repeating a path, a source or a recorded value. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Operation, Result } from "effection"; +import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import type { ExecutionInstallation } from "@executablemd/core/host"; +import { replaysRetainedResult, retainedReplay } from "../src/replay.ts"; +import type { RetainedReplay } from "../src/replay.ts"; +import { WorkflowReplayHistoryError } from "../src/replay.ts"; +import { WorkflowBundleHistoryError } from "../src/bundle.ts"; +import { forkRunRecordEvent } from "../src/journal-events.ts"; +import type { JournalEntry } from "../src/storage/api.ts"; +import type { WorkflowComponentEntry } from "../src/storage/definition.ts"; +import type { WorkflowRunRecord, WorkflowRunStatus } from "../src/storage/record.ts"; + +const ROOT_ID = "a".repeat(64); +const COMMIT = "0".repeat(40); +const SOURCE = "# Retained\n\ndone.\n"; + +function record( + overrides: { + readonly status?: WorkflowRunStatus; + readonly rootDocumentPath?: string; + readonly components?: readonly WorkflowComponentEntry[]; + } = {}, +): WorkflowRunRecord { + const components = overrides.components; + return { + runId: "replay-1", + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: COMMIT, + rootDocumentPath: overrides.rootDocumentPath ?? "flows/root.md", + ...(components === undefined ? {} : { components }), + }, + base: "main", + props: {}, + status: overrides.status ?? "completed", + createdAt: "2026-01-01T00:00:00.000Z", + updatedAt: "2026-01-01T00:00:00.000Z", + }; +} + +let identity = 0; + +function entry(event: DurableEvent): JournalEntry { + identity += 1; + return { eventId: `event-${identity}`, event, workspaceRootId: ROOT_ID }; +} + +/** The root import canonical execution records, as this run recorded it. */ +function rootImport(value: Json): DurableEvent { + return { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value }, + }; +} + +/** One bundled component import, as canonical execution records one. */ +function componentImport(name: string, value: Json): DurableEvent { + return { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name }, + result: { status: "ok", value }, + }; +} + +function rootClose(value: Json): DurableEvent { + return { type: "close", coroutineId: "root", result: { status: "ok", value } }; +} + +/** The terminal core writes when it fails before importing a root document. */ +function preRootClose(path: string, source: string, target: string | null): DurableEvent { + const message = "refused before the root import"; + return rootClose({ + status: "err", + output: "", + error: { name: "Error", message, segment: { message } }, + root_binding: { path, source, target }, + }); +} + +/** An ordinary completed history: the import, then the result. */ +function completedHistory( + path = "flows/root.md", + content = SOURCE, + extra: readonly DurableEvent[] = [], +): JournalEntry[] { + return [ + entry(rootImport({ kind: "repository", path, content })), + ...extra.map((event) => entry(event)), + entry(rootClose({ status: "ok", output: "done.\n", value: "done.\n" })), + ]; +} + +function reason(outcome: Result): string { + if (outcome.ok) { + throw new Error("expected the retained state to be refused"); + } + expect(outcome.error).toEqual(expect.any(WorkflowReplayHistoryError)); + return outcome.error.message; +} + +function admitted(outcome: Result): RetainedReplay { + if (!outcome.ok) { + throw outcome.error; + } + return outcome.value; +} + +/** The run record canonical execution retains before it imports anything. */ +const RUN_RECORD = forkRunRecordEvent({ runId: "replay-1", base: "main", pinnedCommit: COMMIT }); + +/** Run every admission one installation carries over one retained history. */ +function* admit( + installations: readonly ExecutionInstallation[], + retained: readonly DurableEvent[], +): Operation { + for (const installation of installations) { + for (const admission of installation.admissions ?? []) { + try { + yield* admission(retained); + } catch (error) { + return error instanceof Error ? error : new Error(String(error)); + } + } + } + return undefined; +} + +describe("the root a completed run replays on", () => { + // deno-lint-ignore require-yield + it("is the document the retained import selected", function* () { + const replay = admitted(retainedReplay(record(), completedHistory())); + + expect(replay.root).toEqual({ path: "flows/root.md", source: SOURCE, retained: true }); + }); + + // deno-lint-ignore require-yield + it("carries the exact target the import resolved to", function* () { + const history = [ + entry( + rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE, target: "Stage" }), + ), + entry(rootClose({ status: "ok", output: "", value: "" })), + ]; + + expect(admitted(retainedReplay(record(), history)).root).toEqual({ + path: "flows/root.md", + source: SOURCE, + retained: true, + target: "Stage", + }); + }); + + // deno-lint-ignore require-yield + it("carries the selector a recorded failed selection was asked for", function* () { + const history = [ + entry( + rootImport({ + kind: "target-failure", + path: "flows/root.md", + content: SOURCE, + failure: { kind: "no-match", selector: "Missing*", matches: [], available: ["Retained"] }, + }), + ), + entry(rootClose({ status: "err", output: "", error: { name: "E", message: "m" } })), + ]; + + expect(admitted(retainedReplay(record(), history)).root).toEqual({ + path: "flows/root.md", + source: SOURCE, + retained: true, + target: "Missing*", + }); + }); + + // deno-lint-ignore require-yield + it("comes from the terminal when the run failed before importing anything", function* () { + const history = [entry(preRootClose("flows/root.md", SOURCE, null))]; + + expect(admitted(retainedReplay(record(), history)).root).toEqual({ + path: "flows/root.md", + source: SOURCE, + retained: true, + }); + }); + + // deno-lint-ignore require-yield + it("is refused when it is not the document the definition names", function* () { + const shifted = completedHistory("flows/other.md"); + expect(reason(retainedReplay(record(), shifted))).toContain( + "not the document its definition names", + ); + + const bound = [entry(preRootClose("flows/other.md", SOURCE, null))]; + expect(reason(retainedReplay(record(), bound))).toContain( + "not the document its definition names", + ); + }); +}); + +describe("retained state that describes no completed run", () => { + // deno-lint-ignore require-yield + it("refuses a run that has not ended", function* () { + const live: readonly WorkflowRunStatus[] = ["running", "suspended", "interrupted", "cancelled"]; + for (const status of live) { + const outcome = retainedReplay(record({ status }), completedHistory()); + expect([status, reason(outcome).includes("not terminal")]).toEqual([status, true]); + expect([status, replaysRetainedResult(status)]).toEqual([status, false]); + } + const ended: readonly WorkflowRunStatus[] = ["completed", "failed"]; + for (const status of ended) { + expect([status, replaysRetainedResult(status)]).toEqual([status, true]); + } + }); + + // deno-lint-ignore require-yield + it("refuses a lifecycle row that claims completion with no recorded result", function* () { + const history = [ + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + ]; + + expect(reason(retainedReplay(record(), history))).toContain("records no document result"); + }); + + // deno-lint-ignore require-yield + it("refuses a history that continues past the result it records", function* () { + const after = entry({ + type: "yield", + coroutineId: "root", + description: { type: "workspace", name: "write" }, + result: { status: "ok", value: "later" }, + }); + + expect(reason(retainedReplay(record(), [...completedHistory(), after]))).toContain( + "continues past the document result", + ); + expect( + reason( + retainedReplay(record(), [ + ...completedHistory(), + entry(rootClose({ status: "ok", output: "", value: "" })), + ]), + ), + ).toContain("continues past the document result"); + }); + + // deno-lint-ignore require-yield + it("refuses a history recording more than one root import", function* () { + const history = [ + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + entry(rootClose({ status: "ok", output: "", value: "" })), + ]; + + expect(reason(retainedReplay(record(), history))).toContain("more than one root document"); + }); + + // deno-lint-ignore require-yield + it("refuses a root import this version cannot read", function* () { + const unreadable: Json[] = [ + { kind: "repository", path: "flows/root.md" }, + { kind: "repository", path: "flows/root.md", content: SOURCE, extra: 1 }, + { kind: "repository", path: "flows/root.md", content: SOURCE, target: 7 }, + { kind: "registered", origin: "somewhere", reserved: false }, + { kind: "workflow", path: "flows/root.md", sourceHash: "abc", content: SOURCE }, + "not an object at all", + ]; + + for (const value of unreadable) { + const history = [ + entry(rootImport(value)), + entry(rootClose({ status: "ok", output: "", value: "" })), + ]; + const said = reason(retainedReplay(record(), history)); + expect([JSON.stringify(value), said.includes("cannot be read by this version")]).toEqual([ + JSON.stringify(value), + true, + ]); + } + }); + + // deno-lint-ignore require-yield + it("refuses a root import that failed, and a terminal carrying no binding", function* () { + const failed = [ + entry({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "err", error: { name: "Error", message: "gone" } }, + }), + entry(rootClose({ status: "ok", output: "", value: "" })), + ]; + expect(reason(retainedReplay(record(), failed))).toContain("cannot be read by this version"); + + const unbound = [entry(rootClose({ status: "ok", output: "", value: "" }))]; + expect(reason(retainedReplay(record(), unbound))).toContain("cannot be read by this version"); + }); + + // deno-lint-ignore require-yield + it("says what refused without repeating anything the history held", function* () { + const planted = "PLANTED-SECRET-VALUE"; + const history = [ + entry(rootImport({ kind: "repository", path: `flows/${planted}.md`, content: planted })), + entry(rootClose({ status: "ok", output: planted, value: planted })), + ]; + + const said = reason(retainedReplay(record(), history)); + expect(said).not.toContain(planted); + expect(said.length).toBeLessThan(300); + }); +}); + +describe("the bundle a completed replay is held to", () => { + const declared: readonly WorkflowComponentEntry[] = [ + { name: "Stage", path: "flows/Stage.md", sourceHash: "1".repeat(40) }, + { name: "Unused", path: "flows/Unused.md", sourceHash: "2".repeat(40) }, + ]; + + it("grants no authority to import anything", function* () { + const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); + + // Two installations, and neither offers an execution view: a completed + // replay resolves no name, so there is nothing for one to resolve against. + expect(replay.installations).toHaveLength(2); + for (const installation of replay.installations) { + expect(installation.bundle).toBe(undefined); + expect(installation.components).toBe(undefined); + } + }); + + it("admits an import the definition declares, by path and object id", function* () { + const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); + const held = componentImport("Stage", { + kind: "workflow", + path: "flows/Stage.md", + sourceHash: "1".repeat(40), + content: "staged.\n", + }); + + expect(yield* admit(replay.installations, [RUN_RECORD, held])).toBe(undefined); + }); + + it("refuses one recorded under another path, hash or name", function* () { + const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); + const wrong: DurableEvent[] = [ + componentImport("Stage", { + kind: "workflow", + path: "flows/Elsewhere.md", + sourceHash: "1".repeat(40), + content: "staged.\n", + }), + componentImport("Stage", { + kind: "workflow", + path: "flows/Stage.md", + sourceHash: "9".repeat(40), + content: "staged.\n", + }), + componentImport("Undeclared", { + kind: "workflow", + path: "flows/Stage.md", + sourceHash: "1".repeat(40), + content: "staged.\n", + }), + componentImport("Stage", { + kind: "repository", + path: "flows/Stage.md", + content: "staged.\n", + }), + ]; + + for (const event of wrong) { + const refused = yield* admit(replay.installations, [RUN_RECORD, event]); + expect(refused).toEqual(expect.any(WorkflowBundleHistoryError)); + } + }); + + it("leaves a declared member the history never imported alone", function* () { + const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); + + // `Unused` is declared and was never imported. Nothing reads it, nothing + // fetches it, and its absence from the history is not a refusal. + expect(yield* admit(replay.installations, [RUN_RECORD])).toBe(undefined); + }); +}); diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index 533a5fed5..798483202 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -485,6 +485,12 @@ const DENO_ONLY_TOOLING: RuntimeExclusion[] = [ "opens a real node:sqlite run store through @executablemd/workflow/deno to drive runWorkflow() directly; Bun has no node:sqlite at all and Node 22 keeps it behind --experimental-sqlite", issue: "https://github.com/taras/executable.md/issues/366", }, + { + path: "packages/cli/tests/workflow-replay.test.ts", + reason: + "replays completed runs out of a real node:sqlite run store opened through @executablemd/workflow/deno, and reads the answers table it settles; Bun has no node:sqlite at all and Node 22 keeps it behind --experimental-sqlite. The host-neutral half — what a completed run replays on, and what its retained history is held to — runs on every runtime as packages/workflow/tests/replay-inputs.test.ts", + issue: "https://github.com/taras/executable.md/issues/366", + }, ]; /** From 15e9f506e1b1c3ca75b2db207ac882a213c61a46 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 9 Sep 2026 10:13:40 -0300 Subject: [PATCH 63/81] =?UTF-8?q?=F0=9F=90=9B=20Let=20recovery=20decide=20?= =?UTF-8?q?the=20replay,=20and=20hold=20a=20retained=20history=20to=20what?= =?UTF-8?q?=20it=20says?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three corrections to completed replay, each about believing a record instead of establishing what it claims. **Recovery decides, not the status a run still carries.** An executor may commit its document result and disappear before settling, and the run then reads `running` while its journal already holds the outcome. The definition was fetched ahead of the lifecycle transaction that recognizes that, so a bundled run in the supported crash window went to a checkout an ephemeral runner may not have — on the one path required to reach only its durable owner. The definition is now fetched after admission or not at all: the begin transaction reconciles the stale execution, publishes the terminal its retained root result implies, and answers `replay`, and every live-only construction below it is downstream of that answer. A live or partial run still reconstructs and authenticates its whole bundle before any document effect, and a reconstruction it cannot perform leaves the journal and the Workspace root exactly as they were, with the envelope it began closed by the settled recovery at the next acquisition. **The lifecycle row and the recorded result have to be the same outcome.** A row claiming `completed` over a root `Close` that records a failure was accepted, so canonical core could reuse a terminal the run's own state contradicts. `retainedReplay()` now requires `rootOutcome()` — the settled mapping recovery and settlement already publish through — to agree with the retained status and stop reason, and refuses the pairings the lifecycle cannot produce before terminal reuse. **A retained component source is held to the object id, not merely beside it.** The replay admission required the history to repeat the declared hash, which establishes nothing about the bytes recorded next to it: altered source was admitted under the original identity. It now names those bytes the way Git names a blob — the kind, the encoded byte length, a NUL, the content — under the definition's own object format, and requires that name to be the object id the definition declares. --- packages/cli/src/workflow.ts | 89 ++---- packages/cli/tests/workflow-replay.test.ts | 283 +++++++++++++++++- packages/workflow/mod.ts | 2 +- packages/workflow/src/bundle.ts | 64 ++-- packages/workflow/src/git-blob.ts | 149 +++++++++ packages/workflow/src/replay.ts | 60 ++-- .../tests/cloudflare/remote-replay.vitest.ts | 197 ++++++++++++ packages/workflow/tests/git-blob.test.ts | 76 +++++ packages/workflow/tests/replay-inputs.test.ts | 237 +++++++++++++-- 9 files changed, 1003 insertions(+), 154 deletions(-) create mode 100644 packages/workflow/src/git-blob.ts create mode 100644 packages/workflow/tests/git-blob.test.ts diff --git a/packages/cli/src/workflow.ts b/packages/cli/src/workflow.ts index 36d5172a6..56c431906 100644 --- a/packages/cli/src/workflow.ts +++ b/packages/cli/src/workflow.ts @@ -73,10 +73,8 @@ import { retainedSource, validateProps } from "@executablemd/core"; import type { PropsSchema } from "@executablemd/core"; import type { RootDocumentSource } from "@executablemd/core"; import { - definitionComponents, retainedReplay, retainedWorkflowInstallation, - replaysRetainedResult, workflowBundleInstallation, WORKFLOW_RUN_STATUSES, WorkflowLifecycle, @@ -960,17 +958,6 @@ export function runWorkflow( } const { lock: executorLock } = acquired.value; - // A resumed run closed over a component bundle reconstructs it here: under - // the executor lock, from the retained commit, and before the execution - // record exists. A component that is gone, changed, or unreachable leaves - // the run's lifecycle records exactly as they are rather than adding an - // attempt that never began. - const reconstructed = yield* reconstructedSources(request, runId); - if (!reconstructed.ok) { - report(reconstructed.error.message); - return { exitCode: 1 }; - } - // One transaction: whatever the previous workflow executor left is reconciled, this // action is admitted against what that left behind, and the execution is // recorded — or none of it is. A fork's one transaction is its whole @@ -1004,15 +991,23 @@ export function runWorkflow( const completed = frontier.value.some((entry) => isRootClose(entry.event)); // A resume of a run that already ended replays what its own history holds. - // Decided here, from the admitted frontier, and therefore ahead of every - // live-only construction below it: the retained definition, this host's own - // adapter, the suspension controller, the answer provider, the `` - // declaration and `host.attach()`. Each of those is work for an execution - // that is going to import nothing, perform nothing and append nothing. + // The lifecycle decided that, in the transaction above: it reconciled + // whatever the previous executor left, published the canonical terminal a + // retained root result implies, and answered `replay`. Nothing before this + // point may decide it — a run whose executor committed its document result + // and disappeared before settling still reads `running`, and treating that + // status as live is what sent a completed replay to a checkout it may not + // have. + // + // So the definition is fetched here or not at all, and everything below is + // downstream of the same answer: this host's own adapter, the suspension + // controller, the answer provider, the `` declaration and + // `host.attach()` are each work for an execution that is going to import + // nothing, perform nothing and append nothing. const replayed = replay && start === undefined; const prepared = replayed ? retainedReplay(record, frontier.value) - : yield* liveDocument(record, start, database, reconstructed.value); + : yield* liveDocument(record, start, database); if (!prepared.ok) { report(prepared.error.message); return { exitCode: 1 }; @@ -1366,9 +1361,8 @@ function* liveDocument( record: WorkflowRunRecord, start: WorkflowStart | undefined, database: WorkflowRunDatabase, - reconstructed: RetainedSources | undefined, ): Operation> { - const sources = yield* documentSource(start, database, reconstructed); + const sources = yield* documentSource(start, database); if (!sources.ok) { return sources; } @@ -1392,14 +1386,14 @@ function* liveDocument( function* documentSource( start: WorkflowStart | undefined, database: WorkflowRunDatabase, - reconstructed: RetainedSources | undefined, ): Operation> { if (start !== undefined) { return Ok({ source: start.established.source, components: start.established.components }); } - if (reconstructed !== undefined) { - return Ok(reconstructed); - } + // The root and every component this run is closed over, from the retained + // commit, in one read: `loadRetainedDefinition()` reconstructs the bundle + // when the definition names one. The admitted run record is what says which + // commit that is, so this asks the run rather than an earlier snapshot of it. return yield* loadRetainedDefinition(database.record.definition, database.retrieval?.metadata); } @@ -1433,51 +1427,6 @@ function* liveSupport(database: WorkflowRunDatabase): Operation { }; } -/** - * The pinned sources a resumed run closed over a bundle needs before it begins. - * - * Answers with nothing for a `start`, which established its own bundle from Git - * before it asked storage for anything, and for a run whose definition names no - * components — that one keeps loading its root after the run has been admitted, - * because a run that ended is not one to fetch a definition for. - * - * A run this host cannot inspect answers with nothing too. What that run is, - * and whether this action may advance it, is the begin transition's to decide, - * and answering it here would report a different refusal for the same fact. - * - * So does a run whose retained state already ended, and for a reason of its - * own: that run replays what its history holds and imports nothing, so a bundle - * fetched for it would be live retrieval performed for components nobody - * resolves. - */ -function* reconstructedSources( - request: WorkflowRequest, - runId: string, -): Operation> { - if (request.action !== "resume") { - return Ok(undefined); - } - const snapshot = yield* WorkflowLifecycle.operations.inspect(runId); - if (!snapshot.ok) { - return Ok(undefined); - } - // A run whose retained state already ended replays from its own history, and - // the sources that replay is held to come out of that history rather than out - // of a checkout. Reconstructing a bundle from Git for it would be live - // retrieval performed for components nothing is going to import. Whether the - // history actually describes a completed run is decided after admission, from - // the frontier the owner answers with, and a run that claims to have ended - // without one refuses there rather than falling back to this. - if (replaysRetainedResult(snapshot.value.record.status)) { - return Ok(undefined); - } - const { definition } = snapshot.value.record; - if (definitionComponents(definition).length === 0) { - return Ok(undefined); - } - return yield* loadRetainedDefinition(definition, snapshot.value.retrieval?.metadata); -} - /** * How far this invocation has durably got. * diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts index ea4e4abba..b962fb425 100644 --- a/packages/cli/tests/workflow-replay.test.ts +++ b/packages/cli/tests/workflow-replay.test.ts @@ -31,6 +31,7 @@ import { join } from "node:path"; import { tmpdir } from "node:os"; import { DatabaseSync } from "node:sqlite"; import type { Json } from "@executablemd/core"; +import type { DurableEvent } from "@executablemd/durable-streams"; import { executeInstalled } from "@executablemd/core/host"; import { useWorkflowInputDelivery, @@ -40,8 +41,9 @@ import { workflowRunPath, } from "@executablemd/workflow/deno"; import type { WorkflowExecutionTransitions } from "@executablemd/workflow"; -import { Git, WorkflowLifecycle } from "@executablemd/workflow"; +import { forkRunRecordEvent, Git, WorkflowLifecycle } from "@executablemd/workflow"; import type { + WorkflowDefinition, WorkflowHistoryEntry, WorkflowRunDatabase, WorkflowRunStatus, @@ -340,6 +342,8 @@ function statusOf(invocation: Invocation): string | undefined { interface Retained { readonly status: WorkflowRunStatus; readonly executions: number; + /** What each execution ended as, in order. `null` is one still open. */ + readonly ended: (WorkflowRunStatus | null)[]; readonly currentWorkspaceRootId: string; readonly journal: string; } @@ -358,6 +362,7 @@ function* retained(runs: string, runId: string): Operation { return { status: snapshot.value.record.status, executions: snapshot.value.executions.length, + ended: snapshot.value.executions.map((execution) => execution.stopStatus ?? null), currentWorkspaceRootId: snapshot.value.currentWorkspaceRootId, // Identity and content of every retained row, in order: a length would // not notice one rewritten under a new id. @@ -622,9 +627,196 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); }); - it("WRP5: a run that has not ended still reconstructs and attaches", function* () { + it("WRP8: recovers a bundled run whose result committed and whose settlement did not", function* () { const asked: string[] = []; const attached: string[] = []; + const live: Rendered[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + + // The executor committed the document's result and then went without + // settling. This is the supported crash window, not damaged input: the + // run reads `running`, and its journal already holds the outcome. + const crashed = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + refusingSettlement(fixture.runs, attached), + pinnedBody(live), + ); + }); + expect(crashed.exitCode).toBe(1); + const runId = runIdOf(crashed); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("running"); + + // No checkout, no Workspace. Before the correction this reached Git for + // the bundle, because the status the run still carried was `running`. + const resumed = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { resumed, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.resumed.exitCode).toBe(0); + expect(statusOf(outcome.resumed)).toBe("completed"); + // The lifecycle recovered it; nothing was asked of the repository and + // nothing was attached. + expect(asked).toEqual([]); + expect(attached).toHaveLength(1); + expect(replayed[0]?.output).toBe(live[0]?.output); + expect(replayed[0]?.output).toContain("staged."); + expect(replayed[0]?.result?.ok).toBe(true); + + // The frontier is untouched, the stale envelope was closed by the settled + // recovery, and the run is the completed run its history says it is. + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + expect(outcome.after.ended).toEqual(["completed", "completed"]); + expect(outcome.after.status).toBe("completed"); + }); + + it("WRP9: recovers a bundled run whose committed result is a failure", function* () { + const asked: string[] = []; + const attached: string[] = []; + const live: Rendered[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n\n\n", + Unused: "never imported.\n", + }); + const crashed = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + REQUEST, + yield* startFor(fixture), + refusingSettlement(fixture.runs, attached), + pinnedBody(live), + ); + }); + expect(crashed.exitCode).toBe(1); + const runId = runIdOf(crashed); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("running"); + expect(live[0]?.result?.ok).toBe(false); + + const resumed = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { resumed, before, after: yield* retained(fixture.runs, runId) }; + }); + + // The same failure the crashed execution produced, and the output it had + // rendered before it failed. + expect(replayed[0]?.result?.ok).toBe(false); + expect(replayed[0]?.output).toBe(live[0]?.output); + expect(outcome.resumed.exitCode).toBe(1); + expect(asked).toEqual([]); + expect(attached).toHaveLength(1); + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + }); + + it("WRP10: recovers a bundled run whose retained result is a failed terminal", function* () { + const asked: string[] = []; + const attached: string[] = []; + let executed = 0; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + const established = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + + // A root coroutine that ended by raising rather than by producing a + // document result. `rootOutcome()` reads that as the run having failed, + // and names the exact row as its reason. + const runId = yield* seedStaleRun(fixture, established, (definition) => [ + forkRunRecordEvent({ + runId: "unused", + base: established.established.base, + pinnedCommit: definition.objectId, + }), + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { + kind: "repository", + path: definition.rootDocumentPath, + content: established.established.source, + }, + }, + }, + { + type: "close", + coroutineId: "root", + result: { status: "err", error: { name: "Error", message: "the executor died" } }, + }, + ]); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("running"); + + const resumed = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + executed += 1; + return Ok(undefined); + }, + ); + }); + return { resumed, before, after: yield* retained(fixture.runs, runId) }; + }); + + // The lifecycle recovered the canonical failed outcome and then applied the + // settled refusal: a run that failed is not resumed. + expect(outcome.resumed.exitCode).toBe(1); + expect(outcome.resumed.err.join(" ")).toContain("workflow run failed"); + expect(outcome.after.status).toBe("failed"); + expect(outcome.after.ended).toEqual(["failed"]); + // And it got there without a repository, a Workspace or an execution. + expect(asked).toEqual([]); + expect(attached).toEqual([]); + expect(executed).toBe(0); + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + }); + + it("WRP5: a run that has not ended still reconstructs, and its refusal is cleaned up", function* () { + const asked: string[] = []; + const attached: string[] = []; + const rendered: Rendered[] = []; const outcome = yield* scoped(function* () { const fixture = yield* useFixture(BUNDLED_WAITING, { Stage: "staged.\n" }); @@ -654,20 +846,46 @@ describe("what a completed run reaches when it is asked to run again", () => { pinnedBody([]), ); }); - return { refused, before, after: yield* retained(fixture.runs, runId) }; + const stalled = yield* retained(fixture.runs, runId); + + // The cleanup is the settled one, and it is the next acquisition's: it + // closes exactly the envelope that refused and continues the run into + // the wait it was standing at. + const again = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + liveHost(fixture.runs, attached), + pinnedBody(rendered), + ); + }); + return { refused, again, before, stalled, after: yield* retained(fixture.runs, runId) }; }); expect(outcome.refused.exitCode).toBe(1); // The repository was asked, which is the whole distinction. expect(asked.length).toBeGreaterThan(0); + // Nothing was published for a run this invocation could not advance. expect(statusOf(outcome.refused)).toBeUndefined(); - // Refused before it was admitted: the run is left exactly as it was, down - // to its status and the executions it had. - expect(outcome.after.journal).toBe(outcome.before.journal); + + // The lifecycle decided first, so the envelope exists. What it may not + // touch is the frontier: the journal and the Workspace root the run stands + // on are exactly what they were. + expect(outcome.stalled.journal).toBe(outcome.before.journal); + expect(outcome.stalled.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + expect(outcome.stalled.executions).toBe(outcome.before.executions + 1); + expect(outcome.stalled.ended.at(-1)).toBe(null); + + // And the settled recovery closes exactly that envelope as interrupted, + // without inventing an outcome for the run. + expect(outcome.after.ended.slice(0, -1)).toEqual([...outcome.before.ended, "interrupted"]); + expect(statusOf(outcome.again)).toBe("suspended"); expect(outcome.after.status).toBe("suspended"); - expect(outcome.after.executions).toBe(outcome.before.executions); - // One attachment, and it is the live start's. - expect(attached).toHaveLength(1); + expect(outcome.after.journal).toBe(outcome.before.journal); + // Two attachments: the live start's and the recovered continuation's. The + // refused invocation attached nothing. + expect(attached).toHaveLength(2); }); it("WRP6: replays a recorded answer without consuming or appending another", function* () { @@ -809,8 +1027,8 @@ describe("what a completed run reaches when it is asked to run again", () => { }); /** The production host with the settlement its storage refuses. */ -function refusingSettlement(runs: string): WorkflowHost { - const live = liveHost(runs, []); +function refusingSettlement(runs: string, attached: string[] = []): WorkflowHost { + const live = liveHost(runs, attached); return { *useRunHost(): Operation { const transitions = yield* live.useRunHost(); @@ -830,6 +1048,49 @@ function refusingSettlement(runs: string): WorkflowHost { }; } +/** + * A run this host created and then stopped holding, with the history a dead + * executor left behind. + * + * Created through the same transitions production uses and left exactly as a + * lost executor leaves a run: `running`, one execution nobody closed, and a + * journal that already records what the document did. + */ +function* seedStaleRun( + fixture: Fixture, + start: WorkflowStart, + events: (definition: WorkflowDefinition) => readonly DurableEvent[], +): Operation { + return yield* scoped(function* () { + const transitions = yield* useWorkflowRunHost({ root: fixture.runs }); + const runId = crypto.randomUUID(); + const acquired = yield* WorkflowLifecycle.operations.acquireExecutor(runId); + if (!acquired.ok) { + throw acquired.error; + } + if (acquired.value.kind !== "acquired") { + throw new Error(`${runId} already has a live workflow executor`); + } + const begun = yield* transitions.begin(acquired.value.lock, { + runId, + action: "start", + creation: { + definition: start.established.definition, + base: start.established.base, + props: {}, + retrieval: start.established.retrieval, + }, + }); + if (!begun.ok) { + throw begun.error; + } + for (const event of events(start.established.definition)) { + yield* begun.value.database.journal.append(event); + } + return runId; + }); +} + /** * Take every retained event out of one run's journal, leaving its lifecycle row * saying the run ended. diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index dfd04c6bb..1d79d7dcd 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -62,7 +62,7 @@ export { export type { GitApi, GitObjectFormat } from "./src/git.ts"; export { getWorkflowRun, retainedWorkflowInstallation, workflowInstallation } from "./src/run.ts"; export { workflowBundleInstallation, WorkflowBundleHistoryError } from "./src/bundle.ts"; -export { replaysRetainedResult, retainedReplay, WorkflowReplayHistoryError } from "./src/replay.ts"; +export { retainedReplay, WorkflowReplayHistoryError } from "./src/replay.ts"; export type { RetainedReplay } from "./src/replay.ts"; export type { WorkflowRun } from "./src/run.ts"; export { useWorkflowServiceDenial, WorkflowServiceDeniedError } from "./src/service-denial.ts"; diff --git a/packages/workflow/src/bundle.ts b/packages/workflow/src/bundle.ts index 4af79cf71..4f11c64da 100644 --- a/packages/workflow/src/bundle.ts +++ b/packages/workflow/src/bundle.ts @@ -28,7 +28,9 @@ * A completed replay takes the second half without the first, through * `workflowBundleReplayInstallation()`: it imports nothing, so it is handed no * source to import from, and its records are held to the name, path and object - * id the immutable definition declares. + * id the immutable definition declares — with the recorded bytes named as a + * Git blob and required to *be* that object rather than merely to repeat its + * id beside itself. */ import type { DurableEvent, Json, Yield } from "@executablemd/durable-streams"; @@ -37,7 +39,9 @@ import type { JournalAdmission, WorkflowBundleComponent, } from "@executablemd/core/host"; -import type { WorkflowComponentEntry } from "./storage/definition.ts"; +import { definitionComponents } from "./storage/definition.ts"; +import type { WorkflowDefinition } from "./storage/definition.ts"; +import { gitBlobId } from "./git-blob.ts"; /** The root's own import, which is not a bundle member and is admitted elsewhere. */ const ROOT = "__root__"; @@ -72,18 +76,20 @@ const REFUSALS = { * One component a run's definition declares, as an admission holds a retained * import to it. * - * `content` is the exact pinned source, and it is present exactly when this run - * may also import the component. A live or partial execution holds it, so a - * retained record that says something else about the same object is refused - * against the bytes themselves. A completed replay holds none — it imports - * nothing and is given nothing to import — and its records are held to the name, - * path and object id the immutable definition retains, which is the half of - * this comparison that never came from the journal. + * `holds` is where the two halves of this contract differ, and it is the only + * place they may. A live or partial execution has already read the pinned + * source from the definition's own commit, so a retained record is held to + * those exact bytes. A completed replay has no pinned source and no repository + * to ask for one, so it does what Git does: it names the recorded bytes as a + * blob under the definition's own object format and requires that name to be + * the object id the definition declares. Either way the bytes are + * authenticated — repeating an object id beside unrelated bytes is not. */ interface DeclaredComponent { readonly path: string; readonly sourceHash: string; - readonly content?: string; + /** Whether these exact bytes are the object this run's definition names. */ + holds(content: string): boolean; } /** @@ -154,8 +160,8 @@ function importedValue(event: DurableEvent): { value: unknown } | undefined | ty * * Every branch is a decision about what the record *is*, taken before anything * is replayed from it. A declared name must have been recorded as a bundled - * component, with the exact path and object id the definition declares — and - * with this run's exact source too, where this run holds one; an undeclared + * component, with the exact path and object id the definition declares, and + * with bytes that are that object; an undeclared * name must not claim to be one; and a repository selection is admitted only * for the root, which core holds to the run's own root source. */ @@ -197,18 +203,17 @@ function admitImport( } } const read = (member: string) => reading(() => (record as Record)[member]); - if (read("path") !== declared.path || read("sourceHash") !== declared.sourceHash) { - throw new WorkflowBundleHistoryError(REFUSALS.mismatched); + const content = read("content"); + if (typeof content !== "string") { + throw new WorkflowBundleHistoryError(REFUSALS.unreadable); } - // The source only when this run holds one. A replay was given none, so - // comparing the record with itself is what there would be to do, and the - // object id above is the definition's rather than the history's. - if (declared.content !== undefined && read("content") !== declared.content) { + if ( + read("path") !== declared.path || + read("sourceHash") !== declared.sourceHash || + !declared.holds(content) + ) { throw new WorkflowBundleHistoryError(REFUSALS.mismatched); } - if (typeof read("content") !== "string") { - throw new WorkflowBundleHistoryError(REFUSALS.unreadable); - } return; } @@ -266,7 +271,9 @@ export function workflowBundleInstallation( Object.freeze({ path: component.path, sourceHash: component.sourceHash, - content: component.content, + // The bytes themselves, because this run has them: they were read from + // the definition's own commit before it existed. + holds: (content: string) => content === component.content, }), ]), ); @@ -297,12 +304,19 @@ export function workflowBundleInstallation( * ``` */ export function workflowBundleReplayInstallation( - declared: readonly WorkflowComponentEntry[], + definition: WorkflowDefinition, ): ExecutionInstallation { + const { objectFormat } = definition; const index = new Map( - declared.map((entry) => [ + definitionComponents(definition).map((entry) => [ entry.name, - Object.freeze({ path: entry.path, sourceHash: entry.sourceHash }), + Object.freeze({ + path: entry.path, + sourceHash: entry.sourceHash, + // Named the way Git names a blob, under this definition's own object + // format, and required to be the object the definition declares. + holds: (content: string) => gitBlobId(content, objectFormat) === entry.sourceHash, + }), ]), ); return { admissions: [admits(index)] }; diff --git a/packages/workflow/src/git-blob.ts b/packages/workflow/src/git-blob.ts new file mode 100644 index 000000000..2ce920d10 --- /dev/null +++ b/packages/workflow/src/git-blob.ts @@ -0,0 +1,149 @@ +/** + * The object id Git gives one blob, computed rather than asked for. + * + * A workflow definition names each bundled component by the object id of the + * blob it was read from, and that id is identity: changing what a component + * says changes the definition rather than changing what a retained definition + * executes. Holding a retained history to that identity therefore means holding + * the exact bytes it recorded to it — repeating the id beside unrelated bytes + * establishes nothing about the bytes. + * + * A live run has the pinned source in hand and compares bytes with bytes. A + * completed replay has no pinned source and no repository to ask, so it does + * what Git does: frame the content as a blob object and name it. The framing is + * Git's own — the kind, the number of bytes, a NUL, then the content — and the + * number is the length of the *encoded* bytes rather than of the string, so a + * document whose characters and bytes differ in number is named the way Git + * names it. + * + * The arithmetic lives here for the reason `workspace/sha256.ts` gives for + * carrying its own: every host has a SHA-1 already and none of them has one a + * shared module can use. `node:crypto` names a host, and `crypto.subtle` is + * asynchronous — and this runs inside canonical core's synchronous journal + * admission, where there is nothing to await into. FIPS 180-4 is fixed, small + * and has published answers, and the tests hold this to Git's own. + */ + +import { sha256 } from "./workspace/sha256.ts"; +import type { GitObjectFormat } from "./git.ts"; + +const INITIAL = new Uint32Array([0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476, 0xc3d2e1f0]); + +const ROUND = new Uint32Array([0x5a827999, 0x6ed9eba1, 0x8f1bbcdc, 0xca62c1d6]); + +function rotate(value: number, bits: number): number { + return ((value << bits) | (value >>> (32 - bits))) >>> 0; +} + +function padded(input: Uint8Array): Uint8Array { + const length = Math.ceil((input.length + 9) / 64) * 64; + const bytes = new Uint8Array(length); + bytes.set(input); + bytes[input.length] = 0x80; + const bits = BigInt(input.length) * 8n; + for (let index = 0; index < 8; index += 1) { + bytes[length - 1 - index] = Number((bits >> BigInt(index * 8)) & 0xffn); + } + return bytes; +} + +/** The mixing function this round uses, by the quarter it falls in. */ +function mixed(round: number, b: number, c: number, d: number): number { + if (round < 20) { + return (b & c) | (~b & d); + } + if (round < 40) { + return b ^ c ^ d; + } + if (round < 60) { + return (b & c) | (b & d) | (c & d); + } + return b ^ c ^ d; +} + +export function sha1(value: Uint8Array | string): Uint8Array { + const input = typeof value === "string" ? new TextEncoder().encode(value) : value; + const bytes = padded(input); + const state = new Uint32Array(INITIAL); + const words = new Uint32Array(80); + for (let offset = 0; offset < bytes.length; offset += 64) { + for (let index = 0; index < 16; index += 1) { + const at = offset + index * 4; + words[index] = + (((bytes[at] ?? 0) << 24) | + ((bytes[at + 1] ?? 0) << 16) | + ((bytes[at + 2] ?? 0) << 8) | + (bytes[at + 3] ?? 0)) >>> + 0; + } + for (let index = 16; index < 80; index += 1) { + words[index] = rotate( + (words[index - 3] ?? 0) ^ + (words[index - 8] ?? 0) ^ + (words[index - 14] ?? 0) ^ + (words[index - 16] ?? 0), + 1, + ); + } + + let a = state[0] ?? 0; + let b = state[1] ?? 0; + let c = state[2] ?? 0; + let d = state[3] ?? 0; + let e = state[4] ?? 0; + for (let round = 0; round < 80; round += 1) { + const mixture = + (rotate(a, 5) + + mixed(round, b, c, d) + + e + + (ROUND[Math.floor(round / 20)] ?? 0) + + (words[round] ?? 0)) >>> + 0; + e = d; + d = c; + c = rotate(b, 30); + b = a; + a = mixture; + } + state[0] = ((state[0] ?? 0) + a) >>> 0; + state[1] = ((state[1] ?? 0) + b) >>> 0; + state[2] = ((state[2] ?? 0) + c) >>> 0; + state[3] = ((state[3] ?? 0) + d) >>> 0; + state[4] = ((state[4] ?? 0) + e) >>> 0; + } + const digest = new Uint8Array(20); + for (let index = 0; index < state.length; index += 1) { + const word = state[index] ?? 0; + digest[index * 4] = word >>> 24; + digest[index * 4 + 1] = word >>> 16; + digest[index * 4 + 2] = word >>> 8; + digest[index * 4 + 3] = word; + } + return digest; +} + +function hex(digest: Uint8Array): string { + return Array.from(digest, (byte) => byte.toString(16).padStart(2, "0")).join(""); +} + +export function sha1Hex(value: Uint8Array | string): string { + return hex(sha1(value)); +} + +/** The separator Git writes between an object's header and its content. */ +const NUL = "\u0000"; + +/** + * The object id one blob has under a repository's object format. + * + * What `git hash-object -t blob` answers, and nothing else. + */ +export function gitBlobId(content: string, objectFormat: GitObjectFormat): string { + const encoder = new TextEncoder(); + const body = encoder.encode(content); + const header = encoder.encode(`blob ${body.length}${NUL}`); + const framed = new Uint8Array(header.length + body.length); + framed.set(header); + framed.set(body, header.length); + return objectFormat === "sha256" ? hex(sha256(framed)) : hex(sha1(framed)); +} diff --git a/packages/workflow/src/replay.ts b/packages/workflow/src/replay.ts index 014b73866..d4160aeac 100644 --- a/packages/workflow/src/replay.ts +++ b/packages/workflow/src/replay.ts @@ -39,10 +39,9 @@ import { retainedSource } from "@executablemd/core/host"; import type { ExecutionInstallation, RetainedRootDocument } from "@executablemd/core/host"; import { workflowBundleReplayInstallation } from "./bundle.ts"; import { retainedWorkflowInstallation } from "./run.ts"; -import { terminal } from "./lifecycle/policy.ts"; +import { rootOutcome, terminal } from "./lifecycle/policy.ts"; import type { JournalEntry } from "./storage/api.ts"; -import { definitionComponents } from "./storage/definition.ts"; -import type { WorkflowRunRecord, WorkflowRunStatus } from "./storage/record.ts"; +import type { DocumentExecutionCompletion, WorkflowRunRecord } from "./storage/record.ts"; /** The root import a run's own entry is recorded under. */ const ROOT = "__root__"; @@ -88,6 +87,9 @@ const REFUSALS = Object.freeze({ document: "this run's retained root document is not the document its definition names. The run is " + "left exactly as it is.", + disagreed: + "this run's retained state and its recorded document result describe different outcomes, " + + "so neither is the one to replay. The run is left exactly as it is.", }); function refuse(reason: string): Result { @@ -142,6 +144,28 @@ function importedName(event: DurableEvent): string | undefined { return typeof name === "string" ? name : undefined; } +/** + * Whether the two accounts of why this run stopped are the same account. + * + * A canonical outcome that names an event names the exact retained row the + * lifecycle recorded as its reason; one that names none leaves the run with + * none. A host code in that position describes a stop the document itself did + * not record, which is not a terminal to replay from. + */ +function sameReason( + retained: DocumentExecutionCompletion["reason"], + canonical: DocumentExecutionCompletion["reason"], +): boolean { + if (canonical === undefined || retained === undefined) { + return canonical === retained; + } + return ( + canonical.kind === "journal" && + retained.kind === "journal" && + canonical.eventId === retained.eventId + ); +} + /** Whether a retained event is this document execution's own terminal. */ function isRootClose(event: DurableEvent): boolean { return reading(() => event.type === "close" && event.coroutineId === "root") === true; @@ -231,19 +255,6 @@ function boundRoot(close: DurableEvent): RetainedRoot | undefined { return { path, content: source, target: target ?? undefined }; } -/** - * Whether a run in this state replays what it retained rather than continuing. - * - * The one place a caller needs the question answered before it has a frontier - * to read. A resume that is going to replay must not fetch a definition on the - * way in, and the retained status is what says so while the run is still being - * admitted. It is not the decision — `retainedReplay()` is, from the history - * itself — it is what keeps live retrieval from happening ahead of one. - */ -export function replaysRetainedResult(status: WorkflowRunStatus): boolean { - return terminal(status); -} - /** * What this run's owner already holds, as the inputs one canonical replay runs * on — or why the state it holds describes no completed run. @@ -274,6 +285,21 @@ export function retainedReplay( return refuse(REFUSALS.mixed); } + // The document result the journal records, and the lifecycle row that stands + // behind it, have to be the same outcome. `rootOutcome()` is the settled + // mapping — the one recovery and settlement already publish through — so this + // is the lifecycle's own judgment rather than a second copy of it that could + // drift. A row and a result claiming different terminals are damaged retained + // state, and neither of them is the one to replay. + const outcome = rootOutcome(entries); + if ( + outcome === undefined || + outcome.status !== record.status || + !sameReason(record.stopReason, outcome.reason) + ) { + return refuse(REFUSALS.disagreed); + } + const imports = entries.filter((entry) => importedName(entry.event) === ROOT); if (imports.length > 1) { return refuse(REFUSALS.ambiguous); @@ -300,7 +326,7 @@ export function retainedReplay( base: record.base, pinnedCommit: record.definition.objectId, }), - workflowBundleReplayInstallation(definitionComponents(record.definition)), + workflowBundleReplayInstallation(record.definition), ], }); } diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts index bd8e2db5a..9bdb337b1 100644 --- a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -45,6 +45,8 @@ import type { } from "../../src/lifecycle/execution.ts"; import { retainedReplay } from "../../src/replay.ts"; import { retainedWorkflowInstallation } from "../../src/run.ts"; +import { workflowBundleInstallation } from "../../src/bundle.ts"; +import { gitBlobId } from "../../src/git-blob.ts"; let unique = 0; const NOW = 1_800_000_000; @@ -163,6 +165,29 @@ const CREATION: WorkflowRunCreation = { props: {}, }; +/** One component the definition is closed over, named the way Git names it. */ +const STAGE = { name: "Stage", path: "Stage.md", content: "staged.\n" }; +const STAGE_HASH = gitBlobId(STAGE.content, "sha1"); +const BUNDLED_DOCUMENT = "# Remote\n\n\n"; + +const BUNDLED: WorkflowRunCreation = { + definition: { + ...CREATION.definition, + components: [{ name: STAGE.name, path: STAGE.path, sourceHash: STAGE_HASH }], + }, + base: "main", + props: {}, +}; + +/** The run contract every execution of this run installs. */ +function runContract(): ExecutionInstallation { + return retainedWorkflowInstallation({ + runId: RUN_ID, + base: CREATION.base, + pinnedCommit: COMMIT, + }); +} + /** What one document execution rendered, and how it ended. */ interface Rendered { readonly output: string; @@ -323,6 +348,178 @@ describe("a completed run replayed through its own owner", () => { expect(await on(stub, (owner) => owner.holders())).toBe(0); }); + it("recovers a bundled run whose result committed and whose settlement did not", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + // The runner committed the document's result and its connection went + // before it settled. Nothing about time says so; the socket closing does. + const live = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: BUNDLED, + }); + if (!begun.ok) { + throw begun.error; + } + return yield* canonical( + begun.value.database, + retainedSource("README.md", BUNDLED_DOCUMENT), + [runContract(), workflowBundleInstallation([{ ...STAGE, sourceHash: STAGE_HASH }])], + ); + }); + }); + + expect(live.result.ok).toBe(true); + expect(live.output).toContain("staged."); + const before = await ownerState(stub); + // The crash window: the outcome is committed and the lifecycle row is not. + expect(before.run?.["status"]).toBe("running"); + expect(before.executions).toBe(1); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + + const replayed = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + if (!begun.ok) { + throw begun.error; + } + // The owner's own transaction recognized the retained result, closed + // the stale execution and published the terminal it implies. + expect(begun.value.replay).toBe(true); + expect(begun.value.record.status).toBe("completed"); + const frontier = yield* begun.value.database.readJournalEntries(); + if (!frontier.ok) { + throw frontier.error; + } + const prepared = retainedReplay(begun.value.record, frontier.value); + if (!prepared.ok) { + throw prepared.error; + } + const rendered = yield* canonical(begun.value.database, prepared.value.root, [ + ...prepared.value.installations, + ]); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + if (!settled.ok) { + throw settled.error; + } + return rendered; + }); + }); + + // Byte for byte, including the bundled component, from history alone. + expect(replayed.output).toBe(live.output); + expect(replayed.result.ok).toBe(true); + + const after = await ownerState(stub); + expect(after.journal).toEqual(before.journal); + expect(after.currentRootId).toBe(before.currentRootId); + expect(after.published).toEqual(before.published); + expect(after.answers).toEqual(before.answers); + expect(after.run?.["status"]).toBe("completed"); + // The stale envelope closed and one replay envelope opened and closed. + expect(after.executions).toBe(2); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + + it("refuses a lifecycle row its retained result contradicts, and moves nothing", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + // A run whose journal records that its root ended by raising, settled as + // though it had completed. The two cannot both be this run's outcome. + await run(function* () { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + const { database } = begun.value; + const appended = yield* database.transact(function* (transaction) { + yield* transaction.journal.append({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "README.md", content: DOCUMENT }, + }, + }); + // The members in the order the protocol's own parser rebuilds them. + // The owner requires a proposed event to serialize back to the exact + // bytes it was sent as, so a differently ordered record is refused + // while the command is still being read. + yield* transaction.journal.append({ + type: "close", + coroutineId: "root", + result: { status: "err", error: { message: "the executor died", name: "Error" } }, + }); + }); + if (!appended.ok) { + throw appended.error; + } + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + if (!settled.ok) { + throw settled.error; + } + }); + }); + + const before = await ownerState(stub); + expect(before.run?.["status"]).toBe("completed"); + + const refusal = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + if (!begun.ok) { + throw begun.error; + } + const frontier = yield* begun.value.database.readJournalEntries(); + if (!frontier.ok) { + throw frontier.error; + } + const prepared = retainedReplay(begun.value.record, frontier.value); + if (prepared.ok) { + throw new Error("expected the contradictory retained state to be refused"); + } + return prepared.error.message; + }); + }); + + expect(refusal).toContain("describe different outcomes"); + // Nothing this run holds moved, and no replacement outcome was published: + // the one difference is the envelope the begin boundary had to insert, and + // the settled recovery closes exactly that. + const after = await ownerState(stub); + expect(after.journal).toEqual(before.journal); + expect(after.currentRootId).toBe(before.currentRootId); + expect(after.published).toEqual(before.published); + expect(after.answers).toEqual(before.answers); + expect(after.run?.["status"]).toBe("completed"); + expect(after.executions).toBe(before.executions + 1); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + it("hands the run to the next connection when the replay's own ends", async () => { const stub = executor(); const host = lifecycleHost(stub); diff --git a/packages/workflow/tests/git-blob.test.ts b/packages/workflow/tests/git-blob.test.ts new file mode 100644 index 000000000..dde4243b1 --- /dev/null +++ b/packages/workflow/tests/git-blob.test.ts @@ -0,0 +1,76 @@ +/** + * Naming a blob the way Git names one. + * + * A workflow definition holds each bundled component's object id, and a + * completed replay has no repository to ask what a retained source hashes to. + * So it computes the name itself — which is only worth anything if the name it + * computes is Git's. Two authorities settle that here, and neither of them is + * this code: FIPS 180-4's published SHA-1 answers, and the object ids + * `git hash-object -t blob` gave these exact bytes. + * + * The object ids are committed constants rather than a computation. Deriving + * them from the function under test would be a test agreeing with itself, and + * shelling out to Git would make a portable suite depend on a program. The + * end-to-end proof that this agrees with a real repository is + * `packages/cli/tests/workflow-replay.test.ts`, where the definition's hashes + * come from `git rev-parse` and this admission authenticates against them. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { gitBlobId, sha1Hex } from "../src/git-blob.ts"; + +/** Eight UTF-16 units, eleven bytes: `é` is two and the dragon is four. */ +const WIDE = "café \u{1f409}\n"; + +describe("SHA-1, held to its published answers", () => { + // deno-lint-ignore require-yield + it("reproduces the FIPS 180-4 examples", function* () { + expect(sha1Hex("abc")).toBe("a9993e364706816aba3e25717850c26c9cd0d89d"); + expect(sha1Hex("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq")).toBe( + "84983e441c3bd26ebaae4aa1f95129e5e54670f1", + ); + expect(sha1Hex("")).toBe("da39a3ee5e6b4b0d3255bfef95601890afd80709"); + // A million `a`s is the third published example; this is the block-boundary + // half of what it exercises, at a size a test can carry. + expect(sha1Hex("a".repeat(1000))).toBe("291e9a6c66994949b57ba5e650361e98fc36b1ba"); + }); + + // deno-lint-ignore require-yield + it("hashes bytes, not characters", function* () { + expect(sha1Hex(WIDE)).toBe(sha1Hex(new TextEncoder().encode(WIDE))); + }); +}); + +describe("the object id a blob has", () => { + // deno-lint-ignore require-yield + it("is the one Git gives it, under either object format", function* () { + expect(gitBlobId("staged.\n", "sha1")).toBe("4eb53b7fd720524e22040757b43e821f817ff0eb"); + expect(gitBlobId("staged.\n", "sha256")).toBe( + "bee278bf729e0ac11f0bd6bf2ec94b1536d51883bd6e426ac32ec0a94afe76ca", + ); + expect(gitBlobId("never imported.\n", "sha1")).toBe("0b42d358385c85db1957138c7a200ad153514209"); + expect(gitBlobId("", "sha1")).toBe("e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"); + }); + + // deno-lint-ignore require-yield + it("frames the header with the encoded byte length", function* () { + // Eleven bytes, eight units of `String#length`. A framing that used the + // string length would write `blob 8` and name an object Git does not. + expect(new TextEncoder().encode(WIDE).length).toBe(11); + expect(WIDE.length).toBe(8); + expect(gitBlobId(WIDE, "sha1")).toBe("c4ae463ec163e7b0b1a47ca6f0d5a2205d3643dc"); + expect(gitBlobId(WIDE, "sha256")).toBe( + "3a2a85ffaa00d300e360a8f0e3b0d1b13e6bcdabfdcd8124d2f2e3dc062cc9f5", + ); + }); + + // deno-lint-ignore require-yield + it("names different bytes differently", function* () { + expect(gitBlobId("ALTERED\n", "sha1")).toBe("e93f6b023845f2035a5f3d299ae4624802b4e891"); + expect(gitBlobId("ALTERED\n", "sha1")).not.toBe(gitBlobId("staged.\n", "sha1")); + // The framing is what keeps content from being confused with its own + // header: these are not the same object. + expect(gitBlobId("staged.\n", "sha1")).not.toBe(sha1Hex("staged.\n")); + }); +}); diff --git a/packages/workflow/tests/replay-inputs.test.ts b/packages/workflow/tests/replay-inputs.test.ts index a041728a0..a73d1dd17 100644 --- a/packages/workflow/tests/replay-inputs.test.ts +++ b/packages/workflow/tests/replay-inputs.test.ts @@ -19,14 +19,18 @@ import { expect } from "@executablemd/test-support/expect"; import type { Operation, Result } from "effection"; import type { DurableEvent, Json } from "@executablemd/durable-streams"; import type { ExecutionInstallation } from "@executablemd/core/host"; -import { replaysRetainedResult, retainedReplay } from "../src/replay.ts"; +import { retainedReplay } from "../src/replay.ts"; import type { RetainedReplay } from "../src/replay.ts"; import { WorkflowReplayHistoryError } from "../src/replay.ts"; import { WorkflowBundleHistoryError } from "../src/bundle.ts"; import { forkRunRecordEvent } from "../src/journal-events.ts"; import type { JournalEntry } from "../src/storage/api.ts"; import type { WorkflowComponentEntry } from "../src/storage/definition.ts"; -import type { WorkflowRunRecord, WorkflowRunStatus } from "../src/storage/record.ts"; +import type { + WorkflowRunRecord, + WorkflowRunStatus, + WorkflowStopReason, +} from "../src/storage/record.ts"; const ROOT_ID = "a".repeat(64); const COMMIT = "0".repeat(40); @@ -35,17 +39,20 @@ const SOURCE = "# Retained\n\ndone.\n"; function record( overrides: { readonly status?: WorkflowRunStatus; + readonly stopReason?: WorkflowStopReason; readonly rootDocumentPath?: string; + readonly objectFormat?: "sha1" | "sha256"; readonly components?: readonly WorkflowComponentEntry[]; } = {}, ): WorkflowRunRecord { const components = overrides.components; + const stopReason = overrides.stopReason; return { runId: "replay-1", definition: { version: 1, kind: "git", - objectFormat: "sha1", + objectFormat: overrides.objectFormat ?? "sha1", objectId: COMMIT, rootDocumentPath: overrides.rootDocumentPath ?? "flows/root.md", ...(components === undefined ? {} : { components }), @@ -53,6 +60,7 @@ function record( base: "main", props: {}, status: overrides.status ?? "completed", + ...(stopReason === undefined ? {} : { stopReason }), createdAt: "2026-01-01T00:00:00.000Z", updatedAt: "2026-01-01T00:00:00.000Z", }; @@ -227,11 +235,6 @@ describe("retained state that describes no completed run", () => { for (const status of live) { const outcome = retainedReplay(record({ status }), completedHistory()); expect([status, reason(outcome).includes("not terminal")]).toEqual([status, true]); - expect([status, replaysRetainedResult(status)]).toEqual([status, false]); - } - const ended: readonly WorkflowRunStatus[] = ["completed", "failed"]; - for (const status of ended) { - expect([status, replaysRetainedResult(status)]).toEqual([status, true]); } }); @@ -332,12 +335,144 @@ describe("retained state that describes no completed run", () => { }); }); +describe("the lifecycle row and the recorded result have to agree", () => { + /** One history whose root result is the outcome this case is about. */ + function ending(close: DurableEvent): { entries: JournalEntry[]; closeEventId: string } { + const entries = [ + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + entry(close), + ]; + return { entries, closeEventId: entries[1]?.eventId ?? "" }; + } + + const failure = { name: "Error", message: "the run did not finish" }; + + // deno-lint-ignore require-yield + it("accepts a successful result under a completed run", function* () { + const { entries } = ending(rootClose({ status: "ok", output: "", value: "" })); + + expect(admitted(retainedReplay(record(), entries)).root.path).toBe("flows/root.md"); + }); + + // deno-lint-ignore require-yield + it("accepts a failed result under a failed run naming that exact event", function* () { + const { entries, closeEventId } = ending({ + type: "close", + coroutineId: "root", + result: { status: "err", error: failure }, + }); + + const outcome = retainedReplay( + record({ status: "failed", stopReason: { kind: "journal", eventId: closeEventId } }), + entries, + ); + expect(admitted(outcome).root.path).toBe("flows/root.md"); + }); + + // deno-lint-ignore require-yield + it("refuses every pairing the settled lifecycle cannot produce", function* () { + const errored: DurableEvent = { + type: "close", + coroutineId: "root", + result: { status: "err", error: failure }, + }; + const cancelled: DurableEvent = { + type: "close", + coroutineId: "root", + result: { status: "cancelled" }, + }; + const succeeded = rootClose({ status: "ok", output: "", value: "" }); + + const cases: { says: string; close: DurableEvent; status: WorkflowRunStatus; reason?: true }[] = + [ + { says: "a successful result under a failed run", close: succeeded, status: "failed" }, + { says: "a failed result under a completed run", close: errored, status: "completed" }, + { says: "a cancelled result under a completed run", close: cancelled, status: "completed" }, + { says: "a cancelled result under a failed run", close: cancelled, status: "failed" }, + { says: "a failed run naming no reason at all", close: errored, status: "failed" }, + { + says: "a failed run naming another event", + close: errored, + status: "failed", + reason: true, + }, + ]; + + for (const { says, close, status, reason: elsewhere } of cases) { + const { entries } = ending(close); + const outcome = retainedReplay( + record({ + status, + ...(elsewhere === true + ? { stopReason: { kind: "journal", eventId: "event-somewhere-else" } } + : {}), + }), + entries, + ); + expect([says, reason(outcome).includes("describe different outcomes")]).toEqual([says, true]); + } + }); + + // deno-lint-ignore require-yield + it("refuses a failed run whose reason is a host code rather than the event", function* () { + const { entries } = ending({ + type: "close", + coroutineId: "root", + result: { status: "err", error: failure }, + }); + + const outcome = retainedReplay( + record({ status: "failed", stopReason: { kind: "host", code: "document-execution-failed" } }), + entries, + ); + expect(reason(outcome)).toContain("describe different outcomes"); + }); + + // deno-lint-ignore require-yield + it("refuses a completed run carrying a stop reason of its own", function* () { + const { entries, closeEventId } = ending(rootClose({ status: "ok", output: "", value: "" })); + + const outcome = retainedReplay( + record({ status: "completed", stopReason: { kind: "journal", eventId: closeEventId } }), + entries, + ); + expect(reason(outcome)).toContain("describe different outcomes"); + }); +}); + +/** + * The object ids `git hash-object -t blob` gives these exact bytes. + * + * Committed constants rather than a computation, because a test that derived + * them from the same function it is checking would agree with itself. They came + * from Git, and `packages/workflow/tests/git-blob.test.ts` holds the arithmetic + * to Git's own answers and to FIPS 180-4's published ones. + */ +const STAGED = "staged.\n"; +const STAGED_SHA1 = "4eb53b7fd720524e22040757b43e821f817ff0eb"; +const STAGED_SHA256 = "bee278bf729e0ac11f0bd6bf2ec94b1536d51883bd6e426ac32ec0a94afe76ca"; +const UNUSED_SHA1 = "0b42d358385c85db1957138c7a200ad153514209"; +/** Four characters and seven bytes, so Git's framing cannot use the string length. */ +const WIDE = "caf\u00e9 \u{1f409}\n"; +const WIDE_SHA1 = "c4ae463ec163e7b0b1a47ca6f0d5a2205d3643dc"; + describe("the bundle a completed replay is held to", () => { const declared: readonly WorkflowComponentEntry[] = [ - { name: "Stage", path: "flows/Stage.md", sourceHash: "1".repeat(40) }, - { name: "Unused", path: "flows/Unused.md", sourceHash: "2".repeat(40) }, + { name: "Stage", path: "flows/Stage.md", sourceHash: STAGED_SHA1 }, + { name: "Unused", path: "flows/Unused.md", sourceHash: UNUSED_SHA1 }, ]; + /** One retained import of the declared `Stage`, as canonical execution wrote it. */ + function staged(overrides: Record = {}): DurableEvent { + return componentImport("Stage", { + kind: "workflow", + path: "flows/Stage.md", + sourceHash: STAGED_SHA1, + content: STAGED, + ...overrides, + }); + } + it("grants no authority to import anything", function* () { const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); @@ -350,44 +485,86 @@ describe("the bundle a completed replay is held to", () => { } }); - it("admits an import the definition declares, by path and object id", function* () { + it("admits an import whose bytes are the object the definition names", function* () { const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); - const held = componentImport("Stage", { - kind: "workflow", - path: "flows/Stage.md", - sourceHash: "1".repeat(40), - content: "staged.\n", - }); - expect(yield* admit(replay.installations, [RUN_RECORD, held])).toBe(undefined); + expect(yield* admit(replay.installations, [RUN_RECORD, staged()])).toBe(undefined); + }); + + it("admits the same under sha256, and where bytes outnumber characters", function* () { + const cases: { format: "sha1" | "sha256"; hash: string; content: string }[] = [ + { format: "sha256", hash: STAGED_SHA256, content: STAGED }, + { format: "sha1", hash: WIDE_SHA1, content: WIDE }, + ]; + + for (const { format, hash, content } of cases) { + const components: readonly WorkflowComponentEntry[] = [ + { name: "Stage", path: "flows/Stage.md", sourceHash: hash }, + ]; + const replay = admitted( + retainedReplay(record({ components, objectFormat: format }), completedHistory()), + ); + const held = componentImport("Stage", { + kind: "workflow", + path: "flows/Stage.md", + sourceHash: hash, + content, + }); + expect([format, yield* admit(replay.installations, [RUN_RECORD, held])]).toEqual([ + format, + undefined, + ]); + + // The same identity, other bytes. Repeating an object id is not being it. + const altered = componentImport("Stage", { + kind: "workflow", + path: "flows/Stage.md", + sourceHash: hash, + content: `${content}ALTERED\n`, + }); + expect(yield* admit(replay.installations, [RUN_RECORD, altered])).toEqual( + expect.any(WorkflowBundleHistoryError), + ); + } + }); + + it("refuses altered bytes under the object id the definition declares", function* () { + const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); + // The exact declared path and object id, and content that is not that + // object. This is the record a history rewritten in place would carry. + const altered = staged({ content: "ALTERED\n" }); + + const refused = yield* admit(replay.installations, [RUN_RECORD, altered]); + expect(refused).toEqual(expect.any(WorkflowBundleHistoryError)); + // And it says nothing about what it read. + expect(String(refused)).not.toContain("ALTERED"); + expect(String(refused)).not.toContain("flows/Stage.md"); }); - it("refuses one recorded under another path, hash or name", function* () { + it("refuses one recorded under another path, hash, name or kind", function* () { const replay = admitted(retainedReplay(record({ components: declared }), completedHistory())); const wrong: DurableEvent[] = [ - componentImport("Stage", { - kind: "workflow", - path: "flows/Elsewhere.md", - sourceHash: "1".repeat(40), - content: "staged.\n", - }), + staged({ path: "flows/Elsewhere.md" }), + // A different object id, and content that really is that object: the + // definition still does not name it. componentImport("Stage", { kind: "workflow", path: "flows/Stage.md", - sourceHash: "9".repeat(40), - content: "staged.\n", + sourceHash: UNUSED_SHA1, + content: "never imported.\n", }), componentImport("Undeclared", { kind: "workflow", path: "flows/Stage.md", - sourceHash: "1".repeat(40), - content: "staged.\n", + sourceHash: STAGED_SHA1, + content: STAGED, }), componentImport("Stage", { kind: "repository", path: "flows/Stage.md", - content: "staged.\n", + content: STAGED, }), + staged({ content: 7 }), ]; for (const event of wrong) { From f0ea6686ce63f71bca87c6ddd6a70361486e8130 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 9 Sep 2026 10:56:00 -0300 Subject: [PATCH 64/81] =?UTF-8?q?=F0=9F=90=9B=20Replay=20every=20run=20the?= =?UTF-8?q?=20lifecycle=20already=20ended,=20however=20it=20was=20named?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `start` naming a run that has already ended is the same terminal reuse a `resume` of one is, and the begin transaction says so: it holds the supplied definition, base, props and bundle to the immutable record, and answers `replay`. The branch below it asked a second question — whether a candidate definition had been supplied — and a compatible `start --id` always has one. So the shipped compatible-reuse surface took the live path. It skipped the lifecycle/root agreement entirely, so a row saying `completed` over a root that raised was handed to canonical core, which returned the retained failure and left the invocation offering `failed` for a run the owner records as completed — two authoritative outcomes for one run. It also constructed the suspension controller, the answer provider and the live bundle execution view for an execution canonical core returns from before expansion. `replay` is now the whole question. What a caller established describes the request; what the run retained describes the result. The agreement judgment moves to `lifecycle/policy.ts`, beside the mappings it is derived from, and now accounts for both authorities that publish a terminal for one root Close. Recovery reads the coroutine's own settlement, so a document that *returned* a failure recovers as `completed`; the executor that watched it settles the same Close as `failed`, naming the row it failed at. Both are states this system produces. What no settled path produces — a failed row over a document that succeeded, a terminal row over a root that raised or was cancelled and says otherwise, a reason naming something the run does not hold — still refuses before terminal reuse. --- packages/cli/src/workflow.ts | 13 +- packages/cli/tests/workflow-replay.test.ts | 370 ++++++++++++++++-- packages/workflow/src/lifecycle/policy.ts | 93 ++++- packages/workflow/src/replay.ts | 43 +- .../tests/cloudflare/remote-replay.vitest.ts | 87 ++++ 5 files changed, 539 insertions(+), 67 deletions(-) diff --git a/packages/cli/src/workflow.ts b/packages/cli/src/workflow.ts index 56c431906..5d39a7578 100644 --- a/packages/cli/src/workflow.ts +++ b/packages/cli/src/workflow.ts @@ -1004,8 +1004,15 @@ export function runWorkflow( // controller, the answer provider, the `` declaration and // `host.attach()` are each work for an execution that is going to import // nothing, perform nothing and append nothing. - const replayed = replay && start === undefined; - const prepared = replayed + // + // `replay` alone, and deliberately: a `start` naming a run that already + // ended is the same terminal reuse a `resume` of one is, and the candidate + // definition it carried is what proved the two runs are the same run rather + // than a second account of what that run did. The begin transaction has + // already held the supplied definition, base, props and bundle to the + // immutable record; what a caller established describes the request, and + // what the run retained describes the result. + const prepared = replay ? retainedReplay(record, frontier.value) : yield* liveDocument(record, start, database); if (!prepared.ok) { @@ -1015,7 +1022,7 @@ export function runWorkflow( // Nothing beyond this point loads for a completed replay: no controller, no // answer provider, and no import of the adapter either one comes from. - const support = replayed ? undefined : yield* liveSupport(database); + const support = replay ? undefined : yield* liveSupport(database); // Interruption is the outcome nothing else publishes. Registered before the // execution starts, so a scope torn down by Ctrl-C settles the run rather diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts index b962fb425..79eab34f9 100644 --- a/packages/cli/tests/workflow-replay.test.ts +++ b/packages/cli/tests/workflow-replay.test.ts @@ -289,6 +289,8 @@ function invoke( /** What one document execution was given, what it rendered, and how it ended. */ interface Rendered { root: unknown; + /** Whether any installation offered an execution view to import from. */ + imports: boolean; output: string; result: Result | undefined; } @@ -310,7 +312,14 @@ function pinnedBody(seen: Rendered[]): (execution: WorkflowExecution) => Operati next = yield* subscription.next(); } const result = yield* running; - seen.push({ root: { ...execution.root }, output: next.value, result }); + seen.push({ + root: { ...execution.root }, + imports: execution.installations.some( + (installation) => installation.bundle !== undefined, + ), + output: next.value, + result, + }); return result.ok ? Ok(undefined) : Err(new Error(String(result.error.message))); }), ); @@ -341,6 +350,7 @@ function statusOf(invocation: Invocation): string | undefined { /** Everything about a run a replay must not move, read through the host itself. */ interface Retained { readonly status: WorkflowRunStatus; + readonly stopReason: string; readonly executions: number; /** What each execution ended as, in order. `null` is one still open. */ readonly ended: (WorkflowRunStatus | null)[]; @@ -361,6 +371,7 @@ function* retained(runs: string, runId: string): Operation { } return { status: snapshot.value.record.status, + stopReason: JSON.stringify(snapshot.value.record.stopReason ?? null), executions: snapshot.value.executions.length, ended: snapshot.value.executions.map((execution) => execution.stopStatus ?? null), currentWorkspaceRootId: snapshot.value.currentWorkspaceRootId, @@ -755,31 +766,7 @@ describe("what a completed run reaches when it is asked to run again", () => { // A root coroutine that ended by raising rather than by producing a // document result. `rootOutcome()` reads that as the run having failed, // and names the exact row as its reason. - const runId = yield* seedStaleRun(fixture, established, (definition) => [ - forkRunRecordEvent({ - runId: "unused", - base: established.established.base, - pinnedCommit: definition.objectId, - }), - { - type: "yield", - coroutineId: "root", - description: { type: "import_component", name: "__root__" }, - result: { - status: "ok", - value: { - kind: "repository", - path: definition.rootDocumentPath, - content: established.established.source, - }, - }, - }, - { - type: "close", - coroutineId: "root", - result: { status: "err", error: { name: "Error", message: "the executor died" } }, - }, - ]); + const runId = yield* seedStaleRun(fixture, established, raisedHistory(established)); const before = yield* retained(fixture.runs, runId); expect(before.status).toBe("running"); @@ -813,6 +800,279 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); }); + it("WRP11: replays a completed run named again by a compatible start", function* () { + const asked: string[] = []; + const attached: string[] = []; + const live: Rendered[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + const runId = "compatible-1"; + + // Establishing the candidate is what proves the two runs are the same + // run, and it reads the repository. It happens before the invocation, and + // everything the invocation itself asks of Git is recorded separately. + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + { ...REQUEST, id: runId }, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody(live), + ); + }); + expect(started.exitCode).toBe(0); + expect(runIdOf(started)).toBe(runId); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("completed"); + + // The same definition and props, named at the same run. The candidate is + // established under a repository that answers; the invocation runs under + // one that refuses, so anything it asks for after admission is a planted + // failure. + const candidate = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + const again = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + candidate, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { again, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.again.exitCode).toBe(0); + expect(statusOf(outcome.again)).toBe("completed"); + // Nothing was asked of the repository after admission, and nothing was + // attached: the candidate described the request, and the run's own history + // supplied the result. + expect(asked).toEqual([]); + expect(attached).toHaveLength(1); + expect(replayed).toHaveLength(1); + expect(replayed[0]?.root).toEqual({ path: "workflow.md", source: BUNDLED, retained: true }); + expect(replayed[0]?.output).toBe(live[0]?.output); + expect(replayed[0]?.output).toContain("staged."); + expect(replayed[0]?.result?.ok).toBe(true); + // The live run was given a bundle to import from; the replay was not. It + // resolves no name, so it is granted no authority to resolve one. + expect(live[0]?.imports).toBe(true); + expect(replayed[0]?.imports).toBe(false); + + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + expect(outcome.after.status).toBe("completed"); + expect(outcome.after.stopReason).toBe(outcome.before.stopReason); + expect(outcome.after.executions).toBe(outcome.before.executions + 1); + }); + + it("WRP12: refuses a compatible start over a lifecycle row its result contradicts", function* () { + const asked: string[] = []; + const attached: string[] = []; + let executed = 0; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + const established = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + + // The root raised, and the row says the run completed. Two accounts of + // one run, and a replay that reused either would be choosing between them. + const runId = yield* seedStaleRun(fixture, established, raisedHistory(established), { + status: "completed", + }); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("completed"); + + const refused = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + established, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + executed += 1; + return Ok(undefined); + }, + ); + }); + const stalled = yield* retained(fixture.runs, runId); + + // The next acquisition closes exactly the envelope the refusal left, and + // publishes no replacement outcome for the run. + yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + established, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + executed += 1; + return Ok(undefined); + }, + ); + }); + return { refused, before, stalled, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.refused.exitCode).toBe(1); + expect(outcome.refused.err.join(" ")).toContain("describe different outcomes"); + // Refused before terminal reuse, before live support and before any + // attachment: nothing executed and nothing was asked of the repository. + expect(executed).toBe(0); + expect(asked).toEqual([]); + expect(attached).toEqual([]); + expect(statusOf(outcome.refused)).toBeUndefined(); + + // The one difference is the envelope begin had already inserted. + expect(outcome.stalled.journal).toBe(outcome.before.journal); + expect(outcome.stalled.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + expect(outcome.stalled.status).toBe("completed"); + expect(outcome.stalled.stopReason).toBe(outcome.before.stopReason); + expect(outcome.stalled.executions).toBe(outcome.before.executions + 1); + expect(outcome.stalled.ended.at(-1)).toBe(null); + + // And the settled terminal-replay recovery closes that envelope alone: the + // next begin finishes it as interrupted, publishes no outcome for the run, + // and refuses the same contradiction again. + expect(outcome.after.ended).toEqual([...outcome.before.ended, "interrupted", null]); + expect(outcome.after.status).toBe("completed"); + expect(outcome.after.stopReason).toBe(outcome.before.stopReason); + expect(outcome.after.journal).toBe(outcome.before.journal); + }); + + it("WRP13: replays a coherent failed run named again by a compatible start", function* () { + const asked: string[] = []; + const attached: string[] = []; + const replayed: Rendered[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + const established = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + + // A failed row naming the exact retained result it failed at. + const runId = yield* seedStaleRun(fixture, established, raisedHistory(established), { + status: "failed", + reason: "root-close", + }); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("failed"); + + const again = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + established, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { again, before, after: yield* retained(fixture.runs, runId) }; + }); + + // The same failure, replayed rather than retried. + expect(outcome.again.exitCode).toBe(1); + expect(replayed).toHaveLength(1); + expect(replayed[0]?.result?.ok).toBe(false); + expect(outcome.again.err.join(" ")).toContain("the executor died"); + expect(asked).toEqual([]); + expect(attached).toEqual([]); + + // And the retained failed outcome is the one that stands. + expect(outcome.after.status).toBe("failed"); + expect(outcome.after.stopReason).toBe(outcome.before.stopReason); + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); + }); + + it("WRP14: refuses a start whose definition is not the run it names", function* () { + const asked: string[] = []; + const attached: string[] = []; + let executed = 0; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + const runId = "incompatible-1"; + const started = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* invoke( + { ...REQUEST, id: runId }, + yield* startFor(fixture), + liveHost(fixture.runs, attached), + pinnedBody([]), + ); + }); + expect(started.exitCode).toBe(0); + const before = yield* retained(fixture.runs, runId); + + // One component says something else, and it is committed. The bundle is + // definition identity, so this names a run of different code. + yield* writeTextFile(join(fixture.repository, "Stage.md"), "staged differently.\n"); + yield* git(fixture.repository, ["add", "-A"]); + yield* git(fixture.repository, [ + "-c", + "commit.gpgsign=false", + "commit", + "--quiet", + "-m", + "a component changed", + ]); + + const candidate = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + const refused = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + candidate, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + executed += 1; + return Ok(undefined); + }, + ); + }); + return { refused, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.refused.exitCode).toBe(1); + expect(outcome.refused.err.join(" ")).toContain("definition"); + expect(statusOf(outcome.refused)).toBeUndefined(); + // Refused inside the begin transaction, before a replay execution existed. + expect(executed).toBe(0); + expect(attached).toHaveLength(1); + expect(outcome.after.executions).toBe(outcome.before.executions); + expect(outcome.after.journal).toBe(outcome.before.journal); + expect(outcome.after.status).toBe(outcome.before.status); + }); + it("WRP5: a run that has not ended still reconstructs, and its refusal is cleaned up", function* () { const asked: string[] = []; const attached: string[] = []; @@ -1059,7 +1319,8 @@ function refusingSettlement(runs: string, attached: string[] = []): WorkflowHost function* seedStaleRun( fixture: Fixture, start: WorkflowStart, - events: (definition: WorkflowDefinition) => readonly DurableEvent[], + events: (definition: WorkflowDefinition, runId: string) => readonly DurableEvent[], + ending?: { readonly status: WorkflowRunStatus; readonly reason?: "root-close" }, ): Operation { return yield* scoped(function* () { const transitions = yield* useWorkflowRunHost({ root: fixture.runs }); @@ -1084,13 +1345,66 @@ function* seedStaleRun( if (!begun.ok) { throw begun.error; } - for (const event of events(start.established.definition)) { + for (const event of events(start.established.definition, runId)) { yield* begun.value.database.journal.append(event); } + if (ending === undefined) { + return runId; + } + // Settled by this same acquisition, so the run is left the way an executor + // that finished leaves one rather than the way a lost one does. + const entries = yield* begun.value.database.readJournalEntries(); + if (!entries.ok) { + throw entries.error; + } + const close = entries.value.find( + (entry) => entry.event.type === "close" && entry.event.coroutineId === "root", + ); + const settled = yield* transitions.settle(acquired.value.lock, { + executionId: begun.value.execution.executionId, + status: ending.status, + ...(ending.reason === undefined + ? {} + : { reason: { kind: "journal", eventId: close?.eventId ?? "" } }), + }); + if (!settled.ok) { + throw settled.error; + } return runId; }); } +/** The history a run that raised out of its root leaves behind. */ +function raisedHistory( + start: WorkflowStart, +): (definition: WorkflowDefinition, runId: string) => readonly DurableEvent[] { + return (definition, runId) => [ + forkRunRecordEvent({ + runId, + base: start.established.base, + pinnedCommit: definition.objectId, + }), + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { + kind: "repository", + path: definition.rootDocumentPath, + content: start.established.source, + }, + }, + }, + { + type: "close", + coroutineId: "root", + result: { status: "err", error: { message: "the executor died", name: "Error" } }, + }, + ]; +} + /** * Take every retained event out of one run's journal, leaving its lifecycle row * saying the run ended. diff --git a/packages/workflow/src/lifecycle/policy.ts b/packages/workflow/src/lifecycle/policy.ts index 48d7204a7..c7d51caf3 100644 --- a/packages/workflow/src/lifecycle/policy.ts +++ b/packages/workflow/src/lifecycle/policy.ts @@ -15,7 +15,11 @@ import type { DurableEvent } from "@executablemd/durable-streams"; import { WorkflowRequestError } from "../storage/errors.ts"; -import type { DocumentExecutionCompletion, WorkflowRunStatus } from "../storage/record.ts"; +import type { + DocumentExecutionCompletion, + WorkflowRunStatus, + WorkflowStopReason, +} from "../storage/record.ts"; import type { JournalEntry } from "../storage/api.ts"; /** An outcome that already won. A run in one of these is not made mutable again. */ @@ -54,6 +58,93 @@ export function rootOutcome( return undefined; } +/** + * Whether the two accounts of why this run stopped are the same account. + * + * A reason that names an event names the exact retained row; one that names + * none leaves the run with none. + */ +function sameReason( + retained: DocumentExecutionCompletion["reason"], + canonical: DocumentExecutionCompletion["reason"], +): boolean { + if (canonical === undefined || retained === undefined) { + return canonical === retained; + } + return ( + canonical.kind === "journal" && + retained.kind === "journal" && + canonical.eventId === retained.eventId + ); +} + +/** Whether the root's recorded result says the document itself failed. */ +function documentFailed(entries: readonly JournalEntry[]): boolean { + for (const entry of entries) { + const event: DurableEvent = entry.event; + if (event.type !== "close" || event.coroutineId !== "root") { + continue; + } + if (event.result.status !== "ok") { + return true; + } + const value: unknown = event.result.value; + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return false; + } + return Reflect.get(value, "status") === "err"; + } + return false; +} + +/** Whether a reason a settlement chose is one this run's history can carry. */ +function retainedReason( + reason: DocumentExecutionCompletion["reason"], + entries: readonly JournalEntry[], +): boolean { + if (reason === undefined) { + return false; + } + return reason.kind === "host" || entries.some((entry) => entry.eventId === reason.eventId); +} + +/** + * Whether a run in this state is one this retained history can have produced. + * + * Two authorities publish a terminal for one root Close, and they do not always + * publish the same one. `rootOutcome()` above is what *recovery* publishes: it + * reads the coroutine's own settlement, so a document that returned a failure + * closed successfully and recovers as `completed` with no reason. The executor + * that ran the document settles from the document's own result instead, so the + * same Close settles as `failed`, naming whichever retained row it failed at. + * + * Both are states this system produces and neither is damaged. What no settled + * path produces is a run whose status contradicts the result its journal + * records: a `failed` row over a document that succeeded, a terminal row over a + * root that raised or was cancelled and says otherwise, or a reason naming + * something the run does not hold. Those are two accounts of one run, and a + * replay that reused either would be choosing between them. + */ +export function agreesWithRetainedResult( + record: { readonly status: WorkflowRunStatus; readonly stopReason?: WorkflowStopReason }, + entries: readonly JournalEntry[], +): boolean { + const canonical = rootOutcome(entries); + if (canonical === undefined) { + return false; + } + if (record.status === canonical.status) { + return sameReason(record.stopReason, canonical.reason); + } + // The other authority. It differs from recovery in exactly one place — a + // document that returned a failure — and it is the executor that watched it + // do so, so its reason is one it chose rather than one this can derive. + if (record.status !== "failed" || canonical.status !== "completed") { + return false; + } + return documentFailed(entries) && retainedReason(record.stopReason, entries); +} + /** What closing a dead executor's execution makes of it, and of the run. */ export interface Closing { readonly status: WorkflowRunStatus; diff --git a/packages/workflow/src/replay.ts b/packages/workflow/src/replay.ts index d4160aeac..3ecb7c525 100644 --- a/packages/workflow/src/replay.ts +++ b/packages/workflow/src/replay.ts @@ -39,9 +39,9 @@ import { retainedSource } from "@executablemd/core/host"; import type { ExecutionInstallation, RetainedRootDocument } from "@executablemd/core/host"; import { workflowBundleReplayInstallation } from "./bundle.ts"; import { retainedWorkflowInstallation } from "./run.ts"; -import { rootOutcome, terminal } from "./lifecycle/policy.ts"; +import { agreesWithRetainedResult, terminal } from "./lifecycle/policy.ts"; import type { JournalEntry } from "./storage/api.ts"; -import type { DocumentExecutionCompletion, WorkflowRunRecord } from "./storage/record.ts"; +import type { WorkflowRunRecord } from "./storage/record.ts"; /** The root import a run's own entry is recorded under. */ const ROOT = "__root__"; @@ -144,28 +144,6 @@ function importedName(event: DurableEvent): string | undefined { return typeof name === "string" ? name : undefined; } -/** - * Whether the two accounts of why this run stopped are the same account. - * - * A canonical outcome that names an event names the exact retained row the - * lifecycle recorded as its reason; one that names none leaves the run with - * none. A host code in that position describes a stop the document itself did - * not record, which is not a terminal to replay from. - */ -function sameReason( - retained: DocumentExecutionCompletion["reason"], - canonical: DocumentExecutionCompletion["reason"], -): boolean { - if (canonical === undefined || retained === undefined) { - return canonical === retained; - } - return ( - canonical.kind === "journal" && - retained.kind === "journal" && - canonical.eventId === retained.eventId - ); -} - /** Whether a retained event is this document execution's own terminal. */ function isRootClose(event: DurableEvent): boolean { return reading(() => event.type === "close" && event.coroutineId === "root") === true; @@ -286,17 +264,12 @@ export function retainedReplay( } // The document result the journal records, and the lifecycle row that stands - // behind it, have to be the same outcome. `rootOutcome()` is the settled - // mapping — the one recovery and settlement already publish through — so this - // is the lifecycle's own judgment rather than a second copy of it that could - // drift. A row and a result claiming different terminals are damaged retained - // state, and neither of them is the one to replay. - const outcome = rootOutcome(entries); - if ( - outcome === undefined || - outcome.status !== record.status || - !sameReason(record.stopReason, outcome.reason) - ) { + // behind it, have to be the same outcome. The judgment is the lifecycle's own + // — `agreesWithRetainedResult()` sits beside the recovery and settlement + // mappings it is derived from — rather than a second copy of it here that + // could drift. A row and a result claiming different terminals are damaged + // retained state, and neither of them is the one to replay. + if (!agreesWithRetainedResult(record, entries)) { return refuse(REFUSALS.disagreed); } diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts index 9bdb337b1..ad5db74ae 100644 --- a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -348,6 +348,93 @@ describe("a completed run replayed through its own owner", () => { expect(await on(stub, (owner) => owner.holders())).toBe(0); }); + it("replays a completed run its owner is asked to start again", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + const live = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + const rendered = yield* canonical( + begun.value.database, + retainedSource("README.md", DOCUMENT), + [runContract()], + ); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + if (!settled.ok) { + throw settled.error; + } + return rendered; + }); + }); + + const before = await ownerState(stub); + expect(before.run?.["status"]).toBe("completed"); + + // The same creation named at the same run. The owner compares it with the + // immutable record it already holds and answers `replay`, exactly as it + // does for a resume — a caller supplying a candidate definition proves the + // two runs are the same run; it does not make the run live again. + const replayed = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + expect(begun.value.replay).toBe(true); + const frontier = yield* begun.value.database.readJournalEntries(); + if (!frontier.ok) { + throw frontier.error; + } + const prepared = retainedReplay(begun.value.record, frontier.value); + if (!prepared.ok) { + throw prepared.error; + } + const rendered = yield* canonical(begun.value.database, prepared.value.root, [ + ...prepared.value.installations, + ]); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "completed", + }); + if (!settled.ok) { + throw settled.error; + } + return rendered; + }); + }); + + expect(replayed.output).toBe(live.output); + expect(replayed.result.ok).toBe(true); + + const after = await ownerState(stub); + expect(after.journal).toEqual(before.journal); + expect(after.currentRootId).toBe(before.currentRootId); + expect(after.published).toEqual(before.published); + expect(after.answers).toEqual(before.answers); + expect(after.run?.["status"]).toBe("completed"); + expect(after.executions).toBe(before.executions + 1); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + it("recovers a bundled run whose result committed and whose settlement did not", async () => { const stub = executor(); const host = lifecycleHost(stub); From c8fd0f0e2482ca4d1b26d00d8bcfe6b329dff665 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 9 Sep 2026 11:55:34 -0300 Subject: [PATCH 65/81] =?UTF-8?q?=F0=9F=90=9B=20Give=20one=20execution=20o?= =?UTF-8?q?ne=20outcome,=20and=20let=20a=20replay=20change=20nothing=20but?= =?UTF-8?q?=20its=20own=20envelope?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections, both about a run being described twice. **The root Close says what the document did, not merely that it returned.** A durable Close has two layers: the coroutine's own settlement, and — when it settled by returning — the document result it returned. Reading only the outer layer called every finished document a completed one, so recovery published `completed` for a run whose document failed while an uninterrupted settlement published `failed`. The correction had then written that disagreement down as two authorities allowed to differ, and admitted whatever reason the second one carried: an unrelated successful row, an invented host code, or a terminal with no readable document result at all. `rootOutcome()` now reads the whole thing. A returned result decides `completed` or `failed`; an outer error or cancellation keeps its own meaning; and a returned value that is not a document result is `damaged` — refused before canonical core is handed it, rather than becoming an outcome nobody can account for. The failed reason comes from `retainedFailureReason()`, the one rule the live runner already used, which settlement, stale recovery and retained-history admission now all reach. A stale run whose document failed therefore recovers to exactly the `failed` status and reason an uninterrupted settlement publishes; `resume` then applies the settled failed-run refusal and a compatible `start --id` replays that same failure. **A replay closes its own envelope and nothing else.** The remote owner already skipped publication for a terminal run; the local adapter published unconditionally, so even a coherent replay rewrote the run's status, stop reason and `updated_at`. It now applies the same rule the owner does, decided from the retained state inside the same transaction and exact-lock checks. --- packages/cli/src/workflow.ts | 23 +- packages/cli/tests/workflow-replay.test.ts | 143 ++++++++-- packages/workflow/mod.ts | 1 + packages/workflow/src/deno/transitions.ts | 21 +- packages/workflow/src/lifecycle/policy.ts | 270 ++++++++++++------ packages/workflow/src/replay.ts | 27 +- .../tests/cloudflare/remote-replay.vitest.ts | 87 ++++++ packages/workflow/tests/replay-inputs.test.ts | 162 ++++++++++- .../tests/workflow-lifecycle-control.test.ts | 8 +- 9 files changed, 610 insertions(+), 132 deletions(-) diff --git a/packages/cli/src/workflow.ts b/packages/cli/src/workflow.ts index 5d39a7578..eec367fa9 100644 --- a/packages/cli/src/workflow.ts +++ b/packages/cli/src/workflow.ts @@ -73,6 +73,7 @@ import { retainedSource, validateProps } from "@executablemd/core"; import type { PropsSchema } from "@executablemd/core"; import type { RootDocumentSource } from "@executablemd/core"; import { + retainedFailureReason, retainedReplay, retainedWorkflowInstallation, workflowBundleInstallation, @@ -178,8 +179,6 @@ const EXIT_BY_STATUS: Readonly> = Object.freez running: 1, }); -/** A failure the host classified, rather than an exception message it retained. */ -const HOST_FAILURE_CODE = "document-execution-failed"; const HOST_INTERRUPTED_CODE = "executor-interrupted"; const HOST_ORPHANED_CODE = "executor-disappeared"; @@ -435,22 +434,16 @@ function isRootClose(event: DurableEvent): boolean { /** * The stop reason a failure gets. * - * A retained event that already crossed the secret filter is preferable to a - * code, because it says which effect failed. Anything else becomes one - * categorical host code: the alternative is retaining an exception message - * beside the journal that filtered it, which is history nothing has filtered. + * The rule is the lifecycle's own and lives beside the outcome it belongs to: a + * retained event that already crossed the secret filter says which effect + * failed, and a failure the journal holds no row for gets the one categorical + * code. Stale recovery reading a dead executor's journal and a retained history + * held to its lifecycle row reach the same rule, because a reason chosen three + * ways would be three explanations of one failure. */ function* failureReason(database: WorkflowRunDatabase): Operation { const entries = yield* database.readJournalEntries(); - if (entries.ok) { - for (let index = entries.value.length - 1; index >= 0; index -= 1) { - const entry = entries.value[index]; - if (entry !== undefined && entry.event.result.status === "err") { - return { kind: "journal", eventId: entry.eventId }; - } - } - } - return { kind: "host", code: HOST_FAILURE_CODE }; + return retainedFailureReason(entries.ok ? entries.value : []); } /** diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts index 79eab34f9..32479ae87 100644 --- a/packages/cli/tests/workflow-replay.test.ts +++ b/packages/cli/tests/workflow-replay.test.ts @@ -351,6 +351,12 @@ function statusOf(invocation: Invocation): string | undefined { interface Retained { readonly status: WorkflowRunStatus; readonly stopReason: string; + /** + * Which rule chose the reason, in terms two different runs can be compared + * by: a journal reason names a row, and a row's identity is its own run's. + */ + readonly reasonAt: string; + readonly updatedAt: string; readonly executions: number; /** What each execution ended as, in order. `null` is one still open. */ readonly ended: (WorkflowRunStatus | null)[]; @@ -369,9 +375,18 @@ function* retained(runs: string, runId: string): Operation { if (!history.ok) { throw history.error; } + const stopReason = snapshot.value.record.stopReason; + const at = + stopReason === undefined + ? "none" + : stopReason.kind === "host" + ? `host:${stopReason.code}` + : `journal:${history.value.findIndex((entry) => entry.eventId === stopReason.eventId)}`; return { status: snapshot.value.record.status, - stopReason: JSON.stringify(snapshot.value.record.stopReason ?? null), + stopReason: JSON.stringify(stopReason ?? null), + reasonAt: at, + updatedAt: snapshot.value.record.updatedAt, executions: snapshot.value.executions.length, ended: snapshot.value.executions.map((execution) => execution.stopStatus ?? null), currentWorkspaceRootId: snapshot.value.currentWorkspaceRootId, @@ -477,14 +492,18 @@ describe("what a completed run reaches when it is asked to run again", () => { ); // The run is exactly where it was, apart from the one execution envelope - // the lifecycle records for the invocation that replayed it. + // the lifecycle records for the invocation that replayed it. A replay + // observes an outcome that already won, so it republishes nothing — not the + // status, not the reason, and not when the run last moved. expect(outcome.after.status).toBe("completed"); + expect(outcome.after.stopReason).toBe(outcome.before.stopReason); + expect(outcome.after.updatedAt).toBe(outcome.before.updatedAt); expect(outcome.after.journal).toBe(outcome.before.journal); expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); expect(outcome.after.executions).toBe(outcome.before.executions + 1); }); - it("WRP2: replays a retained failure, with the output it had rendered", function* () { + it("WRP2: recovers a stale failure to itself, refuses resume, and replays it", function* () { const asked: string[] = []; const attached: string[] = []; const live: Rendered[] = []; @@ -494,9 +513,8 @@ describe("what a completed run reaches when it is asked to run again", () => { const fixture = yield* useFixture(FAILING); // The document fails and its settlement never lands, so the run is left - // holding a terminal nothing published. The next acquisition recovers it - // from that terminal, which is how a failed document becomes a run whose - // retained state ends without ever being resumable through Git. + // holding a result nothing published. Recovery reads the same result the + // settlement would have, and publishes the same outcome. const started = yield* scoped(function* () { yield* useRepositoryGit(fixture.repository); return yield* invoke( @@ -511,7 +529,26 @@ describe("what a completed run reaches when it is asked to run again", () => { const before = yield* retained(fixture.runs, runId); expect(before.status).toBe("running"); - const resumed = yield* scoped(function* () { + // What an uninterrupted settlement would have published, for comparison + // with what recovery does. + const uninterrupted = yield* scoped(function* () { + const fixtureTwo = yield* useFixture(FAILING); + const settled = yield* scoped(function* () { + yield* useRepositoryGit(fixtureTwo.repository); + return yield* invoke( + { ...REQUEST, id: "settled-1" }, + yield* startFor(fixtureTwo), + liveHost(fixtureTwo.runs, []), + pinnedBody([]), + ); + }); + expect(settled.exitCode).toBe(1); + const state = yield* retained(fixtureTwo.runs, "settled-1"); + return { status: state.status, reason: state.reasonAt }; + }); + + // A resume is what the settled lifecycle refuses for a failed run. + const refused = yield* scoped(function* () { yield* useRefusingGit(asked); return yield* invoke( { ...REQUEST, action: "resume", target: runId }, @@ -520,25 +557,61 @@ describe("what a completed run reaches when it is asked to run again", () => { pinnedBody(replayed), ); }); - return { resumed, before, after: yield* retained(fixture.runs, runId) }; + const recovered = yield* retained(fixture.runs, runId); + + // The same run, named again by a compatible start, replays that failure. + const candidate = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + const again = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + candidate, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { + refused, + again, + before, + recovered, + uninterrupted, + after: yield* retained(fixture.runs, runId), + }; }); - // The same failure the live execution produced, and the partial output it - // had rendered before it failed. + // Recovery published exactly what an uninterrupted settlement publishes — + // one semantic outcome, reached two ways. + expect(outcome.recovered.status).toBe(outcome.uninterrupted.status); + expect(outcome.recovered.status).toBe("failed"); + expect(outcome.recovered.reasonAt).toBe(outcome.uninterrupted.reason); + expect(outcome.recovered.journal).toBe(outcome.before.journal); + // The resume is refused by the settled failed-run rule, without a replay + // envelope of its own. + expect(outcome.refused.exitCode).toBe(1); + expect(outcome.refused.err.join(" ")).toContain("workflow run failed"); + expect(outcome.recovered.executions).toBe(outcome.before.executions); + expect(statusOf(outcome.refused)).toBeUndefined(); + + // The compatible start replays the same failure and the partial output it + // had rendered, reaching no repository and no Workspace. expect(replayed).toHaveLength(1); expect(replayed[0]?.result?.ok).toBe(false); expect(replayed[0]?.output).toBe(live[0]?.output); expect(replayed[0]?.output).toContain("partial line."); - expect(outcome.resumed.exitCode).toBe(1); - - // No provider was reached and no row was added to the history. + expect(outcome.again.exitCode).toBe(1); expect(asked).toEqual([]); expect(attached).toEqual([]); + // And the retained failure is the one that stands, byte for byte. + expect(outcome.after.status).toBe("failed"); + expect(outcome.after.stopReason).toBe(outcome.recovered.stopReason); + expect(outcome.after.updatedAt).toBe(outcome.recovered.updatedAt); expect(outcome.after.journal).toBe(outcome.before.journal); expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); - // The run follows the outcome it actually had: the terminal it replayed - // says the document failed, and that is what the invocation publishes. - expect(statusOf(outcome.resumed)).toBe("failed"); + expect(outcome.after.executions).toBe(outcome.recovered.executions + 1); }); it("WRP3: replays a bundled run without reading one component", function* () { @@ -725,7 +798,9 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(before.status).toBe("running"); expect(live[0]?.result?.ok).toBe(false); - const resumed = yield* scoped(function* () { + // Recovery reads the document's own result, so a run whose document + // failed recovers as failed — and the settled rule then refuses a resume. + const refused = yield* scoped(function* () { yield* useRefusingGit(asked); return yield* invoke( { ...REQUEST, action: "resume", target: runId }, @@ -734,16 +809,41 @@ describe("what a completed run reaches when it is asked to run again", () => { pinnedBody(replayed), ); }); - return { resumed, before, after: yield* retained(fixture.runs, runId) }; + const recovered = yield* retained(fixture.runs, runId); + + const candidate = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + const again = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + candidate, + replayHost(fixture.runs, attached), + pinnedBody(replayed), + ); + }); + return { refused, again, before, recovered, after: yield* retained(fixture.runs, runId) }; }); - // The same failure the crashed execution produced, and the output it had + expect(outcome.recovered.status).toBe("failed"); + expect(outcome.recovered.journal).toBe(outcome.before.journal); + expect(outcome.refused.exitCode).toBe(1); + expect(outcome.refused.err.join(" ")).toContain("workflow run failed"); + expect(outcome.recovered.executions).toBe(outcome.before.executions); + + // The same failure, replayed rather than retried, with the output it had // rendered before it failed. + expect(replayed).toHaveLength(1); expect(replayed[0]?.result?.ok).toBe(false); expect(replayed[0]?.output).toBe(live[0]?.output); - expect(outcome.resumed.exitCode).toBe(1); + expect(outcome.again.exitCode).toBe(1); expect(asked).toEqual([]); expect(attached).toHaveLength(1); + expect(outcome.after.status).toBe("failed"); + expect(outcome.after.stopReason).toBe(outcome.recovered.stopReason); + expect(outcome.after.updatedAt).toBe(outcome.recovered.updatedAt); expect(outcome.after.journal).toBe(outcome.before.journal); expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); }); @@ -871,6 +971,7 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); expect(outcome.after.status).toBe("completed"); expect(outcome.after.stopReason).toBe(outcome.before.stopReason); + expect(outcome.after.updatedAt).toBe(outcome.before.updatedAt); expect(outcome.after.executions).toBe(outcome.before.executions + 1); }); @@ -944,6 +1045,7 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(outcome.stalled.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); expect(outcome.stalled.status).toBe("completed"); expect(outcome.stalled.stopReason).toBe(outcome.before.stopReason); + expect(outcome.stalled.updatedAt).toBe(outcome.before.updatedAt); expect(outcome.stalled.executions).toBe(outcome.before.executions + 1); expect(outcome.stalled.ended.at(-1)).toBe(null); @@ -1002,6 +1104,7 @@ describe("what a completed run reaches when it is asked to run again", () => { // And the retained failed outcome is the one that stands. expect(outcome.after.status).toBe("failed"); expect(outcome.after.stopReason).toBe(outcome.before.stopReason); + expect(outcome.after.updatedAt).toBe(outcome.before.updatedAt); expect(outcome.after.journal).toBe(outcome.before.journal); expect(outcome.after.currentWorkspaceRootId).toBe(outcome.before.currentWorkspaceRootId); }); diff --git a/packages/workflow/mod.ts b/packages/workflow/mod.ts index 1d79d7dcd..4e85f88fc 100644 --- a/packages/workflow/mod.ts +++ b/packages/workflow/mod.ts @@ -63,6 +63,7 @@ export type { GitApi, GitObjectFormat } from "./src/git.ts"; export { getWorkflowRun, retainedWorkflowInstallation, workflowInstallation } from "./src/run.ts"; export { workflowBundleInstallation, WorkflowBundleHistoryError } from "./src/bundle.ts"; export { retainedReplay, WorkflowReplayHistoryError } from "./src/replay.ts"; +export { DOCUMENT_FAILED, retainedFailureReason } from "./src/lifecycle/policy.ts"; export type { RetainedReplay } from "./src/replay.ts"; export type { WorkflowRun } from "./src/run.ts"; export { useWorkflowServiceDenial, WorkflowServiceDeniedError } from "./src/service-denial.ts"; diff --git a/packages/workflow/src/deno/transitions.ts b/packages/workflow/src/deno/transitions.ts index 7e11739c7..8642cada9 100644 --- a/packages/workflow/src/deno/transitions.ts +++ b/packages/workflow/src/deno/transitions.ts @@ -501,8 +501,15 @@ export function* settleExecution( throw new WorkflowDocumentExecutionError(completion.executionId); } const { database } = connection; + const stored = readRunRow(database, path); finish(database, path, completion); - publish(database, path, completion.status, completion.reason); + // A replay closes only its own envelope: the terminal outcome it observed + // is not made mutable again, and nothing about the run — its status, its + // reason or when it last moved — is rewritten by an execution that was + // only ever going to restore what was already there. + if (!terminal(stored.status)) { + publish(database, path, completion.status, completion.reason); + } const record = readRunRow(database, path); if (record.runId !== hold.runId) { throw new WorkflowRunIdMismatchError(hold.runId, path); @@ -836,6 +843,18 @@ export function* cancelRun( } const canonical = rootOutcome(readJournalEntries(database)); + if (canonical?.kind === "damaged") { + // The document finished and this build cannot read what it finished + // as. Cancelling it would replace a result rather than end a run that + // had none, so nothing here changes anything. + return { + kind: "refused" as const, + reason: new WorkflowRequestError( + "workflow run: its root recorded an outcome this version cannot read, so the run " + + "is neither cancelled nor changed.", + ), + }; + } if (canonical !== undefined) { // The document finished before its workflow executor disappeared. Restoring what it // recorded is not cancelling it. diff --git a/packages/workflow/src/lifecycle/policy.ts b/packages/workflow/src/lifecycle/policy.ts index c7d51caf3..e35495325 100644 --- a/packages/workflow/src/lifecycle/policy.ts +++ b/packages/workflow/src/lifecycle/policy.ts @@ -34,115 +34,212 @@ export const INTERRUPTED: DocumentExecutionCompletion["reason"] = Object.freeze( }); /** - * The canonical outcome the root recorded, when it recorded one. + * The one categorical word a failed run has when its journal holds no row that + * says why. * - * A root Close is what proves the document itself finished. Its result decides - * the run's terminal status, and its own event identity is the reason. + * A run that failed always names a reason. Usually that is the exact retained + * row it failed at; a failure the journal has no row for — one raised outside + * any durable operation — has this instead, and nothing else. It is a code + * rather than a message because the alternative is retaining an exception's + * text beside the journal that filtered it. */ -export function rootOutcome( - entries: readonly JournalEntry[], -): { status: WorkflowRunStatus; reason: DocumentExecutionCompletion["reason"] } | undefined { +export const DOCUMENT_FAILED = "document-execution-failed"; + +/** + * Why a failed run stopped, from the history it holds. + * + * The last retained row that failed, and this rule is the whole of it. The + * runner settling a live document, the recovery reading a dead one's journal + * and the admission holding a retained history to its lifecycle row all reach + * it here, because a reason chosen three ways would be three explanations of + * one failure. + */ +export function retainedFailureReason(entries: readonly JournalEntry[]): WorkflowStopReason { + for (let index = entries.length - 1; index >= 0; index -= 1) { + const entry = entries[index]; + if (entry !== undefined && entry.event.result.status === "err") { + return { kind: "journal", eventId: entry.eventId }; + } + } + return { kind: "host", code: DOCUMENT_FAILED }; +} + +/** + * What the root recorded, as one semantic outcome. + * + * A durable `Close` has two layers and both are load-bearing. The outer one is + * the coroutine's own settlement: it raised, it was cancelled, or it *returned* + * — and returning is what a document does whether it succeeded or failed. So an + * outer `ok` says only that the value beneath it is the document's own result, + * and that result's `status` is what decides whether the run completed or + * failed. Reading the outer layer alone calls every finished document a + * completed one. + * + * A returned value that is not a document result at all is neither: it is a + * terminal this build cannot read, and answering `completed` or `interrupted` + * for it would be inventing an outcome for history nobody can account for. + */ +export type RetainedTerminal = + | { + readonly kind: "outcome"; + readonly status: WorkflowRunStatus; + readonly reason: DocumentExecutionCompletion["reason"]; + } + | { readonly kind: "damaged" }; + +export function rootOutcome(entries: readonly JournalEntry[]): RetainedTerminal | undefined { for (const entry of entries) { const event: DurableEvent = entry.event; if (event.type !== "close" || event.coroutineId !== "root") { continue; } - if (event.result.status === "ok") { - return { status: "completed", reason: undefined }; + if (event.result.status !== "ok") { + return { + kind: "outcome", + status: event.result.status === "cancelled" ? "cancelled" : "failed", + reason: { kind: "journal", eventId: entry.eventId }, + }; } - return { - status: event.result.status === "cancelled" ? "cancelled" : "failed", - reason: { kind: "journal", eventId: entry.eventId }, - }; + const document = readDocumentResult(event.result.value); + if (document === undefined) { + return { kind: "damaged" }; + } + return document === "ok" + ? { kind: "outcome", status: "completed", reason: undefined } + : { kind: "outcome", status: "failed", reason: retainedFailureReason(entries) }; } return undefined; } /** - * Whether the two accounts of why this run stopped are the same account. + * The document result a returning root recorded, or nothing when it recorded + * something else. * - * A reason that names an event names the exact retained row; one that names - * none leaves the run with none. + * The shape is canonical core's `DocumentResult` (`packages/core/src/execute.ts`) + * and is parsed as the closed form it is: a success carries its rendered output + * and the value the document produced, a failure carries the output it rendered + * before it failed and the described failure itself, and a terminal core wrote + * before importing anything carries the root it was about beside them. A record + * missing a member, carrying one of the wrong type, or carrying a member this + * form does not have is not a result to reuse. */ -function sameReason( - retained: DocumentExecutionCompletion["reason"], - canonical: DocumentExecutionCompletion["reason"], -): boolean { - if (canonical === undefined || retained === undefined) { - return canonical === retained; +function readDocumentResult(value: unknown): "ok" | "err" | undefined { + const result = plain(value); + if (result === undefined || typeof result["output"] !== "string") { + return undefined; + } + if (result["status"] === "ok") { + return names(result, ["status", "output", "value"]) && "value" in result ? "ok" : undefined; + } + if (result["status"] !== "err") { + return undefined; + } + if (!names(result, ["status", "output", "error", ROOT_BINDING])) { + return undefined; + } + return readDocumentFailure(result["error"]) ? "err" : undefined; +} + +/** Where core records which document a terminal it wrote before importing was about. */ +const ROOT_BINDING = "root_binding"; + +/** Whether a described failure is the closed form core writes. */ +function readDocumentFailure(value: unknown): boolean { + const failure = plain(value); + if ( + failure === undefined || + typeof failure["name"] !== "string" || + typeof failure["message"] !== "string" || + !names(failure, ["name", "message", "segment", "cause", "errors"]) + ) { + return false; + } + const segment = plain(failure["segment"]); + if ( + segment === undefined || + typeof segment["message"] !== "string" || + !names(segment, ["message", "source"]) || + ("source" in segment && typeof segment["source"] !== "string") + ) { + return false; + } + if ("cause" in failure && typeof failure["cause"] !== "string") { + return false; + } + const errors = failure["errors"]; + if (errors === undefined) { + return true; } return ( - canonical.kind === "journal" && - retained.kind === "journal" && - canonical.eventId === retained.eventId + Array.isArray(errors) && + errors.every((entry) => { + const described = plain(entry); + return ( + described !== undefined && + typeof described["name"] === "string" && + typeof described["message"] === "string" && + names(described, ["name", "message"]) + ); + }) ); } -/** Whether the root's recorded result says the document itself failed. */ -function documentFailed(entries: readonly JournalEntry[]): boolean { - for (const entry of entries) { - const event: DurableEvent = entry.event; - if (event.type !== "close" || event.coroutineId !== "root") { - continue; - } - if (event.result.status !== "ok") { - return true; - } - const value: unknown = event.result.value; - if (typeof value !== "object" || value === null || Array.isArray(value)) { - return false; - } - return Reflect.get(value, "status") === "err"; +/** A retained value that is an ordinary object, read once through its own names. */ +function plain(value: unknown): Record | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + const held: Record = {}; + for (const name of Object.keys(value)) { + held[name] = Reflect.get(value, name); } - return false; + return held; } -/** Whether a reason a settlement chose is one this run's history can carry. */ -function retainedReason( - reason: DocumentExecutionCompletion["reason"], - entries: readonly JournalEntry[], +/** Whether a record carries only members this form declares. */ +function names(record: Record, declared: readonly string[]): boolean { + return Object.keys(record).every((name) => declared.includes(name)); +} + +/** + * Whether the two accounts of why this run stopped are the same account. + * + * A reason that names an event names the exact retained row, a categorical one + * carries the same word, and one that names nothing leaves the run with + * nothing. An unrelated row, the wrong row, an invented code and a missing + * reason are each a different explanation of the same failure. + */ +function sameReason( + retained: DocumentExecutionCompletion["reason"], + canonical: DocumentExecutionCompletion["reason"], ): boolean { - if (reason === undefined) { - return false; + if (canonical === undefined || retained === undefined) { + return canonical === retained; } - return reason.kind === "host" || entries.some((entry) => entry.eventId === reason.eventId); + if (canonical.kind === "journal") { + return retained.kind === "journal" && retained.eventId === canonical.eventId; + } + return retained.kind === "host" && retained.code === canonical.code; } /** - * Whether a run in this state is one this retained history can have produced. - * - * Two authorities publish a terminal for one root Close, and they do not always - * publish the same one. `rootOutcome()` above is what *recovery* publishes: it - * reads the coroutine's own settlement, so a document that returned a failure - * closed successfully and recovers as `completed` with no reason. The executor - * that ran the document settles from the document's own result instead, so the - * same Close settles as `failed`, naming whichever retained row it failed at. + * Whether a run in this state is the run this retained history produced. * - * Both are states this system produces and neither is damaged. What no settled - * path produces is a run whose status contradicts the result its journal - * records: a `failed` row over a document that succeeded, a terminal row over a - * root that raised or was cancelled and says otherwise, or a reason naming - * something the run does not hold. Those are two accounts of one run, and a - * replay that reused either would be choosing between them. + * One execution has one winning outcome, and `rootOutcome()` above is it. + * Settlement and stale recovery are two ways of publishing that same semantic + * result, not two authorities allowed to disagree — so a lifecycle row saying + * anything else is a second account of one run, and a replay that reused either + * would be choosing between them. */ export function agreesWithRetainedResult( record: { readonly status: WorkflowRunStatus; readonly stopReason?: WorkflowStopReason }, - entries: readonly JournalEntry[], + canonical: RetainedTerminal, ): boolean { - const canonical = rootOutcome(entries); - if (canonical === undefined) { - return false; - } - if (record.status === canonical.status) { - return sameReason(record.stopReason, canonical.reason); - } - // The other authority. It differs from recovery in exactly one place — a - // document that returned a failure — and it is the executor that watched it - // do so, so its reason is one it chose rather than one this can derive. - if (record.status !== "failed" || canonical.status !== "completed") { - return false; - } - return documentFailed(entries) && retainedReason(record.stopReason, entries); + return ( + canonical.kind === "outcome" && + record.status === canonical.status && + sameReason(record.stopReason, canonical.reason) + ); } /** What closing a dead executor's execution makes of it, and of the run. */ @@ -159,21 +256,28 @@ export interface Closing { * A replay whose terminal state was preserved closes only its own execution: * the authoritative outcome stays exactly as it was. Otherwise a retained root * Close proves the canonical outcome won, and without one the executor was - * interrupted. + * interrupted. A root Close this build cannot read is neither: it proves the + * document finished, so the run is not made interrupted, and it says nothing + * this can publish. */ export function closingOutcome( storedStatus: WorkflowRunStatus, - canonical: - | { status: WorkflowRunStatus; reason: DocumentExecutionCompletion["reason"] } - | undefined, + canonical: RetainedTerminal | undefined, ): Closing { if (terminal(storedStatus)) { return { status: "interrupted", reason: INTERRUPTED, publishes: false }; } - if (canonical !== undefined) { - return { status: canonical.status, reason: canonical.reason, publishes: true }; + if (canonical === undefined) { + return { status: "interrupted", reason: INTERRUPTED, publishes: true }; + } + if (canonical.kind === "damaged") { + // The document finished and this build cannot read what it finished as. + // Calling that an interruption would say the executor went without saying + // anything, which is the one thing this history rules out — so the dead + // executor's execution closes and the run is left exactly as it is. + return { status: "interrupted", reason: INTERRUPTED, publishes: false }; } - return { status: "interrupted", reason: INTERRUPTED, publishes: true }; + return { status: canonical.status, reason: canonical.reason, publishes: true }; } /** diff --git a/packages/workflow/src/replay.ts b/packages/workflow/src/replay.ts index 3ecb7c525..deaabb877 100644 --- a/packages/workflow/src/replay.ts +++ b/packages/workflow/src/replay.ts @@ -39,7 +39,7 @@ import { retainedSource } from "@executablemd/core/host"; import type { ExecutionInstallation, RetainedRootDocument } from "@executablemd/core/host"; import { workflowBundleReplayInstallation } from "./bundle.ts"; import { retainedWorkflowInstallation } from "./run.ts"; -import { agreesWithRetainedResult, terminal } from "./lifecycle/policy.ts"; +import { agreesWithRetainedResult, rootOutcome, terminal } from "./lifecycle/policy.ts"; import type { JournalEntry } from "./storage/api.ts"; import type { WorkflowRunRecord } from "./storage/record.ts"; @@ -90,6 +90,9 @@ const REFUSALS = Object.freeze({ disagreed: "this run's retained state and its recorded document result describe different outcomes, " + "so neither is the one to replay. The run is left exactly as it is.", + damaged: + "this run's recorded document result cannot be read by this version, so there is no " + + "outcome to restore. The run is left exactly as it is.", }); function refuse(reason: string): Result { @@ -263,13 +266,21 @@ export function retainedReplay( return refuse(REFUSALS.mixed); } - // The document result the journal records, and the lifecycle row that stands - // behind it, have to be the same outcome. The judgment is the lifecycle's own - // — `agreesWithRetainedResult()` sits beside the recovery and settlement - // mappings it is derived from — rather than a second copy of it here that - // could drift. A row and a result claiming different terminals are damaged - // retained state, and neither of them is the one to replay. - if (!agreesWithRetainedResult(record, entries)) { + // What the root recorded, read once, by the lifecycle's own judgment: the + // same one stale recovery publishes through and the same one a settlement + // is held to. A second copy of it here would be a second authority. + // + // A terminal this build cannot read is refused before canonical core is + // handed it, because core's rejection would arrive as *this* invocation's + // document failure and be offered as a replacement outcome. + const canonical = rootOutcome(entries); + if (canonical === undefined) { + return refuse(REFUSALS.absent); + } + if (canonical.kind === "damaged") { + return refuse(REFUSALS.damaged); + } + if (!agreesWithRetainedResult(record, canonical)) { return refuse(REFUSALS.disagreed); } diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts index ad5db74ae..0631ee33d 100644 --- a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -44,6 +44,7 @@ import type { WorkflowRunCreation, } from "../../src/lifecycle/execution.ts"; import { retainedReplay } from "../../src/replay.ts"; +import { DOCUMENT_FAILED } from "../../src/lifecycle/policy.ts"; import { retainedWorkflowInstallation } from "../../src/run.ts"; import { workflowBundleInstallation } from "../../src/bundle.ts"; import { gitBlobId } from "../../src/git-blob.ts"; @@ -52,6 +53,8 @@ let unique = 0; const NOW = 1_800_000_000; const COMMIT = "0".repeat(40); const DOCUMENT = "# Remote\n\nthe owner recorded this line.\n"; +/** A document that fails: the name resolves to nothing, and nothing is searched. */ +const FAILING = "# Remote\n\npartial line.\n\n\n"; let keys: TestKeys; beforeAll(async () => { @@ -518,6 +521,90 @@ describe("a completed run replayed through its own owner", () => { expect(await on(stub, (owner) => owner.holders())).toBe(0); }); + it("recovers a run whose committed document result is a failure to failed", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + // The document failed and the connection went before anything settled. The + // coroutine returned, so its own settlement is `ok`; what it returned says + // the document failed, and that is what the run is. + const live = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + return yield* canonical(begun.value.database, retainedSource("README.md", FAILING), [ + runContract(), + ]); + }); + }); + + expect(live.result.ok).toBe(false); + expect(live.output).toContain("partial line."); + const before = await ownerState(stub); + expect(before.run?.["status"]).toBe("running"); + + const replayed = await run(function* (): Operation { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + // A resume is what the settled lifecycle refuses for a failed run, so + // the run is named again by the compatible start it was created with. + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + // The owner's own recovery read the document's result, not the + // coroutine's settlement. + expect(begun.value.record.status).toBe("failed"); + expect(begun.value.replay).toBe(true); + const frontier = yield* begun.value.database.readJournalEntries(); + if (!frontier.ok) { + throw frontier.error; + } + const prepared = retainedReplay(begun.value.record, frontier.value); + if (!prepared.ok) { + throw prepared.error; + } + const rendered = yield* canonical(begun.value.database, prepared.value.root, [ + ...prepared.value.installations, + ]); + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status: "failed", + reason: { kind: "host", code: DOCUMENT_FAILED }, + }); + if (!settled.ok) { + throw settled.error; + } + return rendered; + }); + }); + + // The same failure and the same partial output, from history alone. + expect(replayed.result.ok).toBe(false); + expect(replayed.output).toBe(live.output); + + const recovered = await ownerState(stub); + expect(recovered.run?.["status"]).toBe("failed"); + expect(recovered.journal).toEqual(before.journal); + expect(recovered.currentRootId).toBe(before.currentRootId); + expect(recovered.published).toEqual(before.published); + expect(recovered.answers).toEqual(before.answers); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + it("refuses a lifecycle row its retained result contradicts, and moves nothing", async () => { const stub = executor(); const host = lifecycleHost(stub); diff --git a/packages/workflow/tests/replay-inputs.test.ts b/packages/workflow/tests/replay-inputs.test.ts index a73d1dd17..ca863eac5 100644 --- a/packages/workflow/tests/replay-inputs.test.ts +++ b/packages/workflow/tests/replay-inputs.test.ts @@ -20,6 +20,7 @@ import type { Operation, Result } from "effection"; import type { DurableEvent, Json } from "@executablemd/durable-streams"; import type { ExecutionInstallation } from "@executablemd/core/host"; import { retainedReplay } from "../src/replay.ts"; +import { DOCUMENT_FAILED } from "../src/lifecycle/policy.ts"; import type { RetainedReplay } from "../src/replay.ts"; import { WorkflowReplayHistoryError } from "../src/replay.ts"; import { WorkflowBundleHistoryError } from "../src/bundle.ts"; @@ -97,6 +98,12 @@ function rootClose(value: Json): DurableEvent { return { type: "close", coroutineId: "root", result: { status: "ok", value } }; } +/** A document that decided it failed, as canonical core records one. */ +function documentFailure(output = ""): Json { + const message = "the document refused"; + return { status: "err", output, error: { name: "Error", message, segment: { message } } }; +} + /** The terminal core writes when it fails before importing a root document. */ function preRootClose(path: string, source: string, target: string | null): DurableEvent { const message = "refused before the root import"; @@ -136,6 +143,9 @@ function admitted(outcome: Result): RetainedReplay { return outcome.value; } +/** The reason a failure no retained row identifies is named by. */ +const HOST: WorkflowStopReason = { kind: "host", code: DOCUMENT_FAILED }; + /** The run record canonical execution retains before it imports anything. */ const RUN_RECORD = forkRunRecordEvent({ runId: "replay-1", base: "main", pinnedCommit: COMMIT }); @@ -192,10 +202,12 @@ describe("the root a completed run replays on", () => { failure: { kind: "no-match", selector: "Missing*", matches: [], available: ["Retained"] }, }), ), - entry(rootClose({ status: "err", output: "", error: { name: "E", message: "m" } })), + entry(rootClose(documentFailure())), ]; - expect(admitted(retainedReplay(record(), history)).root).toEqual({ + expect( + admitted(retainedReplay(record({ status: "failed", stopReason: HOST }), history)).root, + ).toEqual({ path: "flows/root.md", source: SOURCE, retained: true, @@ -207,7 +219,11 @@ describe("the root a completed run replays on", () => { it("comes from the terminal when the run failed before importing anything", function* () { const history = [entry(preRootClose("flows/root.md", SOURCE, null))]; - expect(admitted(retainedReplay(record(), history)).root).toEqual({ + // The document failed, so the run failed, and no retained row says where — + // which is exactly what the one categorical code is for. + expect( + admitted(retainedReplay(record({ status: "failed", stopReason: HOST }), history)).root, + ).toEqual({ path: "flows/root.md", source: SOURCE, retained: true, @@ -222,7 +238,7 @@ describe("the root a completed run replays on", () => { ); const bound = [entry(preRootClose("flows/other.md", SOURCE, null))]; - expect(reason(retainedReplay(record(), bound))).toContain( + expect(reason(retainedReplay(record({ status: "failed", stopReason: HOST }), bound))).toContain( "not the document its definition names", ); }); @@ -440,6 +456,144 @@ describe("the lifecycle row and the recorded result have to agree", () => { }); }); +/** One retained effect, settled the way its own operation settled. */ +function effect(name: string, settled: "ok" | "err"): DurableEvent { + return { + type: "yield", + coroutineId: "root", + description: { type: "call", name }, + result: + settled === "ok" + ? { status: "ok", value: name } + : { status: "err", error: { message: `${name} failed`, name: "Error" } }, + }; +} + +describe("what the root recorded, as one outcome", () => { + // deno-lint-ignore require-yield + it("reads the document's own result, not the coroutine's settlement", function* () { + // The coroutine returned, so its settlement is `ok`. What it returned is a + // document that failed, and that is what the run is. + const failing = [ + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + entry(rootClose(documentFailure("partial\n"))), + ]; + expect( + admitted(retainedReplay(record({ status: "failed", stopReason: HOST }), failing)).root, + ).toEqual({ path: "flows/root.md", source: SOURCE, retained: true }); + // And a completed row over the same history is the disagreement. + expect(reason(retainedReplay(record(), failing))).toContain("describe different outcomes"); + }); + + // deno-lint-ignore require-yield + it("names the exact retained row the failure stopped at", function* () { + const rows = [ + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + entry(effect("early", "err")), + entry(effect("between", "ok")), + entry(effect("late", "err")), + entry(rootClose(documentFailure("partial\n"))), + ]; + const late = rows[3]?.eventId ?? ""; + const early = rows[1]?.eventId ?? ""; + const between = rows[2]?.eventId ?? ""; + + // The last row that failed, and only that one. + expect( + admitted( + retainedReplay( + record({ status: "failed", stopReason: { kind: "journal", eventId: late } }), + rows, + ), + ).root.path, + ).toBe("flows/root.md"); + + const wrong: WorkflowStopReason[] = [ + { kind: "journal", eventId: early }, + { kind: "journal", eventId: between }, + { kind: "journal", eventId: "event-somewhere-else" }, + HOST, + { kind: "host", code: "invented-code" }, + ]; + for (const stopReason of wrong) { + const outcome = retainedReplay(record({ status: "failed", stopReason }), rows); + expect([ + JSON.stringify(stopReason), + reason(outcome).includes("describe different outcomes"), + ]).toEqual([JSON.stringify(stopReason), true]); + } + // And a failure that names nothing at all. + expect(reason(retainedReplay(record({ status: "failed" }), rows))).toContain( + "describe different outcomes", + ); + }); + + // deno-lint-ignore require-yield + it("refuses a document result this version cannot read", function* () { + const message = "the document refused"; + const malformed: Json[] = [ + { status: "err" }, + { status: "err", output: "" }, + { status: "err", output: "", error: { name: "Error", message } }, + { status: "err", output: "", error: { name: "Error", message, segment: {} } }, + { status: "err", output: 7, error: { name: "Error", message, segment: { message } } }, + { status: "ok", output: "" }, + { status: "ok", output: "", value: "", extra: 1 }, + { status: "abandoned", output: "", value: "" }, + "not an object at all", + ]; + + for (const value of malformed) { + const history = [ + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + entry(rootClose(value)), + ]; + const said = reason(retainedReplay(record(), history)); + expect([JSON.stringify(value), said.includes("cannot be read by this version")]).toEqual([ + JSON.stringify(value), + true, + ]); + // And it says nothing about what it read. + expect(said).not.toContain("abandoned"); + } + }); + + // deno-lint-ignore require-yield + it("admits the shapes canonical execution actually writes", function* () { + const message = "the document refused"; + const written: { value: Json; status: WorkflowRunStatus }[] = [ + { value: { status: "ok", output: "done\n", value: "done\n" }, status: "completed" }, + { value: { status: "ok", output: "", value: null }, status: "completed" }, + { + value: { + status: "err", + output: "partial\n", + error: { + name: "Error", + message, + segment: { message, source: "flows/root.md" }, + cause: "because", + errors: [{ name: "Error", message }], + }, + }, + status: "failed", + }, + ]; + + for (const { value, status } of written) { + const history = [ + entry(rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE })), + entry(rootClose(value)), + ]; + const outcome = retainedReplay( + record({ status, ...(status === "failed" ? { stopReason: HOST } : {}) }), + history, + ); + expect([JSON.stringify(value), outcome.ok]).toEqual([JSON.stringify(value), true]); + } + }); +}); + /** * The object ids `git hash-object -t blob` gives these exact bytes. * diff --git a/packages/workflow/tests/workflow-lifecycle-control.test.ts b/packages/workflow/tests/workflow-lifecycle-control.test.ts index a20d634fe..2bf03a0f6 100644 --- a/packages/workflow/tests/workflow-lifecycle-control.test.ts +++ b/packages/workflow/tests/workflow-lifecycle-control.test.ts @@ -174,10 +174,16 @@ describe("Tier WLC — cancellation and deletion", () => { transitions, { runId: "closed-1", action: "start", creation: creation() }, function* (begun) { + // The coroutine returned, and what it returned is the document's own + // result: the shape canonical core writes, because that shape is what + // says whether the document completed or failed. yield* begun.database.journal.append({ type: "close", coroutineId: "root", - result: { status: "ok", value: "rendered" }, + result: { + status: "ok", + value: { status: "ok", output: "rendered", value: "rendered" }, + }, }); }, ); From e98e5cd922ee5a8af1ba56d13dc596e669039e7a Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 9 Sep 2026 16:00:54 -0300 Subject: [PATCH 66/81] =?UTF-8?q?=F0=9F=90=9B=20Carry=20an=20unreadable=20?= =?UTF-8?q?terminal=20through=20recovery,=20and=20refuse=20every=20action?= =?UTF-8?q?=20over=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A retained root Close proves the document ended. When its form is not one canonical core writes, this build has no authority to say what it ended as — and the previous correction, having noticed that, then threw the fact away. `closingOutcome()` encoded damage as "interrupted, publishes nothing", both adapters read that as the stored `running` status, and their begins went on to insert another execution, publish `running` and hand the document back to a live invocation. The remote cancel lost it the same way and published `cancelled` over history it could not read, while the local one had grown a check of its own — two providers deciding one lifecycle differently. Damage is now a decision of its own. `Closing` carries it, both reconciliations return it without writing anything at all, and both begins and both cancels refuse with one shared sentence that repeats nothing the history held. The settled behaviour for the stale envelope is the same on both: it is not closed either, because what the execution became is exactly as unreadable as what the run became. The terminal judgment also got stricter in the two places it was loose. A `root_binding` is not decoration a failure may carry: core writes one in exactly one situation, and the whole pre-root form — the empty output, the segment that repeats the failure's own message, the closed path/source/target — is what makes an import-free history attributable to one document. Anything else wearing a binding is a terminal core did not write. And the terminal has to be the frontier: `terminalFrontier()` requires exactly one root Close, last, and both stale recovery and replay admission ask it, so recovery can no longer publish an outcome from the first of two results that replay would refuse whole. --- packages/cli/tests/workflow-replay.test.ts | 217 ++++++++++++++++-- .../workflow/src/cloudflare/lifecycle-link.ts | 26 ++- .../src/cloudflare/owner-lifecycle.ts | 25 +- packages/workflow/src/deno/transitions.ts | 31 ++- packages/workflow/src/lifecycle/policy.ts | 181 ++++++++++++--- .../workflow/src/remote/lifecycle-link.ts | 14 +- packages/workflow/src/remote/lifecycle.ts | 13 +- packages/workflow/src/replay.ts | 34 ++- .../tests/cloudflare/remote-replay.vitest.ts | 86 +++++++ packages/workflow/tests/replay-inputs.test.ts | 63 +++++ 10 files changed, 591 insertions(+), 99 deletions(-) diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts index 32479ae87..db4719e1f 100644 --- a/packages/cli/tests/workflow-replay.test.ts +++ b/packages/cli/tests/workflow-replay.test.ts @@ -1176,6 +1176,151 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(outcome.after.status).toBe(outcome.before.status); }); + it("WRP15: refuses every action over a terminal it cannot read, and moves nothing", function* () { + const asked: string[] = []; + const attached: string[] = []; + let executed = 0; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + const established = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + + // A run left `running`, with an execution nobody closed, over a root + // result this build cannot read. + const runId = yield* seedStaleRun(fixture, established, damagedHistory(established)); + const before = yield* retained(fixture.runs, runId); + expect(before.status).toBe("running"); + expect(before.ended).toEqual([null]); + + const body = () => + // deno-lint-ignore require-yield + function* (): Operation> { + executed += 1; + return Ok(undefined); + }; + + const outcomes = yield* scoped(function* () { + yield* useRefusingGit(asked); + const started = yield* invoke( + { ...REQUEST, id: runId }, + established, + replayHost(fixture.runs, attached), + body(), + ); + const resumed = yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + body(), + ); + const cancelled = yield* scoped(function* () { + const err: string[] = []; + const error = console.error; + yield* ensure(() => { + console.error = error; + }); + console.error = (...parts: unknown[]) => err.push(parts.map(String).join(" ")); + const managed = yield* runWorkflowManagement( + { action: "cancel", runId }, + replayHost(fixture.runs, attached), + ); + return { exitCode: managed.exitCode, out: [], err }; + }); + return { started, resumed, cancelled }; + }); + return { ...outcomes, before, after: yield* retained(fixture.runs, runId) }; + }); + + // Every action refuses, with the one sentence and nothing the history held. + for (const [name, invocation] of Object.entries(outcome)) { + if (name === "before" || name === "after") { + continue; + } + const said = "err" in invocation ? invocation.err.join(" ") : ""; + expect([name, "exitCode" in invocation ? invocation.exitCode : 0]).toEqual([name, 1]); + expect([name, said.includes("cannot read")]).toEqual([name, true]); + expect([name, said.includes("status:")]).toEqual([name, false]); + } + + // No live authority was constructed and no document ran. + expect(executed).toBe(0); + expect(asked).toEqual([]); + expect(attached).toEqual([]); + + // Nothing at all changed: not the run row, not the journal, not the + // Workspace root, and not the execution the previous executor left open. + expect(outcome.after).toEqual(outcome.before); + }); + + it("WRP16: refuses a history holding a second result, without choosing one", function* () { + const asked: string[] = []; + const attached: string[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(PLAIN); + const established = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + + // Two results, and a row recorded after the first of them. No single + // execution produced this, so neither result is the run's. + const runId = yield* seedStaleRun(fixture, established, (definition, id) => { + const events: DurableEvent[] = [ + forkRunRecordEvent({ + runId: id, + base: established.established.base, + pinnedCommit: definition.objectId, + }), + rootImportEvent(definition.rootDocumentPath, established.established.source), + { + type: "close", + coroutineId: "root", + result: { status: "ok", value: { status: "ok", output: "first\n", value: "first\n" } }, + }, + { + type: "close", + coroutineId: "root", + result: { + status: "ok", + value: { status: "ok", output: "second\n", value: "second\n" }, + }, + }, + ]; + return events; + }); + const before = yield* retained(fixture.runs, runId); + + const refused = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + established, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + return Ok(undefined); + }, + ); + }); + return { refused, before, after: yield* retained(fixture.runs, runId) }; + }); + + expect(outcome.refused.exitCode).toBe(1); + expect(outcome.refused.err.join(" ")).toContain("cannot read"); + expect(statusOf(outcome.refused)).toBeUndefined(); + // Neither result was chosen, and nothing was published from either. + expect(outcome.after).toEqual(outcome.before); + expect(asked).toEqual([]); + expect(attached).toEqual([]); + }); + it("WRP5: a run that has not ended still reconstructs, and its refusal is cleaned up", function* () { const asked: string[] = []; const attached: string[] = []; @@ -1477,35 +1622,59 @@ function* seedStaleRun( }); } +/** + * A history whose root recorded a result this build cannot read. + * + * The `Close` says the document ended; what it says it ended as is not the + * closed form canonical core writes, so nothing here can name the outcome. + */ +function damagedHistory( + start: WorkflowStart, +): (definition: WorkflowDefinition, runId: string) => readonly DurableEvent[] { + return (definition, runId) => { + const events: DurableEvent[] = [ + forkRunRecordEvent({ + runId, + base: start.established.base, + pinnedCommit: definition.objectId, + }), + rootImportEvent(definition.rootDocumentPath, start.established.source), + { type: "close", coroutineId: "root", result: { status: "ok", value: { status: "err" } } }, + ]; + return events; + }; +} + +/** The root import canonical execution records before anything else. */ +function rootImportEvent(path: string, content: string): DurableEvent { + return { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path, content } }, + }; +} + /** The history a run that raised out of its root leaves behind. */ function raisedHistory( start: WorkflowStart, ): (definition: WorkflowDefinition, runId: string) => readonly DurableEvent[] { - return (definition, runId) => [ - forkRunRecordEvent({ - runId, - base: start.established.base, - pinnedCommit: definition.objectId, - }), - { - type: "yield", - coroutineId: "root", - description: { type: "import_component", name: "__root__" }, - result: { - status: "ok", - value: { - kind: "repository", - path: definition.rootDocumentPath, - content: start.established.source, - }, + return (definition, runId) => { + const events: DurableEvent[] = [ + forkRunRecordEvent({ + runId, + base: start.established.base, + pinnedCommit: definition.objectId, + }), + rootImportEvent(definition.rootDocumentPath, start.established.source), + { + type: "close", + coroutineId: "root", + result: { status: "err", error: { message: "the executor died", name: "Error" } }, }, - }, - { - type: "close", - coroutineId: "root", - result: { status: "err", error: { message: "the executor died", name: "Error" } }, - }, - ]; + ]; + return events; + }; } /** diff --git a/packages/workflow/src/cloudflare/lifecycle-link.ts b/packages/workflow/src/cloudflare/lifecycle-link.ts index 9067d16de..052262e16 100644 --- a/packages/workflow/src/cloudflare/lifecycle-link.ts +++ b/packages/workflow/src/cloudflare/lifecycle-link.ts @@ -33,12 +33,23 @@ import type { RemoteLifecycleAnswer, RemoteLifecycleLink, } from "../remote/lifecycle-link.ts"; +import type { RemoteLifecycleRefusal } from "../remote/lifecycle-link.ts"; import { WorkflowRecordMalformedError, WorkflowRunConflictError } from "../storage/errors.ts"; import { parseRemoteExecution } from "../remote/records.ts"; import type { AnchoringReadLink, OwnerConnection } from "./client.ts"; import { privateRefusal, storageFailure, translate } from "./client.ts"; -const REFUSALS = new Set(["cancelled", "resume-failed", "terminal"]); +/** The conditions an owner may name, and the only ones this build reads. */ +const REFUSALS: readonly RemoteLifecycleRefusal[] = [ + "cancelled", + "resume-failed", + "terminal", + "damaged-terminal", +]; + +function refusalOf(value: unknown): RemoteLifecycleRefusal | undefined { + return REFUSALS.find((refusal) => refusal === value); +} function fail(reason: string): never { throw new WorkflowRecordMalformedError("lifecycle answer this run's owner returned", reason); @@ -79,13 +90,11 @@ function answered( ); } if (refusal !== null) { - if (typeof refusal !== "string" || !REFUSALS.has(refusal)) { - return fail("it named no condition this build reads"); - } - if (refusal !== "cancelled" && refusal !== "resume-failed" && refusal !== "terminal") { + const named = refusalOf(refusal); + if (named === undefined) { return fail("it named no condition this build reads"); } - return { kind: "refused", refusal }; + return { kind: "refused", refusal: named }; } return { kind: "performed", value: read(held) }; } @@ -306,7 +315,8 @@ export function cloudflareLifecycleLink( const found = members(value, ["conflict", "refusal", "value"]); const refusal = found.get("refusal"); if (refusal !== null) { - if (refusal !== "cancelled" && refusal !== "resume-failed" && refusal !== "terminal") { + const named = refusalOf(refusal); + if (named === undefined) { return Err( new WorkflowRecordMalformedError( "lifecycle answer this run's owner returned", @@ -314,7 +324,7 @@ export function cloudflareLifecycleLink( ), ); } - return Ok({ kind: "refused", refusal }); + return Ok({ kind: "refused", refusal: named }); } const conflict = found.get("conflict"); if (conflict !== null) { diff --git a/packages/workflow/src/cloudflare/owner-lifecycle.ts b/packages/workflow/src/cloudflare/owner-lifecycle.ts index 82e18985e..89b000697 100644 --- a/packages/workflow/src/cloudflare/owner-lifecycle.ts +++ b/packages/workflow/src/cloudflare/owner-lifecycle.ts @@ -62,7 +62,9 @@ export type LifecycleRefusal = /** Resume or start reached a run that was cancelled. */ | "cancelled" /** Cancellation reached a run whose outcome already won. */ - | "terminal"; + | "terminal" + /** The run's own root recorded a result this build cannot read. */ + | "damaged-terminal"; /** What one begin committed, as the runner is allowed to know it. */ export interface BegunValue { @@ -204,12 +206,22 @@ function reconcile( storage: OwnerStorage, stored: WorkflowRunRecord, now: () => string, -): { status: WorkflowRunStatus; recovered: DocumentExecutionRecord | null } { +): { + status: WorkflowRunStatus; + recovered: DocumentExecutionRecord | null; + damaged?: boolean; +} { + const closing = closingOutcome(stored.status, rootOutcome(journalEntries(storage))); + if (closing.damaged) { + // Nothing is decided here and nothing is written: whatever the previous + // executor left stays exactly as it left it, because this build cannot say + // what the document it ran did. + return { status: stored.status, recovered: null, damaged: true }; + } const open = unfinished(storage); if (open.length === 0) { return { status: stored.status, recovered: null }; } - const closing = closingOutcome(stored.status, rootOutcome(journalEntries(storage))); let last: DocumentExecutionRecord | null = null; for (const execution of open) { finish( @@ -274,6 +286,9 @@ export function beginRun( } const recovered = reconcile(storage, stored, now); + if (recovered.damaged === true) { + return { conflict: null, refusal: "damaged-terminal", value: null }; + } // The recovery above stays committed whatever this decides: what the // previous executor's execution became is not undone by this caller being // told it may not continue. @@ -390,6 +405,10 @@ export function cancelRunOnOwner( return { conflict: null, refusal: "terminal", value: null }; } const recovered = reconcile(storage, stored, now); + if (recovered.damaged === true) { + // Cancelling would replace a result rather than end a run that had none. + return { conflict: null, refusal: "damaged-terminal", value: null }; + } if (terminal(recovered.status) || recovered.status === "cancelled") { // The document finished before its executor disappeared. Restoring what // it recorded is not cancelling it. diff --git a/packages/workflow/src/deno/transitions.ts b/packages/workflow/src/deno/transitions.ts index 8642cada9..6d093ea0f 100644 --- a/packages/workflow/src/deno/transitions.ts +++ b/packages/workflow/src/deno/transitions.ts @@ -40,6 +40,7 @@ import { admissionRefusal, type Closing, closingOutcome, + damagedTerminalRefusal, INTERRUPTED, rootOutcome, terminal, @@ -190,6 +191,14 @@ interface Recovery { /** Absent when there is no run yet, which only a `start` may go on from. */ readonly status?: WorkflowRunStatus; readonly closed?: DocumentExecutionRecord; + /** + * Whether this run's own terminal is one this build cannot read. + * + * Carried out of recovery rather than collapsed into the stored status: an + * unreadable terminal is not a run to go on with, and a caller that saw only + * `running` would begin another execution over it. + */ + readonly damaged?: boolean; } interface Refused { @@ -248,6 +257,10 @@ function beginOnce( const recovery = recover(connection, path, hold, request); + if (recovery.damaged === true) { + return { kind: "refused", reason: damagedTerminalRefusal() }; + } + // A file can exist and hold nothing — created by an interrupted attempt, or // left empty by something else. Existence is not a run, so a resume that // reaches one refuses here rather than letting the creation it happens to @@ -633,13 +646,19 @@ interface Reconciled { * addressed to; failing both, the execution was interrupted. */ function reconcile(database: DatabaseSync, path: string, stored: WorkflowRunRecord): Recovery { + const closing = closingOutcome(stored.status, rootOutcome(readJournalEntries(database))); + if (closing.damaged) { + // Nothing is decided here and nothing is written: whatever the previous + // executor left stays exactly as it left it, because this build cannot say + // what the document it ran did. + return { status: stored.status, damaged: true }; + } + const unfinished = reading(database, SELECT_UNFINISHED).all().map(readDocumentExecution); if (unfinished.length === 0) { return { status: stored.status }; } - const closing = closingOutcome(stored.status, rootOutcome(readJournalEntries(database))); - let last: DocumentExecutionRecord | undefined; for (const execution of unfinished) { finish(database, path, { @@ -847,13 +866,7 @@ export function* cancelRun( // The document finished and this build cannot read what it finished // as. Cancelling it would replace a result rather than end a run that // had none, so nothing here changes anything. - return { - kind: "refused" as const, - reason: new WorkflowRequestError( - "workflow run: its root recorded an outcome this version cannot read, so the run " + - "is neither cancelled nor changed.", - ), - }; + return { kind: "refused" as const, reason: damagedTerminalRefusal() }; } if (canonical !== undefined) { // The document finished before its workflow executor disappeared. Restoring what it diff --git a/packages/workflow/src/lifecycle/policy.ts b/packages/workflow/src/lifecycle/policy.ts index e35495325..96cf25483 100644 --- a/packages/workflow/src/lifecycle/policy.ts +++ b/packages/workflow/src/lifecycle/policy.ts @@ -88,27 +88,62 @@ export type RetainedTerminal = | { readonly kind: "damaged" }; export function rootOutcome(entries: readonly JournalEntry[]): RetainedTerminal | undefined { - for (const entry of entries) { - const event: DurableEvent = entry.event; - if (event.type !== "close" || event.coroutineId !== "root") { - continue; - } - if (event.result.status !== "ok") { - return { - kind: "outcome", - status: event.result.status === "cancelled" ? "cancelled" : "failed", - reason: { kind: "journal", eventId: entry.eventId }, - }; - } - const document = readDocumentResult(event.result.value); - if (document === undefined) { - return { kind: "damaged" }; - } - return document === "ok" - ? { kind: "outcome", status: "completed", reason: undefined } - : { kind: "outcome", status: "failed", reason: retainedFailureReason(entries) }; + const frontier = terminalFrontier(entries); + if (frontier.kind === "absent") { + return undefined; } - return undefined; + if (frontier.kind === "mixed") { + // Two results, or work recorded after the one result: a history no single + // execution produced. Choosing one of them would be this build deciding + // which execution the run was. + return { kind: "damaged" }; + } + const event: DurableEvent = frontier.entry.event; + if (event.type !== "close") { + return { kind: "damaged" }; + } + if (event.result.status !== "ok") { + return { + kind: "outcome", + status: event.result.status === "cancelled" ? "cancelled" : "failed", + reason: { kind: "journal", eventId: frontier.entry.eventId }, + }; + } + const document = readDocumentResult(event.result.value); + if (document === undefined) { + return { kind: "damaged" }; + } + return document === "ok" + ? { kind: "outcome", status: "completed", reason: undefined } + : { kind: "outcome", status: "failed", reason: retainedFailureReason(entries) }; +} + +/** + * Where the root's terminal sits in a history, when it sits anywhere. + * + * One execution records one result, and records it last. A history holding two, + * or holding anything after the one it stands behind, is not one execution's — + * and reading it as though the first of them were authoritative is how a + * lifecycle row comes to be published from history nobody can account for. Both + * the recovery that publishes an outcome and the admission that reuses one ask + * this, so neither can decide the question the other way. + */ +export type TerminalFrontier = + | { readonly kind: "absent" } + | { readonly kind: "mixed" } + | { readonly kind: "final"; readonly entry: JournalEntry }; + +export function terminalFrontier(entries: readonly JournalEntry[]): TerminalFrontier { + const closes = entries.filter( + (entry) => entry.event.type === "close" && entry.event.coroutineId === "root", + ); + const only = closes[0]; + if (only === undefined) { + return { kind: "absent" }; + } + return closes.length === 1 && entries[entries.length - 1] === only + ? { kind: "final", entry: only } + : { kind: "mixed" }; } /** @@ -134,15 +169,72 @@ function readDocumentResult(value: unknown): "ok" | "err" | undefined { if (result["status"] !== "err") { return undefined; } - if (!names(result, ["status", "output", "error", ROOT_BINDING])) { - return undefined; + if (ROOT_BINDING in result) { + // A binding is not decoration a failure may carry. Core writes one in + // exactly one situation — it failed before importing anything — and the + // whole form is what makes that import-free history attributable to one + // document at all. Anything else wearing a binding is a terminal core did + // not write. + return readPreRootTerminal(result) ? "err" : undefined; } - return readDocumentFailure(result["error"]) ? "err" : undefined; + return names(result, ["status", "output", "error"]) && readDocumentFailure(result["error"]) + ? "err" + : undefined; } /** Where core records which document a terminal it wrote before importing was about. */ const ROOT_BINDING = "root_binding"; +/** + * Whether a failure carrying a binding is the exact terminal core writes before + * it has imported anything. + * + * The form is `recordedPreRootTerminal()`'s in `packages/core/src/execute.ts`, + * and every part of it is load-bearing: nothing was rendered, so the output is + * empty; no segment failed, so the description repeats the failure's own + * message and says nothing else; and the binding names the document — its path, + * the supplied text for an inline root and nothing for a file one, and the + * selector as written — which is the only thing making a history with no root + * import about one document rather than any. + */ +function readPreRootTerminal(result: Record): boolean { + if (!names(result, ["status", "output", "error", ROOT_BINDING]) || result["output"] !== "") { + return false; + } + const failure = plain(result["error"]); + if ( + failure === undefined || + !names(failure, ["name", "message", "segment", "cause"]) || + typeof failure["name"] !== "string" || + typeof failure["message"] !== "string" || + ("cause" in failure && typeof failure["cause"] !== "string") + ) { + return false; + } + const segment = plain(failure["segment"]); + if ( + segment === undefined || + !names(segment, ["message"]) || + segment["message"] !== failure["message"] + ) { + return false; + } + const binding = plain(result[ROOT_BINDING]); + if (binding === undefined || !names(binding, ["path", "source", "target"])) { + return false; + } + const path = binding["path"]; + const source = binding["source"]; + const target = binding["target"]; + return ( + typeof path === "string" && + (source === null || typeof source === "string") && + (target === null || typeof target === "string") && + "source" in binding && + "target" in binding + ); +} + /** Whether a described failure is the closed form core writes. */ function readDocumentFailure(value: unknown): boolean { const failure = plain(value); @@ -248,6 +340,16 @@ export interface Closing { readonly reason: DocumentExecutionCompletion["reason"]; /** Whether the run's own status follows the execution's, or stays as it is. */ readonly publishes: boolean; + /** + * Whether this run's own terminal is one this build cannot read. + * + * Distinct from "nothing to publish", and the distinction is the whole point: + * a run with nothing to publish is one to go on with, and this is a run whose + * document already ended in a way nothing here can account for. Losing it + * between recovery and the caller is how an unreadable terminal came to + * authorize a second execution. + */ + readonly damaged: boolean; } /** @@ -256,28 +358,43 @@ export interface Closing { * A replay whose terminal state was preserved closes only its own execution: * the authoritative outcome stays exactly as it was. Otherwise a retained root * Close proves the canonical outcome won, and without one the executor was - * interrupted. A root Close this build cannot read is neither: it proves the - * document finished, so the run is not made interrupted, and it says nothing - * this can publish. + * interrupted. A root Close this build cannot read is none of those: it proves + * the document finished, so nothing about the run or the execution it left is + * decided here, and the caller is refused instead. */ export function closingOutcome( storedStatus: WorkflowRunStatus, canonical: RetainedTerminal | undefined, ): Closing { if (terminal(storedStatus)) { - return { status: "interrupted", reason: INTERRUPTED, publishes: false }; + return { status: "interrupted", reason: INTERRUPTED, publishes: false, damaged: false }; } if (canonical === undefined) { - return { status: "interrupted", reason: INTERRUPTED, publishes: true }; + return { status: "interrupted", reason: INTERRUPTED, publishes: true, damaged: false }; } if (canonical.kind === "damaged") { // The document finished and this build cannot read what it finished as. // Calling that an interruption would say the executor went without saying - // anything, which is the one thing this history rules out — so the dead - // executor's execution closes and the run is left exactly as it is. - return { status: "interrupted", reason: INTERRUPTED, publishes: false }; + // anything, which is the one thing this history rules out; closing its + // execution would say the same about the execution. So nothing here is + // decided at all, and the caller is told why. + return { status: storedStatus, reason: undefined, publishes: false, damaged: true }; } - return { status: canonical.status, reason: canonical.reason, publishes: true }; + return { status: canonical.status, reason: canonical.reason, publishes: true, damaged: false }; +} + +/** + * What a caller is told when the run it named holds a terminal nothing can read. + * + * One sentence, shared by every provider and every action, and carrying nothing + * the history held: what is unreadable is retained data, and a diagnostic that + * quoted it would publish exactly what it exists to refuse. + */ +export function damagedTerminalRefusal(): WorkflowRequestError { + return new WorkflowRequestError( + "workflow run: its root recorded a document result this version cannot read, so the run " + + "is neither advanced nor changed. The run is left exactly as it is.", + ); } /** diff --git a/packages/workflow/src/remote/lifecycle-link.ts b/packages/workflow/src/remote/lifecycle-link.ts index eaf4b4682..fef595ef2 100644 --- a/packages/workflow/src/remote/lifecycle-link.ts +++ b/packages/workflow/src/remote/lifecycle-link.ts @@ -42,7 +42,19 @@ export interface RemoteBegun { */ export type RemoteLifecycleAnswer = | { readonly kind: "performed"; readonly value: T } - | { readonly kind: "refused"; readonly refusal: "cancelled" | "resume-failed" | "terminal" }; + | { readonly kind: "refused"; readonly refusal: RemoteLifecycleRefusal }; + +/** + * A condition of the run itself, as an owner names one. + * + * Each is a fact a caller acts on rather than a failure to translate, and each + * carries nothing the run holds. + */ +export type RemoteLifecycleRefusal = + | "cancelled" + | "resume-failed" + | "terminal" + | "damaged-terminal"; /** Which committed checkpoint of which run a fork continues. */ export interface RemoteForkOrigin { diff --git a/packages/workflow/src/remote/lifecycle.ts b/packages/workflow/src/remote/lifecycle.ts index a43e51155..9a4cc5754 100644 --- a/packages/workflow/src/remote/lifecycle.ts +++ b/packages/workflow/src/remote/lifecycle.ts @@ -39,6 +39,7 @@ import type { WorkflowForkRequest, WorkflowRunCreation, } from "../lifecycle/execution.ts"; +import { damagedTerminalRefusal } from "../lifecycle/policy.ts"; import type { WorkflowRunDatabase } from "../storage/api.ts"; import type { CreateWorkflowRunRequest } from "../storage/api.ts"; import type { DocumentExecutionCompletion, WorkflowRunRecord } from "../storage/record.ts"; @@ -60,6 +61,7 @@ import type { RemoteForkCommit, RemoteForkPart, RemoteLifecycleLink, + RemoteLifecycleRefusal, } from "./lifecycle-link.ts"; export type { RemoteExecutorConnection }; @@ -292,6 +294,9 @@ export function* useRemoteLifecycle( } answeredNow(runId, addressed); if (answered.value.kind === "refused") { + if (answered.value.refusal === "damaged-terminal") { + return Err(damagedTerminalRefusal()); + } return Err( new WorkflowRequestError( answered.value.refusal === "terminal" @@ -985,10 +990,10 @@ function creationRequest(runId: string, creation: WorkflowRunCreation): CreateWo }; } -function refusalError( - refusal: "cancelled" | "resume-failed" | "terminal", - runId: string, -): WorkflowStorageError { +function refusalError(refusal: RemoteLifecycleRefusal, runId: string): WorkflowStorageError { + if (refusal === "damaged-terminal") { + return damagedTerminalRefusal(); + } if (refusal === "cancelled") { return new WorkflowRequestError(`workflow run ${JSON.stringify(runId)} was cancelled.`); } diff --git a/packages/workflow/src/replay.ts b/packages/workflow/src/replay.ts index deaabb877..28b409bcf 100644 --- a/packages/workflow/src/replay.ts +++ b/packages/workflow/src/replay.ts @@ -39,7 +39,12 @@ import { retainedSource } from "@executablemd/core/host"; import type { ExecutionInstallation, RetainedRootDocument } from "@executablemd/core/host"; import { workflowBundleReplayInstallation } from "./bundle.ts"; import { retainedWorkflowInstallation } from "./run.ts"; -import { agreesWithRetainedResult, rootOutcome, terminal } from "./lifecycle/policy.ts"; +import { + agreesWithRetainedResult, + rootOutcome, + terminal, + terminalFrontier, +} from "./lifecycle/policy.ts"; import type { JournalEntry } from "./storage/api.ts"; import type { WorkflowRunRecord } from "./storage/record.ts"; @@ -147,11 +152,6 @@ function importedName(event: DurableEvent): string | undefined { return typeof name === "string" ? name : undefined; } -/** Whether a retained event is this document execution's own terminal. */ -function isRootClose(event: DurableEvent): boolean { - return reading(() => event.type === "close" && event.coroutineId === "root") === true; -} - /** The document one retained selection names, as a root source is built from it. */ interface RetainedRoot { readonly path: string; @@ -252,19 +252,20 @@ export function retainedReplay( return refuse(REFUSALS.live); } - const closes = entries.filter((entry) => isRootClose(entry.event)); - const close = closes[0]; - if (close === undefined) { - return refuse(REFUSALS.absent); - } // The terminal is the frontier, and there is one of it. A history holding a // second result, or continuing past the one it stands behind, is one whose // lifecycle row and journal describe different moments of the run — and - // reconciling those is not a replay's to do. - const last = entries[entries.length - 1]; - if (closes.length !== 1 || last === undefined || !isRootClose(last.event)) { + // reconciling those is not a replay's to do. The same reading decides it for + // stale recovery, so neither can accept what the other refuses; what this + // adds is only which way it refused. + const frontier = terminalFrontier(entries); + if (frontier.kind === "absent") { + return refuse(REFUSALS.absent); + } + if (frontier.kind === "mixed") { return refuse(REFUSALS.mixed); } + const close = frontier.entry; // What the root recorded, read once, by the lifecycle's own judgment: the // same one stale recovery publishes through and the same one a settlement @@ -274,10 +275,7 @@ export function retainedReplay( // handed it, because core's rejection would arrive as *this* invocation's // document failure and be offered as a replacement outcome. const canonical = rootOutcome(entries); - if (canonical === undefined) { - return refuse(REFUSALS.absent); - } - if (canonical.kind === "damaged") { + if (canonical === undefined || canonical.kind === "damaged") { return refuse(REFUSALS.damaged); } if (!agreesWithRetainedResult(record, canonical)) { diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts index 0631ee33d..537ca4baf 100644 --- a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -694,6 +694,92 @@ describe("a completed run replayed through its own owner", () => { expect(await on(stub, (owner) => owner.holders())).toBe(0); }); + it("refuses every action over a terminal it cannot read, and moves nothing", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + // A run left `running`, with an execution nobody closed, over a root result + // this build cannot read. + await run(function* () { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + const appended = yield* begun.value.database.transact(function* (transaction) { + yield* transaction.journal.append({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "README.md", content: DOCUMENT }, + }, + }); + yield* transaction.journal.append({ + type: "close", + coroutineId: "root", + result: { status: "ok", value: { status: "err" } }, + }); + }); + if (!appended.ok) { + throw appended.error; + } + }); + }); + + const before = await ownerState(stub); + expect(before.run?.["status"]).toBe("running"); + expect(before.executions).toBe(1); + const heldBefore = await on(stub, (owner) => owner.executionRows()); + expect(heldBefore[0]?.["stopped_at"]).toBe(null); + + // Every action the owner offers over this run, refused by the owner itself. + // Cancellation first: it takes an acquisition of its own, so it cannot ask + // while another connection holds the run. + expect(await on(stub, (owner) => owner.holders())).toBe(0); + const cancelled = await run(function* (): Operation { + return yield* scoped(function* () { + yield* useRemoteLifecycle(host); + const outcome = yield* WorkflowLifecycle.operations.cancel(RUN_ID); + return outcome.ok ? "cancelled" : outcome.error.message; + }); + }); + const refusals = await run(function* (): Operation> { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const started = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + const resumed = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + return { + started: started.ok ? "admitted" : started.error.message, + resumed: resumed.ok ? "admitted" : resumed.error.message, + }; + }); + }); + + for (const said of [refusals["started"], refusals["resumed"], cancelled]) { + expect(String(said)).toContain("cannot read"); + } + + // Nothing at all changed: not the run row, not the journal, not the + // Workspace state, and not the execution the previous executor left open. + const after = await ownerState(stub); + expect(after).toEqual(before); + expect(await on(stub, (owner) => owner.executionRows())).toEqual(heldBefore); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + it("hands the run to the next connection when the replay's own ends", async () => { const stub = executor(); const host = lifecycleHost(stub); diff --git a/packages/workflow/tests/replay-inputs.test.ts b/packages/workflow/tests/replay-inputs.test.ts index ca863eac5..f7289a85e 100644 --- a/packages/workflow/tests/replay-inputs.test.ts +++ b/packages/workflow/tests/replay-inputs.test.ts @@ -558,6 +558,69 @@ describe("what the root recorded, as one outcome", () => { } }); + // deno-lint-ignore require-yield + it("recognizes the exact terminal core writes before it imports anything", function* () { + const message = "refused before the root import"; + const binding = { path: "flows/root.md", source: SOURCE, target: null }; + const failure = { name: "Error", message, segment: { message } }; + + // The one form core can produce here, and the run it describes. + const valid = [ + entry(rootClose({ status: "err", output: "", error: failure, root_binding: binding })), + ]; + expect(retainedReplay(record({ status: "failed", stopReason: HOST }), valid).ok).toBe(true); + + const impossible: Json[] = [ + // A binding that is not one. + { status: "err", output: "", error: failure, root_binding: 7 }, + { status: "err", output: "", error: failure, root_binding: [] }, + // A binding missing a member, carrying an extra one, or mistyping one. + { + status: "err", + output: "", + error: failure, + root_binding: { path: "flows/root.md", source: SOURCE }, + }, + { status: "err", output: "", error: failure, root_binding: { ...binding, extra: 1 } }, + { status: "err", output: "", error: failure, root_binding: { ...binding, path: 7 } }, + { status: "err", output: "", error: failure, root_binding: { ...binding, source: 7 } }, + { status: "err", output: "", error: failure, root_binding: { ...binding, target: 7 } }, + // A binding on a result core could not have written it beside: nothing is + // rendered before the root import, no segment failed, and a failure that + // aggregated others got past it. + { status: "err", output: "partial\n", error: failure, root_binding: binding }, + { + status: "err", + output: "", + error: { name: "Error", message, segment: { message: "something else" } }, + root_binding: binding, + }, + { + status: "err", + output: "", + error: { name: "Error", message, segment: { message, source: "flows/root.md" } }, + root_binding: binding, + }, + { + status: "err", + output: "", + error: { name: "Error", message, segment: { message }, errors: [] }, + root_binding: binding, + }, + { status: "ok", output: "", value: "", root_binding: binding }, + ]; + + for (const value of impossible) { + const said = reason( + retainedReplay(record({ status: "failed", stopReason: HOST }), [entry(rootClose(value))]), + ); + expect([JSON.stringify(value), said.includes("cannot be read by this version")]).toEqual([ + JSON.stringify(value), + true, + ]); + } + }); + // deno-lint-ignore require-yield it("admits the shapes canonical execution actually writes", function* () { const message = "the document refused"; From 856a73b1d0a0b63496e1e486fb41fd08c9788691 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 9 Sep 2026 16:41:49 -0300 Subject: [PATCH 67/81] =?UTF-8?q?=F0=9F=90=9B=20Let=20damage=20outrank=20t?= =?UTF-8?q?he=20row,=20and=20read=20a=20terminal=20against=20the=20history?= =?UTF-8?q?=20around=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two ways an unreadable or impossible retained history still got past the lifecycle. **A terminal row does not vouch for its own journal.** `closingOutcome()` answered the stored status before it looked at the damage, so a run already recorded as `completed` or `failed` came back from both reconciliations with no damaged marker at all, and both begins went on to insert a replay envelope and return `replay: true`. Only afterwards did admission refuse the malformed history — by which time durable execution state had already moved for history that authorizes no execution. Damage is now the first question `closingOutcome()` asks, before the stored status is consulted, so a damaged terminal refuses on both providers for every row it can sit under. **A terminal is only canonical in the history that surrounds it.** A `root_binding` is written by a run that failed before importing anything, and an ordinary document result by one that failed or finished after importing — so the terminal's own members cannot tell a genuine pre-root failure from an ordinary one wearing a binding, and could not tell an ordinary result recorded with no import from a real one. `rootOutcome()` now correlates the two: a bound failure is canonical only with no root import, and an ordinary result only with exactly one root import this coroutine recorded and settled. `rootImports()` is the one reading of that frontier, and retained replay builds its root from the same answer rather than scanning for one of its own. Both rules together say what a workflow run's journal is: a document execution's. The restart harness wrote neither half — no root import, and a plain string where the document result goes — and passed only because a settled `completed` row used to excuse the journal beneath it. It now records the run's own import and returns the result canonical execution returns, both durably, so a second process restores them rather than recording them again. --- packages/cli/tests/workflow-replay.test.ts | 77 +++++++++++++++ packages/workflow/src/lifecycle/policy.ts | 87 ++++++++++++++--- packages/workflow/src/replay.ts | 37 +++---- .../tests/cloudflare/remote-replay.vitest.ts | 79 +++++++++++++++ packages/workflow/tests/replay-inputs.test.ts | 97 +++++++++++++++++++ .../workflow/tests/support/restart-child.ts | 34 +++++-- .../tests/workflow-lifecycle-control.test.ts | 16 ++- .../tests/workflow-run-journal.test.ts | 1 + 8 files changed, 382 insertions(+), 46 deletions(-) diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts index db4719e1f..d7c2cf68a 100644 --- a/packages/cli/tests/workflow-replay.test.ts +++ b/packages/cli/tests/workflow-replay.test.ts @@ -1258,6 +1258,83 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(outcome.after).toEqual(outcome.before); }); + it("WRP17: refuses a terminal row whose own journal it cannot read", function* () { + const asked: string[] = []; + const attached: string[] = []; + let executed = 0; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(BUNDLED, { + Stage: "staged.\n", + Unused: "never imported.\n", + }); + const established = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + + // A row that already says the run ended, over a result nothing can read. + // The row does not vouch for the journal: both accounts have to agree + // before either is reused. + const ended: readonly WorkflowRunStatus[] = ["completed", "failed"]; + const seen: { status: WorkflowRunStatus; before: Retained; after: Retained; said: string }[] = + []; + for (const status of ended) { + const runId = yield* seedStaleRun(fixture, established, damagedHistory(established), { + status, + }); + const before = yield* retained(fixture.runs, runId); + expect([status, before.status]).toEqual([status, status]); + + const said = yield* scoped(function* () { + yield* useRefusingGit(asked); + const started = yield* invoke( + { ...REQUEST, id: runId }, + established, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + executed += 1; + return Ok(undefined); + }, + ); + const resumed = + status === "completed" + ? yield* invoke( + { ...REQUEST, action: "resume", target: runId }, + undefined, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + executed += 1; + return Ok(undefined); + }, + ) + : started; + expect([status, started.exitCode, resumed.exitCode]).toEqual([status, 1, 1]); + expect([status, statusOf(started), statusOf(resumed)]).toEqual([ + status, + undefined, + undefined, + ]); + return `${started.err.join(" ")} ${resumed.err.join(" ")}`; + }); + + seen.push({ status, before, after: yield* retained(fixture.runs, runId), said }); + } + return seen; + }); + + for (const { status, before, after, said } of outcome) { + // Nothing was inserted, closed or published for either row. + expect([status, after]).toEqual([status, before]); + expect([status, said.includes("cannot read")]).toEqual([status, true]); + } + expect(executed).toBe(0); + expect(asked).toEqual([]); + expect(attached).toEqual([]); + }); + it("WRP16: refuses a history holding a second result, without choosing one", function* () { const asked: string[] = []; const attached: string[] = []; diff --git a/packages/workflow/src/lifecycle/policy.ts b/packages/workflow/src/lifecycle/policy.ts index 96cf25483..941e9d762 100644 --- a/packages/workflow/src/lifecycle/policy.ts +++ b/packages/workflow/src/lifecycle/policy.ts @@ -113,11 +113,68 @@ export function rootOutcome(entries: readonly JournalEntry[]): RetainedTerminal if (document === undefined) { return { kind: "damaged" }; } - return document === "ok" + // The terminal has to agree with the history around it. A binding is written + // only by a run that failed before importing anything, so a history that + // imported its root and then recorded one describes two different executions; + // and an ordinary document result is what a run produces *after* importing, + // so one recorded with no import behind it describes a document nothing + // named. Neither is a history any execution can produce, and reading the + // terminal alone cannot tell either of them apart from the real thing. + const imported = rootImports(entries); + if (document.kind === "bound") { + return imported.kind === "none" ? boundOutcome(entries) : { kind: "damaged" }; + } + if (imported.kind !== "one" || imported.entry.event.result.status !== "ok") { + return { kind: "damaged" }; + } + return document.kind === "ok" ? { kind: "outcome", status: "completed", reason: undefined } : { kind: "outcome", status: "failed", reason: retainedFailureReason(entries) }; } +/** The outcome a run that failed before importing anything recorded. */ +function boundOutcome(entries: readonly JournalEntry[]): RetainedTerminal { + return { kind: "outcome", status: "failed", reason: retainedFailureReason(entries) }; +} + +/** + * The root import this history recorded, when it recorded exactly one. + * + * The root's own import is the entry every other record of the run hangs from, + * and canonical core admits a terminal history only when one coroutine — this + * one — recorded exactly one. Both the recovery that publishes an outcome from + * a terminal and the admission that reuses one ask this, so neither can accept + * a history the other refuses. + */ +export type RootImports = + | { readonly kind: "none" } + | { readonly kind: "one"; readonly entry: JournalEntry } + | { readonly kind: "many" }; + +export function rootImports(entries: readonly JournalEntry[]): RootImports { + const imports = entries.filter((entry) => isRootImport(entry.event)); + const only = imports[0]; + if (only === undefined) { + return { kind: "none" }; + } + return imports.length === 1 ? { kind: "one", entry: only } : { kind: "many" }; +} + +function isRootImport(event: DurableEvent): boolean { + return ( + event.type === "yield" && + event.coroutineId === ROOT_COROUTINE && + event.description.type === "import_component" && + event.description.name === ROOT_COMPONENT + ); +} + +/** The coroutine a document execution's own records belong to. */ +const ROOT_COROUTINE = "root"; + +/** The name canonical execution records the run's own document import under. */ +const ROOT_COMPONENT = "__root__"; + /** * Where the root's terminal sits in a history, when it sits anywhere. * @@ -158,13 +215,15 @@ export function terminalFrontier(entries: readonly JournalEntry[]): TerminalFron * missing a member, carrying one of the wrong type, or carrying a member this * form does not have is not a result to reuse. */ -function readDocumentResult(value: unknown): "ok" | "err" | undefined { +function readDocumentResult(value: unknown): { kind: "ok" | "err" | "bound" } | undefined { const result = plain(value); if (result === undefined || typeof result["output"] !== "string") { return undefined; } if (result["status"] === "ok") { - return names(result, ["status", "output", "value"]) && "value" in result ? "ok" : undefined; + return names(result, ["status", "output", "value"]) && "value" in result + ? { kind: "ok" } + : undefined; } if (result["status"] !== "err") { return undefined; @@ -175,10 +234,10 @@ function readDocumentResult(value: unknown): "ok" | "err" | undefined { // whole form is what makes that import-free history attributable to one // document at all. Anything else wearing a binding is a terminal core did // not write. - return readPreRootTerminal(result) ? "err" : undefined; + return readPreRootTerminal(result) ? { kind: "bound" } : undefined; } return names(result, ["status", "output", "error"]) && readDocumentFailure(result["error"]) - ? "err" + ? { kind: "err" } : undefined; } @@ -366,13 +425,11 @@ export function closingOutcome( storedStatus: WorkflowRunStatus, canonical: RetainedTerminal | undefined, ): Closing { - if (terminal(storedStatus)) { - return { status: "interrupted", reason: INTERRUPTED, publishes: false, damaged: false }; - } - if (canonical === undefined) { - return { status: "interrupted", reason: INTERRUPTED, publishes: true, damaged: false }; - } - if (canonical.kind === "damaged") { + // Damage first, and before the stored status is consulted at all. A run whose + // row already says `completed` is not a run whose journal is therefore safe: + // the two accounts have to agree before either is reused, and a row cannot + // vouch for history nothing can read. + if (canonical?.kind === "damaged") { // The document finished and this build cannot read what it finished as. // Calling that an interruption would say the executor went without saying // anything, which is the one thing this history rules out; closing its @@ -380,6 +437,12 @@ export function closingOutcome( // decided at all, and the caller is told why. return { status: storedStatus, reason: undefined, publishes: false, damaged: true }; } + if (terminal(storedStatus)) { + return { status: "interrupted", reason: INTERRUPTED, publishes: false, damaged: false }; + } + if (canonical === undefined) { + return { status: "interrupted", reason: INTERRUPTED, publishes: true, damaged: false }; + } return { status: canonical.status, reason: canonical.reason, publishes: true, damaged: false }; } diff --git a/packages/workflow/src/replay.ts b/packages/workflow/src/replay.ts index 28b409bcf..3586d8ec8 100644 --- a/packages/workflow/src/replay.ts +++ b/packages/workflow/src/replay.ts @@ -41,6 +41,7 @@ import { workflowBundleReplayInstallation } from "./bundle.ts"; import { retainedWorkflowInstallation } from "./run.ts"; import { agreesWithRetainedResult, + rootImports, rootOutcome, terminal, terminalFrontier, @@ -48,11 +49,6 @@ import { import type { JournalEntry } from "./storage/api.ts"; import type { WorkflowRunRecord } from "./storage/record.ts"; -/** The root import a run's own entry is recorded under. */ -const ROOT = "__root__"; - -const IMPORT_COMPONENT = "import_component"; - /** Where core writes the document a terminal it created before importing was about. */ const ROOT_BINDING = "root_binding"; @@ -139,19 +135,6 @@ function recorded(value: unknown): Record | undefined { return held; } -/** Whether a retained event is recognizably a settled component import. */ -function importedName(event: DurableEvent): string | undefined { - if (reading(() => event.type) !== "yield") { - return undefined; - } - const description = reading(() => (event.type === "yield" ? event.description : undefined)); - if (description === undefined || reading(() => description.type) !== IMPORT_COMPONENT) { - return undefined; - } - const name = reading(() => description.name); - return typeof name === "string" ? name : undefined; -} - /** The document one retained selection names, as a root source is built from it. */ interface RetainedRoot { readonly path: string; @@ -274,6 +257,15 @@ export function retainedReplay( // A terminal this build cannot read is refused before canonical core is // handed it, because core's rejection would arrive as *this* invocation's // document failure and be offered as a replacement outcome. + // The root's own import, read by the same rule the lifecycle reads it by, so + // the document this replay is built from is the one the outcome below was + // judged against. Answered here first only so a history recording two says + // so, rather than arriving as damage with nothing to distinguish it. + const imports = rootImports(entries); + if (imports.kind === "many") { + return refuse(REFUSALS.ambiguous); + } + const canonical = rootOutcome(entries); if (canonical === undefined || canonical.kind === "damaged") { return refuse(REFUSALS.damaged); @@ -282,13 +274,8 @@ export function retainedReplay( return refuse(REFUSALS.disagreed); } - const imports = entries.filter((entry) => importedName(entry.event) === ROOT); - if (imports.length > 1) { - return refuse(REFUSALS.ambiguous); - } - - const imported = imports[0]; - const retained = imported === undefined ? boundRoot(close.event) : rootSelection(imported.event); + const retained = + imports.kind === "none" ? boundRoot(close.event) : rootSelection(imports.entry.event); if (retained === undefined) { return refuse(REFUSALS.malformed); } diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts index 537ca4baf..d5df8ec48 100644 --- a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -38,6 +38,7 @@ import type { RemoteLifecycleHost } from "../../src/remote/lifecycle.ts"; import type { RemoteExecutorConnection } from "../../src/remote/lifecycle-link.ts"; import type { RemoteReadPlane } from "../../src/remote/read.ts"; import type { WorkflowRunDatabase } from "../../src/storage/api.ts"; +import type { WorkflowRunStatus } from "../../src/storage/record.ts"; import { WorkflowLifecycle } from "../../src/lifecycle/api.ts"; import type { WorkflowExecutionTransitions, @@ -780,6 +781,84 @@ describe("a completed run replayed through its own owner", () => { expect(await on(stub, (owner) => owner.holders())).toBe(0); }); + it("refuses a terminal row whose own journal it cannot read, and moves nothing", async () => { + const ended: readonly WorkflowRunStatus[] = ["completed", "failed"]; + for (const status of ended) { + const stub = executor(); + const host = lifecycleHost(stub); + + // A row that already says the run ended, over a result nothing can read. + await run(function* () { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + const appended = yield* begun.value.database.transact(function* (transaction) { + yield* transaction.journal.append({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "README.md", content: DOCUMENT }, + }, + }); + yield* transaction.journal.append({ + type: "close", + coroutineId: "root", + result: { status: "ok", value: { status: "err" } }, + }); + }); + if (!appended.ok) { + throw appended.error; + } + const settled = yield* transitions.settle(lock, { + executionId: begun.value.execution.executionId, + status, + ...(status === "failed" ? { reason: { kind: "host", code: DOCUMENT_FAILED } } : {}), + }); + if (!settled.ok) { + throw settled.error; + } + }); + }); + + const before = await ownerState(stub); + expect(before.run?.["status"]).toBe(status); + + const refusals = await run(function* (): Operation> { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const started = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + const resumed = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + return { + started: started.ok ? "admitted" : started.error.message, + resumed: resumed.ok ? "admitted" : resumed.error.message, + }; + }); + }); + + expect(String(refusals["started"])).toContain("cannot read"); + expect(String(refusals["resumed"])).toContain("cannot read"); + // A terminal row does not vouch for the journal beneath it: no replay + // envelope was inserted and nothing was published. + expect(await ownerState(stub)).toEqual(before); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + } + }); + it("hands the run to the next connection when the replay's own ends", async () => { const stub = executor(); const host = lifecycleHost(stub); diff --git a/packages/workflow/tests/replay-inputs.test.ts b/packages/workflow/tests/replay-inputs.test.ts index f7289a85e..506b49d42 100644 --- a/packages/workflow/tests/replay-inputs.test.ts +++ b/packages/workflow/tests/replay-inputs.test.ts @@ -20,6 +20,7 @@ import type { Operation, Result } from "effection"; import type { DurableEvent, Json } from "@executablemd/durable-streams"; import type { ExecutionInstallation } from "@executablemd/core/host"; import { retainedReplay } from "../src/replay.ts"; +import { rootOutcome } from "../src/lifecycle/policy.ts"; import { DOCUMENT_FAILED } from "../src/lifecycle/policy.ts"; import type { RetainedReplay } from "../src/replay.ts"; import { WorkflowReplayHistoryError } from "../src/replay.ts"; @@ -621,6 +622,102 @@ describe("what the root recorded, as one outcome", () => { } }); + // deno-lint-ignore require-yield + it("correlates the terminal with the import history around it", function* () { + const message = "refused before the root import"; + const bound = { + status: "err", + output: "", + error: { name: "Error", message, segment: { message } }, + root_binding: { path: "flows/root.md", source: SOURCE, target: null }, + }; + const imported = rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE }); + const succeeded = { status: "ok", output: "done.\n", value: "done.\n" }; + + // The two histories any execution can produce. + const canonical: { says: string; entries: JournalEntry[]; status: WorkflowRunStatus }[] = [ + { + says: "imported, then a result", + entries: [entry(imported), entry(rootClose(succeeded))], + status: "completed", + }, + { + says: "no import, and the bound failure", + entries: [entry(rootClose(bound))], + status: "failed", + }, + ]; + for (const { says, entries, status } of canonical) { + expect([says, rootOutcome(entries)?.kind]).toEqual([says, "outcome"]); + const outcome = retainedReplay( + record({ status, ...(status === "failed" ? { stopReason: HOST } : {}) }), + entries, + ); + expect([says, outcome.ok]).toEqual([says, true]); + } + + // And the ones none can. A binding is written only by a run that imported + // nothing; an ordinary result is written only by one that imported. + const impossible: { says: string; entries: JournalEntry[]; status: WorkflowRunStatus }[] = [ + { + says: "imported, then the bound failure", + entries: [entry(imported), entry(rootClose(bound))], + status: "failed", + }, + { + says: "no import, and an ordinary failure", + entries: [entry(rootClose(documentFailure("partial\n")))], + status: "failed", + }, + { + says: "no import, and a success", + entries: [entry(rootClose(succeeded))], + status: "completed", + }, + { + says: "an import that failed, then a result", + entries: [ + entry({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "err", error: { message: "gone", name: "Error" } }, + }), + entry(rootClose(succeeded)), + ], + status: "completed", + }, + { + says: "an import another coroutine recorded", + entries: [ + entry({ + type: "yield", + coroutineId: "child", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "flows/root.md", content: SOURCE }, + }, + }), + entry(rootClose(succeeded)), + ], + status: "completed", + }, + ]; + for (const { says, entries, status } of impossible) { + // Stale recovery classifies it as damage rather than publishing from it. + expect([says, rootOutcome(entries)?.kind]).toEqual([says, "damaged"]); + // And admission refuses it rather than building a root from it. + const said = reason( + retainedReplay( + record({ status, ...(status === "failed" ? { stopReason: HOST } : {}) }), + entries, + ), + ); + expect([says, said.includes("cannot be read by this version")]).toEqual([says, true]); + } + }); + // deno-lint-ignore require-yield it("admits the shapes canonical execution actually writes", function* () { const message = "the document refused"; diff --git a/packages/workflow/tests/support/restart-child.ts b/packages/workflow/tests/support/restart-child.ts index 54fc21ca7..a43cef602 100644 --- a/packages/workflow/tests/support/restart-child.ts +++ b/packages/workflow/tests/support/restart-child.ts @@ -26,8 +26,8 @@ import { appendFile } from "node:fs/promises"; import process from "node:process"; -import { durableCall, durableRun } from "@executablemd/durable-streams"; -import type { Workflow } from "@executablemd/durable-streams"; +import { createDurableOperation, durableCall, durableRun } from "@executablemd/durable-streams"; +import type { Json, Workflow } from "@executablemd/durable-streams"; import { main, until } from "effection"; import { WorkflowLifecycle, WorkflowStorageError } from "../../mod.ts"; import { useWorkflowRunHost } from "../../deno.ts"; @@ -40,14 +40,31 @@ const DEFINITION = { rootDocumentPath: "workflows/release.md", } as const; +/** What this run's document is, as its own history records it. */ +const SOURCE = "# Release\n"; + /** * Three durable operations, so replay has an order to preserve. * * Each one's side effect is a line in the marker file, which is what makes * "did this run again" observable from outside the process. + * + * The root's own import comes first and the result is the document result + * canonical execution returns, because a workflow run's journal is a document + * execution's: the lifecycle reads the root import and that result to decide + * what the run became, and a history missing either is one no execution + * produces. Both are durable, so a second process restores them rather than + * recording them again — which is the thing this file exists to observe. */ -function work(marker: string): () => Workflow { - return function* (): Workflow { +function work(marker: string): () => Workflow { + return function* (): Workflow { + yield createDurableOperation( + { type: "import_component", name: "__root__" }, + // deno-lint-ignore require-yield + function* (): Workflow { + return { kind: "repository", path: DEFINITION.rootDocumentPath, content: SOURCE }; + }, + ); const first = yield* durableCall("first", function* () { yield* until(appendFile(marker, "first\n")); return "one"; @@ -60,7 +77,8 @@ function work(marker: string): () => Workflow { yield* until(appendFile(marker, "third\n")); return "three"; }); - return [first, second, third].join(","); + const rendered = [first, second, third].join(","); + return { status: "ok", output: rendered, value: rendered }; }; } @@ -97,7 +115,11 @@ main(function* () { } const { database, execution } = opened.value; - const value = yield* durableRun(work(marker), { stream: database.journal }); + const result = yield* durableRun(work(marker), { stream: database.journal }); + const value = + typeof result === "object" && result !== null && !Array.isArray(result) + ? Reflect.get(result, "value") + : undefined; const settled = yield* transitions.settle(executorLock, { executionId: execution.executionId, diff --git a/packages/workflow/tests/workflow-lifecycle-control.test.ts b/packages/workflow/tests/workflow-lifecycle-control.test.ts index 2bf03a0f6..03ed71af5 100644 --- a/packages/workflow/tests/workflow-lifecycle-control.test.ts +++ b/packages/workflow/tests/workflow-lifecycle-control.test.ts @@ -174,9 +174,19 @@ describe("Tier WLC — cancellation and deletion", () => { transitions, { runId: "closed-1", action: "start", creation: creation() }, function* (begun) { - // The coroutine returned, and what it returned is the document's own - // result: the shape canonical core writes, because that shape is what - // says whether the document completed or failed. + // The history canonical execution writes: the root's own import, then + // the document result it returned. Both halves matter — the result's + // own status is what says whether the document completed or failed, + // and an ordinary result is one a run records after importing. + yield* begun.database.journal.append({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "workflow.md", content: "rendered" }, + }, + }); yield* begun.database.journal.append({ type: "close", coroutineId: "root", diff --git a/packages/workflow/tests/workflow-run-journal.test.ts b/packages/workflow/tests/workflow-run-journal.test.ts index f541a0acd..dc469a450 100644 --- a/packages/workflow/tests/workflow-run-journal.test.ts +++ b/packages/workflow/tests/workflow-run-journal.test.ts @@ -1594,6 +1594,7 @@ describe("Tier WJ — surviving a process", () => { // same sequence, read by a process that never saw them written. expect(after.events).toEqual(before.events); expect(after.events.map((event: { name?: string }) => event.name)).toEqual([ + "__root__", "first", "second", "third", From ecdec5760c7fa5038501a6ca32bfdfd57385ef87 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 9 Sep 2026 20:24:41 -0300 Subject: [PATCH 68/81] =?UTF-8?q?=F0=9F=90=9B=20Count=20every=20root=20imp?= =?UTF-8?q?ort,=20and=20parse=20the=20one=20before=20a=20terminal=20counts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `rootImports()` filtered retained imports by the root coroutine, so a second event naming `import_component/__root__` recorded under a child was ignored rather than refused, and a settled root import was treated as sufficient without reading the selection it carried. Canonical `admitRootHistory()` gathers every event naming the root import first, then requires exactly one in total owned by the root coroutine, and parses it — so lifecycle recovery could publish `completed` or `failed` from a history canonical execution rejects, and retained replay accepted one it rejects too. Uniqueness is now asked of the name, not of the ownership: one shared judgment gathers every event that names the root import, requires exactly one total, requires the root to own it, and parses it into the selection replay rebuilds the document from. Retained replay reads that judgment instead of deciding readability again afterwards, so neither boundary can accept a history the other refuses. --- packages/cli/tests/workflow-replay.test.ts | 115 ++++++++++++++ packages/workflow/src/lifecycle/policy.ts | 108 +++++++++++++- packages/workflow/src/replay.ts | 141 ++---------------- .../tests/cloudflare/remote-replay.vitest.ts | 78 ++++++++++ packages/workflow/tests/replay-inputs.test.ts | 90 +++++++++++ 5 files changed, 395 insertions(+), 137 deletions(-) diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts index d7c2cf68a..3b9e5d65e 100644 --- a/packages/cli/tests/workflow-replay.test.ts +++ b/packages/cli/tests/workflow-replay.test.ts @@ -1398,6 +1398,121 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(attached).toEqual([]); }); + it("WRP18: refuses a root import it cannot hold to one readable selection", function* () { + const asked: string[] = []; + const attached: string[] = []; + + const outcome = yield* scoped(function* () { + const fixture = yield* useFixture(PLAIN); + const established = yield* scoped(function* () { + yield* useRepositoryGit(fixture.repository); + return yield* startFor(fixture); + }); + + const source = established.established.source; + const settled: DurableEvent = { + type: "close", + coroutineId: "root", + result: { status: "ok", value: { status: "ok", output: "done\n", value: "done\n" } }, + }; + + /** What the root import records, said in a way this build cannot hold. */ + const unreadable: Record readonly DurableEvent[]> = { + // A child recorded the root's own import a second time. + duplicated: (path) => [ + rootImportEvent(path, source), + { + type: "yield", + coroutineId: "child", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path, content: source } }, + }, + settled, + ], + // The only import names the root, and the root did not record it. + disowned: (path) => [ + { + type: "yield", + coroutineId: "child", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path, content: source } }, + }, + settled, + ], + // The selection the run replays from is missing the document itself. + contentless: (path) => [ + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: { kind: "repository", path } }, + }, + settled, + ], + // A recorded selection failure with no selector left to replay. + selectorless: (path) => [ + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { + kind: "target-failure", + path, + content: source, + failure: { kind: "no-match", matches: [], available: [] }, + }, + }, + }, + settled, + ], + }; + + const refusals: Record = {}; + for (const [says, history] of Object.entries(unreadable)) { + const runId = yield* seedStaleRun(fixture, established, (definition, id) => [ + forkRunRecordEvent({ + runId: id, + base: established.established.base, + pinnedCommit: definition.objectId, + }), + ...history(definition.rootDocumentPath), + ]); + const before = yield* retained(fixture.runs, runId); + const refused = yield* scoped(function* () { + yield* useRefusingGit(asked); + return yield* invoke( + { ...REQUEST, id: runId }, + established, + replayHost(fixture.runs, attached), + // deno-lint-ignore require-yield + function* (): Operation> { + return Ok(undefined); + }, + ); + }); + const after = yield* retained(fixture.runs, runId); + refusals[says] = { + exitCode: refused.exitCode, + said: refused.err.join(" "), + moved: JSON.stringify(after) !== JSON.stringify(before), + }; + } + return refusals; + }); + + // Each history refuses before the terminal is named, and recovery + // publishes nothing over a run it cannot say the root of. + for (const [says, refusal] of Object.entries(outcome)) { + expect([says, refusal.exitCode]).toEqual([says, 1]); + expect([says, refusal.said.includes("cannot read")]).toEqual([says, true]); + expect([says, refusal.moved]).toEqual([says, false]); + } + expect(asked).toEqual([]); + expect(attached).toEqual([]); + }); + it("WRP5: a run that has not ended still reconstructs, and its refusal is cleaned up", function* () { const asked: string[] = []; const attached: string[] = []; diff --git a/packages/workflow/src/lifecycle/policy.ts b/packages/workflow/src/lifecycle/policy.ts index 941e9d762..58d57090d 100644 --- a/packages/workflow/src/lifecycle/policy.ts +++ b/packages/workflow/src/lifecycle/policy.ts @@ -124,7 +124,7 @@ export function rootOutcome(entries: readonly JournalEntry[]): RetainedTerminal if (document.kind === "bound") { return imported.kind === "none" ? boundOutcome(entries) : { kind: "damaged" }; } - if (imported.kind !== "one" || imported.entry.event.result.status !== "ok") { + if (imported.kind !== "one") { return { kind: "damaged" }; } return document.kind === "ok" @@ -148,27 +148,121 @@ function boundOutcome(entries: readonly JournalEntry[]): RetainedTerminal { */ export type RootImports = | { readonly kind: "none" } - | { readonly kind: "one"; readonly entry: JournalEntry } - | { readonly kind: "many" }; + | { readonly kind: "one"; readonly selection: RetainedRootSelection } + /** More than one retained event names the root import. */ + | { readonly kind: "many" } + /** One does, and it is not a root import this build can read. */ + | { readonly kind: "malformed" }; export function rootImports(entries: readonly JournalEntry[]): RootImports { - const imports = entries.filter((entry) => isRootImport(entry.event)); + // Every event that *names* the root import, whichever coroutine claims it. + // Uniqueness is asked of the name, not of the ownership: a child coroutine + // recording one is a second account of the run's own entry, and canonical + // core refuses that history rather than looking past it. + const imports = entries.filter((entry) => namesRootImport(entry.event)); const only = imports[0]; if (only === undefined) { return { kind: "none" }; } - return imports.length === 1 ? { kind: "one", entry: only } : { kind: "many" }; + if (imports.length !== 1 || only.event.coroutineId !== ROOT_COROUTINE) { + return imports.length === 1 ? { kind: "malformed" } : { kind: "many" }; + } + const selection = rootSelection(only.event); + return selection === undefined ? { kind: "malformed" } : { kind: "one", selection }; } -function isRootImport(event: DurableEvent): boolean { +function namesRootImport(event: DurableEvent): boolean { return ( event.type === "yield" && - event.coroutineId === ROOT_COROUTINE && event.description.type === "import_component" && event.description.name === ROOT_COMPONENT ); } +/** + * The document one retained root import selected, as its own record holds it. + * + * The shapes canonical execution writes for a root and no others: the whole + * document, one exact target, and a selector the document offered no single + * target for. A settlement that failed recorded no selection at all, and a + * successful one that does not hold a complete selection is a root import this + * build cannot read — which is a fact about the history, not about the caller + * asking, so it is decided here rather than by whoever asks next. + */ +export interface RetainedRootSelection { + readonly path: string; + readonly content: string; + readonly target: string | undefined; +} + +function rootSelection(event: DurableEvent): RetainedRootSelection | undefined { + const settlement = plain(event.type === "yield" ? event.result : undefined); + if (settlement === undefined || settlement["status"] !== "ok") { + return undefined; + } + const selection = plain(settlement["value"]); + if (selection === undefined) { + return undefined; + } + const path = selection["path"]; + const content = selection["content"]; + const kind = selection["kind"]; + const members = Object.keys(selection).length; + if (typeof path !== "string" || typeof content !== "string") { + return undefined; + } + + if (kind === "repository") { + const target = selection["target"]; + if (target === undefined) { + return members === 3 ? { path, content, target: undefined } : undefined; + } + return members === 4 && typeof target === "string" ? { path, content, target } : undefined; + } + + if (kind === "target-failure") { + // The selector as the run was asked for it. Handing it back is what makes + // the replayed request the same request: canonical execution resolves it + // against the recorded document again and finds the same failure. + const failure = plain(selection["failure"]); + const selector = failure?.["selector"]; + return members === 4 && typeof selector === "string" + ? { path, content, target: selector } + : undefined; + } + + return undefined; +} + +/** + * The document a terminal written before any import was about. + * + * Read from the binding `readPreRootTerminal()` above has already held to its + * exact closed form, so this only takes what that admitted: nothing decides + * here that was not decided there. + */ +export function preRootSelection( + entries: readonly JournalEntry[], +): RetainedRootSelection | undefined { + const frontier = terminalFrontier(entries); + if (frontier.kind !== "final") { + return undefined; + } + const settlement = plain(frontier.entry.event.result); + const result = plain(settlement?.["value"]); + const binding = plain(result?.[ROOT_BINDING]); + if (binding === undefined) { + return undefined; + } + const path = binding["path"]; + const source = binding["source"]; + const target = binding["target"]; + if (typeof path !== "string" || typeof source !== "string") { + return undefined; + } + return { path, content: source, target: typeof target === "string" ? target : undefined }; +} + /** The coroutine a document execution's own records belong to. */ const ROOT_COROUTINE = "root"; diff --git a/packages/workflow/src/replay.ts b/packages/workflow/src/replay.ts index 3586d8ec8..8d95b8450 100644 --- a/packages/workflow/src/replay.ts +++ b/packages/workflow/src/replay.ts @@ -34,13 +34,13 @@ */ import { Err, Ok, type Result } from "effection"; -import type { DurableEvent } from "@executablemd/durable-streams"; import { retainedSource } from "@executablemd/core/host"; import type { ExecutionInstallation, RetainedRootDocument } from "@executablemd/core/host"; import { workflowBundleReplayInstallation } from "./bundle.ts"; import { retainedWorkflowInstallation } from "./run.ts"; import { agreesWithRetainedResult, + preRootSelection, rootImports, rootOutcome, terminal, @@ -49,9 +49,6 @@ import { import type { JournalEntry } from "./storage/api.ts"; import type { WorkflowRunRecord } from "./storage/record.ts"; -/** Where core writes the document a terminal it created before importing was about. */ -const ROOT_BINDING = "root_binding"; - /** What a completed run hands canonical execution, and nothing else. */ export interface RetainedReplay { /** The root document, exactly as this run's own history recorded it. */ @@ -100,125 +97,6 @@ function refuse(reason: string): Result { return Err(new WorkflowReplayHistoryError(reason)); } -/** - * Read one journal-controlled value, or answer that reading it refused. - * - * As narrow as the reads a retained import is parsed through, and for the same - * reason: a throwing accessor and a proxy trap both mean this record does not - * describe a replay, and nothing wider should be swallowed. - */ -function reading(read: () => T): T | undefined { - try { - return read(); - } catch { - return undefined; - } -} - -/** A recorded value that is an ordinary object, or nothing. */ -function recorded(value: unknown): Record | undefined { - if (typeof value !== "object" || value === null || Array.isArray(value)) { - return undefined; - } - const names = reading(() => Object.keys(value)); - if (names === undefined) { - return undefined; - } - const held: Record = {}; - for (const name of names) { - const member = reading(() => Reflect.get(value, name)); - if (member === undefined) { - return undefined; - } - held[name] = member; - } - return held; -} - -/** The document one retained selection names, as a root source is built from it. */ -interface RetainedRoot { - readonly path: string; - readonly content: string; - readonly target: string | undefined; -} - -/** - * The selection a settled root import recorded. - * - * Only the shapes canonical execution writes for a root: the whole document, - * one exact target, and a selector the document offered no single target for. - * A settlement that failed recorded no selection at all, and is refused by the - * caller rather than guessed at. - */ -function rootSelection(event: DurableEvent): RetainedRoot | undefined { - const settlement = recorded(reading(() => (event.type === "yield" ? event.result : undefined))); - if (settlement === undefined || settlement["status"] !== "ok") { - return undefined; - } - const selection = recorded(settlement["value"]); - if (selection === undefined) { - return undefined; - } - const path = selection["path"]; - const content = selection["content"]; - const kind = selection["kind"]; - const members = Object.keys(selection).length; - if (typeof path !== "string" || typeof content !== "string") { - return undefined; - } - - if (kind === "repository") { - const target = selection["target"]; - if (target === undefined) { - return members === 3 ? { path, content, target: undefined } : undefined; - } - return members === 4 && typeof target === "string" ? { path, content, target } : undefined; - } - - if (kind === "target-failure") { - // The selector as the run was asked for it. Handing it back is what makes - // the replayed request the same request: canonical execution resolves it - // against the recorded document again and finds the same failure. - const failure = recorded(selection["failure"]); - const selector = failure?.["selector"]; - return members === 4 && typeof selector === "string" - ? { path, content, target: selector } - : undefined; - } - - return undefined; -} - -/** - * The document a terminal written before any import was about. - * - * A run can fail before it imports anything, and canonical core records which - * document that failure was about inside the terminal itself. That binding is - * the only account such a history has of its own root, so it is what a replay - * of one is built from. - */ -function boundRoot(close: DurableEvent): RetainedRoot | undefined { - const settlement = recorded(reading(() => close.result)); - if (settlement === undefined || settlement["status"] !== "ok") { - return undefined; - } - const result = recorded(settlement["value"]); - const binding = recorded(result?.[ROOT_BINDING]); - if (binding === undefined || Object.keys(binding).length !== 3) { - return undefined; - } - const path = binding["path"]; - const source = binding["source"]; - const target = binding["target"]; - if (typeof path !== "string" || typeof source !== "string") { - return undefined; - } - if (target !== null && typeof target !== "string") { - return undefined; - } - return { path, content: source, target: target ?? undefined }; -} - /** * What this run's owner already holds, as the inputs one canonical replay runs * on — or why the state it holds describes no completed run. @@ -248,7 +126,6 @@ export function retainedReplay( if (frontier.kind === "mixed") { return refuse(REFUSALS.mixed); } - const close = frontier.entry; // What the root recorded, read once, by the lifecycle's own judgment: the // same one stale recovery publishes through and the same one a settlement @@ -257,14 +134,19 @@ export function retainedReplay( // A terminal this build cannot read is refused before canonical core is // handed it, because core's rejection would arrive as *this* invocation's // document failure and be offered as a replacement outcome. - // The root's own import, read by the same rule the lifecycle reads it by, so - // the document this replay is built from is the one the outcome below was - // judged against. Answered here first only so a history recording two says - // so, rather than arriving as damage with nothing to distinguish it. + // The root's own import, read by the same rule the lifecycle reads it by — + // the same call, answering with the same parsed selection — so the document + // this replay is built from is the one the outcome below was judged against. + // Asked here first only so a history recording two, or recording one this + // build cannot read, says which rather than arriving as undifferentiated + // damage. const imports = rootImports(entries); if (imports.kind === "many") { return refuse(REFUSALS.ambiguous); } + if (imports.kind === "malformed") { + return refuse(REFUSALS.malformed); + } const canonical = rootOutcome(entries); if (canonical === undefined || canonical.kind === "damaged") { @@ -274,8 +156,7 @@ export function retainedReplay( return refuse(REFUSALS.disagreed); } - const retained = - imports.kind === "none" ? boundRoot(close.event) : rootSelection(imports.entry.event); + const retained = imports.kind === "none" ? preRootSelection(entries) : imports.selection; if (retained === undefined) { return refuse(REFUSALS.malformed); } diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts index d5df8ec48..891d4e525 100644 --- a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -781,6 +781,84 @@ describe("a completed run replayed through its own owner", () => { expect(await on(stub, (owner) => owner.holders())).toBe(0); }); + it("refuses a root import no single execution recorded, and moves nothing", async () => { + const stub = executor(); + const host = lifecycleHost(stub); + + // A settled root, over a history in which the run's own import is claimed + // twice: once by the root, once by a child that could not have run it. + await run(function* () { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + const appended = yield* begun.value.database.transact(function* (transaction) { + for (const coroutineId of ["root", "child"]) { + yield* transaction.journal.append({ + type: "yield", + coroutineId, + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path: "README.md", content: DOCUMENT }, + }, + }); + } + yield* transaction.journal.append({ + type: "close", + coroutineId: "root", + result: { + status: "ok", + value: { status: "ok", output: "done.\n", value: "done.\n" }, + }, + }); + }); + if (!appended.ok) { + throw appended.error; + } + }); + }); + + const before = await ownerState(stub); + expect(before.run?.["status"]).toBe("running"); + const heldBefore = await on(stub, (owner) => owner.executionRows()); + + // The terminal reads, and the history it settled cannot say what ran. + const refusals = await run(function* (): Operation> { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const started = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + const resumed = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + return { + started: started.ok ? "admitted" : started.error.message, + resumed: resumed.ok ? "admitted" : resumed.error.message, + }; + }); + }); + + for (const said of Object.values(refusals)) { + expect(String(said)).toContain("cannot read"); + } + + // Neither account of the import was chosen, and recovery published nothing. + const after = await ownerState(stub); + expect(after).toEqual(before); + expect(await on(stub, (owner) => owner.executionRows())).toEqual(heldBefore); + expect(await on(stub, (owner) => owner.holders())).toBe(0); + }); + it("refuses a terminal row whose own journal it cannot read, and moves nothing", async () => { const ended: readonly WorkflowRunStatus[] = ["completed", "failed"]; for (const status of ended) { diff --git a/packages/workflow/tests/replay-inputs.test.ts b/packages/workflow/tests/replay-inputs.test.ts index 506b49d42..08f01865b 100644 --- a/packages/workflow/tests/replay-inputs.test.ts +++ b/packages/workflow/tests/replay-inputs.test.ts @@ -718,6 +718,96 @@ describe("what the root recorded, as one outcome", () => { } }); + // deno-lint-ignore require-yield + it("counts every event that names the root import, whoever recorded it", function* () { + const succeeded = { status: "ok", output: "done.\n", value: "done.\n" }; + const selection = { kind: "repository", path: "flows/root.md", content: SOURCE }; + const owned = entry(rootImport(selection)); + + /** The same import, recorded under another coroutine. */ + function claimed(coroutineId: string): DurableEvent { + return { + type: "yield", + coroutineId, + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: selection }, + }; + } + + const histories: { says: string; entries: JournalEntry[] }[] = [ + { + says: "a child recorded a second one", + entries: [owned, entry(claimed("child")), entry(rootClose(succeeded))], + }, + { + says: "a child recorded the only one", + entries: [entry(claimed("child")), entry(rootClose(succeeded))], + }, + ]; + + for (const { says, entries } of histories) { + // Uniqueness is asked of the name, not of the ownership: a second + // account of the run's own entry is a history canonical core refuses. + expect([says, rootOutcome(entries)?.kind]).toEqual([says, "damaged"]); + expect([says, retainedReplay(record(), entries).ok]).toEqual([says, false]); + } + + // And the one history this rules out nothing about. + expect(rootOutcome([owned, entry(rootClose(succeeded))])?.kind).toBe("outcome"); + expect(retainedReplay(record(), [owned, entry(rootClose(succeeded))]).ok).toBe(true); + }); + + // deno-lint-ignore require-yield + it("holds a settled root import to the selection it must contain", function* () { + const succeeded = { status: "ok", output: "done.\n", value: "done.\n" }; + const unreadable: Json[] = [ + // The content the record must hold, missing. + { kind: "repository", path: "flows/root.md" }, + // A member this form does not have. + { kind: "repository", path: "flows/root.md", content: SOURCE, extra: 1 }, + // A target that is not one. + { kind: "repository", path: "flows/root.md", content: SOURCE, target: 7 }, + // A recorded selection failure with no selector to replay. + { + kind: "target-failure", + path: "flows/root.md", + content: SOURCE, + failure: { kind: "no-match", matches: [], available: [] }, + }, + // A selection kind a root import never records. + { kind: "workflow", path: "flows/root.md", sourceHash: "abc", content: SOURCE }, + ]; + + for (const value of unreadable) { + const entries = [entry(rootImport(value)), entry(rootClose(succeeded))]; + // The lifecycle refuses to call the terminal authoritative… + expect([JSON.stringify(value), rootOutcome(entries)?.kind]).toEqual([ + JSON.stringify(value), + "damaged", + ]); + // …and admission refuses through that same judgment. + const said = reason(retainedReplay(record(), entries)); + expect([JSON.stringify(value), said.includes("cannot be read by this version")]).toEqual([ + JSON.stringify(value), + true, + ]); + } + + // A settlement that failed records no selection at all, and one whose + // settlement cannot be read records none either. + const failed = [ + entry({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "err", error: { message: "gone", name: "Error" } }, + }), + entry(rootClose(succeeded)), + ]; + expect(rootOutcome(failed)?.kind).toBe("damaged"); + expect(retainedReplay(record(), failed).ok).toBe(false); + }); + // deno-lint-ignore require-yield it("admits the shapes canonical execution actually writes", function* () { const message = "the document refused"; From 356d34727a042fdf21681b2f254388fecb51ee35 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Wed, 9 Sep 2026 22:34:40 -0300 Subject: [PATCH 69/81] =?UTF-8?q?=F0=9F=90=9B=20Verify=20a=20retained=20se?= =?UTF-8?q?lection=20against=20the=20document=20that=20recorded=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lifecycle's own reader checked field counts and primitive types, so a retained root import naming a target its document does not offer, spelled a way canonical encoding never spells it, or carrying a "failure" reduced to a selector was read as a complete selection. Canonical `readRootSelection()` does more than recognize a shape: it parses the retained markdown, resolves an exact target against that outline and requires the recorded target back, and re-derives a recorded failure through `findTarget()` requiring the kind, the matches and the whole catalog to agree. History the executor refuses could therefore publish an outcome, insert a replay envelope, and hand execution a selector taken from a forged record. So there is one parser again. It moves out of `execute.ts` into its own module with the reading primitives it needs, canonical execution admits partial histories through it exactly as before, and the lifecycle reaches it through `@executablemd/core/host` — the same reasoning that already carries the Prompt record across that boundary. What comes back is the parser's own copy of the record, so the document a replay is built from is never the object the journal still holds. A selection outcome also has to be able to reach the terminal beside it. A recorded selection failure is raised out of the root import, so the document never ran: a successful result over one is two histories, and the run is damaged rather than completed. Two fixtures recorded selections canonical execution cannot produce — an exact target in a document offering none, and a failure carrying a catalog that document does not have — and now record what the same selector really decides. --- packages/cli/tests/workflow-replay.test.ts | 33 ++- packages/core/host.ts | 15 ++ packages/core/src/execute.ts | 185 +------------- packages/core/src/root-selection.ts | 225 ++++++++++++++++++ packages/workflow/src/lifecycle/policy.ts | 117 +++++---- .../tests/cloudflare/remote-replay.vitest.ts | 88 +++++++ packages/workflow/tests/replay-inputs.test.ts | 196 ++++++++++++++- 7 files changed, 626 insertions(+), 233 deletions(-) create mode 100644 packages/core/src/root-selection.ts diff --git a/packages/cli/tests/workflow-replay.test.ts b/packages/cli/tests/workflow-replay.test.ts index 3b9e5d65e..6309e80f3 100644 --- a/packages/cli/tests/workflow-replay.test.ts +++ b/packages/cli/tests/workflow-replay.test.ts @@ -1398,7 +1398,7 @@ describe("what a completed run reaches when it is asked to run again", () => { expect(attached).toEqual([]); }); - it("WRP18: refuses a root import it cannot hold to one readable selection", function* () { + it("WRP18: refuses a root import it cannot hold to one verified selection", function* () { const asked: string[] = []; const attached: string[] = []; @@ -1449,6 +1449,37 @@ describe("what a completed run reaches when it is asked to run again", () => { }, settled, ], + // A failure record reduced to the selector it was asked for. + forged: (path) => [ + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { + kind: "target-failure", + path, + content: source, + failure: { selector: "Missing" }, + }, + }, + }, + settled, + ], + // An exact target the retained document does not offer. + absent: (path) => [ + { + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { + status: "ok", + value: { kind: "repository", path, content: source, target: "Missing" }, + }, + }, + settled, + ], // A recorded selection failure with no selector left to replay. selectorless: (path) => [ { diff --git a/packages/core/host.ts b/packages/core/host.ts index 0e351f690..d0d34fa05 100644 --- a/packages/core/host.ts +++ b/packages/core/host.ts @@ -142,3 +142,18 @@ export type { */ export { AGENT_PROMPT, parsePromptRecord } from "./src/agent/journal.ts"; export type { PromptRecord } from "./src/agent/journal.ts"; + +/** + * The retained root-import protocol, for a host that reads a retained journal. + * + * Same reasoning as the Prompt record above, with more at stake. A workflow run + * decides from its own retained events whether a completed history may publish + * an outcome and be replayed, and which document that history was about — and + * the only thing that can answer that is the parser canonical execution admits + * partial histories through. Read a second way, the same record would answer to + * a second, weaker protocol: a selection the executor would refuse could + * publish a terminal outcome and authorize a replay. So the parser crosses the + * boundary rather than being described again. + */ +export { recordedRootImport } from "./src/root-selection.ts"; +export type { RootImportRecord, SelectionOutcome } from "./src/root-selection.ts"; diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 8ce006d69..793708161 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -50,6 +50,8 @@ import type { SourcePosition, } from "./types.ts"; import { isJsonObject, parseJson, parseJsonObject } from "./json.ts"; +import { attempt, read, recordedRootImport, UNREADABLE } from "./root-selection.ts"; +import type { SelectionOutcome } from "./root-selection.ts"; import { compilePropsSchema, compileReturnsSchema, @@ -58,7 +60,6 @@ import { } from "./validate.ts"; import { useParseCompiler } from "./components/parse-schema.ts"; import { - documentOutline, isFunctionComponentPath, parseMarkdownDefinition, parseRootMarkdownDefinition, @@ -503,20 +504,6 @@ function isFunctionComponent(value: unknown): value is FunctionComponent { return typeof value === "function"; } -/** - * What one run's selector decided: the whole document, one exact section, or a - * failure that named none. - * - * Selection is compared as an outcome rather than as a target string, because a - * failed selection is an outcome too. Without the third case a journal written - * by one selector that matched nothing would answer a later request for a - * section that does exist. - */ -type SelectionOutcome = - | { kind: "whole" } - | { kind: "exact"; target: string } - | { kind: "failed"; failure: DocumentTargetFailure }; - /** * What the fixed diagnostic says when a recorded root import cannot be read, * and all it says. @@ -540,174 +527,6 @@ const DIFFERENT_PRE_ROOT_DOCUMENT = const ROOT_COROUTINE = "root"; -/** - * What a recorded event turned out to be. - * - * "Not the root import" and "the root import, malformed" are deliberately - * different answers. Collapsing them into one absent value is what would let a - * corrupted record fall through to the recorded terminal result, which is the - * failure this distinction exists to prevent. - */ -type RootImportRecord = - | { kind: "unrelated" } - | { kind: "malformed" } - | { kind: "read"; outline: DocumentOutline; selection: SelectionOutcome }; - -const UNRELATED: RootImportRecord = { kind: "unrelated" }; -const MALFORMED: RootImportRecord = { kind: "malformed" }; - -/** - * Read a value that may refuse to be read. - * - * Every value this boundary touches comes from the journal, and a journal is - * data: a property may be an accessor that throws, a key list may come from a - * Proxy that refuses, and content may be markdown whose frontmatter no parser - * accepts. None of those is a failure of this run — they are ways of saying the - * record cannot be read — so none of them may travel as an error of its own. - * - * Synchronous throughout, so nothing an Effection scope owns passes through - * here: this cannot swallow a cancellation or a durability failure, because - * neither can arise inside a synchronous parse. - */ -function attempt(read: () => T): T | undefined { - try { - return read(); - } catch { - return undefined; - } -} - -/** - * Parse a recorded root import as a closed protocol. - * - * Two selection shapes are supported and nothing else: a repository selection - * with an optional canonical target, and a failed selection with an exact - * failure record. An unknown kind, a missing or mistyped member, an extra - * member, a noncanonical target, and failure data that no selection could have - * produced are each malformed rather than absent. - * - * A result that is not `ok` is left alone. A root import can fail for reasons - * that have nothing to do with selection — an unreadable file — and those - * recorded failures are not this protocol's to interpret. - */ -/** - * A value the journal refused to produce. - * - * Distinct from `undefined`, which is an ordinary absent value. Reading a - * member and finding nothing there, and reading a member that will not say what - * is there, are different facts about a record, and one of them is a refusal: - * conflating them is how "the root import will not say what it settled to" - * became "this is not the root import" and fell through to terminal-result - * reuse. - */ -const UNREADABLE: unique symbol = Symbol("unreadable"); - -/** One read of journal-controlled data: its value, or a refusal. */ -function read(get: () => T): T | typeof UNREADABLE { - try { - return get(); - } catch { - return UNREADABLE; - } -} - -/** The settlements the protocol recognizes as an ordinary failed root import. */ -const SETTLED_FAILURES: readonly string[] = ["err", "cancelled"]; - -function recordedRootImport(event: Yield): RootImportRecord { - // Identification first. An event that will not say what it is cannot be - // claimed as the root import, so it stays unrelated. - const description = read(() => event.description); - if (description === UNREADABLE) { - return UNRELATED; - } - const type = read(() => description.type); - const name = read(() => description.name); - if (type !== "import_component" || name !== "__root__") { - return UNRELATED; - } - - // Identified. From here the event owes this protocol an answer, and every way - // of not giving one is malformed — except the ordinary failed settlement, - // which is a root import that failed for reasons selection knows nothing - // about. - const result = read(() => event.result); - if (result === UNREADABLE || typeof result !== "object" || result === null) { - return MALFORMED; - } - const status = read(() => result.status); - if (status !== "ok") { - return typeof status === "string" && SETTLED_FAILURES.includes(status) ? UNRELATED : MALFORMED; - } - const value = read(() => ("value" in result ? result.value : undefined)); - if (value === UNREADABLE || value === undefined) { - return MALFORMED; - } - return attempt(() => readRootSelection(value)) ?? MALFORMED; -} - -function readRootSelection(value: unknown): RootImportRecord { - // Parsed rather than read in place. `parseJson` walks every property once and - // rebuilds the record, so a trap that throws or a value that is not JSON is - // discovered here — and every read below is of this run's own copy rather - // than of an object the journal still controls. - const record = parseJson(value); - if (!isJsonObject(record)) { - return MALFORMED; - } - const content = record["content"]; - const path = record["path"]; - if (typeof content !== "string" || typeof path !== "string") { - return MALFORMED; - } - const kind = record["kind"]; - const members = Object.keys(record).length; - // Parsing the recorded content is part of reading the record, for every - // shape. It is what the verification below compares against, and doing it - // here means a later read of the same content cannot be the first to - // discover that it does not parse. - const outline = documentOutline(path, content); - - if (kind === "repository") { - const target = record["target"]; - if (target === undefined) { - return members === 3 ? { kind: "read", outline, selection: { kind: "whole" } } : MALFORMED; - } - if (members !== 4 || typeof target !== "string" || !isCanonicalTarget(target)) { - return MALFORMED; - } - // The recorded content is here, so the target is verified against it rather - // than merely parsed: a well-formed target the recorded document does not - // offer describes a selection that never happened. - const resolved = findTarget(outline, target); - if (!resolved.ok || resolved.value.target !== target) { - return MALFORMED; - } - return { kind: "read", outline, selection: { kind: "exact", target } }; - } - - if (kind === "target-failure") { - const failure = recordedDocumentTargetFailure(record["failure"]); - if (members !== 4 || failure === undefined) { - return MALFORMED; - } - // Same standard for a failure: the recorded selector must fail against the - // recorded content in exactly the way the record claims. That verifies the - // catalog and the matches too, which no amount of shape checking could. - const rederived = findTarget(outline, failure.selector); - if (rederived.ok) { - return MALFORMED; - } - const actual = asDocumentTargetError(rederived.error); - if (actual === undefined || !sameDocumentTargetFailure(actual.data, failure)) { - return MALFORMED; - } - return { kind: "read", outline, selection: { kind: "failed", failure } }; - } - - return MALFORMED; -} - /** * What this run's selector decides against the outline the journal recorded. * diff --git a/packages/core/src/root-selection.ts b/packages/core/src/root-selection.ts new file mode 100644 index 000000000..5af9217ea --- /dev/null +++ b/packages/core/src/root-selection.ts @@ -0,0 +1,225 @@ +/** + * The retained root-import protocol (spec §7). + * + * One recorded event decides what a resumed or replayed run is allowed to be a + * continuation of, and reading it is not shape checking. The record has to say + * which document was selected, and the recorded document has to agree: markdown + * that no parser accepts, a target the document does not offer, and a recorded + * failure the same selector would not produce are each a record no execution + * wrote, not a record with a missing field. + * + * It lives on its own because two callers depend on the same answer. Canonical + * execution admits a partial history through it, and a host that reads a + * retained journal — the workflow package, which decides from the same events + * whether a run may publish an outcome or be replayed — reaches it through + * `@executablemd/core/host`. A second reading of the same durable value would + * be a second, weaker protocol, and the day the two stopped agreeing is the day + * a forged selection became executable. + * + * Everything here is synchronous and pure. Nothing an Effection scope owns + * passes through, so no cancellation and no durability failure can be swallowed + * by a parse. + */ + +import type { Yield } from "@executablemd/durable-streams"; +import { isJsonObject, parseJson } from "./json.ts"; +import { documentOutline } from "./definition.ts"; +import { + asDocumentTargetError, + findTarget, + isCanonicalTarget, + recordedDocumentTargetFailure, + sameDocumentTargetFailure, +} from "./document-targets.ts"; +import type { DocumentOutline, DocumentTargetFailure } from "./document-targets.ts"; + +/** + * What one run's selector decided: the whole document, one exact section, or a + * failure that named none. + * + * Selection is compared as an outcome rather than as a target string, because a + * failed selection is an outcome too. Without the third case a journal written + * by one selector that matched nothing would answer a later request for a + * section that does exist. + */ +export type SelectionOutcome = + | { kind: "whole" } + | { kind: "exact"; target: string } + | { kind: "failed"; failure: DocumentTargetFailure }; + +/** + * What a recorded event turned out to be. + * + * "Not the root import" and "the root import, malformed" are deliberately + * different answers. Collapsing them into one absent value is what would let a + * corrupted record fall through to the recorded terminal result, which is the + * failure this distinction exists to prevent. + */ +export type RootImportRecord = + | { kind: "unrelated" } + | { kind: "malformed" } + | { + kind: "read"; + /** The document the record is about, as its own parsed copy. */ + path: string; + content: string; + outline: DocumentOutline; + selection: SelectionOutcome; + }; + +export const UNRELATED: RootImportRecord = { kind: "unrelated" }; +export const MALFORMED: RootImportRecord = { kind: "malformed" }; + +/** + * Read a value that may refuse to be read. + * + * Every value this boundary touches comes from the journal, and a journal is + * data: a property may be an accessor that throws, a key list may come from a + * Proxy that refuses, and content may be markdown whose frontmatter no parser + * accepts. None of those is a failure of this run — they are ways of saying the + * record cannot be read — so none of them may travel as an error of its own. + * + * Synchronous throughout, so nothing an Effection scope owns passes through + * here: this cannot swallow a cancellation or a durability failure, because + * neither can arise inside a synchronous parse. + */ +export function attempt(read: () => T): T | undefined { + try { + return read(); + } catch { + return undefined; + } +} + +/** + * Parse a recorded root import as a closed protocol. + * + * Two selection shapes are supported and nothing else: a repository selection + * with an optional canonical target, and a failed selection with an exact + * failure record. An unknown kind, a missing or mistyped member, an extra + * member, a noncanonical target, and failure data that no selection could have + * produced are each malformed rather than absent. + * + * A result that is not `ok` is left alone. A root import can fail for reasons + * that have nothing to do with selection — an unreadable file — and those + * recorded failures are not this protocol's to interpret. + */ +/** + * A value the journal refused to produce. + * + * Distinct from `undefined`, which is an ordinary absent value. Reading a + * member and finding nothing there, and reading a member that will not say what + * is there, are different facts about a record, and one of them is a refusal: + * conflating them is how "the root import will not say what it settled to" + * became "this is not the root import" and fell through to terminal-result + * reuse. + */ +export const UNREADABLE: unique symbol = Symbol("unreadable"); + +/** One read of journal-controlled data: its value, or a refusal. */ +export function read(get: () => T): T | typeof UNREADABLE { + try { + return get(); + } catch { + return UNREADABLE; + } +} + +/** The settlements the protocol recognizes as an ordinary failed root import. */ +const SETTLED_FAILURES: readonly string[] = ["err", "cancelled"]; + +export function recordedRootImport(event: Yield): RootImportRecord { + // Identification first. An event that will not say what it is cannot be + // claimed as the root import, so it stays unrelated. + const description = read(() => event.description); + if (description === UNREADABLE) { + return UNRELATED; + } + const type = read(() => description.type); + const name = read(() => description.name); + if (type !== "import_component" || name !== "__root__") { + return UNRELATED; + } + + // Identified. From here the event owes this protocol an answer, and every way + // of not giving one is malformed — except the ordinary failed settlement, + // which is a root import that failed for reasons selection knows nothing + // about. + const result = read(() => event.result); + if (result === UNREADABLE || typeof result !== "object" || result === null) { + return MALFORMED; + } + const status = read(() => result.status); + if (status !== "ok") { + return typeof status === "string" && SETTLED_FAILURES.includes(status) ? UNRELATED : MALFORMED; + } + const value = read(() => ("value" in result ? result.value : undefined)); + if (value === UNREADABLE || value === undefined) { + return MALFORMED; + } + return attempt(() => readRootSelection(value)) ?? MALFORMED; +} + +function readRootSelection(value: unknown): RootImportRecord { + // Parsed rather than read in place. `parseJson` walks every property once and + // rebuilds the record, so a trap that throws or a value that is not JSON is + // discovered here — and every read below is of this run's own copy rather + // than of an object the journal still controls. + const record = parseJson(value); + if (!isJsonObject(record)) { + return MALFORMED; + } + const content = record["content"]; + const path = record["path"]; + if (typeof content !== "string" || typeof path !== "string") { + return MALFORMED; + } + const kind = record["kind"]; + const members = Object.keys(record).length; + // Parsing the recorded content is part of reading the record, for every + // shape. It is what the verification below compares against, and doing it + // here means a later read of the same content cannot be the first to + // discover that it does not parse. + const outline = documentOutline(path, content); + + if (kind === "repository") { + const target = record["target"]; + if (target === undefined) { + return members === 3 + ? { kind: "read", path, content, outline, selection: { kind: "whole" } } + : MALFORMED; + } + if (members !== 4 || typeof target !== "string" || !isCanonicalTarget(target)) { + return MALFORMED; + } + // The recorded content is here, so the target is verified against it rather + // than merely parsed: a well-formed target the recorded document does not + // offer describes a selection that never happened. + const resolved = findTarget(outline, target); + if (!resolved.ok || resolved.value.target !== target) { + return MALFORMED; + } + return { kind: "read", path, content, outline, selection: { kind: "exact", target } }; + } + + if (kind === "target-failure") { + const failure = recordedDocumentTargetFailure(record["failure"]); + if (members !== 4 || failure === undefined) { + return MALFORMED; + } + // Same standard for a failure: the recorded selector must fail against the + // recorded content in exactly the way the record claims. That verifies the + // catalog and the matches too, which no amount of shape checking could. + const rederived = findTarget(outline, failure.selector); + if (rederived.ok) { + return MALFORMED; + } + const actual = asDocumentTargetError(rederived.error); + if (actual === undefined || !sameDocumentTargetFailure(actual.data, failure)) { + return MALFORMED; + } + return { kind: "read", path, content, outline, selection: { kind: "failed", failure } }; + } + + return MALFORMED; +} diff --git a/packages/workflow/src/lifecycle/policy.ts b/packages/workflow/src/lifecycle/policy.ts index 58d57090d..6acc646e4 100644 --- a/packages/workflow/src/lifecycle/policy.ts +++ b/packages/workflow/src/lifecycle/policy.ts @@ -14,6 +14,8 @@ */ import type { DurableEvent } from "@executablemd/durable-streams"; +import { recordedRootImport } from "@executablemd/core/host"; +import type { SelectionOutcome } from "@executablemd/core/host"; import { WorkflowRequestError } from "../storage/errors.ts"; import type { DocumentExecutionCompletion, @@ -127,9 +129,16 @@ export function rootOutcome(entries: readonly JournalEntry[]): RetainedTerminal if (imported.kind !== "one") { return { kind: "damaged" }; } - return document.kind === "ok" - ? { kind: "outcome", status: "completed", reason: undefined } - : { kind: "outcome", status: "failed", reason: retainedFailureReason(entries) }; + if (document.kind === "ok") { + // The selection has to be able to lead to the result beside it. A recorded + // selection failure is a document that never ran: canonical execution + // raises it out of the root import, so the only terminal it can reach is a + // failed one. A successful result over it is two histories, not one. + return imported.selection.failed + ? { kind: "damaged" } + : { kind: "outcome", status: "completed", reason: undefined }; + } + return { kind: "outcome", status: "failed", reason: retainedFailureReason(entries) }; } /** The outcome a run that failed before importing anything recorded. */ @@ -182,65 +191,73 @@ function namesRootImport(event: DurableEvent): boolean { /** * The document one retained root import selected, as its own record holds it. * - * The shapes canonical execution writes for a root and no others: the whole - * document, one exact target, and a selector the document offered no single - * target for. A settlement that failed recorded no selection at all, and a - * successful one that does not hold a complete selection is a root import this - * build cannot read — which is a fact about the history, not about the caller - * asking, so it is decided here rather than by whoever asks next. + * Enough of the parsed record to make the same request again, and no more: the + * document itself, the selector it was asked with, and whether that selector + * named a target at all. What proved the record — the outline it was verified + * against — stays with the parser. */ export interface RetainedRootSelection { readonly path: string; readonly content: string; + /** + * The selector to replay the run's request with: the exact target it ran, or + * the selector whose failure it recorded. Absent for a whole document. + */ readonly target: string | undefined; + /** Whether the recorded selection is one that named no single target. */ + readonly failed: boolean; } +/** + * The selection one retained root import recorded, read the way canonical + * execution reads it. + * + * Not read here at all, in fact: `recordedRootImport()` is the parser canonical + * `admitRootSelection()` admits a partial history through, and this asks it the + * same question about the same event. A record it calls malformed is malformed + * for the lifecycle too, so an unparseable document, a target the retained + * document does not offer, a noncanonical target, and a failure record the same + * selector would not re-derive cannot publish an outcome here after the + * executor refused them there. + * + * What comes back is that parser's own copy of the record, so the document a + * replay is built from is never the object the journal still holds. + */ function rootSelection(event: DurableEvent): RetainedRootSelection | undefined { - const settlement = plain(event.type === "yield" ? event.result : undefined); - if (settlement === undefined || settlement["status"] !== "ok") { + if (event.type !== "yield") { return undefined; } - const selection = plain(settlement["value"]); - if (selection === undefined) { + const recorded = recordedRootImport(event); + if (recorded.kind !== "read") { return undefined; } - const path = selection["path"]; - const content = selection["content"]; - const kind = selection["kind"]; - const members = Object.keys(selection).length; - if (typeof path !== "string" || typeof content !== "string") { - return undefined; - } - - if (kind === "repository") { - const target = selection["target"]; - if (target === undefined) { - return members === 3 ? { path, content, target: undefined } : undefined; - } - return members === 4 && typeof target === "string" ? { path, content, target } : undefined; - } - - if (kind === "target-failure") { - // The selector as the run was asked for it. Handing it back is what makes - // the replayed request the same request: canonical execution resolves it - // against the recorded document again and finds the same failure. - const failure = plain(selection["failure"]); - const selector = failure?.["selector"]; - return members === 4 && typeof selector === "string" - ? { path, content, target: selector } - : undefined; - } - - return undefined; + return { + path: recorded.path, + content: recorded.content, + target: selector(recorded.selection), + failed: recorded.selection.kind === "failed", + }; } /** - * The document a terminal written before any import was about. + * What a replay asks for to make the same request again. * - * Read from the binding `readPreRootTerminal()` above has already held to its - * exact closed form, so this only takes what that admitted: nothing decides - * here that was not decided there. + * A recorded failure hands back the selector rather than nothing: canonical + * execution resolves it against the same retained document, finds the same + * failure, and fails the same way. Handing back nothing would ask for the whole + * document instead — a different request that would succeed. */ +function selector(selection: SelectionOutcome): string | undefined { + switch (selection.kind) { + case "whole": + return undefined; + case "exact": + return selection.target; + case "failed": + return selection.failure.selector; + } +} + export function preRootSelection( entries: readonly JournalEntry[], ): RetainedRootSelection | undefined { @@ -260,7 +277,15 @@ export function preRootSelection( if (typeof path !== "string" || typeof source !== "string") { return undefined; } - return { path, content: source, target: typeof target === "string" ? target : undefined }; + return { + path, + content: source, + target: typeof target === "string" ? target : undefined, + // A binding is what a run that failed *before* importing recorded, so it + // holds the document it was asked for rather than the outcome of selecting + // in it. There is no recorded selection failure to disagree with. + failed: false, + }; } /** The coroutine a document execution's own records belong to. */ diff --git a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts index 891d4e525..e3524bc96 100644 --- a/packages/workflow/tests/cloudflare/remote-replay.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-replay.vitest.ts @@ -781,6 +781,94 @@ describe("a completed run replayed through its own owner", () => { expect(await on(stub, (owner) => owner.holders())).toBe(0); }); + it("refuses a recorded selection the retained document does not bear out", async () => { + /** Selections that are well formed and that no execution could have made. */ + const forged: Record = { + absentTarget: { + kind: "repository", + path: "README.md", + content: DOCUMENT, + target: "Missing", + }, + selectorOnly: { + kind: "target-failure", + path: "README.md", + content: DOCUMENT, + failure: { selector: "Missing" }, + }, + }; + + for (const [says, selection] of Object.entries(forged)) { + const stub = executor(); + const host = lifecycleHost(stub); + + await run(function* () { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const begun = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + if (!begun.ok) { + throw begun.error; + } + const appended = yield* begun.value.database.transact(function* (transaction) { + yield* transaction.journal.append({ + type: "yield", + coroutineId: "root", + description: { type: "import_component", name: "__root__" }, + result: { status: "ok", value: selection }, + }); + yield* transaction.journal.append({ + type: "close", + coroutineId: "root", + result: { + status: "ok", + value: { status: "ok", output: "done.\n", value: "done.\n" }, + }, + }); + }); + if (!appended.ok) { + throw appended.error; + } + }); + }); + + const before = await ownerState(stub); + expect([says, before.run?.["status"]]).toEqual([says, "running"]); + const heldBefore = await on(stub, (owner) => owner.executionRows()); + + const refusals = await run(function* (): Operation> { + return yield* scoped(function* () { + const transitions: WorkflowExecutionTransitions = yield* useRemoteLifecycle(host); + const lock = yield* acquired(); + const started = yield* transitions.begin(lock, { + runId: RUN_ID, + action: "start", + creation: CREATION, + }); + const resumed = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + return { + started: started.ok ? "admitted" : started.error.message, + resumed: resumed.ok ? "admitted" : resumed.error.message, + }; + }); + }); + + for (const said of Object.values(refusals)) { + expect([says, String(said).includes("cannot read")]).toEqual([says, true]); + } + + // Recovery published nothing and no envelope was inserted for a run whose + // own record does not say what it selected. + expect([says, await ownerState(stub)]).toEqual([says, before]); + expect([says, await on(stub, (owner) => owner.executionRows())]).toEqual([says, heldBefore]); + expect([says, await on(stub, (owner) => owner.holders())]).toEqual([says, 0]); + } + }); + it("refuses a root import no single execution recorded, and moves nothing", async () => { const stub = executor(); const host = lifecycleHost(stub); diff --git a/packages/workflow/tests/replay-inputs.test.ts b/packages/workflow/tests/replay-inputs.test.ts index 08f01865b..84239ae98 100644 --- a/packages/workflow/tests/replay-inputs.test.ts +++ b/packages/workflow/tests/replay-inputs.test.ts @@ -38,6 +38,15 @@ const ROOT_ID = "a".repeat(64); const COMMIT = "0".repeat(40); const SOURCE = "# Retained\n\ndone.\n"; +/** + * The same document, with a section in it. + * + * An exact target is verified against the retained document, so a fixture that + * records one has to record a document that offers it. `SOURCE` offers none — + * which is what makes it the right document for a recorded selection *failure*. + */ +const SECTIONED = "# Retained\n\ndone.\n\n## Stage\n\nstaged.\n"; + function record( overrides: { readonly status?: WorkflowRunStatus; @@ -179,14 +188,19 @@ describe("the root a completed run replays on", () => { it("carries the exact target the import resolved to", function* () { const history = [ entry( - rootImport({ kind: "repository", path: "flows/root.md", content: SOURCE, target: "Stage" }), + rootImport({ + kind: "repository", + path: "flows/root.md", + content: SECTIONED, + target: "Stage", + }), ), entry(rootClose({ status: "ok", output: "", value: "" })), ]; expect(admitted(retainedReplay(record(), history)).root).toEqual({ path: "flows/root.md", - source: SOURCE, + source: SECTIONED, retained: true, target: "Stage", }); @@ -200,7 +214,9 @@ describe("the root a completed run replays on", () => { kind: "target-failure", path: "flows/root.md", content: SOURCE, - failure: { kind: "no-match", selector: "Missing*", matches: [], available: ["Retained"] }, + // The record as the selector actually fails against this document: + // it offers no targets at all, so the catalog is empty. + failure: { kind: "no-match", selector: "Missing*", matches: [], available: [] }, }), ), entry(rootClose(documentFailure())), @@ -808,6 +824,180 @@ describe("what the root recorded, as one outcome", () => { expect(retainedReplay(record(), failed).ok).toBe(false); }); + // deno-lint-ignore require-yield + it("verifies a recorded selection against the document it recorded", function* () { + const succeeded = { status: "ok", output: "done.\n", value: "done.\n" }; + const unreadable: { says: string; selection: Json }[] = [ + // An exact target the retained document does not offer. Well-formed, and + // a selection that never happened. + { + says: "a target this document has none of", + selection: { + kind: "repository", + path: "flows/root.md", + content: SECTIONED, + target: "Missing", + }, + }, + // The same target, spelled a way canonical encoding does not use. + { + says: "a target that is not canonically encoded", + selection: { + kind: "repository", + path: "flows/root.md", + content: SECTIONED, + target: "stage", + }, + }, + { + says: "a target carrying an encoding the canonical form never has", + selection: { + kind: "repository", + path: "flows/root.md", + content: SECTIONED, + target: "Stage%20", + }, + }, + ]; + + for (const { says, selection } of unreadable) { + const entries = [entry(rootImport(selection)), entry(rootClose(succeeded))]; + expect([says, rootOutcome(entries)?.kind]).toEqual([says, "damaged"]); + expect([says, reason(retainedReplay(record(), entries)).includes("cannot be read")]).toEqual([ + says, + true, + ]); + } + + // Markdown the canonical parser refuses is a document nothing could have + // been selected in. Each boundary gets its own copy of it: `gray-matter` + // caches an empty parse for a source whose frontmatter it just threw on, so + // the refusal is what the *first* read of those bytes answers, and two + // reads of one document would not be two reads of one refusal. + const unparsed = (frontmatter: string): JournalEntry[] => [ + entry( + rootImport({ + kind: "repository", + path: "flows/root.md", + content: `---\n${frontmatter}\n---\n\n# Root\n`, + }), + ), + entry(rootClose(succeeded)), + ]; + expect(rootOutcome(unparsed("returns: [lifecycle"))?.kind).toBe("damaged"); + expect(reason(retainedReplay(record(), unparsed("returns: [admission")))).toContain( + "cannot be read", + ); + + // And the exact target this document does offer, still admitted. + const resolved = [ + entry( + rootImport({ + kind: "repository", + path: "flows/root.md", + content: SECTIONED, + target: "Stage", + }), + ), + entry(rootClose(succeeded)), + ]; + expect(rootOutcome(resolved)?.kind).toBe("outcome"); + expect(admitted(retainedReplay(record(), resolved)).root).toEqual({ + path: "flows/root.md", + source: SECTIONED, + retained: true, + target: "Stage", + }); + }); + + // deno-lint-ignore require-yield + it("holds a recorded selection failure to the failure it re-derives", function* () { + /** The record `"Missing"` actually produces against `SECTIONED`. */ + const derived = { kind: "no-match", selector: "Missing", matches: [], available: ["Stage"] }; + + function failed(failure: Json): JournalEntry[] { + return [ + entry( + rootImport({ + kind: "target-failure", + path: "flows/root.md", + content: SECTIONED, + failure, + }), + ), + entry(rootClose(documentFailure())), + ]; + } + + const forged: { says: string; failure: Json }[] = [ + // A selector is not a failure record, however plausible. + { says: "the selector alone", failure: { selector: "Missing" } }, + { says: "a member short", failure: { kind: "no-match", selector: "Missing", matches: [] } }, + { + says: "a member more", + failure: { ...derived, type: "executablemd.document-target-failure" }, + }, + // The document does fail this selector, but not this way. + { says: "another kind", failure: { ...derived, kind: "invalid-selector" } }, + { says: "another catalog", failure: { ...derived, available: ["Stage", "Other"] } }, + { says: "an emptied catalog", failure: { ...derived, available: [] } }, + { says: "matches nothing matched", failure: { ...derived, matches: ["Stage"] } }, + // A selector whose real outcome is not a failure at all. + { says: "a selector that resolves", failure: { ...derived, selector: "Stage" } }, + ]; + + for (const { says, failure } of forged) { + const entries = failed(failure); + expect([says, rootOutcome(entries)?.kind]).toEqual([says, "damaged"]); + expect([says, reason(retainedReplay(record(), entries)).includes("cannot be read")]).toEqual([ + says, + true, + ]); + } + + // The re-derived record, admitted, and replayed as the same request. + const coherent = failed(derived); + expect(rootOutcome(coherent)).toEqual({ kind: "outcome", status: "failed", reason: HOST }); + expect( + admitted(retainedReplay(record({ status: "failed", stopReason: HOST }), coherent)).root, + ).toEqual({ + path: "flows/root.md", + source: SECTIONED, + retained: true, + target: "Missing", + }); + }); + + // deno-lint-ignore require-yield + it("agrees only with the terminal a failed selection can reach", function* () { + const failure = { + kind: "no-match", + selector: "Missing", + matches: [], + available: ["Stage"], + }; + const selection: Json = { + kind: "target-failure", + path: "flows/root.md", + content: SECTIONED, + failure, + }; + + // A selection that named no target is raised out of the root import, so the + // document never ran. A successful result over it is two histories. + const succeeded = [ + entry(rootImport(selection)), + entry(rootClose({ status: "ok", output: "done.\n", value: "done.\n" })), + ]; + expect(rootOutcome(succeeded)?.kind).toBe("damaged"); + expect(retainedReplay(record(), succeeded).ok).toBe(false); + + // The failed terminal it can reach, with the reason that exact row names. + const failedRun = [entry(rootImport(selection)), entry(rootClose(documentFailure()))]; + expect(rootOutcome(failedRun)).toEqual({ kind: "outcome", status: "failed", reason: HOST }); + expect(retainedReplay(record({ status: "failed", stopReason: HOST }), failedRun).ok).toBe(true); + }); + // deno-lint-ignore require-yield it("admits the shapes canonical execution actually writes", function* () { const message = "the document refused"; From 08fdd40185133ec683bac29ad0e627ccc0aa34dc Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 08:58:34 -0300 Subject: [PATCH 70/81] =?UTF-8?q?=F0=9F=93=9D=20Record=20what=20the=20remo?= =?UTF-8?q?te=20host=20and=20a=20retained=20terminal=20actually=20do?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The documents still called the internal remote lifecycle, transport, fork, inspection and completed replay unbuilt, and still said the neutral transition types were waiting to be exported. All of it is implemented behind the same four-method host, so the inventories now say what is built and, separately, that nothing yet assembles a configured public client, selector, endpoint, release-identity or OIDC supply — which is the whole of what remains before a caller could choose that host. A finished run also has a contract now, in one place. `specs/workflow-spec.md` §9.9 states what a retained terminal is and what may be concluded from it: one final root `Close`, two layers that each decide something, exactly one root import owned by the root coroutine and parsed by canonical execution's own parser against the document it recorded, damage outranking the stored row, and a coherent replay that changes nothing but the execution envelope it opened. The architecture and the Workspace specification reach that statement rather than restating it, and Tier WRH is no longer described as scenarios without tests: WRH1-WRH16 name the committed suites that prove them, and WRH17-WRH22 stay with the machine-wait work they belong to. Documentation only. No production module, test, dependency, lockfile, generated artifact, export or public API changes here. --- architecture.md | 23 +++++++++++++++------ specs/executable-mdx-spec.md | 31 +++++++++++++++++++++++++--- specs/workflow-spec.md | 35 ++++++++++++++++++++++++++++---- specs/workflow-workspace-spec.md | 27 ++++++++++++++++-------- 4 files changed, 94 insertions(+), 22 deletions(-) diff --git a/architecture.md b/architecture.md index c47208a65..76a5ad7ee 100644 --- a/architecture.md +++ b/architecture.md @@ -439,6 +439,8 @@ transaction, never a partial one. The next acquisition performs the ordinary stale-execution recovery and resumes from the exact committed WorkflowRun and Workspace frontier. A completed run replays as it does anywhere, and reading its own history is what it does: it may reach the durable owner holding its retained result — an ephemeral client has nothing else to replay from — while attaching no Workspace, Agent, process, Git, Git-host, Issue, Project, credential or other external-effect provider, performing no effect again and starting no native operation. Lifecycle storage access is not external-effect replay, and only the second is what a completed run must never do. +What that owner implements, it implements behind the four-method `WorkflowHost` boundary and nothing wider: create and lookup, coherent reads, lifecycle transitions and terminal settlement, stale recovery, fork and its staging, Workspace publication, typed delivery and consumption, read-only inspection and history, and canonical completed replay. Which host a caller gets is not yet a decision anybody can make: there is no configured public client factory, no runtime selector, and no endpoint, release-identity or OIDC supplier assembly, so the shipped CLI assembles the local host only. That assembly is the remaining work, and none of the behavior above waits on it. + ## The workflow lifecycle `xmd workflow start [--id=] [--props-*=…] ` and @@ -667,7 +669,7 @@ the requested lifecycle action and inserts one execution record while publishing `running`. A replay of an already completed or failed root instead inserts its execution record while preserving that terminal run state: the replay observes an outcome that already won and does not make the run mutable -again. The settlement transition finishes the record while publishing its +again. The run row is left exactly what it was, `updatedAt` included, and its own settlement closes only the execution envelope it opened — the one durable change a coherent replay makes. The settlement transition finishes the record while publishing its terminal or resumable status. Begin and settlement are separate transactions enclosing the document execution, and each is atomic within itself. @@ -716,6 +718,12 @@ without entering replay. A `cancelled` run reports its retained state without entering replay under either command. None can be advanced under the same run ID. +A retained terminal is read before it is trusted, and one shared judgment does the reading. Both layers of the root `Close` are load-bearing: the outer coroutine settlement says the root returned, was raised out of, or was cancelled, and a returned value's own `status` decides whether the run completed or failed — so reading the outer layer alone would call every finished document a completed one. The history around that terminal has to agree with it. Exactly one final root `Close` may exist, because a second one, or work recorded after the one that is there, is a history no single execution produced and choosing between them would be this build deciding which execution the run was. A run that failed before importing anything carries its root binding and no import; a run that produced an ordinary document result carries exactly one root import, recorded by the root coroutine, and canonical core's own root-selection parser is what reads it — the same function that admits a partial history, so a retained selection the executor would refuse cannot publish an outcome here instead. That parser proves the selection rather than recognizing its shape: the retained document parses, an exact target resolves against it to the exact recorded target, and a recorded selection failure re-derives from the same selector to the same kind, matches and available catalog. A selection that named no target is raised out of the root import, so a successful document result beside one is two histories rather than one. + +Anything those readings refuse is damaged history rather than an outcome, and damage outranks the stored row. A run whose row says `completed` or `failed` over a terminal this build cannot read is not advanced, not re-settled and not published: `start`, `resume` and `cancel` each refuse before an execution record exists, before an acquisition performs anything and before Git, a Workspace or any provider is reached, and the run keeps its row, its journal, its Workspace frontier and whatever unfinished execution the previous executor left open. Stale recovery reads the same judgment first and publishes nothing over damaged history. Both hosts reach these conclusions through the one provider-neutral lifecycle policy; `specs/workflow-spec.md` §9.9 is the normative statement of them. + +A coherent terminal is also where a completed replay gets its inputs. Nothing is fetched for it: the root document is the one the run's own root import retained, the bundle admission is rebuilt from the definition rather than from a repository, and each retained component's bytes are authenticated by naming them the way Git names a blob and comparing that to the object id the definition already holds. So the replay reaches no repository, no working tree and no live component import, and a history whose retained root does not agree with the run's own definition path refuses before anything is replayed from it. + The executor lock remains in the trusted host's outer lifecycle scope; it is not installed as a document provider. A completed replay may hold the lock while recording its document-execution envelope, while the run remains @@ -790,6 +798,8 @@ ordered rows — authored source, cumulative forkability, inherited provenance because two mappings are how history quietly starts meaning different things depending on where it was read. +A durable owner satisfies that surface from one committed reading of its own state: it takes no acquisition, attaches nothing, materializes nothing, appends nothing, and answers status, list and history from the same projection the local provider runs, so a malformed or incomplete page fails the request whole rather than returning part of one. Everything in the rest of this section is the local provider's, because a hot rollback journal and the coordination that recovers one are facts about a file. + The Deno provider first reads each retained database through its ordinary read-only connection. A host crash may leave a hot rollback journal: that is a healthy database one SQLite rollback away from the last committed state, but @@ -3467,6 +3477,7 @@ Status is measured against main. | workflow component bundle | a workflow root declares a closed set of authored Markdown components; the V1 workflow definition optionally carries them as one array sorted by component name, each entry holding the name, its canonical repository-relative path inside the pinned commit and that blob's object ID, and an absent member identifies a run closed over no components — so a definition retained before the member existed reads unchanged. `start` and `resume` read every component from the definition's own pinned commit; the array takes part in definition identity and is compared as part of the same V1 descriptor in compatible reuse; and canonical core resolves those names and holds both live import and retained history to that exact bundle | built on the #301 stack; the full adversarial implementation loop and its scheduling remain unbuilt (#300), and generated XMD admits no bundled Markdown component (#369) | | `workflowInstallation()` / `getWorkflowRun()` | associates one document execution with a workflow run, through an `ExecutionInstallation` the trusted host passes to `executeInstalled()` | built on the #366 stack | | `retainedWorkflowInstallation()` | associates one document execution with a run storage already created, requiring exact journal agreement | built on the #366 stack | +| `retainedReplay()` | assembles what a completed run replays on from the owner's committed state alone — the root document its own root import retained, and the admissions that hold the retained history to the run's immutable definition and its component bundle, each component's bytes authenticated by Git blob identity against the object id the definition holds. It reaches no repository, working tree, provider or live import, and a history it cannot read refuses before anything is replayed from it | built on the #698 stack, both providers | | `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main | | workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; the CLI lifecycle reaches it on the #366 stack | | caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in | built on main | @@ -3495,11 +3506,11 @@ Status is measured against main. | `Git.Push` | publishes the selected checkout's exact current named branch and commit to the same branch on the retained Repository's canonical `origin`, reconciled through the shared Git-host state machine rather than through a Workspace transaction: no props and no component result, no force, no upstream mutation and no implicit staging or committing; the durable request and record carry the Repository's filtered identity without its checkout path, and the transport runs in a provider-owned isolated control repository reading the checkout's objects through an object-source attachment whose alternates chain and object tree are proven contained before the first remote observation, aimed at the exact private retained locator. A destination proven absent is published to once and one already naming this exact commit is adopted; one naming a distinct commit that same authenticated source proves is in this commit's ancestry is a performable pre-state, published over by the same exact non-force refspec and retained as the predecessor with the attested relation, while a divergent commit and one the source cannot read are both conflicts and nothing is fetched to decide either; a completed Push is reconstructed from the Workspace root its own journal event was appended against, read without publishing it or moving the run's frontier, so a branch published more than once resumes | built on the #370 stack, Deno provider only | | `` | upserts one pull request of the selected checkout's current named branch, reconciled through the shared Git-host state machine: a required `title`, an optional positive-integer `number`, an optional `base` defaulting to the Repository's retained initial branch, an optional `draft`, and the rendered content as the body; it renders nothing and returns stable evidence through `as` — the filtered Repository identity, the provider's own stable pull-request identity, number, URL, open state, and the head and base SHAs of the snapshot it finished at. Without a number it creates one pull request for the head/base pair or adopts the compatible one an interrupted attempt left; with a number it brings that exact pull request's title, body, draft state and base to what the request says, records a no-op when they already match, and refuses a number belonging to another repository, opened from another head, or no longer open. It never pushes, never rewrites a head, and never reopens, merges or comments. The run must already hold its own successful `Git.Push` result for that exact Repository identity, head branch, destination ref and commit — proven by a scan of the whole successful history that requires each relevant record's natural key, inputs and result to describe one publication; a branch is published repeatedly, so the whole history is read in order and the run's last publication of that branch decides — an earlier one behind it is history rather than disagreement, while a last one naming another commit is the branch having moved on; that is conflicting, no relevant record at all is missing, and a relevant record that cannot be read whole is unreadable, each failing locally before the Git host is observed; the first adapter works over `github.com` on REST plus the two GraphQL draft transitions, selected from the private retained locator, credentialed from `GH_TOKEN`, then `GITHUB_TOKEN`, then the machine's own `gh` login, issuing each required mutation at most once per attempt and deciding the outcome by one observation, with the locator, endpoint, credential and payload confined to the per-invocation provider closure | built on the #295 stack, Deno provider only | | `` | asks one of two questions, decided by its own shape, through a boundary of its own rather than the Git host's. Self-closing with `url` reads that issue and binds `{ url, title, description, tags, assignee }`; paired with `title` upserts and binds exactly `{ url }`, its rendered content being the description. There is no `description` prop. Props are exactly `url`, `title`, optional `tags`, optional `assignee` and — on a read only — optional `provider`; no repository/token/label/milestone/project/comment/close or approval prop. Both forms render nothing. The form is decided before the tracker is read, before any provider is asked and before an `issue_effect` record exists, and that is where a mixed `url`+`title`, a read carrying content or `tags`/`assignee`, an upsert with no content, an upsert naming a `provider`, and an element that is neither are all refused. A read needs no tracker — its URL is the identity; an upsert requires the nearest lexical `` and takes its discriminator only from there. The tracker carries a credential-free `url` and an optional `provider`; the URL is canonicalized — a credential, a query and a fragment are refused rather than stripped — and a nested tracker replaces the whole value for its descendants, never merging members, with the enclosing one restored on leaving. It is composition data, not authority: the provider holds an adapter-private ceiling beside its credentials, admitted before it connects, so a target outside it sends nothing. One stable contextual operation, `executablemd.workflow.issue`, with `read(url, options)` and `upsert(issue, options)`; a provider is ordinary middleware around it, matching its own URLs without a discriminator and only its own name with one, independently per member, with no host-side resolution. Once middleware matches it owns the answer — it never delegates afterwards, and nothing catches its refusal to try somebody else — and a request everyone delegated reaches `NoIssueProvider` unchanged. `issue_effect` records an operation discriminator with the normalized request and result; both forms replay without reaching `IssueApi` and therefore without network access; only an upsert derives an idempotency key, from the operation, the canonical target and the run's own effect identity. Retention excludes credentials, endpoints, payloads, provider identities, origin markers and host paths. Observing, adopting, creating once and recovering an interrupted creation are the provider's, because they are knowledge about what a service can prove; title is never identity, and tags are a code-point-sorted set. The Deno workflow host installs configured GitHub middleware and installs none otherwise, so absence of configuration is fail-closed | built on the #296 stack; GitHub middleware, Deno host | -| workflow lifecycle inspection and control | reads status/list/history without advancing a run, recovering a private copy when a crashed source needs rollback; enforces the executor lock, refuses live cancellation, cancels non-live runs under that lock and deletes retained state | direct read-only inspection and control built on the #367 stack; coordinated recovered inspection built on the #513 stack, Deno provider only | +| workflow lifecycle inspection and control | reads status/list/history without advancing a run, recovering a private copy when a crashed source needs rollback; enforces the executor lock, refuses live cancellation, cancels non-live runs under that lock and deletes retained state | direct read-only inspection and control built on the #367 stack; coordinated recovered inspection built on the #513 stack, Deno provider only; the durable owner answers the same status, list and history questions from one committed reading, taking no acquisition and appending nothing | | XMD artifact export, inspection and fork source | seals one run's committed retained state, Workspace roots and workflow definition source closure into one immutable `.xmd` evidence file; opens that file read-only for status/history and admits continuation only by creating a new history fork whose lineage names the artifact identity | specified by `specs/xmd-artifact-spec.md`; the version-1 sealed container, its total read-only verifier, `xmd workflow export` and artifact `status`/`history` are built, Deno provider only — the artifact-source fork remains unbuilt. Inspection is two sibling lifecycle operations, `inspectArtifact()` and `historyArtifact()`, taking a path rather than a run id: a run id names live lifecycle authority and a path names immutable evidence, so neither is a mode of the other. They reach no run store, lock, Workspace, definition reader or external provider, and the artifact path never enters the structural answer | | Agent session portability evidence in an XMD artifact | classifies every logical Agent session that contributed a retained Prompt as portable — with ordered provider checkpoint tokens and an opaque Agent session bundle — or as explicitly unavailable, as two content kinds inside the existing version-1 manifest and identity | specified by `specs/xmd-artifact-spec.md` §2.5; the closed union, both content kinds and the complete post-identity profile verifier are built on the #621 stack, Deno provider only. Provider bundle capture, Agent-aware export, intrinsic Agent-aware inspection and artifact-backed fork are unbuilt | | historical authored source | retains an authored durable operation's normalized `SourcePosition` beside its identity, and history parses it or refuses the entry | built on the #367 stack | -| history fork | creates a new run from one compatible checkpoint and retained Workspace root, under a new immutable definition and normalized props | built on the #368 stack, Deno provider only | +| history fork | creates a new run from one compatible checkpoint and retained Workspace root, under a new immutable definition and normalized props | built on the #368 stack; both providers — the durable owner copies the selected prefix and roots into a destination that commits whole or not at all, and neither side is mutated when the fork is refused | | workflow Agent session | a workflow document's `` runs under a profile the host attaches only for a live or partial run: an empty host-owned working directory instead of any Workspace, checkout or caller path, no MCP servers, an empty requested native tool set, and `deny-all` with a permission path that denies every native request and fails the turn that asked without reaching the public permission chain. Within a run a session is identified by the Agent/Session expansion identity the engine derived — the authored name is descriptive, so two sibling `` elements are two sessions — routed inside a placement bound to its element and good for one use, so a kept placement cannot be substituted for the next. The conversation is retained as a row in the run's own database with the provider, resolved agent command and policy fingerprint beside it as compatibility attributes. The order is provider creation, the provider's canonical tagged assertion, the mapping commit, then the first Prompt; occupancy of a provider key is not an assertion, the pre-commit window reconciles only from exactly one, and a missing, conflicting, replaced or ambiguous assertion is one explicit refusal that starts no replacement. Deleting a run removes the row with the run and the provider-session directory beside it, and reports the categories. The profile selects ACP-only capability explicitly — no native-launch advertisement and no client-native attachment advertisement — rather than inheriting the provider package's ordinary-run sets by omission, and it supplies no machine session coordinator, construction-route store or executable observer: a workflow session belongs to a run, and the machine-wide account describes a different thing entirely | built on the #302 stack, with the explicit ACP-only selection from #561; the portable proof that an adapter honours an empty tool set is tracked by #496 and does not widen the ceiling | | generated-XMD admission | admits one Agent-generated fragment through the trusted-host seam: host policy is a `read` table and a `write` table of exact pinned identities, each carrying the authored forms it is admitted for, and an authored `allow` selects a canonical non-empty subset of the closed classes — omitted means `read`. The complete source is preflighted inside one `generated_xmd` durable effect before its first generated effect; only the pinned identity the selected classes hold for that name **and** that form executes; and the admitted source, class selection, selected root, every selected entry with its forms, the identity and form of each element named, and the normalized request policy are retained in that effect's own result — so a continuation restores the decision without reading the current candidate and expands only the retained source. The roots are an as-of-admission retained basis checked by membership — the run's own later root publications and an advanced retained current root pass, while a lost admission root or lost selected root refuses — and every non-root term is checked exactly, refusing a run whose classes, identities, forms or requests have moved. The admission and every nested generated effect are offered inline by the owning expansion in authored order, so a partial continuation restores each completed one without another live execution. Each admitted effect is retained by its own ordinary record, and a read's value is collected while a mutation's is not | built on the #369 stack, continuation basis amended by #589; core owns the mechanics and the workflow policy wrapper is internal | | `` | the workflow host's component an authored document writes where an observation should happen. The host does not register it: it **declares** it to the execution through `ExecutionInstallation.components`, captured before any installation runs, and canonical execution calls its factory once for that attachment with the claimant it minted and registers what comes back. Registration provides availability only — a name a trusted document may write — and carries none of the authority. Its schema is closed on one required `source` string and one optional `allow` array selecting a non-empty duplicate-free subset of the closed effect classes `read` and `write` — omitted means `read` — and paired content is refused. It declares no `returns` and answers with a detached value — `{ observations: [{ name, value }], output }`, each admitted read's own returned value under the name the fragment invoked it by, in invocation order, with whatever the fragment rendered under `output` rather than instead of them, and the pinned identity that produced one left in the retained admission rather than copied here — so an admitted ``, which renders nothing at all, still reaches the document. An admitted mutation contributes no entry and no receipt, so a write-only fragment binds `{ observations: [], output: "" }`; `as` is valid for every selection and binds that same shape. An ordinary `as` captures that value by reference, and an authored `` renders it into the next ``: deciding how a value becomes text is the document's. Every ceiling comes from values the host captured at installation — the run's retained roots and its authoritative current root read from the run's own storage at invocation, as-of-admission provenance a continuation holds by membership so the run's own later publications and an advanced retained current root invalidate nothing, core's pinned self-closing `` read, the write table of core's paired ``, this package's lexical `` built from the definition the ordinary registration owns, and core's self-closing ``, and `` only when the captured request ceiling is non-empty — and no prop, binding, context or middleware return value supplies or widens one. `allow` selects among those tables and adds nothing to them; approval, when a workflow needs one, is authored control flow before the element. Its durable operation is named through that claimant, on the exact invocation the engine handed it and in that invocation's own frame — not from a context a document could rebind, a contextual Api answer, a definition, or a registry answer. Generated source never resolves through the registration: the evaluator consults only its own closed table of pinned identities. It is deliberately not wrapped in `printErrors`, so a refused fragment stops the authored loop rather than becoming text the next turn could read as a read that happened | built on the #302 stack, extended by #369 | @@ -3520,9 +3531,9 @@ Status is measured against main. | `` | updates a protected target ref by compare-and-swap: one non-force update after observing the target equal to the expected commit, adoption of a target already equal to the exact source commit, and refusal of everything else without mutation; remote, ref, credential and non-force policy are host-owned. Not a spelling of `Git.Push`, which advances a branch this run published from a proved ancestry relation | specified by #710; implementation unbuilt | | `` | runs an authored structured argv list natively on the trusted runner against one exact retained Workspace root, under host-owned executable, environment, working-root, per-command duration, whole-run duration, output and process-tree ceilings; a fail-fast pipeline that stops at the first command not exiting `0` and binds the executed prefix — how it completed, how many commands were authored, and one row per command that ran carrying argv, how it ended, which ceiling fired on a timeout, and separately bounded stdout and stderr that state their own truncation; a launch, output-pump or teardown failure binds no result while retaining bounded error evidence, and cancellation commits nothing; absent from the workflow Agent's capabilities and from every generated-XMD write table, and a completed replay runs nothing | specified by #710; implementation unbuilt | | `` | observes that a Git host now records one pull request as merged, at the exact commit a target publication published; a reconciled Git-host observation that mutates nothing, whose only completion is adoption, keyed by the canonical pull-request URL — a merge at another commit conflicts, a pull request still open is temporary unavailability the run waits out under a bounded retry and then a durable machine wait, one closed unmerged conflicts, and publication does not imply any of it | specified by #710; implementation unbuilt | -| remote `WorkflowHost` implementation | keeps the existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — and adds a Cloudflare runtime-named implementation of it beside the Deno one; start, lookup, execute, deliver and inspect stay lifecycle operations reached through those four rather than becoming method names, and a remote host receives no transitions type of its own. One SQLite-backed Durable Object per run is selected from the public run ID, and executor acquisition is an authenticated connection lifetime | specified by #710; implementation unbuilt | -| provider-neutral lifecycle transition types | `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` describe what any host's lifecycle does rather than what one adapter retains, and become package-root public types; the Deno entrypoint may re-export them for source compatibility without owning their meaning, while runtime-specific implementations and retained encodings stay behind their runtime-named entrypoints | neutrality settled by #710; the export move is implementation work | -| same-release runner transport | the messages between an ephemeral runner client and its durable owner are private to one software-factory release: not journaled, exported, authored or supported across independently versioned builds. Connection admission validates an exact immutable build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before parsing, acquisition or state access; there is no cross-version adaptation or downgrade. Privacy of the transport is not privacy of the authority — the acquisition, expected-root validation, owner-side parsing and transactions, content validation, separate no-acquisition delivery and inspection paths, and provider-free completed replay stay public and exact | specified by #710; implementation unbuilt | +| remote `WorkflowHost` implementation | keeps the existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — and adds a Cloudflare runtime-named implementation of it beside the Deno one; start, lookup, execute, deliver and inspect stay lifecycle operations reached through those four rather than becoming method names, and a remote host receives no transitions type of its own. One SQLite-backed Durable Object per run is selected from the public run ID, and executor acquisition is an authenticated connection lifetime | the internal implementation is built: one SQLite-backed Durable Object per run owns create/lookup, coherent reads, lifecycle transitions and settlement, stale recovery, fork and staging, Workspace publication, delivery and consumption, inspection and history, and completed replay, reached through provider-neutral remote adapters behind the same four methods. The configured public host assembly is unbuilt: no client factory, runtime selector, endpoint/release/OIDC supplier assembly or deployment exists, so the shipped CLI assembles the Deno host only | +| provider-neutral lifecycle transition types | `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` describe what any host's lifecycle does rather than what one adapter retains, and become package-root public types; the Deno entrypoint may re-export them for source compatibility without owning their meaning, while runtime-specific implementations and retained encodings stay behind their runtime-named entrypoints | built: neutrality settled by #710, and the types are package-root public types re-exported by the Deno entrypoint for source compatibility | +| same-release runner transport | the messages between an ephemeral runner client and its durable owner are private to one software-factory release: not journaled, exported, authored or supported across independently versioned builds. Connection admission validates an exact immutable build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before parsing, acquisition or state access; there is no cross-version adaptation or downgrade. Privacy of the transport is not privacy of the authority — the acquisition, expected-root validation, owner-side parsing and transactions, content validation, separate no-acquisition delivery and inspection paths, and provider-free completed replay stay public and exact | built for the messages D4 implements: admission validates the exact release identity and refuses a mismatch closed before parsing, acquisition or state access, and the private commands and refusal spellings are named in no public type, journal record, export or documented protocol. The deployment configuration that supplies that identity is part of the unbuilt public host assembly | | factory protocol records | the closed versioned schemas `specs/github-actions-software-factory-spec.md` §11.2 defines — and normatively owns, every other document linking to it rather than restating it — for the subject, stage, implementation revision, handoff, actor, role outcome, invalidation, evidence reference, Planner and Architect verdicts, conflict suspension, Stage 7 decision, merged-observation wait, stage-to-option table, active frontier and the two terminal settlements one issue-driven run retains. Each carries a schema discriminant and a version, and an unknown schema, version, member or enum value refuses rather than being ignored — provider-neutral durable protocol, neither an XMD component nor a TypeScript lifecycle controller | specified by #710; implementation unbuilt | | workflow scheduling (watchers, unattended iteration, remote host selection) | — | #300 | | `` | binds `{ok: true, value}` or `{ok: false, error}`; a failure becomes a bound value, not a raise | defined, unbuilt | diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index a6e89118d..73a0e8da2 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -10044,13 +10044,13 @@ The seven tiers below are the frozen evidence names for the software factory specified by [the software factory](./github-actions-software-factory-spec.md) and by [Workflow workspaces](./workflow-workspace-spec.md) §§3.8, 7.8-7.11, 10.3, 10.5-10.7 -and 13.2, and by [the software factory](./github-actions-software-factory-spec.md) §11.2. Every construct and host they name is **specified; implementation -unbuilt**, so these tiers name the scenarios an implementation is accepted -against rather than tests that exist. Each lists the finite structural +and 13.2, and by [the software factory](./github-actions-software-factory-spec.md) §11.2. Each lists the finite structural scenarios — success, refusal, stale authority, interruption and cancellation, teardown, replay, and denied Agent or generated-XMD authority — and no malformed-input permutation without a distinct structural consequence. +Most of what they name is **specified; implementation unbuilt**, so those tiers name the scenarios an implementation is accepted against rather than tests that exist. Tier WRH is the exception: rows WRH1-WRH16 are implemented and have committed evidence, listed with the tier, while WRH17-WRH22 remain unbuilt with the machine-wait work they belong to. Nothing in the other six tiers is built. + ### Tier WRH — Remote host, executor and delivery separation Defined in [Workflow workspaces](./workflow-workspace-spec.md) §3.8 and §13.2 @@ -10081,6 +10081,31 @@ and [Workflow runs](./workflow-spec.md) §9.8. | WRH15 | Release identity | Connection admission validates an exact immutable runner and owner build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before request parsing, acquisition or state access; no message shape is adapted, downgraded or negotiated, and no transport record is journaled, exported or authored | | WRH16 | Ownership split | Connection admission, request parsing, transaction lifetime and stale recovery belong to the owner; provider attachment and cancellation of its own execution belong to the runner; content is produced by the runner and validated by the owner | +**Where WRH1-WRH16 are proved.** Every row below has committed evidence at two levels: the owner's own behavior against a real Durable Object under `workerd`, and the provider-neutral decisions against a scripted owner. Paths are relative to `packages/`. + +| # | Owner evidence (real Durable Object) | Provider-neutral and host evidence | +|---|---|---| +| WRH1 | `workflow/tests/cloudflare/remote-storage.vitest.ts`, `workflow/tests/cloudflare/remote-owner.vitest.ts` | `workflow/tests/remote-storage.test.ts` | +| WRH2 | `workflow/tests/cloudflare/executor-acquisition.vitest.ts`, `workflow/tests/cloudflare/remote-lifecycle.vitest.ts`, `workflow/tests/cloudflare/remote-publish.vitest.ts` | `workflow/tests/remote-lifecycle.test.ts` | +| WRH3 | `workflow/tests/cloudflare/executor-acquisition.vitest.ts` | `workflow/tests/remote-lifecycle.test.ts` | +| WRH4 | `workflow/tests/cloudflare/remote-lifecycle.vitest.ts` | `workflow/tests/remote-recovery.test.ts` | +| WRH5 | `workflow/tests/cloudflare/remote-publish.vitest.ts`, `workflow/tests/cloudflare/remote-workspace.vitest.ts` | `workflow/tests/remote-publication.test.ts` | +| WRH6 | `workflow/tests/cloudflare/remote-publish.vitest.ts`, `workflow/tests/cloudflare/remote-storage.vitest.ts` | `workflow/tests/remote-publication.test.ts`, `workflow/tests/remote-storage.test.ts` | +| WRH7 | `workflow/tests/cloudflare/remote-delivery.vitest.ts` | `workflow/tests/remote-delivery.test.ts`, `workflow/tests/delivery-gate.test.ts` | +| WRH8 | `workflow/tests/cloudflare/remote-delivery.vitest.ts` | `workflow/tests/remote-delivery.test.ts` | +| WRH9 | `workflow/tests/cloudflare/remote-delivery.vitest.ts` | `workflow/tests/remote-delivery.test.ts`, `workflow/tests/workflow-suspension-answer.test.ts` | +| WRH10 | `workflow/tests/cloudflare/remote-read-plane.vitest.ts` | `workflow/tests/remote-inspection.test.ts` | +| WRH11 | `workflow/tests/cloudflare/executor-acquisition.vitest.ts`, `workflow/tests/cloudflare/remote-lifecycle.vitest.ts` | `workflow/tests/remote-lifecycle.test.ts` | +| WRH12 | `workflow/tests/cloudflare/remote-replay.vitest.ts` | `workflow/tests/replay-inputs.test.ts`, `workflow/tests/git-blob.test.ts`, `cli/tests/workflow-replay.test.ts` | +| WRH13 | — the suites import the runtime-named entrypoint and nothing else does | `workflow/tests/host-neutrality.test.ts`, `workflow/tests/public-entrypoint.test.ts`, `cli/tests/workflow-host-boundary.test.ts` | +| WRH14 | `workflow/tests/cloudflare/remote-lifecycle.vitest.ts` | `cli/tests/workflow-host-boundary.test.ts`, `cli/tests/workflow-cli.test.ts` | +| WRH15 | `workflow/tests/cloudflare/executor-acquisition.vitest.ts`, `workflow/tests/cloudflare/remote-delivery.vitest.ts` | — admission is the owner's, and only the owner can refuse before parsing | +| WRH16 | `workflow/tests/cloudflare/remote-owner.vitest.ts`, `workflow/tests/cloudflare/settle-parser.vitest.ts`, `workflow/tests/cloudflare/owner-storage.vitest.ts` | `workflow/tests/remote-client.test.ts`, `workflow/tests/remote-publication.test.ts` | + +Two rows are narrower than their sentence reads, and the narrowing is not incidental. WRH7's terminal-decision subject is a factory record that does not exist yet, so what is built and proved is answer delivery and its consumption; the plane it arrives on takes no acquisition either way. WRH12 is proved against the whole terminal-history contract of [Workflow runs](./workflow-spec.md) §9.9 rather than against a readable-selection parser alone: `cli/tests/workflow-replay.test.ts` carries the eighteen host-level cases — coherent completed and failed replay, compatible `start --id` and completed `resume`, exact failure recovery, refusal of an unreadable terminal, a duplicated or disowned root import and an unverifiable selection, envelope-only settlement with the terminal run row preserved, and no Git, provider or attachment reached on any of them — and `workflow/tests/cloudflare/remote-replay.vitest.ts` proves the same conclusions where the retained state is a real Durable Object's. + +Fork has no row of its own in this tier because it is not remote-specific: §11 owns it, and the remote implementation is evidenced by `workflow/tests/remote-fork.test.ts`, `workflow/tests/remote-staged-fork.test.ts` and `workflow/tests/cloudflare/remote-fork.vitest.ts`. + ### Tier WGI — Authenticated GitHub ingress Defined in [the software factory](./github-actions-software-factory-spec.md) §5. diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index 409d12ab9..a4f210fb7 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -915,6 +915,36 @@ A later executor is what turns retained delivery into progress: it consumes the value inside the run's own transaction, appends the accepted event once, and only then may execution continue past the wait. +**The owner parses, and the owner transacts.** A request is adopted by the owner alone: it validates the release identity of the connection before parsing anything, parses the request itself rather than accepting a caller's account of it, and opens and commits every transaction. Content arrives content-addressed and is validated against the name it arrived under before it is stored, and against the expected Workspace root before it is published. The runner parses only responses. A caller therefore cannot describe a state change into existence, and a refusal is a refusal of a request the owner read. + +**Creation happens once, and is immutable.** Creating a run initializes pristine storage with the immutable run record, its retrieval metadata, an empty Workspace, the first execution record and `running`, in one transaction. A second compatible creation of the same run id finds that run rather than making another: one owner, one run, one initial root, one lifecycle. A creation whose immutable identity conflicts with what the owner holds is refused, and so is one meeting storage that is damaged or not pristine — in both cases nothing is written, and the distinct condition is what the caller is told. + +**Lookup answers one committed reading, and creates nothing.** An exact lookup returns the complete run storage of §9.4 as a handle the caller may transact against. Absent, foreign, incompatible, damaged, unparseable, wrong-run and closed-scope lookups stay distinct conditions; none of them creates, repairs or partially answers, and a handle whose scope has ended answers nothing. + +**One execution is one envelope.** Beginning inserts exactly one document-execution record — its identity minted by the caller so a retried request is the same bytes and an owner cannot begin a second execution for a request it already answered — and publishes the run state that goes with it in the same transaction, against the exact expected root. Settlement closes that same record with the semantic outcome and its exact stop reason, in one transaction, and only a settled record's status may be reported. An injected failure on either side exposes the whole old state or the whole new one. + +**Stale recovery needs no timer.** An unfinished record found after acquisition belonged to the previous executor and is proven stale by the closed connection rather than by a timestamp, PID, timeout or status row. The next acquisition reconciles the retained root history first, restores or closes that execution accordingly, and only then begins another; a committed effect is never repeated, and cancellation instead finishes that exact stale execution and publishes `cancelled` without beginning one. + +**Fork copies from one selected source, or does nothing.** A fork names one source prefix and root and produces one destination run and one lineage. The destination commits whole or not at all, its copied prefix outlives the source, and an incompatible selection or a failure part-way mutates neither the source nor the destination. + +**A handle's authority ends with its scope.** Teardown closes the handle and, for an executor, releases ownership; it rolls back nothing already committed and settles nothing that was not settled. Closing the connection is the only staleness proof a remote host needs, and a closed one authorizes nothing while leaving every committed transaction exactly as it was. + +### 9.9 A retained terminal, and what may be concluded from it + +A finished run is read before it is trusted, and one shared judgment does the reading, because a conclusion reached two ways is two contracts. Recovery publishing an outcome and admission reusing one ask the same functions of the same events. + +**One semantic outcome.** A root `Close` has two layers and both decide something. The outer layer is the coroutine's own settlement: it returned, it was raised out of, or it was cancelled. Returning is what a document does whether it succeeded or failed, so an outer `ok` says only that the value beneath it is the document's own result, and that result's `status` is what decides `completed` or `failed`. A returned value that is not a document result at all is neither outcome. A failed run names its reason exactly: the last retained row that failed, or — for a failure raised outside any durable operation — one categorical code, and never a message lifted out of an exception. + +**One final terminal.** Exactly one final root `Close` may exist. A second one, or any work recorded after the one that is there, is a history no single execution produced; the run is damaged rather than resolved, because choosing between them would be this build deciding which execution the run was. + +**The terminal has to agree with the history around it.** A run that failed before importing anything carries the root binding core writes for exactly that case, and no root import. A run that produced an ordinary document result carries exactly one root import: exactly one retained event names it, whichever coroutine recorded it, and that one event belongs to the root coroutine. Either shape found with the other's history is damaged. + +**The root import is read by canonical execution's own parser.** The same function that admits a partial history parses the retained selection here, so a selection the executor would refuse cannot publish an outcome instead. It proves rather than recognizes: the retained document parses; an exact target is canonically encoded and resolves against that document to the exact target recorded; and a recorded selection failure re-derives from the same selector to the same kind, matches and available catalog, so a failure record reduced to its selector, carrying another catalog, or naming a selector that actually resolves is not a failure any selection produced. A selection that named no target is raised out of the root import, so the document never ran and a successful result beside one is two histories rather than one. + +**Damage outranks the row.** History those readings refuse is damaged, and damage decides before the stored status does. A run whose row says `completed` or `failed` over a terminal that cannot be read is not advanced, not re-settled and not published: start, resume and cancel each refuse before an execution record exists, before an acquisition performs anything, and before Git, a Workspace or any provider is reached. What the run keeps is exactly what it had — the row, the journal, the Workspace frontier and any unfinished execution the previous executor left open. Stale recovery reads the same judgment first and publishes nothing over damaged history. + +**A coherent terminal replays, and changes only its own envelope.** A completed or failed run named again is replayed rather than refused, under whichever command named it. The replay opens its own document-execution record and closes it, and the run row is left exactly what it was, `updatedAt` included: an outcome that already won does not become mutable again by being read. What the replay runs on comes from the run's own retained state and nowhere else — the root document its root import retained, and the component bundle rebuilt from the immutable definition with each retained component's bytes named the way Git names a blob and compared to the object id the definition holds. No repository, working tree, live import, Workspace, Agent, process, Git-host, Issue, Project or credential provider is reached, no effect is performed again, no native operation starts, no retained answer is consumed and no event is appended. A retained root that does not agree with the run's own definition path refuses before anything is replayed from it. + ## 10. The document filesystem of a run A host attaches one run's Workspace to a document execution with @@ -981,7 +1011,4 @@ uncontained filesystem this boundary exists to prevent. ## 11. Intentionally excluded -Public `xmd workflow` lifecycle commands; lifecycle transition policy, executor -leases and stale-owner recovery; public root selection, history checkpoints and -forks; workflow-owned worktrees; and deterministic Git and GitHub effects. -Retained roots and private restoration do not expose any of those behaviors. +Public `xmd workflow` lifecycle commands; public root selection, history checkpoints and forks; workflow-owned worktrees; and deterministic Git and GitHub effects. Retained roots and private restoration do not expose any of those behaviors. The lifecycle policy those commands are built on — which transitions exist, what a stale executor's unfinished execution becomes, and what a retained terminal permits — is specified by §9.6, §9.8 and §9.9 here and by [Workflow workspaces](./workflow-workspace-spec.md) §3, and reaching it is not something a retained root or a restoration does. diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 6dad7b28e..09fbaebb5 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -2502,6 +2502,8 @@ Workspace, Agent or external providers. It still reconstructs the bundle and still applies that admission, so retained output is accepted only for a history this run is a run of. +Where the inputs for that come from depends on what the host still has. A local `resume` reads the retained components from the retained commit, because the repository is there. A completed replay may have nothing but the run's own storage — an ephemeral runner holds no clone — so it is assembled from retained state alone: the root document its own root import retained, and the bundle rebuilt from the immutable definition with each retained component's bytes named the way Git names a blob and compared to the object id the definition already holds. That is the same admission reaching the same conclusion from evidence the run carries, rather than a weaker one. Which histories are readable enough to be replayed at all, and what a replay may change, is [Workflow runs](./workflow-spec.md) §9.9. + A partial replay that reaches a completed Git-host effect may still reconstruct what that effect needed locally — a Push rebuilds its checkout from the Workspace in order to name the request it is asking about — and then hands back the @@ -3754,7 +3756,7 @@ import nothing Cloudflare-specific. **The host assembly contract does not change.** `WorkflowHost` keeps its four methods — `useRunHost()`, `useLifecycle()`, `useDelivery()` and `attach()` — and the Cloudflare adapter is one more implementation of them beside the Deno one. Starting, looking up, executing, delivering into and inspecting a run are lifecycle operations reached *through* that boundary, exactly as they are locally; they are not replacement method names, and no fifth method appears. A remote host receives no transitions type of its own either. What a remote adapter changes is where each of those four reaches, not what the shared CLI asks for. -**The transition types those methods speak are provider-neutral.** `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` describe what any host's lifecycle does, not what one adapter retains, and they are already defined in the provider-neutral lifecycle module. They become package-root public types, and the Deno entrypoint may keep re-exporting them for source compatibility without owning their meaning. Runtime-specific implementations and retained encodings — SQLite, DOFS, run-id hashing, filesystem paths — stay behind their runtime-named entrypoints, which is the boundary that rationale was always about. That export move is the first implementation story's work; #710 settles that the types are neutral, and performs no production change. +**The transition types those methods speak are provider-neutral.** `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` describe what any host's lifecycle does, not what one adapter retains, and they are already defined in the provider-neutral lifecycle module. They become package-root public types, and the Deno entrypoint may keep re-exporting them for source compatibility without owning their meaning. Runtime-specific implementations and retained encodings — SQLite, DOFS, run-id hashing, filesystem paths — stay behind their runtime-named entrypoints, which is the boundary that rationale was always about. They are package-root public types now, and the Deno entrypoint re-exports them for source compatibility without owning their meaning. Executor ownership is one authenticated WebSocket connection whose lifetime is the acquisition. The owner registers the exact acquisition on admission and invalidates it on close; there is no duration, expiry, renewal, heartbeat, generation record or liveness poll, and a close rolls back nothing already committed. Every mutating transaction validates that exact acquisition and the expected Workspace root together. @@ -3792,9 +3794,15 @@ Privacy of the transport is not privacy of the authority. These constraints are Delivery and inspection reach the owner without an acquisition, under §3.8, on authenticated paths of their own. +#### What the owner implements, and what assembling it still needs + +Everything above is implemented behind the four methods and nothing wider: creating a run and finding it, coherent reads of its committed state, lifecycle transitions and terminal settlement, stale recovery at the next acquisition, fork and the staged candidate a fork is admitted by, root validation and atomic Workspace publication, typed delivery and its later consumption, read-only inspection and history, and canonical completed replay. The owner refuses in its own vocabulary and mutates nothing when it does, on every one of those paths. + +What does not exist yet is the assembly that would let a caller choose that host: there is no configured public client factory, no runtime selector, no endpoint, release-identity or OIDC supplier assembly, no CLI flag, prop or environment reading for any of it, and no deployment. The shipped CLI assembles the local host only, and none of the behavior above waits on that assembly — what it waits on is somebody deciding, in configuration, which host a run belongs to. + #### What completed replay does and does not reach -A completed run replays there as it does locally: it attaches no Workspace, Agent, process, Git, Git-host, Issue, Project or credential provider and performs no effect a second time. It does reach the run's durable owner, because that is where the retained result is; an ephemeral client holds nothing of its own to replay from. Reading retained completion from the owner that holds it is not attaching a provider, and the distinction is the whole point of the rule: what a completed replay must not do is contact an *external* service or repeat an effect, not refrain from reading its own history. +A completed run replays there as it does locally: it attaches no Workspace, Agent, process, Git, Git-host, Issue, Project or credential provider and performs no effect a second time. It does reach the run's durable owner, because that is where the retained result is; an ephemeral client holds nothing of its own to replay from. Reading retained completion from the owner that holds it is not attaching a provider, and the distinction is the whole point of the rule: what a completed replay must not do is contact an *external* service or repeat an effect, not refrain from reading its own history. What it reads, and what makes a retained terminal readable at all, is [Workflow runs](./workflow-spec.md) §9.9: one final root `Close`, one semantic outcome, a root import parsed by canonical execution's own parser and verified against the document it recorded, damage outranking the stored row, and a coherent replay changing nothing but the execution envelope it opened. ## 14. Contract inventory @@ -3804,17 +3812,17 @@ A completed run replays there as it does locally: it attaches no Workspace, Agen | retained run record and filtered journal | built by #291 | | caller-owned storage transaction | built by #291; Workspace mutations join it in #365 | | provider-backed retained Workspace | document filesystem built by #366 and repository composition by #293; document deletion (§10.1) built by #567 for both providers; process capabilities unbuilt (#218) | -| `xmd workflow start` / `resume` | built by #366, Deno entrypoints only; both acquire #367's executor lock | +| `xmd workflow start` / `resume` | built by #366, Deno entrypoints only; both acquire #367's executor lock. The same two lifecycle actions are implemented against the durable owner, where the acquisition is the connection rather than the lock; no CLI selector reaches them yet | | ``, `` and `` composition | built by #293, Deno provider only | | transactional Git components (`Git.Switch`, `Git.Add`, `Git.Commit`) | built by #294, Deno provider only | | `` read and upsert, and the `issue_effect` boundary (§10.3) | built by #296; GitHub middleware, Deno host | | ``, ``, `` (§7.7) | built by #576; GitHub middleware, Deno host. Named by canonical URL and asked of `PullRequestApi`, which carries the upsert too; ordinary durable reads rather than reconciled effects, inheritable by a fork; complete or unavailable, never truncated. Which URLs may be read is operator configuration | -| lifecycle status/list/history | built by #367 | -| lifecycle cancel/delete and executor lock | built by #367 | +| lifecycle status/list/history | built by #367; the durable owner answers the same three from one committed reading, taking no acquisition, attaching nothing and appending nothing | +| lifecycle cancel/delete and executor lock | built by #367; remote cancellation follows what the run retains and takes an acquisition of its own, and a terminal the owner cannot read refuses cancellation without mutating anything | | durable suspension request and executor-lock release | built by #367 | -| `xmd workflow answer` and the `suspension_answer` effect | built by #300 | +| `xmd workflow answer` and the `suspension_answer` effect | built by #300; delivery and consumption are implemented on the owner's own authenticated path, which takes no acquisition, and the response schema is judged by the boundary that writes the value (§3.5) | | workflow scheduling (watchers, unattended iteration, remote hosts) | #300 | -| history fork | built (§11); Deno provider only | +| history fork | built (§11); both providers — the owner copies one selected prefix and its roots into a destination that commits whole or not at all, over a staged candidate assembled without contacting it, and a refusal mutates neither side | | XMD artifact export, inspection and fork source | specified in `specs/xmd-artifact-spec.md`; `xmd workflow export` and artifact `status`/`history` are built, Deno provider only. The artifact-backed fork remains unbuilt | | Agent session portability evidence in an artifact | specified in `specs/xmd-artifact-spec.md` §2.5; the format and its complete verifier are built. Provider bundle capture, Agent-aware export, intrinsic Agent-aware inspection and artifact-backed fork are unbuilt | | workflow Agent isolation | built by #302: no directory attachment, an empty host-owned working directory, no MCP servers, an empty requested tool set and deny-all with a failing permission path; the portable no-tool proof is tracked by #496 | @@ -3832,7 +3840,8 @@ A completed run replays there as it does locally: it attaches no Workspace, Agen | `Evidence.Run` trusted native evidence execution (§10.5) | specified by #710; implementation unbuilt | | `Project.Status` and the Project-provider boundary (§10.6) | specified by #710; implementation unbuilt | | factory protocol records consumed by these effects | specified by #710 and owned normatively by [the software factory](./github-actions-software-factory-spec.md) §11.2, which this specification links to rather than duplicating: `Git.Merge`'s publish ceiling reads the Stage 7 decision, `PullRequest.Merged`'s wait is one of those records, and `Project.Status` projects a stage through the configured stage-to-option table | -| remote lifecycle host, executor connection, versioned runner transport and remote topology (§3.8, §13.2) — the existing four-method `WorkflowHost` boundary, with a Cloudflare implementation beside the Deno one | specified by #710; implementation unbuilt | -| terminal-decision delivery on the delivery plane (§3.8) | specified by #710; implementation unbuilt | +| remote lifecycle host, executor connection, versioned runner transport and remote topology (§3.8, §13.2) — the existing four-method `WorkflowHost` boundary, with a Cloudflare implementation beside the Deno one | the internal implementation is built by #698: one SQLite-backed Durable Object per run, connection-lifetime acquisition, owner-side parsing and transactions, release identity at admission, and provider-neutral remote adapters behind the same four methods. The configured public host assembly — client factory, runtime selector, endpoint/release/OIDC supplier assembly, CLI configuration and deployment — is unbuilt | +| terminal-decision delivery on the delivery plane (§3.8) | specified by #710; implementation unbuilt. Answer delivery and consumption on that plane are built by #698; a terminal decision as a delivery subject belongs to the factory records that define it | +| canonical completed replay from retained state (§9, §13.2) | built by #698 for both providers: the retained root document and the bundle admission come from the run's own committed state, component bytes are authenticated by Git blob identity, an unreadable terminal, import or selection refuses before anything is replayed, and a coherent replay changes only the execution envelope it opened | | Worker JavaScript | deferred | | bundled workerd local host | omitted; POC #347 / PR #348 retained as provider evidence | From f01c32c17b7b31e1ca1d38fe0da37236e2daf0e2 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 10:53:01 -0300 Subject: [PATCH 71/81] =?UTF-8?q?=E2=9C=A8=20Reach=20a=20run's=20owner=20o?= =?UTF-8?q?ver=20three=20requests,=20and=20assemble=20one=20host=20from=20?= =?UTF-8?q?them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner had three planes and no way in. It has one now: a path names the plane, one run id selects the object arithmetically, and a gateway forwards on that id alone — it parses no command, verifies no token and commits nothing, because an owner that trusted a gateway about any of those would have moved its own admission outside itself. The executor plane performs a real upgrade, checks the release before the token and the token before the run, and takes the acquisition last. A runner reaches all three through one configured client: an already-selected run id, one credential-free endpoint parsed before anything is sent, the exact release identity, a token minted for the immediate request, and the HTTP and WebSocket I/O its host performs. It is bound to that one run and refuses another before a token exists. Trusted code assembles it into the same four methods the local entrypoint installs — nothing selects it, reads a flag or an environment variable for it, or deploys it. Composing that exposed a real gap in the runner's Workspace coordinator. Its routed journal was not the handle's journal, so a document run through the CLI would have published its effects beside the transaction that made them and the provenance check would have refused every one. The remote handle now routes its own journal and records the provenance taken over it, the way the local handle always has, and the coordinator opens the run from the handle its own lifecycle produced rather than a second one. That makes the attachment exact. A handle remembers the link it was opened from; attaching compares that link, by identity, to the ones this runner's own acquisitions produced. Two clients on two owners can hold handles that agree about run id, root and anchor, so nothing a handle says about itself could settle which run it is — where it came from can. Two suites found real defects while proving this: a token was minted for a run the client is not bound to, and admission refusals were read through the command vocabulary, so a release or token category raised a malformed answer instead of an unavailability. Both are fixed here. --- architecture.md | 10 +- packages/cli/src/remote-workflow.ts | 142 ++++++++ .../cli/tests/remote-workflow-host.test.ts | 342 ++++++++++++++++++ packages/workflow/cloudflare.ts | 40 ++ packages/workflow/deno.json | 2 +- packages/workflow/deno.ts | 48 +++ .../workflow/src/cloudflare/configured.ts | 250 +++++++++++++ packages/workflow/src/cloudflare/endpoint.ts | 83 +++++ packages/workflow/src/cloudflare/gateway.ts | 45 +++ packages/workflow/src/cloudflare/owner.ts | 102 +++++- packages/workflow/src/cloudflare/routes.ts | 101 ++++++ packages/workflow/src/deno/remote-runner.ts | 164 +++++++++ packages/workflow/src/remote/database.ts | 61 +++- packages/workflow/src/remote/workspace.ts | 39 +- .../cloudflare/remote-owner-routes.vitest.ts | 330 +++++++++++++++++ .../cloudflare/remote-workspace.vitest.ts | 20 +- .../cloudflare/support/executor-object.ts | 14 +- packages/workflow/tests/remote-runner.test.ts | 221 +++++++++++ .../workflow/tests/remote-workspace.test.ts | 63 +++- specs/executable-mdx-spec.md | 2 + specs/workflow-spec.md | 2 + specs/workflow-workspace-spec.md | 16 +- 22 files changed, 2055 insertions(+), 42 deletions(-) create mode 100644 packages/cli/src/remote-workflow.ts create mode 100644 packages/cli/tests/remote-workflow-host.test.ts create mode 100644 packages/workflow/src/cloudflare/configured.ts create mode 100644 packages/workflow/src/cloudflare/endpoint.ts create mode 100644 packages/workflow/src/cloudflare/gateway.ts create mode 100644 packages/workflow/src/cloudflare/routes.ts create mode 100644 packages/workflow/src/deno/remote-runner.ts create mode 100644 packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts create mode 100644 packages/workflow/tests/remote-runner.test.ts diff --git a/architecture.md b/architecture.md index 76a5ad7ee..00320b4b8 100644 --- a/architecture.md +++ b/architecture.md @@ -439,7 +439,11 @@ transaction, never a partial one. The next acquisition performs the ordinary stale-execution recovery and resumes from the exact committed WorkflowRun and Workspace frontier. A completed run replays as it does anywhere, and reading its own history is what it does: it may reach the durable owner holding its retained result — an ephemeral client has nothing else to replay from — while attaching no Workspace, Agent, process, Git, Git-host, Issue, Project, credential or other external-effect provider, performing no effect again and starting no native operation. Lifecycle storage access is not external-effect replay, and only the second is what a completed run must never do. -What that owner implements, it implements behind the four-method `WorkflowHost` boundary and nothing wider: create and lookup, coherent reads, lifecycle transitions and terminal settlement, stale recovery, fork and its staging, Workspace publication, typed delivery and consumption, read-only inspection and history, and canonical completed replay. Which host a caller gets is not yet a decision anybody can make: there is no configured public client factory, no runtime selector, and no endpoint, release-identity or OIDC supplier assembly, so the shipped CLI assembles the local host only. That assembly is the remaining work, and none of the behavior above waits on it. +What that owner implements, it implements behind the four-method `WorkflowHost` boundary and nothing wider: create and lookup, coherent reads, lifecycle transitions and terminal settlement, stale recovery, fork and its staging, Workspace publication, typed delivery and consumption, read-only inspection and history, and canonical completed replay. + +A runner reaches it over three requests, and the path says which plane. One run id selects its owner arithmetically, a gateway forwards on that id alone, and the executor plane is a real upgrade whose admission order — release, then token, then the run — is the owner's own. One configured client carries all three: an already-selected run id, one credential-free endpoint parsed at construction, the release identity, a token minted per request, and the I/O its host performs. It is bound to that one run, refuses another before a token exists, and keeps endpoint, release and token in closure state that reaches no record, event, error or document-visible value. Trusted code assembles that client into the same four methods, and an attachment is bound to the exact storage handle the begin transition produced — compared by identity to the link this runner's own acquisition opened, because two clients on two owners can hold handles that agree about everything except where they came from. + +What no code chooses is which host a run belongs to. There is no runtime or CLI selector, no ambient endpoint, release-identity or OIDC source, and no deployment: the shipped entrypoints install the local host, and the explicit installer is the seam a later trusted assembly configures. ## The workflow lifecycle @@ -3531,9 +3535,9 @@ Status is measured against main. | `` | updates a protected target ref by compare-and-swap: one non-force update after observing the target equal to the expected commit, adoption of a target already equal to the exact source commit, and refusal of everything else without mutation; remote, ref, credential and non-force policy are host-owned. Not a spelling of `Git.Push`, which advances a branch this run published from a proved ancestry relation | specified by #710; implementation unbuilt | | `` | runs an authored structured argv list natively on the trusted runner against one exact retained Workspace root, under host-owned executable, environment, working-root, per-command duration, whole-run duration, output and process-tree ceilings; a fail-fast pipeline that stops at the first command not exiting `0` and binds the executed prefix — how it completed, how many commands were authored, and one row per command that ran carrying argv, how it ended, which ceiling fired on a timeout, and separately bounded stdout and stderr that state their own truncation; a launch, output-pump or teardown failure binds no result while retaining bounded error evidence, and cancellation commits nothing; absent from the workflow Agent's capabilities and from every generated-XMD write table, and a completed replay runs nothing | specified by #710; implementation unbuilt | | `` | observes that a Git host now records one pull request as merged, at the exact commit a target publication published; a reconciled Git-host observation that mutates nothing, whose only completion is adoption, keyed by the canonical pull-request URL — a merge at another commit conflicts, a pull request still open is temporary unavailability the run waits out under a bounded retry and then a durable machine wait, one closed unmerged conflicts, and publication does not imply any of it | specified by #710; implementation unbuilt | -| remote `WorkflowHost` implementation | keeps the existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — and adds a Cloudflare runtime-named implementation of it beside the Deno one; start, lookup, execute, deliver and inspect stay lifecycle operations reached through those four rather than becoming method names, and a remote host receives no transitions type of its own. One SQLite-backed Durable Object per run is selected from the public run ID, and executor acquisition is an authenticated connection lifetime | the internal implementation is built: one SQLite-backed Durable Object per run owns create/lookup, coherent reads, lifecycle transitions and settlement, stale recovery, fork and staging, Workspace publication, delivery and consumption, inspection and history, and completed replay, reached through provider-neutral remote adapters behind the same four methods. The configured public host assembly is unbuilt: no client factory, runtime selector, endpoint/release/OIDC supplier assembly or deployment exists, so the shipped CLI assembles the Deno host only | +| remote `WorkflowHost` implementation | keeps the existing four-method host boundary — `useRunHost()`, `useLifecycle()`, `useDelivery()`, `attach()` — and adds a Cloudflare runtime-named implementation of it beside the Deno one; start, lookup, execute, deliver and inspect stay lifecycle operations reached through those four rather than becoming method names, and a remote host receives no transitions type of its own. One SQLite-backed Durable Object per run is selected from the public run ID, and executor acquisition is an authenticated connection lifetime | built: one SQLite-backed Durable Object per run owns create/lookup, coherent reads, lifecycle transitions and settlement, stale recovery, fork and staging, Workspace publication, delivery and consumption, inspection and history, and completed replay, reached over three request planes and one gateway that routes by run id; one configured client — run id, credential-free endpoint, release identity, per-request token, host I/O — is bound to one owner, and trusted code assembles it into the same four methods with an attachment bound by identity to the handle its own lifecycle opened. Unbuilt: any selector or ambient source that would choose this host, and any deployment | | provider-neutral lifecycle transition types | `WorkflowExecutionTransitions`, `WorkflowBeginRequest`, `WorkflowExecutionBegun`, `WorkflowForkRequest`, `WorkflowForkSelection` and `WorkflowRunCreation` describe what any host's lifecycle does rather than what one adapter retains, and become package-root public types; the Deno entrypoint may re-export them for source compatibility without owning their meaning, while runtime-specific implementations and retained encodings stay behind their runtime-named entrypoints | built: neutrality settled by #710, and the types are package-root public types re-exported by the Deno entrypoint for source compatibility | -| same-release runner transport | the messages between an ephemeral runner client and its durable owner are private to one software-factory release: not journaled, exported, authored or supported across independently versioned builds. Connection admission validates an exact immutable build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before parsing, acquisition or state access; there is no cross-version adaptation or downgrade. Privacy of the transport is not privacy of the authority — the acquisition, expected-root validation, owner-side parsing and transactions, content validation, separate no-acquisition delivery and inspection paths, and provider-free completed replay stay public and exact | built for the messages D4 implements: admission validates the exact release identity and refuses a mismatch closed before parsing, acquisition or state access, and the private commands and refusal spellings are named in no public type, journal record, export or documented protocol. The deployment configuration that supplies that identity is part of the unbuilt public host assembly | +| same-release runner transport | the messages between an ephemeral runner client and its durable owner are private to one software-factory release: not journaled, exported, authored or supported across independently versioned builds. Connection admission validates an exact immutable build or protocol fingerprint from trusted deployment configuration and refuses a mismatch closed, before parsing, acquisition or state access; there is no cross-version adaptation or downgrade. Privacy of the transport is not privacy of the authority — the acquisition, expected-root validation, owner-side parsing and transactions, content validation, separate no-acquisition delivery and inspection paths, and provider-free completed replay stay public and exact | built: admission validates the exact release identity and refuses a mismatch closed before parsing, acquisition or state access, and the private paths, header names, commands and refusal spellings are named in no public type, journal record, export or documented protocol — one refusal category crosses, the one that says another live executor holds the run. Which release identity a deployment supplies is deployment configuration, and no deployment exists | | factory protocol records | the closed versioned schemas `specs/github-actions-software-factory-spec.md` §11.2 defines — and normatively owns, every other document linking to it rather than restating it — for the subject, stage, implementation revision, handoff, actor, role outcome, invalidation, evidence reference, Planner and Architect verdicts, conflict suspension, Stage 7 decision, merged-observation wait, stage-to-option table, active frontier and the two terminal settlements one issue-driven run retains. Each carries a schema discriminant and a version, and an unknown schema, version, member or enum value refuses rather than being ignored — provider-neutral durable protocol, neither an XMD component nor a TypeScript lifecycle controller | specified by #710; implementation unbuilt | | workflow scheduling (watchers, unattended iteration, remote host selection) | — | #300 | | `` | binds `{ok: true, value}` or `{ok: false, error}`; a failure becomes a bound value, not a raise | defined, unbuilt | diff --git a/packages/cli/src/remote-workflow.ts b/packages/cli/src/remote-workflow.ts new file mode 100644 index 000000000..e9e480d14 --- /dev/null +++ b/packages/cli/src/remote-workflow.ts @@ -0,0 +1,142 @@ +/** + * The remote workflow host — one run, one owner, assembled explicitly. + * + * The local host is chosen by which entrypoint is running. This one is not + * chosen at all: it is constructed by trusted code that already holds the four + * things it needs — which run, which owner endpoint, which release both sides + * agreed on, and how to mint a short-lived token — and hands them in. There is + * no flag, no environment variable, no prop and no runtime detection that + * reaches it, because a host that could be selected by ambient configuration + * would be a host somebody could redirect. + * + * What it returns is the same `WorkflowHost` the Deno host implements, with the + * same four methods. `xmd workflow` asks them the same questions in the same + * order; the answers come from a Durable Object instead of a file. + * + * The I/O adapters are here because here is where a runtime may be named. The + * workflow package performs no `fetch` and constructs no `WebSocket` of its + * own — it is handed both, so a test supplies deterministic transports and + * proves the same assembly. + */ + +import { ensure, resource, until, type Operation } from "effection"; +import { remoteOwnerClient, useRemoteWorkflowRunner } from "@executablemd/workflow/deno"; +import type { + OwnerHttpRequest, + OwnerHttpResponse, + OwnerSocket, + OwnerTransport, + OwnerUpgrade, + OwnerUpgradeRefused, +} from "@executablemd/workflow/deno"; +import type { WorkflowExecutionTransitions, WorkflowRunDatabase } from "@executablemd/workflow"; +import type { WorkflowHost } from "./workflow.ts"; + +/** What trusted code supplies to reach one run on one owner. */ +export interface RemoteWorkflowConfiguration { + /** The already-selected public run id. Never derived here. */ + readonly runId: string; + /** The credential-free owner endpoint, parsed once when this is built. */ + readonly endpoint: string; + /** The exact immutable release identity this deployment admits. */ + readonly release: string; + /** A fresh short-lived token for the immediate request. */ + token(): Operation; + /** + * Where this runner assembles fork candidates. + * + * Runner-local scratch, and nothing durable lives in it. Explicit for the + * same reason everything else here is: a directory read from the environment + * is a directory somebody else can choose. + */ + readonly scratchRoot: string; + /** + * The HTTP and WebSocket I/O to perform, when the platform's own will not do. + * + * Absent means this runtime's `fetch` and `WebSocket`, which is what a real + * runner uses. A test supplies its own and proves the same assembly against a + * transport it controls. + */ + readonly transport?: OwnerTransport; +} + +/** + * Assemble one remote host for one run. + * + * Everything it installs belongs to the calling scope: the executor connection + * an execution acquires, the read and delivery planes, the temporary trees a + * Workspace mutation materializes into, and the storage handles opened along + * the way all end when that scope does. + */ +export function* useRemoteWorkflowHost( + configuration: RemoteWorkflowConfiguration, +): Operation { + const client = remoteOwnerClient({ + runId: configuration.runId, + endpoint: configuration.endpoint, + release: configuration.release, + token: () => configuration.token(), + transport: configuration.transport ?? platformTransport(), + }); + const runner = yield* useRemoteWorkflowRunner({ + owner: client, + scratchRoot: configuration.scratchRoot, + }); + return { + useRunHost(): Operation { + return runner.useRunHost(); + }, + useLifecycle(): Operation { + return runner.useLifecycle(); + }, + useDelivery(): Operation { + return runner.useDelivery(); + }, + attach(database: WorkflowRunDatabase, operation: Operation): Operation { + // Only the exact handle this host's own lifecycle opened is attachable, + // and the runner proves that by identity rather than by comparing what + // the handle says about itself. + return runner.attach(database, operation); + }, + }; +} + +/** The two pieces of I/O this runtime already has, named once. */ +function platformTransport(): OwnerTransport { + return { + *request(request: OwnerHttpRequest): Operation { + const response = yield* until( + fetch(request.url, { + method: "POST", + headers: { ...request.headers }, + body: request.body, + }), + ); + // Read to completion here, so nothing downstream holds a body that has + // to be drained or cancelled. + return { status: response.status, body: yield* until(response.text()) }; + }, + + connect(upgrade: OwnerUpgrade): Operation { + return resource(function* (provide) { + const socket = new WebSocket(upgrade.url, [...upgrade.protocols]); + // The socket belongs to this scope from the moment it exists, so a + // cancellation between opening and handing it over still closes it. + yield* ensure(() => { + socket.close(); + }); + const settled = new Promise((resolve) => { + socket.addEventListener("open", () => resolve(socket), { once: true }); + // A refused upgrade closes without ever opening. The owner answered a + // status and a category, and a standard client is shown neither, so + // what travels is that the owner could not be reached for this + // request — its own vocabulary reaches the planes that can carry it. + const refused = () => resolve({ refusal: "command:unavailable" }); + socket.addEventListener("error", refused, { once: true }); + socket.addEventListener("close", refused, { once: true }); + }); + yield* provide(yield* until(settled)); + }); + }, + }; +} diff --git a/packages/cli/tests/remote-workflow-host.test.ts b/packages/cli/tests/remote-workflow-host.test.ts new file mode 100644 index 000000000..9191d5766 --- /dev/null +++ b/packages/cli/tests/remote-workflow-host.test.ts @@ -0,0 +1,342 @@ +/** + * Tier WRH14 — the configured remote host, as trusted code constructs it. + * + * What this file is about is the assembly and its boundaries: that the host has + * the same four methods the local one has and nothing more, that it is bound to + * one run and refuses another before a token is minted or anything is sent, + * that the two request planes take no acquisition while execution takes one, + * and that a storage handle it did not open cannot be attached to a document. + * + * Everything here goes through the published entrypoints and a transport this + * test owns. Nothing imports a provider-private module, and the owner is + * scripted rather than real — what a real Durable Object does with these + * requests is proved against one in + * `packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts`, because + * hibernation, an actual upgrade and a real transaction are not things a script + * can claim. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, type Operation, resource, scoped } from "effection"; +import { WorkflowInputDelivery, WorkflowLifecycle } from "@executablemd/workflow"; +import type { WorkflowRunDatabase } from "@executablemd/workflow"; +import type { + OwnerHttpRequest, + OwnerHttpResponse, + OwnerSocket, + OwnerTransport, + OwnerUpgrade, + OwnerUpgradeRefused, + SocketListener, +} from "@executablemd/workflow/deno"; +import { OwnerEndpointError } from "@executablemd/workflow/deno"; +import { useRemoteWorkflowHost } from "../src/remote-workflow.ts"; +import type { WorkflowHost } from "../src/workflow.ts"; + +const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +const OTHER_RUN = "4bxsfqu1yxtsmfg6aaccq1sxf1a4z456bf614gt4d6t31nqdqwzz"; +const ENDPOINT = "https://owner.example/workflow"; +const RELEASE = "factory-2026.09.10-abcdef"; + +/** Everything one scripted owner was asked, and what it answered. */ +interface Scripted { + /** Every ordinary request, in order. */ + readonly requests: OwnerHttpRequest[]; + /** Every upgrade, in order. */ + readonly upgrades: OwnerUpgrade[]; + /** Every token this client minted, in order. */ + readonly tokens: string[]; + /** Every socket handed out, and whether it is still open. */ + readonly sockets: { readonly protocols: readonly string[]; closed: boolean }[]; + readonly transport: OwnerTransport; + token(): Operation; +} + +/** + * One owner, scripted. + * + * `answer` decides what an ordinary request comes back as; `upgrade` decides + * whether the executor plane hands over a socket or a refusal. Both record + * everything, because most of what this file proves is what was *not* asked. + */ +function scripted( + options: { + answer?: (request: OwnerHttpRequest) => OwnerHttpResponse; + upgrade?: string; + } = {}, +): Scripted { + const requests: OwnerHttpRequest[] = []; + const upgrades: OwnerUpgrade[] = []; + const tokens: string[] = []; + const sockets: { readonly protocols: readonly string[]; closed: boolean }[] = []; + let minted = 0; + return { + requests, + upgrades, + tokens, + sockets, + // deno-lint-ignore require-yield + *token(): Operation { + const token = `token-${(minted += 1)}`; + tokens.push(token); + return token; + }, + transport: { + // deno-lint-ignore require-yield + *request(request: OwnerHttpRequest): Operation { + requests.push(request); + return ( + options.answer?.(request) ?? { + status: 200, + body: JSON.stringify({ outcome: "refused", refusal: "command:absent" }), + } + ); + }, + connect(upgrade: OwnerUpgrade): Operation { + return resource(function* (provide) { + upgrades.push(upgrade); + if (options.upgrade !== undefined) { + yield* provide({ refusal: options.upgrade }); + return; + } + const held = { protocols: upgrade.protocols, closed: false }; + sockets.push(held); + const socket: OwnerSocket = { + send(): void {}, + close(): void { + held.closed = true; + }, + addEventListener(_type: string, _listener: SocketListener): void {}, + removeEventListener(_type: string, _listener: SocketListener): void {}, + }; + yield* ensure(() => { + held.closed = true; + }); + yield* provide(socket); + }); + }, + }, + }; +} + +/** The host, built the way trusted code builds one. */ +function* host( + owner: Scripted, + runId: string = RUN_ID, + endpoint: string = ENDPOINT, +): Operation { + return yield* useRemoteWorkflowHost({ + runId, + endpoint, + release: RELEASE, + token: () => owner.token(), + scratchRoot: "/tmp/xmd-remote-host-test", + transport: owner.transport, + }); +} + +/** What reading a handle nothing opened would do, if anything read one. */ +function refuse(): never { + throw new Error("PLANTED-FOREIGN-DATABASE-USED"); +} + +/** A storage handle nothing opened: shaped like one, and one nothing may use. */ +function foreignDatabase(): WorkflowRunDatabase { + return { + get record() { + return refuse(); + }, + get retrieval() { + return refuse(); + }, + get journal() { + return refuse(); + }, + readJournalEntries: refuse, + transact: refuse, + replaceRetrievalMetadata: refuse, + readDocumentExecutions: refuse, + }; +} + +describe("the configured remote workflow host", () => { + it("has the four methods a host has, and no others", function* () { + const owner = scripted(); + const assembled = yield* scoped(function* () { + const built = yield* host(owner); + return Object.keys(built).toSorted(); + }); + expect(assembled).toEqual(["attach", "useDelivery", "useLifecycle", "useRunHost"]); + // Constructing a host reaches no owner: no token was minted, no request was + // sent and nothing was upgraded. + expect(owner.tokens).toEqual([]); + expect(owner.requests).toEqual([]); + expect(owner.upgrades).toEqual([]); + }); + + it("refuses an endpoint that cannot address an owner, before anything else", function* () { + const owner = scripted(); + const refused: Record = {}; + const offered: Record = { + "endpoint-absent": "", + "endpoint-unparseable": "not a url", + "endpoint-scheme": "ftp://owner.example", + "endpoint-credentials": "https://user:secret@owner.example", + "endpoint-query": "https://owner.example/workflow?token=x", + "endpoint-fragment": "https://owner.example/workflow#fragment", + }; + for (const [expected, endpoint] of Object.entries(offered)) { + refused[expected] = yield* scoped(function* () { + try { + // Parsed by the client's own construction, which building the host + // reaches before it installs anything at all. + yield* host(owner, RUN_ID, endpoint); + return "admitted"; + } catch (error) { + return error instanceof OwnerEndpointError ? error.refusal : "other"; + } + }); + } + expect(refused).toEqual(Object.fromEntries(Object.keys(offered).map((key) => [key, key]))); + // Every one of them refused here, and none of them minted a token. + expect(owner.tokens).toEqual([]); + expect(owner.requests).toEqual([]); + }); + + it("reads through the request plane, taking no acquisition", function* () { + const owner = scripted(); + const outcome = yield* scoped(function* () { + const built = yield* host(owner); + yield* built.useLifecycle(); + const inspected = yield* WorkflowLifecycle.operations.inspect(RUN_ID); + return inspected.ok ? "answered" : inspected.error.name; + }); + // The owner answered `absent`, which is a fact about the run rather than a + // failure of the plane. + expect(outcome).toBe("WorkflowRunNotFoundError"); + // One ordinary request, on the read path of the configured endpoint, with + // one freshly minted token beside the body rather than inside it. + expect(owner.requests).toHaveLength(1); + expect(owner.requests[0]?.url).toBe(`${ENDPOINT}/runs/${RUN_ID}/read`); + expect(owner.requests[0]?.headers["authorization"]).toBe("Bearer token-1"); + expect(owner.requests[0]?.body).not.toContain("token-1"); + // And nothing was acquired to answer it. + expect(owner.upgrades).toEqual([]); + expect(owner.sockets).toEqual([]); + }); + + it("delivers through its own request plane, taking no acquisition", function* () { + const owner = scripted({ + answer: () => ({ + status: 200, + body: JSON.stringify({ outcome: "refused", refusal: "command:not-suspended" }), + }), + }); + const outcome = yield* scoped(function* () { + const built = yield* host(owner); + yield* built.useDelivery(); + const delivered = yield* WorkflowInputDelivery.operations.deliver({ + runId: RUN_ID, + suspensionId: "suspension-1", + value: "answered", + secretDetection: false, + }); + return delivered.ok ? "retained" : "refused"; + }); + expect(outcome).toBe("refused"); + expect(owner.requests).toHaveLength(1); + expect(owner.requests[0]?.url).toBe(`${ENDPOINT}/runs/${RUN_ID}/delivery`); + expect(owner.upgrades).toEqual([]); + }); + + it("is bound to one run, and refuses another before minting a token", function* () { + const owner = scripted(); + const outcomes = yield* scoped(function* () { + const built = yield* host(owner); + yield* built.useLifecycle(); + yield* built.useDelivery(); + const inspected = yield* WorkflowLifecycle.operations.inspect(OTHER_RUN); + const delivered = yield* WorkflowInputDelivery.operations.deliver({ + runId: OTHER_RUN, + suspensionId: "suspension-1", + value: "answered", + secretDetection: false, + }); + return { + inspected: inspected.ok ? "answered" : inspected.error.message, + delivered: delivered.ok ? "retained" : "refused", + }; + }); + // Refused because this owner's plane is one run's, whichever layer says so + // first — and said without a token having been minted for it. + expect(outcomes.inspected).toContain("other than"); + expect(outcomes.delivered).toBe("refused"); + // The refusal happened here: no token was minted, and nothing was sent. + expect(owner.tokens).toEqual([]); + expect(owner.requests).toEqual([]); + expect(owner.upgrades).toEqual([]); + }); + + it("acquires one socket for execution, and gives it up with its scope", function* () { + const owner = scripted(); + const acquired = yield* scoped(function* () { + const built = yield* host(owner); + yield* built.useRunHost(); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + return taken.ok ? taken.value.kind : `failed:${taken.error.message}`; + }); + expect(acquired).toBe("acquired"); + // One upgrade, on the executor path, offering this build's protocol with + // the release and a fresh token beside it — and the URL carries neither. + expect(owner.upgrades).toHaveLength(1); + expect(owner.upgrades[0]?.url).toBe(`${ENDPOINT}/runs/${RUN_ID}/executor`); + expect(owner.upgrades[0]?.url).not.toContain("token"); + expect(owner.upgrades[0]?.protocols).toEqual([ + "executablemd.workflow.owner.v1", + RELEASE, + "token-1", + ]); + // No ordinary request was needed to execute, and the socket is closed now + // that the scope that acquired it has ended. + expect(owner.requests).toEqual([]); + expect(owner.sockets).toHaveLength(1); + expect(owner.sockets[0]?.closed).toBe(true); + }); + + it("reports a run another executor holds, rather than failing", function* () { + const owner = scripted({ upgrade: "acquisition:already-running" }); + const outcome = yield* scoped(function* () { + const built = yield* host(owner); + yield* built.useRunHost(); + const acquired = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + return acquired.ok ? acquired.value.kind : `failed:${acquired.error.message}`; + }); + expect(outcome).toBe("already-running"); + expect(owner.sockets).toEqual([]); + }); + + it("attaches nothing it did not open", function* () { + const owner = scripted(); + const refused = yield* scoped(function* () { + const built = yield* host(owner); + yield* built.useRunHost(); + try { + yield* built.attach(foreignDatabase(), never()); + return "attached"; + } catch (error) { + return error instanceof Error ? error.message : "other"; + } + }); + expect(refused).toContain("not opened by this remote host"); + // Refused before the handle was read at all: the planted accessors say so, + // and no temporary tree, materialization or request happened either. + expect(owner.requests).toEqual([]); + }); +}); + +/** An operation an attachment must never reach. */ +// deno-lint-ignore require-yield +function* never(): Operation { + throw new Error("PLANTED-ATTACHED-OPERATION-RAN"); +} diff --git a/packages/workflow/cloudflare.ts b/packages/workflow/cloudflare.ts index 3b2f079c0..e2ac10924 100644 --- a/packages/workflow/cloudflare.ts +++ b/packages/workflow/cloudflare.ts @@ -42,3 +42,43 @@ export type { OwnerNamespace, RunIdRefusal } from "./src/cloudflare/routing.ts"; export { WorkflowObjectStorageError } from "./src/cloudflare/recognition.ts"; export type { RecognitionFailure } from "./src/cloudflare/recognition.ts"; + +/** + * The supported request boundary, for the Worker in front of these owners. + * + * A gateway routes and forwards; it does not parse a private command, verify a + * token, or report that either was already checked. `ownerFor` selects the + * object arithmetically from the public run id, and the owner it reaches makes + * every decision itself. + */ +export { ownerRoute } from "./src/cloudflare/gateway.ts"; + +/** + * One configured client for one run's owner, for a trusted runner. + * + * The minimum a host must supply is an already-selected run id, one + * credential-free endpoint, the exact release identity, a token operation, and + * the HTTP and WebSocket I/O to perform. Endpoint, release and token stay in + * the client's closure; the private commands, refusal spellings and route + * shapes stay inside the adapter. + */ +export { remoteOwnerClient } from "./src/cloudflare/configured.ts"; +export type { + OwnerHttpRequest, + OwnerHttpResponse, + OwnerTransport, + OwnerUpgrade, + OwnerUpgradeRefused, + RemoteOwnerClient, + RemoteOwnerConfiguration, +} from "./src/cloudflare/configured.ts"; +export { OwnerEndpointError } from "./src/cloudflare/endpoint.ts"; +export type { EndpointRefusal } from "./src/cloudflare/endpoint.ts"; + +/** + * The socket shape a host's `connect` provides. + * + * Part of the transport contract rather than of the protocol: a runtime's own + * `WebSocket` satisfies it, and what travels over it stays private. + */ +export type { OwnerSocket, SocketListener } from "./src/remote/client.ts"; diff --git a/packages/workflow/deno.json b/packages/workflow/deno.json index ab4ae02a3..9be6e877f 100644 --- a/packages/workflow/deno.json +++ b/packages/workflow/deno.json @@ -9,6 +9,6 @@ "./credential-helper": "./src/deno/composition/credential-helper.ts" }, "publish": { - "exclude": ["!vendor/cloudflare-computer-dofs/generated/**/*.d.ts"] + "exclude": ["!vendor/cloudflare-computer-dofs/generated/**/*.d.ts", "!src/cloudflare"] } } diff --git a/packages/workflow/deno.ts b/packages/workflow/deno.ts index 012dada52..0d61a9551 100644 --- a/packages/workflow/deno.ts +++ b/packages/workflow/deno.ts @@ -140,3 +140,51 @@ export type { SuspensionControllerOptions, SuspensionNotice, } from "./src/deno/suspension.ts"; + +/** + * One runner for a run whose storage is somewhere else. + * + * The same four things this entrypoint's local host installs, composed over a + * configured owner client instead of a directory: the executor lifecycle and + * its transitions, the no-acquisition read and delivery planes, and the + * Workspace attachment for a live or partial execution. Native Git, evidence + * processes and Agent clients stay on this side, as they do locally; the owner + * runs none of them. + * + * Which owner, which release and which token are the caller's to supply — this + * reads no flag, environment variable or prop for any of them. + */ +export { useRemoteWorkflowRunner } from "./src/deno/remote-runner.ts"; +export type { + RemoteRunnerOwner, + RemoteWorkflowRunner, + RemoteWorkflowRunnerOptions, +} from "./src/deno/remote-runner.ts"; + +/** + * One configured client for one run's owner, for a trusted runner. + * + * Published here as well as from `./cloudflare` because a runner is where one + * is constructed and `./cloudflare` is the owner's entrypoint: it names the + * Durable Object runtime, so it resolves inside a Worker and nowhere else. The + * module is the same one either way. + * + * The minimum a host supplies is an already-selected run id, one + * credential-free endpoint, the exact release identity, an operation that mints + * a short-lived token, and the HTTP and WebSocket I/O to perform. Endpoint, + * release and token stay in the client's closure; the route shapes, private + * commands and refusal spellings stay inside the adapter. + */ +export { remoteOwnerClient } from "./src/cloudflare/configured.ts"; +export type { + OwnerHttpRequest, + OwnerHttpResponse, + OwnerTransport, + OwnerUpgrade, + OwnerUpgradeRefused, + RemoteOwnerClient, + RemoteOwnerConfiguration, +} from "./src/cloudflare/configured.ts"; +export { OwnerEndpointError } from "./src/cloudflare/endpoint.ts"; +export type { EndpointRefusal } from "./src/cloudflare/endpoint.ts"; +export type { OwnerSocket, SocketListener } from "./src/remote/client.ts"; diff --git a/packages/workflow/src/cloudflare/configured.ts b/packages/workflow/src/cloudflare/configured.ts new file mode 100644 index 000000000..636a7f5c1 --- /dev/null +++ b/packages/workflow/src/cloudflare/configured.ts @@ -0,0 +1,250 @@ +/** + * One configured client for one run's owner. + * + * This is the supported way a trusted runner reaches a deployment: an + * already-selected run id, one credential-free endpoint, the exact release this + * build talks to, and an operation that mints a short-lived token when a plane + * needs one. Nothing here reads a flag, an environment variable, a document + * prop or a global; a caller that has not been given these values cannot + * construct one, which is the point. + * + * It is bound to one run. Every plane requires the configured id before a token + * is minted, before a URL is built and before any I/O happens, so a client + * cannot be walked across a namespace and there is nothing here that could + * enumerate one. + * + * The three planes are three requests, not three protocols this publishes. What + * crosses on each of them — the paths, the header names, the private commands + * and the refusal spellings — stays inside this adapter, and the endpoint, + * release and token stay in its closure: no workflow record, journal event, + * public error or document-visible value carries any of them. + */ + +import { Err, Ok, type Operation, type Result } from "effection"; +import type { RemoteExecutorConnection } from "../remote/lifecycle-link.ts"; +import type { OwnerSocket } from "../remote/client.ts"; +import type { RemoteDeliveryLink } from "../remote/answer-link.ts"; +import type { RemoteReadPlane } from "../remote/read.ts"; +import { WorkflowRequestError } from "../storage/errors.ts"; +import { cloudflareDeliveryLink } from "./delivery-client.ts"; +import { cloudflareReadPlane } from "./read-client.ts"; +import { useExecutorConnection } from "./executor-connection.ts"; +import { parseOwnerEndpoint, type OwnerEndpoint } from "./endpoint.ts"; +import { admitRunId } from "./routing.ts"; +import { storageFailure } from "./client.ts"; +import { RELEASE_HEADER, upgradeProtocols } from "./routes.ts"; + +/** One ordinary request to an owner, as a host performs it. */ +export interface OwnerHttpRequest { + readonly url: string; + readonly headers: Readonly>; + readonly body: string; +} + +/** What an owner answered an ordinary request with. */ +export interface OwnerHttpResponse { + readonly status: number; + readonly body: string; +} + +/** One upgrade request, as a host performs it. */ +export interface OwnerUpgrade { + readonly url: string; + /** The subprotocols to offer, in the order this build offers them. */ + readonly protocols: readonly string[]; +} + +/** An upgrade the owner refused, with the category it refused under. */ +export interface OwnerUpgradeRefused { + readonly refusal: string; +} + +/** + * The I/O a host performs on this client's behalf. + * + * Explicit because performing it is the one thing a runtime has to supply and + * this package will not reach for: `fetch` and `WebSocket` are the runner's, + * named where the runner is assembled. Nothing here decides anything about a + * run — a transport that answered on its own would be an owner. + */ +export interface OwnerTransport { + /** Perform one request and answer with what came back. */ + request(request: OwnerHttpRequest): Operation; + /** + * Open one socket, or answer with the category the owner refused under. + * + * The socket must be open when this returns: the first command goes out + * immediately, and a client that sent into a connecting socket would lose it. + * The returned socket belongs to the calling scope. + */ + connect(upgrade: OwnerUpgrade): Operation; +} + +/** What a trusted host supplies to reach one run's owner. */ +export interface RemoteOwnerConfiguration { + /** The run this client is bound to. Selected by the caller, never derived. */ + readonly runId: string; + /** Where this deployment's owners are. Credential-free, and parsed once. */ + readonly endpoint: string; + /** The exact immutable release identity both sides must agree on. */ + readonly release: string; + /** A fresh short-lived token for the immediate request, minted per request. */ + token(): Operation; + /** The HTTP and WebSocket I/O this client performs through. */ + readonly transport: OwnerTransport; +} + +/** One run's owner, reached over its three planes. */ +export interface RemoteOwnerClient { + /** The run this client is bound to, as it was configured. */ + readonly runId: string; + /** + * Admit one executor connection for this run, owned by the calling scope. + * + * `already-running` is the owner's answer that another live executor holds + * the run — a fact about the run rather than a failure of this call. + */ + admit(runId: string): Operation>; + /** The no-acquisition read plane for this run. */ + reads(runId: string): Operation>; + /** The no-acquisition delivery plane for this run. */ + readonly delivery: RemoteDeliveryLink; +} + +/** The one upgrade refusal that is a fact about the run rather than about the connection. */ +const ALREADY_RUNNING = "acquisition:already-running"; + +/** What a caller is told when it addresses a run this client is not bound to. */ +function foreign(runId: string, bound: string): Error { + // Neither id is quoted. What went wrong is that a client bound to one run was + // asked about another, and a diagnostic naming them would put a caller's own + // addressing mistake into a message this run may keep. + return new WorkflowRequestError( + runId === bound + ? "this workflow run id cannot address a workflow owner." + : "this client is bound to one workflow run and was asked about another.", + ); +} + +/** + * Build one client for one run. + * + * The endpoint is parsed here, so an operator learns about a credential, a + * fragment, a query or a scheme this build does not speak before a token is + * minted or anything is opened. + */ +export function remoteOwnerClient(configuration: RemoteOwnerConfiguration): RemoteOwnerClient { + const bound = admitRunId(configuration.runId); + const endpoint: OwnerEndpoint = parseOwnerEndpoint(configuration.endpoint); + const { release, transport } = configuration; + + /** Hold every plane to the one run this client was configured for. */ + function admitted(runId: string): boolean { + return runId === bound; + } + + /** + * Carry one plane's request, with the admission that plane already built. + * + * The token comes from the caller rather than from here: a plane mints one + * for the request it is about to make, and minting a second would be two + * credentials for one question. + */ + function* post( + admission: { readonly release: string; readonly token: string; readonly runId: string }, + plane: "read" | "delivery", + body: string, + ): Operation { + // The bound run decides before anything is built or sent. + if (!admitted(admission.runId)) { + throw foreign(admission.runId, bound); + } + const answered = yield* transport.request({ + url: endpoint.planeUrl(admission.runId, plane), + headers: { + [RELEASE_HEADER]: admission.release, + authorization: `Bearer ${admission.token}`, + "content-type": "application/json", + }, + body, + }); + if (answered.status !== 200) { + // The owner answers both request planes with an envelope, so any other + // status is the request never having reached one. Nothing about the + // response travels: a status is not a refusal category. + throw storageFailure("command:unavailable"); + } + return answered.body; + } + + return { + runId: bound, + + *admit(runId: string): Operation> { + if (!admitted(runId)) { + return Err(foreign(runId, bound)); + } + return yield* useExecutorConnection( + { + *open(forRun: string): Operation> { + if (!admitted(forRun)) { + return Err(foreign(forRun, bound)); + } + const token = yield* configuration.token(); + const opened = yield* transport.connect({ + url: endpoint.planeUrl(forRun, "executor"), + protocols: upgradeProtocols(release, token), + }); + if (!("refusal" in opened)) { + return Ok(opened); + } + // The owner refused the connection. One category is a fact about + // the run and is reported as one; every other refusal an admission + // can produce — a release, a token, a run id — is this connection + // not being admitted, and the word for it stays here rather than + // becoming a public compatibility surface. + return opened.refusal === ALREADY_RUNNING + ? Ok("already-running") + : Err(storageFailure("command:unavailable")); + }, + ids: () => { + let command = 0; + return () => `command-${(command += 1)}`; + }, + }, + runId, + ); + }, + + // deno-lint-ignore require-yield + *reads(runId: string): Operation> { + if (!admitted(runId)) { + return Err(foreign(runId, bound)); + } + return Ok( + cloudflareReadPlane( + { send: (admission, body) => post(admission, "read", body) }, + release, + configuration.token, + runId, + ), + ); + }, + + delivery: cloudflareDeliveryLink( + { send: (admission, body) => post(admission, "delivery", body) }, + { + release, + // The delivery plane is told which run each request is for, so the + // binding is checked here — before a token exists for a run this + // client was never configured to answer for. + *token(runId: string): Operation { + if (!admitted(runId)) { + throw foreign(runId, bound); + } + return yield* configuration.token(); + }, + }, + ), + }; +} diff --git a/packages/workflow/src/cloudflare/endpoint.ts b/packages/workflow/src/cloudflare/endpoint.ts new file mode 100644 index 000000000..4408b145c --- /dev/null +++ b/packages/workflow/src/cloudflare/endpoint.ts @@ -0,0 +1,83 @@ +/** + * Where this deployment's owners are, parsed once and then never re-read. + * + * One value from trusted configuration, held to what the supported routes can + * mean before a token is minted or a socket is opened. A credential in it, a + * fragment, a query, a scheme this build does not speak or a path it never + * writes are configuration mistakes, and an operator learns about them at + * construction rather than in the middle of a run. + * + * It never travels. The endpoint is host closure state: no workflow record, no + * journal event, no diagnostic and no document-visible value carries it, so a + * run cannot report where its owner was reached and a document cannot ask. + */ + +import { planePath, type OwnerPlane } from "./routes.ts"; + +/** Why an endpoint cannot address this deployment's owners. */ +export type EndpointRefusal = + | "endpoint-absent" + | "endpoint-unparseable" + | "endpoint-scheme" + | "endpoint-credentials" + | "endpoint-query" + | "endpoint-fragment"; + +export class OwnerEndpointError extends Error { + override name = "OwnerEndpointError"; + + constructor(readonly refusal: EndpointRefusal) { + super(`this workflow owner endpoint cannot be used (${refusal})`); + } +} + +/** One deployment's owner endpoint, normalized. */ +export interface OwnerEndpoint { + /** Where one run's plane is, as an absolute URL. */ + planeUrl(runId: string, plane: OwnerPlane): string; +} + +/** The schemes an owner is reached over. */ +const SCHEMES: readonly string[] = ["https:", "http:"]; + +/** + * Parse one endpoint, or refuse it. + * + * `http:` is admitted beside `https:` because a local owner — `workerd` on a + * loopback address — is how this is exercised without a deployment. Which + * scheme an operator may configure is deployment policy above this, and + * nothing here weakens transport security on its own. + */ +export function parseOwnerEndpoint(value: unknown): OwnerEndpoint { + if (typeof value !== "string" || value === "") { + throw new OwnerEndpointError("endpoint-absent"); + } + let url: URL; + try { + url = new URL(value); + } catch { + throw new OwnerEndpointError("endpoint-unparseable"); + } + if (!SCHEMES.includes(url.protocol)) { + throw new OwnerEndpointError("endpoint-scheme"); + } + // A credential in a configured endpoint would be a credential this client + // sends on every request without ever having been given one to hold. + if (url.username !== "" || url.password !== "") { + throw new OwnerEndpointError("endpoint-credentials"); + } + if (url.search !== "") { + throw new OwnerEndpointError("endpoint-query"); + } + if (url.hash !== "") { + throw new OwnerEndpointError("endpoint-fragment"); + } + // The base path, without a trailing separator, so the plane path below is the + // only thing that decides the shape of what follows. + const base = `${url.origin}${url.pathname.replace(/\/+$/, "")}`; + return { + planeUrl(runId: string, plane: OwnerPlane): string { + return `${base}${planePath(runId, plane)}`; + }, + }; +} diff --git a/packages/workflow/src/cloudflare/gateway.ts b/packages/workflow/src/cloudflare/gateway.ts new file mode 100644 index 000000000..d252b9071 --- /dev/null +++ b/packages/workflow/src/cloudflare/gateway.ts @@ -0,0 +1,45 @@ +/** + * The Worker in front of these owners, as far as this package decides it. + * + * One job: find which run a request is for, and hand the request to that run's + * object. The run id comes out of the path, is admitted before it reaches + * `idFromName` — that call answers with an object for any string, so a + * mistyped id would otherwise address a fresh, empty owner rather than fail — + * and the request travels on unopened. + * + * Nothing else happens here. The body is not read, the headers are not + * inspected, no token is verified and no state is touched: an owner that + * trusted a gateway's account of any of those would have moved its own + * admission outside itself. + */ + +import { admitRunId, ownerFor, type OwnerNamespace } from "./routing.ts"; +import { routeOf } from "./routes.ts"; + +/** What a stub has to offer for a request to be forwarded to it. */ +export interface OwnerStub { + fetch(request: Request): Promise; +} + +/** + * Forward one request to the owner of the run it names. + * + * Answers `404` for a path this build does not write and `400` for a run id + * that cannot address an owner — neither of which reaches an object at all. + */ +export async function ownerRoute( + namespace: OwnerNamespace, + request: Request, +): Promise { + const route = routeOf(new URL(request.url).pathname); + if (route === undefined) { + return new Response("route", { status: 404 }); + } + let runId: string; + try { + runId = admitRunId(route.runId); + } catch { + return new Response("run-id", { status: 400 }); + } + return await ownerFor(namespace, runId).fetch(request); +} diff --git a/packages/workflow/src/cloudflare/owner.ts b/packages/workflow/src/cloudflare/owner.ts index 758a3f1cc..5ec256251 100644 --- a/packages/workflow/src/cloudflare/owner.ts +++ b/packages/workflow/src/cloudflare/owner.ts @@ -18,10 +18,15 @@ * Delivery and inspection arrive over ordinary requests, take no acquisition, * and cannot move the lifecycle — which is why they are separate methods here * rather than commands on the socket. + * + * `fetch` is where those three meet an actual request. It reads which plane was + * addressed and nothing else: the upgrade, the admission order and every + * decision stay in the three methods below, so a gateway in front of this + * routes bytes and cannot pre-approve any of it. */ import { DurableObject } from "cloudflare:workers"; -import type { Operation } from "effection"; +import { run, type Operation } from "effection"; import { OwnerTransactions } from "./owner-transaction.ts"; import { acquireExecutor, @@ -51,6 +56,13 @@ import { crossSecretGate } from "./owner-gate.ts"; import { dispatchCommand } from "./dispatcher.ts"; import { WorkflowRecordMalformedError } from "../storage/errors.ts"; import { discardPriorAcquisitions, PRIVATE_OBJECT_NAMES } from "./private-schema.ts"; +import { + RELEASE_HEADER, + routeOf, + selectedProtocol, + upgradeAdmission, + type RouteAdmission, +} from "./routes.ts"; import { declaredObjects, holdsNoRun, @@ -322,6 +334,85 @@ export abstract class WorkflowOwnerObject extends DurableObject { * other state — foreign, damaged, a version this build does not implement — * is recognition's to refuse rather than initialization's to overwrite. */ + /** + * Answer one request on one of this owner's three planes. + * + * The runtime callback boundary this host adapts at, so the Effection scope + * is opened here and closed before a response leaves: every plane below is an + * operation, and none of them may outlive the request that asked. + * + * Which plane is decided by the path, and the run id it names is admitted by + * the plane rather than here — a path that says nothing this build writes is + * refused before an admission exists at all. + */ + override async fetch(request: Request): Promise { + const url = new URL(request.url); + const route = routeOf(url.pathname); + if (route === undefined) { + return new Response("route", { status: 404 }); + } + if (route.plane === "executor") { + return await this.#upgrade(request, route.runId); + } + const admission: RouteAdmission = { + release: request.headers.get(RELEASE_HEADER), + token: bearer(request.headers.get("authorization")), + runId: route.runId, + }; + const body = await request.text(); + const answered = await run(() => + route.plane === "read" ? this.read(admission, body) : this.deliver(admission, body), + ); + // Both planes answer rather than raise, so the status says only that this + // owner answered; what it answered is the envelope, and a refusal category + // is the same word on either plane. + return new Response(JSON.stringify(answered), { + status: 200, + headers: { "content-type": "application/json" }, + }); + } + + /** + * Take one executor connection, or refuse the upgrade. + * + * The socket the owner accepts is the runtime's own server half, created + * here; the client half is handed back with the 101 and is the only end the + * caller ever holds. A refusal closes nothing because nothing was accepted: + * `admit()` takes the acquisition last, so a release, token or run-id + * refusal leaves this object exactly as it was. + */ + async #upgrade(request: Request, runId: string): Promise { + if (request.headers.get("upgrade")?.toLowerCase() !== "websocket") { + return new Response("upgrade", { status: 400 }); + } + const admission = upgradeAdmission(request.headers.get("sec-websocket-protocol"), runId); + const pair = new WebSocketPair(); + const client = pair[0]; + const server = pair[1]; + if (client === undefined || server === undefined) { + return new Response("internal", { status: 500 }); + } + try { + await run(() => + this.admit( + { runId: admission.runId, release: admission.release, token: admission.token }, + server, + ), + ); + } catch (error) { + // The refusal category, and nothing else. A caller learns which of the + // ordered checks said no; it learns nothing about this object's state. + return new Response(refusalOf(error), { status: 403 }); + } + return new Response(null, { + status: 101, + webSocket: client, + // Selected explicitly: a handshake that offered a subprotocol and got + // none back is one a standard client fails. + headers: { "sec-websocket-protocol": selectedProtocol() }, + }); + } + open(runId: string, initializeRun: () => void): void { admitRunId(runId); if (isPristine(declaredObjects(this.owned))) { @@ -332,6 +423,15 @@ export abstract class WorkflowOwnerObject extends DurableObject { } } +/** The token one `Authorization` header carries, if it carries one. */ +function bearer(header: string | null): string | null { + if (header === null) { + return null; + } + const [scheme, value] = header.split(" "); + return scheme?.toLowerCase() === "bearer" && value !== undefined && value !== "" ? value : null; +} + /** * A fresh correlation value for one acquisition. * diff --git a/packages/workflow/src/cloudflare/routes.ts b/packages/workflow/src/cloudflare/routes.ts new file mode 100644 index 000000000..e0a6b52ee --- /dev/null +++ b/packages/workflow/src/cloudflare/routes.ts @@ -0,0 +1,101 @@ +/** + * Where a request reaches one run's owner, and what it carries outside its body. + * + * Three planes, three paths, and the path is what says which. A gateway routes + * on the run id in it and forwards the request whole; nothing here parses a + * command, verifies a token or decides anything about a run — the owner does + * all three, and a boundary that pre-decided any of them would be a second + * authority the owner would have to trust. + * + * Private to one release, like everything else that crosses. The paths and the + * header names are this build talking to itself: they are journaled by neither + * side, exported by neither, and named in no public type. What is public is the + * endpoint an operator configures and the three planes' behavior. + * + * The executor plane carries its admission in `Sec-WebSocket-Protocol` because + * that is the one header a standard `WebSocket` client can set. It is a + * transport header on a request that is never retained, and the alternative — + * a token in the URL — would put a credential somewhere URLs get written down. + */ + +/** The one path prefix every plane hangs from. */ +const RUNS = "runs"; + +/** Which plane a path names. */ +export type OwnerPlane = "executor" | "read" | "delivery"; + +/** The order the subprotocol carries admission in. */ +const PROTOCOL = "executablemd.workflow.owner.v1"; + +/** What a request said about itself, outside its body. */ +export interface RouteAdmission { + readonly release: string | null; + readonly token: string | null; + readonly runId: string | null; +} + +/** The header a release identity travels in. */ +export const RELEASE_HEADER = "x-executablemd-workflow-release"; + +/** + * The one subprotocol this build offers, and the two values beside it. + * + * Offered in a fixed order so the owner reads by position rather than by + * guessing which value is which, and the name is first so the owner can echo + * exactly one selected protocol back — a handshake that selected none is one a + * standard client fails. + */ +export function upgradeProtocols(release: string, token: string): readonly string[] { + return [PROTOCOL, release, token]; +} + +/** The admission one upgrade request carries, read back out of its protocols. */ +export function upgradeAdmission(header: string | null, runId: string | null): RouteAdmission { + const offered = (header ?? "").split(",").map((value) => value.trim()); + const [name, release, token] = offered; + return name === PROTOCOL + ? { release: release ?? null, token: token ?? null, runId } + : { release: null, token: null, runId }; +} + +/** The protocol an owner selects, so the handshake completes. */ +export function selectedProtocol(): string { + return PROTOCOL; +} + +/** Where one plane of one run's owner is, under a normalized endpoint. */ +export function planePath(runId: string, plane: OwnerPlane): string { + return `/${RUNS}/${encodeURIComponent(runId)}/${plane}`; +} + +/** + * Which run and plane a path names, or nothing. + * + * Read from the end, because a configured endpoint may carry a path of its own + * and what is in front of the route belongs to the deployment. The three + * segments the route is made of are exact all the same: a tail this build does + * not write names no plane, and the only sender is this build. + */ +export function routeOf(pathname: string): { runId: string; plane: OwnerPlane } | undefined { + const segments = pathname.split("/").filter((segment) => segment !== ""); + const route = segments.slice(-3); + if (route.length !== 3 || route[0] !== RUNS) { + return undefined; + } + const [, encoded, plane] = route; + if (encoded === undefined || encoded === "" || plane === undefined) { + return undefined; + } + if (plane !== "executor" && plane !== "read" && plane !== "delivery") { + return undefined; + } + let runId: string; + try { + runId = decodeURIComponent(encoded); + } catch { + // A percent sequence this build never wrote. The id it would name is not + // one to guess at. + return undefined; + } + return { runId, plane }; +} diff --git a/packages/workflow/src/deno/remote-runner.ts b/packages/workflow/src/deno/remote-runner.ts new file mode 100644 index 000000000..ffa1f6cba --- /dev/null +++ b/packages/workflow/src/deno/remote-runner.ts @@ -0,0 +1,164 @@ +/** + * One runner, for one run whose storage is somewhere else. + * + * The four things a host has to be able to do — move the run's lifecycle, read + * it, deliver into it, and attach it to a document execution — over + * one configured owner client. Everything underneath is already built and + * proved: the provider-neutral executor lifecycle, the no-acquisition read and + * delivery planes, and the runner's Workspace coordinator. What this adds is + * the composition, and the one thing composition has to get right. + * + * That one thing is the handoff from the lifecycle to the attachment. A begin + * transition hands back a storage handle; an attachment needs the Workspace + * runtime for the *same* run, over the same connection. Matching a run id, a + * root or an anchor would be enough for two clients on two owners to satisfy — + * their records can be identical — so nothing here matches a field. The handle + * says which link it was opened from, this runner remembers the links its own + * acquisitions produced, and both answers have to be the same object. + * + * Native work stays here. Materialization, the invocation-owned temporary + * trees and the containment-checked filesystem are the runner's own, and the + * owner runs none of them. + * + * One seam is deliberately not installed. The remote run-storage provider + * answers `create` and `lookup` over an executor link, and a link is an + * acquisition — so installing it here would mean holding one run's executor + * connection open for as long as the host lives, whether or not anything ever + * executed. Creation and lookup happen where the acquisition already is: the + * begin transition takes one, creates or finds the run inside it, and gives it + * back when its scope ends. + */ + +import type { Operation, Result } from "effection"; +import type { WorkflowExecutionTransitions } from "../lifecycle/execution.ts"; +import type { WorkflowRunDatabase } from "../storage/api.ts"; +import { + remoteRunOrigin, + type RemoteRunLink, + type RemoteWorkspaceLink, +} from "../remote/database.ts"; +import type { RemoteExecutorConnection } from "../remote/lifecycle-link.ts"; +import type { RemoteDeliveryLink } from "../remote/answer-link.ts"; +import type { RemoteReadPlane } from "../remote/read.ts"; +import { installRemoteInputDelivery } from "../remote/delivery.ts"; +import { useRemoteLifecycleReads } from "../remote/inspection.ts"; +import { + useRemoteRun, + useRemoteWorkspaceEffects, + withRemoteWorkspaceEffects, +} from "../remote/workspace.ts"; +import { WorkflowRequestError } from "../storage/errors.ts"; +import { installRemoteWorkflowLifecycle } from "./remote-host.ts"; +import { createRemoteWorkspaceFilesystem } from "./remote-workspace-files.ts"; +import { runnerFiles, useRunnerTrees } from "./remote-files.ts"; + +/** The owner this runner reaches, as a configured client supplies it. */ +export interface RemoteRunnerOwner { + /** The run this client is bound to. */ + readonly runId: string; + /** Admit one executor connection for this run, owned by the calling scope. */ + admit(runId: string): Operation>; + /** The no-acquisition read plane for this run. */ + reads(runId: string): Operation>; + /** The no-acquisition delivery plane for this run. */ + readonly delivery: RemoteDeliveryLink; +} + +/** What a trusted host supplies to assemble one runner. */ +export interface RemoteWorkflowRunnerOptions { + /** The configured client for the one owner this runner works against. */ + readonly owner: RemoteRunnerOwner; + /** + * Where this runner assembles fork candidates. + * + * Runner-local scratch, and nothing durable: an absolute directory this + * process may write, supplied explicitly rather than read from anywhere. + */ + readonly scratchRoot: string; +} + +/** What a host installs for a run whose storage is somewhere else. */ +export interface RemoteWorkflowRunner { + /** Install the executor lifecycle for this owner; hand back its transitions. */ + useRunHost(): Operation; + /** Install status, list and history over the no-acquisition read plane. */ + useLifecycle(): Operation; + /** Install typed answer delivery over the no-acquisition delivery plane. */ + useDelivery(): Operation; + /** Attach this run's Workspace to one live or partial document execution. */ + attach(database: WorkflowRunDatabase, operation: Operation): Operation; +} + +/** + * Assemble one runner for one owner. + * + * The scope that asks owns everything this installs: the acquisition it takes, + * the temporary trees it materializes into, and the handles it opens all end + * when that scope does. + */ +export function* useRemoteWorkflowRunner( + options: RemoteWorkflowRunnerOptions, +): Operation { + const { owner } = options; + /** + * The links this runner's own acquisitions produced. + * + * Keyed by the link object, so membership is identity. A link from another + * client — or a value shaped like one — is not in here, and a handle opened + * from it cannot be attached however closely its record matches. + */ + const admitted = new WeakMap(); + + return { + *useRunHost(): Operation { + const transitions = yield* installRemoteWorkflowLifecycle({ + root: options.scratchRoot, + *admit(runId: string): Operation> { + const connection = yield* owner.admit(runId); + if (connection.ok && connection.value !== "already-running") { + admitted.set(connection.value.link, connection.value.link); + } + return connection; + }, + source: (runId: string) => owner.reads(runId), + }); + return transitions; + }, + + *useLifecycle(): Operation { + const plane = yield* owner.reads(owner.runId); + if (!plane.ok) { + throw plane.error; + } + yield* useRemoteLifecycleReads(plane.value); + }, + + *useDelivery(): Operation { + yield* installRemoteInputDelivery(owner.delivery); + }, + + *attach(database: WorkflowRunDatabase, operation: Operation): Operation { + const origin = remoteRunOrigin(database); + const link = origin === undefined ? undefined : admitted.get(origin.link); + if (link === undefined) { + // Not a handle this runner's own lifecycle opened. Nothing about the + // handle is quoted back: what is wrong is which handle it is, and a + // diagnostic naming a run would name the wrong one. + throw new WorkflowRequestError( + "this workflow run storage was not opened by this remote host, and cannot be attached.", + ); + } + const host = runnerFiles(); + const trees = yield* useRunnerTrees(); + const run = yield* useRemoteRun({ + link, + database, + files: host, + trees, + createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), + }); + yield* useRemoteWorkspaceEffects(run); + return yield* withRemoteWorkspaceEffects(run, operation); + }, + }; +} diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index 2245390fc..adbe557ef 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -33,7 +33,13 @@ import { type Result, resource, } from "effection"; -import type { DurableEvent, DurableStream, Json } from "@executablemd/durable-streams"; +import { + establishJournalProvenance, + type DurableEvent, + type DurableStream, + type JournalProvenance, + type Json, +} from "@executablemd/durable-streams"; import type { JournalEntry, WorkflowRunDatabase, WorkflowRunTransaction } from "../storage/api.ts"; import { WorkflowDatabaseClosedError, @@ -55,6 +61,7 @@ import type { RemoteRetainedAnswer } from "./answer-link.ts"; import type { RemoteInvocationSnapshot } from "./records.ts"; import type { CreateWorkflowRunRequest } from "../storage/api.ts"; import type { WorkspaceRootManifest } from "../workspace/root-manifest.ts"; +import { routeRemoteRunJournal } from "./journal-route.ts"; /** What a remote handle needs to answer everything the interface asks. */ export interface RemoteRunLink extends OwnerLink { @@ -242,6 +249,44 @@ function createTurns(): Turns { }; } +/** What one handle was opened from, for a host that has to prove it was. */ +export interface RemoteRunOrigin { + /** The exact link this handle reads, writes and commits through. */ + readonly link: RemoteRunLink; + /** The provenance taken over this handle's routed journal. */ + readonly provenance: JournalProvenance; +} + +/** + * What each handle was opened from. + * + * Held beside the handle rather than on it: `WorkflowRunDatabase` is the same + * interface both hosts implement, and a link or a witness on it would be a + * capability every caller of either could reach. A `WeakMap` keyed by the exact + * handle answers only for a handle this module built, and a second loaded copy + * cannot answer for one of these at all. + * + * What a host does with the answer is compare it — by object identity, to the + * connection it holds — so a handle from another client, or a look-alike with + * the same run id and root, is refused before an effect exists. + */ +const origins = (() => { + const held = new WeakMap(); + return { + remember(database: WorkflowRunDatabase, origin: RemoteRunOrigin): void { + held.set(database, origin); + }, + of(database: WorkflowRunDatabase): RemoteRunOrigin | undefined { + return held.get(database); + }, + }; +})(); + +/** What this handle was opened from, if this module opened it. */ +export function remoteRunOrigin(database: WorkflowRunDatabase): RemoteRunOrigin | undefined { + return origins.of(database); +} + /** Open one scope-owned lease on a run whose storage is somewhere else. */ export function useRemoteRunDatabase( link: RemoteRunLink, @@ -366,7 +411,7 @@ export function useRemoteRunDatabase( }, get journal(): DurableStream { - return ordinary; + return routed; }, transact, @@ -421,6 +466,18 @@ export function useRemoteRunDatabase( }, }; + // The same shape the local handle has: what a caller runs its document on + // is the routed journal, so a Workspace effect's publication lands inside + // the transaction that made the change rather than beside it, and the + // provenance a coordinator compares is taken over that exact stream. Built + // here because here is where the handle exists — a caller that assembled + // the pair itself could pair one run's journal with another's storage. + const routed: DurableStream = routeRemoteRunJournal(handle, ordinary); + origins.remember( + handle, + Object.freeze({ link, provenance: establishJournalProvenance(routed) }), + ); + yield* ensure(() => { closed = true; }); diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts index 16074ab31..46954e28e 100644 --- a/packages/workflow/src/remote/workspace.ts +++ b/packages/workflow/src/remote/workspace.ts @@ -65,10 +65,9 @@ import type { RemoteReadLink } from "./read.ts"; import type { RemoteInvocationSnapshot } from "./records.ts"; import { withRemoteJournalRoute } from "./journal-route.ts"; import { resource } from "effection"; -import { establishJournalProvenance, type DurableStream } from "@executablemd/durable-streams"; -import { useRemoteRunDatabase, type RemoteWorkspaceLink } from "./database.ts"; +import type { DurableStream } from "@executablemd/durable-streams"; +import { remoteRunOrigin, useRemoteRunDatabase, type RemoteWorkspaceLink } from "./database.ts"; import { installRemoteSuspensionAnswers } from "./answers.ts"; -import { routeRemoteRunJournal } from "./journal-route.ts"; import type { TemporaryTrees } from "./invocation.ts"; @@ -203,28 +202,44 @@ export interface RemoteRunOptions { * and an invocation admitted from one run would commit to the other. */ readonly link: RemoteWorkspaceLink; + /** + * The handle that link produced, or nothing to open one now. + * + * A host whose lifecycle already opened this run passes the exact handle its + * begin transition returned, so the document, its journal and this + * coordinator are one association rather than two views of one run. The + * handle has to be one this build opened from a link — its routed journal + * and the provenance taken over it are what a coordinator compares — and a + * foreign or fabricated one is refused here, before any effect exists. + */ + readonly database?: WorkflowRunDatabase; readonly files: RunnerFiles; readonly trees: TemporaryTrees; createFilesystem(at: HostPath, authorize: () => void): WorkspaceFilesystem; - /** The run's ordinary journal, which this routes and takes provenance over. */ - readonly journal: DurableStream; } /** * Open one remote run: its database, its routed journal and its provenance. * - * The database is created here from the same link the runtime reads through, so + * The handle carries all three. It is opened from the same link the runtime + * reads through — here, or by the lifecycle that began this execution — so * "this runtime belongs to this handle" is true by construction rather than by * a check that could be passed with another handle. */ export function useRemoteRun(options: RemoteRunOptions): Operation { return resource(function* (provide) { - const database = yield* useRemoteRunDatabase( - options.link, - yield* options.link.frontierSnapshot(), - ); - const journal = routeRemoteRunJournal(database, options.journal); - const provenance = establishJournalProvenance(journal); + const database = + options.database ?? + (yield* useRemoteRunDatabase(options.link, yield* options.link.frontierSnapshot())); + const journal = database.journal; + const origin = remoteRunOrigin(database); + // The handle has to be one this build opened, and opened from this exact + // link. A handle another client opened describes the same run and answers + // to another owner, and that is the pairing this refuses. + if (origin === undefined || origin.link !== options.link) { + unavailable("this is not a remote run handle this build opened from this link."); + } + const provenance = origin.provenance; // Installed here because here is where a run's three halves exist at once: // the acquisition it is reached through, the handle its execution transacts // on, and the witness over the exact journal an answer would be published diff --git a/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts new file mode 100644 index 000000000..d0a3660a7 --- /dev/null +++ b/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts @@ -0,0 +1,330 @@ +/** + * Tier WRH — the supported request boundary, on real workerd. + * + * The three planes stop being methods here and become requests. What that adds + * is everything a script cannot claim: the namespace arithmetic that decides + * which object answers, an actual WebSocket upgrade whose handshake a standard + * client would accept, the admission order as a caller experiences it through a + * status, and what a read or a delivery does while an executor is live. + * + * The client is the production one, configured exactly as a runner configures + * it, with its I/O pointed at the stub instead of at a network. So what is + * under test is the pair — this build's client against this build's owner — and + * not either half against a fixture of the other. + */ + +import { env, runInDurableObject } from "cloudflare:test"; +import { beforeAll, describe, expect, it } from "vitest"; +import { run, until, type Operation } from "effection"; +import { remoteOwnerClient } from "../../src/cloudflare/configured.ts"; +import type { + OwnerHttpRequest, + OwnerHttpResponse, + OwnerTransport, + OwnerUpgrade, + OwnerUpgradeRefused, +} from "../../src/cloudflare/configured.ts"; +import type { OwnerSocket, SocketListener } from "../../src/remote/client.ts"; +import { ownerRoute } from "../../src/cloudflare/gateway.ts"; +import { planePath, selectedProtocol } from "../../src/cloudflare/routes.ts"; +import type { ExecutorObject } from "./support/executor-object.ts"; +import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; +import { generateKeys, signToken, type TestKeys } from "./support/tokens.ts"; + +/** The clock the owner is configured with, so a token's window is exact. */ +const NOW = 1_800_000_000; +const ENDPOINT = "https://owner.invalid/workflow"; + +let keys: TestKeys; + +beforeAll(async () => { + keys = await generateKeys(); +}); + +let unique = 0; + +/** + * One run id, so a test's owner is its own. + * + * The id is what selects the object, so a fresh id is a fresh owner — which is + * also the arithmetic under test. + */ +function runOf(): string { + unique += 1; + return `${RUN_ID.slice(0, 40)}${unique}${Math.floor(Math.random() * 1_000_000)}`; +} + +/** The object the gateway would reach for this run. */ +function stubFor(runId: string) { + return env.EXECUTOR.get(env.EXECUTOR.idFromName(runId)); +} + +function on( + stub: ReturnType, + body: (instance: ExecutorObject) => T, +): Promise> { + return runInDurableObject(stub, body) as Promise>; +} + +/** A token this owner admits. */ +async function token(): Promise { + return await signToken(keys, { ...VALID_CLAIMS, iat: NOW - 10, nbf: NOW - 10, exp: NOW + 600 }); +} + +/** + * The namespace, as the gateway addresses it. + * + * `idFromName` and `get` and nothing else: the gateway is handed exactly what + * it needs to route, so a test cannot accidentally prove that it reached for + * more. + */ +const namespace = { + idFromName: (name: string) => env.EXECUTOR.idFromName(name), + get: (id: DurableObjectId) => env.EXECUTOR.get(id), +}; + +/** + * Every request one client made, and the transport that made it. + * + * The transport routes through the gateway, exactly as a Worker would: the + * request is built from the client's own URL and headers, and nothing about it + * is adjusted on the way. + */ +function transportTo( + routed: { readonly urls: string[] }, + route: (request: Request) => Promise, +): OwnerTransport { + return { + *request(request: OwnerHttpRequest): Operation { + routed.urls.push(request.url); + const response = yield* until( + route( + new Request(request.url, { + method: "POST", + headers: request.headers, + body: request.body, + }), + ), + ); + return { status: response.status, body: yield* until(response.text()) }; + }, + + *connect(upgrade: OwnerUpgrade): Operation { + routed.urls.push(upgrade.url); + const response = yield* until( + route( + new Request(upgrade.url, { + headers: { + upgrade: "websocket", + "sec-websocket-protocol": upgrade.protocols.join(", "), + }, + }), + ), + ); + const socket = response.webSocket; + if (socket === null) { + return { refusal: yield* until(response.text()) }; + } + // Selected explicitly by the owner, which is what a standard client + // requires when it offered a subprotocol at all. + expect(response.headers.get("sec-websocket-protocol")).toBe(selectedProtocol()); + socket.accept(); + return ownerSocket(socket); + }, + }; +} + +/** + * The runtime's socket, as the client's contract sees it. + * + * A `WebSocket` here carries the runtime's own event types, and the client + * needs only the data. The same adapter a runner's host supplies. + */ +function ownerSocket(socket: WebSocket): OwnerSocket { + const listeners = new Map(); + return { + send: (data) => socket.send(data), + close: () => socket.close(), + addEventListener(type, listener) { + const forward: EventListener = (event) => { + const data: unknown = Reflect.get(event, "data"); + listener(typeof data === "string" ? { data } : {}); + }; + listeners.set(listener, forward); + socket.addEventListener(type, forward); + }, + removeEventListener(type, listener) { + const found = listeners.get(listener); + if (found !== undefined) { + socket.removeEventListener(type, found); + } + }, + }; +} + +/** One configured client for one owner, with the gateway in front of it. */ +async function client(runId: string) { + await on(stubFor(runId), (owner) => + owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW), + ); + const minted = await token(); + const routed = { urls: [] as string[] }; + return { + routed, + client: remoteOwnerClient({ + runId, + endpoint: ENDPOINT, + release: POLICY.release, + // deno-lint-ignore require-yield + *token(): Operation { + return minted; + }, + transport: transportTo(routed, (request) => ownerRoute(namespace, request)), + }), + }; +} + +describe("the owner's request boundary", () => { + it("routes one run id to one object, arithmetically", async () => { + const runId = runOf(); + // The gateway is given the namespace and the request; which object answers + // is `idFromName` and nothing else, so the same id answers twice and a + // different id answers from somewhere else. + const first = await ownerRoute( + namespace, + new Request(`${ENDPOINT}${planePath(runId, "read")}`), + ); + expect(first.status).toBe(200); + const answered = await first.json(); + // Nothing is stored under that id, and the owner says so rather than + // creating anything. + // No release travelled on this bare request, and that is the first thing + // the owner asks about — before the token, and before the run. + expect(answered).toEqual({ outcome: "refused", refusal: "release:release-absent" }); + + // A path this build does not write reaches no object at all. + expect((await ownerRoute(namespace, new Request(`${ENDPOINT}/nope`))).status).toBe(404); + // Neither does an id that cannot address one. + expect( + (await ownerRoute(namespace, new Request(`${ENDPOINT}/runs/${"x".repeat(600)}/read`))).status, + ).toBe(400); + }); + + it("upgrades the executor plane, and takes the acquisition last", async () => { + const runId = runOf(); + const built = await client(runId); + const outcome = await run(function* () { + const admitted = yield* built.client.admit(runId); + if (!admitted.ok) { + return `failed:${admitted.error.message}`; + } + if (admitted.value === "already-running") { + return "already-running"; + } + // The connection is the acquisition, and the owner sees exactly one. + expect(yield* until(on(stubFor(runId), (owner) => owner.holders()))).toBe(1); + return "admitted"; + }); + expect(outcome).toBe("admitted"); + expect(built.routed.urls).toEqual([`${ENDPOINT}/runs/${runId}/executor`]); + // The scope that admitted it has ended, so the socket has gone and the run + // has no executor. + expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); + }); + + it("refuses an upgrade in the settled order, and takes nothing when it does", async () => { + const runId = runOf(); + await on(stubFor(runId), (owner) => + owner.configure([{ kid: keys.kid, jwk: keys.publicJwk }], NOW), + ); + const routed = { urls: [] as string[] }; + const transport = transportTo(routed, (request) => ownerRoute(namespace, request)); + + /** One upgrade attempt with exactly these admission values. */ + async function attempt(release: string, minted: string): Promise { + return await run(function* () { + const configured = remoteOwnerClient({ + runId, + endpoint: ENDPOINT, + release, + // deno-lint-ignore require-yield + *token(): Operation { + return minted; + }, + transport, + }); + const admitted = yield* configured.admit(runId); + return admitted.ok + ? admitted.value === "already-running" + ? "already-running" + : "admitted" + : admitted.error.message; + }); + } + + // A build this owner will not talk to is refused before the token is read, + // which is why a deliberately unusable token refuses the connection all the + // same — and the refusal a caller sees names neither the build nor the + // token, because a private category is not a public contract. + expect(await attempt("another-build", "not a token")).toContain("refused the operation"); + expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); + // An authenticated build with an unusable token is refused before the run + // is named, and still takes nothing. + expect(await attempt(POLICY.release, "not a token")).toContain("refused the operation"); + expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); + // And the one that passes both. + expect(await attempt(POLICY.release, await token())).toBe("admitted"); + expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); + }); + + it("answers a read while an executor is live, and takes no acquisition", async () => { + const runId = runOf(); + const built = await client(runId); + const outcome = await run(function* () { + const admitted = yield* built.client.admit(runId); + if (!admitted.ok || admitted.value === "already-running") { + return "not-admitted"; + } + expect(yield* until(on(stubFor(runId), (owner) => owner.holders()))).toBe(1); + // The read plane, while that acquisition is held. Nothing is stored, so + // the owner answers that the run is absent — over an ordinary request, + // with no second acquisition and no effect on the first. + const plane = yield* built.client.reads(runId); + if (!plane.ok) { + return `no-plane:${plane.error.message}`; + } + const inspected = yield* plane.value.inspect(); + // The acquisition is untouched by the read, whichever way the owner + // answered it. + expect(yield* until(on(stubFor(runId), (owner) => owner.holders()))).toBe(1); + return inspected.ok ? "answered" : "refused"; + }); + // Nothing is stored, so the owner refuses rather than inventing a run. + expect(outcome).toBe("refused"); + expect(built.routed.urls).toEqual([ + `${ENDPOINT}/runs/${runId}/executor`, + `${ENDPOINT}/runs/${runId}/read`, + ]); + expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); + }); + + it("retains a delivered value over its own request, taking no acquisition", async () => { + const runId = runOf(); + const built = await client(runId); + const outcome = await run(function* () { + // Nothing is stored, so this run is not waiting for anything — which is + // the owner's answer rather than a failure of the plane, and it is + // reached without an acquisition existing at any point. + const retained = yield* built.client.delivery.retain({ + runId, + suspensionId: "suspension-1", + answer: "answered", + secretDetection: false, + }); + return retained.ok ? "retained" : retained.error.name; + }); + expect(outcome).not.toBe("retained"); + expect(built.routed.urls).toEqual([`${ENDPOINT}/runs/${runId}/delivery`]); + expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); + }); +}); diff --git a/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts b/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts index 433108066..3106b56b0 100644 --- a/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-workspace.vitest.ts @@ -18,7 +18,7 @@ import { env, runInDurableObject } from "cloudflare:test"; import { beforeAll, describe, expect, it } from "vitest"; -import { InMemoryStream, type Workflow, type Json } from "@executablemd/durable-streams"; +import { type Workflow, type Json } from "@executablemd/durable-streams"; import { run, type Operation } from "effection"; import type { ExecutorObject } from "./support/executor-object.ts"; import { POLICY, RUN_ID, VALID_CLAIMS } from "./support/executor-object.ts"; @@ -138,7 +138,6 @@ function* opened(socket: WebSocket): Operation { files: host.files, trees: host.trees, createFilesystem: (at) => host.workspace(at("/")), - journal: new InMemoryStream(), }); } @@ -171,9 +170,11 @@ describe("the coordinator against a real owner", () => { yield* withRemoteWorkspaceEffects(opening, durableRun(workflow, { stream: opening.journal })); return yield* opening.journal.readAll(); }); - // The result travelled inside the commit, so the ordinary journal never - // saw it. - expect(outcome.filter((event) => event.type === "yield")).toHaveLength(0); + // The result travelled inside the commit rather than beside it: the run's + // journal — which is this owner's — holds the effect's row exactly once, + // and the owner's own row below shows it was written by the transaction + // that moved the root. + expect(outcome.filter((event) => event.type === "yield")).toHaveLength(1); const after = await on(stub, (owner) => owner.published()); // Content, root, references, mapping, pointer and the journal row moved @@ -181,8 +182,12 @@ describe("the coordinator against a real owner", () => { expect(after["currentRootId"]).not.toBe(before["currentRootId"]); expect(after["roots"]).toBe(2); expect(after["repositories"]).toEqual([{ name: "app", checkout_path: "/app" }]); + // Both rows the run wrote name the published root: the effect's own result, + // committed by the transaction that moved the pointer, and the terminal the + // completed run appended afterwards against the root it ended on. expect(after["events"]).toEqual([ expect.objectContaining({ workspace_root_id: after["currentRootId"] }), + expect.objectContaining({ workspace_root_id: after["currentRootId"] }), ]); // A fresh admitted invocation, through the production read link: the owner @@ -258,9 +263,12 @@ describe("the coordinator against a real owner", () => { expect(after["currentRootId"]).toBe(before["currentRootId"]); expect(after["roots"]).toBe(before["roots"]); expect(after["repositories"]).toEqual([]); - // One row, and it names the root the run is still on. + // Two rows, and both name the root the run is still on: the effect's own + // filtered failure, committed in its transaction, and the terminal the + // failed run appended afterwards. Neither moved the Workspace. expect(after["events"]).toEqual([ expect.objectContaining({ workspace_root_id: before["currentRootId"] }), + expect.objectContaining({ workspace_root_id: before["currentRootId"] }), ]); }); }); diff --git a/packages/workflow/tests/cloudflare/support/executor-object.ts b/packages/workflow/tests/cloudflare/support/executor-object.ts index d75abf7bf..467e4d250 100644 --- a/packages/workflow/tests/cloudflare/support/executor-object.ts +++ b/packages/workflow/tests/cloudflare/support/executor-object.ts @@ -26,6 +26,7 @@ import { STAGING_TABLE, } from "../../../src/cloudflare/private-schema.ts"; import { MARKER_TABLE } from "../../../src/cloudflare/marker.ts"; +import { routeOf } from "../../../src/cloudflare/routes.ts"; /** The identities this owner is configured to admit. */ export const POLICY: AdmissionPolicy = { @@ -773,7 +774,18 @@ export class ExecutorObject extends WorkflowOwnerObject { } } - async fetch(request: Request): Promise { + /** + * The legacy upgrade these suites were written against, and the real one. + * + * A request on one of the supported routes goes to the base object, which is + * where the production boundary lives. Everything else keeps the header shape + * the suites here already use — they are about admission and acquisition + * rather than about how a request is addressed. + */ + override async fetch(request: Request): Promise { + if (routeOf(new URL(request.url).pathname) !== undefined) { + return await super.fetch(request); + } const pair = new WebSocketPair(); const client = pair[0]; const server = pair[1]; diff --git a/packages/workflow/tests/remote-runner.test.ts b/packages/workflow/tests/remote-runner.test.ts new file mode 100644 index 000000000..4b95c120b --- /dev/null +++ b/packages/workflow/tests/remote-runner.test.ts @@ -0,0 +1,221 @@ +/** + * Tier WRH14 — the runner's four methods, and the handoff between two of them. + * + * A begin transition hands back a storage handle. An attachment needs the + * Workspace runtime for the *same* run, over the same connection — and two + * clients on two owners can hold handles whose run id, root and anchor are + * identical, so nothing a handle says about itself can establish that. What + * establishes it is where the handle came from. + * + * So this file is about which handles attach and which do not. An attachment + * that succeeds here has opened the run from the exact link its own acquisition + * produced, taken the provenance of that handle's own journal, and installed + * the coordinator for it — every one of which has to line up, or the attachment + * raises instead. What a real owner does with the commit such an attachment + * produces is proved against one in `tests/cloudflare/remote-workspace.vitest.ts`. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { Ok, type Operation, scoped } from "effection"; +import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; +import type { ExecutorLock } from "../src/lifecycle/api.ts"; +import type { WorkflowExecutionTransitions } from "../src/lifecycle/execution.ts"; +import type { WorkflowRunDatabase } from "../src/storage/api.ts"; +import { useRemoteWorkflowRunner } from "../src/deno/remote-runner.ts"; +import type { RemoteRunnerOwner, RemoteWorkflowRunner } from "../src/deno/remote-runner.ts"; +import type { RemoteReadPlane } from "../src/remote/read.ts"; +import { WorkflowRequestError } from "../src/storage/errors.ts"; +import { installedHost, RUN_ID, type Script } from "./support/remote-lifecycle-host.ts"; + +/** One scripted owner, and every run its acquisitions were opened for. */ +function ownerOf(script: Script = {}): { owner: RemoteRunnerOwner; acquisitions: string[] } { + const acquisitions: string[] = []; + const host = installedHost({ ...script, opened: acquisitions }); + return { + acquisitions, + owner: { + runId: RUN_ID, + admit: (runId: string) => host.admit(runId), + // deno-lint-ignore require-yield + *reads(runId: string) { + return Ok(readPlane(runId)); + }, + delivery: { + // deno-lint-ignore require-yield + *wait(): Operation { + throw new Error("PLANTED-DELIVERY-WAIT-REACHED"); + }, + // deno-lint-ignore require-yield + *retain(): Operation { + throw new Error("PLANTED-DELIVERY-RETAIN-REACHED"); + }, + }, + }, + }; +} + +/** + * One read plane, which answers nothing and takes nothing. + * + * What the tests below need of it is that installing it and asking it a + * question require no acquisition; what it would answer is the read plane's own + * contract and is proved where that is under test. + */ +function unanswered(): never { + throw new WorkflowRequestError("this scripted plane answers no read"); +} + +function readPlane(runId: string): RemoteReadPlane { + return { + runId, + // deno-lint-ignore require-yield + *inspect() { + return unanswered(); + }, + // deno-lint-ignore require-yield + *history() { + return unanswered(); + }, + // deno-lint-ignore require-yield + *forkSource() { + return unanswered(); + }, + }; +} + +/** One runner over one scripted owner. */ +function assembled(owner: RemoteRunnerOwner, scratch: string): Operation { + return useRemoteWorkflowRunner({ owner, scratchRoot: `/tmp/xmd-remote-runner-${scratch}` }); +} + +/** Take this run's acquisition, or say why it could not be taken. */ +function* acquired(): Operation { + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + if (!taken.ok) { + throw taken.error; + } + if (taken.value.kind !== "acquired") { + throw new Error("expected the executor acquisition to be taken"); + } + return taken.value.lock; +} + +/** Begin one execution, and hand back the handle it produced. */ +function* opened( + transitions: WorkflowExecutionTransitions, + lock: ExecutorLock, +): Operation { + const begun = yield* transitions.begin(lock, { runId: RUN_ID, action: "resume" }); + if (!begun.ok) { + throw begun.error; + } + return begun.value.database; +} + +/** What an attachment runs. Reaching it at all is the claim. */ +// deno-lint-ignore require-yield +function* attached(): Operation { + return "attached"; +} + +/** Attach one handle through one runner, and report what came back. */ +function* attaching(runner: RemoteWorkflowRunner, handle: WorkflowRunDatabase): Operation { + try { + return yield* scoped(() => runner.attach(handle, attached())); + } catch (error) { + return error instanceof Error ? error.message : "other"; + } +} + +function planted(): never { + throw new Error("PLANTED-FOREIGN-HANDLE-READ"); +} + +/** A handle nothing opened: shaped like one, and one nothing may read. */ +function foreignHandle(): WorkflowRunDatabase { + return { + get record() { + return planted(); + }, + get retrieval() { + return planted(); + }, + get journal() { + return planted(); + }, + readJournalEntries: planted, + transact: planted, + replaceRetrievalMetadata: planted, + readDocumentExecutions: planted, + }; +} + +describe("a runner for a run whose storage is somewhere else", () => { + it("attaches the handle its own lifecycle opened, and no other", function* () { + const outcome = yield* scoped(function* () { + const first = ownerOf(); + const second = ownerOf(); + const one = yield* assembled(first.owner, "one"); + const transitions = yield* one.useRunHost(); + const database = yield* opened(transitions, yield* acquired()); + // A second runner over a second owner, with an acquisition and a handle + // of its own. Its scripted owner answers with the same record, root and + // anchor, so the two handles agree about everything except where they + // came from. + return yield* scoped(function* () { + const other = yield* assembled(second.owner, "two"); + const theirs = yield* other.useRunHost(); + const another = yield* opened(theirs, yield* acquired()); + return { + own: yield* attaching(one, database), + theirs: yield* attaching(other, another), + crossed: yield* attaching(other, database), + back: yield* attaching(one, another), + foreign: yield* attaching(one, foreignHandle()), + acquisitions: [...first.acquisitions, ...second.acquisitions], + }; + }); + }); + // Attaching succeeded, which means the run was opened from the exact link + // this runner's acquisition produced, the provenance of that handle's own + // journal was taken, and the coordinator was installed for it. + expect(outcome.own).toBe("attached"); + expect(outcome.theirs).toBe("attached"); + // Neither runner can attach the other's handle, in either direction. + expect(outcome.crossed).toContain("not opened by this remote host"); + expect(outcome.back).toContain("not opened by this remote host"); + expect(outcome.foreign).toContain("not opened by this remote host"); + // One acquisition per runner, and neither of them for the other's run. + expect(outcome.acquisitions).toEqual([RUN_ID, RUN_ID]); + }); + + it("reads and delivers without taking an acquisition", function* () { + const outcome = yield* scoped(function* () { + const scripted = ownerOf(); + const built = yield* assembled(scripted.owner, "planes"); + yield* built.useLifecycle(); + yield* built.useDelivery(); + const inspected = yield* trapped(WorkflowLifecycle.operations.inspect(RUN_ID)); + return { + inspected, + // Nothing was acquired to install either plane or to answer with them. + acquisitions: scripted.acquisitions, + }; + }); + // The scripted plane answers no read, which is the plane refusing rather + // than an acquisition that was never taken. + expect(outcome.inspected).toContain("answers no read"); + expect(outcome.acquisitions).toEqual([]); + }); +}); + +/** Run one operation and report what it refused with, if it refused. */ +function* trapped(operation: Operation): Operation { + try { + yield* operation; + return "answered"; + } catch (error) { + return error instanceof Error ? error.message : "other"; + } +} diff --git a/packages/workflow/tests/remote-workspace.test.ts b/packages/workflow/tests/remote-workspace.test.ts index 341dcb4b6..1fa164fdf 100644 --- a/packages/workflow/tests/remote-workspace.test.ts +++ b/packages/workflow/tests/remote-workspace.test.ts @@ -286,7 +286,6 @@ function* owner(captured: CapturedWorkspace) { requests, next: () => `owner-${(identifier += 1)}`, commits: scripted.commits, - journal: new InMemoryStream(), }; } @@ -323,7 +322,6 @@ function* harness( files: runnerFiles(), trees: yield* useRunnerTrees(), createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), - journal: new InMemoryStream(), }); return { run, @@ -359,7 +357,16 @@ function* invocation( const raised = yield* trapped( withRemoteWorkspaceEffects(held.run, durableRun(workflow, { stream: held.run.journal })), ); - return { raised, events: yield* held.run.journal.readAll() }; + // Read back through the run's own journal, which is the owner's. An owner + // this harness scripted to refuse cannot answer that read, and the tests + // that ask for one are not the tests that scripted a refusal. + let events: DurableEvent[] = []; + try { + events = yield* held.run.journal.readAll(); + } catch { + events = []; + } + return { raised, events }; }); } @@ -373,6 +380,35 @@ function yielded(events: readonly DurableEvent[]): DurableEvent[] { return events.filter((event) => event.type === "yield"); } +/** + * The commands that carried a Workspace effect, out of everything the owner was + * asked to commit. + * + * A run's journal lives on its owner, so an ordinary append — the root `Close` + * of the invocation below, for one — reaches the owner as a commit of its own. + * That is the run being persisted where it belongs, and it is not what these + * tests are counting: what they are counting is how many times the coordinator + * proposed a Workspace transaction. + */ +function workspaceIntents(commits: readonly Record[]): Record[] { + return commits.filter((intent) => { + const events = intent["events"]; + return ( + Array.isArray(events) && + events.some((event) => { + const parsed: unknown = typeof event === "string" ? JSON.parse(event) : event; + const description = + parsed !== null && typeof parsed === "object" ? Reflect.get(parsed, "description") : null; + return ( + description !== null && + typeof description === "object" && + Reflect.get(description, "type") === "workspace" + ); + }) + ); + }); +} + describe("the runner's Workspace coordinator", () => { it("commits Files, one mapping and the filtered result as one intent", function* () { yield* scoped(function* () { @@ -391,8 +427,9 @@ describe("the runner's Workspace coordinator", () => { expect(raised).toBe(undefined); // Exactly one intent, carrying all three things together. - expect(held.commits).toHaveLength(1); - const intent = held.commits[0] ?? {}; + const intents = workspaceIntents(held.commits); + expect(intents).toHaveLength(1); + const intent = intents[0] ?? {}; expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); const mappings = intent["mappings"]; expect(Array.isArray(mappings) && mappings).toHaveLength(1); @@ -421,8 +458,9 @@ describe("the runner's Workspace coordinator", () => { expect(String(raised)).toContain("this Workspace effect refused"); // One commit, and it proposes nothing about the Workspace. - expect(held.commits).toHaveLength(1); - const intent = held.commits[0] ?? {}; + const intents = workspaceIntents(held.commits); + expect(intents).toHaveLength(1); + const intent = intents[0] ?? {}; expect(intent["publication"]).toBe(null); expect(intent["mappings"]).toEqual([]); expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); @@ -488,7 +526,7 @@ describe("the runner's Workspace coordinator", () => { expect(raised).not.toBe(undefined); // The owner said no, so nothing is promoted and nothing private crossed. expect(String(raised)).not.toContain("command:"); - expect(held.commits).toHaveLength(1); + expect(workspaceIntents(held.commits)).toHaveLength(1); }); }); @@ -580,7 +618,6 @@ describe("the runner's Workspace coordinator", () => { files: runnerFiles(), trees: yield* useRunnerTrees(), createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), - journal: new InMemoryStream(), }; let executed = 0; @@ -605,8 +642,10 @@ describe("the runner's Workspace coordinator", () => { // nothing else: no root, no content, no staging, no commit. expect(executed).toBe(1); expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); - expect(b.commits).toHaveLength(1); - expect(yielded(yield* a.journal.readAll())).toEqual([]); + expect(workspaceIntents(b.commits)).toHaveLength(1); + // A was never asked for anything else either — no read, no staging and + // no commit reached it after the one snapshot above. + expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); }); }); @@ -729,7 +768,7 @@ describe("the runner's Workspace coordinator", () => { // What must not happen is claiming it did. expect(raised).not.toBe(undefined); expect(String(raised)).not.toContain("command:"); - expect(held.commits).toHaveLength(1); + expect(workspaceIntents(held.commits)).toHaveLength(1); }); }); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 73a0e8da2..2bae3548b 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -10051,6 +10051,8 @@ malformed-input permutation without a distinct structural consequence. Most of what they name is **specified; implementation unbuilt**, so those tiers name the scenarios an implementation is accepted against rather than tests that exist. Tier WRH is the exception: rows WRH1-WRH16 are implemented and have committed evidence, listed with the tier, while WRH17-WRH22 remain unbuilt with the machine-wait work they belong to. Nothing in the other six tiers is built. +WRH14 covers the configured host as well as the boundary it satisfies: `packages/cli/tests/remote-workflow-host.test.ts` proves that the explicit installer has exactly the four methods, that it is bound to one run and refuses another before a token is minted, that reads and deliveries take no acquisition while execution takes one socket, and that a storage handle it did not open cannot be attached; `packages/workflow/tests/remote-runner.test.ts` proves the lifecycle-to-attachment handoff, including that two clients holding handles which agree about run id, root and anchor still cannot attach each other's; and `packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts` proves the request boundary against a real Durable Object — namespace routing, an actual upgrade whose subprotocol is selected, the admission order as a status, and a read and a delivery answered while an executor is live. No selector chooses that host: the shipped Deno and compiled entrypoints install the local one and Node and Bun remain unsupported. + ### Tier WRH — Remote host, executor and delivery separation Defined in [Workflow workspaces](./workflow-workspace-spec.md) §3.8 and §13.2 diff --git a/specs/workflow-spec.md b/specs/workflow-spec.md index a4f210fb7..c38d37387 100644 --- a/specs/workflow-spec.md +++ b/specs/workflow-spec.md @@ -927,6 +927,8 @@ only then may execution continue past the wait. **Fork copies from one selected source, or does nothing.** A fork names one source prefix and root and produces one destination run and one lineage. The destination commits whole or not at all, its copied prefix outlives the source, and an incompatible selection or a failure part-way mutates neither the source nor the destination. +**The planes are three requests, and one client carries them.** Which plane a request is for is its path, and the run id in that path is what selects the owner — arithmetically, through the namespace's own naming, with a gateway that forwards on that id and decides nothing else. A trusted host reaches all three through one configured client bound to one run: it holds an already-selected run id, one credential-free endpoint parsed before anything is sent, the exact release identity, and an operation that mints a short-lived token for the immediate request. Every plane requires the configured run id before a token exists, a URL is built or any I/O happens; the endpoint, the release and the token stay in the host's closure and reach no record, event or diagnostic; and the request shapes, header names and refusal spellings stay private to one release, save for the one category that is a fact about the run — another live executor holds it. + **A handle's authority ends with its scope.** Teardown closes the handle and, for an executor, releases ownership; it rolls back nothing already committed and settles nothing that was not settled. Closing the connection is the only staleness proof a remote host needs, and a closed one authorizes nothing while leaving every committed transaction exactly as it was. ### 9.9 A retained terminal, and what may be concluded from it diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 09fbaebb5..f75cae5b2 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -3794,11 +3794,19 @@ Privacy of the transport is not privacy of the authority. These constraints are Delivery and inspection reach the owner without an acquisition, under §3.8, on authenticated paths of their own. -#### What the owner implements, and what assembling it still needs +#### What the owner implements, and how a runner reaches it Everything above is implemented behind the four methods and nothing wider: creating a run and finding it, coherent reads of its committed state, lifecycle transitions and terminal settlement, stale recovery at the next acquisition, fork and the staged candidate a fork is admitted by, root validation and atomic Workspace publication, typed delivery and its later consumption, read-only inspection and history, and canonical completed replay. The owner refuses in its own vocabulary and mutates nothing when it does, on every one of those paths. -What does not exist yet is the assembly that would let a caller choose that host: there is no configured public client factory, no runtime selector, no endpoint, release-identity or OIDC supplier assembly, no CLI flag, prop or environment reading for any of it, and no deployment. The shipped CLI assembles the local host only, and none of the behavior above waits on that assembly — what it waits on is somebody deciding, in configuration, which host a run belongs to. +The three planes are three requests, and the path says which. One public run id selects its owner arithmetically through the namespace's own `idFromName`, and a gateway in front of the objects routes on that id and forwards the request whole — it parses no command, verifies no token and commits nothing, because an owner that trusted a gateway's account of any of those would have moved its own admission outside itself. The executor plane is a real WebSocket upgrade: the owner creates the pair, checks the release before the token and the token before the run, registers the acquisition last, and hands back the client half. The read and delivery planes are ordinary authenticated requests that accept no socket and take no acquisition. + +A runner reaches all three through one configured client, which is the supported public surface: an already-selected run id, one credential-free endpoint parsed at construction, the exact release identity, an operation that mints a short-lived token for the immediate request, and the HTTP and WebSocket I/O the host performs. The client is bound to that one run — every plane requires the configured id before a token exists, a URL is built or anything is sent — and the endpoint, the release and the token stay in its closure, reaching no workflow record, journal event, public error or document-visible value. What crosses on the planes stays private: the paths, the header names, the commands and the refusal spellings are one release talking to itself, and the one refusal category that is a fact about the run rather than about the connection — another live executor holds it — is the only one that becomes a public answer. + +Trusted code assembles that client into the same four-method host the local entrypoint installs. `useRunHost()` installs the executor lifecycle over the client's acquisition and returns the provider-neutral transitions; `useLifecycle()` installs status, list and history over the read plane, where the complete domain of one bound owner is zero or one coherent snapshot and no namespace-wide claim; `useDelivery()` installs typed answer delivery over the delivery plane; and `attach()` installs the runner's Workspace coordinator around a live or partial execution. A completed replay reaches none of it: it reads the owner and opens and closes its own execution envelope, and `attach()` is never called. + +**An attachment is bound to the handle that produced it.** The handle a begin transition hands back carries its own routed journal and the provenance taken over it, and it remembers the exact link it was opened from. Attaching compares that link to the ones this runner's own acquisitions produced, by object identity. Two clients on two owners can hold handles whose run id, root and journal anchor are identical, so nothing a handle says about itself could establish which run it is; what establishes it is where it came from, and a handle another client opened — or one shaped like a handle and opened by nothing — is refused before a temporary tree, a materialization or a request exists. + +What does not exist yet is anything that would *choose* this host. There is no runtime or CLI selector, no ambient endpoint, release-identity or OIDC source, no flag, prop or environment reading for any of it, and no deployment. The shipped Deno and compiled entrypoints install the local host, Node and Bun remain unsupported hosts, and the explicit installer above is the seam a later trusted factory assembly passes its configuration to. #### What completed replay does and does not reach @@ -3812,7 +3820,7 @@ A completed run replays there as it does locally: it attaches no Workspace, Agen | retained run record and filtered journal | built by #291 | | caller-owned storage transaction | built by #291; Workspace mutations join it in #365 | | provider-backed retained Workspace | document filesystem built by #366 and repository composition by #293; document deletion (§10.1) built by #567 for both providers; process capabilities unbuilt (#218) | -| `xmd workflow start` / `resume` | built by #366, Deno entrypoints only; both acquire #367's executor lock. The same two lifecycle actions are implemented against the durable owner, where the acquisition is the connection rather than the lock; no CLI selector reaches them yet | +| `xmd workflow start` / `resume` | built by #366, Deno entrypoints only; both acquire #367's executor lock. The same two lifecycle actions are implemented against the durable owner, where the acquisition is the connection rather than the lock, and trusted code assembles that host explicitly (§13.2); no selector chooses it, so the shipped entrypoints are local | | ``, `` and `` composition | built by #293, Deno provider only | | transactional Git components (`Git.Switch`, `Git.Add`, `Git.Commit`) | built by #294, Deno provider only | | `` read and upsert, and the `issue_effect` boundary (§10.3) | built by #296; GitHub middleware, Deno host | @@ -3840,7 +3848,7 @@ A completed run replays there as it does locally: it attaches no Workspace, Agen | `Evidence.Run` trusted native evidence execution (§10.5) | specified by #710; implementation unbuilt | | `Project.Status` and the Project-provider boundary (§10.6) | specified by #710; implementation unbuilt | | factory protocol records consumed by these effects | specified by #710 and owned normatively by [the software factory](./github-actions-software-factory-spec.md) §11.2, which this specification links to rather than duplicating: `Git.Merge`'s publish ceiling reads the Stage 7 decision, `PullRequest.Merged`'s wait is one of those records, and `Project.Status` projects a stage through the configured stage-to-option table | -| remote lifecycle host, executor connection, versioned runner transport and remote topology (§3.8, §13.2) — the existing four-method `WorkflowHost` boundary, with a Cloudflare implementation beside the Deno one | the internal implementation is built by #698: one SQLite-backed Durable Object per run, connection-lifetime acquisition, owner-side parsing and transactions, release identity at admission, and provider-neutral remote adapters behind the same four methods. The configured public host assembly — client factory, runtime selector, endpoint/release/OIDC supplier assembly, CLI configuration and deployment — is unbuilt | +| remote lifecycle host, executor connection, versioned runner transport and remote topology (§3.8, §13.2) — the existing four-method `WorkflowHost` boundary, with a Cloudflare implementation beside the Deno one | built by #698: one SQLite-backed Durable Object per run, connection-lifetime acquisition, owner-side parsing and transactions, release identity at admission, the three request planes and the gateway that routes to them, one configured client bound to one run, and an explicit trusted assembly of all of it into the same four methods. What remains unbuilt is choosing it: no runtime or CLI selector, no ambient endpoint/release/OIDC source, and no deployment | | terminal-decision delivery on the delivery plane (§3.8) | specified by #710; implementation unbuilt. Answer delivery and consumption on that plane are built by #698; a terminal decision as a delivery subject belongs to the factory records that define it | | canonical completed replay from retained state (§9, §13.2) | built by #698 for both providers: the retained root document and the bundle admission come from the run's own committed state, component bytes are authenticated by Git blob identity, an unreadable terminal, import or selection refuses before anything is replayed, and a coherent replay changes only the execution envelope it opened | | Worker JavaScript | deferred | From 2adb4fc91d11beb07960722194e6d731305f2389 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 12:35:37 -0300 Subject: [PATCH 72/81] =?UTF-8?q?=F0=9F=90=9B=20Give=20a=20remote=20run=20?= =?UTF-8?q?the=20attachment=20a=20document=20actually=20reaches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fourth method installed the Workspace coordinator and stopped. So an authored `` on a remote run resolved through whatever Files provider was in scope — a runner has the ambient host one installed by its entrypoint — and a document wrote to the runner's own filesystem instead of the run's Workspace. The new regression reproduces exactly that against the previous revision. An attachment now installs what a live or partial document reaches: the logical Workspace working directory, the document filesystem, the Repository, Worktree and `` composition, transactional Git, the composition components, the pull-request and Issue middleware, elicitation, and whatever Agent profile its host configured. The set is inseparable — the Files provider alone would resolve authored paths against the surrounding host's working directory — and it is one set: `withDocumentCapabilities()` is what both hosts install, so there is one account of what ``, `` and the Git components mean. What a host contributes to those rules is two things. The effect a mutation becomes, and the savepoint that undoes a part of one mutation which could not be finished. Locally that is the lease it validated and a real transaction savepoint; on a runner it is the exact remote run its own acquisition opened, and an attempt restored from the accepted root — correct because one effect performs one mutation. Which of the two answers is decided by the exact storage handle. An attachment registers its binding under the handle it was given and removes it when the attachment ends, so a handle another client opened finds that client's binding or none, a handle nobody attached is refused rather than performed, and no context value is what says a filesystem, journal or publication target belongs to this run. --- architecture.md | 2 + .../workflow/src/deno/composition/effects.ts | 7 +- packages/workflow/src/deno/remote-files.ts | 4 + packages/workflow/src/deno/remote-runner.ts | 35 +- packages/workflow/src/deno/transaction.ts | 43 +- .../workflow/src/deno/workspace/effect.ts | 15 + packages/workflow/src/deno/workspace/files.ts | 12 +- packages/workflow/src/deno/workspace/host.ts | 90 ++-- packages/workflow/src/remote/invocation.ts | 21 + packages/workflow/src/remote/materialize.ts | 7 + packages/workflow/src/remote/workspace.ts | 19 + packages/workflow/src/workspace/effects.ts | 107 ++++ packages/workflow/src/workspace/savepoint.ts | 56 +++ .../cloudflare/remote-owner-routes.vitest.ts | 72 ++- .../tests/cloudflare/support/worker-files.ts | 11 + .../workflow/tests/remote-publication.test.ts | 7 +- packages/workflow/tests/remote-runner.test.ts | 459 +++++++++++++++++- specs/executable-mdx-spec.md | 2 +- specs/workflow-workspace-spec.md | 6 +- 19 files changed, 891 insertions(+), 84 deletions(-) create mode 100644 packages/workflow/src/workspace/effects.ts create mode 100644 packages/workflow/src/workspace/savepoint.ts diff --git a/architecture.md b/architecture.md index 00320b4b8..2c6ed23a1 100644 --- a/architecture.md +++ b/architecture.md @@ -443,6 +443,8 @@ What that owner implements, it implements behind the four-method `WorkflowHost` A runner reaches it over three requests, and the path says which plane. One run id selects its owner arithmetically, a gateway forwards on that id alone, and the executor plane is a real upgrade whose admission order — release, then token, then the run — is the owner's own. One configured client carries all three: an already-selected run id, one credential-free endpoint parsed at construction, the release identity, a token minted per request, and the I/O its host performs. It is bound to that one run, refuses another before a token exists, and keeps endpoint, release and token in closure state that reaches no record, event, error or document-visible value. Trusted code assembles that client into the same four methods, and an attachment is bound to the exact storage handle the begin transition produced — compared by identity to the link this runner's own acquisition opened, because two clients on two owners can hold handles that agree about everything except where they came from. +A remote attachment installs what a local one installs, because the document rules are written once. Filesystem, Repository, Worktree, ``, Git, composition components, pull-request and Issue middleware, elicitation and any configured Agent profile all come from the same modules; what each host contributes is the effect a mutation becomes and the savepoint that undoes an unfinishable part of one. The exact storage handle decides which of those two bindings answers, so an authored `` on a remote run becomes a durable Workspace effect against the invocation's materialized tree and never falls through to the runner's own filesystem — and a handle no attachment registered is refused rather than performed. + What no code chooses is which host a run belongs to. There is no runtime or CLI selector, no ambient endpoint, release-identity or OIDC source, and no deployment: the shipped entrypoints install the local host, and the explicit installer is the seam a later trusted assembly configures. ## The workflow lifecycle diff --git a/packages/workflow/src/deno/composition/effects.ts b/packages/workflow/src/deno/composition/effects.ts index efc86fc88..c28f1bfb6 100644 --- a/packages/workflow/src/deno/composition/effects.ts +++ b/packages/workflow/src/deno/composition/effects.ts @@ -21,9 +21,10 @@ import { RepositoryCompositionProtocolError, } from "../../composition/errors.ts"; import type { WorkflowRunDatabase } from "../../storage/api.ts"; +import { workspaceEffectsFor } from "../../workspace/effects.ts"; +import type { WorkspaceFilesystem } from "../../workspace/filesystem.ts"; import { WorkflowStorageError } from "../../storage/errors.ts"; import { savepoint } from "../transaction.ts"; -import { createWorkspaceEffect } from "../workspace/effect.ts"; import { isJournalableWorkspaceFailure } from "../workspace/errors.ts"; import type { DenoWorkspaceFilesystem } from "../workspace/filesystem.ts"; import type { WorkspaceMetadata } from "../workspace/repositories.ts"; @@ -153,11 +154,11 @@ function* compositionEffect( database: WorkflowRunDatabase, description: EffectDescription, perform: ( - filesystem: DenoWorkspaceFilesystem, + filesystem: WorkspaceFilesystem, metadata: WorkspaceMetadata, ) => Operation, ): Workflow { - return yield createWorkspaceEffect(database, description, perform); + return yield workspaceEffectsFor(database).create(description, perform); } /** diff --git a/packages/workflow/src/deno/remote-files.ts b/packages/workflow/src/deno/remote-files.ts index 1e807706b..8244385d8 100644 --- a/packages/workflow/src/deno/remote-files.ts +++ b/packages/workflow/src/deno/remote-files.ts @@ -103,6 +103,10 @@ export function runnerFiles(): RunnerFiles { yield* until(mkdir(path, { recursive: false, mode })); }, + *removeTree(path: string): Operation { + yield* until(rm(path, { recursive: true, force: true })); + }, + *writeFile(path: string, bytes: Uint8Array, mode: number): Operation { yield* until(writeFile(path, bytes, { mode })); }, diff --git a/packages/workflow/src/deno/remote-runner.ts b/packages/workflow/src/deno/remote-runner.ts index ffa1f6cba..936a690e9 100644 --- a/packages/workflow/src/deno/remote-runner.ts +++ b/packages/workflow/src/deno/remote-runner.ts @@ -30,6 +30,7 @@ */ import type { Operation, Result } from "effection"; +import type { DurableEffect, EffectDescription, Json } from "@executablemd/durable-streams"; import type { WorkflowExecutionTransitions } from "../lifecycle/execution.ts"; import type { WorkflowRunDatabase } from "../storage/api.ts"; import { @@ -43,10 +44,14 @@ import type { RemoteReadPlane } from "../remote/read.ts"; import { installRemoteInputDelivery } from "../remote/delivery.ts"; import { useRemoteLifecycleReads } from "../remote/inspection.ts"; import { + createRemoteWorkspaceEffect, useRemoteRun, useRemoteWorkspaceEffects, withRemoteWorkspaceEffects, } from "../remote/workspace.ts"; +import { useWorkspaceEffects, type WorkspaceMutation } from "../workspace/effects.ts"; +import { withDocumentCapabilities } from "./workspace/host.ts"; +import type { WorkflowWorkspaceOptions } from "./workspace/host.ts"; import { WorkflowRequestError } from "../storage/errors.ts"; import { installRemoteWorkflowLifecycle } from "./remote-host.ts"; import { createRemoteWorkspaceFilesystem } from "./remote-workspace-files.ts"; @@ -75,6 +80,17 @@ export interface RemoteWorkflowRunnerOptions { * process may write, supplied explicitly rather than read from anywhere. */ readonly scratchRoot: string; + /** + * What a live or partial attachment installs beyond the run's own Workspace. + * + * The host-owned inputs the optional capabilities need — the credential + * helper, the Issue and pull-request configuration, and the Agent profile + * installer — supplied explicitly by whoever assembled this runner. Nothing + * here is read from a flag, an environment variable, a prop or a global, and + * an absent member installs the capability's unconfigured behavior rather + * than a different one. + */ + readonly capabilities?: WorkflowWorkspaceOptions; } /** What a host installs for a run whose storage is somewhere else. */ @@ -158,7 +174,24 @@ export function* useRemoteWorkflowRunner( createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), }); yield* useRemoteWorkspaceEffects(run); - return yield* withRemoteWorkspaceEffects(run, operation); + // The document's own capabilities, over this exact binding. Installed + // together because either half alone is wrong: the rules without the + // binding would reach whatever filesystem an entrypoint left in scope, + // and the binding without the rules would be a coordinator nothing asks. + yield* useWorkspaceEffects(database, { + create( + description: EffectDescription, + mutate: WorkspaceMutation, + ): DurableEffect { + return createRemoteWorkspaceEffect(run, description, (filesystem, metadata) => + mutate(filesystem, metadata), + ); + }, + }); + return yield* withRemoteWorkspaceEffects( + run, + withDocumentCapabilities(database, operation, options.capabilities ?? {}), + ); }, }; } diff --git a/packages/workflow/src/deno/transaction.ts b/packages/workflow/src/deno/transaction.ts index 1d979f8a6..43fe167c7 100644 --- a/packages/workflow/src/deno/transaction.ts +++ b/packages/workflow/src/deno/transaction.ts @@ -15,9 +15,7 @@ * database, so nothing above this boundary can reach SQLite through it. */ -import { type Api, createApi } from "@effectionx/context-api"; import { type Context, createContext, type Operation } from "effection"; -import { WorkflowTransactionError } from "../storage/errors.ts"; import type { RunTransaction } from "./connections.ts"; import type { SavepointManager } from "./savepoints.ts"; @@ -63,41 +61,14 @@ export function* holdsTransactionOn(path: string): Operation { return false; } -export interface TransactionApi { - /** - * Run `body` inside a savepoint, discarding its work if it fails. - * - * Answers with what the body answered. A failure rolls the savepoint back - * and propagates, leaving the surrounding transaction open and free to - * continue or to fail on its own terms. - */ - savepoint(body: Operation): Operation; -} - -/** No transaction is open in this scope, so there is nothing to nest inside. */ -export class NoOpenTransactionError extends WorkflowTransactionError { - override name = "NoOpenTransactionError"; - - constructor() { - super( - "a savepoint needs a transaction to be inside, and this scope is not inside one. " + - "Take savepoints within the body a transaction hands you.", - ); - } -} - -export const Transaction: Api = createApi( - "executablemd.workflow.deno.savepoint", - { - // deno-lint-ignore require-yield - *savepoint(_body: Operation): Operation { - throw new NoOpenTransactionError(); - }, - }, -); +import { Transaction } from "../workspace/savepoint.ts"; -/** The savepoint operation, for whoever is inside a transaction. */ -export const savepoint: TransactionApi["savepoint"] = Transaction.operations.savepoint; +export { + NoOpenTransactionError, + savepoint, + Transaction, + type TransactionApi, +} from "../workspace/savepoint.ts"; /** What the open transaction installs so `savepoint()` can answer. */ export function useTransactionSavepoints( diff --git a/packages/workflow/src/deno/workspace/effect.ts b/packages/workflow/src/deno/workspace/effect.ts index c0e58086c..066e1415a 100644 --- a/packages/workflow/src/deno/workspace/effect.ts +++ b/packages/workflow/src/deno/workspace/effect.ts @@ -22,6 +22,10 @@ import { savepoint } from "../transaction.ts"; import { isJournaledEffectFailure } from "./errors.ts"; import type { DenoWorkspaceFilesystem } from "./filesystem.ts"; import type { WorkspaceMetadata } from "./repositories.ts"; +import { + useWorkspaceEffects as bindWorkspaceEffects, + type WorkspaceMutation, +} from "../../workspace/effects.ts"; import { type PrivateWorkspaceTransaction, withPrivateWorkspaceTransaction, @@ -227,6 +231,17 @@ export function withWorkspaceEffects( return unavailable(); } connections.validateLease(database); + // The lease this host just validated, as the one binding the shared + // document rules build their effects through. It closes with this scope, + // so an effect created afterwards finds no binding at all. + yield* bindWorkspaceEffects(database, { + create( + description: EffectDescription, + mutate: WorkspaceMutation, + ): DurableEffect { + return createWorkspaceEffect(database, description, mutate); + }, + }); return yield* withWorkspaceCoordinationProvider(coordinator(connections, database), operation); }); } diff --git a/packages/workflow/src/deno/workspace/files.ts b/packages/workflow/src/deno/workspace/files.ts index 6f2a72b30..a4fbe7e0e 100644 --- a/packages/workflow/src/deno/workspace/files.ts +++ b/packages/workflow/src/deno/workspace/files.ts @@ -73,8 +73,9 @@ import type { } from "@executablemd/runtime"; import type { EffectDescription, Json, Workflow } from "@executablemd/durable-streams"; import type { WorkflowRunDatabase } from "../../storage/api.ts"; +import { workspaceEffectsFor } from "../../workspace/effects.ts"; +import type { WorkspaceFilesystem } from "../../workspace/filesystem.ts"; import { savepoint } from "../transaction.ts"; -import { createWorkspaceEffect } from "./effect.ts"; import { journalableWorkspaceCode } from "./errors.ts"; import type { DenoWorkspaceFilesystem, DenoWorkspaceStat } from "./filesystem.ts"; import { @@ -277,9 +278,14 @@ function* describeFileEffect( function* fileEffect( database: WorkflowRunDatabase, description: EffectDescription, - perform: (filesystem: DenoWorkspaceFilesystem) => Operation>, + perform: (filesystem: WorkspaceFilesystem) => Operation>, ): Workflow { - return yield createWorkspaceEffect(database, description, (filesystem) => perform(filesystem)); + // The binding the host attached for this exact handle. Which host answers is + // decided by the handle; that the handle is this run's was proved when the + // attachment registered it. + return yield workspaceEffectsFor(database).create(description, (filesystem) => + perform(filesystem), + ); } /** diff --git a/packages/workflow/src/deno/workspace/host.ts b/packages/workflow/src/deno/workspace/host.ts index cba1bf0fd..05a866abb 100644 --- a/packages/workflow/src/deno/workspace/host.ts +++ b/packages/workflow/src/deno/workspace/host.ts @@ -143,45 +143,65 @@ export interface WorkflowAgentAttachment { export type WorkflowAgentInstaller = (attachment: WorkflowAgentAttachment) => Operation; +/** + * Run `operation` with the document's own capabilities installed for this run. + * + * Everything an authored document reaches, and nothing about where the run's + * storage lives. The caller has already bound this run's Workspace effects — + * locally to a validated lease, on a runner to the exact remote run its own + * acquisition opened — and every rule below builds its effects through that one + * binding. Which means both hosts get the same ``, ``, + * ``, `` and Git behavior, with authority staying in whichever + * binding was attached. + * + * The set is inseparable for the reason the module note above gives: the Files + * provider alone would resolve a document's paths against whatever working + * directory the surrounding host adapter answers with. + */ +export function withDocumentCapabilities( + database: WorkflowRunDatabase, + operation: Operation, + options: WorkflowWorkspaceOptions = {}, +): Operation { + return scoped(function* () { + yield* useLogicalWorkspaceCwd(); + yield* useWorkflowFiles(database); + const composition = { + ...options.composition, + ...(options.helper === undefined ? {} : { helper: options.helper }), + }; + yield* useRepositoryComposition(database, composition); + yield* useGitComposition(database, composition); + if (options.gitHubIssues !== undefined) { + yield* useGitHubIssues(options.gitHubIssues); + } + yield* useCompositionComponents(); + // Ordinary middleware, installed the way the Issue adapter is: it owns + // the URLs it recognizes and delegates the rest. + // Installed on every live or partial attachment, configured or not: the + // configuration governs URL reads, and `` must keep working + // on a host that authorizes none. + yield* useGitHubPullRequests( + database, + composition.host ?? denoRepositoryHost(), + options.gitHubPullRequests ?? {}, + ); + // After the composition components and inside this attachment: a + // completed replay never reaches here, so it registers no second `Elicit` + // and installs no provider for work that is not going to happen. + yield* useWorkflowElicitation(); + if (options.agent !== undefined) { + yield* options.agent({ runId: database.record.runId, database }); + } + return yield* operation; + }); +} + /** Run `operation` with this run's Workspace attached to the document. */ export function withWorkflowWorkspace( database: WorkflowRunDatabase, operation: Operation, options: WorkflowWorkspaceOptions = {}, ): Operation { - return withWorkspaceEffects( - database, - scoped(function* () { - yield* useLogicalWorkspaceCwd(); - yield* useWorkflowFiles(database); - const composition = { - ...options.composition, - ...(options.helper === undefined ? {} : { helper: options.helper }), - }; - yield* useRepositoryComposition(database, composition); - yield* useGitComposition(database, composition); - if (options.gitHubIssues !== undefined) { - yield* useGitHubIssues(options.gitHubIssues); - } - yield* useCompositionComponents(); - // Ordinary middleware, installed the way the Issue adapter is: it owns - // the URLs it recognizes and delegates the rest. - // Installed on every live or partial attachment, configured or not: the - // configuration governs URL reads, and `` must keep working - // on a host that authorizes none. - yield* useGitHubPullRequests( - database, - composition.host ?? denoRepositoryHost(), - options.gitHubPullRequests ?? {}, - ); - // After the composition components and inside this attachment: a - // completed replay never reaches here, so it registers no second `Elicit` - // and installs no provider for work that is not going to happen. - yield* useWorkflowElicitation(); - if (options.agent !== undefined) { - yield* options.agent({ runId: database.record.runId, database }); - } - return yield* operation; - }), - ); + return withWorkspaceEffects(database, withDocumentCapabilities(database, operation, options)); } diff --git a/packages/workflow/src/remote/invocation.ts b/packages/workflow/src/remote/invocation.ts index 923df77c7..656706d27 100644 --- a/packages/workflow/src/remote/invocation.ts +++ b/packages/workflow/src/remote/invocation.ts @@ -85,6 +85,15 @@ export interface Attempt extends SealableAttempt { readonly at: HostPath; /** What the attempt describes right now, captured and checked locally. */ capture(): Operation; + /** + * Put this attempt back to the accepted root it started from. + * + * What a savepoint means on a tree. One mutation performs one change, so a + * part of it that cannot be finished leaves nothing to keep — and the + * accepted materialization is still exactly what the owner confirmed, so the + * attempt is rebuilt from it rather than repaired. + */ + restore(): Operation; } /** @@ -182,6 +191,18 @@ export function useAttempt( return yield* captureWorkspace(files, at(root), reject); }, + *restore(): Operation { + yield* files.removeTree(root); + yield* files.makeDirectory(root, 0o700); + yield* materializeWorkspaceRoot( + files, + reads, + at(root), + materialization.workspaceRootId, + reject, + ); + }, + /** * Seal this attempt into the proposal the owner will decide. * diff --git a/packages/workflow/src/remote/materialize.ts b/packages/workflow/src/remote/materialize.ts index 114bb00ed..d0523f0fc 100644 --- a/packages/workflow/src/remote/materialize.ts +++ b/packages/workflow/src/remote/materialize.ts @@ -92,6 +92,13 @@ export interface RunnerFiles { /** The same, for a link's own permissions. `undefined` where unsupported. */ readonly setLinkMode: ((path: string, mode: number) => Operation) | undefined; readFile(path: string): Operation; + /** + * Remove one tree this invocation owns, and everything under it. + * + * What an attempt is put back with. Only ever called with a directory this + * invocation materialized, and only by the attempt that owns it. + */ + removeTree(path: string): Operation; /** One directory's entries, described without following a link. */ list(path: string): Operation; /** One path, described without following a link. */ diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts index 46954e28e..c1c698098 100644 --- a/packages/workflow/src/remote/workspace.ts +++ b/packages/workflow/src/remote/workspace.ts @@ -54,6 +54,7 @@ import type { WorkspaceMetadata } from "../workspace/metadata.ts"; import type { AgentSessions } from "../storage/agent-session.ts"; import { activeWorkspaceRoute, type WorkspaceRoute } from "./database.ts"; import { createInvocationMappings } from "./mappings.ts"; +import { Transaction } from "../workspace/savepoint.ts"; import { type Attempt, type Materialization, @@ -460,6 +461,24 @@ function* coordinateTransaction( let result: DurableResult; try { const value = yield* scoped(function* () { + // The savepoint the shared rules ask for when part of one mutation + // cannot be finished. The attempt is disposable by construction, so + // undoing that part is restoring the attempt from the accepted root — + // correct because one effect performs one mutation, so nothing else in + // this body has changed anything a caller still needs. + yield* Transaction.around( + { + *savepoint([body]: [Operation]): Operation { + try { + return yield* body; + } catch (error) { + yield* attempt.restore(); + throw error; + } + }, + }, + { at: "min" }, + ); yield* WorkspaceMutation.around( { *run([candidate, mutate]): Operation { diff --git a/packages/workflow/src/workspace/effects.ts b/packages/workflow/src/workspace/effects.ts new file mode 100644 index 000000000..71c9dec52 --- /dev/null +++ b/packages/workflow/src/workspace/effects.ts @@ -0,0 +1,107 @@ +/** + * How a document's Workspace work becomes one durable effect, whichever host + * holds the run. + * + * There is one set of rules for what ``, ``, ``, + * `` and the Git components mean, and there are two places a run's storage + * can live. Writing the rules twice is how the two would start to differ, so + * the rules stay where they are and this is the one thing they ask a host for: + * turn this mutation into the effect that performs it. + * + * The host answers with an effect bound to what it already proved. The local + * host closes over the lease it validated; the runner closes over the exact + * remote run its own acquisition opened. Neither authority is in this contract + * and neither can be supplied by a caller. + * + * Which binding answers is decided by the exact handle, not by anything a scope + * can reach. An attachment registers its binding under the handle it was given + * — the one its own lifecycle produced — and unregisters it when the attachment + * ends. So a document holding a second run's handle finds that run's binding or + * none, a handle nobody attached finds none, and a replaceable context value is + * never what says a database, journal or publication target belongs to a run. + */ + +import { ensure, type Operation } from "effection"; +import type { DurableEffect, EffectDescription, Json } from "@executablemd/durable-streams"; +import type { WorkflowRunDatabase } from "../storage/api.ts"; +import { WorkflowTransactionError } from "../storage/errors.ts"; +import type { WorkspaceFilesystem } from "./filesystem.ts"; +import type { WorkspaceMetadata } from "./metadata.ts"; + +/** + * What a Workspace mutation is given. + * + * The authoritative filesystem first, because most mutations are only about + * bytes. Retained Repository and Worktree identity follows it, in the same + * transaction, so a mutation that needs both commits both or neither. + */ +export type WorkspaceMutation = ( + filesystem: WorkspaceFilesystem, + metadata: WorkspaceMetadata, +) => Operation; + +/** What a host contributes: one effect for one mutation of one run. */ +export interface WorkspaceEffectBinding { + /** + * The effect that performs this mutation. + * + * Synchronous, because an effect is a value a document yields rather than + * work of its own: building one is not a place a scope may suspend, and a + * `Workflow` body yields effects and nothing else. + */ + create( + description: EffectDescription, + mutate: WorkspaceMutation, + ): DurableEffect; +} + +const bindings = (() => { + const held = new WeakMap(); + return { + attach(database: WorkflowRunDatabase, binding: WorkspaceEffectBinding): void { + held.set(database, binding); + }, + detach(database: WorkflowRunDatabase): void { + held.delete(database); + }, + of(database: WorkflowRunDatabase): WorkspaceEffectBinding | undefined { + return held.get(database); + }, + }; +})(); + +/** + * Bind this run's Workspace effects for as long as the attachment lasts. + * + * Registered for the exact handle the host attached and removed when that scope + * ends, however it ends — so an effect created after the attachment is over + * finds nothing rather than a stale filesystem or a closed transaction. + */ +export function* useWorkspaceEffects( + database: WorkflowRunDatabase, + binding: WorkspaceEffectBinding, +): Operation { + bindings.attach(database, binding); + yield* ensure(() => { + bindings.detach(database); + }); +} + +/** + * The binding this run is attached through. + * + * Refuses rather than answering with nothing: a document that reached a + * Workspace operation with no attachment is asking for a mutation no host has + * agreed to perform, and performing it against whatever filesystem happens to + * be in scope is the one outcome that must not be possible. + */ +export function workspaceEffectsFor(database: WorkflowRunDatabase): WorkspaceEffectBinding { + const binding = bindings.of(database); + if (binding === undefined) { + throw new WorkflowTransactionError( + "this workflow run has no Workspace attachment, so a Workspace effect cannot be " + + "created. A host attaches one for a live or partial document execution.", + ); + } + return binding; +} diff --git a/packages/workflow/src/workspace/savepoint.ts b/packages/workflow/src/workspace/savepoint.ts new file mode 100644 index 000000000..ed909bfe6 --- /dev/null +++ b/packages/workflow/src/workspace/savepoint.ts @@ -0,0 +1,56 @@ +/** + * A savepoint inside one Workspace mutation, whichever host is performing it. + * + * One mutation may change several things and then find that it cannot finish + * one of them. What the shared Files and composition rules do about that is + * ask for the failed part to be undone and carry on — a write that created two + * parent directories and was then refused leaves neither behind, and the + * refusal is journaled against the Workspace as it was. + * + * How that is undone is the host's. The local host takes a real SQLite + * savepoint inside the transaction it is already in; the runner works in a + * disposable attempt and restores it from the accepted root. Both answer the + * same question, so the rules above them do not know which one they are + * running on — and a scope with no host answering at all refuses rather than + * performing work nothing can take back. + */ + +import { type Api, createApi } from "@effectionx/context-api"; +import type { Operation } from "effection"; +import { WorkflowTransactionError } from "../storage/errors.ts"; + +export interface TransactionApi { + /** + * Run `body` inside a savepoint, discarding its work if it fails. + * + * Answers with what the body answered. A failure rolls the savepoint back + * and propagates, leaving the surrounding transaction open and free to + * continue or to fail on its own terms. + */ + savepoint(body: Operation): Operation; +} + +/** No transaction is open in this scope, so there is nothing to nest inside. */ +export class NoOpenTransactionError extends WorkflowTransactionError { + override name = "NoOpenTransactionError"; + + constructor() { + super( + "a savepoint needs a transaction to be inside, and this scope is not inside one. " + + "Take savepoints within the body a transaction hands you.", + ); + } +} + +export const Transaction: Api = createApi( + "executablemd.workflow.deno.savepoint", + { + // deno-lint-ignore require-yield + *savepoint(_body: Operation): Operation { + throw new NoOpenTransactionError(); + }, + }, +); + +/** The savepoint operation, for whoever is inside a transaction. */ +export const savepoint: TransactionApi["savepoint"] = Transaction.operations.savepoint; diff --git a/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts index d0a3660a7..e38e27a67 100644 --- a/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts @@ -15,7 +15,7 @@ import { env, runInDurableObject } from "cloudflare:test"; import { beforeAll, describe, expect, it } from "vitest"; -import { run, until, type Operation } from "effection"; +import { run, scoped, until, type Operation } from "effection"; import { remoteOwnerClient } from "../../src/cloudflare/configured.ts"; import type { OwnerHttpRequest, @@ -277,6 +277,76 @@ describe("the owner's request boundary", () => { expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); }); + it("keeps the upgraded socket authoritative between separate object accesses", async () => { + const runId = runOf(); + const built = await client(runId); + const outcome = await run(function* (): Operation> { + return yield* scoped(function* () { + const admitted = yield* built.client.admit(runId); + if (!admitted.ok || admitted.value === "already-running") { + return { admitted: false }; + } + // The acquisition is registered on the socket's attachment, which is + // what an evicted object reads back — nothing about it is in memory. + const before = yield* until(on(stubFor(runId), (owner) => owner.holders())); + // Read again through a separate access to the object. Nothing about + // the acquisition is in this instance's memory — it is the socket's + // serialized attachment, which is the same mechanism a hibernated + // object restores from — and the socket the runner still holds is the + // one that has to remain authoritative either way. + const evicted = yield* until(on(stubFor(runId), (owner) => owner.holders())); + const opened = yield* admitted.value.link.open(runId, null); + return { + admitted: true, + before, + evicted, + opened: opened.ok, + after: yield* until(on(stubFor(runId), (owner) => owner.holders())), + }; + }); + }); + expect(outcome).toEqual({ + admitted: true, + before: 1, + evicted: 1, + // The run is not stored, so opening says so — over the same acquisition, + // which the owner still recognizes after the eviction. + opened: false, + after: 1, + }); + // And the acquisition ends with the scope that held it, not with the object. + expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); + }); + + it("lets a closed socket authorize nothing, and admits the next executor", async () => { + const runId = runOf(); + const built = await client(runId); + const closed = await run(function* (): Operation> { + const first = yield* scoped(function* () { + const admitted = yield* built.client.admit(runId); + if (!admitted.ok || admitted.value === "already-running") { + return { admitted: false }; + } + // Ending the connection is how a runner stops being the executor. + yield* admitted.value.close(); + return { + admitted: true, + // Nothing may be asked of the run over a connection that is gone. + refused: !(yield* admitted.value.link.open(runId, null)).ok, + holders: yield* until(on(stubFor(runId), (owner) => owner.holders())), + }; + }); + // And the next acquisition is admitted, because the first one released + // ownership by closing rather than by any lease expiring. + const second = yield* scoped(function* () { + const admitted = yield* built.client.admit(runId); + return admitted.ok && admitted.value !== "already-running"; + }); + return { ...first, second }; + }); + expect(closed).toEqual({ admitted: true, refused: true, holders: 0, second: true }); + }); + it("answers a read while an executor is live, and takes no acquisition", async () => { const runId = runOf(); const built = await client(runId); diff --git a/packages/workflow/tests/cloudflare/support/worker-files.ts b/packages/workflow/tests/cloudflare/support/worker-files.ts index c694c34fa..d44b2a1ba 100644 --- a/packages/workflow/tests/cloudflare/support/worker-files.ts +++ b/packages/workflow/tests/cloudflare/support/worker-files.ts @@ -118,6 +118,17 @@ export function createWorkerFiles(): { nodes.set(path, { kind: "directory", mode, mtime: (clock += 1) }); }, + // deno-lint-ignore require-yield + *removeTree(path): Operation { + // Every node at or under this path, so what a savepoint restores into is + // an empty directory rather than one holding part of a failed attempt. + for (const held of Array.from(nodes.keys())) { + if (held === path || held.startsWith(`${path}/`)) { + nodes.delete(held); + } + } + }, + // deno-lint-ignore require-yield *writeFile(path, bytes, mode): Operation { requireParent(path); diff --git a/packages/workflow/tests/remote-publication.test.ts b/packages/workflow/tests/remote-publication.test.ts index 8deed2789..0d2839c87 100644 --- a/packages/workflow/tests/remote-publication.test.ts +++ b/packages/workflow/tests/remote-publication.test.ts @@ -469,9 +469,12 @@ describe("what the production runner publishes", () => { yield* until(writeFile(attempt.at("/NOTES.md"), "never published\n", { mode: 0o644 })); // Everything a caller can reach by name. Reading where the Workspace is, - // reading what an attempt holds — and nothing that moves either. + // reading what an attempt holds, and putting an attempt back to the + // accepted root — and nothing that moves either. `restore` throws this + // attempt's own work away; it publishes nothing, and the accepted + // materialization it restores from is what the owner already confirmed. expect(Object.keys(materialization).toSorted()).toEqual(["at", "workspaceRootId"]); - expect(Object.keys(attempt).toSorted()).toEqual(["at", "capture"]); + expect(Object.keys(attempt).toSorted()).toEqual(["at", "capture", "restore"]); const reachable = [ ...Object.getOwnPropertyNames(attempt), ...Object.getOwnPropertyNames(materialization), diff --git a/packages/workflow/tests/remote-runner.test.ts b/packages/workflow/tests/remote-runner.test.ts index 4b95c120b..836224433 100644 --- a/packages/workflow/tests/remote-runner.test.ts +++ b/packages/workflow/tests/remote-runner.test.ts @@ -17,7 +17,18 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { Ok, type Operation, scoped } from "effection"; +import { Ok, type Operation, scoped, until } from "effection"; +import { mkdir, writeFile } from "node:fs/promises"; +import { collect, execute, inlineSource } from "@executablemd/core"; +import { API, useHostFiles } from "@executablemd/runtime"; +import type { HostFilesEvent } from "@executablemd/runtime"; +import type { Json } from "@executablemd/durable-streams"; +import { encodeBase64 } from "../src/cloudflare/encoding.ts"; +import { cloudflareReadLink, cloudflareRunLink } from "../src/cloudflare/client.ts"; +import { cloudflareLifecycleLink } from "../src/cloudflare/lifecycle-link.ts"; +import { type OwnerSocket, type SocketListener, useOwnerConnection } from "../src/remote/client.ts"; +import { captureWorkspace, type CapturedWorkspace } from "../src/remote/materialize.ts"; +import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; import type { ExecutorLock } from "../src/lifecycle/api.ts"; import type { WorkflowExecutionTransitions } from "../src/lifecycle/execution.ts"; @@ -151,6 +162,296 @@ function foreignHandle(): WorkflowRunDatabase { }; } +/** + * One owner, scripted at the wire. + * + * Everything above it is production code: the real client, the real lifecycle, + * the real database handle, the real coordinator and the real runner + * facilities. What this stands in for is the object that would answer — so what + * a test can say afterwards is what the runner actually sent it, and what it + * committed. + */ +function scriptedOwner(captured: CapturedWorkspace) { + const sent: Record[] = []; + const commits: Record[] = []; + let currentRoot = captured.root.rootId; + let refusal: string | undefined; + let lost = false; + + function frontier(): Record { + return { + record: { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-10T00:00:00.000Z", + updatedAt: "2026-09-10T00:00:00.000Z", + }, + retrieval: null, + workspaceRootId: currentRoot, + journalEventId: null, + }; + } + + function begun(executionId: string): Record { + return { + frontier: frontier(), + // Exactly the members an execution that has not stopped declares. + execution: { executionId, startedAt: "2026-09-10T00:00:01.000Z" }, + replay: false, + recovered: null, + }; + } + + function answer(request: Record): Record { + const command = request["command"]; + if (command === "open" || command === "frontier") { + return { outcome: "performed", value: frontier() }; + } + if (command === "begin") { + // A lifecycle answer is one of three fields and never two: the value, a + // refusal, or the immutable fields a creation conflicts on. + return { + outcome: "performed", + value: { + conflict: null, + refusal: null, + value: begun(String(request["executionId"])), + }, + }; + } + if (command === "mappings") { + return { + outcome: "performed", + value: { + workspaceRootId: currentRoot, + journalEventId: null, + repositories: [], + worktrees: [], + agentSessions: [], + }, + }; + } + if (command === "root") { + return { + outcome: "performed", + value: { workspaceRootId: currentRoot, manifest: captured.root.manifest }, + }; + } + if (command === "content") { + const digest = String(request["digest"]); + const bytes = + request["kind"] === "manifest" + ? captured.contents.get(digest)?.manifestBytes + : captured.blobs.get(digest); + if (bytes === undefined) { + throw new Error("asked for content this owner does not hold"); + } + return { + outcome: "performed", + value: { + kind: request["kind"], + digest, + size: bytes.length, + bytes: encodeBase64(bytes), + }, + }; + } + if (command === "stage") { + const encoded = String(request["bytes"] ?? ""); + const padding = encoded.endsWith("==") ? 2 : encoded.endsWith("=") ? 1 : 0; + return { + outcome: "performed", + value: { + kind: request["kind"], + digest: request["digest"], + size: (encoded.length / 4) * 3 - padding, + }, + }; + } + if (command === "settle") { + return { outcome: "performed", value: { status: "completed" } }; + } + commits.push(request); + const publication = request["publication"]; + // Scripted for the Workspace proposal rather than for every append: an + // owner that refused the run's own journal rows would end the document + // before it ever reached the effect under test. + const proposing = publication !== null && publication !== undefined; + if (lost && proposing) { + return { outcome: "lost" }; + } + if (refusal !== undefined && proposing) { + return { outcome: "refused", refusal }; + } + const events = Array.isArray(request["events"]) ? request["events"] : []; + // The owner publishes what it validated, and the frontier moves with it. + if (publication !== null && publication !== undefined) { + currentRoot = String(Reflect.get(publication, "proposedWorkspaceRootId")); + } + return { + outcome: "performed", + value: { + workspaceRootId: currentRoot, + journalEventIds: events.map((_entry, index) => `event-${index}`), + }, + }; + } + + const listeners = new Map>(); + const socket: OwnerSocket = { + send(data: string): void { + const request: Record = JSON.parse(data); + sent.push(request); + const response = answer(request); + if (response["outcome"] === "lost") { + for (const listener of listeners.get("close") ?? []) { + listener({}); + } + return; + } + for (const listener of listeners.get("message") ?? []) { + listener({ data: JSON.stringify({ id: request["id"], ...response }) }); + } + }, + close(): void {}, + addEventListener(type, listener): void { + const found = listeners.get(type) ?? new Set(); + found.add(listener); + listeners.set(type, found); + }, + removeEventListener(type, listener): void { + listeners.get(type)?.delete(listener); + }, + }; + + return { + socket, + sent, + commits, + get currentRoot(): string { + return currentRoot; + }, + refuse(reason: string): void { + refusal = reason; + }, + lose(): void { + lost = true; + }, + }; +} + +/** A small starting tree, captured so the scripted owner can serve it. */ +function* startingTree(): Operation { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); + const root = yield* trees.create("source"); + yield* until(writeFile(`${root}/README.md`, "starting\n", { mode: 0o644 })); + yield* until(mkdir(`${root}/docs`, { mode: 0o755 })); + return yield* captureWorkspace( + files, + (logical) => (logical === "/" ? root : `${root}${logical}`), + (reason) => { + throw new Error(reason); + }, + ); +} + +/** One runner over a scripted owner reached through the production client. */ +function* wired(captured: CapturedWorkspace): Operation<{ + owner: ReturnType; + runner: RemoteWorkflowRunner; +}> { + const owner = scriptedOwner(captured); + const connection = yield* useOwnerConnection(owner.socket); + let identifier = 0; + const next = () => `command-${(identifier += 1)}`; + const reads = cloudflareReadLink(connection, next, RUN_ID); + const runner = yield* useRemoteWorkflowRunner({ + owner: { + runId: RUN_ID, + // deno-lint-ignore require-yield + *admit() { + return Ok({ + link: cloudflareRunLink(connection, next, RUN_ID), + lifecycle: cloudflareLifecycleLink(connection, reads, next), + // deno-lint-ignore require-yield + *close(): Operation {}, + }); + }, + // deno-lint-ignore require-yield + *reads(runId: string) { + return Ok(readPlane(runId)); + }, + delivery: { + // deno-lint-ignore require-yield + *wait(): Operation { + throw new Error("PLANTED-DELIVERY-WAIT-REACHED"); + }, + // deno-lint-ignore require-yield + *retain(): Operation { + throw new Error("PLANTED-DELIVERY-RETAIN-REACHED"); + }, + }, + }, + scratchRoot: "/tmp/xmd-remote-runner-live", + }); + return { owner, runner }; +} + +/** + * The ambient host filesystem a runtime entrypoint installs, watched. + * + * The real provider rather than a stand-in, at the position a host installs it + * and with a working directory a workflow run must never resolve against. What + * a test says afterwards is whether a document reached it at all. + */ +function* useHostSpy(): Operation { + const seen: HostFilesEvent[] = []; + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return "/nowhere-the-workflow-may-reach"; + }, + }, + { at: "min" }, + ); + yield* useHostFiles({ observe: (event) => seen.push(event) }); + return seen; +} + +/** + * The commits that proposed a Workspace, out of everything the owner was asked + * to commit. + * + * A run's journal lives on its owner, so every ordinary append — the root + * import, a component import, the terminal — reaches it as a commit of its own. + * What a Workspace effect adds to one is the publication, and that is what + * these tests are counting. + */ +function published(commits: readonly Record[]): Record[] { + return commits.filter((intent) => { + const publication = intent["publication"]; + return publication !== null && publication !== undefined; + }); +} + +/** One authored document, executed as this run's root inside the attachment. */ +function document(source: string, database: WorkflowRunDatabase): Operation { + return scoped(function* () { + return yield* collect(yield* execute({ ...inlineSource(source), stream: database.journal })); + }); +} + describe("a runner for a run whose storage is somewhere else", () => { it("attaches the handle its own lifecycle opened, and no other", function* () { const outcome = yield* scoped(function* () { @@ -190,6 +491,162 @@ describe("a runner for a run whose storage is somewhere else", () => { expect(outcome.acquisitions).toEqual([RUN_ID, RUN_ID]); }); + it("makes an authored File write a remote Workspace effect", function* () { + const outcome = yield* scoped(function* () { + const captured = yield* startingTree(); + const before = captured.root.rootId; + const { owner, runner } = yield* wired(captured); + const transitions = yield* runner.useRunHost(); + const database = yield* opened(transitions, yield* acquired()); + + // The ambient host filesystem, installed the way a runtime entrypoint + // installs it and outside the attachment. A workflow document must never + // reach it. + const host = yield* useHostSpy(); + + const output = yield* runner.attach( + database, + document( + ["# Remote", "", 'written by the document'].join("\n"), + database, + ), + ); + return { output: String(output), host, owner, before }; + }); + + // The document ran to completion — `` renders nothing, so what it + // wrote is visible in what the owner was asked to commit, below — and the + // ambient host filesystem was never asked for anything at all. + expect(outcome.output.trimEnd()).toBe("# Remote"); + expect(outcome.host).toEqual([]); + // It materialized the exact retained root, then proposed one commit: the + // new root and the journal row describing the effect, together. + const asked = outcome.owner.sent.map((request) => request["command"]); + expect(asked).toContain("mappings"); + expect(asked).toContain("root"); + const proposals = published(outcome.owner.commits); + expect(proposals).toHaveLength(1); + const intent = proposals[0] ?? {}; + expect(intent["expectedWorkspaceRootId"]).toBe(outcome.before); + // One transaction carried both halves: the file the document wrote, and + // the journal row describing the effect that wrote it. + expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); + expect(String(intent["events"])).toContain("workspace_file"); + expect(JSON.stringify(intent["publication"])).toContain("/NOTES.md"); + // And the owner's frontier moved to what it published, which is what a + // later read of this run observes. + expect(outcome.owner.currentRoot).not.toBe(outcome.before); + expect(intent["publication"]).toEqual( + expect.objectContaining({ proposedWorkspaceRootId: outcome.owner.currentRoot }), + ); + }); + + it("installs the whole live set, and resolves a document's paths inside the run", function* () { + const outcome = yield* scoped(function* () { + const captured = yield* startingTree(); + const { owner, runner } = yield* wired(captured); + const transitions = yield* runner.useRunHost(); + const database = yield* opened(transitions, yield* acquired()); + const host = yield* useHostSpy(); + // `` is the lexical half of the composition and `` is the + // document filesystem: a write inside a directory the document named + // proves both, and proves the path resolved inside the run's own + // Workspace rather than against the host working directory above. + const output = yield* runner.attach( + database, + document( + [ + "# Remote", + "", + '', + "", + ' nested', + "", + "", + ].join("\n"), + database, + ), + ); + return { output: String(output), host, owner }; + }); + + expect(outcome.host).toEqual([]); + const proposals = published(outcome.owner.commits); + expect(proposals).toHaveLength(1); + // The file landed under the directory the document named, inside the run. + expect(JSON.stringify(proposals[0]?.["publication"])).toContain("/docs/inner.md"); + }); + + it("keeps a refused, failed or cancelled attachment to one owner transaction", function* () { + const outcomes = yield* scoped(function* () { + /** One document write, under an owner scripted to answer this way. */ + function* attempt( + script: (owner: ReturnType) => void, + body?: (result: string) => Operation, + ): Operation<{ said: string; commits: number; root: string; before: string }> { + return yield* scoped(function* () { + const captured = yield* startingTree(); + const { owner, runner } = yield* wired(captured); + const transitions = yield* runner.useRunHost(); + const database = yield* opened(transitions, yield* acquired()); + script(owner); + let said: string; + try { + const rendered = yield* runner.attach( + database, + document( + ["# Remote", "", 'written by the document'].join("\n"), + database, + ), + ); + said = body === undefined ? String(rendered) : yield* body(String(rendered)); + } catch (error) { + said = error instanceof Error ? `raised:${error.name}` : "raised:other"; + } + return { + said, + commits: published(owner.commits).length, + attempts: owner.commits.filter((intent) => intent["publication"] !== null).length, + root: owner.currentRoot, + before: captured.root.rootId, + }; + }); + } + + return { + // The owner refuses the commit: nothing is promoted, and the run is + // still on the root it started from. + refused: yield* attempt((owner) => owner.refuse("command:stale-root")), + // The answer never arrives: whether the owner committed is exactly what + // cannot be known, and nothing here claims it did. + lost: yield* attempt((owner) => owner.lose()), + // The document fails after its own effect committed. The effect's + // transaction is the one visible outcome; nothing else is sent. + failed: yield* attempt( + () => undefined, + // deno-lint-ignore require-yield + function* (): Operation { + throw new Error("PlantedDocumentFailure"); + }, + ), + }; + }); + + // A refused commit leaves the frontier where it was, and the run learns it + // rather than being told the write succeeded. + expect(outcomes.refused.root).toBe(outcomes.refused.before); + expect(outcomes.refused.commits).toBe(1); + expect(outcomes.refused.said).toContain("raised:"); + // A lost answer is the same: one attempt, and no claim either way. + expect(outcomes.lost.root).toBe(outcomes.lost.before); + expect(outcomes.lost.commits).toBe(1); + expect(outcomes.lost.said).toContain("raised:"); + // A document that failed afterwards published its effect and nothing else. + expect(outcomes.failed.commits).toBe(1); + expect(outcomes.failed.root).not.toBe(outcomes.failed.before); + expect(outcomes.failed.said).toBe("raised:Error"); + }); + it("reads and delivers without taking an acquisition", function* () { const outcome = yield* scoped(function* () { const scripted = ownerOf(); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 2bae3548b..6d2dbdd31 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -10051,7 +10051,7 @@ malformed-input permutation without a distinct structural consequence. Most of what they name is **specified; implementation unbuilt**, so those tiers name the scenarios an implementation is accepted against rather than tests that exist. Tier WRH is the exception: rows WRH1-WRH16 are implemented and have committed evidence, listed with the tier, while WRH17-WRH22 remain unbuilt with the machine-wait work they belong to. Nothing in the other six tiers is built. -WRH14 covers the configured host as well as the boundary it satisfies: `packages/cli/tests/remote-workflow-host.test.ts` proves that the explicit installer has exactly the four methods, that it is bound to one run and refuses another before a token is minted, that reads and deliveries take no acquisition while execution takes one socket, and that a storage handle it did not open cannot be attached; `packages/workflow/tests/remote-runner.test.ts` proves the lifecycle-to-attachment handoff, including that two clients holding handles which agree about run id, root and anchor still cannot attach each other's; and `packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts` proves the request boundary against a real Durable Object — namespace routing, an actual upgrade whose subprotocol is selected, the admission order as a status, and a read and a delivery answered while an executor is live. No selector chooses that host: the shipped Deno and compiled entrypoints install the local one and Node and Bun remain unsupported. +WRH14 covers the configured host as well as the boundary it satisfies, including what its fourth method installs: `packages/workflow/tests/remote-runner.test.ts` runs an authored `` — and an authored `` inside a `` — through the public attachment with the ambient host filesystem installed outside it, and proves the host provider is never asked, the exact retained root is materialized, one commit carries the new root and the effect's own journal row together, and the owner's frontier moves to what it published; the same file covers a refused commit, a lost answer and a document that fails afterwards, each leaving one attempt and the prior or new complete transaction as the only visible outcome. `packages/cli/tests/remote-workflow-host.test.ts` proves that the explicit installer has exactly the four methods, that it is bound to one run and refuses another before a token is minted, that reads and deliveries take no acquisition while execution takes one socket, and that a storage handle it did not open cannot be attached; `packages/workflow/tests/remote-runner.test.ts` proves the lifecycle-to-attachment handoff, including that two clients holding handles which agree about run id, root and anchor still cannot attach each other's; and `packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts` proves the request boundary against a real Durable Object — namespace routing, an actual upgrade whose subprotocol is selected, the admission order as a status, and a read and a delivery answered while an executor is live. No selector chooses that host: the shipped Deno and compiled entrypoints install the local one and Node and Bun remain unsupported. ### Tier WRH — Remote host, executor and delivery separation diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index f75cae5b2..8400e3e6e 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -3802,7 +3802,11 @@ The three planes are three requests, and the path says which. One public run id A runner reaches all three through one configured client, which is the supported public surface: an already-selected run id, one credential-free endpoint parsed at construction, the exact release identity, an operation that mints a short-lived token for the immediate request, and the HTTP and WebSocket I/O the host performs. The client is bound to that one run — every plane requires the configured id before a token exists, a URL is built or anything is sent — and the endpoint, the release and the token stay in its closure, reaching no workflow record, journal event, public error or document-visible value. What crosses on the planes stays private: the paths, the header names, the commands and the refusal spellings are one release talking to itself, and the one refusal category that is a fact about the run rather than about the connection — another live executor holds it — is the only one that becomes a public answer. -Trusted code assembles that client into the same four-method host the local entrypoint installs. `useRunHost()` installs the executor lifecycle over the client's acquisition and returns the provider-neutral transitions; `useLifecycle()` installs status, list and history over the read plane, where the complete domain of one bound owner is zero or one coherent snapshot and no namespace-wide claim; `useDelivery()` installs typed answer delivery over the delivery plane; and `attach()` installs the runner's Workspace coordinator around a live or partial execution. A completed replay reaches none of it: it reads the owner and opens and closes its own execution envelope, and `attach()` is never called. +Trusted code assembles that client into the same four-method host the local entrypoint installs. `useRunHost()` installs the executor lifecycle over the client's acquisition and returns the provider-neutral transitions; `useLifecycle()` installs status, list and history over the read plane, where the complete domain of one bound owner is zero or one coherent snapshot and no namespace-wide claim; `useDelivery()` installs typed answer delivery over the delivery plane; and `attach()` installs everything a live or partial document execution reaches. A completed replay reaches none of it: it reads the owner and opens and closes its own execution envelope, and `attach()` is never called. + +**A remote attachment installs the same capabilities a local one does.** The logical Workspace working directory, the document `API.Files` provider, the Repository, Worktree and `` composition, the transactional Git components, the composition components, the pull-request and Issue middleware, elicitation and — where the host supplied one — the Agent profile. They are one set for the reason §10.1 gives: the Files provider alone would resolve an authored path against whatever working directory the surrounding host adapter answers with, so a workflow document would write to the runner's own filesystem instead of the run's Workspace. The document rules are written once and reach both hosts, and each host contributes two things to them: the effect its mutations become, and the savepoint that undoes a part of one mutation which could not be finished. Locally that effect is bound to the validated lease and the savepoint is a real transaction savepoint; on a runner the effect is bound to the exact remote run and the savepoint restores the disposable attempt from the accepted root. Which binding answers is decided by the exact storage handle the attachment registered — not by a context value, a run id or anything else a caller could hold — so a handle another client opened finds that client's binding or none, and a document with no attachment is refused rather than performing its write somewhere. + +The host-owned inputs those capabilities need — the credential helper, the Issue and pull-request configuration, the Agent profile installer — are explicit configuration to whoever assembles the runner. Nothing about them is read from a flag, an environment variable, a document prop or a global, and an absent one installs that capability's unconfigured behavior rather than a different capability. **An attachment is bound to the handle that produced it.** The handle a begin transition hands back carries its own routed journal and the provenance taken over it, and it remembers the exact link it was opened from. Attaching compares that link to the ones this runner's own acquisitions produced, by object identity. Two clients on two owners can hold handles whose run id, root and journal anchor are identical, so nothing a handle says about itself could establish which run it is; what establishes it is where it came from, and a handle another client opened — or one shaped like a handle and opened by nothing — is refused before a temporary tree, a materialization or a request exists. From c1ad42231e6c96b3e9c3cd92975782a996123f50 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 14:33:39 -0300 Subject: [PATCH 73/81] =?UTF-8?q?=F0=9F=90=9B=20Let=20a=20runner=20attach?= =?UTF-8?q?=20a=20checkout=20and=20retain=20a=20session=20without=20the=20?= =?UTF-8?q?local=20store?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two capabilities were assembled remotely in name only. A Repository or Worktree committed its identity and its content through the run's own binding and then reattached its checkout through `transactWorkspaceRoots`, whose implementation is the local storage provider's private workspace — which a runner never installs. An Agent profile read and committed its session mapping through the same door. So a remote run could publish a checkout it could not then attach, and establish a conversation whose mapping nothing retained. A host now answers four questions for one run rather than two: the effect a mutation becomes, the savepoint that undoes an unfinishable part of one, the read an ephemeral attachment needs, and the transaction a session mapping is retained by. Locally all four come from the lease this handle already validated. Remotely the read takes one coherent owner snapshot, materializes the selected root into a directory the invocation owns, hands the attachment only the retained metadata and that filesystem, and closes before native Git runs — no owner read and no transaction held across a subprocess. The mapping transaction stages from the admitted state on the runner, where the provider is, and submits bounded deltas to one owner transaction that revalidates each subject and commits them all or refuses them together. No provider call happens inside it, a body that failed or was cancelled sends nothing, and a conflicting assertion never replaces what is retained. A handle binds its host when it is opened, not only when a document is attached: reading a retained mapping needs no Workspace, and a caller that opened storage alone has always been able to. An attachment binds a narrower one over the top and gives the outer one back when it ends. The public configuration is the published one. `capabilities` on the runner and on the configured host accept what a *host* owns — the issue tracker, the readable pull requests, the credential helper, the Agent profile — and no substituted repository host, Git-host transport or invocation observer, each of which is a seam a credential this run acquires would become visible through. One projection does that for both, member by member, and a type-level regression fails if the seam comes back. --- architecture.md | 2 +- packages/cli/src/remote-workflow.ts | 48 ++ .../cli/tests/remote-workflow-host.test.ts | 96 ++++ packages/workflow/deno.ts | 2 +- .../workflow/src/deno/composition/effects.ts | 4 +- .../workflow/src/deno/composition/provider.ts | 13 +- .../src/deno/composition/repository.ts | 4 +- .../workflow/src/deno/composition/worktree.ts | 4 +- packages/workflow/src/deno/database.ts | 7 + packages/workflow/src/deno/remote-runner.ts | 39 +- .../workflow/src/deno/workspace/effect.ts | 56 +- packages/workflow/src/deno/workspace/files.ts | 6 +- .../workflow/src/deno/workspace/published.ts | 39 +- packages/workflow/src/remote/collector.ts | 32 +- packages/workflow/src/remote/database.ts | 20 +- packages/workflow/src/remote/workspace.ts | 108 +++- packages/workflow/src/workspace/effects.ts | 87 +++- .../cloudflare/remote-owner-routes.vitest.ts | 15 +- packages/workflow/tests/remote-runner.test.ts | 491 ++++++++---------- .../tests/support/remote-owner-script.ts | 278 ++++++++++ specs/workflow-workspace-spec.md | 2 +- 21 files changed, 1021 insertions(+), 332 deletions(-) create mode 100644 packages/workflow/tests/support/remote-owner-script.ts diff --git a/architecture.md b/architecture.md index 2c6ed23a1..beadceeaa 100644 --- a/architecture.md +++ b/architecture.md @@ -443,7 +443,7 @@ What that owner implements, it implements behind the four-method `WorkflowHost` A runner reaches it over three requests, and the path says which plane. One run id selects its owner arithmetically, a gateway forwards on that id alone, and the executor plane is a real upgrade whose admission order — release, then token, then the run — is the owner's own. One configured client carries all three: an already-selected run id, one credential-free endpoint parsed at construction, the release identity, a token minted per request, and the I/O its host performs. It is bound to that one run, refuses another before a token exists, and keeps endpoint, release and token in closure state that reaches no record, event, error or document-visible value. Trusted code assembles that client into the same four methods, and an attachment is bound to the exact storage handle the begin transition produced — compared by identity to the link this runner's own acquisition opened, because two clients on two owners can hold handles that agree about everything except where they came from. -A remote attachment installs what a local one installs, because the document rules are written once. Filesystem, Repository, Worktree, ``, Git, composition components, pull-request and Issue middleware, elicitation and any configured Agent profile all come from the same modules; what each host contributes is the effect a mutation becomes and the savepoint that undoes an unfinishable part of one. The exact storage handle decides which of those two bindings answers, so an authored `` on a remote run becomes a durable Workspace effect against the invocation's materialized tree and never falls through to the runner's own filesystem — and a handle no attachment registered is refused rather than performed. +A remote attachment installs what a local one installs, because the document rules are written once. Filesystem, Repository, Worktree, ``, Git, composition components, pull-request and Issue middleware, elicitation and any configured Agent profile all come from the same modules; what each host contributes is four things — the effect a mutation becomes, the savepoint that undoes an unfinishable part of one, the read an ephemeral checkout attachment needs, and the transaction an Agent-session mapping is retained by. The exact storage handle decides which host answers all four, so an authored `` on a remote run becomes a durable Workspace effect against the invocation's materialized tree and never falls through to the runner's own filesystem, a reattachment reads one owner snapshot into a tree the invocation owns and closes it before Git runs, an Agent-session mapping is retained by the run's own owner in one transaction the provider is never called from inside, and a handle no attachment registered is refused rather than performed. What no code chooses is which host a run belongs to. There is no runtime or CLI selector, no ambient endpoint, release-identity or OIDC source, and no deployment: the shipped entrypoints install the local host, and the explicit installer is the seam a later trusted assembly configures. diff --git a/packages/cli/src/remote-workflow.ts b/packages/cli/src/remote-workflow.ts index e9e480d14..a223ac03c 100644 --- a/packages/cli/src/remote-workflow.ts +++ b/packages/cli/src/remote-workflow.ts @@ -21,6 +21,7 @@ import { ensure, resource, until, type Operation } from "effection"; import { remoteOwnerClient, useRemoteWorkflowRunner } from "@executablemd/workflow/deno"; +import type { WorkflowWorkspaceOptions } from "@executablemd/workflow/deno"; import type { OwnerHttpRequest, OwnerHttpResponse, @@ -58,6 +59,21 @@ export interface RemoteWorkflowConfiguration { * transport it controls. */ readonly transport?: OwnerTransport; + /** + * What a live or partial attachment installs beyond the run's own Workspace. + * + * The host-owned inputs and only those: which issue tracker this program + * authorizes, which pull requests a document may read, how this host + * assembles its credential helper, and which Agent profile it installs. + * There is no member for a substituted repository host, a Git-host transport + * or an invocation observer, because each of those is a seam through which a + * credential this run acquires would become visible to whoever supplied it. + * + * Explicit, like everything else here. Nothing is read from a flag, an + * environment variable, a document prop or a global, and an absent member + * keeps the capability's unconfigured behavior. + */ + readonly capabilities?: WorkflowWorkspaceOptions; } /** @@ -81,6 +97,11 @@ export function* useRemoteWorkflowHost( const runner = yield* useRemoteWorkflowRunner({ owner: client, scratchRoot: configuration.scratchRoot, + // Projected member by member, as the published boundary is everywhere + // else: a spread would carry whatever else a caller put on the object. + ...(configuration.capabilities === undefined + ? {} + : { capabilities: permitted(configuration.capabilities) }), }); return { useRunHost(): Operation { @@ -140,3 +161,30 @@ function platformTransport(): OwnerTransport { }, }; } + +/** + * The capability inputs this host passes on, and the whole of them. + * + * Named one at a time rather than forwarded: what a trusted caller may + * configure is a closed list, and reading a property nobody declared is how a + * getter somebody else wrote gets to run. + */ +function permitted(options: WorkflowWorkspaceOptions): WorkflowWorkspaceOptions { + return { + ...(options.gitHubIssues === undefined ? {} : { gitHubIssues: options.gitHubIssues }), + ...(options.gitHubPullRequests === undefined + ? {} + : { + gitHubPullRequests: { + ...(options.gitHubPullRequests.allowed === undefined + ? {} + : { allowed: options.gitHubPullRequests.allowed }), + ...(options.gitHubPullRequests.endpoint === undefined + ? {} + : { endpoint: options.gitHubPullRequests.endpoint }), + }, + }), + ...(options.helper === undefined ? {} : { helper: options.helper }), + ...(options.agent === undefined ? {} : { agent: options.agent }), + }; +} diff --git a/packages/cli/tests/remote-workflow-host.test.ts b/packages/cli/tests/remote-workflow-host.test.ts index 9191d5766..ef1197ef2 100644 --- a/packages/cli/tests/remote-workflow-host.test.ts +++ b/packages/cli/tests/remote-workflow-host.test.ts @@ -32,7 +32,15 @@ import type { } from "@executablemd/workflow/deno"; import { OwnerEndpointError } from "@executablemd/workflow/deno"; import { useRemoteWorkflowHost } from "../src/remote-workflow.ts"; +import type { RemoteWorkflowConfiguration } from "../src/remote-workflow.ts"; import type { WorkflowHost } from "../src/workflow.ts"; +import { + document, + published, + scriptedOwner, + startingTree, + useHostSpy, +} from "../../workflow/tests/support/remote-owner-script.ts"; const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; const OTHER_RUN = "4bxsfqu1yxtsmfg6aaccq1sxf1a4z456bf614gt4d6t31nqdqwzz"; @@ -160,6 +168,24 @@ function foreignDatabase(): WorkflowRunDatabase { }; } +/** + * What a caller may configure, at the type level. + * + * The published boundary excludes a substituted repository host, a Git-host + * transport and an invocation observer, because each is a seam through which a + * credential this run acquires would become visible to whoever supplied it. + * That exclusion is a property of the *type*, so this is where it is asserted: + * adding `composition` back to what the public configuration accepts stops this + * file compiling. + */ +type Capabilities = NonNullable; +type NoComposition = "composition" extends keyof Capabilities ? never : true; +type NoObserver = "observe" extends keyof Capabilities ? never : true; +type NoAccess = "access" extends keyof NonNullable + ? never + : true; +const NARROW: [NoComposition, NoObserver, NoAccess] = [true, true, true]; + describe("the configured remote workflow host", () => { it("has the four methods a host has, and no others", function* () { const owner = scripted(); @@ -168,6 +194,9 @@ describe("the configured remote workflow host", () => { return Object.keys(built).toSorted(); }); expect(assembled).toEqual(["attach", "useDelivery", "useLifecycle", "useRunHost"]); + // And what it may be configured with is the host-owned list, proved above + // where the property lives. + expect(NARROW).toEqual([true, true, true]); // Constructing a host reaches no owner: no token was minted, no request was // sent and nothing was upgraded. expect(owner.tokens).toEqual([]); @@ -316,6 +345,73 @@ describe("the configured remote workflow host", () => { expect(owner.sockets).toEqual([]); }); + it("runs an authored File through the configured public host", function* () { + const outcome = yield* scoped(function* () { + const captured = yield* startingTree(); + const owner = scriptedOwner(captured); + // The configured public host, over a transport whose socket is that + // scripted owner. Everything between the two is production code: the + // client, its three planes, the runner and the attachment. + const built = yield* useRemoteWorkflowHost({ + runId: RUN_ID, + endpoint: ENDPOINT, + release: RELEASE, + // deno-lint-ignore require-yield + *token(): Operation { + return "token-1"; + }, + scratchRoot: "/tmp/xmd-remote-public-host", + transport: { + // deno-lint-ignore require-yield + *request(): Operation { + throw new Error("PLANTED-REQUEST-PLANE-REACHED"); + }, + connect(): Operation { + return resource(function* (provide) { + yield* provide(owner.socket); + }); + }, + }, + }); + const transitions = yield* built.useRunHost(); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + if (!taken.ok || taken.value.kind !== "acquired") { + throw new Error("expected the configured host to take the acquisition"); + } + const begun = yield* transitions.begin(taken.value.lock, { + runId: RUN_ID, + action: "resume", + }); + if (!begun.ok) { + throw begun.error; + } + const database = begun.value.database; + const ambient = yield* useHostSpy(); + const rendered = yield* built.attach( + database, + document( + ["# Remote", "", 'through the public host'].join("\n"), + database, + ), + ); + return { + attached: String(rendered).trimEnd(), + owner, + before: captured.root.rootId, + ambient, + }; + }); + + expect(outcome.attached).toBe("# Remote"); + // The ambient host filesystem was never asked, and the owner received one + // proposal carrying the new root and the effect's own journal row. + expect(outcome.ambient).toEqual([]); + const proposals = published(outcome.owner.commits); + expect(proposals).toHaveLength(1); + expect(proposals[0]?.["expectedWorkspaceRootId"]).toBe(outcome.before); + expect(JSON.stringify(proposals[0]?.["publication"])).toContain("/NOTES.md"); + }); + it("attaches nothing it did not open", function* () { const owner = scripted(); const refused = yield* scoped(function* () { diff --git a/packages/workflow/deno.ts b/packages/workflow/deno.ts index 0d61a9551..6c044ba78 100644 --- a/packages/workflow/deno.ts +++ b/packages/workflow/deno.ts @@ -115,7 +115,7 @@ export type { AgentSessionResolution, ProviderAssertion, } from "./src/deno/workspace/agent-sessions.ts"; -export { transactAgentSessions } from "./src/deno/workspace/private.ts"; +export { transactAgentSessions } from "./src/workspace/effects.ts"; export type { AgentSessions } from "./src/deno/workspace/agent-sessions.ts"; export { WORKSPACE_GIT_ADD, diff --git a/packages/workflow/src/deno/composition/effects.ts b/packages/workflow/src/deno/composition/effects.ts index c28f1bfb6..3d04ed0b8 100644 --- a/packages/workflow/src/deno/composition/effects.ts +++ b/packages/workflow/src/deno/composition/effects.ts @@ -21,7 +21,7 @@ import { RepositoryCompositionProtocolError, } from "../../composition/errors.ts"; import type { WorkflowRunDatabase } from "../../storage/api.ts"; -import { workspaceEffectsFor } from "../../workspace/effects.ts"; +import { workspaceHostFor } from "../../workspace/effects.ts"; import type { WorkspaceFilesystem } from "../../workspace/filesystem.ts"; import { WorkflowStorageError } from "../../storage/errors.ts"; import { savepoint } from "../transaction.ts"; @@ -158,7 +158,7 @@ function* compositionEffect( metadata: WorkspaceMetadata, ) => Operation, ): Workflow { - return yield workspaceEffectsFor(database).create(description, perform); + return yield workspaceHostFor(database).create(description, perform); } /** diff --git a/packages/workflow/src/deno/composition/provider.ts b/packages/workflow/src/deno/composition/provider.ts index 404d1de25..d5242d4bb 100644 --- a/packages/workflow/src/deno/composition/provider.ts +++ b/packages/workflow/src/deno/composition/provider.ts @@ -38,13 +38,12 @@ import type { import type { GitPushOutcome, GitPushRequest } from "../../composition/git-push-records.ts"; import type { RepositoryRecord, WorktreeRecord } from "../../composition/records.ts"; import type { WorkflowRunDatabase } from "../../storage/api.ts"; -import { transactWorkspaceRoots } from "../workspace/private.ts"; -import type { PrivateWorkspaceTransaction } from "../workspace/private.ts"; +import { workspaceHostFor } from "../../workspace/effects.ts"; +import type { WorkspaceAttachmentView } from "../../workspace/effects.ts"; import { gitSession, type GitSession } from "./git.ts"; import { denoRepositoryHost, type RepositoryHost } from "./host.ts"; import type { GitAuthentication } from "./authentication.ts"; import type { HelperAssembly } from "./credential-helper.ts"; -import { WORKSPACE_REPOSITORY, WORKSPACE_WORKTREE } from "./effects.ts"; import { stale, type Attached, type StaleReason } from "./identity.ts"; import { createRepository, @@ -109,11 +108,15 @@ function* attach( database: WorkflowRunDatabase, host: RepositoryHost, subject: string, - prepare: (workspace: PrivateWorkspaceTransaction, root: string) => Operation, + prepare: (workspace: WorkspaceAttachmentView, root: string) => Operation, disagreement: (git: GitSession, attached: Attached) => Operation, ): Operation { const root = yield* host.useDirectory(); - const prepared = yield* transactWorkspaceRoots(database, (workspace) => prepare(workspace, root)); + // Through whichever host attached this run. Locally that is the validated + // lease and its transaction; on a runner it is one coherent owner snapshot + // materialized into a directory this invocation owns. Either way the read is + // over before Git runs, so nothing is held open across a subprocess. + const prepared = yield* workspaceHostFor(database).read((workspace) => prepare(workspace, root)); if (!prepared.ok) { throw prepared.error; } diff --git a/packages/workflow/src/deno/composition/repository.ts b/packages/workflow/src/deno/composition/repository.ts index a52a7f643..4162c7926 100644 --- a/packages/workflow/src/deno/composition/repository.ts +++ b/packages/workflow/src/deno/composition/repository.ts @@ -23,7 +23,7 @@ import { type RepositoryCreationRequest, type RepositoryRecord, } from "../../composition/records.ts"; -import type { PrivateWorkspaceTransaction } from "../workspace/private.ts"; +import type { WorkspaceAttachmentView } from "../../workspace/effects.ts"; import { checkoutPrimary, checkoutReadable, @@ -165,7 +165,7 @@ export function* performRepository( } export function* prepareRepositoryAttachment( - workspace: PrivateWorkspaceTransaction, + workspace: WorkspaceAttachmentView, root: string, record: RepositoryRecord, subject: string, diff --git a/packages/workflow/src/deno/composition/worktree.ts b/packages/workflow/src/deno/composition/worktree.ts index 3c26bd456..d314c9663 100644 --- a/packages/workflow/src/deno/composition/worktree.ts +++ b/packages/workflow/src/deno/composition/worktree.ts @@ -23,7 +23,7 @@ import { type WorktreeRecord, } from "../../composition/records.ts"; import type { StoredRepository } from "../workspace/repositories.ts"; -import type { PrivateWorkspaceTransaction } from "../workspace/private.ts"; +import type { WorkspaceAttachmentView } from "../../workspace/effects.ts"; import { addWorktree, checkoutReadable, @@ -189,7 +189,7 @@ export function* performWorktree( } export function* prepareWorktreeAttachment( - workspace: PrivateWorkspaceTransaction, + workspace: WorkspaceAttachmentView, root: string, record: WorktreeRecord, subject: string, diff --git a/packages/workflow/src/deno/database.ts b/packages/workflow/src/deno/database.ts index 00e262223..d540facb6 100644 --- a/packages/workflow/src/deno/database.ts +++ b/packages/workflow/src/deno/database.ts @@ -49,6 +49,8 @@ import { } from "../storage/record.ts"; import { insertJournalEvent, readJournalEntries } from "./journal.ts"; import { routeWorkflowRunJournal } from "./journal-route.ts"; +import { denoWorkspaceHost } from "./workspace/effect.ts"; +import { useWorkspaceHost } from "../workspace/effects.ts"; import type { RunConnection, RunConnectionLease, @@ -93,6 +95,11 @@ export function openWorkflowRunDatabase( yield* ensure(() => { handle.close(); }); + // This host's answers for this exact handle, for as long as the handle is + // open. An attachment binds a narrower one over the top while a document + // runs; a caller that only opened storage still has the two reads a + // retained mapping and an ephemeral attachment need. + yield* useWorkspaceHost(handle.database, denoWorkspaceHost(handle.database)); yield* provide(handle.database); }); } diff --git a/packages/workflow/src/deno/remote-runner.ts b/packages/workflow/src/deno/remote-runner.ts index 936a690e9..ac8ea83f2 100644 --- a/packages/workflow/src/deno/remote-runner.ts +++ b/packages/workflow/src/deno/remote-runner.ts @@ -45,13 +45,21 @@ import { installRemoteInputDelivery } from "../remote/delivery.ts"; import { useRemoteLifecycleReads } from "../remote/inspection.ts"; import { createRemoteWorkspaceEffect, + readRemoteWorkspace, + transactRemoteAgentSessions, useRemoteRun, useRemoteWorkspaceEffects, withRemoteWorkspaceEffects, } from "../remote/workspace.ts"; -import { useWorkspaceEffects, type WorkspaceMutation } from "../workspace/effects.ts"; +import { + useWorkspaceHost, + type WorkspaceAttachmentView, + type WorkspaceMutation, +} from "../workspace/effects.ts"; +import type { AgentSessions } from "../storage/agent-session.ts"; import { withDocumentCapabilities } from "./workspace/host.ts"; -import type { WorkflowWorkspaceOptions } from "./workspace/host.ts"; +import { permittedWorkspaceOptions } from "./workspace/published.ts"; +import type { WorkflowWorkspaceOptions } from "./workspace/published.ts"; import { WorkflowRequestError } from "../storage/errors.ts"; import { installRemoteWorkflowLifecycle } from "./remote-host.ts"; import { createRemoteWorkspaceFilesystem } from "./remote-workspace-files.ts"; @@ -89,6 +97,13 @@ export interface RemoteWorkflowRunnerOptions { * here is read from a flag, an environment variable, a prop or a global, and * an absent member installs the capability's unconfigured behavior rather * than a different one. + * + * The published options, not the broad internal ones. A substituted + * repository host, a Git-host transport or an invocation observer is a seam + * through which a credential this run acquires would become visible to + * whoever supplied it, and this is a public export: what it accepts is what + * a *host* owns, and the projection into the internal shape is explicit + * rather than a spread of whatever arrived. */ readonly capabilities?: WorkflowWorkspaceOptions; } @@ -178,7 +193,7 @@ export function* useRemoteWorkflowRunner( // together because either half alone is wrong: the rules without the // binding would reach whatever filesystem an entrypoint left in scope, // and the binding without the rules would be a coordinator nothing asks. - yield* useWorkspaceEffects(database, { + yield* useWorkspaceHost(database, { create( description: EffectDescription, mutate: WorkspaceMutation, @@ -187,10 +202,26 @@ export function* useRemoteWorkflowRunner( mutate(filesystem, metadata), ); }, + + read( + body: (view: WorkspaceAttachmentView) => Operation, + ): Operation> { + return readRemoteWorkspace(run, body); + }, + + sessions( + body: (sessions: AgentSessions) => Operation, + ): Operation> { + return transactRemoteAgentSessions(run, body); + }, }); return yield* withRemoteWorkspaceEffects( run, - withDocumentCapabilities(database, operation, options.capabilities ?? {}), + withDocumentCapabilities( + database, + operation, + permittedWorkspaceOptions(options.capabilities ?? {}), + ), ); }, }; diff --git a/packages/workflow/src/deno/workspace/effect.ts b/packages/workflow/src/deno/workspace/effect.ts index 066e1415a..3f903d2c9 100644 --- a/packages/workflow/src/deno/workspace/effect.ts +++ b/packages/workflow/src/deno/workspace/effect.ts @@ -6,7 +6,7 @@ import { type Result as DurableResult, serializeError, } from "@executablemd/durable-streams"; -import { ensure, type Operation, scoped } from "effection"; +import { ensure, type Operation, scoped, type Result } from "effection"; import type { WorkflowRunDatabase, WorkflowRunTransaction } from "../../storage/api.ts"; import { WorkflowTransactionError } from "../../storage/errors.ts"; import { @@ -22,12 +22,16 @@ import { savepoint } from "../transaction.ts"; import { isJournaledEffectFailure } from "./errors.ts"; import type { DenoWorkspaceFilesystem } from "./filesystem.ts"; import type { WorkspaceMetadata } from "./repositories.ts"; -import { - useWorkspaceEffects as bindWorkspaceEffects, - type WorkspaceMutation, +import type { + WorkspaceAttachmentView, + WorkspaceHostBinding, + WorkspaceMutation, } from "../../workspace/effects.ts"; +import type { AgentSessions } from "../../storage/agent-session.ts"; import { type PrivateWorkspaceTransaction, + transactAgentSessions as transactDenoAgentSessions, + transactWorkspaceRoots, withPrivateWorkspaceTransaction, workflowRunTransactionToken, } from "./private.ts"; @@ -231,17 +235,6 @@ export function withWorkspaceEffects( return unavailable(); } connections.validateLease(database); - // The lease this host just validated, as the one binding the shared - // document rules build their effects through. It closes with this scope, - // so an effect created afterwards finds no binding at all. - yield* bindWorkspaceEffects(database, { - create( - description: EffectDescription, - mutate: WorkspaceMutation, - ): DurableEffect { - return createWorkspaceEffect(database, description, mutate); - }, - }); return yield* withWorkspaceCoordinationProvider(coordinator(connections, database), operation); }); } @@ -285,3 +278,36 @@ export function createWorkspaceEffect( workspaceEffectOwners.claim(executionIdentity, database); return createOwnedDurableWorkspaceOperation(description, execute, executionIdentity); } + +/** + * What this host answers for one of its own handles. + * + * The three things the shared document rules ask a host for, over the lease + * this handle already validated. Built for the handle rather than for a scope, + * because the lease is the authority: a handle another provider opened is not + * one this can answer for, and validating it is what says so. + */ +export function denoWorkspaceHost(database: WorkflowRunDatabase): WorkspaceHostBinding { + return { + create( + description: EffectDescription, + mutate: WorkspaceMutation, + ): DurableEffect { + return createWorkspaceEffect(database, description, mutate); + }, + + read( + body: (view: WorkspaceAttachmentView) => Operation, + ): Operation> { + // The same transaction this host has always used for an ephemeral + // attachment, handing the body only the two members it may see. + return transactWorkspaceRoots(database, (workspace) => + body({ filesystem: workspace.filesystem, metadata: workspace.metadata }), + ); + }, + + sessions(body: (sessions: AgentSessions) => Operation): Operation> { + return transactDenoAgentSessions(database, body); + }, + }; +} diff --git a/packages/workflow/src/deno/workspace/files.ts b/packages/workflow/src/deno/workspace/files.ts index a4fbe7e0e..5ba455905 100644 --- a/packages/workflow/src/deno/workspace/files.ts +++ b/packages/workflow/src/deno/workspace/files.ts @@ -73,7 +73,7 @@ import type { } from "@executablemd/runtime"; import type { EffectDescription, Json, Workflow } from "@executablemd/durable-streams"; import type { WorkflowRunDatabase } from "../../storage/api.ts"; -import { workspaceEffectsFor } from "../../workspace/effects.ts"; +import { workspaceHostFor } from "../../workspace/effects.ts"; import type { WorkspaceFilesystem } from "../../workspace/filesystem.ts"; import { savepoint } from "../transaction.ts"; import { journalableWorkspaceCode } from "./errors.ts"; @@ -283,9 +283,7 @@ function* fileEffect( // The binding the host attached for this exact handle. Which host answers is // decided by the handle; that the handle is this run's was proved when the // attachment registered it. - return yield workspaceEffectsFor(database).create(description, (filesystem) => - perform(filesystem), - ); + return yield workspaceHostFor(database).create(description, (filesystem) => perform(filesystem)); } /** diff --git a/packages/workflow/src/deno/workspace/published.ts b/packages/workflow/src/deno/workspace/published.ts index e1fb6d601..779073ea1 100644 --- a/packages/workflow/src/deno/workspace/published.ts +++ b/packages/workflow/src/deno/workspace/published.ts @@ -26,7 +26,10 @@ import type { GitHubIssuesOptions } from "../issue/github.ts"; import type { GitHubPullRequestsOptions } from "../composition/pull-request-reads.ts"; import type { HelperAssembly } from "../composition/credential-helper.ts"; import { withWorkflowWorkspace as withBroadWorkspace } from "./host.ts"; -import type { WorkflowAgentInstaller } from "./host.ts"; +import type { + WorkflowAgentInstaller, + WorkflowWorkspaceOptions as BroadWorkflowWorkspaceOptions, +} from "./host.ts"; /** * What a host may configure, and the whole of it. @@ -62,16 +65,19 @@ export interface WorkflowWorkspaceOptions { readonly agent?: WorkflowAgentInstaller; } -/** Run `operation` with this run's Workspace attached, as a host installs it. */ -export function withWorkflowWorkspace( - database: WorkflowRunDatabase, - operation: Operation, - options: WorkflowWorkspaceOptions = {}, -): Operation { - // Projected member by member. A spread would carry whatever else a caller put - // on the object, and reading an unknown property is how a getter somebody - // else wrote gets to run. - return withBroadWorkspace(database, operation, { +/** + * The broad options these narrow ones permit, and nothing else. + * + * Member by member, because a spread would carry whatever else a caller put on + * the object and reading an unknown property is how a getter somebody else + * wrote gets to run. Shared with the runner assembly, which projects the same + * published options into the same internal shape: two projections of one + * boundary would eventually differ, and the difference would be a seam. + */ +export function permittedWorkspaceOptions( + options: WorkflowWorkspaceOptions, +): BroadWorkflowWorkspaceOptions { + return { ...(options.gitHubIssues === undefined ? {} : { gitHubIssues: options.gitHubIssues }), ...(options.gitHubPullRequests === undefined ? {} @@ -89,5 +95,14 @@ export function withWorkflowWorkspace( }), ...(options.helper === undefined ? {} : { helper: options.helper }), ...(options.agent === undefined ? {} : { agent: options.agent }), - }); + }; +} + +/** Run `operation` with this run's Workspace attached, as a host installs it. */ +export function withWorkflowWorkspace( + database: WorkflowRunDatabase, + operation: Operation, + options: WorkflowWorkspaceOptions = {}, +): Operation { + return withBroadWorkspace(database, operation, permittedWorkspaceOptions(options)); } diff --git a/packages/workflow/src/remote/collector.ts b/packages/workflow/src/remote/collector.ts index e531ca2e7..c4bea4db2 100644 --- a/packages/workflow/src/remote/collector.ts +++ b/packages/workflow/src/remote/collector.ts @@ -180,6 +180,7 @@ export function transactRemotely( enlist: EnlistWorkspace, anchor: TransactionAnchor, consume: EnlistAnswer, + enlistMappings: EnlistMappings, ) => Operation, ): Operation> { return call(function* (): Operation> { @@ -246,7 +247,7 @@ export function transactRemotely( if (!live) { throw new RemoteTransactionError("transaction-closed"); } - if (enlisted !== undefined) { + if (enlisted !== undefined || stagedMappings !== undefined) { throw new RemoteTransactionError("publication-already-enlisted"); } if (mappings.length > MAX_MAPPINGS) { @@ -257,6 +258,29 @@ export function transactRemotely( enlisted = { attempt, mappings: Object.freeze(mappings.map(detachMapping)) }; }; + let stagedMappings: readonly RetainedMapping[] | undefined; + /** + * How a transaction retains mappings without proposing a Workspace. + * + * An Agent-session mapping is a fact about a conversation rather than + * about bytes: the run's Workspace does not move, and nothing is + * published. It is here rather than as an option on `enlist` because the + * two are different proposals — one carries a root and one does not — + * and one transaction makes at most one of them. + */ + const enlistMappings: EnlistMappings = (mappings: readonly RetainedMapping[]): void => { + if (!live) { + throw new RemoteTransactionError("transaction-closed"); + } + if (enlisted !== undefined || stagedMappings !== undefined) { + throw new RemoteTransactionError("publication-already-enlisted"); + } + if (mappings.length > MAX_MAPPINGS) { + throw new RemoteTransactionError("too-many-mappings"); + } + stagedMappings = Object.freeze(mappings.map(detachMapping)); + }; + let consumption: AnswerConsumption | undefined; /** * How an answer claim spends the retained value inside this transaction. @@ -297,6 +321,7 @@ export function transactRemotely( journalEventId: starting.journalEventId, }, consume, + enlistMappings, ), ); } finally { @@ -315,7 +340,7 @@ export function transactRemotely( // A private snapshot. The collector's own array never leaves. events: appended.map((event) => structuredClone(event)), publication: sealed?.publication ?? null, - mappings: sealed?.mappings ?? [], + mappings: sealed?.mappings ?? stagedMappings ?? [], bytes: sealed?.bytes ?? new Map(), answer: consumption ?? null, }); @@ -363,6 +388,9 @@ export type EnlistWorkspace = ( mappings?: readonly RetainedMapping[], ) => void; +/** How a transaction retains mappings with no Workspace proposal at all. */ +export type EnlistMappings = (mappings: readonly RetainedMapping[]) => void; + /** * How an answer claim designates the retained value this transaction spends. * diff --git a/packages/workflow/src/remote/database.ts b/packages/workflow/src/remote/database.ts index adbe557ef..c933cedaa 100644 --- a/packages/workflow/src/remote/database.ts +++ b/packages/workflow/src/remote/database.ts @@ -55,7 +55,12 @@ import type { WorkflowRunRecord, } from "../storage/record.ts"; import { createTransactionGate, type OwnerLink, transactRemotely } from "./collector.ts"; -import type { EnlistAnswer, EnlistWorkspace, TransactionAnchor } from "./collector.ts"; +import type { + EnlistAnswer, + EnlistMappings, + EnlistWorkspace, + TransactionAnchor, +} from "./collector.ts"; import type { RemoteContent, RemoteContentRequest, RemoteFrontierSnapshot } from "./read.ts"; import type { RemoteRetainedAnswer } from "./answer-link.ts"; import type { RemoteInvocationSnapshot } from "./records.ts"; @@ -158,6 +163,8 @@ export interface WorkspaceRoute { readonly database: WorkflowRunDatabase; readonly transaction: WorkflowRunTransaction; readonly enlist: EnlistWorkspace; + /** How this transaction retains mappings with no Workspace proposal. */ + readonly enlistMappings: EnlistMappings; /** Where this transaction began, so a coordinator can prove it has not drifted. */ readonly anchor: TransactionAnchor; /** @@ -379,7 +386,7 @@ export function useRemoteRunDatabase( return yield* transactRemotely( link, gate, - function* (transaction, enlist, anchor, consume) { + function* (transaction, enlist, anchor, consume, enlistMappings) { // The marker and the route are installed for the body's scope // alone. Outside it neither exists, so a retained transaction // object reaches nothing and an unrelated scope is not mistaken for @@ -388,7 +395,14 @@ export function useRemoteRunDatabase( handle, enclosing: yield* ActiveTransaction.get(), }); - yield* ActiveRoute.set({ database: handle, transaction, enlist, anchor, consume }); + yield* ActiveRoute.set({ + database: handle, + transaction, + enlist, + enlistMappings, + anchor, + consume, + }); return yield* body(transaction); }, ); diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts index c1c698098..390d61edf 100644 --- a/packages/workflow/src/remote/workspace.ts +++ b/packages/workflow/src/remote/workspace.ts @@ -45,15 +45,17 @@ import { type Result as DurableResult, serializeError, } from "@executablemd/durable-streams"; -import { ensure, type Operation, scoped } from "effection"; +import { ensure, Err, Ok, type Operation, type Result, scoped } from "effection"; import type { WorkflowRunDatabase, WorkflowRunTransaction } from "../storage/api.ts"; import { WorkflowTransactionError } from "../storage/errors.ts"; import type { WorkspaceFilesystem } from "../workspace/filesystem.ts"; import { isJournaledEffectFailure } from "../workspace/failure.ts"; import type { WorkspaceMetadata } from "../workspace/metadata.ts"; import type { AgentSessions } from "../storage/agent-session.ts"; +import type { WorkspaceAttachmentView } from "../workspace/effects.ts"; import { activeWorkspaceRoute, type WorkspaceRoute } from "./database.ts"; import { createInvocationMappings } from "./mappings.ts"; +import type { RetainedMapping } from "./publication.ts"; import { Transaction } from "../workspace/savepoint.ts"; import { type Attempt, @@ -355,6 +357,110 @@ export function createRemoteWorkspaceEffect( return createOwnedDurableWorkspaceOperation(description, execute, executionIdentity); } +/** + * Read this run's retained Workspace for one ephemeral attachment. + * + * The two things an attachment may see, over a tree this invocation owns. The + * root is materialized here and the materialization is this scope's, so the + * export the body takes out of it survives and the tree does not: native Git + * runs afterwards, against files, with no owner read and no transaction held + * open across it. + * + * Nothing durable happens. No root is published, no mapping is staged, and the + * owner is asked for one coherent snapshot and the content that snapshot names. + */ +export function readRemoteWorkspace( + run: RemoteRun, + body: (view: WorkspaceAttachmentView) => Operation, +): Operation> { + return scoped(function* () { + const bound = bindings.of(run); + if (bound === undefined) { + unavailable("this is not a remote run this build opened."); + } + const { runtime } = bound; + const reject = (reason: string): never => unavailable(reason); + try { + const snapshot = yield* runtime.reads.invocationSnapshot(); + const materialization = yield* useMaterialization( + runtime.files, + runtime.trees, + runtime.reads, + snapshot.workspaceRootId, + reject, + ); + // Live for as long as this read is. An attachment that kept the + // filesystem would be holding a directory this scope is about to remove. + let live = true; + yield* ensure(() => { + live = false; + }); + const authorize = (): void => { + if (!live) { + unavailable("this remote Workspace read is over."); + } + }; + const mappings = createInvocationMappings(snapshot, authorize); + const filesystem = runtime.createFilesystem(materialization.at, authorize); + return Ok(yield* body({ filesystem, metadata: mappings.metadata })); + } catch (error) { + return Err(error instanceof Error ? error : new Error(String(error))); + } + }); +} + +/** + * Read and commit this run's Agent-session mappings, in one transaction. + * + * Two halves that must not be confused. The body runs on the runner, where the + * provider is, and the mappings it reads are the ones the owner admitted; what + * it changed is collected as bounded deltas and submitted to the owner, which + * revalidates the subject under its own transaction and commits all of them or + * none. No provider call happens inside that transaction — there is no + * transaction open while the body runs — and a body that failed or was + * cancelled sends nothing at all. + */ +export function transactRemoteAgentSessions( + run: RemoteRun, + body: (sessions: AgentSessions) => Operation, +): Operation> { + return scoped(function* () { + const bound = bindings.of(run); + if (bound === undefined) { + unavailable("this is not a remote run this build opened."); + } + const { runtime, database } = bound; + let value: T; + let deltas: readonly RetainedMapping[]; + try { + const snapshot = yield* runtime.reads.invocationSnapshot(); + const mappings = createInvocationMappings(snapshot, () => undefined); + // The body first, and outside any transaction: establishing or asserting + // a conversation is provider work, and an owner transaction is not a + // place a provider call may happen. + value = yield* body(mappings.agentSessions); + deltas = mappings.deltas(); + } catch (error) { + return Err(error instanceof Error ? error : new Error(String(error))); + } + if (deltas.length === 0) { + // Nothing was staged, so there is nothing for the owner to decide. + return Ok(value); + } + // One transaction that carries mappings and nothing else: no event, no + // publication, no answer. The owner revalidates each subject against what + // it holds and commits every delta or refuses them together. + return yield* database.transact(function* (transaction) { + const route = yield* activeWorkspaceRoute(database, transaction); + if (route === undefined) { + unavailable("this transaction is not the one this run's mappings may be retained in."); + } + route.enlistMappings(deltas); + return value; + }); + }); +} + function coordinator(run: BoundRun): WorkspaceCoordinationProvider { return { *run(authority: WorkspaceCoordinationAuthority): Operation { diff --git a/packages/workflow/src/workspace/effects.ts b/packages/workflow/src/workspace/effects.ts index 71c9dec52..5aa7c66e9 100644 --- a/packages/workflow/src/workspace/effects.ts +++ b/packages/workflow/src/workspace/effects.ts @@ -21,12 +21,13 @@ * never what says a database, journal or publication target belongs to a run. */ -import { ensure, type Operation } from "effection"; +import { ensure, type Operation, type Result } from "effection"; import type { DurableEffect, EffectDescription, Json } from "@executablemd/durable-streams"; import type { WorkflowRunDatabase } from "../storage/api.ts"; import { WorkflowTransactionError } from "../storage/errors.ts"; import type { WorkspaceFilesystem } from "./filesystem.ts"; import type { WorkspaceMetadata } from "./metadata.ts"; +import type { AgentSessions } from "../storage/agent-session.ts"; /** * What a Workspace mutation is given. @@ -40,8 +41,29 @@ export type WorkspaceMutation = ( metadata: WorkspaceMetadata, ) => Operation; -/** What a host contributes: one effect for one mutation of one run. */ -export interface WorkspaceEffectBinding { +/** + * What an ephemeral attachment is allowed to see. + * + * A checkout is rebuilt from the Workspace and proved to be the retained one on + * every partial execution, and doing that needs two things: the bytes, and the + * record that names them. It needs nothing else — no publication, no capture, + * no restore, no root selection — so this is the whole of what it is handed, + * and a host cannot hand it more by supplying a wider object. + */ +export interface WorkspaceAttachmentView { + readonly filesystem: WorkspaceFilesystem; + readonly metadata: WorkspaceMetadata; +} + +/** + * What a host contributes for one run, and the whole of it. + * + * Three things, because a document reaches exactly three kinds of Workspace + * work: the effect a mutation becomes, the read an ephemeral attachment needs, + * and the transaction an Agent-session mapping is retained by. Everything else + * the rules do is arithmetic above these. + */ +export interface WorkspaceHostBinding { /** * The effect that performs this mutation. * @@ -53,18 +75,37 @@ export interface WorkspaceEffectBinding { description: EffectDescription, mutate: WorkspaceMutation, ): DurableEffect; + /** + * Read this run's retained Workspace for one ephemeral attachment. + * + * Nothing durable happens here: a checkout is exported to a host directory + * and proved, and the Workspace is left exactly as it was found. The scope + * this opens closes before native work runs against what it exported, so no + * transaction and no owner read is held across a Git process. + */ + read(body: (view: WorkspaceAttachmentView) => Operation): Operation>; + /** + * Read and commit this run's Agent-session mappings, in one transaction. + * + * The narrow half of the run's retained state, and the only half a host + * installing an Agent profile needs. A conversation and the row naming it are + * one fact, so the mapping this returns is retained by the run's own owner or + * by nothing — and the provider that establishes the conversation is never + * called from inside that transaction. + */ + sessions(body: (sessions: AgentSessions) => Operation): Operation>; } const bindings = (() => { - const held = new WeakMap(); + const held = new WeakMap(); return { - attach(database: WorkflowRunDatabase, binding: WorkspaceEffectBinding): void { + attach(database: WorkflowRunDatabase, binding: WorkspaceHostBinding): void { held.set(database, binding); }, detach(database: WorkflowRunDatabase): void { held.delete(database); }, - of(database: WorkflowRunDatabase): WorkspaceEffectBinding | undefined { + of(database: WorkflowRunDatabase): WorkspaceHostBinding | undefined { return held.get(database); }, }; @@ -77,13 +118,23 @@ const bindings = (() => { * ends, however it ends — so an effect created after the attachment is over * finds nothing rather than a stale filesystem or a closed transaction. */ -export function* useWorkspaceEffects( +export function* useWorkspaceHost( database: WorkflowRunDatabase, - binding: WorkspaceEffectBinding, + binding: WorkspaceHostBinding, ): Operation { + // What was bound before, restored when this scope ends. A handle's own host + // binds one when it opens the handle at all — reading a mapping needs no + // attachment — and an attachment binds a narrower one for as long as the + // document runs. Detaching to nothing would leave the handle unusable for + // the rest of its own life. + const outer = bindings.of(database); bindings.attach(database, binding); yield* ensure(() => { - bindings.detach(database); + if (outer === undefined) { + bindings.detach(database); + return; + } + bindings.attach(database, outer); }); } @@ -95,7 +146,7 @@ export function* useWorkspaceEffects( * agreed to perform, and performing it against whatever filesystem happens to * be in scope is the one outcome that must not be possible. */ -export function workspaceEffectsFor(database: WorkflowRunDatabase): WorkspaceEffectBinding { +export function workspaceHostFor(database: WorkflowRunDatabase): WorkspaceHostBinding { const binding = bindings.of(database); if (binding === undefined) { throw new WorkflowTransactionError( @@ -105,3 +156,19 @@ export function workspaceEffectsFor(database: WorkflowRunDatabase): WorkspaceEff } return binding; } + +/** + * Read and commit this run's Agent-session mappings, wherever it lives. + * + * The same name and the same narrow behavior a host installing an Agent profile + * has always used, answered by whichever host attached this exact handle. A + * conversation and the row naming it are one fact, so what the body stages is + * retained by the run's own owner or by nothing — and no provider call happens + * inside that transaction. + */ +export function transactAgentSessions( + database: WorkflowRunDatabase, + body: (sessions: AgentSessions) => Operation, +): Operation> { + return workspaceHostFor(database).sessions(body); +} diff --git a/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts b/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts index e38e27a67..7122e09d1 100644 --- a/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts +++ b/packages/workflow/tests/cloudflare/remote-owner-routes.vitest.ts @@ -13,7 +13,7 @@ * not either half against a fixture of the other. */ -import { env, runInDurableObject } from "cloudflare:test"; +import { env, evictDurableObject, runInDurableObject } from "cloudflare:test"; import { beforeAll, describe, expect, it } from "vitest"; import { run, scoped, until, type Operation } from "effection"; import { remoteOwnerClient } from "../../src/cloudflare/configured.ts"; @@ -277,7 +277,7 @@ describe("the owner's request boundary", () => { expect(await on(stubFor(runId), (owner) => owner.holders())).toBe(0); }); - it("keeps the upgraded socket authoritative between separate object accesses", async () => { + it("keeps the upgraded socket authoritative across a real eviction", async () => { const runId = runOf(); const built = await client(runId); const outcome = await run(function* (): Operation> { @@ -289,12 +289,13 @@ describe("the owner's request boundary", () => { // The acquisition is registered on the socket's attachment, which is // what an evicted object reads back — nothing about it is in memory. const before = yield* until(on(stubFor(runId), (owner) => owner.holders())); - // Read again through a separate access to the object. Nothing about - // the acquisition is in this instance's memory — it is the socket's - // serialized attachment, which is the same mechanism a hibernated - // object restores from — and the socket the runner still holds is the - // one that has to remain authoritative either way. + // The object is really evicted, with this client's socket still open. + // Nothing about the acquisition is in the instance that comes back: it + // is the socket's serialized attachment, which is what a hibernated + // object restores from. + yield* until(evictDurableObject(stubFor(runId))); const evicted = yield* until(on(stubFor(runId), (owner) => owner.holders())); + // The same client, over the same link, after that eviction. const opened = yield* admitted.value.link.open(runId, null); return { admitted: true, diff --git a/packages/workflow/tests/remote-runner.test.ts b/packages/workflow/tests/remote-runner.test.ts index 836224433..542af43a8 100644 --- a/packages/workflow/tests/remote-runner.test.ts +++ b/packages/workflow/tests/remote-runner.test.ts @@ -17,18 +17,10 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { Ok, type Operation, scoped, until } from "effection"; -import { mkdir, writeFile } from "node:fs/promises"; -import { collect, execute, inlineSource } from "@executablemd/core"; -import { API, useHostFiles } from "@executablemd/runtime"; -import type { HostFilesEvent } from "@executablemd/runtime"; +import { Ok, type Operation, scoped, sleep, spawn, suspend } from "effection"; import type { Json } from "@executablemd/durable-streams"; -import { encodeBase64 } from "../src/cloudflare/encoding.ts"; import { cloudflareReadLink, cloudflareRunLink } from "../src/cloudflare/client.ts"; import { cloudflareLifecycleLink } from "../src/cloudflare/lifecycle-link.ts"; -import { type OwnerSocket, type SocketListener, useOwnerConnection } from "../src/remote/client.ts"; -import { captureWorkspace, type CapturedWorkspace } from "../src/remote/materialize.ts"; -import { runnerFiles, useRunnerTrees } from "../src/deno/remote-files.ts"; import { WorkflowLifecycle } from "../src/lifecycle/api.ts"; import type { ExecutorLock } from "../src/lifecycle/api.ts"; import type { WorkflowExecutionTransitions } from "../src/lifecycle/execution.ts"; @@ -37,7 +29,22 @@ import { useRemoteWorkflowRunner } from "../src/deno/remote-runner.ts"; import type { RemoteRunnerOwner, RemoteWorkflowRunner } from "../src/deno/remote-runner.ts"; import type { RemoteReadPlane } from "../src/remote/read.ts"; import { WorkflowRequestError } from "../src/storage/errors.ts"; -import { installedHost, RUN_ID, type Script } from "./support/remote-lifecycle-host.ts"; +import { installedHost, type Script } from "./support/remote-lifecycle-host.ts"; +import { + document, + published, + RUN_ID, + scriptedOwner, + type ScriptedRetention, + startingTree, + useHostSpy, +} from "./support/remote-owner-script.ts"; +import { useOwnerConnection } from "../src/remote/client.ts"; +import { transactAgentSessions, workspaceHostFor } from "../src/workspace/effects.ts"; +import type { CapturedWorkspace } from "../src/remote/materialize.ts"; +import { locatorFingerprintOf } from "../src/composition/locator.ts"; +import { agentSessionKey } from "../src/storage/agent-session.ts"; +import { durableRun, type Workflow } from "@executablemd/durable-streams"; /** One scripted owner, and every run its acquisitions were opened for. */ function ownerOf(script: Script = {}): { owner: RemoteRunnerOwner; acquisitions: string[] } { @@ -162,215 +169,15 @@ function foreignHandle(): WorkflowRunDatabase { }; } -/** - * One owner, scripted at the wire. - * - * Everything above it is production code: the real client, the real lifecycle, - * the real database handle, the real coordinator and the real runner - * facilities. What this stands in for is the object that would answer — so what - * a test can say afterwards is what the runner actually sent it, and what it - * committed. - */ -function scriptedOwner(captured: CapturedWorkspace) { - const sent: Record[] = []; - const commits: Record[] = []; - let currentRoot = captured.root.rootId; - let refusal: string | undefined; - let lost = false; - - function frontier(): Record { - return { - record: { - runId: RUN_ID, - definition: { - version: 1, - kind: "git", - objectFormat: "sha1", - objectId: "0".repeat(40), - rootDocumentPath: "README.md", - }, - base: "main", - props: {}, - status: "running", - createdAt: "2026-09-10T00:00:00.000Z", - updatedAt: "2026-09-10T00:00:00.000Z", - }, - retrieval: null, - workspaceRootId: currentRoot, - journalEventId: null, - }; - } - - function begun(executionId: string): Record { - return { - frontier: frontier(), - // Exactly the members an execution that has not stopped declares. - execution: { executionId, startedAt: "2026-09-10T00:00:01.000Z" }, - replay: false, - recovered: null, - }; - } - - function answer(request: Record): Record { - const command = request["command"]; - if (command === "open" || command === "frontier") { - return { outcome: "performed", value: frontier() }; - } - if (command === "begin") { - // A lifecycle answer is one of three fields and never two: the value, a - // refusal, or the immutable fields a creation conflicts on. - return { - outcome: "performed", - value: { - conflict: null, - refusal: null, - value: begun(String(request["executionId"])), - }, - }; - } - if (command === "mappings") { - return { - outcome: "performed", - value: { - workspaceRootId: currentRoot, - journalEventId: null, - repositories: [], - worktrees: [], - agentSessions: [], - }, - }; - } - if (command === "root") { - return { - outcome: "performed", - value: { workspaceRootId: currentRoot, manifest: captured.root.manifest }, - }; - } - if (command === "content") { - const digest = String(request["digest"]); - const bytes = - request["kind"] === "manifest" - ? captured.contents.get(digest)?.manifestBytes - : captured.blobs.get(digest); - if (bytes === undefined) { - throw new Error("asked for content this owner does not hold"); - } - return { - outcome: "performed", - value: { - kind: request["kind"], - digest, - size: bytes.length, - bytes: encodeBase64(bytes), - }, - }; - } - if (command === "stage") { - const encoded = String(request["bytes"] ?? ""); - const padding = encoded.endsWith("==") ? 2 : encoded.endsWith("=") ? 1 : 0; - return { - outcome: "performed", - value: { - kind: request["kind"], - digest: request["digest"], - size: (encoded.length / 4) * 3 - padding, - }, - }; - } - if (command === "settle") { - return { outcome: "performed", value: { status: "completed" } }; - } - commits.push(request); - const publication = request["publication"]; - // Scripted for the Workspace proposal rather than for every append: an - // owner that refused the run's own journal rows would end the document - // before it ever reached the effect under test. - const proposing = publication !== null && publication !== undefined; - if (lost && proposing) { - return { outcome: "lost" }; - } - if (refusal !== undefined && proposing) { - return { outcome: "refused", refusal }; - } - const events = Array.isArray(request["events"]) ? request["events"] : []; - // The owner publishes what it validated, and the frontier moves with it. - if (publication !== null && publication !== undefined) { - currentRoot = String(Reflect.get(publication, "proposedWorkspaceRootId")); - } - return { - outcome: "performed", - value: { - workspaceRootId: currentRoot, - journalEventIds: events.map((_entry, index) => `event-${index}`), - }, - }; - } - - const listeners = new Map>(); - const socket: OwnerSocket = { - send(data: string): void { - const request: Record = JSON.parse(data); - sent.push(request); - const response = answer(request); - if (response["outcome"] === "lost") { - for (const listener of listeners.get("close") ?? []) { - listener({}); - } - return; - } - for (const listener of listeners.get("message") ?? []) { - listener({ data: JSON.stringify({ id: request["id"], ...response }) }); - } - }, - close(): void {}, - addEventListener(type, listener): void { - const found = listeners.get(type) ?? new Set(); - found.add(listener); - listeners.set(type, found); - }, - removeEventListener(type, listener): void { - listeners.get(type)?.delete(listener); - }, - }; - - return { - socket, - sent, - commits, - get currentRoot(): string { - return currentRoot; - }, - refuse(reason: string): void { - refusal = reason; - }, - lose(): void { - lost = true; - }, - }; -} - -/** A small starting tree, captured so the scripted owner can serve it. */ -function* startingTree(): Operation { - const files = runnerFiles(); - const trees = yield* useRunnerTrees(); - const root = yield* trees.create("source"); - yield* until(writeFile(`${root}/README.md`, "starting\n", { mode: 0o644 })); - yield* until(mkdir(`${root}/docs`, { mode: 0o755 })); - return yield* captureWorkspace( - files, - (logical) => (logical === "/" ? root : `${root}${logical}`), - (reason) => { - throw new Error(reason); - }, - ); -} - /** One runner over a scripted owner reached through the production client. */ -function* wired(captured: CapturedWorkspace): Operation<{ +function* wired( + captured: CapturedWorkspace, + retained: ScriptedRetention = {}, +): Operation<{ owner: ReturnType; runner: RemoteWorkflowRunner; }> { - const owner = scriptedOwner(captured); + const owner = scriptedOwner(captured, retained); const connection = yield* useOwnerConnection(owner.socket); let identifier = 0; const next = () => `command-${(identifier += 1)}`; @@ -407,51 +214,6 @@ function* wired(captured: CapturedWorkspace): Operation<{ return { owner, runner }; } -/** - * The ambient host filesystem a runtime entrypoint installs, watched. - * - * The real provider rather than a stand-in, at the position a host installs it - * and with a working directory a workflow run must never resolve against. What - * a test says afterwards is whether a document reached it at all. - */ -function* useHostSpy(): Operation { - const seen: HostFilesEvent[] = []; - yield* API.Env.around( - { - // deno-lint-ignore require-yield - *cwd(): Operation { - return "/nowhere-the-workflow-may-reach"; - }, - }, - { at: "min" }, - ); - yield* useHostFiles({ observe: (event) => seen.push(event) }); - return seen; -} - -/** - * The commits that proposed a Workspace, out of everything the owner was asked - * to commit. - * - * A run's journal lives on its owner, so every ordinary append — the root - * import, a component import, the terminal — reaches it as a commit of its own. - * What a Workspace effect adds to one is the publication, and that is what - * these tests are counting. - */ -function published(commits: readonly Record[]): Record[] { - return commits.filter((intent) => { - const publication = intent["publication"]; - return publication !== null && publication !== undefined; - }); -} - -/** One authored document, executed as this run's root inside the attachment. */ -function document(source: string, database: WorkflowRunDatabase): Operation { - return scoped(function* () { - return yield* collect(yield* execute({ ...inlineSource(source), stream: database.journal })); - }); -} - describe("a runner for a run whose storage is somewhere else", () => { it("attaches the handle its own lifecycle opened, and no other", function* () { const outcome = yield* scoped(function* () { @@ -647,6 +409,215 @@ describe("a runner for a run whose storage is somewhere else", () => { expect(outcomes.failed.said).toBe("raised:Error"); }); + it("reads the retained Workspace an ephemeral attachment needs, and keeps nothing", function* () { + /** A repository this owner already retains, at a path the root contains. */ + const stored = { + record: { + name: "app", + locatorFingerprint: locatorFingerprintOf("https://git.example.invalid/octo/app.git"), + requestedBase: null, + creationCommit: "9".repeat(40), + primaryBranch: "main", + objectFormat: "sha1", + checkoutPath: "/docs", + }, + locator: "https://git.example.invalid/octo/app.git", + }; + + const outcome = yield* scoped(function* () { + const captured = yield* startingTree(); + const { owner, runner } = yield* wired(captured, { repositories: [stored] }); + const transitions = yield* runner.useRunHost(); + const database = yield* opened(transitions, yield* acquired()); + return yield* runner.attach( + database, + (function* (): Operation> { + // Exactly what a Repository reattachment asks of the run: the record + // that names the checkout, and the bytes at it. No Deno lease, no + // Deno private workspace, and no transaction held while it reads. + const read = yield* workspaceHostFor(database).read(function* (view) { + const record = view.metadata.readRepository("app"); + const entries = yield* view.filesystem.readdir("/"); + const readme = yield* view.filesystem.readTextFile("/README.md"); + return { + named: record?.record.checkoutPath, + locator: record?.locator, + entries: entries.map((entry) => entry.name).toSorted(), + readme, + }; + }); + if (!read.ok) { + throw read.error; + } + return { ...read.value, commits: published(owner.commits).length }; + })(), + ); + }); + + // The retained record and the retained bytes, from the owner's own + // snapshot and the root it named. + expect(outcome["named"]).toBe("/docs"); + expect(outcome["locator"]).toBe("https://git.example.invalid/octo/app.git"); + expect(outcome["entries"]).toEqual(["README.md", "docs"]); + expect(outcome["readme"]).toBe("starting\n"); + // Nothing durable happened: no proposal, no publication, no mapping. + expect(outcome["commits"]).toBe(0); + }); + + it("retains an Agent-session mapping at the owner, in one transaction", function* () { + // The key is derived from the identity rather than chosen: a record whose + // key does not follow from what it names is one no owner retains. + const identity = { + provider: "acpx", + agentCommand: "/usr/bin/claude", + sessionIdentity: "expansion-1", + }; + const session = { + ...identity, + sessionKey: agentSessionKey(identity), + policy: "policy-1", + assertion: { kind: "acp", value: "conversation-1" }, + createdAt: "2026-09-10T00:00:00.000Z", + }; + + const outcomes = yield* scoped(function* () { + /** One Agent-session body, under an owner that already retains this. */ + function* attempt( + retained: ScriptedRetention, + body: (sessions: { + read(key: string): unknown; + commit(record: typeof session): void; + }) => Operation, + ): Operation<{ said: string; mappings: number; publications: number }> { + return yield* scoped(function* () { + const captured = yield* startingTree(); + const { owner, runner } = yield* wired(captured, retained); + const transitions = yield* runner.useRunHost(); + const database = yield* opened(transitions, yield* acquired()); + let said: string; + try { + said = yield* runner.attach( + database, + (function* (): Operation { + const committed = yield* transactAgentSessions(database, body); + return committed.ok ? committed.value : `refused:${committed.error.name}`; + })(), + ); + } catch (error) { + said = error instanceof Error ? `raised:${error.name}` : "raised:other"; + } + const mapped = owner.commits.filter((intent) => { + const mappings = intent["mappings"]; + return Array.isArray(mappings) && mappings.length > 0; + }); + return { + said, + mappings: mapped.length, + publications: published(owner.commits).length, + }; + }); + } + + return { + // Nothing retained yet: the mapping is staged and the owner commits it. + retained: yield* attempt({}, function* (sessions) { + sessions.commit(session); + return "committed"; + }), + // The same mapping already retained: reading it is enough, and there is + // nothing for the owner to decide. + already: yield* attempt({ agentSessions: [session] }, function* (sessions) { + return sessions.read(session.sessionKey) === undefined ? "absent" : "read"; + }), + // A different conversation under the same identity: refused where the + // rules live, and never sent. + conflicting: yield* attempt( + { agentSessions: [session] }, + // deno-lint-ignore require-yield + function* (sessions) { + sessions.commit({ ...session, assertion: { kind: "acp", value: "another" } }); + return "committed"; + }, + ), + // A body that failed after staging sends no mapping at all. + failed: yield* attempt( + {}, + // deno-lint-ignore require-yield + function* (sessions) { + sessions.commit(session); + throw new Error("PlantedAgentFailure"); + }, + ), + }; + }); + + // One mapping-only transaction, and no Workspace proposal with it. + expect(outcomes.retained.said).toBe("committed"); + expect(outcomes.retained.mappings).toBe(1); + expect(outcomes.retained.publications).toBe(0); + // Reading what the owner admitted stages nothing. + expect(outcomes.already.said).toBe("read"); + expect(outcomes.already.mappings).toBe(0); + // A conflicting assertion never replaces the retained one. + expect(outcomes.conflicting.said).toContain("refused:"); + expect(outcomes.conflicting.mappings).toBe(0); + // And a failure before the commit retains nothing. + expect(outcomes.failed.said).toContain("refused:"); + expect(outcomes.failed.mappings).toBe(0); + }); + + it("cancels an in-flight attachment without proposing anything", function* () { + const outcome = yield* scoped(function* () { + const captured = yield* startingTree(); + const { owner, runner } = yield* wired(captured); + const transitions = yield* runner.useRunHost(); + const database = yield* opened(transitions, yield* acquired()); + const reached = { inside: false }; + // The attachment is halted while the document is inside its own effect, + // which is where a real cancellation arrives: between the mutation and + // the commit the collector would have sent. + const running = yield* spawn(() => + runner.attach( + database, + (function* (): Operation { + const binding = workspaceHostFor(database); + yield* (function* (): Operation { + const effect = binding.create( + { type: "workspace", name: "cancelled" }, + function* (filesystem): Operation { + yield* filesystem.writeFile("/NOTES.md", "never committed\n", 0o644); + reached.inside = true; + // Nothing settles this: the halt below is what ends it. + yield* suspend(); + return "unreachable"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + return yield* durableRun(workflow, { stream: database.journal }); + })(); + return "unreachable"; + })(), + ), + ); + // Let the effect get inside its mutation, then halt the attachment. + while (!reached.inside) { + yield* sleep(1); + } + yield* running.halt(); + return { owner, inside: reached.inside }; + }); + + // The document got as far as writing into its attempt, and the owner was + // never asked to commit any of it. + expect(outcome.inside).toBe(true); + expect(published(outcome.owner.commits)).toEqual([]); + // The attachment's own scope is over, so the temporary trees it + // materialized into are gone with it. + expect(outcome.owner.commits.every((intent) => intent["publication"] === null)).toBe(true); + }); + it("reads and delivers without taking an acquisition", function* () { const outcome = yield* scoped(function* () { const scripted = ownerOf(); diff --git a/packages/workflow/tests/support/remote-owner-script.ts b/packages/workflow/tests/support/remote-owner-script.ts new file mode 100644 index 000000000..8714c64f4 --- /dev/null +++ b/packages/workflow/tests/support/remote-owner-script.ts @@ -0,0 +1,278 @@ +/** + * One owner, scripted at the wire, and what a document needs around it. + * + * Shared because two suites ask the same question of the same production stack + * from two different heights: the configured public host, and the runner it + * assembles. A second copy of this owner would be a second protocol, and the + * day the two disagreed one of those suites would be proving nothing. + */ + +import { type Operation, scoped, until } from "effection"; +import { mkdir, writeFile } from "node:fs/promises"; +import { collect, execute, inlineSource } from "@executablemd/core"; +import { API, useHostFiles } from "@executablemd/runtime"; +import type { HostFilesEvent } from "@executablemd/runtime"; +import type { Json } from "@executablemd/durable-streams"; +import { encodeBase64 } from "../../src/cloudflare/encoding.ts"; +import type { OwnerSocket, SocketListener } from "../../src/remote/client.ts"; +import { captureWorkspace, type CapturedWorkspace } from "../../src/remote/materialize.ts"; +import { runnerFiles, useRunnerTrees } from "../../src/deno/remote-files.ts"; +import type { WorkflowRunDatabase } from "../../src/storage/api.ts"; + +/** The run every scripted owner here answers about. */ +export const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + +/** + * One owner, scripted at the wire. + * + * Everything above it is production code: the real client, the real lifecycle, + * the real database handle, the real coordinator and the real runner + * facilities. What this stands in for is the object that would answer — so what + * a test can say afterwards is what the runner actually sent it, and what it + * committed. + */ +export interface ScriptedRetention { + /** Repositories this owner already retains, as its snapshot reports them. */ + readonly repositories?: readonly Record[]; + /** Agent sessions this owner already retains. */ + readonly agentSessions?: readonly Record[]; +} + +export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRetention = {}) { + const sent: Record[] = []; + const commits: Record[] = []; + let currentRoot = captured.root.rootId; + let refusal: string | undefined; + let lost = false; + + function frontier(): Record { + return { + record: { + runId: RUN_ID, + definition: { + version: 1, + kind: "git", + objectFormat: "sha1", + objectId: "0".repeat(40), + rootDocumentPath: "README.md", + }, + base: "main", + props: {}, + status: "running", + createdAt: "2026-09-10T00:00:00.000Z", + updatedAt: "2026-09-10T00:00:00.000Z", + }, + retrieval: null, + workspaceRootId: currentRoot, + journalEventId: null, + }; + } + + function begun(executionId: string): Record { + return { + frontier: frontier(), + // Exactly the members an execution that has not stopped declares. + execution: { executionId, startedAt: "2026-09-10T00:00:01.000Z" }, + replay: false, + recovered: null, + }; + } + + function answer(request: Record): Record { + const command = request["command"]; + if (command === "open" || command === "frontier") { + return { outcome: "performed", value: frontier() }; + } + if (command === "begin") { + // A lifecycle answer is one of three fields and never two: the value, a + // refusal, or the immutable fields a creation conflicts on. + return { + outcome: "performed", + value: { + conflict: null, + refusal: null, + value: begun(String(request["executionId"])), + }, + }; + } + if (command === "mappings") { + return { + outcome: "performed", + value: { + workspaceRootId: currentRoot, + journalEventId: null, + repositories: retained.repositories ?? [], + worktrees: [], + agentSessions: retained.agentSessions ?? [], + }, + }; + } + if (command === "root") { + return { + outcome: "performed", + value: { workspaceRootId: currentRoot, manifest: captured.root.manifest }, + }; + } + if (command === "content") { + const digest = String(request["digest"]); + const bytes = + request["kind"] === "manifest" + ? captured.contents.get(digest)?.manifestBytes + : captured.blobs.get(digest); + if (bytes === undefined) { + throw new Error("asked for content this owner does not hold"); + } + return { + outcome: "performed", + value: { + kind: request["kind"], + digest, + size: bytes.length, + bytes: encodeBase64(bytes), + }, + }; + } + if (command === "stage") { + const encoded = String(request["bytes"] ?? ""); + const padding = encoded.endsWith("==") ? 2 : encoded.endsWith("=") ? 1 : 0; + return { + outcome: "performed", + value: { + kind: request["kind"], + digest: request["digest"], + size: (encoded.length / 4) * 3 - padding, + }, + }; + } + if (command === "settle") { + return { outcome: "performed", value: { status: "completed" } }; + } + commits.push(request); + const publication = request["publication"]; + // Scripted for the Workspace proposal rather than for every append: an + // owner that refused the run's own journal rows would end the document + // before it ever reached the effect under test. + const proposing = publication !== null && publication !== undefined; + if (lost && proposing) { + return { outcome: "lost" }; + } + if (refusal !== undefined && proposing) { + return { outcome: "refused", refusal }; + } + const events = Array.isArray(request["events"]) ? request["events"] : []; + // The owner publishes what it validated, and the frontier moves with it. + if (publication !== null && publication !== undefined) { + currentRoot = String(Reflect.get(publication, "proposedWorkspaceRootId")); + } + return { + outcome: "performed", + value: { + workspaceRootId: currentRoot, + journalEventIds: events.map((_entry, index) => `event-${index}`), + }, + }; + } + + const listeners = new Map>(); + const socket: OwnerSocket = { + send(data: string): void { + const request: Record = JSON.parse(data); + sent.push(request); + const response = answer(request); + if (response["outcome"] === "lost") { + for (const listener of listeners.get("close") ?? []) { + listener({}); + } + return; + } + for (const listener of listeners.get("message") ?? []) { + listener({ data: JSON.stringify({ id: request["id"], ...response }) }); + } + }, + close(): void {}, + addEventListener(type, listener): void { + const found = listeners.get(type) ?? new Set(); + found.add(listener); + listeners.set(type, found); + }, + removeEventListener(type, listener): void { + listeners.get(type)?.delete(listener); + }, + }; + + return { + socket, + sent, + commits, + get currentRoot(): string { + return currentRoot; + }, + refuse(reason: string): void { + refusal = reason; + }, + lose(): void { + lost = true; + }, + }; +} + +/** A small starting tree, captured so the scripted owner can serve it. */ +export function* startingTree(): Operation { + const files = runnerFiles(); + const trees = yield* useRunnerTrees(); + const root = yield* trees.create("source"); + yield* until(writeFile(`${root}/README.md`, "starting\n", { mode: 0o644 })); + yield* until(mkdir(`${root}/docs`, { mode: 0o755 })); + return yield* captureWorkspace( + files, + (logical) => (logical === "/" ? root : `${root}${logical}`), + (reason) => { + throw new Error(reason); + }, + ); +} + +/** + * The ambient host filesystem a runtime entrypoint installs, watched. + * + * The real provider rather than a stand-in, at the position a host installs it + * and with a working directory a workflow run must never resolve against. What + * a test says afterwards is whether a document reached it at all. + */ +export function* useHostSpy(): Operation { + const seen: HostFilesEvent[] = []; + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return "/nowhere-the-workflow-may-reach"; + }, + }, + { at: "min" }, + ); + yield* useHostFiles({ observe: (event) => seen.push(event) }); + return seen; +} + +/** + * The commits that proposed a Workspace, out of everything the owner was asked + * to commit. + * + * A run's journal lives on its owner, so every ordinary append — the root + * import, a component import, the terminal — reaches it as a commit of its own. + * What a Workspace effect adds to one is the publication, and that is what + * these tests are counting. + */ +export function published(commits: readonly Record[]): Record[] { + return commits.filter((intent) => { + const publication = intent["publication"]; + return publication !== null && publication !== undefined; + }); +} + +/** One authored document, executed as this run's root inside the attachment. */ +export function document(source: string, database: WorkflowRunDatabase): Operation { + return scoped(function* () { + return yield* collect(yield* execute({ ...inlineSource(source), stream: database.journal })); + }); +} diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 8400e3e6e..a349facc6 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -3804,7 +3804,7 @@ A runner reaches all three through one configured client, which is the supported Trusted code assembles that client into the same four-method host the local entrypoint installs. `useRunHost()` installs the executor lifecycle over the client's acquisition and returns the provider-neutral transitions; `useLifecycle()` installs status, list and history over the read plane, where the complete domain of one bound owner is zero or one coherent snapshot and no namespace-wide claim; `useDelivery()` installs typed answer delivery over the delivery plane; and `attach()` installs everything a live or partial document execution reaches. A completed replay reaches none of it: it reads the owner and opens and closes its own execution envelope, and `attach()` is never called. -**A remote attachment installs the same capabilities a local one does.** The logical Workspace working directory, the document `API.Files` provider, the Repository, Worktree and `` composition, the transactional Git components, the composition components, the pull-request and Issue middleware, elicitation and — where the host supplied one — the Agent profile. They are one set for the reason §10.1 gives: the Files provider alone would resolve an authored path against whatever working directory the surrounding host adapter answers with, so a workflow document would write to the runner's own filesystem instead of the run's Workspace. The document rules are written once and reach both hosts, and each host contributes two things to them: the effect its mutations become, and the savepoint that undoes a part of one mutation which could not be finished. Locally that effect is bound to the validated lease and the savepoint is a real transaction savepoint; on a runner the effect is bound to the exact remote run and the savepoint restores the disposable attempt from the accepted root. Which binding answers is decided by the exact storage handle the attachment registered — not by a context value, a run id or anything else a caller could hold — so a handle another client opened finds that client's binding or none, and a document with no attachment is refused rather than performing its write somewhere. +**A remote attachment installs the same capabilities a local one does.** The logical Workspace working directory, the document `API.Files` provider, the Repository, Worktree and `` composition, the transactional Git components, the composition components, the pull-request and Issue middleware, elicitation and — where the host supplied one — the Agent profile. They are one set for the reason §10.1 gives: the Files provider alone would resolve an authored path against whatever working directory the surrounding host adapter answers with, so a workflow document would write to the runner's own filesystem instead of the run's Workspace. The document rules are written once and reach both hosts, and each host contributes four things to them: the effect its mutations become, the savepoint that undoes a part of one mutation which could not be finished, the read an ephemeral attachment needs, and the transaction an Agent-session mapping is retained by. Locally the effect is bound to the validated lease, the savepoint is a real transaction savepoint, and both reads are that lease's own transaction. On a runner the effect is bound to the exact remote run; the savepoint restores the disposable attempt from the accepted root; the attachment read takes one coherent owner snapshot, materializes the selected root into a directory this invocation owns, hands the attachment only the retained metadata and that filesystem, and closes before native Git runs — so no owner read and no transaction is held across a subprocess; and an Agent-session mapping is staged from the admitted state on the runner, where the provider is, then submitted as bounded deltas to one owner transaction that revalidates each subject and commits them all or refuses them together. No provider call happens inside that transaction, a body that failed or was cancelled sends nothing, and a conflicting assertion never replaces a retained one. Which binding answers is decided by the exact storage handle the attachment registered — not by a context value, a run id or anything else a caller could hold — so a handle another client opened finds that client's binding or none, and a document with no attachment is refused rather than performing its write somewhere. The host-owned inputs those capabilities need — the credential helper, the Issue and pull-request configuration, the Agent profile installer — are explicit configuration to whoever assembles the runner. Nothing about them is read from a flag, an environment variable, a document prop or a global, and an absent one installs that capability's unconfigured behavior rather than a different capability. From 0e7ff6d4aa17c62c3377631ae4242a6c333b4180 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 15:01:20 -0300 Subject: [PATCH 74/81] =?UTF-8?q?=E2=9C=85=20Prove=20a=20Repository,=20a?= =?UTF-8?q?=20Git=20mutation=20and=20an=20Agent=20session=20on=20a=20remot?= =?UTF-8?q?e=20run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both capabilities were argued from below rather than executed. The Repository evidence preloaded a record and called the attachment read directly; the Agent evidence called the session transaction directly, against an owner whose later snapshots never included the mapping it had just accepted. Either one could pass while the composed path failed, which is exactly what happened twice before. So the scripted owner now adopts what it performs. A commit that is performed moves the pointer to the root it published, keeps the bytes it was staged, and merges the mappings it validated — so a later coherent snapshot answers with the run as it became, and a reattachment or a continuation reads what was actually retained. On that owner, one authored document through the configured public host clones a real repository on the runner, retains the checkout and its immutable identity in one owner transaction, reattaches it, and switches its branch in a second transaction that starts from the root the first one published. The remote is then deleted and the same document runs again: it reconstructs the checkout from the committed frontier, retains no second Repository, and reads the branch's own file back. The retained record names the checkout by its logical Workspace path and the remote by a fingerprint — no locator, no host path, no transport detail. The Agent path is driven through `capabilities.agent` on the configured host, with the shipped policy and the shipped transaction and only the provider standing in. It establishes, commits and then prompts, in that order; a later attachment reattaches the exact retained assertion and creates nothing; a failure before the commit retains no mapping; and a different conversation under the same identity is refused before any replacement. The contract inventory says which providers each capability has now, without implying that anything selects the remote one. --- .../cli/tests/remote-workflow-host.test.ts | 347 ++++++++++++++++-- .../tests/support/remote-owner-script.ts | 89 ++++- specs/workflow-workspace-spec.md | 6 +- 3 files changed, 402 insertions(+), 40 deletions(-) diff --git a/packages/cli/tests/remote-workflow-host.test.ts b/packages/cli/tests/remote-workflow-host.test.ts index ef1197ef2..5eb0fa245 100644 --- a/packages/cli/tests/remote-workflow-host.test.ts +++ b/packages/cli/tests/remote-workflow-host.test.ts @@ -41,8 +41,24 @@ import { startingTree, useHostSpy, } from "../../workflow/tests/support/remote-owner-script.ts"; +import { useBareRemote } from "../../workflow/tests/support/git-remotes.ts"; +import { + agentSessionKey, + resolveAgentSession, + transactAgentSessions, +} from "@executablemd/workflow/deno"; +import type { ProviderAssertion } from "@executablemd/workflow/deno"; const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; + +/** The session this run's Agent profile is about. */ +const IDENTITY = { + provider: "acpx", + agentCommand: "/usr/bin/claude", + sessionIdentity: "expansion-1", +}; +const POLICY = "policy-1"; +const ASSERTED: ProviderAssertion = { kind: "acp", value: "conversation-1" }; const OTHER_RUN = "4bxsfqu1yxtsmfg6aaccq1sxf1a4z456bf614gt4d6t31nqdqwzz"; const ENDPOINT = "https://owner.example/workflow"; const RELEASE = "factory-2026.09.10-abcdef"; @@ -149,6 +165,40 @@ function refuse(): never { throw new Error("PLANTED-FOREIGN-DATABASE-USED"); } +/** The configured public host, over a scripted owner's socket. */ +function hostFor( + owner: { readonly socket: OwnerSocket }, + capabilities?: NonNullable, +): Operation { + return useRemoteWorkflowHost({ + ...(capabilities === undefined ? {} : { capabilities }), + runId: RUN_ID, + endpoint: ENDPOINT, + release: RELEASE, + // deno-lint-ignore require-yield + *token(): Operation { + return "token-1"; + }, + scratchRoot: "/tmp/xmd-remote-public-host", + transport: { + // deno-lint-ignore require-yield + *request(): Operation { + throw new Error("PLANTED-REQUEST-PLANE-REACHED"); + }, + connect(): Operation { + return resource(function* (provide) { + yield* provide(owner.socket); + }); + }, + }, + }); +} + +/** One authored document, executed as this run's root. */ +function documentOf(source: string, database: WorkflowRunDatabase): Operation { + return document(source, database); +} + /** A storage handle nothing opened: shaped like one, and one nothing may use. */ function foreignDatabase(): WorkflowRunDatabase { return { @@ -352,27 +402,7 @@ describe("the configured remote workflow host", () => { // The configured public host, over a transport whose socket is that // scripted owner. Everything between the two is production code: the // client, its three planes, the runner and the attachment. - const built = yield* useRemoteWorkflowHost({ - runId: RUN_ID, - endpoint: ENDPOINT, - release: RELEASE, - // deno-lint-ignore require-yield - *token(): Operation { - return "token-1"; - }, - scratchRoot: "/tmp/xmd-remote-public-host", - transport: { - // deno-lint-ignore require-yield - *request(): Operation { - throw new Error("PLANTED-REQUEST-PLANE-REACHED"); - }, - connect(): Operation { - return resource(function* (provide) { - yield* provide(owner.socket); - }); - }, - }, - }); + const built = yield* hostFor(owner); const transitions = yield* built.useRunHost(); const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); if (!taken.ok || taken.value.kind !== "acquired") { @@ -412,6 +442,281 @@ describe("the configured remote workflow host", () => { expect(JSON.stringify(proposals[0]?.["publication"])).toContain("/NOTES.md"); }); + it("clones, retains and reattaches a Repository, then publishes a Git mutation", function* () { + const outcome = yield* scoped(function* () { + const remote = yield* useBareRemote({ + commits: [ + { + message: "first", + entries: [ + { path: "which.txt", content: "main\n" }, + { path: "nested/note.md", content: "note\n" }, + ], + }, + { + message: "release", + branch: "release", + entries: [{ path: "which.txt", content: "release\n" }], + }, + ], + }); + const captured = yield* startingTree(); + const owner = scriptedOwner(captured); + const source = [ + "# Remote", + "", + ``, + '', + '', + "", + "switched to: {which}", + "", + ].join("\n"); + + /** One document execution through the configured public host. */ + function* runThrough(authored: string): Operation<{ output: string; ambient: unknown[] }> { + return yield* scoped(function* () { + const built = yield* hostFor(owner); + const transitions = yield* built.useRunHost(); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + if (!taken.ok || taken.value.kind !== "acquired") { + throw new Error("expected the configured host to take the acquisition"); + } + const begun = yield* transitions.begin(taken.value.lock, { + runId: RUN_ID, + action: "resume", + }); + if (!begun.ok) { + throw begun.error; + } + const ambient = yield* useHostSpy(); + const rendered = yield* built.attach( + begun.value.database, + documentOf(authored, begun.value.database), + ); + return { output: String(rendered), ambient }; + }); + } + + const first = yield* runThrough(source); + const afterFirst = owner.commits.length; + + // The remote is gone before the continuation runs. A replay that cloned + // again would have nowhere to clone from, which is the point. + yield* remote.remove(); + const again = yield* runThrough(source); + const continuation = owner.commits.slice(afterFirst); + + return { first, again, continuation, owner }; + }); + + // The document cloned on the runner, switched the checkout and read the + // branch's own file back — all against runner-owned materialization. + expect(outcome.first.output).toContain("switched to: release"); + expect(outcome.first.ambient).toEqual([]); + // The owner retained the Repository identity with the root that holds its + // checkout: one transaction carrying the mapping and the publication. + const proposals = published(outcome.owner.commits); + const retaining = proposals.filter((intent) => { + const mappings = intent["mappings"]; + return ( + Array.isArray(mappings) && mappings.some((m) => Reflect.get(m, "kind") === "repository") + ); + }); + expect(retaining).toHaveLength(1); + expect(JSON.stringify(retaining[0]?.["publication"])).toContain("/project"); + // The retained *record* names the checkout by its logical Workspace path + // and the remote by a fingerprint. No locator and no host path is in it: + // the locator travels beside the record, which is where a reattachment + // reads it from and where it is not part of retained identity. + const retainedMappings = retaining[0]?.["mappings"]; + const proposed = Array.isArray(retainedMappings) ? retainedMappings[0] : undefined; + const record = JSON.stringify(Reflect.get(proposed ?? {}, "record")); + expect(record).toContain("locatorFingerprint"); + expect(record).toContain('"checkoutPath":"/repositories/'); + expect(record).not.toContain("/tmp"); + expect(record).not.toContain("/var/folders"); + expect(record).not.toContain("xmd-remote-"); + expect(record).not.toContain('locator"'); + // The Git mutation is its own owner transaction, and it starts from the + // root the Repository creation published: one atomic step after another, + // never one proposal carrying both. + const gitProposal = proposals.find((intent) => { + const held = intent["mappings"]; + return Array.isArray(held) && !held.some((m) => Reflect.get(m, "kind") === "repository"); + }); + expect(gitProposal).not.toBe(undefined); + expect(gitProposal?.["expectedWorkspaceRootId"]).toBe( + Reflect.get(retaining[0]?.["publication"] ?? {}, "proposedWorkspaceRootId"), + ); + expect(Array.isArray(gitProposal?.["events"]) && gitProposal?.["events"]).toHaveLength(1); + + // The continuation reconstructed the retained checkout from the owner's + // committed frontier, with no remote left to clone from. + expect(outcome.again.output).toContain("switched to: release"); + expect(outcome.again.ambient).toEqual([]); + // And it retained no second Repository: the recorded creation restored + // rather than cloning again. + const recreated = outcome.continuation.filter((intent) => { + const mappings = intent["mappings"]; + return ( + Array.isArray(mappings) && mappings.some((m) => Reflect.get(m, "kind") === "repository") + ); + }); + expect(recreated).toEqual([]); + }); + + it("installs a configured Agent profile, retains its mapping and reattaches it", function* () { + /** + * The profile this host configures, driving the shipped session policy. + * + * The provider itself stands in — establishing a conversation needs an + * agent process, and what is under test is which durable identity this run + * accepts. Everything around it is production code: the same + * `resolveAgentSession` the shipped profile calls, inside the same + * `transactAgentSessions` it commits through, reached through the + * configured host's own `capabilities.agent`. + */ + function profile( + log: string[], + asserted: ProviderAssertion, + failBeforeCommit = false, + ): (attachment: { readonly database: WorkflowRunDatabase }) => Operation { + return ({ database }) => + (function* (): Operation { + log.push("installed"); + const key = agentSessionKey(IDENTITY); + const committed = yield* transactAgentSessions(database, function* (sessions) { + const retained = sessions.read(key); + // The provider is asked here, outside the owner's transaction — + // this stands in for that — and the policy decides what the run + // accepts. + log.push(retained === undefined ? "provider:create" : "provider:assert"); + const resolution = resolveAgentSession(retained, POLICY, [asserted], IDENTITY); + if (failBeforeCommit) { + throw new Error("PlantedProfileFailure"); + } + if (retained === undefined && resolution.kind === "reattach") { + sessions.commit(resolution.record); + log.push("committed"); + } else { + log.push("reattached"); + } + }); + if (!committed.ok) { + throw committed.error; + } + // Only after the mapping is the run's does the profile speak to the + // conversation at all. + log.push("prompt"); + })(); + } + + const outcome = yield* scoped(function* () { + const captured = yield* startingTree(); + const owner = scriptedOwner(captured); + + /** One attachment with this profile configured, reporting what it did. */ + function* attaching( + log: string[], + asserted: ProviderAssertion, + failBeforeCommit = false, + ): Operation { + return yield* scoped(function* () { + const built = yield* hostFor(owner, { + agent: profile(log, asserted, failBeforeCommit), + }); + const transitions = yield* built.useRunHost(); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + if (!taken.ok || taken.value.kind !== "acquired") { + throw new Error("expected the configured host to take the acquisition"); + } + const begun = yield* transitions.begin(taken.value.lock, { + runId: RUN_ID, + action: "resume", + }); + if (!begun.ok) { + throw begun.error; + } + try { + yield* built.attach( + begun.value.database, + documentOf("# Remote\n", begun.value.database), + ); + return "attached"; + } catch (error) { + return error instanceof Error ? `raised:${error.message}` : "raised:other"; + } + }); + } + + const mapped = (): number => + owner.commits.filter((intent) => { + const mappings = intent["mappings"]; + return ( + Array.isArray(mappings) && + mappings.some((mapping) => Reflect.get(mapping, "kind") === "agent-session") + ); + }).length; + + // A profile that fails before its mapping commits. + const failing: string[] = []; + const refused = yield* attaching(failing, ASSERTED, true); + const afterFailure = mapped(); + + // Then one that establishes and commits. + const first: string[] = []; + const created = yield* attaching(first, ASSERTED); + const afterCreate = mapped(); + + // A later attachment, from what the owner now retains. + const second: string[] = []; + const again = yield* attaching(second, ASSERTED); + const afterReattach = mapped(); + + // And one whose provider asserts a different conversation. + const conflicting: string[] = []; + const replaced = yield* attaching(conflicting, { + kind: ASSERTED.kind, + value: "another-conversation", + }); + + return { + refused, + failing, + afterFailure, + created, + first, + afterCreate, + again, + second, + afterReattach, + replaced, + conflicting, + }; + }); + + // The installer ran inside the attachment, and a failure before the commit + // retained nothing. + expect(outcome.failing).toEqual(["installed", "provider:create"]); + expect(outcome.refused).toContain("raised:"); + expect(outcome.afterFailure).toBe(0); + // The next attachment establishes it: policy, then commit, then the first + // prompt — in that order, and one mapping at the owner. + expect(outcome.created).toBe("attached"); + expect(outcome.first).toEqual(["installed", "provider:create", "committed", "prompt"]); + expect(outcome.afterCreate).toBe(1); + // A later attachment reattaches the exact retained assertion, and neither + // creates a session nor retains a second mapping. + expect(outcome.again).toBe("attached"); + expect(outcome.second).toEqual(["installed", "provider:assert", "reattached", "prompt"]); + expect(outcome.afterReattach).toBe(1); + // A different conversation under the same identity is refused before any + // replacement, and still nothing more is retained. + expect(outcome.replaced).toContain("raised:"); + expect(outcome.conflicting).toEqual(["installed", "provider:assert"]); + }); + it("attaches nothing it did not open", function* () { const owner = scripted(); const refused = yield* scoped(function* () { diff --git a/packages/workflow/tests/support/remote-owner-script.ts b/packages/workflow/tests/support/remote-owner-script.ts index 8714c64f4..549e54fa9 100644 --- a/packages/workflow/tests/support/remote-owner-script.ts +++ b/packages/workflow/tests/support/remote-owner-script.ts @@ -13,7 +13,7 @@ import { collect, execute, inlineSource } from "@executablemd/core"; import { API, useHostFiles } from "@executablemd/runtime"; import type { HostFilesEvent } from "@executablemd/runtime"; import type { Json } from "@executablemd/durable-streams"; -import { encodeBase64 } from "../../src/cloudflare/encoding.ts"; +import { decodeBase64, encodeBase64 } from "../../src/cloudflare/encoding.ts"; import type { OwnerSocket, SocketListener } from "../../src/remote/client.ts"; import { captureWorkspace, type CapturedWorkspace } from "../../src/remote/materialize.ts"; import { runnerFiles, useRunnerTrees } from "../../src/deno/remote-files.ts"; @@ -42,8 +42,31 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet const sent: Record[] = []; const commits: Record[] = []; let currentRoot = captured.root.rootId; + let currentManifest = captured.root.manifest; let refusal: string | undefined; let lost = false; + // What this owner has accepted, as it would then hold it. A commit that is + // performed moves the root, keeps the bytes it was staged, and merges the + // mappings it validated — so a later coherent snapshot answers with what the + // run actually became rather than with what it started as. + const blobs = new Map(captured.blobs); + const manifests = new Map(); + for (const [digest, content] of captured.contents) { + manifests.set(digest, content.manifestBytes); + } + const staged = new Map(); + const repositories = new Map>(); + const worktrees = new Map>(); + const sessions = new Map>(); + for (const stored of retained.repositories ?? []) { + const record = stored["record"]; + if (record !== null && typeof record === "object") { + repositories.set(String(Reflect.get(record, "name")), stored); + } + } + for (const record of retained.agentSessions ?? []) { + sessions.set(String(record["sessionKey"]), record); + } function frontier(): Record { return { @@ -101,24 +124,21 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet value: { workspaceRootId: currentRoot, journalEventId: null, - repositories: retained.repositories ?? [], - worktrees: [], - agentSessions: retained.agentSessions ?? [], + repositories: [...repositories.values()], + worktrees: [...worktrees.values()], + agentSessions: [...sessions.values()], }, }; } if (command === "root") { return { outcome: "performed", - value: { workspaceRootId: currentRoot, manifest: captured.root.manifest }, + value: { workspaceRootId: currentRoot, manifest: currentManifest }, }; } if (command === "content") { const digest = String(request["digest"]); - const bytes = - request["kind"] === "manifest" - ? captured.contents.get(digest)?.manifestBytes - : captured.blobs.get(digest); + const bytes = request["kind"] === "manifest" ? manifests.get(digest) : blobs.get(digest); if (bytes === undefined) { throw new Error("asked for content this owner does not hold"); } @@ -134,14 +154,13 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet } if (command === "stage") { const encoded = String(request["bytes"] ?? ""); - const padding = encoded.endsWith("==") ? 2 : encoded.endsWith("=") ? 1 : 0; + const bytes = decodeBase64(encoded); + // Held until a commit adopts them, exactly as staged bytes are: a + // proposal that is refused leaves nothing behind. + staged.set(`${String(request["kind"])}:${String(request["digest"])}`, bytes); return { outcome: "performed", - value: { - kind: request["kind"], - digest: request["digest"], - size: (encoded.length / 4) * 3 - padding, - }, + value: { kind: request["kind"], digest: request["digest"], size: bytes.length }, }; } if (command === "settle") { @@ -160,10 +179,48 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet return { outcome: "refused", refusal }; } const events = Array.isArray(request["events"]) ? request["events"] : []; - // The owner publishes what it validated, and the frontier moves with it. + // The owner publishes what it validated, and everything moves with it: the + // pointer, the content it was staged, and the mappings it accepted. A + // later snapshot then answers with the run as it now is. if (publication !== null && publication !== undefined) { currentRoot = String(Reflect.get(publication, "proposedWorkspaceRootId")); + currentManifest = String(Reflect.get(publication, "proposedManifest")); + const held = Reflect.get(publication, "content"); + for (const piece of Array.isArray(held) ? held : []) { + const kind = String(Reflect.get(piece, "kind")); + const digest = String(Reflect.get(piece, "digest")); + const bytes = staged.get(`${kind}:${digest}`); + if (bytes === undefined) { + throw new Error(`the runner published ${kind} ${digest} without staging it`); + } + (kind === "manifest" ? manifests : blobs).set(digest, bytes); + } + } + for (const mapping of Array.isArray(request["mappings"]) ? request["mappings"] : []) { + const kind = String(Reflect.get(mapping, "kind")); + const record = Reflect.get(mapping, "record"); + if (record === null || typeof record !== "object") { + throw new Error("the runner proposed a mapping with no record"); + } + if (kind === "repository") { + // Retained the way a snapshot reports one: the record, and the locator + // beside it, which the owner keeps out of the record itself. + repositories.set(String(Reflect.get(record, "name")), { + record, + locator: Reflect.get(mapping, "locator") ?? null, + }); + } + if (kind === "worktree") { + worktrees.set( + `${String(Reflect.get(record, "repositoryName"))}/${String(Reflect.get(record, "name"))}`, + record as Record, + ); + } + if (kind === "agent-session") { + sessions.set(String(Reflect.get(record, "sessionKey")), record as Record); + } } + staged.clear(); return { outcome: "performed", value: { diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index a349facc6..5b54ecaf6 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -3825,8 +3825,8 @@ A completed run replays there as it does locally: it attaches no Workspace, Agen | caller-owned storage transaction | built by #291; Workspace mutations join it in #365 | | provider-backed retained Workspace | document filesystem built by #366 and repository composition by #293; document deletion (§10.1) built by #567 for both providers; process capabilities unbuilt (#218) | | `xmd workflow start` / `resume` | built by #366, Deno entrypoints only; both acquire #367's executor lock. The same two lifecycle actions are implemented against the durable owner, where the acquisition is the connection rather than the lock, and trusted code assembles that host explicitly (§13.2); no selector chooses it, so the shipped entrypoints are local | -| ``, `` and `` composition | built by #293, Deno provider only | -| transactional Git components (`Git.Switch`, `Git.Add`, `Git.Commit`) | built by #294, Deno provider only | +| ``, `` and `` composition | built by #293; both owners by #698 — one set of rules over whichever host the exact storage handle is bound to. Native Git runs on the runner against invocation-owned materialization in either case; what differs is where the checkout's bytes and its immutable identity are retained, and which read an ephemeral reattachment goes through. No selector chooses the remote owner (§13.2) | +| transactional Git components (`Git.Switch`, `Git.Add`, `Git.Commit`) | built by #294; both owners by #698 — the mutation runs on the runner and its resulting root and filtered journal result are published in one transaction by whichever owner holds the run. No selector chooses the remote one (§13.2) | | `` read and upsert, and the `issue_effect` boundary (§10.3) | built by #296; GitHub middleware, Deno host | | ``, ``, `` (§7.7) | built by #576; GitHub middleware, Deno host. Named by canonical URL and asked of `PullRequestApi`, which carries the upsert too; ordinary durable reads rather than reconciled effects, inheritable by a fork; complete or unavailable, never truncated. Which URLs may be read is operator configuration | | lifecycle status/list/history | built by #367; the durable owner answers the same three from one committed reading, taking no acquisition, attaching nothing and appending nothing | @@ -3838,7 +3838,7 @@ A completed run replays there as it does locally: it attaches no Workspace, Agen | XMD artifact export, inspection and fork source | specified in `specs/xmd-artifact-spec.md`; `xmd workflow export` and artifact `status`/`history` are built, Deno provider only. The artifact-backed fork remains unbuilt | | Agent session portability evidence in an artifact | specified in `specs/xmd-artifact-spec.md` §2.5; the format and its complete verifier are built. Provider bundle capture, Agent-aware export, intrinsic Agent-aware inspection and artifact-backed fork are unbuilt | | workflow Agent isolation | built by #302: no directory attachment, an empty host-owned working directory, no MCP servers, an empty requested tool set and deny-all with a failing permission path; the portable no-tool proof is tracked by #496 | -| workflow Agent session retention | built by #302: a row in the run's own database, keyed by the engine-derived Session expansion identity alone — the authored name is descriptive — with provider, agent command and policy fingerprint beside it as compatibility attributes. The mapping commits after the provider's canonical tagged assertion and before the first Prompt; occupancy of a provider key is never identity, and missing, mismatched, replaced or ambiguous assertions each refuse instead of starting a replacement session | +| workflow Agent session retention | built by #302, and by #698 for a run whose storage is elsewhere: the provider is established or asserted on the runner, outside any transaction, and the mapping is submitted as bounded deltas to one owner transaction that revalidates the subject and commits or refuses it whole. a row in the run's own database, keyed by the engine-derived Session expansion identity alone — the authored name is descriptive — with provider, agent command and policy fingerprint beside it as compatibility attributes. The mapping commits after the provider's canonical tagged assertion and before the first Prompt; occupancy of a provider key is never identity, and missing, mismatched, replaced or ambiguous assertions each refuse instead of starting a replacement session | | generated-XMD admission | built by #369, through `@executablemd/core/host`; the workflow policy wrapper is internal. Host policy is a read table and a write table of exact pinned identities, each entry carrying the authored forms it is admitted for, and an authored `allow` selects a canonical subset of the closed classes `read` and `write` — omitted means `read`. The complete fragment is preflighted inside one `generated_xmd` effect before its first generated effect | | `` and the authored loop | built by #302 and #369: a workflow-host component with a closed schema of one required `source` and one optional `allow`, declared to the execution rather than registered by the attachment — canonical execution calls the host's factory with the claimant it minted and registers what comes back, which provides availability only. Its ceilings come from the run's own storage, core's pinned `` read and write identities and this package's lexical ``; iteration, branching, approval and exhaustion are ordinary Markdown | | generated-XMD mutation-proposal admission | built by #369 and #567: the standard Deno profile's write table is core's paired `File:write`, this package's lexical `Dir` and core's self-closing `File.Delete`, in that retained order and followed by any host extension; admitted mutations run as the ordinary components they are through the run's effect transactions, a generated deletion publishing the same `workspace_file` effect an authored one does; the evaluator adds no receipt or result entry, so a write-only fragment still binds `{ observations: [], output: "" }`; and approval is authored control flow before the element. Local Git, Git-host, issue, process, execution, credential and external-write effects are outside the class | From 2f5352237c3da18330076f320b7d48187c30ab53 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 16:10:35 -0300 Subject: [PATCH 75/81] =?UTF-8?q?=E2=9C=85=20Prove=20the=20remote=20contin?= =?UTF-8?q?uation=20and=20Agent=20profile=20the=20tests=20claimed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two D5 regressions asserted structure they did not reach. The scripted owner discarded every event it accepted, so a mapping and a root sat beside an empty journal — a state no owner can produce from a successful effect. It now retains the ordered journal: each entry with the id the owner minted for it and the Workspace root the transaction that appended it selected, appended in the same step that moves the root and merges the mappings. `frontier` and the mappings snapshot name the terminal event, and a `journal` read answers one anchored page at a time from exactly where the client says it is. So the Repository regression can be about continuation. The first execution clones, retains the creation whole, and is cancelled with the Git mutation's proposal still in flight: the owner decided the creation and never decided the mutation, which is a prefix an owner can be holding. The continuation reads that prefix back in anchored pages, restores the recorded creation with no remote left to clone from, rebuilds the checkout from the root the replayed record selected — the live switch starts from that root and moves a checkout really on `main` — and performs only the work the cancellation left undone. The Agent regression installed its own policy. It now installs the shipped `useWorkflowAgentProfile()` through the public configuration and runs an authored Agent/Session/Prompt document, with only the agent process and its own session store substituted. Sampled at the owner, the order is the contract: the conversation exists, the owner then accepts which one it is, and only then does anything prompt it. A restart reattaches that exact provider-native session and prompts only the unfinished turn; a completed document restores without a provider at all; an establishment that fails and one cancelled in flight propose no mapping; and a provider asserting a different conversation is refused before a replacement session or a prompt. --- .../cli/tests/remote-workflow-host.test.ts | 797 +++++++++++++----- .../tests/support/remote-owner-script.ts | 64 +- 2 files changed, 663 insertions(+), 198 deletions(-) diff --git a/packages/cli/tests/remote-workflow-host.test.ts b/packages/cli/tests/remote-workflow-host.test.ts index 5eb0fa245..13cfb4af4 100644 --- a/packages/cli/tests/remote-workflow-host.test.ts +++ b/packages/cli/tests/remote-workflow-host.test.ts @@ -18,7 +18,18 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { ensure, type Operation, resource, scoped } from "effection"; +import { ensure, type Operation, resource, scoped, sleep, spawn, withResolvers } from "effection"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { readTextFile } from "@effectionx/fs"; +import { agentIdentityComponents, collect, retainedSource } from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import type { + AcpRuntimeDoctorReport, + AcpRuntimeHandle, + AcpRuntimeOptions, + ProbeCapableRuntime, +} from "@executablemd/acp"; import { WorkflowInputDelivery, WorkflowLifecycle } from "@executablemd/workflow"; import type { WorkflowRunDatabase } from "@executablemd/workflow"; import type { @@ -42,23 +53,14 @@ import { useHostSpy, } from "../../workflow/tests/support/remote-owner-script.ts"; import { useBareRemote } from "../../workflow/tests/support/git-remotes.ts"; -import { - agentSessionKey, - resolveAgentSession, - transactAgentSessions, -} from "@executablemd/workflow/deno"; -import type { ProviderAssertion } from "@executablemd/workflow/deno"; +import { useWorkflowAgentProfile, workflowSessionPolicyDigest } from "../src/workflow-agent.ts"; +import type { WorkflowAgentProfileOptions } from "../src/workflow-agent.ts"; +import { createFakeAcp, makeStore, tripwireAcp } from "./support/fake-acp.ts"; +import type { FakeAcp } from "./support/fake-acp.ts"; +import { useTempDirectory } from "@executablemd/test-support/temp"; const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; -/** The session this run's Agent profile is about. */ -const IDENTITY = { - provider: "acpx", - agentCommand: "/usr/bin/claude", - sessionIdentity: "expansion-1", -}; -const POLICY = "policy-1"; -const ASSERTED: ProviderAssertion = { kind: "acp", value: "conversation-1" }; const OTHER_RUN = "4bxsfqu1yxtsmfg6aaccq1sxf1a4z456bf614gt4d6t31nqdqwzz"; const ENDPOINT = "https://owner.example/workflow"; const RELEASE = "factory-2026.09.10-abcdef"; @@ -442,7 +444,7 @@ describe("the configured remote workflow host", () => { expect(JSON.stringify(proposals[0]?.["publication"])).toContain("/NOTES.md"); }); - it("clones, retains and reattaches a Repository, then publishes a Git mutation", function* () { + it("clones and retains a Repository, then continues its Git mutation from that history", function* () { const outcome = yield* scoped(function* () { const remote = yield* useBareRemote({ commits: [ @@ -462,6 +464,11 @@ describe("the configured remote workflow host", () => { }); const captured = yield* startingTree(); const owner = scriptedOwner(captured); + // Installed around both executions, at the position a runtime entrypoint + // installs it and with a working directory a workflow run must never + // resolve against: anything either execution let fall through to the + // caller's filesystem is visible here rather than silent. + const ambient = yield* useHostSpy(); const source = [ "# Remote", "", @@ -474,9 +481,12 @@ describe("the configured remote workflow host", () => { ].join("\n"); /** One document execution through the configured public host. */ - function* runThrough(authored: string): Operation<{ output: string; ambient: unknown[] }> { + function* runThrough( + authored: string, + socket: OwnerSocket, + ): Operation<{ output: string; failure: string }> { return yield* scoped(function* () { - const built = yield* hostFor(owner); + const built = yield* hostFor({ socket }); const transitions = yield* built.useRunHost(); const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); if (!taken.ok || taken.value.kind !== "acquired") { @@ -489,42 +499,65 @@ describe("the configured remote workflow host", () => { if (!begun.ok) { throw begun.error; } - const ambient = yield* useHostSpy(); - const rendered = yield* built.attach( - begun.value.database, - documentOf(authored, begun.value.database), - ); - return { output: String(rendered), ambient }; + try { + const rendered = yield* built.attach( + begun.value.database, + documentOf(authored, begun.value.database), + ); + return { output: String(rendered), failure: "" }; + } catch (error) { + return { + output: "", + failure: error instanceof Error ? error.message : "other", + }; + } }); } - const first = yield* runThrough(source); - const afterFirst = owner.commits.length; + // The first execution clones, retains the Repository, and is cancelled + // with the Git mutation's proposal still in flight. So the owner decided + // the creation and never decided the mutation, and what it holds is the + // prefix it accepted rather than a history nobody wrote. + const withheld: Record[] = []; + const proposing = withResolvers(); + const attempt = yield* spawn(() => + runThrough( + source, + withholding(owner.socket, withheld, () => proposing.resolve()), + ), + ); + yield* proposing.operation; + yield* attempt.halt(); - // The remote is gone before the continuation runs. A replay that cloned - // again would have nowhere to clone from, which is the point. + const accepted = owner.commits.length; + const asked = owner.sent.length; + const prefix = owner.entries(); + + // The remote is gone before the continuation runs, so nothing it does + // can involve the network — and what it continues from is the journal the + // owner retained beside the root and the mapping. yield* remote.remove(); - const again = yield* runThrough(source); - const continuation = owner.commits.slice(afterFirst); + const again = yield* runThrough(source, owner.socket); - return { first, again, continuation, owner }; + return { + withheld, + again, + ambient, + retained: prefix, + creation: owner.commits.slice(0, accepted), + continuation: owner.commits.slice(accepted), + replayed: owner.sent.slice(asked), + owner, + }; }); - // The document cloned on the runner, switched the checkout and read the - // branch's own file back — all against runner-owned materialization. - expect(outcome.first.output).toContain("switched to: release"); - expect(outcome.first.ambient).toEqual([]); - // The owner retained the Repository identity with the root that holds its - // checkout: one transaction carrying the mapping and the publication. - const proposals = published(outcome.owner.commits); - const retaining = proposals.filter((intent) => { - const mappings = intent["mappings"]; - return ( - Array.isArray(mappings) && mappings.some((m) => Reflect.get(m, "kind") === "repository") - ); - }); + // One proposal reached the owner, carrying the Repository mapping and the + // root that holds its checkout; the mutation's proposal reached it never. + const retaining = published(outcome.creation); expect(retaining).toHaveLength(1); expect(JSON.stringify(retaining[0]?.["publication"])).toContain("/project"); + expect(outcome.withheld).toHaveLength(1); + expect(only(outcome.withheld[0])).toContain('"type":"workspace_git_switch"'); // The retained *record* names the checkout by its logical Workspace path // and the remote by a fingerprint. No locator and no host path is in it: // the locator travels beside the record, which is where a reattachment @@ -538,183 +571,296 @@ describe("the configured remote workflow host", () => { expect(record).not.toContain("/var/folders"); expect(record).not.toContain("xmd-remote-"); expect(record).not.toContain('locator"'); - // The Git mutation is its own owner transaction, and it starts from the - // root the Repository creation published: one atomic step after another, - // never one proposal carrying both. - const gitProposal = proposals.find((intent) => { - const held = intent["mappings"]; - return Array.isArray(held) && !held.some((m) => Reflect.get(m, "kind") === "repository"); - }); - expect(gitProposal).not.toBe(undefined); - expect(gitProposal?.["expectedWorkspaceRootId"]).toBe( - Reflect.get(retaining[0]?.["publication"] ?? {}, "proposedWorkspaceRootId"), + + // What the owner holds is one coherent prefix: the creation's own journal + // row, carrying the root that transaction published, and nothing from the + // transaction it never decided. A root and a mapping beside a journal + // missing the transaction that created them is not a state this owner can + // be in, and neither is a journal holding a transaction the owner refused + // to decide. + const creationRoot = Reflect.get( + retaining[0]?.["publication"] ?? {}, + "proposedWorkspaceRootId", ); - expect(Array.isArray(gitProposal?.["events"]) && gitProposal?.["events"]).toHaveLength(1); + const repositoryEvent = outcome.retained.find((entry) => + entry.record.includes('"type":"workspace_repository"'), + ); + expect(repositoryEvent?.workspaceRootId).toBe(creationRoot); + expect( + outcome.retained.filter((entry) => entry.record.includes('"type":"workspace_git_switch"')), + ).toEqual([]); + expect(outcome.retained.at(-1)?.workspaceRootId).toBe(creationRoot); + + // The continuation read that prefix — anchored pages, from the terminal + // event the frontier named. + const pages = outcome.replayed.filter((request) => request["command"] === "journal"); + expect(pages.length > 0).toBe(true); + expect(pages[0]?.["anchorEventId"]).toBe(outcome.retained.at(-1)?.eventId); - // The continuation reconstructed the retained checkout from the owner's - // committed frontier, with no remote left to clone from. + // The recorded creation restored rather than cloning again — the remote it + // was cloned from no longer exists — and the checkout the Git mutation + // needed was reconstructed from the root that replayed record selected. + // The live switch started from exactly that root and moved a checkout that + // really was on `main`, which is what a checkout rebuilt from the recorded + // Workspace and proved against the record looks like. + expect(outcome.again.failure).toBe(""); + const mutation = published(outcome.continuation); + const switched = mutation.find((intent) => + only(intent).includes('"type":"workspace_git_switch"'), + ); + expect(switched?.["expectedWorkspaceRootId"]).toBe(creationRoot); + expect(only(switched)).toContain('"before":{"branch":"main"'); + expect(only(switched)).toContain('"after":{"branch":"release"'); + // And then the branch's own file, read live from that same checkout. + const read = mutation.find((intent) => only(intent).includes('"type":"workspace_file"')); + expect(only(read)).toContain('"content":"release'); expect(outcome.again.output).toContain("switched to: release"); - expect(outcome.again.ambient).toEqual([]); - // And it retained no second Repository: the recorded creation restored - // rather than cloning again. - const recreated = outcome.continuation.filter((intent) => { + + // Two transactions and no third: only the work the cancellation left + // undone. The mutation's own journal row carries the root it published, + // and that root is the run's — a read moves nothing, so the switch is the + // last thing that moved it. + expect(mutation).toHaveLength(2); + const mutationRoot = Reflect.get(switched?.["publication"] ?? {}, "proposedWorkspaceRootId"); + const gitEvent = outcome.owner + .entries() + .find((entry) => entry.record.includes('"type":"workspace_git_switch"')); + expect(gitEvent?.workspaceRootId).toBe(mutationRoot); + expect(outcome.owner.currentRoot).toBe(mutationRoot); + // Nothing retained a second Repository, and the ambient host filesystem was + // asked for nothing by either execution. + const recreated = mutation.filter((intent) => { const mappings = intent["mappings"]; return ( Array.isArray(mappings) && mappings.some((m) => Reflect.get(m, "kind") === "repository") ); }); expect(recreated).toEqual([]); + expect(outcome.ambient).toEqual([]); }); - it("installs a configured Agent profile, retains its mapping and reattaches it", function* () { - /** - * The profile this host configures, driving the shipped session policy. - * - * The provider itself stands in — establishing a conversation needs an - * agent process, and what is under test is which durable identity this run - * accepts. Everything around it is production code: the same - * `resolveAgentSession` the shipped profile calls, inside the same - * `transactAgentSessions` it commits through, reached through the - * configured host's own `capabilities.agent`. - */ - function profile( - log: string[], - asserted: ProviderAssertion, - failBeforeCommit = false, - ): (attachment: { readonly database: WorkflowRunDatabase }) => Operation { - return ({ database }) => - (function* (): Operation { - log.push("installed"); - const key = agentSessionKey(IDENTITY); - const committed = yield* transactAgentSessions(database, function* (sessions) { - const retained = sessions.read(key); - // The provider is asked here, outside the owner's transaction — - // this stands in for that — and the policy decides what the run - // accepts. - log.push(retained === undefined ? "provider:create" : "provider:assert"); - const resolution = resolveAgentSession(retained, POLICY, [asserted], IDENTITY); - if (failBeforeCommit) { - throw new Error("PlantedProfileFailure"); - } - if (retained === undefined && resolution.kind === "reattach") { - sessions.commit(resolution.record); - log.push("committed"); - } else { - log.push("reattached"); - } - }); - if (!committed.ok) { - throw committed.error; - } - // Only after the mapping is the run's does the profile speak to the - // conversation at all. - log.push("prompt"); - })(); - } + it("prompts through the shipped Agent profile, and retains the conversation it got", function* () { + const root = yield* useTempDirectory("xmd-remote-agent-"); + const source = yield* readTextFile(join(FIXTURES, "claude-session.md")); const outcome = yield* scoped(function* () { const captured = yield* startingTree(); const owner = scriptedOwner(captured); + // One provider store across every attachment below: a provider keeps its + // sessions across processes, so a fresh one would be a provider that + // forgot rather than a run that came back. + const store = makeStore(); - /** One attachment with this profile configured, reporting what it did. */ - function* attaching( - log: string[], - asserted: ProviderAssertion, - failBeforeCommit = false, - ): Operation { - return yield* scoped(function* () { - const built = yield* hostFor(owner, { - agent: profile(log, asserted, failBeforeCommit), - }); - const transitions = yield* built.useRunHost(); - const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); - if (!taken.ok || taken.value.kind !== "acquired") { - throw new Error("expected the configured host to take the acquisition"); - } - const begun = yield* transitions.begin(taken.value.lock, { - runId: RUN_ID, - action: "resume", - }); - if (!begun.ok) { - throw begun.error; - } - try { - yield* built.attach( - begun.value.database, - documentOf("# Remote\n", begun.value.database), - ); - return "attached"; - } catch (error) { - return error instanceof Error ? `raised:${error.message}` : "raised:other"; - } - }); - } + // The first attachment is cancelled while the second prompt's turn is in + // flight. The session has been established and its mapping has committed + // by then, so what the cancellation leaves unfinished is the turn rather + // than the retention — which is what gives the restart below something + // to resolve from. + const live = createFakeAcp(); + live.script({ reply: "the reviewer saw the release notes" }); + live.script({ reply: "", manual: true }); + const marks: Mark[] = []; + const interrupted = yield* spawn(() => + attaching(sampled(owner.socket, live, marks), root, source, { + createRuntime: live.create, + sessionStore: store, + }), + ); + yield* live.startedTurns(2); + yield* interrupted.halt(); + const created = retained(owner); + const asserted = storeAssertions(store); - const mapped = (): number => - owner.commits.filter((intent) => { - const mappings = intent["mappings"]; - return ( - Array.isArray(mappings) && - mappings.some((mapping) => Reflect.get(mapping, "kind") === "agent-session") - ); - }).length; - - // A profile that fails before its mapping commits. - const failing: string[] = []; - const refused = yield* attaching(failing, ASSERTED, true); - const afterFailure = mapped(); - - // Then one that establishes and commits. - const first: string[] = []; - const created = yield* attaching(first, ASSERTED); - const afterCreate = mapped(); - - // A later attachment, from what the owner now retains. - const second: string[] = []; - const again = yield* attaching(second, ASSERTED); - const afterReattach = mapped(); - - // And one whose provider asserts a different conversation. - const conflicting: string[] = []; - const replaced = yield* attaching(conflicting, { - kind: ASSERTED.kind, - value: "another-conversation", + // The restart: the same run, the same provider store, and a provider that + // answers the turn the first attempt never finished. + const resumed = createFakeAcp(); + resumed.script({ reply: "and they recommended shipping it" }); + const again = yield* attaching(owner.socket, root, source, { + createRuntime: resumed.create, + sessionStore: store, + }); + + // And once the document has finished, a further attachment restores it + // from what the owner retains and reaches no provider at all — not even + // to create a runtime. + const reached: string[] = []; + const replayed = yield* attaching(owner.socket, root, source, { + createRuntime: tripwireAcp((what) => reached.push(what)), + sessionStore: store, }); return { - refused, - failing, - afterFailure, created, - first, - afterCreate, + asserted, + marks, + establishedFirst: established(live), + promptedFirst: [...live.prompts], again, - second, - afterReattach, - replaced, - conflicting, + establishedAgain: established(resumed), + promptedAgain: [...resumed.prompts], + reattached: retained(owner), + held: storeAssertions(store), + replayed, + reached, }; }); - // The installer ran inside the attachment, and a failure before the commit - // retained nothing. - expect(outcome.failing).toEqual(["installed", "provider:create"]); - expect(outcome.refused).toContain("raised:"); - expect(outcome.afterFailure).toBe(0); - // The next attachment establishes it: policy, then commit, then the first - // prompt — in that order, and one mapping at the owner. - expect(outcome.created).toBe("attached"); - expect(outcome.first).toEqual(["installed", "provider:create", "committed", "prompt"]); - expect(outcome.afterCreate).toBe(1); - // A later attachment reattaches the exact retained assertion, and neither - // creates a session nor retains a second mapping. + // One session, established on the runner, and one mapping at the owner + // carrying exactly what the provider asserted about it — under the shipped + // session policy rather than a digest this test invented. + expect(outcome.establishedFirst).toHaveLength(1); + expect(outcome.created).toHaveLength(1); + expect(member(outcome.created[0], "provider")).toBe("acpx"); + expect(member(outcome.created[0], "policy")).toBe(workflowSessionPolicyDigest()); + expect(member(member(outcome.created[0], "assertion"), "kind")).toBe("acpx.agentSessionId"); + expect([String(member(member(outcome.created[0], "assertion"), "value"))]).toEqual( + outcome.asserted, + ); + // The order is the whole of it, sampled at the owner: the conversation + // existed, the owner then accepted which one it was, and only then did + // anything prompt it. + expect(outcome.marks).toEqual([{ ensured: 1, prompts: 0 }]); + expect(outcome.promptedFirst).toHaveLength(2); + expect(outcome.promptedFirst[0]).toContain("What did the reviewer see?"); + + // The restart reattaches the exact conversation: the same placement, the + // same provider-native identity, no second session and no second mapping. expect(outcome.again).toBe("attached"); - expect(outcome.second).toEqual(["installed", "provider:assert", "reattached", "prompt"]); - expect(outcome.afterReattach).toBe(1); - // A different conversation under the same identity is refused before any - // replacement, and still nothing more is retained. - expect(outcome.replaced).toContain("raised:"); - expect(outcome.conflicting).toEqual(["installed", "provider:assert"]); + expect(outcome.establishedAgain).toEqual(outcome.establishedFirst); + expect(outcome.held).toEqual(outcome.asserted); + expect(outcome.reattached).toEqual(outcome.created); + // And it prompted only the work the cancellation left unfinished. + expect(outcome.promptedAgain).toHaveLength(1); + expect(outcome.promptedAgain[0]).toContain("And what did they recommend?"); + + // A completed document restores without a provider. + expect(outcome.replayed).toBe("attached"); + expect(outcome.reached).toEqual([]); + }); + + it("proposes no mapping for a session that never became one", function* () { + const captured = yield* startingTree(); + const root = yield* useTempDirectory("xmd-remote-agent-window-"); + const source = yield* readTextFile(join(FIXTURES, "claude-session.md")); + + // A provider whose establishment fails outright. + const failed = yield* scoped(function* () { + const owner = scriptedOwner(captured); + const outcome = yield* attaching(owner.socket, root, source, { + createRuntime: establishing( + () => {}, + () => Promise.reject(new Error("PlantedEstablishFailure")), + [], + ), + sessionStore: makeStore(), + }); + return { outcome, proposed: retained(owner) }; + }); + + // And one cancelled with the establishment still in flight — a real + // Effection cancellation of the attachment, in the window between asking a + // provider for a conversation and retaining which one it is. + const cancelled = yield* scoped(function* () { + const owner = scriptedOwner(captured); + const store = makeStore(); + const asking = withResolvers(); + const closed: string[] = []; + let answer: (handle: AcpRuntimeHandle) => void = () => {}; + const attempt = yield* spawn(() => + attaching(owner.socket, root, source, { + createRuntime: establishing( + () => asking.resolve(), + () => + new Promise((resolve) => { + answer = resolve; + }), + closed, + ), + sessionStore: store, + }), + ); + yield* asking.operation; + const halting = yield* spawn(() => attempt.halt()); + // Cancellation is delivered on microtasks, so by the next macrotask the + // provider's own cleanup is what is waiting for this answer rather than + // the run. `closed` below is what confirms this stood in that window: a + // provider that answers a cancelled establishment has a live session to + // give back, and giving it back is the only thing left to do with it. + yield* sleep(0); + answer(ESTABLISHED_LATE); + yield* halting; + return { + closed, + proposed: retained(owner), + asserted: storeAssertions(store), + }; + }); + + expect(failed.outcome).toContain("raised:"); + expect(failed.proposed).toEqual([]); + // The cancellation landed where it was aimed, and what the provider + // answered afterwards was closed rather than adopted. + expect(cancelled.closed).toEqual(["cancelled before the session was established"]); + // Nothing was retained and nothing was asserted, so a later attachment + // resolves from an empty run rather than from a conversation nobody can + // name. + expect(cancelled.proposed).toEqual([]); + expect(cancelled.asserted).toEqual([]); + }); + + it("refuses a conversation the provider replaced, before prompting or retaining", function* () { + const captured = yield* startingTree(); + const root = yield* useTempDirectory("xmd-remote-agent-conflict-"); + const source = yield* readTextFile(join(FIXTURES, "claude-session.md")); + + const outcome = yield* scoped(function* () { + // One run that established a session, so what the owner below retains is + // a mapping this stack actually wrote rather than one this test composed. + const establishedRun = scriptedOwner(captured); + const store = makeStore(); + const live = createFakeAcp(); + live.script({ reply: "the reviewer saw the release notes" }); + live.script({ reply: "and they recommended shipping it" }); + const first = yield* attaching(establishedRun.socket, root, source, { + createRuntime: live.create, + sessionStore: store, + }); + const record = retained(establishedRun); + + // The same run as an owner holds it, and a provider whose store now + // asserts a different conversation under the same placement. + const owner = scriptedOwner(captured, { agentSessions: record }); + const replacement = makeStore(); + for (const [key, held] of store.records) { + replacement.records.set(key, { + ...held, + agentSessionId: "another-conversation", + }); + } + const provider = createFakeAcp(); + const refused = yield* attaching(owner.socket, root, source, { + createRuntime: provider.create, + sessionStore: replacement, + }); + return { + first, + record, + refused, + ensured: provider.ensured.length, + prompts: provider.prompts.length, + proposed: retained(owner), + }; + }); + + expect(outcome.first).toBe("attached"); + expect(outcome.record).toHaveLength(1); + // Refused where the decision belongs: before a replacement session is + // established and before anything is prompted. The owner was asked to + // retain nothing, so what it holds is still the conversation this run had. + expect(outcome.refused).toContain("different durable identity"); + expect(outcome.ensured).toBe(0); + expect(outcome.prompts).toBe(0); + expect(outcome.proposed).toEqual([]); }); it("attaches nothing it did not open", function* () { @@ -741,3 +887,264 @@ describe("the configured remote workflow host", () => { function* never(): Operation { throw new Error("PLANTED-ATTACHED-OPERATION-RAN"); } + +/** Where the workflow Agent documents this suite drives live. */ +const FIXTURES = join(dirname(fileURLToPath(import.meta.url)), "fixtures", "workflow-agent"); + +/** + * The journal records one proposal carries, as the owner receives them. + * + * Read out of the intent rather than re-encoded, because a record is a string + * on the wire and searching its JSON encoding would be searching the escaping. + */ +function only(intent: Record | undefined): string { + const events = intent?.["events"]; + return (Array.isArray(events) ? events : []).map((event) => String(event)).join(""); +} + +/** One member of a value nothing has checked. */ +function member(value: unknown, name: string): unknown { + return value !== null && typeof value === "object" ? Reflect.get(value, name) : undefined; +} + +/** + * The Agent-session mapping records this owner was asked to retain, in order. + * + * Read back out of the intents it received, so what is counted is what crossed + * rather than what this process believes it staged. + */ +function retained(owner: { + readonly commits: readonly Record[]; +}): Record[] { + return owner.commits.flatMap((intent) => { + const proposed = intent["mappings"]; + return (Array.isArray(proposed) ? proposed : []) + .filter((mapping) => member(mapping, "kind") === "agent-session") + .map((mapping) => JSON.parse(JSON.stringify(member(mapping, "record")))); + }); +} + +/** The distinct sessions this provider was asked to establish. */ +function established(fake: FakeAcp): string[] { + return [...new Set(fake.ensured.map((input) => input.sessionKey))].sort(); +} + +/** Every provider-native identity the substituted store currently holds. */ +function storeAssertions(store: ReturnType): string[] { + return [...store.records.values()] + .flatMap((record) => (record.agentSessionId === undefined ? [] : [record.agentSessionId])) + .sort(); +} + +/** What the provider had done by the time the owner accepted a mapping. */ +interface Mark { + readonly ensured: number; + readonly prompts: number; +} + +/** + * The owner's socket, with the provider sampled at each mapping commit. + * + * The scripted owner answers inside `send`, so what is read after it returns is + * what the provider had done at the moment the mapping was accepted. That is + * the only place the order between establishing a conversation, retaining which + * one it is, and prompting it can be observed at all — afterwards, all three + * have happened. + */ +function sampled(socket: OwnerSocket, fake: FakeAcp, marks: Mark[]): OwnerSocket { + return { + send(data: string): void { + const intent: Record = JSON.parse(data); + const proposed = intent["mappings"]; + const carries = (Array.isArray(proposed) ? proposed : []).some( + (mapping) => member(mapping, "kind") === "agent-session", + ); + socket.send(data); + if (carries) { + marks.push({ + ensured: fake.ensured.length, + prompts: fake.prompts.length, + }); + } + }, + close(): void { + socket.close(); + }, + addEventListener(type: "message" | "close" | "error", listener: SocketListener): void { + socket.addEventListener(type, listener); + }, + removeEventListener(type: "message" | "close" | "error", listener: SocketListener): void { + socket.removeEventListener(type, listener); + }, + }; +} + +/** What a turn against a session that was never established would do. */ +function tooEarly(): never { + throw new Error("PLANTED-TURN-WITHOUT-A-SESSION"); +} + +/** + * A provider whose establishment does one thing: what a case here tells it to. + * + * Two of the cases are about the window between asking a provider for a + * conversation and retaining which one it is. Nothing is retained inside it, so + * what has to be driven is the provider's own answer — one that fails, and one + * that never comes. + */ +function establishing( + asking: () => void, + answer: () => Promise, + closed: string[], +): (options: AcpRuntimeOptions) => ProbeCapableRuntime { + return function create(): ProbeCapableRuntime { + return { + doctor(): Promise { + return Promise.resolve({ ok: true, message: "fake agent ready" }); + }, + ensureSession(): Promise { + asking(); + return answer(); + }, + startTurn: tooEarly, + runTurn: tooEarly, + cancel(): Promise { + return Promise.resolve(); + }, + close(input: { readonly handle: AcpRuntimeHandle; readonly reason: string }): Promise { + closed.push(input.reason); + return Promise.resolve(); + }, + }; + }; +} + +/** + * The session a cancelled establishment answers with, too late to be used. + * + * A provider asked for a conversation answers whether or not anybody is still + * waiting, so this is a live session with nothing left to do with it but give + * it back. + */ +const ESTABLISHED_LATE: AcpRuntimeHandle = { + sessionKey: "cancelled-session", + backend: "acpx", + runtimeSessionName: "cancelled-session", + acpxRecordId: "cancelled-session", + backendSessionId: "acp:cancelled-session", + agentSessionId: "agent-session:cancelled-session", +}; + +/** + * One authored Agent document, executed as this run's root inside the + * attachment. + * + * Installed the way `xmd` itself installs it: `` names durable work + * after its own invocation, so the execution is told about the identity + * components rather than having them registered around it. + */ +function prompting(source: string, database: WorkflowRunDatabase): Operation { + return scoped(function* () { + return yield* collect( + yield* executeInstalled( + { + ...retainedSource("workflows/claude-session.md", source), + stream: database.journal, + }, + [{ components: agentIdentityComponents() }], + ), + ); + }); +} + +/** + * One attachment with the shipped Agent profile configured, and what it did. + * + * The profile is `useWorkflowAgentProfile()` itself, passed through the public + * configuration's `capabilities.agent`; only the agent process and the store it + * keeps its own sessions in are substituted. + */ +function attaching( + socket: OwnerSocket, + root: string, + source: string, + provider: { + readonly createRuntime: WorkflowAgentProfileOptions["createRuntime"]; + readonly sessionStore: WorkflowAgentProfileOptions["sessionStore"]; + }, +): Operation { + return scoped(function* () { + const built = yield* hostFor( + { socket }, + { + agent: (attachment) => + useWorkflowAgentProfile({ + root, + attachment, + defaultAgent: "claude", + ...provider, + }), + }, + ); + const transitions = yield* built.useRunHost(); + const taken = yield* WorkflowLifecycle.operations.acquireExecutor(RUN_ID); + if (!taken.ok || taken.value.kind !== "acquired") { + throw new Error("expected the configured host to take the acquisition"); + } + const begun = yield* transitions.begin(taken.value.lock, { + runId: RUN_ID, + action: "resume", + }); + if (!begun.ok) { + throw begun.error; + } + try { + yield* built.attach(begun.value.database, prompting(source, begun.value.database)); + return "attached"; + } catch (error) { + return error instanceof Error ? `raised:${error.message}` : "raised:other"; + } + }); +} + +/** + * The owner's socket, with one Workspace proposal held back. + * + * How a partial history is produced without inventing one. The Repository + * creation commits whole — root, staged content, mapping and its own journal + * row — and the proposal after it is still in flight when the run is + * cancelled: it never reaches the owner, so the owner never decides it and + * appends nothing for it. What is left is a prefix an owner can actually be + * holding, with real live work after it. + */ +function withholding( + socket: OwnerSocket, + withheld: Record[], + reached: () => void, +): OwnerSocket { + return { + send(data: string): void { + const intent: Record = JSON.parse(data); + const publication = intent["publication"]; + const mappings = intent["mappings"]; + const creation = (Array.isArray(mappings) ? mappings : []).some( + (mapping) => member(mapping, "kind") === "repository", + ); + if (withheld.length === 0 && publication !== null && publication !== undefined && !creation) { + withheld.push(intent); + reached(); + return; + } + socket.send(data); + }, + close(): void { + socket.close(); + }, + addEventListener(type: "message" | "close" | "error", listener: SocketListener): void { + socket.addEventListener(type, listener); + }, + removeEventListener(type: "message" | "close" | "error", listener: SocketListener): void { + socket.removeEventListener(type, listener); + }, + }; +} diff --git a/packages/workflow/tests/support/remote-owner-script.ts b/packages/workflow/tests/support/remote-owner-script.ts index 549e54fa9..d6d29fb08 100644 --- a/packages/workflow/tests/support/remote-owner-script.ts +++ b/packages/workflow/tests/support/remote-owner-script.ts @@ -22,6 +22,9 @@ import type { WorkflowRunDatabase } from "../../src/storage/api.ts"; /** The run every scripted owner here answers about. */ export const RUN_ID = "5cktgrv2zyutngh7bbddr2tyg2b5a567cg725hu5e7u42orerxaa"; +/** How many entries one anchored page carries, so paging is exercised at all. */ +const JOURNAL_PAGE = 2; + /** * One owner, scripted at the wire. * @@ -55,6 +58,17 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet manifests.set(digest, content.manifestBytes); } const staged = new Map(); + /** + * The filtered journal, as the owner keeps it. + * + * Each entry carries the id this owner minted for it and the Workspace root + * the transaction that appended it selected — the publication's proposed root + * when it published one, and the root the transaction expected when it did + * not. A root and a mapping beside an empty journal is not a state a real + * owner can reach, so this keeps all three or none. + */ + const journal: { eventId: string; record: string; workspaceRootId: string }[] = []; + let minted = 0; const repositories = new Map>(); const worktrees = new Map>(); const sessions = new Map>(); @@ -87,7 +101,7 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet }, retrieval: null, workspaceRootId: currentRoot, - journalEventId: null, + journalEventId: journal.at(-1)?.eventId ?? null, }; } @@ -123,13 +137,43 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet outcome: "performed", value: { workspaceRootId: currentRoot, - journalEventId: null, + journalEventId: journal.at(-1)?.eventId ?? null, repositories: [...repositories.values()], worktrees: [...worktrees.values()], agentSessions: [...sessions.values()], }, }; } + if (command === "journal") { + // One anchored page per request, continuing exactly where the client + // says it is. The anchor is the terminal event the frontier named, so a + // page that ran past it or stopped short of it would be a history no + // reader could assemble. + const anchorEventId = request["anchorEventId"]; + const afterEventId = request["afterEventId"] ?? null; + const from = + afterEventId === null + ? 0 + : journal.findIndex((entry) => entry.eventId === afterEventId) + 1; + if (from === 0 && afterEventId !== null) { + throw new Error("the runner asked to continue from an event this owner never minted"); + } + const page = journal.slice(from, from + JOURNAL_PAGE); + return { + outcome: "performed", + value: { + anchorEventId, + afterEventId, + entries: page.map((entry, index) => ({ + eventId: entry.eventId, + previousEventId: index === 0 ? afterEventId : (page[index - 1]?.eventId ?? null), + record: entry.record, + workspaceRootId: entry.workspaceRootId, + })), + done: from + page.length >= journal.length, + }, + }; + } if (command === "root") { return { outcome: "performed", @@ -221,11 +265,21 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet } } staged.clear(); + // Appended in the same step that moved the root and merged the mappings: + // the events, each carrying the root this transaction selected. + for (const record of events) { + minted += 1; + journal.push({ + eventId: `owner-event-${minted}`, + record: String(record), + workspaceRootId: currentRoot, + }); + } return { outcome: "performed", value: { workspaceRootId: currentRoot, - journalEventIds: events.map((_entry, index) => `event-${index}`), + journalEventIds: journal.slice(journal.length - events.length).map((e) => e.eventId), }, }; } @@ -264,6 +318,10 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet get currentRoot(): string { return currentRoot; }, + /** The filtered journal this owner retains, as it would answer a read. */ + entries(): readonly { eventId: string; record: string; workspaceRootId: string }[] { + return journal.map((entry) => ({ ...entry })); + }, refuse(reason: string): void { refusal = reason; }, From d1cddb77f29c1ea288f3cb930fa04094a5401886 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 17:17:21 -0300 Subject: [PATCH 76/81] =?UTF-8?q?=E2=9C=85=20Make=20the=20replay=20and=20c?= =?UTF-8?q?onflict=20evidence=20discriminate=20what=20it=20claims?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two assertions could not fail for the reason they were written. The replay check looked only at commits that published. A Repository effect that ran a second time while the owner already held a compatible mapping would neither clone nor publish a root: it would return that mapping and append its own event, in a commit that publishes nothing — exactly the shape the check had filtered away first. It now reads every commit the continuation made, and what the owner ends up holding: no `workspace_repository` event and no Repository mapping after the prefix, and one Repository effect row in the whole journal, the one already in it. The conflict case transplanted a mapping into a second owner that had never committed it and held neither the root nor the journal of the transaction that did. It now runs on one owner: the session is established through the shipped profile, the execution is cancelled with its first Prompt in flight, and the provider then comes back holding a different conversation under the same placement. Nothing else changes. The continuation refuses before a replacement session exists — the provider is never started — before anything is prompted, and before any commit at all, so the mapping, the root and the journal are the ones the interrupted run left. The refusal arrives as the cause of the divergence that stopping before a retained history is, so the assertions read the whole cause chain rather than the outermost wrapper. Beside them, the scripted owner answers a commit with the identities that commit minted rather than with a slice computed from the journal's length, and a mappings-only intent is now shown being answered with none. Its paged journal is capped at the anchor it was asked for and refuses an anchor it never minted or a cursor outside that prefix, which the Repository case reads back directly once the journal has run past the prefix its continuation consumed. --- .../cli/tests/remote-workflow-host.test.ts | 299 +++++++++++++++--- .../tests/support/remote-owner-script.ts | 48 ++- 2 files changed, 281 insertions(+), 66 deletions(-) diff --git a/packages/cli/tests/remote-workflow-host.test.ts b/packages/cli/tests/remote-workflow-host.test.ts index 13cfb4af4..e03ea639f 100644 --- a/packages/cli/tests/remote-workflow-host.test.ts +++ b/packages/cli/tests/remote-workflow-host.test.ts @@ -506,10 +506,7 @@ describe("the configured remote workflow host", () => { ); return { output: String(rendered), failure: "" }; } catch (error) { - return { - output: "", - failure: error instanceof Error ? error.message : "other", - }; + return { output: "", failure: chain(error) }; } }); } @@ -539,6 +536,35 @@ describe("the configured remote workflow host", () => { yield* remote.remove(); const again = yield* runThrough(source, owner.socket); + // The same anchored prefix, read again now that the journal has run past + // it. An owner answers the prefix a reader anchored — not the history + // that arrived afterwards — and refuses to answer at all for an anchor it + // never minted or a cursor outside that prefix. + const terminal = String(prefix.at(-1)?.eventId); + const reread = { + head: ask(owner, { command: "journal", anchorEventId: terminal, afterEventId: null }), + rest: ask(owner, { + command: "journal", + anchorEventId: terminal, + afterEventId: prefix[1]?.eventId ?? null, + }), + atEnd: ask(owner, { + command: "journal", + anchorEventId: terminal, + afterEventId: terminal, + }), + beyond: ask(owner, { + command: "journal", + anchorEventId: terminal, + afterEventId: owner.entries()[prefix.length]?.eventId ?? null, + }), + unknown: ask(owner, { + command: "journal", + anchorEventId: "owner-event-nothing", + afterEventId: null, + }), + }; + return { withheld, again, @@ -547,6 +573,7 @@ describe("the configured remote workflow host", () => { creation: owner.commits.slice(0, accepted), continuation: owner.commits.slice(accepted), replayed: owner.sent.slice(asked), + reread, owner, }; }); @@ -582,9 +609,7 @@ describe("the configured remote workflow host", () => { retaining[0]?.["publication"] ?? {}, "proposedWorkspaceRootId", ); - const repositoryEvent = outcome.retained.find((entry) => - entry.record.includes('"type":"workspace_repository"'), - ); + const repositoryEvent = outcome.retained.find((entry) => isRepositoryEffect(entry.record)); expect(repositoryEvent?.workspaceRootId).toBe(creationRoot); expect( outcome.retained.filter((entry) => entry.record.includes('"type":"workspace_git_switch"')), @@ -596,6 +621,22 @@ describe("the configured remote workflow host", () => { const pages = outcome.replayed.filter((request) => request["command"] === "journal"); expect(pages.length > 0).toBe(true); expect(pages[0]?.["anchorEventId"]).toBe(outcome.retained.at(-1)?.eventId); + // And the prefix a reader anchors is the prefix it gets, however far the + // journal has run since: the whole of it in pages, nothing that arrived + // after it, and no answer at all for an anchor this owner never minted or + // a cursor outside that prefix. + expect(listed(outcome.reread.head)).toEqual( + outcome.retained.slice(0, 2).map((entry) => entry.eventId), + ); + expect(member(outcome.reread.head["value"], "done")).toBe(false); + expect(listed(outcome.reread.rest)).toEqual( + outcome.retained.slice(2).map((entry) => entry.eventId), + ); + expect(member(outcome.reread.rest["value"], "done")).toBe(true); + expect(listed(outcome.reread.atEnd)).toEqual([]); + expect(member(outcome.reread.atEnd["value"], "done")).toBe(true); + expect(String(outcome.reread.beyond["raised"])).toContain("beyond the prefix it anchored"); + expect(String(outcome.reread.unknown["raised"])).toContain("never minted"); // The recorded creation restored rather than cloning again — the remote it // was cloned from no longer exists — and the checkout the Git mutation @@ -616,7 +657,7 @@ describe("the configured remote workflow host", () => { expect(only(read)).toContain('"content":"release'); expect(outcome.again.output).toContain("switched to: release"); - // Two transactions and no third: only the work the cancellation left + // Two publications and no third: only the work the cancellation left // undone. The mutation's own journal row carries the root it published, // and that root is the run's — a read moves nothing, so the switch is the // last thing that moved it. @@ -627,15 +668,24 @@ describe("the configured remote workflow host", () => { .find((entry) => entry.record.includes('"type":"workspace_git_switch"')); expect(gitEvent?.workspaceRootId).toBe(mutationRoot); expect(outcome.owner.currentRoot).toBe(mutationRoot); - // Nothing retained a second Repository, and the ambient host filesystem was - // asked for nothing by either execution. - const recreated = mutation.filter((intent) => { - const mappings = intent["mappings"]; - return ( - Array.isArray(mappings) && mappings.some((m) => Reflect.get(m, "kind") === "repository") - ); - }); - expect(recreated).toEqual([]); + + // And the creation restored rather than running again, which is a claim + // about *every* commit the continuation made rather than about the two + // that published. A Repository effect that executed a second time while + // the owner already held a compatible mapping would neither clone nor + // publish a root — it would return that mapping and append its own event + // — so a check that looked only at publications could not see it. This + // one looks at the whole sequence, and at what the owner ends up holding: + // one Repository effect row in the journal, the one already in the + // retained prefix, and no Repository mapping proposed again. + expect(records(outcome.continuation).filter(isRepositoryEffect)).toEqual([]); + expect(mappingsOf(outcome.continuation, "repository")).toEqual([]); + const repositoryRows = outcome.owner + .entries() + .filter((entry) => isRepositoryEffect(entry.record)); + expect(repositoryRows).toHaveLength(1); + expect(repositoryRows[0]?.eventId).toBe(repositoryEvent?.eventId); + // The ambient host filesystem was asked for nothing by either execution. expect(outcome.ambient).toEqual([]); }); @@ -693,6 +743,7 @@ describe("the configured remote workflow host", () => { created, asserted, marks, + enlisted: enlistment(owner), establishedFirst: established(live), promptedFirst: [...live.prompts], again, @@ -716,6 +767,13 @@ describe("the configured remote workflow host", () => { expect([String(member(member(outcome.created[0], "assertion"), "value"))]).toEqual( outcome.asserted, ); + // It crossed as a mappings-only intent — one mapping, no event and no + // publication — and the owner answered it with no minted identity at all. + // One identity per proposed event is the client's rule, and a transaction + // that proposed none is answered with none. + expect(outcome.enlisted?.publication).toBe(null); + expect(outcome.enlisted?.events).toEqual([]); + expect(outcome.enlisted?.journalEventIds).toEqual([]); // The order is the whole of it, sampled at the owner: the conversation // existed, the owner then accepted which one it was, and only then did // anything prompt it. @@ -808,59 +866,93 @@ describe("the configured remote workflow host", () => { expect(cancelled.asserted).toEqual([]); }); - it("refuses a conversation the provider replaced, before prompting or retaining", function* () { - const captured = yield* startingTree(); + it("refuses a conversation the provider replaced, and leaves the run as it was", function* () { const root = yield* useTempDirectory("xmd-remote-agent-conflict-"); const source = yield* readTextFile(join(FIXTURES, "claude-session.md")); const outcome = yield* scoped(function* () { - // One run that established a session, so what the owner below retains is - // a mapping this stack actually wrote rather than one this test composed. - const establishedRun = scriptedOwner(captured); + const captured = yield* startingTree(); + const owner = scriptedOwner(captured); const store = makeStore(); + + // One run, one owner, and the state that run actually left there. The + // session is established through the shipped profile and the execution + // is cancelled with its first Prompt genuinely in flight: the mapping + // commits before that Prompt begins, so what this owner is holding is + // the conversation, the root and the journal prefix of the transaction + // that put it there — with a turn still unfinished, which is what gives + // the attachment below something to continue. const live = createFakeAcp(); - live.script({ reply: "the reviewer saw the release notes" }); - live.script({ reply: "and they recommended shipping it" }); - const first = yield* attaching(establishedRun.socket, root, source, { - createRuntime: live.create, - sessionStore: store, - }); - const record = retained(establishedRun); + live.script({ reply: "", manual: true }); + const attempt = yield* spawn(() => + attaching(owner.socket, root, source, { + createRuntime: live.create, + sessionStore: store, + }), + ); + yield* live.startedTurns(1); + yield* attempt.halt(); + + const before = { + root: owner.currentRoot, + journal: owner.entries(), + mappings: owner.agentSessions(), + commits: owner.commits.length, + }; - // The same run as an owner holds it, and a provider whose store now - // asserts a different conversation under the same placement. - const owner = scriptedOwner(captured, { agentSessions: record }); - const replacement = makeStore(); + // The provider comes back holding a different conversation under the + // same placement. Nothing else changes: the same owner, the same run, + // the same store, the same configured host. for (const [key, held] of store.records) { - replacement.records.set(key, { - ...held, - agentSessionId: "another-conversation", - }); + store.records.set(key, { ...held, agentSessionId: "another-conversation" }); } const provider = createFakeAcp(); + provider.script({ reply: "a turn nothing may reach" }); const refused = yield* attaching(owner.socket, root, source, { createRuntime: provider.create, - sessionStore: replacement, + sessionStore: store, }); + return { - first, - record, + before, refused, - ensured: provider.ensured.length, + established: provider.ensured.length, + started: provider.started, prompts: provider.prompts.length, - proposed: retained(owner), + continuation: owner.commits.slice(before.commits), + after: { + root: owner.currentRoot, + journal: owner.entries(), + mappings: owner.agentSessions(), + }, }; }); - expect(outcome.first).toBe("attached"); - expect(outcome.record).toHaveLength(1); - // Refused where the decision belongs: before a replacement session is - // established and before anything is prompted. The owner was asked to - // retain nothing, so what it holds is still the conversation this run had. + // The interrupted run really did leave a conversation, a root and a + // journal behind — otherwise there is nothing here to conflict with. + expect(outcome.before.mappings).toHaveLength(1); + expect(outcome.before.journal.length > 0).toBe(true); + // The refusal is the shipped policy's own. A continuation that stops before + // its retained history is exhausted is a divergence, and this one carries + // the refusal as its cause: what the run refused about is still what the + // failure says. + expect(outcome.refused).toContain("Divergence"); expect(outcome.refused).toContain("different durable identity"); - expect(outcome.ensured).toBe(0); + // It happened where the decision belongs: before a replacement session was + // established — the provider was never started at all — and before + // anything was prompted. + expect(outcome.established).toBe(0); + expect(outcome.started).toBe(false); expect(outcome.prompts).toBe(0); - expect(outcome.proposed).toEqual([]); + // And before any mapping proposal — before any commit at all: this owner + // was not asked to retain, replace or forget anything. + expect(mappingsOf(outcome.continuation, "agent-session")).toEqual([]); + expect(outcome.continuation).toEqual([]); + // What it holds is what it held: the same mapping, the same root, and the + // same journal, entry for entry. + expect(outcome.after.mappings).toEqual(outcome.before.mappings); + expect(outcome.after.root).toBe(outcome.before.root); + expect(outcome.after.journal).toEqual(outcome.before.journal); }); it("attaches nothing it did not open", function* () { @@ -902,6 +994,113 @@ function only(intent: Record | undefined): string { return (Array.isArray(events) ? events : []).map((event) => String(event)).join(""); } +/** + * Every journal record a sequence of commits proposed, in order. + * + * The whole sequence, not the publications in it: durable work that restores + * from a compatible mapping publishes nothing and still appends its own event, + * so a claim about what a continuation did has to be a claim about every commit + * it made. + */ +function records(commits: readonly Record[]): string[] { + return commits.flatMap((intent) => { + const events = intent["events"]; + return (Array.isArray(events) ? events : []).map((event) => String(event)); + }); +} + +/** Every mapping of one kind a sequence of commits proposed, in order. */ +function mappingsOf(commits: readonly Record[], kind: string): unknown[] { + return commits.flatMap((intent) => { + const mappings = intent["mappings"]; + return (Array.isArray(mappings) ? mappings : []).filter( + (mapping) => member(mapping, "kind") === kind, + ); + }); +} + +/** + * The mappings-only commit this run enlisted its Agent session through, and + * what the owner answered it. + * + * Read out of the wire traffic rather than reconstructed: what is being checked + * is the shape of an intent that carries a mapping and nothing else, and the + * shape of the answer to it. + */ +function enlistment(owner: { + readonly sent: readonly Record[]; + readonly answered: readonly Record[]; +}): { publication: unknown; events: unknown; journalEventIds: unknown } | undefined { + const intent = owner.sent.find( + (request) => + request["command"] === "commit" && mappingsOf([request], "agent-session").length > 0, + ); + if (intent === undefined) { + return undefined; + } + const given = owner.answered.find((answer) => answer["id"] === intent["id"]); + return { + publication: intent["publication"], + events: intent["events"], + journalEventIds: member(given?.["value"], "journalEventIds"), + }; +} + +/** + * One request straight to an owner, answered the way it answers the client. + * + * The scripted owner answers inside `send`, so this is its own answer to + * exactly this request rather than a reconstruction of one. A request it + * refuses to answer at all comes back as what it raised. + */ +function ask( + owner: { readonly socket: OwnerSocket }, + request: Record, +): Record { + let answer: Record = {}; + const listener: SocketListener = (event) => { + answer = JSON.parse(String(event.data)); + }; + owner.socket.addEventListener("message", listener); + try { + owner.socket.send(JSON.stringify({ id: "read-1", ...request })); + } catch (error) { + return { raised: chain(error) }; + } finally { + owner.socket.removeEventListener("message", listener); + } + return answer; +} + +/** The event identities one journal answer listed, in order. */ +function listed(answer: Record): string[] { + const entries = member(answer["value"], "entries"); + return (Array.isArray(entries) ? entries : []).map((entry) => String(member(entry, "eventId"))); +} + +/** + * One failure and everything it was caused by, in order. + * + * A failure this stack reports is often a wrapper over the decision that caused + * it — a run that refuses before its retained history is exhausted is reported + * as a divergence carrying that refusal — and a test asserting on the outermost + * message alone would be asserting on the wrapper. + */ +function chain(error: unknown): string { + const messages: string[] = []; + let current: unknown = error; + while (current instanceof Error && messages.length < 8) { + messages.push(current.message); + current = current.cause; + } + return messages.length === 0 ? String(error) : messages.join(" <- "); +} + +/** Whether one journal record is a Repository effect's own result. */ +function isRepositoryEffect(record: string): boolean { + return record.includes('"type":"workspace_repository"'); +} + /** One member of a value nothing has checked. */ function member(value: unknown, name: string): unknown { return value !== null && typeof value === "object" ? Reflect.get(value, name) : undefined; @@ -1102,7 +1301,7 @@ function attaching( yield* built.attach(begun.value.database, prompting(source, begun.value.database)); return "attached"; } catch (error) { - return error instanceof Error ? `raised:${error.message}` : "raised:other"; + return `raised:${chain(error)}`; } }); } diff --git a/packages/workflow/tests/support/remote-owner-script.ts b/packages/workflow/tests/support/remote-owner-script.ts index d6d29fb08..00ff432e1 100644 --- a/packages/workflow/tests/support/remote-owner-script.ts +++ b/packages/workflow/tests/support/remote-owner-script.ts @@ -43,6 +43,7 @@ export interface ScriptedRetention { export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRetention = {}) { const sent: Record[] = []; + const answered: Record[] = []; const commits: Record[] = []; let currentRoot = captured.root.rootId; let currentManifest = captured.root.manifest; @@ -146,11 +147,16 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet } if (command === "journal") { // One anchored page per request, continuing exactly where the client - // says it is. The anchor is the terminal event the frontier named, so a - // page that ran past it or stopped short of it would be a history no - // reader could assemble. - const anchorEventId = request["anchorEventId"]; + // says it is. The anchor is the terminal event the reader started from, + // and the page is capped at it however far the journal has since run: + // a page that ran past the anchor would hand a reader history that was + // not there when it decided where the end was. + const anchorEventId = String(request["anchorEventId"] ?? ""); const afterEventId = request["afterEventId"] ?? null; + const anchor = journal.findIndex((entry) => entry.eventId === anchorEventId); + if (anchor === -1) { + throw new Error("the runner anchored a read to an event this owner never minted"); + } const from = afterEventId === null ? 0 @@ -158,7 +164,10 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet if (from === 0 && afterEventId !== null) { throw new Error("the runner asked to continue from an event this owner never minted"); } - const page = journal.slice(from, from + JOURNAL_PAGE); + if (from > anchor + 1) { + throw new Error("the runner asked to continue from beyond the prefix it anchored"); + } + const page = journal.slice(from, Math.min(from + JOURNAL_PAGE, anchor + 1)); return { outcome: "performed", value: { @@ -170,7 +179,7 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet record: entry.record, workspaceRootId: entry.workspaceRootId, })), - done: from + page.length >= journal.length, + done: from + page.length >= anchor + 1, }, }; } @@ -266,21 +275,20 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet } staged.clear(); // Appended in the same step that moved the root and merged the mappings: - // the events, each carrying the root this transaction selected. + // the events, each carrying the root this transaction selected. What is + // answered is what this transaction minted and nothing else — one identity + // per proposed event, in order, and an empty list for a commit that + // proposed none, which is what a mappings-only intent is. + const appended: string[] = []; for (const record of events) { minted += 1; - journal.push({ - eventId: `owner-event-${minted}`, - record: String(record), - workspaceRootId: currentRoot, - }); + const eventId = `owner-event-${minted}`; + appended.push(eventId); + journal.push({ eventId, record: String(record), workspaceRootId: currentRoot }); } return { outcome: "performed", - value: { - workspaceRootId: currentRoot, - journalEventIds: journal.slice(journal.length - events.length).map((e) => e.eventId), - }, + value: { workspaceRootId: currentRoot, journalEventIds: appended }, }; } @@ -290,6 +298,9 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet const request: Record = JSON.parse(data); sent.push(request); const response = answer(request); + // What it answered, beside what it was asked. A test asserting on the + // shape of an answer should read the answer rather than re-derive it. + answered.push({ id: request["id"], command: request["command"], ...response }); if (response["outcome"] === "lost") { for (const listener of listeners.get("close") ?? []) { listener({}); @@ -314,10 +325,15 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet return { socket, sent, + answered, commits, get currentRoot(): string { return currentRoot; }, + /** The Agent-session mappings this owner retains, as it would report them. */ + agentSessions(): readonly Record[] { + return [...sessions.values()].map((record) => ({ ...record })); + }, /** The filtered journal this owner retains, as it would answer a read. */ entries(): readonly { eventId: string; record: string; workspaceRootId: string }[] { return journal.map((entry) => ({ ...entry })); From 452265f4916b69adc4003999c1411b6d2c59f34a Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 17:46:03 -0300 Subject: [PATCH 77/81] =?UTF-8?q?=E2=9C=85=20Refuse=20a=20journal=20cursor?= =?UTF-8?q?=20at=20its=20anchor,=20the=20way=20the=20owner=20does?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The anchored journal contract admits a cursor only strictly before the anchor: `readJournalPage()` refuses `afterSequence >= anchorSequence`, because a cursor at the end of a snapshot is a reader asking for a page after its own snapshot ends. The scripted owner refused only a cursor past the anchor and answered one at the anchor with an empty successful page, and the direct read that checks the anchored prefix recorded that answer as valid. So a client that asked for another page after reaching its terminal event could have passed this suite while the owner it will actually talk to refused. Both cursors are now the same refusal, and the test expects it. Nothing else moves: a null cursor still starts at the first entry, a cursor strictly before the anchor still continues after it, the page is still capped at the anchor however far the journal has run since, and an anchor this owner never minted is still refused as one. --- .../cli/tests/remote-workflow-host.test.ts | 14 ++++++++----- .../tests/support/remote-owner-script.ts | 20 ++++++++++++------- 2 files changed, 22 insertions(+), 12 deletions(-) diff --git a/packages/cli/tests/remote-workflow-host.test.ts b/packages/cli/tests/remote-workflow-host.test.ts index e03ea639f..f56d80020 100644 --- a/packages/cli/tests/remote-workflow-host.test.ts +++ b/packages/cli/tests/remote-workflow-host.test.ts @@ -539,7 +539,8 @@ describe("the configured remote workflow host", () => { // The same anchored prefix, read again now that the journal has run past // it. An owner answers the prefix a reader anchored — not the history // that arrived afterwards — and refuses to answer at all for an anchor it - // never minted or a cursor outside that prefix. + // never minted, or for a cursor that is not inside the snapshot that + // anchor names: at the anchor is already outside it. const terminal = String(prefix.at(-1)?.eventId); const reread = { head: ask(owner, { command: "journal", anchorEventId: terminal, afterEventId: null }), @@ -548,7 +549,7 @@ describe("the configured remote workflow host", () => { anchorEventId: terminal, afterEventId: prefix[1]?.eventId ?? null, }), - atEnd: ask(owner, { + atAnchor: ask(owner, { command: "journal", anchorEventId: terminal, afterEventId: terminal, @@ -633,9 +634,12 @@ describe("the configured remote workflow host", () => { outcome.retained.slice(2).map((entry) => entry.eventId), ); expect(member(outcome.reread.rest["value"], "done")).toBe(true); - expect(listed(outcome.reread.atEnd)).toEqual([]); - expect(member(outcome.reread.atEnd["value"], "done")).toBe(true); - expect(String(outcome.reread.beyond["raised"])).toContain("beyond the prefix it anchored"); + // A cursor at the anchor and a cursor past it are the same refusal, and it + // is the owner's own: `readJournalPage()` refuses a cursor whose sequence + // is at or after the anchor's. Answering either with an empty page would be + // a fixture admitting a cursor state the real boundary rejects. + expect(String(outcome.reread.atAnchor["raised"])).toContain("outside its anchored snapshot"); + expect(String(outcome.reread.beyond["raised"])).toContain("outside its anchored snapshot"); expect(String(outcome.reread.unknown["raised"])).toContain("never minted"); // The recorded creation restored rather than cloning again — the remote it diff --git a/packages/workflow/tests/support/remote-owner-script.ts b/packages/workflow/tests/support/remote-owner-script.ts index 00ff432e1..4389b4b1c 100644 --- a/packages/workflow/tests/support/remote-owner-script.ts +++ b/packages/workflow/tests/support/remote-owner-script.ts @@ -151,22 +151,28 @@ export function scriptedOwner(captured: CapturedWorkspace, retained: ScriptedRet // and the page is capped at it however far the journal has since run: // a page that ran past the anchor would hand a reader history that was // not there when it decided where the end was. + // + // A cursor is admitted only where the real owner admits one — strictly + // before the anchor, which `readJournalPage()` enforces by refusing + // `afterSequence >= anchorSequence`. A cursor *at* the anchor is a reader + // asking for a page after the end of its own snapshot, and answering it + // with an empty page here would let a client pass this suite while the + // owner it will actually talk to refuses. const anchorEventId = String(request["anchorEventId"] ?? ""); const afterEventId = request["afterEventId"] ?? null; const anchor = journal.findIndex((entry) => entry.eventId === anchorEventId); if (anchor === -1) { throw new Error("the runner anchored a read to an event this owner never minted"); } - const from = - afterEventId === null - ? 0 - : journal.findIndex((entry) => entry.eventId === afterEventId) + 1; - if (from === 0 && afterEventId !== null) { + const after = + afterEventId === null ? -1 : journal.findIndex((entry) => entry.eventId === afterEventId); + if (afterEventId !== null && after === -1) { throw new Error("the runner asked to continue from an event this owner never minted"); } - if (from > anchor + 1) { - throw new Error("the runner asked to continue from beyond the prefix it anchored"); + if (after >= anchor) { + throw new Error("a journal cursor is outside its anchored snapshot"); } + const from = after + 1; const page = journal.slice(from, Math.min(from + JOURNAL_PAGE, anchor + 1)); return { outcome: "performed", From a8c9f321900abcb35e0d2fc735b7a34f0d17373f Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 19:27:24 -0300 Subject: [PATCH 78/81] =?UTF-8?q?=F0=9F=A7=B9=20Let=20the=20runner's=20Wor?= =?UTF-8?q?kspace=20tests=20own=20their=20own=20scope?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `local/no-redundant-test-scope` refuses a `scoped()` call that wraps a whole `it()` body: the BDD adapter already runs each test in its own scope, so such a wrapper opens a lifetime that begins and ends exactly where the test's own does, and leaves a reader two candidate owners for the test's resources. Twelve of them were in the runner's Workspace suite. This is the rule's own fix — each body dedented one level, nothing else moved — applied because the rule is only reachable through the full `deno task lint`, which runs its JS plugin, and the focused per-file checks that carried these commits never loaded it. --- .../workflow/tests/remote-workspace.test.ts | 586 +++++++++--------- 1 file changed, 279 insertions(+), 307 deletions(-) diff --git a/packages/workflow/tests/remote-workspace.test.ts b/packages/workflow/tests/remote-workspace.test.ts index 1fa164fdf..7e851de57 100644 --- a/packages/workflow/tests/remote-workspace.test.ts +++ b/packages/workflow/tests/remote-workspace.test.ts @@ -411,223 +411,242 @@ function workspaceIntents(commits: readonly Record[]): Record { it("commits Files, one mapping and the filtered result as one intent", function* () { - yield* scoped(function* () { - const held = yield* harness(); - const { raised, events } = yield* invocation( - held, - "write", - function* (filesystem, metadata): Operation { - yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); - yield* filesystem.mkdir("/app", { mode: 0o755 }); - metadata.insertRepository(repository()); - // Read-your-writes: its own insert, before anything is committed. - return metadata.readRepository("app")?.record.checkoutPath ?? "missing"; - }, - ); - expect(raised).toBe(undefined); - - // Exactly one intent, carrying all three things together. - const intents = workspaceIntents(held.commits); - expect(intents).toHaveLength(1); - const intent = intents[0] ?? {}; - expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); - const mappings = intent["mappings"]; - expect(Array.isArray(mappings) && mappings).toHaveLength(1); - expect((mappings as Record[])[0]?.["kind"]).toBe("repository"); - expect(intent["publication"]).not.toBe(null); - // The result travelled in this same intent rather than through the - // ordinary journal, so nothing was written before the owner agreed. - expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); - expect(yielded(events)).toHaveLength(0); - }); + const held = yield* harness(); + const { raised, events } = yield* invocation( + held, + "write", + function* (filesystem, metadata): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + yield* filesystem.mkdir("/app", { mode: 0o755 }); + metadata.insertRepository(repository()); + // Read-your-writes: its own insert, before anything is committed. + return metadata.readRepository("app")?.record.checkoutPath ?? "missing"; + }, + ); + expect(raised).toBe(undefined); + + // Exactly one intent, carrying all three things together. + const intents = workspaceIntents(held.commits); + expect(intents).toHaveLength(1); + const intent = intents[0] ?? {}; + expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); + const mappings = intent["mappings"]; + expect(Array.isArray(mappings) && mappings).toHaveLength(1); + expect((mappings as Record[])[0]?.["kind"]).toBe("repository"); + expect(intent["publication"]).not.toBe(null); + // The result travelled in this same intent rather than through the + // ordinary journal, so nothing was written before the owner agreed. + expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); + expect(yielded(events)).toHaveLength(0); }); it("journals a documented failure against the unchanged root, and keeps nothing", function* () { - yield* scoped(function* () { - const held = yield* harness(); - const { raised } = yield* invocation( - held, - "refuse", - function* (filesystem, metadata): Operation { - yield* filesystem.writeFile("/SCRATCH.md", "discarded\n", 0o644); - yield* filesystem.remove("/README.md"); - metadata.insertRepository(repository()); - throw new DocumentedFailure("this Workspace effect refused"); - }, - ); - expect(String(raised)).toContain("this Workspace effect refused"); - - // One commit, and it proposes nothing about the Workspace. - const intents = workspaceIntents(held.commits); - expect(intents).toHaveLength(1); - const intent = intents[0] ?? {}; - expect(intent["publication"]).toBe(null); - expect(intent["mappings"]).toEqual([]); - expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); - expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); - - // That the owner still holds the starting root after this is a claim - // about storage, and it is made against real owner storage in - // `remote-workspace.vitest.ts`. What is settled here is that nothing was - // proposed: no publication, no mapping, and the root this commit expected - // is the one the invocation was admitted from. - }); + const held = yield* harness(); + const { raised } = yield* invocation( + held, + "refuse", + function* (filesystem, metadata): Operation { + yield* filesystem.writeFile("/SCRATCH.md", "discarded\n", 0o644); + yield* filesystem.remove("/README.md"); + metadata.insertRepository(repository()); + throw new DocumentedFailure("this Workspace effect refused"); + }, + ); + expect(String(raised)).toContain("this Workspace effect refused"); + + // One commit, and it proposes nothing about the Workspace. + const intents = workspaceIntents(held.commits); + expect(intents).toHaveLength(1); + const intent = intents[0] ?? {}; + expect(intent["publication"]).toBe(null); + expect(intent["mappings"]).toEqual([]); + expect(intent["expectedWorkspaceRootId"]).toBe(held.captured.root.rootId); + expect(Array.isArray(intent["events"]) && intent["events"]).toHaveLength(1); + + // That the owner still holds the starting root after this is a claim + // about storage, and it is made against real owner storage in + // `remote-workspace.vitest.ts`. What is settled here is that nothing was + // proposed: no publication, no mapping, and the root this commit expected + // is the one the invocation was admitted from. }); it("prevents the document from running when the admitted root is unreachable", function* () { - yield* scoped(function* () { - const held = yield* harness(() => emptySnapshot("f".repeat(64))); - let executed = 0; - // deno-lint-ignore require-yield - yield* invocation(held, "unreachable", function* (): Operation { - executed += 1; - return "ran"; - }); - // Materialization is before the transaction, so this never reaches the - // anchor check — and it must still leave the run exactly as it was. - expect(executed).toBe(0); - expect(held.commits).toEqual([]); + const held = yield* harness(() => emptySnapshot("f".repeat(64))); + let executed = 0; + // deno-lint-ignore require-yield + yield* invocation(held, "unreachable", function* (): Operation { + executed += 1; + return "ran"; }); + // Materialization is before the transaction, so this never reaches the + // anchor check — and it must still leave the run exactly as it was. + expect(executed).toBe(0); + expect(held.commits).toEqual([]); }); it("refuses before the document runs when the run moved since admission", function* () { - yield* scoped(function* () { - // The root still materializes, so the invocation gets all the way to the - // transaction; the journal anchor is what has moved. Nothing later could - // notice on its own — both answers were true when they were given. - const held = yield* harness((rootId) => ({ - ...emptySnapshot(rootId), - journalEventId: "event-from-another-moment", - })); - let executed = 0; - // deno-lint-ignore require-yield - const { raised } = yield* invocation(held, "drifted", function* (): Operation { - executed += 1; - return "ran"; - }); - expect(String(raised)).toContain("moved past"); - expect(executed).toBe(0); - expect(held.commits).toEqual([]); + // The root still materializes, so the invocation gets all the way to the + // transaction; the journal anchor is what has moved. Nothing later could + // notice on its own — both answers were true when they were given. + const held = yield* harness((rootId) => ({ + ...emptySnapshot(rootId), + journalEventId: "event-from-another-moment", + })); + let executed = 0; + // deno-lint-ignore require-yield + const { raised } = yield* invocation(held, "drifted", function* (): Operation { + executed += 1; + return "ran"; }); + expect(String(raised)).toContain("moved past"); + expect(executed).toBe(0); + expect(held.commits).toEqual([]); }); it("leaves the accepted Workspace alone when the owner refuses the commit", function* () { - yield* scoped(function* () { - const held = yield* harness(); - held.refuse("command:stale-root"); - const { raised } = yield* invocation( - held, - "refused", - function* (filesystem): Operation { - yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); - return "ran"; - }, - ); - expect(raised).not.toBe(undefined); - // The owner said no, so nothing is promoted and nothing private crossed. - expect(String(raised)).not.toContain("command:"); - expect(workspaceIntents(held.commits)).toHaveLength(1); + const held = yield* harness(); + held.refuse("command:stale-root"); + const { raised } = yield* invocation(held, "refused", function* (filesystem): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + return "ran"; }); + expect(raised).not.toBe(undefined); + // The owner said no, so nothing is promoted and nothing private crossed. + expect(String(raised)).not.toContain("command:"); + expect(workspaceIntents(held.commits)).toHaveLength(1); }); it("cannot pair one run's handle with another run's link, journal or provenance", function* () { + // Two owners, deliberately begun from the same root and the same empty + // journal. Every structural value they hold is equal; only the objects + // differ, and only the objects decide. + const tree = yield* startingTree(); + const a = yield* harness(emptySnapshot, tree); + const b = yield* harness(emptySnapshot, tree); + expect(a.run.database.record.runId).toBe(b.run.database.record.runId); + + // An effect made against A, coordinated under B. yield* scoped(function* () { - // Two owners, deliberately begun from the same root and the same empty - // journal. Every structural value they hold is equal; only the objects - // differ, and only the objects decide. - const tree = yield* startingTree(); - const a = yield* harness(emptySnapshot, tree); - const b = yield* harness(emptySnapshot, tree); - expect(a.run.database.record.runId).toBe(b.run.database.record.runId); - - // An effect made against A, coordinated under B. - yield* scoped(function* () { - yield* useRemoteWorkspaceEffects(b.run); - const effect = createRemoteWorkspaceEffect(a.run, { type: "workspace", name: "a" }, own); - function* workflow(): Workflow { - yield effect; - } - const raised = yield* trapped( - withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: b.run.journal })), - ); - expect(String(raised)).toContain("foreign"); - }); + yield* useRemoteWorkspaceEffects(b.run); + const effect = createRemoteWorkspaceEffect(a.run, { type: "workspace", name: "a" }, own); + function* workflow(): Workflow { + yield effect; + } + const raised = yield* trapped( + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: b.run.journal })), + ); + expect(String(raised)).toContain("foreign"); + }); - // A's coordinator installed, B's binding asked to use it. - yield* scoped(function* () { - yield* useRemoteWorkspaceEffects(a.run); - const effect = createRemoteWorkspaceEffect(b.run, { type: "workspace", name: "b" }, own); - function* workflow(): Workflow { - yield effect; - } - const raised = yield* trapped( - withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: b.run.journal })), - ); - expect(String(raised)).toContain("no remote Workspace coordinator is installed"); - }); + // A's coordinator installed, B's binding asked to use it. + yield* scoped(function* () { + yield* useRemoteWorkspaceEffects(a.run); + const effect = createRemoteWorkspaceEffect(b.run, { type: "workspace", name: "b" }, own); + function* workflow(): Workflow { + yield effect; + } + const raised = yield* trapped( + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: b.run.journal })), + ); + expect(String(raised)).toContain("no remote Workspace coordinator is installed"); + }); - // B throughout, running over A's journal. The provenance is A's. - yield* scoped(function* () { - yield* useRemoteWorkspaceEffects(b.run); - const effect = createRemoteWorkspaceEffect(b.run, { type: "workspace", name: "c" }, own); - function* workflow(): Workflow { - yield effect; - } - const raised = yield* trapped( - withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: a.run.journal })), - ); - expect(String(raised)).toContain("provenance"); - }); - - // A value shaped like a binding is not one. - const forged = { database: b.run.database, journal: b.run.journal }; - expect( - String(yield* trapped(useRemoteWorkspaceEffects(forged as unknown as RemoteRun))), - ).toContain("not a remote run this build opened"); - - // Neither owner was asked for anything, and neither journal moved. - expect([a.commits, b.commits]).toEqual([[], []]); - expect(yielded(yield* a.run.journal.readAll())).toEqual([]); - expect(yielded(yield* b.run.journal.readAll())).toEqual([]); + // B throughout, running over A's journal. The provenance is A's. + yield* scoped(function* () { + yield* useRemoteWorkspaceEffects(b.run); + const effect = createRemoteWorkspaceEffect(b.run, { type: "workspace", name: "c" }, own); + function* workflow(): Workflow { + yield effect; + } + const raised = yield* trapped( + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: a.run.journal })), + ); + expect(String(raised)).toContain("provenance"); }); + + // A value shaped like a binding is not one. + const forged = { database: b.run.database, journal: b.run.journal }; + expect( + String(yield* trapped(useRemoteWorkspaceEffects(forged as unknown as RemoteRun))), + ).toContain("not a remote run this build opened"); + + // Neither owner was asked for anything, and neither journal moved. + expect([a.commits, b.commits]).toEqual([[], []]); + expect(yielded(yield* a.run.journal.readAll())).toEqual([]); + expect(yielded(yield* b.run.journal.readAll())).toEqual([]); }); it("cannot be opened from one owner's reads and another owner's commits", function* () { - yield* scoped(function* () { - // The construction the correction closes. Two owners, deliberately begun - // from one captured tree, so their root, anchor and run record are equal - // and only the objects differ. Before this, `useRemoteRun` took the - // database/commit link and the Workspace read link separately, and this - // combination produced a legitimate binding: the invocation would be - // admitted from A's mappings and content and commit its result to B. - const tree = yield* startingTree(); - const a = yield* owner(tree); - const b = yield* owner(tree); - - // Each link is built from one connection and carries its own reads, so - // reading through one reaches that owner and no other. - const linkA = cloudflareRunLink(a.connection, a.next, RUN_ID); - const linkB = cloudflareRunLink(b.connection, b.next, RUN_ID); - yield* linkA.invocationSnapshot(); - expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); - expect(b.requests).toEqual([]); - - expect(ONE_LINK).toBe(true); - const options: RemoteRunOptions = { - link: linkB, - files: runnerFiles(), - trees: yield* useRunnerTrees(), - createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), - }; + // The construction the correction closes. Two owners, deliberately begun + // from one captured tree, so their root, anchor and run record are equal + // and only the objects differ. Before this, `useRemoteRun` took the + // database/commit link and the Workspace read link separately, and this + // combination produced a legitimate binding: the invocation would be + // admitted from A's mappings and content and commit its result to B. + const tree = yield* startingTree(); + const a = yield* owner(tree); + const b = yield* owner(tree); + + // Each link is built from one connection and carries its own reads, so + // reading through one reaches that owner and no other. + const linkA = cloudflareRunLink(a.connection, a.next, RUN_ID); + const linkB = cloudflareRunLink(b.connection, b.next, RUN_ID); + yield* linkA.invocationSnapshot(); + expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); + expect(b.requests).toEqual([]); + + expect(ONE_LINK).toBe(true); + const options: RemoteRunOptions = { + link: linkB, + files: runnerFiles(), + trees: yield* useRunnerTrees(), + createFilesystem: (at, authorize) => createRemoteWorkspaceFilesystem(at, authorize), + }; + + let executed = 0; + const run = yield* useRemoteRun(options); + yield* useRemoteWorkspaceEffects(run); + const effect = createRemoteWorkspaceEffect( + run, + { type: "workspace", name: "one-owner" }, + function* (filesystem): Operation { + executed += 1; + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + return "ran"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + yield* withRemoteWorkspaceEffects(run, durableRun(workflow, { stream: run.journal })); + + // Everything the invocation read and everything it committed went to B. + // A answered the one snapshot this test asked it for directly, and + // nothing else: no root, no content, no staging, no commit. + expect(executed).toBe(1); + expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); + expect(workspaceIntents(b.commits)).toHaveLength(1); + // A was never asked for anything else either — no read, no staging and + // no commit reached it after the one snapshot above. + expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); + }); + + it("refuses before the split, not after: the other run's journal stays empty", function* () { + // The discriminator. Before this correction, B's transaction would enlist + // the Workspace while the publication appended through A's journal, and a + // refusal from B would leave A holding an event for a commit that never + // happened. The refusal has to come first. + const tree = yield* startingTree(); + const a = yield* harness(emptySnapshot, tree); + const b = yield* harness(emptySnapshot, tree); + b.refuse("command:stale-root"); - let executed = 0; - const run = yield* useRemoteRun(options); - yield* useRemoteWorkspaceEffects(run); + yield* scoped(function* () { + yield* useRemoteWorkspaceEffects(b.run); const effect = createRemoteWorkspaceEffect( - run, - { type: "workspace", name: "one-owner" }, + b.run, + { type: "workspace", name: "split" }, function* (filesystem): Operation { - executed += 1; yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); return "ran"; }, @@ -635,55 +654,16 @@ describe("the runner's Workspace coordinator", () => { function* workflow(): Workflow { yield effect; } - yield* withRemoteWorkspaceEffects(run, durableRun(workflow, { stream: run.journal })); - - // Everything the invocation read and everything it committed went to B. - // A answered the one snapshot this test asked it for directly, and - // nothing else: no root, no content, no staging, no commit. - expect(executed).toBe(1); - expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); - expect(workspaceIntents(b.commits)).toHaveLength(1); - // A was never asked for anything else either — no read, no staging and - // no commit reached it after the one snapshot above. - expect(a.requests.map((request) => request["command"])).toEqual(["mappings"]); + const raised = yield* trapped( + withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: a.run.journal })), + ); + expect(String(raised)).toContain("provenance"); }); - }); - - it("refuses before the split, not after: the other run's journal stays empty", function* () { - yield* scoped(function* () { - // The discriminator. Before this correction, B's transaction would enlist - // the Workspace while the publication appended through A's journal, and a - // refusal from B would leave A holding an event for a commit that never - // happened. The refusal has to come first. - const tree = yield* startingTree(); - const a = yield* harness(emptySnapshot, tree); - const b = yield* harness(emptySnapshot, tree); - b.refuse("command:stale-root"); - - yield* scoped(function* () { - yield* useRemoteWorkspaceEffects(b.run); - const effect = createRemoteWorkspaceEffect( - b.run, - { type: "workspace", name: "split" }, - function* (filesystem): Operation { - yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); - return "ran"; - }, - ); - function* workflow(): Workflow { - yield effect; - } - const raised = yield* trapped( - withRemoteWorkspaceEffects(b.run, durableRun(workflow, { stream: a.run.journal })), - ); - expect(String(raised)).toContain("provenance"); - }); - // No commit reached either owner, and A holds no event for work that - // happened somewhere else. - expect([a.commits, b.commits]).toEqual([[], []]); - expect(yielded(yield* a.run.journal.readAll())).toEqual([]); - }); + // No commit reached either owner, and A holds no event for work that + // happened somewhere else. + expect([a.commits, b.commits]).toEqual([[], []]); + expect(yielded(yield* a.run.journal.readAll())).toEqual([]); }); it("refuses a binding whose scope has closed", function* () { @@ -714,89 +694,81 @@ describe("the runner's Workspace coordinator", () => { }); it("refuses a Files capability kept past the invocation that owned it", function* () { - yield* scoped(function* () { - const held = yield* harness(); - let escaped: WorkspaceFilesystem | undefined; - let metadata: WorkspaceMetadata | undefined; - // deno-lint-ignore require-yield - yield* invocation(held, "captured", function* (filesystem, held): Operation { - escaped = filesystem; - metadata = held; - return "ran"; - }); - const wrote = yield* trapped(escaped?.writeFile("/LATE.md", "too late") ?? sleep(0)); - expect(String(wrote)).toContain("stale"); - expect(() => metadata?.insertRepository(repository())).toThrow(); + const held = yield* harness(); + let escaped: WorkspaceFilesystem | undefined; + let metadata: WorkspaceMetadata | undefined; + // deno-lint-ignore require-yield + yield* invocation(held, "captured", function* (filesystem, held): Operation { + escaped = filesystem; + metadata = held; + return "ran"; }); + const wrote = yield* trapped(escaped?.writeFile("/LATE.md", "too late") ?? sleep(0)); + expect(String(wrote)).toContain("stale"); + expect(() => metadata?.insertRepository(repository())).toThrow(); }); it("authorizes only paths beneath the attempt this invocation owns", function* () { - yield* scoped(function* () { - const held = yield* harness(); - const refused: string[] = []; - // deno-lint-ignore require-yield - const { raised } = yield* invocation(held, "escape", function* (filesystem): Operation { - return yield* (function* (): Operation { - for (const path of ["/../escaped", "/docs/../../escaped"]) { - const failure = yield* trapped(filesystem.writeFile(path, "outside")); - refused.push(String(failure)); - } - // Both ends of a rename: checking one would let the other leave. - refused.push(String(yield* trapped(filesystem.rename("/README.md", "/../moved")))); - // The Workspace root itself is a directory this invocation owns. - const entries = yield* filesystem.readdir("/"); - return entries.map((entry) => entry.name).toSorted(); - })(); - }); - expect(raised).toBe(undefined); - expect(refused).toHaveLength(3); - for (const failure of refused) { - expect(failure).toContain("outside the tree this invocation owns"); - } + const held = yield* harness(); + const refused: string[] = []; + // deno-lint-ignore require-yield + const { raised } = yield* invocation(held, "escape", function* (filesystem): Operation { + return yield* (function* (): Operation { + for (const path of ["/../escaped", "/docs/../../escaped"]) { + const failure = yield* trapped(filesystem.writeFile(path, "outside")); + refused.push(String(failure)); + } + // Both ends of a rename: checking one would let the other leave. + refused.push(String(yield* trapped(filesystem.rename("/README.md", "/../moved")))); + // The Workspace root itself is a directory this invocation owns. + const entries = yield* filesystem.readdir("/"); + return entries.map((entry) => entry.name).toSorted(); + })(); }); + expect(raised).toBe(undefined); + expect(refused).toHaveLength(3); + for (const failure of refused) { + expect(failure).toContain("outside the tree this invocation owns"); + } }); it("claims nothing when the answer to its commit is lost", function* () { - yield* scoped(function* () { - const held = yield* harness(); - held.lose(); - const { raised } = yield* invocation(held, "lost", function* (filesystem): Operation { - yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); - return "ran"; - }); - // Whether the owner committed is exactly what cannot be known from here. - // What must not happen is claiming it did. - expect(raised).not.toBe(undefined); - expect(String(raised)).not.toContain("command:"); - expect(workspaceIntents(held.commits)).toHaveLength(1); + const held = yield* harness(); + held.lose(); + const { raised } = yield* invocation(held, "lost", function* (filesystem): Operation { + yield* filesystem.writeFile("/NOTES.md", "written by the effect\n", 0o644); + return "ran"; }); + // Whether the owner committed is exactly what cannot be known from here. + // What must not happen is claiming it did. + expect(raised).not.toBe(undefined); + expect(String(raised)).not.toContain("command:"); + expect(workspaceIntents(held.commits)).toHaveLength(1); }); it("sends nothing and keeps nothing when the invocation is cancelled", function* () { - yield* scoped(function* () { - const held = yield* harness(); - yield* useRemoteWorkspaceEffects(held.run); - const effect = createRemoteWorkspaceEffect( - held.run, - { type: "workspace", name: "cancelled" }, - function* (filesystem): Operation { - yield* filesystem.writeFile("/SLOW.md", "in progress\n", 0o644); - yield* sleep(10_000); - return "never"; - }, - ); - function* workflow(): Workflow { - yield effect; - } - const task = yield* spawn(() => - withRemoteWorkspaceEffects(held.run, durableRun(workflow, { stream: held.run.journal })), - ); - yield* sleep(0); - yield* task.halt(); - // Cancellation is control flow: nothing was claimed, and nothing was sent. - expect(held.commits).toEqual([]); - expect(yielded(yield* held.run.journal.readAll())).toHaveLength(0); - }); + const held = yield* harness(); + yield* useRemoteWorkspaceEffects(held.run); + const effect = createRemoteWorkspaceEffect( + held.run, + { type: "workspace", name: "cancelled" }, + function* (filesystem): Operation { + yield* filesystem.writeFile("/SLOW.md", "in progress\n", 0o644); + yield* sleep(10_000); + return "never"; + }, + ); + function* workflow(): Workflow { + yield effect; + } + const task = yield* spawn(() => + withRemoteWorkspaceEffects(held.run, durableRun(workflow, { stream: held.run.journal })), + ); + yield* sleep(0); + yield* task.halt(); + // Cancellation is control flow: nothing was claimed, and nothing was sent. + expect(held.commits).toEqual([]); + expect(yielded(yield* held.run.journal.readAll())).toHaveLength(0); }); }); From fc5481b8107a9c00b730dbed9219959d8e437d20 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 19:59:09 -0300 Subject: [PATCH 79/81] =?UTF-8?q?=F0=9F=9A=9A=20Call=20the=20shared=20undo?= =?UTF-8?q?=20an=20undo,=20and=20leave=20savepoints=20to=20the=20host=20th?= =?UTF-8?q?at=20takes=20them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Tier DLC — DLC13` holds the provider-neutral Workspace coordination surface to a vocabulary no host owns, and a savepoint is one host's answer rather than the question: it is what SQLite offers, and naming a shared contract after it is how a neutral boundary quietly becomes one provider's. The guard caught two files doing exactly that — `src/workspace/savepoint.ts`, whose Api member, context identity and refusal text all said savepoint, and `src/remote/workspace.ts`, which implemented that member. So the contract now asks for what it wants: `Transaction.undoable(body)`, in `src/workspace/undoable.ts`. Nothing else changes. The local host still nests a real SQLite savepoint, and the module it installs from re-exports the operation under that name, so its own call sites read as they always did. The runner still restores its disposable attempt from the accepted root. No behavior, retained state, durable identity or authority moves; the failing name does. The guard only runs in the full battery, which is why the D-phase focused evidence never reached it. --- packages/workflow/src/deno/transaction.ts | 11 ++-- packages/workflow/src/remote/workspace.ts | 10 ++-- packages/workflow/src/workspace/savepoint.ts | 56 ------------------- packages/workflow/src/workspace/undoable.ts | 57 ++++++++++++++++++++ 4 files changed, 69 insertions(+), 65 deletions(-) delete mode 100644 packages/workflow/src/workspace/savepoint.ts create mode 100644 packages/workflow/src/workspace/undoable.ts diff --git a/packages/workflow/src/deno/transaction.ts b/packages/workflow/src/deno/transaction.ts index 43fe167c7..f46b9db55 100644 --- a/packages/workflow/src/deno/transaction.ts +++ b/packages/workflow/src/deno/transaction.ts @@ -61,14 +61,17 @@ export function* holdsTransactionOn(path: string): Operation { return false; } -import { Transaction } from "../workspace/savepoint.ts"; +import { Transaction } from "../workspace/undoable.ts"; +// The shared contract calls it an undo, because a savepoint is this host's own +// answer to it rather than the question. Inside this adapter it is a savepoint, +// which is what it is here. export { NoOpenTransactionError, - savepoint, Transaction, type TransactionApi, -} from "../workspace/savepoint.ts"; + undoable as savepoint, +} from "../workspace/undoable.ts"; /** What the open transaction installs so `savepoint()` can answer. */ export function useTransactionSavepoints( @@ -77,7 +80,7 @@ export function useTransactionSavepoints( ): Operation { return Transaction.around( { - *savepoint([body]: [Operation]): Operation { + *undoable([body]: [Operation]): Operation { return yield* savepoints.operation(transaction, body); }, }, diff --git a/packages/workflow/src/remote/workspace.ts b/packages/workflow/src/remote/workspace.ts index 390d61edf..f82a4ef9a 100644 --- a/packages/workflow/src/remote/workspace.ts +++ b/packages/workflow/src/remote/workspace.ts @@ -56,7 +56,7 @@ import type { WorkspaceAttachmentView } from "../workspace/effects.ts"; import { activeWorkspaceRoute, type WorkspaceRoute } from "./database.ts"; import { createInvocationMappings } from "./mappings.ts"; import type { RetainedMapping } from "./publication.ts"; -import { Transaction } from "../workspace/savepoint.ts"; +import { Transaction } from "../workspace/undoable.ts"; import { type Attempt, type Materialization, @@ -567,14 +567,14 @@ function* coordinateTransaction( let result: DurableResult; try { const value = yield* scoped(function* () { - // The savepoint the shared rules ask for when part of one mutation - // cannot be finished. The attempt is disposable by construction, so - // undoing that part is restoring the attempt from the accepted root — + // The undo the shared rules ask for when part of one mutation cannot + // be finished. The attempt is disposable by construction, so undoing + // that part is restoring the attempt from the accepted root — // correct because one effect performs one mutation, so nothing else in // this body has changed anything a caller still needs. yield* Transaction.around( { - *savepoint([body]: [Operation]): Operation { + *undoable([body]: [Operation]): Operation { try { return yield* body; } catch (error) { diff --git a/packages/workflow/src/workspace/savepoint.ts b/packages/workflow/src/workspace/savepoint.ts deleted file mode 100644 index ed909bfe6..000000000 --- a/packages/workflow/src/workspace/savepoint.ts +++ /dev/null @@ -1,56 +0,0 @@ -/** - * A savepoint inside one Workspace mutation, whichever host is performing it. - * - * One mutation may change several things and then find that it cannot finish - * one of them. What the shared Files and composition rules do about that is - * ask for the failed part to be undone and carry on — a write that created two - * parent directories and was then refused leaves neither behind, and the - * refusal is journaled against the Workspace as it was. - * - * How that is undone is the host's. The local host takes a real SQLite - * savepoint inside the transaction it is already in; the runner works in a - * disposable attempt and restores it from the accepted root. Both answer the - * same question, so the rules above them do not know which one they are - * running on — and a scope with no host answering at all refuses rather than - * performing work nothing can take back. - */ - -import { type Api, createApi } from "@effectionx/context-api"; -import type { Operation } from "effection"; -import { WorkflowTransactionError } from "../storage/errors.ts"; - -export interface TransactionApi { - /** - * Run `body` inside a savepoint, discarding its work if it fails. - * - * Answers with what the body answered. A failure rolls the savepoint back - * and propagates, leaving the surrounding transaction open and free to - * continue or to fail on its own terms. - */ - savepoint(body: Operation): Operation; -} - -/** No transaction is open in this scope, so there is nothing to nest inside. */ -export class NoOpenTransactionError extends WorkflowTransactionError { - override name = "NoOpenTransactionError"; - - constructor() { - super( - "a savepoint needs a transaction to be inside, and this scope is not inside one. " + - "Take savepoints within the body a transaction hands you.", - ); - } -} - -export const Transaction: Api = createApi( - "executablemd.workflow.deno.savepoint", - { - // deno-lint-ignore require-yield - *savepoint(_body: Operation): Operation { - throw new NoOpenTransactionError(); - }, - }, -); - -/** The savepoint operation, for whoever is inside a transaction. */ -export const savepoint: TransactionApi["savepoint"] = Transaction.operations.savepoint; diff --git a/packages/workflow/src/workspace/undoable.ts b/packages/workflow/src/workspace/undoable.ts new file mode 100644 index 000000000..4d89f5892 --- /dev/null +++ b/packages/workflow/src/workspace/undoable.ts @@ -0,0 +1,57 @@ +/** + * Undoing part of one Workspace mutation, whichever host is performing it. + * + * One mutation may change several things and then find that it cannot finish + * one of them. What the shared Files and composition rules do about that is + * ask for the failed part to be undone and carry on — a write that created two + * parent directories and was then refused leaves neither behind, and the + * refusal is journaled against the Workspace as it was. + * + * How that undo is performed is the host's, and its name here is deliberately + * not the local host's: that host nests a real SQLite savepoint inside the + * transaction it is already in, and the runner works in a disposable attempt + * and restores it from the accepted root. Both answer the same question, so + * the rules above them do not know which one they are running on — and a scope + * with no host answering at all refuses rather than performing work nothing can + * take back. + */ + +import { type Api, createApi } from "@effectionx/context-api"; +import type { Operation } from "effection"; +import { WorkflowTransactionError } from "../storage/errors.ts"; + +export interface TransactionApi { + /** + * Run `body` so that its work is discarded if it fails. + * + * Answers with what the body answered. A failure undoes what the body did + * and propagates, leaving the surrounding transaction open and free to + * continue or to fail on its own terms. + */ + undoable(body: Operation): Operation; +} + +/** No transaction is open in this scope, so there is nothing to undo inside. */ +export class NoOpenTransactionError extends WorkflowTransactionError { + override name = "NoOpenTransactionError"; + + constructor() { + super( + "undoing part of a mutation needs a transaction to be inside, and this scope is not " + + "inside one. Ask for one within the body a transaction hands you.", + ); + } +} + +export const Transaction: Api = createApi( + "executablemd.workflow.workspace.undoable", + { + // deno-lint-ignore require-yield + *undoable(_body: Operation): Operation { + throw new NoOpenTransactionError(); + }, + }, +); + +/** The undo operation, for whoever is inside a transaction. */ +export const undoable: TransactionApi["undoable"] = Transaction.operations.undoable; From c469b48c3f69db4b3719097277a1b897de803f34 Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 20:35:15 -0300 Subject: [PATCH 80/81] =?UTF-8?q?=F0=9F=94=A7=20Record=20which=20two=20rem?= =?UTF-8?q?ote=20suites=20belong=20to=20the=20Deno=20host?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Discovery puts every new `*.test.ts` in all three runtime jobs, so staying out of one is a deliberate act this file records. Two of the remote suites never made that act, and the Node job failed on them exactly as the file predicts: `createWorkflowRun` takes a run's advisory lock through `Deno.FsFile.tryLock`, and no Deno runtime is present. Both belong to the local host by subject rather than by accident. The interoperability suite exists to show the two capture implementations agree, so its fixture has to be produced by the local one — a real `node:sqlite` store, a run created through the Deno provider, a DOFS capture — before the runner materializes and recaptures it. The staged fork suite assembles its candidate through the Deno adapter's own connections, staging directory and lifecycle installation. Neither claim survives being made portable, and the portable halves of both are already elsewhere: `remote-fork.test.ts` for what a staged candidate must be over the no-acquisition plane, and every other Tier WRH suite for what the runner does. Node now reports 4442 passing and none failing. --- scripts/runtime-test-exclusions.ts | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index 798483202..86c52709f 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -491,6 +491,18 @@ const DENO_ONLY_TOOLING: RuntimeExclusion[] = [ "replays completed runs out of a real node:sqlite run store opened through @executablemd/workflow/deno, and reads the answers table it settles; Bun has no node:sqlite at all and Node 22 keeps it behind --experimental-sqlite. The host-neutral half — what a completed run replays on, and what its retained history is held to — runs on every runtime as packages/workflow/tests/replay-inputs.test.ts", issue: "https://github.com/taras/executable.md/issues/366", }, + { + path: "packages/workflow/tests/remote-interoperability.test.ts", + reason: + "the subject is that the two capture implementations agree, so the fixture has to be produced by the local one: it opens a real node:sqlite store, creates the run through the Deno provider — which takes that run's advisory lock through Deno.FsFile.tryLock — and captures it with the Deno DOFS provider before the runner materializes and recaptures it. Neither half is portable, and there is nothing left of the claim without both", + issue: "https://github.com/taras/executable.md/issues/698", + }, + { + path: "packages/workflow/tests/remote-staged-fork.test.ts", + reason: + "assembles the fork candidate through the Deno adapter itself — its connections, its staging directory and its remote lifecycle installation — so creating the run takes an advisory lock through Deno.FsFile.tryLock. What a staged candidate must be, independent of the adapter that builds one, is proved over the no-acquisition plane on every runtime as packages/workflow/tests/remote-fork.test.ts", + issue: "https://github.com/taras/executable.md/issues/698", + }, ]; /** From b6c127f9944c0414a661767adfdc94941ab04ccb Mon Sep 17 00:00:00 2001 From: Min Kim Date: Thu, 10 Sep 2026 20:42:44 -0300 Subject: [PATCH 81/81] =?UTF-8?q?=F0=9F=94=A7=20Run=20the=20workerd=20suit?= =?UTF-8?q?e=20through=20the=20vitest=20the=20plugin=20resolves?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `deno task test:cloudflare` failed with "Vitest failed to find the runner" and collected no tests at all. The cause is two installs: the Workers plugin imports `vitest` and resolves pnpm's copy, while `node_modules/.bin/vitest` is whatever wrote that link last — and a `deno task` re-materializes `node_modules/.bin` into Deno's layout before it runs anything, so in any tree that `deno task setup` prepared, the CLI and the plugin are different vitest instances and the pool never takes over. Naming the module instead of the link settles it: `node ./node_modules/vitest/vitest.mjs` is the same copy the plugin resolves, whichever installer touched `.bin` last. CI already ran this script the one way that happened to work — `pnpm test:cloudflare`, with no Deno layout in the job at all — and still does; what changes is that the alias beside it works too, on the tree a contributor actually has. `deno task test:cloudflare`: 14 files, 196 tests, 0 failures. --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 4a867b228..c2a0f0e51 100644 --- a/package.json +++ b/package.json @@ -78,7 +78,7 @@ "test:deno": "deno task test", "lint": "oxlint -c .oxlintrc.json --ignore-pattern 'scripts/tests/fixtures/**' --ignore-pattern '**/npm/**' --ignore-pattern 'packages/workflow/vendor/cloudflare-computer-dofs/**' --ignore-pattern 'packages/acp/vendor/acpx/**' packages scripts .reviews/components && oxfmt --check packages scripts .reviews/components/*.ts", "fmt": "oxfmt --write packages scripts .reviews/components/*.ts", - "test:cloudflare": "vitest run --config vitest.config.ts", + "test:cloudflare": "node ./node_modules/vitest/vitest.mjs run --config vitest.config.ts", "check:cloudflare": "tsc -p packages/workflow/tsconfig.cloudflare.json" }, "workspaces": [