Skip to content

A worker that needs its task token to finish starting cannot be dispatched - #760

Merged
jorgemanrubia merged 1 commit into
mainfrom
acp-arming-before-handshake
Sep 18, 2026
Merged

jorgemanrubia merged 1 commit into
mainfrom
acp-arming-before-handshake

Conversation

@jorgemanrubia

Copy link
Copy Markdown
Member

An agent that will not finish starting until its MCP servers have connected cannot be dispatched. Its server is waiting for the task token, and the connector will not hand the token over until the start it is blocking has returned. Both sides wait out a timeout — the bridge's 30-second dial, or the driver's two-minute handshake — and every dispatch through such an agent fails.

Nothing in the connector prevented this; only adapter behaviour did. It was hit for real while the Codex harness row was being built: a fake that bound at session/new made every ACP dispatch fail after 30 seconds.

Originally tracked in The ACP handshake needs a token the socket is not yet armed to hand over, which named the hazard rather than fixing it, because changing dispatch ordering on a guess is worse than documenting it. This is the fix, with the guess retired: the deadlock is a failing test first.

The shape of it

The token socket accepts nothing until it is told the worker's process group, and it was told only once the driver had handed back a ready session. For the ACP driver that is the wrong side of a long wait: the whole handshake — including the adapter's own /mcp read-back, which is a real prompt turn — runs inside the start.

So a worker is now announced when its process exists rather than when its session is ready. Every driver starts its agent through one function, so that is where the announcement is made and no driver can forget it. A new driver invariant says so.

What moving it widens, since something did rely on the old order

The token window starts when the socket is armed, so the handshake is inside it now. Three reasons that is right rather than merely tolerable:

  • The concern this window was written against was a launcher that takes its time, and that is untouched — the arming is after the fork returns, so a slow sandbox launcher still spends none of it.
  • It is no smaller for the case it exists for. An agent starts its MCP servers during its handshake, so the bridge's connection is already waiting in the listener's backlog and is accepted the instant the socket is armed.
  • Stretching it to cover both would be the worse fix. The socket is armed for the worker's whole process group — the agent's own tree — so a longer armed window is a longer window in which the agent's own tools could ask for the token instead of its MCP server.

The window now means what its own documentation always said it meant: once the worker exists, not once its session is ready.

The compatibility check measured nothing about this

It armed the socket right after the start returned, which is what it did under either ordering, so it passed under both and could not have caught a regression. It now arms where the dispatcher arms, and fails outright on the old order.

That check needs the pinned adapters and model quota, so it cannot prove its own sensitivity in CI. Its verdict is therefore a small object driven through both orderings against the real driver in the ordinary suite: the new one passes, the old one fails with "after NewSession returned", and removing the announcement from production turns the first half red.

Proof

Every test here was watched failing before it passed. The deadlock reproduces in about five seconds with the dispatcher's wiring removed, and in milliseconds with it.

Copilot AI balanced review requested due to automatic review settings September 18, 2026 14:46
@github-actions github-actions Bot added commands CLI command implementations tests Tests (unit and e2e) labels Sep 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The ordering fix is centralized across all spawn drivers and covered by focused regression tests.

Pull request overview

Moves token-socket arming to worker creation, preventing MCP-dependent handshakes from deadlocking.

Changes:

  • Adds SessionConfig.Started and invokes it centrally from StartWorker.
  • Arms the dispatcher’s token socket before driver handshakes.
  • Adds regression and ordering tests across dispatcher and ACP paths.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File summaries
File Description
internal/connector/tokensocket.go Documents the revised token window.
internal/connector/recovery_acp_test.go Updates recovery-test ordering rationale.
internal/connector/driver/worker.go Announces workers immediately after startup.
internal/connector/driver/worker_other.go Updates the non-Unix signature.
internal/connector/driver/drivertest/drivertest.go Adapts test worker startup.
internal/connector/driver/driver.go Defines the new driver invariant and callback.
internal/connector/driver/driver_test.go Updates worker tests for SessionConfig.
internal/connector/driver/codex/codex.go Passes full configuration to StartWorker.
internal/connector/driver/claude/claude.go Passes full configuration to StartWorker.
internal/connector/driver/announce_test.go Tests centralized worker announcements.
internal/connector/driver/acp/compat_test.go Arms the compatibility socket during startup.
internal/connector/driver/acp/arming_test.go Tests ACP handshake ordering.
internal/connector/driver/acp/acp.go Passes full configuration to StartWorker.
internal/connector/dispatcher.go Arms sockets through the startup callback.
internal/connector/dispatcher_test.go Models pre-session handshakes in the fake driver.
internal/connector/dispatcher_arming_test.go Reproduces and verifies the deadlock fix.
internal/commands/connect_worker_unix_test.go Adapts worker test setup.
Review details
  • Files reviewed: 17/17 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…shake returns

The socket accepts nothing until AllowGroup names the worker's process group,
and the connector named it only once Driver.NewSession had returned. For the
ACP driver the whole handshake runs inside NewSession, the adapter's own /mcp
read-back — a real prompt turn — included. An adapter that will not finish
that handshake until its MCP servers have connected therefore waits for a
token the connector cannot hand over until the handshake blocking it has
finished, and both sides sit there until the bridge's 30-second dial or the
driver's two-minute handshake runs out. A fake that bound at session/new hit
exactly this while the Codex harness row was being built.

SessionConfig.Started announces the worker's process the moment the fork
returns. It is called from driver.StartWorker, the one place every driver
starts its agent, so no driver can forget it; the dispatcher wires it to
AllowGroup. Written up as driver invariant 7. The dispatcher still arms on
Session.Process() afterwards as a backstop for a driver that announces
nothing — AllowGroup takes only the first group it is given, so where the
driver did announce, that changes nothing.

What the earlier arming widens is the token window, and that is the one thing
that relied on the old order: it starts at AllowGroup, so the handshake is
inside it now. The card-23 review's concern was a launcher that takes its
time, which is preserved — the arming is after the fork. The window is no
smaller for the case it is for, since an agent starts its MCP servers during
its handshake and their connection is already waiting in the listener's
backlog. And it is deliberately not stretched to cover both: the socket is
armed for the worker's whole process group, which is the agent's own tree, so
a longer armed window is a longer window in which the agent's tools could ask
for the token instead of its MCP server. DefaultTokenWindow now says that
rather than the old order.

The compat check armed on Session.Process() right after NewSession under
either ordering, so it passed under both and would not have caught a
regression. It arms from cfg.Started now, where the dispatcher does, and
fails outright on the old order. Since it cannot run without the pinned
adapters and model quota, its verdict is a small object driven through both
orderings against the real driver in the ordinary suite
(TestTheArmingCheckTellsTheTwoOrderingsApart).
@jorgemanrubia
jorgemanrubia force-pushed the acp-arming-before-handshake branch from a1ed6ff to 21df8f8 Compare September 18, 2026 15:00
@jorgemanrubia

Copy link
Copy Markdown
Member Author

🤖 TestRateLimiterWaitNeverSleepsPastTheDeadline failed on the first run of this branch. It is not this change, and it is not one of the four known load-dependent failures either — it is a fifth, and it is on main.

It cannot see this change. The test-dependency closure of internal/resilience is internal/resilience and the standard library — no internal package at all, so nothing here reaches it. The package is also byte-identical to main (git diff origin/main -- internal/resilience is empty).

Reproduced on pristine main. A detached worktree at 63aba329, under CPU load (160 spinners on 32 cores), -count=40, failing on three consecutive runs with exactly the message CI printed:

--- FAIL: TestRateLimiterWaitNeverSleepsPastTheDeadline (0.23s)
    Error: Received unexpected error:
           Too many requests (client limit 10/s); waited 0s

Idle, the same command passes 30 and 40 iterations without a failure.

Why it is load-sensitive. The test sets a 200ms server block against a 220ms budget — 20ms of slack over a 200ms sleep. waitSince sleeps the block out and loops; if the wake-up is late, remaining <= 0 on the next pass and it returns the client-limit gate error rather than the token. So any scheduler delay over 20ms fails it, and the error is the deadline one, not the elapsed-time assertion the test's own name is about.

Filing it here rather than fixing it: it is not in this change's territory, and the fix is someone's call about whether that budget should be a real margin or the clock should be injected the way jitter already is.

The other failure on that run, TestAnUnsafeSessionStillReportsItsRefusals, was fixed by Four driver failures on CI and this branch is now rebased onto it.

@jorgemanrubia
jorgemanrubia merged commit b67f1d3 into main Sep 18, 2026
25 checks passed
@jorgemanrubia
jorgemanrubia deleted the acp-arming-before-handshake branch September 18, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commands CLI command implementations tests Tests (unit and e2e)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants