Read the conversation that asked before taking a lock on the one being answered in - #305
Open
kevin9327 wants to merge 1 commit into
Open
Read the conversation that asked before taking a lock on the one being answered in#305kevin9327 wants to merge 1 commit into
kevin9327 wants to merge 1 commit into
Conversation
…g answered in A delivery takes the run lock on the addressed Bot's conversation, and then reads the history of the conversation that ASKED — a different thread, which the lock never protected. Between the two sits the only `await` in the delivery that is not inside the `try` whose `finally` gives the lock back. `historyOrEmpty` answers a missing thread with nothing and rethrows everything else on purpose: a 500 from the platform means an outage or a bad key, not an empty conversation. So one 500, one expired key, one dropped connection, and the lock is held by a delivery that has already gone. Nothing renews it and nothing releases it, so it sits until the platform's own TTL — about two minutes — during which the person cannot start a run in that conversation at all. The hop's retry a minute later then collides with the lock it is still holding itself, gets null from `acquire`, reports the conversation as busy and spends one of its five attempts on that. So the same outage costs the person a working conversation and costs the hop an attempt, for a read that did not need the lock in the first place. Hoisted above the acquire. Nothing else moves: the ask is still appended with the platform's own run id, which is only known after the lock, and the history is of another conversation so reading it a moment earlier changes nothing about what the addressed Bot is shown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevin9327
requested review from
MikeRyanDev,
davidmckayv,
guidovizoso and
tylerslaton
as code owners
August 31, 2026 11:27
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
guidovizoso
added a commit
that referenced
this pull request
Aug 31, 2026
The history read is the one call in a delivery that throws on a platform error, and it sat after the lock was acquired but before the try whose finally gives the lock back: a 500 from the platform leaked the lock for its full TTL. On a forward hop that lock is a scratch thread and the leak costs nothing; on a relay it is the asking conversation itself, so the person could not type for two minutes, the retry a minute later collided with the hop's own leftover hold and spent an attempt on it, and a relay that ran out of attempts vanished without a notice. The read is of the conversation that asked, which the lock — taken on the conversation being answered in — never protected. Hoisted above the acquire it fails before anything is held, and the happy path holds the lock for less of the turn too. Raised by kevin9327 on #290, who had the same reordering up as #305 against main; a test now pins the ordering.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
server/src/agents/handoff-delivery.tstakes the run lock on the conversation the answer lands in,and then reads the history of the conversation that asked — a different thread, which the lock
never protected:
:212const held = await lock.acquire({ threadId: where.threadId, … }):248await history({ threadId: work.threadId, actorId: work.actorId }):265try { … } finally { clearInterval(heartbeat); await lock.release(…) }The read at
:248is the oneawaitin the delivery that sits between the acquire and thefinallythat gives the lock back.
historyOrEmpty(server/src/copilot.ts:953-963) answers a missing thread with nothing and rethrowseverything else, deliberately — "A 500 from the platform means an outage or a bad key". So one 500,
one expired key, one dropped connection, and the delivery throws with the lock still held:
(
handoff-runner.ts:393-399), which is correct.where.threadId— the addressed Bot's own conversation with that person — is neverreleased and never renewed (the heartbeat is only armed at
:261, after the read). It sits untilthe platform's TTL,
THREAD_LOCK_TTL_SECONDS = 120(copilot.ts:993). For those two minutes theperson cannot start a run in that conversation.
nullfrom
acquire, throws "is busy with another run" and spends one of its five attempts on it.The module already argues this case for the other direction. The
finallyat:348-359says:That is exactly what a failed history read produces — it just happens one line above where the
finallystarts.The fix
Hoist the read above the acquire. It is a read of another conversation, so the lock was doing nothing
for it, and moving it earlier also shortens the window the lock is held on the happy path. Nothing
else moves: the ask is still appended with the platform's own run id, which is only known after the
lock.
Where it runs
platform's, shared by every replica; a replica that leaked one made the conversation unusable
from all of them for the TTL, and the hop's own retry — which any replica may claim — walked
into it. After this, a failed read leaves nothing behind for another replica to trip over.
NXthrough the platform, and isstill the thing that serialises runs in a conversation. What changed is that it is no longer
held across an operation that can throw outside its
finally.Boundary and audit
retried, and
agent.handoff_retried/agent.handoff_failedare written by the runner exactlyas before. This changes only what is left behind on the way out.
Changelog
CHANGELOG.md, underUnreleased.Proof
server/tests/agent-handoff-delivery.test.tsgains ahistoryErroroption to the existing stubharness and one case: a history read that throws, asserting the whole lock sequence is empty. It is
red on
mainand green here.Against
main'shandoff-delivery.tswith the new test in place:With the change:
The sequence is asserted rather than the absence of a release, because a delivery that acquired and
then released would also be correct and the assertion should say which of the two happened.