Skip to content

Read the conversation that asked before taking a lock on the one being answered in - #305

Open
kevin9327 wants to merge 1 commit into
CopilotKit:mainfrom
kevin9327:fix/handoff-history-before-lock
Open

Read the conversation that asked before taking a lock on the one being answered in#305
kevin9327 wants to merge 1 commit into
CopilotKit:mainfrom
kevin9327:fix/handoff-history-before-lock

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

What this changes

server/src/agents/handoff-delivery.ts takes the run lock on the conversation the answer lands in,
and then reads the history of the conversation that asked — a different thread, which the lock
never protected:

  • :212 const held = await lock.acquire({ threadId: where.threadId, … })
  • :248 await history({ threadId: work.threadId, actorId: work.actorId })
  • :265 try { … } finally { clearInterval(heartbeat); await lock.release(…) }

The read at :248 is the one await in the delivery that sits between the acquire and the finally
that gives the lock back.

historyOrEmpty (server/src/copilot.ts:953-963) answers a missing thread with nothing and rethrows
everything else, deliberately — "A 500 from the platform means an outage or a bad key". So one 500,
one expired key, one dropped connection, and the delivery throws with the lock still held:

  1. The hop rejects and goes back on the queue with a 60 s delay
    (handoff-runner.ts:393-399), which is correct.
  2. The lock on where.threadId — the addressed Bot's own conversation with that person — is never
    released and never renewed (the heartbeat is only armed at :261, after the read). It sits until
    the platform's TTL, THREAD_LOCK_TTL_SECONDS = 120 (copilot.ts:993). For those two minutes the
    person cannot start a run in that conversation.
  3. The retry a minute later collides with the lock the hop is still holding itself, gets null
    from acquire, throws "is busy with another run" and spends one of its five attempts on it.

The module already argues this case for the other direction. The finally at :348-359 says:

Given back whatever happened. Left held, the conversation is unusable by anybody until the lock
expires: the person cannot ask a follow-up and the next hop is refused, which turns one failed
delivery into a conversation that has stopped working.

That is exactly what a failed history read produces — it just happens one line above where the
finally starts.

The fix

Hoist the read above the acquire. It is a read of another conversation, so the lock was doing nothing
for it, and moving it earlier also shortens the window the lock is held on the happy path. Nothing
else moves: the ask is still appended with the platform's own run id, which is only known after the
lock.

Where it runs

  • New state that outlives a request? None. One local moved earlier in the same function.
  • What happens on the second replica? Better, and this is the point. The lock is the
    platform's, shared by every replica; a replica that leaked one made the conversation unusable
    from all of them for the TTL, and the hop's own retry — which any replica may claim — walked
    into it. After this, a failed read leaves nothing behind for another replica to trip over.
  • Anything serialised? Unchanged. The lock is still taken NX through the platform, and is
    still the thing that serialises runs in a conversation. What changed is that it is no longer
    held across an operation that can throw outside its finally.
  • Anything fanned out to a browser? No.
  • New listener, port, or schedule? None.

Boundary and audit

  • Every acting call still goes through the gateway: unchanged.
  • New refusals and new failures each write a row. No new outcome — the hop already rejects and is
    retried, and agent.handoff_retried / agent.handoff_failed are written by the runner exactly
    as before. This changes only what is left behind on the way out.
  • Nothing new is trusted from the client.

Changelog

  • CHANGELOG.md, under Unreleased.

Proof

server/tests/agent-handoff-delivery.test.ts gains a historyError option to the existing stub
harness and one case: a history read that throws, asserting the whole lock sequence is empty. It is
red on main and green here.

Against main's handoff-delivery.ts with the new test in place:

143 |     expect(lockCalls).toEqual([]);
error: expect(received).toEqual(expected)
- []
+ [ "acquire" ]
(fail) turning a hop into a turn > a history the platform will not hand back leaves no lock behind
 17 pass, 1 fail

With the change:

bun test server/tests/agent-handoff-delivery.test.ts server/tests/agent-handoff-runner.test.ts \
         server/tests/agent-handoff.test.ts server/tests/agent-handoff-tool.test.ts
                       -> 60 pass, 0 fail (113 expect() calls)
bun run format:check   -> Checked 487 files. No fixes applied.
bun run lint           -> Checked 490 files. No fixes applied.
bun run typecheck      -> app / server / worker all exit 0

The sequence is asserted rather than the absence of a release, because a delivery that acquired and
then released would also be correct and the assertion should say which of the two happened.

…g answered in

A delivery takes the run lock on the addressed Bot's conversation, and then reads the history of the
conversation that ASKED — a different thread, which the lock never protected. Between the two sits
the only `await` in the delivery that is not inside the `try` whose `finally` gives the lock back.

`historyOrEmpty` answers a missing thread with nothing and rethrows everything else on purpose: a 500
from the platform means an outage or a bad key, not an empty conversation. So one 500, one expired
key, one dropped connection, and the lock is held by a delivery that has already gone. Nothing renews
it and nothing releases it, so it sits until the platform's own TTL — about two minutes — during
which the person cannot start a run in that conversation at all.

The hop's retry a minute later then collides with the lock it is still holding itself, gets null from
`acquire`, reports the conversation as busy and spends one of its five attempts on that. So the same
outage costs the person a working conversation and costs the hop an attempt, for a read that did not
need the lock in the first place.

Hoisted above the acquire. Nothing else moves: the ask is still appended with the platform's own run
id, which is only known after the lock, and the history is of another conversation so reading it a
moment earlier changes nothing about what the addressed Bot is shown.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

guidovizoso added a commit that referenced this pull request Aug 31, 2026
The history read is the one call in a delivery that throws on a platform
error, and it sat after the lock was acquired but before the try whose finally
gives the lock back: a 500 from the platform leaked the lock for its full TTL.
On a forward hop that lock is a scratch thread and the leak costs nothing; on
a relay it is the asking conversation itself, so the person could not type for
two minutes, the retry a minute later collided with the hop's own leftover
hold and spent an attempt on it, and a relay that ran out of attempts vanished
without a notice.

The read is of the conversation that asked, which the lock — taken on the
conversation being answered in — never protected. Hoisted above the acquire it
fails before anything is held, and the happy path holds the lock for less of
the turn too. Raised by kevin9327 on #290, who had the same reordering up as
#305 against main; a test now pins the ordering.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant