A refusal could be thrown away while an earlier one was being written - #767
jorgemanrubia wants to merge 2 commits into
Conversation
A refusal Codex wrote could be thrown away: never recorded anywhere, not merely missing from a turn's result. Writing a refusal to the ledger is allowed ten seconds and ran on the goroutine reading the worker's output, while the read end of that pipe was released a couple of seconds after the worker was terminated, whatever that goroutine was doing. Closing it discards what the worker wrote and nobody has parsed. No bound could be put on the reading while those shared a goroutine. Counted in wall time it expires inside a ledger write, and the reader comes back to find its own pipe closed with the worker's next refusal still in it — the loss this exists to end. Counted in time spent on the pipe, a descendant outside the worker's group writing without stop keeps the pipe busy and holds the session's shutdown open for as long as it lives. Those are not two bugs to balance; they are one structure making the bound impossible to place. So the reader only reads, and a scribe writes. Everything the reader does between reads is microseconds now, which is what lets the pipe's drain bound be a plain clock and mean what it says. The pipe belongs to the reader while it is reading: termination asks it to stop rather than closing the descriptor under it, the deadline on the pipe has one owner, and the descriptor is released when the reader says it is through. The queue between them is bounded, and it never drops. A refusal handed over when it is full waits, which is backpressure onto the worker — what a slow ledger already did to this driver when both ran on one goroutine, so it is not a new way to fail. Dropping would be the old one wearing a new hat. Its size is derived rather than chosen: a drain can only produce what a pipe and a scanner buffer can hold, which is fewer refusals than this, so the drain cannot block on it. The scribe is drained before the session's updates close, which is where the dispatcher anchors "every refusal the driver read has been through the recorder", and before Close returns, so a caller that closes and then asks the ledger is not asking early. A refusal read after the scribe has gone — by an ending that is not the reader's — is written by whoever read it rather than lost. Held past the grace and the drain budget together, with a refusal in the pipe the whole time, this fails 10 runs of 10 on main and passes 5 of 5 here. The test's second refusal is said only once the ledger is holding the first, so it is in the pipe and nowhere else.
There was a problem hiding this comment.
🟡 Changes recommended
The asynchronous handoff weakens refusal durability and does not reliably bound shutdown.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Separates Codex output reading from ledger writes to prevent refusals being discarded during shutdown.
Changes:
- Adds reader-owned pipe draining and shutdown signaling.
- Introduces a bounded refusal queue and scribe goroutine.
- Adds regression coverage for slow ledger writes.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
File summaries
| File | Description |
|---|---|
| internal/connector/driver/worker.go | Adds reader-controlled pipe lifetime and drain deadlines. |
| internal/connector/driver/worker_other.go | Adds platform stub methods for the reader lifecycle. |
| internal/connector/driver/codex/codex.go | Moves refusal recording to an asynchronous scribe. |
| internal/connector/driver/codex/codex_test.go | Tests slow-ledger refusal preservation. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
A refusal Codex wrote could be thrown away: recorded nowhere, not merely missing from a turn's result. The read end of the worker's output was released a couple of seconds after the worker was terminated, whatever the goroutine reading it was doing, and closing it discards what the worker wrote and nobody has parsed. That goroutine is allowed to be slow — writing a refusal to the ledger is given ten seconds, and the driver's own contract puts that write on the reader, before the update for it is emitted — so the clock ran out inside a write and took the pipe away with the worker's next refusal still in it. The first shape I tried for this moved the ledger write to a goroutine of its own so the reader would only ever read. It does not survive contact with the contract it would have to break: a refusal held only in a queue is not recorded, and publishing the update before the write lands is the thing the contract exists to forbid. It also buys nothing — the same writes still have to finish before the session's updates close, so the shutdown is no shorter for having moved them. That shape is abandoned. What was wrong was the unit. A ledger write moves no bytes, so a drain counted in bytes cannot be spent by one. The reader now stops after taking more from the pipe than the worker could possibly have left there — a worker is ended before its reader is asked, so what remains of its output is at most one pipe's worth, and the budget is far more than a pipe holds. Past that, what is still arriving can only be a descendant that left the worker's group and is writing without end, which is the one case that has no other ending. The pipe belongs to the reader while it is reading, so termination asks it to stop rather than closing the descriptor under it, the deadline on the pipe has one owner, and the descriptor is released when the reader says it is through. A cancel asks too: ending the worker is not the end of its output when a descendant holds the pipe open, and a canceled turn only the reader can finish would never be finished. Held past every clock in this path, with a refusal sitting in the pipe the whole time, this fails 10 runs of 10 on main. A stop that closes the pipe loses what is in it, 10 of 10; a drain counted in time loses the line that arrives while the reader works, 10 of 10; and without a budget a reader follows a descendant forever, 3 of 3.
There was a problem hiding this comment.
🟡 Changes recommended
The byte-based drain can block shutdown for hours and may truncate output from enlarged pipes.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
internal/connector/driver/codex/codex.go:338
- The PR description still presents a reader/scribe split, bounded queue, and clock-based drain, but this revision intentionally keeps
RecordRefusalsynchronous on the reader and implements a byte budget with no scribe or queue. Please update the description and evidence so the architecture being reviewed matches this implementation.
// The worker's output is released when this session has read it, not on
// a clock: what the reader does between reads — writing a refusal to the
// ledger — is allowed to take longer than any such clock.
worker.ReadingDone(s.readerEnd)
- Files reviewed: 5/5 changed files
- Comments generated: 2
- Review effort level: Balanced
| // arriving can only be a descendant that left the worker's group and is | ||
| // writing without end — the one case that has no other ending, and one the | ||
| // connector had already decided not to wait for. | ||
| const drainBudget = 1 << 20 |
| if o.drained >= drainBudget { | ||
| // More than the worker could have left: what is still | ||
| // arriving is a descendant's, and no longer waited for. | ||
| return 0, io.EOF | ||
| } | ||
| window = drainWindow |
A refusal Codex wrote could be thrown away — recorded nowhere, not merely missing from a turn's result. The read end of the worker's output was released a couple of seconds after the worker was terminated, whatever the goroutine reading it was doing, and closing it discards whatever the worker wrote that nobody has parsed. That goroutine is allowed to be slow: writing a refusal to the ledger is given ten seconds, and the driver's own contract puts that write on the reader, before the update for it is emitted. So the clock ran out inside a write and took the pipe away with the worker's next refusal still in it.
Carded as The worker's output pipe is closed on a clock, and a refusal can be lost with it, after being found and taken back out of An unsafe session could report fewer refusals than it made.
What was wrong was the unit, not the goroutine
The card proposed separating the pipe read from the ledger write so the reader would only ever read. I built that first and it does not survive contact with the contract it would have to break.
driver.gosays a refusal "is never held only in a session's memory", and that "the recorder is called before an update is emitted, so a worker that exits between a refusal and its result has already recorded it." A queue in memory is what that forbids; publishing the update before the write lands is the window it exists to close. It also buys nothing: the same writes still have to finish before the session's updates close — where the dispatcher anchors "every refusal the driver read has been through the recorder" — so moving them to another goroutine only moves which goroutine waits.A ledger write moves no bytes. So the drain is counted in bytes, and the ten seconds a write may take cannot spend it. The write stays where the contract puts it.
The budget is derived, and against a bound that does not involve the scanner: a worker is ended before its reader is asked, so what remains of its output is at most one pipe's worth, and the budget is far more than a pipe holds on any system this runs on. The worker's own last lines are therefore never the ones cut off. Past the budget, what is still arriving can only be a descendant that left the worker's group and is writing without end — the one case that has no other ending, and one the connector had already decided not to wait for.
The pipe belongs to the reader while it is reading
Termination asks the reader to stop rather than closing the descriptor under it. The deadline on the pipe has one owner, so nothing can cut short a window the reader opened for itself. The descriptor is released when the reader says it is through, rather than on a timer that cannot know. And a cancel asks too: ending the worker is not the end of its output when a descendant holds the pipe open, so a canceled turn that only the reader can finish would otherwise never be finished.
Every path, and what happens when the budget runs out
What can end a turn is unchanged by this. The reader's own ending,
turn.completed,turn.failed, and the unsafe or canceled endings reached from any of them are all the reader, after it has parsed the stream to that point. The policy check claims the turn before ending the worker, then waits for the reader. The prompt's writer can still finish a turn ahead of the reader, in both its branches, and needs a worker that stopped reading its input while still writing.Where a refusal is recorded is also unchanged, which is the point: on the reader, at the moment it is read, before the update for it is emitted.
Evidence
The test makes its ordering rather than waiting for one: the second refusal is said only once the ledger is holding the first, so it is in the pipe and nowhere else — the reader cannot have taken it into its own buffer, because it was not there to take. The fake fails loudly if that trigger never comes rather than saying its events anyway. No assertion runs on a helper goroutine.
main, the refusal reaching neither the ledger nor the resultgo build ./...,go vet ./...,gofmt -l .,golangci-lint run --build-tags dev ./...andBASECAMP_NO_KEYRING=1 go test -tags dev ./...are all clean, as isgo test -tags dev -race ./internal/connector/..., and so areGOOS=windowsandGOOS=darwinbuilds.