Skip to content

A refusal could be thrown away while an earlier one was being written - #767

Draft
jorgemanrubia wants to merge 2 commits into
mainfrom
reader-and-ledger-are-two-goroutines
Draft

jorgemanrubia wants to merge 2 commits into
mainfrom
reader-and-ledger-are-two-goroutines

Conversation

@jorgemanrubia

@jorgemanrubia jorgemanrubia commented Sep 18, 2026

Copy link
Copy Markdown
Member

A refusal Codex wrote could be thrown away — recorded nowhere, not merely missing from a turn's result. The read end of the worker's output was released a couple of seconds after the worker was terminated, whatever the goroutine reading it was doing, and closing it discards whatever the worker wrote that nobody has parsed. That goroutine is allowed to be slow: writing a refusal to the ledger is given ten seconds, and the driver's own contract puts that write on the reader, before the update for it is emitted. So the clock ran out inside a write and took the pipe away with the worker's next refusal still in it.

Carded as The worker's output pipe is closed on a clock, and a refusal can be lost with it, after being found and taken back out of An unsafe session could report fewer refusals than it made.

What was wrong was the unit, not the goroutine

The card proposed separating the pipe read from the ledger write so the reader would only ever read. I built that first and it does not survive contact with the contract it would have to break. driver.go says a refusal "is never held only in a session's memory", and that "the recorder is called before an update is emitted, so a worker that exits between a refusal and its result has already recorded it." A queue in memory is what that forbids; publishing the update before the write lands is the window it exists to close. It also buys nothing: the same writes still have to finish before the session's updates close — where the dispatcher anchors "every refusal the driver read has been through the recorder" — so moving them to another goroutine only moves which goroutine waits.

A ledger write moves no bytes. So the drain is counted in bytes, and the ten seconds a write may take cannot spend it. The write stays where the contract puts it.

The budget is derived, and against a bound that does not involve the scanner: a worker is ended before its reader is asked, so what remains of its output is at most one pipe's worth, and the budget is far more than a pipe holds on any system this runs on. The worker's own last lines are therefore never the ones cut off. Past the budget, what is still arriving can only be a descendant that left the worker's group and is writing without end — the one case that has no other ending, and one the connector had already decided not to wait for.

The pipe belongs to the reader while it is reading

Termination asks the reader to stop rather than closing the descriptor under it. The deadline on the pipe has one owner, so nothing can cut short a window the reader opened for itself. The descriptor is released when the reader says it is through, rather than on a timer that cannot know. And a cancel asks too: ending the worker is not the end of its output when a descendant holds the pipe open, so a canceled turn that only the reader can finish would otherwise never be finished.

Every path, and what happens when the budget runs out

path what it does on expiry
exec, when the process exits or after its wait delay closes the write end the reader drains what is buffered, then sees the end of file — nothing lost
asking the reader to stop sets one field; touches neither the descriptor nor its deadline
the reader, once asked reads until a window of its own finds the pipe empty, or the drain budget of bytes runs out what a descendant is still writing is abandoned; never the worker's own output, which was all in the pipe before the asking and is far smaller than the budget
the Worker's release after termination closes the read end once the reader says it is through; the old clock only where no reader registered
closing the output outright closes the read end at once, discarding what is in it now only for releasing a descriptor nobody is reading

What can end a turn is unchanged by this. The reader's own ending, turn.completed, turn.failed, and the unsafe or canceled endings reached from any of them are all the reader, after it has parsed the stream to that point. The policy check claims the turn before ending the worker, then waits for the reader. The prompt's writer can still finish a turn ahead of the reader, in both its branches, and needs a worker that stopped reading its input while still writing.

Where a refusal is recorded is also unchanged, which is the point: on the reader, at the moment it is read, before the update for it is emitted.

Evidence

The test makes its ordering rather than waiting for one: the second refusal is said only once the ledger is holding the first, so it is in the pipe and nowhere else — the reader cannot have taken it into its own buffer, because it was not there to take. The fake fails loudly if that trigger never comes rather than saying its events anyway. No assertion runs on a helper goroutine.

without the change with it
a ledger held past every clock in this path, with a refusal in the pipe 10 failures in 10 on main, the refusal reaching neither the ledger nor the result 0 in 5
a stop that closes the pipe 10 in 10 0 in 5
a drain counted in time rather than bytes 10 in 10 0 in 5
no budget at all, against a descendant writing without end never ends, 3 in 3 0 in 5
a stop that installs its own deadline 20 in 20 0 in 5

go build ./..., go vet ./..., gofmt -l ., golangci-lint run --build-tags dev ./... and BASECAMP_NO_KEYRING=1 go test -tags dev ./... are all clean, as is go test -tags dev -race ./internal/connector/..., and so are GOOS=windows and GOOS=darwin builds.

A refusal Codex wrote could be thrown away: never recorded anywhere,
not merely missing from a turn's result. Writing a refusal to the
ledger is allowed ten seconds and ran on the goroutine reading the
worker's output, while the read end of that pipe was released a couple
of seconds after the worker was terminated, whatever that goroutine was
doing. Closing it discards what the worker wrote and nobody has parsed.

No bound could be put on the reading while those shared a goroutine.
Counted in wall time it expires inside a ledger write, and the reader
comes back to find its own pipe closed with the worker's next refusal
still in it — the loss this exists to end. Counted in time spent on the
pipe, a descendant outside the worker's group writing without stop keeps
the pipe busy and holds the session's shutdown open for as long as it
lives. Those are not two bugs to balance; they are one structure making
the bound impossible to place.

So the reader only reads, and a scribe writes. Everything the reader
does between reads is microseconds now, which is what lets the pipe's
drain bound be a plain clock and mean what it says. The pipe belongs to
the reader while it is reading: termination asks it to stop rather than
closing the descriptor under it, the deadline on the pipe has one owner,
and the descriptor is released when the reader says it is through.

The queue between them is bounded, and it never drops. A refusal handed
over when it is full waits, which is backpressure onto the worker —
what a slow ledger already did to this driver when both ran on one
goroutine, so it is not a new way to fail. Dropping would be the old one
wearing a new hat. Its size is derived rather than chosen: a drain can
only produce what a pipe and a scanner buffer can hold, which is fewer
refusals than this, so the drain cannot block on it.

The scribe is drained before the session's updates close, which is where
the dispatcher anchors "every refusal the driver read has been through
the recorder", and before Close returns, so a caller that closes and
then asks the ledger is not asking early. A refusal read after the
scribe has gone — by an ending that is not the reader's — is written by
whoever read it rather than lost.

Held past the grace and the drain budget together, with a refusal in the
pipe the whole time, this fails 10 runs of 10 on main and passes 5 of 5
here. The test's second refusal is said only once the ledger is holding
the first, so it is in the pipe and nowhere else.
Copilot AI balanced review requested due to automatic review settings September 18, 2026 19:14
@github-actions github-actions Bot added the tests Tests (unit and e2e) label Sep 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The asynchronous handoff weakens refusal durability and does not reliably bound shutdown.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Separates Codex output reading from ledger writes to prevent refusals being discarded during shutdown.

Changes:

  • Adds reader-owned pipe draining and shutdown signaling.
  • Introduces a bounded refusal queue and scribe goroutine.
  • Adds regression coverage for slow ledger writes.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File summaries
File Description
internal/connector/driver/worker.go Adds reader-controlled pipe lifetime and drain deadlines.
internal/connector/driver/worker_other.go Adds platform stub methods for the reader lifecycle.
internal/connector/driver/codex/codex.go Moves refusal recording to an asynchronous scribe.
internal/connector/driver/codex/codex_test.go Tests slow-ledger refusal preservation.
Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 3
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/connector/driver/codex/codex.go Outdated
Comment thread internal/connector/driver/codex/codex.go Outdated
Comment thread internal/connector/driver/codex/codex.go Outdated
A refusal Codex wrote could be thrown away: recorded nowhere, not
merely missing from a turn's result. The read end of the worker's output
was released a couple of seconds after the worker was terminated,
whatever the goroutine reading it was doing, and closing it discards
what the worker wrote and nobody has parsed. That goroutine is allowed
to be slow — writing a refusal to the ledger is given ten seconds, and
the driver's own contract puts that write on the reader, before the
update for it is emitted — so the clock ran out inside a write and took
the pipe away with the worker's next refusal still in it.

The first shape I tried for this moved the ledger write to a goroutine
of its own so the reader would only ever read. It does not survive
contact with the contract it would have to break: a refusal held only in
a queue is not recorded, and publishing the update before the write
lands is the thing the contract exists to forbid. It also buys nothing —
the same writes still have to finish before the session's updates close,
so the shutdown is no shorter for having moved them. That shape is
abandoned.

What was wrong was the unit. A ledger write moves no bytes, so a drain
counted in bytes cannot be spent by one. The reader now stops after
taking more from the pipe than the worker could possibly have left
there — a worker is ended before its reader is asked, so what remains of
its output is at most one pipe's worth, and the budget is far more than
a pipe holds. Past that, what is still arriving can only be a descendant
that left the worker's group and is writing without end, which is the
one case that has no other ending.

The pipe belongs to the reader while it is reading, so termination asks
it to stop rather than closing the descriptor under it, the deadline on
the pipe has one owner, and the descriptor is released when the reader
says it is through. A cancel asks too: ending the worker is not the end
of its output when a descendant holds the pipe open, and a canceled turn
only the reader can finish would never be finished.

Held past every clock in this path, with a refusal sitting in the pipe
the whole time, this fails 10 runs of 10 on main. A stop that closes the
pipe loses what is in it, 10 of 10; a drain counted in time loses the
line that arrives while the reader works, 10 of 10; and without a budget
a reader follows a descendant forever, 3 of 3.
Copilot AI review requested due to automatic review settings September 18, 2026 19:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The byte-based drain can block shutdown for hours and may truncate output from enlarged pipes.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

internal/connector/driver/codex/codex.go:338

  • The PR description still presents a reader/scribe split, bounded queue, and clock-based drain, but this revision intentionally keeps RecordRefusal synchronous on the reader and implements a byte budget with no scribe or queue. Please update the description and evidence so the architecture being reviewed matches this implementation.
	// The worker's output is released when this session has read it, not on
	// a clock: what the reader does between reads — writing a refusal to the
	// ledger — is allowed to take longer than any such clock.
	worker.ReadingDone(s.readerEnd)
  • Files reviewed: 5/5 changed files
  • Comments generated: 2
  • Review effort level: Balanced

// arriving can only be a descendant that left the worker's group and is
// writing without end — the one case that has no other ending, and one the
// connector had already decided not to wait for.
const drainBudget = 1 << 20
Comment on lines +84 to +89
if o.drained >= drainBudget {
// More than the worker could have left: what is still
// arriving is a descendant's, and no longer waited for.
return 0, io.EOF
}
window = drainWindow
@jorgemanrubia
jorgemanrubia marked this pull request as draft September 18, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Tests (unit and e2e)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants