Skip to content

The intake feed test fails under -race on CI, on an assertion that was never due - #758

Merged
jorgemanrubia merged 1 commit into
mainfrom
fix/intake-feed-flake
Sep 18, 2026
Merged

jorgemanrubia merged 1 commit into
mainfrom
fix/intake-feed-flake

Conversation

@jorgemanrubia

@jorgemanrubia jorgemanrubia commented Sep 18, 2026

Copy link
Copy Markdown
Member

Main's Race Detection job is red, and one of the reasons is this test: the
end-to-end intake feed test fails under -race on CI with a bare assertion
that never became true — on main's own push run at 66b1a3a
(run 35349470544)
and on at least one pull request built from it. The race detector reported no
data race. There is none.

The test was reading across an order it had not waited for. It serves a live
event on the socket, waits for that event to reach the ledger, and then reads
the durable feed position. A fresh ledger has no position to resume, so the
entry is present-class: the feed holds the page's position through the walk
and saves it only after the drain has accepted every event buffered during the
walk. The live event is one of those. Its row in the ledger says the drain
reached it, not that the save has run.

Instrumented to report what was true at the instant of that read, under twenty
competing processes on two cores, the position was absent and arrived 3.8ms,
33.2ms and 34.8ms later — with the walk's one page delivered, both pointer
lines written and both ids handed over. Everything the two events owed was
done; only the walk's position was outstanding. Under -race on a loaded
runner that window is wide enough to lose, which is exactly what CI kept
losing.

So the assertion was wrong, not the code: nothing guarantees the position is
durable at the moment the live event's row appears, and the feed's ordering —
deliver, drain, then save what the entry held — is deliberate. The test now
waits for the position to be saved before reading it, and waits for the
pointer lines and the queue hand-off too, since the ledger row is written
before both. Waiting for the condition rather than widening a timeout is the
same fix shape as
The recovery harness reads a worker's word once the worker has said it,
and the restart test next door already worked this way.

Under load on two pinned cores, over nine batches: 15 of 245 runs failed
before, always at that line and never with a data race; 0 of 90 after, the
after batches alternating with before batches in the same sessions. Unloaded,
both are clean. The test introduced in
Take the account event feed into a durable ledger
is the only thing that changes here.

The end-to-end intake test served a live event on the socket, waited for it
to reach the ledger, and read the durable position there. A fresh ledger has
no position to resume, so the entry is present-class: the feed HOLDS the
page's position through the walk and saves it only after the drain has
accepted every event buffered during it. The live event is one of those. Its
row in the ledger says the drain reached it, not that the save has run, and
the read that followed landed between the two — require.True over an `ok`
that was not yet due.

Under -race everything runs several times slower and that gap widens. It is
what failed the Race Detection job on main at 66b1a3a (run 35349470544) and
on at least one pull request built from it. The race detector reported no
data race, because there is none: the check read across an order it had not
waited for.

Instrumented to say what was true at the moment of that read, under twenty
competing processes on two cores: the position was ABSENT and arrived 3.8ms,
33.2ms and 34.8ms later on the three iterations of forty that caught it —
with one poll page walked, both pointer lines written and both ids handed
over. Everything the events owed was done; only the walk's position was
outstanding.

The check now waits for the position to be saved before it reads it — the
condition, not a duration, the same shape as #752. The counts below it are
waited on too: the ledger row is written before the pointer line and the
hand-off, so the row says nothing about either, which the restart test next
door already knew and said. That wait is for "at least two", so the
equalities still measure — a third pointer line or a third id fails them
rather than satisfying them.
Copilot AI balanced review requested due to automatic review settings September 18, 2026 14:32
@github-actions github-actions Bot added the tests Tests (unit and e2e) label Sep 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The waits correctly reflect the asynchronous ordering and preserve exact result assertions.

Pull request overview

Updates the intake feed test to wait for asynchronous side effects before asserting results.

Changes:

  • Waits for checkpoint persistence.
  • Waits for pointer output and queue handoff.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File summaries
File Description
internal/connector/intake_feed_test.go Synchronizes assertions with feed processing.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@jorgemanrubia
jorgemanrubia merged commit 4e1bb89 into main Sep 18, 2026
26 checks passed
@jorgemanrubia
jorgemanrubia deleted the fix/intake-feed-flake branch September 18, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Tests (unit and e2e)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants