connectd: add --dev-max-wake-delay-ms - #9481
Conversation
The wake-delay watchdog (write_to_subd's empty-queue check) measures the gap between handing a peer message to the subdaemon queue and the subdaemon consuming it, which includes lightningd-master's time to spawn the subdaemon: under CI load that alone can exceed the hardcoded 5000 msec and fail runs with 'wake delay for WIRE_OPEN_CHANNEL' BROKEN (ElementsProject#9268) -- tests/test_connection.py:: test_funding_cancel_race spins up 100 nodes and hit a 11068 msec window. dev_lightningd_is_slow exists for the same class but is only settable by the memleak dev-RPC. Make the threshold a developer option (default unchanged at 5000) so load-heavy tests can raise it; wired through connectd_init. The regression test freezes the accepter's master past the default threshold and expects a clean run with the option raised. Changelog-Fixes: ElementsProject#9268
Andezion
left a comment
There was a problem hiding this comment.
The PR says Fixes #9268, but it does not fix the flaky test named in that issue. test_funding_cancel_race is the test that hit the 11068ms delay in ci, and it still calls node_factory.get_nodes(num, opts={}) - no dev-max-wake-delay-ms is passed there. The pr only adds a separate, new test (test_dev_max_wake_delay) that exercises the new option in isolation
The 100+ nodes this test spawns starve the scheduler enough on loaded CI runners that connectd's wake-delay watchdog fires BROKEN on the delayed WIRE_OPEN_CHANNEL (the 11068ms case in ElementsProject#9268) long before the funding-cancel race under test is affected. Give every node the raised threshold (--dev-max-wake-delay-ms=60000) so the incidental telemetry stops failing the test while genuine multi-minute hangs still trip it. Fixes: ElementsProject#9268 Signed-off-by: Amperstrand <amperstrand@localhost>
|
Fair point — the test that actually hit #9268 never got the knob. Wired now: every node in |
The wake-delay watchdog (
write_to_subd's empty-queue check) measures the gap between handing a peer message to the subdaemon queue and the subdaemon consuming it, which includes lightningd-master's time to spawn the subdaemon: under CI load that alone can exceed the hardcoded 5000 msec and fail runs withwake delay for WIRE_OPEN_CHANNELBROKEN (#9268) — tests/test_connection.py::test_funding_cancel_race spins up 100 nodes on a fast runner and hit a 11068 msec window.dev_lightningd_is_slowexists for the same class but is only settable by the memleak dev-RPC.This makes the threshold a developer option (
--dev-max-wake-delay-ms, default unchanged at 5000) so load-heavy tests can raise it; wired throughconnectd_init. The regression test freezes the accepter's master past the default threshold and expects a clean run with the option raised.deterministic repro results (master build, regtest pyln) — expand
Fuller repro record — the trigger recipe with its two ordering traps, all cell results, and the control arm: https://gist.github.com/Amperstrand/b50c4d604f88016047e104dc65f866c6
Fixes #9268