ci: size the mcp-host-config timeout against its measured tail - #505
Merged
Conversation
`timeout-minutes: 3` sat inside the range of a legitimate passing run of this job, so it decided pass/fail on network luck rather than on the assertions it makes. The dominant cost is the last step's `scripts/verify_mcp_hosts.py`, which boots the plugin through `uvx` against a deliberately cold uv cache (UV_CACHE_DIR/UV_TOOL_DIR under RUNNER_TEMP, --allow-bootstrap-network) — a full network package resolve+build every run. Two samples on the same PR, both PASSing the same assertions: run 34217620888 job 102033099129 — bootstrap 23.98s, job 52s run 34210128951 job 102009142706 — bootstrap 179.86s, job 194s A 7.5x spread, entirely in that bootstrap. The old bound came from ceil(2 * 61 / 60) against run 33951959734 (2026-09-05, max 61s) — a 2x margin computed on a sample that never saw the tail. Re-applying the same convention to the measured max gives ceil(2 * 194 / 60) = 7. The cold cache is the point of the test (it is the path a first-time user hits), so the variance cannot be removed without destroying what the job asserts; the bound has to cover it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
13 tasks
cdeust
marked this pull request as ready for review
September 8, 2026 12:11
cdeust
pushed a commit
that referenced
this pull request
Sep 8, 2026
Brings in #505 (mcp-host-config timeout sized to its measured tail), so this PR's CI runs under a bound that no longer decides on network luck. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
cdeust
pushed a commit
that referenced
this pull request
Sep 8, 2026
Brings in #503 (floor check non-blocking + --no-regression gate) and #505 (mcp-host-config timeout sized to its measured tail), so this PR is verified against the CI its merge will actually land on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mcp-host-config(Validate MCP host configurations) carriedtimeout-minutes: 3. 180 s sits inside the range of a legitimate, passing run of that job, so the job decided pass/fail on network luck rather than on the assertions it makes. It killed PR #504 twice on a diff that touches nothing it exercises.Root cause
The job's cost is not the config parsing — those three validators take ~6 s combined. It is the last step's
scripts/verify_mcp_hosts.py, which boots the plugin throughuvxagainst a deliberately cold uv cache (UV_CACHE_DIR/UV_TOOL_DIRunderRUNNER_TEMP,--allow-bootstrap-network), paying a full network package resolve + build on every run.Two samples, same PR, same assertions, both
PASS:initialize + discovery + memory_stats)A 7.5× spread, entirely inside that bootstrap. The 179.86 s run passed its assertions and was killed anyway, at 194 s, by the 180 s ceiling.
The old bound was
ceil(2 * 61 / 60) = 3, sourced to run 33951959734 (2026-09-05, max 61 s). The 2× margin was real; the sample it was computed against had simply never seen this job's tail.Change
timeout-minutes: 3→7, re-applying the repository's own convention (ceil(2 * max_seconds / 60)) to the measured max:ceil(2 * 194 / 60) = 7. The# source:comment records both samples, the job ids, and why the variance exists.Not attempted: warming the cache to remove the variance. The coldness is the point of the test — it is the path a first-time user hits — so flattening it would destroy what the job asserts. The bound has to cover the tail instead.
Test plan
yaml.safe_loadon the edited workflow;jobs.mcp-host-config.timeout-minutes == 7, job name unchanged.actionlint+shellcheckrun on workflow files in theLintjob of this PR's own CI — the authoritative check for this file.No job added or removed, so the
Check the aggregate CI gate covers every jobassertion inLintis unaffected.All existing tests pass. (no application code touched; this PR's own CI run is the verification)
New tests added for new behavior. (N/A — a workflow bound, not behaviour; the two measurements in the source comment are the evidence)
Type of change
Coding-standards compliance
7cites two dated, job-identified measurements and the arithmetic that derives it from them.Breaking changes
None. A ceiling is raised; nothing that passed before fails now.
Reviewer checklist
# source:block in the workflow is the documentation for this constant).🤖 Generated with Claude Code
https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
Generated by Claude Code