Skip to content

ci: size the mcp-host-config timeout against its measured tail - #505

Merged
cdeust merged 1 commit into
mainfrom
fix/mcp-host-config-timeout
Sep 8, 2026
Merged

ci: size the mcp-host-config timeout against its measured tail#505
cdeust merged 1 commit into
mainfrom
fix/mcp-host-config-timeout

Conversation

@cdeust

@cdeust cdeust commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Summary

mcp-host-config (Validate MCP host configurations) carried timeout-minutes: 3. 180 s sits inside the range of a legitimate, passing run of that job, so the job decided pass/fail on network luck rather than on the assertions it makes. It killed PR #504 twice on a diff that touches nothing it exercises.

Root cause

The job's cost is not the config parsing — those three validators take ~6 s combined. It is the last step's scripts/verify_mcp_hosts.py, which boots the plugin through uvx against a deliberately cold uv cache (UV_CACHE_DIR/UV_TOOL_DIR under RUNNER_TEMP, --allow-bootstrap-network), paying a full network package resolve + build on every run.

Two samples, same PR, same assertions, both PASS:

Run Job Bootstrap (initialize + discovery + memory_stats) Whole job
34217620888 102033099129 23.98 s 52 s
34210128951 102009142706 179.86 s 194 s

A 7.5× spread, entirely inside that bootstrap. The 179.86 s run passed its assertions and was killed anyway, at 194 s, by the 180 s ceiling.

The old bound was ceil(2 * 61 / 60) = 3, sourced to run 33951959734 (2026-09-05, max 61 s). The 2× margin was real; the sample it was computed against had simply never seen this job's tail.

Change

timeout-minutes: 37, re-applying the repository's own convention (ceil(2 * max_seconds / 60)) to the measured max: ceil(2 * 194 / 60) = 7. The # source: comment records both samples, the job ids, and why the variance exists.

Not attempted: warming the cache to remove the variance. The coldness is the point of the test — it is the path a first-time user hits — so flattening it would destroy what the job asserts. The bound has to cover the tail instead.

Test plan

  • yaml.safe_load on the edited workflow; jobs.mcp-host-config.timeout-minutes == 7, job name unchanged.

  • actionlint + shellcheck run on workflow files in the Lint job of this PR's own CI — the authoritative check for this file.

  • No job added or removed, so the Check the aggregate CI gate covers every job assertion in Lint is unaffected.

  • All existing tests pass. (no application code touched; this PR's own CI run is the verification)

  • New tests added for new behavior. (N/A — a workflow bound, not behaviour; the two measurements in the source comment are the evidence)

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Coding-standards compliance

  • §8 Every numeric constant sourced: the new 7 cites two dated, job-identified measurements and the arithmetic that derives it from them.
  • §9 No dead code.
  • Other sections N/A — single-line workflow bound plus its provenance comment.

Breaking changes

None. A ceiling is raised; nothing that passed before fails now.

Reviewer checklist

  • CHANGELOG.md — not updated; CI-internal bound with no user-facing surface. Say the word if you want an entry anyway.
  • Documentation updated (the # source: block in the workflow is the documentation for this constant).
  • No secrets / credentials / PII in the diff.
  • CI passes on the latest commit — pending.

🤖 Generated with Claude Code

https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1


Generated by Claude Code

`timeout-minutes: 3` sat inside the range of a legitimate passing run of
this job, so it decided pass/fail on network luck rather than on the
assertions it makes.

The dominant cost is the last step's `scripts/verify_mcp_hosts.py`, which
boots the plugin through `uvx` against a deliberately cold uv cache
(UV_CACHE_DIR/UV_TOOL_DIR under RUNNER_TEMP, --allow-bootstrap-network) —
a full network package resolve+build every run. Two samples on the same
PR, both PASSing the same assertions:

  run 34217620888 job 102033099129 — bootstrap  23.98s, job  52s
  run 34210128951 job 102009142706 — bootstrap 179.86s, job 194s

A 7.5x spread, entirely in that bootstrap. The old bound came from
ceil(2 * 61 / 60) against run 33951959734 (2026-09-05, max 61s) — a 2x
margin computed on a sample that never saw the tail. Re-applying the same
convention to the measured max gives ceil(2 * 194 / 60) = 7.

The cold cache is the point of the test (it is the path a first-time user
hits), so the variance cannot be removed without destroying what the job
asserts; the bound has to cover it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
@cdeust
cdeust marked this pull request as ready for review September 8, 2026 12:11
@cdeust
cdeust merged commit 6fc287f into main Sep 8, 2026
28 checks passed
cdeust pushed a commit that referenced this pull request Sep 8, 2026
Brings in #505 (mcp-host-config timeout sized to its measured tail), so
this PR's CI runs under a bound that no longer decides on network luck.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
cdeust pushed a commit that referenced this pull request Sep 8, 2026
Brings in #503 (floor check non-blocking + --no-regression gate) and
#505 (mcp-host-config timeout sized to its measured tail), so this PR
is verified against the CI its merge will actually land on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
@cdeust
cdeust deleted the fix/mcp-host-config-timeout branch September 8, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants