Skip to content

ci: complete comparative benchmark contract for #309 - #373

Merged
codeforester merged 3 commits into
ci/312-20260918-deduplicate-invariant-gatesfrom
ci/309-20260918-comparative-benchmark-contract
Sep 21, 2026
Merged

codeforester merged 3 commits into
ci/312-20260918-deduplicate-invariant-gatesfrom
ci/309-20260918-comparative-benchmark-contract

Conversation

@codeforester

@codeforester codeforester commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Complete the comparative benchmark contract for #309. This PR is stacked on #372 (ci/312-20260918-deduplicate-invariant-gates); after #372 merges, retarget this PR to main for final review.

Changes

  • Measure fresh-process import, cold invocation, warm dispatch, lifecycle, production, and feature scenarios for Click, Typer, Cyclopts, and base-cli.
  • Require all comparator metrics and base-cli scenarios needed by --check.
  • Gate performance budgets only for base-cli; comparator warm timings remain informational inputs for comparison.
  • Publish versioned JSON reports with source, environment, framework versions, p95/median/MAD, and lifecycle comparisons.
  • Write platform-specific GitHub summaries and retain dated JSON artifacts for 90 days, including WSL.
  • Document scenario semantics, local commands, thresholds, and calibration evidence.

Review follow-up

  • Scoped the warm-invocation budget failure to base-cli so dependency noise cannot fail the framework gate.
  • Kept lifecycle comparison in the same ambient Click runner environment as the comparator and documented that intentional measurement choice.
  • Extracted the shared subprocess PYTHONPATH setup into _subprocess_environment().

Validation

  • Full local validation passed: 534 tests, 193 subtests, coverage, typing, lint, contracts, benchmark, and security gates.
  • Comparative benchmark passed with 31 samples per scenario on local macOS for all four frameworks.
  • Documentation, changelog, YAML, Ruff, and benchmark self-tests passed.

Comment thread scripts/benchmark_runtime.py Outdated
Comment thread scripts/benchmark_runtime.py
Comment thread scripts/benchmark_runtime.py Outdated
@codeforester
codeforester merged commit 26ea363 into ci/312-20260918-deduplicate-invariant-gates Sep 21, 2026
120 of 121 checks passed
@codeforester
codeforester deleted the ci/309-20260918-comparative-benchmark-contract branch September 21, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant