docs: eval-return-experiments, the twelve experiments and the scorecard that grades them

[buzzdan]

Summary

Adds docs/eval-return-experiments.md, the design for measuring whether linter-driven development is worth its tokens, and its line in docs/index.md. Nothing in it has run yet; this records the plan so the evals repository can build against it.

Part one, is the plugin worth its tokens. Eight experiments that add the arms every baseline has been missing: the same task with no plugin, with the generated coding-rules handbook alone, with the handbook plus the rule greps as lint gates (the token-free delivery), and with the full plugin, across Sonnet, Opus and Fable. Experiment 7 (writing features from a fixed spec, then reviewing the result for the plants it added) starts with four cells and a decision table for which cell comes next. Experiment 8 (the review task on the same grid) separates the twelve rules from the machinery that applies them. Experiment 3 (a cheaper model with the plugin against a stronger one without, on the refactor oracles) stays a separate arm gated on 2 and 7.

Part two, does the code LDD refactored pay off for the next agent. Four experiments on py-mini-ldd, py-mini after one full plugin refactor pass committed as the plugin left it, against the dirty original, with the plugin out of the picture: comprehension questions, adding a feature, fixing an injected bug, five features in sequence. A hand-finished clean twin is an optional third arm.

The scorecard. Three layers every run is graded on: correctness with differential testing, pass^k and an optional mutation score; structure as deltas in ISO 25010 terms via the SIG maintainability model's risk bands, with the tools the postcheck runs today separated from the ones still to add; design as the plugin's plant count labelled rules conformance plus an independent pairwise, blinded judge. The section states which rules structural metrics can see (R3, R4, R5, R7, R8, R11) and which only a review or judge can, lists the measures to avoid on the evidence, and gives the method rules from the 2023 to 2026 agent-evaluation literature, including a decay check that re-runs gate 1 on each major model release. A note covers TypeSafe's Jev as a candidate scorer for the judge, gated on calibration and excluded from any WEKA code.

Ranking and a gated run order. The ranking is by decisions changed per dollar. The run order is six gates: the most important experiment runs first with its pass criterion written down before it starts, including a cost-per-passing-feature bound, and nothing further is spent on a fail. All budgets assume five runs per cell.

Checks

• check-repo-brain.sh over the docs: clean on every commit. The wrapper scripts/check-docs.sh could not run in this environment because rsync is missing, so the gate was run directly on the repo root.
• No generated plugin or handbook files are touched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FvrtWhSgzg6DGB8kcdUZVN

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved inconsistencies affect budgets, reproducibility, metric definitions, and experiment scope.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2

Open (3)

• Experiment 8 cannot substitute for experiment 3 · New
• Planned analysis tools are presented as already available · New
• R7 coverage is inconsistently classified · New

What changed in this PR

Adds a draft design for evaluating plugin value, downstream code quality, and return on tokens across twelve experiments.

Changes:

• Defines experiments, budgets, metrics, and execution order.
• Adds a correctness, structure, and design scorecard.
• Links the new guide from the documentation index.

File	Description
docs/​eval-return-experiments.md	Experimental plan, scoring framework, and execution guidance
docs/​index.md	Links the new evaluation document

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.