Skip to content

Community readiness: public benchmark, real-output docs, MCP 2.x fix, release gates - #26

Draft
denfry wants to merge 12 commits into
mainfrom
chore/community-readiness
Draft

Community readiness: public benchmark, real-output docs, MCP 2.x fix, release gates#26
denfry wants to merge 12 commits into
mainfrom
chore/community-readiness

Conversation

@denfry

@denfry denfry commented Sep 4, 2026

Copy link
Copy Markdown
Owner

What

Community-readiness pass: reproducible public benchmark, real-output docs and demo, three shipped-path bug fixes, release/CI gates, contributor onboarding, launch kit. Ten atomic commits; see docs/COMMUNITY_AUDIT.md for every finding with severity and status.

Fixes in shipped paths

  • MCP server did not start on mcp 2.x (FastMCP → MCPServer rename) and codebase-index mcp --root <repo> was rejected. Both fixed; verified against mcp 1.29.1 and 2.1.1. The MCP tests had been skipping silently in CI; they now fail loudly if our module breaks.
  • Plugin bin/ wrappers refused architecture, diff-impact, path, describe; parity test added.
  • Benchmark leakage: changelog-style files were never excluded from the eval corpus and CHANGES.rst was not covered.

Evidence

  • New tests/eval/run_baselines.py: index vs rg+80-line windows and repo-map-style context on Flask/Gson/Fastify at pinned commits, git-derived ground truth, one tokenizer on both sides, paired significance. Logged run: tests/eval/results/2026-09-04-public-baselines.md. Pooled n=450: hit@3 0.547 vs 0.304, MRR 0.456 vs 0.263, p<0.001, 3.8k vs 3.5k context tokens.
  • That run found the read plan cost 6.8k tokens (whole-class ranges); retrieval.max_read_lines (default 120) caps it with additive truncated/line_end_full fields, quality unchanged.
  • The private-repo "13× fewer tokens" headline is withdrawn everywhere.
  • Full suite green (coverage 84.9%), ruff, mypy, skill-sync, check_versions.py, check_links.py, python -m build + twine check + release_smoke.py. Every README command was run on Flask; outputs are in examples/demo/EXPECTED_OUTPUT.md.

Maintainer actions after merge

Enable Discussions, upload assets/social-preview.png, run scripts/apply_labels.sh, confirm private vulnerability reporting, optionally record the GIF from docs/demo.tape. Then cut 1.9.1.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL

denfry and others added 12 commits September 4, 2026 16:07
@denfry @claude
test(eval): exclude changelog-style files from the corpus, not just f…
170bcb8 
…rom answers

gen_queries documented that CHANGELOG_EXCLUDES was applied by the harness, but
the harness never imported it: for a git-derived query set the changelog (and
Flask-style CHANGES.rst, which the deny pattern also missed) stayed indexed and
could outrank the implementation it paraphrases. Both lists now cover CHANGES*,
HISTORY*, NEWS* and RELEASE_NOTES*, and build_corpus_index applies them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
fix(plugin): allow architecture, diff-impact, path and describe in bi…
b98bb5a 
…n/ wrappers

The plugin wrappers whitelisted ten subcommands while the skill template's cbx
allowed fourteen, so plugin users could not run the four commands SKILL.md
routes to. A parity test now pins bin/cbx, bin/cbx.ps1 and both template
wrappers to the same set.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
feat(retrieval): cap recommended_reads at max_read_lines (default 120)
0d83adb 
A symbol-aligned chunk can be a whole class (1,500 lines in Flask's app.py),
and the read plan handed that span to the agent verbatim. On the public
baseline benchmark that made the index's follow-through reads cost more than
three 80-line grep windows. Reads are now capped at the definition head;
capped entries carry additive truncated/line_end_full fields so the agent can
read on deliberately. retrieval.max_read_lines=0 restores the old behaviour.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
ci(release): version-consistency, link and packaging gates; changelog…
53165f6 
…-driven release notes

- scripts/check_versions.py: __version__ must match plugin.json, the
  requirements.lock tag, the three .skill_version stamps and a CHANGELOG heading.
- scripts/check_links.py: relative Markdown links and image sources must resolve.
- scripts/release_notes.py: the GitHub release body is the CHANGELOG section for
  the tag, and an empty section fails the release job.
- CI lint job runs both checks; a new package job builds, twine-checks and runs
  the clean-venv install smoke on every PR instead of only at tag time.
- release.yml refuses a tag whose name does not match __version__.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
docs(community): contributor onboarding, issue templates, support and…
cbaba61 
… security policy

- docs/DEVELOPMENT.md: fifteen-minute path from clone to a merged PR, with
  verified commands and recipes for languages, graph edges, ranking signals,
  MCP tools and CLI commands.
- CONTRIBUTING.md: contribution areas mapped to labels, evidence rules for
  retrieval-quality PRs, and the fork workflow; unverified uv/ruff-format
  instructions removed.
- SECURITY.md: supported line is 1.9.x, private vulnerability reporting via
  GitHub advisories, explicit statement that checksums/SBOMs are not yet shipped.
- Issue templates: concise bug/feature forms plus benchmark-report and
  language-support forms; blank issues off; SUPPORT.md; labels.yml with an
  apply script; placeholder FUNDING.yml removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
test(eval): index vs rg+window and repo-map-style baselines on public…
a509092 
… repositories

run_eval.py compares the ranker with its own earlier versions; it cannot say
whether an index beats not having one. run_baselines.py asks that question on
Flask, Gson and Fastify at pinned commits, with git-derived ground truth
(150 newest localised commits each), symmetric tiktoken accounting on every
side, and paired bootstrap/permutation significance. The logged run is under
tests/eval/results/.

Pooled over 450 queries: hit@3 0.547 (index) vs 0.304 (rg + 80-line windows),
MRR 0.456 vs 0.263, p < 0.001 on every metric, at 3.8k vs 3.5k context tokens
per query once the read plan is capped (6.8k before the cap). Repo-map-style
context is scored on answer presence under a 2k/8k budget.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
docs: consolidate duplicate pages and replace fabricated examples wit…
b66572b 
…h real output

- One canonical page per topic: RETRIEVAL.md, SCHEMA.md, SECURITY_MODEL.md;
  the former duplicates are redirect stubs so inbound links keep working.
  SCHEMA.md now matches storage/schema.sql (the old copy described columns
  and tables that do not exist).
- QUICKSTART and INSTALLATION showed invented output (an AuthService.ts table
  with a Score column, a fake doctor transcript); both now show output captured
  on pallets/flask at a pinned commit.
- FAQ/INSTALLATION/MCP/ARCHITECTURE: version pins to 1.9.0, full MCP tool
  list, correct config path (.claude/cache/codebase-index/config.json),
  doctor's real check list, unverified install paths labelled as such.
- Root ROADMAP.md is a milestone history; docs/ROADMAP.md owns forward work.
- installer.md translated to English; flags verified against install.sh.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
docs(demo): reproducible Flask demo, real-output terminal card, bench…
d0f4269 
…mark chart, architecture diagram

- examples/demo/run_demo.sh|.ps1 clone pallets/flask at a pinned commit and run
  the Find / Trace / Predict scenarios; EXPECTED_OUTPUT.md is the captured run.
- assets/demo-terminal.svg is rendered from that transcript by
  scripts/gen_terminal_svg.py; the previous demo.png was a mock-up with invented
  paths and is removed (gen_assets.py no longer builds it).
- assets/benchmark.svg is rendered from the logged public baseline JSON by
  scripts/gen_benchmark_chart.py, so the chart cannot drift from the run.
- assets/architecture.svg: index and query paths on one diagram.
- docs/DEMO.md + docs/demo.tape: exact steps for a VHS/asciinema recording.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
fix(mcp): start on mcp 2.x (MCPServer) and accept --root on the mcp s…
87146cf 
…ubcommand

mcp 2.0 renamed FastMCP to MCPServer and removed mcp.server.fastmcp, so
`codebase-index mcp` failed with "needs the optional extra" on a fresh
install, and every MCP test skipped silently in CI because the skip guard
wrapped our own module's import. The server now imports MCPServer first and
falls back to FastMCP on 1.x; verified against mcp 1.29.1 and 2.1.1 (41 tests
each, plus a stdio initialize / tools/list / healthcheck round-trip). Tests now
skip only when the SDK itself is absent.

`--root` was a global option only, so the documented
`codebase-index mcp --root <repo>` form was rejected; the subcommand accepts it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
docs: rebuild README around real output and the public benchmark; aud…
5be25cd 
…it and launch kit

- README: one-sentence proposition, agents list, terminal card rendered from a
  real Flask transcript, 60-second try-it, why-this-exists with grep / repo-map /
  cloud comparison, evidence section with the public baseline chart, then
  architecture, privacy, languages, status, contributing, roadmap.
- BENCHMARKS.md leads with the public baseline run and withdraws the private
  55k LOC Java headline (asymmetric accounting, not reproducible); COMPARISON.md
  and the older result logs point at the new run.
- docs/COMMUNITY_AUDIT.md: P0-P3 findings with status and before/after scores.
- docs/COMMUNITY_LAUNCH.md: HN / Reddit / X / Discord / release drafts using
  only logged numbers, a day-0 to month-1 sequence, and the feedback loop;
  replaces the stale docs/SEO.md.
- examples/demo-project (a project that did not exist) removed in favour of
  examples/demo; package keywords extended (code-graph, impact-analysis,
  mcp-server, claude-code-plugin).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
docs(readme): absolute image URLs so the PyPI project page renders th…
0276877 
…e visuals

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
@denfry @claude
docs(launch): tick the checklist items applied via gh (labels, Discus…
97aca99 
…sions, About, PVR)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@denfry