Skip to content

fix(source-ledger): keep glob wildcards in register waiver lines, plus a privacy fix that unblocks the leak guard - #63

Closed
MendixMau wants to merge 2 commits into
masterfrom
fix/source-ledger-glob-waivers
Closed

MendixMau wants to merge 2 commits into
masterfrom
fix/source-ledger-glob-waivers

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Two commits. The first one is the reason the second one could be committed at all, so please read them in order.

1. privacy: genericize a client project name in three tracked files

bin/check-no-client-data.sh was red on a clean tree, before any change of mine. A client project name was on four lines across CHANGELOG.md, project-bin/wf-set-call-captions.py and skills/learned-workflow-patterns.md. This repo is public, so it was published.

All four are field-run credits and one aside in prose. None is load bearing: the name says who the run was for, and every finding stands without it. Replaced with the descriptive anonymisation the denylist file itself prescribes, matching how other field runs are credited in the changelog.

This is a forward fix only. The name stays in the history of the two commits that introduced it. Removing it from history is a separate decision and is deliberately not taken here.

One thing worth fixing in the guard itself, not done in this PR: the denylist entry that catches this name was added on 2026-08-20, after the commits that introduced it. The guard hard-fails on an absent denylist but has no notion of a stale one, so nothing re-scanned the tree when the entry was added. It surfaced only because a later commit happened to run the hook.

2. fix(source-ledger): keep glob wildcards in register waiver lines

The bug

The register parser stripped every literal * from a line before matching it:

l = re.sub(r'^[ \t>*_-]+', '', line.rstrip('\n')).replace('*', '')

The .replace('*', '') was there to remove markdown bold. It also destroyed glob wildcards in Waived source <glob>: reason lines, so InputCodeExamples/SomeCorpus/* became InputCodeExamples/SomeCorpus/ and then matched zero files through fnmatch.

Directory level waivers were silently non-functional. Only exact paths and basenames ever worked. Silently is the operative word: the line is accepted, it just waives nothing, so the ledger reports the files as missing and the author assumes they wrote the glob wrong.

Found on a migration whose corpus mixed one in-scope source folder with about 1,800 unrelated workshop example files across 8 directories. The only workaround the bug left was one waiver line per file, which is not a workaround at that scale.

The fix, and why it is two halves not one

Dropping the blanket strip is only half of it. Doing just that trades one silent bug for a worse one, in the opposite direction:

register line before naive fix this PR
Waived source dir/*: r key dir/, matches nothing dir/*, correct dir/*, correct
Waived source **one.pdf**: r key one.pdf, correct **one.pdf**, also waives one.pdf.bak one.pdf, correct
**Waived source** one.pdf: r key one.pdf, correct no match at all one.pdf, correct

A waiver means "out of scope", so a widened key silently hides files nobody named. That is the more dangerous direction of the two, which is why bold markers are now removed precisely rather than by stripping every star. The lookarounds require a path character beside the **, so a bold pair hugging a path is removed while dir/** and **/*.bak survive untouched.

Verification

Adversarially regression tested before opening this, not just unit tested:

  • Every register on this machine was diffed old parser against new. 25 registers, 4535 lines, 12 Waived source lines. Exactly one register parses differently, and it parses differently in the intended direction. No register anywhere uses a bold spelling, and gate-check.sh --waive never writes one, so the bold cases are a latent hazard for hand-edited registers rather than a live regression.
  • On the project that found it: waived 4, missing 1805, FAIL before; waived 1809, missing 0, PASS after, with the 3 genuinely in-scope files EXTRACTED and name-verified.
  • Tests: tests/wave2/test-source-ledger.sh 43/43 (was 42, new T13), tests/wave2/test-source-sufficiency-gate.sh 7/7.

New fixture T13 pins both halves: a directory glob waives its 3 files, two bold spellings waive exactly their own file, and the neighbouring .bak is left alone.

Two pre-existing items found while testing, not touched here

Both predate this change and neither is caused by it:

  1. gate-check.sh register_set_line and reg_field still gsub(/\*/, "") when looking up an existing label, so re-running --waive 'dir/*' appends a second line rather than replacing the first. Harmless today because the dict's last write wins, but the writer and the reader now disagree about what * means.
  2. In source-ledger.sh the waiver branch runs before the on-disk check, so a waived file that is no longer on disk never reports MISSING.

Happy to fix either in a follow-up if you want them.

🤖 Generated with Claude Code

MendixMau and others added 2 commits September 16, 2026 11:12
The repo's own leak guard, bin/check-no-client-data.sh, was red on a clean
tree: a client project name reached four lines across CHANGELOG.md,
project-bin/wf-set-call-captions.py and skills/learned-workflow-patterns.md.
The repo is public, so it was published. Because the guard fails the whole
tree rather than the staged diff, it also blocked every commit here until
this was cleared.

All four are field-run credits and one aside in prose. None is load bearing:
the name identifies who the run was for, and the finding stands without it.
Replaced with the descriptive anonymisation the denylist file itself asks
for, matching how other field runs are credited in this changelog.

This is a forward fix only. The name remains in the history of the two
commits that introduced it and in the GitHub UI for those commits; removing
it from history is a separate decision, deliberately not taken here.

Worth noting for the guard itself: the denylist entry that caught this was
added on 2026-08-20, after the commits that introduced the name, and the
guard has no notion of a stale denylist. It only found this because a later
commit happened to run it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The register parser stripped every literal '*' from a line before matching
it, a blanket .replace('*', '') meant to remove markdown bold. It also
destroyed glob wildcards in "Waived source <glob>: reason" lines, so
"InputCodeExamples/SomeCorpus/*" became ".../SomeCorpus/" and matched zero
files through fnmatch. Directory level waivers, the documented use of the
waiver vocabulary that --waive shares, were silently non-functional; only
exact paths and basenames ever worked.

Found on a migration whose corpus mixed one in-scope source folder with
about 1,800 unrelated workshop example files across 8 directories. The only
workaround the bug left was a waiver line per file, which is not a
workaround at that scale.

Bold markers are now removed precisely instead. The lookarounds require a
path character beside the '**', so "dir/**" and "**/*.bak" survive while
"**legacy/one.pdf**" loses its markers. Doing only the first half would have
traded one silent bug for a worse one: a bold key would have carried its
asterisks into the fnmatch pattern and waived "legacy/one.pdf.bak" as well.
A waiver means out of scope, so a widened key hides files nobody named.

Verified on the project that found it: waived 4/1812 FAIL before, 1809/1812
PASS after, with the 3 in-scope files EXTRACTED and name-verified. New
fixture T13 pins both halves. tests/wave2/test-source-ledger.sh 43/43,
tests/wave2/test-source-sufficiency-gate.sh 7/7.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MendixMau pushed a commit that referenced this pull request Sep 16, 2026
…ricize PR #63's identifiers

F1 regression fix in bin/source-ledger.sh: the leading-marker strip ate an OPENING single '*'
as a list bullet while only '**' was handled on the CLOSING side, so a single-asterisk emphasis
spelling ("- *Waived source legacy/one.pdf*: r" or "- Waived source *legacy/one.pdf*: r") left
one wildcard behind and silently widened the waiver onto the neighbouring .bak file — the exact
failure PR #63's own fix (the T13 bold-vs-glob distinction) set out to close, one marker
narrower. Emphasis stripping is now symmetric for both single-* and ** around the key, while a
real trailing glob ("legacy/*", "dir/**", "**/*.bak") still survives.

Also genericizes three client identifiers PR #63 introduced (verified absent from origin/master
before this branch): the source-ledger.sh comment's example path, the company-abbreviation
phrase in test-source-ledger.sh, and the CHANGELOG credit line. Moves that CHANGELOG line into
today's (2026-09-16) section, where it belongs by date, with a short follow-up clause. Extends
the --waive success message and interview-protocol.md's waiver-syntax note with one sentence
each: a glob is accepted, and '*' matches across '/' so a bare directory name already waives
everything under it (fnmatch semantics were previously unstated).

New T14 in tests/wave2/test-source-ledger.sh pins: single-* emphasis (both spellings) waives
exactly its named file and not a neighbouring .bak; a bare 'dir/*' and 'dir/**' both cover a
nested file, proven in separate projects so neither spelling's match masks the other's.

Deliberately NOT included (separate PRs): the item-1 duplicate-waiver-line fix in
bin/gate-check.sh (register_set_line / reg_field), and the item-2 "no longer on disk" note on
an already-waived, deleted file.

Fixture counts: tests/wave2/test-source-ledger.sh 43/43 -> 46/46 (3 new T14 assertions);
tests/wave2/test-source-sufficiency-gate.sh unchanged at 7/7. 16-case parser probe (14 original
+ 2 regression cases): 16/16 correct. bin/check-scripts.sh 76/76 shell + 20/20 node;
bin/check-portability.sh clean, 127 files; check-no-client-data.sh clean, 547 tracked files
(name check off, no denylist in this environment).

The two identifiers already on origin/master before PR #63 are scrubbed separately in PR #79.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
MendixMau added a commit that referenced this pull request Sep 16, 2026
…e-emphasis fix

fix(source-ledger): single-* emphasis no longer widens a waiver (supersedes #63)
@MendixMau

Copy link
Copy Markdown
Owner Author

Superseded by #80, now merged as dc3b236. It carries both of this PR's commits unchanged plus the follow-up the triage found: single-* emphasis around a waiver key no longer widens the waiver (the same widening this PR's own commit message called the worse failure, one marker narrower), the --waive message and interview-protocol.md now say globs are accepted, and fixture T14 pins it (46/46). Thank you for the field find on the 1,800-file corpus, it is credited on the CHANGELOG line.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant