Add poka-yoke skill: make misuse unrepresentable rather than warned against by rainmanjam · Pull Request #2792 · github/awesome-copilot

rainmanjam · 2026-08-25T00:34:39Z

What this adds

skills/poka-yoke/ — mistake-proofing applied to code. The skill pushes toward devices that make a wrong action impossible or self-announcing, rather than instructions asking someone to avoid it, and it refuses to accept a comment or a convention as a fix.

# the mistake is expressible: nothing stops refunding an order that was never paid
def refund(order: dict) -> Refund: ...

# the mistake is no longer expressible
def refund(order: PaidOrder) -> Refund: ...

The gap it addresses

Your contributing guide asks submissions not to duplicate existing model strengths without meaningful uplift, so here is the specific behaviour and its measurement.

Given a design, models readily list what to fix. They rarely state what the fix makes impossible, which is the difference between advice you agree with and a constraint you can rely on. Across 591 blind-graded runs and six model families, responses named the foreclosed set 42% of the time unprompted and 81% with this skill applied. Assertions were written before the runs; the grader never saw which configuration produced a response.

And its limits, which are in the skill body rather than omitted:

Every run was the first turn of a fresh session. That measures the ceiling, not what survives a long working session.
The comparison was against no methodology, not a different one, so it does not establish that this particular method produced the gain.
It costs something measurable: responses became somewhat worse at spotting the specific defect already on the page while becoming better at changing the shape that allowed it. For finding the bug in front of you, a reviewer is the better tool.

Bundled assets

All self-contained, no network access, no dependencies, 76 KB total.

Path	What it is
`scripts/detect_hazards.py`	Standard-library scanner. 42 pattern rules across 20 hazard shapes in TypeScript, Python, Go, Rust and SQL. Exits non-zero on findings; a scan of zero files exits non-zero rather than reporting a clean bill of health
`references/hazard-catalog.md`	The taxonomy, with the device that closes each shape
`references/lang-*.md`	Language-specific patterns

Checks

npm run skill:validate → ✅ poka-yoke is valid (all 413 skills pass)
npm run build → README tables regenerated, included in this PR
Scanner smoke-tested from inside the skill folder against an external repo
No existing skill covers poka-yoke, mistake-proofing, or hazard-shape analysis; nearest neighbours are domain-specific reviewers

Source

MIT. Raw benchmark runs, the harness, the grader prompts and the assertion checklists are public at https://github.com/rainmanjam/poka-yoke.

…gainst Mistake-proofing, applied to code. The skill pushes toward devices that make a wrong action impossible or self-announcing, rather than instructions asking someone to avoid it, and refuses to accept a comment or a convention as a fix. The gap it addresses. Given a design, models readily list what to fix and rarely state what the fix makes impossible. Measured across 591 blind-graded runs and six model families, responses named the foreclosed set 42% of the time unprompted and 81% with this skill applied. Assertions were written before the runs and graded blind to configuration. Stated with its limits, because they matter: every run was the first turn of a fresh session, so this measures the ceiling rather than what survives a long working session; the comparison was against no methodology rather than a different one, so it does not establish that this particular method caused the gain; and the skill costs something measurable, making responses somewhat worse at spotting the specific defect already on the page while better at changing the shape that allowed it. All of that is in the skill body rather than omitted. Bundled, all self-contained, no network access and no dependencies: scripts/detect_hazards.py standard-library scanner, 42 pattern rules across 20 hazard shapes, five languages references/hazard-catalog.md the taxonomy with device per shape references/lang-*.md Python, TypeScript, Go and Rust patterns Verified with npm run skill:validate and npm run build. Raw benchmark runs, the harness and the assertion checklists are public at https://github.com/rainmanjam/poka-yoke

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-08-25T00:36:35Z

🔍 Vally Lint Results

✅ All checks passed

Scope	Checked
Skills	1
Agents	0
Total	1

Severity	Count
❌ Errors	0
⚠️ Warnings	0
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	✅ poka-yoke (2/2 checks passed)
ℹ️	✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️	✓ spec-compliance: All spec checks passed.
ℹ️	✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️	✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️	1 skill(s) linted, 1 passed

Full linter output

### Linting skills/poka-yoke
✅ poka-yoke (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

codespell failed the PR: it reads HasTable as a misspelling of hashtable. The identifier is a legitimate generic parameter on a typestate builder, QueryBuilder<HasTable, HasWhere>, so this is a false positive, but it is our file breaking their gate and the fix costs nothing. HasFrom is also the better name: the flag tracks whether .from() has been called, not whether a table exists. Renamed in the upstream repository too, so the two copies do not diverge on day one. Verified the whole bundle is codespell-clean locally.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-08-25T00:38:02Z

🔒 PR Risk Scan Results

Scanned 6 changed file(s).

Severity	Count
🔴 High	0
🟠 Medium	0
ℹ️ Info	1

Severity	Rule	File	Line	Match
ℹ️	`skill-script-touched`	`skills/poka-yoke/scripts/detect_hazards.py`	1	`skills/poka-yoke/scripts/detect_hazards.py`

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

…verification (#16) The README claimed models name what a design forecloses 42% of the time unprompted and 81% with the skills, "across 80 graded verdicts". Recomputed from the stored gradings, over the six models the summary describes, it is 45% -> 80% across 132 verdicts. An exhaustive search over every subset of scenarios finds no combination producing 42% -> 81%. The gradings have not moved since 3eb21d4 (2026-08-23) and the figure was published after, so this was never a case of data drifting out from under a number. It survived because the test that verifies the trade table skipped that row, with a comment saying it was "covered by its own aggregate elsewhere". There was no elsewhere. The single most-quoted number in the project was the one figure exempt from the device that checks the others, and it is the same sentence now sitting in the OpenAI bundle and in github/awesome-copilot#2792. Also: - The average hid where the effect lives. Asked squarely to design an interface models already do this 77% of the time; the skills add eleven points. The gains are in the scenarios where nobody asked for a design review: build-endpoint 14% -> 79%, build-agent-feature 33% -> 83%, build-form 29% -> 64%. The per-scenario table is published alongside the mean. - preregister_arms.py now hashes what a run can reach (markdown, python, shell) rather than only markdown, so the treatment arm's bundled scripts can no longer change under a green check — a blind spot that ran one way, in favour of the arm being tested. Packaging is excluded deliberately: manifests carry the version and would fail the lock on every release for a reason unrelated to the treatment. - --check now refuses arms that are registered but not runnable. with_defensive has 2 of 10 routes and with_placebo 5, and the lock was green. A sweep would have routed half its scenarios to a missing sub-skill and reported the absence of instructions as a control. - by_model carries runs, min_cell_n and median_cell_n. The most-read table gave a reader no way to tell a row built from single-run cells from one built from seven-run cells. - The narrowing warning is now a refusal. CLAUDE.md says it "went unread three times in one session" because output gets piped through tail. It happened a fourth time during this review: a bare --aggregate-only zeroed every published pass rate and the warning scrolled past above the tail window. A notice you can pipe away is not a device. - Aggregation pairs on the arms that have runs. CONFIGS grew from two to four when the control arms were written, and `all(... for c in CONFIGS)` then made `paired` empty for every model. Nobody re-aggregated between the expansion and now, so the first re-aggregation silently emptied every row. - A router scenario. Ten specialists had scenarios; the entry point had none, and its claim that the method works on a runbook or a checklist had never been tested. The subject is deliberately not software. Claude-Session: https://claude.ai/code/session_01VW5SAyUSPxJnpLijYSYums

The submitted numbers (42% -> 81%) could not be reproduced from the upstream gradings. Recomputed over the six models the sentence describes: 45% -> 80% across 132 verdicts. Adds the per-scenario breakdown, which is the more useful claim: the gains are in tasks where nobody asked for a design review.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

rainmanjam · 2026-08-25T17:24:17Z

Correcting a figure in this submission before it is reviewed, rather than after.

The evidence paragraph claimed 42% → 81%. Auditing the upstream benchmark this morning, I
could not reproduce that from the stored gradings under any population — an exhaustive search
over subsets of scenarios found nothing yielding it. Recomputed over the six model families
the sentence describes: 45% → 80% across 132 graded verdicts. The direction and size of
the effect are unchanged; the numbers were not reproducible and now are.

It survived because the upstream test that verifies the other published figures skipped that
one row. That is fixed upstream too, so the figure is now checked against the raw gradings on
every commit.

I have also added the per-scenario breakdown, which is the more useful claim: the average was
hiding where the effect lives. Asked squarely to design an interface, models already name what
a design forecloses 77% of the time and the skills add eleven points. The large gains are in
tasks where nobody asked for a design review — endpoint 14% → 79%, agent feature 33% → 83%,
form 29% → 64%.

One other thing worth flagging, since it was in this file as submitted. The line
# CI, exits non-zero on findings was not true when I opened this PR — the detector ended
in a bare return 0 and printed findings while exiting 0. That is fixed in upstream v0.2.0
and I verified both claims in this skill against the current code before pushing: findings
exit 1, a zero-file scan exits 2.

No other changes. Apologies for the churn.

github-actions · 2026-08-25T17:24:37Z

🟡 Contributor Reputation Check: MEDIUM risk

Check	Risk
Profile	MEDIUM
Credential audit	NONE

Maintainers: please review this contributor before merging.
See the workflow run for full details.
Automated check powered by AGT.

Copilot AI balanced review requested due to automatic review settings August 25, 2026 00:34

rainmanjam requested a review from aaronpowell as a code owner August 25, 2026 00:34

Copilot AI reviewed Aug 25, 2026

github-actions Bot added new-submission PR adds at least one new contribution skills PR touches skills labels Aug 25, 2026

Copilot AI review requested due to automatic review settings August 25, 2026 00:37

Copilot AI reviewed Aug 25, 2026

Copilot AI review requested due to automatic review settings August 25, 2026 17:24

Copilot AI reviewed Aug 25, 2026

github-actions Bot added the needs-review:MEDIUM Contributor reputation check flagged MEDIUM risk label Aug 25, 2026

aaronpowell approved these changes Aug 25, 2026

View reviewed changes

aaronpowell merged commit 7c36d87 into github:main Aug 25, 2026
16 checks passed

rainmanjam mentioned this pull request Aug 26, 2026

Add the ten poka-yoke specialist skills #2808

Open

Navigation Menu

Uh oh!

Add poka-yoke skill: make misuse unrepresentable rather than warned against - #2792

Add poka-yoke skill: make misuse unrepresentable rather than warned against#2792
aaronpowell merged 3 commits into
github:mainfrom
rainmanjam:add-poka-yoke-skill

rainmanjam commented Aug 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

rainmanjam commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rainmanjam commented Aug 25, 2026

What this adds

The gap it addresses

Bundled assets

Checks

Source

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 25, 2026

🔍 Vally Lint Results

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 25, 2026

🔒 PR Risk Scan Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

rainmanjam commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants