Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/design/mechanisms.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr

## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that

Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix, and is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage) and nameparser/_pipeline/_pieces.py over pieces: _is_suffix_piece, _is_leading_title, _leading_titles, _peel_walk, _peel_trailing and _segment_holds_no_name are called by both stages, _is_title_piece and _trailing_start by group alone — `_trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead.
Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix, and is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, is_leading_title, leading_titles, peel_walk, peel_trailing and segment_holds_no_name are called by both stages, is_title_piece and trailing_start by group alone — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead.

## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins

Expand Down
24 changes: 12 additions & 12 deletions nameparser/_pipeline/_assign.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@
effective_script, is_suffix_lenient, resolve_script_set,
)
from nameparser._pipeline._pieces import (
_is_suffix_piece, _leading_titles, _peel_trailing, _peel_walk,
_segment_holds_no_name,
is_suffix_piece, leading_titles, peel_trailing, peel_walk,
segment_holds_no_name,
)
from nameparser._pipeline._state import (
ParseState, PendingAmbiguity, Structure, WorkToken,
Expand All @@ -60,14 +60,14 @@ def _set_roles(tokens: list[WorkToken], piece: tuple[int, ...],
# rules.md#H2: "an abbreviation opening the part of the name that
# carries the given name — the whole name, or the part after a
# family comma — reads as a title even when unlisted" -- the count is
# _pieces._leading_titles since #424 (its test, _is_leading_title, is
# _pieces.leading_titles since #424 (its test, is_leading_title, is
# the leading-particle scan's too); the roles are set here.
def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...],
ptags: tuple[frozenset[str], ...],
tokens: list[WorkToken]) -> int:
"""Assign TITLE to the leading title pieces and return the first
non-title index."""
n = _leading_titles(pieces, ptags, tokens)
n = leading_titles(pieces, ptags, tokens)
for k in range(n):
_set_roles(tokens, pieces[k], Role.TITLE)
return n
Expand Down Expand Up @@ -175,7 +175,7 @@ def _assign_main(seg_idx: int, state: ParseState,
flagged = [k for k in rest if "suffix" in ptags[k]]
for k in flagged:
_set_roles(tokens, pieces[k], Role.SUFFIX)
rest = _peel_walk(n, ptags)
rest = peel_walk(n, ptags)
if not rest:
return None
# rules.md#N3: "a name that is only a nickname and one name word
Expand All @@ -188,15 +188,15 @@ def _assign_main(seg_idx: int, state: ParseState,
_set_roles(tokens, pieces[rest[0]], Role.FAMILY)
return None
# peel the trailing suffix run: k = first index in rest from which
# every piece is a suffix. The walk is _pieces._peel_trailing since
# every piece is a suffix. The walk is _pieces.peel_trailing since
# #425 -- one walk, shared with the bound-given reserve, and
# documented there. Every bare ambiguous acronym it had to resolve
# is one coin-flip each, in either direction, so the report
# collects rather than overwrites. Deferred to after assignment
# because the wording reads the role back, and which role "not
# peeled" means depends on name_order. (The roman-numeral fork
# needs no such deferral and is reported here.)
peeled = _peel_trailing(rest, pieces, ptags, tokens)
peeled = peel_trailing(rest, pieces, ptags, tokens)
if peeled.numeral is not None:
# a trailing single letter is a name part unless it happens
# to be a roman numeral -- and V/X/I are ordinary middle
Expand Down Expand Up @@ -300,15 +300,15 @@ def assign(state: ParseState) -> ParseState:
# positional read peels a trailing suffix first: 'Smith Jr.,
# Mr.' has two pieces and one name, and read positionally lost
# its family (the code review).
no_name = _segment_holds_no_name(state.pieces[1],
no_name = segment_holds_no_name(state.pieces[1],
state.piece_tags[1], tokens)
if no_name and sum(
1 for k, piece in enumerate(fam_pieces)
if not _is_suffix_piece(piece, fam_tags[k], tokens)) > 1:
if not is_suffix_piece(piece, fam_tags[k], tokens)) > 1:
order = _assign_main(0, state, tokens, ambiguities)
else:
for k, piece in enumerate(fam_pieces):
if k > 0 and _is_suffix_piece(piece, fam_tags[k], tokens):
if k > 0 and is_suffix_piece(piece, fam_tags[k], tokens):
_set_roles(tokens, piece, Role.SUFFIX)
else:
_set_roles(tokens, piece, Role.FAMILY)
Expand Down Expand Up @@ -338,7 +338,7 @@ def assign(state: ParseState) -> ParseState:
if no_name:
for k, piece in enumerate(pieces):
_set_roles(tokens, piece,
Role.SUFFIX if _is_suffix_piece(
Role.SUFFIX if is_suffix_piece(
piece, ptags[k], tokens)
else Role.TITLE)
n = len(pieces)
Expand All @@ -363,7 +363,7 @@ def assign(state: ParseState) -> ParseState:
# initial, so strict only
last_of_two = (m == len(pieces) - 1
and len(state.segments) == 2)
if _is_suffix_piece(pieces[m], ptags[m], tokens) or (
if is_suffix_piece(pieces[m], ptags[m], tokens) or (
last_of_two and len(pieces[m]) == 1
and is_suffix_lenient(
tokens[pieces[m][0]].text, state.lexicon)):
Expand Down
Loading