diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 8fc606ab..bb470729 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix, and is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage) and nameparser/_pipeline/_pieces.py over pieces: _is_suffix_piece, _is_leading_title, _leading_titles, _peel_walk, _peel_trailing and _segment_holds_no_name are called by both stages, _is_title_piece and _trailing_start by group alone — `_trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix, and is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, is_leading_title, leading_titles, peel_walk, peel_trailing and segment_holds_no_name are called by both stages, is_title_piece and trailing_start by group alone — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 6696597d..6a0bfdbf 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -42,8 +42,8 @@ effective_script, is_suffix_lenient, resolve_script_set, ) from nameparser._pipeline._pieces import ( - _is_suffix_piece, _leading_titles, _peel_trailing, _peel_walk, - _segment_holds_no_name, + is_suffix_piece, leading_titles, peel_trailing, peel_walk, + segment_holds_no_name, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, @@ -60,14 +60,14 @@ def _set_roles(tokens: list[WorkToken], piece: tuple[int, ...], # rules.md#H2: "an abbreviation opening the part of the name that # carries the given name — the whole name, or the part after a # family comma — reads as a title even when unlisted" -- the count is -# _pieces._leading_titles since #424 (its test, _is_leading_title, is +# _pieces.leading_titles since #424 (its test, is_leading_title, is # the leading-particle scan's too); the roles are set here. def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], ptags: tuple[frozenset[str], ...], tokens: list[WorkToken]) -> int: """Assign TITLE to the leading title pieces and return the first non-title index.""" - n = _leading_titles(pieces, ptags, tokens) + n = leading_titles(pieces, ptags, tokens) for k in range(n): _set_roles(tokens, pieces[k], Role.TITLE) return n @@ -175,7 +175,7 @@ def _assign_main(seg_idx: int, state: ParseState, flagged = [k for k in rest if "suffix" in ptags[k]] for k in flagged: _set_roles(tokens, pieces[k], Role.SUFFIX) - rest = _peel_walk(n, ptags) + rest = peel_walk(n, ptags) if not rest: return None # rules.md#N3: "a name that is only a nickname and one name word @@ -188,7 +188,7 @@ def _assign_main(seg_idx: int, state: ParseState, _set_roles(tokens, pieces[rest[0]], Role.FAMILY) return None # peel the trailing suffix run: k = first index in rest from which - # every piece is a suffix. The walk is _pieces._peel_trailing since + # every piece is a suffix. The walk is _pieces.peel_trailing since # #425 -- one walk, shared with the bound-given reserve, and # documented there. Every bare ambiguous acronym it had to resolve # is one coin-flip each, in either direction, so the report @@ -196,7 +196,7 @@ def _assign_main(seg_idx: int, state: ParseState, # because the wording reads the role back, and which role "not # peeled" means depends on name_order. (The roman-numeral fork # needs no such deferral and is reported here.) - peeled = _peel_trailing(rest, pieces, ptags, tokens) + peeled = peel_trailing(rest, pieces, ptags, tokens) if peeled.numeral is not None: # a trailing single letter is a name part unless it happens # to be a roman numeral -- and V/X/I are ordinary middle @@ -300,15 +300,15 @@ def assign(state: ParseState) -> ParseState: # positional read peels a trailing suffix first: 'Smith Jr., # Mr.' has two pieces and one name, and read positionally lost # its family (the code review). - no_name = _segment_holds_no_name(state.pieces[1], + no_name = segment_holds_no_name(state.pieces[1], state.piece_tags[1], tokens) if no_name and sum( 1 for k, piece in enumerate(fam_pieces) - if not _is_suffix_piece(piece, fam_tags[k], tokens)) > 1: + if not is_suffix_piece(piece, fam_tags[k], tokens)) > 1: order = _assign_main(0, state, tokens, ambiguities) else: for k, piece in enumerate(fam_pieces): - if k > 0 and _is_suffix_piece(piece, fam_tags[k], tokens): + if k > 0 and is_suffix_piece(piece, fam_tags[k], tokens): _set_roles(tokens, piece, Role.SUFFIX) else: _set_roles(tokens, piece, Role.FAMILY) @@ -338,7 +338,7 @@ def assign(state: ParseState) -> ParseState: if no_name: for k, piece in enumerate(pieces): _set_roles(tokens, piece, - Role.SUFFIX if _is_suffix_piece( + Role.SUFFIX if is_suffix_piece( piece, ptags[k], tokens) else Role.TITLE) n = len(pieces) @@ -363,7 +363,7 @@ def assign(state: ParseState) -> ParseState: # initial, so strict only last_of_two = (m == len(pieces) - 1 and len(state.segments) == 2) - if _is_suffix_piece(pieces[m], ptags[m], tokens) or ( + if is_suffix_piece(pieces[m], ptags[m], tokens) or ( last_of_two and len(pieces[m]) == 1 and is_suffix_lenient( tokens[pieces[m][0]].text, state.lexicon)): diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index bfe0050d..fa25a5cb 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -23,7 +23,7 @@ The piece-level predicates moved to _pieces in #439 -- the S2 trailing peel, the leading-title and title-piece tests, the suffix-piece test, the no-name-segment test. Most are shared with -assign; _is_title_piece and _trailing_start are group's alone and +assign; is_title_piece and trailing_start are group's alone and travelled because the shared ones call them. They had collected here by import direction rather than by topic (assign imported group and could not be imported back), which is the accumulation @@ -41,15 +41,14 @@ from nameparser._lexicon import _title_key from nameparser._pipeline._pieces import ( - _is_leading_title, _is_suffix_piece, _is_title_piece, - _leading_titles, _peel_trailing, _peel_walk, _segment_holds_no_name, - _trailing_start, + is_leading_title, is_suffix_piece, is_title_piece, + leading_titles, peel_trailing, peel_walk, segment_holds_no_name, + trailing_start, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, ) -from nameparser._pipeline._vocab import D as _D -from nameparser._pipeline._vocab import PH as _PH +from nameparser._pipeline._vocab import D, PH from nameparser._pipeline._vocab import delimiter_cores from nameparser._types import AmbiguityKind, Role @@ -81,7 +80,7 @@ class BoundJoin(IntEnum): # rules.md#S2: "a trailing word of the suffix vocabulary reads as a # suffix" -- group does not decide that; it stops before whatever -# _trailing_start says the run is, so the chain and the maiden walk +# trailing_start says the run is, so the chain and the maiden walk # end where assign's peel begins (#424). # rules.md#P2: "a particle joins the words after it into one name # part, the join running until the next particle starts a group of @@ -163,14 +162,14 @@ def _maiden_take(pieces: Sequence[Sequence[int]], # marker ends the maiden name, and so does the trailing numeral as # assign will read it, which the suffix-piece test does not see # (#424): 'John née Jones Smith V' took the V as maiden text. The - # numeral only -- _trailing_start says why the acronym fork is + # numeral only -- trailing_start says why the acronym fork is # left to assign here. Read from the MARKER, not after it: a # numeral straight after the marker then has the piece before it # the fork wants, and 'Jane Smith née V' declines like 'Jane Smith # née PhD' -- nothing after the marker but a suffix, so the marker # stays a word -- as 1.4.0 read it. skip = frozenset(range(len(pieces))) - frozenset(seen) - trailing = _trailing_start(seen[m], pieces, ptags, tokens, skip, + trailing = trailing_start(seen[m], pieces, ptags, tokens, skip, numeral_only=True) # The fork reads the piece before the numeral, and the take # REMOVES that piece: afterwards assign sees the piece before the @@ -187,13 +186,13 @@ def _maiden_take(pieces: Sequence[Sequence[int]], left = [i for i in seen if i < seen[m] or i >= trailing] view = [pieces[i] for i in left] view_tags = [ptags[i] for i in left] - if _trailing_start(_leading_titles(view, view_tags, tokens), + if trailing_start(leading_titles(view, view_tags, tokens), view, view_tags, tokens, numeral_only=True) == len(view): trailing = len(pieces) j = m + 1 while (j < len(seen) and seen[j] < trailing - and not _is_suffix_piece(pieces[seen[j]], ptags[seen[j]], + and not is_suffix_piece(pieces[seen[j]], ptags[seen[j]], tokens)): j += 1 # j == m + 1 means nothing followed the marker but a suffix, so the @@ -217,9 +216,9 @@ def _is_rootname(piece: Sequence[int], ptags: Set[str], tokens: Sequence[WorkToken]) -> bool: if len(piece) == 1 and "initial" in tokens[piece[0]].tags: return False - return not (_is_title_piece(piece, ptags, tokens) + return not (is_title_piece(piece, ptags, tokens) or _is_prefix_piece(piece, ptags, tokens) - or _is_suffix_piece(piece, ptags, tokens)) + or is_suffix_piece(piece, ptags, tokens)) def _group_segment(seg: tuple[int, ...], additional: int, @@ -238,13 +237,13 @@ def _group_segment(seg: tuple[int, ...], additional: int, ambiguities = [] def title(k: int) -> bool: - return _is_title_piece(pieces[k], ptags[k], tokens) + return is_title_piece(pieces[k], ptags[k], tokens) def prefix(k: int) -> bool: return _is_prefix_piece(pieces[k], ptags[k], tokens) def suffix(k: int) -> bool: - return _is_suffix_piece(pieces[k], ptags[k], tokens) + return is_suffix_piece(pieces[k], ptags[k], tokens) def conj(k: int) -> bool: return _is_conj_piece(pieces[k], ptags[k], tokens) @@ -302,8 +301,8 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), while k < len(pieces) - 1: a, b = pieces[k], pieces[k + 1] if (len(a) == 1 and len(b) == 1 - and _PH.fullmatch(tokens[a[0]].text) - and _D.fullmatch(tokens[b[0]].text)): + and PH.fullmatch(tokens[a[0]].text) + and D.fullmatch(tokens[b[0]].text)): merge(k, k + 2, add={"suffix"}) else: k += 1 @@ -436,7 +435,7 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # word and left assign two pieces where the fork counted # three). The scan asks assign's own test. leading = next((k for k in range(len(pieces)) - if not _is_leading_title(pieces[k], ptags[k], + if not is_leading_title(pieces[k], ptags[k], tokens) or prefix(k)), 0) # rules.md#P2: "a trailing suffix begins" -- where it begins @@ -453,8 +452,8 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # van der Berg V' read family 'van der Berg V'). The chain # takes both forks, and asks again after its merges whether # the acronym still has the pieces the fork counted (below). - name_start = _leading_titles(pieces, ptags, tokens) - tail = len(pieces) - _trailing_start(name_start, pieces, ptags, + name_start = leading_titles(pieces, ptags, tokens) + tail = len(pieces) - trailing_start(name_start, pieces, ptags, tokens) def chain(tail: int) -> None: k = 0 @@ -476,7 +475,7 @@ def chain(tail: int) -> None: # A fork whose two sides are decided in different stages # needs an emitter in each. # - # Narrow, and #367 is why. `all(_is_leading_title(...))` + # Narrow, and #367 is why. `all(is_leading_title(...))` # says every piece ahead of this one is a title, and the # loop skipped k == leading, so `leading` is STRICTLY # before k -- and being before k it is one of those titles, @@ -518,7 +517,7 @@ def chain(tail: int) -> None: if (j > k + 1 and "vocab:particle-ambiguous" in tokens[pieces[k][0]].tags - and all(_is_leading_title(pieces[x], ptags[x], + and all(is_leading_title(pieces[x], ptags[x], tokens) for x in range(k))): i = pieces[k][0] @@ -551,8 +550,8 @@ def chain(tail: int) -> None: kept = ([list(q) for q in pieces], [set(t) for t in ptags], len(ambiguities)) chain(tail) - left = len(pieces) - _trailing_start( - _leading_titles(pieces, ptags, tokens), pieces, ptags, + left = len(pieces) - trailing_start( + leading_titles(pieces, ptags, tokens), pieces, ptags, tokens) if left < tail: pieces[:], ptags[:] = kept[0], kept[1] @@ -569,7 +568,7 @@ def chain(tail: int) -> None: # title test does not see H2's unlisted abbreviations, and # 'Xyz. abdul John Smith' joined nothing where 'Dr. abdul John # Smith' read given 'abdul John'. - fk = _leading_titles(pieces, ptags, tokens) + fk = leading_titles(pieces, ptags, tokens) if (bound_join is not BoundJoin.DISABLED and fk + 1 < len(pieces) and len(pieces[fk]) == 1 @@ -609,14 +608,14 @@ def chain(tail: int) -> None: # nothing joined, 'abdul Smith Ma' peels the acronym # unjoined and keeps it joined. Shapes pinned in # test_group.py. - rest = _peel_walk(fk, ptags) - before = _peel_trailing(rest, pieces, ptags, tokens) + rest = peel_walk(fk, ptags) + before = peel_trailing(rest, pieces, ptags, tokens) view, view_tags = list(pieces), list(ptags) view[fk:fk + 2] = [pieces[fk] + pieces[fk + 1]] view_tags[fk:fk + 2] = [joined_tags(fk, fk + 2, drop={"title"})] - view_rest = _peel_walk(fk, view_tags) - after = _peel_trailing(view_rest, view, view_tags, tokens) + view_rest = peel_walk(fk, view_tags) + after = peel_trailing(view_rest, view, view_tags, tokens) same_suffixes = ( [tuple(view[j]) for j in view_rest[after.names:]] == [tuple(pieces[j]) for j in rest[before.names:]]) @@ -713,7 +712,7 @@ def group(state: ParseState) -> ParseState: # rather than tested. one_entry = tail or ( family_comma and seg_idx == 1 - and _segment_holds_no_name(pieces, ptags, tokens)) + and segment_holds_no_name(pieces, ptags, tokens)) if one_entry: # v1 renders each tail COMMA SEGMENT as one suffix entry # ('Smith, V MD' -> suffix 'V MD'); a delimiter core inside @@ -754,7 +753,7 @@ def group(state: ParseState) -> ParseState: # glued across the writer's own comma ('Smith Jr., Mr. # Jr.' rendered suffix 'Jr. Jr.') -- the inverse of the # bug this block exists to fix. - in_entry = tail or _is_suffix_piece( + in_entry = tail or is_suffix_piece( pieces[k], ptags[k], tokens) for pos, i in enumerate(pieces[k]): if pos > 0 or (in_entry and entry_open): diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index f227d026..e1cddb46 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -13,9 +13,9 @@ grouping owned them but because assign imported group and could not be imported back, so group was the only place both stages could reach. They arrived there that way across three PRs -- #424 brought -_is_leading_title, _leading_titles and _trailing_start, #425 the peel -(_peel_walk, _peel_trailing), #429 _segment_holds_no_name. -_is_title_piece and _is_suffix_piece are older than any of that: they +is_leading_title, leading_titles and trailing_start, #425 the peel +(peel_walk, peel_trailing), #429 segment_holds_no_name. +is_title_piece and is_suffix_piece are older than any of that: they were group's from its first commit, and travel because the others call them. @@ -25,11 +25,18 @@ piece predicate may not depend on a stage, in either direction. The S2 trailing peel travels as the unit decisions.md describes -- -_peel_walk, _peel_trailing and _trailing_start together -- though only +peel_walk, peel_trailing and trailing_start together -- though only the first two cross a stage boundary. Layering: imports _state and _vocab only; _group and _assign import it, and neither of the two it imports imports it back. + +Naming follows _vocab's: inside an already-private module the leading +underscore marks module-PRIVATE, so the names other stages call are +bare and only the internals keep it (_PERIOD_ABBREV here). Getting that +backwards -- which this module did until the underscores came off -- +costs a reader the one cheap way to tell a shared predicate from a +helper. """ from __future__ import annotations @@ -44,7 +51,7 @@ # rules.md#H3: "successive title words at the start of the part # carrying the given name chain into one title; a title word # elsewhere in the name does not" -def _is_title_piece(piece: Sequence[int], ptags: Set[str], +def is_title_piece(piece: Sequence[int], ptags: Set[str], tokens: Sequence[WorkToken]) -> bool: if "title" in ptags: return True @@ -64,34 +71,34 @@ def _is_title_piece(piece: Sequence[int], ptags: Set[str], # carries the given name — the whole name, or the part after a # family comma — reads as a title even when unlisted" # (history: decisions.md#H2) -def _is_leading_title(piece: Sequence[int], ptags: Set[str], +def is_leading_title(piece: Sequence[int], ptags: Set[str], tokens: Sequence[WorkToken]) -> bool: - if _is_title_piece(piece, ptags, tokens): + if is_title_piece(piece, ptags, tokens): return True return (len(piece) == 1 and bool(_PERIOD_ABBREV.match(tokens[piece[0]].text))) -def _leading_titles(pieces: Sequence[Sequence[int]], +def leading_titles(pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken]) -> int: """How many leading pieces assign peels as titles: the first non-title index. A title needs a following piece, unless the whole segment is one title (v1 parity). One definition, read by assign (which sets the roles) and by the chain's trailing-run walk; the - leading-particle scan shares the predicate, _is_leading_title, + leading-particle scan shares the predicate, is_leading_title, but stops at a title-and-particle word (P4, #367, #424).""" n = 0 while n < len(pieces): if ((n + 1 < len(pieces) or len(pieces) == 1) - and _is_leading_title(pieces[n], ptags[n], tokens)): + and is_leading_title(pieces[n], ptags[n], tokens)): n += 1 continue break return n -def _is_suffix_piece(piece: Sequence[int], ptags: Set[str], +def is_suffix_piece(piece: Sequence[int], ptags: Set[str], tokens: Sequence[WorkToken]) -> bool: if "suffix" in ptags: return True @@ -101,7 +108,7 @@ def _is_suffix_piece(piece: Sequence[int], ptags: Set[str], return "vocab:suffix" in tags and "initial" not in tags -def _segment_holds_no_name(pieces: Sequence[Sequence[int]], +def segment_holds_no_name(pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken]) -> bool: """The segment is titles and suffixes only ('John Smith, Dr.', @@ -113,7 +120,7 @@ def _segment_holds_no_name(pieces: Sequence[Sequence[int]], Smith' with the honorific moved, and 'John Smith, Mr. Jr.' the same with the postnominal along -- so the pre-comma name keeps its positional read instead of being merged. Uses the same - _is_leading_title predicate the peel does, period-abbreviation + is_leading_title predicate the peel does, period-abbreviation inference included, so the two cannot disagree about what a title is; a suffix piece counts as what it is, so a mixed run like 'Smith, Dr. Jr.' is a title and a postnominal, each read where it @@ -128,14 +135,14 @@ def _segment_holds_no_name(pieces: Sequence[Sequence[int]], True does NOT mean "every piece is a suffix" -- the title tolerance is the whole point, and a true segment can still hold pieces assign routes to TITLE, so a caller rendering the segment as one unit must - ask _is_suffix_piece per piece as well. What assuming otherwise cost + ask is_suffix_piece per piece as well. What assuming otherwise cost is recorded at the one-entry join in group(), the caller that made the assumption. """ if not pieces: return False - return all(_is_suffix_piece(pieces[k], ptags[k], tokens) - or _is_leading_title(pieces[k], ptags[k], tokens) + return all(is_suffix_piece(pieces[k], ptags[k], tokens) + or is_leading_title(pieces[k], ptags[k], tokens) for k in range(len(pieces))) @@ -161,9 +168,9 @@ class Peel(NamedTuple): # shape any word can wear and does not. A BARE ambiguous acronym is # consumed only when the name has words to spare" # (v1's are_suffixes tail rule, with the roman-numeral special) -def _peel_walk(start: int, ptags: Sequence[Set[str]], +def peel_walk(start: int, ptags: Sequence[Set[str]], skip: Set[int] = frozenset()) -> list[int]: - """The indices _peel_trailing walks: `start` to the segment's end, + """The indices peel_trailing walks: `start` to the segment's end, minus the group-flagged credential pieces (the Ph. D. merge), which assign reads as suffixes at any position, and minus `skip` -- a tail segment's delimiter cores, which are structure rather @@ -176,7 +183,7 @@ def _peel_walk(start: int, ptags: Sequence[Set[str]], if j not in skip and "suffix" not in ptags[j]] -def _trailing_start(start: int, pieces: Sequence[Sequence[int]], +def trailing_start(start: int, pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], skip: Set[int] = frozenset(), numeral_only: bool = False) -> int: @@ -197,17 +204,17 @@ def _trailing_start(start: int, pieces: Sequence[Sequence[int]], piece, the one before the numeral, and _maiden_take re-asks it with the piece the take leaves there; the acronym is left to assign.""" - rest = _peel_walk(start, ptags, skip) - peeled = _peel_trailing(rest, pieces, ptags, tokens) + rest = peel_walk(start, ptags, skip) + peeled = peel_trailing(rest, pieces, ptags, tokens) if numeral_only: return rest[-1] if peeled.numeral is not None else len(pieces) return rest[peeled.names] if peeled.names < len(rest) else len(pieces) -def _peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], +def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken]) -> Peel: - """The S2 trailing peel over `rest`, a _peel_walk list. In the + """The S2 trailing peel over `rest`, a peel_walk list. In the piece layer rather than in assign because group's bound-given reserve asks the same question of the view the join would leave (#425): one walk, so the reserve and the assignment cannot drift. Pure -- the ambiguities are @@ -218,7 +225,7 @@ def _peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], k = len(rest) while k > 0: piece = pieces[rest[k - 1]] - if _is_suffix_piece(piece, ptags[rest[k - 1]], tokens): + if is_suffix_piece(piece, ptags[rest[k - 1]], tokens): k -= 1 continue # a final single letter that is a roman numeral, after a piece diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 447da311..63e831cc 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -2159,7 +2159,7 @@ def __post_init__(self) -> None: "_normalize strips the trailing period, " "so the vocabulary sees 씨 either way -- the initial " "veto was the only thing rejecting the written form, " - "and literally the veto: _is_suffix_piece is " + "and literally the veto: is_suffix_piece is " "'vocab:suffix' in tags and 'initial' not in tags, and " "'씨.' carried both, so the suffix-shaped piece went to " "the given. 1.4.0 read this first '씨.' / last 김민준 -- " @@ -2361,7 +2361,7 @@ def __post_init__(self) -> None: classification="fix(#307)", notes="the post-comma run is normally the given name -- " "'김민준, 태호' gives given 태호 -- and group's " - "_is_suffix_piece diverts this one because 씨 is a " + "is_suffix_piece diverts this one because 씨 is a " "single-token piece carrying vocab:suffix. NOT the " "lenient comma gate, which an earlier note named: " "measured, lenient_comma_suffixes=False leaves this " diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index e7a5b6f7..5605b01d 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -366,7 +366,7 @@ def test_positional_segment_zero_reports_the_particle_fork() -> None: def test_a_credential_run_after_a_family_comma_reads_as_suffixes() -> None: # 'Smith, Jr.' -- the peel's whole-segment exception claimed this - # even with 'jr' out of TITLES, because _is_leading_title also + # even with 'jr' out of TITLES, because is_leading_title also # infers a title from the period-abbreviation shape. The slot after # a family comma IS postnominal position, so a run that is nothing # but suffix pieces is read as one before the peel gets a chance