Block citations resolve to bounding boxes: get_block() and resolve_citations() - #503
Merged
Merged
Conversation
…tations()
resolve_citations(answer, doc_ids=None) reads the <cite doc= page= block=/>
and <doc=…;page=…;block=…> tags of a cited answer and returns one entry per
distinct citation with the document id and, for block-level citations on
cloud documents, the block's page, bbox, type and text from the new
GET /doc/{doc_id}/block/{block_id}/ route, wrapped as get_block(). Local
mode stays page-level; a name shared by two documents raises instead of
guessing; a block the document lacks keeps its entry without a bbox.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…kups _CITE_ATTR_RE pairs its quotes, so an apostrophe inside a quoted name (Moody's Outlook.pdf) stays in the name instead of ending it. _OLD_CITATION_RE stops a name at < or >, so a <doc=name> tag without its own ';' no longer swallows the text up to the next tag's ';' and that citation with it. The block lookup keeps its bbox-less entry on a status-less raise (local mode, where pages have no blocks) and on 403, as doc_targeting_block does; a transport failure still propagates with its status. The library listing is _all_documents(), which tolerates an absent 'total' and ends on the empty page; ids are deduplicated before the collision check, so a document re-served after an upload shifted the window is one document, not a collision with itself. doc_ids=[] fails loud like chat(doc_id=[]) instead of resolving nothing. Docstrings: the bbox coordinate space on both get_block(); the shared-document promise dropped, since the metadata route is owner-only.
_CITE_TAG_RE's second alternative (the paired form) was unreachable: the first alternative matches the opening tag whenever a '>' follows, and without one neither alternative can match. Dropping it, stopping the tag body at '<' and letting the group absorb the whitespace makes an unterminated '<cite ' scan linear: '<cite ' plus 4000 spaces went from 2 minutes to 0.1 ms, 99 KB of unterminated tags from 27 minutes to about a millisecond. Output is identical on every shape tried, the paired form included.
Two holes of the kinds already closed, left open one line and one loop
away from their siblings.
The old format's block group still crossed tag boundaries the way its
name group used to: "<doc=a.pdf;page=3;block=p3_text_5 <cite doc=..." put
the whole following citation inside block_id and, since a block lookup
that 404s is now kept without a bbox, lost it silently. It stops at the
tag's edge like the name beside it; a block id has never contained < or
>, the server writes p{page}_{type}_{seq}.
The library listing reads entries with .get() like the rest of the
codebase does: _all_documents() tolerates a listing whose shape varies,
so reading its result with raw subscripts put the KeyError back one
level down. An entry without a name or an id cannot answer a citation
either way, so it is skipped.
…like chat() add() trims both values itself: the name, which each caller used to trim, and the block id, which neither did. "block=p3_text_5 >" and block=" p3_text_5 " now look up p3_text_5. A padded id 404s, and a 404 is kept without a bbox, so the padding lost the bbox silently and handed the padded id back to the caller. _CITE_ATTR_RE anchors the attribute name at a word boundary. Without \b the greedy (\w+) restarted at every character of a long word inside a tag body: '<cite ' + 64 KB of x + '>' cost 19 s on the caller's thread. Anchored, 64 KB is 1.6 ms and 1 MB 27 ms. Output is identical: the leftmost match of (\w+)= always begins at a word boundary (0 diffs on 30,000 random inputs). The linear-time test now reaches this expression; its inputs had no '>' and never did. The scope parameter is doc_id, the name chat(), chat_completions() and document_context() give the same str-or-list argument, and the one the docstring already used to describe it. The integration builders keep their doc_ids.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two client methods for turning block-level citations into page coordinates:
client.get_block(doc_id, block_id)— one layout block as the API returns it (doc_id, page, block_id, bbox, block_type, text), viaGET /doc/{doc_id}/block/{block_id}/.bboxis[x0, y0, x1, y1]in thousandths of the page's width and height (0-1000), origin top-left. Cloud-only; local mode raises, because local page content has no blocks.client.resolve_citations(answer, doc_id=None)— parses the<cite doc= page= block=/>tags own-modelchat(citations=True)writes and the<doc=…;page=…;block=…>tags the managed chat writes (the cloud parser's expressions, tightened: quotes paired, names and tag bodies stopped at the tag's edge, attribute names anchored at a word boundary, so an unterminated or malformed tag scans in linear time; names and block ids trimmed), deduplicates, maps document names to ids, and mergesget_block()'s fields into each block-level entry. Output is the managed chat'scitationsentry shape plusdoc_idandtext.Semantics
doc_id— whatchat(doc_id=...)took — scopes the name → id lookup; without it the client lists your own library (100 per page, ids deduplicated).doc_id=[]fails loud likechat(doc_id=[]).PageIndexAPIErrornaming both ids — no guessing.bbox; so does a block that cannot be looked up (local mode, a document you cannot read). Every other status re-raises.doc_id: None. The markdown / footnote formats ofcitation_prompt()are prose and are not parsed.Depends on
VectifyAI/pageindex-compute PR #607 — the block route — squash-merged into
devas 6842852a and deployed to Dev (Sep 13). Live-verified there against a document submitted withbeta_headers=["block_reference"]:get_block()returns the bbox;resolve_citations()merges it on both the library and thedoc_id=path, keeps a missing block without a bbox, returnsdoc_id: Nonefor an unknown name, and resolves the<doc=…;page=…;block=…>tag the managed chat itself wrote (enable_citations=True) to a table block's bbox. Tests here run against fakes.Tests
11 in
tests/test_client.py: parser cases for both tag formats (padded names and block ids trimmed), a linear-time bound on unterminated tags and on a tag whose body is one long word, cloud resolution (found / missing block / unknown document /doc_idscope including a repeated id / argument guards includingdoc_id=[]), an apostrophe inside a quoted name next to a bare<doc=…>tag, full-library paging, a listing withouttotalthat re-serves a document after an upload shifted the window, shared-name refusal, a 403 block kept and a 500 re-raised, request wiring, and local mode's page-level result plus the cloud-only refusal onget_blockand the bbox-less block entry. Full suite 619 green, pyright clean.Docs (
chat.mdx/documents.mdx) and the release note follow with the release.