Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a registry-level suppression guard to prevent project-wide fallback resolution from creating fabricated CALLS edges when the caller explicitly imports a homonymous symbol from an external (non-indexed) package.
Changes:
- Introduces
cbm_suppress_external_import_shadow()to suppress fallback-resolved edges for bare identifiers that are imported from non-relative specifiers but do not bind in the import map. - Applies the guard consistently in both sequential (
pass_calls.c) and parallel (pass_parallel.c) emission paths. - Adds focused unit + pipeline tests reproducing #1355 and validating behavior for sequential vs parallel indexing.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_registry.c | Adds unit tests for the new suppression predicate’s contract and edge cases. |
| tests/test_pipeline.c | Adds an end-to-end fixture + assertions to verify the regression is fixed in both sequential and parallel resolvers. |
| src/pipeline/registry.c | Implements cbm_suppress_external_import_shadow() plus helpers. |
| src/pipeline/pipeline.h | Exposes the new suppression predicate in the public pipeline header. |
| src/pipeline/pass_parallel.c | Invokes the guard in the parallel call-resolution emission path. |
| src/pipeline/pass_calls.c | Threads file_imports into resolution and invokes the guard in the sequential emission path. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| char *old_workers = getenv("CBM_WORKERS"); | ||
| char *saved_workers = old_workers ? strdup(old_workers) : NULL; | ||
| char *old_single = getenv("CBM_INDEX_SINGLE_THREAD"); |
| /* Enough files that CBM_WORKERS can take the fused-parallel path; the same | ||
| * tree is then indexed by each resolver in turn, because the guard lives at | ||
| * two independent emit sites (pass_calls.c and pass_parallel.c). */ | ||
| write_external_import_shadow_fixture(tmp, 50); |
| * "/abs/x") rather than an external package. Package specifiers are everything | ||
| * else — "drizzle-orm", "rxjs/operators", "@scope/pkg". */ | ||
| static bool specifier_is_relative(const char *module_path) { | ||
| return module_path && (module_path[0] == '.' || module_path[0] == '/'); |
|
Thank you for the detailed reproduction, the sequential and parallel coverage, and for documenting the known workspace tradeoff explicitly. This changes project-wide import-resolution behavior, so we are reviewing the approach carefully and will come back with a maintainer decision before asking you to rework anything. Our review queue is currently full, so this may take a little time. Thank you for your patience and for making the tradeoff visible. |
9ce0c1c to
f78bfe2
Compare
|
Reviewed. The predicate is well built, the reproduction is convincing, and there is a standing constraint on this class of change that decides the timing rather than the merit. The predicate itselfI traced it, and the conjunction is tight in the right way: strategy limited to the four project-wide guesses, callee must be a bare identifier (no Excluding relative specifiers on purpose is the right line. A relative path names something inside the tree, so a missing And the reproduction earns its conclusion. What holds it upThis PR is one of a named cluster — #1128, #1324, #1386, #1702 and this one — and there is a standing decision that they are judged against one shared census rather than one at a time, because whichever lands first shifts the baseline the others are measured on. You already spotted the nearest neighbour yourself: #1386 guards the Python side of the same issue and lands at the same two emission sites. So this is a sequencing constraint, not a verdict. Concretely, what would clear it. A change of this shape was accepted recently on four things together, and yours currently shows two of them:
One interaction to check while you are at it: you drop Two smaller thingsYour known limitation is real and honestly stated. A bare specifier that does name in-project code but fails to materialise an Your |
A bare call whose name the calling file imports from a package outside the
indexed tree was bound to whatever project symbol shared the simple name.
An `import { eq, sql } from "drizzle-orm"` plus an unrelated local module
exporting `eq`/`sql` produced two CALLS edges from the caller into that
module, strategy unique_name — the same defect on the sequential and the
fused-parallel resolver. The TS-LSP is not involved: it declines the
external names, and the textual registry fallback fires anyway.
The package materializes no node, so resolve_import_node returns NULL, no
IMPORTS edge is written, and the per-file import map carries no key for the
name. Strategies 1-2 miss and resolve_name_lookup binds the call by simple
name. Confidence does not separate the cases: the fabricated edge lands at
0.75 whenever the file also imports anything from the target's module.
Add cbm_suppress_external_import_shadow(), a pure predicate beside the
existing perl/tsjs/cross-language guards and called at the same two emit
sites. It drops the edge only when the callee is a bare identifier, the
strategy is a project-wide guess (suffix_match / unique_name /
field_type_hint / fuzzy), the file imports that exact local name from a
non-relative specifier, and no import-map key binds it. Relative specifiers
are excluded on purpose: they name a path inside the tree, so a missing
IMPORTS edge there is an in-project resolution gap and the fallback can
still be right. A name imported both relatively and from a package keeps
its edge, independently of extraction order.
Addresses the external-import sub-case of DeusData#1355.
Signed-off-by: Yyunozor <yyunozor@icloud.com>
- specifier_is_relative() now also recognizes Windows drive-letter
(C:\ / C:/) and UNC (\\server\share) specifiers as in-tree, not
external packages. pr-smoke runs this pipeline on Windows, and the
previous POSIX-only check ('.' / '/') classified those specifiers as
external, which could suppress a real edge on that platform only.
Added external_import_shadow_windows_relative_specifier_kept
(tests/test_registry.c): red before this fix, green after.
- old_workers / old_single in the DeusData#1355 pipeline test are now
const char*, matching getenv()'s read-only contract.
- The pad_files literal (50) is now a named
EXTERNAL_IMPORT_SHADOW_PARALLEL_PAD constant with a comment tying it
to MIN_FILES_FOR_PARALLEL (a private #define in pipeline.c, not
reachable for a static_assert from tests), so a future bump to that
threshold can't silently drop the fixture back onto the
sequential-only path.
Signed-off-by: Yyunozor <yyunozor@icloud.com>
…ort guard A per-language blast-radius census on twelve public repositories found one regression class in the DeusData#1355 guard, and it is not per-language: it is per-topology. In a workspace monorepo a BARE specifier can still name code inside the indexed tree. `import { eq } from "drizzle-orm"` written inside the drizzle-orm repository is a sibling package, not a dependency. Measured on drizzle-team/drizzle-orm at b7862528, full index, both resolvers: the guard removed 2609 CALLS edges. Resolving each removed edge's specifier against the repository's own package manifests puts 1377 of them (52.8%) on the file the specifier actually names. 690 of those are strict: the specifier names a subpath, that subpath maps to one directory, and the target file is inside it. The other 687 come from the bare package root, where the check can only confirm the target is somewhere in the package, so read 690 as the defensible floor. Only 373 removed edges pointed at a third-party package, which is the defect DeusData#1355 reports. The same census on eleven other repositories (flask, scrapy, express, zustand, got, cobra, gin, chi, ripgrep, gson, jq) removed 12 CALLS edges in total: nine fabricated, one lost (a documentation example importing the repository's own published name), two weak links between a shell command and a manifest key. The regression is specific to trees that ship the package they import. Consult the pipeline package map before calling a specifier external. It is keyed by the `name` of every manifest found in the tree, so a workspace registers each of its own packages there, and a subpath specifier is walked back one slash at a time ("drizzle-orm/pg-core" -> "drizzle-orm"). A specifier the tree itself claims is now treated exactly like a relative one -- kept. The check is deliberately a NAME test rather than a resolution test. A workspace package usually points `main` at a build artifact ("./index.cjs") that is not checked in, so asking whether the entry file exists in the graph answers "no" for exactly the monorepos this has to protect. Whether the tree claims the name is decidable from the manifest alone. Passing NULL restores the previous specifier-shape-only contract, which is what the unit tests exercise. After the change the same census removes 391 edges on drizzle-orm, 375 of them CALLS, and every one resolves to a third-party package; of the other eleven repositories only got changes, recovering its self-referencing edge. Signed-off-by: Yyunozor <yyunozor@icloud.com>
f78bfe2 to
797a11a
Compare
|
Thank you for the review. I ran the census. It found a regression class in my Summary
Full census: per-repository tables, edges read by hand, the #1907 fixture, caveats3. Measured blast radius, per languageTwelve public repositories, one index with
Go, Rust, Java and C are byte-identical before and after. I also built a traced drizzle-orm is the problem. I classified all 2609 removed CALLS edges
The 1377 are not all equally strong evidence, so here is the split. In 690 the I read 14 of them by hand. The other eleven repositories removed 12 CALLS edges in total: express 3,
9 + 1 + 2 = 12. 4. Why there is no per-language gateThe census answers this better than an argument would. The blast radius is not What separates the safe repositories from drizzle-orm is not the language. It is So the correct seam is the package map, which is what you called the right The new commit consults Same census after the change:
got is the only one of the eleven that changes: the self-referencing All 375 resolve to a third-party package. I read 10: Tests: Each emission site has its own red proof. Replacing The #1907 interactionYour reasoning holds, and here is the measurement. Numbers below are from my
The absolute totals do not travel between indexing sessions. Four fresh What does hold in both sessions is the comparison, which is what the question So my drop-list naming The seam does exist. I built a fixture with Two notesThe branch was 106 commits behind, so I rebased it onto The known limitation is now narrower but still real. A bare specifier that names The "per topology, not per language" claim rests on one workspace. drizzle-orm Rebasing this changes the diff you already traced. If you would rather judge the Repository SHAs used: drizzle-orm Measurement caveats. Running the same |
|
One correction to my last note: I have since run the full suite on this branch ( Branch: 7821 passed, 1 failed, 7 skipped, 141/141 suites. Bare parent The one failure on the branch is |
|
Census done — the third item on the bar I gave you, measured per-language blast radius on real graphs (ten languages, ~3.1M CALLS edges, every removed edge classified). The JS/TS/Java/PHP effect is correct and valuable: koel's
Same sign-off, then merge on green. Thanks for the careful predicate work — the drizzle-orm repro is what made the census worth running. |
… packages JVM package imports were read as external. A Kotlin or Java specifier is a dotted package path whether the package belongs to the indexed tree or to a dependency -- `import org.jetbrains.exposed.v1.tests.shared.assertEquals` and `import kotlin.test.assertEquals` have the same shape, and neither is "relative". The package map cannot separate them either: the manifests it reads for the JVM (pom.xml, build.gradle) name build artifacts, not packages. So every package-path specifier fell through to "external" and the guard cut calls into the project's own code. Measured on JetBrains/Exposed at 0e4d81a5, full index: the guard removed 426 CALLS edges, all of them suffix_match, all of them into the repository's own exposed-tests/.../shared/Assert.kt. Every callee was assertEquals, assertTrue or assertFalse -- names that file declares and that the test files import twice, once from the project package and once from kotlin.test. The double import is what leaves the import map without a key and hands the call to the project-wide guess. The evidence that separates the two is the tree's own `package` declarations, and the pipeline already collects them: the import passes build a namespace map keyed by the dot-normalized `package` / `namespace` / `use` of every indexed file, to resolve namespace imports. Publish that map for the duration of the run instead of freeing it at the end of the import pass, and consult it before calling a package-path specifier external. The trailing member segment is walked off one at a time, the same way the namespace-map lookup in cbm_pipeline_resolve_import_node does it. This keeps the effect the census called valuable. `java.net.URL` walks to "java.net" and then "java", neither of which any project file declares, so a bare `URL` bound to an in-tree URL class is still cut. Only specifiers the tree itself claims are kept. Passing NULL restores the previous contract, which is what the unit tests exercise. After the change Exposed carries 31195 CALLS edges, the same as the merge-base, with none removed and none added -- stable across 28 indexations of the repository (8 with this branch, 20 single-threaded). Total edge counts are not a usable invariant here: roughly one indexation in ten materializes a `.properties` resource file as a File node under Folder nodes instead of a Module with its Variables, on the merge-base binary as well, and the file it picks varies. That flip never touches a CALLS edge and disappears entirely under CBM_INDEX_SINGLE_THREAD=1. The walk that strips the trailing member segment has no floor, on purpose: a third-party package published under a prefix the tree declares reads as in-tree and the guard stands down, leaving exactly the edge the merge-base emits. A call-resolution suppressor has to fail on that side. Signed-off-by: Yyunozor <yyunozor@icloud.com>
The DeusData#1355 guard dropped the call before the emitters ran, so it did not only remove the fabricated CALLS edge it is meant to remove: it also skipped the route, HTTP, CONFIG and URL classification that happens inside emit_classified_edge / emit_service_edge. A PR that claims to touch nothing but weak same-name CALLS edges was silently costing Route nodes, and with them every HANDLES edge that pass_route_nodes.c later bridges onto those routes. The shape is not hypothetical. A route registration whose router is a dependency is exactly a bare call bound by a package specifier: the resolved QN carries the router library name, emit_classified_edge reads it as CBM_SVC_ROUTE_REG, and the first argument is path-shaped. On a 66-file TypeScript fixture main mints __route__ANY__/orders and the guard removed it, on both resolvers. The fix is the seam the DeusData#592/DeusData#606 member guard already uses, for the same reason and with the same comment: join the predicate to drop_plain_call and let the emitter decide. Only the plain-CALLS fall-through is suppressed, so the fabricated edge still goes and every service edge stays main-identical. The two resolvers keep identical gates, as the notes in both files require. The new pipeline test asserts both halves on both resolvers: the Route node survives, and a second bare call from the same external package in the same file is still cut, so the first assertion cannot pass by the guard having been disarmed. Signed-off-by: Yyunozor <yyunozor@icloud.com>
|
Both items are fixed, in two commits. Numbers below are from my run. The Summary
Detail: the Kotlin mechanism, the route mechanism, an indexing flip I had to chase, the census, the red proof, and what I could not measureWhy Kotlin, and why only three names
Ten read by hand, all the same shape: An indexing flip, and why I stopped quoting total edge countsMy first draft of this comment said Exposed came out "byte-identical, 263895 Roughly one indexation in ten materializes a single It is not this diff. Evidence, 58 indexations of Exposed, repository tree
The majority state of this branch is byte-identical to the majority state of The route mechanism
It comes from Fixture, 66 TypeScript files.
Identical under flask is not evidence for this, and I want to say soThe raw census shows the published head removing 22 HANDLES and adding 10 on CensusFull index per repository per binary. "removed" means present in the merge-base
¹ upstream noise, measured above. A dash means zero changed edges. For gson and the Go repositories I am Red proof, one site at a time, on the committed stateEach site broken alone, rebuilt, suites run, restored from
The last two have no red test. In the test harness the project name comes from They are not dead code. The same fixture indexed through the CLI with a short Formatting checked with Per-language by evidenceI did not gate the declared-package check by language in the code. It is a What I could not measureI did not find the Java repository behind your 88 HANDLES and 5 Route nodes. I ran |
DeusData
left a comment
There was a problem hiding this comment.
Thank you — and I mean that with some weight. You took the census seriously enough to find a regression in your own patch, reported it first, and then fixed both of my findings with a mechanism rather than a special case. That is a rare way to work, and it is why this review went as deep as it did. Here is what I checked, what holds, and the one thing that does not yet.
What holds
Both 5 September items are really fixed on a full index, and the tests bind. I reverted the production hunks only, tests untouched:
81820262(guard runs after route classification —drop_plain_callinstead of the earlycontinue), both sites reverted →pipeline+registry: 355 passed, 1 failed
FAIL tests/test_pipeline.c:6471: sequential == 4, expected 0 == 0138a6225(declared-package set via the published namespace map), mechanism disabled atregistry.c:816→ 4 failures, among them
external_import_shadow_declared_package_kept FAIL tests/test_registry.c:1239
FAIL tests/test_pipeline.c:6364: parallel == 4, expected 0 == 0- whole predicate disabled → 3 failures at
test_pipeline.c:6127,:6237,:6471.
The predicate is cheap and safe in the ways this codebase has been burned before. No registry access at all; no lookup inside a registration loop; cost per resolved call is O(K + I·D) over this file's import-map keys and imports, and only after the cheap filters (weak strategy, bare callee, file has imports) — calls resolved by import_map / same_module / lsp_* leave after four strcmps. Zero allocations (one CBM_SZ_512 stack buffer, over-long specifiers return false). g_nsmap is written once before the resolve workers start and only read afterwards, the same single-writer contract as g_pkgmap, and it owns its keys, so extending its lifetime to the end of the run is safe under spill. The complexity suite is green on the merge result.
It merges onto today's main cleanly and the result is green: no textual conflict, builds with -Wall -Wextra -Werror under ASan+UBSan, complexity 5 · extraction 350 · registry 71 · pipeline 285 · cross_repo 8 · parallel 74 · lang_contract 41 · incremental 163 = 997 passed, 0 failed; the memory-core linter shows no growth from this PR. main's newer is_test_flags parameter lives in candidate ranking (best_by_import_distance), which you do not touch and your verdict does not depend on. (Your head is 21 commits behind main, not 300 — your own merge of 20 September brought it nearly up to date.)
The one blocking finding: the JVM fix does not survive an incremental re-index
g_nsmap is built by cbm_pipeline_namespace_map_build[_names] from the file list of the current pass. On a full index that is every file. On the incremental / closure_repair route it is changed_files only (pipeline_incremental.c:1254 and :1385 hand that list to cbm_build_registry_from_cache / cbm_pipeline_pass_definitions). So when someone edits only app/Query.kt, the file that declares package com.example.util is not in the map, specifier_names_declared_package answers "not declared", and the guard suppresses the real edge again — the Kotlin regression from 5 September, coming back through the watcher.
Reproduced on your own Kotlin fixture: full index, then rewrite only the caller file without changing a single name, re-run. With an attribution control (the predicate returning false at entry) to prove it is the guard and nothing else:
guard armed (this PR): round=0 route=0 (full index) true_edge=1 fabricated_edge=0
round=1 route=closure_repair true_edge=0 fabricated_edge=0 <- real edge LOST
guard disarmed: round=0 route=0 (full index) true_edge=1 fabricated_edge=1
round=1 route=closure_repair true_edge=1 fabricated_edge=1
(The fabricated_edge column is the good news: on a full index the guard does exactly what it promises.) The TypeScript-workspace side is immune — same probe, true_edge=1 in both rounds and both arms — because the pkgmap comes from cbm_pkgmap_build_from_repo, a tree-wide manifest walk that does not depend on which files changed.
Beyond the lost edges this makes the graph depend on how it was built — full and incremental disagree about the same source tree — which is a property we work hard to keep out of this project. On an Exposed-shaped repository every edited test file would lose its assertEquals / assertTrue edges until the next full index.
Fix shape, your choice: source the declared-package set tree-wide on the incremental routes too (seed it from the existing graph/store, or persist it next to the pkgmap), or have the guard stand down when the map is known to be partial. The second is the smaller change and errs in the safe direction (it keeps an edge main already has). Either way it needs a RED test on the closure_repair route — the probe above is that test almost verbatim (cbm_pipeline_incremental_test_last_route() gives you the route to assert on), and I am happy to paste it in full if useful.
Smaller things
- An inert assertion, which you already suspected.
tests/test_pipeline.c:6363—ASSERT_EQ(sequential, 0)in…keeps_declared_package_issue1355— still passes with the mechanism disabled; only:6364(parallel) goes red. Your explanation is right: the sequentialbuild_import_mapbinds both names by import map under the temp-dir project name, so the guard is never reached. That leaves the sequentialg_nsmappublication inpass_definitions.cand thensmapargument inpass_calls.cwithout a red test. A fixture that actually reaches the sequential guard closes it. - Say in the PR text what
81820262decided. Of the two options I gave for the route-classification item you took "run after classification", which deliberately keeps the 88 JavaHANDLESedges / 5Routenodes I had called spurious — the branch ismain-identical there by design. That is a defensible choice; it should be stated, so nobody later reads it as an oversight. - The per-language question (my 1 September item 4) is still open, and it is mine to settle, not yours. The predicate takes no
lang, and the namespace map is not language-scoped, so in a polyglot repository a Kotlin or PHP package name can make a JS bare specifier read as in-tree. It fails permissive only — it can keep an edgemainalready has, never remove one — which is the right direction. This project has a standing rule that suppressors are scoped per language by evidence; your "per topology, not per language" argument is a serious one and I will take it to that decision rather than ask you to guess the answer. Nothing to do on your side until I come back on it.
Landing order, so you do not resolve a conflict twice
A change of ours, #2227, edits the same drop_plain_call expression in pass_calls.c and pass_parallel.c and will conflict textually with this PR there (registry.c, pipeline.h and both test files auto-merge). The resolution is mechanical and the semantics compose — #2227's exemption covers member calls, yours fires only on bare callees:
(weak_member && !cbm_weak_member_unique_name_exempt(...)) || weak_local_binding ||
cbm_suppress_external_import_shadow(...)Since this needs one more push anyway, the cheapest order for you is: we land #2227, you merge main in, and resolve that one expression in the same push as the incremental fix. I will tell you here the moment #2227 is in. (main also currently carries two reds that are not yours — a linter ratchet and a worker-policy script, both fixed by #2257 — so expect your CI to show them until that lands.)
Not checked on my side, so you know the limits of this review: no corpus re-run (the census numbers are yours and mine from earlier in the thread; the incremental finding is proven on the fixture, not measured on a real repository), macOS arm64 only, and the new Windows drive-letter/UNC specifier handling was read, not executed.
This is close. Thank you for the persistence — a month on one PR is a lot to ask of anyone.
Addresses the external-import sub-case of #1355: the caller's own source already says the callee is not a project symbol.
Complements #1386, which guards the Python side of the same issue with a generic-name list. This one keys on import evidence rather than on names and is not language-gated; the repro and tests here are TypeScript. Both land at the same two emission sites — happy to rebase onto #1386 whenever it lands.
Reproduction
src/queries.tsdoesimport { eq, sql } from "drizzle-orm";src/text-utils.tsexports unrelated localeq/sqlhelpers. Onmain(34d18ae):Two fabricated
CALLSedges, identical underCBM_INDEX_SINGLE_THREAD=1andCBM_WORKERS=4. The TS-LSP is not at fault:normalize, imported relatively from the very module the guess picked, resolves vialsp_ts_import@ 0.95. It declines the external names; the textual fallback fires regardless.Cause
drizzle-ormis outside the indexed tree, socbm_pipeline_resolve_import_nodereturns NULL, noIMPORTSedge is written, and the import map has no key foreq. Strategies 1-2 miss andresolve_name_lookupbinds by simple name. Confidence does not separate the cases: the fabricated edge reaches 0.75 when the file also imports from the target's module.Fix
cbm_suppress_external_import_shadow()— a pure predicate beside the existingcbm_perl_*/cbm_tsjs_*/cbm_suppress_cross_language_*guards. It drops the edge only when the callee is a bare identifier, the strategy is a project-wide guess (suffix_match/unique_name/field_type_hint/fuzzy), the file imports that name from a non-relative specifier, and no import-map key binds it.Relative specifiers are excluded on purpose: they name a path inside the tree, so a missing
IMPORTSedge there is an in-project gap and the fallback may still be right.Verification
scripts/test.sh: branch 7569 / 1 / 8, parent 7565 / 1 / 8. Delta is exactly the 4 new tests; the pre-existing failure (test_cli.c:8914) is identical on both sides.buildQuery → eqedge present, on both resolvers.workspaces,pnpm-workspace.yaml, tsconfigpaths, barrel and./x.jsimports keep their edges; Javaimport staticand Pythonfrom pkg import fare untouched.Known limitation
A bare specifier that does name in-project code but fails to materialize an
IMPORTSedge (the workspace case in #1732) now loses its same-name fallback edge instead of keeping it at reduced confidence. A natural follow-up would consult the package map first.