[DE-8690] Local EvaluationV2 over parquet: nucleus/evalv2 + nu evalv2 run (WIP) - #482
Draft
luke-e-schaefer wants to merge 1 commit into
Draft
[DE-8690] Local EvaluationV2 over parquet: nucleus/evalv2 + nu evalv2 run (WIP)#482luke-e-schaefer wants to merge 1 commit into
nu evalv2 run (WIP)#482luke-e-schaefer wants to merge 1 commit into
Conversation
…2 run` (WIP) Adds the SDK-side local harness for the EvaluationV2 portable kernel: run an evaluation offline over exported parquet using the shared evalv2-core kernel, so results match a platform run by construction (parity by design). - nucleus/evalv2/parquet_io.py: ParquetSource (streaming sorted merge-join over predictions + ground_truth [+ items] parquets -> ItemBundles; O(one item+batch) memory) and ParquetSink (result bundle: matches/per_threshold parquet + summary/charts JSON). - nucleus/evalv2/run.py + cli.py: run_local_eval + `nu evalv2 run` (--predictions/--ground-truth/--items/--out/--config/--iou-type/ --class-agnostic/--min-prediction-score); config JSON reuses the kernel's exclusion-rule parser + rollup groups. - cli/nu.py: register the `evalv2` group (guarded import so `nu` still works if the optional evalv2-core dep isn't installed). Draft/WIP. `nu evalv2 upload` (push result bundle to the platform ingest endpoint) is a follow-up. Tests run isolated from the heavy SDK import via --confcutdir + importlib. Before ready: wire evalv2-core as a real dependency (private index), version bump + CHANGELOG. Ref: https://linear.app/scale-epd/issue/DE-8690 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Run EvaluationV2 offline over exported parquet using the shared
evalv2-corekernel (lives in scaleapipackages/evalv2-core). Because the SDK and the platform worker import the same kernel, local results match a platform run by construction (parity by design); pycocotools is the source of truth.In this PR
nucleus/evalv2/parquet_io.py—ParquetSource(a streaming sorted merge-join overpredictions+ground_truth[+items] parquets →ItemBundles; O(one item + batch) memory, so it handles tens of millions of rows without OOM) andParquetSink(writes the result bundle:matches/per_thresholdparquet +summary/chartsJSON).nucleus/evalv2/run.py+cli.py—run_local_eval(...)andnu evalv2 run(--predictions/--ground-truth/--items/--out/--config/--iou-type/--class-agnostic/--min-prediction-score); the--configJSON reuses the kernel's exclusion-rule parser + rollup groups.cli/nu.py— registers theevalv2command group (guarded import sonustill works if the optionalevalv2-coredep isn't installed).Still planned
nu evalv2 upload— push the result bundle to the platform ingest endpoint (async job), to view the eval in-platform.Verification / caveats
CliRunner) run without the heavy SDK import via--confcutdir+importlib(the repo-rootconftest.pyimportsnucleus+ requires an API key).evalv2-coreas a real dependency (private index), version bump +CHANGELOG.mdper repo convention.🤖 Generated with Claude Code