feat(appkit): managed eval datasets + turn semantics (stack 3/5) - #479
Open
MarioCadenas wants to merge 4 commits into
Open
feat(appkit): managed eval datasets + turn semantics (stack 3/5)#479MarioCadenas wants to merge 4 commits into
MarioCadenas wants to merge 4 commits into
Conversation
Contributor
|
This pull request has had no activity for 23 days and has been marked as stale. It will be closed in 7 days if there is no further activity. Add a comment, push a commit, or apply the |
Contributor
|
This pull request has been automatically closed because it had no activity for a month. Feel free to reopen it if you would like to continue the work. |
MarioCadenas
force-pushed
the
pr/agent-evals-3-datasets
branch
3 times, most recently
from
September 2, 2026 09:01
037a2e8 to
2b76763
Compare
MarioCadenas
force-pushed
the
pr/agent-evals-3-datasets
branch
from
September 3, 2026 09:55
2b76763 to
8a009ec
Compare
Contributor
📦 Bundle size reportCompared against
|
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 1.1 MB (+41 KB) | 399 KB (+17 KB) |
| Type declarations | 412 KB (+26 KB) | 148 KB (+12 KB) |
| Source maps | 2.2 MB (+83 KB) | 748 KB (+34 KB) |
| Other | 11 KB | 3.7 KB |
| Total | 3.7 MB (+149 KB) | 1.3 MB (+62 KB) |
Per-entry composition (own code — deps external (as shipped))
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
. |
95 KB (-1 B) | 2.5 KB | 98 KB (-1 B) | external | 313 KB |
./beta |
88 KB (+11 KB) | 455 B (-1 B) | 88 KB (+11 KB) | external | 264 KB (+34 KB) |
./testing |
17 KB | 0 B | 17 KB | external | 51 KB |
./tsdown |
520 B | 0 B | 520 B | external | 813 B |
./type-generator |
23 KB | 0 B | 23 KB | external | 65 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
. |
index.js |
initial | 91 KB |
. |
utils.js |
initial | 4.0 KB |
. |
remote-tunnel-manager.js |
lazy | 2.5 KB |
./beta |
beta.js |
initial | 71 KB |
./beta |
stream-manager.js |
initial | 5.8 KB |
./beta |
wide-event-emitter.js |
initial | 3.2 KB |
./beta |
databricks.js |
initial | 3.2 KB |
./beta |
configuration.js |
initial | 2.2 KB |
./beta |
service-context.js |
initial | 1.3 KB |
./beta |
client.js |
initial | 434 B |
./beta |
client-options.js |
initial | 219 B |
./beta |
supervisor-api.js |
lazy | 192 B |
./beta |
databricks.js |
lazy | 141 B |
./beta |
index.js |
lazy | 122 B |
./testing |
index.js |
initial | 17 KB |
./tsdown |
index.js |
initial | 520 B |
./type-generator |
index.js |
initial | 23 KB |
@databricks/appkit-ui
npm tarball (packed): 350 KB (-4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 395 KB | 132 KB |
| Type declarations | 229 KB | 84 KB (-1 B) |
| Source maps | 766 KB | 253 KB |
| CSS | 16 KB | 3.2 KB |
| Total | 1.4 MB | 473 KB (-1 B) |
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
./js |
5.3 KB | 49 KB | 55 KB | 208 KB | 14 KB |
./js/beta |
20 B | 0 B | 20 B | 0 B | 0 B |
./react |
432 KB | 49 KB | 481 KB | 1.3 MB | 177 KB |
./react/beta |
1.0 KB | 0 B | 1.0 KB | 0 B | 1.9 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
./js |
index.js |
initial | 5.2 KB |
./js |
chunk |
initial | 120 B |
./js |
apache-arrow |
lazy | 49 KB |
./js/beta |
beta.js |
initial | 20 B |
./react |
index.js |
initial | 430 KB |
./react |
tslib |
initial | 2.1 KB |
./react |
apache-arrow |
lazy | 49 KB |
./react/beta |
beta.js |
initial | 1.0 KB |
Contributor
🤖 AppKit PR bot🔬 Run evalsStart an eval for this PR from the evals-monitor app: Go to Evals Monitor → 📦 Try this PR's app templateScaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh run download 33772632142 -R databricks/appkit -n appkit-template-0.71.0-pr.202e62c-pr-agent-evals-3-datasets-479 -D appkit-pr-479 \
&& unzip -o "appkit-pr-479/appkit-template-0.71.0-pr.202e62c-pr-agent-evals-3-datasets-479.zip" -d "appkit-pr-479" \
&& databricks apps init --template "appkit-pr-479"The template pins |
MarioCadenas
force-pushed
the
pr/agent-evals-3-datasets
branch
from
September 3, 2026 11:59
8a009ec to
8165cd5
Compare
Let an agent eval sweep a Databricks managed evaluation dataset instead of a
single hardcoded turn, and make one-shot vs multi-turn conversations explicit:
- defineEval gains `dataset: { table, limit? }`. The runner reads the UC table
(via SQLWarehouseConnector over the SQL Statement Execution API — the Python
datasets API needs Spark), then runs the test once per row with the row bound
to `t.input`/`t.expected`.
- readEvalDataset reuses the SQL connector's result transform, which already
JSON-parses `inputs`/`expectations` columns whether stored as JSON or structs.
- resolveWorkspaceClient builds the WorkspaceClient the connector needs, from
the same profile/host/token as resolveDatabricksAuth. CLI: `--warehouse`.
- Each dataset row gets a fresh driver so rows are independent conversations
(no thread/context bleed between rows).
- Turn semantics: consecutive `t.send`s share one thread (multi-turn); new
`t.reset()` starts a fresh conversation for independent one-shot checks in a
single test.
- Example dataset.eval.ts + unit tests for the reader and reset.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…al_dataset Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
LLM-judge assertions (`t.judge.*`) now fail the eval on a miss, like the deterministic assertions — previously they were soft-by-default and a failing guideline left the run green. `.atLeast(n)` sets the pass threshold without demoting; chain `.soft()` to keep a judge as a tracked-only metric. Surfaces real signal: e.g. a dataset row whose guidelines the agent can't satisfy now fails the run (non-zero exit) instead of passing silently. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
MarioCadenas
force-pushed
the
pr/agent-evals-3-datasets
branch
from
September 3, 2026 13:49
8165cd5 to
cc876d7
Compare
The eval CLI carried two flags for one physical SQL warehouse: --warehouse-id (MLflow assessment writes to UC-backed traces) and --warehouse (managed-dataset reads). Collapse to --warehouse-id, feeding both paths; the MLflow write path keeps its MLFLOW_TRACING_SQL_WAREHOUSE_ID env fallback. Also apply oxfmt to eval files left unformatted by an earlier rebase. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 3/5 · targets
pr/agent-evals-2-framework(review after #2).Managed datasets and explicit conversation semantics.
dataset: { table }sweeps a Databricks managed evaluation dataset (a Unity Catalogcatalog.schema.tableread via the SQL Statement Execution API — the Python datasets API needs Spark). One run per row;t.input/t.expectedbind each row.--warehouse+ auth required.send), multi-turn (consecutivesends share a thread), andt.reset()for an independent turn in one test. Each dataset row gets a fresh driver so rows don't bleed..atLeast(n)sets the threshold,.soft()opts out.