Skip to content

feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5) - #478

Open
MarioCadenas wants to merge 1 commit into
mainfrom
pr/agent-evals-2-framework
Open

feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5)#478
MarioCadenas wants to merge 1 commit into
mainfrom
pr/agent-evals-2-framework

Conversation

@MarioCadenas

Copy link
Copy Markdown
Collaborator

Stack 2/5 · targets pr/agent-evals-1-tracing (review after #1).

The core eval framework, plus LLM-as-judge and the MLflow REST connector.

  • Authoring (defineEval): drive an agent over HTTP against a running app; assert with t.succeeded(), t.calledTool(), t.check(value, matcher) (includes/equals/matches). Gate-by-default, .soft() to demote.
  • Native MLflow Evaluation runs: a run tagged genai_evaluate; each turn's trace links via mlflow.sourceRun; per-assertion feedback written via the assessments REST API.
  • LLM-as-judge (t.judge.factuality/closedQA/custom) via autoevals → a Databricks serving endpoint.
  • connectors/mlflow: MlflowClient (host/token, post/postResult, serving URL) + resolveDatabricksAuth/resolveWorkspaceClient (OAuth from a CLI profile — no hand-set PAT). Extracted so both evals and future callers share the REST/auth layer.
  • appkit agent eval CLI.

Squashed history note: contains the framework, judge, and connector-extraction commits.

@MarioCadenas
MarioCadenas requested a review from a team as a code owner July 16, 2026 14:26
@MarioCadenas
MarioCadenas requested review from pkosiec and removed request for a team July 16, 2026 14:26
Base automatically changed from pr/agent-evals-1-tracing to main August 13, 2026 09:27
@MarioCadenas
MarioCadenas force-pushed the pr/agent-evals-2-framework branch from 0b55719 to 1431ce7 Compare August 18, 2026 12:57
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.1 MB (+43 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.1 MB (+35 KB) 397 KB (+15 KB)
Type declarations 406 KB (+20 KB) 145 KB (+9.3 KB)
Source maps 2.2 MB (+72 KB) 745 KB (+30 KB)
Other 11 KB 3.7 KB
Total 3.7 MB (+128 KB) 1.3 MB (+54 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 95 KB (+1 B) 2.5 KB 98 KB (+1 B) external 313 KB
./beta 81 KB (+4.5 KB) 455 B (-1 B) 81 KB (+4.5 KB) external 242 KB (+12 KB)
./testing 17 KB 0 B 17 KB external 51 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 23 KB 0 B 23 KB external 65 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 91 KB
. utils.js initial 4.0 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 65 KB
./beta stream-manager.js initial 5.8 KB
./beta wide-event-emitter.js initial 3.2 KB
./beta databricks.js initial 3.2 KB
./beta configuration.js initial 2.1 KB
./beta service-context.js initial 1.3 KB
./beta client.js initial 434 B
./beta client-options.js initial 219 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 141 B
./beta index.js lazy 122 B
./testing index.js initial 17 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 23 KB

@databricks/appkit-ui

npm tarball (packed): 350 KB (-4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 395 KB 132 KB
Type declarations 229 KB 84 KB (-1 B)
Source maps 766 KB 253 KB
CSS 16 KB 3.2 KB
Total 1.4 MB 473 KB (-1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 20 B 0 B 20 B 0 B 0 B
./react 432 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 1.0 KB 0 B 1.0 KB 0 B 1.9 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 20 B
./react index.js initial 430 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 1.0 KB

@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 33763255268 -R databricks/appkit -n appkit-template-0.71.0-pr.97064b1-pr-agent-evals-2-framework-478 -D appkit-pr-478 \
  && unzip -o "appkit-pr-478/appkit-template-0.71.0-pr.97064b1-pr-agent-evals-2-framework-478.zip" -d "appkit-pr-478" \
  && databricks apps init --template "appkit-pr-478"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

@MarioCadenas
MarioCadenas force-pushed the pr/agent-evals-2-framework branch 2 times, most recently from e32afdd to 65de478 Compare August 21, 2026 10:37
@MarioCadenas
MarioCadenas force-pushed the pr/agent-evals-2-framework branch 2 times, most recently from 1f15f16 to 274ec9d Compare September 1, 2026 13:20
@MarioCadenas
MarioCadenas changed the base branch from main to fix/agents-mlflow-single-provider September 2, 2026 14:39
@MarioCadenas
MarioCadenas force-pushed the pr/agent-evals-2-framework branch from da02130 to 5fe6166 Compare September 3, 2026 09:39
Base automatically changed from fix/agents-mlflow-single-provider to main September 3, 2026 11:59
@MarioCadenas
MarioCadenas force-pushed the pr/agent-evals-2-framework branch from 5fe6166 to 74929cc Compare September 3, 2026 11:59
Rebuilt on the current fix/agents-mlflow-single-provider base (#545 was
force-pushed onto v0.70.0, so the branch's merged copies were stale). The
eval framework is self-contained — it imports nothing from the agents
plugin — so every #545 file is taken from the base untouched; only the
framework's own files plus additive seam edits (beta/connectors/cli
exports, autoevals dep) are applied.

Framework: discover server/agents/<agent>/evals/*.eval.ts, drive a running
app over HTTP+SSE, run deterministic assertions + LLM-as-judge (autoevals),
and report results + per-trace assessments to MLflow (classic V3 + UC V4).
Includes the mlflow REST connector and the `appkit agent eval` CLI.

Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@MarioCadenas
MarioCadenas force-pushed the pr/agent-evals-2-framework branch from 74929cc to f8102dd Compare September 3, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant