Rebuild Evaluations page around runs with detailed history and scoring - #471
Merged
Merged
Conversation
Ports the MetaStrip from #470 so the evaluation runs list can keep its cost, movement and status columns aligned on every row — a failed run with nothing to put in a column prints a dash there rather than sliding its neighbours over. The trace and interaction lists take the same strip, as that change has it; the source is identical to the branch so it no-ops once #470 merges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Every client wants to know what operating an agent will cost, and an evaluation run is the closest thing to a controlled measurement: a suite's replays are simulated user–agent interactions, a sampling run's cohorts are real ones. But the run also asks a judge model to score, recommend and rule, and a run that reported that spend in the same number overstated the first. The runners now meter every judge call under what it was for, through the `kind:` keyword ActiveAgent::Evals::Judge hands a block that accepts it, and persist the total as scores["_judge_usage"] — calls, tokens, estimated cost, model and a by-kind tally. A generation-sampling run records scores["_cohorts"] beside it: per model, how many generations were sampled, how many cleared every criterion, their latency and tokens, and what those interactions cost to serve. EvaluationRun#usage reports both sides, the agent's with a per-interaction rate, and nil for a run that recorded neither. The API numbers runs oldest-first (`number`), reports `run_count` and a `previous_run` summary per evaluation, so the dashboard can say "Run #3" and "+3 passed vs #2" without a request per evaluation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Implements the "Evaluations Redesign v2" design (its REVIEW EVALS RUNS section) in the engine, where the platform's earlier implementation was lost when it swapped its own dashboard for this one. Evaluations are the top level; every run is kept and listed with its movement against the run before it; a sampling evaluation's run opens to a page of its own — a scorecard per model cohort, the judge's verdict, the criteria × models matrix and what the run asks to fix — at /evaluations/:id/runs/:run_id. A scenario suite's runs are the same list, full width, and a row selects the run its model scorecards, fix items and scenario matrix show; the deep link selects it too. What a run cost is shown as two figures everywhere it appears — on the run row, on the run page, on a page tile and in the footer: the agent's spend with its per-interaction rate, the operating figure a budget is set against, apart from the judge's own offline calls. The shared vocabulary (criterion labels and expectations, cohorts, scorecards, movement, spend, the sampling run's fix items) lives in utils/evaluationRuns.mjs, pinned by node tests. The report route, the ?evaluation= links and the e2e hooks (evaluation-card, data-telemetry, score-source-telemetry) are kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR restructures the Evaluations page to make runs the primary organizational unit, with full run history, detailed scoring breakdowns, and improved cost tracking. The changes separate sampling evaluations from scenario suites in the UI while sharing common run list and scoring components.
Key Changes
Frontend Architecture
evaluationRuns.mjs): Centralized definitions for criteria types, run scoring, cost calculations, and run metadata. Pure functions with no dependencies enable reusable logic across components and testability.EvaluationsView.jsxinto focused modules:EvaluationForm.jsx: Creation form with criteria selectionEvaluationRunDetail.jsx: Sampling run detail page with scorecard per model, verdict, criteria matrix, and fix recommendationsRunsList.jsx: Shared run history list showing pass rates, movement vs previous run, and costsModelScorecard.jsx: Per-model cohort summary (passed/total, scores, tokens, cost)SpendStrip.jsx: Cost breakdown (agent operating cost vs judge evaluation cost)CriteriaFooter.jsx: Evaluation configuration display/evaluations/:id/runs/:runId) and suite runs, with browser history supportBackend Enhancements
_cohorts) for sampling runs, enabling "5/12 passed" display per model without re-reading generations_judge_usagefield records judge model's calls, tokens, and cost, distinguishing evaluation overhead from agent operating costGET /api/evaluations/:idreturns latest 20 runs with total countData Model
EvaluationRungainscohortsandjudge_usageaccessors for structured score metadatallm_judgetype with custom prompts alongside rule-based and telemetry criteria_-prefixed metadata fieldsUI/UX Improvements
+3 passed vs #2,partial run,#1 failedNotable Implementation Details
evaluationRuns.mjsimports nothing, enabling node tests without browser dependencies_modelsfieldhttps://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd