feat(appkit): eval suite features — concurrency, retries, reporters (stack 4/5) - #480
Open
MarioCadenas wants to merge 8 commits into
Open
feat(appkit): eval suite features — concurrency, retries, reporters (stack 4/5)#480MarioCadenas wants to merge 8 commits into
MarioCadenas wants to merge 8 commits into
Conversation
Contributor
|
This pull request has had no activity for 23 days and has been marked as stale. It will be closed in 7 days if there is no further activity. Add a comment, push a commit, or apply the |
Contributor
|
This pull request has been automatically closed because it had no activity for a month. Feel free to reopen it if you would like to continue the work. |
MarioCadenas
force-pushed
the
pr/agent-evals-3-datasets
branch
from
September 1, 2026 16:32
ce9424b to
a360c4b
Compare
MarioCadenas
force-pushed
the
pr/agent-evals-4-suite
branch
from
September 1, 2026 16:32
74a473e to
2dc29fe
Compare
MarioCadenas
force-pushed
the
pr/agent-evals-3-datasets
branch
from
September 1, 2026 17:27
a360c4b to
037a2e8
Compare
MarioCadenas
force-pushed
the
pr/agent-evals-4-suite
branch
from
September 1, 2026 17:27
2dc29fe to
ed23b41
Compare
…al CLI Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
The eval driver captured tool-call names but discarded their arguments. Parse the function-call `arguments` JSON into `DriveResult.toolCallDetails` (the later `done` event's fuller args win over the initial `added`), and expose `t.calledToolWith(name, expected)` — passes when a call to `name` had args that deep-contain `expected` (nested-aware partial match; extra args ignored). Gate by default, like `calledTool`. `toolCalls: string[]` is unchanged. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
A managed-dataset row's `inputs.messages` can be a full conversation, not just one question. Add `userTurns(input)` to extract every user-message content in order; the example dataset eval replays them against one thread so the agent sees the accumulating conversation. Interleaved assistant/system turns are ignored — the agent generates its own responses. Single-user-turn rows are unchanged (one send). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add `--min-pass-rate <0..1>` to the agent eval CLI: instead of requiring every eval to pass, exit non-zero only when the aggregate pass rate falls below the threshold. `summarize()` now returns `passRate` (passed / scored, excluding skips; 1 when nothing scored). Without the flag, behavior is unchanged (any gate failure fails the run). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Wire three eval fields that were declared but inert:
- `timeoutMs` — runEval races the test against a per-eval timeout and records a
clean non-passing result ("eval timed out after Nms") instead of hanging.
Precedence: def.timeoutMs > runner/CLI --timeout > unbounded. Timer is always
cleared.
- `evals.config.ts` (defineEvalConfig) — discovered per-agent and loaded via the
tsx loader; its maxConcurrency/timeoutMs apply as defaults (CLI flag > config >
built-in). Judge model still comes from the CLI (needs creds the config lacks).
- `tags` + `--tag <tag...>` — run only evals whose tags intersect the filter.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add `--retries <n>` / `RunEvalsOptions.retries`: re-run an eval up to N extra times when it fails on an infrastructure error (a thrown error or timeout — `result.error` set), to absorb transient turn/stream flakiness. Assertion failures are never retried — a wrong reply is real signal, and retrying a flaky judge until it passes would corrupt the result. Each attempt gets a fresh driver. Extracted as `runWithRetries` (unit-tested for attempt counting, stop-on-success, and no-retry-on-assertion-failure). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add `--reporter <text|json|junit>` (default text) and `--output <file>` to the
agent eval CLI. `formatResultsJson` emits `{summary, results}`; `formatResultsJUnit`
emits a `<testsuite>` with a `<testcase>` per eval (`<failure>`/`<skipped>` as
appropriate, all values XML-escaped). In json/junit mode the per-eval streaming
is suppressed and human banners go to stderr so stdout stays clean for piping or
`--output`; exit-code and pass-rate gating are unchanged.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
The discoverEvalConfigs test still built fixtures under config/agents/ from the pre-rename stack; the runner now discovers per-agent configs under server/agents/ (the folder-per-agent layout). Align the fixtures so the test matches the runner. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
MarioCadenas
force-pushed
the
pr/agent-evals-3-datasets
branch
from
September 2, 2026 09:01
037a2e8 to
2b76763
Compare
MarioCadenas
force-pushed
the
pr/agent-evals-4-suite
branch
from
September 2, 2026 09:03
ed23b41 to
1e835b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4/5 · targets
pr/agent-evals-3-datasets(review after #3).CI-grade suite features.
t.calledToolWith(name, expected)(deep-partial arg match).userTurns(input)replays a row's fullmessages[]against one thread.--concurrency N/maxConcurrency— bounded, order-preserving pool.--min-pass-rate 0..1— gate on aggregate rate instead of all-must-pass.timeoutMs, load per-direvals.config.ts,--tagfiltering.--retries N— re-run only on infra failures (turn/timeout), never on assertion failures.--reporter json|junit+--outputfor CI.Includes two small dedup refactors (credential resolution; agent-dir listing / int parsing).