Skip to content

docs(evals): mark batch evaluation as legacy #1867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hassiebp merged 7 commits into from
Sep 7, 2026
Merged

docs(evals): mark batch evaluation as legacy #1867

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e6c3916
fix(evals): use v2 observations for batch evaluation
cursoragent Sep 7, 2026
65f638a
fix(evals): retain v3 batch evaluation fallback
cursoragent Sep 7, 2026
7607236
fix(evals): preserve v3 resume filters
cursoragent Sep 7, 2026
334ac15
test(evals): correct fallback filter fixture
cursoragent Sep 7, 2026
8d1993d
fix(evals): make v2 observation reads opt in
cursoragent Sep 7, 2026
22aa20e
fix(evals): restrict v2 reads to observations
cursoragent Sep 7, 2026
ff416d2
docs(evals): mark batch evaluation as legacy
cursoragent Sep 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion langfuse/_client/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3272,7 +3272,7 @@ def run_batched_evaluation(
resume_from: Optional[BatchEvaluationResumeToken] = None,
verbose: bool = False,
) -> BatchEvaluationResult:
"""Fetch traces or observations and run evaluations on each item.
"""Fetch traces or observations using legacy read APIs and evaluate each item.

This method provides a powerful way to evaluate existing data in Langfuse at scale.
It fetches items based on filters, transforms them using a mapper function, runs
Expand All @@ -3288,6 +3288,11 @@ def run_batched_evaluation(
it memory-efficient for large datasets. It includes comprehensive error handling,
retry logic, and resume capability for long-running evaluations.

Legacy platform compatibility:
This method reads traces from `GET /api/public/traces` and observations
from the legacy `GET /api/public/observations` endpoint. It is supported
with Langfuse platform v3 and is not yet supported with platform v4.

Args:
scope: The type of items to evaluate. Must be one of:
- "traces": Evaluate complete traces with all their observations
Expand Down
6 changes: 5 additions & 1 deletion langfuse/batch_evaluation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -855,12 +855,16 @@ async def run_async(
verbose: bool = False,
resume_from: Optional[BatchEvaluationResumeToken] = None,
) -> BatchEvaluationResult:
"""Run batch evaluation asynchronously.
"""Run batch evaluation asynchronously using legacy read APIs.

This is the main implementation method that orchestrates the entire batch
evaluation process: fetching items, mapping, evaluating, creating scores,
and tracking statistics.

This runner reads traces from `GET /api/public/traces` and observations
from the legacy `GET /api/public/observations` endpoint. It is supported
with Langfuse platform v3 and is not yet supported with platform v4.

Args:
scope: The type of items to evaluate ("traces", "observations").
mapper: Function to transform API response items to evaluator inputs.
Expand Down