fix: validation

[jakelorocco]

Pull Request

Issue

Fixes #426

Description

I believe I covered all the parts of the initial issue.

1. Dead validation context — avalidate's throwaway SimpleContext() deleted. Validation runs over the caller's context, in their own context type. output now designates the target and is appended only when context.last_output() is not output

2. Provenance loss — Requirement._output: str → _validation_target: Span, bound on a copy via _bind_validation_target, exposed through parts() so generate_walk can await it, rendered preferring a Component parsed_repr over the raw string.

3. Dead validation_ctx (bug: validation_ctx parameter in SamplingStrategy.sample() exists but is not being used in the validation call #668) — honored in sampling/base.py, budget_forcing.py, sofai.py; majority_voting.py verified. The old validation_ctx or context default was itself wrong (pre-generation context)

4. Judge prompt reworded — Requirement.jinja2, scoping the verdict to the output while telling the model the conversation is there to interpret the requirement. Got better results and fewer tokens across several models (granite 4.2 as well)

5. genstub preconditions judged over a fresh ChatContext() instead of leaking session history

6. Fixing alora requirement validation especially when auto-routing: when a reroute would happen but the context renders nothing, an explicit ALoraRequirement raises ValueError naming the context, the adapter, and both ways out; an auto-rerouted plain Requirement disables the reroute and falls back to LLMaJ with a warn-once

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code was added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

• Component
• Requirement
• Sampling Strategy
• Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

[planetf1]

One non-blocking question inline.

[planetf1]

Small question on the input semantics: the docstring says it is prepended to the validation context and appears before output, but this starts from the existing context and appends input.

When output is omitted and the context already ends in a ModelOutputThunk, the rendered order becomes … → output → input. Is that intentional? If so, could the docstring describe that ordering; otherwise this needs a regression test and adjustment.

[jakelorocco]

I'll push a change modifying the docstring. The current behavior (and intended) is that given a validation context, an input, and an output; we append the input to the context and then the output.

Yes, the scenario you are describing is a real effect of using input without an output. It's probably not ideal, but is a side-effect of having the option. Our validation paths for sampling strategies no longer utilize it, so we should be okay here.

[jakelorocco]

This PR is ready for re-review.

[planetf1]

This guard checks last_output(), but that only looks back 3 nodes by default, while Context.as_list() asserts whole-chain uniqueness by identity. If output is still in the context but further back than 3 nodes, it gets appended a second time here and the whole call blows up with AssertionError: There might be a cycle in the context tree.

I reproduced this directly against this head (not just traced it):

ctx = ChatContext().add(Message("user", "...")).add(tA).add(Message("user", "...")).add(tB)
await avalidate(reqs=[req], context=ctx, backend=backend, output=tA)
# AssertionError: There might be a cycle in the context tree.

Reachability: none of Mellea's own internal callers (sampling strategies, genstub) hit this today, since they always validate the just-generated output, which is always the context's tail. But it's trivially reachable for anyone using the parameter the way its own docstring describes: two instruct() calls, then m.validate(reqs, output=first_result) — validating something other than the latest output is the whole point of output=.

It's also a regression from the old behaviour — before this PR output went into a throwaway SimpleContext, so validating an older output worked fine.

Skipping the append instead of raising isn't a safe fix either, since Requirement.validate re-derives its target from ctx.last_output() — so a silently-skipped append would judge the wrong output. I'd suggest raising a clear ValueError here when output is present in the chain but isn't the tail, telling the caller to pass a context whose last entry is that output.

[jakelorocco]

I've changed this to always append unless the output is already the last thing in the context. I think that makes more sense. I changed the cycle logic as well to be node based instead of the data the node holds.

[planetf1]

Since parts() now exposes the bound target, and backends await every uncomputed leaf from parts() before generating, validating ≥2 requirements against the same uncomputed thunk means two tasks both end up calling astream() on it at once via asyncio.gather (functional.py:1723).

astream()'s own docstring says not to do this — two consumers race on the same queue, one grabs the completion signal, and the other waits forever for a chunk that's never coming.

I reproduced it: same setup, 1 requirement finishes instantly, 2 requirements hangs indefinitely. avalidate never returns. This is easy to hit — aact(..., await_result=False) plus more than one requirement is normal usage, nothing exotic.

Fix: await the target once before the gather, e.g.

target = validation_target_ctx.last_output()
if target is not None and not target.is_computed():
    await target.avalue()

[jakelorocco]

Done.