feat(evals): RubyLLM judge/replay glue and ProviderKey credential helpers for hosts

[TonsOfFun]

Why

A host app that drives RubyLLM acts_as_chat conversations and runs ActiveAgent::Evals suites against them, with the actionagent dashboard mounted, currently carries three pieces of generic RubyLLM-to-ActiveAgent glue. ActiveAgent::Evals::Correlation (1.6.3) already covered trace correlation; this PR moves the remaining two into the gems so every such host stops re-deriving them:

1. a Judge whose completions come from a RubyLLM chat, traced through the correlation;
2. a Replay built from the messages a conversation stored (tool calls with MCP-style errors, tokens, answer);
3. the owner's saved dashboard provider keys, as a hash or written onto a RubyLLM.context config block.

API

ActiveAgent::Evals::RubyLLM (activeagent)

Loaded by an explicit require "active_agent/evals/ruby_llm", which requires ruby_llm itself. require "active_agent/evals" alone still loads without the gem; the standalone load test now checks both directions.

judge = ActiveAgent::Evals::RubyLLM.judge(
  label: "claude-opus-5", model: "claude-opus-5", provider: :anthropic,
  context: RubyLLM,            # or a RubyLLM.context built with the host's keys
  correlation: correlation,    # optional: each call traced via Correlation#judge(kind)
  assume_model_exists: true,   # default
  **chat_options
)
# => Judge answering context.chat(model:, provider:, assume_model_exists:, **chat_options)
#      .with_instructions(instructions).ask(prompt).content

replay = ActiveAgent::Evals::RubyLLM.replay(chat.messages.order(:id), duration_ms: 1_234, metadata: { "chat_id" => chat.id })

replay(messages, answer: nil, duration_ms: nil, error: nil, metadata: {}) accepts an Array or a relation (to_a) of acts_as_chat records or RubyLLM::Message values:

• tool_calls: every message's calls in id order, { "name", "arguments", "error", "detail" }; the tool message answering a call (matched on tool_call_id) whose content is JSON with an "error" key marks it errored with that error as detail.
• input_tokens / output_tokens: summed over assistant messages from input_tokens/output_tokens (RubyLLM 1.x) or tokens.input/tokens.output (RubyLLM 2.x).
• answer: the last assistant message's content unless given.
• role may be a String or a Symbol; tool_calls may be a has_many, a Hash keyed by call id, or nil.

ActionAgent::ProviderKey (actionagent)

ActionAgent::ProviderKey.credentials_for(owner)   # => { "anthropic" => "sk-ant-...", "openai" => "sk-..." }
ActionAgent::ProviderKey.apply_to(config, owner: owner)   # config.openai_api_key = ..., per saved key

Only KEY_PROVIDERS rows with a present credential; ollama hosts are left out. config is duck-typed (anything with <provider>_api_key= writers), so the engine gains no RubyLLM dependency. A credential that cannot be decrypted is skipped with a warning naming the error class only, never the value.

Migrating a host

• Replace the host's extract_tool_calls / token-summing / replay-building code with ActiveAgent::Evals::RubyLLM.replay(messages, duration_ms:, metadata:).
• Replace the hand-written judge block with ActiveAgent::Evals::RubyLLM.judge(label:, model:, provider:, context:, correlation:).
• Replace the ProviderKey.for_owner(owner).where(...) loop that builds a RubyLLM.context with RubyLLM.context { |config| ActionAgent::ProviderKey.apply_to(config, owner: owner) }.
• Add require "active_agent/evals/ruby_llm" where the suite is run.

Docs and changelog

• docs/framework/evaluations.md: new "RubyLLM hosts" section showing Correlation + judge + replay together.
• docs/framework/self-hosted-observability.md: "Using an owner's provider keys outside the engine" next to the provider-key description; a pointer from the RubyLLM dashboard guide.
• CHANGELOG entries under Unreleased. No version bump.

Validation

• bin/test test/evals/*_test.rb actionagent/test/credentials_test.rb actionagent/test/dashboard_assistant_service_test.rb actionagent/test/provider_key_test.rb: 182 runs, 0 failures (includes the 16 new tests).
• rubocop --config .rubocop.yml on every changed Ruby file: no offenses.
• npx vitepress build docs: builds; the new anchors resolve.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V2Eg7xu5GJ5UsSw8GmGNix

---

Generated by Claude Code