Skip to content

chat: keep vendor fields on tool calls in both directions - #92

Merged
CMGS merged 2 commits into
mainfrom
fix/chat-tool-call-vendor-fields
Sep 18, 2026
Merged

CMGS merged 2 commits into
mainfrom
fix/chat-tool-call-vendor-fields

Conversation

@CMGS

@CMGS CMGS commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Problem

On Google's OpenAI-compatible endpoint (/v1beta/openai), Gemini 3 returns a per-call thought signature at tool_calls[i].extra_content.google.thought_signature. The next turn must echo it back, or Google answers 400 INVALID_ARGUMENT: Function call is missing a thought_signature. The gateway dropped the field in three places, so every Gemini 3 tool loop through it failed on turn 2 with 424 (original_status_code: 400).

# Site Path Effect
1 chat request parse: typed ChatMessage.tool_calls: Vec<ToolCall>, then to_value every request that replays history the echo never reaches the vendor, which is the 400
2 buffered render: from_value::<Vec<ToolCall>> stream: false the client never receives the signature
3 merge_tool_call_fragments copies only id/type/name/arguments stream replay after an output-DLP hit, and the thinking-strip path the replayed call loses it

The live stream path and wire_messages already forwarded the field.

Fix

  • ChatMessage.tool_calls becomes Option<Vec<Value>>, the same shape ChunkDelta.tool_calls and ChatMsg.tool_calls already use. ToolCall and FunctionCall had no other users, so they are removed.
  • Request: m.tool_calls.map(Value::Array) moves the calls. This removes the old per-request to_value, which re-serialized every historical call, arguments included.
  • Buffered render: forwards the output of openai_tool_calls(..) unchanged. The chat: render anthropic tool_use blocks as OpenAI tool_calls #28 500 arm existed because the typed conversion could fail, and before chat: render anthropic tool_use blocks as OpenAI tool_calls #28 that failure fell back to []. With no conversion there is nothing to drop, and the stream surface already forwarded the same payload.
  • Accumulator: the first value wins for every fragment field except index and function. Inside function, the name is still first-wins and arguments still append.

Rejected alternatives:

  • extra_content: Option<Value> on ToolCall keeps the typed round trip. Every request would then copy each historical signature once more, because to_value serializes by reference, and the next vendor field would be dropped again.
  • #[serde(flatten)] buffers every call through serde's Content on every request.

Ownership

  • Request: the body parse allocates the signature once. After that it only moves: ChatMsg, then wire_messages, then the upstream body.
  • Buffered response: moves from GatewayResponse.tool_calls into the response with no clone.
  • Stream: one clone per call into the accumulator. The fragment itself goes to the client, and the DLP-hit replay needs its own copy. This follows the same rule as the existing id/type clone.

Behavior changes

  • A malformed tool call echoed by a client (missing type, object arguments) is no longer rejected with a gateway 400. It reaches the vendor, which returns a 4xx. That does not trigger fallback, because is_upstream_fault falls back only on 5xx/429.
  • Buffered chat replies from an anthropic-messages model now keep the index that tool_use_to_tool_calls sets; the typed parse used to strip it. I probed OpenAI, DeepSeek, DashScope, SiliconFlow and Moonshot directly: all accept index and extra_content on an echoed call.
  • Blocklist: security_check walks every string leaf, opaque proofs included (SignedThinking::Visit). The signature is now scanned the same way an Anthropic thinking signature already is.
  • DLP redaction cannot realistically rewrite standard base64: emails need @, secrets need -/_ or AKIA[0-9A-Z]{16}, and the CN-mobile rule needs 1[3-9] followed by 9 more digits.

Hot path

The A/B ran on an Apple M2 Max, with both versions of openai.rs in one process via #[path]. Each case ran 41 rounds with alternating arm order; the whole run was then repeated with the order swapped. The table shows the mean of the two runs' medians. The two runs agree within 2%.

Case Old New Δ
request, 1 call, no signature 1245 ns 1125 ns −119 ns (−9.5%)
request, 10 calls, no signature 9.59 µs 8.20 µs −1.39 µs (−14.5%)
request, 100 calls, no signature 90.1 µs 79.9 µs −10.1 µs (−11.2%)
request, 10 calls × 432 B signature 10.6 µs 11.9 µs +1.27 µs
request, 100 calls × 432 B signature 100.3 µs 118.3 µs +18.0 µs
request, 100 calls × 4 KiB signature 134.8 µs 171.6 µs +37.2 µs
buffered render, 1 / 4 calls, no signature 735 / 1983 ns 712 / 1873 ns −24 / −112 ns
buffered render, 1 / 4 calls × 432 B 949 / 2850 ns 1163 / 3644 ns +214 / +793 ns
stream accumulate, 6 fragments, no signature 937 ns 857 ns −81 ns
stream accumulate, 6 fragments, 432 B / 4 KiB 958 / 953 ns 1145 / 1163 ns +184 / +208 ns
  • Buffered-render rows include one input clone in both arms: 259 ns for 1 call and 1056 ns for 4 calls.
  • The live signatures measured 296–432 B.
  • Requests without vendor fields, which covers every non-Gemini client, get cheaper on all three paths.
  • With signatures, the added cost is the signature bytes themselves. The old code parsed past them and dropped them; carrying them to the vendor is the fix.

Evidence

  • New tests: tool_calls_keep_vendor_fields_both_ways (protocol), tool_call_fragments_keep_vendor_fields_first_wins (engine), and chat_tool_calls_keep_vendor_fields_both_ways (e2e: buffered, live stream, and DLP-hit replay). Each one fails on the old code:
    • with src at main, the e2e fails on the upstream-body assertion (left: Null);
    • with only the old accumulator restored, it fails on dlp=true only;
    • the engine unit test fails on the old merge.
  • cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace are clean: 31 suites, 622 passed, 0 failed. The env-gated PG/Redis suites were not run.
  • Live check: a two-turn gemini-3.6-flash tool loop (reasoning_effort: high) through a local gateway. The client echoes the turn-1 assistant message verbatim. The branch passed 3 of 3 runs in each mode, plus the side-by-side run in the table.
Google direct Gateway at main 0c087b99 This branch
buffered: turn 1 client sees extra_content yes no yes
buffered: turn 2 stop 424 (upstream 400) stop
stream: turn 1 client sees extra_content yes yes yes
stream: turn 2 stop 424 (upstream 400) stop

Not in this PR

  • The native gemini protocol engine (VertexEngine) has no tool calling: tools never reaches the body and functionCall parts are not parsed. openai-chat against /v1beta/openai is the only Gemini tool route.
  • The DLP-hit stream replay emits tool-call deltas without index (pre-existing; visible in this PR's e2e output). Strict stream parsers such as the AI SDK's zod schema reject those deltas.
  • /v1/messages served by an openai-chat model cannot carry the signature, because Anthropic tool_use blocks have no slot for it.

Gemini 3 on its OpenAI-compatible endpoint returns a thought signature at
tool_calls[i].extra_content.google.thought_signature and rejects the next
turn with 400 INVALID_ARGUMENT when the echoed call lacks it. The typed
ToolCall (id/type/function) dropped the field at three sites: the request
parse, the buffered response render, and the stream accumulator that the
DLP-hit replay rebuilds from. A tool loop through the gateway failed on
turn 2 with 424.

ChatMessage.tool_calls is now raw JSON like ChunkDelta.tool_calls and
ChatMsg.tool_calls: the request moves into ChatMsg without the per-call
to_value copy, and the buffered render forwards what the engine returned.
The accumulator keeps the first value of every fragment field except index
and function. The typed render's 500 arm goes with it; nothing converts,
so nothing can drop.
Raw chat tool calls can carry non-object values after vendor-field passthrough. Reject those shapes as a client error before Responses conversion and move valid fields without reserializing them.
@CMGS
CMGS merged commit 2ead154 into main Sep 18, 2026
2 checks passed
@CMGS
CMGS deleted the fix/chat-tool-call-vendor-fields branch September 18, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant