Add tool calling to LlamaLanguageModel, including streaming - #227
Merged
Merged
Conversation
llama_chat_apply_template has no parameter for tool definitions, so tool support is implemented at the prompt layer. The tool syntax is detected from the model's embedded chat template: Hermes-style JSON (Qwen 2.5/3 and most ChatML fine-tunes), Qwen 3.5's XML function/parameter form, and the Gemma 4 canonical format with its token-quoted argument notation. Definitions are rendered into the system prompt following each template's own wording and placement, past tool turns are replayed in the native markup (including Gemma 4's open-model-turn continuation), and calls are parsed back out of generated text and run through the resolve-and-continue loop used by the MLX and Ollama backends, with the same delegate hooks, iteration cap, and repeated-signature guard. Generation stops early once a complete tool-call block is produced. Non-streaming respond() only; streamResponse() ignores session tools as before. (cherry picked from commit 89cf703)
streamResponse() now runs the same resolve-and-continue loop as respond(), yielding snapshots that carry the cumulative visible text and the tool-call and tool-output entries produced so far. Because llama tool calls arrive as text rather than parsed events, snapshots withhold any trailing partial match of a call-start marker until the next token confirms or breaks it, so markup never appears mid-stream. Gemma 4 emits thought-channel spans without being asked: thinking is opt-in via a system-turn token this backend never injects, and the canonical template ships a strip_thinking macro for consumers. Both respond() and streamResponse() now remove completed spans and withhold unclosed ones, recognizing the canonical marker spelling and the variant observed from deployed quantizations. (cherry picked from commit 436f229)
…fixes at end of round respond() only kept the visible text of the final round, so a preamble before a tool call was dropped while the streaming path kept it; accumulate it the same way. The streaming path also committed each round's text with a trailing marker prefix still held back, so an answer that legitimately ends in one lost that suffix. Release it once the round has ended.
There was a problem hiding this comment.
🟡 Changes recommended
Prompt opt-outs, malformed Qwen calls, and cross-round text boundaries require correction.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds prompt-layer tool calling to the Llama backend, including streamed execution and transcript replay.
Changes:
- Supports Hermes JSON, Qwen XML, and Gemma tool formats.
- Adds tool-resolution loops for standard and streaming responses.
- Adds format and model integration tests.
File summaries
| File | Description |
|---|---|
LlamaToolCallFormatTests.swift |
Tests tool formatting, parsing, replay, and streaming. |
LlamaToolCallFormat.swift |
Implements tool syntax detection, rendering, and parsing. |
LlamaLanguageModel.swift |
Integrates tool execution into generation and streaming. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟡 Changes recommended
Gemma schema and result serialization can lose types, and non-streaming output handling is inconsistent.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Balanced
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@james-333i's #216, cherry-picked onto
mainwith his authorship intact: the two tool-calling commits, without the copies of #213, #214, #215, and #181 that #216 carried and that have since landed on their own.Original description and review discussion: #216.