RFC: prompt caching needs a breakpoint in the right place, not just an exposed flag

[IsmaelMartinez]

Summary

On Anthropic and Bedrock, prompt caching needs an explicit cache breakpoint. #2610 landed the config pass-through that lets one be declared, which was the right first step.

On its own it wins little, because the cacheable region depends on where shared content sits in the prompt and on where a breakpoint can physically land. And as merged it cannot be switched on at all by callers addressing Claude through a Bedrock application inference profile. There is one exception, described below, that is worth taking regardless.

Revised 12-08-2026: #2610 landed the pass-through, so the old item 1 is done; added the model-gate item in its place. See the edit history for what changed.

Where a breakpoint can land

PR-Agent builds a single system message from a plain string:

messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]

Given a string, litellm's AnthropicCacheControlHook sets cache_control on the message as a whole; given a list it sets it on the last content block. The injection point's index selects a message, not a content block. So the breakpoint always lands at the end of the system prompt, and a single system message cannot be split.

One consequence is already correct: the diff sits in the user message, after the breakpoint, so the variable content is outside the cached region with no work needed.

The other consequence is that the cached prefix is the whole system prompt, so the cache key is whatever that prompt varies on.

What that means per tool

A single-chunk run makes four calls, not three, because each /improve chunk runs a suggestions call plus a mandatory self-reflect call.

call	system prompt varies on	reusable
/describe	repo_context, skills_context, extra_instructions, custom_labels_class	per repo
/review	the same, plus num_pr_files	per PR, so effectively never
/improve suggestions	the same, plus num_code_suggestions	per repo
/improve reflect	nothing at all	globally

The reflect prompt is the interesting one. It interpolates no variables, so it is a static block of roughly 1.7k tokens that is byte-identical for every repo and every PR. It is the only globally shared prefix in the pipeline, it clears the 1,024 token minimum on the models that matter, and it is cacheable today with no prompt restructuring at all.

/review is the opposite case. num_pr_files moves its prefix on nearly every PR, so its cache entry is written and almost never read.

The placement problem

For the other three calls, tool-specific text comes first in the prompt files and skills_context / extra_instructions are interpolated after it. Two consequences:

• No cross-tool hits. /describe, /review and /improve diverge in the first few lines, and caching is prefix-based. Same conclusion as [Feature Request] Prompt caching #1353, and correct for the current layout.
• The cached prefix ends up tied to per-repo content, so quiet repos write entries that expire unread and pay the write premium for nothing.

Hoisting the shared context into a leading block is necessary but not sufficient, because of the end-of-message constraint above: reordering leaves the breakpoint downstream of both the shared block and the tool-specific text. Getting a genuine cross-tool prefix needs either the system content emitted as structured content blocks with cache_control on the shared one, or the tool-specific text moved out of the system message.

Landed, with one gap

#2610 merged the pass-through on 08-08-2026, superseding #2405. It takes a native TOML array, so the TypeError from json.loads on a Dynaconf list is gone; it resolves and validates the value ahead of the retry wrapper, so a malformed config raises ValueError instead of being wrapped as an APIError and retried; it uses setdefault rather than overwriting kwargs; and it ships unit tests.

The gap is the gate. The kwarg is only passed when "claude" in model.lower(). Callers addressing Claude through a Bedrock application inference profile pass bedrock/converse/arn:aws:bedrock:<region>:<account>:application-inference-profile/<id>, which contains no claude, so the feature silently no-ops and logs a debug line. That is the same failure mode as item 5 below, and as temperature handling before it: a capability gated on a model name that an opaque profile ARN can never satisfy. Gating on the provider prefix, or honouring an explicit opt-in when the config is set, would fix it.

To do

1. Fix that gate, so the feature merged in feat(litellm): pass-through cache_control_injection_points for Anthropic prompt caching #2610 can actually be enabled by application inference profile callers.
2. Enable a breakpoint for the self-reflect call. This is the one immediate win and it depends on nothing else.
3. Restructure the system prompt so a shared leading block can carry the breakpoint: structured content blocks, or tool-specific text out of the system message. Needs a maintainer view, since it changes prompt layout.
4. Document the ordering: shared content first, repo-specific last.
5. Document TTL behaviour. is_claude_4_5_on_bedrock no longer uses a hardcoded name list; since #31929 it reads cache_creation_input_token_cost_above_1hr from the pricing JSON, so Sonnet 5 and Opus 5 pick up the 1h TTL from litellm 1.93.0 and requirements.txt is past that. The gap left to document is that the lookup needs a resolvable model name, so callers addressing Bedrock through an application inference profile ARN get an opaque id with no pricing entry and silently fall back to 5m. Nothing in feat(litellm): pass-through cache_control_injection_points for Anthropic prompt caching #2610 mentions TTL, so this is still open.

The case against

Caching is not free. Writes cost 1.25x at 5 minutes and 2x at 1 hour, reads cost 0.1x, so break-even is a 21.7 or 52.6 per cent hit rate. Below that this raises the bill, and a user enabling it on a quiet repo gets no signal.

A single PR run writes four entries and reads none, since the four system prompts all differ. Reads only arrive from later runs. Chunked suggestions calls do not help each other either: parallel_calls dispatches them through asyncio.gather, and a cache entry only becomes available once the first response begins, so concurrent siblings each write their own.

Item 3 also reorders prompt content, which carries a quality risk worth testing rather than assuming.

To be clear, #1353 asked to cache the diff across commands. That is still infeasible and this does not revive it. Only the leading shared context, and the static reflect prompt, become cacheable.

Items 1 and 2 are both small and self-contained. Happy to send a patch for either; item 3 is the one I would rather agree first.

[IsmaelMartinez]

Two corrections to my own list, plus one thing I missed. All checked against source.

Item 4 is already done. is_claude_4_5_on_bedrock no longer uses a hardcoded name list; since #31929 it reads cache_creation_input_token_cost_above_1hr from the pricing JSON, so Sonnet 5 and Opus 5 get the 1h TTL from litellm 1.93.0 and requirements.txt is past that. What is left to document is the gap it leaves: the lookup needs a resolvable model name, and a Bedrock application inference profile ARN reduces to an opaque id with no pricing entry, so those callers silently get 5m.

Item 2 needs more than hoisting. pr-agent builds a single system message from a plain string, and the hook sets cache_control on the whole message; the injection point's index selects a message, not a content block. So the breakpoint always lands at the end of the system prompt, and hoisting shared context to the front leaves it downstream of exactly the region we wanted to isolate. A cross-tool prefix needs the system content emitted as structured blocks with cache_control on the shared one, or the tool-specific text moved out of the system message.

The easy win is /improve's self-reflect call. Its system prompt interpolates no variables at all: a static block of roughly 1.7k tokens, identical for every repo and every PR. That makes it the one globally shared prefix in the pipeline, and it is cacheable today with no layout change and no dependency on item 2. The reviewer prompt is the opposite case: it interpolates num_pr_files, so its prefix shifts with the PR's file count and its cache entry is nearly always a wasted write.

Two things temper that. A single-PR run writes four entries and reads none, since /describe, /review, suggestions and reflect all have different system prompts, so reads only come from later runs. And chunked suggestions calls do not help each other: parallel_calls dispatches them through asyncio.gather, and per Anthropic's docs a cache entry only becomes available once the first response begins, so concurrent siblings each write their own. Whether it nets out positive depends on how often reflect calls recur inside the TTL, which is worth measuring rather than assuming.

---

Update: I have edited the RFC body above to fold these corrections into the proposal itself, so it now reads as one proposal rather than proposal plus errata. The issue edit history has the before and after if you want to see exactly what changed.

[xyzs996]

On the break-even arithmetic in The case against — the 21.7% / 52.6% thresholds fall out of r = 0.1, and that holds for exactly half the catalog but not the rest.

I keep a table that re-reads OpenRouter's catalog daily. Of the 40 non-batch models on it today, every one publishes a cache-read price, and as a fraction of list input those come out:

cache read ÷ input	models
0.083	1
0.100	20
0.104 – 0.186	10
0.200	7
0.250 – 0.254	2

Through the same formula, h = (w − 1) / (w − r):

r	break-even at w = 1.25	at w = 2.0
0.083	21.4%	52.2%
0.100	21.7%	52.6%
0.200	23.8%	55.6%
0.254	25.1%	57.3%

So the numbers in the RFC are the floor rather than the typical case: 19 of the 40 need a higher hit rate than stated — up to 3.4 points more at the 5-minute TTL and 4.7 at the hour. That doesn't change the conclusion for Anthropic itself, which is 0.1. But PR-Agent reaches models through litellm, and on a route like Grok 4.6 (0.250) or Kimi K2.7 Code (0.254) the "quiet repo pays the write premium for nothing" region is meaningfully wider than the note implies — worth a sentence if the doc lands as written.

One assumption I expected to break and it doesn't: for the 8 models on the table that publish a separate long-context tier, the cache-read ratio is identical either side of the step — Grok 4.20 is 0.160 below 200k and 0.160 above, Gemini 3.1 Pro Preview 0.100 / 0.100, GPT-5.5 0.100 / 0.100. The absolute bill steps, the break-even hit rate does not. So a PR big enough to push the prompt past 200k needs no separate arithmetic, which is one fewer case for item 5 to document.

To check any of it:

curl -s https://cdn.jsdelivr.net/gh/xyzs996/llm-api-pricing@main/data/prices.csv

Columns are input_per_million, cached_input_per_million, long_input_per_million, long_cached_input_per_million; rebuilt daily, CC BY 4.0. Cache write premium isn't in that data, so I can't second-guess the 1.25 / 2.0 side — those I took from your note as given.

[IsmaelMartinez]

Closing this as duplicated and completed. The same RFC exists as issue #2601, which is where the work was actually tracked, and that closed as completed on 23-08-2026.

One item from here is still live and has its own issue: the model gate cannot match a Bedrock application inference profile, since it tests "claude" in model.lower(). That is tracked as #3216.