The Anthropic API caches prompt prefixes (cache_control breakpoints). A cache hit prices cached input at about 10%. More important here, it cuts time to first token a lot, because the provider skips re-prefilling the transcript. Our sessions fit this well: a stable system prompt, then tool contracts, then an append-only transcript.

pi-ai 1.0 asks for short cache retention by default: it sets cache breakpoints for Anthropic and sends each durable conversation's session id as the cache key where a provider routes by one. What is left is to check whether usage's cacheRead (already recorded in RunUsage) shows hits in prod, and whether long retention pays for itself.

Breakpoint placement: end of system prompt, end of tool definitions, and a moving breakpoint at the second-to-last turn. Keep the prefix byte-stable (see lever 4).

The Responses API supports store: true + previous_response_id. The client sends only the new turn and the provider keeps the context. Per-turn upload drops from O(session) to O(delta). That is the biggest single win for the large dev sessions that produced 91 GB of egress in 33h. Pi SDK support still needs checking. Fall back to full replay where it is not supported.

Old tool results are the dead weight. A 64 KB exec output that stopped mattering three turns ago is re-serialized, re-uploaded, and re-prefilled every turn, and on a cache miss it breaks the prefix. Elide or summarize tool results older than N turns, and leave a model-visible [output elided] note. This cuts bytes and event-loop serialization time. It also makes prompt-cache prefixes shorter to rebuild after a miss. The append-time spill of oversized outputs (a follow-up from the finished perf-squeeze plan) does the same job on the storage side. Session continuation is a separate approach that starts a fresh session.

Prompt caching only pays when the prefix is identical across turns. Since the move to durable sessions the worst hazards are gone: the system prompt no longer carries the clock, the transcript is a list of stored entries that never change, and the per-turn state blocks (plan, context usage, session status, time) ride behind each turn's input without being stored. Hazards that remain:

A delegated child's result is an edit of its placeholder entry, so a park-resume keeps the cache up to that entry and loses it from there on. A turn that rebuilds its whole context from the log (a restart repair, a session without a store) starts with a cold cache.

System-prompt resolution embeds remote hm:// docs (cached 5 min). A re-fetch that changes bytes mid-session breaks the whole cache. Consider pinning the resolved prompt for the session's lifetime.

Tool promotion (touch-expand) changes the tool list mid-session. That can't be avoided when it happens. Tool definitions should sit in their own cache segment, so a promotion only invalidates from that point on.

provider.request_gap measures everything before the request leaves. Known costs in that window: opening the session's durable store, which reads its main file, and importing what reached the log since the last turn. A turn that has to rebuild its context (a restart repair, a session without a store) also pays #piMessages decoding the full CBOR transcript. Both grow with session length. If prod shows this gap growing, the fix is keeping a store open between the turns of a busy session.

Each tool batch costs a full provider round trip on a growing context. Two behavioural levers need no protocol work:

Tool prompts should keep pushing agents to batch independent tool calls in one turn (parallel calls in one response), instead of one call per turn.

Long tool outputs where the model only needs a slice (big file reads, listings) should come back already bounded. The smaller each result, the cheaper every later turn. Output caps mostly do this already. Audit the stragglers.