# Prompt caching

The cache serves the unchanged start of your input cheaply; one early change makes everything after it fresh again. Learn what breaks the prefix.

> Prompt caching serves the unchanged start of your input from fast storage at a low price. It matches exactly from the first token, so one early change makes everything after it fresh again.

Source: https://ai-sw-factory.mellicci.dev/cost/prompt-caching

**Diagram:** The cache serves the longest identical start of the input; the first changed segment makes it and everything after it fresh.

- Same prefix, next call:
  - Cached, read cheaply:
    - System and tools — tool definitions included
    - →
    - Instruction files — always loaded
    - →
    - History
  - →
  - New input — fresh input
- One early segment changed:
  - System and tools — a server was toggled
  - →
  - Fresh again, written anew:
    - Instruction files
    - →
    - History
    - →
    - New input

Every call re-sends the whole context, as [tokens, turns and context](https://ai-sw-factory.mellicci.dev/cost/tokens-turns-and-context) explains. Prompt caching is what keeps that re-reading cheap. The provider remembers the start of your input from a recent call, and if the start of this call is the same, it serves that part as a cache read.

The match is exact and runs from the first token. The cache holds a **prefix**: the first N tokens must be identical, byte for byte, to a recent call. It does not know two prefixes are "almost the same".

## Why it matters

Most of a long session's input is old material that has not changed. When it is read from the cache, it is the cheapest token in the system. When something early changes, that whole stretch is billed again, and as a cache write it is dearer than fresh input (see [why tokens cost differently](https://ai-sw-factory.mellicci.dev/cost/why-tokens-cost-differently)). A single toggle can turn a cheap reply into the most expensive one of the hour.

## How it works

The harness assembles the input in a fixed order, and the cache compares it with the last call from the top. The first difference ends the match.

| What changes | Where it sits | What the next call does |
|---|---|---|
| New user message, new tool result | End of input | Prefix still matches; only the new part is fresh |
| Edited [instruction file](https://ai-sw-factory.mellicci.dev/fundamentals/instruction-files) | Near the start | Everything after the edit is fresh or written again |
| Tool or [MCP server](https://ai-sw-factory.mellicci.dev/fundamentals/mcp) toggled | Tool definitions, near the start | Almost the whole input is rebuilt |
| Different system prompt or model setting | Very start | The whole input is rebuilt |
| A timestamp near the top | Near the start | Differs on every call, so the cache never matches past it |

The rule that follows is **stable material first, volatile material last**. The harness controls most of the order; you control what you change, and when.

Which parts count toward the cache and how it is applied differ by provider. How long an entry survives is a separate question: see [cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime).

## Example

The `ticket-service` team's agent spend doubled last month. One developer, deep in a long session, enabled an extra MCP server to look up a tracker issue. The tool definitions sit at the start of the input, so the next call matched almost nothing. Compare it with the call before.

| Tokens, by kind | Call without the toggle | Call after the toggle |
|---|---|---|
| Cache read | 120,000 | 0 |
| Cache write | 2,000 | 128,000 |
| Fresh input | 500 | 500 |

The extra server added about 6,000 tokens of definitions, and the old 120,000 tokens no longer matched. The harness rebuilt the whole prefix as a cache write instead of reading it. The fix is to set tools up before the session starts, not during it. If you need a different set, a fresh session pays the rebuild once, on purpose.

## What helps, what hurts

Helps:

- Choose tools, servers and instruction files before the first prompt; the prefix then stays fixed.
- Keep always-loaded material small and steady. [Skills](https://ai-sw-factory.mellicci.dev/fundamentals/skills) load details only when needed, so the always-loaded prefix stays short.
- Put changing facts (dates, task notes) in your messages, at the end, not in files loaded at the start.
- Start a new session when you really need a different setup, and see [cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime) for when the cache lapses anyway.

Hurts:

- Toggling an MCP server or tool mid-session.
- Editing an instruction file during a session.
- Anything that stamps a changing value near the top, such as a timestamp.
- Switching the model or system prompt in the middle of a long conversation.

## Trade-offs

- A fixed setup is less flexible: you may carry a tool you rarely use, or restart to add one.
- Starting fresh instead of toggling drops the conversation's context, so you re-brief the agent.
- Keeping instruction files short helps the prefix but leaves less standing guidance.
- A cache write costs more than fresh input, so caching only pays when the prefix is reused.

## See it in your own sessions

The tell-tale is a share of input served from cache that drops suddenly, or cache writes where you expected reads. One way to see this in your own sessions is [Agent Tokenwatch](https://github.com/mellicci/agent-tokenwatch) ([project page](https://mellicci.dev/work/agent-tokenwatch)), an open-source command-line tool for Claude Code, Codex CLI and GitHub Copilot CLI. It records only token counts and structure, never prompts, code or tool contents. Its status line shows cache health per reply, and the session retrospective shows when and why the cached share fell.

## Related

- [Why tokens cost differently](https://ai-sw-factory.mellicci.dev/cost/why-tokens-cost-differently): the price order of reads, writes and fresh input
- [Cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime): how long a cached prefix lives
- [Instruction files](https://ai-sw-factory.mellicci.dev/fundamentals/instruction-files): always-loaded, part of the prefix
- [MCP](https://ai-sw-factory.mellicci.dev/fundamentals/mcp): tool definitions are part of the prefix
- [Skills](https://ai-sw-factory.mellicci.dev/fundamentals/skills): progressive disclosure keeps the prefix small

## Key terms

- **Cached prefix** - the unchanged start of the input, identical token for token to a recent call, so it is served as a cache read.
- **Cache break** - a change early in the input that stops the match, so every token after it is fresh or written again.
- **Stable first, volatile last** - the ordering rule that keeps the prefix long, with rarely changing material at the start and changing material at the end.