# Measuring cost

Find where your tokens went by reading counts per reply: tokens by kind, cache share, misses, the few costliest replies and context growth.

> Measuring cost means reading token counts by kind for each reply and each session, then asking a few fixed questions of them. Counts and structure are enough; you do not need to read prompts or code.

Source: https://ai-sw-factory.mellicci.dev/cost/measuring-cost

**Diagram:** Measuring cost turns a session into replies, splits each into token kinds, and asks four questions of the result.

- Session — many replies
- → split into
- Replies — tool calls included
- → counted by kind
- Tokens by kind — fresh, cache read, cache write, output
- → ask
- Where did it go?:
  - Costliest replies
  - Cache share
  - Cache misses
  - Context growth

You cannot tune what you cannot see, and a monthly total hides almost everything useful. The useful view is smaller: how many tokens of each kind each reply used, and how those add up across a session.

A reply is the unit, not a message. One reply includes every tool call the model makes before it answers, so a one-line prompt can be a large reply. See [Tokens, turns and context](https://ai-sw-factory.mellicci.dev/cost/tokens-turns-and-context).

## Why it matters

Without measurement, spend rises and the only lever is blame or guesswork. With it, you can tell a cold cache from a bloated context from an output-heavy task, and each has a different fix.

Cost is concentrated. A handful of replies usually explain most of a session's tokens, so finding them beats trimming everything a little.

## How it works

Four questions, each answered from counts alone.

| Question | Look at | A likely cause when it looks bad |
|---|---|---|
| Which replies cost most? | Tokens by kind, ranked per reply | One event, not a habit; check what happened just before it |
| How much input came from the cache? | Cache read as a share of all input | A changed prefix or an expired cache: [Prompt caching](https://ai-sw-factory.mellicci.dev/cost/prompt-caching), [Cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime) |
| Where are the misses? | Replies with a large cache write or fresh input where a read was expected | A break, a toggled tool or an edited instruction near the top |
| How big is the context over time? | Input size per reply across the session | Output piling up in the conversation: [Context growth](https://ai-sw-factory.mellicci.dev/cost/context-growth) |

Read the kinds against the ordering from [Why tokens cost differently](https://ai-sw-factory.mellicci.dev/cost/why-tokens-cost-differently): a cache read is the cheapest, output the most expensive. Two replies with the same total tokens can differ a lot in cost.

**Estimates versus bills.** A figure computed locally from token counts and assumed rates is an estimate. Rates change, and plans differ in how they bill. The provider's billing is authoritative; use local numbers to find causes and compare runs, not to reconcile an invoice.

**Compare like with like.** Cost per day or per message rewards doing less. Cost per finished task is the fairer unit, and it is what [Your first cost experiment](https://ai-sw-factory.mellicci.dev/cost/your-first-cost-experiment) uses.

**Privacy.** None of this needs prompts, code or transcripts. Counts, timestamps and structure are enough, so you can measure without storing anything sensitive.

## Example

The `ticket-service` team reads one developer's day: 140 replies, but 5 of them account for about half the tokens. Ranking replies by total and labeling each by its dominant kind shows five different events, not one habit. The numbers are illustrative.

| Reply | Dominant kind | What happened | Page |
|---|---|---|---|
| 1 | Cache write, about 90,000 tokens | First reply after a long lunch break | [Cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime) |
| 2 | Cache write, about 70,000 tokens | A tool server toggled, so the prefix changed and was rebuilt | [Prompt caching](https://ai-sw-factory.mellicci.dev/cost/prompt-caching) |
| 3 | Cache read, about 180,000 tokens | A late reply in a session still carrying a pasted 40,000-token log | [Context growth](https://ai-sw-factory.mellicci.dev/cost/context-growth) |
| 4 | Output, about 12,000 tokens | Long reasoning on a simple rename | [Model and reasoning choice](https://ai-sw-factory.mellicci.dev/cost/model-and-reasoning-choice) |
| 5 | Fresh input, about 40,000 tokens of tool output | A vague prompt triggered wide exploration | [Prompts and exploration](https://ai-sw-factory.mellicci.dev/cost/prompts-and-exploration) |

Five causes, five different fixes. A flat "use the agent less" would have addressed none of them.

For a measured case beyond this illustration, see [Token cost: is it cheaper to let an agent manage Kubernetes with CLI or MCP?](https://www.mellicci.dev/writing/cli-or-mcp-kubernetes-agent-token-cost). It compares setups over 40 agent runs on five Kubernetes operations, measured per session against independently verified success. The plain CLI setup used 76% fewer tokens per successful task than the MCP setup and succeeded more often, and the difference came from the context carried into every call, not from the output.

## What helps, what hurts

Helps:

- Rank replies by cost before changing anything; the top few tell you where to look.
- Track cache share over a session, because a sudden drop marks a miss you can name.
- Plot context size per reply, which shows [growth](https://ai-sw-factory.mellicci.dev/cost/context-growth) before it becomes a bill.
- Compare cost per finished task, over similar tasks.

Hurts:

- Judging by a total or an average, which hides the few replies that matter.
- Counting messages instead of replies, so tool-heavy work looks free.
- Treating a local estimate as the invoice.
- Comparing a quiet day with a busy one and calling the gap a trend.

## Trade-offs

- Local estimates are quick but approximate; the bill is exact but arrives late and is coarse.
- Per-reply detail finds causes but takes attention; a weekly view is cheaper to read and less precise.
- Cost per finished task needs you to define "finished", and a cheaper run that does worse work is not a win.

## See it in your own sessions

One way to see this in your own sessions is [Agent Tokenwatch](https://github.com/mellicci/agent-tokenwatch), an open-source Node.js command-line tool for Claude Code, Codex CLI and GitHub Copilot CLI. It records token counts and structure locally and never prompts, code, transcripts, file paths or tool contents.

- **Status line:** shows the live session, with its tokens and cache health (warm or cold).
- **Retrospective skills:** show what a session or a period cost and what drove it, compared with your own average.
- **Cost coach:** walks you through one expensive moment, such as a cold break or a huge tool output.

```sh
npm install -g agent-tokenwatch
tokenwatch install --agents all --scope user
tokenwatch doctor
```

More on [the project page](https://mellicci.dev/work/agent-tokenwatch).

## Related

- [Tokens, turns and context](https://ai-sw-factory.mellicci.dev/cost/tokens-turns-and-context) - why a reply is the unit.
- [Prompt caching](https://ai-sw-factory.mellicci.dev/cost/prompt-caching) - what a cache hit needs.
- [Cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime) - why breaks cause misses.
- [Context growth](https://ai-sw-factory.mellicci.dev/cost/context-growth) - why replies get dearer.
- [Your first cost experiment](https://ai-sw-factory.mellicci.dev/cost/your-first-cost-experiment) - measure, change one thing, compare.

## Key terms

- **Token kind** - fresh input, cache read, cache write or output, each priced differently.
- **Cache share** - the fraction of input served from the cache.
- **Cache miss** - a reply whose prefix was not found in the cache.
- **Estimate** - a local figure from counts; the provider's billing is authoritative.
- **Cost per finished task** - total cost of the replies needed to finish one task.