# Your first cost experiment

Reduce agent cost with evidence: change one thing, compare cost per finished task on similar work, check quality, then keep or drop it.

> A cost experiment changes one habit, compares cost per finished task before and after on similar work, and checks quality. You keep the change only if cost drops and the work stays as good.

Source: https://ai-sw-factory.mellicci.dev/cost/your-first-cost-experiment

**Diagram:** An experiment is a loop: measure, change one thing, run it on similar work, compare cost and quality, then keep or drop.

- Measure baseline — cost per finished task
- Pick one change — the biggest driver first
- Run on similar work — same kind of tasks
- Compare — cost and quality together
- Keep or drop — record the decision
- Centre: One variable

A cost experiment is a small, controlled test of one habit. You decide the metric and the quality check before you start, run the change on work that resembles your baseline, and let the numbers decide.

This page ties the module together. Every earlier page named a mechanism and a habit that follows from it; here you test whether the habit pays off in your own work.

## Why it matters

Cost advice is easy to believe and hard to verify. A change that seems obviously cheaper can do nothing, or it can move cost from one kind of token to another.

Two traps are common. If you change three habits at once, you cannot tell which one mattered. If you judge by cost alone, a cheaper run that fails more tests looks like a win until the rework arrives.

## How it works

| Step | What you do | Why |
|---|---|---|
| Start from the biggest driver | Use what you measured in [Measuring cost](https://ai-sw-factory.mellicci.dev/cost/measuring-cost) | A small change to a big driver beats a big change to a small one |
| Change one thing | One habit, nothing else | Otherwise you cannot attribute the result |
| Choose the metric in advance | [Cost per finished task](https://ai-sw-factory.mellicci.dev/cost/measuring-cost), in tokens by kind | Per message hides re-reading; per day depends on how much you worked |
| Match the work | Baseline and comparison windows cover similar tasks | A week of renames and a week of a new feature are not comparable |
| Add a quality check | Failing tests after merge, reverted changes, rework | Cheaper work that is worse work is not a saving |
| Collect enough runs | Enough finished tasks to see past normal variation | A handful of tasks can swing either way by chance |
| Decide and record | Keep, drop, or try the next variable | An unrecorded result gets re-argued next quarter |

Candidate changes come from the earlier pages:

- a fresh session per task ([Context growth](https://ai-sw-factory.mellicci.dev/cost/context-growth))
- the tool set fixed before the session starts ([Prompt caching](https://ai-sw-factory.mellicci.dev/cost/prompt-caching))
- compacting or handing over before a break ([Cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime))
- a smaller model for mechanical work ([Model and reasoning choice](https://ai-sw-factory.mellicci.dev/cost/model-and-reasoning-choice))
- specific prompts ([Prompts and exploration](https://ai-sw-factory.mellicci.dev/cost/prompts-and-exploration))
- budgets on loops ([Automation at scale](https://ai-sw-factory.mellicci.dev/cost/automation-at-scale))

## Example

The `ticket-service` team (a TypeScript API, four developers, each with a coding agent) measured their spend and found most of it was cache reads from long-running sessions that stayed open across many issues. They test one change: a fresh session per issue. For two weeks they work as usual; for the next two they start a new session for each issue, with a short written handover when a task spans days.

Both windows cover similar work: bug fixes and small features from the same backlog. They chose the metric in advance, and the quality check is tests failing after merge. Token counts below are in thousands per finished task (illustrative).

| | Long sessions (weeks 1-2) | Fresh per issue (weeks 3-4) |
|---|---|---|
| Tasks finished | 22 | 24 |
| Fresh input | 9 | 8 |
| Cache write | 38 | 45 |
| Cache read | 1,900 | 620 |
| Output | 14 | 15 |
| Tests failing after merge | 3 | 2 |

Cache reads fall by about two thirds, cache writes rise a little because each session builds its own prefix, and output is flat. Quality did not slip. The team keeps the change, records it with the numbers, and picks the next variable from the same measurement: the model used for renames.

## A real experiment

[Token cost: is it cheaper to let an agent manage Kubernetes with CLI or MCP?](https://www.mellicci.dev/writing/cli-or-mcp-kubernetes-agent-token-cost) runs this method on real work. Four toolchains were each run twice on five Kubernetes operations, 40 runs in all, and every run was judged by an independent check of whether the task really succeeded. The metric was tokens per verified success, chosen in advance. It also shows two of the traps on this page in its own data:

- **Noise.** One change saved 38% in the first round and used 45% more in the second. A single before-and-after comparison would have proved whichever story you wanted.
- **Check the mechanism.** An installed output-compression tool turned out to remove about a tenth of a percent of the output. Installing an optimization is not evidence that it worked.

## What helps, what hurts

Helps:

- Writing the metric, the quality check and the decision rule before the first run, so the result cannot bend your reading of it.
- Comparing like with like, because [context growth](https://ai-sw-factory.mellicci.dev/cost/context-growth) makes the same task cost more late in a long session.
- Keeping the quality check cheap and automatic, such as a count of failing tests or reverted changes.

Hurts:

- Changing a model, a prompt style and a session habit in the same fortnight.
- Judging by one good day or one expensive session instead of a window.
- Counting only token totals, which reward doing less work, not finishing it.

## Trade-offs

- A clean experiment takes weeks. For a small team that is slow, but a wrong guess that sticks costs longer.
- Similar work is never identical. Expect noise, and treat a small difference as no difference.
- A quality check catches rework you can count. It misses subtler losses, so review a sample of the work as well.
- "Drop" is a valid result. A change that does not pay off still taught you something about your drivers.

## See it in your own sessions

One way to run this on your own data is [Agent Tokenwatch](https://github.com/mellicci/agent-tokenwatch), an open-source Node.js command-line tool (`npm install -g agent-tokenwatch`, command `tokenwatch`) for Claude Code, Codex CLI and GitHub Copilot CLI. It is local and metadata-only: it records token counts and structure, never prompts, code, transcripts, file paths or tool contents.

Its experiment designer skill defines an experiment and tracks it, and it judges experiments already running before it proposes new ones. Its overall retrospective gives you the baseline and comparison windows from your own telemetry, so you compare real periods instead of remembered ones. See also [the project page](https://mellicci.dev/work/agent-tokenwatch).

## Related

- [Measuring cost](https://ai-sw-factory.mellicci.dev/cost/measuring-cost) - find the biggest driver first.
- [Context growth](https://ai-sw-factory.mellicci.dev/cost/context-growth) - why a fresh session per task can pay off.
- [Cache lifetime](https://ai-sw-factory.mellicci.dev/cost/cache-lifetime) - what a break does to the cache.
- [Automation at scale](https://ai-sw-factory.mellicci.dev/cost/automation-at-scale) - budgets for loops.

## Key terms

- **Baseline window** - a period of normal work, measured before the change.
- **Comparison window** - a period of similar work with the one change in place.
- **Quality check** - a rework measure a cheaper run must not worsen.