Why these 5 numbers matter

Of twelve token optimizers measured end to end on real coding sessions, exactly one made sessions cheaper by a statistically meaningful margin — nine made no measurable difference, and two made sessions more expensive than using no tool at all. That is the single most useful fact in this category, and it contradicts almost every product page in it. The gap comes from what gets measured. Most tools in this space report compression ratios: how much smaller a file, a log or a message becomes after processing. Those numbers are real and often impressive. They also don't tell you what you want to know, because an agent that compresses each file but then reads forty of them, or that adds a round-trip to decide what to compress, can easily end up costing more. Compression is an input. Session cost is the outcome. Only the outcome shows up on your invoice. An upfront disclosure, because it changes how you should read this: the benchmark below is maintained by me, and my own tool ranks first in it. It's public, reproducible, and I've kept the unflattering results in — but it is not an independent third party, and you shouldn't treat it as one. The method is published so you can disagree with it specifically rather than in general.

1. How many optimizers actually make sessions cheaper?

One of twelve (THOL, 2026). Nine of the twelve landed statistically indistinguishable from the control — meaning that across 170 runs each, you could not tell from the bill whether the tool was installed. Two finished measurably worse than no tool at all. This is the number that should reframe how you shop in this category. The default outcome of installing a token optimizer is nothing measurable happens. Not harm, usually — just no effect on the thing you installed it for. Any tool asking you to trust a compression percentage is asking you to trust an input metric while the outcome metric sits unmeasured. The nine no-effect tools are not badly built. Several are impressive engineering. They simply optimize one layer — command output, or message compression, or retrieval — in a session whose cost is spread across all of them plus the transcript replay that dominates everything.

2. How much cheaper is the top-ranked tool?

23.2% cheaper than running no tool at all, measured across 17 tasks and 170 runs (THOL, 2026). That's the aggregate figure over the full task set, including the short tasks where there's little to save. Two things worth saying about that number. First, it's smaller than the headline percentages this category advertises, and deliberately so — it's an end-to-end session cost, not a best-case compression ratio on a favourable file. Second, it's the only figure in the twelve that survived statistical testing against the control. If you're comparing it against a competitor's "90% reduction" claim, you're comparing two different quantities. Ask what the other number is measuring before concluding it's four times better.

3. Where does the saving actually concentrate?

On long sessions: 38.9% cost reduction above 200,000 tokens (THOL, 2026), measured over 7 tasks and 70 runs. This is the most actionable number on the page. Savings are not evenly distributed — they scale with session length, because the mechanism being attacked is the transcript and context that get replayed on every turn. A short session has little accumulated context to trim. A long one has a great deal. Which maps neatly onto where the pain is. Nobody worries about the cost of a five-minute session. The sessions that blow through a plan's usage limit or produce a surprising invoice are the long ones — and those are where the measured effect roughly doubles.

4. Can an optimizer make things worse?

Yes — the worst-performing tool made sessions 55.7% more expensive than using no tool at all (THOL, 2026), and a second tool finished 23.4% more expensive. This is not a hypothetical risk, and the tool in question is one of the most popular in the category by GitHub stars. Its per-message compression works as advertised. The problem is everything around it: the processing overhead, the extra round-trips, and compression decisions that don't pay for themselves once the agent's real behaviour is in the loop. The general lesson is uncomfortable but useful: popularity and measured effect are uncorrelated here. A joke tool that makes the model answer in telegraphic "caveman" style outranked several serious engineering efforts. Stars measure enthusiasm, not outcomes.

5. How much measurement is behind each result?

170 runs per tool, across 17 deterministic coding tasks (THOL, 2026), with results computed from successful runs only. The reason for that volume is variance. Agent sessions are noisy — the same task run twice can differ substantially in cost depending on which files the agent decides to read. A single before/after comparison, which is how most tool benchmarks in this space are conducted, cannot distinguish a real effect from that noise. Most of the "no measurable difference" verdicts above would have looked like clear wins or losses in a one-run test. That's also why nine tools land in the no-effect bucket rather than being ranked confidently against each other. With enough runs, small differences stop looking meaningful.

Source notes

Every figure on this page comes from the Token-Harness Optimizer Leaderboard (THOL), a public benchmark I maintain: pi-infected.github.io/token-harness-optimizer-leaderboard. The raw results, including the per-tool and per-task data behind each number, are published there. For operators who want the provenance in full: the current campaign runs on Claude Code 2.1.206 with Claude Sonnet, in fully isolated headless sessions using deterministic fixtures and programmatic verification against ground truth. Cost ratios are geometric means versus a no-tool control, with 95% confidence intervals from bootstrap resampling; "no measurable difference" means the interval spans 1.0. The long-session figure is restricted to tasks exceeding 200,000 tokens. Tokenade's own entry was measured on version 0.8.13 under identical conditions. This is internal data, not a third-party study — see the disclosure at the top. It is reproducible rather than independent, which are different guarantees. Where a tool is absent from the current campaign, this site says so instead of substituting the vendor's own published claim.

See also:

Best Claude Code token optimizers — the field ranked on these criteria
ccusage alternatives — measuring spend versus reducing it
Headroom alternatives — the tool behind number 4
Claude Code pricing — what the bill is made of in the first place

Token Optimizer Benchmark: 5 Key Numbers

Key figures