How we measure token savings
Every saving this site quotes comes from one public, reproducible benchmark. Here is exactly what it measures, what it leaves out, and how to check it yourself.
What is measured
THOL (Token-Harness Optimizer Leaderboard) measures one outcome: did the coding task succeed, and what did the whole session cost in US dollars — with each optimizer installed, and with plain Claude Code as the control. Compression ratios on a fixed file are not measured; they are an input, the bill is the outcome.
Protocol
Each run is a fully isolated headless Claude Code session: throwaway HOME, throwaway workspace, strict MCP config, one pinned model (Claude Sonnet) and one pinned Claude Code version. Tasks use deterministic fixtures, and programmatic verifiers score the result against a ground truth that is never present in the workspace. Each tool runs 10 times per task.
What is left out
Tasks that plain Claude Code finishes in under about five turns are dropped: every tool lands on the same near-zero cost there, so only fixed overhead would show. The headline table is restricted to long sessions — tasks where plain Claude Code uses more than 200,000 tokens — because that is where real agent work, and real bills, are.
How results are aggregated
Cost is aggregated as the geometric mean of per-task cost ratios against the control, so no single task dominates; a cost-weighted total agrees with it to within a point. Every successful run is used as is. Runs vary from one to the next, so small gaps between neighbouring tools are not a ranking — the control's own run-to-run spread is published as the noise floor.
The published result
On long sessions, Tokenade cuts the cost of a session by 39% compared with plain Claude Code. That is the only figure this site quotes from the benchmark.
What you will actually see
The 39% is measured on long sessions — tasks where the agent uses more than 200,000 tokens — because that is where the replayed context, the command output and the file reads pile up. On short tasks there is little to trim: every tool in the benchmark lands close to plain Claude Code there, so expect a small effect on quick one-off questions and the largest one on long, multi-step work. It is a session-cost figure, not a compression ratio on a single file, so it is not comparable to per-file percentages quoted elsewhere.
Impartiality
The benchmark is maintained by Tokenade's author, Paul Irolla. The conflict is handled by construction: the harness cannot tell tools apart, install steps follow each tool's own documentation, verifiers are frozen before any run and check task outcomes rather than tool behaviour, the control is a first-class entry, and all raw data — including runs unfavourable to Tokenade — is published unedited. It is reproducible, not independent.
Check it yourself
The leaderboard publishes the aggregates (results.json) and every run (results.sqlite), with the full method and exact reproduction steps in its repository README.
Open the leaderboard and its raw data →
Cite this benchmark
Irolla, P. (2026). Token-Harness Optimizer Leaderboard (THOL): an end-to-end benchmark of coding-agent token optimizers.
@misc{irolla2026thol,
author = {Irolla, Paul},
title = {Token-Harness Optimizer Leaderboard (THOL): an end-to-end benchmark of coding-agent token optimizers},
year = {2026},
howpublished = {\url{https://pi-infected.github.io/token-harness-optimizer-leaderboard/}},
note = {Methodology: https://tokenade.net/en/methodology}
}