THOL Explained: Do Token Optimizers Really Save?

Every tool that promises to cut an AI agent's bill quotes an impressive number: "this file shrinks by 80%", "90% fewer tokens". Those numbers are real, and they almost never tell you what happens to your bill.

THOL is a public benchmark built to answer that one question. It gives an AI agent the same jobs over and over, with no tool and then with each token optimizer installed, and compares what each whole session cost. This page explains how it works, shows what it found, and tells you how to read it. The raw data and the full protocol are public; links are at the end.

THOL in one minute

THOL stands for Token-Harness Optimizer Leaderboard. In one sentence: same jobs, same agent, same model, with and without each tool, many times over, and the bill of the whole session at the end.

The result Tokenade quotes from it: on long sessions, Tokenade makes a session 39% cheaper than the same agent with no tool. That is the only Tokenade figure this site uses.

Where an AI agent's bill really comes from

To understand why THOL measures whole sessions, look at what the agent sends to the model. At every step it sends the entire conversation so far: your request, every file it has read, every command result. Then it gets a short answer and does it again.

Nine tokens out of ten are history being re-sent. That is why a 600-line build log is expensive: the agent does not pay for it once, it pays for it at every following step. And it is why shrinking one message tells you little: what matters is what stays in the conversation, and how many steps the agent needs.

Why "this file shrinks by 80%" is the wrong number

A compression rate describes one message, usually chosen to look good. A session is different. If a tool shrinks output too aggressively, the agent no longer finds what it needs: it reads the file again, retries the command, takes extra steps. Each extra step re-sends the whole conversation. The "saving" can end up costing more than it saved.

A tool also adds its own weight: instructions loaded into the agent, extra tool descriptions sent at every step. A per-file percentage ignores all of that. The bill at the end of the session does not.

How one THOL run works

A failed run never counts as a saving: a tool that makes sessions cheap by making them fail does not win.

The 17 jobs

The tasks mix everyday coding work (fixing bugs, adding features, refactoring, migrating, understanding an unfamiliar codebase) with jobs that are not code: finding one line in a large log, writing a report from a PDF, auditing a website's SEO, researching open-source projects on the web. Seven of them are "long": the agent on its own uses more than 200,000 tokens to finish.

Why single out long sessions? On a quick job there is almost nothing to trim, so every tool lands close to the agent on its own and only each tool's fixed cost shows. Long sessions are where real work, and real bills, are.

Same job, different bill

Here is a detail most benchmarks skip: even with no tool at all, running the same job twice does not cost the same. The agent takes a different path, reads different files, takes more or fewer steps.

This is why THOL runs everything ten times and publishes this spread: a small gap between two tools can be pure chance. Only a difference larger than this natural noise means something.

The result on long sessions

For each task, THOL divides the cost with the tool by the cost with no tool, then averages those ratios across tasks in a way that stops one unusual task from dominating (a geometric mean). The chart shows the result as a change in the cost of a session: below zero is cheaper, above zero is more expensive.

How to read it:

• One tool is measurably cheaper. On long sessions, Tokenade comes out 39% cheaper than the agent on its own, a gap well beyond the noise.
• Most tools change nothing you could measure. Their bars lean one way or the other, but the gap stays inside the natural run-to-run spread: with these 70 runs, you could not tell from the bill whether they were installed.
• One tool makes sessions more expensive. Compressing every message is not free if the agent then has to work harder.

What you will actually see

The 39% is a session-cost figure measured on long sessions. On long, multi-step work, expect the biggest effect. On a quick one-off question, expect a small one: there is little to remove.

It is not a compression rate on a single file, so it cannot be compared with the per-file percentages quoted elsewhere: those describe one message, this describes the bill.

How Tokenade counts your own savings

THOL measures Tokenade in a lab. On your computer, Tokenade keeps its own ledger: each time it changes what reaches the model, it writes down the size before, the size after and the difference. Tokens are counted on your machine; no text is sent anywhere to be counted.

Three corrections keep that figure honest:

1. Only real savings are booked. A change that saves nothing, or adds text, counts as zero.
2. A saving is capped at what your agent would really have shown the model. Agents already cut very large results on their own; counting the full raw size would overstate the benefit.
3. What Tokenade adds is subtracted. The short instructions it gives your agent are charged against the savings.

What cannot be measured exactly, such as the whole file the agent did not have to read, goes to a separate estimate shown next to the measured total and never added to it. The full method is in how savings are measured.

Who runs THOL, and why you can still trust it

THOL is maintained by Tokenade's author, Paul Irolla, so it is not independent, and this page does not pretend otherwise. The conflict of interest is handled by how the test is built:

• every tool goes through the same test bench, installed exactly as its own documentation says;
• the answer checkers are fixed before any run and judge the job, not the tool;
• the agent with no tool is tested like any other entry;
• every run, including the ones that do not favour Tokenade, is published unedited.

It is reproducible rather than independent: anyone can rerun it and compare.

Check it yourself

• The live leaderboard: THOL leaderboard
• The aggregated data behind every chart on this page: results.json
• The full protocol, harness and reproduction steps: THOL repository on GitHub (the benchmark is open source, MIT)

Cite this benchmark

Irolla, P. (2026). Token-Harness Optimizer Leaderboard (THOL): an end-to-end benchmark of coding-agent token optimizers.

FAQ

Why only long sessions?

On short jobs there is almost nothing to trim, so every tool lands close to the agent on its own and the comparison only shows each tool's fixed cost. Long sessions, over 200,000 tokens, are where the bill is.

Why does a tool with a big compression rate not win?

Because the bill comes from the whole session. If compressed output forces the agent to read again or take extra steps, each step re-sends the conversation and the saving disappears.

Does a failed run count?

No. Only successful runs are compared, so a tool cannot look cheap by making the agent give up.

Is Tokenade open source like THOL?

No. THOL, the benchmark, is open source (MIT). Tokenade, the product, is not.