Output compaction and the stash

Every output Tokenade handles goes through the same pipeline: secrets are redacted, a compactor that knows the command or format removes noise, and whatever is still too large is folded into a preview while the complete original is kept in a local stash. The agent sees the preview and a hash it can query. Nothing is lost, and nothing grows.

The pipeline

Capture. tokenade wrap runs the command (or a hook receives the tool result). Capture caps are 16 MB for stdout and 4 MB for stderr.
Redact. Credentials and other secret-shaped values are replaced with <redacted> before anything is stored or shown.
Compact. A compactor removes what the model doesn't need (see below).
Fold and stash. If the result is still above the stash threshold, the complete output is written to the stash and the agent gets a folded preview with a recovery banner.
Guard. If the compacted version would be larger than the input, the raw bytes are returned instead.

Choosing a compactor

Tokenade recognises the command you ran and uses a compactor that knows its output: git, cargo, npm, pytest, kubectl, docker, terraform and many more.

When the command is not one it knows, Tokenade recognises the format of the output instead: JSON, YAML, CSV, XML, Markdown, diffs, stack traces, tables, package installs, logs, schemas, notebooks and others. Source code is passed through unchanged.

You can add your own rules in TOML; see custom compactors.

The fold and the banner

When output is still above the threshold after compaction, the complete text goes to the stash and the agent receives a short preview. The banner says how much was folded and how to get it back:

[tokenade:seq 85479 tokens folded away; the stash holds the COMPLETE output — this blob is ~85690 tokens — too large to dump whole. recover with `tokenade expand-ref f6351f5ead9a` · `--prompt "q1, q2"` to ask (RAG, several at once) · `--grep <pat>` exact lines. Any other hash above covers only what was shown.]

The hash is 12 hex characters. To get details back, see get folded output back.

Thresholds and limits

Setting	Default	Override
Stash threshold, command output	5,000 tokens	`TOKENADE_DISCLOSE_THRESHOLD`
Stash threshold, file reads	25,000 tokens	`TOKENADE_READ_DISCLOSE_THRESHOLD`
Stash directory	`~/.cache/tokenade/stash/` on Linux, the platform cache directory elsewhere	`TOKENADE_STASH_DIR`
Stash size cap	2 GiB, oldest evicted first	`TOKENADE_STASH_MAX_DIR_BYTES`
Single stash file	50 MB
Stash lifetime	7 days since last use

How much to keep: read mode

TOKENADE_READ_MODE scales how much of a long listing or read stays inline:

Mode	Effect
`aggressive`	Keeps less inline than the default
`task`	Default
`reference`	Keeps more inline, for configs, schemas and full diffs
`entropy`	Keeps the most informative lines, in their original order

A fold that honours the mode names it in its banner, for example [tokenade:git-ls-files·aggressive]. Not every compactor honours the mode. tokenade read-mode shows the active mode.

Deduplication

Tokenade remembers content it has already delivered. When the same file or the same output comes back unchanged, the agent gets a short §ref:HASH§ pointer instead of the full text a second time. A repeated identical poll, such as checking a build's status in a loop, is answered with a short "unchanged" note.

Secret redaction

Tokenade masks known secrets in every output it handles: Authorization headers, api_key= and *_TOKEN=, *_KEY=, *_SECRET= assignments, AWS, GitHub, Stripe and other provider keys, private keys, card numbers and high-entropy strings. Redaction runs before deduplication and stashing, so the stash never holds what was redacted. Redaction continues even when compaction is turned off or your license is inactive.

If something slipped through before an upgrade, tokenade scrub-cache re-applies the patterns to what is already on disk.

What is never done

Output is never made larger: the anti-inflation guard returns raw bytes with a passthrough_inflated label.
Exit codes are never changed.
tokenade read on a source file returns it unchanged.
Negligible savings are not booked (see how savings are measured).