# Output compaction and the stash

> How Tokenade shrinks command output: a per-command compactor first, then a fold that keeps the complete original in a local stash you can query by hash.

Source: https://tokenade.net/en/docs/compaction · Updated: 2026-10-07

Every output Tokenade handles goes through the same pipeline: secrets are redacted, a compactor that knows the command or format removes noise, and whatever is still too large is folded into a preview while the complete original is kept in a local stash. The agent sees the preview and a hash it can query. Nothing is lost, and nothing grows.

## The pipeline

1. **Capture.** `tokenade wrap` runs the command (or a hook receives the tool result). Capture caps are 16 MB for stdout and 4 MB for stderr.
2. **Redact.** Credentials and other secret-shaped values are replaced with `<redacted>` before anything is stored or shown.
3. **Compact.** A compactor removes what the model doesn't need (see below).
4. **Fold and stash.** If the result is still above the stash threshold, the complete output is written to the stash and the agent gets a folded preview with a recovery banner.
5. **Guard.** If the compacted version would be larger than the input, the raw bytes are returned instead.

## Choosing a compactor

Tokenade recognises the command you ran and uses a compactor that knows its output: `git`, `cargo`, `npm`, `pytest`, `kubectl`, `docker`, `terraform` and many more.

When the command is not one it knows, Tokenade recognises the format of the output instead: JSON, YAML, CSV, XML, Markdown, diffs, stack traces, tables, package installs, logs, schemas, notebooks and others. Source code you are likely to edit (a file under about 20,000 tokens inside a git repository) is passed through unchanged; other source files above 3 KB have their comments stripped, and `TOKENADE_NO_CODE_STRIP=1` turns that off.

You can add your own rules in TOML; see [custom compactors](https://tokenade.net/en/docs/custom-compactors).

## The fold and the banner

When output is still above the threshold after compaction, the complete text goes to the stash and the agent receives a short preview. The banner says how much was folded and how to get it back:

```text
[tokenade:seq 85479 tokens folded away; the stash holds the COMPLETE output — this blob is ~85690 tokens — too large to dump whole. recover with `tokenade expand-ref f6351f5ead9a` · `--prompt "q1, q2"` to ask (RAG, several at once) · `--grep <pat>` exact lines. Any other hash above covers only what was shown.]
```

The hash is 12 hex characters. To get details back, see [get folded output back](https://tokenade.net/en/docs/expand-ref-disclose).

## Thresholds and limits

| Setting | Default | Override |
| --- | --- | --- |
| Stash threshold, command output | 5,000 tokens | `TOKENADE_DISCLOSE_THRESHOLD` |
| Stash threshold, file reads | 25,000 tokens | `TOKENADE_READ_DISCLOSE_THRESHOLD` |
| Stash directory | `~/.cache/tokenade/stash/` on Linux, the platform cache directory elsewhere | `TOKENADE_STASH_DIR` |
| Stash size cap | 2 GiB, oldest evicted first | `TOKENADE_STASH_MAX_DIR_BYTES` |
| Single stash file | 50 MB | |
| Stash lifetime | 7 days since last use | |

## How much to keep: read mode

`TOKENADE_READ_MODE` scales how much of a long listing or read stays inline:

| Mode | Effect |
| --- | --- |
| `aggressive` | Keeps less inline than the default |
| `task` | Default |
| `reference` | Keeps more inline, for configs, schemas and full diffs |
| `entropy` | Keeps the most informative lines, in their original order |

A fold that honours the mode names it in its banner, for example `[tokenade:git-ls-files·aggressive]`. Not every compactor honours the mode. `tokenade read-mode` shows the active mode.

## Deduplication

Tokenade remembers content it has already delivered. When the same file or the same output comes back unchanged, the agent gets a short `§ref:HASH§` pointer instead of the full text a second time. A repeated identical poll, such as checking a build's status in a loop, is answered with a short "unchanged" note.

## Secret redaction

Tokenade masks known secrets in every output it handles: `Authorization` headers, `api_key=` and `*_TOKEN=`, `*_KEY=`, `*_SECRET=` assignments, AWS, GitHub, Stripe and other provider keys, private keys, card numbers and high-entropy strings. Redaction runs before deduplication and stashing, so the stash never holds what was redacted. Redaction continues even when compaction is turned off.

If something slipped through before an upgrade, [`tokenade scrub-cache`](https://tokenade.net/en/docs/maintenance) re-applies the patterns to what is already on disk.

## What is never done

- Output is never made larger: the anti-inflation guard returns raw bytes with a `passthrough_inflated` label.
- Exit codes are never changed.
- Files on disk are never modified: when `tokenade read` strips comments from a source file, only the delivered copy changes.
- Negligible savings are not booked (see [how savings are measured](https://tokenade.net/en/docs/savings-measurement)).

## Related

- [Get folded output back](https://tokenade.net/en/docs/expand-ref-disclose)
- [How hooks intercept tool calls](https://tokenade.net/en/docs/hooks)
- [wrap, raw and filter](https://tokenade.net/en/docs/wrap-raw-filter)
- [Custom compactors and filters](https://tokenade.net/en/docs/custom-compactors)
- Article: [Output filtering for command logs](https://tokenade.net/en/articles/output-filtering-command-logs)
