Docs / Commands

wrap, raw and filter

Updated

TL;DR — `tokenade wrap '<cmd>'` runs a command, compacts stdout and stderr with the right per-command compactor and keeps the exit code. `tokenade raw` bypasses compaction. `cmd | tokenade filter <cmd>` compacts output you already have.

tokenade wrap runs a shell command and returns a compacted version of its output, with the exit code unchanged. It is what your agent's hook calls for every shell command, and you can call it by hand for any agent. raw is the escape hatch that runs a command without compaction; filter compacts output you captured earlier.

wrap

tokenade wrap [--shell <posix|powershell|cmd>] [--split-streams] <shell-cmd>

Alias: exec-compact. Quote the command as one argument: tokenade wrap 'git log --stat -20'.

FlagEffect
--shell posixDefault. sh -c (Git Bash on Windows)
--shell powershellpowershell.exe -Command (or pwsh)
--shell cmdcmd.exe /C
--split-streamsCompact stdout and stderr separately and return each to its own stream, so redirections behave as without the wrapper. Chronological interleaving is lost

Without --shell, the shell is detected from the environment and falls back to posix. Pass it when you wrap a PowerShell or cmd command by hand.

What wrap does, in order:

  1. Runs the command and captures stdout and stderr together (caps: 16 MB stdout, 4 MB stderr).
  2. Redacts secrets.
  3. Compacts the output with a compactor that knows that command, or that recognises its format (JSON, logs, diffs, tables and the like).
  4. If the result is still large, stashes the complete output on disk and returns a preview with a hash you can query. See output compaction and the stash.
  5. Exits with the inner command's exit code (128 + signal when it was killed by a signal).
$ tokenade wrap 'seq 1 20000'
[tokenade:seq] 20000 line(s)
1
2
3
…
19999
20000
[tokenade:seq 85479 tokens folded away; the stash holds the COMPLETE output — this blob is ~85690 tokens — too large to dump whole. recover with `tokenade expand-ref f6351f5ead9a` · `--prompt "q1, q2"` to ask (RAG, several at once) · `--grep <pat>` exact lines. …]
$ tokenade wrap 'exit 7'; echo $?
7

Gotcha: stderr is merged

wrap merges stderr into stdout before compacting. tokenade wrap 'cmd' 2>err.log therefore writes an empty err.log. Redirect inside the wrapped command (tokenade wrap 'cmd 2>err.log') or use --split-streams.

raw

tokenade raw <command> [args...]
tokenade raw '<command line>'

Aliases: bypass, noproxy. Runs the command with no compaction and sets TOKENADE_HOOK_DISABLED=1 so the agent hooks let it through too. Output streams are inherited, the exit code matches the inner command, and nothing is written to the savings ledger.

You usually don't need it. If you already ran a command through Tokenade and need the exact bytes, tokenade expand-ref <hash> returns them from the stash instantly, without re-running anything. raw is for what the stash can't give you: re-executing without compaction, interactive runs, or debugging a compactor.

For commands whose output folds well (grep, rg, ag, find, ls, wc, head, tail, awk, sed, cat), raw refuses and exits 2, because re-running would spend the original call's tokens a second time:

$ tokenade raw ls
[tokenade] refusing bypass for `ls` — re-running it would spend the original call's tokens again.
Most of what you want is already recoverable: `tokenade expand-ref <hash>` …

Override with TOKENADE_FORCE_BYPASS=1 tokenade raw … when you really need the original stream.

If you bypass often (raw, or any hook skipped through TOKENADE_HOOK_DISABLED=1), each further bypass prints a one-line reminder on stderr pointing at expand-ref. The reminder cannot be silenced.

filter

<cmd> 2>&1 | tokenade filter <argv...>

Compacts stdin as if it had been produced by the command you name. Useful when the output is already captured:

cargo build 2>&1 | tokenade filter cargo build
cat /tmp/log.txt | tokenade filter git log
FlagEffect
--deliverAlso run the second stage: stash what is still too big and point back at what was dropped, exactly as wrap does

Without --deliver, filter applies only the per-command compactor and does not stash. On output that compactor leaves alone it returns the input unchanged where wrap would still fold it. tokenade read - is different again: it detects the format without a command hint.

Auto-wrapping in your shell

tokenade shell-init prints a shell fragment that routes noisy commands through tokenade wrap when stdin is not a terminal, which means for agents, not for you typing at a prompt. It covers the commands Tokenade knows how to compact, limited to what is actually on your PATH.

eval "$(tokenade shell-init)" # sh, bash, zsh
tokenade shell-init --shell fish | source # fish
Invoke-Expression (& tokenade shell-init --shell powershell | Out-String)

Flags: --shell sh|bash|zsh|fish|powershell|nushell|csh|cmd, --all (skip the PATH filter). On csh/tcsh and cmd.exe, aliases go through tokenade shellwrap, which passes through at a terminal and wraps otherwise. tokenade install sets this up for you; you only need it by hand for an agent Tokenade can't hook.