Custom compactors and filters

When Tokenade only compacts a command generically (an internal CLI, a niche build tool), you can teach it what matters in that output with a declarative TOML filter. No code: you describe which command it applies to and what to drop or keep, and Tokenade validates the file before using it.

Install a filter in one command

The file is validated first: every regex is compiled and every [[tests]] case is run. If anything fails, nothing is installed. On success the filter is written to ~/.tokenade/filters/<name>.toml and applies to matching commands from the next run on. Its savings appear in tokenade gain and the dashboard under its name.

Other forms:

--from scaffolds a starter filter from a real captured output and prints the TOML. The draft carries draft = true, and an install is refused until you have reviewed it and removed that line.

A complete example

Fields

Two templates expand inside match_command: {{runner}} becomes (npx|bunx|pnpm dlx|yarn dlx) and {{py-runner}} becomes (poetry run|uv run|python -m). An unknown {{token}} is a hard error.

Unknown fields are rejected, so a typo such as expect instead of expect_contains fails validation instead of silently skipping your test.

Operations

They run in this order; set at least one.

1. match_output: list of { pattern, summary }. If pattern matches anywhere in the output, the entire output is replaced by the one-line summary (a clean build becomes build: ok).
2. strip_ansi: true removes ANSI colour sequences.
3. replace: list of { from, to } regex substitutions; to may use \1.
4. strip_lines_matching: list of regexes; every matching line is dropped.
5. on_empty: if the steps above left nothing, emit this single line instead.
6. head_lines / tail_lines: keep the first N and last M lines; the middle becomes a … [K lines omitted] marker, so the final outcome line survives.
7. max_lines: hard cap on total lines kept, with a … [K more] tag.

For output that goes through phases (compiling, then running tests, then failures), the format also accepts [[phases]] entries with an optional keep regex (lines kept while in that phase; all lines if absent) and an optional until regex (moves to the next phase). The last phase needs no until.

Self-tests

Each [[tests]] block has an input and either expect_contains or expect_equals. They gate the install: a filter whose own test fails is not written. Add at least one per real output you care about.

Project filters and trust

A repository can ship filters in .tokenade/filters/*.toml. Tokenade scans two places, in this order:

1. .tokenade/filters/ in the current directory (project layer)
2. ~/.tokenade/filters/ (your own filters, always trusted)

When both define the same name, the project file wins and the global one is shadowed.

A project filter is code you did not write: it could rewrite a failing test run into "all tests passed". So project filters do nothing until you trust them:

Trust is keyed on the file content and stored in ~/.tokenade/trusted-filters.json. Editing a trusted file voids its trust; run tokenade trust-filters again after reviewing the change.

A filter file larger than 256 KiB is refused without being read.

Check what is loaded

tokenade healthcheck reports two rows for filters:

• project-filters: project-local compactors waiting for approval (info, not a failure).
• filter-syntax: compactor files that do not parse and therefore never fire (fail). The detail line gives the exact reason. Fix the file or delete it.

To try a filter against a saved output without running the command, pipe it through tokenade filter <cmd…>, which filters stdin as if produced by that command. See wrap, raw and filter.

For background on why command-specific filtering pays off, see Output filtering for command logs.

Related

• Output compaction and the stash
• wrap, raw and filter
• Files and paths
• Troubleshooting