# Custom compactors and filters

> Write a TOML filter that tightens the output of your own commands, install it with tokenade add-compactor, and share project filters safely.

Source: https://tokenade.net/en/docs/custom-compactors · Updated: 2026-10-07

When Tokenade only compacts a command generically (an internal CLI, a niche build tool), you can teach it what matters in that output with a declarative TOML filter. No code: you describe which command it applies to and what to drop or keep, and Tokenade validates the file before using it.

## Install a filter in one command

```bash
tokenade add-compactor --file mytool.toml
```

The file is validated first: every regex is compiled and every `[[tests]]` case is run. If anything fails, nothing is installed. On success the filter is written to `~/.tokenade/filters/<name>.toml` and applies to matching commands from the next run on. Its savings appear in `tokenade gain` and the dashboard under its `name`.

Other forms:

```bash
tokenade add-compactor < mytool.toml        # read the TOML from stdin
tokenade add-compactor --list               # list installed user compactors
tokenade add-compactor --from sample.txt --command "mytool status"
```

`--from` scaffolds a starter filter from a real captured output and prints the TOML. The draft carries `draft = true`, and an install is refused until you have reviewed it and removed that line.

## A complete example

```toml
name = "mytool-status"
match_command = "^mytool\\s+status"
strip_ansi = true
strip_lines_matching = ["^DEBUG", "^\\s*$"]
match_output = [{ pattern = "All systems go", summary = "mytool: ok" }]
head_lines = 40
tail_lines = 5
on_empty = "mytool: ok"

[[tests]]
input = "DEBUG boot\nAll systems go\n"
expect_contains = "mytool: ok"
```

## Fields

| Field | Required | Meaning |
|---|---|---|
| `name` | yes | Report label and file name. |
| `match_command` | yes | Regex tested against the whole command line. When several filters match, the one covering the **longest** span wins. |
| `priority` | no (default `0`) | Tie-break between equally specific matches only; higher wins. Never overrides a longer match. |
| `draft` | no | Set by `--from`. Remove it to install. |

Two templates expand inside `match_command`: `{{runner}}` becomes `(npx|bunx|pnpm dlx|yarn dlx)` and `{{py-runner}}` becomes `(poetry run|uv run|python -m)`. An unknown `{{token}}` is a hard error.

Unknown fields are rejected, so a typo such as `expect` instead of `expect_contains` fails validation instead of silently skipping your test.

## Operations

They run in this order; set at least one.

1. `match_output`: list of `{ pattern, summary }`. If `pattern` matches anywhere in the output, the **entire** output is replaced by the one-line `summary` (a clean build becomes `build: ok`).
2. `strip_ansi`: `true` removes ANSI colour sequences.
3. `replace`: list of `{ from, to }` regex substitutions; `to` may use `\1`.
4. `strip_lines_matching`: list of regexes; every matching line is dropped.
5. `on_empty`: if the steps above left nothing, emit this single line instead.
6. `head_lines` / `tail_lines`: keep the first N and last M lines; the middle becomes a `… [K lines omitted]` marker, so the final outcome line survives.
7. `max_lines`: hard cap on total lines kept, with a `… [K more]` tag.

For output that goes through phases (compiling, then running tests, then failures), the format also accepts `[[phases]]` entries with an optional `keep` regex (lines kept while in that phase; all lines if absent) and an optional `until` regex (moves to the next phase). The last phase needs no `until`.

## Self-tests

Each `[[tests]]` block has an `input` and either `expect_contains` or `expect_equals`. They gate the install: a filter whose own test fails is not written. Add at least one per real output you care about.

## Project filters and trust

A repository can ship filters in `.tokenade/filters/*.toml`. Tokenade scans two places, in this order:

1. `.tokenade/filters/` in the current directory (project layer)
2. `~/.tokenade/filters/` (your own filters, always trusted)

When both define the same `name`, the project file wins and the global one is shadowed.

A project filter is code you did not write: it could rewrite a failing test run into "all tests passed". So project filters do **nothing** until you trust them:

```bash
tokenade trust-filters --list   # show untrusted project filters, trust nothing
tokenade trust-filters          # trust the project filters in this directory
```

Trust is keyed on the file **content** and stored in `~/.tokenade/trusted-filters.json`. Editing a trusted file voids its trust; run `tokenade trust-filters` again after reviewing the change.

A filter file larger than 256 KiB is refused without being read.

## Check what is loaded

`tokenade healthcheck` reports two rows for filters:

- `project-filters`: project-local compactors waiting for approval (info, not a failure).
- `filter-syntax`: compactor files that do not parse and therefore never fire (fail). The detail line gives the exact reason. Fix the file or delete it.

To try a filter against a saved output without running the command, pipe it through `tokenade filter <cmd…>`, which filters stdin as if produced by that command. See [wrap, raw and filter](https://tokenade.net/en/docs/wrap-raw-filter).

For background on why command-specific filtering pays off, see [Output filtering for command logs](https://tokenade.net/en/articles/output-filtering-command-logs).

## Related

- [Output compaction and the stash](https://tokenade.net/en/docs/compaction)
- [wrap, raw and filter](https://tokenade.net/en/docs/wrap-raw-filter)
- [Files and paths](https://tokenade.net/en/docs/files-and-paths)
- [Troubleshooting](https://tokenade.net/en/docs/troubleshooting)
