Docs / Commands
web and search
Updated
tokenade web fetches a page and returns it as compact markdown, without scripts, menus or markup. tokenade search runs one query across several independent search engines and returns merged, deduplicated results. Both take several targets at once and run them concurrently.
web
| Form | Example |
|---|---|
| One URL | tokenade web https://example.com |
| Several URLs, fetched concurrently | tokenade web https://a.com https://b.com |
| Piped HTML | curl -s https://example.com | tokenade web - |
| Saved file | tokenade web saved.html |
| Flag | Effect |
|---|---|
--prompt "q1, q2" | Return only the passages that answer each question, under its own heading. Also --question, -q |
--max-tokens N | Bounds the input that gets parsed, not the reply. The page is still converted whole, so the answer is the same size with or without it. To bound the reply, ask a question with --prompt |
# Example Domain
This domain is for use in documentation examples without needing permission. …
--prompt "can match guards bind variables, what is exhaustiveness checking"
### Q1 · can match guards bind variables
[tokenade: relative links below resolve under https://doc.rust-lang.org]
…
match a_number {
Some(x) if x <= 5 => println!("0 to 5 num = {x}"),
Some(x @ 6..=10) => println!("6 to 10 num = {x}"),
…
Links in the page are kept usable: relative links are rewritten from the site root, links into the same page become anchors, and links to other sites are written in full.
search
tokenade search "<q1>" "<q2>" SEVERAL queries, run concurrently
tokenade search <file>|- compact a SAVED results page
tokenade search --diagnose probe every engine
| Flag | Effect |
|---|---|
--answer | Fetch the top results and return the passages, not links |
--json | Machine-readable output: engines, query, refused, results[] with title, url, snippet |
--diagnose | Probe every engine and report which ones answer |
--html, --serp | Treat the argument as a saved results page even if it reads like a query |
Each quoted argument is a separate query. Results are merged, deduplicated and ranked by how many engines agree. The output is title, URL and snippet only, which costs a fraction of reading a results page.
[tokenade:search via yahoo] "rust anyhow vs thiserror":
1. Rust Error Handling: thiserror, anyhow, and When to Use Each
https://momori.dev/posts/rust-error-handling-thiserror-anyhow/
Feb 6, 2024 · In this blog post, we'll explore strategies for …
2. Rust Error Handling Compared: anyhow vs thiserror vs snafu
https://dev.to/leapcell/rust-error-handling-compared-anyhow-vs-thiserror-vs-snafu-2003
…
Engines used: Bing, Brave, DuckDuckGo, Ecosia, Marginalia, Mojeek and Yahoo. --diagnose shows which are healthy from your network:
probing 7 engines with "python asyncio vs threading"
engine bytes hits ms verdict
ecosia 0 0 1705 unreachable
bing 132143 10 223 ok
brave 365044 20 816 ok
duckduckgo 34856 10 696 ok
mojeek 5516 0 132 thin
yahoo 165192 7 842 ok
marginalia 1029 0 188 thin
4/7 engines healthy
Follow a result with tokenade web <url> to read the page itself.
Inside your agent
On Claude Code, Tokenade's hooks also handle the agent's own web tools:
- WebSearch: the search runs locally across the same engines and replaces the native result. If the engines fail or are too slow, the native tool runs instead.
- WebFetch: intercepted only when it pays (PDFs and other binaries, very large pages); everything else goes through untouched.
See how hooks intercept tool calls for the full list.
Turning web features off
| Variable | Effect |
|---|---|
TOKENADE_NO_WEBSEARCH=1 | Tokenade no longer replaces or compacts the agent's web search, tokenade search refuses to run (exit 3), and the rules Tokenade writes stop mentioning it |
TOKENADE_NO_WEBFETCH=1 | Tokenade no longer intercepts the agent's page fetches |
Use these when web queries must leave only through your AI vendor. For Claude Code, set them in the env block of ~/.claude/settings.json, and run tokenade install once with the variable set in your shell so the written rules are updated.
Exit codes and gotchas
0success,1the page could not be fetched,2usage error (unknown flag, a path-shaped argument that does not exist),3search disabled byTOKENADE_NO_WEBSEARCH.- A mistyped flag is refused, not sent as a query.
tokenade search /path/that/does/not/exist.htmlis refused instead of being searched, so a local path never leaves your machine.- Quote each multi-word query:
tokenade search "q1" "q2"runs two searches. - A results page larger than the stash threshold is folded; recover details with
tokenade expand-ref.