Claude Code for Data Science: Lighter Context

Your AI agent opens a notebook and swallows every base64 plot in it. It runs a query and the whole result set lands in the conversation. By the third iteration on a model, the context is mostly data you already looked at. Tokenade compacts notebooks, tables and command output on your computer before the agent sees them, and keeps the complete version one command away.

Sources on this page were read on 2026-10-15; Tokenade's own figure comes from the published THOL benchmark.

What you deal with every day

Data work with Claude Code is mainstream: Dataquest's guide for data scientists (2025-10-21) covers project setup, EDA and tests, and dbt Labs ships official agent skills for analytics engineering (listed 2026-06-03). The same few things fill the context in every project.

Notebooks are heavy JSON

An .ipynb file is JSON, and its saved outputs ride along. neuroai.science (2026-01-29) explains why Jupyter and Claude don't mix well: plots "are embedded in the json file underlying the jupyter notebook in base64 format, and that eats up large chunks of the context window".

The authors of the nb-claude-skill plugin advertise that a 20-cell notebook "can be 50 000+ tokens as raw JSON", measured against their own renderer.

Tables tokenise badly

Delimiters, decimals and timestamps break into many tokens per cell. Writing about uploads to Claude, fast.io (updated 2026-10-07) notes that "an uncompressed CSV file measuring only a few megabytes on disk can expand into hundreds of thousands of tokens". The same arithmetic applies when an agent prints a CSV or a SELECT * to its terminal.

Script output lands whole

Dataquest's guide shows the ! bash mode, which "adds complete output to Claude Code's context". It also warns that "accumulated conversation history consumes tokens". Training logs, dbt run output and test reports are re-sent on every following turn until the session ends or compacts.

Credentials sit next to the data

Warehouse passwords in profiles.yml, tokens in .env, connection strings echoed by a debug print: a data stack is full of secrets, and an agent that reads files and runs commands will meet them.

What Tokenade changes for data work

Tokenade sits between your AI agent and its tools, on the computer where you work.

Output compacted by format. When a command isn't one Tokenade knows by name, it recognises the shape of the output instead: CSV, JSON, tables, logs, notebooks, stack traces. Known commands such as pytest and git get their own compactor. See output compaction.
Big reads folded, not lost. A file read above about 25,000 tokens, or command output above about 5,000, becomes a preview plus a hash. The complete output stays in a local stash; the agent can grep it or ask it questions instead of reloading it.
Spreadsheets as text. tokenade read forecast.xlsx returns the sheets as text, which helps with the Excel files business teams send. See Read documents and media.
No paying twice. An unchanged file read again, or the same status check polled in a loop, comes back as a short pointer or an "unchanged" note.
Credentials masked. Keys, tokens, private keys and high-entropy strings are replaced with <redacted> before output reaches the agent.

On long agent sessions, Tokenade is 39% cheaper than working without it, measured on the THOL benchmark. On a subscription, that means more iterations per usage window. tokenade gain shows the saving on your own projects, per command.

What stays on your computer

Compaction runs locally. Your data, query results and code never pass through Tokenade's servers; Tokenade sends usage counts only, such as token numbers and the program name (python, dbt), never arguments or file paths (full list).

What the agent does see, its provider receives, with or without Tokenade. Compaction reduces how much of a result set reaches the model, but it is not a data-protection control. Credential masking is automatic; e-mail addresses, phone numbers and IP addresses are masked only if you set TOKENADE_REDACT_PII=1. Names and other personal fields in your data are not masked.

Tokenade works with agents that run on your computer: Claude Code (including the Code tab of Claude Desktop), Codex, Cursor, Copilot CLI and others (compatibility). It does not run inside Jupyter itself, and it has no effect in Claude Cowork.

Setting it up

Install the CLI and run tokenade install. It detects your agents and wires each one through hooks or a plugin (install guide).
Work as usual: ask the agent to profile a dataset, fix a dbt model or rerun the tests.
Run tokenade gain after a session to see which commands and files cost the most, and what was saved.

FAQ

Will the agent miss rows or columns?

The agent sees a compacted view and a hash. When it needs more, it asks the stash with --grep or --prompt, or restores the full output with tokenade expand-ref. Nothing is deleted, and output is never made larger than the original.

Does it work in Jupyter or VS Code notebooks?

Tokenade acts on the agent, not the notebook interface. When Claude Code, Codex or Cursor reads an .ipynb file or runs a script, the result goes through Tokenade. A chat assistant built into Jupyter is not covered.

Can it keep client data away from the model?

No. It reduces how much output reaches the model and masks credentials, but whatever the agent reads still goes to its provider. Use your organisation's data rules as you would without Tokenade.

What about dbt and SQL output?

Output from commands Tokenade has no dedicated compactor for is handled by format: tables, logs and JSON are compacted, and anything still long is folded with the full text kept locally.

Claude Code for data science without drowning the context in notebooks and CSVs