Benchmark

Measured on the handlers, not estimated.

Claude Code already limits long tool output, so counting the raw output as the saving overstates it. Every number here is measured against what Claude Code alone would put in context under its own documented limits, with the real Kiasi handlers on deterministic inputs. Where Claude Code alone sends less, the card says so.

python3 scripts/benchmark.py

Later steps that re-send it: 10

0tokens kept off later steps, all scenarios, versus Claude Code alone (negative where Kiasi sends more)

Loading benchmark.json…

How it is measured

Each scenario is a shape-realistic tool output generated deterministically by scripts/benchmark.py, then replayed through the same handlers that run in a session: the caps and the Bash clean in core/caps.py, and the out-of-context run and distill tools in sandbox.py. Nothing is sampled and nothing is a model call, so the numbers repeat exactly.

Three numbers per scenario. Raw output is what the tool produced. Claude Code alone is the same output under Claude Code's documented limits, through the tool path the scenario really uses without Kiasi, with the exit status that fits the story. With Kiasi is what the handler put in context. The headline compares the last two.

Far past Claude Code's limits, Claude Code alone already sends less. It saves a huge MCP result or a long successful command to a file and sends a path, with a short preview at most, so on those rows Kiasi sends more. Kiasi differs where it does something else with the output: a failing run keeps the failing lines and a path instead of a blind head and tail, outputs under the limits are still trimmed and stripped of colour codes and redraws, session transcripts are capped, and an unchanged file read a second time is not sent again.

What this does not measure

Claude Code's path message is modelled with a realistic length, not quoted from its own text.
The inputs are synthetic: shape-realistic and generated by the script, not recorded from real sessions.
The count of later steps is an assumption, not an observation. The slider above shows how much it matters.
There is no paired A/B on real tasks yet, so nothing here says the model finishes the same work with the shorter context.
How cached re-reads count toward plan limits is not modelled.

Next: a paired A/B on real tasks

Not run yet

The plan is to run the same tasks with and without Kiasi, compare tokens used and whether the task was completed, and publish the runs. Until that exists, this page is a measurement of the handlers, not of end-to-end outcomes.

The full table

Generated from benchmark.json. The same table is in BENCHMARK.md, and a test fails if the two drift.