condense-json 1.0 shrinks LLM’s log store, not its prompt tokens

TL;DR

• condense-json 1.0 swaps repeated JSON substrings for {"$r": [...]} reference tokens.
• PR #1586 wires it into LLM 0.32 as a third dedup layer over message hashing and vacuum.
• Run llm logs backup first: the new schema drops raw provider payloads like logprobs.
• TOON is the library to reach for when the goal is prompt-token savings, not disk savings.

Only one tech feature today, and its framing is the whole story. condense-json 1.0 — the new Python library Simon Willison tagged and wired into LLM 0.32 via PR #1586 — is a storage-side compression tool: it dedups repeated JSON substrings inside the local SQLite log database, adding a third layer on top of message hashing and vacuum. It does not shrink the JSON you send to a model. That’s a different problem, and Willison’s own writeup points readers at TOON for it, which claims ~40% prompt-token cuts on mixed shapes.

The distinction matters because the naming invites the blur — “condense JSON” sounds like something you’d reach for to cut inference bills, and it isn’t. Before you upgrade, the ops note is unambiguous: run llm logs backup first, because the new schema drops raw provider payloads like logprobs.

condense-json 1.0 shrinks LLM’s SQLite logs, not prompts

Source: simon-willison · published 2026-08-02

TL;DR

• Simon Willison tagged condense-json 1.0, a Python library that swaps repeated JSON substrings for {"$r": [...]} reference tokens.
• PR #1586 wires it into LLM 0.32 as the third dedup layer — message hashing, substring condensation, SQLite vacuum.
• Run llm logs backup before upgrading: the new schema drops raw provider payloads like logprobs.
• For prompt-token savings, look at TOON — claims ~40% cuts vs. standard JSON on mixed shapes.

What the library actually does

condense_json(input_json, replacements) walks a JSON tree, finds any string or substring matching a key in a replacements dict, and rewrites it into a {"$r": ["prefix ", {"$": "1"}, " suffix"]} form that points back at the shared value. uncondense_json reverses it. The output is still valid JSON — no custom parser, no lossy encoding, no prompt-time behavior change. Independent coverage frames it as a “smart reduce” that preserves the data model while optimizing storage and downstream model consumption ¹.

Where it fits: the LLM 0.32 dedup stack

The 1.0 stamp is best read alongside Willison’s concurrent llm 0.32 release, which replaces the old text-in/text-out log abstraction with typed Message parts and content-addressable hash IDs. Forked conversations now share storage as trees rather than duplicating linear turns ². condense-json operates one layer below that: after messages are hash-deduplicated, it strips repeated substrings — system prompts, tool schemas, JSON tool arguments — inside the payloads themselves.

flowchart TB
    A[LLM conversation turns] --> B[Layer 1: hash-based message IDs<br/>dedupe whole messages, enable forking]
    B --> C[Layer 2: condense-json<br/>dedupe substrings inside payloads]
    C --> D[Layer 3: SQLite VACUUM<br/>reclaim freed pages on disk]

Willison’s own rc1 notes confirm the migration is designed to be non-disruptive — llm logs transparently unions the legacy and new tables so old history keeps working ³.

Costs the release note glosses over

Two caveats the announcement skips. First, upgraders should run llm logs backup before jumping to 0.32: the new schema drops some raw provider payloads, notably logprobs, so the compression story comes with a fidelity trade-off ⁴. Second, condense-json’s storage wins are partially offset by SQLite’s own quirks. On WAL-mode databases — which is most of them now — a VACUUM to actually reclaim freed pages can temporarily double the file size before a wal_checkpoint(TRUNCATE) completes ⁵. First-run maintenance on a large log file will spike disk usage before it drops.

Not the same fight as TOON

condense-json is the conservative choice: stay inside the JSON spec, stay reversible, only touch storage. The louder competing school in late 2026 is Token-Oriented Object Notation and similar tabular re-encodings, which claim roughly 40% token reductions — one benchmark measured 2,744 tokens versus 4,545 for standard JSON — and up to 15% inference-cost savings in production pipelines ⁶.

Critics argue that because LLMs are predominantly trained on JSON, any savings from custom formats must be weighed against potential degradation in reasoning ⁶.

That’s a real trade-off condense-json sidesteps entirely by never touching what the model sees. The 1.0 isn’t competing with TOON; it’s competing with a bloated logs.db. Pick the tool for the layer you actually have a problem at.

Footnotes

1. neura.market coverage of condense-json 1.0 — https://www.neura.market/news/condense-json-1-0-shrinks-repeated-json-data

   condense-json is positioned as a ‘smart reduce’ strategy that preserves the original data model while optimizing it for both storage and subsequent model consumption

   ↩

2. the-ai-beat.com on LLM 0.32 conversation redesign — https://www.the-ai-beat.com/blog/2026-07-31-tooling-simon-willison-s-llm-tool-gets-proper-conversation

   By using hash-based IDs for individual messages, the database can now automatically deduplicate content… enables llm to represent conversations as trees rather than linear logs, allowing users to fork a conversation from any point without redundant data storage

   ↩

3. Simon Willison, llm 0.32 rc1 post — https://simonwillison.net/2026/Jul/30/llm-rc1/

   the llm logs command now merges data from both legacy and new tables to provide a unified history

   ↩

4. steveash/hitchhikers-guide notes on llm 0.32 — https://github.com/steveash/hitchhikers-guide-to-ai-native-engineering/blob/main/source-notes/blog-simonwillison-llm032a0.md

   users should perform a manual backup using llm logs backup before upgrading, as older raw provider payloads (like log probabilities) are no longer persisted in the new format

   ↩

5. photostructure.com on SQLite vacuum — https://photostructure.com/coding/how-to-vacuum-sqlite/

   in Write-Ahead Logging (WAL) mode, a VACUUM operation can temporarily double the disk footprint because the entire rebuilt database is written into the WAL file before being merged

   ↩

6. simonwillison.net /llm/ tag notes on TOON-style formats — https://simonwillison.net/2026/Apr/29/llm/

   TOON averaged 2,744 tokens compared to 4,545 for standard JSON—a 39.6% saving… critics argue that because LLMs are predominantly trained on JSON, any savings from custom formats must be weighed against potential degradation in reasoning

   ↩ ↩²