JS Wei (Jack) Sun

Willison's llm-echo 0.5 ships the agent mocks OpenAI's SDK still lacks

A single-maintainer plugin ships the agent-mock primitive OpenAI's own Agents SDK still carries as an open issue.

Willison’s llm-echo 0.5 ships the agent mocks OpenAI’s SDK still lacks

TL;DR

  • llm-echo 0.5 simulates system-prompt, tool-definition, and thinking-token overhead for agent harness tests.
  • OpenAI’s Agents SDK still tracks the same mock primitive as an open GitHub issue.
  • Companion llm-tools-datasette bump clears dependency debt for a typed-event llm core release.

Today’s tech section is a single feature, and the editorial point is the asymmetry it exposes. A one-maintainer plugin, llm-echo 0.5, now simulates the full overhead an agent runtime inflicts on a prompt — system prompt bytes, tool definitions, thinking-token budget — so developers can test harnesses without paying for a real inference call. The same primitive sits in OpenAI’s Agents SDK repo as an open issue, waiting on first-class mocks for tool calls and reasoning tokens.

That gap — hobbyist plugin ships the test fixture, vendor SDK links to a GitHub thread — is the whole frame. The companion llm-tools-datasette bump clears dependency debt for a core llm release built around typed events, which is the shape the mocks need to interop with. The agent-testing layer is being built in the open, one plugin at a time, ahead of the SDK that nominally owns it.

Willison’s llm-echo 0.5 mocks tool calls and reasoning tokens

Source: simon-willison · published 2026-10-18

TL;DR

  • llm-echo 0.5 now simulates system-prompt, tool-definition and thinking-token overhead — not just mirroring the user prompt back.
  • Same-day llm-tools-datasette bump clears dependency debt ahead of a core llm release built around typed events.
  • OpenAI’s Agents SDK has an open issue asking for first-class mocks of tool calls and reasoning tokens — the same primitive llm-echo just shipped.

Two plugin bumps, one tell

Simon Willison shipped llm-echo 0.5 and llm-tools-datasette 0.1.1 on the same day, and framed both as side-effects of testing pull requests against an in-progress llm core release. That framing undersells what the diff shows. llm-echo is a debug “model” that returns the request to you instead of calling an LLM; version 0.5 had to grow new simulation surface — system prompts, tool-call metadata, attachment overhead, reasoning tokens — because the core library no longer treats a model response as a text string. The LLM release series index tracks a steady 2026 march toward typed event streams covering tool calls, tool results and reasoning traces 12, and llm-echo is finally catching up to it.

The datasette-tools bump is the mundane other half: HTTP dependencies that used to be inherited implicitly from llm are being declared explicitly as the core refactors. Together they read as a maintainer clearing the debug and packaging decks right before cutting a release.

Why reasoning-token mocking matters now

The reason simulating thinking blocks is suddenly worth a point release is that llm is becoming an agentic harness, not a single-prompt CLI. llm-anthropic now exposes server-side tools — WebSearch, WebFetch, CodeExecution — through the unified -T flag 3, and the earlier 0.x release notes added the plugin surface for usage accounting and attachments that 0.5 is now exercising end-to-end 4. In an agentic loop, a debug model that doesn’t count tool-definition tokens or reasoning tokens will silently under-report cost. Any logging or budget-enforcement code built on top of --usage would ship broken, and nobody would notice until the first production bill.

The example in the release makes the shape concrete:

uvx --with llm-echo llm -m echo 'hi there' -o thinking 1 --usage

returns a reasoning line, a structured JSON dump of the full request state, and simulated input/output token counts. That is enough to exercise a cost tracker or a conversation-history serializer without burning a provider call.

Where it sits in the mocking landscape

llm-echo is deliberately minimal, and the contrast with heavier tools is the interesting part. MockServer takes the opposite approach — it returns byte-for-byte correct OpenAI, Anthropic and Gemini responses, including full streaming SSE framing and tool-call metadata, so an unmodified SDK can be pointed at a local container 5. The two live at different layers: MockServer verifies that downstream parsers handle real wire formats; llm-echo verifies that Willison’s own CLI assembles prompts, attachments and usage counters correctly before dispatch.

The gap llm-echo 0.5 closes isn’t Willison-specific either. An open issue on openai/openai-agents-python requests exactly these primitives — first-class mocking hooks for tool calls and reasoning tokens — because hand-rolled stubs keep failing strict message-order validation 6.

llm-echo just happens to be the first place in this stack where the primitive actually exists.

That’s the signal in the housekeeping: typed events, server-side tools and reasoning-token accounting are no longer optional surface area for anything that wraps an LLM.

Further reading

Footnotes

  1. simonwillison.net — LLM releases series index — https://simonwillison.net/series/llm-releases/

    A chronological series tracking every LLM core and plugin release, showing a cadence that accelerated through 2026 around reasoning traces, typed events and server-side tools.

    ↩
  2. simonwillison.net — llm-tool-use tag — https://simonwillison.net/tags/llm-tool-use/

    Model responses are no longer treated as simple text strings but as a stream of typed events, including tool calls and tool results.

    ↩
  3. simonwillison.net — model-context-protocol tag — https://simonwillison.net/tags/model-context-protocol/

    llm-anthropic now supports server-side tools like WebSearch, WebFetch and CodeExecution, invoked through the unified -T interface.

    ↩
  4. simonwillison.net — LLM 0.x release notes (Apr 2026) — https://simonwillison.net/2026/Apr/29/llm/

    Earlier LLM releases added the plugin surface for usage accounting and attachments that llm-echo 0.5 now simulates end-to-end.

    ↩
  5. MockServer — LLM Response Mocking docs — https://www.mock-server.com/mock_server/llm_response_mocking.html

    MockServer returns byte-for-byte correct provider responses for OpenAI, Anthropic and Gemini, including full streaming SSE framing and tool call metadata, without network activity.

    ↩
  6. GitHub issue — openai/openai-agents-python #678 — https://github.com/openai/openai-agents-python/issues/678

    Users request first-class mocking hooks for tool calls and reasoning tokens because stubbing them manually is brittle and often fails on strict message-order validation.

    ↩
Jack Sun

Jack Sun, writing.

Engineer · Bay Area

Hands-on with agentic AI all day — building frameworks, reading what industry ships, occasionally writing them down.

Digest
All · AI Tech · AI Research · AI News
Writing
Essays
Elsewhere
Subscribe
All · AI Tech · AI Research · AI News · Essays

© 2026 Wei (Jack) Sun · jacksunwei.me Built on Astro · hosted on Cloudflare