Willison's llm-echo 0.5 ships the agent mocks OpenAI's SDK still lacks
A single-maintainer plugin ships the agent-mock primitive OpenAI's own Agents SDK still carries as an open issue.
Willison’s llm-echo 0.5 ships the agent mocks OpenAI’s SDK still lacks
TL;DR
- llm-echo 0.5 simulates system-prompt, tool-definition, and
thinking-token overhead for agent harness tests. - OpenAI’s Agents SDK still tracks the same mock primitive as an open GitHub issue.
- Companion llm-tools-datasette bump clears dependency debt for a typed-event
llmcore release.
Today’s tech section is a single feature, and the editorial point is the asymmetry it exposes. A one-maintainer plugin, llm-echo 0.5, now simulates the full overhead an agent runtime inflicts on a prompt — system prompt bytes, tool definitions, thinking-token budget — so developers can test harnesses without paying for a real inference call. The same primitive sits in OpenAI’s Agents SDK repo as an open issue, waiting on first-class mocks for tool calls and reasoning tokens.
That gap — hobbyist plugin ships the test fixture, vendor SDK links to a GitHub thread — is the whole frame. The companion llm-tools-datasette bump clears dependency debt for a core llm release built around typed events, which is the shape the mocks need to interop with. The agent-testing layer is being built in the open, one plugin at a time, ahead of the SDK that nominally owns it.
Willison’s llm-echo 0.5 mocks tool calls and reasoning tokens
Source: simon-willison · published 2026-10-18
TL;DR
- llm-echo 0.5 now simulates system-prompt, tool-definition and
thinking-token overhead — not just mirroring the user prompt back. - Same-day llm-tools-datasette bump clears dependency debt ahead of a core
llmrelease built around typed events. - OpenAI’s Agents SDK has an open issue asking for first-class mocks of tool calls and reasoning tokens — the same primitive
llm-echojust shipped.
Two plugin bumps, one tell
Simon Willison shipped llm-echo 0.5 and llm-tools-datasette 0.1.1 on the same day, and framed both as side-effects of testing pull requests against an in-progress llm core release. That framing undersells what the diff shows. llm-echo is a debug “model” that returns the request to you instead of calling an LLM; version 0.5 had to grow new simulation surface — system prompts, tool-call metadata, attachment overhead, reasoning tokens — because the core library no longer treats a model response as a text string. The LLM release series index tracks a steady 2026 march toward typed event streams covering tool calls, tool results and reasoning traces 12, and llm-echo is finally catching up to it.
The datasette-tools bump is the mundane other half: HTTP dependencies that used to be inherited implicitly from llm are being declared explicitly as the core refactors. Together they read as a maintainer clearing the debug and packaging decks right before cutting a release.
Why reasoning-token mocking matters now
The reason simulating thinking blocks is suddenly worth a point release is that llm is becoming an agentic harness, not a single-prompt CLI. llm-anthropic now exposes server-side tools — WebSearch, WebFetch, CodeExecution — through the unified -T flag 3, and the earlier 0.x release notes added the plugin surface for usage accounting and attachments that 0.5 is now exercising end-to-end 4. In an agentic loop, a debug model that doesn’t count tool-definition tokens or reasoning tokens will silently under-report cost. Any logging or budget-enforcement code built on top of --usage would ship broken, and nobody would notice until the first production bill.
The example in the release makes the shape concrete:
uvx --with llm-echo llm -m echo 'hi there' -o thinking 1 --usage
returns a reasoning line, a structured JSON dump of the full request state, and simulated input/output token counts. That is enough to exercise a cost tracker or a conversation-history serializer without burning a provider call.
Where it sits in the mocking landscape
llm-echo is deliberately minimal, and the contrast with heavier tools is the interesting part. MockServer takes the opposite approach — it returns byte-for-byte correct OpenAI, Anthropic and Gemini responses, including full streaming SSE framing and tool-call metadata, so an unmodified SDK can be pointed at a local container 5. The two live at different layers: MockServer verifies that downstream parsers handle real wire formats; llm-echo verifies that Willison’s own CLI assembles prompts, attachments and usage counters correctly before dispatch.
The gap llm-echo 0.5 closes isn’t Willison-specific either. An open issue on openai/openai-agents-python requests exactly these primitives — first-class mocking hooks for tool calls and reasoning tokens — because hand-rolled stubs keep failing strict message-order validation 6.
llm-echojust happens to be the first place in this stack where the primitive actually exists.
That’s the signal in the housekeeping: typed events, server-side tools and reasoning-token accounting are no longer optional surface area for anything that wraps an LLM.
Further reading
- llm-tools-datasette 0.1.1 — simon-willison
Footnotes
-
simonwillison.net — LLM releases series index — https://simonwillison.net/series/llm-releases/
↩A chronological series tracking every LLM core and plugin release, showing a cadence that accelerated through 2026 around reasoning traces, typed events and server-side tools.
-
simonwillison.net — llm-tool-use tag — https://simonwillison.net/tags/llm-tool-use/
↩Model responses are no longer treated as simple text strings but as a stream of typed events, including tool calls and tool results.
-
simonwillison.net — model-context-protocol tag — https://simonwillison.net/tags/model-context-protocol/
↩llm-anthropic now supports server-side tools like WebSearch, WebFetch and CodeExecution, invoked through the unified -T interface.
-
simonwillison.net — LLM 0.x release notes (Apr 2026) — https://simonwillison.net/2026/Apr/29/llm/
↩Earlier LLM releases added the plugin surface for usage accounting and attachments that llm-echo 0.5 now simulates end-to-end.
-
MockServer — LLM Response Mocking docs — https://www.mock-server.com/mock_server/llm_response_mocking.html
↩MockServer returns byte-for-byte correct provider responses for OpenAI, Anthropic and Gemini, including full streaming SSE framing and tool call metadata, without network activity.
-
GitHub issue — openai/openai-agents-python #678 — https://github.com/openai/openai-agents-python/issues/678
↩Users request first-class mocking hooks for tool calls and reasoning tokens because stubbing them manually is brittle and often fails on strict message-order validation.