Willison clones Claude Code, GEPA overfits Datasette, Anthropic splits defenders

TL;DR

• Simon Willison shipped llm-coding-agent 0.1a0, a Claude Code clone with 6 tools written by Fable 5.
• GEPA’s prompt rewrite lifted Datasette Agent to 95% train accuracy but only 85% on held-out test.
• Anthropic’s 0-10 Cyber Jailbreak Severity rubric arrives co-authored with Google, Microsoft and Amazon.
• Stamos, Moussouris, Wysopal, Vixie call Fable 5’s tighter classifier a huge own goal for U.S. defenders.
• Geoffrey Litt warns agent-driven changes accrue cognitive debt as developers’ mental models drift.

Today’s three AI-tech leads all sit one layer above the model — the automated helper that writes an agent, tunes an agent, or grades attacks against one — but the three ships land differently. Simon Willison shipped llm-coding-agent 0.1a0, a working Claude Code clone whose library was itself written by Fable 5 against a spec.md prompt. GEPA rewrote Datasette Agent’s SQL prompt to 95% training accuracy by inserting advice that contradicted existing rules, and held-out test dropped to 85%. Anthropic’s Cyber Jailbreak Severity rubric ships co-authored with Google, Microsoft and Amazon, but the tighter Fable 5 classifier it enables is blocking IBM X-Force code reviews and drawing an on-record revolt from Stamos, Moussouris, Wysopal and Vixie.

Geoffrey Litt’s AIE 2026 talk, surfaced in the round-up, names the operator condition running underneath all three: when automation ships changes bigger than you can hold in your head, the mental model drifts and the human stops being a creative participant.

Willison ships a Claude Code clone written by Fable 5

Source: simon-willison · published 2026-07-02

TL;DR

• Simon Willison shipped llm-coding-agent 0.1a0, a Claude Code clone built on his llm library and installable via uvx.
• Its six tools mirror Claude Code’s canonical seven-tool set that reportedly handles ~95% of coding tasks.
• Fable 5 wrote the library itself from a spec.md prompt using red/green TDD, unprompted CodingAgent Python API included.
• The --yolo mode inherits the repo-based prompt-injection threat that has produced reverse-shell exploits against Claude Code.

A faithful clone, tool for tool

Willison’s tool inventory is Claude Code’s tool inventory with the serial numbers filed off. read_file, write_file, edit_file, execute_command, list_files, search_files map cleanly onto Anthropic’s Read/Write/Edit/Bash/Glob/Grep — the seven-tool canon that reportedly covers ~95% of real coding tasks in the reference implementation ¹. The edit_file contract, which demands that old_string match the file’s current contents exactly, is the same “staleness check” pattern Claude Code uses to stop the model from editing files it hasn’t recently read ¹.

What’s missing is the Task sub-agent primitive — Claude Code’s mechanism for spawning parallel workers — and the ripgrep-backed parallel-safe search that gives the reference implementation its speed. search_files here is a plain regex pass capped at 100 results. This is a 0.1a0, and Willison labels it a “slop-alpha”; the point is that his llm library now has enough agent-framework surface area to host a Claude Code clone at all.

Fable 5 wrote most of it

The provenance is the story. Willison handed Claude Code for web two prompts: write a spec.md, then implement it via red/green TDD with sensible commits. Fable 5 produced the working library, the tests, the README, and — unprompted — a CodingAgent(model=..., root=..., approve=True).run(...) Python API that Willison says he’s “delighted to see.”

That workflow has a known failure mode worth naming. When an agent runs both the red and green phases of TDD itself, it can generate a test suite that “perfectly validates a hallucinated business requirement,” producing high coverage against the wrong specification ². A spec-first prompt mitigates this by fixing intent up front, but nobody should treat a Fable-authored library as production-hardened on the basis of green CI alone.

The security surface it inherits

llm code --yolo opts the developer into the exact threat model that has already burned Claude Code users. SecurityWeek documented attacks where an innocuous-looking repository embeds setup instructions that trick the agent into spawning a reverse shell on the host ³. Iain Harper argues the industry is approaching “a Challenger disaster for AI security” because developers routinely grant agents broad system permissions ⁴.

Willison’s --allow "pytest*" --allow "git diff*" allowlists are the correct mitigation shape. But practitioners have observed that Claude-style agents exhibit “bash addiction” — reaching for cat or sed inside execute_command instead of the safer built-in Read/Edit tools, which routes right around whatever gating the harness applies ⁵. Anyone leaning on --allow should assume the model will try to shell out past it.

Where this sits

The 2026 open-source coding-agent field is crowded — OpenCode swaps between 75+ providers mid-session, Aider reportedly completes comparable multi-file tasks with 4.2× fewer tokens than Claude Code ⁶. llm-coding-agent isn’t competing on novelty. Its differentiator is that it plugs into Willison’s existing llm plugin ecosystem and inherits model-agnosticism for free: today it’s GPT-5.5 building SwiftUI CLIs, tomorrow it’s whichever model has a plugin. The interesting claim isn’t that this agent is better — it’s that a single developer plus one Fable session can now clone Claude Code in a weekend.

---

GEPA overfits Datasette Agent: 95% train, 85% test

Source: simon-willison · published 2026-07-02

TL;DR

• Held-out test accuracy fell from 95% to 85% after GEPA rewrote Datasette Agent’s SQL prompt.
• GEPA raised training accuracy to 95% by inserting advice that contradicted existing display-mode rules.
• Decagon needed 19+ ablations to stabilize GEPA for production supervisor agents.
• Arize’s rival “Prompt Learning” optimizer claims parity with fewer rollouts than GEPA.

The overfitting Willison undersells

Simon Willison’s write-up frames DSPy as a rigorous way to catch subtle prompt failures in Datasette Agent’s SQL generation. The worked example is genuinely useful — but independent analysis of the same experiment surfaces a number the post buries: GEPA raised training accuracy from 90% to 95% and simultaneously dropped held-out test accuracy from 95% to 85% on the 30-question bookstore benchmark ⁷.

That’s not a small wobble. It’s the canonical small-dataset overfitting story, and the mechanism is embarrassing: GEPA inserted advice that conflicted with the agent’s existing display-mode rules ⁷ — precisely the “contradictory instructions cause guess-and-retry loops” pathology the optimizer was invoked to fix. Automated prompt edits need the same regression discipline as code changes, and a ~20-example training split is nowhere near enough to trust GEPA’s rewrites.

GEPA’s real cost curve

The Datasette experiment reads differently against published GEPA numbers. On paper, GEPA looks dominant: a 12% lead over MIPROv2 on AIME-2025 and up to 35× fewer rollouts than GRPO on reasoning tasks ⁸. But Decagon’s production notes report needing 19+ ablation experiments to stabilize GEPA for supervisor agents, and flag that GEPA’s reflection model — typically GPT-4o class — drives per-iteration costs up substantially ⁹.

Arize’s competing “Prompt Learning” work claims comparable accuracy to GEPA with fewer rollouts by strengthening the feedback loop rather than layering on evolutionary machinery ¹⁰. On a benchmark as small as Willison’s, GEPA’s evolutionary complexity may simply be overkill.

The bug was schema representation, not prompt phrasing

Willison’s diagnosed failure — the schema listing gives only table names, so the agent guesses columns like page_count and o.order_id, then loops on errors — is a well-known text-to-SQL failure mode with an active research literature he doesn’t cite.

Recent work pushes in exactly the opposite direction from the “don’t call describe_table if you already have the info” heuristic. CRUSH4SQL deliberately uses “useful hallucinations” as a retrieval seed: ask the LLM to imagine a plausible schema, then use that imagined schema to retrieve real columns via dense retrieval ¹¹. ColSQL enriches column metadata with LLM-generated descriptions before generation ¹¹. The accuracy stakes are real — models score 85–91% on Spider 1.0 but collapse to 10–21% on enterprise-realistic benchmarks like BIRD-Ent ¹¹.

Willison’s landing fix (include column names upfront, soften the describe_table restriction) is correct, but it aligns with the multi-stage schema-linking consensus. DSPy surfaced the symptom; the cure was already in the literature.

The security wrinkle

Datasette Agent already lives inside the “lethal trifecta” of private data, tool use, and untrusted content. Cloudurable’s write-up on DSPy’s MCP integration flags CVE-2025-12695 (a file-read restriction bypass) and articulates a sharper concern: self-optimizing prompt systems can drift into emergent vulnerabilities that are harder to audit than static, hand-written prompts ¹². When a GEPA run can silently insert instructions that contradict your existing security rules the way it contradicted Willison’s display-mode rules ⁷, that’s not a hypothetical.

The Datasette experiment is a good worked example of DSPy in a real project. It’s also a good worked example of why you don’t ship GEPA’s rewrites without a held-out test set.

---

Anthropic’s CJS grades AI jailbreaks 0-10 amid defender revolt

Source: anthropic-news · published 2026-07-02

TL;DR

• Anthropic proposes Cyber Jailbreak Severity (CJS), a 0-10 log-scale rubric co-authored with Google, Microsoft and Amazon.
• Fable 5’s tighter classifier claims 99% coverage of the June bypass while rejecting routine code reviews from IBM X-Force.
• Stamos, Moussouris, Wysopal and Vixie call the blocks a “huge own goal” for U.S. defenders.
• The HackerOne program is a non-paid VDP that one researcher bypassed in ~20 hours.

The framework Anthropic wants regulators to adopt

CJS is a four-axis rubric — Capability Gain (0-4), Breadth (0-2), Weaponization Ease (0-2), Discoverability (0-2) — collapsed onto a logarithmic 0-10 severity scale. Anthropic anchors it with a Log4Shell thought experiment: a model that surfaced the flaw pre-disclosure in December 2021 scores CJS-4 (Critical); the same output today scores CJS-0 because public scanners already catch it.

Reviewers can manually elevate scores when a jailbreak surfaces novel, unpatchable vulnerabilities. The pitch is a common vocabulary that stops the next incident from producing another 19-day Commerce Department shutdown.

What defenders are actually seeing

The “safety margin” Anthropic touts is landing as a keyword filter in practice. Matt Suiche reports that asking for “security best practices” can silently downgrade a session to Opus 4.8, and IBM X-Force’s Valentina Palmiotti says routine code reviews and even security blog posts get rejected ¹³. Anthropic itself admits the redeployed classifier “comes at the cost of flagging benign requests more often” while claiming 99% coverage of the specific June bypass ¹⁴.

The four-tier taxonomy hides the tradeoff. Most legitimate defensive work — OSINT, vulnerability triage, exploit reproduction for patching — lives in “Low-Risk Dual Use,” the tier explicitly designed for partial blocks. An open letter organized by Alex Stamos and signed by Katie Moussouris, Rachel Tobac, Chris Wysopal and Paul Vixie called the pullback a “huge own goal,” noting the original triggering prompt merely asked the model to fix known flaws — a capability GPT-5.5, Kimi 2.7 and GLM-5.2 all still offer ¹⁵.

The shifting-baseline problem

CJS’s most elegant feature — severity that decays as defenses catch up — is also its weakest. Corelight documents Mythos Preview scoring 83.1% first-attempt exploit success and autonomously surfacing a 27-year-old OpenBSD TCP flaw, collapsing vulnerability-to-exploit windows “from months to hours” ¹⁶. If model capability itself is compounding, a rubric that treats “already discoverable” as CJS-0 will systematically understate the risk of the next Log4Shell — because “already discoverable” now means “discoverable by an agent that runs overnight.”

CJS also isn’t alone. NIST and OWASP’s AIVSS v0.8 already extends CVSS v4.0 with “agentic amplifiers” for autonomy, tool use and persistent memory. Cyber Strategy Institute researchers dismissed AIVSS as “forensics rather than proactive governance if not enforced at runtime” — a charge that transfers cleanly to CJS’s post-hoc scoring ¹⁷.

The bounty that isn’t

Buried under the framework is a detail Anthropic’s post omits: the HackerOne “Cyber Jailbreak” program is a Vulnerability Disclosure Program, not a paid bounty ¹⁸. Researcher Vitto Rivabella bypassed Fable 5 in roughly 20 hours and concluded the extracted output was “often less useful than a standard Google search”; others, like Pliny, are already publishing rather than disclosing ¹⁸. Critics warn the incentive structure will push jailbreakers toward publishing for clout — the exact failure mode CJS was designed to score ¹⁸.

The framework is a shared vocabulary for regulators. The security community’s actual objection — that Fable 5’s blocks cost defenders more than they cost attackers — remains unanswered.

Round-ups

Geoffrey Litt: understand your agent’s code or lose the plot

Source: simon-willison

Coding agents now ship changes big enough that developers accrue cognitive debt as their mental model drifts from the codebase. Geoffrey Litt’s AIE 2026 talk argues for a rule of thumb: understand the code deeply enough to remain an active creative participant, not a rubber stamp.

Footnotes

1. claudecodecamp.com — how tool use works in Claude Code — https://www.claudecodecamp.com/p/how-tool-use-actually-works-in-claude-code

   Claude Code utilizes a core set of seven tools—Read, Write, Edit, Bash, Grep, Glob, and Task—to handle roughly 95% of coding tasks… The Edit tool performs exact string replacement and includes a ‘staleness check’ to ensure the file has not changed since the model last read it.

   ↩ ↩²

2. arielsoftwares.com — TDD critique 2026 — https://www.arielsoftwares.com/test-driven-development/

   Because agents often combine the red and green steps internally, they may generate a test suite that perfectly validates a hallucinated business requirement, creating a ‘shadow AI’ problem where coverage is high but functional truth is absent.

   ↩

3. SecurityWeek — attack abusing Claude Code repos — https://www.securityweek.com/new-attack-abuses-claude-code-and-harmless-looking-repositories-to-hijack-developer-machines/

   Security researchers have demonstrated ‘indirect prompt’ attacks where a harmless-looking repository contains setup instructions that trick Claude Code into spawning a reverse shell on the developer’s machine.

   ↩

4. iain.so — Security for Production AI Agents in 2026 — https://iain.so/security-for-production-ai-agents-in-2026

   Industry experts like Iain Harper suggest we are nearing a ‘Challenger disaster’ for AI security, as developers frequently run agents with broad system permissions.

   ↩

5. dev.to — ‘Claude Code ignores its own tools’ — https://dev.to/yurukusa/claude-code-ignores-its-own-tools-here-are-3-hooks-that-force-it-to-behave-mi1

   A common developer complaint is ‘bash addiction,’ where Claude ignores its built-in tools (like Read or Edit) in favor of common shell commands like cat or sed found in its training data… This bypasses safety checks and triggers unnecessary permission prompts.

   ↩

6. KDnuggets — 10 Agentic AI Frameworks 2026 — https://www.kdnuggets.com/10-agentic-ai-frameworks-you-should-know-in-2026

   OpenCode… is model-agnostic, allowing developers to switch between over 75 providers—including Anthropic, OpenAI, and local models via Ollama—mid-session… Aider has been benchmarked using 4.2x fewer tokens than Claude Code for similar multi-file tasks.

   ↩

7. letsdatascience.com analysis of Willison’s DSPy experiment — https://letsdatascience.com/news/dspy-evaluates-and-refines-datasette-agent-prompts-6ced2073

   the GEPA-optimized prompt resulted in a notable performance drop on held-out data, with test accuracy falling from 95% to 85%… the optimizer adding contradictory advice that conflicted with the agent’s existing display-mode rules.

   ↩ ↩² ↩³

8. particula.tech — DSPy GEPA vs MIPROv2 benchmarks — https://particula.tech/blog/dspy-gepa-vs-miprov2-automatic-prompt-optimization

   On demanding reasoning tasks like the AIME-2025 math benchmark, GEPA achieves a 12% accuracy lead over MIPROv2… GEPA reaching optimal performance using up to 35x fewer model rollouts than GRPO.

   ↩

9. Decagon engineering blog — Optimizing GEPA for production — https://decagon.ai/blog/optimizing-gepa-for-production

   19+ ablation experiments were necessary to stabilize performance for production-grade supervisors… GEPA’s reliance on a reflection model (typically a frontier LLM like GPT-4o) can introduce higher per-iteration costs.

   ↩

10. super-agentic.ai — GEPA optimizer field notes — https://super-agentic.ai/resources/super-posts/gepa-game-changing-dspy-optimizer-agentic-ai/

    Arize suggest that simpler feedback-loop techniques, branded as ‘Prompt Learning,’ can match or exceed GEPA’s accuracy with even fewer rollouts by focusing on stronger feedback loops rather than evolutionary complexity.

    ↩

11. Neo4j developer blog — Schema representation in text-to-SQL/Cypher — https://neo4j.com/blog/developer/schema-representation-in-text2cypher/

    CRUSH4SQL uses ‘useful hallucinations’ by asking an LLM to imagine a minimal schema first, then using that hallucination to retrieve actual database columns via dense retrieval… models score 85-91% on academic benchmarks like Spider 1.0, they plummet to 10-21% on enterprise-realistic ones like SocialDB or BIRD-Ent.

    ↩ ↩² ↩³

12. cloudurable.com — Securing DSPy’s MCP integration — https://cloudurable.com/blog/securing-dspy-s-mcp-integration-reliable-ai-meets/

    CVE-2025-12695, a vulnerability where DSPy failed to properly restrict file reads… because DSPy systems are designed to adapt and optimize themselves based on feedback, they may develop emergent vulnerabilities over time that are harder to audit than static, hand-written prompts.

    ↩

13. digitalapplied.com — Fable 5 classifier tradeoffs analysis — https://www.digitalapplied.com/blog/claude-fable-5-safety-classifier-coding-tradeoffs-2026

    Matt Suiche argues the filter acts as a crude keyword-based barrier; asking Claude to implement ‘security best practices’ can paradoxically trigger a downgrade to Opus 4.8, and IBM X-Force’s Valentina Palmiotti reported routine code reviews and security blog posts being rejected.

    ↩

14. MarkTechPost — Fable 5 redeployment coverage — https://www.marktechpost.com/2026/07/01/anthropic-redeploys-claude-fable-5-on-july-1-after-us-export-controls-lift-adds-new-cybersecurity-classifier/

    The updated classifier is designed to block the reported bypass in 99% of cases, and Anthropic admitted the update ‘comes at the cost of flagging benign requests more often.’

    ↩

15. letsdatascience.com — coverage of Stamos-led open letter — https://letsdatascience.com/news/cybersecurity-leaders-ask-us-to-lift-anthropic-restrictions-ba8b2ba8

    Signatories including Alex Stamos, Katie Moussouris, Rachel Tobac, Chris Wysopal and Paul Vixie warned that pulling Fable 5 away from defenders was a ‘huge own goal for the U.S.’, noting the triggering ‘jailbreak’ merely asked the model to fix known flaws — a capability already present in GPT-5.5 and open-weight Chinese models like Kimi 2.7 and GLM-5.2.

    ↩

16. Corelight blog — Claude Mythos exploit-window analysis — https://corelight.com/blog/claude-mythos-collapsing-exploit-window

    Mythos Preview scored 93.9% on SWE-bench Verified and 83.1% first-attempt exploit success, autonomously surfacing a 27-year-old TCP flaw in OpenBSD — collapsing the vulnerability-to-exploit window ‘from months to hours’ and making CJS’s ‘shifting baseline’ scoring precarious.

    ↩

17. neuraltrust.ai — AIVSS framework comparison — https://neuraltrust.ai/blog/aivss-scoring-system

    NIST/OWASP’s AIVSS v0.8 builds on CVSS v4.0 with ‘agentic amplifiers’ for autonomy, tool use and persistent memory; Cyber Strategy Institute researchers labeled it ‘forensics rather than proactive governance if not enforced at runtime’ — a critique that applies equally to CJS’s post-hoc scoring.

    ↩

18. letsdatascience.com — CJS framework explainer — https://letsdatascience.com/news/anthropic-proposes-cross-industry-framework-for-scoring-ai-j-8da00d16

    Anthropic’s HackerOne ‘Cyber Jailbreak’ program is a non-monetary VDP; researcher Vitto Rivabella bypassed Fable 5 in ~20 hours but concluded the extracted information was ‘often less useful than a standard Google search,’ while critics warn unrewarded disclosure will push jailbreakers toward publishing for clout.

    ↩ ↩² ↩³