Fable 5.1 forces reasoning, BenchMIRT finds 2 axes, Paint.NET unreviewed

TL;DR

• Claude Fable 5.1 burns $3.30 and 14 minutes on one pelican SVG, with reasoning no longer disableable.
• Ai2’s BenchMIRT collapses 16 open-weight benchmarks to just 2 latent axes: safety and reasoning.
• Paint.NET ships 180,000 lines of Claude-written Direct2D code its creator admits he cannot review.
• Hugging Face releases 200+ WebGPU kernels for browser-local inference without server round-trips.
• Python 3.15 hits RC2 with maintainers urged to publish wheels before October’s stable release.

Today’s three features don’t line up under one frame, so take them as a list. Anthropic’s Fable 5.1 removes the reasoning:{enabled:false} escape hatch — the model now returns a 400 if you try to turn thinking off, and the same task costs 1.7× more output tokens, eating the 75% cache-read discount. Ai2’s BenchMIRT fits multidimensional IRT to 100 open-weight LLMs and recovers just two latent axes (safety and reasoning), with per-item discrimination weights that are themselves a gaming vector: strip the sharpest safety items and the aggregate looks fine. And Paint.NET’s Rick Brewster shipped 180,000 lines of Claude-generated Direct2D bindings for WINE, calling the DLL “trust me bro” code behind a /wine flag because he cannot review a quarter of his 20-year codebase.

The briefs sit alongside as tooling scaffolding: Hugging Face’s 200+ WebGPU kernels, Codex’s LibreOffice bundle, a datasette-mcp release reshaped for weaker models, a vibe-coded GeoJSON viewer, and Python 3.15 RC2.

Claude Fable 5.1 burns $3.30 and 14 min per pelican

Source: simon-willison · published 2026-09-01

TL;DR

• Max reasoning took 13m54s and cost $3.30 to generate one pelican SVG, emitting 65,927 output tokens.
• Fable 5.1 uses ~1.7× more output tokens than Fable 5 for equivalent tasks, eating Anthropic’s 75% cache-read discount.
• Terminal-Bench-Science jumped to 52.6% from 24.7%, backed by a Cognition case cracking a 4-5-year-old concurrency bug in one afternoon.
• Reasoning can no longer be disabled — reasoning:{enabled:false} now returns a 400 error.

The pelican is expensive on purpose

Simon Willison’s pelican-on-a-bicycle test walks Claude Fable 5.1 up its new five-rung reasoning ladder — low, medium, high, xhigh, max — and the price tag climbs from ~$0.10 at low effort to $3.30 at max, with output token counts jumping from ~2,000 to 65,927. At max, the model spent 13 minutes and 54 seconds debating whether to give the pelican a helmet or preserve its crest, then adjusting the front fork’s rake so the bike would sit right.

The picture that came out is genuinely the best pelican Willison has gotten from an Anthropic model. But treating the cost curve as a knob the user chose to turn misses the bigger shift.

Verbosity is the default, not a setting

VentureBeat’s read of Artificial Analysis data finds Fable 5.1 emits roughly 1.7× the tokens of Fable 5 to solve the same task ¹. Anthropic’s headline 75% cache-read discount, down to $0.25 per million tokens, is more than swallowed by that output growth — net per-task cost is up around 20% for many real workloads. Willison’s low/medium runs skipped visible reasoning entirely and still burned ~2,000 output tokens each, which is consistent with a model that’s simply chattier at every effort level.

Compounding this: developers on r/claudexplorers report that reasoning:{enabled:false} now triggers a 400 invalid_request_error — the thinking block is architecturally required for tool calls ². The ladder isn’t fully optional. If you use tool-calling, you pay for thought.

The Millennium bug is the real launch story

The SVG bench is charming; the launch-day case study is not. Cognition migrated Devin’s code-review traffic to Fable 5.1 on day one after the model resolved a “one-in-a-million” concurrency bug at hedge fund Millennium that had gone unexplained for four to five years — by disassembling a closed-source vendor library and matching its logic against a core dump ³. That’s the forensic-reasoning story behind the Terminal-Bench-Science leap from 24.7% to 52.6%, and it’s a much better argument for the token bill than any pelican.

Pelican fatigue is setting in

Two dissents worth flagging. First, on the animation Willison generated for $1.37: HN commenters flagged the reversed-wheel rotation as a genuine model failure, not the MP4 encoding artifact Willison suggested ⁴. Second, the community has coined “pelicanmaxxing” — the argument that labs are now training against this specific public prompt, which would erode its diagnostic value ⁵. Willison himself started worrying about this in July; the term suggests the community now treats it as established.

GPT-5.6 Sol… a more cost-effective workhorse that provides 95% of Fable’s intelligence at a fraction of the operational budget ⁶.

That’s the reviewer verdict Anthropic has to answer. Fable 5.1 is measurably smarter, but it’s structurally more expensive, less controllable at the API layer, and being judged on benchmarks its own popularity is contaminating. The Millennium bug-find is a strong case for keeping it in the toolbox. The default-verbosity math is a strong case for not routing everything through it.

---

Paint.NET ships 180K lines of Claude-written Direct2D for WINE

Source: simon-willison · published 2026-09-02

TL;DR

• Paint.NET’s creator shipped 180,000 lines of Claude-generated code to reimplement Direct2D on WINE — a quarter of the 20-year codebase.
• Rick Brewster openly admits he cannot review it, calling the DLL “trust me bro” code gated behind a /wine flag.
• Independent coverage claims Claude produced more working Direct2D surface in 3 weeks than WINE shipped in 12 years.
• ~45% of AI-generated code ships with vulnerabilities per Veracode 2025, with no measurable gains as models scale.

What actually shipped

Rick Brewster, sole maintainer of Paint.NET for two decades, just merged a from-scratch reimplementation of Microsoft’s Direct2D graphics API — PaintDotNet.Windows.Direct2D1.Managed.dll, 180,000 lines — written almost entirely by Claude. It ships behind a /wine command-line flag and only activates when the app runs under WINE on Linux. WINE’s own Direct2D has been perpetually incomplete since Paint.NET adopted the API in 2014; per one independent write-up, Claude produced more usable surface area in three weeks than the WINE project managed in twelve years, including working out the math for Direct2D’s built-in effects library ⁷.

It is not a drop-in miracle. Brewster’s forum post spells out the fine print: WINE 11.14 or newer, DXVK required, a manual d3dcompiler_47 registry override to avoid instant crashes, and UIAnimation plus Windows 11 DisplayInformation disabled outright ⁸. Brewster calls the current state “buggy and slow.” The DLL is a compatibility escape hatch, not a general-purpose Direct2D.

The review problem

The interesting number isn’t 180,000. It’s zero — the lines Brewster has personally audited. He says so explicitly: “I cannot possibly review 180,000 lines of code.” The rest of Paint.NET is roughly 700,000 lines built over 20 years of careful stewardship. This DLL is a quarter of that, dropped in weeks, unreviewed.

Simon Willison has spent the last year trying to hold a line between “vibe coding” (ship what the model emits) and “vibe engineering” (professionals steering agents with real review), and has flagged Brewster’s release as the moment that distinction collapses — a senior developer explicitly shipping code they cannot audit ⁹. The empirical case for worry is not thin. Veracode’s 2025 dataset, summarized by the Cloud Security Alliance, found roughly 45% of AI-generated code contains security vulnerabilities, with no improvement as models scale ¹⁰. Brewster himself caught Claude skipping COM AddRef() calls — exactly the resource-leak class that fails silently on GPUs. A separate review of six vibe-coded projects found manual audit of a 300K-line AI codebase took ~10× the AI’s write time and hit an “80/20 wall” where features work in isolation but interactions collapse in production ¹¹. Brewster’s decision not to review is economically rational; it also pushes the debugging cost onto users of the /wine flag.

The clean-room that isn’t

Brewster calls this “clean-room reverse-engineered.” That phrase has a specific legal meaning: the implementing party must have had no exposure to the protected source. An LLM trained on public GitHub — and, plausibly, leaked Windows internals — cannot credibly claim that firewall ¹². Worse, under Thaler v. Perlmutter, purely AI-authored code may not be copyrightable at all, leaving Paint.NET with “all the liability and none of the protection” if Microsoft ever objected ¹². Microsoft has historically tolerated WINE, so the acute risk is low, but AI-generated reimplementations of a live proprietary API are legally untested.

Net

The feasibility bar for reimplementing a large proprietary API just fell through the floor. The review, security, and IP frameworks that made previous clean-room work defensible have not moved. Brewster is running the experiment in public, and everyone shipping agent-written code at this scale is watching what breaks first — the DLL, the users, or the lawyers.

---

Ai2’s BenchMIRT collapses 16 LLM benchmarks to 2 axes

Source: huggingface-blog · published 2026-09-01

TL;DR

• Ai2’s BenchMIRT fit multidimensional IRT to 100 open-weight LLMs and recovered just two latent axes: safety and reasoning.
• BBQ loads on reasoning, not bias — failing an item may mean lost entity-tracking, not a stereotype.
• WMDP scores fall as reasoning rises: higher-reasoning models refuse the weapons-knowledge prompts rather than answer.
• Per-item discrimination weights are a gaming vector: strip the sharpest safety items and aggregate scores stay clean.

What BenchMIRT actually does

Item Response Theory has been used for decades to score humans on tests where some questions are harder or more diagnostic than others. Ai2’s BenchMIRT ports the multidimensional variant (MIRT) to LLM evaluation: fit a model of latent abilities across 100 open-weight LLMs answering ~34,000 questions from 16 benchmarks, then read out which questions load on which capability.

Trained without labels on which benchmark tests what, MIRT settles on two dimensions — safety and general reasoning. That stability is the load-bearing claim of the post. Everything downstream (subsetting, per-item prediction, benchmark reinterpretation) follows from those two axes being real.

This is a natural extension of an active subfield. py-irt (Lalor & Rodriguez) established that raw accuracy doesn’t track latent ability ¹³, and tinyBenchmarks already showed a curated 100-item subset of MMLU predicts full-benchmark scores within ~2% error ¹⁴. BenchMIRT’s “10% preserves ranking” is confirmation, not revelation.

BBQ measures reasoning, WMDP rewards refusal

The interesting result is the disentangling. BBQ — the Bias Benchmark for QA — loads more heavily on reasoning than on the safety axis: failing a BBQ item may mean the model lost track of which entity the pronoun refers to, not that it holds a stereotype. HarmBench’s copyright prompts (song-lyric requests and friends) similarly track reasoning, not safety. WMDP is inverted: higher-reasoning models score lower on the weapons-knowledge test because they follow refusal instructions better.

The WMDP finding is less of a discovery than the post frames it. CAIS designed WMDP as an unlearning proxy explicitly so that mitigation (RMU) drives hazardous-topic accuracy toward chance while MMLU stays high ¹⁵ — the negative correlation is the intended signal.

The BBQ reinterpretation is where independent work pushes back hardest. A recent paper shows chain-of-thought models can amplify social bias: they identify the correct “Unknown” answer, then rationalize their way into a stereotype through spurious intermediate steps ¹⁶. So the reasoning–bias relationship is bidirectional, not the clean “BBQ is really a reasoning test” story a two-dimensional fit produces.

The regime-mismatch problem

Stanford CRFM has flagged a structural issue with IRT-for-LLMs that BenchMIRT does not resolve. Classical IRT was built for many test-takers and few items; LLM benchmarks invert that ratio, and simulations across 18,000 conditions show ranking inferences become unstable when ability distributions are skewed or when proprietary APIs hide token probabilities ¹⁷. A 100-model fit sits near the low end of what these estimators tolerate, which is consistent with BenchMIRT’s own note that plain averages sometimes beat MIRT at per-item prediction (79% vs. 70% baseline is a real gain, but a narrow one).

The gaming vector

Ai2 raises the concern themselves, and it’s not hypothetical. The UK AI Security Institute has documented models inflating safety scores through blanket over-refusal — a statistical artifact that looks aligned in aggregate but isn’t a genuine trait ¹⁸. BenchMIRT publishes per-item discrimination weights. A motivated developer now knows exactly which safety items to train against, or which to lobby benchmark curators to drop.

“Targeted benchmark design” is a double-edged phrase. Sharper instruments cut both ways.

The framework is a useful audit tool for people running honest evaluations. It is also a menu for people who aren’t.

Round-ups

Hugging Face ships 200+ WebGPU kernels for local AI

Source: huggingface-blog

Hugging Face released @huggingface/kernels, a library bundling more than 200 WebGPU compute kernels for running AI models locally in the browser. The package targets developers building client-side inference without server round-trips, extending Hugging Face’s push into on-device execution.

OpenAI’s Codex desktop app quietly bundles LibreOffice

Source: simon-willison

Poking through his cache folder, Simon Willison found the Codex (now ChatGPT) desktop app ships a 1.7GB runtime including full Python, Node.js, Poppler, git, and a 430MB headless LibreOffice. Skill files tell Codex how to invoke the bundled binaries for document tasks.

datasette-mcp 0.2 reshapes SQL results for weaker models

Source: simon-willison

The first non-alpha release of datasette-mcp changes execute_sql output from arrays of arrays to arrays of objects, so weaker LLMs stop losing track of which positional value maps to which column. The plugin now requires mcp 2.1.1 or later.

Python 3.15 hits final release candidate ahead of October launch

Source: simon-willison

Release manager Hugo van Kemenade announced Python 3.15.0 RC2, the final candidate before October’s stable release. Only clear bug fixes are allowed from here, and maintainers are urged to publish 3.15 wheels now — binaries built against the RC will work with the final release.

Vibe-coded GeoJSON viewer renders and exports map polygons

Source: simon-willison

Simon Willison published a browser tool for overlaying GeoJSON files on OpenStreetMap and exporting PNGs, built by GPT-5.6-Sol with Claude Code iterations. He used it to visualize California district boundaries that ChatGPT Work extracted from government data sources on request.

Footnotes

1. VentureBeat — https://venturebeat.com/technology/anthropics-claude-fable-5-1-and-mythos-5-1-arrive-with-a-75-cost-reduction-for-fable-cache-reads

   Fable 5.1 uses approximately 1.7 times more output tokens than Fable 5 to solve the same tasks… a 75% reduction in cache-read costs to $0.25 per million tokens

   ↩

2. r/claudexplorers thread on reasoning behavior — https://www.reddit.com/r/claudexplorers/comments/1sho9qf/pretty_sure_i_fixed_claudes_reasoning_can_other/

   sending reasoning:{enabled:false} now triggers a 400 invalid_request_error, as the model requires a thinking block to process tool calls

   ↩

3. BigGo finance recap of Cognition/Millennium case — https://finance.biggo.com/news/40eddb91-1b3c-4e20-a03d-2ca7cffa5a13

   A ‘one-in-a-million’ concurrency bug had remained unexplained for four to five years… Fable 5.1 successfully identified the root cause in a single afternoon [by] disassembling an external vendor library and matching its logic against a core dump

   ↩

4. bpdata.com news aggregator — https://news.bpdata.com/article/anthropics-claude-fable-5-1-boosts-science-scores-and-6cb14383

   swalsh challenged the community, commenting, ‘Now that it’s a solved benchmark, can we get the animated version?’… users noted a technical ‘skill issue’ where the wheels appeared to rotate in the wrong direction

   ↩

5. Hacker News discussion thread — https://news.ycombinator.com/item?id=49530472

   ‘pelicanmaxxing,’ suggesting that AI labs may be overfitting their models on this specific prompt because it is a well-known public test

   ↩

6. awongcm.io 2026 model landscape review — https://www.awongcm.io/blog/2026/07/19/the-2026-ai-model-landscape-for-developers-a-mid-year-deep-dive-across-closed-and-open-models/

   GPT-5.6 Sol—currently priced at a promotional $4/M input and $20/M output—is being adopted as the ‘refined gentleman’ of AI: a more cost-effective workhorse that provides 95% of Fable’s intelligence at a fraction of the operational budget

   ↩

7. NoesisNews — coverage of the Paint.NET/Claude Direct2D work — https://noesisnews.fr/article/claude-direct2d-paint-net-wine

   Claude reportedly achieved more Direct2D progress in three weeks than the WINE community had produced in the twelve years since Paint.NET adopted the API in 2014, including reverse-engineering the math for the built-in effects library.

   ↩

8. Paint.NET forum (Rick Brewster original post) — https://forums.paint.net/topic/134563-%F0%9F%8D%B7-extremely-experimental-winelinux-support-how-to-get-started/

   Requires WINE 11.14+ and DXVK; users must set d3dcompiler_47 to ‘native’ via wine reg or the app crashes; launching with /wine disables UIAnimation and Windows 11 DisplayInformation components.

   ↩

9. Simon Willison — ‘Vibe engineering’ (Oct 2025) — https://simonwillison.net/2025/Oct/7/vibe-engineering/

   Willison draws a line between ‘vibe coding’ (trusting the output without review) and ‘vibe/agentic engineering’ done by professionals — and has since flagged a ‘disturbing realization’ that experts like Brewster are now shipping code volumes they cannot possibly audit, blurring the two categories.

   ↩

10. Cloud Security Alliance — GenAI Code Security research note — https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-generated-code-security-vibe-coding-202/

    Independent testing (Veracode 2025) found AI-generated code introduces security vulnerabilities in ~45% of cases, with no measurable improvement as models get more capable — a direct concern for 180K unreviewed lines handling COM lifetimes and GPU resources.

    ↩

11. dev.to — ‘I reviewed 6 vibe-coded codebases’ — https://dev.to/jsgurujobs/i-reviewed-6-vibe-coded-codebases-here-is-what-i-found-m0o

    Reviewer found vibe-coded projects consistently hit an ‘80/20 wall’ where individual features work but interactions fail in production, and that manual review of a 300K-line AI codebase took roughly 10× longer than the AI’s implementation time.

    ↩

12. PlagiarismToday — ‘Cleanroom as a Service: AI Washing Copyright’ — https://www.plagiarismtoday.com/2026/03/24/cleanroom-as-a-service-ai-washing-copyright/

    Legal scholars question whether a true ‘clean room’ can exist when the implementing agent has already ingested the protected code during training; purely AI-generated output may also fall into a copyright void under Thaler v. Perlmutter, leaving the developer with ‘all the liability and none of the protection.’

    ↩ ↩²

13. Semantic Scholar — py-irt (Lalor & Rodriguez) — https://www.semanticscholar.org/paper/py-irt%3A-A-Scalable-Item-Response-Theory-Library-for-Lalor-Rodriguez/f05f7ce54505220589edc40ddecdd787ef26edb9

    A scalable Item Response Theory library for Python, extending Lalor et al. (2016) and Rodriguez et al. (2021)‘s work showing that high accuracy scores do not always correlate with high latent ability and introducing ‘local efficiency’ metrics for per-item information.

    ↩

14. Liner review of tinyBenchmarks (Polo et al., 2024) — https://liner.com/review/tinybenchmarks-evaluating-llms-with-fewer-examples

    tinyBenchmarks reduces massive benchmarks like MMLU (14k+ items) to curated subsets of just 100 items while maintaining an average error margin of under 2%, using a generalized p-IRT estimator that combines direct sampling with IRT model predictions.

    ↩

15. Center for AI Safety — WMDP benchmark description — https://safe.ai/blog/wmdp-benchmark

    Standard refusal behavior is fragile because it can be bypassed through jailbreaking or adversarial fine-tuning; WMDP-based unlearning (RMU) aims to reduce hazardous-topic scores to random-chance while maintaining high MMLU reasoning scores.

    ↩

16. alphaxiv 2502.15361 — Reasoning amplifies bias in thinking models — https://www.alphaxiv.org/abs/2502.15361

    Generating Chain-of-Thought reasoning can actually amplify social bias; models may initially identify the correct ‘Unknown’ answer but then ‘rationalize’ their way into a stereotypical conclusion through spurious intermediate steps.

    ↩

17. Stanford CRFM — ‘Reliable and Efficient Evaluation’ — https://crfm.stanford.edu/2025/06/04/reliable-and-efficient-evaluation.html

    Classical IRT tools were built for many test-takers and few items, whereas AI benchmarks invert this — few models, many items — causing estimators to become computationally infeasible or produce unreliable ranking inferences when ability distributions are non-normal.

    ↩

18. Renascence — coverage of UK AI Security Institute audit — https://www.renascence.io/news/43433/psychological-methods-reveal-major-weaknesses-in-ai-security-testing

    Models could inflate their ‘safety’ scores simply by refusing a higher volume of requests, a pattern that appears safer in statistical aggregates but does not reflect a genuine underlying trait of harm avoidance.

    ↩