smolvm clears sandbox tests, LFM2.5 keeps 97% at 4-bit, Willison vs METR

TL;DR

• smolvm 1.8.3 posts 0.6-1.5s cold starts on Firecracker, behind Daytona’s sub-90ms lead.
• Liquid AI’s 4-bit LFM2.5 keeps 96.5-97.4% of BF16 accuracy across 230M-2.6B sizes.
• Willison pegs agent-assisted senior devs at ~1,000 lines a day, roughly 5× the pre-AI baseline.
• METR’s 2025 RCT clocks experienced developers 19% slower with AI on real tasks.
• DORA 2025 ties a 25% rise in AI use to a 7.2% drop in delivery stability.

Today’s tech pool is three developer-facing drops with no shared vendor: Claude Fable 5’s validation of the smolvm agent sandbox on Firecracker, Liquid AI’s 4-bit LFM2.5 checkpoints at four sizes, and Simon Willison’s argument that agent-assisted senior devs ship roughly 1,000 lines of code a day.

The third drop is where the day gets loud. METR’s 2025 RCT clocked experienced developers 19% slower with AI on real tasks — while those same devs predicted a 24% speedup — and DORA 2025 tied a 25% rise in AI use to a 7.2% drop in delivery stability. Willison’s number and the field data are pointing in opposite directions, and the smolvm and LFM2.5 pieces sit alongside as the concrete infrastructure the disagreement is actually about.

smolvm clears sandbox tests, but mount granularity bites

Source: simon-willison · published 2026-08-19

TL;DR

• Claude Fable 5 validated smolvm 1.8.3 at 0.6–1.5s cold / ~50ms warm starts on Firecracker.
• Independent benchmarks rank smolvm behind Daytona’s sub-90ms leader, near E2B’s 150–717ms range.
• Mounting at /workspace silently wipes the internal storage disk — a real footgun for the “designated files” pattern.
• The Plan B pivot to GitHub Actions runners rides on undocumented KVM support with no SLA.

What actually got tested

Simon Willison handed Claude Fable 5 a concrete brief: put smolmachines through its paces as a fast, secure sandbox for untrusted Python and JavaScript, with RAM/CPU caps, no network, and filesystem access limited to designated files. Fable’s own environment couldn’t run it — the Claude Code for web container is itself a Firecracker guest with no /dev/kvm — so the agent stood up a temporary GitHub Actions workflow, ran the test battery on an Ubuntu runner, and tore the workflow down in the final commit. Willison’s framing is that this is Fable being “relentlessly proactive.” That’s true. It’s also worth reading what the tests actually prove, and what they don’t.

Where smolvm sits in the field

smolvm is one of a cohort of developer-first microVM runtimes chasing the AI-agent code-execution niche. Fable’s measured numbers — 0.6–1.5s cold, ~50ms warm — land it in E2B’s neighborhood (150–717ms create times on Firecracker) and well ahead of Modal’s ~2.4s, but behind Daytona’s optimized sub-90ms cold starts ¹. An independent April review clocked smolvm at 200–500ms cold, still 2–4× faster than plain Docker on macOS ². The architectural argument for choosing Firecracker over a syscall-interception approach like gVisor holds up: ~5 MiB per-instance overhead versus 50–100 MiB for gVisor’s Sentry, and none of the 20–40% syscall tax on I/O-heavy workloads like package installs ³. For agents that spin up a fresh VM per task, that overhead compounds fast.

The caveats Willison’s post skips

The happy-path guarantees — hardware isolation, no-network default, timeout enforcement — check out. The footguns are in smolmachines’ own security docs. Mounts are directory-granular only; you cannot mount a single file, which is exactly the pattern the brief asked for. Worse, mounting a host directory at the default /workspace path overrides the internal storage-disk workspace entirely, a silent data-loss trap for anyone reaching for the obvious ergonomics ⁴. The HN launch thread (~480 points) piled on other rough edges: curl | bash install, CLI glitches with backspace and network symbols on ARM64 macOS, and libkrun’s virtio-fs as the plausible weak seam pending private mount namespaces ⁵.

The Plan B is more fragile than it reads

GitHub Actions runners do expose /dev/kvm on standard x64 Linux SKUs, but that capability is officially undocumented, carries no SLA, and is absent on ARM and macOS runners entirely ⁶. Community benchmarks also report meaningful slowdowns under nested virtualization versus bare metal, which means the 0.6–1.5s cold numbers are a soft upper bound rather than a floor. Anyone copying the reproduction recipe is depending on Azure runner behavior that could change without notice.

Takeaway

smolvm is a legitimate entrant, not a category-definer. Its real differentiation is macOS ergonomics and single-binary portability, not raw speed or novel isolation. If you’re evaluating it for a “run this untrusted transform against these three files” workflow, budget time for the mount-granularity gap before the sandbox itself.

---

Liquid AI’s 4-bit LFM2.5 keeps 97% of BF16 accuracy

Source: huggingface-blog · published 2026-08-19

TL;DR

• Liquid AI shipped Q4_0 GGUF checkpoints for LFM2.5 at 230M, 350M, 1.2B and 2.6B, recovering 96.5–97.4% of BF16 accuracy.
• The recipe is quantization-aware distillation: a 4-bit student trained against a BF16 teacher with tempered Top-K KL divergence.
• Decode throughput runs 4–33% faster on the small models and 3–14% faster on the 1.2B/2.6B.
• Ships as a static Q4_0 with no imatrix, skipping the dynamic bit-allocation that peers rely on.

What QAD actually is

Quantization-Aware Distillation is QAT with a teacher. Liquid trains the 4-bit student directly against the BF16 model using a tempered, decoupled Top-K objective that minimizes KL divergence between the two output distributions, with offline logit caching so you don’t have to hold both models in VRAM at once ⁷. This is the same shape as the recipe NVIDIA has been publishing for NVFP4 — QAT-plus-distillation clears 90% baseline recovery at 4-bit and is more stable than either post-training quantization or vanilla QAT on multi-stage pipelines ⁸. Liquid’s 96–97% numbers are therefore not an outlier claim; they’re roughly where the state of the art has landed.

The comparison the blog post glosses

The eye-catching line is that QAD Q4_0 matches Unsloth’s UD-Q4_K_XL on reasoning and instruction-following. That framing deserves an asterisk. When Unsloth ran the analogous experiment on Google’s Gemma QAT release, a naive 4-bit conversion of the QAT weights scored only ~70% top-1; it took Unsloth’s dynamic mixed-precision layer selection to push it to 85.6% — a 15.6-point recovery on top of the QAT training itself ⁹. That is, most of what the ecosystem calls “recovered accuracy” comes from dynamic bit allocation, not from the training objective alone.

Independent analysis has made the sharper version of the point: K-quants like Q4_K_M can beat uniform 4-bit QAT precisely because they keep embeddings and early attention layers at higher precision, whereas a strictly uniform 4-bit budget struggles with long-context coherence ¹⁰. Liquid shipped Q4_0 as a static quant with no importance matrix ¹¹, which means the fair comparison is QAD vs. plain PTQ Q4_0 — a weak baseline — not QAD vs. the best K-quant pipeline available today.

Practitioner reality and a security wrinkle

Community testers roughly agree with Liquid’s own positioning: the 1.2B and 2.6B are competent “worker agent” models for extraction and routing, but the 2.6B has been observed failing complex tool-calling flows — confused working-directory state, brittle system-prompt adherence ¹¹. Read the BFCLv4 numbers with that in mind.

Separately, the broader GGUF-Q4_0 surface just got riskier. The ICML 2025 “Mind the Gap” paper demonstrated a practical backdoor that appears benign in FP16 and only activates after Q4_0 rounding — the quantization error itself provides enough mathematical slack to hide the trigger ¹². QAD doesn’t cause this. But it lands weights into an ecosystem where “the quantized checkpoint behaves like the full-precision one” is no longer a free assumption.

Net

QAD is a real, well-motivated technique, and the recovery numbers are in line with what NVIDIA and Unsloth report for the same class of method. The interesting gap isn’t the training recipe — it’s that Liquid stopped short of the dynamic imatrix layer that peers use to close the last few points. A UD-style variant of these checkpoints would be a more honest bar for the “matches Q4_K_M” claim.

---

Willison’s 1,000-line day meets METR’s 19% slowdown

Source: simon-willison · published 2026-08-19

TL;DR

• Willison pegs agent-assisted output at ~1,000 lines/day, vs. 50-200 in the pre-AI era, gated on senior-level review skill.
• METR’s 2025 RCT found the opposite direction: experienced devs took 19% longer with AI while predicting a 24% speedup.
• GitClear logged a ~8× jump in duplicated code blocks in 2024 as refactoring hit historic lows — the “weird bumps” quantified.
• DORA 2025 tied a 25% rise in AI use to a 7.2% drop in delivery stability.

The 1,000-line day

On the Talking Postgres podcast, Simon Willison made a claim that will annoy anyone who has ever taught junior engineers not to measure productivity in lines of code: he thinks LOC is a defensible metric again. His numbers — 50-60 lines on a normal pre-AI day, 200 on a great one, ~1,000 with a competent agent driver — are meant to argue that the throughput shift is large enough to be worth naming, provided the code stays “maintainable, tested, all of that.”

The independent data is unkind to the net productivity read. METR’s 2025 randomized trial of experienced open-source contributors found AI users took 19% longer on real tasks while predicting a 24% speedup ¹³. Swarmia’s telemetry does confirm the raw volume story — median PR batch size roughly doubled between Q1 2025 and Q1 2026 as agentic tools went mainstream ¹⁴ — but batch size measures code produced, not value shipped. DORA’s 2025 report tightens the vice: a 25% increase in AI use correlated with a 7.2% decline in delivery stability, and roughly a third of developers say they don’t trust the code they’re merging ¹⁵.

Willison’s direction — volume up — holds. His implied magnitude of useful throughput does not.

The Winchester Mystery House has receipts

The stronger half of the argument is the one Willison and co-host Claire Giordano frame with Fred Brooks’ “conceptual integrity” and the Winchester Mystery House — 140 rooms bolted on with no master plan. When adding a feature costs an hour instead of a week, the natural friction that used to filter out bad ideas disappears, and software grows “weird bumps in funny different directions.”

This is the part of the podcast with hard numbers behind it. GitClear’s analysis of 150M+ lines of change found duplicated code blocks rose nearly eightfold in 2024 while refactoring activity dropped to historic lows ¹⁶ — a direct measurement of the sprawl pattern. A separate analysis notes a “brownfield barrier” around 100K LOC, where agents start “chasing their own tails” because they can no longer navigate the systems they helped write ¹⁷. The house doesn’t just get sprawling; it becomes uninhabitable to the contractor.

Cognitive capacity is the real ceiling

Willison’s cleanest insight is that cognitive capacity — not typing speed — is why you still need a team. Practitioners running his workflow agree in a way that undercuts his throughput math. One developer’s 50-day Claude Code journal describes running four parallel agents as “mentally exhausting,” leaving him “wiped out by noon” ¹⁸. If a senior can supervise agents for half a day before decision fatigue sets in, the effective multiplier is nowhere near 100×. Add DORA’s trust gap ¹⁵ — reviewers auditing code they distrust burn more attention than authors writing from scratch — and the discipline Willison prescribes starts looking less like a virtue and more like the actual bottleneck.

The takeaway isn’t that Willison is wrong about the shift. It’s that “1,000 lines/day” is the wrong scoreboard for the game he’s describing.

Round-ups

LLMs revive user-extensible software via cheap sandboxed plugins

Source: simon-willison

Jeremy Morrell argues web apps should ship a solid core and let users extend it with LLM-authored add-ons. Modern browser sandbox primitives handle the security boundary, while LLMs slash authoring cost — reopening a design space that died with the plugin era.

Footnotes

1. Blaxel — Best cloud sandboxes for AI agents 2026 — https://blaxel.ai/blog/best-cloud-sandboxes-ai-agents-2026

   Daytona leads in raw speed with optimized configurations hitting sub-90ms startup times, while E2B follows closely with a measured 150ms to 717ms create time… Modal typically shows higher cold start latencies of roughly 2.4 seconds.

   ↩

2. mywrittenword.com — smolvm vs E2B comparison — https://mywrittenword.com/2026/04/26/smolvm-firecracker-microvm-ai-agent-sandbox-e2b-comparison-2026/

   Cold boot times typically range between 200ms and 500ms depending on the host environment… significantly faster than the 500ms–2s range often seen with standard Docker containers on macOS.

   ↩

3. Northflank — Firecracker vs gVisor — https://northflank.com/blog/firecracker-vs-gvisor

   Firecracker’s memory overhead is remarkably low, often cited at ~5 MiB per instance… gVisor’s Sentry process requires between 50 MiB and 100 MiB per sandbox… syscall interception slowing heavy file and network operations by 20% to 40% or more.

   ↩

4. Celesto.ai smolvm security docs — https://docs.celesto.ai/smolvm/concepts/security

   SmolVM currently supports mounting entire directories only, rather than individual files… mounting a host directory at the default /workspace path will override the internal storage-disk workspace entirely, which can lead to data loss or configuration conflicts.

   ↩

5. Hacker News Show HN thread (item 47808268) — https://news.ycombinator.com/item?id=47808268

   Critics on Hacker News raised concerns about the installation method — specifically the use of ‘piping to bash’ for the install script — and noted bugs in the CLI’s handling of terminal backspaces and network symbols on ARM64 macOS.

   ↩

6. GitHub issue — KVM availability on GitHub Actions runners — https://github.com/randax/talos-box/issues/74

   GitHub does not provide a Service Level Agreement (SLA) for general KVM usage, effectively treating it as an ‘as-is’ feature that could be restricted without notice… ARM-based runners and macOS runners remain significant gaps in support.

   ↩

7. Liquid AI blog (QAD post) — https://www.liquid.ai/blog/qad

   QAD distills a high-precision teacher directly into a 4-bit student, using a tempered, decoupled Top-K knowledge distillation objective that minimizes KL divergence between teacher and student distributions.

   ↩

8. NVIDIA developer blog — QAT/NVFP4 — https://developer.nvidia.com/blog/how-quantization-aware-training-enables-low-precision-accuracy-recovery/

   Quantization-aware training with distillation recovers over 90% of baseline accuracy at 4-bit, outperforming both PTQ and vanilla QAT in stability for complex multi-stage models.

   ↩

9. Kaitchup Substack — Unsloth’s QAT writeup — https://kaitchup.substack.com/p/unsloths-quantization-aware-training

   A naive conversion of the Gemma QAT checkpoint yielded only ~70% top-1 accuracy; Unsloth Dynamic optimization pushed it to 85.6%, a +15.6 point recovery over the base QAT release.

   ↩

10. Kaitchup Substack — weekly roundup — https://kaitchup.substack.com/p/this-week-arcee-trinity-and-quantization

    K-quants like Q4_K_M can outperform native QAT checkpoints because they preserve sensitive layers (embeddings, early attention) at higher precision, whereas uniform 4-bit QAT can struggle with long-context coherence.

    ↩

11. HuggingFace LFM2.5-350M-GGUF repo / LocalLLaMA discussion — https://huggingface.co/LiquidAI/LFM2.5-350M-GGUF

    The QAD GGUF was released as a static quant with no importance matrix, and independent testers reported the 2.6B struggled with complex tool calls, occasionally becoming confused about working directories or system prompts.

    ↩ ↩²

12. arXiv 2505.23786 — ‘Mind the Gap’ GGUF backdoor — https://arxiv.org/pdf/2505.23786

    A model can be trained to appear benign in FP16 but exhibit malicious behavior only after quantization to GGUF; the quantization error provides enough mathematical flexibility to hide triggers activated by Q4_0 rounding.

    ↩

13. r/ExperiencedDevs summarizing METR 2025 RCT — https://www.reddit.com/r/ExperiencedDevs/comments/1rnkv2t/the_ai_coding_productivity_data_is_in_and_its_not/

    developers using AI actually saw a 19% increase in task completion time compared to those working manually… they had predicted AI would save them 24% of their time

    ↩ ↩²

14. Swarmia productivity impact study — https://www.swarmia.com/blog/productivity-impact-of-ai-coding-tools/

    median batch size roughly doubled (growing 97.5% to 109%) between Q1 2025 and Q1 2026 as agentic AI adoption became mainstream

    ↩ ↩²

15. DevOps.com on DORA 2025 report — https://devops.com/dora-2025-faster-but-are-we-any-better/

    a 25% increase in AI use was linked to a 7.2% decline in delivery stability… roughly one-third of developers do not trust the code generated by AI

    ↩ ↩² ↩³

16. GitClear ‘Coding on Copilot’ report — https://www.gitclear.com/coding_on_copilot_data_shows_ais_downward_pressure_on_code_quality

    the prevalence of duplicated code blocks rose nearly eightfold in 2024, while refactoring activity dropped to historic lows

    ↩ ↩²

17. Votee.ai ‘The Mythical Agent-Minute’ — https://votee.ai/resources/the-mythical-agent-minute

    as projects reach approximately 100,000 lines of code, AI agents begin ‘chasing their own tails’ because they struggle to navigate the intricate logic of existing systems

    ↩

18. dev.to ‘50 days of Claude Code’ journal — https://dev.to/yurukusa/i-tracked-my-claude-code-usage-50-days-straight-my-hip-flexors-have-opinions-f34

    running four agents simultaneously… is ‘mentally exhausting,’ often leaving him ‘wiped out’ by noon

    ↩