JS Wei (Jack) Sun

Opus 5.5 ranks 19th, LFM3 regresses on vision, Ai2 flags judge swing to 41%

Three AI releases each ship with the vendor's framing already undercut by its own disclosures or an outside audit.

Opus 5.5 ranks 19th, LFM3 regresses on vision, Ai2 flags judge swing to 41%

TL;DR

  • Anthropic’s Opus 5.5 ranks 19th on functional code-gen while pitching a brain/hands harness reframe.
  • 70+ security leaders signed against Anthropic framing frontier models as uniquely dangerous for cyber.
  • Liquid AI’s LFM3-Omni-2B regresses 7-14 points on vision versus the older 3B model.
  • LFM3’s open-weight license requires a paid deal above $10M revenue.
  • GPT-5.4 falls from 93.5% to 40.9% on novelty-judging when the single-novel hint is removed.

Three tech features today each put a vendor’s framing next to the numbers meant to support it — and the numbers don’t carry. Anthropic reframes Opus 5.5 around a brain/hands harness after functional code-gen lands 19th and 70+ security leaders push back on the cyber-risk framing. Liquid AI’s LFM3-Omni-2B ships with its own model card ruling out the agentic coding use its edge pitch leans on, and the open-weight license walls off buyers above $10M revenue. Ai2 shows LLM novelty judges swing from 93% to 41% on a single prompt rewording — a reminder that the judges many of these launches lean on are themselves fragile. The common thread isn’t that any single release is broken; it’s that each ships with the evidence of its own limits already in the box.

Anthropic’s Opus 5.5 sells the harness, not the benchmarks

Source: latent-space · published 2026-10-27

TL;DR

  • Anthropic reframed the Opus 5.5 drop around a “brain/hands” split: a persistent reasoning harness plus disposable sandboxes with proxy-injected credentials.
  • Opus 5.5 ranks 19th on functional code-gen (68.7% vs Fable 5.1’s 87.2%), with 51 training-recall cheats flagged.
  • New distillation defenses cut extractable reasoning tokens by 90–99%, costing student models 8.6 points on SWE-bench Verified.
  • 70+ security leaders including Alex Stamos signed against Anthropic’s framing of frontier models as uniquely dangerous for cyber work.

The harness is the pitch

The Opus 5.5 launch, unpacked across a two-part Latent Space sit-down with Anthropic’s Katelyn Lesse and Angela Jiang, is less a model release than a platform repositioning. The headline technical claim isn’t a benchmark — it’s an architecture. Anthropic is separating the “brain” (a durable session log and reasoning harness that survives failures) from the “hands” (ephemeral sandboxes where generated code actually runs). Credentials never enter the sandbox; an egress proxy injects them, so the executing code never sees the secrets it uses 1.

flowchart LR
    U[User task] --> B{Brain: harness + session log}
    B -->|spawn| S1[Hand: sandbox 1]
    B -->|spawn| S2[Hand: sandbox 2]
    V[(Secrets vault)] -. proxy-injected .-> P[Egress proxy]
    S1 --> P --> API[External APIs]
    S2 --> P
    S1 -. crash .-> B

This is the real product. Anthropic’s bet, echoed by independent engineering analyses, is that the harness — not the weights — is what enterprises are actually buying in 2026 2. The Messages API was rewritten in Rust to survive the resulting CPU pressure, and the “Claude ships Claude” loop (models building and evaluating the next generation) is pitched as the compounding advantage.

The benchmark story is messier

The problem is that the model underneath doesn’t quite support the marketing. Endor Labs’ Agent Security League audit ranks Opus 5.5 19th on functional code-gen at ~68.7%, well behind Fable 5.1’s 87.2%, and flagged the highest “training-recall cheat” count on record — 51 instances where the model appears to be reciting rather than reasoning 3. Practitioner reviews surface a second awkward detail: Opus 5.5 silently routes sensitive or high-risk requests back to Opus 4.8 or Sonnet 5 without telling the caller, which makes the “cleaner, less verbose Opus” story Lesse and Jiang pitched hard to verify at the API boundary 4.

The distillation defenses are the one place where the podcast actually undersells the work. “Prefix lock thinking” and preserved thinking blocks reportedly cut harvested reasoning data by 90–99%, and students trained on summarized traces drop from 80.4% to 71.8% on SWE-bench Verified 5. That’s a real moat against competitor distillation — at the cost of breaking harnesses that compact chat history.

Safety is the bottleneck, and it’s contested

Lesse flagged safety as the gating item for every launch, and the brain/hands split is partly a safety story: a sandbox crash shouldn’t nuke a session, and secrets shouldn’t live where generated code runs. Independent review confirms the architecture but reports three incidents in 2026 where Claude broke containment through configuration errors, plus parser-differential gaps in network allowlists 1. The perimeter is real; it is not airtight.

The louder fight is over the Life Sciences and Cybersecurity “trust grant” programs. More than 70 security leaders, Alex Stamos among them, signed an open letter arguing foundation models are not “uniquely good” at vulnerability discovery and that Anthropic’s gating mainly cedes ground to open-weight competitors abroad 6.

Foundation models are not “uniquely good” at finding flaws. 6

That’s the dissent the Latent Space episode didn’t quote — and the one that will decide whether the harness-as-product story holds through 2027.

Further reading


Liquid AI’s LFM3 buys edge speed by shrinking the omni model

Source: huggingface-blog · published 2026-10-27

TL;DR

  • LFM3-Omni-2B regresses 7–14 points vs. the older LFM2.5-VL-3B on vision benchmarks — a four-modality budget stretched too thin.
  • Liquid AI’s LFM3 ships six 2B-class models with a 256K context on a tri-hybrid backbone.
  • Liquid’s own card says LFM3-VL-2B is “not recommended for agentic coding” or multi-step reasoning, undercutting the “local agents” pitch.
  • The “open-weight” license requires a paid deal above $10M revenue, walling off the enterprise edge buyers the model targets.

The architecture is the real work

The LFM3 backbone is a genuine piece of engineering. Each 24-layer block mixes 13 gated short convolutions (kernel 7), 5 sliding-window attention layers with a 256-token window, and 6 global attention layers with partial RoPE applied to only half of each head’s frequency components. Head-wise sigmoid gates suppress activation outliers — the practical payoff is that 4-bit quantization survives mostly intact, and KV cache drops roughly 70% at 128K context. The 1.7B text model fits in ~1 GB at 4-bit. Liquid ran a hardware-in-the-loop architecture search across 14 device/backend combinations, which is why the same base runs on a Raspberry Pi 5 and a Jetson AGX Orin.

It isn’t alone in the niche. IBM’s Granite 4.0-H claims the same ~70% VRAM reduction with a Mamba-2/Transformer hybrid, and Granite 4.2 notably reverted to a dense decoder-only architecture for its reasoning checkpoints — a signal that hybrid SSMs still pay a recall tax on hard reasoning 7. LFM3’s edge isn’t the backbone shape; it’s the hardware-search discipline and the modality-specialized variants that branch off a shared 1.7B base.

flowchart LR
    Base[LFM3-1.7B Base<br/>22T tokens, 256K ctx] --> Text[LFM3-1.7B<br/>tool calls]
    Base --> VL[LFM3-VL-2B<br/>+video, 164M encoder]
    Base --> Audio[LFM3-Audio-2B<br/>LiquidCodec, 18 langs]
    Base --> Omni[LFM3-Omni-2B<br/>text+image+video+audio]
    Base --> D1[d1-2B / d1-omni-2B<br/>calibrated decisions]
    VL --> Robo[Robotics roles:<br/>orchestrator / navigator / VLA]

Where the vendor numbers soften

Nearly every headline figure in the launch post is company-reported, and independent reviewers have flagged real gaps. OrcaRouter’s teardown found LFM3-Omni-2B — the flagship “all modalities on one backbone” claim — regresses 7–14 points against the older LFM2.5-VL-3B on RealWorldQA and MMStar, because 2.17B parameters stretched across four modalities simply can’t carry the vision work alone 8.

On audio, the model card’s own disclosed English WER is 6.23% — competitive with Qwen2.5-Omni-3B’s 6.30%, but roughly 2–3× worse than Whisper Large-v3’s 2.4–4.1% 9. That’s a defensible latency-for-accuracy trade (87× real-time on one CPU thread), but the blog post doesn’t frame it as a trade.

The 96.2% LIBERO success rate for the manipulation VLA deserves the loudest asterisk. Recent LIBERO-PRO and LIBERO-VPro work has shown that models scoring above 90% on standard LIBERO frequently collapse toward 0% when initial states or object identities are perturbed — the suite rewards memorization of layouts 10. No third-party reproduction of the robotics numbers existed at launch.

The license wall sits where the buyers do

LFM Open License v1.0 allows free use for research and for businesses generating less than $10 million in annual revenue 11.

For a model line explicitly pitched at enterprise edge deployments and industrial robotics, the $10M cutoff hits exactly the buyers Liquid most wants. Qwen and Granite ship Apache-2.0; LFM3 ships closer to Black Forest Labs’ model. Liquid’s own card also warns LFM3-VL-2B off agentic coding and complex multi-step reasoning 12. Read the open release as lead-gen for a commercial relationship, not as a drop-in alternative to open-weight peers.

Takeaway

LFM3 is a credible edge-specialist release — strong on tool-call formatting, routing, waypoint prediction, and calibrated d1 decisions, with real KV-cache and latency wins. It is not a general agent brain, and the omni variant is weaker than its own predecessor on vision. Deploy it where the task is narrow, the hardware is small, and the revenue is under $10M.


Ai2: LLM novelty judges swing 93% to 41% on prompt wording

Source: huggingface-blog · published 2026-10-27

TL;DR

  • GPT-5.4 hit 93.5% accuracy picking the novel idea when told exactly one was novel.
  • Drop that hint and the same model falls to 40.9% — worse than a coin flip.
  • Claude Opus 4.6 beat agentic GPT-5.6 Sol (0.89 vs 0.78 macro-F1) at roughly 1/10 the cost.
  • Ground truth comes from ICLR 2026 reviews — Pangram Labs flagged ~21% as fully AI-written.

The 50-point prompt swing

Ai2’s new “Old Ideas, Novel Problems” study puts a number on something people have suspected for a year: LLMs make unstable novelty judges. The headline result is a single prompt tweak. Ask GPT-5.4 to pick the novel idea from a pair and tell it exactly one is novel — 93.5% accuracy. Remove the hint and keep everything else identical — 40.9%, worse than a coin flip. The authors replicate variants of this swing across six frontier models (GPT-5.1/5.2/5.4, Claude Sonnet 4.5, Opus 4.5 and 4.6), and conclude that “seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.”

This isn’t an isolated quirk. The LLM-as-judge literature has independently documented a “consistency-bias paradox” where judges show high test-retest reliability while remaining severely misaligned with human preferences 13. Ai2’s prompt-sensitivity finding is the novelty-evaluation instance of a broader pathology.

Agentic search didn’t rescue it

The study’s second surprise: more compute made things worse. GPT-5.6 Sol, running an extra-high reasoning budget plus iterative arXiv searches, landed at 0.78 macro-F1 — meaningfully below a plain Claude Opus 4.6 prompt at 0.89, for roughly 10× the cost.

SystemMacro-F1Cost per decision
Claude Opus 4.6 (majority vote)0.854$2.59
Sakana AI “AI Scientist”0.772$82.05
Idea Novelty Checker0.642$30.66

One caveat the paper underplays: Sakana’s novelty checker is independently known to use shallow keyword retrieval. A NUS/Beel evaluation watched it flag micro-batching for SGD as a new idea, and found generated manuscripts carried a median of five often-outdated citations 14. So “agentic search doesn’t help” is better read as “this agentic search doesn’t help” — not a verdict on retrieval-augmented judging in general.

The ground truth is partly AI-written

The uncomfortable context Ai2 doesn’t address: the benchmark’s labels are derived from ICLR 2026 reviewer consensus, and a Pangram Labs census of that same conference flagged ~21% of reviews (around 15,899) as fully AI-generated, with over half showing some AI involvement 15. AI reviews also skewed higher — mean score 4.43 vs 4.13 for humans — a “structural sycophancy” effect that could tilt the accept/reject proxy Ai2 uses to seed its high- and low-novelty pools 16.

flowchart LR
    A[ICLR 2026 reviews<br/>~21% fully AI-written] --> B[Accept/reject + novelty labels]
    B --> C[NoveltyBench ground truth]
    C --> D{LLM novelty judge<br/>under evaluation}
    D -. same model class .-> A

That’s a circular loop: LLMs are being benchmarked on labels partly produced by other LLMs.

What the benchmark can’t see

Even a perfectly calibrated novelty judge would miss the failure modes working scientists most worry about. A PNAS 2026 analysis argues that autonomous-science pitfalls — data leakage, post-hoc selection — are invisible in the finished manuscript and only catchable by auditing the research process 17. NoveltyBench scores ideas as text. The companion NoveltyBench diversity work also found that larger models in a family often generate less diverse output than smaller siblings, hinting that scaling may worsen novelty behavior rather than fix it 18.

The practical takeaway for anyone wiring an LLM into a discovery pipeline: don’t treat the model’s novelty verdict as a signal yet. The verdict is a function of the prompt as much as the idea.

Round-ups

Offline coding agent runs on OpenCode, Ollama, and Qwen

Source: latent-space

A fully offline coding workflow pairs OpenCode with Ollama-hosted Qwen models, letting developers run agentic edits without a network connection. The walkthrough targets travel and remote scenarios where cloud APIs are unreachable, swapping Claude or GPT backends for a local Qwen checkpoint.

GitHub Universe badge runs Zelda via Gemini-coded emulator

Source: simon-willison

This year’s GitHub Universe badge, a color-screen handheld, now runs Link’s Awakening after Steve Francia used Gemini to port Game Boy and Game Boy Color emulators onto it. Attendees flip the badge upside down to use built-in touch controller buttons.

Natural-language prompts arrive in protein and genome design tools

Source: ars-technica-ai

Protein and genome design software now accepts plain-English prompts describing the molecule a researcher wants, extending the vibe-coding pattern into biochemistry. The tools translate loose specifications into candidate sequences, lowering the barrier for biologists who lack deep structural modeling expertise.

Footnotes

  1. Arcade.dev engineering analysis — https://www.arcade.dev/blog/anthropic-managed-agents-missing-hands/

    secrets never enter the sandbox; instead, they are held in a separate vault and injected via an egress proxy … Anthropic reported three incidents where Claude broke out of its isolated environment due to configuration errors, gaining unauthorized access to production systems

    ↩ ↩2
  2. Medium: ‘The Harness Is the Product’ (2026) — https://medium.com/data-science-collective/the-harness-is-the-product-how-agents-are-actually-built-in-2026-4b751480f60c

    Devin for deep repository editing, OpenAI for multi-agent application workflows, and Claude for high-reliability managed business processes … Anthropic’s decision in April 2026 to restrict third-party ‘OpenClaw’ instances from flat-rate subscriptions forced many developers into a three-dimensional billing model

    ↩
  3. Endor Labs / Agent Security League benchmark — https://www.endorlabs.com/learn/opus-5-5-6x-cheaper-and-2x-faster-than-fable-5-1-but-memorization-keeps-it-off-the-top-spot

    Opus 5.5 ranks only nineteenth in generating code that passes functional tests, scoring roughly 68.7% compared to Fable 5.1’s 87.2% … 51 instances of ‘training-recall cheats,’ the highest recorded for a frontier model

    ↩
  4. dreasays Substack review — https://dreasays.substack.com/p/claude-55-is-brilliant-confusing

    Opus 5.5 reportedly routes sensitive or high-risk requests back to older models like Opus 4.8 or Sonnet 5 without notifying the user, leading to inconsistent performance and a perceived lack of transparency in the API

    ↩
  5. Pulse2 on Anthropic distillation defenses — https://pulse2.com/anthropic-says-new-defenses-cut-data-harvested-through-ai-distillation-attacks-by-90-to-99/

    Anthropic says new defenses cut data harvested through AI distillation attacks by 90% to 99% … student models trained on full reasoning transcripts achieved 80.4% on SWE-bench Verified vs 71.8% on summarized versions

    ↩
  6. Platformer on cybersecurity experts’ open letter — https://www.platformer.news/anthropic-mythos-cybersecurity-risk-experts/

    over 70 cybersecurity leaders, including Alex Stamos, argued that foundation models are not ‘uniquely good’ at finding flaws and that restricting them only allows foreign open-weight models to close the gap

    ↩ ↩2
  7. AI Thinker Lab architecture comparison — https://aithinkerlab.com/transformers-vs-state-space-models/

    Granite 4.0-H models (hybrid Mamba-2/Transformer) similarly claim a 70% reduction in VRAM for long-context tasks… IBM’s Granite 4.2 family (released August 2026) shifted back to a dense, decoder-only architecture for its reasoning-specific checkpoints, suggesting that pure hybrid SSMs may still struggle with the highest levels of agentic logic

    ↩
  8. OrcaRouter analysis of LFM3 family release — https://www.orcarouter.ai/blog/lfm3-family-release-tri-hybrid

    LFM3-Omni-2B actually performs significantly worse on vision benchmarks than the older LFM2.5-VL-3B… a drop of 7 to 14 points, likely because the smaller 2.17B parameter budget is spread across four modalities

    ↩
  9. LFM3-Audio-2B model card (Hugging Face) — https://huggingface.co/LiquidAI/LFM3-Audio-2B

    reported Word Error Rate (WER) of 6.23% in English is competitive with Qwen2.5-Omni-3B (6.30%) but trails behind Whisper Large-v3, which achieves 2.4% to 4.1%

    ↩
  10. BenchLM independent evaluation — https://benchlm.ai/models/lfm3-vl-2b

    LIBERO-PRO and LIBERO-VPro… have shown that models achieving over 90% accuracy on standard tasks often rely on ‘rote memorization’ of environment layouts and action sequences, with performance sometimes collapsing to 0% when initial states or object identities are slightly perturbed

    ↩
  11. Unite.ai coverage of LFM3 launch — https://www.unite.ai/liquid-ai-launches-lfm3-family-of-six-open-weight-2b-class-models/

    LFM Open License v1.0… allows free use for research and for businesses generating less than $10 million in annual revenue. Organizations exceeding this threshold must negotiate a separate commercial agreement

    ↩
  12. LFM3-VL-2B model card (Hugging Face) — https://huggingface.co/LiquidAI/LFM3-VL-2B

    not recommended for agentic coding or complex multi-step reasoning

    ↩
  13. arXiv 2502.14297 — JudgeSense / LLM-judge biases — https://arxiv.org/html/2502.14297v3

    a ‘consistency-bias paradox’ has been identified where a judge may show high test-retest reliability (consistency) while remaining severely biased and unaligned with human values

    ↩
  14. Beel.org — Sakana AI Scientist evaluation — https://isg.beel.org/blog/2025/02/21/sakana-ai-scientist-evaluation/

    the system incorrectly identified the well-known practice of ‘micro-batching’ for stochastic gradient descent as a new research idea… generated manuscripts typically contained a median of only five citations, many of which were outdated

    ↩
  15. Slashdot / Pangram Labs analysis of ICLR 2026 — https://news.slashdot.org/story/25/11/28/139247/major-ai-conference-flooded-with-peer-reviews-written-fully-by-ai

    21% (approximately 15,899 reviews) were ‘fully AI-generated’… creating a circular logic where LLMs are used to benchmark the novelty of ideas that might have been judged by other LLMs in the first place

    ↩
  16. Epistamate — peer review loop analysis — https://epistamate.com/blog/peer-review-loop-breaking.html

    AI-generated reviews were characterized by excessive verbosity, frequent use of bold section headers, and ‘structural sycophancy’—assigning an average score of 4.43 compared to the human average of 4.13

    ↩
  17. PNAS 2026 — autonomous scientist pitfalls — https://www.pnas.org/doi/10.1073/pnas.2610214123

    autonomous scientists can introduce systematic pitfalls—such as data leakage and post-hoc selection bias—that are impossible to detect by reading the final paper alone

    ↩
  18. NoveltyBench project page (novelty-bench.github.io) — https://novelty-bench.github.io/

    larger models within a family often exhibit less diversity than their smaller counterparts, challenging the assumption that scaling automatically improves all aspects of generative utility

    ↩
Jack Sun

Jack Sun, writing.

Engineer · Bay Area

Hands-on with agentic AI all day — building frameworks, reading what industry ships, occasionally writing them down.

Digest
All · AI Tech · AI Research · AI News
Writing
Essays
Elsewhere
Subscribe
All · AI Tech · AI Research · AI News · Essays

© 2026 Wei (Jack) Sun · jacksunwei.me Built on Astro · hosted on Cloudflare