NeMo triples MoE throughput, meta-harnesses lift Qwen3.5 44%, Cohere leads FFASR

TL;DR

• NVIDIA’s NeMo AutoModel posts 3.4-3.7× MoE fine-tune throughput vs HF Transformers v5 on 8×H100.
• CVE-2025-33236 hits NeMo at CVSS 7.8: hardcoded trust_remote_code=True enables silent RCE from Hub checkpoints.
• Stanford’s Meta-Harness lifts Claude Opus 4.6 to 76.4% on TerminalBench-2, beating hand-tuned Terminus-KIRA.
• Xiaomi’s HarnessX adds 44% to Qwen3.5 on embodied planning and 14.5% averaged across 15 benchmarks.
• Cohere Transcribe tops the new FFASR leaderboard at 17.9% WER, ahead of IBM Granite and NVIDIA Parakeet.

Today’s three AI-tech leads sit on different layers of the stack with no shared thread. NVIDIA’s NeMo AutoModel posts a 3.4× MoE fine-tune throughput jump over HF Transformers v5 — paired with a CVSS 7.8 RCE from a hardcoded trust_remote_code=True in its importers, plus checkpoint failures on GB200 and MXFP8 memory bloat. Stanford and Xiaomi push the meta-harness layer as the real control plane for agent performance, with Claude Opus 4.6 hitting 76.4% on TerminalBench-2 and small models like Qwen3.5 gaining the most from the scaffold rewrite. And Cohere Transcribe takes the top slot on Treble’s brand-new far-field ASR leaderboard at 17.9% WER, on a simulated-room eval whose own users question the wave-simulator’s sim-to-real fidelity.

Each is a vendor or lab planting a flag on a layer that wasn’t seriously measured a quarter ago — training scaffolding, agent scaffolding, evaluation scaffolding — even if today’s news doesn’t connect them.

NVIDIA NeMo AutoModel triples MoE fine-tune throughput

Source: huggingface-blog · published 2026-06-24

TL;DR

• NeMo AutoModel posts 3.4–3.7× throughput and 29–32% lower peak memory vs HF Transformers v5 on 8×H100.
• CVE-2025-33236 (CVSS 7.8) — NeMo hardcodes trust_remote_code=True in importers, enabling silent RCE from a malicious Hub checkpoint.
• The baseline is “vanilla” HF v5, not Unsloth, which fits a Qwen3-30B LoRA in 17.5 GB on one RTX 5090.
• Release notes flag checkpoint failures on GB200, StarCoder2 LayerNorm bugs, and MXFP8 memory bloat — not production-ready on Blackwell.

What’s actually fast

The engineering is real. Sharding Mixture-of-Experts weights with Expert Parallelism (EP=8) drops the per-GPU expert footprint from ~55 GiB to ~6.8 GiB on a 30B model, which is the move that turns Qwen3-30B-A3B fine-tuning from 68.2 GiB peak to 48.1 GiB. DeepEP fuses all-to-all token routing with expert compute, shaving up to 47% off iteration cost on DeepSeek V3-scale models. TransformerEngine handles fused attention, linear, and RMSNorm. The frontier flex — full fine-tuning the 550B Nemotron 3 Ultra across 128 H100s where vanilla Transformers OOMs — is genuinely new headroom.

The drop-in API is the other half of the story: swap AutoModelForCausalLM for NeMoAutoModelForCausalLM and checkpoints stay compatible with vLLM and SGLang. That ergonomic surface is the actual product.

The benchmark framing deserves a second look

Independent analysis flags that NVIDIA’s 3.7× is measured against unoptimized HF Transformers v5, with no Kernel Hub or Triton fusions in the baseline ¹. The chart effectively moves the conversation from model architecture to NVIDIA’s execution layer — and once a team builds on that layer, leaving it is costly ¹. A like-for-like read across the fine-tuning landscape:

AutoModel’s edge is multi-node linear scaling with HF compatibility, not single-GPU efficiency. NVIDIA’s own Megatron-Bridge docs steer the largest jobs toward Parallel Folding, which decouples attention and expert parallelism in ways AutoModel does not ³.

Security and stability gotchas

The friction-free importer is also the attack surface. Cato CTRL’s CVE-2025-33236 (CVSS 7.8) reports that NeMo hardcodes trust_remote_code=True across multiple model loaders, bypassing the safety prompt HF users rely on and enabling silent code execution from a poisoned Hub checkpoint ⁴. The same ergonomics that sell the drop-in API are what defeat the guardrail.

NVIDIA’s release notes confirm the library is pre-stable: checkpoint save/load broken on GB200, LayerNorm-with-bias accuracy regressions on StarCoder2, and MXFP8 memory bloat under investigation ⁵. The 550B Nemotron Ultra demo is impressive on paper — ~2,922 GB of BF16 optimizer state on 128 H100s — but community testers already report results “highly sensitive to quantization backend and chat template” ⁶.

Takeaway

If you train MoE models on 8+ H100s and live inside the HF API, AutoModel is a real throughput unlock worth piloting. Pin a version, audit the importers before pointing them at public checkpoints, and don’t deploy on Blackwell until the checkpoint and MXFP8 bugs clear ⁵. If you’re a single-GPU LoRA shop, this isn’t the tool — Unsloth still owns that ground ².

---

Meta-harnesses rewrite agent scaffolds, lifting Qwen3.5 44%

Source: latent-space · published 2026-06-25

TL;DR

• Stanford’s Meta-Harness pushed Claude Opus 4.6 to 76.4% on TerminalBench-2, beating hand-engineered Terminus-KIRA (74.7%) and vanilla Claude Code (58.0%).
• Xiaomi’s HarnessX shows small models gain most — +44% for Qwen3.5 on embodied planning, +14.5% averaged across 15 benchmarks.
• Databricks (Omnigent) and IBM (CUGA) race to own the meta-layer as a control plane for multi-agent systems.
• Anthropic’s data: ~37% of multi-agent failures are coordination bugs — the skeptics’ core case against the whole frame.

The result that started the season

Latent Space’s “Meta-Harness Summer” framing rests on one concrete result. Stanford’s IRIS Lab built an outer-loop agent — Claude Code with filesystem access to every prior run — that consumes up to ~10M tokens of execution traces per step and rewrites the harness code itself. Paired with Claude Opus 4.6, the rewritten harness scored 76.4% on TerminalBench-2, beating the hand-engineered Terminus-KIRA baseline (74.7%) and crushing stock Claude Code at 58.0% ⁷.

The shape of the system matters more than the number:

flowchart LR
    A[Execution traces<br/>~10M tokens] --> B{Proposer<br/>Claude Code}
    B -->|rewrites| C[Harness code]
    C --> D[Inner agent<br/>Claude Opus 4.6]
    D --> E[TerminalBench task]
    E -->|new trace| A

The proposer doesn’t tune prompts. It edits the scaffold — tool definitions, retry logic, planning structure — between runs, using past failures as the training signal. No gradient updates, no fine-tuning, just code-writing-code.

Why small-model shops care

The politically interesting data point comes from Xiaomi. HarnessX reports an average +14.5% gain across 15 model-benchmark combinations, but the headline is a +44% jump for open-weight Qwen3.5 on embodied planning ⁸. If that generalizes, harness work is a disproportionate rescue for models that can’t afford a frontier-lab budget — inverting the usual “wait for the next checkpoint” reflex.

That’s the wedge Databricks and IBM are pushing into. Omnigent is pitched as a pluggable meta-harness that combines Claude Code with custom agents under shared policy, including “Identity Injection” so sub-agents use credentials from env vars they can’t read or leak ⁹. IBM’s CUGA takes the governance-first route with a hierarchical Two-Loop planner/executor, claiming #1 on AppWorld (750 tasks across 457 APIs) and 61.7% on WebArena — though reviewers call setup “cumbersome” and flag failure modes like correctly navigating a travel tool while picking the wrong destination ¹⁰.

The dissent is loud

Two pushbacks deserve weight. First, Anthropic’s own numbers (synthesized by CIO) say multi-agent systems use ~15× more tokens than chat and that ~37% of failures are coordination bugs. Databricks engineer Sandipan Bhaumik — whose employer is selling a meta-harness — warns that scaling from one to fifty agents is “a distributed systems problem nobody’s discussing,” with silent handoff failures and untraceable decisions ¹¹.

Second, Latent Space published the counter-case to its own framing weeks ago:

Many harness gains sit inside benchmark noise, and a basic scaffold on a stronger frontier model often matches an engineered one.

If true, every meta-harness is a bridge that gets bulldozed by the next model release ¹².

What’s actually at stake

The field is converging on “harness > prompt” as 2026’s unit of progress and splitting hard on whether the meta-layer is a durable platform or scaffolding for a problem the next base model deletes. The most defensible claim in the cluster is the small-model lift ⁸ — that’s a real argument for open-weight shops to invest in self-rewriting scaffolds now, regardless of where GPT-6 lands. The least defensible is that orchestrating fifty agents pays for its own coordination tax in production ¹¹.

---

Cohere tops new FFASR far-field benchmark at 17.9% WER

Source: huggingface-blog · published 2026-06-24

TL;DR

• Cohere Transcribe leads the new Far-Field ASR leaderboard at 17.9% WER, ahead of IBM Granite and NVIDIA Parakeet.
• Every tested model takes a 3–4× WER hit moving from clean near-field audio to simulated rooms at low SNR.
• Held-out simulated audio blocks test-set gaming, a deliberate defense against leaderboard contamination.
• Treble’s own users question the wave-simulator’s sim-to-real fidelity, citing hand-tuned scattering coefficients.

What the leaderboard actually shows

Hugging Face and Treble Technologies launched the Far-Field ASR (FFASR) Leaderboard to score ASR models on simulated reverberant rooms instead of clean audiobook clips. The headline result isn’t in the announcement post — it’s in the third-party recaps: Cohere Transcribe tops the far-field board at 17.9% WER, IBM Granite Speech trails by ~2 percentage points, and NVIDIA Parakeet by ~3.6 ¹³. Those same models score roughly 5–7% on the standard Open ASR Leaderboard, so the move from anechoic to a furnished room at low SNR costs every model a 3–4× WER multiplier ¹³. That penalty — not the rank order — is the load-bearing finding.

The evaluation runs 2,000 held-out anechoic samples through a hybrid wave-based simulator across 14 furnished rooms (20–470 m³), and splits results by SNR band: high (>14 dB), mid (8–12 dB), and low (<6 dB). Latency is measured as RTFx on an NVIDIA L4. Critically, submitters never see the audio, an “audio-in-the-vault” design that Appen flags as a deliberate defense against the test-set contamination that has dogged earlier leaderboards ¹⁴.

How much to trust the simulation

FFASR’s authority rests entirely on the fidelity of Treble’s wave-based engine. That assumption has independent critics. Practicing acousticians using Treble’s commercial SDK report needing to apply “unreal scattering coefficients” of 0.6–0.7 to match measured reverberation times, a process they describe as “guessing” ¹⁵. If material parameters need hand-tuning to match physical rooms, the simulated room impulse responses may systematically mis-rank models that happen to be tuned to real microphone-array quirks. The companion Treble10 dataset (Interspeech 2025) is more upfront about its envelope: a DG-FEM solver only operates below ~3.2 kHz, with geometric acoustics filling in above that ¹⁶ — which covers most speech energy, but leaves the upper consonant band on a different physical model.

The benchmark FFASR isn’t

Compared to CHiME (spontaneous multi-party meetings) and VOiCES (real audio replayed in real rooms), FFASR sits closer to VOiCES and inherits the same critique: scripted speech, no interruptions, no overlap, no filler words — none of the behaviors that define real conversation ¹⁷. There’s also a sharper trap in the framing. FFASR suggests developers compare dry-vs-far-field gaps to decide whether to add “acoustic preprocessing.” But Deepgram and others have documented a denoising paradox: front-end enhancement and beamforming raise perceptual SNR while degrading Whisper’s WER, because the model is sensitive to enhancement artifacts ¹⁸.

Whisper’s zero-shot performance can actually drop when audio is preprocessed for denoising. ¹⁸

A leaderboard that rewards raw far-field WER without modeling that tradeoff risks pushing teams toward pipelines that hurt the metric the benchmark itself measures.

Net read

FFASR fills a genuine evaluation gap and the held-out design is a real improvement over Open ASR. But until the roadmap items — multi-talker scenes, microphone arrays, echo cancellation — land, FFASR rankings are best read as a relative robustness signal on a known-narrow slice of the hands-free problem, not as a deployment oracle.

Round-ups

Willison converts MDN browser-compat data to a 66MB SQLite DB

Source: simon-willison

Inspired by Mozilla’s new MDN MCP server, Simon Willison piped the mdn/browser-compat-data repo through sqlite-utils into a queryable database. A GitHub Actions workflow force-pushes the build to an orphan branch, exploiting GitHub’s open CORS headers so Datasette Lite can load it directly in the browser.

Tom MacWright: LLM-generated job applications erase the candidate

Source: simon-willison

Hiring managers are seeing fully synthetic stacks — AI-written resumes linking to AI-built portfolios and AI-authored GitHub commits. MacWright argues the polish backfires: a perfected, generic application reveals nothing true about the applicant, only which tools they ran the pipeline through.

Footnotes

1. Surf AI analysis — https://asksurf.ai/pulse/en/nvidia-nemo-moe-training

   NVIDIA’s comparisons are framed against a ‘vanilla’ Transformers v5 setup that lacks hardware-specific optimizations… the benchmark methodology effectively moves the performance conversation away from model architecture and toward the proprietary execution layer.

   ↩ ↩²

2. Spheron Network (Axolotl vs Unsloth vs Torchtune comparison) — https://www.spheron.network/blog/axolotl-vs-unsloth-vs-torchtune/

   Unsloth’s Triton-based grouped-GEMM and LoRA kernels enable 16-bit LoRA training of Qwen3 30B on ~63GB VRAM and 4-bit fitting in 17.5GB — viable on a single RTX 5090 — whereas NeMo AutoModel targets multi-node H100/B200 clusters.

   ↩ ↩²

3. NVIDIA Megatron-Bridge MoE optimization docs — https://docs.nvidia.com/nemo/megatron-bridge/nightly/training/moe-optimization.html

   Megatron Core’s ‘Parallel Folding’ decouples attention and expert layer configurations to maximize hardware utilization on Blackwell and Hopper, pushing scaling further than AutoModel’s EP-only approach.

   ↩ ↩²

4. Cato Networks (Cato CTRL security research) — https://www.catonetworks.com/blog/cato-ctrl-new-vulnerabilities-in-nvidia-nemo-and-meta-pytorch/

   CVE-2025-33236 (CVSS 7.8): NeMo hardcodes trust_remote_code=True across multiple model importers, allowing silent remote code execution when a user imports a malicious model from a public repository.

   ↩

5. NVIDIA NeMo AutoModel release notes — https://docs.nvidia.com/nemo/automodel/whats-new/release-notes

   Known issues include checkpoint saving/loading failures on Blackwell (GB200) hardware, LayerNorm-with-bias accuracy discrepancies in models like StarCoder2, and higher-than-expected memory usage with MXFP8 mixed precision currently under investigation.

   ↩ ↩²

6. AIMLAPI blog on Nemotron-3-Ultra-550B — https://aimlapi.com/blog/nvidia-nemotron-3-ultra-the-550b-model-agents-have-been-waiting-for

   Full fine-tuning materializes ~2,922 GB of optimizer state in BF16, requiring 128 H100s — out of reach for most independent labs; testers report ‘weird results’ in local stress tests, with performance highly sensitive to quantization backend and chat template.

   ↩

7. Yoonho Lee (Stanford IRIS) project page — https://yoonholee.com/meta-harness/

   Meta-Harness uses an agentic ‘proposer’ (Claude Code) with filesystem access to past iterations, consuming up to ~10M tokens of execution traces per step to rewrite harness code; pairing Claude Opus 4.6 with Meta-Harness scored 76.4% on TerminalBench-2, beating hand-engineered Terminus-KIRA (74.7%) and Claude Code (58.0%).

   ↩

8. VentureBeat — HarnessX coverage — https://venturebeat.com/orchestration/xiaomis-harnessx-rewrites-its-own-ai-scaffolding-mid-task-and-smaller-models-gain-the-most

   Xiaomi’s HarnessX reported an average +14.5% gain across 15 model-benchmark combinations, with a +44% boost for open-weight Qwen3.5 on embodied planning — suggesting smaller models benefit most from self-rewriting scaffolds.

   ↩ ↩²

9. Databricks blog — Omnigent launch — https://www.databricks.com/blog/introducing-omnigent-meta-harness-combine-control-and-share-your-agents

   Omnigent is positioned as a ‘meta-harness’ that lets developers combine multiple agents (Claude Code, custom agents) under shared policies, with ‘Identity Injection’ so agents use credentials from env vars without being able to see or leak them.

   ↩

10. IBM Research blog — CUGA — https://research.ibm.com/blog/cuga-agent-framework

    CUGA’s hierarchical ‘Two-Loop’ (outer planner / inner executor) hit #1 on AppWorld (750 tasks across 457 APIs) and 61.7% on WebArena, but reviewers flag setup as ‘cumbersome’ and note failure modes like correctly navigating travel tools while picking wrong destinations due to lack of live web data.

    ↩

11. CIO.com — ‘True multi-agent collaboration doesn’t work’ — https://www.cio.com/article/4143420/true-multi-agent-collaboration-doesnt-work.html

    Anthropic research suggests multi-agent systems use ~15× more tokens than chat, and coordination errors account for ~37% of system failures; Databricks’ Bhaumik warns scaling from one to fifty agents creates ‘a distributed systems problem nobody’s discussing,’ with silent handoff failures and untraceable decisions.

    ↩ ↩²

12. Latent Space — ‘Is Harness Engineering Real?’ (prior issue) — https://www.latent.space/p/ainews-is-harness-engineering-real

    Critics of the ‘Big Harness’ camp argue many scaffolding gains are noise within margin of error, with a basic scaffold often matching engineered systems when paired with a stronger frontier model — implying meta-harnesses may be obsoleted by the next model jump.

    ↩

13. LocalAIMaster — Parakeet vs Whisper comparison — https://localaimaster.com/blog/parakeet-vs-whisper

    Cohere Transcribe leads FFASR far-field at 17.9% WER, ~2pp ahead of IBM Granite Speech and ~3.6pp ahead of NVIDIA Parakeet, while the same models score ~5% WER on near-field Open ASR — a 3–4× degradation from clean to far-field conditions.

    ↩ ↩²

14. Appen — Leaderboard integrity blog — https://www.appen.com/blog/hugging-face-open-llm-leaderboard

    FFASR uses a held-out evaluation set where audio is never exposed to submitters, a ‘Benchmaxxer Repellant’ design intended to prevent the test-set contamination that has manipulated previous leaderboards.

    ↩

15. Treble Technologies — BRAS validation docs — https://docs.treble.tech/validation/bras

    Practitioners report needing to apply ‘unreal scattering coefficients’ (0.6–0.7) to match measured reverberation times, a process described as ‘guessing’ that undermines the software’s physics-based claims.

    ↩

16. arXiv — Treble10 dataset paper (Interspeech 2025) — https://arxiv.org/abs/2606.05678

    Treble10 ships >3,000 broadband RIRs across ten furnished rooms paired with LibriSpeech utterances in six configurations (mono, 8th-order Ambisonics, 6-channel device), using a Discontinuous Galerkin FEM solver below ~3.2 kHz and geometric acoustics above.

    ↩

17. ACL Anthology — CHiME/VOiCES survey (LREC 2022) — https://aclanthology.org/2022.lrec-1.209.pdf

    VOiCES provides a controlled environment for studying acoustic distortions but fails to capture the linguistic nuances and human behaviors — interruptions, filler words, overlapping speech — that define the CHiME challenges.

    ↩

18. Deepgram — Noise-robust ASR best practices — https://deepgram.com/learn/noise-robust-speech-recognition-methods-best-practices

    Traditional speech enhancement and beamforming, while improving perceptual quality for humans, often introduce artifacts that degrade ASR accuracy — Whisper’s zero-shot performance can actually drop when audio is preprocessed for denoising.

    ↩ ↩²