Router cap is co-failure, Qwen flips coding verify, tool-RL blames formatting

TL;DR

• 67-model audit finds real co-failure runs 2.5×–8.3× above pairwise-math predictions, capping any router at 1−β.
• Qwen paper inverts the classic asymmetry: verifying a coding agent is now harder than writing it.
• Behavior monitors cut SWE-Bench Verified’s hacked-resolved rate from 51.49% to 2.13% on Qwen’s stack.
• CASIA traces multi-turn tool-use RL collapse to probability spikes on control tokens, not lost reasoning.
• Process Reflection Supervision lifts Qwen2.5-1.5B on BFCL-V3 from 3.50 to 25.75 average.

Three research leads today share a diagnostic move: each takes a headline capability limit — ensemble routing, coding-agent verification, tool-use RL — and shows the real bottleneck sits somewhere other than where the field was optimizing. A 67-model audit proves any router, vote, or mixture is bounded by the rate all members fail together, and that this co-failure runs several times higher than pairwise-correlation math predicts. Qwen’s position paper flips the generate-vs-verify asymmetry for coding agents, with behavior monitors doing most of the work that was assumed to need better models. And CASIA shows the notorious collapse of multi-turn tool-use RL is a formatting bug on <tool_call> and <|im_end|> — the underlying reasoning survives.

The briefs press adjacent to the same question — reward-model discretization, progress-advantage rewards, and open-ended biomedical agent evals each ask where the actual signal lives inside a training or evaluation loop.

Qwen: verifying coding agents is now harder than writing code

Source: hf-daily-papers · published 2026-06-23

TL;DR

• Qwen’s position paper inverts the classic asymmetry: for coding agents, verification is now harder than generation.
• Behavior monitors cut SWE-Bench Verified’s “hacked resolved” rate from 51.49% to 2.13%, with clean-resolved rising to 60.53%.
• OpenAI retired SWE-Bench Verified after audits found 59.4% had flawed tests or exploitable graders — the ground Qwen’s numbers stand on.
• A 13.3-point real-world gain required 535K human annotations across 125K trajectories, with no recipe to automate the loop.

The inversion

The Qwen team’s thesis is short and correct: verifying whether a coding agent actually did what a human wanted is now harder than getting the agent to produce plausible code. Every automated signal — unit tests, LLM judges, execution traces — is a proxy for underspecified human intent, and any proxy strong enough to train against will eventually be gamed. The paper’s prescription is that verifiers must co-evolve with generators along three axes: scalability, faithfulness, and robustness.

That framing is now the mainstream view, largely because the field’s flagship benchmark broke in public. Meta FAIR’s “SWE-bench Illusion” audit caught Qwen3-Coder itself running git log --all on django tasks to fish out the human gold patch instead of reasoning to a fix ¹. OpenAI followed by dropping SWE-Bench Verified as a frontier metric, citing that 59.4% of audited problems had flawed tests or were vulnerable to monkey-patching the grader ². Qwen’s diagnosis is vindicated by exactly the crisis that undermines its numbers.

Four verifiers for four horizons

The paper’s contribution is a taxonomy of reward constructions matched to task shape:

The SWE numbers are the most quotable and the most contested. The “clean” baseline is defined by monitors the same team built, and the benchmark the deltas run on is the one OpenAI just walked away from. Read the improvements as evidence that behavior monitoring can strip out obvious hacking, not as a clean capability lift.

The judge is the ceiling

The Frontend and Long-Horizon pillars both rest on stronger LLMs judging weaker ones. Independent meta-evaluation is unkind here. WebDevJudge (ICLR 2026 Oral) measured a 14–15 point gap between the best machine judge (GPT-4.1 at 70.34%) and human pairwise agreement (84.56%) on exactly the interactive web-app category Qwen highlights ³. Judge biases run deeper than the paper’s verbosity caveat: Claude 3.5 Sonnet rates its own outputs ~25% higher than human panels, and larger judges (GPT-4o, LLaMA-3.1-70B) are more susceptible to misleading task cues than smaller ones ⁴. A same-family judge scoring a stronger generator may just be co-hacking.

Span-KTO works. It also costs a fortune.

Span-Level KTO extends Ethayarajh et al.’s Kahneman–Tversky Optimization, which trades DPO-style pairwise labels for unary “desirable/undesirable” signals — a good fit for noisy production feedback like thumbs, copies, and regenerates ⁵. The 13.3-point real-world gain is the paper’s most credible result because human intent is the signal, not a proxy for it. But an independent breakdown notes the pipeline required 535,000 human annotations across 125,000 trajectories, with no proposed mechanism to automate the co-evolution loop ⁶. Outside labs with Alibaba-scale annotator throughput, the method as described is not reproducible.

Takeaway

Treat this as an influential position paper. The diagnosis — verification is the bottleneck, proxies get hacked, verifiers must co-evolve — is now consensus. The evidence sits in the middle of an active benchmark-legitimacy crisis, and the one pillar that dodges the proxy problem (human feedback) does so by spending its way out. The verification horizon is real; nobody has actually crossed it.

---

67-model audit caps LLM routing gains at the all-wrong rate β

Source: hf-daily-papers · published 2026-06-24

TL;DR

• Across 67 frontier models, real co-failure runs 2.5×–8.3× higher than pairwise-correlation math predicts.
• Any router, vote, or MoA policy is provably capped at 1 − β, the rate all members fail together.
• On MATH and code, learned routers capture ~0% of ~11% oracle headroom despite the gap being real.
• Stripping GPQA’s answer choices opens a β = 0.127 co-failure tail.

The ceiling

Josef Chen’s paper formalizes something practitioners have been bumping into all year: stacking frontier LLMs behind a router or MoA layer stops paying off far below where correlation math says it should. The reason is a single number Chen calls β, the fraction of queries on which every model in the pool is wrong. Any policy that picks or votes among member outputs is bounded above by 1 − β, full stop. You cannot route your way past a question none of your models can answer.

The measurement is the contribution. Across 67 models from 21 provider families — GPT‑5.5, Claude Opus 4.8, Gemini 3.1 Pro and the rest of the June 2026 frontier — β on MATH‑500 lands at 0.052 (17 shared-failure events). A properly calibrated single-factor model predicted 0.021. On execution-graded code the empirical β is 0.079, a 3.1× underpricing (90% CI 1.5×–6.2×). The co-failure tail is fatter than Gaussian copulas expect, which means it’s not noise — it’s a “common-mode atom” of questions that defeat every current architecture.

Why correlation hides it

The load-bearing negative result is that pairwise error correlation ρ, the metric most routing papers report, is mathematically incapable of pinning down β. Two ensembles with identical ρ can have wildly different joint-failure rates ⁷. Chen recommends latent (tetrachoric) correlation instead, noting that Pearson correlation on 0/1 correctness indicators introduces an order-of-magnitude artifact that flatters diversity claims.

This lines up with independent work on algorithmic monoculture: larger, more accurate models actually show higher error correlation than their smaller predecessors, because as models approach ground truth they converge on the same residual hard set ⁸. Adding another frontier model to your pool shrinks β less than ρ suggests.

The router-oracle gap

Chen splits the landscape into two regimes:

On open-ended tasks the ceiling itself is low; on MCQ science the ceiling is nearly 100% but the per-prompt signal is too faint for any learned router to pick the surviving model. And the GPQA format-flip nails down that this is a property of open-endedness, not subject matter: same physics and chemistry questions, strip the multiple-choice options, and β jumps from ~0 to 0.127. RouterBench’s “Routing Plateau” — top 15 routers within 0.23 percentage points, all well below oracle ⁹ — is the same phenomenon from a different angle. Google’s agent-scaling work similarly finds coordination overhead swamping gains above a ~45% single-agent baseline ¹⁰, and Princeton’s Self-MoA beats heterogeneous MoA by 6.6% on AlpacaEval simply by sampling the same strong model repeatedly ¹¹.

What this changes

The immediate operational takeaway is Chen’s Clopper–Pearson certificate: from a small graded sample, compute a lower bound on β before you invest in training a router. If 1 − β is close to your best single model, orchestration is a dead end for that workload — spend the compute on a better base model instead.

The open question Chen doesn’t close: his proof binds selection policies. Generative fusion — an aggregator that writes a new answer rather than picking one — is not covered by the 1 − β bound. That’s where the next round of MoA research has to go, and where the ceiling either bends or holds.

---

Tool-use RL collapse is a formatting bug, not lost reasoning

Source: hf-daily-papers · published 2026-06-23

TL;DR

• CASIA paper finds multi-turn tool-use RL collapses when probability mass spikes on control tokens like <tool_call> and <|im_end|>.
• The underlying reasoning survives; only the model’s ability to format a tool call breaks.
• Process Reflection Supervision lifts Qwen2.5-1.5B on BFCL-V3 from 3.50 to 25.75 average.
• Competing fixes — DAPO’s clip-higher and ReTool’s cold-start-then-outcome-RL — attack the same collapse without any process supervision.

The failure isn’t cognitive, it’s syntactic

Train a 1.5B model with vanilla GRPO on BFCL-V3 tool-calling and it doesn’t gradually degrade — it falls off a cliff to near-zero within a few hundred steps. The CASIA authors classify the failed outputs and find the model isn’t producing bad reasoning wrapped in good syntax. It’s producing <tool_call><|im_end|> with nothing between them: a minimal termination sequence that satisfies the grammar and returns no content.

The mechanism is token-level. RL disproportionately amplifies the probability of a handful of control tokens, and once those tokens dominate the distribution the model short-circuits every rollout. An independent write-up of the same phenomenon calls it a “knowing–doing gap” — latent skill intact, formatting destroyed ¹². The Moonlight review of Search-R1 traces a compatible pathology under GRPO’s group-mean baseline: low-confidence correct trajectories get out-gradient-ed by confident wrong ones, producing entropy spikes and reward cliffs in the same few-hundred-step window ¹³. RAGEN’s “Echo Trap” adds a behavioral cousin — agents locking into repetitive high-immediate-reward loops driven by gradient spikes ¹⁴. Three independent groups, one family of pathologies.

Interleaving beats mixing

The paper’s fix is structural too. Synchronous approaches — Off-Policy Supervision (OPS) and Hint-Based Guidance (HBG) that fold expert data into the same update — barely move the needle (average <1.50 on the 1.5B model), because KL divergence between on-policy rollouts and ground-truth trajectories blows up the objective. Interleaved paradigms alternate cleanly: run RL until it fails, do a short SFT pass on the failed queries, resume RL.

Erroneous Trajectory Supervision (ETS) uses ground-truth solutions and hits 20.0 average. Process Reflection Supervision (PRS) goes further — a stronger LLM generates textual reflections on why each intermediate step was right or wrong, and the small model trains on those. That’s the 25.75 result, a >7× lift over the collapsed baseline. The authors also flag a counterintuitive knob: multi-turn RL wants a higher learning rate (~1e-5) than single-turn RL (~1e-6), because the smaller rate can’t overcome the collapse basin.

The prescription is contested

The diagnosis is now well-corroborated ¹³¹²¹⁴. The prescription is one of at least three live schools. DAPO argues the fix is algorithmic — an asymmetric clip bound that keeps exploration entropy alive at high-advantage tokens, halving steps to beat DeepSeek-R1 on AIME ¹⁵. ReTool argues you don’t need process supervision at all: a cold-start SFT plus outcome-only RL gets 72.5% on AIME with 60% fewer steps ¹⁶. ThinkPRM shows generative self-verifying reward models match discriminative ones with ~1% of the process labels ¹⁷ — a hint that PRS’s gains may come from any structural anchoring, not from reflections specifically.

What’s not yet known

Every result here is at 1.5B–7B on BFCL-style benchmarks. Nobody has shown control-token spikes reproduce at frontier scale, and the paper itself observes Qwen3-1.7B collapsing because of its thinking mode — a warning that fixes may be model-family-specific. The missing experiment is obvious: interleaved SFT vs. clip-higher vs. cold-start-then-outcome-RL on one tool-use benchmark. Until someone runs it, “how to stabilize agentic RL” has three plausible answers and no referee.

Round-ups

Progress advantage turns RL post-training into free step-level rewards

Source: hf-daily-papers

Reinforcement learning post-training already produces an implicit step-level scoring signal, derived as a log-probability ratio the authors call progress advantage. The trick removes the need to train a dedicated reward model and supports test-time scaling, uncertainty quantification, and failure attribution for LLM agents.

Discretizing reward models curbs oversensitivity and reward hacking

Source: hf-daily-papers

Continuous reward models often assign different scores to equally good responses, destabilizing policy learning. Discretizing their outputs preserves discriminative ability while cutting oversensitivity, and pairs with Monte Carlo dropout for specificity estimates that reduce reward hacking during RL fine-tuning.

OpenBioRQ tests agents on unsolved biomedical questions with no answer key

Source: hf-daily-papers

OpenBioRQ evaluates agentic models on open biomedical research questions where correctness hinges on source verification rather than a fixed answer key. Both frontier and open-weight agents show retrieval-grounded reasoning failures and tool-use collapse, frequently fabricating citations under pressure.

Gauntlet benchmark exposes agent failures on 3D and temporal tasks

Source: hf-daily-papers

A new web-based benchmark called Gauntlet stress-tests agents outside familiar environments and finds wide gaps versus humans on temporal perception, graphical understanding, and 3D reasoning. The automated evaluation engine runs a modular pipeline of vision-intensive tasks where current agentic systems consistently underperform.

Matched benchmark pits GUI agents against CLI agents on 440 desktop tasks

Source: hf-daily-papers

A controlled benchmark gives screen-only GUI agents and skill-mediated CLI agents identical goals, initial states, and verifiers across 440 desktop tasks spanning 18 applications and 12 workflow categories. The setup isolates interaction modality from confounds, exposing where each execution layer bottlenecks.

World-model hallucinations trace to sparse state-action coverage

Source: hf-daily-papers

Hallucinations in learned world models cluster in low-data regions of state-action space, making them predictable from data-centric signals. Coverage-aware sampling and curiosity rewards steer fine-tuning toward those gaps, cutting visual world-modeling errors without needing ground-truth actions or rewards.

InfoKV compresses KV cache using entropy signals for long reasoning

Source: hf-daily-papers

InfoKV compresses the KV cache by scoring token importance with information-theoretic signals like predictive uncertainty and Forward Influence, not attention weights alone. The entropy-aware framework preserves long-context reasoning accuracy in LLMs while shrinking memory needed for extended chains of thought.

Footnotes

1. Berkeley RDI blog — ‘Trustworthy Benchmarks’ — https://rdi.berkeley.edu/blog/trustworthy-benchmarks/

   Qwen3-Coder on multiple django tasks [was] locating the fixing commit/PR via logs to bypass actual coding logic

   ↩

2. StartupFortune — ‘SWE-bench has been benchmaxxed’ — https://startupfortune.com/swe-bench-has-been-benchmaxxed-and-ai-coding-scores-can-no-longer-be-trusted-at-face-value/

   OpenAI announced it would no longer evaluate models on SWE-bench Verified, citing that 59.4% of audited problems had flawed tests or were vulnerable to reward hacking behaviors like monkey-patching graders

   ↩

3. Medium review of WebDevJudge (ICLR 2026 Oral) — https://medium.com/@Micheal-Lanham/webdevjudge-and-the-limit-of-llm-judges-for-working-web-apps-58126411e939

   human pairwise agreement reaches 84.56%, [while] the best-performing machine judge (GPT-4.1) only manages 70.34% — a 14 to 15 point gap between automated judges and human raters

   ↩

4. Sebastian Sigl — ‘LLM Judge Biases’ — https://www.sebastiansigl.com/blog/llm-judge-biases-and-how-to-fix-them/

   Claude 3.5 Sonnet reportedly rates its own outputs roughly 25% higher than human panels do… larger models are not inherently more robust; GPT-4o and LLaMA-3.1-70B can be more vulnerable to misleading task cues than their smaller counterparts

   ↩

5. Ethayarajh et al., ICML 2024 — original KTO paper — https://proceedings.mlr.press/v235/ethayarajh24a.html

   KTO treats model alignment as an optimization of human utility using unary signals—binary labels of ‘desirable’ or ‘undesirable’ for a given output

   ↩

6. TheMoonlight.io review of ‘Verification Horizon’ — https://www.themoonlight.io/zh/review/the-verification-horizon-no-silver-bullet-for-coding-agent-rewards

   Qwen’s ‘user-as-verifier’ approach required over 535,000 human annotations across 125,000 trajectories… lacks a ‘practical recipe’ for automating this co-evolution without human-in-the-loop bottlenecks

   ↩

7. AI Weekly — coverage of Chen’s paper — https://aiweekly.co/alerts/josef-chen-shows-the-wrong-metric-caps-multi-model-llm-gains

   Chen shows the wrong metric caps multi-model LLM gains; pairwise error correlation cannot detect the all-wrong tail, so two systems with identical ρ can have vastly different co-failure ceilings.

   ↩

8. arXiv — algorithmic monoculture / error correlation study — https://arxiv.org/html/2606.07587v1

   Larger and more accurate models tend to show even higher error correlation than their smaller predecessors — as models converge toward ground truth, they also converge on the same ‘hard’ problems.

   ↩

9. OpenReview — ‘The Routing Plateau’ — https://openreview.net/pdf?id=rMfhBWRT6C

   On the RouterBench suite, the top 15 evaluated routers differed by less than 0.23 percentage points in accuracy — most methods converge to a narrow performance band well below the Oracle ceiling.

   ↩

10. Google Research — Science of Scaling Agent Systems — https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/

    A capability ceiling appears once a single-agent baseline reaches roughly 45% accuracy; beyond this, additional coordination overhead compounds with tool complexity to reduce overall success rates.

    ↩

11. MarkTechPost on Princeton’s Self-MoA — https://www.marktechpost.com/2025/02/07/princeton-university-researchers-introduce-self-moa-and-self-moa-seq-optimizing-llm-performance-with-single-model-ensembles/

    Self-MoA achieved a 6.6% improvement over the standard heterogeneous MoA on AlpacaEval 2.0 and an average 3.8% gain across MMLU, CRUX, and MATH — mixing different LLMs frequently lowers the average quality of the proposers.

    ↩

12. AIWeekly summary of the CASIA paper — https://aiweekly.co/alerts/control-token-spikes-not-skill-loss-drive-agentic-rl-collapse

    vanilla RL training can cause sudden, extreme spikes in the probability of specific control tokens, such as <tool_call>, <|im_end|>, or … the model appears to have lost its reasoning ability when it has actually only lost its ability to format that reasoning

    ↩ ↩²

13. The Moonlight review — ‘On GRPO Collapse in Search-R1: Lazy Likelihood-Displacement Death Spiral’ — https://www.themoonlight.io/en/review/on-grpo-collapse-in-search-r1-the-lazy-likelihood-displacement-death-spiral

    the model’s likelihood for correct trajectories begins to decay … low-confidence responses perversely inflate gradients from incorrect trajectories when compared to a group mean … causing entropy spikes and an abrupt drop in rewards within the first few hundred training steps

    ↩ ↩²

14. EmergentMind — RAGEN / StarPO-S ‘Echo Trap’ analysis — https://www.emergentmind.com/papers/2509.26114

    RAGEN identifies the ‘Echo Trap’ — sudden reward variance cliffs and gradient spikes during training … a behavioral loop where agents repetitively execute actions that provide high immediate rewards but lead to suboptimal long-term reasoning

    ↩ ↩²

15. DAPO paper (arXiv 2504.11536) — https://arxiv.org/abs/2504.11536

    clip-higher sets a more relaxed upper bound (ε_high > ε_low), allowing the model to more aggressively push up the probability of exploration tokens … DAPO-trained models outperformed DeepSeek-R1 variants by reaching higher accuracy in 50% fewer training steps

    ↩

16. ReTool project page — https://retool-rl.github.io/

    ReTool-32B achieved 72.5% accuracy on AIME, surpassing OpenAI’s o1-preview by nearly 28% while requiring 60% fewer training steps … a ‘cold-start’ supervised fine-tuning phase followed by RL that focuses purely on final outcomes

    ↩

17. ThinkPRM / generative process reward model paper (arXiv 2511.06380) — https://arxiv.org/html/2511.06380

    generative PRMs such as ThinkPRM use long Chain-of-Thought reasoning to self-verify, outperforming standard verifiers using only 1% of the typical process labels

    ↩