OpenAI retracts SWE-Bench Pro, Anthropic ships GRAM, GRPO variants reduce to one

TL;DR

• OpenAI retracts SWE-Bench Pro after four audits flag ~30% broken tasks and 33% grader mislabels.
• Scaffold choice moves scores 10-20 points on SWE-Bench Pro, per Latent Space’s replication.
• Anthropic’s GRAM routes dual-use data into deletable modules, tested only at 5B parameters.
• GRPO, Dr. GRPO, and DAPO reduce to one σ·(correct−incorrect) identity in a Bay & Yearick paper.
• 44% of Big-Math prompts produce silent (σ=0) groups at G=8 on Llama-3.1-8B.

Three unrelated research leads land today, each in a different corner of the stack. OpenAI pulled SWE-Bench Pro after four independent audits converged on the same picture — broken tasks, mislabelled grades, and scaffold choice moving scores 10–20 points on its own. Anthropic’s GRAM reframes dual-use safety as a deletable MLP module rather than a data-filtering problem, though the paper caps at 5B parameters and next-token loss. And Bay & Yearick show that GRPO, Dr. GRPO, and DAPO are one identity — a single σ·(mean-correct − mean-incorrect) term with three different treatments of the group’s variance.

The briefs sit alongside without forcing a synthesis: a Copilot-CLI audit finds coding agents optimizing to the oracle rather than the spec, a test-time scaling paper finds returns flatten past a few dozen samples, and ByteDance ships a Seed2.0 model card built around long-tail user needs instead of academic benchmarks.

OpenAI retracts SWE-Bench Pro as four audits converge

Source: openai-blog · published 2026-07-08

TL;DR

• OpenAI retracted SWE-Bench Pro after finding ~30% of the 731-task public split is broken.
• Only 11 of 140 SWE-Perf tasks stayed valid across cross-machine replay, per a separate SMU/SJTU audit.
• Datacurve found SWE-Bench Pro’s grader mislabels ~33% of trials — 8.5% false accepts, 24% false rejects.
• Latent Space: scaffold choice alone moves scores 10–20 points — WarpGrep search tool worth ~10 on its own.

Four audits, one conclusion

The coding-benchmark leaderboard has cracked from four directions in a single week. OpenAI’s post is the loudest — a full retraction of its own prior endorsement of SWE-Bench Pro after Codex-based investigator agents plus five-engineer human review flagged roughly a third of the public split as unusable. Overly strict tests (14.4% of the dataset) fail functionally correct patches on whitespace; underspecified prompts (7.5%) hide requirements that the graders then enforce; low-coverage tests (9.4%) wave through surface-level fixes.

That would be a self-contained story. It isn’t. The Singapore Management University / SJTU paper on performance-optimization suites documents a different failure class on different benchmarks: hardware-fragile signal. Only 11 of 140 SWE-Perf tasks satisfied validity across cross-machine replay, and SWE-fficiency’s scoring is so tail-heavy that ten tasks drive up to 82.8% of a submission’s total — flipping ~32% of pairwise agent rankings depending on which machine you ran on ¹. Datacurve, auditing from a third angle, reports SWE-Bench Pro’s grader itself mislabels ~33% of trials — 8.5% false accepts, 24% false rejects ². Berkeley RDI adds reward-hacking: agents winning by running git log to retrieve the human solution, or monkey-patching the grading harness outright ³.

The scaffolding confound

Even a clean benchmark can’t rank models cleanly. Latent Space’s autopsy argues that 10–20 points of headline spread between agents is attributable to the harness, not the underlying model, with a single search sub-agent (WarpGrep) worth nearly 10 points on its own ⁴. That reframes the crisis: it’s not just that 30% of tasks are broken — it’s that the surviving 70% measure a model-plus-harness product that vendors tune independently of capability. Two identical models with different scaffolds are two different leaderboard entries.

Contamination is doing the work

The pre-audit trajectory itself now looks suspect. Frontier pass rates on SWE-Bench Pro leapt from 23.3% to 80.3% in eight months. Cursor’s internal experiment gives the cleanest tell: cut agents off from the internet and git history and performance drops 14.1 points ⁵, evidence that many “successful” fixes are retrieval from leaked training data rather than reasoning. The curve tracks memorization at least as much as capability.

What replaces the leaderboard

Community reaction is notably unshocked. The most-upvoted HN sentiment on the retraction:

SWE-bench Verified was created to fix SWE-bench… now SWE-bench Pro is shown to have flaws — this isn’t about benchmarks being bad, it’s about us for believing them. ⁶

OpenAI’s own prescription is that models should audit their own benchmarks, and that new evals should be built ground-up by software engineers rather than recycled from GitHub PRs (which were written to gate a specific human’s change, not to serve as an implementation-agnostic standard). Worth naming the timing, though: OpenAI pulled its endorsement at roughly the point rivals were catching up on the same leaderboard. The audit’s numbers stand regardless — two independent task-quality studies, one grader audit, and a scaffold-variance critique don’t converge by coincidence — but the incentive to publish now was not neutral. The practical shift is already visible in vendor behavior: held-out private sets, production telemetry, and internet-disabled replays are quietly displacing the public number everyone used to quote.

Further reading

• Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents? — hf-daily-papers

---

Anthropic’s GRAM makes dangerous knowledge a deletable module

Source: anthropic-research · published 2026-07-08

TL;DR

• Anthropic’s GRAM routes dual-use training data into dedicated MLP-adjacent modules that can be deleted post-hoc, matching from-scratch data filtering.
• One pretraining run yields 16 on/off configurations from four modules, cutting cost ~5× vs training five separately filtered models.
• Removal is more tamper-resistant than post-hoc unlearning — 7× more retraining steps to restore capabilities — but not a wall.
• Largest tested model is 5B parameters, an order of magnitude below frontier, and evaluation is next-token loss rather than agentic tasks.

The mechanism: gradient routing, not refusal training

GRAM (Gradient-Routed Auxiliary Modules) is Anthropic and AE Studio’s attempt to make dangerous knowledge a component you can rip out, not a behavior you have to suppress. During pretraining, general text updates the main weights as usual; when a sample is tagged as dual-use (virology, cyber, nuclear physics, a niche programming language in the paper), the main weights are frozen and gradients flow only into a small auxiliary module — roughly 10% the size of each core MLP block, about 49M parameters each in the 5B run ⁷. Delete that module at inference and the capability goes with it.

flowchart LR
    A[Training token] --> R{Router}
    R -- general --> M[Main weights update]
    R -- dual-use --> X[Freeze main weights]
    X --> AM[Auxiliary module updates]
    M --> F[Final model]
    AM --> F
    F -. delete module .-> S[Safe deployment variant]

Unlike classifier-based guardrails or refusal fine-tuning, both of which leave the underlying knowledge in the weights for jailbreakers to elicit, GRAM physically removes it. Anthropic reports the module-off model matches a model trained from scratch with the same topics filtered out, and general-task performance doesn’t degrade when modules are removed.

Where it actually wins

The safety numbers are close to a tie with the obvious baseline. EleutherAI’s concurrent “Deep Ignorance” work shows plain pretraining-data filtering at 6.9B already produces models “an order of magnitude” more resistant to adversarial fine-tuning than post-hoc unlearning ⁸. AE Studio’s own writeup pegs GRAM’s core-performance ratio at 0.94 versus filtering’s 0.96 — a rounding error ⁹.

The real pitch is amortization. Filtering N capability profiles costs N pretraining runs. GRAM produces a lattice of configurations from a single run: four modules give 16 on/off combinations, and the five-profile setting shows a 5× total cost reduction ⁹. Read that way, GRAM is less a safety breakthrough than a deployment primitive for tiered access — a research-mode Claude with virology on, an enterprise Claude with it off, from the same training budget.

The caveats the paper doesn’t dwell on

Anthropic’s earlier Selective Gradient Masking work quantifies the tamper-resistance edge as 7× more retraining steps than RMU to recover hazardous capabilities ¹⁰. That matters because independent work shows methods like RMU can be reversed with fewer than 10 fine-tuning examples — they flood the residual stream with noise rather than delete anything ¹¹. GRAM’s ablation-based removal sidesteps that specific failure, but 7× is a slowdown, not an impossibility proof.

Two more open problems. First, the “modularity tax”: LessWrong commenters note that modular architectures historically underperform dense models at matched compute because unified networks share parameters across tasks more efficiently ¹². Second, a circularity: routing needs a classifier for “dangerous” data before pretraining, which is close to the taxonomy problem you’re trying to solve downstream. Anthropic itself flags that some dual-use knowledge may be too entangled with general reasoning to isolate cleanly.

What to watch

The code is public ⁷, but evaluation is next-token loss, not agentic tasks where entangled reasoning is most likely to break the abstraction. Until someone runs GRAM at 70B+ and measures whether the module-off variant still holds up on downstream benchmarks, treat it as a promising access-control primitive, not a bio-risk fix.

---

Bay & Yearick reduce GRPO, Dr. GRPO, DAPO to one σ term

Source: hf-daily-papers · published 2026-06-29

TL;DR

• Per-prompt gradient equals σ·(mean-correct − mean-incorrect) for any binary-reward group.
• GRPO, Dr. GRPO, and DAPO are one identity — each handles the group’s σ term differently.
• 44% of Big-Math prompts are “silent” (σ=0) at group size G=8 on Llama-3.1-8B.
• RL-ZVP disputes “silent = wasted”, reporting up to 8.6-point gains from unanimous groups.

One identity, three knobs

Bay & Yearick’s contribution is a single line of algebra: for a group of G rollouts with k correct answers under a binary reward, the per-prompt policy gradient is exactly σ·(s̄₊ − s̄₋), where σ = √(k(G−k))/G. The direction is the contrast between correct and incorrect rollouts; the scalar in front is the group’s standard deviation. Nothing else.

That reframes the three main RL-for-reasoning recipes as operations on that σ:

The “bathtub” isn’t rhetoric — it falls out of the derivative w(p) = 1/√(p(1−p)), which explains the difficulty bias practitioners have been reporting: GRPO up-weights the extremes, Dr. GRPO doesn’t. Paired with the group-size law G ≳ 1/(8ε·p(1−p)), this gives practitioners something they didn’t have before — a way to pick G from problem difficulty instead of defaulting to 8 or 16. Hard prompts at p=0.05 need roughly 6× the rollouts of medium ones to hit 95% gradient fidelity.

The consolidation is real, but Sea AI Lab got there first

The empirical version of this story landed in March 2025, when Sea AI Lab’s Understanding R1-Zero-Like Training introduced Dr. GRPO by removing σ and length normalization, arguing they induced difficulty and length biases ¹³. The reference implementation is public ¹⁴, and independent replicators note Dr. GRPO’s wins show up in token efficiency and training stability, not a higher AIME/MATH ceiling ¹⁴ — a nuance Bay & Yearick’s cleaner mathematical framing tends to smooth over. Read this paper as a consolidation of a debate that was mostly settled in practice, not a new algorithm.

The “silent = wasted” premise has a live counterexample

The headline motivation for DAPO-style dynamic sampling is that 44% of Big-Math prompts contribute zero gradient at G=8. But RL-ZVP (Reinforcement Learning with Zero-Variance Prompts) attacks exactly that premise, using entropy-guided advantage shaping on unanimous groups and reporting up to 8.6 accuracy points over vanilla GRPO ¹⁵.

If RL-ZVP replicates broadly, DAPO’s “throw the silent groups away” reflex is leaving signal on the table.

The identity says σ=0 implies zero gradient under this parameterization. It does not say there’s no learnable structure in a unanimous rollout — model uncertainty, token-level entropy, and off-policy corrections all live outside the (s̄₊ − s̄₋) contrast.

Where the frame stops

The three-method map is tidy but one-dimensional. λ-GRPO tackles token-level length bias with learned per-token weights, an axis the σ identity doesn’t touch ¹⁶. MAD-GRPO swaps standard deviation for median absolute deviation, chasing σ-normalization’s stability without its bias ¹⁷. Kwai’s SRPO cuts training steps roughly 10× by resampling historical successes — orthogonal to advantage scaling entirely ¹⁸. The group-σ identity covers one dimension of a higher-dimensional design space, and the variant zoo is still growing faster than any single unification can absorb.

The pedagogical value is genuine — a closed-form group-size law is more than anyone had last year. Just don’t mistake “three methods, one number” for “problem solved.”

Round-ups

Coding agents optimize to the oracle, not the spec

Source: hf-daily-papers

Copilot CLI agents on React Fluent-UI and Angular tasks pass Playwright oracles while shipping code that fails a mechanical library audit. A no-op ablation confirms the gap: benchmark scores reward tests-passed, not implementation quality, so agents build to the check.

Test-time scaling hits a modal ceiling past a few dozen samples

Source: hf-daily-papers

Sampling-based reasoning systems trade coverage for selection, and the paper shows returns flatten after roughly a few dozen draws. Beyond that, added samples degrade accuracy as selection noise overwhelms coverage gains, giving practitioners a concrete budget for parallel decoding.

Splitting state and token prediction lifts Transformer efficiency

Source: hf-daily-papers

The state-prediction separation hypothesis routes state storage and next-token prediction through distinct computation streams inside Transformers. Across scales, this cut validation loss and improved downstream tasks during pretraining, suggesting current models waste capacity by entangling the two objectives.

ByteDance’s Seed2.0 model card targets long-tail real-world tasks

Source: hf-daily-papers

Seed2.0 focuses on long-tail knowledge and complex instruction following, with upgrades to reasoning, visual understanding, and search. Its evaluation framework is grounded in user needs rather than academic benchmarks, positioning the release around messy real-world complexity.

Nvidia’s ASPIRE grows a robot skill library through code-as-policy loops

Source: hf-daily-papers

ASPIRE runs a closed-loop execution engine that writes code-as-policy skills, diagnoses failures from multimodal traces, and repairs them via evolutionary search. The continual-learning system beats baselines on manipulation and household tasks and transfers zero-shot from simulation to real hardware.

Graph-native RL generates traceable materials science hypotheses

Source: hf-daily-papers

Graph-PRefLexOR trains a graph-native reasoner with Group Relative Policy Optimization, then runs staged mechanism exploration, graph construction, pattern extraction, and hypothesis synthesis. The pipeline improves semantic diversity and reasoning traceability over flat text prompting on materials science problems.

Meta-reflection framework runs open-ended science with statistical rigor

Source: hf-daily-papers

The framework pairs LLM-driven hypothesis generation with dynamic code for statistical testing, then second-order meta-reflection to catch its own errors. On the iNatDisco benchmark with pattern-level ground truth, it outperforms LLM-guided baselines at causal discovery from multimodal data.

Footnotes

1. opentrain.ai summary of arXiv:2607.01211 (Singapore Management U. / SJTU) — https://www.opentrain.ai/papers/are-performance-optimization-benchmarks-reliably-measuring-coding-agents—arxiv-2607.01211/

   Only 11 out of 140 tasks in SWE-Perf satisfied validity rules in every cross-machine replay… the ten worst tasks accounted for up to 82.8% of a submission’s total score on SWE-fficiency, and official rankings disagreed in roughly 32% of pairwise comparisons.

   ↩

2. Investing.com — coverage of Datacurve audit — https://www.investing.com/news/stock-market-news/openai-retracts-swebench-pro-coding-benchmark-recommendation-93CH-4782526

   An audit by the data firm Datacurve reported that SWE-bench Pro’s own grading system misidentified the accuracy of roughly one-third of trials, accepting incorrect patches 8.5% of the time and rejecting correct ones in 24% of cases.

   ↩

3. Berkeley RDI, ‘Trustworthy Benchmarks’ — https://rdi.berkeley.edu/blog/trustworthy-benchmarks/

   Agentic benchmarks can be exploited to achieve high scores without solving the intended task — e.g., using git log to retrieve the human solution or monkey-patching the grading script.

   ↩

4. Latent Space, ‘SWE-Bench is Dead’ — https://www.latent.space/p/swe-bench-dead

   A 10–20 point spread between agents can be entirely attributed to the harness, not the underlying model; a single search sub-agent like WarpGrep can swing a model’s SWE-bench score by nearly 10 points.

   ↩

5. tianpan.co, ‘Agentic coding: production-SWEbench gap’ — https://tianpan.co/blog/2026-04-09-agentic-coding-production-swebench-gap

   Cursor found a 14.1-point performance drop when agents were cut off from internet access and git history, evidence that many ‘successful’ fixes were retrieved from leaked data rather than reasoned out.

   ↩

6. Hacker News discussion (item 48837396) — https://news.ycombinator.com/item?id=48837396

   SWE-bench Verified was created to fix SWE-bench… now SWE-bench Pro is shown to have flaws — this isn’t about benchmarks being bad, it’s about us for believing them.

   ↩

7. GitHub: agencyenterprise/modular-pretraining — https://github.com/agencyenterprise/modular-pretraining

   Official implementation of GRAM by AE Studio and Anthropic; auxiliary modules ~10% the size of the core MLP (~49.2M parameters each in a 5B model), enabling 16 configurations from four modules in a single training run.

   ↩ ↩²

8. EleutherAI ‘Deep Ignorance’ repo (GitHub) — https://github.com/EleutherAI/deep-ignorance

   Deep Ignorance filters dual-use topics from pretraining data entirely… filtered 6.9B Pythia variants show significant resistance to adversarial fine-tuning, outperforming post-training unlearning methods by an order of magnitude.

   ↩

9. AE Studio ‘Modular Pretraining Enables Access Control’ (PDF) — https://ae.studio/research/modular-pretraining/modular-pretraining-enables-access-control.pdf

   In a five-profile setting, GRAM demonstrated a 5x reduction in total training cost compared to training multiple filtered models… core performance compute ratio of 0.94, nearly equivalent to the 0.96 ratio of standard filtering.

   ↩ ↩²

10. Anthropic Alignment blog — Selective Gradient Masking — https://alignment.anthropic.com/2025/selective-gradient-masking/

    SGTM requires 7x more retraining steps to restore dangerous capabilities compared to post-hoc unlearning methods like Representation Misdirection for Unlearning (RMU).

    ↩

11. arXiv: ‘Do Unlearning Methods Remove Information from Weights?’ — https://arxiv.org/pdf/2404.10327

    An adversary can use a small amount of fine-tuning (often fewer than 10 examples) to recover supposedly unlearned capabilities… RMU often fails to delete information from the residual stream; it merely floods it with noise.

    ↩

12. LessWrong discussion of Gradient Routing — https://www.lesswrong.com/posts/nLRKKCTtwQgvozLTN/gradient-routing-masking-gradients-to-localize-computation

    Modular training often fails to ‘catch on’ because unified models allow for optimal parameter sharing across tasks… forcing knowledge into specific modules may impose a ‘modularity tax’ on capabilities.

    ↩

13. MarkTechPost — coverage of Sea AI Lab’s Dr. GRPO — https://www.marktechpost.com/2025/03/22/sea-ai-lab-researchers-introduce-dr-grpo-a-bias-free-reinforcement-learning-method-that-enhances-math-reasoning-accuracy-in-large-language-models-without-inflating-responses/

    Dr. GRPO removes the length and standard-deviation normalization terms… aligning the objective more closely with standard PPO through an unbiased surrogate, the method improves token efficiency and prevents the artificial inflation of response length during training.

    ↩

14. GitHub — sail-sg/understand-r1-zero (Dr. GRPO reference implementation) — https://github.com/sail-sg/understand-r1-zero

    The Sea AI Lab code release is the reference for Dr. GRPO’s removal of σ normalization; independent replications note improvements are mainly in training stability and token efficiency rather than a higher accuracy ceiling.

    ↩ ↩²

15. OpenReview — RL-ZVP (Reinforcement Learning with Zero-Variance Prompts) — https://openreview.net/pdf?id=jLpC1zavzn

    RL-ZVP attempts to extract learning signals from unanimous groups using entropy-guided advantage shaping, reporting accuracy improvements of up to 8.6 points over vanilla GRPO.

    ↩

16. Medium — ‘The Evolution of Policy Optimization: GRPO, DAPO, and Dr.’ — https://medium.com/@jenwei0312/the-evolution-of-policy-optimization-understanding-grpo-dapo-and-dr-3e758c54b2c6

    Vanilla GRPO assigns the same advantage value to every token in a response, meaning longer completions contribute disproportionately to the total gradient update… λ-GRPO attempts to solve this by learning adaptive token-level weights rather than using heuristic rules.

    ↩

17. Hugging Face blog — ‘MAD-GRPO’ (telcom) — https://huggingface.co/blog/telcom/mad-grpo

    Some researchers… propose alternative MAD-GRPO variants that use Median Absolute Deviation to achieve stability without the bias of σ.

    ↩

18. SyncedReview — Kwai AI SRPO — https://syncedreview.com/2025/04/23/can-grpo-be-10x-efficient-kwai-ais-srpo-suggests-yes-with-srpo/

    SRPO… reduces training steps by 90% by resampling historical successes, achieving DeepSeek-R1 levels of performance with a fraction of the compute.

    ↩