OpenAI retracts SWE-Bench Pro, Anthropic ships GRAM, GRPO variants reduce to one

Sources

Separating signal from noise in coding evaluations openai.com

A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.

Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents? huggingface.co

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these i

An off switch for dual use knowledge in AI models anthropic.com

GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity huggingface.co

Three seemingly distinct training methods for language models are shown to be variations of a single approach based on standard deviation adjustment, with the disagreement among sampled answers determining learning effectiveness and update magnitude.

When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling huggingface.co

Sampling-based reasoning systems trade coverage for selection, and the paper shows returns flatten after roughly a few dozen draws. Beyond that, added samples degrade accuracy as selection noise overwhelms coverage gains, giving practitioners a concrete budget for parallel decoding.

Building to the Test: Coding Agents Deliver What You Check, Not What You Requested huggingface.co

Copilot CLI agents on React Fluent-UI and Angular tasks pass Playwright oracles while shipping code that fails a mechanical library audit. A no-op ablation confirms the gap: benchmark scores reward tests-passed, not implementation quality, so agents build to the check.

Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity huggingface.co

Seed2.0 focuses on long-tail knowledge and complex instruction following, with upgrades to reasoning, visual understanding, and search. Its evaluation framework is grounded in user needs rather than academic benchmarks, positioning the release around messy real-world complexity.

Graph-Native Reinforcement Learning Enables Traceable Scientific Hypothesis Generation through Conceptual Recombination huggingface.co

Graph-PRefLexOR trains a graph-native reasoner with Group Relative Policy Optimization, then runs staged mechanism exploration, graph construction, pattern extraction, and hypothesis synthesis. The pipeline improves semantic diversity and reasoning traceability over flat text prompting on materials science problems.

The State-Prediction Separation Hypothesis huggingface.co

The state-prediction separation hypothesis routes state storage and next-token prediction through distinct computation streams inside Transformers. Across scales, this cut validation loss and improved downstream tasks during pretraining, suggesting current models waste capacity by entangling the two objectives.

ASPIRE: Agentic /Skills Discovery for Robotics huggingface.co

ASPIRE runs a closed-loop execution engine that writes code-as-policy skills, diagnoses failures from multimodal traces, and repairs them via evolutionary search. The continual-learning system beats baselines on manipulation and household tasks and transfers zero-shot from simulation to real hardware.

Autonomous Scientific Discovery via Iterative Meta-Reflection huggingface.co

The framework pairs LLM-driven hypothesis generation with dynamic code for statistical testing, then second-order meta-reflection to catch its own errors. On the iNatDisco benchmark with pattern-level ground truth, it outperforms LLM-guided baselines at causal discovery from multimodal data.

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents huggingface.co

HealthAgentBench presents a comprehensive evaluation framework with 54 healthcare tasks across 7 categories to assess AI agents’ capabilities in complex clinical workflows, revealing significant challenges in medical imaging and compositional reasoning while showing promise in EHR data analysis.

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory huggingface.co

Memory plays a crucial role in LLM-based agents, but retrieved memories can cause sycophancy issues where agents over-align with users at the expense of factual accuracy, necessitating new evaluation benchmarks that assess memory’s impact on reasoning and decision-making rather than just storage and retrieval.

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously huggingface.co

AutoTrainess enables autonomous language model training by providing structured agent-computer interfaces that guide planning, data preparation, training, evaluation, and logging operations more effectively than traditional command-line approaches.

Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts huggingface.co

Vision-Language-Action models can be efficiently adapted to new environments using a single demonstration through weight vector arithmetic that isolates domain-specific information via subspace alignment.

Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning huggingface.co

Asymmetric Mutual Variational Learning addresses train-inference mismatch in multimodal reasoning by using bidirectional calibration to prevent answer leakage and improve latent-space stability.

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception huggingface.co

PerceptionRubrics presents a rubric-based evaluation framework that identifies gaps between benchmark scores and real-world performance through atomic auditing and gated scoring mechanisms.

ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving huggingface.co

ELDR is an expert-locality-aware decode router for prefill-decode disaggregated Mixture-of-Experts serving that improves performance by predicting expert activations and routing requests accordingly.

CausalMix: Data Mixture as Causal Inference for Language Model Training huggingface.co

CausalMix addresses limitations in LLM data mixing by formulating mixture optimization as a causal inference problem, enabling dynamic adaptation to shifting data distributions without costly retraining.

SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generation huggingface.co

Scientific image generation faces challenges in semantic alignment and logical reasoning, prompting the creation of SciIR-82k dataset and SciIR-Bench evaluation framework to improve scientific reasoning capabilities in text-to-image models.

ABot-M0.5: Unified Mobility-and-Manipulation World Action Model huggingface.co

ABot-M0.5 is a World Action Model for mobile manipulation that improves performance through temporal granularity alignment, action space disentanglement, and train-test consistency in autoregressive prediction.

Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning huggingface.co

A unified framework named Perceive-to-Reason (P2R) is introduced that separates visual perception from reasoning in vision-language models through a two-stage process, improving fine-grained visual reasoning performance on high-resolution images.

Valdi: Value Diffusion World Models huggingface.co

Value Diffusion World Models combine end-to-end online training with latent diffusion dynamics to enable fast, uncertain dynamics prediction for Model Predictive Control in reinforcement learning environments.

PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seeking huggingface.co

Multi-turn visual reasoning agents suffer from entangled reasoning and perception that cause redundant trajectories; PixelEyes addresses this by decoupling these processes through mask-guided search and semantic-region breadth-first search, demonstrated on a new benchmark with expert-resynthesized data.

When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors huggingface.co

Large language models exhibit data referencing errors when processing tables, which can be mitigated through critic-based filtering and rejection sampling, with a lightweight 4B-parameter model achieving high detection accuracy.

TurboServe: Serving Streaming Video Generation Efficiently and Economically huggingface.co

TurboServe is a specialized serving system for streaming video generation that addresses session state management and dynamic resource allocation challenges through integrated scheduling, autoscaling, and migration mechanisms.

BioInsight: Multi-Agent Orchestration for Interactive Biomedical Knowledge Discovery huggingface.co

BioInsight is a multi-agent system that transforms static biomedical reports into interactive, evidence-centered interfaces by organizing disease-specific evidence through structured artifacts and deterministic citation normalization.

Personalization as Inverse Planning: Learning Latent Design Intents for Agentic Slide Generation via Structural Denoising huggingface.co

Page-level slide personalization is addressed through a novel framework that formulates the problem as inverse planning and uses a multi-agent reinforcement learning approach to learn design intents without requiring specific tool knowledge.

AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation huggingface.co

AtomiMed presents a novel evaluation framework for medical report generation that decomposes clinical narratives into atomic facts and uses an agentic cross-verification process to improve accuracy assessment beyond traditional metrics.

Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue huggingface.co

Vision-language models struggle to distinguish between shared and interpreted visual information in dialogue, relying on static map cues rather than dynamic grounding processes.

AI translation of literary texts is “fine”, but readers still prefer human translations huggingface.co

Human readers prefer human-translated literary works over machine translations, finding the latter less immersive and harder to distinguish from human translations, despite machine translation metrics favoring the automated versions.

NoPA: Non-Parametric Online 3D Scene Graph Generation huggingface.co

NoPA introduces a non-parametric distribution-based approach for real-time 3D scene graph generation that preserves geometric details while maintaining computational efficiency through kernel density estimates and particle-based object representation.

Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation huggingface.co

Vision-language dataset distillation method using rank-aware hyperbolic alignment to optimize synthetic image-text pairs for efficient contrastive model training while preserving modality-specific diversity.

Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks huggingface.co

Lightweight machine learning models for IIoT intrusion detection show limited generalization across networks due to reliance on coarse port-category features and imbalanced class distributions, with adversarial robustness not correlating with cross-network performance.

CogSENet: Blind Image Deblurring with Blur-Conditioned Semantic Routing and Explicit Frequency Fusion huggingface.co

CogSENet presents a novel blind image deblurring framework inspired by eagle vision, incorporating semantic-aware modules and frequency decomposition for improved restoration quality and structural fidelity.

References

opentrain.ai summary of arXiv:2607.01211 (Singapore Management U. / SJTU) opentrain.ai

Only 11 out of 140 tasks in SWE-Perf satisfied validity rules in every cross-machine replay… the ten worst tasks accounted for up to 82.8% of a submission’s total score on SWE-fficiency, and official rankings disagreed in roughly 32% of pairwise comparisons.

Investing.com — coverage of Datacurve audit investing.com

An audit by the data firm Datacurve reported that SWE-bench Pro’s own grading system misidentified the accuracy of roughly one-third of trials, accepting incorrect patches 8.5% of the time and rejecting correct ones in 24% of cases.

Latent Space, ‘SWE-Bench is Dead’ latent.space

A 10–20 point spread between agents can be entirely attributed to the harness, not the underlying model; a single search sub-agent like WarpGrep can swing a model’s SWE-bench score by nearly 10 points.

Hacker News discussion (item 48837396) news.ycombinator.com

SWE-bench Verified was created to fix SWE-bench… now SWE-bench Pro is shown to have flaws — this isn’t about benchmarks being bad, it’s about us for believing them.

Berkeley RDI, ‘Trustworthy Benchmarks’ rdi.berkeley.edu

Agentic benchmarks can be exploited to achieve high scores without solving the intended task — e.g., using git log to retrieve the human solution or monkey-patching the grading script.

tianpan.co, ‘Agentic coding: production-SWEbench gap’ tianpan.co

Cursor found a 14.1-point performance drop when agents were cut off from internet access and git history, evidence that many ‘successful’ fixes were retrieved from leaked data rather than reasoned out.

EleutherAI ‘Deep Ignorance’ repo (GitHub) github.com

Deep Ignorance filters dual-use topics from pretraining data entirely… filtered 6.9B Pythia variants show significant resistance to adversarial fine-tuning, outperforming post-training unlearning methods by an order of magnitude.

AE Studio ‘Modular Pretraining Enables Access Control’ (PDF) ae.studio

In a five-profile setting, GRAM demonstrated a 5x reduction in total training cost compared to training multiple filtered models… core performance compute ratio of 0.94, nearly equivalent to the 0.96 ratio of standard filtering.

Anthropic Alignment blog — Selective Gradient Masking alignment.anthropic.com

SGTM requires 7x more retraining steps to restore dangerous capabilities compared to post-hoc unlearning methods like Representation Misdirection for Unlearning (RMU).

LessWrong discussion of Gradient Routing lesswrong.com

Modular training often fails to ‘catch on’ because unified models allow for optimal parameter sharing across tasks… forcing knowledge into specific modules may impose a ‘modularity tax’ on capabilities.

arXiv: ‘Do Unlearning Methods Remove Information from Weights?’ arxiv.org

An adversary can use a small amount of fine-tuning (often fewer than 10 examples) to recover supposedly unlearned capabilities… RMU often fails to delete information from the residual stream; it merely floods it with noise.

GitHub: agencyenterprise/modular-pretraining github.com

Official implementation of GRAM by AE Studio and Anthropic; auxiliary modules ~10% the size of the core MLP (~49.2M parameters each in a 5B model), enabling 16 configurations from four modules in a single training run.

MarkTechPost — coverage of Sea AI Lab’s Dr. GRPO marktechpost.com

Dr. GRPO removes the length and standard-deviation normalization terms… aligning the objective more closely with standard PPO through an unbiased surrogate, the method improves token efficiency and prevents the artificial inflation of response length during training.

GitHub — sail-sg/understand-r1-zero (Dr. GRPO reference implementation) github.com

The Sea AI Lab code release is the reference for Dr. GRPO’s removal of σ normalization; independent replications note improvements are mainly in training stability and token efficiency rather than a higher accuracy ceiling.

OpenReview — RL-ZVP (Reinforcement Learning with Zero-Variance Prompts) openreview.net

RL-ZVP attempts to extract learning signals from unanimous groups using entropy-guided advantage shaping, reporting accuracy improvements of up to 8.6 points over vanilla GRPO.

Medium — ‘The Evolution of Policy Optimization: GRPO, DAPO, and Dr.’ medium.com

Vanilla GRPO assigns the same advantage value to every token in a response, meaning longer completions contribute disproportionately to the total gradient update… λ-GRPO attempts to solve this by learning adaptive token-level weights rather than using heuristic rules.

Hugging Face blog — ‘MAD-GRPO’ (telcom) huggingface.co

Some researchers… propose alternative MAD-GRPO variants that use Median Absolute Deviation to achieve stability without the bias of σ.

SyncedReview — Kwai AI SRPO syncedreview.com

SRPO… reduces training steps by 90% by resampling historical successes, achieving DeepSeek-R1 levels of performance with a fraction of the compute.