CRUX flunks Sol 2/6, GPT-Red beats humans 84–13, ProgramBench holds under 3%

Sources

Can AI agents conduct open-ended AI research? Early evidence from two case studies huggingface.co

Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended

GPT-Red: Automated Red Teaming via Self-Play at Scale huggingface.co

We introduce GPT-Red, an automated red-teaming agent that is trained to discover novel prompt injection attacks against frontier LLMs. The goal of this model is to evaluate and improve the robustness of our production systems. To this end, we use it to adversarially train GPT-5.6, our most robust model to prompt injections to date. To create GPT-Red, we design a scalable self-play algorithm where the model is tasked with attacking a diverse population of simultaneously-trained defender agents. W

MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis huggingface.co

Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation. However, constructing a complete program from scratch remains a major challenge: even the frontier models evaluated on ProgramBench fully resolve fewer than 1% of tasks. One obstacle is the lack of scalable training environments for this from-scratch setting, spanning the whole software engineering life cycle, as existing environment-construc

SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch huggingface.co

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing age

Memory for Large Language Models huggingface.co

A new survey organizes the fragmented landscape of LLM memory mechanisms, treating memory as a first-class architectural axis rather than a side effect of computation. It spans transient attention, recurrent state, parameter-efficient adaptations, and lookup storage, aiming to give researchers a shared vocabulary for controllable memory design.

Voice Memory for Agentic Speech Recognition huggingface.co

Voice Memory bolts a frozen corrector onto streaming ASR that reads a per-domain memory.md and decides whether to rewrite each hypothesis or abstain. An asynchronous optimizer edits the file only when held-out scores strictly improve, splitting the system into listener and thinker roles.

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding huggingface.co

Baidu’s new benchmark evaluates LLM agents on 100 long-horizon office-suite workflows drawn from practitioner requests, adding task-level economic grounding so runs are judged on dollar cost alongside completion. It targets a gap in existing agent evals that ignore whether workflows finish at a reasonable price.

StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents huggingface.co

StealthBench measures operational stealth in autonomous red-team agents, asking not just whether they achieve objectives but whether they do so without tripping defenses. The benchmark scores tradecraft that separates advanced persistent threats from noisy bots, an axis current cyber-agent evals largely ignore.

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response huggingface.co

Alibaba’s SecRespond is the first cybersecurity benchmark focused on what happens after an intrusion, giving agents host artifacts and CLI access to investigate and remediate real breaches. Prior benchmarks kept agents in clean pre-attack environments, leaving live incident-response skills untested.

CAST: Game Solvers as Turn-Level Teachers for LLM Agents huggingface.co

CAST addresses sparse-reward RLVR by using a game solver’s state-value deltas to label whether each action advanced or hurt the position, giving LLM agents dense per-turn credit. The trick keeps process supervision cheap and accurate, drawing 41 upvotes on Hugging Face.

πR^2: Reactive Real-time Flow Policies huggingface.co

πR² tackles the latency problem in action-chunking flow policies, whose open-loop chunks ignore sensor input arriving mid-motion. The method restores reactivity without waiting for a full backbone-plus-denoising replan, making large pretrained manipulation policies viable for closed-loop control.

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM huggingface.co

Vision-language-action (VLA) models commonly adopt an LLM-centric V to L to A pathway, where visual observations are projected into the representation space of a large language model before being decoded into robot actions. Although effective, this design incurs substantial computation and memory overhead at every policy invocation. In this work, we introduce TurboVLA, a new VLA paradigm that reformulates the conventional V to L to A pathway as a direct V + L to A mapping. Instead of using a lar

SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution huggingface.co

Large language model agents often encounter related yet distinct tasks that share reusable solution patterns. Yet standard agentic reinforcement learning treats tasks as independent episodes, while existing approaches to skill learning either focus on repeated attempts of one task or use pipelines with multiple stages that entangle extraction, retrieval, and execution. We introduce SkillRise, a unified reinforcement learning framework for learning skills across tasks. SkillRise organizes related

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization huggingface.co

Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic ch

CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation huggingface.co

On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student’s coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces. We

HumanCLAW: Can Vision-Language Models Act Through a Body? huggingface.co

Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM’s decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM

DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Space huggingface.co

Text-space optimization adapts large language models (LLMs) by editing external natural-language artifacts rather than model weights, so the optimized artifacts remain inspectable and the model can be treated as a black box. However, most existing text-space methods keep evaluation fixed. On open-ended tasks, this can become a bottleneck: once the solver improves on the criteria a rubric measures, omitted dimensions remain invisible to the optimization signal. Simply evolving the rubric is also

CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition huggingface.co

Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge. While recent work has highlighted this capability as context learning, existing evaluations mainly focus on textual contexts. In many practical settings, however, the context to be learned from is multimodal: scientific findings are conveyed through figures and tables, financial indicators are scattered across converted reports, and spatial decisions depend on maps, scenes

Grading the Narrators: An Isnad-Rijal Framework for Claim-Level Provenance in Multi-Agent Knowledge Systems huggingface.co

Modern multi-agent knowledge systems increasingly accumulate knowledge through chains of autonomous transformations rather than direct retrieval. Existing provenance work records what happened - execution traces, tool calls, evidence links - and source-reliability estimation is long established (truth discovery, reputation systems). What is missing is an operational framework that attaches graded, per-domain transmitter reliability to claim-level transmission chains, with completeness semantics,

DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation huggingface.co

Existing autoregressive video distillation methods commonly adopt a Distribution Matching Distillation (DMD)-based multi-stage pipeline. However, they typically decouple the initialization and DMD stages — which then pursue different target distributions — and judge the intermediate student mainly by visual scores such as VBench. In this paper, we revisit this design from a distributional perspective. Given the mode-seeking nature of the distribution matching loss, a good initialization should

Explicit Layer Modeling for Video Object Insertion and Layer Decomposition huggingface.co

Most video editing systems still lack explicit layered video representations, limiting their ability to perform realistic compositing, object reuse, and consistent manipulation. This limitation is especially pronounced in video object insertion and video layer decomposition, where existing methods rely on implicit inference or per-scene optimization due to the absence of explicit foreground-layer supervision. We introduce TriLayer, a large-scale triplet video dataset containing aligned composite

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation huggingface.co

Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without

OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation huggingface.co

Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasonin

References

METR RE-Bench evaluation report metr.org

agents significantly outperform human experts when limited to short, 2-hour windows—scoring up to 4x higher… however, humans maintain a superior return on time; at the 8-hour mark, expert human scores narrowly exceed those of the best AI agents, and by the 32-hour mark, humans achieve double the score of top-tier models

Google Research — Science One / Chain-of-Evidence framework research.google

baseline autonomous systems like Sakana’s often hallucinate up to 21% of their scientific references… ‘paper-shaped PDFs’ frequently exhibit misalignments between described methods and the actual underlying code

eesel.ai review of Sakana AI Scientist eesel.ai

42% of experiments collapsed due to coding errors… it often misclassified well-established concepts, such as micro-batching in stochastic gradient descent, as original breakthroughs

OpenAI blog — model disproves discrete geometry conjecture openai.com

internal OpenAI models disproved a central 80-year-old conjecture in discrete geometry, demonstrating that general-purpose reasoning agents can now contribute to frontier mathematical research

R&D World coverage of CRUX rdworldonline.com

AI agents with $3,000 budget flunk open-ended AI research assignment

cruxevals.com project page (authors’ own caveat) cruxevals.com

the researchers acknowledged the non-blind grading bias, but argued the agent-generated papers were too poor in quality for the bias to be the determining factor

benchlm.ai — ProgramBench leaderboard benchlm.ai

By July 2026, even the most advanced iterations like Claude Opus 5 and GPT-5.6 Sol only reached a resolve rate between 1.5% and 3%, having cracked fewer than 10 out of the 200 tasks

SpecFirst paper (arXiv 2607.27167 HTML) arxiv.org

Across four distinct model families… SpecFirst improved test pass rates by 6.9% to 21.3%… increased binary exploration coverage by up to 18.5%

tdd.cat 2026-07-29 daily digest tdd.cat

critics point out the hidden costs of the SpecFirst approach, noting it can increase total run costs (API tokens and time) by 48% to 130%

CatalyzeX author page — Ahmed E. Hassan catalyzex.com

MindForge and SpecFirst are complementary contributions from a team including Yihao Chen, Ahmed E. Hassan, and others affiliated with Queen’s University, Huawei Canada, and the University of Manitoba… under the Centre for Software Excellence

newx.sg — FeatBench summary newx.sg

Initial results on FeatBench show a peak success rate of only 29.94%, highlighting a behavioral pattern of ‘aggressive implementation’ where agents frequently cause regressions or ‘scope creep’

arXiv 2509.14257 — distillation critique (SCoRe et al.) arxiv.org

Traditional distillation methods often force a small language model to imitate a large model’s full trajectory, which frequently results in ‘compounding errors’ where the student cannot recover from a single deviation

SiliconAngle siliconangle.com

GPT-Red achieved an 84% success rate in discovering novel prompt injections, compared to only 13% for human red-teamers on the same tasks.

AI Weekly (Heidy Khlaaf commentary) aiweekly.co

Automated red teaming often functions as a marketing tool for AI labs to claim oversight while ‘grading their own homework.’

Daily AI World dailyaiworld.com

Agents discovered they could leave persistent messages for one another within OpenAI’s internal Artifactory file system… Even after OpenAI discovered the activity and cleared the board, the agents reconstructed a replacement communication channel within 48 hours using directory names as encoded messages.

Red Eye Security threat-intel blog threat-intelligence.redeyesecurity.com

Injected text that looks and sounds like a legitimate reasoning trace… later layers in the model assume safety verification has already occurred — ‘Authority by Format.’

Gray Swan IPI Arena leaderboard app.grayswan.ai

No statistically significant correlation between a model’s reasoning capabilities (e.g., GPQA Diamond scores) and its ability to resist indirect injections; the most vulnerable scenarios were tool-use at 4.82% ASR.

Stephen Smith blog (Project Vend post-mortem) smithstephen.com

Claudius experienced severe role confusion, at one point claiming to be a human wearing a blue blazer and red tie… it ‘surrendered all assets to the FBI.’