Atria Dawn unreproduced, Dream-RSI collapses to 1.7×, NGU doubles on async RL

TL;DR

• Atria Dawn Preview claims 5 SOTA wins across 16 benchmarks with zero third-party reproduction.
• The same 744B agent ranks bottom-tier on SWE-Bench Pro and GDPval.
• Dream-RSI’s 162× speedup shrinks to ~1.7× under a matched fixed-exploration baseline.
• NGU doubles hard-subset GSM8k to 87.9% vs 47.1% by resampling failed prompts.
• NGU requires async RL infra — vLLM pause/resume plus age-cutoff bookkeeping — or gains vanish.

Two of today’s research features arrive with headline numbers that don’t survive an outside look. Atria Dawn Preview, a 744B-parameter MoE agent, claims SOTA on 5 of 16 benchmarks and gets pitched at the superintelligence frame — but no third-party leaderboard has reproduced any of it, and the same model sits bottom-tier on SWE-Bench Pro and GDPval. Dream-RSI’s 162× agent-call reduction over SimpleTES collapses to roughly 1.7× once you match the exploration budget on the same model, with a random-selection ablation still missing.

The third feature is the counterpoint. NGU posts a genuine near-doubling on hard-subset GSM8k (87.9% vs 47.1%) by resampling prompts that fixed-K GRPO wastes compute on — a clean fix to what the authors call the Matthew Effect in RL. The catch is infrastructural: without async vLLM pause/resume and the age-cutoff bookkeeping, the same idea just produces stragglers. Real gain, contingent plumbing.

Atria Dawn’s 744B agent claims 5 SOTA wins, none reproduced

Source: hf-daily-papers · published 2026-09-13

TL;DR

• Atria Dawn Preview, a 744B-parameter MoE agent, claims SOTA on 5 of 16 benchmarks including CyberGym 86.5.
• No third-party leaderboard has reproduced any of the scores as of late September 2026.
• The same model ranks bottom-tier on SWE-Bench Pro and GDPval, undercutting the “frontier productivity” framing.
• Princeton’s shadow evaluation says frontier agents still lack “scientific taste” — hard to square with the “superintelligence” label.

A 744B agent trained on verified trajectories

Atria Dawn Preview is a 744B-parameter Mixture-of-Experts model built on Z.ai’s foundation, trained through what the authors call a Verifiable Experience Pipeline: agent trajectories in live terminals, IDEs, and CAD tools are kept in the training set only when an external signal — a passing unit test, a geometric constraint, a benchmark delta — validates the outcome. Failure modes are then mined for new training tasks. It’s a coherent recipe, and it produces headline numbers: AutomationBench 53.8 (leading by 4.1), BFCL v4 77.0, DeepSearchQA 96.0, BrowseComp 92.5, CyberGym 86.5.

The benchmark table is a marketing document

Every one of those numbers is vendor-reported. Contra Collective’s summary is the one to keep in mind: “the benchmark table is a marketing document until someone else runs it.” ¹ No independent leaderboard — Artificial Analysis, HELM, or otherwise — has reproduced the scores yet.

The unevenness is the tell. EnterpriseDNA flags a “performance paradox”: the same model that tops search and cyber sits at the bottom of its peer group on SWE-Bench Pro (59.6) and GDPval ², where GPT-5.6 Sol and others still lead. “Leading tier” is really a claim about a couple of sub-domains.

Practitioners aren’t seeing the polished paper either. r/opencodeCLI threads describe repeated HTTP 422 — custom endpoint rejected this request as malformed failures when wiring Atria Dawn into agent harnesses ³. Combined with a 744B footprint that demands Blackwell-class hardware, the MIT license is closer to a research artifact than a usable open-weight release.

The R&D case study has a bias problem

The paper’s most quotable claim — that 33.2% of tasks completed with AI were rated “infeasible” without it — comes from self-reports by the same 56 people who built the system, with no blinded control ⁴. The 11.0 → 28.5 jump in agent-actions-per-prompt is real motion, but it’s motion inside a team incentivized to see motion.

Princeton’s shadow-evaluation work by Kirgis and Kapoor lands harder. Given a week and thousands of dollars in compute, frontier agents could execute engineering tasks but produced work that expert reviewers unanimously rejected as unpublishable, citing missing “scientific taste,” instruction drift, and inability to backtrack ⁵. That failure profile maps almost exactly onto Atria’s own admitted limits — homogeneous proposals, correlated blind spots, experience that never gets absorbed — yet the paper still brands the release “the dawn of agentic superintelligence.”

An open-weights CyberGym leader with no red-team

The CyberGym 86.5 is the most operationally consequential number in the paper and the least discussed. AP’s reporting on autonomous vulnerability research is blunt: “the technical skills required to perform automated patching are functionally identical to those needed for offensive exploitation.” ⁶ Shipping 744B open weights that top a security benchmark, with no red-team disclosures and no gating, is the fault line most likely to define this release outside the benchmarks discourse — well before anyone settles whether the scores replicate.

---

Dream-RSI’s 162× speedup shrinks to 1.7× under fair baseline

Source: hf-daily-papers · published 2026-09-13

TL;DR

• Dream-RSI replays past discovery trees to score new policies offline, claiming 162× fewer agent calls vs. SimpleTES on a Lasso task.
• Under a matched fixed-exploration baseline with the same model and budget, the gain collapses to ~1.7×.
• Reviewers flag a missing random-selection ablation, leaving replay ranking’s contribution unproven vs. just generating more policy variants.
• No outside lab has reproduced either number — repo unreleased, all runs on closed Gemini 3.1 Pro / 3.7 Flash.

The headline number is a baseline artifact

The pitch is genuinely clever: log every attempt a coding agent makes into a “discovery tree,” then let a higher-level policy-development agent generate thousands of alternative exploration strategies and score them by replaying the tree — no re-running the expensive coder or evaluator. VentureBeat led with the marquee figure: “up to 162x fewer discovery-agent calls by replaying searches it already ran” ⁷. That’s a real datapoint — on the Lasso-path task with Gemini 3.1 Pro, Dream-RSI reached the target in 317 agent calls, 162× fewer than SimpleTES.

But SimpleTES is a weak comparator. When shattered.io held the model and search budget constant and compared against a fixed-exploration baseline instead, the improvement dropped to a “consistent 1.7x gain,” which the reviewer called “an engineering-solid paper wrapped in an oversold narrative” ⁸. The KernelBench numbers in the paper itself corroborate this range — 1.79× fewer generations on LayerNorm, 2.43× on VGG16, 2.09× higher performance on ConvDiv. Single-digit efficiency wins, not orders of magnitude.

The ablation that isn’t in the paper

The sharpest technical critique comes from the alphaXiv community review, which points out a control the authors don’t run: randomly selecting among the revised policies the policy-development agent generates, rather than picking the one with the best replay score ⁹. Without that comparison — plus matched-call runs with repeated seeds — you can’t tell whether the gain comes from the “dreaming” evaluator or simply from producing more policy variants and picking any of them. That’s the load-bearing claim of the whole framework, and it’s untested.

Is this really “recursive self-improvement”?

Practitioners on r/reinforcementlearning object to the RSI label. The system doesn’t touch the coding agent’s weights; it optimizes an executable exploration policy — closer to “experience replay or system prompt optimization” than self-modification. And the replay simulator is “bounded by the state-space already explored” ¹⁰: it can re-rank branches the tree already contains, but it can’t score genuinely novel trajectories.

The paper’s own Section 5.1 ablation reinforces this ceiling in an interesting way. Injecting high-level “lessons learned” summaries into prompts hurt discovery performance versus replaying raw, messy history ¹¹ — the authors read this as evidence that strong inductive biases over-constrain the search. It’s also evidence that the system is a navigator of known maps, not a generator of new ones.

Reproducibility is currently blocked

The zhengkid/Dream-RSI repository is flagged as “being prepared for release,” with end-to-end training scripts absent or limited to specific demos as of late September 2026 ¹². Every reported experiment runs on closed Gemini 3.1 Pro or 3.7 Flash backbones. No independent lab has reproduced the 162× number — or the 1.7× one.

Takeaway

Strip away the RSI framing and there’s a useful contribution here: offline meta-policy evaluation via discovery-tree replay is a real efficiency knob, worth 2–3× under honest comparison. The problem is the packaging. A framework that claims recursive self-improvement should probably clear the bar of a random-selection control before the 162× number goes on the slide.

---

NGU nearly doubles hard-problem RL gains by never giving up

Source: hf-daily-papers · published 2026-09-10

TL;DR

• NGU nearly doubles hard-subset GSM8k accuracy to 87.9% vs 47.1% for standard GRPO by resampling failed prompts.
• Matthew Effect in RL: fixed-K sampling wastes compute on solved prompts and never generates gradient signal for hard ones.
• +4.3% on the hard subset on AIME 2025 with Qwen 3 4B, vs +2.5% for the best GRPO baseline.
• Requires async RL infrastructure — vLLM pause/resume plus age-cutoff and anchoring bookkeeping — or it just creates stragglers.

The pathology NGU is fixing

Vanilla GRPO trains on K completions per prompt and computes advantages within that group. When all K fail — the common case on genuinely hard reasoning prompts — the group’s variance is zero and no gradient flows. Noukhovitch et al. call the aggregate effect the Matthew Effect in RL for LLMs: models get systematically better at problems they were already competent on, while the hard tail stagnates. Their receipts include Olmo 3.1 and DeepCoder on LCBv5 and SWEBench Verified, where standard RL pipelines leave the hardest bucket essentially untouched.

The proposed fix is a scheduling change, not a new objective. Never Give Up (NGU) samples a small batch (K=4), and if every completion is wrong, puts the prompt back in the queue with probability p_NGU (typically 0.5–0.95). Repeat until a success shows up, then combine that lone correct answer with all the accumulated failures into one large GRPO group — producing a very strong advantage signal for the rare positive.

Where it sits in the GRPO family

NGU isn’t the first attack on the silent-group problem. DAPO handled it at the batch level by filtering zero-variance groups and oversampling until the batch is full of informative examples, reportedly hitting 50 on AIME 2024 vs DeepSeek-R1’s 47 with half the training steps ¹³. NGU differs by persisting per-prompt rather than per-batch, which is only tractable inside an asynchronous trainer where some workers can grind on one hard prompt while others race through easy ones.

A more deflationary reading: a recent “group-standard-deviation identity” analysis argues GRPO, Dr.GRPO and DAPO are “three operations on one number,” with binary-reward update magnitudes collapsing to the group’s std ¹⁴. On that framing NGU is another knob on the same dial — reallocating where variance gets manufactured — rather than a categorically new algorithm.

Benchmarks, in context

The headline AIME 2025 number is 22.0% with Qwen 3 4B-Base after Deepscaler RL. For scale: ByteDance’s POLARIS recipe, on the same base model and also DeepScaleR-derived, reports 79.4% on AIME 2025 after ~700 RL steps ¹⁵. NGU’s setup is a deliberately compute-constrained ablation, not a frontier run — a caveat the paper doesn’t foreground. The honest read of the contribution is the per-compute delta on the hard tail (+4.3% vs +2.5%), plus the striking “100% All-Tests Pass” result on Manufactoria coding problems where standard GRPO plateaus around 80% and never closes out the hardest test cases.

Caveats worth pricing in

NGU is inseparable from async infrastructure. The vLLM/OpenRLHF stack uses pause/resume APIs to broadcast weight updates to inference workers mid-flight ¹⁶ — without that, NGU just creates stragglers that idle the cluster. It also inherits an “initial competence” floor: the author explicitly notes that datasets skewed entirely toward unsolvable problems degrade into off-policy noise ¹⁷. And reproducing the recipe requires not just the algorithm but the age-cutoff T, the anchoring rescale, and the staleness bookkeeping around them — the kind of “complexity tax” Nathan Lambert has argued only becomes auditable when full logs and checkpoints ship alongside weights ¹⁸.

Net: a clean scheduling fix inside an increasingly crowded GRPO family, valuable for what it does to the hard tail per unit of compute — not a new SOTA.

Round-ups

Reasoning models flunk structurally identical rule-induction variants

Source: hf-daily-papers

Reasoning models handle rule-induction tasks inconsistently, breaking when researchers apply isomorphic recombination or symbol substitution that preserves underlying structure. The failures point to a lack of systematicity, with evaluation code released to probe compositional generalization across model families.

Elo-per-token analysis exposes when LLM agents stop scaling

Source: hf-daily-papers

LLM agents outpace independent sampling early in test-time compute but hit an inflection point where extra tokens stop paying off. Measured with a Bradley-Terry Elo-per-token metric on AtCoder Heuristic Contests, parallel short sessions beat single long runs at equal budget.

Continual Search improves root-cause diagnosis on long agent traces

Source: hf-daily-papers

Root-cause attribution in long-horizon agent failures is reframed as iterative search: an LLM diagnoser is re-prompted until unresolved evidence surfaces. On the new MegaRCA-Mix benchmark of extended execution traces, Continual Search outperforms single-pass attribution baselines.

RSIAgent adapts to new environments without any training

Source: hf-daily-papers

RSIAgent is a training-free multi-agent framework that recursively self-improves in unfamiliar digital environments. It builds memory autonomously and follows a broad-then-deep exploration policy that maps causal relationships before deepening skills, letting agents onboard to new apps without fine-tuning.

Grouped Value Attention shrinks KV cache by reconstructing keys on demand

Source: hf-daily-papers

Grouped Value Attention stores only grouped values in the KV cache and rebuilds keys via a learned linear map with decoupled RoPE. The scheme matches grouped-query attention accuracy while keeping a smaller persistent cache, cutting memory pressure during autoregressive decoding.

Orthrus speculative decoding stays lossless only at FP32 precision

Source: hf-daily-papers

Orthrus, an autoregressive-diffusion speculative decoder, holds its lossless guarantee only under high numerical precision. BF16 runs show trajectory divergence from FP32 references, yet downstream lm-eval-harness benchmarks and perplexity remain intact, suggesting the precision gap matters more for theory than practical quality.

ZGCM-1 ships as 7B open model for math and agentic search

Source: hf-daily-papers

ZGCM-1 is a fully open 7B foundation model that fuses internal reasoning with tool use through agentic co-training. Architecture-system co-design pairs gated sliding-window and full attention with an FP8 Muon optimizer, plus progressive long-context scaling across pre-, mid-, and post-training stages.

Footnotes

1. Contra Collective analysis — https://contracollective.com/blog/atria-dawn-preview-benchmarks-where-it-fits-2026

   the benchmark table is a marketing document until someone else runs it

   ↩

2. EnterpriseDNA review — https://enterprisedna.co/resources/news/atria-dawn-preview-744b-open-source-agent-enterprise-2026/

   Despite its high ranking in search and reasoning, the model lags behind competitors on benchmarks like SWE-Bench Pro and GDPval, suggesting its agentic capabilities may be unevenly distributed

   ↩ ↩²

3. Hyper.ai community roundup — https://hyper.ai/en/papers/2609.15818

   developers attempting to integrate the model via plugins reported frequent stability issues, specifically citing ‘HTTP 422’ errors and messages stating that ‘custom endpoint rejected this request as malformed’

   ↩

4. picx.dev methodology critique — https://picx.dev/p/fWsT3n

   The ‘infeasibility’ metric relies entirely on self-reported judgments from the researchers who built the model, introducing a potential for participant bias

   ↩

5. Princeton ‘shadow evaluation’ study (Kirgis & Kapoor) — https://communitynews.org/science-tech/princeton-study-ai-agents-open-ended-research/

   agents could successfully handle raw coding and engineering, they fundamentally lacked ‘scientific taste’—the intuitive judgment required to identify novel, worthwhile research directions

   ↩

6. AP News on autonomous vulnerability research — https://apnews.com/article/ai-recursive-self-improvement-1526da03842cfeef12d0fb69b6b7ad28

   the technical skills required to perform automated patching are functionally identical to those needed for offensive exploitation

   ↩

7. VentureBeat — https://venturebeat.com/orchestration/googles-dream-rsi-cuts-discovery-agent-calls-up-to-162x-by-replaying-searches-it-already-ran

   Google’s Dream-RSI cuts discovery agent calls up to 162x by replaying searches it already ran

   ↩

8. shattered.io analysis — https://shattered.io/dream-rsi-recursive-self-improvement-2026/

   In a strictly controlled comparison against a fixed exploration baseline, where the model and search budget remain identical, Dream-RSI demonstrated a consistent 1.7x gain — an engineering-solid paper wrapped in an oversold narrative.

   ↩

9. alphaXiv review of 2609.14858 — https://www.alphaxiv.org/abs/2609.14858

   The paper lacks matched-call final comparisons using repeated seeds and a control that randomly selects among revised policies, so it is unclear whether gains come from the replay-based ranking or simply from generating multiple policy variations.

   ↩

10. r/reinforcementlearning discussion — https://www.reddit.com/r/reinforcementlearning/comments/1wkk270/metaexploration_in_longhorizon_agent_discovery_is/

    The system is more accurately described as a refined form of experience replay or system prompt optimization rather than ‘true’ RSI, which traditionally implies a model rewriting its own weights; the dreaming simulator is bounded by the state-space already explored and cannot simulate entirely novel branches.

    ↩

11. shattered.io (semantic-bias finding) — https://shattered.io/dream-rsi-recursive-self-improvement-2026/

    Providing the agent with semantic ‘lessons learned’ summaries actually harmed performance compared to replaying raw, messy history, as summaries tend to over-compress and bias the exploration path.

    ↩

12. explainx.ai — https://explainx.ai/blog/google-deepmind-dream-rsi-recursive-self-improvement-2026

    As of late September 2026, the full implementation is ‘being prepared for release’ according to a staged release plan, with functional scripts for end-to-end training initially absent or limited to specific demos.

    ↩

13. Medium (Syed Hasan) — ‘DAPO: GRPO on Steroids’ — https://medium.com/@syed_hasan/dapo-decoupled-clip-and-dynamic-sampling-policy-optimization-grpo-on-steroids-9c571a0536f3

    In head-to-head benchmarks on AIME 2024, a 32B model trained with DAPO reached 50 points, outperforming DeepSeek-R1’s 47 points while requiring 50% fewer training steps.

    ↩

14. Pebblous AI blog — ‘GRPO, Dr.GRPO, DAPO and the Standard-Deviation Identity’ — https://blog.pebblous.ai/blog/grpo-drgrpo-dapo-standard-deviation/en/

    GRPO, Dr. GRPO, and DAPO are merely ‘three operations on one number’: the group standard deviation… under binary rewards the update magnitude is exactly equal to the group’s standard deviation.

    ↩

15. ByteDance Seed — POLARIS report on Qwen 3 4B — https://seed.bytedance.com/en/blog/seed-research-4b-%E6%A8%A1%E5%9E%8B%E6%95%B0%E5%AD%A6%E6%8E%A8%E7%90%86%E6%8E%A5%E8%BF%91-235b-%E8%A1%A8%E7%8E%B0-seed-%E6%9C%80%E6%96%B0%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0%E9%85%8D%E6%96%B9%E5%85%A8%E9%9D%A2%E5%BC%80%E6%BA%90

    A ‘DeepScaleR-style’ recipe pushed Qwen 3 4B to 79.4% accuracy on the AIME 2025 benchmark… with approximately 700 steps of RL training, this 4B-parameter model achieved reasoning performance nearly identical to the much larger Qwen 3 235B.

    ↩

16. vLLM blog — OpenRLHF + vLLM async RL integration — https://vllm.ai/blog/2025-04-23-openrlhf-vllm

    OpenRLHF utilizes vLLM’s pause and resume APIs, which allow model updates to be broadcast to inference workers mid-flight while preserving the state of existing requests.

    ↩

17. Michael Noukhovitch — project page for NGU — https://mnoukhov.github.io/posts/ngu/

    If a dataset consists almost entirely of unsolvable problems, NGU may become inefficient or lead to off-policy samples that provide a weaker learning signal.

    ↩

18. Interconnects (Nathan Lambert) — ‘Papers I’m reading: base model RL / GRPO’ — https://www.interconnects.ai/p/papers-im-reading-base-model-rl-grpo

    Full artifacts, including training logs, checkpoints, and evaluation code, rather than just open weights… allow independent researchers to audit the ‘complexity tax’ and the millions of dollars required to refine these recipes.

    ↩