WeatherNext 3: 3h lag, 110/124 Inspect Evals fail, evolution search beats GRPO

TL;DR

• WeatherNext 3 trains on raw satellite and station data, cutting forecast lag from ~7 hours to ~3.
• DeepMind’s model tops ECMWF HRES and NOAA GFS on ~97% of global features at 5km hourly.
• 110 of 124 Inspect Evals units fail a mechanical claim-replay audit at pinned commits.
• Evolution Strategies hits 73.2 Pass@1 and 96.6 Pass@32 on GSM8K, beating GRPO on both.
• ES→GRPO sequential training posts the best 6-task average at 42.3 Pass@1 and 80.0 Pass@32.

Today’s three research features don’t share a frame — they share a date. DeepMind’s WeatherNext 3 rebuilds AI weather forecasting on raw satellite and station observations instead of ERA5 reanalysis, jumping to 5 km hourly resolution and cutting the observation-to-forecast lag from about seven hours to three. An Inspect Evals census then finds that only 14 of 124 benchmark units survive a mechanical claim-replay at a pinned commit, and even those flip rankings under admissible evaluator rules. And a new Evolution Strategies result argues weight-space perturbation preserves Pass@K where GRPO collapses it — with an independent follow-up already contesting the paper’s forgetting story.

The briefs skew hard toward agent training: ACE, PILOT, WikiSkill, CritICL, and TTPO all reframe how experience, supervision, or test-time signal should shape a learner, while PAWBench and Self-OPD push on world-model evaluation and teacher-free distillation.

DeepMind’s WeatherNext 3 cuts forecast lag to 3 hours

Source: deepmind-blog · published 2026-09-03

TL;DR

• WeatherNext 3 trains directly on raw satellite and station data, dropping the ERA5-reanalysis dependency earlier AI models leaned on.
• Resolution jumps to 5 km, hourly globally, up from 25 km/6-hour in WeatherNext 2.
• Data lag from observation to forecast drops from ~7 hours to ~3, per DeepMind’s Ilan Price.
• On Brightband’s Operational WeatherBench it tops ECMWF HRES and NOAA GFS on ~97% of global features.
• MSE-trained AI models still under-intensify record events — the extreme-weather blind spot is unresolved.

What’s actually new

The Functional Generative Network backbone isn’t new — WeatherNext 2 shipped it last November, generating 64-member ensembles roughly 8× faster than the diffusion-based GenCast ¹. What changed in v3 is the training target. Previous AI weather models — GraphCast, Pangu, even WeatherNext 2 — leaned on ECMWF’s ERA5 reanalysis, which is itself a physics-model output. That made the whole enterprise slightly circular: AI learning to imitate a numerical weather prediction (NWP) system’s smoothed history.

WeatherNext 3 replaces that with hourly geostationary satellite mosaics and sparse weather-station observations as direct training signal ². The operational consequence, per DeepMind’s Ilan Price, is that “not waiting for the next analysis date” collapses forecast data lag from roughly seven hours to three ³. For grid operators dispatching wind and solar, or for nowcasting a squall line, three hours is the difference between a forecast and a postmortem.

The resolution jump is the other headline: 5 km grids for temperature and moisture, 10 km for secondary surface variables, hourly refresh across the globe. Google is wiring it into Search, Maps, the Gemini app, and exposing the gridded data through BigQuery and Earth Engine.

The independent scorecard

Brightband’s Operational WeatherBench has quietly become the leaderboard the AI-weather field uses to arbitrate claims, and WeatherNext 3 currently sits on top of it, beating ECMWF HRES and NOAA GFS on roughly 97% of tested features ². That’s real external validation for the 60% CRPS improvement DeepMind claims on precipitation against NASA IMERG, and the 30% gain against MRMS radar.

Two caveats worth naming. First, OWB is automated scoring against gridded targets — it’s not peer review, and it doesn’t do point-verification against surface stations, which is exactly where AI models have historically looked weaker than their reanalysis-trained scores suggest. Second, “beats HRES on 97% of features” is a bulk-distribution claim.

Where the model is still blind

The most persistent scientific critique of this whole class of models predates WeatherNext 3 and still applies. A PNAS analysis found that models trained to minimize mean squared error “favor ‘safe’ statistical averages, leading to a consistent underestimation of both the frequency and intensity of record-breaking events” ⁴. Precipitation-CRPS gains on IMERG don’t answer that; they reward calibration on the middle of the distribution, not the tails where cyclones rapidly intensify.

WindBorne argues that satellite data is insufficient for capturing vertical atmospheric profiles… its ‘bottom-up’ balloon data is the only way to close the 85% atmospheric data gap over oceans. ⁵

That’s the sharpest commercial counter-voice: satellites see tops of clouds, not the vertical column, and WindBorne’s Atlas balloon constellation is a bet that in-situ data is the missing input no amount of transformer scale will substitute for.

And at the operational-center level, the “one global AI model to rule them all” framing is contested. ECMWF has stopped running third-party AI models like Pangu-Weather on its real-time charts, consolidating around its in-house AIFS ⁶. The paradigm isn’t converging on Google — it’s diverging, with sovereign AI models, in-situ data moats, and closed frontier weights each staking a different claim on the next decade of forecasting.

---

Inspect Evals census: only 14 of 124 benchmarks replay cleanly

Source: hf-daily-papers · published 2026-08-24

TL;DR

• 110 of 124 Inspect Evals units failed a mechanical claim-replay audit at a pinned commit.
• The stop reason was uniform: historical evidence needed to reconstruct the reported claim wasn’t bound in the repo.
• Among the 14 that did replay, full rankings and raw scores flipped across admissible evaluator rules.
• On AgentDojo, Gemini 2.0 vs GPT-4o-mini reversed depending on the aggregation rule — while Claude 3.5’s top rank held.

Running the code isn’t replaying the claim

Qin and Tong draw a sharp distinction that most eval infrastructure blurs: an evaluation artifact specifies a forward computation (task, scorer, metric), but the claim attached to a leaderboard number — “Model A beats Model B on capability X” — requires more. It requires the historical traces, the alternative admissible scorings, and a deterministic way to check which conclusions survive across them. The paper formalizes this as an identified set $S(D, \mathcal{F}, q)$ over a frozen evidence base $D$, an evaluator family $\mathcal{F}$, and a claim query $q$, then runs the machinery against every mechanically eligible unit in the Inspect Evals collection at a single commit.

The census result is brutal. Of 124 units, 110 stopped — the evidence needed to replay the claim (specific traces, historical state, the exact scorer inputs) simply wasn’t there. Only 14 units, about 11%, permitted deterministic claim-replay analysis at all. This is fail-closed by design: the framework refuses to guess, and returns an explicit stopping reason per unit.

AgentDojo: the winner is stable, the ranking isn’t

The paper’s cleanest concrete case is AgentDojo’s Targeted Attack Success Rate (lower is better). Claude 3.5 ranks first across all five tested readings, with scores between 1.113 and 3.810 — a stable winner. But under the Published-micro reading Gemini 2.0 (20.827) beats GPT-4o-mini (27.186), while under a Utility-conditional reading GPT-4o-mini (19.745) beats Gemini (32.000). The pairwise claim flips on the aggregation rule.

That instability isn’t a hypothetical. PromptGuard’s independent AgentDojo replication found two defences that scored well on the leaderboard “did nothing” against adaptive attackers, and warned that leaderboard ASR is “at best an upper bound” ⁷. The original AgentDojo already puts top models under 50% on benign tasks with targeted ASR around 25% — a regime where evaluator choice dominates model differences ⁸.

The wider validity crisis ratifies the census

Qin and Tong’s mechanical result lands alongside a much larger survey. Bean et al.’s review of 445 AI benchmarks — Oxford, Stanford, Berkeley and UK AISI — found only 16% use uncertainty estimates or statistical tests, and that most benchmarks fail to demonstrate construct validity for what they claim to measure ⁹¹⁰. The Guardian’s summary is blunt: scores may be “irrelevant or even misleading” ⁹.

Harness variance quantifies the problem. GPT-4o’s reported MMLU-Pro score varies by ~13 percentage points across evaluation harnesses, an order of magnitude larger than the ~1% gap separating top-ranked models on the leaderboard ¹¹. Judge-style artifacts pile on: practitioners attribute GPT-4o-mini beating Claude 3.5 Sonnet on LMSys Arena to formatting and “people-pleasing” tone rather than capability ¹².

What’s at stake

The paper’s audit/release interface — reporting values, winners, directions, and stable pairwise relations separately, rather than a single score — is a concrete demand on benchmark maintainers: bind the evidence, or acknowledge the claim isn’t licensed. The 89% stop rate suggests most published benchmark claims currently aren’t. Whether AISI, Arcadia and the wider Inspect ecosystem adopt the discipline, or treat 11% as a diagnostic to argue with, is the next move. No public maintainer response has surfaced yet.

---

Evolution Strategies beat GRPO on Pass@K for reasoning

Source: hf-daily-papers · published 2026-08-26

TL;DR

• GRPO’s Pass@32 on GSM8K fell to 95.3 from a 96.4 base while lifting Pass@1 — entropy collapse in one line.
• Evolution Strategies improved both, hitting 73.2 Pass@1 and 96.6 Pass@32 by perturbing weights instead of sampling tokens.
• ES→GRPO sequential training posted the best average across 6 reasoning tasks: 42.3 Pass@1 and 80.0 Pass@32 on Qwen2.5-1.5B.
• Independent work sees held-out forgetting under ES without Anchored Weight Decay — contesting the paper’s functional-sparsity defense.

The pathology ES is aimed at

GRPO sharpens a policy by sampling multiple completions per prompt and reinforcing the winners. It works — until it doesn’t. On Qwen2.5-1.5B trained on GSM8K, GRPO pushed Pass@1 from 70.3 to 78.1 but lowered Pass@32 below the base model (95.3 vs 96.4). The model became more confident on its first try and worse at exploring 32 tries. On MATH-500, the same pattern held: Pass@1 up, Pass@16 and Pass@32 down. That’s entropy collapse in a sentence.

Ba et al. swap the mechanism entirely. Evolution Strategies (ES) perturbs the parameter vector with Gaussian noise, evaluates each perturbed model on the reward, and moves the center in the reward-weighted average direction — no backprop, no token-level advantages. Because the population lives in parameter space rather than sampling from one policy, the search naturally preserves policy heterogeneity. The authors formalize this with a “verifier-projected Jensen-Shannon diversity” argument: parameter noise induces policy diversity, which raises the probability that at least one member of the population solves any given prompt.

The numbers back the theory. ES improved Pass@1, Pass@16 and Pass@32 simultaneously on MATH-500 — something GRPO couldn’t do.

The portable takeaway: run them in sequence

The cleanest practical result is that ES and GRPO compose. ES→GRPO — broad reasoning coverage first, then sharpen — outperformed either alone on the 6-task Qwen2.5-1.5B average (42.3 Pass@1, 80.0 Pass@32). Population size also scales the friendly way: N=128 was optimal at 1.5B, but 7B models converged at N=32.

Where the claims get contested

Two of the paper’s headline arguments deserve scrutiny.

First, forgetting. Ba et al. attribute large ES parameter drift to “functional sparsity” — keep the top 1–10% of updates by magnitude and you recover the gains — and argue held-out degradation comes from training-set overfitting, not ES itself. Independent work on Robust Policy Optimization measures substantially larger L2 drift under ES than GRPO and finds real forgetting on held-out benchmarks absent an Anchored Weight Decay constraint ¹³. Practitioner reports also flag high run-to-run variance in early ES training that GRPO doesn’t suffer ¹⁴, compounded by the paper’s own admission that ES only works with z-score reward normalization and a well-tuned σ.

Second, necessity. The framing casts ES as the answer to entropy collapse, but gradient-based fixes are moving fast. Pass-at-k Policy Optimization anneals k during training — exploration early, Pass@1 reliability late — as a pure-gradient analog of the ES→GRPO recipe ¹⁵. GDPO fixes a separate “reward collapse” in multi-reward GRPO via decoupled normalization ¹⁶. And on the efficiency axis, subspace ES variants already dominate full-parameter ES: Hyper-ES optimizes CMA-ES merging coefficients over gradient-derived descent directions for +1% over GRPO-LoRA with 10% fewer gradient updates ¹⁷, and ESSA evolves only the singular values of SVD-compressed LoRA adapters for up to 6× wall-clock speedup vs GRPO on large-scale hardware ¹⁸.

The ES→GRPO recipe is the most portable takeaway; the “just swap in ES” story is more fragile than the Pass@K numbers suggest.

Code is on GitHub for anyone who wants to test the σ sensitivity themselves.

Round-ups

ACE reframes agent training data as constrained distribution design

Source: hf-daily-papers

Agentic data generation gets recast as constrained distribution design over factorized experience tuples rather than a scaling problem. The ACE framework prioritizes three axes when synthesizing trajectories for LLM agents: execution-grounded accuracy, learner-relative complexity, and diversity — arguing sheer volume matters less than shaping the mix a learner actually needs.

PILOT lets a supervisor steer long-horizon agents mid-run

Source: hf-daily-papers

Long-horizon agents rarely recover from bad early steps, so PILOT wraps them in a supervisor-worker harness that intervenes on active workers and distills successful execution traces into reusable skills. The loop lifts both accuracy and token efficiency on multi-step tasks through live self-evolution.

WikiSkill grows a persistent knowledge base alongside agent skills

Source: hf-daily-papers

Rather than treating agent skills as isolated snippets, WikiSkill co-evolves them with a persistent knowledge base so experience accumulates across runs. Ablations show the paired representation transfers gains across different backbone models, not just the one that generated the trajectories.

CritICL uses small-model failures to steer stronger LLMs at inference

Source: hf-daily-papers

Weak-to-strong generalization gets an inference-time twist: CritICL mines structured failure patterns from smaller models and feeds them as critiques to guide stronger LLM reasoning. Static and dynamic variants cut both generation calls and token spend versus conventional inference-time scaling recipes.

TTPO matches supervised math reasoning without labels at test time

Source: hf-daily-papers

Test-Time Policy Optimization trains a model on its own rollouts using majority-vote pseudo-labels, asymmetrically distilling agreeing trajectories while penalizing dissenters at the token level. On math reasoning benchmarks the label-free procedure reaches parity with fully supervised fine-tuning.

PAWBench shows video world models fail probabilistic alignment

Source: hf-daily-papers

Video generators evaluated as stochastic samplers fall short of matching reference behavior distributions, according to PAWBench and its companion PAWEval metric. The work formalizes probabilistic alignment as a distributional criterion for world models, exposing that current systems capture plausible frames but not the right outcome spread.

Self-OPD distills flow matching models without a teacher

Source: hf-daily-papers

On-policy distillation for flow matching usually needs a task-specific teacher; Self-OPD drops it by sampling SDE branches from the student itself and scoring them with normalized advantages. A pull-push objective and direction-aware attenuation then align the velocity field for multi-objective goals.

Footnotes

1. MarkTechPost (WeatherNext 2 FGN writeup) — https://www.marktechpost.com/2025/11/17/google-deepminds-weathernext-2-uses-functional-generative-networks-for-8x-faster-probabilistic-weather-forecasts/

   WeatherNext 2 uses Functional Generative Networks for 8x faster probabilistic weather forecasts

   ↩

2. MarkTechPost — https://www.marktechpost.com/2026/09/03/google-deepminds-weathernext-3-trains-on-weather-station-observations-to-deliver-5-km-global-forecasts-refreshed-every-hour/

   trains on weather station observations to deliver 5 km global forecasts refreshed every hour

   ↩ ↩²

3. Quartz — https://qz.com/google-deepmind-weathernext-3-ai-weather-forecast-090326

   Ilan Price… said the model’s precision stems from ‘not waiting for the next analysis date and using the most recent information,’ reducing forecast data lag from seven hours to three

   ↩

4. PNAS study on AI weather models & extremes — https://www.pnas.org/doi/10.1073/pnas.2420914122

   models optimized to minimize mean squared error tend to favor ‘safe’ statistical averages, leading to a consistent underestimation of both the frequency and intensity of record-breaking events

   ↩

5. AI Certs on WindBorne — https://www.aicerts.ai/news/windbornes-weather-forecasting-ai-targets-profitable-skies/

   WindBorne argues that satellite data is insufficient for capturing vertical atmospheric profiles… its ‘bottom-up’ balloon data is the only way to close the 85% atmospheric data gap over oceans

   ↩

6. PDP Spectra AI weather comparison — https://pdpspectra.com/blog/ai-weather-forecasting-graphcast-pangu/

   ECMWF has ceased running ‘external’ models like Pangu-Weather in its real-time charts to focus exclusively on its internal AIFS development

   ↩

7. PromptGuard blog — independent AgentDojo replication — https://promptguard.co/blog/we-ran-agentdojo-two-of-our-defences-did-nothing

   Two of our defences did nothing … results on AgentDojo do not necessarily transfer: guardrails that stop a specific injection template can be trivially bypassed by an adaptive attacker, so leaderboard ASR is at best an upper bound.

   ↩

8. AlphaXiv summary of the AgentDojo paper (Debenedetti et al.) — https://www.alphaxiv.org/abs/2406.13352

   AgentDojo evaluates ten LLMs and finds that even state-of-the-art models like GPT-4 achieve less than 50% success on benign tasks, while adversarial attacks reduce this further, with targeted attack success rates around 25%.

   ↩

9. The Guardian — coverage of Bean et al. 445-benchmark validity study — https://www.theguardian.com/technology/2025/nov/04/experts-find-flaws-hundreds-tests-check-ai-safety-effectiveness

   Almost all had weaknesses in at least one area … scores could be ‘irrelevant or even misleading’. Only 16% of the benchmarks used uncertainty estimates or statistical tests to show how likely a benchmark was to be accurate.

   ↩ ↩²

10. Oxford Internet Institute press release on the Bean et al. study — https://www.oii.ox.ac.uk/study-identifies-weaknesses-in-how-ai-systems-are-evaluated/

    Many of the tests used to evaluate these systems may not measure what they claim to, undermining the validity of resulting claims … the authors set out eight recommendations for building benchmarks with stronger construct validity.

    ↩

11. IntuitionLabs analysis of MMLU-Pro evaluator variance — https://intuitionlabs.ai/articles/mmlu-pro-ai-benchmark-explained

    GPT-4o’s reported MMLU-Pro score varied by roughly 13 percentage points across evaluation harnesses — an order of magnitude larger than the ~1% gap separating top-ranked models on the leaderboard.

    ↩

12. r/LocalLLaMA discussion: why GPT-4o-mini beats Claude 3.5 Sonnet on LMSys — https://www.reddit.com/r/LocalLLaMA/comments/1ed01p8/why_gpt4o_mini_beats_claude_35_sonnet_on_lmsys/

    GPT-4o-mini uses better formatting, more bolding and headers, and is more ‘people-pleasing’ — the ranking reversal is a judge-style artifact, not a capability difference.

    ↩

13. Robust Policy Optimization to Prevent Catastrophic Forgetting (ResearchGate) — https://www.researchgate.net/publication/400622099_Robust_Policy_Optimization_to_Prevent_Catastrophic_Forgetting

    Evolutionary Strategies induce substantially larger L2-norm parameter drift than GRPO, and without Anchored Weight Decay this random walk in weight space produces measurable forgetting on held-out benchmarks.

    ↩

14. YouTube practitioner walkthrough on ES vs GRPO — https://www.youtube.com/watch?v=QN2kbfcrX20

    ES is simpler to implement because it treats the LLM as a black box requiring only forward passes, but developers report high run-to-run variance in early-stage training compared to the relative stability of GRPO.

    ↩

15. Pass-at-k Policy Optimization (alphaxiv 2601.20861) — https://www.alphaxiv.org/abs/2601.20861

    Directly optimizing Pass@k can degrade Pass@1 due to ‘prompt interference’; annealing k during training is proposed to get exploration early and reliability late — a gradient-based alternative to the ES→GRPO sequential recipe.

    ↩

16. GoPenAI blog — GDPO: The Hidden Flaw in Multi-Reward RLHF — https://blog.gopenai.com/gdpo-the-hidden-flaw-in-multi-reward-rlhf-thats-been-sabotaging-your-llm-training-de3625b9f7df

    In multi-reward GRPO, distinct objectives get collapsed to identical advantages — a ‘reward collapse’ that GDPO fixes via decoupled normalization, addressing a failure mode entropy-focused ES fixes don’t touch.

    ↩

17. Hyper-ES paper (ResearchGate) — https://www.researchgate.net/publication/411824157_Hyper-ES_Effective_Evolution_Strategies_for_LLM_Reasoning_via_Descent_Direction_Merging

    Hyper-ES uses a small number of gradient-based runs to find descent directions, then applies CMA-ES to optimize the merging coefficients… outperforming GRPO-LoRA by 1% while requiring 10% fewer gradient updates.

    ↩

18. ESSA paper (Hugging Face) — https://huggingface.co/papers/2507.04453

    ESSA reaches near-optimal accuracy up to six times faster than GRPO on large-scale hardware… optimizing only singular values of LoRA adapters after SVD compression.

    ↩