WeatherNext 3: 3h lag, 110/124 Inspect Evals fail, evolution search beats GRPO

Sources

Introducing WeatherNext 3, our most advanced and accurate global weather AI model deepmind.google

What Does an Evaluation License? A Commit-Bound Census of Claim-Relative Inference in Inspect Evals huggingface.co

Evaluation artifacts specify a forward computation: a task, scorer, and reported metric. They do not necessarily license the claim attached to that metric because the historical evidence and alternative semantics needed to replay it may be unbound. We formalize this missing claim-replay layer through a frozen substrate D, a grounded family F, a claim query q, and the resulting identified set. We then census all 124 mechanically eligible Inspect Evals units at a pinned commit. Every unit receives

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO huggingface.co

Evolution strategies improve reasoning diversity and Pass@K over GRPO through sparse functional updates and population diversity, supporting a hybrid training approach.

What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents huggingface.co

Agentic data generation gets recast as constrained distribution design over factorized experience tuples rather than a scaling problem. The ACE framework prioritizes three axes when synthesizing trajectories for LLM agents: execution-grounded accuracy, learner-relative complexity, and diversity — arguing sheer volume matters less than shaping the mix a learner actually needs.

CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes huggingface.co

Weak-to-strong generalization gets an inference-time twist: CritICL mines structured failure patterns from smaller models and feeds them as critiques to guide stronger LLM reasoning. Static and dynamic variants cut both generation calls and token spend versus conventional inference-time scaling recipes.

PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents huggingface.co

Long-horizon agents rarely recover from bad early steps, so PILOT wraps them in a supervisor-worker harness that intervenes on active workers and distills successful execution traces into reusable skills. The loop lifts both accuracy and token efficiency on multi-step tasks through live self-evolution.

PAWBench: How Far Are We from Probabilistically Aligned World Modeling? huggingface.co

Video generators evaluated as stochastic samplers fall short of matching reference behavior distributions, according to PAWBench and its companion PAWEval metric. The work formalizes probabilistic alignment as a distributional criterion for world models, exposing that current systems capture plausible frames but not the right outcome spread.

WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution huggingface.co

Rather than treating agent skills as isolated snippets, WikiSkill co-evolves them with a persistent knowledge base so experience accumulates across runs. Ablations show the paired representation transfers gains across different backbone models, not just the one that generated the trajectories.

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher huggingface.co

On-policy distillation for flow matching usually needs a task-specific teacher; Self-OPD drops it by sampling SDE branches from the student itself and scoring them with normalized advantages. A pull-push objective and direction-aware attenuation then align the velocity field for multi-objective goals.

TTPO: Test-Time Policy Optimization huggingface.co

Test-Time Policy Optimization trains a model on its own rollouts using majority-vote pseudo-labels, asymmetrically distilling agreeing trajectories while penalizing dissenters at the token level. On math reasoning benchmarks the label-free procedure reaches parity with fully supervised fine-tuning.

CaSKG: Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrieval huggingface.co

CaSKG calibrates procedural skill relations via counterfactual-causal graph construction to improve compact, executable retrieval for LLM agents.

Magpie: Real-Time World Renderer for Interactive Games huggingface.co

Magpie is a real-time generative rendering system that separates gameplay logic from visual generation to preserve interactive designability while reducing asset requirements for game prototypes.

Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models huggingface.co

Game engines provide executable verification and long-horizon trajectories for reinforcement learning post-training of spatial world models, motivating a human-engine verification paradigm.

TacForcing: Streaming Action Generation with Execution-Time Tactile Feedback huggingface.co

TacForcing is a streaming action-generation framework that integrates real-time tactile feedback during execution via a streaming action expert and execution-aware tactile attention, improving contact-rich manipulation.

Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization huggingface.co

Zero-WAM enables robotic manipulation of unseen tasks by conditioning a causal video-action model on in-context human video guidance, supported by an automatically generated dataset and a future-chunk prediction objective.

Aphanta: Diagnosing Task-Aligned Image-Edited Intermediates for Multimodal Reasoning huggingface.co

Aphanta evaluates when image-editing intermediates improve multimodal reasoning by testing direct, editor-generated, and idealized visual states across tasks.

UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City huggingface.co

UrbanGround evaluates whether multimodal language model agents can sustain reliable navigation and spatial reasoning in a realistic 3D city replica, revealing that local perceptual skills fail to compose into extended goal-directed behavior.

GameWAM: A World Action Model for Video Games huggingface.co

GameWAM is a unified world-action model for native video-game control that jointly predicts future visuals and executable keyboard-mouse actions using block-causal flow matching, mode-specific distributions, and block-cycle replanning.

Procedura: Agentic 3D Modeling with Procedural Control huggingface.co

Procedura is a 3D modeling agent that generates editable, part-structured procedural assemblies with sharp geometry and validated articulation from text prompts.

Luce: Relightable Gaussians for 3D Asset Generation huggingface.co

Luce unifies geometry and PBR materials in a voxelized Gaussian cloud, using a variational autoencoder and rectified-flow transformer to generate relightable 3D assets from single images.

Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning huggingface.co

An agentic framework combining LLMs and VLMs enables consistent, multi-instruction editing of long multi-shot videos while preserving spatiotemporal structure.

EditaLive! Unified Character Video Editing for Live Streaming huggingface.co

EditaLive enables real-time human-centric live-stream video editing by adapting an image animation model to causal streaming generation with distilled two-step sampling and sparse attention.

CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension huggingface.co

CaRGo-T improves multimodal humor understanding by modeling causal relationships as graph-based reasoning structures interpreted by vision-language models.

Training Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Report huggingface.co

Harness-Aware Training enables compact models to adapt to evolving digital-avatar harness configurations with low latency and high accuracy.

References

MarkTechPost marktechpost.com

trains on weather station observations to deliver 5 km global forecasts refreshed every hour

MarkTechPost (WeatherNext 2 FGN writeup) marktechpost.com

WeatherNext 2 uses Functional Generative Networks for 8x faster probabilistic weather forecasts

PNAS study on AI weather models & extremes pnas.org

models optimized to minimize mean squared error tend to favor ‘safe’ statistical averages, leading to a consistent underestimation of both the frequency and intensity of record-breaking events

AI Certs on WindBorne aicerts.ai

WindBorne argues that satellite data is insufficient for capturing vertical atmospheric profiles… its ‘bottom-up’ balloon data is the only way to close the 85% atmospheric data gap over oceans

Quartz qz.com

Ilan Price… said the model’s precision stems from ‘not waiting for the next analysis date and using the most recent information,’ reducing forecast data lag from seven hours to three

PDP Spectra AI weather comparison pdpspectra.com

ECMWF has ceased running ‘external’ models like Pangu-Weather in its real-time charts to focus exclusively on its internal AIFS development

The Guardian — coverage of Bean et al. 445-benchmark validity study theguardian.com

Almost all had weaknesses in at least one area … scores could be ‘irrelevant or even misleading’. Only 16% of the benchmarks used uncertainty estimates or statistical tests to show how likely a benchmark was to be accurate.

Oxford Internet Institute press release on the Bean et al. study oii.ox.ac.uk

Many of the tests used to evaluate these systems may not measure what they claim to, undermining the validity of resulting claims … the authors set out eight recommendations for building benchmarks with stronger construct validity.

AlphaXiv summary of the AgentDojo paper (Debenedetti et al.) alphaxiv.org

AgentDojo evaluates ten LLMs and finds that even state-of-the-art models like GPT-4 achieve less than 50% success on benign tasks, while adversarial attacks reduce this further, with targeted attack success rates around 25%.

PromptGuard blog — independent AgentDojo replication promptguard.co

Two of our defences did nothing … results on AgentDojo do not necessarily transfer: guardrails that stop a specific injection template can be trivially bypassed by an adaptive attacker, so leaderboard ASR is at best an upper bound.

IntuitionLabs analysis of MMLU-Pro evaluator variance intuitionlabs.ai

GPT-4o’s reported MMLU-Pro score varied by roughly 13 percentage points across evaluation harnesses — an order of magnitude larger than the ~1% gap separating top-ranked models on the leaderboard.

r/LocalLLaMA discussion: why GPT-4o-mini beats Claude 3.5 Sonnet on LMSys reddit.com

GPT-4o-mini uses better formatting, more bolding and headers, and is more ‘people-pleasing’ — the ranking reversal is a judge-style artifact, not a capability difference.

Hyper-ES paper (ResearchGate) researchgate.net

Hyper-ES uses a small number of gradient-based runs to find descent directions, then applies CMA-ES to optimize the merging coefficients… outperforming GRPO-LoRA by 1% while requiring 10% fewer gradient updates.

ESSA paper (Hugging Face) huggingface.co

ESSA reaches near-optimal accuracy up to six times faster than GRPO on large-scale hardware… optimizing only singular values of LoRA adapters after SVD compression.

Robust Policy Optimization to Prevent Catastrophic Forgetting (ResearchGate) researchgate.net

Evolutionary Strategies induce substantially larger L2-norm parameter drift than GRPO, and without Anchored Weight Decay this random walk in weight space produces measurable forgetting on held-out benchmarks.

GoPenAI blog — GDPO: The Hidden Flaw in Multi-Reward RLHF blog.gopenai.com

In multi-reward GRPO, distinct objectives get collapsed to identical advantages — a ‘reward collapse’ that GDPO fixes via decoupled normalization, addressing a failure mode entropy-focused ES fixes don’t touch.

Pass-at-k Policy Optimization (alphaxiv 2601.20861) alphaxiv.org

Directly optimizing Pass@k can degrade Pass@1 due to ‘prompt interference’; annealing k during training is proposed to get exploration early and reliability late — a gradient-based alternative to the ES→GRPO sequential recipe.

YouTube practitioner walkthrough on ES vs GRPO youtube.com

ES is simpler to implement because it treats the LLM as a black box requiring only forward passes, but developers report high run-to-run variance in early-stage training compared to the relative stability of GRPO.