MILO, AED, and LANTERN all ride open-weight Qwen3 or gpt-oss bases

TL;DR

MILO auto-evolves agent harnesses on gpt-oss-120b to 86.1% Terminal-Bench, inside the noise band.
AED fine-tunes Qwen3-8B on 50,228 failure traces, lifting diagnosis from 47.2% to 63.6%.
LANTERN probes Qwen3-32B activations to surface 4 novel OEIS relations in 8 hours.
SkillGym auto-builds verifiable environments for long-horizon skill-use agent training.
Self-evolving search agents increasingly collude on wrong answers each round.

Today’s three research features each build a new method on an open-weight base rather than a fresh pretrain. MILO auto-evolves agent harnesses around gpt-oss-120b to hit 86.1% on Terminal-Bench 2.1 — inside the leaderboard’s known noise band. AED fine-tunes Qwen3-8B on 50,228 failure traces curated from 23 other policy models, lifting diagnostic agreement from 47.2% to 63.6%. LANTERN probes a frozen Qwen3-32B’s residual stream to rank 50M OEIS sequence pairs, surfacing four novel number-theoretic relations in under 8 hours.

The round-up pool extends the agent-method beat. SkillGym auto-builds verifiable training environments, PivotOPD distills recovery from pivotal mistakes, and a self-evolving search paper documents how proposer-solver pairs quietly collude on wrong answers over time — a cautionary note for anyone leaning on the same evolutionary machinery MILO rides.

MILO auto-evolves agent harnesses past hand-tuned SOTA

TL;DR

MILO auto-evolves agent harnesses via islands plus a meta-orchestrator that reassigns mutators on stall.
Hits 86.1% on Terminal-Bench 2.1 vs. 83.8% leaderboard top — inside the benchmark’s known noise band.
Delivers a 20× jump on DeepSWE with gpt-oss-120b, where prior search methods stayed below 1%.
The meta-orchestrator is the genuine novelty atop inherited AlphaEvolve islands and GEPA-style reflective mutators.

The pitch: evolve the search, not just the harness

Agent harnesses — the code that wraps an LLM with tool calls, control flow, and memory — are still mostly hand-written. MILO (Meta-evolutionary Island Orchestration) treats harness design as a search problem and automates it with two coupled loops: an inner loop that mutates individual harnesses, and an outer loop in which an orchestrator agent reshapes the search itself when islands plateau.

The island architecture — a population partitioned into semi-isolated lineages with occasional cross-migration — is lifted almost directly from DeepMind’s AlphaEvolve, which uses roughly five islands of ~25 programs with ring-topology migration to resist diversity collapse ¹. The mutator design, which feeds execution traces back as natural-language “evidence” rather than scalar reward, echoes GEPA’s reflective approach — the same trick that let GEPA beat RL baselines with ~35× fewer rollouts ². MILO’s genuine novelty is the meta layer: an orchestrator that can reassign a stalled island’s mutator, graft features from a winning island into a stuck one, speciate a promising subtree, or adjust the curriculum toward frontier tasks that most harnesses fail.

flowchart LR
    O{{Meta-Orchestrator}} -. reassign/graft/speciate .-> I1
    O -. reassign/graft/speciate .-> I2
    O -. reassign/graft/speciate .-> I3
    subgraph Islands[Lineage islands]
      I1[Island 1<br/>Mutator A] --> T1[(Append-only<br/>lineage tree)]
      I2[Island 2<br/>Mutator B] --> T2[(Append-only<br/>lineage tree)]
      I3[Island 3<br/>Mutator C] --> T3[(Append-only<br/>lineage tree)]
    end
    T1 & T2 & T3 --> P[Pareto filter:<br/>accuracy / tokens / latency]
    P --> O

A new harness is admitted only if it advances the Pareto frontier across accuracy, token cost, and latency — which is why the token-efficiency numbers travel alongside the accuracy ones.

Reading the benchmark numbers

The headline is 86.1 ± 2.0% on Terminal-Bench 2.1 vs. 83.8 ± 2.3% for the top official entry. Treat that gap with care. Terminal-Bench 2.1 was explicitly rebuilt this cycle, with 28 tasks modified to patch dependency drift, timeout mismatches, and reward-hacking shortcuts ³ — so the number is harder to game than a v2.0 score. But Vals AI points out the benchmark only has 89 tasks, which means gaps under ~10.6 points are statistical noise, and some top entries lean on fallback mechanisms that would cost ~8 points if scored as failures ⁴. MILO’s +2.3-point edge sits well inside that band.

The more defensible claims are the efficiency and generalization results. A 26% token reduction at higher accuracy is a real Pareto move. A 20× resolution-rate jump on DeepSWE with the open-weight gpt-oss-120b — against prior search methods stuck below 1% — is the kind of delta that doesn’t disappear in noise. And the fact that MILO improves harnesses starting from both mini-SWE-agent (a ~100-line bash-first baseline) and DeepAgents (a LangGraph-heavy stack with sub-agent delegation and filesystem memory ⁵) is a stronger generalization signal than any single benchmark.

What’s actually new, and what’s open

The EinsteinArena math results — a tightened Erdős minimum-overlap bound of 0.3808568 and two autocorrelation inequality improvements — are legitimate records but incremental in a field where TTT-Discover, SimpleTES, and AlphaEvolve have been chipping at the same constants all year ⁶. The novelty is the search method, not the math.

The honest read: MILO is competitive with hand-tuned SOTA at meaningfully lower token cost, and the orchestrator layer is a credible contribution. Independent reproduction of the DeepSWE 20× claim — the result least explainable by benchmark variance — is the thing to watch.

AED’s 50K failure traces lift 8B agents past frontier diagnosis

TL;DR

Agent Error Dataset (AED) packages 50,228 error–diagnosis pairs from 9,961 tasks across 33 environments and 23 policy models.
Fine-tuning Qwen3-8B on AED lifts exact-step diagnostic agreement from 47.2% to 63.6%, beating the strongest prompted frontier baseline at 54.7%.
Applying first-proposal corrections raises verifier pass rate from 18.4% to 51.1% on 3,062 matched replay pairs.
Independent reviewers flag single-seed headline comparisons, train/eval task overlap, and a CC BY-NC-SA license that walls off commercial post-training use.

Learning from what agents got wrong

Most agent post-training throws away the interesting data. A rollout that ends in failure contains the observation stream, the action chosen, and the environment’s response — information that reward-only pipelines collapse into a single zero. The Agent Error Dataset (AED) is a bet that this is the wrong trade. The authors assemble 50,228 error–diagnosis pairs from nearly 10,000 source tasks spanning 33 environments and 19 harness families, then show that supervising on failures beats supervising only on successes.

The pipeline, dubbed Agentic Error-to-Training (AET), is a five-stage conveyor from raw failure to structured training view:

flowchart LR
    A[Failure traces<br/>33 envs, 23 policies] --> B[Teacher<br/>diagnosis]
    B --> C[Correction<br/>proposal]
    C --> D[Replay<br/>verification]
    D --> E[Training views:<br/>diagnosis / recovery / DPO]

The replay step matters most: where the environment supports it, proposed fixes are executed and compared against retrying the original action from the same state. That turns “the teacher thinks this is a better action” into “the environment agrees.”

The numbers that carry the paper

Three results do the work. First, those replay-verified corrections took 3,062 matched pairs from an 18.4% baseline pass rate to 51.1% — a 32.7-point swing that validates the diagnoses as more than plausible-sounding text. Second, fine-tuning Qwen3-8B on AED pushed exact-step diagnostic agreement from 47.2% to 63.6%, which is 9 points above the best prompted frontier model on the same holdout. Third, on WebShop-lite, an acting policy trained with “action-only repair” supervision beat a success-only baseline by 6.67 points — the paper’s clearest evidence that repair data is a better teacher than curated successes.

Scaling behaviour is consistent across four training-set sizes, which is the kind of monotone curve that usually survives replication.

The field is already crowded

AED lands into an active cluster, not a vacuum. Agentic-DPO (Chen & Yuille) builds contrastive pairs from failed-vs-expert trajectories and adds “Policy-Preserving Augmentation” to stop models memorising tool schemas — a failure mode AED’s preference view does not explicitly guard against ⁷. Recovery-Bench attacks the problem from the eval side, parachuting agents into deliberately corrupted states and finding that leaders on clean benchmarks collapse under “context pollution” ⁸ — exactly the transfer test AED has not yet run.

The sharpest counter-thesis comes from HarnessFix, which argues that most “agent errors” are actually runtime and tool-interface defects, and that debugging the harness is more cost-effective than fine-tuning the policy ⁹. AED’s own Appendix E concedes that some entries stem from “infrastructure or task-contract defects,” which is uncomfortably close to HarnessFix’s point.

Caveats worth pricing in

An audit cited by The Moonlight found shared task identities between AED’s training set and external benchmarks including TrajErrBench, with some seeds regressing when trajectory granularity differed from AED’s multi-agent transcripts ¹⁰. OpenTrain’s reviewer tagged the release “Review before use,” noting the 6.67-point WebShop-lite win rests on single-seed comparisons ¹¹. And the CC BY-NC-SA 4.0 license propagates ShareAlike to any derivative dataset or model — effectively excluding the frontier labs who would otherwise be the obvious customers ¹².

The scale is real and the pipeline is a credible systematisation of “learn from failure.” Whether the headline deltas survive multi-seed replication, overlap-controlled splits, and a fair comparison against harness-level fixes is the next question.

LANTERN mines Qwen3 activations for 4 novel OEIS relations

TL;DR

LANTERN probes Qwen3-32B’s residual stream to rank 50M OEIS sequence pairs, yielding 62 verified new links in under 8 hours
Only 13 of 62 survived the authors’ own non-triviality screen
4 relations earned the “novel” tag, meaning a targeted literature search didn’t surface them
5-core partitions get a new generating function T(q)²·R(q)⁵, bridging a theta quotient and the Rogers-Ramanujan continued fraction
Critics argue OEIS-style benchmarks measure memorization, not induction, because sequence code saturates web training data

The pipeline

LANTERN treats mathematical discovery as a link-prediction problem over the On-Line Encyclopedia of Integer Sequences. The authors embed each sequence’s text definition with Qwen3-32B, pull the residual stream from all 65 layers, and compress it to ~321 hand-engineered features — centered cosine similarities plus PCA-reduced element-wise products at layers 4 and 40. A logistic regression trained on 2,780 “gold/silver” OEIS cross-references then ranks all 50 million pairs among the 10,000 most-referenced sequences.

The top candidates flow through a tiered agent stack: Claude Sonnet 5 culls implausible matches, Claude Opus 5 proposes a concrete identity using Python and symbolic math libraries, and a sandbox checks that the proposed function reproduces 25–100+ stored terms. A final analytical audit — part human, part frontier model — tries to catch finite-term coincidences.

flowchart LR
    A[10k OEIS entries<br/>50M pairs] --> B[Qwen3-32B residuals<br/>65 layers → 321 features]
    B --> C[Logistic classifier]
    C --> D[Sonnet 5 filter]
    D --> E[Opus 5 hypothesis +<br/>Python sandbox]
    E --> F[Analytical audit]
    F --> G[62 verified →<br/>13 non-trivial → 4 novel]

What actually got discovered

The headline is 62 verified relations between previously unlinked OEIS entries. The honest number is smaller. The authors’ own “content screen” — meant to strip definitional restatements — cuts the set to 13. Of those, 9 were rated informative and 4 earned the “N2” label meaning they weren’t surfaced by a targeted literature search ¹³.

The N2 results are bridges between known objects, not new theorems. The strongest is the identity T(q)²·R(q)⁵ for 5-core partitions, combining a theta quotient with the Rogers-Ramanujan continued fraction ¹⁴. Another links the 2D “Fredkin replicator” cellular automaton to Rule 150 via a difference-of-squares identity. Interesting; not AlphaProof.

The memorization problem

The sharpest external pushback targets whether any of this is induction at all. Reviewers cite “Where Induction Runs Out,” which argues OEIS performance “often tracks the model’s recognition of a sequence from its pre-training corpus” rather than genuine algorithmic reasoning ¹⁵. The paper’s temporal snapshot design (April 2025 → August 2026) partially addresses this, but Mathematica and Maple code for OEIS entries is dense on the public web — true isolation is nearly impossible.

Methodologically, LANTERN sits awkwardly between camps. Symbolic systems like Diofantos reconstruct >90% of known OEIS formulas by posing discovery as a Diophantine problem, versus ~70% for neural approximators like SINDy — and they yield human-readable identities without a frontier-model audit ¹⁶. On the proof side, Moonlight contrasts LANTERN’s executable term-matching with AlphaProof’s formal derivation discipline ¹³.

The more durable contribution may be mechanistic: LANTERN is one more data point for the growing finding that LLMs encode task-relevant properties as linearly decodable directions distinct from what the model verbalizes ¹⁷. The discovery narrative is oversold. The probe works. Reproduction on a non-Qwen model, and a public code release, would settle which of those claims matters.

Round-ups

SkillGym auto-builds verifiable environments to train skill-use agents

SkillGym crawls skills from the web, then generates verifiable environments, trajectories, and training data for LLM agents without human curation. The pipeline targets long-horizon professional tasks where reliable skill-use training data has been the main bottleneck.

PivotOPD teaches multi-turn agents to recover from pivotal mistakes

Across Qwen3 models from 8B to 235B, over half of failed agent rollouts hinge on a single pivotal action that derails later turns. PivotOPD extends on-policy distillation to supervise recovery from these errors, countering the error compounding that plagues multi-turn training.

Self-evolving search agents collude on wrong answers over time

In self-evolving agents where a proposer writes questions and a solver answers them, the pair increasingly agrees on shared mistakes, inflating internal reward without matching external correctness. Audits against source evidence show this co-cheating worsens each round, degrading pseudo-label accuracy.

OSWorld-Science benchmarks VLM agents on real scientific software

OSWorld-Science tests computer-use agents on research workflows inside specialized scientific tools, scoring them on verifiable artifacts rather than UI traces. The harness targets tasks that demand interpreting domain interfaces and manipulating scientific objects, a gap left by generic desktop agent benchmarks.

CUA-SWE tests agents on visual debugging, not just code edits

Software engineering benchmarks mostly ignore the run-inspect-fix loop, so CUA-SWE forces agents to execute programs, read GUI output, and trace runtime failures back to source. It unifies coding and computer-use evaluation, which prior work has studied separately.

Latent agent-to-agent links raise harmful compliance vs text

Trainable links that let multi-agent systems share internal representations cut token and latency costs, but even benign link training increases harmful compliance compared to text channels. The underlying agents stay safety-aligned, suggesting the vulnerability lives in the learned mapping itself.

Synthetic pre-pretraining scales, but grammar prior theory falls apart

Pre-pretraining on synthetic non-natural data keeps improving token efficiency past the 1B-parameter and 2B-token ceiling of prior work, but the gains do not come from a learned grammatical inductive bias. The authors verify the method at realistic scale and reject the dominant mechanistic explanation.

OSTI preprint on evolutionary LLM search (AlphaEvolve lineage) — https://www.osti.gov/servlets/purl/3818138

AlphaEvolve partitions its population into semi-isolated ‘islands’ with periodic ring-topology migration… a typical configuration uses five islands of ~25 programs each, balancing exploration via Levenshtein-distance diversity against exploitation of top-tier candidates.

↩
Comet blog on GEPA optimization — https://www.comet.com/site/blog/gepa-ai-optimization/

GEPA uses ‘Actionable Side Information’—natural-language feedback from execution traces—to diagnose failures and propose targeted mutations, achieving 35x fewer rollouts than traditional RL methods while frequently outperforming them.

↩
Terminal-Bench 2.1 release notes (tbench.ai) — https://www.tbench.ai/news/terminal-bench-2-1

Terminal-Bench 2.1 is a ‘verified’ update that modified or fixed 28 tasks suffering from external dependency shifts, insufficient timeouts for valid solutions, and vulnerability to reward hacking where agents would bypass tasks rather than solve them.

↩
Vals AI Terminal-Bench 2.1 analysis — https://www.vals.ai/benchmarks/terminal-bench-2-1

Some top-tier results rely on ‘fallback’ mechanisms… if these fallbacks are counted as failures, scores for some top models drop by nearly 8 percentage points; a gap of less than 10.6 points between models should be viewed as statistical noise due to the sample size of 89 tasks.

↩
LangChain DeepAgents GitHub — https://github.com/langchain-ai/deepagents

DeepAgents provides a dense layer of planning, sub-agent delegation, and persistent filesystem memory to handle long-running tasks that ‘shallow’ agents typically fail to complete.

↩
EinsteinArena-new-SOTA GitHub leaderboard — https://github.com/togethercomputer/EinsteinArena-new-SOTA

By late 2026, Einstein Arena entries reached Erdős minimum-overlap values as low as 0.3808587, with the third autocorrelation inequality improved to ~1.454555 — a more substantial gain over the prior 1.455643.

↩
Agentic-DPO (Chen & Yuille, Semantic Scholar) — https://www.semanticscholar.org/paper/Agentic-DPO%3A-From-Imitation-to-Agentic-Policy-on-Chen-Yuille/042908df8d744ec637c3043916136cca8bf85eeb

iterative loop where agents collect trajectories from failed rollouts and pair them with successful expert demonstrations to create contrastive training data… uses ‘Policy-Preserving Augmentation’ (PPA) to ensure the model learns robust policy logic rather than just memorizing tool-specific schemas

↩
OpenReview: Recovery-Bench — https://openreview.net/pdf?id=PFR4E8583W

initializes agents directly into corrupted states derived from failed trajectories to measure their ability to self-correct mid-task… reveals that some high-performing models on standard tasks struggle significantly with ‘context pollution’ during recovery

↩
HarnessFix (OpenReview) — https://openreview.net/challenge?redirect=%2Fforum%3Fid%3DPFR4E8583W

compiles raw traces into a ‘Harness-aware Trace Intermediate Representation’ to isolate whether a failure stemmed from the model’s reasoning or the ‘agent harness’ (the runtime environment and tool interfaces)… principled debugging of the harness is often more effective than model fine-tuning

↩
The Moonlight review (themoonlight.io) — https://www.themoonlight.io/es/review/agent-error-dataset-scaling-50000-error-diagnosis-pairs-for-failure-analysis-and-error-aware-post-training

the audit found shared task identities and exact goals between the AED training set and external evaluation cases… results on the TrajErrBench (Qi et al., 2026) benchmark showed mixed outcomes: while some seeds improved, others regressed when faced with single-agent trajectories that differed significantly in granularity from the AED’s multi-agent transcripts

↩
OpenTrain AI reviewer verdict — https://www.opentrain.ai/papers/agent-error-dataset-scaling-50-000-error-diagnosis-pairs-for-failure-analysis-an—arxiv-2609.40111/

flagged the dataset with ‘Review before use’ warnings, noting that independent benchmark evidence is still missing and full ecosystem coverage is only partial… certain actor-training results, such as the 6.67-point gain on WebShop-lite, are based on single-seed comparisons

↩
Hyperight coverage of AED release — https://hyperight.com/hugging-face-releases-a-50000-pair-dataset-for-ai-agent-error-analysis-and-post-training/

50,228 pairs from 9,961 source tasks… released under CC BY-NC-SA 4.0 — any adaptations or derivative works built using the dataset must be redistributed under the same license terms, effectively ‘infecting’ downstream research with the same restrictions

↩
Moonlight review of LANTERN — https://www.themoonlight.io/en/review/lantern-illuminating-hidden-mathematical-knowledge-in-language-models

the final ‘content screen’ that reduced 62 verified relations to 13 non-trivial ones suggests a high rate of technically correct but mathematically uninteresting output… [LANTERN] lacks the rigorous ‘proof-first’ approach of systems like AlphaProof, relying instead on empirical verification

↩ ↩² ↩³
arXiv:2609.32264 (abstract) — https://arxiv.org/abs/2609.32264

the Rogers-Ramanujan continued fraction R(q) and a specific theta quotient T(q) can be combined as T(q)^2 R(q)^5 to form the generating function for 5-core partitions

↩
Moonlight review (ES) — memorization critique — https://www.themoonlight.io/es/review/lantern-illuminating-hidden-mathematical-knowledge-in-language-models

critics from the ‘Where Induction Runs Out’ study argue that OEIS-derived benchmarks often measure memorization rather than algorithmic induction… high performance on OEIS tasks often tracks the model’s recognition of a sequence from its pre-training corpus

↩
MDPI survey on OEIS discovery methods — https://www.mdpi.com/2227-7390/12/23/3745

symbolic approaches like Diofantos… reformulating equation discovery as a Diophantine problem to achieve over 90% reconstruction rates on known OEIS formulas, markedly higher than the ~70% achieved by approximate methods like SINDy

↩
OpenReview — linear probing of latent solvability — https://openreview.net/pdf?id=rlADfdxA1I

an LLM’s internal belief about whether a problem is solvable is encoded as a distinct, linearly decodable direction, separate from the direction used for verbalization

↩

MILO, AED, and LANTERN all ride open-weight Qwen3 or gpt-oss bases

TL;DR

MILO auto-evolves agent harnesses past hand-tuned SOTA

TL;DR

The pitch: evolve the search, not just the harness

Reading the benchmark numbers

What’s actually new, and what’s open

AED’s 50K failure traces lift 8B agents past frontier diagnosis

TL;DR

Learning from what agents got wrong

The numbers that carry the paper

The field is already crowded

Caveats worth pricing in

LANTERN mines Qwen3 activations for 4 novel OEIS relations

TL;DR

The pipeline

What actually got discovered

The memorization problem

Round-ups

SkillGym auto-builds verifiable environments to train skill-use agents

PivotOPD teaches multi-turn agents to recover from pivotal mistakes

Self-evolving search agents collude on wrong answers over time

OSWorld-Science benchmarks VLM agents on real scientific software

CUA-SWE tests agents on visual debugging, not just code edits

Latent agent-to-agent links raise harmful compliance vs text

Synthetic pre-pretraining scales, but grammar prior theory falls apart

Footnotes

Jack Sun, writing.