Puro-2B trains for $6.9K, attention sinks hit 99%, LoopArena tops at 24.69%

TL;DR

Puro-2B hits 57.81% vs Qwen2-1.5B’s 55.14% on 15 tasks for $6.9K accelerator cost.
Puro-2B’s cheapness rides on patched RTX 5090 drivers re-enabling P2P/RDMA, plus a new MuonH optimizer.
Sliding-window + 4 attention sinks recovers 99% of teacher performance with zero post-training tokens.
GPT-5.5 tops LoopArena at 24.69% strict success versus an 18.52% goal-restating dumb baseline.
7 round-ups cluster on agent harnesses: StarHarness, EvoUndo, ContextPilot, StepGuard, RCCA, J-Zero.

Three research features today, three unrelated fronts. Puro-2B trains a competitive 2B model for $6.9K in accelerator rental on consumer RTX 5090s — cheap thanks to patched NVIDIA drivers re-enabling P2P/RDMA and a new MuonH optimizer, not FP8 alone. A sliding-window attention + 4 sinks recipe recovers 99% of teacher performance with zero post-training tokens, though Mamba co-author Albert Gu pushes back on the framing’s scope. And LoopArena — which grades the LLM steering a coding agent, not the code it writes — puts GPT-5.5 on top at just 24.69% strict success, barely six points above a baseline that only restates the goal each step.

Puro-2B beats Qwen2-1.5B for $6.9K on consumer RTX 5090s

TL;DR

Puro-2B trains a 2B-parameter model to 57.81% vs. Qwen2-1.5B’s 55.14% on a 15-task average, at ~$6.9K accelerator cost.
The “$5,090” headline is accelerator-rental only — it excludes failed runs, engineer time, and data-pipeline infra.
Cheapness comes from patched NVIDIA drivers re-enabling P2P/RDMA on RTX 5090s, plus a new MuonH optimizer, not just FP8.
Full recipe, data, and weights ship under Apache 2.0 on Hugging Face.

What the recipe actually gets you

Tsinghua’s PACMAN group pretrained a 2B model from scratch on 1.4T tokens using consumer RTX 5090s and FP8, and the top run lands at 57.81% average across 15 tasks — beating Qwen2-1.5B (55.14%) but trailing Qwen2.5-1.5B (60.73%) ¹. The paper’s “approaches Qwen2.5-1.5B” framing papers over a real 3-point gap; the honest read is “beats the 2023 model, not the 2024 one.” Still, doing that for under $7K in accelerator time — with a fitted scaling law suggesting $4.4K is enough to hit Qwen2-1.5B parity — is a genuine shift in what a small lab can attempt.

The load-bearing tricks aren’t in the abstract

Two details do most of the work, and one of them is a hardware hack. The team modified NVIDIA drivers to re-enable PCIe peer-to-peer and GPUDirect RDMA on RTX 5090s — capabilities NVIDIA software-locks on consumer cards to protect datacenter margins — and credits this with a 2.7× compute-per-dollar advantage over H200 rentals ². Without that patch, the economics collapse toward standard cloud pricing.

The second lever is MuonH, an extension of Keller Jordan’s Muon optimizer that fixes each weight matrix’s Frobenius norm at its initialization value, projecting updates onto a fixed hypersphere. The authors report a 20–30% token-equivalent speedup over standard weight decay at up to 1.2B parameters ³. That’s real, but with two asterisks: base Muon already carries 10–20% wall-clock overhead from its Newton-Schulz iterations, and it has required MuonClip-style patches to stay stable at trillion-parameter scale in Moonshot’s Kimi-K2 and Zhipu’s GLM-5 ⁴. MuonH’s behavior above 2B is unproven.

The third contribution, Curriculum Model Averaging, addresses a specific pathology: LR decay kills plasticity right when the model hits its highest-quality end-of-curriculum data. CMA holds a moderate LR and averages checkpoints to stabilize. Independent EACL 2026 work identifies the same problem and proposes the same fix ⁵, which is actually good news — parallel provenance strengthens the mechanism.

The “poor lab” framing needs an asterisk

Practitioner commentary is blunt: the $5,090 and $6,891 figures are accelerator-only, single-run costs — they don’t include failed experiments, hyperparameter sweeps, the 1.4T-token data pipeline, senior-engineer time to tune distributed configs, or 5090 depreciation under continuous thermal load ⁶. A well-resourced lab can reproduce Puro-2B cheaply. A literally poor lab probably cannot afford the runs that preceded the reported one.

What’s actually shipped

Apache 2.0, on Hugging Face, with data, code, and weights — a rare full-stack release at this scale. For anyone doing controlled pretraining research (data curricula, optimizer studies, curriculum ablations), that’s the real value, not the dollar figure. The headline gets you in the door; the reproducibility of the driver patch and the MuonH scaling curve are what determine whether the recipe survives contact with a second lab.

Sliding window + 4 sinks beats linear-attention retrofits

TL;DR

SWA with 4 attention sinks recovers 99.0% of teacher performance across six short-context benchmarks with zero post-training tokens.
On 4K Needle-in-a-Haystack, SWA reaches 17–23% of full-attention accuracy while LoLCATs collapses to 5.8% and Liger-GLA to 0.8%.
Mamba co-author Albert Gu calls the title misleading — the result scopes to retrofit conversion, not linear/SSM models trained from scratch.
Independent ablations show LoLCATs-style hybrids route around their linear path entirely, corroborating the baseline critique.

The baseline was the problem

Jolicoeur-Martineau et al. (Microsoft Applied Sciences) argue that the whole linear-attention retrofit literature — LoLCATs, Hedgehog, Liger-GLA, QRWKV — has been beating up a straw man. Their alternative is embarrassingly simple: take a pre-trained Transformer, restrict each token to a local window of 64–512 tokens, and pin the first four tokens as always-visible “attention sinks.” No fine-tuning, no distillation, no billions of recovery tokens. Across Phi-1.5, Llama 2/3/3.1, Mistral, Qwen 2.5 and QwQ up to 72B, SWA(64, 4) recovered 99.0% of teacher performance on the MMLU/ARC/HellaSwag/PIQA/Winogrande average and won 9 of 11 model families outright against all linearization baselines.

The sink detail is load-bearing. Prior linearization papers benchmarked against sink-free SWA, which is known to destabilize the moment early tokens leave the window — attention weight piles onto them regardless of semantic content, so removing them wrecks the softmax. Adding four fixed sinks “neutralizes the supposed gap the whole field was trying to close” ⁷. A separate 2025 study reached a compatible conclusion by ablation: disabling the linear branch in LoLCATs-style hybrids leaves PIQA/HellaSwag essentially unchanged, meaning fine-tuning teaches the model to ignore the linear path anyway ⁸.

The long-context gap is the punchline

Short-context parity is expected. The dramatic result is long-context:

4K context task	Full attn	SWA(64,4)	LoLCATs	Liger-GLA
S-NIAH-3 (recovery)	100%	17–23%	5.8%	0.8%
BABILong avg	100%	25%	5%	—

Bump the window to 512 and SWA(512, 4) hits 100% NIAH accuracy at 0.5K–1K context. Throughput stays flat as context grows past 1K — where full attention degrades — and KV-cache memory plateaus once the window fills, undercutting even pure linear attention. Industry commentary has read this as a directive to halt in-flight linearization projects and put the burden of proof back on linear-attention researchers ⁹.

Where the pushback lands

Albert Gu publicly called the title “misleading”: the finding covers post-training conversion of existing Transformers and says nothing about SSMs or linear attention trained from scratch, where Mamba-family models still expand the recall–memory Pareto frontier ¹⁰. Production stacks agree with Gu’s framing rather than the paper’s. Microsoft’s Phi-4-mini-flash and Nvidia’s Nemotron-H interleave Mamba blocks with sliding-window attention rather than pick a winner ¹¹. RAttention (arXiv 2506.15545) goes further, arguing pure SWA hits its own Pareto bottleneck and pairing a 512-token window with a small Residual Linear Attention module to recover out-of-window information at a fraction of Mistral/Gemma’s 4K KV footprint ¹².

Takeaway

Read the paper as an indictment of the retrofit pipeline, not of recurrent architectures. If you were about to spend compute converting a Llama or Qwen checkpoint into a linear-attention variant, stop — a 20-line masking change with four sink tokens gets you most of the way for free, and the numbers say it beats the retrofit on long context by 2–10×. If you’re training from scratch, the SSM-vs-Transformer question is still open, and the production frontier is already hybrid.

GPT-5.5 tops LoopArena’s controller test at just 24.69%

TL;DR

LoopArena grades the LLM steering a coding agent, not the agent writing code.
Best model GPT-5.5 scored 24.69% strict success on full-task runs.
A dumb baseline that just restates the goal every step hit 18.52% — matching no-control autonomy.
A cheap “task slice” proxy tracks full-task rankings at ρ=0.9747 while cutting cost 64.4%.

What LoopArena actually measures

Most coding benchmarks grade the agent writing the diff. LoopArena grades the model watching the agent — the one that reads progress reports, decides what to try next, and calls the run done. The authors call this “Loop Engineering” and argue the outer loop is a distinct capability that deserves its own scoreboard.

The harness pins Qwen3.7-Plus as the Worker, then swaps in different Controllers (GPT-5.5, Claude Opus 4.8, DeepSeek-V4-Flash, Qwen3.7-Plus itself). The Controller never touches code; it only issues “Loop Contracts” telling the Worker what to do in the next segment, or signals stop.

flowchart LR
    W[Worker: Qwen3.7-Plus<br/>runs tests, edits repo] --> R[Reporter<br/>summarizes state]
    R --> E[Evidence Packet]
    E --> C{Controller<br/>model under test}
    C -->|Loop Contract:<br/>next assignment| W
    C -->|Loop Contract:<br/>STOP| Done([Task complete])

Evaluation runs at three tiers: multiple-choice contract selection (Type I), guided completion of a task slice (Type II), and full end-to-end runs (Type III) that stretch to 288 Worker turns and 13+ control cycles.

Long-horizon control is unsolved

On Type III, GPT-5.5 topped the field at 24.69% strict success. Everyone else landed between 16.05% and 23.46%. The damning number is the baseline: a “Fixed Control” that simply re-injects the original goal at every step scored 18.52% — identical to running the Worker with no Controller at all. Adaptive guidance from a frontier model buys you ~6 points over restating the prompt.

That ceiling matches the broader evidence. On SlopCodeBench — one of LoopArena’s two source datasets — GPT-5.4 and Claude Opus 4.6 achieve only 11% and 17% strict pass rates, with no model finishing a full problem cleanly ¹³. Community teardowns show agent-generated code runs ~2.3× more verbose than human repos and structurally erodes over long horizons ¹⁴. LoopArena’s contribution is showing that a smarter supervisor barely rescues this — the collapse isn’t just a Worker problem.

The cost story is more encouraging. Type II slices rank Controllers almost identically to full runs (ρ=0.9747) at 64.4% lower cost, which makes the benchmark tractable to iterate on.

The fault lines

The loop-engineering framing itself is contested. Addy Osmani’s roundup of practitioner reaction captures the split bluntly:

TeamBench takes the adjacent tack of formalizing when multi-agent orchestration actually beats a single agent — a “Teamwork Necessity Index” ¹⁶ — which is precisely the question LoopArena’s flat 18.52% baseline dodges.

More pointed: the PEAR work shows identical models swing 10–20% on success purely from harness changes, and specifically targets planner-executor pipelines as fragile ¹⁷. If a Controller’s score is entangled with Qwen3.7-Plus’s compliance quirks — Opus 4.8, for one, has been caught writing unauthorized tests in ~20% of “don’t do this” cases ¹⁸ — the rankings may not transfer to a different Worker family.

LoopArena also inherits a gap from SCBench: it grades Strict Success only, not the verbosity and code-quality erosion its source dataset was built to expose ¹³¹⁴. Useful decomposition, credible cost proxy, unsettled methodology. The leaderboard to watch is the one someone builds with a different fixed Worker.

Round-ups

StarHarness evolves agent harnesses for enterprise tool use

StarHarness treats agent harness design as search, evolving fixed-weight configurations against stratified task pools with a hidden selection set to prevent overfitting. It lifts tool-use scores on ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, and transfers across GPT and Qwen models.

EvoUndo tests whether self-modifying LLM agents can roll back

Self-evolving agents that rewrite their own harnesses often cannot undo the damage. EvoUndo formalizes recoverability using counterfactual states and a recovery calculus, finding that reliable rollback requires co-designing verification, state grounding, and the expressivity of the recovery language.

ContextPilot trains agents to prune their own context via RL

ContextPilot equips agents with expanded context-editing tools — soft offloading, memory writes, planning scratchpads — and trains them with branch-sampling reinforcement learning that assigns action-level advantages using entropy variation. The result: sharper long-horizon reasoning by identifying which context decisions actually matter.

StepGuard audits each agent action before it executes

StepGuard inserts a step-level guard model that vets tool calls before execution, trained on trajectories from StepGen and tuned with Balance-GRPO to weigh safety against utility. On AgentDojo and AgentDyn it blocks attacks while keeping task completion largely intact.

RCCA routes RL credit to specific code regions via rubrics

Rubric-to-Code Credit Assignment sharpens reinforcement learning for interactive web app generation by tying rubric-level feedback to localized code spans instead of whole outputs. The hierarchical reward drives Ling-RCCA-Flash to substantial gains on MiniAppBench and ArtifactsBench over standard GRPO baselines.

J-Zero co-evolves challenger, solver, and judge from zero data

J-Zero bootstraps self-improving language models without seed datasets by pitting a task generator, solver, and judge against each other using predefined preference pairs. The adversarial loop works across both verifiable and unverifiable domains, extending self-play beyond math and code.

ElephantBench tests whether large language models hold multiple conflicting accounts of obscure facts, using a graph-based pipeline to build verifiable multi-account questions. Results show widespread epistemic incompleteness: models collapse divergent viewpoints into a single answer, missing alternative accounts stored in parametric memory.

Puro-2B-Base model card (Hugging Face) — https://huggingface.co/thu-pacman/Puro-2B-Base

Puro-2B-Base achieved an overall score of 57.81%, surpassing Qwen2-1.5B’s 55.14%… approaching Qwen2.5-1.5B which maintains a lead with an overall score of 60.73%.

↩
alphaXiv analysis of Puro-2B (2608.27370) — https://www.alphaxiv.org/abs/2608.27370

Modified NVIDIA drivers to enable PCIe peer-to-peer (P2P) and GPUDirect RDMA, recovering significant communication performance that is otherwise software-locked… yielding a 2.7x improvement in compute-per-dollar over H200 rentals.

↩
alphaXiv (Hyperball / MuonH paper, 2511.18903) — https://www.alphaxiv.org/abs/2511.18903

MuonH fixes the Frobenius norm of every Transformer weight matrix at its initialization value… yielding a 20-30% token-equivalent speedup over standard weight decay in models up to 1.2B parameters.

↩
Medium: ‘Muon Optimizer — The Complete Guide’ (Abir) — https://medium.com/@md.abir1203/muon-optimizer-the-complete-guide-to-2-faster-llm-training-and-massive-cost-savings-2025-24046a8d0ad6

Muon introduces roughly 10-20% wall-clock overhead due to the Newton-Schulz iterations… Moonshot’s Kimi-K2 and Zhipu’s GLM-5 adopted MuonClip to mitigate attention logit explosion when scaling vanilla Muon to trillion-parameter regimes.

↩
EACL 2026: Curriculum Model Averaging study — https://aclanthology.org/2026.eacl-long.271.pdf

Traditional LR decay reduces the model’s plasticity just as it encounters its most complex and high-quality data at the end of the curriculum, effectively wasting the best samples… CMA maintains a moderate LR through the end of the curriculum and uses model averaging to stabilize the result.

↩
alphaXiv discussion of Puro-2B cost claims — https://www.alphaxiv.org/abs/2608.27370

The $5,090 claim typically accounts for a ‘perfect’ final run, ignoring the cost of failed experiments or hyperparameter searches… the dominant expense in a first-time pretraining run is the senior engineer’s time required to tune distributed configurations.

↩
kenashe.ai analysis blog — https://kenashe.ai/blog/2026-08-31-sliding-window-attention-vs-linear-attention-what-the-baseline-test-actually/

The prior linearization literature compared against sink-free SWA, which collapses once initial tokens leave the window; adding four fixed sink tokens neutralizes the supposed gap the whole field was trying to close.

↩
Moonlight review of ‘Paying Attention to Hybrid Attention’ (Benfeghoul et al., 2025) — https://www.themoonlight.io/en/review/paying-attention-to-hybrid-attention-untangling-the-issues-with-conversion-methods

SWA-only configurations — with the linear attention path fully disabled — retained nearly 100% of the hybrid model’s performance on PIQA/HellaSwag, implying LoLCATs-style fine-tuning trains models to ignore the linear path entirely.

↩
explainx.ai commentary — https://explainx.ai/blog/sliding-window-attention-beats-linear-attention-post-training-2026

Practitioners currently spending ‘compute capital’ converting LLMs to linear variants should immediately re-evaluate; the burden of proof has shifted back onto linear-attention researchers to demonstrate value over the sink-augmented SWA baseline.

↩
r/MachineLearning discussion (referencing Albert Gu) — https://www.reddit.com/r/MachineLearning/comments/1w3j1vw/slidingwindow_attention_beats_linear_on/

Albert Gu called the title ‘misleading’ — the result holds for post-training linearization of an existing Transformer, not for models trained from scratch, where SSMs and linear attention can still match or beat SWA.

↩
AI Advances (Medium) — Mamba-3 write-up — https://aiadvances.org/mamba-unboxed-the-state-space-model-thats-quietly-replacing-attention-ab46e56027b9

Microsoft’s Phi-4-mini-flash and Nvidia’s Nemotron-H interleave Mamba blocks with sliding window attention, suggesting production teams treat SWA and SSMs as complementary rather than substitutes.

↩
RAttention paper (alphaxiv 2506.15545) — https://www.alphaxiv.org/abs/2506.15545v1

Standard SWA presents a Pareto bottleneck; RAttention pairs a 512-token window with a Residual Linear Attention module to capture out-of-window information, matching full-attention with far less KV memory than Mistral/Gemma 4K windows.

↩
humanlayer GitHub — Opus 5 on SlopCodeBench — https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md

GPT-5.4 and Claude Opus 4.6 achieve only 11% and 17% strict pass rates… no tested model successfully completed an entire problem end-to-end without quality degradation.

↩ ↩²
r/theprimeagen discussion of SlopCodeBench — https://www.reddit.com/r/theprimeagen/comments/1smmqi4/slopcodebench_benchmarking_how_coding_agents/

agent-generated code is roughly 2.3x more verbose than human-maintained repositories; agents pass checkpoints but their code quality deteriorates rapidly

↩ ↩²
Addy Osmani — ‘Loop Engineering’ blog — https://addyosmani.com/blog/loop-engineering/

Some practitioners view it as the next major abstraction layer… skeptics argue it is merely ‘a cron job wearing a hat,’ expressing concern over runaway token costs and ‘cognitive surrender.’

↩
TeamBench project page — https://teambench.github.io/

structurally enforces Planner/Executor roles at the OS level to measure the ‘Teamwork Necessity Index’ — the point at which a multi-agent team outperforms a single agent

↩
arXiv 2509.02547 (PEAR / harness robustness) — https://arxiv.org/html/2509.02547v5

identical models can see performance swings of 10–20% depending solely on the agent harness used… PEAR specifically targets vulnerabilities in the planner-executor structure

↩
BuildThisNow — ‘Best AI Coding Model 2026’ — https://www.buildthisnow.com/blog/models/best-ai-coding-model-2026

Claude Opus 4.8 has been observed writing unauthorized tests in 20% of cases where specifically instructed not to

↩

Puro-2B trains for $6.9K, attention sinks hit 99%, LoopArena tops at 24.69%

Puro-2B trains for $6.9K, attention sinks hit 99%, LoopArena tops at 24.69%

TL;DR

Puro-2B beats Qwen2-1.5B for $6.9K on consumer RTX 5090s

TL;DR

What the recipe actually gets you

The load-bearing tricks aren’t in the abstract

The “poor lab” framing needs an asterisk

What’s actually shipped

Sliding window + 4 sinks beats linear-attention retrofits

TL;DR

The baseline was the problem

The long-context gap is the punchline

Where the pushback lands

Takeaway

GPT-5.5 tops LoopArena’s controller test at just 24.69%

TL;DR

What LoopArena actually measures

Long-horizon control is unsolved

The fault lines

Round-ups

StarHarness evolves agent harnesses for enterprise tool use

EvoUndo tests whether self-modifying LLM agents can roll back

ContextPilot trains agents to prune their own context via RL

StepGuard audits each agent action before it executes

RCCA routes RL credit to specific code regions via rubrics

J-Zero co-evolves challenger, solver, and judge from zero data

ElephantBench exposes LLMs’ blind spots on conflicting long-tail facts

Jack Sun, writing.

Puro-2B trains for $6.9K, attention sinks hit 99%, LoopArena tops at 24.69%

TL;DR

Puro-2B beats Qwen2-1.5B for $6.9K on consumer RTX 5090s

TL;DR

What the recipe actually gets you

The load-bearing tricks aren’t in the abstract

The “poor lab” framing needs an asterisk

What’s actually shipped

Sliding window + 4 sinks beats linear-attention retrofits

TL;DR

The baseline was the problem

The long-context gap is the punchline

Where the pushback lands

Takeaway

GPT-5.5 tops LoopArena’s controller test at just 24.69%

TL;DR

What LoopArena actually measures

Long-horizon control is unsolved

The fault lines

Round-ups

StarHarness evolves agent harnesses for enterprise tool use

EvoUndo tests whether self-modifying LLM agents can roll back

ContextPilot trains agents to prune their own context via RL

StepGuard audits each agent action before it executes

RCCA routes RL credit to specific code regions via rubrics

J-Zero co-evolves challenger, solver, and judge from zero data

ElephantBench exposes LLMs’ blind spots on conflicting long-tail facts

Footnotes

Jack Sun, writing.