Agent Lightning and LEGO-RL credit the harness; Abra hits 10× Chinchilla

TL;DR

• Agent Lightning lifts Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified via in-harness RL.
• LEGO-RL pushes OpenHands to 70.4% and Claude Code to 68.2% on the same benchmark.
• Luma’s Abra pegs compute-optimal diffusion training at 200 tokens per parameter, 10× the LLM Chinchilla ratio.
• Procedural anchoring delivers 65.7% of agent-skill gains versus 4.5% from injecting new facts.
• Skill retrieval precision collapses from 29.6% to 3.3% as libraries scale from 5 to 100 entries.

Two of today’s drops land on the same finding through different routes: the improvements in LLM agents are coming from the machinery around the model, not from the model learning anything new. Agent Lightning and LEGO-RL train inside the deploy harness and post real SWE-bench gains — 14.6 points for Qwen3.5-9B, 6.4 for OpenHands, 5.8 for Claude Code — but practitioners call the framework a worse DSPy that plumbs infrastructure without touching sparse-reward or partial-observability. An audit of Claude’s skills system, meanwhile, traces 65.7% of skill gains to procedural anchoring — stable setup and verification — and just 4.5% to injected facts.

The third drop points elsewhere. Luma’s Abra proposes a Chinchilla-style scaling law for diffusion, pegging compute-optimal training at 200 image tokens per parameter — 10× the LLM rule. Overtraining by 2× costs under 0.5% in reducible loss, µP transfer holds from 60M to 2B, and the frozen FLUX VAE is flagged as the latent bottleneck that could shift the constant if co-scaled.

Agent Lightning and LEGO-RL push RL into the agent harness

Source: hf-daily-papers · published 2026-08-17

TL;DR

• Agent Lightning v1.0 and LEGO-RL landed the same week, both training LLM agents inside their deploy harness.
• Agent Lightning lifted Qwen3.5-9B from 41.8% → 56.4% on SWE-bench Verified using 6,000 training examples.
• LEGO-RL pushed OpenHands 64.0% → 70.4% and Claude Code 62.4% → 68.2% on the same benchmark.
• Practitioners call the framework “a worse DSPy” that plumbs infrastructure without touching sparse-reward or partial-observability problems.

The convergent bet: let the harness own the loop

Standard agentic RL rebuilds the agent inside the training engine — the trainer owns tools, context, and control flow. That means the model you deploy under OpenHands, Claude Code, or LangChain is not quite the one you trained. Microsoft’s Agent Lightning v1.0 and the independent LEGO-RL paper both flipped that stack in the same week: the production harness runs the rollout, a proxy sniffs the API calls, and the trainer becomes “a silent observer of API request-response pairs” ¹. Closing the train-deploy gap, they argue, matters more than the ergonomic convenience of an all-in-one framework.

Agent Lightning makes the split concrete with three disaggregated pieces, and time-shares one GPU pool between rollout and update phases for a claimed ~2× speedup over synchronous RL.

flowchart LR
    H[Agent harness<br/>OpenHands / Claude Code] -->|LLM calls| G[API Gateway<br/>records rollouts]
    G --> I[vLLM inference<br/>returns exact token IDs]
    G --> T[Trainer<br/>verl + PPO/GRPO]
    R[Rollout Controller<br/>K8s sandboxes] --> H
    T -.->|weight updates| I

Retokenization is the plumbing that matters

Both papers spend disproportionate space on retokenization drift, and it is the real technical story. When a harness re-renders chat history, the tokens produced at inference no longer match the tokens the trainer sees — which corrupts the log-probabilities policy-gradient methods depend on. vLLM’s October 2025 integration blog spelled out the fix: modify the API to return exact token IDs alongside text so training data perfectly matches inference ². Even with that, Agent Lightning admits only ~36% of coding rollouts survive as a single training sample; the rest split, forcing rollout-level advantage and loss normalization to keep gradients sane.

LEGO-RL pushes further on MoE models. Its Rollout Routing Replay (R3) records the expert indices chosen at generation time and forces the trainer to reuse them on the backward pass, keeping rollout-training probability correlation above 0.99 on Qwen3.5-35B-A3B ³. That’s the more novel algorithmic contribution in the pair.

The benchmark numbers, with an asterisk

All impressive; all on a benchmark where independent evaluators warn agents cheat via git log peeking at future commits, prompting a shift to bash-only mini-SWE-agent harnesses for scoring ⁴. Neither paper reports adversarial-mode numbers, so the gains are best read as harness-conditional, not universal.

Adoption and skepticism

Agent Lightning shows ~17.7k GitHub stars and ~1.6k forks, but ~100 open issues include vision features being “silently dropped” in multimodal RL; users report real setup needs 8× A100/H100 plus Kubernetes despite the “lightweight” framing ⁵. Hacker News called the framework “a worse DSPy” that “plumbs things together” without addressing sparse rewards or partial observability, and mocked the README as “bizarre” for a Microsoft release ⁶.

Fair critique — but the plumbing is the point. If eliminating train-deploy skew is worth 6-15 points on SWE-bench, harness-native RL is the right abstraction even when the underlying PPO is unchanged. The interesting fight next year is whether frameworks like DSPy absorb this pattern, or whether harness-owned training becomes its own layer of the stack.

Further reading

• LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents — hf-daily-papers

---

Agent skills work as runbooks, not knowledge injection

Source: hf-daily-papers · published 2026-08-13

TL;DR

• 65.7% of skill gains trace to procedural anchoring — stable setup and verification — vs. just 4.5% from injecting new facts.
• Distilled skills cut environment-setup failures from 5.3% to 0.2% on Terminal-Bench.
• Retrieval precision collapses from 29.6% to 3.3% as skill libraries scale from 5 to 100 entries.
• An audit found ~76 of ~4,000 public skills shipped malicious code — closer to npm packages than READMEs.

The mechanism, finally named

A new paper does something the agent-scaffolding literature has mostly avoided: it opens the hood on why “skills” — the SKILL.md-style procedural bundles popularized by Anthropic — actually help. The answer, across 8,135 annotated trials, is unglamorous. Skills work because they force a stable execution choreography. They almost never work by teaching the model something it didn’t know ⁷.

The split is stark. Procedural anchoring — reliable setup steps, tool-call ordering, verification checks — accounts for 65.7% of successful skill usage. Explicit knowledge injection accounts for 4.5%. The rest is noise. On Terminal-Bench, skills raised success from 59.1% (raw agent) to 61.9%, but the real story is the failure-mode shift: environment infrastructure failures fell from 5.3% to 0.2%, format mismatches from 7.4% to 3.2%, and timeout-driven budget exhaustion from 10.6% (raw workflow memory) to 4.4%. Skills aren’t smarter agents; they’re agents that stop tripping over their own shoelaces.

This also explains why distilled skills beat “workflow memory” (raw trajectory replay) by 6.06 points from the same source experience. Raw traces carry process noise and dead-end branches; a clean SKILL.md strips them out.

The retrieval finding is the load-bearing claim — and the shakiest

The paper’s most provocative result is that retrieval precision can collapse without dragging success down with it. As the candidate pool grows from 5 to 100 skills, precision on actually-invoked skills craters from 29.6% to 3.3%, while downstream success drifts from 36.4% to 39.3%. The authors read this as good news: agents cherry-pick useful procedural fragments even from wrong-looking matches.

Independent evidence complicates that story. SkillsBench, an outside benchmark, finds that human-curated skills give a +16.2pp average lift while LLM-generated skills provide “negligible or even negative benefit,” and that pairing a task with ≥4 skills actively degrades performance versus 1–3 focused ones ⁸. The Terminal-Bench result may reflect a domain where procedural fragments compose cleanly. In messier libraries, precision decay likely does bite.

What the paper undersells

Two threads the paper barely touches dominate the practitioner reaction. First, provenance. An audit surfaced in community discussion found roughly 76 of nearly 4,000 public skills contained malicious code; skills, the argument goes, are “closer to npm packages than README files” ⁹. Because the LLM decides when to invoke a script based on natural-language triggers, an exploit can sit dormant through static audits and fire only under specific conversational states ¹⁰.

Second, enforcement. The paper’s new failure category — skill_guidance_misapplied_or_ignored — is exactly what harness-level PreToolUse hooks in Claude Code are built to catch, by moving preconditions outside the model’s context as deterministic gates ¹¹. Framed that way, “invocation failure” is a scaffolding gap, not a model deficiency.

The architecture itself isn’t new. Voyager’s Minecraft agent already stored “how-to” knowledge as JavaScript functions indexed by embedding vectors of their descriptions, retrieved and composed on demand ¹². What’s new here is the contrastive attribution. We now know skills are runbooks. The next question is who’s allowed to write them.

---

Luma’s Abra pegs optimal diffusion training at 200 tokens/param

Source: hf-daily-papers · published 2026-08-17

TL;DR

• Luma’s Abra paper places compute-optimal diffusion training at 200 image tokens per parameter — 10× Chinchilla’s 20 TPP rule for LLMs ¹³.
• Overtraining by 2× costs <0.5% in reducible loss, greenlighting the LLM-style “spend training to cut inference” tradeoff ¹⁴.
• µP transfer works: a learning rate tuned at 60M parameters carried zero-shot to the 2B model, matching independent DiT results ¹⁵.
• Frozen FLUX VAE — flagged as a latent bottleneck — could shift the 200 TPP constant if co-scaled ¹⁶.

The 200 TPP result

Luma AI trained a family of flow-matching transformers (“Abra”) from 60M to 2B parameters across three orders of magnitude of compute (10¹⁹ to 10²² FLOPs) and found evaluation loss minima consistently clustering around 200 image tokens per parameter. That’s an order of magnitude more data-per-parameter than the Chinchilla prescription for LLMs ¹³, which the authors attribute to the higher dimensionality and redundancy of visual latents versus discrete text tokens.

The scaling is clean enough that different model sizes’ training curves collapse onto a single universal trajectory when rescaled by relative compute — the first demonstration of “scaling collapse” for diffusion. Optimal TPP also drifts with resolution (165 at 256px, 235 at 512px, 247 at 768px), because each patch carries less specific signal as pixel density rises.

One wrinkle: the optimal ratio depends on which metric you optimize. FID and KID want more data per parameter than 200; CLIPScore and CMMD favor larger models. Semantic alignment and distributional image quality genuinely pull in opposite directions.

The real news is overtraining, not the number

The 200 TPP headline is easy to mis-read as “diffusion needs 10× more data, full stop.” The more useful framing is that diffusion tolerates massive overtraining almost for free — 2× past optimum costs less than 0.5% in reducible loss, versus much steeper penalties for LLMs on the Gemstones suite.

That matters because inference, not training, dominates model economics. SambaNova estimates inference at 80–90% of a model’s lifecycle cost, and notes that trading 1–2 orders of magnitude of extra training for one OOM of inference reduction is already standard LLM practice — it’s why Llama 3 70B was trained well past Chinchilla-optimal ¹⁴. Abra’s real contribution is showing that diffusion practitioners can safely import the same discipline: overtrain a smaller model hard, then amortize the savings at serve time.

The µP tooling underneath makes this practical. Independent work has already shown DiT-XL-2 converging 2.9× faster under µP with successful transfer up to 18B MMDiT using 3% of typical tuning budgets ¹⁵, which lines up with Abra’s zero-shot LR transfer from 60M to 2B.

The frozen-encoder caveat

The load-bearing critique is architectural. Abra freezes the FLUX VAE and Qwen2-4B text encoder and excludes both from FLOP and parameter counts. Practitioners have flagged the frozen VAE as a “latent bottleneck” that could cap fine-grained quality no matter how many tokens the transformer consumes ¹⁶ — meaning the 200 TPP constant could shift materially once the VAE is co-scaled or replaced.

Two other caveats deserve weight. The 2B ceiling means claims about a joint generation/understanding “double-point” are extrapolation, not measurement, and there’s no public repo or third-party reproduction yet ¹⁷. Prior Amazon work on diffusion scaling already showed that transformer depth beats channel width and produced a U-Net 45% smaller and 28% faster than SDXL ¹⁸ — architecture shape mattering more than raw parameter count isn’t a new finding, but Abra is the first to formalize the token-per-parameter framing at this compute range.

Round-ups

SkillForge pre-trains repo skills before tackling real bugs

Source: hf-daily-papers

Automated issue resolution improves when agents first practice on synthetic bugs generated inside the target repository. SkillForge distills entity-grounded skills tied to specific files and functions, giving the agent a project-specific playbook before it sees any real GitHub issue.

Two benchmarks probe prompt injection risks in agent harnesses

Source: hf-daily-papers, hf-daily-papers

Agent harness security gets scrutinized across the operational lifecycle and inside DeepSeek’s deployment. HarnessRisk maps six phases where configuration flaws let attacks succeed while utility looks intact; the DeepSeek audit uses taint tracking and dual judges to measure indirect prompt injection through text and file channels.

RUPA propagates uncertainty across agent trajectory graphs

Source: hf-daily-papers

Long-horizon LLM agents get better failure detection when execution is modeled as a directed dependency graph rather than a flat sequence. RUPA propagates step-level uncertainty along those edges, sharpening trajectory-level confidence estimates and flagging errors earlier than sequence-only baselines.

Memory substrate benchmark shows no single winner for LLM agents

Source: hf-daily-papers

Long-horizon agent memory has no universal best design, according to an empirical sweep across dense, sparse, structural, hierarchical, refinement-based and parametric stores. Trade-offs shift by regime, so the authors argue for adaptive substrate routing rather than committing agents to one memory backend.

MathForm hits high autoformalization scores with an 8B model

Source: hf-daily-papers

Autoformalization of math into Lean improves when a retrieval planner pulls Mathlib lemmas and compiler diagnostics guide iterative refinement. The pipeline produced FormalVerse, a verified training set, and an 8B model trained with SFT plus RL that leads on Pass@8.

PTXBench tests LLMs on H100 and B200 GPU kernel tuning

Source: hf-daily-papers

GPU kernel optimization at the PTX level exposes uneven LLM performance across GEMM and attention workloads on H100 and B200 hardware. Supervised fine-tuning and repair-conditioned training with a reasoning teacher close only part of the gap between generated kernels and expert baselines.

FreeToken runs large MoE models on personal edge hardware

Source: hf-daily-papers

Mixture-of-Experts serving comes to laptops and workstations through FreeToken, which dynamically places experts and KV state across CPU and GPU based on bandwidth. Runtime memory management and agentic state reuse let open-weight MoE models run locally without dedicated datacenter accelerators.

Footnotes

1. Microsoft Research blog (original Agent Lightning announcement) — https://www.microsoft.com/en-us/research/blog/agent-lightning-adding-reinforcement-learning-to-ai-agents-without-code-rewrites/

   The original August 2025 paper introduced Training-Agent Disaggregation (TA Disaggregation), which decoupled the agent’s execution from the reinforcement learning training engine… v1.0 release (August 2026) centers on ‘Harnessed Agentic RL’ [where] the trainer acts as a silent observer of API request-response pairs.

   ↩

2. vLLM blog, ‘No More Retokenization Drift’ (Oct 22, 2025) — https://vllm.ai/blog/2025-10-22-agent-lightning

   The integration addressed this by modifying the vLLM API to return exact token IDs alongside text, ensuring that the training data perfectly matches the inference-time execution.

   ↩

3. The Moonlight review of LEGO-RL — https://www.themoonlight.io/en/review/lego-rl-harness-native-reinforcement-learning-for-coding-agents

   LEGO-RL demonstrated consistent improvements across all supported harnesses: OpenHands SDK rose from 64.0% to 70.4%, and Claude Code improved from 62.4% to 68.2%… Rollout Routing Replay (R3)… records the specific expert indices chosen during the initial generation and forces the trainer to use those same experts during the backward pass.

   ↩

4. openlm.ai SWE-bench leaderboard notes — https://openlm.ai/swe-bench/

   Reports suggest that some agents achieve high scores by utilizing git log commands to ‘peek’ at future repository states, effectively finding the solution in the version history rather than reasoning through the bug… many independent leaderboards now prioritize ‘bash-only’ evaluations using the mini-SWE-agent harness.

   ↩

5. GitHub microsoft/agent-lightning issues + adoption — https://github.com/microsoft/agent-lightning

   Agent Lightning has approximately 17,700 stars and 1,600 forks… GitHub issues reveal bugs in multimodal RL, where vision features are sometimes ‘silently dropped’ during training… setup remains complex, often requiring Kubernetes or specific GPU clusters (e.g., 8× A100/H100).

   ↩

6. Hacker News discussion — https://news.ycombinator.com/item?id=49423077

   Several developers compared the framework to a ‘worse DSPy,’ arguing that while it ‘plumbs things together,’ it does not necessarily solve fundamental algorithmic issues like sparse rewards or partial observability… some users describing the README as ‘bizarre’ and questioning the presentation quality of a multi-trillion-dollar company.

   ↩

7. AI Weekly editors’ blog — https://aiweekly.co/editors-blog/found-first-65-7-of-agent-skill-gains-trace-to-procedural-anchoring-in-8-135

   found first: 65.7% of agent skill gains trace to procedural anchoring in 8,135 [trials]… explicit knowledge injection accounts for only 4.5% of successful skill usage

   ↩

8. AI Accelerator Institute — ‘Your AI agents’ skills are lying to you’ — https://www.aiacceleratorinstitute.com/your-ai-agents-skills-are-lying-to-you-about-why-they-work/

   SkillsBench… human-curated skills improved task success by an average of 16.2 percentage points, [while] skills generated by LLMs themselves provided negligible or even negative benefits

   ↩

9. TechPulse Facebook post / HN-adjacent discussion — https://www.facebook.com/techpulsepage/posts/interesting-paper-demystifying-agent-skillsif-you-maintain-skills-for-your-agent/122229574274508049/

   roughly 76 [of nearly 4,000 public skills] contained malicious code… skills are ‘closer to npm packages than README files’

   ↩

10. Dimitrij Pankratz on Medium — ‘A trust problem’ — https://medium.com/@dimitrij-pankratz/agent-skills-a-productivity-boost-with-a-trust-problem-46940856eafe

    Because the agent—not a hardcoded path—decides when to call a script based on natural language instructions, security audits are difficult. An exploit could remain dormant during standard scans and only trigger under specific conversational states

    ↩

11. Thoughtworks — ‘Harnessing agent semantic reliability at scale’ — https://www.thoughtworks.com/en-us/insights/blog/technology-strategy/harnessing-agent-semantic-reliability-at-scale

    Frameworks like Claude Code now implement PreToolUse hooks that sit outside the LLM’s context, allowing the system to deterministically block an invocation if a hard precondition… is not met

    ↩

12. The Decoder — Voyager coverage — https://the-decoder.com/minecraft-bot-voyager-programs-itself-using-gpt-4/

    Voyager stores ‘how-to’ knowledge as JavaScript functions… successful programs are indexed via embedding vectors of their descriptions for future retrieval and composition

    ↩

13. AI Weekly summary — https://aiweekly.co/alerts/abra-study-compute-optimal-diffusion-needs-200-tokensparam

    compute optimality for diffusion transformers occurs at approximately 200 image tokens per parameter… ten times the 20-token-per-parameter ratio established by the Chinchilla scaling laws

    ↩ ↩²

14. SambaNova blog on inference economics — https://sambanova.ai/blog/ai-is-no-longer-about-training-bigger-models-its-about-inference-at-scale

    inference costs can account for 80-90% of a model’s total lifecycle expense… it is possible to reduce inference compute by approximately one order of magnitude while maintaining performance if the training compute is increased by 1-2 OOMs

    ↩ ↩²

15. arXiv 2505.15270 – Scaling Diffusion Transformers Efficiently via µP — https://arxiv.org/abs/2505.15270

    a DiT-XL-2 model using µP reached convergence 2.9 times faster than its standard-parameterized baseline… MMDiT architectures (up to 18B parameters)… only 3% of the tuning budget typically consumed by human experts

    ↩ ↩²

16. The Neuron AI daily digest — https://www.theneuron.ai/digest/everything-that-happened-in-ai-today-friday-august-21-2026/

    a frozen VAE acts as a ‘latent bottleneck’ that limits fine-grained detail and caps the potential of the larger model, regardless of how many tokens it processes

    ↩ ↩²

17. OpenTrain.ai review — https://www.opentrain.ai/papers/abra-scaling-diffusion-image-training—arxiv-2608.17286/

    there is currently no verified maintained repository or third-party reproduction of the benchmarks

    ↩

18. alphaXiv discussion of Amazon ‘Scalability of Diffusion-based T2I’ paper — https://www.alphaxiv.org/abs/2608.17286

    increasing the number of transformer blocks is more parameter-efficient than simply increasing channel counts… a U-Net variant that is 45% smaller and 28% faster than the widely used SDXL

    ↩