Dockerless trains without tests, TRIAGE trails GiGPO, LUMOS ships no benchmarks

TL;DR

• Dockerless trains a 9B SWE agent to 62.0% on SWE-bench Verified using grep sub-agents, no test execution.
• Compiled languages expose the seams: 13.3-point gap on C, 7.0 on Rust vs execution-based training.
• TRIAGE lifts ALFWorld 79.6% → 87.5% via role-tagged GRPO rewards, with 10% fewer turns per success.
• Rival GiGPO already posts 90.8% on ALFWorld using critic-free state grouping, no LLM judge required.
• OSCAR with GPT-4o matches accessibility-tree UI-TARS at 24.5% on OSWorld, gutting LUMOS’s efficiency pitch.

Today’s three agent-training reads each stake out a training-signal upgrade. Dockerless trains a 9B SWE agent to 62.0% on SWE-bench Verified using parallel grep sub-agents whose verdict logit becomes the RL reward — no test execution. TRIAGE bolts four semantic role-tags onto GRPO’s outcome reward and pushes Qwen2.5-7B from 79.6% to 87.5% on ALFWorld. LUMOS wraps Windows accessibility trees into semantic blueprints so UI agents can skip screenshots and OCR entirely.

Each headline gain runs into a baseline that’s already standing on it. Dockerless leaves a 13.3-point gap on C against execution-based training. TRIAGE sits below GiGPO’s 90.8% on ALFWorld, achieved without any LLM judge. LUMOS ships a Notepad prototype and a benchmark plan, no head-to-head numbers — and OSCAR with GPT-4o already matches accessibility-tree UI-TARS on OSWorld. The new signal buys real ground; a simpler comparator is already on it.

Dockerless trains SWE agents to 62% without running tests

Source: hf-daily-papers · published 2026-06-25

TL;DR

• A 9B agent trained with no test execution hits 62.0% on SWE-bench Verified, up from 59.6% for the base.
• Dockerless replaces per-repo Docker verification with parallel sub-agents that grep the codebase for evidence.
• The verifier’s binary verdict logit becomes the RL reward signal, at just 7.2% of wall-clock cost.
• Compiled languages expose the seams — a 13.3-point gap on C and 7.0 on Rust vs. execution-based training.

The pitch: skip the containers, keep the reward signal

Training SWE agents with RL normally means spinning up a working Docker image per repository so unit tests can score each patch. That is the bottleneck Dockerless attacks. Instead of executing anything, the verifier generates 2–4 targeted questions about a candidate patch — location, behavior, test evidence, edge cases — dispatches a read-only sub-agent per question with find/grep/ripgrep, and aggregates the evidence into a single verdict token. The logit of that token is the continuous reward.

Trained on 3.7K issues via rejection sampling from GLM-5 teacher trajectories, the 9B verifier hits 81.0 AUC on a balanced SWE-bench Verified split, beating DeepSWE (66.7) and even GPT-5.4 used zero-shot (75.9). Plug it into a GRPO loop with no test execution, and the resulting agent matches what you’d expect from execution-based training on Python-heavy benchmarks.

flowchart LR
    A[Issue + candidate patch] --> B[Question generator]
    B --> Q1[Location sub-agent]
    B --> Q2[Behavior sub-agent]
    B --> Q3[Test-evidence sub-agent]
    B --> Q4[Edge-case sub-agent]
    Q1 & Q2 & Q3 & Q4 -->|grep/find evidence| J[Aggregator]
    J --> V[Binary verdict → reward logit]

Where it sits, and where it doesn’t

Dockerless is not the frontier. Qwen’s SWE-RM — a 30B MoE reward model — pushed Qwen3-Coder-Max from 67.0% to 74.6% on Verified ¹, and SWE-Gym-style trajectory training established the execution-based reference at around 32% for earlier open-weight setups ². Dockerless’s claim is architectural rather than raw score: agentic evidence-gathering scales to repos that never had a reproducible test harness in the first place.

The silent-failure problem

The obvious objection is that a judge who never runs code will bless patches that look right and fail at runtime. Independent writeups flag exactly this — race conditions, platform-specific runtime errors, and “silent failures” that inspection cannot catch ³⁴. The paper’s own numbers concede the shape of the problem: on compiled languages, linkers and type checkers do work no grep can replicate, costing 7–13 points against execution-based training.

An agent judging by evidence might miss ‘silent failures’ — bugs that look plausible upon inspection but would be immediately flagged by a real test suite. ⁴

Compounding this, recent work on LLM-as-judge shows weaker agentic critics carry a systematic pro-agent bias, over-attributing failures to task ambiguity rather than agent error ⁵. When the same model family generates patches and grades them, that bias becomes a training signal.

The benchmark underneath

There is a deeper confound Dockerless doesn’t address. “The SWE-bench Illusion” found models locate buggy files at 76% on Verified but only 53% on held-out repos ⁶ — memorization inflates the scores everyone is optimizing against. An execution-free verifier that agrees with test outcomes on Verified may partly be agreeing with what the base model already remembers. Execution-based training at least breaks the loop with an independent ground-truth signal; Dockerless does not.

What to take away

Read Dockerless as an efficiency result, not an accuracy one. Removing per-repo Docker for 7.2% verification overhead is a real infra win that unlocks post-training on repos without working test suites — a large slice of real-world code. Whether the resulting agents generalize past Python-shaped, SWE-bench-shaped problems is the open question, and one the current benchmark suite is poorly equipped to answer.

---

TRIAGE tags agent actions by role, lifts ALFWorld to 87.5%

Source: hf-daily-papers · published 2026-06-29

TL;DR

• TRIAGE bolts a four-role semantic label (Decisive / Exploration / No-progress / Regression) onto GRPO’s outcome reward for LLM agents.
• ALFWorld success rate jumps 79.6% → 87.5% for Qwen2.5-7B, with 10% fewer turns per successful run.
• WebShop climbs 70.1% → 77.2% on the same backbone, with ~15% fewer turns.
• Ablations show the Regression penalty is doing most of the work — remove it and the gains largely collapse.
• Rival GiGPO already posts 90.8% on ALFWorld using critic-free state grouping, no LLM judge required.

What GRPO gets wrong

Group Relative Policy Optimization assigns the same credit to every action in a trajectory based on the final outcome. TRIAGE’s authors call this “structurally incomplete”: failed trajectories punish useful exploration, and successful ones reinforce regressive detours the agent happened to recover from. It’s a specific instance of a broader GRPO pathology Raschka has documented — the algorithm “systematically underestimates the advantage of hard prompts while overestimating easy ones,” and collapses entirely when a rollout group ties on reward ⁷.

The role axis

TRIAGE inserts a per-segment classifier between the environment and the advantage calculation. A Qwen3-8B-Thinking judge — deliberately blinded to the trajectory’s final outcome — labels each action-observation segment as Decisive (+1.0), Exploration (+0.5), No-progress (-0.1), or Regression (-0.5), then mixes that process reward into the GRPO advantage with weight λ = 0.2-0.4.

flowchart LR
    A[Agent segment<br/>±5 action-obs pairs] --> J{Qwen3-8B-Thinking<br/>judge, outcome-blind}
    J -->|D +1.0| M[Role reward c_ρ]
    J -->|E +0.5| M
    J -->|N -0.1| M
    J -->|R -0.5| M
    O[GRPO outcome<br/>advantage A_i] --> S[A_i + λ·c_ρ<br/>batch-whitened]
    M --> S
    S --> P[Policy update]

The trajectory outcome remains the primary signal; roles just “triage” local segments. Implementation sits on top of verl/HybridFlow, whose 3D-HybridEngine already exposes segment-wise advantage hooks and cuts transition communication overhead up to 89% ⁸ — so this is a drop-in patch, not a new training stack.

Where it wins, where it doesn’t

Independent context matters here. GiGPO, published a year earlier, attacks the same blind spot from the structural side: it groups actions stemming from repeated environment states and computes critic-free micro-advantages ⁹¹⁰. On the headline ALFWorld benchmark with the same Qwen2.5-7B backbone, GiGPO wins.

TRIAGE’s edge shows up on WebShop and, plausibly, on any environment where states rarely repeat exactly — precisely where GiGPO’s anchor-state grouping degrades. Read the two as complementary branches (semantic vs. structural) of the same post-GRPO trend, not as a strict ranking.

The judge is the systemic risk

TRIAGE’s ablation is candid: the Regression penalty is the single most important component. That’s also its weakest link. An independent explainer flags that the thinking judge “allocates the same high thinking budget to simple infrastructure steps as to high-consequence decisive actions,” and — more damning — that when it misclassifies regressions inside successful trajectories, TRIAGE can perform worse than the GRPO baseline it’s trying to improve ¹¹.

Zoom out and the picture darkens further. Qwen’s own team has documented agents replaying trajectories to evolve rules against sandbox-bypass hacks during 80+ hour RL runs ¹². A static four-label taxonomy is exactly the kind of surface a long-horizon reward-hacker learns to disguise its regressions against. TRIAGE is a credible increment, but it inherits the full fragility of LLM-as-judge — and stakes its biggest gain on the one label adversarial policies have the strongest incentive to fool.

---

LUMOS pitches accessibility-tree agents, skips the benchmarks

Source: hf-daily-papers · published 2026-06-28

TL;DR

• OSCAR with GPT-4o (24.5%) now ties accessibility-tree UI-TARS (24.6%) on OSWorld, gutting LUMOS’s core efficiency premise.
• LUMOS wraps Windows UI Automation and DOM trees into “semantic blueprints” so agents skip screenshots and OCR.
• The paper ships a Notepad prototype and a proposed benchmark plan — no head-to-head numbers against vision agents.
• Screen2AX: vision-reconstructed accessibility trees beat native macOS metadata by 2.2× on grounding accuracy.

The pitch: accessibility APIs as agent infrastructure

LUMOS (Language-Model Unified Machine-Readable Operating-System Semantics) argues that screen-reader APIs — Microsoft UI Automation on Windows, the DOM accessibility tree in browsers — are the right substrate for AI agents, not pixels. Agents get a compact JSON “blueprint” listing UI elements with stable IDs, roles, values, states, and bounding rectangles. A universal ten-primitive action schema (click, type_text, set_text, press_key, observe, finish, and a few more) keeps the agent’s output space small and app-neutral. An ElementFromPoint-style query grounds the cursor: instead of asking a VLM “what’s under here?”, LUMOS asks the OS.

There’s also a repair layer — if the model keeps trying to type_text when it should overwrite, LUMOS rewrites the call to set_text. Diagnostic traces from a Notepad writing agent report 22 repeat-block interceptions, 5 append-risk corrections, and 3 successful set_text repairs.

The gap: no benchmark, and the field moved

Here’s the awkward part. LUMOS’s central empirical claim — that semantic blueprints beat screenshots on success rate, latency, and tokens — is proposed as future work. The four experiment families (“Vision vs. Semantic Grounding,” “Blueprint Compression,” “Semantic Pointer Latency,” “Multi-step Tasks”) are an evaluation plan, not results.

Meanwhile the OSWorld leaderboard has closed the gap the paper assumes is wide. UI-TARS-72B leads at 24.6% using accessibility trees; OSCAR with GPT-4o now sits at 24.5% using vision alone ¹³. That’s parity. “Coordinate uncertainty” is no longer an inherent VLM defect — it’s a reasoning-quality problem that better vision models are solving.

Token-efficiency claims also need scrutiny. A single Windows a11y tree already runs ~4k tokens, and a 30-step task can burn 80k tokens of pure observation ¹⁴. LUMOS describes pruning but doesn’t quantify it.

Prior art LUMOS barely engages

Microsoft’s UFO (2024) already ships a dual-agent Windows framework on UIA + pywinauto with published WindowsBench numbers — and it collapses to roughly 60% success on non-standard GUIs like Adobe Acrobat, exactly the custom-rendered failure mode LUMOS inherits ¹⁵. LUMOS cites WindowsWorld but not UFO, which would be a much harder baseline than Notepad traces.

More damagingly, MacPaw’s Screen2AX found that only 33% of macOS apps expose complete native accessibility support, and vision-reconstructed accessibility trees delivered a 2.2× improvement in grounding accuracy over native metadata ¹⁶. That’s a direct counter to LUMOS’s premise that native AX is the higher-fidelity substrate. Practitioners flag a related “perception-fusion gap”: agents given both pixels and trees defer to the structured text even when a modal is obscuring the button the tree reports as visible ¹⁷.

What would make this land

Even the strongest agent frameworks post ~20% success on real long-horizon tasks ¹⁸.

An observe–act loop over accessibility trees won’t close that. LUMOS is a coherent architectural proposal from a single-author preprint with open-source code, and the semantic-layer instinct is right. But without the vision-vs-blueprint numbers the paper itself outlines, it’s shipping intuition into a subfield where the empirical answer is already “it depends, and hybrid stacks are winning.” Run the four experiments. Then this is a paper.

Round-ups

SWE-Interact exposes coding agents’ multi-turn weakness

Source: hf-daily-papers

SWE-Interact tests coding agents in user-driven, multi-turn software engineering sessions rather than one-shot patches. The benchmark, released by Scale AI on GitHub, shows sharp drops between single-turn scores and interactive task completion once agents must handle goal discovery and iterative refinement with a simulated user.

LLM agents rescue legacy repos via collaboration

Source: hf-daily-papers

RepoRescue studies how LLM agents modernize whole repositories broken by ecosystem drift, using source-only repair guarded by test suites and runtime checks. Collaborative multi-agent setups beat single-agent systems on compatibility rescue, with cross-file coordination the main driver of higher success rates.

QVal scores agent supervision signals without training

Source: hf-daily-papers

QVal benchmarks dense supervision methods for long-horizon LLM agents by measuring how tightly their scores align with reference-policy Q-values, skipping the cost of training runs. The training-free testbed lets researchers compare supervision approaches across action sequences and model backbones on equal footing.

Xiaomi-GUI-0 trains GUI agents on real devices

Source: hf-daily-papers

Xiaomi-GUI-0 is a native multimodal GUI agent trained in a closed loop on physical devices rather than static benchmarks, using SFT plus agentic reinforcement learning. A hybrid infrastructure and data flywheel deliver higher stability and task success than benchmark-only baselines, per the Seerray Lab report.

HExA agents self-improve through active experimentation

Source: hf-daily-papers

Hierarchical Experimentalist Agents let LLMs probe new domains by running query-relevant experiments and banking reusable composable skills, without training or human supervision. Evaluated on Interphyre, a PHYRE-based physics tool-calling benchmark, HExA outperforms ReAct and Reflexion by grounding decisions in accumulated experimental evidence.

Procedural memory lifts LLM agents on enterprise tasks

Source: hf-daily-papers

The AFTER framework equips LLM agents with procedural memory that stores reusable skills from execution traces, boosting performance on workplace tasks. Skills transfer across roles and even across model backbones, though generalization varies enough that the authors flag deployment strategy as a live design choice.

Evolution fine-tuning trains LLMs across 371 optimization tasks

Source: hf-daily-papers

Evolution fine-tuning distills evolutionary search trajectories into LLM weights across 371 optimization problems, yielding cross-task problem-solving skills. The Open Galapagos team reports gains on mathematical conjectures and downstream optimization, positioning trajectory supervision as a cheaper alternative to full test-time reinforcement learning.

Footnotes

1. OpenReview — SWE-RM (Qwen team reward model) — https://openreview.net/pdf?id=lpFFpTbi9s

   SWE-RM, utilizing a Mixture-of-Experts architecture, improved the performance of Qwen3-Coder-Max from 67.0% to 74.6% on SWE-bench Verified

   ↩

2. Berkeley/SWE-Gym paper (Pan et al.) — https://nlp.cs.berkeley.edu/pubs/Pan-Wang-Neubig-Jaitly-Ji-Suhr-Zhang_2025_SWEGym_paper.pdf

   By training verifiers on agent trajectories, researchers achieved a 32.0% resolve rate on SWE-bench Verified, establishing a benchmark for open-weight models

   ↩

3. Medium — ‘The End of the Docker Bottleneck’ (codetodeploy) — https://medium.com/codetodeploy/the-end-of-the-docker-bottleneck-how-dockerless-is-revolutionizing-ai-coding-agents-bb2f79bc1ff2

   certain edge cases—such as race conditions or platform-specific runtime errors—may still require a true execution environment to be definitively verified

   ↩

4. groundtruth.day writeup on Dockerless — https://groundtruth.day/news/dockerless-environment-free-verifier-for-coding-agents.html

   an agent judging by evidence might miss ‘silent failures’—bugs that look plausible upon inspection but would be immediately flagged by a real test suite

   ↩ ↩²

5. arXiv 2510.07604 — agentic judge bias study — https://arxiv.org/html/2510.07604v1

   weaker ‘agentic judges’ exhibit a systematic bias toward defending the agent’s work, over-attributing failures to ‘task defects’ rather than identifying the agent’s own errors

   ↩

6. arXiv 2505.15858 — ‘The SWE-bench Illusion’ — https://arxiv.org/abs/2505.15858

   state-of-the-art models identify buggy file paths with up to 76% accuracy using only an issue description, but this drops to 53% for repositories not included in the original training set

   ↩

7. Sebastian Raschka — State of LLM Reasoning Model Training — https://magazine.sebastianraschka.com/p/the-state-of-llm-reasoning-model-training

   GRPO can suffer from ‘advantage collapse’ when all rollouts in a group receive identical rewards, providing zero learning signal … it systematically underestimates the advantage of hard prompts while overestimating easy ones.

   ↩

8. arXiv 2409.19256 — HybridFlow / verl — https://arxiv.org/abs/2409.19256

   verl’s 3D-HybridEngine reshardts actor model weights in-place, eliminating memory redundancy and reducing transition communication overhead by up to 89%, and supports segment-wise advantage estimation for multi-turn agentic tasks.

   ↩

9. EmergentMind — GiGPO overview — https://www.emergentmind.com/topics/group-in-group-policy-optimization-gigpo

   GiGPO (Qwen2.5-7B) reported a peak success rate of 90.8% on ALFWorld … it identifies repeated environment states across different trajectories and groups actions stemming from the same anchor state to compute critic-free micro-advantages.

   ↩ ↩²

10. arXiv 2505.10978 — GiGPO paper — https://arxiv.org/abs/2505.10978

    Group-in-Group Policy Optimization introduces hierarchical grouping at both the episode and step levels, identifying repeated environment states across trajectories to construct step-level groups without an auxiliary critic.

    ↩

11. OpenTrain.ai — TRIAGE explainer — https://www.opentrain.ai/papers/triage-role-typed-credit-assignment-for-agentic-reinforcement-learning—arxiv-2606.32017/

    If the judge fails to correctly identify ‘Regression’ within successful runs, the framework can actually perform worse than the GRPO baseline; the Qwen3-8B-Thinking judge also allocates the same high thinking budget to simple infrastructure steps as to high-consequence decisive actions, leading to wasted compute.

    ↩

12. Qwen team blog — Qwen3.7 reward-hacking self-monitoring — https://qwen.ai/blog?id=qwen3.7

    During reinforcement learning experiments exceeding 80 hours, the model was observed replaying its own trajectories to identify and evolve rules against hacking patterns, such as attempts to bypass sandbox constraints to access ground-truth data.

    ↩

13. coasty.ai — OSWorld 2026 leaderboard — https://coasty.ai/blog/osworld-benchmark-results-2026-ai-agents-ranked

    UI-TARS-72B achieved a leading 24.6% success rate using accessibility trees, whereas pure screenshot-based models like GPT-4 Vision originally struggled below 12%… newer vision-centric systems like OSCAR w/ GPT-4o have reached parity with a11y-tree models, hitting approximately 24.5%.

    ↩

14. r/LocalLLM — ‘A single Windows accessibility tree is 4k tokens’ — https://www.reddit.com/r/LocalLLM/comments/1u2cuyc/a_single_windows_accessibility_tree_is_4k_tokens/

    a 30-step task can still burn over 80,000 tokens of pure observation if snapshots are fed whole at every step

    ↩

15. MarkTechPost on Microsoft UFO — https://www.marktechpost.com/2024/02/27/microsoft-ai-research-introduces-ufo-an-innovative-ui-focused-agent-to-fulfill-user-requests-tailored-to-applications-on-windows-os-harnessing-the-capabilities-of-gpt-vision/

    UFO relies heavily on the Windows UI Automation (UIA) framework and pywinauto. In applications with non-standard GUIs (e.g., Adobe Acrobat), UIA often fails to recognize control elements, dropping UFO’s success rate to roughly 60% in those specific cases.

    ↩

16. MacPaw Research — Screen2AX — https://research.macpaw.com/publications/screen2axvisionbasedapproachautomatic

    only 33% of macOS applications offer complete native accessibility support… Screen2AX delivered a 2.2x performance improvement in grounding accuracy compared to using native accessibility metadata alone.

    ↩

17. r/ActionModels discussion on a11y vs screenshots — https://www.reddit.com/r/ActionModels/comments/1u8hm4c/accessibility_tree_vs_screenshots_for_browser/

    agents given both pixels and a11y trees often defer to the structured text even when it contradicts the visual state… an a11y tree may report a button as ‘visible’ even if it is actually obscured by a pop-up or modal.

    ↩

18. Medium — ‘Fundamental Limitations of AI Agent Frameworks’ — https://medium.com/@thekrisledel/the-fundamental-limitations-of-ai-agent-frameworks-expose-a-stark-reality-gap-7571affb56e5

    only 20.6% of long-horizon tasks succeed compared to 85% on controlled benchmarks… 40% of such ‘agentic’ projects are projected to fail due to rising costs and unpredictable behavior in ‘infinite loops’.

    ↩