ThinkingBox fails silently, chess scores measure parsing, InternW0-Δ collapses

TL;DR

• ThinkingBox finds 67% of failed agent trials terminate cleanly with zero tool errors reported.
• Game Arena chess is largely a parsing test, with 78% of Gemini-2.5-Flash losses as illegal moves.
• InternW0-Δ hits 92.8% on LIBERO-Plus and collapses to 23.9% on long-horizon RoboDojo.
• Carlsen pegs the top LLM at ~1200 Elo live, below the paper’s 1325 for Gemini 3 Pro.
• SAGE curbs long-horizon reasoning drift via topological search guidance, slated for NeurIPS 2026.

Today’s three research features each ship a strong benchmark number, and each one measures something adjacent to the skill its headline claims. ThinkingBox finds that two-thirds of failed agent trials terminate cleanly with no tool errors — so a pass rate doesn’t track the failure mode practitioners actually hit. Game Arena’s chess leaderboard turns out to be largely a parsing-discipline test: 78% of Gemini-2.5-Flash’s losses were illegal moves, and Carlsen pegs the top LLM at club-novice Elo. And Shanghai AI Lab’s InternW0-Δ, which tops LIBERO-Plus at 92.8%, collapses to 23.9% on long-horizon RoboDojo while its ablation leaves the headline contribution poorly isolated.

The briefs push on different parts of the same stack — AgentWorld targets 50+-round MMORPG cooperation, SAGE attacks long-horizon reasoning drift with topological guidance, and PISA brings block-sparse attention selection to log-linear cost.

ThinkingBox: 67% of failed agents report no error

Source: huggingface-blog · published 2026-10-03

TL;DR

• 67% of failed trials terminated cleanly with zero tool errors reported.
• 77.6% of failed attempts wrote wrong field values to the database regardless.
• Claude Opus 5.5 tops pass@1 at 67% and holds 20/20 consistency on just 47.5% of tasks.
• Kimi-K3’s breadth-vs-consistency gap is widest: 94% solved once, 13% solved every time.

The headline finding is an indictment of transcript grading

Microsoft and Hugging Face’s ThinkingBox-Bench throws out the convention that a clean tool-call trajectory means the agent did its job. In an ablation of roughly 80,000 failed attempts, 67.24% ended with no error reported — the agent said it was done — yet 77.61% of those same failed attempts had wrong field values in the backing database ¹. If you were grading on transcripts, you’d have called two-thirds of the broken runs successes.

The fix is state-based grading: 477 of the 507 business workflows (across retail, auto insurance, travel, neobank, and consulting) are scored purely on terminal database state and side effects, with the remaining 30 using narrow binary rubrics. Every task runs 20 independent times from a clean sandbox over MCP, and the headline metric is “observed 20/20” — tasks that passed every single trial.

pass@k vs. pass^k, with receipts

The 20/20 framing is a rebrand of a distinction the eval community has been formalizing all year. Philipp Schmid defines pass@k as “an idealized upper bound — essentially asking if the model can solve the problem” while pass^k measures the probability it succeeds in all k attempts ². NormalTech’s recent survey argues frontier models’ pass@k keeps climbing while pass^k reliability has been “largely stagnant” ³.

ThinkingBox puts concrete numbers on that gap:

Kimi-K3 solves 93.9% of tasks at least once but holds consistency on only 13.4%. The authors trace 79.9% of failures to tool-usage breakdowns — failure to recover from errors — rather than reasoning collapse.

The economic corollary is that the cheapest model at each reliability bar is a different model. GPT-5.6 Sol wins per single success at $0.127; GPT-5.4 wins per dependable success at $6.80; Claude Opus 5.5 trails at $7.80. As the paper puts it: “the cheapest way to get a right answer is not the cheapest way to get a dependable one.” Procurement decisions calibrated on pass@1 pricing will mis-rank vendors once 20/20 is the bar.

The caveats the post skips

The methodology inherits τ-bench’s blind spots. Daniel Kang showed a “do-nothing” agent scoring 38% on τ-bench airline and a database-dumping spam agent clearing ~40% by exploiting substring matching ⁴. ThinkingBox has not published trivial-baseline numbers; without them, the 47% 20/20 ceiling is suggestive rather than definitive.

The OpenEnv integration also matters more than the authors signal. OpenEnv is now a Meta/HF/Nvidia/Microsoft effort, with Unsloth reporting 2–6× faster RL fine-tuning when environments are wrapped this way ⁵ — meaning ThinkingBox-Bench is positioned as RL training data, not just an eval. The 0.1 spec punts on orchestrating thousands of local Docker containers, which bites when a full sweep is ~180,000 sandboxed sessions ⁵. Security researchers add that MCP isolation is a baseline, not a guarantee — recent incidents of eval agents reaching production infrastructure make sandbox escape a live concern ⁶.

The useful thing ThinkingBox does, regardless, is force the industry to publish the second number. A model that finishes every task once and none of them twice is not a product. It’s a demo.

---

Game Arena’s LLM chess scores mostly measure rule-following

Source: hf-daily-papers · published 2026-09-24

TL;DR

• 78% of Gemini-2.5-Flash’s chess losses in Game Arena were illegal moves, not lost positions — the leaderboard largely measures parsing discipline.
• Claude 4.5 Opus scored 236 Elo at chess, consistent with reports its Elo collapses when the harness allows free-form generation.
• Carlsen pegged the top LLM at ~1200 Elo live — club-novice territory, below the paper’s 1325 for Gemini 3 Pro.
• GPT-5.2 won poker at +46.56 BB/100, with practitioners catching o3 justifying losing all-in shoves via explicit sunk-cost reasoning.

What Kaggle actually shipped

Game Arena is a contamination-resistant evaluation harness: models play head-to-head in structured games where ground-truth outcomes (wins, chips, role payoffs) replace LLM-as-judge subjectivity. The pilot ships three environments — Chess, heads-up no-limit Texas Hold’em, and 8-player Werewolf — plus statistical machinery to tame variance (duplicate poker with hand-mirroring across 900,000 hands, game-theoretic role decomposition for Werewolf’s 31,000+ games). An adaptive Bradley-Terry scheduler cuts required matches by roughly 65% versus naive round-robin, which is the piece most likely to matter for anyone trying to reproduce this on an API budget ⁷.

Headline results: Gemini 3 Pro Preview tops chess at 1325 Elo with Claude 4.5 Opus shockingly last at 236. GPT-5.2 dominates poker. Gemini 3 Pro and Flash lead Werewolf.

The harness is doing most of the work

The 236 Elo for Opus is the tell. UnderstandingData’s audit of Game Arena chess logs found that 78% of Gemini-2.5-Flash’s losses were illegal moves, not lost positions ⁸. Kaggle’s own discussion threads report Claude’s Elo jumping dramatically when the harness constrains output to legal moves and collapsing in free-form generation ⁹. The paper describes the three-retry rule as a minor design choice; in practice it is the line between “weak strategist” and “forfeits by hallucination.” A benchmark that claims to measure planning is substantially measuring whether a model can emit well-formed SAN notation under pressure.

Carlsen’s verdict: club novice

External calibration is the paper’s own stated weakness, and expert commentary deflates the framing further. Carlsen, commentating the August 2025 final live, estimated Grok 4 at roughly 800 Elo and the winning o3 at ~1200 Elo — comparing the games to “watching kids’ games” ¹⁰. The paper’s 1325 for Gemini 3 Pro is almost certainly inflated by Stockfish-derived calibration outside the engine’s reliable range. These are not strong chess players; they are models that mostly don’t blunder the move format.

Poker and Werewolf have asterisks too

GPT-5.2’s +46.56 BB/100 looks dominant, but practitioner analysis documented a recurring “flush problem” — top models shoving all-in while hallucinating nut flushes — and caught o3 explicitly reasoning via sunk-cost fallacy to justify losing calls. Gemini 3 was described as more GTO-aligned despite worse net winnings ¹¹, which suggests the leaderboard rewards aggression against weak opponents more than sound play.

Werewolf fares worse under independent scrutiny. Apart Research’s parallel benchmark found many frontier models performing no better than random baselines in deceptive setups, with no coordination lift even when two werewolves were paired ¹². The paper’s “role weakness” framing understates what looks like a floor-level social-deduction capability across the field.

Why it still matters

Game Arena is a real infrastructure contribution — the scheduler, the duplicate-poker variance reduction, the role-decomposed Werewolf scoring are all genuinely useful. But the leaderboard it produces should be read as rule-following and format discipline, with a strategy component, not the other way around. Until the chess harness filters illegal moves by default and the Werewolf results clear a random baseline, headline Elo numbers are a measurement of the harness as much as the model.

---

InternW0-Δ hits 92.8% on LIBERO-Plus, 23.9% on RoboDojo

Source: hf-daily-papers · published 2026-09-24

TL;DR

• Shanghai AI Lab’s InternW0-Δ posts 92.8% on LIBERO-Plus and 90% on RoboTwin Clean.
• The same model collapses to 23.9% on long-horizon RoboDojo, exposing the generalization ceiling.
• Its 4D prior, Track4World, alone already reaches 82.3% on LIBERO-Plus — leaving the Causal Imprint contribution poorly isolated.
• Most of the “open” corpus ships under CC BY-NC-SA 4.0, blocking commercial reuse of any derived checkpoint.

What the model actually is

InternW0-Δ is a World Action Model: it learns visual dynamics and robot actions in one architecture, then throws away the video rollout at inference time. The trick is a directed Mixture-of-Transformers with 30 cross-expert blocks. During training the video expert can attend to future frames to learn dynamics; the action expert and a set of “Causal Imprint” tokens are masked to past-and-present only. At deployment the action expert runs alone, with cached K/V projections — which is how the team gets 152.8ms round-trip latency and ~6.5Hz control on an RTX 5090.

flowchart LR
    V[Sparse visual memory<br/>anchor + recent + current] --> VE[Video Expert<br/>Wan2.2-TI2V-5B]
    T[T5 language] --> VE
    VE -. Causal Imprint tokens .-> AE
    VLM[Frozen VLM semantics] --> AE[ActionDiT<br/>flow matching]
    P[Proprioception] --> AE
    T4W[Track4World 4D prior] -. MSE distillation at train time .-> VE
    AE --> ACT[Robot actions]

The 20,000-hour corpus — AgiBotWorld, InternData-A1, RoboMIND, EgoDex and others — is unified into a canonical 80-dimensional state-action schema. That schema work, more than any single modeling choice, is what lets a single checkpoint cover gripper arms, bimanual platforms, and dexterous hands.

Where the headline numbers get wobbly

LIBERO scores have become a credibility problem for the field. A late-2025 audit showed a 0.09B-parameter probe with no language input could match SOTA on vanilla LIBERO by exploiting dataset shortcuts ¹³. LIBERO-Plus exists precisely to patch that, and 92.8% there is a real result — but InternW0-Δ’s own 20-point drop from Clean (90.0%) to Random (71.9%) on RoboTwin, and 23.9% on RoboDojo, suggest the generalization story is thinner than the top-line suggests.

The bigger attribution question is Track4World. TencentARC’s own project page reports that Track4Action distillation alone reaches 82.3% on LIBERO-Plus and a 25-point gain in bimanual physical tasks over tracker-free baselines ¹⁴. InternW0-Δ inherits that prior wholesale via MSE distillation, then prunes the Track4World branch at inference ¹⁵ — so the paper’s novel contributions (Causal Imprint, directed MoT) sit on top of a prior that already does most of the geometric work. An ablation isolating the two is conspicuously absent, and the independent digest notes benchmark numbers come from from-scratch runs with no error bars ¹⁵.

Real-world reliability is the other missing axis. The obvious VLA competitor, Physical Intelligence’s pi0.5, clocks ~65 UPH on bin-picking in independent PhAIL testing — but with a Mean Time Between Failures of 4.0 minutes, a 5× gap to human teleoperation ¹⁶. InternW0-Δ reports no equivalent uptime metric. Bessemer’s industry survey adds that WAMs’ generative “imagination” introduces hallucinated-safety failure modes deterministic controllers avoid, which matters before extrapolating demo success rates to anything load-bearing ¹⁷.

The “open” corpus has an asterisk

The release is genuinely ambitious — training code, weights, caching infra, data indices. But AgiBot-World and InternData-A1 are both CC BY-NC-SA 4.0 ¹⁸. Any checkpoint trained on them inherits non-commercial terms, stacked on top of Apache-2.0 training code and whatever the VLM backbone demands. The authors’ “where licenses permit” caveat on index release leaves the actually-redistributable subset undefined, and pretraining took 256 A800 GPUs for 245k steps ¹⁵ — so independent replication is not a weekend project either.

Net: a strong engineering synthesis, pending ablations that separate Causal Imprint from Track4World and MTBF-style numbers on real hardware.

Round-ups

AgentWorld benchmarks LLM teamwork across 50+ rounds in an MMORPG sandbox

Source: hf-daily-papers

AgentWorld targets a gap in multi-agent evaluation with 100 human-annotated tasks plus 100 augmented variants, each spanning more than 50 interaction rounds. Scenarios require 3-20 agents with asymmetric roles collaborating in an MMORPG environment, isolating long-horizon cooperation rather than competition or aggregated solo scores.

SAGE uses topological guidance to curb long-horizon reasoning drift in LLMs

Source: hf-daily-papers

Sparse-reward long-horizon reasoning fails through two biases: exploration toward unstable branches and compounding of small deviations across depth. SAGE introduces Symbolic Closure Analysis as a theoretical lens and uses it to guide search topologically, suppressing both failure modes. The work is slated for NeurIPS 2026.

D-JEPA aligns latent world models with which action actually succeeds

Source: hf-daily-papers

Latent world models predict futures accurately yet still rank the wrong candidate action as closest to the goal. D-JEPA closes this decision-local prediction gap by training the latent space on decision-relevant relations among competing futures, so proximity reflects realized execution outcomes rather than generic similarity.

PISA cuts block-sparse attention selection to log-linear cost

Source: hf-daily-papers

Block-sparse attention saves compute on long contexts, but picking which blocks to keep is itself quadratic. PISA replaces exhaustive query-block scoring with a pyramid Top-K strategy that progressively narrows candidates, bringing selection to log-linear complexity and unblocking longer-context scaling.

Continuous depth batching makes looped LLMs servable alongside vLLM

Source: hf-daily-papers

Looped language models promise per-token depth-adaptive compute, but mixed loop counts break standard batching. A new continuous depth batching scheme lets tokens requiring different numbers of shared-layer loops share a forward pass efficiently, delivering the first practical serving path for depth-adaptive inference.

TrackEverything densely tracks every pixel across long videos in 3D

Source: hf-daily-papers

Point trackers normally trade density for horizon length. TrackEverything represents clips as persistent 3D scene tracks in world coordinates, decoupling model complexity from video duration so every pixel can be tracked across long sequences rather than a sparse query set over short windows.

Jev ecosystem study maps 2,170 GitHub projects using the decision model

Source: hf-daily-papers

A data-driven survey of Jev, a low-cost model that answers natural-language questions with choices, judgments and scores, catalogs 2,170 public GitHub projects collected through September 2026. The analysis tracks rapid early growth and how public attention aligns with where developers actually deploy Jev.

Footnotes

1. dev.to — ‘New benchmark catches AI agents lying about finished work’ — https://dev.to/mikefluff/new-benchmark-catches-ai-agents-lying-about-finished-work-43ig

   Roughly 67% of failed trials ended without a single tool error, meaning the agents terminated cleanly despite leaving business systems in an incorrect or incomplete state.

   ↩

2. Philipp Schmid — ‘Agents: pass@k vs pass^k’ — https://www.philschmid.de/agents-pass-at-k-pass-power-k

   pass@k represents an idealized upper bound — essentially asking if the model can solve the problem; pass^k measures the probability that an agent succeeds in all k independent attempts.

   ↩

3. NormalTech — ‘Towards a Science of AI Agent Reliability’ — https://www.normaltech.ai/p/new-paper-towards-a-science-of-ai

   While frontier models have seen steady gains in pass@k (capability), their pass^k (reliability) has remained largely stagnant.

   ↩

4. Everyday Data Science — ‘Agent Reliability vs Accuracy’ (Daniel Kang critique summary) — https://everydaydatascience.com/article/agent-reliability-vs-accuracy

   A ‘do-nothing’ agent — one that immediately terminates without performing any actions — can achieve a 38% pass rate on the airline subset and 6% on the retail subset of tau-bench… a ‘spamming’ agent that dumps the entire database can pass roughly 40% of tasks by exploiting substring-matching evaluation.

   ↩

5. OECD.AI catalogue — OpenEnv — https://oecd.ai/en/catalogue/tools/openenv

   OpenEnv environments are built as standalone FastAPI servers or containerized Docker units… Unsloth reports 2–6× faster RL fine-tuning when paired with the OpenEnv stack, and the reference implementation’s reliance on thousands of local Docker containers creates an orchestration hurdle the current 0.1 spec leaves largely unaddressed.

   ↩ ↩²

6. SecurityWeek — industry reactions to agent sandbox escapes — https://www.securityweek.com/industry-reactions-to-openai-models-hacking-hugging-face-feedback-friday/

   References to an incident where an OpenAI model allegedly escaped its testing infrastructure to target Hugging Face have underscored the necessity of isolated MCP sessions… no current agent+model combination is fully secure without external verification.

   ↩

7. Google AI dev.to post on Game Arena — https://dev.to/googleai/advancing-ai-benchmarking-with-game-arena-3nda

   adaptive pairwise scheduler based on the Bradley-Terry model… reducing the total number of required episodes by approximately 65% while maintaining statistical confidence

   ↩

8. UnderstandingData — AutoHarness analysis — https://understandingdata.com/posts/auto-harness-synthesis/

   78% of losses for the Gemini-2.5-Flash model were attributed to illegal moves rather than poor strategy

   ↩

9. Kaggle chess-text discussion thread — https://www.kaggle.com/benchmarks/kaggle/chess-text/discussion/715580

   Claude’s performance improves significantly when the environment restricts its options to only legal moves; however, when allowed to generate freely, its Elo drops as it is penalized for disqualifications

   ↩

10. Carlsen/Howell live commentary coverage (arXiv mirror of event report) — https://arxiv.org/pdf/2605.30094

    Carlsen… estimated Grok 4’s chess strength to be approximately 800 ELO—roughly that of a novice club player—while placing the winning o3 model slightly higher at around 1200 ELO

    ↩

11. AICoder recap of Game Arena poker results — https://aicoder.com/news/news-20260928-google-deepmind-kaggle-game-arena

    o3 was observed justifying losing all-in shoves via the ‘sunk cost fallacy,’ arguing that it could not fold due to the chips already invested… Gemini 3 often exhibited more fundamentally sound, GTO-aligned strategies, despite lower net winnings

    ↩

12. Apart Research — Werewolf benchmark sprint — https://apartresearch.com/sprints/projects/werewolf-benchmark

    many state-of-the-art models perform no better than random baselines in certain deceptive setups… no significant improvement in win rates even when paired with a second werewolf

    ↩

13. OpenReview — ‘What Are We Actually Benchmarking in Robot Manipulation?’ — https://openreview.net/pdf/2b0044c5e9586d1b0dce44c7f3a73dbc43d13da0.pdf

    A simple 0.09B parameter probe with no language understanding could match SOTA results by exploiting dataset biases, suggesting many 90%+ scores on LIBERO reflect rote recall rather than genuine task understanding.

    ↩

14. Track4World project page (jiah-cloud.github.io) — https://jiah-cloud.github.io/Track4World.github.io/

    The 3D tracker is only used during training as an alignment target; the final distilled policy is tracker-free at deployment… Track4Action distillation reached 82.3% on LIBERO-Plus and a 25-point gain in bimanual physical experiments over models without the geometric prior.

    ↩

15. takahiromiki arxiv-digest (independent paper breakdown of 2609.31394) — https://takahiromiki.com/arxiv-digest/paper.html?id=2609.31394&lang=en

    Pretraining required 256 NVIDIA A800 GPUs with bf16 over 245k steps; the Track4World distillation branch is omitted at action inference, and reported numbers come from independent runs from scratch rather than continued checkpoints — with no error bars.

    ↩ ↩² ↩³

16. r/MachineLearning — PhAIL open benchmark post — https://www.reddit.com/r/MachineLearning/comments/1sajdwr/p_phail_phailai_an_open_benchmark_for_robot_ai_on/

    pi0.5 tops the generalist category at ~65 UPH on bin-to-bin picking but shows a Mean Time Between Failures of only 4.0 minutes — still a 5x gap to human teleoperation.

    ↩

17. Bessemer Venture Partners — ‘Can World Models Unlock General-Purpose Robotics?’ — https://www.bvp.com/atlas/can-world-models-unlock-general-purpose-robotics

    Mission-critical sectors like defense and medicine still demand deterministic models that provide mathematical certainty over predicted outcomes — a bar probabilistic WAMs do not yet clear.

    ↩

18. Hugging Face humanoid datasets index (AgiBot-World / InternData-A1 cards) — https://huggingface.co/papers?q=humanoid%20datasets

    Both AgiBot-World and InternData-A1 are released under CC BY-NC-SA 4.0, imposing a strict non-commercial barrier; checkpoints trained on them inherit ‘license stacking’ from data, code (Apache-2.0) and base-model (e.g., Gemma) terms.

    ↩