JS Wei (Jack) Sun

Hugging Face sunsets torch_xla, IBM skips fine-tuning, Willison skips review

Hugging Face retires torch_xla, IBM tops a benchmark without fine-tuning, and Willison ships a game with no human code review.

Hugging Face sunsets torch_xla, IBM skips fine-tuning, Willison skips review

TL;DR

  • Hugging Face’s TorchTPU makes TPU a first-class PyTorch device via .to("tpu"), eager by default.
  • PyTorch/XLA archives 2026-11-15, under a month after TorchTPU’s unveiling.
  • IBM’s ALTK-Evolve tops τ²-bench retail at 67.5% Pass^4, beating fine-tuned RAFT-30B by 6.1 points with no weight updates.
  • ALTK-Evolve runs 7× cheaper than ACE on AppWorld at comparable accuracy on gpt-oss-120b.
  • Simon Willison shipped Battle Drones in 321 Claude Code commits, with AI-vs-AI playtests cutting stalemates from 14% to 0.9%.

Three tech features today, three different orthodoxies dropped. Hugging Face’s TorchTPU makes TPU an eager-by-default PyTorch device and sunsets torch_xla on November 15 — the XLA compile step most of the TPU world treated as the price of admission is now optional. IBM’s ALTK-Evolve tops τ²-bench retail with GPT-4.1 plus a learned playbook, beating a fine-tuned 30B model by 6.1 points without touching a single weight. And Simon Willison shipped a browser RTS in 321 Claude Code commits with almost no human review, letting an AI-vs-AI playtest loop replace QA entirely.

The caveats scale with the ambition. TorchTPU still forces custom kernels into Pallas or Helion, not Triton. ALTK-Evolve’s lossless prompt adds 32–65% to per-conversation cost. And Willison’s game eats a 53% visibility penalty on Steam for the AI disclosure. The orthodoxies are negotiable; the bills aren’t.

Hugging Face wires TPUs into PyTorch as torch_xla sunsets

Source: huggingface-blog · published 2026-10-20

TL;DR

  • Hugging Face’s TorchTPU makes TPU a first-class PyTorch device via .to("tpu"), eager by default.
  • PyTorch/XLA archives November 15, 2026 — less than a month after TorchTPU’s unveiling.
  • Compiled inference cuts Qwen-Image-2.1 from 8.0s to 3.5s per 1024² image on Cloud TPU v6e.
  • Catch: custom kernels must be written in Pallas or Helion, not Triton.

What actually ships

TorchTPU plugs into PyTorch via the PrivateUse1 extension point, which is why model.to("tpu") behaves like .to("cuda") instead of the lazy-tensor dance torch_xla forced on users 1. Eager mode is the default; torch.compile(backend="tpu") is the opt-in path to graph-level optimization. Transformers v5 exposes device_map="tpu" in from_pretrained and a TensorParallelConfig that shards weights across chips during load. Diffusers gets the same treatment — pipe.to("tpu") for prototyping, pipe.transformer.compile(backend="tpu") for production — and Hugging Face reports 96.5% of the Transformers test suite passing on TPU, with 34 hardware-specific bugs filed upstream and 19 already closed.

The headline inference numbers come from Cloud TPU v6e: Qwen-Image-2.1 drops from 8.0s to 3.5s per image under compile, MiniMax-H3 video+audio from 262s to 188s. The outlier is kernel-level — the Tokamax ragged-dot kernel from the Kernels Hub pushes Qwen3-30B-A3B decode to 21.5× faster at batch 64, and Gemma 4 26B-A4B to 12.8×, because MoE routing is exactly the workload XLA used to handle badly.

The deprecation shock the post soft-pedals

Hugging Face describes Optimum TPU as “largely obsolete.” The underlying move is sharper: Google and Meta announced TorchTPU at PyTorch Conference 2026 and simultaneously set the pytorch/xla repo to be archived on November 15, 2026 2. Teams running torch_xla in production have weeks, not quarters, to port. Google’s own TorchTitan integration is the only large-scale reference point so far — 52% Model FLOPs Utilization on Qwen 3 8B, scaling to 8,192 Ironwood chips 3 — and that number is self-reported with no independent reproduction yet.

The serving story is also broader than the HF post implies. Google shipped out-of-tree vLLM-TorchTPU and SGLang-TorchTPU plugins plus a new C++ library, TPU Sync, that splits prefill and decode across TPU hosts for disaggregated serving 4. That is where the per-token-cost claims against B200/B300 will actually land or not.

The Pallas tax

The “native PyTorch” framing has a sharp edge: custom ops don’t carry over. Kernel authors must write in Pallas (JAX-based) or the newer Helion DSL, not Triton 5. Any team with a hand-tuned Triton kernel is rewriting it, and GPU-ecosystem staples like bitsandbytes and several PEFT variants still need refactoring to land. Practitioner commentary has been blunt about the second-order problem — TPUs remain, as one write-up put it, “unobtanium” for anyone outside Google Cloud and partners like Anthropic and Meta 6.

The frictionless .to("tpu") call is real. The frictionless supply of TPUs to run it on is not.

Takeaway

TorchTPU fixes the specific thing that made torch_xla miserable — the lazy-tensor debugging loop — and does it with a credible eager-mode API and real inference speedups. But it arrives bundled with a four-week XLA archive deadline, a kernel language most PyTorch developers haven’t written, and a hardware access story that hasn’t changed. The ergonomic gap with CUDA has closed; the ecosystem gap has not.


IBM’s ALTK-Evolve tops τ²-bench retail without fine-tuning

Source: huggingface-blog · published 2026-10-20

TL;DR

  • GPT-4.1 + a learned playbook took #1 on τ²-bench retail at 67.5% Pass^4, beating fine-tuned RAFT-30B-A3B by 6.1 points.
  • No weight updates: guidelines are distilled from past trajectories by an LLM and appended to the system prompt.
  • 7× cheaper than ACE on AppWorld with gpt-oss-120b (116K vs 777K tokens per task) at comparable accuracy.
  • τ²-bench scores only the final DB state, rewarding agents that reach the right outcome via a user-hostile path.
  • The “lossless” prompt adds 32–65% to per-conversation cost, partly recoverable via prompt caching.

The headline result

IBM Research’s ALTK-Evolve takes an off-the-shelf GPT-4.1 agent, bolts a markdown “playbook” of distilled behavioral rules onto its system prompt, and lands at the top of the τ²-bench retail leaderboard — ahead of RAFT-30B-A3B, a model specifically fine-tuned on synthetic τ²-bench retail trajectories. The lift is 10 points of Pass^4 (57.5% → 67.5%) with zero gradient updates 7. On the airline domain paired with GPT-5.5, the same approach hits 82.0% Pass^4 against a frontier-leaderboard ceiling of 66–72%.

The mechanism is unglamorous and that is the point. An LLM reads successful and failed training trajectories, extracts rules (“authenticate the user before any mutation,” “summarize the transaction before commit”), consolidates duplicates, and the result lives in the prompt. IBM’s companion write-up emphasizes variance reduction over mean accuracy: 5-of-5 task completion rose from 53% to 69%, which is the number that actually matters for production reliability 8.

Why this beats ACE on cost

The more interesting comparison isn’t against fine-tuning — it’s against Stanford/SambaNova’s ACE, the other prominent learned-memory framework. On AppWorld with DeepSeek-V3.2, ALTK-Evolve matches ACE’s accuracy using 263K tokens per task against ACE’s 634K; swap in gpt-oss-120b and the ratio widens to 116K vs 777K, roughly one-seventh the inference cost 9. The architectural difference is selective retrieval versus full-playbook injection every turn.

That matters more now because ACE is under reproducibility pressure. Entropi.ai’s 2026 memory survey reports multiple ACE claims failed to replicate in third-party testing, including an “abbreviation kit” that was lossy rather than lossless and regressed accuracy from 96.6% to 84.2% 10. The learned-memory field is noisier than the leaderboards imply, and ALTK-Evolve’s independent corroboration 7 is a real asset.

Where the benchmark cracks

τ²-bench’s state-based verification is strict, but Layer3Labs flags the obvious blind spot:

The binary success metric focuses only on the final database state and ignores the quality of the path taken, potentially rewarding agents that use convoluted or frustrating conversational routes as long as the technical result is correct. 11

For a customer service benchmark, scoring only the end state means an agent can learn rules that pass verification while subjecting users to a bad conversation. Cost, safety, and emotional unpredictability aren’t scored at all.

The second caveat is self-inflicted. The “lossless” configuration adds hundreds of guidelines and raises per-conversation cost 32% in airline and 65% in retail. Practitioners writing on context engineering warn this is exactly the regime where attention dilution kicks in — large just-in-case prompts degrade focus on the primary task 12. IBM’s own data back this up: the airline domain performed better with the trimmed “sup3” set (support ≥ 3) than with everything. Prompt caching recovers much of the token cost in production, but the attention-budget cost doesn’t cache.

Takeaway

The durable signals are the retail SOTA with independent verification and the token-efficiency win over ACE. “Prompt-engineered memory beats fine-tuning” is a real result on this benchmark — but on a benchmark whose metric design and sibling frameworks are both actively contested.


Willison ships vibe-coded RTS with audio-modem multiplayer

Source: simon-willison · published 2026-10-20

TL;DR

  • Simon Willison shipped Battle Drones, a browser RTS built in 321 Claude Code commits, almost none human-reviewed.
  • An Ultracode run with 1,200 AI-vs-AI playtests cut 25-minute stalemates from 14% to 0.9%.
  • Multiplayer syncs two phones via an audio modem with no server required.
  • Independent coverage flags a 53% visibility penalty for AI-disclosed games on Steam Next Fest.

The “fun” pivot is the actual result

Battle Drones is a 2D-then-3D Command & Conquer tribute that Willison steered entirely through natural-language prompts to Claude Code, Opus 5.5 and GPT-6 Astra. The buildings, 3D models, sprite generation (via OpenAI’s image API), reactive procedural music, and even a determinism patch replacing Math.sin/Math.cos with custom implementations to stop Chrome and Safari desyncing replays — all agent-authored. ExplainX’s independent write-up singles out that floating-point fix as the single most impressive moment: the model identified the drift problem and shipped the workaround without being asked 13.

But the headline capability isn’t the asset pipeline, it’s the balancing pass. In “Ultracode” mode, Claude Code orchestrates many subagents at once; Willison kicked off an overnight run and came back to a game tuned against 1,200 simulated matches. Stalemates after 25 minutes dropped from 14% to 0.9%; wins by sides trailing by 800+ scrap rose from 24% to 46%. That’s an agent closing a feedback loop that would normally require a human designer with a spreadsheet.

The caveat Willison soft-pedals: Ultracode is a quota bomb. Independent write-ups describe it as routinely converting five-minute edits into 40-agent pipelines, with “task drift” by the fifth or sixth round of subagent review where “the implementation can deviate so far from the original prompt that it becomes worse than useless” 14. Battle Drones is a best-case demo — a low-stakes greenfield project where more exploration is strictly better. The same mechanism pointed at a production codebase is where the critiques bite.

The audio modem is cool, but not novel

Willison claims audio-sync multiplayer is “a first for an RTS” and names only Alon Horesh’s Siner and Georgi Gerganov’s ggwave as prior art. The lineage is actually much longer. Hackaday traces data-over-sound back to Apple II and C64 cassette loaders, through Chirp.io’s Skylanders integration 15. ggwave itself is a well-characterized reference point: 64–160 bps throughput, roughly three-foot range, significant degradation under ambient noise, and ultrasonic modes requiring 48 kHz sampling that many low-power devices can’t sustain in real time 16.

Any claim of substantially higher throughput than ggwave’s envelope deserves independent benchmarking before anyone treats Battle Drones’ audio layer as a reusable pattern. The pragmatic admission is in the update: real players got headaches from the chirps, and Willison had Claude build a stateless WebSockets relay on Fly.io for $2.19/month. The static-HTML purity didn’t survive first contact with a second human.

Agent battles measure prompts, not models

The Claude-Haiku-5.5-vs-GPT-6-Luna replay is the kind of thing that invites a leaderboard, and Willison hints he wants to build one. The academic literature on LLMs playing RTS games (Orak, RTSGameBench) is cautionary: frontier models form plausible high-level strategies but collapse on micro-execution like unit positioning, and “small prompt variations can drastically shift an agent’s behavior” 17. A Battle Drones league table would be measuring tool-schema sensitivity as much as model skill.

What this is evidence of

Battle Drones is a legitimate capability demo for 2026-era coding agents in a greenfield domain where Willison has no prior expertise. It’s not evidence the pipeline produces anything a market will pay for. Steam Next Fest’s October 2026 cohort crossed 8,000 demos with 36% carrying mandatory AI disclosures (up from 20% in June), and disclosed titles reportedly suffered up to 53% worse visibility and reviews 18. Willison explicitly excludes himself from that arena. The gap between “fun on a plane” and “survives a storefront” is still the whole game.

Round-ups

PyTorch 2.15 upgrades Apple Silicon MPS backend

Source: huggingface-blog

PyTorch 2.15 lands expanded Metal Performance Shaders support for Apple Silicon, closing more gaps with CUDA for local training and inference on Mac. Hugging Face walks through the new operator coverage and performance wins developers see when running transformer workloads on M-series chips.

AnyLanguageModel 1.0 unifies local LLMs on Apple

Source: huggingface-blog

AnyLanguageModel hits 1.0 after a year shipping a single Swift API across Apple’s on-device model stack. The release consolidates MLX, Core ML, and llama.cpp backends so iOS and macOS developers swap local models without rewriting inference code.

eMudhra CA compromise forces root store purge

Source: simon-willison

Private keys for eight eMudhra intermediate CA certificates leaked, and they chain to roots trusted by Mozilla, Microsoft, Apple, and Chrome. Any bundled trust store needs updating — uv 0.13.4, pip 26.2.2, and certifi 2026.10.19 already ship patched lists.

Mollick charts 30 months of AI progress

Source: one-useful-thing

Ethan Mollick marks the launch of his new book with a chart-driven recap of the 30 months since the last edition, tracking benchmark jumps, cost collapses, and adoption curves across frontier models to show where the exponential actually bent.

Footnotes

  1. PyTorch Blog — Introducing TorchTPU — https://pytorch.org/blog/introducing-torchtpu/

    TorchTPU integrates directly via the PrivateUse1 extension point, allowing developers to target TPUs using standard .to(“tpu”) semantics

    ↩
  2. dev.to (Gabriel Anhaia) — PyTorch/XLA deprecation note — https://dev.to/gabrielanhaia/google-open-sourced-torchtpu-pytorchxla-is-deprecated-and-gets-archived-november-15-3cab

    the pytorch/xla GitHub repository will be archived on November 15, 2026, becoming a read-only resource

    ↩
  3. Google Developers Blog — TorchTitan integration — https://developers.googleblog.com/integrating-torchtpu-into-torchtitan-and-optimizing-models-for-tpu/

    achieved 52% Model FLOPs Utilization (MFU) on Qwen 3 8B models, demonstrating near-linear scaling across up to 8,192 chips

    ↩
  4. Google Developers Blog — vLLM & SGLang on TPU — https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu/

    a new C++ library, TPU Sync, [enables] disaggregated serving, where prefill and decode phases are split across different TPU hosts to minimize latency

    ↩
  5. Google Cloud Blog — TorchTPU open source — https://cloud.google.com/blog/topics/ai-infrastructure/torchtpu-is-now-open-source-run-native-pytorch-on-tpus

    custom kernel development is restricted to JAX-based Pallas or the newer Helion DSL, rather than native PyTorch Triton kernels

    ↩
  6. shattered.io — practitioner commentary — https://shattered.io/google-torchtpu-pytorch-nvidia-lock-in-2026/

    developers labeling TPUs as ‘unobtanium’ for anyone outside Google Cloud or its top-tier partners like Anthropic and Meta

    ↩
  7. Unite.AI coverage of IBM Research report — https://www.unite.ai/ibm-research-reports-learned-memory-tops-tau-2-bench-service-benchmark/

    A standard GPT-4.1 agent reached the top of the retail leaderboard, improving its Pass^4 score from 57.5% to 67.5% — surpassing RAFT-30B-A3B, which had been specifically fine-tuned on synthetic τ²-bench retail data.

    ↩ ↩2
  8. IBM Research — ALTK-Evolve Consistency blog — https://huggingface.co/blog/ibm-research/altk-evolve-consistency

    The toolkit raised the share of tasks successfully completed in five out of five attempts from 53% to 69%, targeting the ‘consistency gap’ between one-off success and repeatable reliability.

    ↩
  9. Developers Digest — ACE vs ALTK-Evolve breakdown — https://www.developersdigest.tech/blog/ace-altk-evolve-agent-memory-delivery-cost-2026

    On AppWorld with DeepSeek-V3.2, ALTK-Evolve hit comparable accuracy to ACE using only 263K tokens per task versus 634K; on gpt-oss-120b the gap widened to 116K vs 777K — roughly one-seventh the inference cost.

    ↩
  10. Entropi.ai 2026 AI Memory Survey — https://entropi.ai/blog/context-engineering-ai-memory-landscape-2026

    Several core features of the ACE framework failed to reproduce in third-party testing; a reported +34% boost did not reproduce, and the AI-readable abbreviation kit was lossy rather than lossless, causing a regression from 96.6% to 84.2%.

    ↩
  11. Layer3Labs τ²-bench guide — https://www.layer3labs.io/guides/tau-bench-explained

    The binary success metric focuses only on the final database state and ignores the quality of the path taken, potentially rewarding agents that use convoluted or frustrating conversational routes as long as the technical result is correct.

    ↩
  12. Agenteer — context bloat analysis — https://agenteer.com/blog/the-two-context-bloat-problems-every-ai-agent-builder-must-understand/

    Prompt bloat occurs when developers overload the input with excessive safety layers and ‘just-in-case’ context, leading to high latency and ‘attention dilution’ where the model loses focus on the primary task.

    ↩
  13. ExplainX — independent write-up of Battle Drones — https://explainx.ai/blog/battle-drones-simon-willison-vibe-coded-rts-audio-multiplayer-claude-code-321-commits-2026

    The model not only recognized the potential for floating-point drift but also implemented the complex fix [replacing Math.sin/cos] without direct human instruction… open questions remain about the reliability of such custom math fixes across older hardware or unconventional JavaScript runtimes.

    ↩
  14. MindStudio — ‘What is Ultracode Mode’ — https://www.mindstudio.ai/blog/what-is-ultra-code-mode-claude-code

    Ultracode encourages ‘token-maxing’ to achieve the highest possible quality, it can burn through weekly usage limits in minutes by turning five-minute tasks into 40-agent pipelines… by the fifth or sixth round of subagent reviews, the implementation can deviate so far from the original prompt that it becomes ‘worse than useless’.

    ↩
  15. Hackaday — ggwave (2022) — https://hackaday.com/2022/07/06/ggwave-sings-the-songs-of-your-data/

    ggwave uses FSK with Reed-Solomon error correction to transmit 8–16 bytes per second between air-gapped devices… the concept of data-over-sound dates back to cassette-tape loaders for the Apple II and Commodore 64.

    ↩
  16. Hackster.io — ggwave profile — https://www.hackster.io/news/georgi-gerganov-s-ggwave-offers-air-gapped-comms-between-microcontrollers-smartphones-via-sound-3b2020c00ccd

    Standard protocols deliver 64–160 bits per second… performance degrades rapidly with distance (limited to roughly three feet) and loud background noise. Ultrasonic modes require 48 kHz sampling that many low-power microcontrollers cannot process in real-time.

    ↩
  17. arXiv — Orak RTS/game benchmark for LLM agents — https://arxiv.org/html/2506.03610v3

    Current agents struggle with ‘tactical vs. strategic’ differentiation; an agent might formulate a logical high-level plan but fail at ‘micro’ tasks like unit positioning or responding to a fast-moving threat… small prompt variations can drastically shift an agent’s behavior.

    ↩
  18. Kotaku — Steam Next Fest coverage — https://kotaku.com/steam-next-fest-has-become-a-massive-disaster-and-a-grim-look-at-the-future-2000748008

    Over 36% of October’s demos carry a mandatory generative AI disclosure, a sharp rise from 20% in June 2026… games with AI disclosures suffer a performance gap of up to 53% in visibility and reviews compared to those without them.

    ↩
Jack Sun

Jack Sun, writing.

Engineer · Bay Area

Hands-on with agentic AI all day — building frameworks, reading what industry ships, occasionally writing them down.

Digest
All · AI Tech · AI Research · AI News
Writing
Essays
Elsewhere
Subscribe
All · AI Tech · AI Research · AI News · Essays

© 2026 Wei (Jack) Sun · jacksunwei.me Built on Astro · hosted on Cloudflare