JS Wei (Jack) Sun

TorchNeuron skips 22k PyTorch tests for Trainium, Scheff buries puzzle-coding

A PyTorch backend ships without 22k upstream tests, and a veteran declares puzzle-coding dead — both for AI throughput.

TorchNeuron skips 22k PyTorch tests for Trainium, Scheff buries puzzle-coding

TL;DR

  • HuggingFace’s TorchNeuron exposes AWS Trainium as a native PyTorch device, skipping ~22k upstream tests.
  • Trainium2 at ~$4.80/chip-hour undercuts H100 at ~$9.80, with Anthropic running 65% of Claude on it.
  • Jeremy Scheff declared programming-as-puzzle dead after an agent produced a 2–3× rewrite.
  • Anthropic junior hiring collapsed from 27% to 9% in a single year.
  • METR trial found senior devs who felt 20% faster were actually 19% slower in familiar code.

Both of today’s tech features cash in an established engineering discipline for the sake of AI-era throughput. TorchNeuron ships AWS Trainium as a native PyTorch device by walking away from roughly 22,000 upstream tests, dynamic shapes, and anything that doesn’t fit a static KV cache — the payoff is Trainium2 at ~$4.80/chip-hour against an H100 at ~$9.80, and Anthropic reportedly running 65% of Claude compute on the backend. Jeremy Scheff makes the parallel trade on the human side, declaring programming-as-puzzle dead after an agent produced a 2–3× speedup on a 3,000-line file he’d deferred for years, with Anthropic’s collapsed junior hiring and METR’s 19%-slower senior devs as the surrounding receipts. The TorchNeuron piece shows what the infrastructure concession looks like; Scheff shows what the career concession looks like. Geoffrey Huntley’s git-pre-commit counter is the only piece arguing the craft can be migrated rather than retired.

TorchNeuron ships for Trainium, skips 22k PyTorch tests

Source: huggingface-blog · published 2026-10-19

TL;DR

  • Hugging Face’s TorchNeuron exposes AWS Trainium as a native PyTorch device, retiring the XLA-based optimum-neuron stack.
  • Public beta on Oct 20, 2026 claims 89% coverage of the top-100 text-generation models out of the box.
  • The beta skips ~22,000 upstream PyTorch tests, forbids dynamic shapes, and still requires static KV caches.
  • Trainium2 at ~$4.80/chip-hour undercuts an H100 at ~$9.80, with Anthropic reportedly running 65% of Claude compute on it.

From “think like a compiler” to a normal device

The old optimum-neuron stack sat on PyTorch XLA, which meant lazy tensors, static shapes, forced recompilations on graph breaks, and a bespoke trainer. TorchNeuron throws that out: Neuron is now a regular torch.device, FSDP2 and tensor parallelism work through standard DeviceMesh/DistributedConfig, and TRL’s SFTTrainer runs unmodified. The same Hugging Face script that trained on an H100 can now target a trn2.48xlarge and produce a hardware-agnostic checkpoint that uploads straight to the Hub.

That’s the architectural win, and it’s a real one. The headline coverage number — 89% of the top-100 text-gen models working out of the box, including large Qwen3 variants and the FLUX diffusion family — is the kind of claim that would have been laughable on the XLA stack a year ago. A native torch.compile path and the new Neuron Kernel Interface (NKI) also give performance engineers somewhere to go when the default kernels fall short; AWS cites a 2.4× speedup on Vision Transformers from NKI-authored ops.

Where the beta frays

Independent reads are more cautious. CloudComputing News notes that the initial release skips roughly 22,000 PyTorch upstream tests and still lacks dynamic input shapes 1. Hugging Face’s own guidance — deactivate automatic dynamic shapes, use static caches — is effectively confirmation. Ecosystem depth is the other quiet problem: there are roughly 50 CUDA Stack Overflow answers for every one Neuron answer 1, so any off-path compile error is still a lonely debugging session.

The surrounding stack is also churning. In under two years AWS has routed users from transformers-neuronx to NeuronX Distributed Inference and now to vLLM Neuron 2. Teams that invested in NxDI in 2025 are being redirected again, which blunts the “stable foundation” framing on the inference side. Training (TorchNeuron + FSDP2 + TRL) looks like the more durable bet.

The strongest tell that friction remains non-trivial is that AWS is simultaneously shipping an LLM agent — neuron-framework-autoport — to automate Hugging Face model porting, hitting ~77.3% pass rates on internal NKI generation benchmarks 3. A vendor that both claims “native, no rewrite” and ships a code-rewriting agent is telling on itself.

Why the migration math still works

The economics are what make the caveats tolerable. Trainium2 instances run around $4.80 per chip-hour on-demand versus ~$9.80 for an H100, and Anthropic — which reportedly runs 65% of its compute on Trainium2 — gets effective rates as low as $0.50/chip-hour on long-term commits 4. Project Rainier, Anthropic’s 500k–1M Trainium2 cluster for training and serving Claude, is the clearest executive-level vote of confidence 5. Deepgram, on the smaller end, reports ~60% more input audio seconds per dollar than its GPU baseline 6.

The honest framing is “credible second source for amortized training workloads,” not “CUDA replacement.”

If your workload is repeatable, large-scale, and already written in idiomatic PyTorch, TorchNeuron closes most of the developer-experience gap that kept Trainium a niche choice. If you need dynamic shapes, exotic ops, or a mature community to Google your way out of errors, the beta label is doing real work.

Further reading


Scheff buries programming-as-puzzle after a 2–3× agent rewrite

Source: simon-willison · published 2026-10-19

TL;DR

  • Jeremy Scheff declared programming-as-puzzle dead after an agent delivered 2–3× speedups and rewrote a 3,000-line file he’d deferred for years 7.
  • Junior technical hiring at Anthropic collapsed from 27% to 9% in a single year 8.
  • METR trial: senior devs who felt 20% faster with AI were actually 19% slower in familiar codebases 9.
  • Geoffrey Huntley’s counter: migrate the craft into git pre-commit hooks that discipline agents deterministically 10.

The capitulation, in Scheff’s own numbers

Simon Willison’s repost of Jeremy Scheff’s “obituary for programming” is less a eulogy than a receipt. Scheff spent years resisting coding agents specifically to preserve the “tricky little logic puzzle” that made the job worth doing. Then he tried one. A single “make it faster” prompt returned 2–3× runtime gains, and an AI-driven overhaul cleared a 3,000-line file he had been putting off “for years” 7. The puzzle wasn’t taken from him by a vendor keynote; it was taken by an agent that cleared his backlog in an afternoon.

Ryan Dahl, from the opposite end of the seniority ladder, voices the same resignation about low-level work: he no longer believes engine engineering “is where I can do the most important work,” citing a “gravity well of compatibility” that blocks the big problems 11. Two very different programmers, same shrug.

The labor market priced it in before the mourners arrived

Scheff’s emotional reckoning trails a structural one. Anthropic’s employment report shows the share of junior roles in technical hiring falling from 27% to 9% in a year, with internal agents clocking roughly six hours of work for every one hour logged by a human 8. The “programming” that remains is being funneled toward senior reviewers who can absorb that output — which is precisely the cohort writing the obituaries. The role didn’t disappear; it got compressed into supervision, and the entry ramp got pulled up behind it.

The productivity story is shakier than the grief

Here the obituary gets complicated. A longitudinal arXiv study documents a “productivity-experience paradox”: the share of developers reporting a worsened experience — eroded flow state, higher cognitive load — doubled from 14% to 27%. More damning, METR’s randomized trial found that experienced developers who felt 20% faster with AI were actually 19% slower on complex tasks in codebases they knew well 9. On Lobste.rs, where Willison found Scheff’s piece, a parallel thread succeeded in deactivating the “vibecoding” tag entirely, with users arguing it had crossed from “useless” to “harmful” hype 12.

The grief may be real while the productivity gain that justifies it is partially illusory for the senior cohort doing the mourning.

Where the craft actually migrates

Against the funereal mood, Geoffrey Huntley offers a working-engineer rebuttal: the craft didn’t die, it moved up a layer into enforcement. If an agent repeatedly violates a project convention, don’t update an AGENTS.md file and hope — implement a git pre-commit hook that fails deterministically and injects the error back into the agent’s context 10. Convention becomes rail. The puzzle-solving instinct still has a home; it’s just that the puzzle is now “how do I build a system that keeps a non-deterministic worker inside the lines,” not “how do I write this inner loop.”

That’s the honest read on Scheff’s post. One kind of programmer lost their favorite part of the job this year. Another kind — the one who liked building the scaffolding more than filling it in — just inherited the whole stack.

Further reading

Footnotes

  1. CloudComputing News — https://www.cloudcomputing-news.net/news/aws-torchneuron-pytorch-trainium-chips/

    The initial release skips approximately 22,000 PyTorch tests and lacks support for advanced features such as dynamic input shapes… for every one Stack Overflow answer regarding AWS Neuron, there are roughly 50 for CUDA-related issues

    ↩ ↩2
  2. Hugging Face optimum-neuron vLLM plugin docs — https://huggingface.co/docs/optimum-neuron/en/guides/vllm_plugin

    AWS moved transformers-neuronx into maintenance mode, advising users to transition to NeuronX Distributed Inference (NxDI). However, by late 2026, AWS further updated this guidance, placing NxDI itself into maintenance and directing developers toward vLLM Neuron

    ↩
  3. AWS ML Blog — Neuron Agentic Development — https://aws.amazon.com/blogs/machine-learning/stop-hand-tuning-kernels-how-neuron-agentic-development-accelerates-aws-trainium-optimizations/

    AI-driven agents such as neuron-framework-autoport are designed to automate the end-to-end porting of Hugging Face models… agents have achieved pass rates of approximately 77.3% on standard NKI generation benchmarks

    ↩
  4. Medium — AWS AI chip pricing analysis — https://medium.com/@cli_87015/aws-built-its-own-ai-chips-and-what-that-means-for-pricing-749ffcb3bf43

    Trainium2 instances are priced at approximately $4.80 per chip-hour, roughly half the on-demand cost of a single NVIDIA H100 at ~$9.80… Anthropic, which reportedly runs 65% of its compute on Trainium2, can achieve effective rates as low as $0.50 per chip-hour through long-term commitments

    ↩
  5. Anthropic press release — AWS compute partnership — https://www.anthropic.com/news/anthropic-amazon-compute

    Project Rainier has deployed clusters containing between 500,000 and 1,000,000 Trainium2 chips to train and serve Claude models; Anthropic’s Head of Compute James Bradbury: ‘performance and scale aren’t just technical requirements—they’re essential’

    ↩
  6. AWS Trainium customer page (Deepgram quote) — https://aws.amazon.com/ai/machine-learning/trainium/customers/

    Deepgram reported that Trainium processed approximately 60% more input audio seconds per dollar than their existing GPU baselines

    ↩
  7. Jeremy Scheff (dumbmatter) — original post, via search summary — https://www.youtube.com/watch?v=I8F5wagYFuc

    by simply instructing an agent to ‘make the code faster,’ he achieved 2x to 3x performance increases… he used AI agents to overhaul a 3,000-line file… that he had been ‘putting off for years’

    ↩ ↩2
  8. Anthropic — Employment at Anthropic research report — https://www.anthropic.com/research/employment-at-anthropic

    the share of junior roles in technical hiring plummeted from 27% to just 9% over the last year… AI agents now log approximately six hours of active work for every one hour logged by a human employee

    ↩ ↩2
  9. arXiv longitudinal study on AI coding adoption (2605.23135) — https://arxiv.org/html/2605.23135v1

    the proportion of developers reporting a worsened experience (such as eroded flow state and increased cognitive load) doubled from 14% to 27%… METR found that while experienced developers felt 20% faster with AI, they were actually 19% slower on complex tasks in familiar codebases

    ↩ ↩2
  10. Geoffrey Huntley — ghuntley.com/toronto — https://ghuntley.com/toronto/

    if an agent repeatedly ignores a project convention… developers should not just update a markdown file but implement a git pre-commit hook… the hook fails deterministically and injects the error message back into the agent’s context

    ↩ ↩2
  11. Simon Willison quoting Ryan Dahl (HN) — https://simonwillison.net/2026/Oct/13/geoffrey-huntley/

    he no longer believes low-level engine engineering ‘is where I can do the most important work,’ citing a ‘gravity well of compatibility’ that prevents solving ‘big problems’

    ↩
  12. zottmann.org — ‘On software developers mourning the craft’ — https://zottmann.org/2026/06/25/on-software-developers-mourning-the.html

    a meta-discussion erupted regarding the ‘vibecoding’ tag, with some users successfully advocating for its deactivation. Critics argued the term had crossed from being ‘useless’ to ‘harmful’ hype

    ↩
Jack Sun

Jack Sun, writing.

Engineer · Bay Area

Hands-on with agentic AI all day — building frameworks, reading what industry ships, occasionally writing them down.

Digest
All · AI Tech · AI Research · AI News
Writing
Essays
Elsewhere
Subscribe
All · AI Tech · AI Research · AI News · Essays

© 2026 Wei (Jack) Sun · jacksunwei.me Built on Astro · hosted on Cloudflare