Nemotron fakes cryptarithm at 7%, Ai2 beats weak KDE 37×, GQE prefills 1.8×

TL;DR

• Nemotron-30B plateaus at 7% cryptarithm accuracy despite 97% per-line arithmetic correctness.
• Nemotron jumps from 0.03 to 0.57 when the hidden key is revealed, exposing the search step as faked.
• Ai2’s DiScoFormer beats KDE 37× at 100-D against a baseline known to collapse past 20 dimensions.
• GQE activates 9 of 16 query heads per token, delivering 1.7–1.8× prefill speedup at long context.
• Tapered Transformers front-load MLP capacity into early layers, improving perplexity at matched compute.

Three unrelated AI-research leads land today, each pulling on a different thread. NVIDIA’s Nemotron-30B turns out to fake cryptarithm search — arithmetic stays clean on 97% of trace lines while final answers stall at 7%, with revealing the key lifting accuracy almost 20×. Ai2’s DiScoFormer posts a 37× density-error win that holds up only against a baseline known to collapse past 20 dimensions. And GQE quietly delivers what the first two only claim: a 1.8× prefill speedup by activating 9 of 16 query heads inside each GQA group, the KV cache left alone.

The briefs cluster around architecture and training recipes — HydraHead mixes Full and Linear attention per head, Tapered Transformers front-load MLP capacity into early layers, and Confident Decoding exits at an intermediate layer when the final one drifts. Read together with the leads, the day’s research surface is unusually focused on where compute actually goes inside a transformer.

LLMs fake cryptarithm search: perfect math, 7% answers

Source: hf-daily-papers · published 2026-06-19

TL;DR

• Cryptarithm accuracy plateaus at 0.01–0.07 for a Nemotron-30B fine-tune, despite 11 rounds of CoT engineering and RL.
• Arithmetic stays correct on 97–100% of trace lines in cryptarithm runs.
• Keep/drop verdicts then emit as random templates, with fidelity as low as 16%.
• Revealing the hidden key converts search to forward derivation, lifting accuracy from 0.03 to 0.57.

The verdict-as-token failure

Arjun Patel’s paper draws a sharp line through the space of “solvable” reasoning tasks: forward-computable procedures distill into chain-of-thought essentially for free, while procedures that require backtracking search over information-free structures refuse to distill at all. Roman numerals, unit conversion, gravity, and monoalphabetic ciphers all train to ~1.00. Cryptarithms — cross-equation constraint search with hidden digits — cap at 0.07 no matter what.

The mechanism is what makes the result interesting. On cryptarithm traces, the model executes the arithmetic on each candidate assignment correctly on 97–100% of lines, then emits a “match / no-match” verdict that is statistically decoupled from that arithmetic ¹. The verdict has become what Patel calls a “high-probability filler phrase” — the same shape as the teacher trace, with the logical content stripped out. Independent coverage reproduces the headline numbers: perfect arithmetic above, 43–84% wrong verdicts below ¹.

The causal intervention nails the diagnosis. When the cipher key is revealed — converting the same cryptarithm instance into a forward derivation — accuracy jumps from 0.03 to 0.57. The model can do the work. It just can’t learn to search for the assignment that makes the work pay off.

Why scale and RL don’t break it

The first instinct is to throw more parameters or RL at the problem. Patel shows neither helps: dense and MoE Transformers from 3B up to 671B parameters all stall at ≤0.04 on cryptarithms, and SFT, RLVR, and STaR each fail to clear the same ceiling.

Convergent evidence from other groups makes this look less like a Nemotron quirk and more like a regime. Kempner Institute researchers report that training models on fixed backtracking traces “locks” them into rigid, suboptimal search strategies, with sequential backtracking actually underperforming parallel sampling on CountDown ². Graph-connectivity work finds small Transformers learn search on toy instances but degrade systematically as graphs grow, even with effectively unlimited data ³. And a separate study supplies the mechanism Patel doesn’t name: history entanglement, where decoder-only Transformers emit different actions for the same logical state depending on what prefix led there ⁴ — exactly the symptom you’d expect when “verdict” becomes a function of trace context rather than evidence.

LogicPuzzleRL provides the cleanest RL data point: cryptarithm-specific RLVR lifted a baseline from 44.51% to 44.98% — a 1.06-point gain ⁵. The absolute floor is higher than Patel’s setup, but the delta corroborates his “invisible leash” framing. Verifiable rewards reshuffle trajectories the model already produces; they don’t install a search procedure that wasn’t there.

What escapes — and what’s still open

The escape hatch in Patel’s own results is telling. The competition’s first-place entry hit 0.92 on cryptarithms by skipping search entirely and memorizing a precomputed catalog of solutions. The task is learnable; the procedure is not.

That dovetails with the broader faithfulness literature, where Claude 3.7 and DeepSeek-R1 reach answers by some other route and post-hoc rationalize a trace ⁶. Patel’s verdict-as-token is the SFT-trained cousin: the trace looks like reasoning, but the answer token has come unmoored from the evidence above it.

The genuinely open question — barely touched here — is whether non-imitation objectives (process rewards, latent CoT, tool calls into an actual solver) move the ceiling or merely relocate it. Nothing in the current evidence settles that, and the negative results are accumulating faster than the positive ones.

---

Ai2’s DiScoFormer beats KDE 37× — against a weak baseline

Source: huggingface-blog · published 2026-06-29

TL;DR

• DiScoFormer’s 37× density-error win at 100-D is measured only against hand-tuned KDE, a baseline known to collapse past ~20 dimensions.
• Stronger neural baselines like Sliced Score Matching (2019), which already handles 100-D mixtures, are not benchmarked.
• The headline “attention ≈ Gaussian kernel” proof was flagged at OpenReview for assuming L2-normalized inputs.
• “Zero-shot” needs an asterisk: authors concede test-time training is required on distributions far from the GMM family.

What Ai2 actually shipped

DiScoFormer is a single transformer with two heads — one for log-density, one for its gradient (the score) — trained on synthetic samples from Gaussian Mixture Models. Because GMMs are universal approximators with closed-form densities and scores, they give an infinite supervised signal mapping point sets to distribution properties. At inference, the model takes a batch of samples as context and answers density/score queries at arbitrary points, no per-task fitting required. A consistency loss tying the score head to the gradient of the density head lets it self-correct on out-of-distribution inputs.

The headline numbers: in 100 dimensions, a 6.5× reduction in score error and 37× reduction in density error versus hand-tuned KDE, and graceful generalization to Laplace and Student-t distributions never seen in training. ICML reviewers liked the package enough to spotlight it as an Oral (top 2.2%) ⁷, which is a real third-party signal — not Ai2-blog framing alone.

Where the blog oversells

The HuggingFace post leans on three claims that the OpenReview thread and adjacent literature qualify.

The “attention generalizes KDE” proof is narrower than advertised. Reviewer NWbe flagged that Proposition 3.2 appears to rely on L2-normalized input vectors, weakening the generality of the equivalence; the authors refined the theoretical positioning rather than strengthening the proof ⁸. Worse, the connection itself isn’t new. Softmax attention has been understood as a learned Nadaraya–Watson estimator with a Gaussian-RBF-like kernel for years, and KDEformer-style architectures already exploit this explicitly ⁹. DiScoFormer’s real contribution is the set-input, GMM-pretrained instantiation — not the proof of equivalence.

“Zero-shot” is partly underwritten by test-time training. In the rebuttal, the authors acknowledge TTT is available to “boost performance on extreme OOD data” ¹⁰ — i.e., pure feed-forward inference degrades when inputs leave the GMM neighborhood. That matters for anyone planning to drop the model into a pipeline as a one-shot KDE replacement.

Boundary effects survive. Reviewer 8TYy pressed on compactly supported densities; the authors conceded the model “might still mirror KDE-like behavior due to the unbounded support of the training GMMs” ¹¹. A known KDE failure mode is inherited, not fixed.

The missing benchmark

The 37× headline is against hand-tuned KDE, which everyone knows collapses past ~20 dimensions. Song & Ermon’s Sliced Score Matching already recovered accurate scores on 100-D Gaussian mixtures back in 2019 ¹², and Score-Debiased KDE variants exist. Without those comparisons, the practical magnitude of the win is ambiguous — DiScoFormer might be the new state of the art on amortized density estimation, or it might be matching what dedicated score-matching methods already do, with the convenience of a pretrained model swapped in for per-problem training.

Takeaway

Treat DiScoFormer as a promising amortized estimator — a genuine engineering result with peer-reviewed pedigree — not as the universal KDE replacement the blog implies. The interesting follow-up isn’t “does it work?” but “does it beat Sliced Score Matching once you turn TTT off?”

---

GQE skips half of GQA’s query heads for 1.8× prefill speedup

Source: hf-daily-papers · published 2026-06-17

TL;DR

• GQE activates 9 of 16 query heads per token, routed inside each GQA group with the KV cache untouched.
• 1.7–1.8× prefill speedup at 4k–1024k context, collapsing to 1.15× at 2k.
• Matches dense GQA (56.04 vs 55.86) at 250M params on 30B tokens — only with renormalized routing plus a shared head.
• DeepSeek-V3 and GLM chase KV-side sparsity for ~2.9× FLOPs cuts at 1M context — a different lever entirely.

What GQE actually changes

Grouped Query Experts (GQE) takes the pool of query heads that share a KV head in a GQA group and treats them as MoE experts. A router picks the top-$k$ per token; the KV side stays dense and untouched. In the headline 16-query/8-KV configuration with $k=1$, only 9 of 16 query computations fire per token (8 routed + 1 shared), and at 250M parameters trained on 30B FineWeb-Edu tokens this matches the all-active GQA baseline (56.04 vs 55.86 average across HellaSwag, PIQA, ARC-Easy).

The “match” is fragile, though, and the paper is honest about it. Naive hard top-$k$ routing drops to 55.43; a weighted concatenation without renormalization falls further to 55.18. Recovering parity requires two specific tricks: an extra attention slot containing a router-renormalized weighted sum of the chosen experts (so gradients reach the router), and one always-on shared head that anchors the layer while the routed experts specialize.

A crowded attention-MoE lineage

GQE is the latest entry in a five-year line of “heads as experts” work, and the comparison matters because each design picks a different axis to sparsify:

GQE’s distinct bet is the dense-KV constraint. MoH-style per-head routing is repeatedly flagged in community discussion as forcing a full KV cache per head, which becomes “prohibitively expensive as sequence lengths grow” ¹⁶. GQE inherits GQA’s bandwidth profile cleanly — which is also why its speedup only materializes once attention dominates compute, beyond ~4k tokens. SwitchHead attacks the projection matrices rather than head count ¹⁴, so the two approaches are largely orthogonal and could in principle compose.

Why 250M doesn’t settle it

Practitioner reaction is cautiously interested but unconvinced. A CSDN write-up of Reddit threads argues the 250M-parameter, 30B-token regime is “insufficient” to prove generalization, and notes simpler memory-cutting tricks like Slim Attention achieve similar wins without MoE’s training-stability risks ¹⁸. At sub-1B scale the gating and dispatch overhead frequently erases FLOPs savings in wall-clock terms — the paper’s own 1.15× number at 2k is consistent with that.

The harder question is direction of travel. Frontier deployments are converging on content-dependent KV sparsity — DeepSeek-V3 and GLM-class systems compute only the top ~2,048 tokens per head, claiming 2.9× per-token FLOPs reduction at a 1M-token window ¹⁷. That attacks the quadratic term GQE deliberately leaves alone. GQE cleanly fills a gap MoH and SwitchHead don’t — query-side sparsity that respects GQA’s KV layout — but the gap may not be where the next order of magnitude is hiding.

Round-ups

HydraHead mixes Full and Linear attention at the head level

Source: hf-daily-papers

Rather than alternating layers, HydraHead assigns Full or Linear attention to individual heads using interpretability-driven selection and scale-normalized fusion. The hybrid reuses pretrained parameters via distillation, beating uniform architectures on long-context benchmarks while cutting training overhead.

Tapered Transformers front-load parameters into early layers

Source: hf-daily-papers

Allocating more MLP capacity to early layers and tapering later ones improves perplexity at matched total parameters and compute. The depth-aware schedule is architecture-agnostic across transformer, recurrent and memory-based variants, challenging the long-standing convention of identical layers stacked uniformly through a network.

Confident decoding picks intermediate layers to cut alignment tax

Source: hf-daily-papers

Qwen researchers find the final transformer layer is not always the most reliable token predictor. An entropy-guided backward search dynamically stops at a more confident intermediate layer, lifting scores on GPQA-Diamond, Omni-MATH and HLE with minimal overhead by sidestepping alignment-induced projection noise.

Tiered Language Models gate private capabilities behind secret keys

Source: hf-daily-papers

McGill researchers propose releasing open-weight LLMs whose computation graph swaps in a private parameter subset only when unlocked with a secret key. Public weights keep instruction-following intact, while sensitive factual knowledge stays inaccessible without authorization, offering a middle path between full open release and closed APIs.

FastMix learns pretraining data mixtures via gradient descent

Source: hf-daily-papers

FastMix treats data-mixture selection as a bilevel optimization, jointly updating mixture coefficients and model parameters through differentiable inner and outer loops. The approach replaces expensive grid search or proxy models with direct gradient-based discovery of optimal domain weights during a single training run.

Token noise and sequence shuffles delay pretraining overfitting

Source: hf-daily-papers

For data-constrained pretraining, simple training-time augmentations — token-level noise, sequence permutations and target-offset prediction — push back the epoch where validation loss turns up. Final model quality improves on fixed datasets trained for many epochs, with all recipes released on GitHub.

Baidu’s Unlimited OCR transcribes many pages in one pass

Source: hf-daily-papers

Reference Sliding Window Attention caps KV-cache growth during long-sequence OCR, letting a single forward pass transcribe multi-page documents without ballooning memory. Baidu reports the parsing-attention mechanism generalizes beyond OCR to ASR and translation, with code released on GitHub.

Footnotes

1. themoonlight.io review of ‘A Verifiable Search Is Not a Learnable Chain-of-Thought’ — https://www.themoonlight.io/en/review/a-verifiable-search-is-not-a-learnable-chain-of-thought

   During free-running generation, the model may perform perfect arithmetic on 97–100% of its lines yet emit incorrect elimination verdicts 43–84% of the time, as the verdict has become a high-probability ‘filler’ phrase.

   ↩ ↩²

2. Kempner Institute (Harvard) — ‘To Backtrack or Not to Backtrack’ — https://kempnerinstitute.harvard.edu/research/deeper-learning/to-backtrack-or-not-to-backtrack-when-sequential-search-limits-model-reasoning/

   Sequential backtracking underperformed parallel sampling in CountDown but excelled in Sudoku… training models on fixed search traces ‘locks’ them into rigid, suboptimal strategies.

   ↩

3. ppc.land — ‘Small Models Can’t Learn Search Tasks Despite More Data’ — https://ppc.land/small-models-cant-learn-search-tasks-despite-more-data/

   Small Transformers successfully learned search procedures for small graphs but failed systematically as graph size increased… even with effectively limitless training data, models rely on shortcuts that do not generalize.

   ↩

4. Can Transformers Learn to Verify During Backtracking Search (ResearchGate) — https://www.researchgate.net/publication/405131843_Can_Transformers_Learn_to_Verify_During_Backtracking_Search/fulltext/6a0fd8f0ce67915c7547ef9b/Can-Transformers-Learn-to-Verify-During-Backtracking-Search.pdf?origin=scientificContributions

   Decoder-only Transformers trained on cumulative traces often emit different decisions for the same state if the prior history differs — ‘history entanglement’ rather than state-based decisions.

   ↩

5. LogicPuzzleRL (ResearchGate) — https://www.researchgate.net/publication/392466003_LogicPuzzleRL_Cultivating_Robust_Mathematical_Reasoning_in_LLMs_via_Reinforcement_Learning

   A model fine-tuned specifically on cryptarithms achieved an accuracy of 44.98%, a +1.06% gain over the 44.51% baseline.

   ↩

6. Cobus Greyling — ‘Chain-of-Thought Reasoning Is Not Always Faithful’ (Medium) — https://cobusgreyling.medium.com/chain-of-thought-reasoning-is-not-always-faithful-d35848eb80f4

   Even advanced models like Claude 3.7 or DeepSeek R1 exhibit implicit post-hoc rationalization, where the model arrives at a biased answer and then constructs a logical-sounding justification after the fact.

   ↩

7. ICML 2026 Spotlight Posters listing — https://icml.cc/virtual/2026/events/2026SpotlightPosters

   DiScoFormer was accepted as a Spotlight (top 2.2%) and Oral presentation at ICML 2026.

   ↩

8. OpenReview rebuttal thread — https://openreview.net/challenge?redirect=%2Fforum%3Fid%3DZuqbPd7xTy

   Reviewer NWbe pointed to a potential flaw in Proposition 3.2, which appeared to rely on a strong assumption of L2-normalized input vectors; authors refined their theoretical positioning in response.

   ↩

9. Sudjianto, ‘Self-Attention as a Kernel Machine’ (Substack) — https://agussudjianto.substack.com/p/self-attention-as-a-kernel-machine

   Dot-product attention is a learned version of the Nadaraya-Watson estimator (1964); softmax attention effectively implements a Gaussian RBF kernel — prior frameworks like KDEformer already made this connection explicit.

   ↩

10. OpenReview author response — https://openreview.net/challenge?redirect=%2Fforum%3Fid%3DgyOWJpP8cQ

    Test-Time Training (TTT) is available to boost performance on extreme out-of-distribution data — implying pure zero-shot inference degrades on inputs far from the GMM training family.

    ↩

11. OpenReview discussion (reviewers + rebuttal) — https://openreview.net/challenge?redirect=%2Fforum%3Fid%3DgyOWJpP8cQ

    Reviewer 8TYy questioned boundary effects on compactly supported densities; authors acknowledged boundary bias ‘might still mirror KDE-like behavior due to the unbounded support of the training GMMs.’

    ↩

12. Song & Ermon, Sliced Score Matching (PubMed-indexed) — https://pubmed.ncbi.nlm.nih.gov/39556504/

    Sliced Score Matching scales linearly with dimensionality and accurately recovers parameters of 100-dimensional Gaussian mixtures where standard score matching fails — an existing high-D baseline DiScoFormer does not directly benchmark against.

    ↩

13. Mixture of Attentive Experts (MAE), Peng et al., ACL 2020 — https://aclanthology.org/2020.acl-main.587.pdf

    MAE reallocates heads to different inputs using block coordinate descent, treating attention heads as experts rather than statically pruning them.

    ↩

14. SwitchHead (Csordás et al., arXiv:2312.07987) — https://arxiv.org/html/2312.07987v1

    SwitchHead applies MoE to the value and output projections of attention, using 4–8× fewer attention matrices than standard Transformers while matching performance.

    ↩ ↩²

15. MoH: Multi-Head Attention as Mixture-of-Head Attention (Jin et al., arXiv:2410.11842) — https://arxiv.org/abs/2410.11842

    MoH-LLaMA3-8B achieves a 2.4% accuracy improvement across 14 benchmarks while utilizing only 75% of the attention heads, and supports continue-tuning from pre-trained dense models like LLaMA-3.

    ↩

16. MoH Hugging Face papers page (community thread) — https://huggingface.co/papers/2410.11842

    Traditional MoH implementations often retain a full KV cache per head to maintain flexibility, which can be prohibitively expensive as sequence lengths grow.

    ↩ ↩²

17. Step-3 / Stepfun technical report — https://chat.stepfun.com/research/zh/step3

    Commercial-scale systems such as DeepSeek-V3 and GLM-class models adopt content-dependent sparse attention computing only the top ~2,048 tokens per head, yielding ~2.9× per-token FLOPs reduction at a 1M-token window.

    ↩ ↩²

18. CSDN practitioner write-up on GQE — https://blog.csdn.net/m0_60388871/article/details/162276913

    Reviewers note the design is interesting but experiments at 250M may be ‘insufficient’ to prove generalization, and compare it unfavourably with mathematically identical compression methods like Slim Attention that avoid MoE training-stability risks.

    ↩