Subquadratic's 12M-token breakthrough is a finetune with a self-run eval
Every URL the pipeline pulled into ranking for this issue — primary sources plus the supporting and contradicting findings each Researcher returned. Inline citations in the issue point back here.
Sources
A startup claims it broke through a bottleneck that’s holding back LLMs technologyreview.com
Miami-based AI startup Subquadratic came out of stealth mode last month with a huge claim. It announced that it had solved a mathematical bottleneck that had been holding back large language models for almost a decade. The details were thin, and many people were unconvinced. But Subquadratic has started to bring the receipts, sharing the…
References
The Next Web thenextweb.com
Will Depue tweeted that SubQ is ‘almost surely a sparse attention finetune of Kimi or DeepSeek’… Subquadratic’s CTO Alex Whedon eventually confirmed on X that the company used open-source weights as a starting point due to limited initial funding.
SiliconAngle siliconangle.com
Subquadratic debuted with $29 million in seed funding at a reported $500 million post-money valuation, led by Tinder co-founder Justin Mateen (JAM Fund) and former SoftBank Vision Fund partner Javier Villamizar.
Subquadratic technical report (subq.ai) subq.ai
SSA achieves 98% retrieval accuracy at a 12-million-token context window while attending to only 0.13% of token pairs, with a 56x prefill speedup over FlashAttention-2 at 1M tokens and 99.12% on RULER at 128K.
DeepSeek NSA paper (arXiv 2503.01868) arxiv.org
Native Sparse Attention uses a hierarchical three-branch design (sliding window, compressed tokens, selective blocks) and is ‘natively trainable,’ reporting up to 11.6x decoding speedups on 64k sequences — establishing prior art for content-dependent sparse attention that predates SubQ’s launch.
AI Agents Directory aiagentsdirectory.com
Critics like Stepan Goncharov labeled the initial results as ‘cherry-picked,’ pointing out that SubQ compared its performance against older versions of FlashAttention rather than current industry standards, and the Appen evaluation was commissioned by Subquadratic itself.
Galileo AI (Mamba scaling analysis) galileo.ai
Pure subquadratic models often struggle with multi-hop reasoning and precise needle-in-a-haystack retrieval because they compress context into a fixed-size state bottleneck; the industry has converged on hybrid designs (Jamba, Zamba2, Nemotron-3, IBM Bamba) rather than pure subquadratic stacks.
