Qwen 3.8 27B ties GPT-5.6 Luna on score, burns 2.3× the tokens
Qwen 3.8 27B hits GPT-5.6 Luna's benchmark score but spends 2.3× the tokens getting there, a latency tax that bites agentic loops.
Qwen 3.8 27B ties GPT-5.6 Luna on score, burns 2.3× the tokens
TL;DR
- Qwen 3.8 27B scored 52 on the Artificial Analysis Index, tying GPT-5.6 Luna.
- The open model burns 2.3× more tokens than Luna on equivalent tasks.
- One SVG prompt consumed >22,000 reasoning tokens under the default overthinking setting.
- A Q4_K_M quant fits ~17 GB VRAM at 99.97% quality on a single RTX 3090.
Today’s tech pool is a single release, and the interesting number isn’t the benchmark score — it’s the token count next to it. Qwen 3.8 27B clears 52 on the Artificial Analysis Index, tying GPT-5.6 Luna and trailing the 753B GLM-5.2 by a single point. On the leaderboard, that’s a headline. In an agentic loop, it’s a bill: the same tasks that Luna finishes cost Qwen roughly 2.3× the tokens, and the default reasoning trace will happily spend >22,000 of them drawing one SVG. The weights are open, a Q4_K_M quant runs near-lossless on a single RTX 3090, and the parity is real — but parity in score isn’t parity in inference cost, and that gap is where the deployment math actually happens.
Qwen 3.8 27B matches GPT-5.6 Luna but burns 2.3x the tokens
Source: simon-willison · published 2026-08-17
TL;DR
- Qwen 3.8 27B scored 52 on the Artificial Analysis Index, tying GPT-5.6 Luna and trailing 753B GLM-5.2 by 1 point.
- 2.3x more tokens than GPT-5.6 Luna on equivalent tasks — a latency tax that bites in agentic loops.
- >22,000 reasoning tokens to draw a single SVG, an extreme overthinking default.
- A Q4_K_M quant fits in ~17 GB VRAM at 99.97% quality, running near-lossless on a single RTX 3090.
The parity number is real
At 27B parameters, Alibaba’s new Qwen is inside a point of models 25–60x its size. GLM-5.2 (753B) and DeepSeek V4 Pro 0813 (1.7T) both score 53; GPT-5.6 Luna, size undisclosed but presumed much larger, ties Qwen at 52. Vendor benchmarks back the story: 61.7 on SWE-bench Pro, 90.3 on LiveCodeBench v6, and a DeepSWE 1.1 jump from 13.3 on Qwen 3.6 to 42.2 on the new model 1. Day-zero llama.cpp support and a native 262k context window make it the strongest open-weights option for local agentic work right now 1.
The community’s quant results seal the “runs on your desk” pitch. Q4_K_M lands at ~17.1 GB VRAM with 99.97% of full-precision quality — indistinguishable from Q8_0 in blind eval, and notably better than the newer NVFP4 format 2. A single 3090 is enough.
| Model | Params | Index score |
|---|---|---|
| GLM-5.2 (max) | 753B | 53 |
| DeepSeek V4 Pro 0813 (max) | 1.7T | 53 |
| GPT-5.6 Luna (max) | undisclosed | 52 |
| Qwen 3.8 27B | 27B | 52 |
The overthinking tax
The score buys itself with tokens. Simon Willison’s follow-up documents the model consuming over 22,000 reasoning tokens to produce a simple SVG graphic, and calls setting reasoning_effort to medium or low “essential” for anything routine 3. Hacker News commenters describe the internal <think> traces as a “caveman”-style shorthand driven by what one reader-engineer dubbed a “desired oververbosity 9” parameter, routinely pushing past 10,000 reasoning tokens on straightforward prompts 4.
A separate HN thread quantifies the aggregate cost: Qwen 3.8 consumes up to 2.3x more tokens than GPT-5.6 Luna for the same task 5. In batch eval that shows up as equal accuracy at higher latency; in a live agent loop with tool calls, it’s the difference between snappy and sluggish.
Parity on the Intelligence Index masks a latency tax — 2.3x more tokens than GPT-5.6 Luna for the same task. 5
Cracks worth naming
Not every benchmark flatters the release. Willison’s Aug 16 writeup flags a regression on specialized general knowledge: Qwen 3.8 27B reportedly lost a majority of medical benchmark trials against its own predecessor, Qwen 3.6 27B 6. That fits a pattern where reasoning-heavy post-training trades domain recall for chain-of-thought depth — a real cost if you were planning to lean on the model for factual work rather than derivation.
The broader benchmaxxing critique applies too: a one-point gap on an aggregate index where top models now cluster within noise, graded partly by LLM-as-judge, is not a crisp signal.
Net read
Qwen 3.8 27B is the sharpest test yet of the “small model + huge reasoning budget” thesis. As a locally deployable weight release it’s remarkable 21. As a drop-in for agentic production it’s more complicated — the 52 is bought with verbosity that eats the economic advantage the parameter count implies 35, and the medical regression 6 suggests the reasoning gains weren’t free elsewhere either. Clip the effort knob before you benchmark it against your own workload.
Footnotes
-
kie.ai — Qwen 3.8 27B model overview — https://kie.ai/blog/qwen-3-8-27b-27b-dense-multimodal-local-model
↩ ↩2 ↩3Vendor-reported 61.7 on SWE-bench Pro and 90.3 on LiveCodeBench v6; DeepSWE 1.1 jumped from 13.3 (Qwen 3.6) to 42.2 with day-zero llama.cpp support and native 262k context.
-
r/LocalLLM — ‘I benchmarked every Qwen 3.8 27B quant that fits’ — https://www.reddit.com/r/LocalLLM/comments/1vr4iqj/i_benchmarked_every_qwen_38_27b_quant_that_fits/
↩ ↩2Q4_K_M retains 99.97% of original model quality at ~17.1 GB VRAM, effectively indistinguishable from Q8_0; the newer NVFP4 format underperforms standard K-quants.
-
Simon Willison Substack — ‘Qwen 3.8 27B is excellent but it defaults to overthinking’ — https://simonw.substack.com/p/qwen-38-27b-is-excellent-but-it-defaults
↩ ↩2The model consumed over 22,000 reasoning tokens to produce a simple SVG graphic; setting reasoning_effort to ‘medium’ or ‘low’ is essential for routine tasks.
-
Hacker News discussion (item 49304017) — https://news.ycombinator.com/item?id=49304017
↩The 27B model ‘punches above its weight’ via RLVR, but generates upwards of 10,000 reasoning tokens on straightforward tasks — a ‘caveman’ note-form thinking style with an internal ‘desired oververbosity 9’ parameter.
-
Hacker News thread linked from Simon Willison (49334544) — https://news.ycombinator.com/item?id=49334544
↩ ↩2 ↩3Parity on the Intelligence Index masks a ‘latency tax’ — Qwen 3.8 consumes up to 2.3x more tokens than GPT-5.6 Luna for the same task, making it slower in agentic workflows despite matching scores.
-
Simon Willison — earlier Qwen 3.8 27B writeup (Aug 16) — https://simonwillison.net/2026/Aug/16/qwen-38-27b/
↩ ↩2Independent tests suggest a regression in specialized general knowledge — the model reportedly lost a majority of medical benchmark trials to its predecessor Qwen 3.6 27B.
