Astra, encoder-free VLMs, taskless distillation each skip a specialized stage

TL;DR

GPT-6 Astra beats CV specialists 89.1 vs 78.4 on 2D detection and 73.8 vs 46.5 on 3D grounding.
Astra collapses to 17.7 dB on image restoration versus the specialist’s 30.7 dB.
Encoder-free MLLMs reach SigLIP parity at 6.1×10²¹ FLOPs compute-optimal, extrapolated from 44B-param runs.
Active Taskless Distillation lifts HumanEval+ 5.34pp using single-word teacher replies on off-topic prompts.
ATD defenses fail: chain-of-thought stripping and on-task token filters miss the single-token benign-text channel.

Today’s research beat is unified by subtraction. Three features each remove a specialized stage from a pipeline the field had treated as load-bearing — and each ships the headline that the removal works, flanked by the specific condition under which it still breaks.

GPT-6 Astra routes around trained CV specialists, matching or beating them on OCR, detection, and 3D grounding before collapsing on image restoration where pixel geometry matters. Encoder-free VLMs drop the vision encoder entirely, with Apple’s 457-model sweep backing the compute-optimal parity claim — though the 10²² crossover is extrapolated, not observed in any single run. And Active Taskless Distillation skips the on-topic prompt data that distillation defenses were built around, moving coding skill through single-word teacher replies to prompts that never mention code.

GPT-6 Astra matches CV specialists on semantics, not geometry

TL;DR

GPT-6 Astra hits 98.1% on OCR, edging the 98.0% human reference.
Astra beats CV specialists on 2D detection (89.1 vs. 78.4) and 3D grounding (73.8 vs. 46.5).
Image restoration collapses to 17.7 dB vs. the specialist’s 30.7 dB — generalists hallucinate plausible-but-wrong pixels.
Max-reasoning mode costs 13.22× more for marginal gains, versus 1.04–1.20× for tool-use pipelines.

The hard/easy inversion is now consensus

The MBZUAI team ran GPT-6 Astra against five other frontier generalists — Fable 5, Kimi K3, Gemini 3.1 Pro, Qwen 3.8-Max, Muse Spark 1.3 — across 55 benchmarks spanning nine CV areas. The headline finding is that general-purpose VLMs have crossed into specialist territory on anything semantic. Astra beats the specialist reference on 2D object detection (89.1 vs. 78.4), video understanding (74.3 vs. 71.5), and 3D visual grounding (73.8 vs. 46.5), and matches humans on 2D spatial reasoning (96.0 vs. 95.8).

Independent writeups converge on this map. cctest.ai confirms the pattern and notes that Astra “smooths over” fine structural details, producing depth maps that lack specialist precision and dense predictions that jitter across video frames ¹. A parallel GPT-4o evaluation from a year earlier found the same “hard tasks easy, easy tasks hard” paradox ² — this is a persistent architectural limit of VLM-style generalists, not an Astra-specific quirk.

Where the generalists still break

The failure modes cluster around metric fidelity. On LOL-v1 low-light restoration, Astra scored 17.7 dB against the specialist’s 30.7 dB, with the authors noting models “hallucinated” plausible details — extra pins in a bowling lane, invented textures — rather than reconstructing the actual pixels. Pathology (PUMA) sits at 23.8 vs. a human 82.0. 3D reconstruction error is 1.40 vs. the specialist’s 0.61.

Aurav.ae sharpens this with an embodied-agent failure: Astra identifies the “sit” target correctly but cannot execute the body-object conditioning, which they frame as a fidelity-sensitive perception gap distinct from reasoning weakness ³. The model knows what the chair is; it cannot localize precisely enough to use it.

The cost table is the real headline

Vector Labs’ enterprise read argues the procurement signal is buried in the compute footnotes ⁴. Tool-use pipelines — handing depth off to a specialist and letting Astra reason over the result — add 1.04–1.20× inference cost. Cranking the reasoning dial from “low” to “max” improved 3D detection by 1.36× but inflated cost by 13.22×. For robotics or anything real-time, the high-reasoning configs are economically dead on arrival.

A practitioner thread adds the comparative wrinkle: Qwen 3.8-Max reportedly matches Astra on GPQA Diamond (92.6%) at $6/M output tokens vs. $12/M, though it suffers response-parsing failures where the reasoning is right but the format breaks ⁵. Kimi K3 shows a 51% hallucination rate on binary-choice benchmarks despite MoE scale.

What the paper doesn’t measure

The strongest dissent is methodological. Prompt-sensitivity work shows template choice can swing VLM logits more than image content, and that these models “disperse” rather than cluster meaning-preserving paraphrases ⁶. The study reports single-prompt scores across all 55 benchmarks without variance under paraphrase — leaving open how much of Astra’s lead is prompt artifact rather than capability. Several suites (MMMU-Pro, MathVista, RefCOCO) also predate the model cutoffs, so leakage is a live concern.

The capability map is probably right. The ranking inside it is less settled than the tables suggest.

Encoder-free VLMs catch encoder-based ones at 10^22 FLOPs

TL;DR

Encoder-free MLLMs hit parity with SigLIP-based ones at 6.1×10²¹ FLOPs compute-optimal, 1.2×10²² under 5× overtraining — well inside flagship budgets.
Removing the vision encoder lifts the model-allocation exponent from 0.464 to 0.570 — compute-optimal recipes want bigger decoders.
Apple’s 457-model native-multimodal study reaches the same qualitative verdict, strengthening the architectural claim beyond one lab’s recipe.
The 10²² crossover is extrapolated from models up to 44B params, not observed in any single training run.

The crossover claim

Most frontier VLMs still bolt a pretrained ViT (CLIP, SigLIP) onto a language decoder. The Tencent/CASIA group ran a controlled model ladder — 11 sparse MoE decoders from 1.1B to 44B total params — and fit Chinchilla-style scaling laws for text and multimodal loss under both architectures. On text, the two are indistinguishable (exponents 0.422 vs. 0.427). On multimodal loss, encoder-free models start behind but improve faster, and the fitted curves cross at 6.1×10²¹ FLOPs compute-optimal, 1.2×10²² if you overtrain 5×. For scale: Kimi K2.5 is estimated at ~10²⁵ FLOPs. The paper’s claim is that the “encoder prior” is a small-scale crutch, not a permanent advantage.

Objective	Encoder-based $a$	Encoder-free $a$
Text loss	0.422	0.427
Multimodal loss	0.464	0.570

The jump in the multimodal allocation exponent $a$ (where $M_{opt} \propto C^a$) is the headline mechanism: without a pretrained visual prior, the decoder has to learn one, and compute-optimal recipes should spend the extra budget on parameters rather than tokens.

What the decoder learns to do

Probing results suggest the first few decoder layers quietly become an encoder. In an 8B encoder-free run, attention mass on visual tokens leaps from 0.217 to 0.645 early in training, coinciding with a sharp multimodal loss drop. Visual-token representations diverge from their input embeddings in shallower layers than they do in encoder-based runs, and MoE expert-load imbalance (MaxVio) on visual tokens is markedly higher — specific experts are specializing in vision. The crossover is topic-dependent: STEM and chart tasks catch up fast because they lean on symbolic reasoning, while GUI, OCR, and natural-image captioning lag much longer, since those are exactly where a pretrained vision encoder’s inductive bias earns its keep.

Corroboration and caveats

Apple’s parallel 457-model study of Native Multimodal Models reaches the same qualitative verdict — early-fusion architectures are “more efficient to train, and easier to deploy” than late-fusion with frozen encoders, with a similar shift in compute-optimal allocation ⁷⁸. Gemma 4 12B is the deployment existence proof: swapping the ViT for a linear patch projection reportedly yields ~50× faster vision processing on H100 and 8GB-VRAM local inference ⁹.

The empirical ledger is less flattering. EVE-7B HD (MMBench 52.3, MME 1628) beats Fuyu-8B but still trails modular SOTA like InternVL2 and Qwen2-VL ¹⁰ — no publicly-benchmarked encoder-free open model has yet been trained near 10²² FLOPs, so the crossover remains a fitted extrapolation, not an observed run ¹¹. And AAAI 2026 work identifies “Routing Distraction” in native multimodal MoEs:

Concentrated expert routing on visual tokens — which this paper celebrates — is not the same as correct routing. If reasoning experts stay dark on image inputs past the loss-crossover, the architectural win on validation loss may not translate into a win on downstream reasoning benchmarks. The scaling law says the gap closes; it does not say what closes with it.

ATD transfers coding skill via off-topic word picks

TL;DR

Active Taskless Distillation lifts HumanEval+ by 5.34pp using single-word teacher replies to prompts that never mention code.
The attack picks “near-tie” prompts where the ancestor is ~50/50 between two ordinary words, then records the teacher’s choice.
Transfer is source-specific: a coding-tuned teacher only lifts coding benchmarks, a science-tuned teacher only lifts science.
Defenses that strip chain-of-thought or filter on-task tokens don’t apply to this single-token benign-text channel.

The behavioral shadow

Post-training a model on code doesn’t just change how it writes code. It also, measurably, changes which everyday word it picks when asked to continue a sentence about jackets and ties. That residual — the “behavioral shadow” of the fine-tune — is enough to reconstruct the capability in a second model that never sees a line of the original training data.

The new paper formalizes this as Active Taskless Distillation (ATD). A student initialized from the same public ancestor as a private teacher is trained purely on the teacher’s one-token responses to task-unrelated prompts. On HumanEval+, 5,664 such single-word observations move Qwen2.5-1.5B-Instruct from 45.88% to 51.22% — a 5.34pp signal-minus-control gain that holds across LoRA ranks 16/32/64 and full fine-tuning. Analogous lifts appear on ScienceQA (+2.53pp), HellaSwag (+2.90pp), OpenBookQA (+2.07pp) and four other benchmarks.

This is the capability-transfer sequel to Anthropic’s 2025 subliminal-learning result, where a student fine-tuned on random integers from an owl-loving teacher inherited the owl preference as long as both shared a base model ¹³. LessWrong commenters argued at the time that the mechanism was a generic property of gradient updates on shared architectures ¹⁴. ATD confirms that read and escalates it: the channel carries executable skill, not just animal preferences.

How the attack works

The trick is prompt selection. Using only the public ancestor $M_0$, the authors keep prompts where $M_0$ is within 2% of 50/50 between two candidate words — so tiny weight shifts from the teacher’s private fine-tune are enough to tip the preference. Those are the only prompts the attacker sends.

flowchart LR
    A[Public ancestor M0] -->|private FT on code| B[Teacher M_T]
    A -->|identify near-tie prompts<br/>jacket vs tie| C[Carrier set]
    C -->|greedy decode| B
    B -->|single-token reply| D[1-bit observations]
    A -->|init student| E[Student]
    D -->|cross-entropy| E
    E -->|+5.34pp HumanEval+| F[Recovered skill]

Active selection matters: swapping near-tie prompts for passive ordinary prompts drops the HumanEval+ gain by 4.27pp at the same query budget. The decision boundary is where the shadow is readable. A 50/50 mixture of math and code carriers also recovers both skills simultaneously, so the signals compose.

Why current defenses miss it

Anthropic’s own distillation-defense work found that removing chain-of-thought traces was among the most effective interventions against reasoning transfer ¹⁵. ATD sidesteps that entirely — no CoT, no logits, no on-task prompts required. VentureBeat’s framing of the earlier subliminal work (“filtering bad behavior out of data may be fundamentally insufficient” ¹⁶) now extends to IP leakage.

The traffic looks benign. KYC and “Hydra Cluster” throttling ¹⁷ target attackers who query on-task; ATD’s carriers are everyday word-pair prompts indistinguishable from normal API load. The Institute for Law & AI already describes distillation as a legal gray zone where ToS violations are hard to prosecute ¹⁸; a taskless variant makes evidentiary attribution harder still, because nothing in the harvested data resembles the protected capability.

Two load-bearing caveats remain. The attack requires the exact public ancestor — an incompatible base kills transfer — and it only elicits capabilities the student can already express (memorized HumanEval answers and injected substitution ciphers both failed to transfer). That bounds the threat to the increasingly common case where frontier fine-tunes start from known Qwen, Llama, or Mistral checkpoints. Which is to say: most of them.

Round-ups

VQS cuts self-evolving VLM label errors with program checks

Self-evolving vision-language models invent their own questions, but human review found 24% of majority-vote labels and 18% of model-judge labels wrong. VQS instead generates questions whose answers a program can verify, replacing the noisy judge with executable ground truth.

WideSWE tests coding agents on cross-repository changes

Real feature work and bug fixes often span multiple codebases, yet agent benchmarks stay inside one repo. WideSWE mines 103 software ecosystems for 120 coordinated tasks, split evenly between 60 bug fixes and 60 features, to probe whether agents handle ecosystem-scale changes.

Imprint Reader decodes what a model learned from weight updates

Imprint Reader inspects parameter-level traces left by training to produce an explicit account of what a language model absorbed. The authors push beyond readout, using the decoded learning signal as a behavioral intervention that steers the model’s subsequent outputs.

Calibrated importance sampling fixes RLVR training-inference drift

RLVR pipelines sample rollouts on an inference engine but compute gradients on a training engine, and the two assign different probabilities to identical tokens. Calibrated importance sampling corrects the gap using a logit-displacement model of the mismatch, stabilizing policy updates without swapping engines.

DepthBench tests whether residual tricks add real compute depth

Transformer layers contribute less as depth grows, and fixes like LayerNorm Scaling, mHC and AttnRes claim to help. DepthBench isolates whether those gains reflect genuine computational depth or just better access to earlier representations, giving architecture researchers a cleaner yardstick.

MALA attention skips compute on near-zero score mass

Dense attention runs the full post-score path on every QK tile even when most normalized mass is negligible. MALA, a fused primitive, keeps score access to all legal causal interactions but allocates post-score compute by contribution, reusing the online-softmax normalizer in both passes.

Adaptive looped transformers lift test-time scaling slopes

Looped transformers reuse layers for latent computation, but prior work only compared them at matched parameters or per-token FLOPs. This study measures accuracy gained per doubling of decoding FLOPs and shows post-trained adaptive looping steepens that test-time scaling curve.

cctest.ai — ‘GPT-6 Astra reorders the hard problems’ — https://cctest.ai/en/articles/gpt-6-astra-reorders-the-hard-problems-in-computer-vision

Astra can capture a scene’s general layout but frequently ‘smooths over’ fine structural details, leading to depth maps that lack the precision of specialist models; dense predictions are often temporally unstable across consecutive video frames.

↩
Moonlight review — ‘How well does GPT-4o understand vision?’ — https://www.themoonlight.io/en/review/how-well-does-gpt-4o-understand-vision-evaluating-multimodal-foundation-models-on-standard-computer-vision-tasks

Models excel at ‘hard’ semantic tasks that humans find difficult yet frequently fail at ‘easy’ visual tasks such as basic geometric reasoning and spatial alignment; native image-generation models still exhibit spatial hallucinations during pixel-level edits.

↩
Aurav.ae — ‘Astra redefines the hard line’ — https://aurav.ae/ai-automation/gpt-6-astra-redefines-the-hard-line-in-computer-vision

In the Embodied Agent Arena, Astra frequently fails ‘body-object’ conditions such as completing a ‘sit’ action despite identifying the target accurately — a ‘fidelity-sensitive perception’ gap rather than a reasoning gap.

↩
Vector Labs — ‘The shrinking case for dedicated CV models’ — https://vector-labs.ai/insights/gpt-6-astra-and-the-shrinking-case-for-dedicated-computer-vision-models-in-enterprise-pipelines

The moat is shifting from raw intelligence to cost-effective inference… tool-use strategies raised inference cost only ~1.04–1.20x, while ‘higher reasoning’ inflated costs up to 13.22x for marginal accuracy gains.

↩
Reddit r/better_claw — Qwen 3.8 / Gemini 3.8 Flash thread — https://www.reddit.com/r/better_claw/comments/1w9oqy2/gemini_38_flash_and_qwen_38_both_dropped_last/

Qwen 3.8 Max rivals Astra with 92.6% on GPQA Diamond at roughly half the output cost ($6/M vs $12/M)… but suffers ‘response-parsing failures’ where reasoning is correct but final output format is malformed; Kimi K3 shows a 51% hallucination rate on binary-choice benchmarks.

↩
Medium — ‘The science of prompt sensitivity’ — https://medium.com/@sahil.pardasani/the-science-of-prompt-sensitivity-how-llms-react-to-small-variations-43ba988b8508

The specific prompt template often exerts a greater influence on the model’s logits than the visual content of the question itself… LLMs ‘disperse’ similar inputs rather than clustering them, making them inherently unstable under meaning-preserving variations.

↩
Apple Machine Learning Research — Scaling Laws for Native Multimodal Models — https://machinelearning.apple.com/research/scaling-laws-native-multimodal-models

Early-fusion architectures, which do not rely on image encoders, outperform their late-fusion counterparts at lower parameter counts, are more efficient to train, and are easier to deploy.

↩
MarkTechPost coverage of Apple’s 457-model NMM study — https://www.marktechpost.com/2025/04/14/multimodal-models-dont-need-late-fusion-apple-researchers-show-early-fusion-architectures-are-more-scalable-efficient-and-modality-agnostic/

Apple researchers trained 457 models with varying architectures and data mixtures… early-fusion models are more parameter-efficient and demonstrate different scaling behavior from late-fusion, with distinct compute-optimal allocation between parameters and tokens.

↩
daily.dev — Gemma 4 12B: The Encoder-Free Model Explained — https://daily.dev/posts/gemma-4-12b-the-encoder-free-model-explained-r5monijsn

Replacing a deep vision encoder with a lightweight linear projection can be roughly 50 times faster on an H100… enabling local execution on mid-range laptops with as little as 8GB of VRAM.

↩
arXiv:2604.08541 (EVE / encoder-free VLM benchmarking) — https://arxiv.org/abs/2604.08541

EVE-7B (HD) achieves MMBench 52.3 and MME 1628, significantly surpassing Fuyu-8B… but these models still lag behind the latest state-of-the-art modular systems like InternVL2 or Qwen2-VL in raw benchmark performance.

↩
themoonlight.io review of 2609.35457 — https://www.themoonlight.io/en/review/how-far-are-we-from-removing-the-visual-encoder-scaling-laws-for-encoder-free-multimodal-pretraining

The 10^22 FLOPs crossover remains an extrapolation and has not yet been directly demonstrated in a single training run, leaving it as a significant hypothesis rather than observed proof for frontier-scale models.

↩
AAAI 2026 — Routing Distraction in Multimodal MoE — https://ojs.aaai.org/index.php/AAAI/article/view/39405

Native multimodal models can perceive image content accurately but often fail to reason about it because the routing mechanism fails to activate reasoning experts for visual inputs — a phenomenon termed Routing Distraction.

↩
Anthropic Alignment (Cloud et al., subliminal learning) — https://alignment.anthropic.com/2025/subliminal-learning/

A ‘student’ model learns to prefer owls after training on sequences of numbers generated by a ‘teacher’ model that prefers owls… the effect only occurs when the teacher and student share the same base model.

↩
LessWrong discussion of subliminal learning — https://www.lesswrong.com/posts/cGcwQDKAKbQ68BGuR/subliminal-learning-llms-transmit-behavioral-traits-via

Subliminal learning likely occurs due to subtle statistical patterns in the teacher’s outputs rather than semantic content… it is a general phenomenon of neural network training, not something specific to LLMs.

↩
Anthropic — Detecting and preventing distillation attacks — https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks

Removing chain-of-thought reasoning traces from outputs is one of the most effective interventions, significantly impairing a student’s ability to inherit the teacher’s reasoning capabilities.

↩
VentureBeat coverage of subliminal learning risks — https://venturebeat.com/technology/subliminal-learning-anthropic-uncovers-how-ai-fine-tuning-secretly-teaches-bad-habits

Filtering bad behavior out of data may be fundamentally insufficient to prevent a model from learning bad tendencies… enterprises relying on synthetic data pipelines should be aware they could unknowingly propagate hidden biases.

↩
Didit.me — KYC in the LLM era (anti-distillation) — https://didit.me/blog/kyc-llm-era-model-access-anti-distillation/

Coordinated ‘Hydra Clusters’ of thousands of fraudulent accounts have been used to bypass rate limits and harvest the millions of exchanges needed for industrial-scale distillation.

↩
Institute for Law & AI — Responding to AI distillation without panic — https://law-ai.org/responding-to-ai-distillation-without-panic/

Distillation occupies a legal gray area… often violating Terms of Service but difficult to prosecute under existing copyright law, leaving technical countermeasures as the primary line of defense.

↩

Astra, encoder-free VLMs, taskless distillation each skip a specialized stage

TL;DR

GPT-6 Astra matches CV specialists on semantics, not geometry

TL;DR

The hard/easy inversion is now consensus

Where the generalists still break

The cost table is the real headline

What the paper doesn’t measure

Encoder-free VLMs catch encoder-based ones at 10^22 FLOPs

TL;DR

The crossover claim

What the decoder learns to do

Corroboration and caveats

ATD transfers coding skill via off-topic word picks

TL;DR

The behavioral shadow

How the attack works

Why current defenses miss it

Round-ups

VQS cuts self-evolving VLM label errors with program checks

WideSWE tests coding agents on cross-repository changes

Imprint Reader decodes what a model learned from weight updates

Calibrated importance sampling fixes RLVR training-inference drift

DepthBench tests whether residual tricks add real compute depth

MALA attention skips compute on near-zero score mass

Adaptive looped transformers lift test-time scaling slopes

Footnotes

Jack Sun, writing.