GenCeption cuts vision data 500×, Qwen two-hop hits 11%, Soofi S tops German

TL;DR

• DeepMind’s GenCeption matches vision specialists on 500× less training data, contested by V-JEPA proponents.
• Fine-tuned Qwen and Llama recall 100% of injected facts, chaining just 10.4-18.2% on two-hop questions.
• Self-patching lifts two-hop accuracy to 54.2% on Qwen-3B by relocating facts to middle layers.
• ROME critiques counter that causal tracing finds bottlenecks, not storage sites, undercutting the misalignment diagnosis.
• Soofi S 30B tops the fully-open German leaderboard by +6.3 over Apertus 70B, flagged for QA-base contamination.

Today’s three research leads share a rhythm: land a marquee number, then absorb a counter-claim from outside the paper. DeepMind’s GenCeption matches vision specialists on 500× less data, and V-JEPA proponents reject the generative premise it’s built on. A layer-misalignment paper explains why fine-tuned Qwen and Llama chain facts poorly on two-hop questions, and prior critiques of ROME-style causal tracing question whether the diagnosis is real or a tracing artifact. Munich’s Soofi S 30B tops the fully-open German+English leaderboard, and community auditors flag a QA-base slice in the disclosed training mix as possible GPQA contamination. The pattern isn’t internal caveats disclosed in the same release — it’s outside camps meeting each win with a specific, named objection.

DeepMind’s GenCeption hits vision SOTA with 500× less data

Source: hf-daily-papers · published 2026-07-09

TL;DR

• GenCeption matches specialist vision models using up to 500× less training data.
• A fine-tuned WAN 2.1 video diffusion backbone runs perception in one forward pass — no denoising loop.
• Trade press calls this vision’s “next-token” moment for pretraining.
• V-JEPA proponents reject the generative premise as wasted capacity on pixel noise.

The bet

Google DeepMind’s argument in GenCeption is that text-to-video generation is to computer vision what next-token prediction was to NLP: a pretraining objective rich enough to force the model to internalize 3D geometry, object permanence, and vision-language alignment as a side effect. The Decoder frames it as vindication that “video generators already contain the world models computer vision has been missing” ¹. The concrete evidence: a single unified model, prompted with text, that produces state-of-the-art depth (AbsRel 0.048 on KITTI), surface normals (mAE 29.3 on Sintel), camera pose (ATE 0.050 on Sintel), and referring segmentation (70.0 J&F on MeViS) — using synthetic data an order of magnitude smaller than what specialists like DepthAnything 3 or SAM 3 train on.

How the trick works

GenCeption starts from WAN 2.1, an off-the-shelf text-to-video diffusion transformer, and re-purposes it as a feed-forward perception engine rather than an iterative generator.

flowchart LR
    V[Input video] --> E[VAE encoder]
    T[Text prompt<br/>'depth' / 'normals' / 'mask'] --> D
    E --> D[DiT backbone<br/>t=0, velocity negated]
    D -->|dense tasks| R[3-channel RGB<br/>depth, normals, seg, raymap]
    D -->|sparse tasks| K[Learnable tokens → MLP<br/>2D/3D keypoints]

Two design moves do most of the work. First, since the backbone was trained with Rectified Flow to predict velocity v = ε - x₀, negating the output at t=0 aligns it with the target latent in a single deterministic step — no denoising loop. Second, everything is coerced into either a 3-channel RGB “video” (depth maps, normal maps, even 6D camera raymaps encoded as “Rothko” images) or a set of appended learnable tokens for sparse coordinates. One L2 loss, no task-specific heads.

The payoff is data efficiency, not just accuracy. GenCeption-L hits average AbsRel 0.071 across depth benchmarks using only 7,500 synthetic RenderPeople videos, and generalizes zero-shot from synthetic humans to real animals and robots.

Where the claims wobble

Two caveats the coverage isn’t flagging. The generation-as-perception idea is not new: Marigold showed in 2024 that fine-tuning Stable Diffusion for depth works precisely because a good image generator has to encode 3D structure ². GenCeption’s real novelty is scaling that intuition to video and collapsing it to one step — meaningful, but not a paradigm birth.

The benchmark story is also regime-dependent. Independent leaderboards put UniDepthV2 ahead on KITTI at AbsRel 0.055 under a metric (not affine-invariant) evaluation ³, and the two numbers aren’t directly comparable. None of the trade coverage says so.

The sharper philosophical dissent comes from the V-JEPA camp. XenoSpectrum’s read is that GenCeption is “a representation learner rather than a predictive physics engine” ⁴ — perceptually rich, but not evidence of causal understanding.

Critics, including Yann LeCun, argue this ‘generative trap’ wastes massive computational resources on high-entropy noise — such as the exact movement of individual leaves in the wind — which is irrelevant to high-level understanding. ⁵

What’s actually settled

The data-efficiency result is the durable finding, and it’s a real one: a pre-trained video generator plus a modest synthetic corpus can go toe-to-toe with specialist pipelines trained on orders of magnitude more labeled data. The “world model” framing is contested, the SOTA leadership is regime-dependent, and the 14B model runs at ~8 FPS on TPU v6e for 81 frames ⁶ — too slow for the robotics use cases that would most benefit from a true video-native world model. Worth watching whether anyone can reproduce the data-efficiency curve on an open backbone.

---

Fine-tuned Qwen, Llama hit 100% recall but 11% reasoning

Source: hf-daily-papers · published 2026-07-09

TL;DR

• Fine-tuned Qwen-2.5 and Llama-3 hit near-100% recall on injected facts.
• The same facts score just 10.4-18.2% when chained into two-hop questions.
• The authors blame layer misalignment: SFT stores facts in early or late layers, away from middle reasoning circuits.
• Self-patching relocates hidden states to middle layers, lifting two-hop accuracy to 54.2% on Qwen-3B and 45.8% on Llama-8B.
• Prior ROME critiques find causal tracing identifies bottlenecks, not storage sites, so “misalignment” may be a tracing artifact.

The Knowing–Using Gap, quantified

The paper formalizes a familiar frustration into two numbers. Across models from 1B to 8B parameters fine-tuned on new biomedical (STaRK-Prime) and academic (STaRK-MAG) facts, single-hop recall saturates near 100% while multi-hop “Chaining” accuracy stalls between 10.4% and 18.2%. A temporal lag of 4-9 epochs separates memorization from any generalization at all — and in many runs generalization never emerges. Novelty is controlled: base models scored under 6% on these facts before training.

None of this is unprecedented. Gekhman et al. showed last year that fine-tuning is a poor knowledge-injection channel and that forced acquisition “linearly increases hallucination” ⁷. Allen-Zhu & Li’s Physics of Language Models 3.1 went further, demonstrating that models can hit zero training loss on facts and still fail 100% of extraction queries when pre-training paraphrase diversity is missing ⁸. The Mem2Gen paper’s contribution is not the phenomenon but the mechanistic story it tells about why.

A routing problem, not a knowledge problem

The core method is self-patching: for a failed two-hop query, extract the anchor entity’s hidden state at some source layer $l_s$ and inject it at target layer $l_t$ on a fresh forward pass, sweeping all pairs. Where patching fixes the answer, the knowledge existed — it just wasn’t being routed to a layer that could compute with it.

flowchart LR
    A[Anchor entity<br/>at source layer l_s] -->|extract hidden state| B[Fresh forward pass]
    B -->|inject at target l_t| C[Target middle layer]
    C --> D{Answer correct?}
    D -->|Yes| E[Knowledge existed,<br/>was mis-routed]
    D -->|No| F[Knowledge absent<br/>or unusable]

Tracking these “permeation maps” across training epochs, the authors observe that successful generalization corresponds to patch-effective regions expanding until they cover the diagonal (the model has learned to route facts to itself). Failed cases halt short of the diagonal, with gradients vanishing once memorization is satisfied. Facts get stranded.

Turning that diagnostic into a fix, a fixed non-oracle heuristic — patching along Late→Middle and Early→Middle pairs chosen per architecture — recovers 58-75% of the oracle-patching gain. On simpler intersection tasks, patching moves accuracy from ~79% to ~98%, nearly closing the gap. Notably, scaling hurts: injecting more facts widens the gap, consistent with the routing-capacity framing.

Where the localization claim wobbles

The remedy inherits ROME/MEMIT’s premise that factual computation localizes cleanly to specific layers, and that premise is contested. Hase et al.’s reinvestigation of ROME argued that causal tracing identifies “bottlenecks or retrieval paths rather than the actual storage site,” and that editing traced layers isn’t always the most effective update site ⁹. If self-patching’s Late→Middle pattern is picking up a computational bottleneck rather than a real storage location, the heuristic may generalize worse than the numbers suggest.

Two adjacent literatures raise further doubt. Scaled model editing work shows ROME collapses base capabilities under ~100 edits and MEMIT under ~1,000 ¹⁰ — an open question for any deployment that would patch thousands of facts. And practitioners have reported grokking-like delayed generalization in ordinary fine-tuning ¹¹, offering an alternative reading of the 4-9 epoch lag: a phase transition between circuits rather than a spatial routing wait. An independent review also flagged that the 58-75% “oracle headroom” is measured against a self-defined ceiling with no plug-in implementation for standard open-weight stacks ¹².

What’s actually useful here

Read as a deployable fix, self-patching is not that — it needs per-architecture layer choices and inference-time bookkeeping. Read as a training-time probe, it’s genuinely useful: permeation maps give researchers a visible signal for whether an SFT run is depositing knowledge where reasoning can reach it, well before evaluation catches the failure. That alone is worth the paper.

---

Soofi S beats Apertus on German benchmarks, trails it on openness

Source: hf-daily-papers · published 2026-07-09

TL;DR

• Soofi S 30B-A3B tops the fully-open German+English leaderboard: +6.3 over Apertus 70B, +2.8 over Olmo 3 32B.
• The 8–9× throughput headline is inherited from NVIDIA’s Nemotron 3 Nano stack, not a Soofi contribution.
• Community auditors flag QA-base in the disclosed training mix as possible contamination of GPQA-style evals.
• Sovereignty pitch wins on Munich’s 10,000-GPU infra, loses on the gated release with an unfinished license.

The benchmark lead is real, with an asterisk

A German consortium has shipped Soofi S 30B-A3B, a hybrid Mamba-2 / MoE / Transformer model that beats every other “fully open” release on both German and English aggregates — +2.8 points over Olmo 3 32B in English and +6.3 over Apertus 70B in German ¹³. The model activates 3.2B of 31.6B parameters per token, was pretrained on 26.68T tokens with German upsampled to 15.3% during annealing, and claims a 1M-token context window.

The lead is genuine but comes with community caveats. LocalLLaMA-adjacent auditors noticed that the released Hugging Face artifact is gated, the model card still contains TODO placeholders for license and context length, and the disclosed training mix includes AIML-TUDA/QA-base — paraphrased NLP benchmarks that plausibly leak into GPQA-style evaluations ¹⁴. The Soofi team’s own honesty helps and hurts: they concede a narrower lead on the contamination-aware LiveCodeBench, which is both a credibility signal and a hint the aggregates flatter the model.

The architecture is borrowed

The headline throughput number — 8–9× the decode TPS of dense 14–24B models at 40K context, essentially flat from 4K to 256K — is impressive, but it isn’t Soofi’s engineering. The backbone is a direct reuse of NVIDIA’s Nemotron 3 Nano: 23 Mamba-2 layers, 23 granular MoE layers (128 experts, top-6 + 2 shared), and 6 GQA layers sparsely interleaved to bound the KV cache.

flowchart TB
    subgraph Stack["Nemotron 3 Nano backbone (reused by Soofi S)"]
        M[23× Mamba-2 layers<br/>fixed recurrent state]
        E[23× Granular MoE layers<br/>128 experts, top-6 + 2 shared]
        A[6× GQA attention layers<br/>bounded KV cache]
    end
    Stack --> T[8–9× decode throughput<br/>vs. dense 14–24B at 40K ctx]
    D[German-weighted 27T-token corpus<br/>+ WSD schedule + open checkpoints] --> S[Soofi S contribution]
    Stack --> S

NVIDIA already benchmarks that reference stack at ~3.3× the throughput of Qwen3-30B-A3B on a single H200 ¹⁵. Soofi’s contribution is the German data mix, the WSD training recipe, and the radical-openness posture around checkpoints and logs — not the transformer-Mamba hybrid itself.

That reframing matters. Analyst coverage pushed back on the 27T-token / 30B-param ratio as Chinchilla-violating overtraining; lead author Michael Fromm counters that MoE-Mamba hybrids with 3B active params absorb repetition differently than dense models ¹⁶. Testable, but so far untested by third parties.

Sovereignty means infrastructure — not openness

The “trained on German soil” framing checks out. The Munich Industrial AI Cloud that consumed 253,000 B200 GPU-hours houses ~10,000 Blackwell GPUs and ~1,000 DGX B200 systems, delivers ~0.5 ExaFLOPS, and runs on Eisbach-canal cooling and 100% renewables ¹⁷. As a European compute-sovereignty milestone, that is real.

The openness framing is shakier. Apertus — the Swiss model Soofi outscores — ships Apache 2.0 with fully open weights and training data across 1,800+ languages ¹⁸. Soofi’s preview is gated, has an unfinished custom license, and cannot redistribute 1.3% of its Phase 1 corpus (the commercially licensed Genios press archives). Soofi wins the benchmark comparison against Apertus and loses the openness comparison to the same model.

What’s actually new here

Strip the layers back and Soofi S is a well-executed German-language investment on top of NVIDIA’s reference architecture, deployed on genuinely sovereign infrastructure, with an openness posture stronger than most weights-only releases but weaker than the Swiss competitor it beats on benchmarks. That’s a useful contribution to European AI capacity. It’s not the architectural or openness breakthrough the paper’s framing implies.

Round-ups

Long-Horizon-Terminal-Bench grades agents on 46 multi-hour shell tasks

Source: hf-daily-papers

Long-Horizon-Terminal-Bench pushes terminal agents past short, outcome-only tests with 46 tasks across nine categories, from system administration to data engineering. Dense reward-based grading credits partial progress instead of pass/fail, exposing where agents stall on multi-step workflows that current benchmarks miss entirely.

Self-guided test-time training sharpens long-context LLM recall

Source: hf-daily-papers

Long-context accuracy degrades as inputs grow because models fail to locate relevant evidence. Self-Guided Test-Time Training treats the prompt itself as a training example, fine-tuning instance-specific parameters at inference so the model adapts to the passage it must actually answer from.

TOP-D stabilizes on-policy distillation with a proximal teacher

Source: hf-daily-papers

Trust Region Policy Distillation reworks on-policy distillation by dynamically constructing a nearby teacher, taming the gradient variance that has made OPD unstable. The authors prove monotonic improvement and global convergence, giving distillation the trust-region guarantees that PPO brought to policy optimization.

Visual pretraining reclaims figures and layouts text tokenizers discard

Source: hf-daily-papers

Scalable Visual Pretraining for Language Intelligence argues that converting documents and web pages to plain text throws away equations, figures and page structure that carry real knowledge. The paper pretrains language models directly on visual renderings of these sources to preserve signals lost by tokenization.

FLUX diffusion transformer repurposed for depth, segmentation and matting

Source: hf-daily-papers

Instead of decoding tokens into RGB pixels, this work maps a pretrained DiT’s outputs directly to task-native fields via a token-local linear head and task LoRAs. FLUX-Klein hits state-of-the-art on KITTI depth, trimap-free matting, normals, saliency, pose and referring segmentation with minimal added parameters.

PanoWorld exploits panorama rotation symmetry for long-range memory

Source: hf-daily-papers

PanoWorld reframes camera motion in omnidirectional video as translations under fixed headings, using rotation equivariance to keep long-range memory consistent. Dense Panoramic Ray-Conditioning and a geometry-aware memory bank let the world model recall distant scene structure that flat-camera generators lose.

KronQ adds gradient covariance to LLM post-training quantization

Source: hf-daily-papers

GPTQ and its heirs assume every output channel contributes equally to layer reconstruction, relying only on input activation statistics. KronQ folds a Kronecker-factored gradient covariance into the Hessian, weighting channels by their true impact on loss and improving low-bit quantization quality without retraining.

Footnotes

1. The Decoder — https://the-decoder.com/google-deepmind-argues-video-generators-already-contain-the-world-models-computer-vision-has-been-missing/

   Google DeepMind argues video generators already contain the world models computer vision has been missing

   ↩

2. Marigold paper (arXiv 2402.14973) — https://arxiv.org/abs/2402.14973

   fine-tuning Stable Diffusion for monocular depth estimation, arguing that a model capable of generating a realistic ‘photo’ must inherently understand spatial relationships

   ↩

3. Roboflow blog on DepthAnything 3 — https://blog.roboflow.com/depth-anything-3/

   UniDepthV2 as the leader on KITTI with an AbsRel of 0.055 (5.53%), outperforming DA3 in raw in-domain metric accuracy

   ↩

4. XenoSpectrum — https://xenospectrum.com/en/genception-video-generation-world-model/

   it remains a ‘representation learner’ rather than a predictive physics engine

   ↩

5. ScriptsHub (V-JEPA analysis) — https://scriptshub.net/resources/blogs/v-jepa-world-models-predictive-learning-for-physical-ai/

   Critics, including Yann LeCun, argue this ‘generative trap’ wastes massive computational resources on high-entropy noise—such as the exact movement of individual leaves in the wind—which is irrelevant to high-level understanding

   ↩

6. AIbase — https://www.aibase.com/news/29703

   the 14-billion-parameter version of GenCeption takes approximately ten seconds to process just 81 frames, limiting its utility for real-time applications like robotics or AR

   ↩

7. Gekhman et al., EMNLP 2024 — ‘Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?’ — https://arxiv.org/abs/2405.05904

   LLMs struggle to acquire new factual knowledge through fine-tuning… as the model eventually learns new knowledge, it linearly increases its tendency to hallucinate.

   ↩

8. Allen-Zhu & Li, ‘Physics of Language Models 3.1: Knowledge Storage and Extraction’ (arXiv:2309.14316) — https://arxiv.org/abs/2309.14316

   Models can achieve near-zero training loss on facts yet fail 100% of the time to answer questions about those facts if the training data lacked diversity; instruction fine-tuning cannot recover knowledge that was memorized but not properly augmented during pre-training.

   ↩

9. LessWrong — Hase et al. critique, ‘But is it really in Rome? An investigation of the ROME model editing technique’ — https://www.lesswrong.com/posts/QL7J9wmS6W2fWpofd/but-is-it-really-in-rome-an-investigation-of-the-rome-model

   Causal tracing identifies computational bottlenecks or retrieval paths rather than the actual storage site of the data — editing the layers tracing points to is not always the most effective way to update knowledge.

   ↩

10. Gupta et al., ‘Model Editing at Scale leads to Gradual and Catastrophic Forgetting’ (scalable-model-editing.github.io) — https://scalable-model-editing.github.io/catastrophic/

    ROME and MEMIT suffer from two phases of degradation: gradual forgetting followed by catastrophic forgetting, with MEMIT stable up to ~1,000 edits and ROME collapsing under 100.

    ↩

11. r/LocalLLaMA thread — ‘Has anyone seen grokking during LLM finetuning?’ — https://www.reddit.com/r/LocalLLaMA/comments/1r1760x/has_anyone_seen_grokking_during_llm_finetuning/

    A model might appear to have reached a plateau in performance while fine-tuning on a specific logic or reasoning task, only to ‘grok’ the underlying principle much later — early stopping may prevent models from reaching full reasoning potential.

    ↩

12. TheMoonlight.io review of the Mem2Gen paper — https://www.themoonlight.io/en/review/towards-mechanistically-understanding-why-memorized-knowledge-fails-to-generalize-in-large-language-model-finetuning

    The ‘oracle headroom’ metric of which their heuristic recovers 58–75% is a self-defined ceiling that may not reflect real-world performance across diverse architectures, and there is no plug-in fix for popular open-weight models.

    ↩

13. The Decoder — https://the-decoder.com/german-ai-consortium-releases-soofi-s-an-open-30b-model-that-tops-benchmarks-in-both-english-and-german/

    Soofi S achieved the highest English and German aggregate scores among fully open models… +2.8 points over Olmo 3 32B and +6.3 points over Apertus 70B

    ↩

14. Smol AI newsletter (LocalLLaMA roundup) — https://news.smol.ai/issues/26-07-15-thinky-inkling/

    gated on Hugging Face, model card with ‘TODO’ license and context length, and allegations the training mix (notably AIML-TUDA/QA-base) may have leaked into evaluation sets

    ↩

15. NVIDIA developer blog (Nemotron 3 Super) — https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/

    Nemotron-3 Nano achieves up to 3.3x higher throughput than Qwen3-30B-A3B on single-H200 serving with a 23 Mamba-2 / 23 MoE / 6 GQA layer stack

    ↩

16. connic.co analyst brief — https://connic.co/blog/soofi-s-german-open-model

    27 trillion tokens for a 30B model is heavily overtrained by Chinchilla standards; lead Michael Fromm counters that hybrid Mamba-MoE stacks benefit differently from massive data repetition than dense models do

    ↩

17. T-Systems press release (Industrial AI Cloud Munich) — https://www.t-systems.com/dk/en/insights/newsroom/news/germanys-first-ai-factory-for-industry-goes-into-operation-1138922

    the Munich site houses ~10,000 Blackwell GPUs including ~1,000 DGX B200 systems, delivers ~0.5 ExaFLOPS, and is cooled with Eisbach canal water on 100% renewable energy

    ↩

18. effektiv.ch (Apertus release analysis) — https://www.effektiv.ch/en/blog/apertus-release

    Apertus is Apache 2.0 with fully open weights and training data trained on 1,800+ languages on the Swiss ‘Alps’ supercomputer — a stricter reading of ‘open’ than Soofi’s gated, custom-license preview

    ↩