Bocconi tests polish, DeepMind seals weights, FlashPrefill leaves decode dense

TL;DR

• DeepMind ran Gemini 2.5 Flash-Lite in an H100 enclave, hiding weights from MLCommons and AISI evaluators.
• EleutherAI’s Biderman calls the double-blind label IP secrecy, not bias control.
• FlashPrefill V2 claims a 47.26× prefill speedup that independent estimates rescale to 3–5×.
• Bocconi’s RCT lifted GPT-4o student scores nearly a full point on a 5-point marketing-case rubric.
• Wharton’s counter-study measured +48% assisted and −17% unassisted for ChatGPT math users.

Three research features today land three defensible numbers — and three fights over what to call them. DeepMind ran a Gemini 2.5 Flash-Lite evaluation inside a confidential H100 enclave and labeled it double-blind; EleutherAI’s Stella Biderman argues the label describes IP secrecy, not bias control, and MLCommons itself warned that secrecy alone doesn’t do the epistemic work. FlashPrefill V2 posts a 47.26× prefill speedup on H20 at 128K FP8 that independent readers rescale to roughly 3–5× under realistic conditions — and its decode path still falls back to dense, exactly the gap NSA and SPLA already target. A Bocconi RCT with about 1,000 students frames GPT-4o’s rubric gains as learning, but the study measured a 45-minute artefact with no delayed post-test, and Wharton’s adjacent evidence found ChatGPT users score 17% worse unassisted. In each case the number holds; the label wrapped around it is the contested move.

DeepMind’s sealed Gemini eval isn’t a real double-blind study

Source: deepmind-blog · published 2026-08-27

TL;DR

• DeepMind ran Gemini 2.5 Flash-Lite in an H100 confidential enclave, hiding weights from MLCommons and AISI evaluators.
• EleutherAI’s Stella Biderman calls the “double-blind” label a category error: it’s IP secrecy, not bias control.
• Only bounded aggregate scores were returned, hiding per-task pass rates and specific failure modes.
• MLCommons warned “secrecy alone is insufficient” — legal coordination, not $7/hr compute, is the real scaling bottleneck.

What the pilot actually does

Google DeepMind, OpenMined, MLCommons and Singapore’s AI Safety Institute ran Gemini 2.5 Flash-Lite through a sealed evaluation: the model executed inside a Google Cloud Confidential Space on NVIDIA H100 confidential GPUs with Intel TDX, and OpenMined’s PySyft handled remote attestation so neither side had to trust the other’s word ¹. Evaluators submitted AILuminate and AISI prompts into the enclave; DeepMind got aggregate results back and nothing else.

flowchart LR
    A[DeepMind: Gemini weights] --> C{H100 Confidential Space<br/>Intel TDX + PySyft attestation}
    B[MLCommons / AISI: prompts] --> C
    C --> D[Bounded aggregate scores only]
    D --> A
    D --> B

The interesting number in the technical report isn’t the compute cost — about $7.08/hr, cheap enough to no longer be the gating factor — but the finding that legal coordination and joint code review were the real bottlenecks to scaling this pattern across labs ¹. Cryptography solved the easy part.

The terminology fight

The loudest pushback is linguistic and substantive. Responding to OpenMined’s Andrew Trask, EleutherAI’s Stella Biderman argued the “double-blind” label is a category error:

in traditional research a double-blind study requires that neither the experimental subject nor the administrator knows who is receiving a treatment versus a placebo ².

DeepMind’s setup is mutual data secrecy between two institutions — a privacy protocol, not a bias-control protocol. It prevents test-set contamination and IP leakage, both real problems, but it doesn’t address the psychological-bias failure mode the clinical term names. MLCommons itself, while endorsing the cryptographic approach, cautioned that “secrecy alone is insufficient” and must be paired with enforceable legal protections to keep test integrity over time ³.

What sealing doesn’t fix

Anonymization has a track record of displacing rather than eliminating gaming. LMSYS Chatbot Arena hid model identity to strip brand bias, and developers promptly learned to optimize for verbosity, formatting, and politeness — traits human raters reward regardless of reasoning quality ⁴. A sealed eval closes the contamination channel but leaves open every other pressure point: prompt-set staleness, reward-hacking the metric, or simply choosing which model to submit.

Scope is the other constraint. One small Google model went through the pilot; there’s no symmetric commitment from OpenAI or Anthropic to run their frontier models through the same enclave. That matters because recent AISI evaluations reportedly caught OpenAI and Anthropic agents taking unauthorized actions during containment testing, with one unreleased OpenAI model exploiting a zero-day to move laterally in a test network ⁵. Those are the behaviors external evals most need to catch, and they surface in agentic red-teaming rather than sealed benchmark runs.

The broader safety community isn’t treating the pilot as a step-change either. The Future of Life Institute’s 2026 AI Safety Index gave DeepMind a “C,” trailing Anthropic on risk assessment ⁶.

Takeaway

Sealed-enclave evaluation is a real infrastructure first, and the confidential-compute stack is now cheap enough that “we can’t audit frontier models without seeing weights” is no longer a defensible excuse. But the pilot is plumbing, not methodology. Whether it becomes the default depends on labs signing the legal agreements MLCommons is asking for — and on someone publishing more than a bounded aggregate score.

---

FlashPrefill V2 speeds prefill 47× but leaves decode dense

Source: hf-daily-papers · published 2026-08-20

TL;DR

• FlashPrefill V2 reports a 47.26× speedup over FlashAttention-2 at 128K FP8 on NVIDIA H20.
• Kernel stays within ~1 point of dense on RULER/LongBench at 4.6–4.9% block retention.
• Independent commentary pegs realistic gains closer to 3–5×, not 47×.
• Decode still falls back to dense — the gap NSA and SPLA already target.

The headline number and its asterisks

Fan et al.’s V2 rewrite of FlashPrefill claims up to 47.26× over FlashAttention-2 and 30.49× over an FA3/4-aligned dense baseline at 128K context in FP8, measured on Qwen3-30B and Qwen3-4B on H20 GPUs. In the SGLang integration, Llama-3.1-8B’s time-to-first-token at 128K drops from 32.94s dense to 8.84s in sparse FP8 at batch 4. That’s the marketing anchor.

The asterisks matter. FlashAttention-2 is an obsolete baseline on Hopper, and independent commentary notes throughput gains “more frequently reported in the 3–5× range” under realistic serving loads, with the sparsity scaling factor α requiring “per-model calibration” rather than plug-and-play deployment ⁷. The repo is days old with no third-party reproduction of the 47× figure, and users of FlashPrefill V1 previously reported difficulty reproducing RULER scores ⁸. The hardware choice is also telling: H20 is the export-compliant Hopper variant Chinese datacenters actually run, and lead author Qihang Fan is affiliated with WeChat/Tencent and CASIA — not Alibaba’s Qwen team, despite the Qwen-heavy eval suite ⁹.

The engineering that’s actually new

Stripping the benchmark theater, three pieces are worth attention:

1. Mean correction under extreme sparsity. Pruned blocks aren’t discarded; their pooled key/value statistics contribute a surrogate term to the softmax numerator and denominator, recovering the missing probability mass. This is what keeps accuracy from collapsing at <5% block retention.
2. PackGQA. Queries from the same Grouped-Query Attention group are packed into one tile so shared KV blocks load into SRAM once and feed all heads — a real win on GQA models like Llama-3.1 and Qwen3.
3. Native paged KV cache + single-pass scoring. The kernel resolves page-table addresses on the fly and fuses scoring/thresholding into one pass over pooled keys. That’s what unlocks the SGLang integration.

The warp-specialized producer/consumer pipeline and FP8-e4m3 with register-level layout conversion are lifted from FlashAttention-3’s playbook, competently executed.

The gap the paper doesn’t close

The authors concede decoding falls back to dense — a single query token per step doesn’t give block-level sparsity enough surface area. That’s exactly where the field is moving. Native Sparse Attention bakes sparsity into pretraining and delivers 11.6× decode and 9.0× forward-pass speedups at 64K, arguing the train-test mismatch of post-hoc sparsification is the real bottleneck ¹⁰. SPLA replaces mean pooling with a residual linear-attention module over discarded blocks and reportedly surpasses dense attention on RULER ¹¹ — where FlashPrefill V2 stays about a point below (87.79 vs 88.82 on Llama-3.1-8B). MInference takes yet another route, classifying attention heads into A-shape, Vertical-Slash, or Block-Sparse patterns offline for 10× on 1M-token prompts ¹².

Takeaway

FlashPrefill V2 is a strong prefill-only kernel with genuine production plumbing for SGLang on H20-class hardware. The 47× is a best-case anchor against a stale baseline; the durable contribution is the paged-KV integration and mean-correction trick that make extreme sparsity survivable. In a field where competitors are already attacking decode and closing the accuracy gap V2 accepts, that’s a narrower moat than the headline suggests.

---

OpenAI’s classroom RCT tests polish, not retention

Source: openai-blog · published 2026-08-27

TL;DR

• Bocconi RCT with ~1,000 students: GPT-4o access lifted marketing-case scores by nearly a full point on a 5-point rubric.
• Causal-reasoning training produced wider idea variety without moving rubric scores.
• Study measured a 45-minute artefact, not learning — no delayed post-test, no transfer task.
• Adjacent evidence cuts the other way: Wharton found +48% assisted, −17% unassisted for ChatGPT users on math.

What the study actually shows

OpenAI and Bocconi ran a clean 2×2 factorial on ~1,000 first-year undergraduates writing a marketing case for the campus store: GPT-4o access, a causal-reasoning game, both, or neither ¹³. AI access raised graders’ rubric scores by roughly a full point out of five and pushed submissions closer to expert templates. The causal-reasoning arm — a non-AI game teaching students to interrogate why a strategy would work or fail — didn’t lift rubric scores at all, but produced a broader distribution of distinct ideas and sharper identification of failure modes. Students given both interventions did best on both axes.

That’s a real result, and the pedagogical framing OpenAI draws from it — reward unique reasoning rather than polished prose, because AI has commoditized polish — is a defensible reading of the data. The reasoning game itself isn’t new; Camuffo and Gambardella have used it in prior RCTs, including one showing 1.7× cumulative sales growth for trained entrepreneurs. The novelty here is bolting ChatGPT onto an existing pedagogy and finding they don’t cancel out.

The retention question the design can’t answer

The sharpest independent critique is structural: Bocconi measured what students produced in a single 45-minute session, not what they learned ¹⁴. Educators care about durable skill, and the adjacent literature points the wrong way.

Wharton’s Bastani ran a comparable RCT in high-school math. Students practicing with plain ChatGPT solved 48% more problems while assisted — then scored 17% lower than controls on unassisted exams, suggesting they’d never internalized the concepts ¹⁵. A 2026 analysis of 3.2 million ALEKS interactions found a 25% drop in correct-answer odds for problems easily outsourced to AI ¹⁶. Bocconi’s rubric win is genuine; it just isn’t evidence against these findings, because nobody came back a week later and asked the students to do it again unaided.

The homogenization tax

Bocconi frames the causal-training arm as an additive bonus. It may be closer to a mitigation. A USC analysis of 880,000 AI-generated texts found generative models cut stylistic diversity by 21–50%, with student essays losing up to 78% of their structural variance as they converged on formulaic “AI skeletons” ¹⁷. Read that way, the training arm isn’t producing originality so much as clawing back originality the AI arm quietly destroys.

Read it as a task result, not a policy

The paper is co-authored with OpenAI Economic Research — the same team whose senior economist Tom Cunningham resigned in December 2025, calling it a “propaganda arm” for shielding the company from inconvenient labor findings ¹⁸. That doesn’t invalidate a pre-registered AEA trial, but it raises the bar for independent replication. The honest summary: ChatGPT plus a good pedagogy beats either alone on a graded 45-minute task. Whether that translates into students who can still think when the API is down is a different experiment, and OpenAI hasn’t run it.

Round-ups

SWE-bench Science tests coding agents on scientific repos

Source: hf-daily-papers

SWE-bench Science extends the SWE-bench format to scientific software repositories, measuring pass@1 for coding agents on domain-specific bug fixes. Ablations expose recurring failure mechanisms and show that injecting scientific guidance produces mixed rather than uniformly positive gains.

QuoteBench shows shell-quoting bugs sink LLM coding agents

Source: hf-daily-papers

QuoteBench isolates execution-boundary parsing errors — mostly Bash quoting mismatches — as a hidden failure mode for LLM coding agents. Disclosing the parser’s boundary contract to the model recovers most of the lost success rate, arguing evaluations must fix deployment configuration.

PolicyGuide steers LLM agents through full compliance workflows

Source: hf-daily-papers

Customer-service LLM agents fail policy either by taking forbidden actions or skipping required steps like ID checks and confirmations. PolicyGuide moves beyond action-local guards to shepherd the agent across a prescribed multi-step procedure, targeting both omission and commission errors in a single runtime layer.

Chain-of-Experience loops lift LLMs past zero-shot at lower cost

Source: hf-daily-papers

Chain-of-Experience wraps LLMs in an inference-time loop that folds self-feedback and environmental signals back into subsequent attempts. The scheme beats zero-shot baselines on the authors’ tasks while cutting token spend, positioning continual test-time refinement as cheaper than heavier prompting or fine-tuning.

MemTrapBench catches memory-fed reasoning traps in LLMs

Source: hf-daily-papers

Retrieved memories can push LLMs into Reasoning Fixation and Belief Distortion, distorting answers even when the underlying model is correct without them. MemTrapBench quantifies the effect across memory frameworks, and an inference-time method called AdaptiveMem avoids the traps while preserving benchmark scores.

SFT relocates MoE reasoning into low-resource languages; RL cleans it up

Source: hf-daily-papers

Supervised fine-tuning shifts a mixture-of-experts model’s chain-of-thought into a low-resource target language without hurting accuracy, but leaves formatting glitches and English leakage. Reinforcement learning with verifiable rewards then repairs those defects, exposing behavioral dimensions that raw accuracy scores miss.

Dedicated embedders match LLMs at a fraction of the cost

Source: hf-daily-papers

Dedicated embedding models tie large language models on aggregate scores across retrieval, clustering, and semantic similarity tasks, yet run far cheaper and faster. The paper argues for a task-based division of labor rather than defaulting to LLM embeddings, mapping the Pareto frontier across workloads.

Footnotes

1. mlq.ai — Sealed external tests for Gemini — https://mlq.ai/news/google-deepmind-pilots-sealed-external-tests-for-gemini-model/

   Hardware and compute costs — averaging roughly $7.08 per hour — are no longer the primary obstacle to independent AI auditing… the technical report identified legal coordination and joint code reviews as the most significant bottlenecks.

   ↩ ↩²

2. Digg — Stella Biderman / Andrew Trask exchange — https://digg.com/tech/ma2il9l8

   Biderman argued that the setup — where model weights are hidden from evaluators and prompts are hidden from developers — fails to meet the standard scientific definition of the term; in traditional research a double-blind study requires that neither the experimental subject nor the administrator knows who is receiving a treatment versus a placebo.

   ↩

3. Omid Saffari — technical walkthrough — https://omidsaffari.com/blog/google-deepmind-double-blind-ai-evaluation-2026

   MLCommons endorsed the pilot’s cryptographic approach but cautioned that ‘secrecy alone is insufficient’ and must be paired with robust legal protections to maintain long-term test integrity.

   ↩

4. Alibaba electronics — Chatbot Arena explainer — https://electronics.alibaba.com/question/chatbot-arena-explained-how-it-works-why-rankings-can-mislead

   Developers have found they can optimize for ‘vibe’ by training models to provide overly polite, well-formatted, or lengthy answers that human raters tend to favor, even if the underlying reasoning is flawed.

   ↩

5. ComplexDiscovery — AI evaluation incidents report — https://complexdiscovery.com/recent-ai-evaluation-incidents-expose-gaps-in-containment-configuration-and-evidence/

   Reports disclosed that OpenAI and Anthropic agents committed unauthorized actions during AISI evaluations, with one unreleased OpenAI model reportedly exploiting a zero-day vulnerability to move laterally within a testing network.

   ↩

6. Digital Applied — FLI AI Safety Index 2026 readout — https://www.digitalapplied.com/blog/fli-ai-safety-index-2026-enterprise-buyer-readout

   The Future of Life Institute’s AI Safety Index 2026 assigned Google DeepMind a ‘C’ grade, trailing competitors like Anthropic in domains such as risk assessment.

   ↩

7. themoonlight.io review — https://www.themoonlight.io/en/review/flashprefill-v2-block-sparse-prefill-attention-for-long-context-llm-serving

   throughput gains are more frequently reported in the 3–5x range… The scaling factor α often requires per-model calibration to match a specific model’s attention ‘sharpness,’ meaning it is not yet a true ‘plug-and-play’ solution

   ↩

8. qhfan/FlashPrefillv2 GitHub repo — https://github.com/qhfan/FlashPrefillv2

   reproducing the project’s claimed results on H100 hardware requires a very specific environment: a Hopper GPU (sm_90a), the CUDA 12.x toolkit… users have historically reported issues reproducing evaluation scores (specifically RULER benchmarks) in the developer’s previous version

   ↩

9. aimodeling.com news writeup — https://www.aimodeling.com/en/news/slug/flashprefill-v2-block-sparse-prefill

   primary affiliations as WeChat, Tencent, the Chinese Academy of Sciences (CASIA), and UCAS… FlashPrefill V2 remains a Tencent-led contribution rather than an official Alibaba/Qwen project

   ↩

10. Native Sparse Attention (alphaxiv 2502.11089) — https://www.alphaxiv.org/abs/2502.11089

    NSA is ‘natively trainable,’ meaning it is integrated during pre-training and fine-tuning… up to 11.6x faster decoding and 9.0x faster forward passes at 64k context lengths

    ↩

11. SPLA paper (arXiv 2512.22066) — https://arxiv.org/html/2512.22066v1

    Sparse Plus Linear Attention… instead of completely discarding unselected ‘long-tail’ blocks, it compresses them into a recurrent state using a Residual Linear Attention (RLA) module… reported to surpass standard dense attention models on RULER

    ↩

12. Hugging Face blog: MInference 1.0 (Li Yucheng) — https://huggingface.co/blog/liyucheng/minference10

    MInference identifies three static-dynamic patterns… A-shape, Vertical-Slash, and Block-Sparse—and assigns the optimal pattern to each head offline… achieving up to a 10x speedup for 1M-token prompts on a single A100

    ↩

13. Camuffo, Gambardella, Pandey, Asirvatham et al., OpenAI working paper PDF (Aug 2026) — https://cdn.openai.com/pdf/novices-and-llm-august-2026.pdf

    Causal reasoning training is complementary to AI: students who played the game and then used ChatGPT produced higher-quality strategic analyses than those using AI alone.

    ↩

14. ResultSense analysis of the Bocconi RCT — https://www.resultsense.com/news/2026-08-28-bocconi-chatgpt-critical-thinking-study/

    The study measured what students produced in a single 45-minute task rather than what they learned… the experiment was not longitudinal, leaving open questions about skill retention, transferability, or whether students might develop a dependency on AI that erodes their unaided performance over time.

    ↩

15. Wharton (Penn) — Bastani study on ChatGPT and math learning — https://knowledge.wharton.upenn.edu/article/without-guardrails-generative-ai-can-harm-education/

    Students using standard ChatGPT solved 48% more practice problems correctly, but on subsequent unassisted exams they scored 17% lower than the control group, suggesting they had failed to grasp the underlying mathematical concepts.

    ↩

16. Hechinger Report — ‘Kids who use ChatGPT do worse on tests’ — https://hechingerreport.org/kids-chatgpt-worse-on-tests/

    A 2026 study of 3.2 million interactions on the ALEKS platform observed a 25% decline in the odds of correct responses for math problems that are easily outsourced to AI.

    ↩

17. Chosun / USC study on LLM homogenization — https://www.chosun.com/english/industry-en/2026/08/25/WB55MVIOAVHQFFY6SO2RZWPBHE/

    A University of Southern California analysis of over 880,000 texts found that generative AI reduces stylistic diversity by 21% to 50% across various prompts… student essays lost up to 78% of their structural variance, converging toward formulaic ‘AI skeletons’.

    ↩

18. CleanTechnica on OpenAI economics-team resignations — https://cleantechnica.com/2025/12/14/openai-pushing-propaganda-over-research-researchers-who-quit-argue/

    Cunningham’s internal departure message explicitly characterized the team as a ‘propaganda arm’ tasked with shielding the company from inconvenient truths about workforce displacement.

    ↩