JS Wei (Jack) Sun

Qwen needs Python, Antigravity's demo was in ChEMBL, DeltaNet beats Transformers

Qwen's reasoning win needed a Python sandbox, Antigravity's drug demo was already in ChEMBL, and DeltaNet beat Transformers where theory said they'd fail.

Qwen needs Python, Antigravity’s demo was in ChEMBL, DeltaNet beats Transformers

TL;DR

  • Google Antigravity ships cheminformatics skills on Gemini 3.8 Flash, compressing medicinal-chem loops to 1 hour.
  • Antigravity’s lorlatinib walkthrough re-derives a 2017 macrocyclization already in ChEMBL and training data.
  • Frontier Transformers crater to 13% on StateBench’s longest General Finite tracking sequences.
  • Gated DeltaNet generalizes 250×, solving 64,000-step tracking after training on 256 steps.
  • Qwen 3.8 27B identified Breakout, Pong, and Pac-Man from raw policy weights in 46-65 minutes each.

Three AI tech showcases today each collapse under a different critique. Google Antigravity compresses multi-day medicinal chemistry into an hour, but its walkthrough re-derives a 2017 drug whose answer is already in ChEMBL and Gemini’s training corpus — a benchmark-leak problem. Ai2’s StateBench shows frontier Transformers collapsing to 13% on hard state tracking while Gated DeltaNet generalizes 250× past its training length — an architectural ceiling problem, solved by a different architecture rather than a bigger Transformer. And Qwen 3.8 27B classifies Atari games from raw policy weights on a laptop, but the headline Qwen3.5-0.8B math fix needed a Python sandbox and a 50-forward-pass budget — a scaffold-credit problem.

Round-ups cover CWE-bench v2, a cleaner vulnerability-detection benchmark meant to undo earlier score inflation, and IBM Palette, a small-model approach to on-brand slide generation.

DeepMind’s Antigravity cuts drug-design loops to one hour

Source: deepmind-blog · published 2026-11-03

TL;DR

  • Google Antigravity ships cheminformatics skills on Gemini 3.8 Flash, compressing multi-day medicinal-chem loops to one hour.
  • The showcase walkthrough re-derives lorlatinib from crizotinib — a 2017 macrocyclization whose answer is already in ChEMBL and the training data.
  • 10,000 scientists get 6 months of free top-tier access, a generous footprint for untested prompt-injection risk.
  • Frontier models cluster within 5 points on DrugDiscoveryBench 2026, making harness quality — not Gemini — Antigravity’s real battleground.

What actually ships

Antigravity is pitched as a “science workbench” with a clean division of labor: Gemini 3.8 Flash handles orchestration, database navigation and scientific reasoning, while RDKit and connectors to ChEMBL and PubChem do the deterministic chemistry. The useful plumbing is the representation layer — the agent unifies IUPAC names, Markush structures and SDF files, then exports into industry-standard formats that drop into existing lab pipelines 1. For medicinal chemists who spend real hours reconciling molecule encodings, that is not nothing.

The marquee demo walks through rigidification: evolving the “floppy” first-generation kinase inhibitor crizotinib (five rotatable bonds) into lorlatinib, a macrocycle with zero rotatable bonds and higher potency against ALK-positive lung cancer. The agent curates compounds from literature, flags brain-penetrant candidates, runs matched molecular pair analysis, benchmarks ML models with scaffold and chronological splits, and even draws curved-arrow mechanisms for the proposed cyclization.

The lorlatinib demo is retrospective

The problem with that storyline is that lorlatinib shipped in 2018. Both molecules are deeply represented in the public SAR corpus the agent is trained on and queries at inference. Showing an LLM “discover” a known rigidified scaffold is a workflow demo, not evidence of generative capability. Derek Lowe’s standing critique applies cleanly: AI handles well-structured subproblems, but the dominant causes of clinical failure — target choice and human toxicity — sit outside what any agent framework currently touches 2. Independent coverage of the launch makes the same point more bluntly: changing a single atom can redefine a compound, and “a guessing model can quietly ruin a multi-million-dollar analysis” 3.

Competition and the harness problem

Antigravity is not the only pitch. DeepMind’s sister company Isomorphic Labs just shipped IsoDDE, a specialist stack that reportedly more than doubles AlphaFold 3’s accuracy on novel protein–ligand systems and beats physics-based methods on binding affinity 4. That is a different philosophy — purpose-built engines over general agents — and the benchmark picture argues it matters. On DrugDiscoveryBench 2026’s 82 expert tasks, GPT-5.5 hit 51.6%, Gemini 3.5 Flash 50.0%, and Claude Opus 4.8 47.2%, with harness changes swinging results by more than 10 points 5. Antigravity’s edge, if there is one, lives in the skills layer, not the Gemini weights underneath.

The security problem nobody is demoing

Handing an agent terminal access plus live connectors to public chemistry databases is a new attack surface. Red-team work on pharma LLM deployments has shown indirect prompt injection through a single poisoned database entry can hijack an agent’s reasoning, and 83% of pharma organizations currently lack basic automated safeguards against AI-related data breaches 6. Rolling that stack out to 10,000 outside scientists on a free tier is a generous way to find the failure modes.

flowchart LR
    A[ChEMBL / PubChem] --> B{Antigravity agent<br/>Gemini 3.8 Flash}
    L[Literature PDFs] --> B
    B --> R[RDKit deterministic tools]
    B --> E[Export: SDF / IUPAC / Markush]
    E --> W[Lab pipeline]
    P[Poisoned DB entry] -. indirect prompt injection .-> B
    B -. hallucinated 'verified' .-> W

The plumbing is real. The lorlatinib victory lap is not a prospective result, and the interesting question — whether a general agent beats specialist engines like IsoDDE on prospective benchmarks — remains open.


Ai2’s StateBench drops frontier LLMs to 13% on hard tracking

Source: huggingface-blog · published 2026-11-03

TL;DR

  • Frontier models crater to 13% accuracy on StateBench’s longest “General Finite” sequences, where variables reference other variables.
  • Gated DeltaNet generalizes 250× — trained on 256-step sequences, it solves 64,000-step tracking that Transformers cannot learn even with direct training.
  • StateBench operationalizes a 2024 theorem proving Transformers and linear SSMs cannot solve $S_5$ word problems.
  • Format sensitivity is a red flag: narrative passes, spreadsheet fails on identical logic.

What StateBench actually measures

Ai2’s new benchmark sorts state-tracking problems into four tiers borrowed from Krohn–Rhodes algebra: R-trivial (set once), Aperiodic (overwrite with a constant), Soluble (self-referential updates like x = x + 5), and General Finite (assignments by reference, modeled on the $S_5$ permutation group). The last tier is the interesting one — it is the formal boundary where standard attention and linear-recurrent architectures are provably out of expressive range, as Merrill, Petty and Sabharwal established in The Illusion of State in State-Space Models 7. StateBench is the empirical arm of that theoretical program, not an independent discovery. The four levels operationalize a complexity hierarchy Ai2’s group has been developing for two years.

That lineage matters because it reframes the headline. Frontier models aren’t failing a hard benchmark — they are failing in the exact pattern a published complexity proof predicted.

The 13% cliff and the architecture split

GPT-6 Sol, GPT-6 Luna, Claude Opus 5.5 and Gemini 3.8 Flash all handle R-trivial and Aperiodic reliably. Accuracy collapses on Soluble as context grows, and on the longest General Finite sequences falls to 13% or lower — numbers independent coverage has corroborated 8. This matches what RULER has been showing for over a year: “effective context” is far smaller than the advertised window the moment the task stops being lookup 9. StateBench extends that critique from “find the fact” to “maintain the variable.”

The architectural experiment is sharper than the frontier-model table. Ai2 trained 275M-parameter models from scratch on StateBench data. Transformers still failed General Finite even with direct training. A hybrid using Gated DeltaNet, trained on 256-assignment sequences, generalized perfectly to 64,000 — a 250× jump. The implication: pretrained hybrids underperform not because the architecture can’t do it, but because web text doesn’t contain enough dense state-update sequences to teach the recurrent memory to activate.

Why the Gated DeltaNet win deserves asterisks

The 250× result is real but already a moving target. NVIDIA’s Gated DeltaNet-2 decouples the erase and write gates the original tied to a single scalar, specifically because the shared gate hurt multi-key retrieval 10. DeltaProduct replaces the rank-1 delta update with Householder products to keep effective rank stable as sequences grow 11. Both suggest vanilla GDN is a floor. But they also raise a co-design worry: how much of GDN’s advantage here is StateBench and GDN being developed by overlapping communities?

A quieter reproducibility trap, from Sebastian Raschka’s from-scratch implementation: Parity-style tracking only generalizes if the beta parameter lives in $(0, 2)$ to permit negative eigenvalues. Without that, accuracy collapses to random 12. Ai2 should disclose the parameterization alongside the length-generalization claim.

The format-sensitivity finding — narrative pass, spreadsheet fail on identical logic — is the one every reader should sit with. If presentation changes the score, the benchmark is still partly measuring familiarity, not pure state.


Qwen 3.8 27B reads toy policy weights on a laptop

Source: simon-willison · published 2026-11-03

TL;DR

  • Qwen 3.8 27B on a laptop identified Breakout, Pong, and Pac-Man from raw policy weights with no tools.
  • Each game took 46-65 minutes and tens of thousands of reasoning tokens to decode.
  • Freeway — the one obscure 1981 Atari title in the set — was misidentified as a Pac-Man variant.
  • The headline Qwen3.5-0.8B math fix needed a Python sandbox and a 50-forward-pass budget, not pure reasoning.
  • On Gemma 3 270M, which lacks Qwen’s repetition-loop pathology, the same recipe produced negligible gains.

The laptop replication

Simon Willison took the “which game was this policy trained on?” prompts from Takehi et al.’s Sparks of Weight-Space Programming and pointed a local Qwen 3.8 27B (17GB quantized, LM Studio) at them. After 43,506 reasoning tokens and 46 minutes, it called Pac-Man at 72% confidence. Pong landed at 75% after an hour, Breakout at 88% after 52 minutes. Freeway — a 1981 Atari title that’s genuinely obscure — got misidentified as Pac-Man.

That’s a meaningful data point. The paper framed weight-reading as an emergent property of frontier models like the hypothetical GPT-6 Astra. A 27B model you can fit in a backpack clears the same bar, just slowly.

What the critics actually object to

The two most-cited rebuttals — Matthew Honnibal’s and Sean Goedecke’s — don’t dispute the numbers. They dispute the framing. The paper mashes together two very different experiments: LLMs eyeballing tiny 17K-172K-parameter policy nets with no tools, and LLMs running inside a Python sandbox with a 50-forward-pass evaluation budget to iteratively patch Qwen3.5-0.8B 1314.

Goedecke’s deflation is sharper still. The game-identification prompts hand the model explicitly labeled input channels (“agent”, “moving objects”, “walls”). A three-line numpy snippet computing L2 norms of the first-layer weights produces heatmaps where a human can see the paddle row for Breakout or the maze for Pac-Man by eye 15.

identifying the game from labeled-channel toy weights is ‘relatively trivial’

Willison’s own reasoning trace supports this — Qwen spends its 43K tokens mapping channel indices to grid coordinates, the exact bookkeeping numpy would do in milliseconds.

The write-side claim is narrower than it sounds

An independent reproduction from the authors’ bit-exact diffs confirms the headline: MATH-500 rises from 7.4% to ~50%, GSM8K from 13.8% to 67.1%. The mechanism is concrete — 131,072 entries edited across down_proj columns in layers 15 and 19, cutting token-limit terminations from 93% to ~33% 16. In other words, the “fix” kills a documented reasoning loop that Qwen’s own model card warns about. It doesn’t teach math. It stops the model from looping until it runs out of context.

That explains the Gemma 3 270M negative result cleanly: no repetition pathology, no gains 17. The authors concede the method “removes a barrier” rather than “adds an ability.”

The technique itself has a lineage the paper underplays. Weight orthogonalization to bake a contrastive activation direction into weights is the core of Arditi et al.’s refusal-direction work, Turner’s ActAdd, Rimsky’s CAA, and the abliteration toolchain routinely shipped against Qwen on HuggingFace 18. The Astra fix is a plausibly LLM-automated application of that recipe, not a new paradigm.

What’s actually new

Strip the framing and one interesting residual claim survives: a frontier LLM can autonomously run a mechanistic-interpretability workflow — spot a failure direction, localize it to specific layers, patch it — that previously needed a human researcher driving the notebook. Willison’s laptop run validates the weakest half of that. The age of source code is not over.

Round-ups

CWE-bench v2 expands security benchmark coverage and transparency

Source: huggingface-blog

CWE-bench v2 updates the vulnerability-detection benchmark with broader CWE coverage, cleaner ground-truth labels, and fuller documentation of how samples were sourced. The revision targets reproducibility gaps that let earlier code-security evaluations overstate model performance on real-world bug classes.

IBM’s Palette teaches small models to match a brand’s slide style

Source: huggingface-blog

Palette, from IBM Research, fine-tunes a compact model to generate slide decks in a user’s own design language rather than generic templates. The approach learns layout, typography, and color conventions from example decks so outputs stay on-brand without manual restyling.

Footnotes

  1. Chenab Times — DeepMind cheminformatics coverage — https://thechenabtimes.com/2026/11/04/google-deepmind-enhances-drug-discovery-with-ai-powered-agentic-cheminformatics/

    Antigravity unifies IUPAC names, Markush structures and SDF files, and allows export in industry-standard formats so outputs feed directly into existing lab pipelines — but DeepMind reiterates that proposed macrocycles remain testable hypotheses requiring wet-lab validation.

    ↩
  2. Bio-IT World — Derek Lowe interview — https://www.bio-itworld.com/news/2025/04/15/derek-lowe-on-ai-in-drug-discovery-between-hype-and-hope

    AI excels at well-structured problems like protein folding but still struggles with the messy reality of small-molecule discovery; the primary causes of drug failure — target selection and human toxicity — remain outside the current reach of AI models.

    ↩
  3. GlobAI — independent coverage of the Antigravity launch — https://www.globai.org/blog/google-deepmind-unveils-agentic-cheminformatics-on-antigravity

    Experts emphasize the distinction between a worked demonstration and a validated drug result; changing a single atom can redefine a compound’s properties, and a guessing model can quietly ruin a multi-million-dollar analysis.

    ↩
  4. Isomorphic Labs — IsoDDE announcement — https://www.isomorphiclabs.com/articles/the-isomorphic-labs-drug-design-engine-unlocks-a-new-frontier

    The Isomorphic Labs Drug Design Engine more than doubles the accuracy of AlphaFold 3 on novel protein–ligand systems in the internal Runs N’ Poses benchmark, outperforming gold-standard physics-based methods for binding-affinity prediction.

    ↩
  5. Vibe Leaderboard — DrugDiscoveryBench 2026 — https://www.vibeleaderboard.ai/intel/41bc8788-b663-4944-91ae-350197a572eb

    On 82 expert-curated tasks, GPT-5.5 led at 51.6%, Gemini 3.5 Flash reached 50.0%, and Claude Opus 4.8 scored 47.2% — no single model dominates, and a superior agent harness can swing results by over 10 percentage points.

    ↩
  6. IntuitionLabs — Pharma LLM security report — https://intuitionlabs.ai/articles/pharma-llm-security-red-teaming-prompt-injection

    Indirect prompt injection via a malicious database entry can hijack an agent’s reasoning process; 83% of pharmaceutical organizations lack basic automated safeguards against AI-related data breaches.

    ↩
  7. Merrill, Petty & Sabharwal — ‘The Illusion of State in State-Space Models’ (arXiv 2404.08819) — https://arxiv.org/abs/2404.08819

    Despite their recurrent formulation, common SSMs like S4 and Mamba are expressively confined to TC⁰ and provably cannot solve NC¹-complete word problems such as S₅ permutation composition.

    ↩
  8. SuperpowerDaily — independent coverage — https://superpowerdaily.com/posts/ai2-finds-leading-ai-models-fall-to-13-or-less-on-its-longest-hardest-tracking-tasks

    StateBench fills a ‘measurement gap,’ since traditional needle-in-a-haystack tests ignore the procedural complexity of enterprise state tracking; frontier models fall to 13% or less on the longest, hardest tracking tasks.

    ↩
  9. llm-stats.com — RULER benchmark summary — https://llm-stats.com/benchmarks/ruler

    RULER’s 13 tasks across retrieval, multi-hop tracing, aggregation, and QA consistently show ‘effective context’ is far smaller than claimed context, with 128K-window models cliffing at 32K–64K on non-lookup tasks.

    ↩
  10. NVIDIA Research — ‘Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention’ — https://research.nvidia.com/publication/2026-12_gated-deltanet-2-decoupling-erase-and-write-linear-attention

    The original Gated DeltaNet tied erase and write to a single scalar gate; decoupling them improves multi-key retrieval and needle-in-a-haystack performance.

    ↩
  11. DeltaProduct paper (arXiv 2502.10297) — https://arxiv.org/html/2502.10297v3

    DeltaProduct uses Householder products to achieve higher-rank state transitions, overcoming the rank-1 limitation of the original delta rule and stabilizing effective rank as sequences grow.

    ↩
  12. Sebastian Raschka — ‘LLMs from Scratch’ DeltaNet chapter — https://sebastianraschka.com/llms-from-scratch/ch04/08_deltanet/

    Achieving perfect generalization on Parity-like state tracking requires β_t ∈ (0, 2) to permit negative eigenvalues; missing this parameterization yields near-random accuracy.

    ↩
  13. R&D World — summary of Honnibal/Goedecke critiques — https://www.rdworldonline.com/mit-team-says-gpt-6-astra-can-fix-an-ai-model-by-editing-its-weights-critics-say-the-method-was-standard-ml-engineering/

    Critics argue the headline Qwen repair required a Python sandbox and a budget of 50 forward passes on practice problems — closer to an automated ML engineer than a ‘native’ weights reader; the ‘no-tools’ setting only held for toy 17K-weight networks.

    ↩
  14. Hacker News discussion (item 50355251) — https://news.ycombinator.com/item?id=50355251

    Commenters flagged the paper-shaped blog post as bypassing peer review while adopting formal paper aesthetics, and noted that Astra’s success hinges on a 50-pass interactive search budget — low-budget variants show much smaller gains.

    ↩
  15. Sean Goedecke, ‘LLMs are not superhuman…’ (via daily.dev) — https://daily.dev/posts/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights-yrkwtj9xo

    A three-line numpy snippet computing L2 norms of first-layer weights produces heatmaps that make the game obvious to any human — a paddle row for Breakout, a maze for Pac-Man — so identifying the game from labeled-channel toy weights is ‘relatively trivial’.

    ↩
  16. HuggingFace repo RikiyaT/Qwen3.5-0.8B-weight-space-programming — https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming

    Independent reproduction from released bit-exact diffs lifts MATH-500 from 7.4% → ~50% and GSM8K from 13.8% → 67.1%; the edit modifies 131,072 entries across down_proj columns in layers 15 and 19, and cuts the token-limit termination rate from 93% to ~33%.

    ↩
  17. CryptoBriefing — Gemma 3 negative result — https://cryptobriefing.com/gpt-6-astra-edits-qwen-weights-math/

    Applied to Gemma 3 270M, which does not exhibit the Qwen-style repetition loop, the same weight-editing approach yielded negligible math gains — the authors concede the method ‘removes a barrier’ rather than ‘adding an ability.’

    ↩
  18. Coral Flavor — abliteration / steering prior art — https://coralflavor.com/blog/abliterated-vs-fine-tuned-the-state-of-uncensored-ai-models-in-2026

    Abliteration uses weight orthogonalization to bake a contrastive activation direction permanently into weights — a lineage running through Arditi et al. (2024), Turner’s ActAdd, and Rimsky’s CAA; critics say Astra’s Qwen ‘fix’ is a specific application of this well-known steering-vector technique.

    ↩
Jack Sun

Jack Sun, writing.

Engineer · Bay Area

Hands-on with agentic AI all day — building frameworks, reading what industry ships, occasionally writing them down.

Digest
All · AI Tech · AI Research · AI News
Writing
Essays
Elsewhere
Subscribe
All · AI Tech · AI Research · AI News · Essays

© 2026 Wei (Jack) Sun · jacksunwei.me Built on Astro · hosted on Cloudflare