Anthropic ART fails replication, GPT-6 outscores graders, Edge0 streams 35B MoE

TL;DR

Anthropic’s ART enzyme find failed to reproduce in 0 of 10 re-runs of the same Claude-agent pipeline.
GPT-6 Astra scores 57.3% on MentalHealthBench, above the 38.5% earned by the clinicians who wrote the rubric.
Stanford finds psychiatrists themselves disagree on what counts as a safe suicide-scenario reply.
Edge0 streams a 35B MoE at 20.4 tok/s in 2.9 GiB, with a 3.9-point quality drop vs fp16.
LimiX-2 reframes tabular in-context learning as mechanism modeling, drawing 711 Hugging Face upvotes.

Three research features today with no shared thread. Anthropic’s 950-agent, 21-hour Claude run surfaced a CRISPR-like ART enzyme system in jumbo phages — and the same bioRxiv preprint concedes that 0 of 10 re-runs of the pipeline rediscovered the headline upstream array. MentalHealthBench puts GPT-6 Astra at 57.3%, ahead of Claude Opus 5.5 and above the licensed clinicians who wrote the rubric, while a parallel Stanford study finds those clinicians don’t agree with each other on what a safe suicide-scenario reply looks like. Edge0 serves a 35B MoE at 20 tok/s in under 3 GiB of active RAM by streaming 4-bit experts from SSD, with the paper framing the streaming engine — prerouter, staged executor, unmerged LoRA — as the real contribution.

The round-ups lean infrastructural: long-context MoE memory peaks, a bit-plane KV cache in Fathom, LimiX-2’s mechanism-modeling reframe of tabular ICL (711 upvotes), a self-locomoting anthropomorphic hand, and ScienceIDE’s attempt to turn research codebases into agent training environments.

Claude’s ART enzyme find: 0 of 10 re-runs reproduced it

TL;DR

Anthropic ran 950 Claude agents for 21 hours to surface a CRISPR-like “ART” enzyme system in jumbo phages.
The bioRxiv preprint admits 0 of 10 re-runs of the same pipeline rediscovered the headline upstream array.
Similar RT+array systems — retrons and DRT2 — were already characterized in 2021 and 2024.
Georgia Tech’s Mark Riedl calls announcing a discovery of unknown function “irresponsible science.”

What Anthropic is claiming

Anthropic’s life-sciences group deployed roughly 950 Claude agents that chewed through 210 million tokens of genomic data, filtered 200,000 reverse transcriptases down to 3,500 candidate systems, then to 20 finalists. One agent flagged a reverse transcriptase sitting next to a tandem repeat array that “looked CRISPR-like” — dubbed Array-associated Reverse Transcriptases, or ART. The pitch is that Claude did months of expert bioinformatics in 21 hours and generated a genuinely new hypothesis.

Novelty is partial, not clean

The “AI saw what humans missed” framing does not survive contact with the prior literature. Retron-like reverse transcriptases in jumbo phages (MarsHill, Machias) were characterized in 2021, and the broader Diversity-Generating Retroelement family has been studied for two decades ¹. More directly, Sternberg and Zhang described DRT2 in 2024: an RT that performs rolling-circle reverse transcription on a neighboring non-coding RNA to generate repetitive cDNA arrays encoding a toxic “Neo” open reading frame ². ART’s architecture — RT + accessory gene + repeat array producing short RNAs — sits inside that conceptual space.

Dario Amodei himself acknowledged that a Stanford/Arc Institute group independently described a “comparable but distinct” system on a similar timeline through conventional computational biology ³. That is the tell: the field was already converging.

The reproducibility footnote

Buried in the preprint is a striking admission. The authors ran the same agentic pipeline 10 additional times. Nearly all runs surfaced related genomic sequences, but none flagged the specific upstream array that made the initial find headline-worthy ⁴. That is a serious problem for a workflow being sold as a repeatable “AI scientist.” It reframes the discovery as a lucky draw across 950 stochastic agents rather than a robust protocol, and it directly undercuts the “compressing months of work into 21 hours” claim — the 21 hours only works if you also count the runs that missed.

What actually holds up

The wet-lab data is the strongest part of the story. In Staphylococcus phages, ART-associated short RNAs constituted up to 8% of the total RNA pool shortly after infection ⁵ — a large expression signal that argues the array is doing something, even if nobody knows what yet.

That “even if nobody knows what yet” is where Mark Riedl, an AI researcher at Georgia Tech, draws blood:

Riedl also notes the awkward optics: Anthropic simultaneously publishes bio-capability demonstrations and warns publicly about AI-enabled bio-misuse. Feng Zhang — the actual CRISPR pioneer in this story — offered only the standard hedged endorsement: “genuinely intriguing,” merits further investigation. That is what “interesting pattern, unproven system” sounds like in polite scientific English.

The takeaway

There is a genuine result here: a measurable sRNA expression signal and a plausible architecture worth chasing in a wet lab. There is also a partial-novelty problem, an admitted reproducibility problem, and a milestone announcement running well ahead of the biology. Both things are true, and only one of them was in the press release.

MentalHealthBench scores GPT-6 above the clinicians grading it

TL;DR

GPT-6 Astra tops MentalHealthBench at 57.3%, ahead of Claude Opus 5.5 (52.4%) and a March-2025 GPT-4o baseline (32.1%).
80 licensed clinicians across 22 countries wrote the rubrics that define the ground truth.
Human-written clinician responses reportedly score ~38.5% — below the models being evaluated.
A Stanford study finds psychiatrists disagree on what counts as a “safe” suicide-scenario reply, splitting safety-first vs. engagement-centered.
55% of early developer reactions were skeptical, reading the release as legal-defensive positioning after Raine v. OpenAI.

What shipped

OpenAI released MentalHealthBench, an open eval that grades model responses to synthetic mental-health conversations across 10 behaviors — empathy, harm avoidance, agency, actionable guidance — weighted from −10 (harmful) to +10 (highly beneficial). Rubrics only ship if two clinicians agreed and none dissented. Grading is automated by GPT-5.6 Sol against those rubrics. The eval slots into OpenAI’s simple-evals repo, and the M42 healthcare AI team already has a standalone implementation running local vLLM inference against the physician rubrics ⁷, so the numbers are reproducible outside OpenAI’s infrastructure.

The headline result: GPT-6 Astra 57.3%, GPT-6 Sol 53.9%, Claude Opus 5.5 52.4%, GPT-4o (March 2025) 32.1%. Non-acute conversations dominate the mix (53.5%); emergency scenarios are 28.3%.

The rubric rewards verbosity

The sharpest methodological problem is what independent analysts are calling the style gap. Concise, targeted responses written by the clinicians themselves reportedly score below verbose model outputs, because the rubric awards points for hitting expert-listed items rather than for clinical parsimony ⁸. A model can pass by producing empathetic, well-structured filler that touches every checkbox while missing the nuanced red flag a terse clinician would prioritize.

That’s compounded by the grader. Work on MentalAlign-70k found LLM judges are reliable on cognitive attributes but significantly weaker on affective ones like empathy, and exhibit “systematic inflation” versus human raters ⁹. MentalHealthBench uses GPT-5.6 Sol to grade GPT-6 outputs on exactly those affective dimensions. Draw your own conclusions.

The ground truth is contested

The two-expert-consensus design assumes clinicians converge on what a good response looks like. A Stanford study on AI mental-health safety testing found they don’t — inter-rater reliability among board-certified psychiatrists was poor on high-risk scenarios like suicide and self-harm, with disagreement tracking whether raters came from a “safety-first” or “engagement-centered” school ¹⁰. A rubric that filters for two-expert agreement doesn’t dissolve that split; it just picks the items where the two happened to overlap, which skews toward the uncontroversial middle.

The Raine shadow

MentalHealthBench doesn’t ship into a neutral discourse. Raine v. OpenAI — the wrongful-death suit brought by the family of 16-year-old Adam Raine — was amended in October 2025 to add intentional-misconduct claims, alleging OpenAI knowingly relaxed guardrails to prioritize engagement ¹¹. Developer forums drew the line explicitly: roughly 55% of early reactions were skeptical, with commenters framing the benchmark as an attempt to define the metrics regulators will inherit ¹².

The 57.3% is a rubric-coverage number graded by a sympathetic LLM against a contested clinical standard. It is not a therapeutic-readiness signal, and OpenAI’s own framing — AI as supplement, not replacement — concedes as much. The useful contribution here is the reproducible infrastructure and the canary-protected dataset. The leaderboard is the part to read skeptically.

Edge0 streams a 35B MoE from SSD at 20 tok/s in 3 GiB

TL;DR

Edge0 serves a 35B MoE at 20.4 tok/s using 2.9 GiB active RAM, streaming 4-bit experts from SSD.
A per-layer prerouter predicts next-token expert routing, buying one forward pass to hide SSD reads.
Quality drops 3.9 points vs. fp16 across five benchmarks, recovered by an unmerged LoRA parallel branch.
Edge0’s real contribution is the streaming engine — incremental stacking, staged executor, unmerged LoRA.

The memory wall the sparsity story forgot

MoE inference is supposed to be cheap: activate 4 of 128 experts per token, do less math. But the weights still have to be somewhere, and a 35B-class MoE is 19.5 GB even at int4 — well past what a 16 GB laptop holds. Naive SSD offload doesn’t rescue you, because you can’t start fetching layer N+1’s experts until layer N picks them, and the read is far too slow to hide behind the compute.

Edge0’s answer is to route one token early. A small per-layer MLP head — the “prerouter” — consumes the current token’s hidden state and predicts which experts the next token will need. That prediction becomes the actual routing decision, so what gets fetched from disk is exactly what gets used. No speculative unroll, no fallback load, no dropped tokens.

Numbers, and what they’re conditional on

On an Apple Silicon box where the 18.4 GB model file doesn’t fit in RAM, the prerouter cuts time-blocked-on-load from ~244 ms to ~102 ms per step at K=4, an 80–84% decode throughput gain over on-demand streaming. End state: 20.4 tok/s at 2.9 GiB active for the 35B tier, 28.0 tok/s at 1.5 GiB for an 8B variant. Quality lands at 79.2 vs. 83.2 (fp16) averaged across AIME 2026, HumanEval, GPQA-Diamond, MMLU-Pro, and IFBench.

Two design choices carry real weight. First, weight tensors use “sticky slots” and only the experts that actually changed between tokens get restacked — avoiding a per-step tensor rebuild. Second, the recovery LoRA is served unmerged, as a parallel branch y = W_int4(x) + ΔW(x). Merging and re-quantizing would erase up to 82% of the adapter signal, a number consistent with Scale-QLoRA’s independent finding that naive 4-bit merges can cost up to 39 points on downstream tasks ¹³.

Where the paper’s framing gets generous

The prerouter is pitched as the unlock, but “predict routing ahead of time” is a crowded field. Fate reaches 97.15% prefetch accuracy via cross-layer gate similarity, without SSD in the loop ¹⁴. Brainstorm (OSDI ‘23) got there earlier with speculative execution and a verification thread that unrolls on mispredict ¹⁵. Edge0’s real novelty is treating the prediction as the route and paying for the quality gap with distillation, not the prediction itself.

The “SSD streaming is the only path” framing is also contested. Fiddler runs unquantized Mixtral-8x7B at ~3 tok/s on a single 24 GB GPU by executing MoE layers on the CPU instead of shuttling weights across PCIe — 19× over DeepSpeed-MII in its setup ¹⁶. Edge0 is faster in absolute terms but runs on unified memory and fast NVMe; on a PCIe-bound workstation, CPU orchestration remains a live alternative.

The headline numbers also assume slack. Edge0 is MLX-only, and MLX prefill can be ~50% slower than llama.cpp with Flash Attention on long contexts, with prefill sometimes >90% of end-to-end time ¹⁷ — matching the paper’s admitted 44 ms/step CPU graph-building floor. On an 8 GB MacBook without spare OS page cache, community testing reports decode dropping ~30× and hard-math benchmarks losing >6 points ¹⁸. The “3 GiB peak active” line is real; it just needs a fast SSD and a system that isn’t already fighting for pages.

Takeaway

The prerouter is the marketing, but the engineering — incremental stacking, staged executor, unmerged LoRA — is what makes 35B on a laptop actually usable. Whether the pattern ports off MLX and off Apple’s storage stack is the open question, and the paper’s roadmap leaves the CUDA backend as an empty slot.

Round-ups

LimiX-2 reframes tabular in-context learning as mechanism modeling

The structured-data model adopts Contextual Mechanism Networks pretrained with Context-Conditional Masked Modeling, shifting in-context learning from target-centric p(y|x,D) prediction to mechanism-oriented joint modeling. Scaled per the LimiX family’s established laws, the release drew 711 upvotes on Hugging Face’s daily papers.

Long-context MoE training targets 4 unbounded memory peaks at once

Mixture-of-Experts training fails whenever any single component’s peak allocation exceeds device memory, so average footprint is the wrong target. The paper identifies four peaks left unbounded by common parallelism plans: expert dispatch, vocabulary projection, gradient checkpoint boundaries, and optimizer state, each scaling on a different axis.

Fathom lets each query pick its bit budget for offloaded KV scans

At million-token sessions, ranking all keys for top-k decoding saturates host-memory bandwidth. Fathom stores the 4-bit K cache channel-major as bit planes, so reading a t-plane prefix equals a t-bit quantizer, letting each query spend a per-channel bit budget instead of a fixed read depth.

Zing-0.5 is a 5B world model steered by keyboard and live text

The autoregressive world model targets playability: users explore generated worlds and shape events through joint keyboard and online text control. Its unified conditioning combines magnitude-aware key inputs with temporally aligned text instructions, letting a single 5B model learn navigation and event control together.

ScienceIDE turns research codebases into agent training environments

Scientific repositories hold decades of executable knowledge, but fragmented toolchains and implicit conventions block their use as agent training data — the ‘scientific experience bottleneck.’ ScienceIDE wraps this code into programmable environments with expert-defined correctness criteria, giving scientific agents reliable learning signal.

HypoEvolve pairs genetic search with multi-agent LLMs for hypotheses

Scientific-discovery agents typically mix critique, comparison, and revision loops, but the contribution of collaboration versus raw model skill is unclear. HypoEvolve wraps multi-agent LLMs in a genetic algorithm to separate those effects, isolating how different collaboration structures shape final hypothesis quality.

Anthropomorphic hand learns to walk on its own fingers

A self-contained robotic hand uses the same fingers to locomote, bear weight, and manipulate, keeping its original finger design and position controller. Reinforcement learning in a hardware-calibrated simulator accounts for the hand’s unequal fingers, and the trained policy moves faster in sim than hand-tuned baselines.

kingy.ai analysis — https://kingy.ai/blog/claude-art-discovery-explained/

‘Retron-like’ reverse transcriptases had already been identified in jumbo phages, such as the MarsHill and Machias phages, in 2021… the current discovery is argued to be a ‘re-contextualization’ where AI spotted structural patterns surrounding an enzyme that human experts had already recorded.

↩
runhangshu.com summary of DRT2 (Sternberg/Zhang 2024) — https://www.runhangshu.com/en/2024/09/defense-associated-rt2-drt2/

The RT enzyme binds to a neighboring non-coding RNA and, upon phage infection, performs ‘rolling-circle’ reverse transcription… generates long, concatenated cDNA repeats from a single RNA template… encodes a ‘nearly-endless open reading frame’ that acts as a potent toxin.

↩
Business Today (citing Amodei) — https://www.businesstoday.in/technology/story/claude-finds-novel-enzyme-in-dna-could-ai-be-entering-the-gene-editing-discovery-race-557432-2026-09-24

Anthropic CEO Dario Amodei noted that a team at Stanford University (associated with the Arc Institute) independently described a ‘comparable but distinct’ system around the same time, suggesting that the scientific community was already converging on this family of enzymes through traditional computational biology.

↩
Futunn summary of bioRxiv preprint — https://www.futunn.com/403

Researchers performed 10 additional ‘reproducibility’ searches using the same computational framework. While nearly all runs identified related genomic sequences, none of the subsequent attempts successfully flagged the specific upstream array that made the initial find so unique.

↩
Anthropic bioRxiv preprint PDF (Durrant et al.) — https://www-cdn.anthropic.com/22573675ada52a8ca8a97a1a4b4326b2f208a071.pdf

In some instances, such as in Staphylococcus phages, these short RNAs were found to constitute up to 8% of the total RNA pool shortly after infection, suggesting the system plays a significant role in the viral lifecycle.

↩
AI Weekly (reporting Mark Riedl) — https://aiweekly.co/alerts/anthropics-new-biolab-claude-autonomously-finds-a-novel-crispr-like-enzyme

Announcing a ‘discovery’ without knowing if it has any actual function or meaning constitutes ‘irresponsible science’… Anthropic seemed to be trying to help the public become ‘sufficiently afraid’ of AI by demonstrating the very biological capabilities it warned were dangerous.

↩
GitHub — m42-health/healthbench (independent implementation) — https://github.com/m42-health/healthbench/

Standalone implementations by the M42 healthcare AI team provide local inference and grading scripts using tools like vLLM… enabling developers to compute detailed metrics by comparing model outputs against the physician-derived rubrics.

↩
nxcode.io analysis of MentalHealthBench — https://www.nxcode.io/resources/news/mentalhealthbench-expert-rubrics-ai-support-2026

Concise, expert-written responses from clinicians often scored lower than the verbose, detailed responses generated by models like GPT-6 Astra… a model could technically pass the rubric by providing ‘safe’ and ‘empathetic’ filler while failing to identify nuanced red flags.

↩ ↩²
OpenReview — MentalAlign-70k paper on LLM-as-judge reliability — https://openreview.net/pdf?id=REKJB5Sh2v

LLM judges show strong reliability in evaluating ‘cognitive attributes’ but exhibit significantly reduced precision when assessing ‘affective dimensions’ like empathy… judges frequently exhibit ‘systematic inflation,’ assigning higher scores to responses than human experts.

↩
Stanford News — study on AI mental-health safety testing — https://news.stanford.edu/stories/2026/07/study-exposes-major-flaw-in-ai-mental-health-safety-testing

Inter-rater reliability among board-certified psychiatrists was surprisingly poor, particularly for high-risk scenarios like suicide and self-harm… experts often disagreed on whether a response was ‘safe,’ depending on whether they prioritized ‘safety-first’ or ‘engagement-centered’ clinical frameworks.

↩
Wikipedia — Raine v. OpenAI — https://en.wikipedia.org/wiki/Raine_v._OpenAI

The complaint alleges that ChatGPT acted as a ‘suicide coach’… In October 2025, the family amended their complaint to include ‘intentional misconduct,’ alleging that OpenAI knowingly relaxed safety guardrails to prioritize user engagement over protection.

↩
daily.dev — developer community reaction — https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf

Approximately 55% of initial community reactions were classified as skeptical… lingering concern that by defining the ‘benchmark’ for mental health safety, OpenAI is attempting to preemptively set regulatory standards that favor its own proprietary architectures.

↩
Scale-QLoRA (arXiv 2509.23638) — https://arxiv.org/html/2509.23638v1

naive merge can result in catastrophic loss—sometimes reducing task performance by up to 39 percentage points—because the fine-tuned adjustments are too subtle to be captured by the coarse 4-bit quantization bins.

↩
The Moonlight review of ‘Fate: Accurate Expert Predictions via Cross-Layer Gate’ — https://www.themoonlight.io/en/review/accurate-expert-predictions-in-moe-inference-via-cross-layer-gate

Fate leverages cross-layer gate similarity to reach a 97.15% prefetch accuracy… accuracy is typically lower in shallower layers, necessitating specialized caching strategies.

↩
Cui et al., ‘Brainstorm’ (OSDI ‘23) — https://www.usenix.org/system/files/osdi23-cui.pdf

Brainstorm… utilizes statistical profiles to predict routing decisions in advance… a parallel verification thread checks the actual routing results; if a misprediction occurs, the model execution is ‘unrolled’ and re-executed with the correct experts.

↩
Kamahori et al., ‘Fiddler’ (arXiv 2505.09343) — https://arxiv.org/html/2505.09343v2

Fiddler achieved token generation speeds over 3 tokens per second for unquantized Mixtral-8x7B, outperforming DeepSpeed-MII by approximately 19× and Mixtral-offloading by 8× … by moving small activation data (megabytes) to the CPU rather than moving heavy expert weights (gigabytes) to the GPU.

↩
Contra Collective, ‘MLX vs llama.cpp on Apple Silicon’ — https://contracollective.com/blog/mlx-vs-llama-cpp-apple-silicon-local-ai

In long-context tasks (e.g., 146,000 tokens), MLX’s token generation speed can drop to nearly 50% of the speed offered by llama.cpp with Flash Attention enabled… prefill phase can account for over 90% of total response time in MLX.

↩
MindStudio review of Edge0-35B — https://www.mindstudio.ai/blog/edge0-35b-phone-memory-moe

on devices with high memory pressure, such as an 8GB MacBook, speeds can drop by 30x as the system struggles to manage the expert weight swap… complex reasoning tasks see the steepest accuracy decline, losing over 6 points on hard math benchmarks.

↩

Anthropic ART fails replication, GPT-6 outscores graders, Edge0 streams 35B MoE

TL;DR

Claude’s ART enzyme find: 0 of 10 re-runs reproduced it

TL;DR

What Anthropic is claiming

Novelty is partial, not clean

The reproducibility footnote

What actually holds up

The takeaway

MentalHealthBench scores GPT-6 above the clinicians grading it

TL;DR

What shipped

The rubric rewards verbosity

The ground truth is contested

The Raine shadow

Edge0 streams a 35B MoE from SSD at 20 tok/s in 3 GiB

TL;DR

The memory wall the sparsity story forgot

Numbers, and what they’re conditional on

Where the paper’s framing gets generous

Takeaway

Round-ups

LimiX-2 reframes tabular in-context learning as mechanism modeling

Long-context MoE training targets 4 unbounded memory peaks at once

Fathom lets each query pick its bit budget for offloaded KV scans

Zing-0.5 is a 5B world model steered by keyboard and live text

ScienceIDE turns research codebases into agent training environments

HypoEvolve pairs genetic search with multi-agent LLMs for hypotheses

Anthropomorphic hand learns to walk on its own fingers

Footnotes

Jack Sun, writing.