PAW echoes Sakana, WARP merges beat oracle, GRPO tops SDPO variants

TL;DR

• PAW compiles English specs into 23MB LoRAs a frozen 0.6B interpreter hot-swaps at runtime.
• WARP infers domain mixes from released weights, scoring 0.046 MAE on BERT via merged pseudo-checkpoints.
• GRPO ends a 4-task RL curriculum at 57.29% on AIME, improving across every stage.
• LOCOS isolates the attention heads that synthesize retrieval answers rather than copy them.
• MrFlow delivers 25× speedups on text-to-image diffusion with no retraining.

Each of today’s three AI-research features lands the same shape from a different subfield: a newer, fancier method gets matched or beaten by something simpler or older. PAW ships a natural-language-to-LoRA compiler whose actual novelty is packaging — Sakana’s 2025 Text-to-LoRA already demonstrated the hypernetwork trick. WARP reconstructs a model’s training-domain mix from its weights, and its simulated pseudo-checkpoints beat the real training-trajectory oracle because model-merging smooths away mini-batch noise. And a continual-RL comparison finds plain GRPO outperforming dense self-distillation across a four-task schedule, where the fashionable EMA-teacher variants collapse to 0.00% by task three.

PAW compiles English specs into 23MB swappable LoRAs

Source: hf-daily-papers · published 2026-07-01

TL;DR

• Program-as-Weights compiles a natural-language spec into a 23MB LoRA that a frozen 0.6B interpreter hot-swaps at runtime.
• The 0.6B interpreter hits 73.78% on FuzzyBench vs. 68.70% for Qwen3-32B prompting, at ~50× less memory.
• Sakana’s Text-to-LoRA shipped the same text→adapter hypernetwork trick in 2025, so PAW’s novelty is the packaging.
• FuzzyBench-10M is LLM-generated and LLM-judged, so the headline gap likely reflects in-distribution fit, not general reasoning.

The compiler/interpreter split

PAW’s framing is the story. A “fuzzy function” — filter these logs by importance, repair this malformed JSON, rank by intent — is compiled once from a natural-language spec into a small LoRA, then executed locally by a tiny frozen model. The heavy lifting is a 4B compiler; the runtime is a 0.6B interpreter (or even GPT-2 124M) that hot-swaps the generated adapter.

flowchart LR
    S[NL spec] --> PC[Pseudo Compiler<br/>Qwen3-4B frozen]
    PC --> PP[Pseudo-program<br/>restatement + I/O examples]
    S --> LC[LoRA Compiler 4B]
    PP --> LC
    LC --> M[LoRA Mapper<br/>+ shared bases]
    M --> W[(23MB LoRA weights)]
    PP --> I[Interpreter 0.6B<br/>frozen]
    W -. hot-swap .-> I
    U[User input] --> I
    I --> O[Output]

The pseudo-program is the discrete half of the artifact; the LoRA is the continuous half. Only the compiler and mapper train — gradients flow back through the frozen interpreter, so the compiler learns to write weights the interpreter can actually run.

What the numbers show

On FuzzyBench, a 0.6B interpreter with PAW weights beats Qwen3-32B prompting by five points (73.78% vs. 68.70%) while fitting in ~430MB of quantized disk. On a MacBook M3 it clocks 31.6 tok/s with a 0.48s load. Under “heavy noise” — typos, grammar errors, ambiguous specs — accuracy drops only ~3.7 points, which the authors credit to the pseudo-compiler denoising intent before the interpreter sees it. The multimodal result is more striking: a Qwen3-VL-4B compiler generates weights for a text-only 0.6B interpreter that outperforms specialized 4B vision models on Circuit (0.274 vs. 0.196) and Chemical (0.414 vs. 0.258) diagram tasks.

Prior art and the “LoRA blob” problem

The paper undersells its lineage. Sakana AI’s Text-to-LoRA already demonstrated a task encoder plus MLP blocks emitting A/B low-rank matrices for a frozen base, matching hand-tuned oracle adapters zero-shot ¹. Hyperformer and HyperLoRA go further back. PAW’s novelty is the developer product — compiler/interpreter metaphor, one artifact per function, GGUF tooling — not the hypernetwork mechanism.

The sharper critique is debuggability. On Hacker News, practitioners argue that if an LLM can compile a spec, it should emit auditable Python — trusting a generated adapter is like “randomly testing FIR coefficients until output looks right” ². Independent reviews of hypernetwork-generated weights describe them as lossy compression, typically 95–98% of a hand-trained adapter with “intruder dimensions” that accelerate forgetting ³. ALCHEmist — the symbolic-code baseline PAW compares against — is more brittle under noise but genuinely inspectable ⁴; the paper treats the trade-off as settled, and it isn’t.

Then there’s the benchmark. FuzzyBench-10M was generated by “gpt-5.2” and verified by two-LLM agreement. Recent work on synthetic evals warns this is a hall of mirrors: same-family generator-and-judge setups inflate scores that don’t transfer ⁵.

Where PAW actually wins

Deployment economics. A frozen 23MB .paw artifact per function, running locally, means sensitive logs and PII never leave the host — a real EU AI Act story — and it dodges the vendor-API drift that quietly breaks production prompts ⁶. Whether or not the hypernetwork trick generalizes beyond FuzzyBench, “one compiled adapter per fuzzy function, 30 tok/s on a laptop” is a genuinely new ops pattern. That’s the claim worth watching, not the 73.78%.

---

WARP infers training-data mixes from released weights

Source: hf-daily-papers · published 2026-07-01

TL;DR

• WARP reverse-engineers domain mixtures from a base + fine-tuned checkpoint pair, hitting 0.046 MAE on BERT and 0.104 on GPT-2-Small.
• The method fabricates pseudo-checkpoints via model merging (LERP, SLERP, TIES) — no access to the real training trajectory needed.
• Simulated trajectories beat the oracle using real checkpoints (0.047 vs 0.074 MAE on BERT), because merging smooths away mini-batch noise.
• Results are confined to BERT and GPT-2-Small on 3–10 domain taxonomies, well short of forensic-grade evidence against a frontier LLM.

The setup: weights leak more than you think

Modern foundation-model releases hand you the finished parameters but keep the data recipe locked up. WARP, out of UW–Madison’s SprocketLab, argues that recipe leaks anyway — the geometry between a base model $\theta_{\text{base}}$ and its fine-tuned descendant $\theta_{\text{ref}}$ encodes which domains did the pulling, and in what proportion.

The pipeline has three moves. First, interpolate between the two checkpoints to fabricate $T$ pseudo-checkpoints along a plausible training path. Second, on a small probing set (2,500 examples per candidate domain), compute how each domain’s gradient aligns with the trajectory direction — a modified Mimic Score — and stack the results into a $K \times T$ footprint matrix. Third, map that footprint to a mixture, either with a parameter-free softmax readout or a supervised 2-layer MLP trained on synthetic fine-tunes with known mixtures.

On four classification datasets (SNLI, AGNews, Yelp, Yahoo) treating classes as domains, the best configuration — Supervised TIES — hits 0.046 mean absolute error on BERT and 0.104 on GPT-2-Small. The paper’s headline comparison is against sample-level Membership Inference (0.063 and 0.141 respectively).

The counterintuitive result

The most interesting finding isn’t the MAE number, it’s that fake trajectories beat real ones. When the authors swapped in actual intermediate checkpoints as an “Oracle” baseline, performance got worse — 0.074 MAE versus 0.047 for Supervised LERP on BERT. Independent explainers corroborate the explanation ⁷: real SGD paths are noisy from stochastic batching and LR schedules, while LERP/TIES trajectories provide a smoothed geodesic that’s easier for the MLP to read.

The robustness numbers also hold up across training regimes — 9-epoch short runs (0.130 MAE), 12-epoch converged (0.147), 18-epoch overtrained (0.124) — suggesting the geometric footprint is a stable artifact of the mixture, not of when you stopped.

Where the framing gets soft

Two caveats deserve to sit next to the results. First, the MI baseline WARP beats is a weaker target than the paper implies. NeurIPS 2024 work ⁸ showed that blind baselines — which never query the target model at all — often match state-of-the-art MIAs, because “successful” membership inference frequently detects distribution shift rather than genuine memorization. Mixture recovery is a different task, but “outperforms MI” is a low bar in 2026.

Second, WARP isn’t alone. LLMSurgeon ⁹ attacks the same problem black-box, inverting a calibrated confusion matrix on generated text, and reports ~95% coarse overlap on LLMScan. The head-to-head comparison isn’t in either paper.

Stakes: audit tool or exfiltration primitive?

Coverage frames WARP two ways at once: a lever against “access asymmetry” for researchers auditing proprietary systems ¹⁰, and a mechanism for competitors to clone hard-won data-curation IP ¹¹. Any legal deployment collides with proposed Federal Rule of Evidence 707 ¹², which would require machine-generated inferences to meet Daubert-style error-rate and peer-review standards — a threshold the Advisory Committee has deferred voting on precisely because black-box methods resist cross-examination. A 0.046 MAE on BERT is a proof of concept; it is not evidence a court will accept against GPT-5. The scaling question — gradients across pseudo-checkpoints for a frontier-scale model — is the one the paper explicitly punts on.

---

GRPO beats dense self-distillation on continual retention

Source: hf-daily-papers · published 2026-07-01

Source: hf-daily-papers · published 2026-07-01

TL;DR

• SDPO with a frozen EMA teacher hits 84.29% on Math500 after single-domain training — 7 points above GRPO.
• A 5% EMA variant collapses to 0.00% on every benchmark after the third sequential task.
• GRPO ends the four-task curriculum at 57.29% on AIME, improving across stages instead of forgetting.
• A restart-and-freeze teacher (StableSDPO) recovers AIME to 55.00% from a matched-variant collapse.

The claim: density is the bug, not the feature

Through late 2025 the practitioner line — anchored by Thinking Machines Lab’s widely circulated post — treated on-policy distillation as a “cheap and powerful” middle path between SFT and RL, with dense per-token teacher rewards seen as strictly better than sparse outcome rewards ¹³. “Denser ≠ Better” is the sharpest counter to that story. The authors agree SDPO wins on immediate specialization, but argue the same signal density that accelerates learning is what drives parameter drift and continual-learning collapse. The formal contribution is an excess-KL-drift analysis: privileged teacher projections provably induce more drift than sequence-level rewards, which directly raises the risk of forgetting.

The numbers are stark

On MATH alone, SDPO with a frozen EMA teacher (SDPO₀) lifted a Qwen3-4B base from 32.71% to 56.42% on AIME and from 73.85% to 84.29% on Math500, comfortably beating GRPO’s 44.67 / 76.84. Then the authors ran the four-stage MATH→SCIENCE→TOOLUSE→CODING curriculum. That same SDPO₀ variant’s MATH score fell back to 34.38% — essentially base-model performance — by the end of the CODING stage. Push the teacher freshness up to a 5% EMA rate (SDPO₅%) and the model collapses to 0.00% on every benchmark after stage three. GRPO on the same sequence held its earlier gains and finished at 57.29% on AIME.

SVD analysis explains the gap: SDPO produces much larger spectral shifts and subspace rotations in the weight matrices, and its response embeddings drift across the manifold rather than staying clustered. The teacher, being a slightly-lagged copy of the student, amplifies whatever formatting artifacts the student already emits — think endlessly repeating \boxed tokens — into a self-reinforcing feedback loop.

Independent corroboration

The failure mode is not isolated. A concurrent April 2026 paper, Rethinking On-Policy Distillation, reports up to a 17% relative accuracy drop on AIME 2024/2025 under privileged self-distillation and pins it on “prefix drift” — the teacher’s token-level rewards degrade as trajectories lengthen ¹⁴. That work adds a second axis the SDPO-CL authors don’t emphasize: thinking-pattern consistency. A weaker teacher whose top-k distribution overlaps the student can beat a stronger but stylistically mismatched one ¹⁵. Two papers, two mechanistic angles, same conclusion.

Where the fixes land

StableSDPO — periodically copy the student’s weights into the teacher, then freeze — is the paper’s own remediation, and the recovery from 34.38% to 55.00% on AIME under a matched 5% schedule is real ¹⁶. Amazon’s PGPO takes a different route, using a mutual-information estimate to modulate step sizes and dampen late-training collapse ¹⁷. Neither closes the retention gap with GRPO. The honest read: on-policy distillation is a narrow-specialization tool that needs guardrails, not the new post-training default the mid-2025 discourse suggested ¹⁸. If your pipeline runs multiple sequential fine-tuning stages, sequence-level rewards remain the safer primitive.

Round-ups

LOCOS pinpoints attention heads behind non-literal retrieval

Source: hf-daily-papers

Logit-Contribution Scoring measures how each attention head’s output-value circuit feeds answer-token logits, isolating the heads that synthesize context rather than copy it. The method beats prior attribution techniques on retrieval benchmarks including MuSiQue and BABILong, and ships with open code.

Transfer-Aware Curriculum picks RLVR domains that generalize

Source: hf-daily-papers

TAC runs a bandit-style online curriculum over reasoning domains, using gradient-geometry alignment to estimate which training mixes transfer broadest. Prioritizing high-transfer domains during GRPO lifts macro-averaged accuracy across held-out tasks versus uniform sampling.

AutoMem trains LLMs to manage their own memory

Source: hf-daily-papers

AutoMem treats memory as a learnable cognitive skill, jointly optimizing both the memory structure and the agent’s proficiency at file-system-style operations over it. The framework delivers sizable gains on long-horizon agent trajectories where fixed memory schemes degrade.

AgenticSTS benchmarks long-horizon agents inside Slay the Spire 2

Source: hf-daily-papers

AgenticSTS enforces a bounded contract: each turn reassembles a fresh prompt from typed retrieval over explicit memory layers, letting researchers ablate individual memory components. Tested on Slay the Spire 2, isolating strategic-skill memory measurably improves conditional reasoning.

FlashMorph converts Transformers to hybrid attention under a budget

Source: hf-daily-papers

FlashMorph frames the choice of which layers keep full attention versus swap to linear attention as a budget-constrained subset optimization, solved with morphable layerwise gates and linearization regularization. The resulting hybrids retain long-context retrieval accuracy while cutting inference cost.

MrFlow speeds text-to-image diffusion 25x without retraining

Source: hf-daily-papers

MrFlow generates at low resolution, upsamples in pixel space with a pretrained GAN, then re-injects noise for a short high-resolution refinement pass. The staged pipeline exploits quadratic token savings to reach 25x speedups on flow-matching models with no training or runtime changes.

Task-agnostic inverse-dynamics pretraining boosts VLA robots

Source: hf-daily-papers

The two-stage recipe first learns physical competence via self-supervised inverse dynamics on unlabeled robot data, then adds lightweight language grounding. On the SIMPLER benchmark and WidowX hardware, it beats behavior cloning while using far fewer expert demonstrations.

Footnotes

1. Sakana AI — Text-to-LoRA — https://sakana.ai/doc-to-lora/

   Text-to-LoRA (T2L) uses a task encoder to extract vector representations from text descriptions, which are then passed through MLP blocks to generate the A and B low-rank matrices for a base model… matching or exceeding the performance of manually fine-tuned ‘oracle’ adapters in zero-shot settings.

   ↩

2. Hacker News thread #48774368 (user mathisfun123) — https://news.ycombinator.com/item?id=48774368

   compared to randomly testing coefficients in signal processing without a clear understanding of the resulting filter’s behavior… a LoRA blob remains an opaque behavioral artifact where the developer ‘doesn’t even know what they are getting out of it’

   ↩

3. gopubby.com — LoRA generation review — https://ai.gopubby.com/we-can-now-generate-loras-in-less-than-1-minute-94736f6079de

   T2L’s weight generation is often described as ‘lossy compression’; it can maintain the performance of the adapters it was trained to reconstruct, but its ability to generalize to entirely unseen tasks remains a point of academic debate… often results in slightly lower downstream accuracy than a model fine-tuned on a specific dataset.

   ↩

4. Burnyverse survey — Neurosymbolic AI — https://burnyverse.com/Exocortex/Artificial+Intelligence

   ALCHEmist achieves performance comparable to or better than large-scale LLM annotators while reducing total labeling costs by a factor of approximately 500x… standard symbolic code can become brittle compared to newer neurosymbolic methods that treat programs as differentiable weights.

   ↩

5. arXiv 2510.13259 — synthetic benchmark validity — https://arxiv.org/html/2510.13259v1

   static, model-generated datasets often fail to capture the ‘messiness’ and unpredictable edge cases of real-world production environments… using the same LLM to both generate the benchmark and perform the task can lead to overinflated accuracy scores that do not translate to general capabilities.

   ↩

6. Hyperion Consulting — AI Research Decoded — https://hyperion-consulting.io/en/insights/ai-research-decoded-from-fuzzy-logic-to-autonomous-agents-the-next-wave-of-deployable-ai

   Because inference is local, sensitive data (such as internal logs or user documents) never leaves the host environment, aligning with high-risk compliance standards like the EU AI Act… a compiled PAW artifact is a frozen, deterministic binary that remains stable over time.

   ↩

7. Emergent Mind — WARP paper explainer — https://www.emergentmind.com/papers/2607.01686

   Simulated trajectories often perform better than the true training trajectory… merging-based paths provide a smoothed version of the transition, making it easier for the MLP to extract the underlying signal.

   ↩

8. NeurIPS 2024 poster — ‘Blind Baselines Beat Membership Inference Attacks’ — https://neurips.cc/virtual/2024/poster/95944

   Blind baselines—which simply look at data distributions without querying the target model—can often outperform state-of-the-art MIAs, suggesting many claimed MIA successes are actually detecting distribution shifts rather than true membership.

   ↩

9. arXiv — LLMSurgeon (Luo et al., 2605.30348) — https://arxiv.org/abs/2605.30348

   LLMSurgeon estimates a calibrated soft confusion matrix on held-out reference data and solves a constrained inverse linear problem to de-blur biased predictions, recovering domain proportions from generated text alone.

   ↩

10. AI Weekly — ‘UW-Madison’s WARP recovers training data mixtures from weights’ — https://aiweekly.co/alerts/uw-madisons-warp-recovers-training-data-mixtures-from-weights

    The framework addresses ‘access asymmetry’ where researchers study public models but remain blind to proprietary data distributions—information effectively teleported into the released weights.

    ↩

11. AI Native Foundation Daily Digest (2026-07-03) — https://ainativefoundation.org/ai-native-daily-paper-digest-20260703/

    This transparency is a double-edged sword: while it aids in accountability, it also allows competitors to clone high-performing data recipes, potentially eroding the competitive advantage of organizations that invest heavily in data curation.

    ↩

12. Minnesota Journal of Law, Science & Tech — Proposed FRE 707 — https://journals.law.umn.edu/mjlst/2026/01/26/proposed-rule-of-evidence-707-machine-experts

    Any inference that a model was trained on a specific dataset must be supported by a demonstrable error rate and peer-reviewed methodology; the Advisory Committee deferred a final vote citing the black-box nature of AI as a hurdle to cross-examination.

    ↩

13. Medium explainer of Thinking Machines Lab’s on-policy distillation post — https://medium.com/@maheshlambe/mira-murati-thiking-machines-on-policy-distillation-cheap-accuracy-real-gains-85ac523b20f7

    On-policy distillation is framed as a ‘cheap and powerful’ middle path between SFT and RL, providing dense per-token feedback on student-generated trajectories to avoid the sparse rewards that make standard RL expensive and unstable.

    ↩

14. Rethinking On-Policy Distillation (arXiv 2607.06855) — https://arxiv.org/html/2607.06855v1

    In privileged self-distillation settings, researchers observed a relative accuracy drop of up to 17% on AIME 2024 and AIME 2025 across multiple models… as the reasoning trace grows, the ‘prefix drift’ between student and teacher increases, making the teacher’s token-level rewards less reliable.

    ↩

15. chrisliu298/awesome-on-policy-distillation (GitHub) — https://github.com/chrisliu298/awesome-on-policy-distillation

    Success depends strictly on ‘thinking-pattern consistency’ between teacher and student; if their underlying logic differs, the distillation often fails… a teacher with lower benchmark scores but higher initial thinking-pattern overlap can be more effective than a ‘smarter’ teacher that thinks differently.

    ↩

16. alphaXiv discussion of Denser ≠ Better (StableSDPO results) — https://www.alphaxiv.org/abs/2607.01763

    On the AIME benchmark, a standard SDPO model with a 5% EMA rate collapsed to 34.38% performance; however, replacing it with a matched ‘restart-and-freeze’ strategy restored performance to 55.00%.

    ↩

17. Amazon Science — Physics-Guided Policy Optimization with Self-Distillation — https://www.amazon.science/publications/physics-guided-policy-optimization-with-self-distillation

    PGPO uses a mutual-information estimate to modulate step sizes, preventing the late-training collapse observed in vanilla SDPO.

    ↩

18. ICML 2026 poster — Self-Distillation Policy Optimization (LAS group) — https://icml.cc/virtual/2026/poster/65174

    SDPO formalizes self-distillation as a process where a model acts as its own probabilistic oracle by conditioning on rich textual feedback, converting outcome-based rewards into granular next-token predictions.

    ↩