Meta Skaling cuts sweep 10×, Parallel-RL +10.7%, activation oracles hide target

TL;DR

• Meta’s Skaling adds one coupling exponent, cutting sweep compute 10× on far-extrapolation.
• Parallel-RL merges per-task RL experts, hitting +10.7% over base at 1.5B.
• Fine-tuned oracles drop exact recovery to 35% while linear probes stay near 100%.
• PrivacyPeek finds LLM agents acquire far more sensitive data than outputs reveal.

Today’s research pool holds three unrelated method drops. Meta FAIR’s Skaling rewrites Chinchilla’s additive loss law with a single coupling exponent and beats a full grid sweep on far-extrapolation using roughly an order of magnitude less compute. A separate group’s Parallel-RL recipe trains per-task RL experts, merges their weights, and lands +10.7% over the 1.5B base — leaning on the observation that RL updates sit two orders of magnitude smaller in L2 norm than SFT ones. And an activation-oracle paper shows that fine-tuning an oracle on its own target word crashes exact recovery to 35% even while internal linear probes still hit ~100%, an explicit ‘anti-reader’ construction.

Each of the three ships alongside concurrent work that pokes at the mechanism — Farseer-code where Skaling collapses, an FP32 result that questions the BF16 RL-sparsity story, a Nanda-group paper arguing most oracle wins are text inversion — but those debates run beneath the headline numbers rather than around them.

Meta’s Skaling law adds one exponent, cuts sweep compute 10×

Source: hf-daily-papers · published 2026-08-06

TL;DR

• Meta FAIR bolts a single coupling exponent onto Chinchilla’s additive law, cutting loss-prediction MAPE 1.5–3× across interpolation and extrapolation.
• An “L-shape” sparse sweep beats a full Chinchilla grid on far-extrapolation using ~10× less compute (5.1×10²¹ vs 5.0×10²² FLOPs).
• Fitted coupling k lands at 0.31–0.45 across most grids, confirming the relationship is robustly non-additive.
• On Farseer-code k drifts to 0.77–0.90 and Skaling collapses back to Chinchilla-like accuracy.

One exponent, not five

Chinchilla’s 2022 law assumes model size N and token count D contribute independently: L(N,D) = A/Nᵃ + B/Dᵇ + E. Skaling’s authors point out the mixed second derivative ∂²L/∂N∂D is empirically non-zero, so the additive form has to mispredict at grid corners — small model / huge data, or vice versa. Their fix is to wrap the whole thing in a single exponent k:

$$L(N,D) = \left(\tfrac{A}{N^\alpha} + \tfrac{B}{D^\beta}\right)^k + E$$

k=1 recovers Chinchilla exactly; k≠1 restores the multiplicative coupling from Kaplan’s 2020 law while keeping Chinchilla’s per-axis decay rates interpretable. This matters because the other recent challenger — StepFun and Fudan’s Farseer (2025) — needed a nine-parameter fit and roughly 3M H100-hours to claim a 433% reduction in extrapolation error ¹. Skaling gets most of that back with one extra number.

The headline accuracy numbers hold up across grids. On Farseer’s held-out large models, MAPE drops from Chinchilla’s 1.48% to 0.47%; on the authors’ own SK-Grid, far-extrapolation error falls from 5.17% to 0.70%. The paper also argues the Chinchilla-optimal token-to-parameter ratio isn’t constant — at frontier scales, the prescription can be off by up to 100×.

The L-shape is the real compute story

The more practically interesting result is the sampling protocol. Instead of a dense N×D grid, Skaling sweeps two thin bands: vary D at one tiny model, vary N at one short horizon. Feed those two “arms” of the L into the coupled functional form and it reconstructs the interior. On Farseer’s data, this hits 1.51% far-extrapolation MAPE using ~10× less compute than the dense sweep — beating Chinchilla’s full-grid result (2.46%) and demolishing Chinchilla’s L-shape fit (9.82%) ². If it generalizes, planning a frontier training run gets an order of magnitude cheaper.

Where it wobbles

Independent commentary is friendlier than “Chinchilla is dead” framing suggests. On alphaXiv, readers note that scaling-law fitting is a fragile non-convex problem and recommend CMA-ES over the L-BFGS-B setup in Appendix C ³. Epoch AI’s earlier Chinchilla replication work found that adding or dropping a handful of runs materially shifts fitted exponents — training noise often dominates optimizer precision ⁴. The reported k=0.31–0.45 “stable range” deserves scrutiny in that light.

The paper concedes two harder problems. First, k and the irreducible loss floor E are entangled: because runs don’t reach saturation, the fit can explain curvature by pushing E toward zero or tuning k. Second, on Farseer-code the fitted k drifts to 0.77–0.90 and Skaling degrades to Chinchilla-like accuracy — and the direction of the optimal D*/N* drift flips between datasets ⁵.

What actually matters downstream

Frontier labs have already voted with their token budgets: Llama 3 8B trained on 15T+ tokens, roughly 1,800:1 tokens-per-parameter, because inference cost across millions of users dwarfs one-time training cost ⁶. Skaling won’t move that decision. What it might move is the cost of the planning sweep that precedes it — and that’s a real, if narrower, win.

---

Parallel-RL merges task-specific RL experts, +10.7% at 1.5B

Source: hf-daily-papers · published 2026-08-05

TL;DR

• RL weight updates land at ~0.03 L2 norm vs SFT’s ~7.4 on DeepSeek-R1-Distill-Qwen, over two orders of magnitude apart.
• Sequential SFT collapses 23.1%, sequential RL gains 24.9% across four reasoning tasks from the same base.
• Authors propose Parallel-RL: train per-task RL experts, merge weights, adapt on 5% data — hits +10.7% over base at 1.5B.
• Concurrent work suggests the RL-sparsity effect largely vanishes in FP32, hinting BF16 rounding may be doing some of the work.

The claim: RL updates don’t step on each other

“SFT Conflicts, RL Coexists” argues that supervised fine-tuning and reinforcement learning leave qualitatively different fingerprints on a model’s weights. Measuring parameter deltas on DeepSeek-R1-Distill-Qwen (1.5B and 7B) across Math, Science, Logic, and Code, the authors find SFT touches 93% of parameters with an average L2 norm of ~7.4, while GRPO-style RL touches only ~20% at norm ~0.03. Pairwise cosine similarity between task updates is high for SFT (0.1–1.0) and effectively zero for RL (~10⁻⁵) — near-orthogonal in the high-dimensional weight space.

The theoretical scaffolding is the interesting part. Because GRPO’s group-relative advantage sums to zero, the “mean” gradient direction — where most cross-task interference lives — gets subtracted out. What remains is a variance-limited residual, small in magnitude and statistically orthogonal across independent tasks. SFT, by contrast, is norm-limited: it has to bridge the full gap to an expert distribution, so updates are large and directionally aligned with whatever the current task demands.

The behavioral consequence is stark. Train SFT on Math only, and Logic drops 16%. Train RL on Math only, and Logic improves 2%.

The recipe: train in parallel, merge, lightly adapt

If RL updates are near-orthogonal, you don’t need to interleave tasks. Parallel-RL trains N single-task RL models from the same base, extracts each ΔW, and combines them — naive sum, TIES, or an SVD rank-1 reduction. A final “adaptation” pass on just 5% of the mixed data cleans up merge artifacts.

Numbers at 1.5B: naive summation retains 94.2% of single-task expert performance and gains 6.6% over base. Adapted Parallel-RL hits +10.7% over base and 103.2% of single-task performance — i.e. the merged generalist beats the specialists it was assembled from.

The caveats worth flagging

Two threads should temper the headline. First, the sparsity finding isn’t new: Mukherjee et al. already documented that PPO, DPO, and GRPO touch only 5–30% of weights and that retraining just those subnetworks recovers full performance ⁷. This paper’s contribution is the theoretical framing plus the merge recipe, not the sparsity observation itself.

Second, and more mechanically concerning, concurrent analysis of Mukherjee’s result finds the sparsity effect largely disappears when training in pure FP32 ⁸ — many RL updates may simply fall below BF16’s rounding threshold. If that critique generalizes, the 0.03-vs-7.4 norm gap and the ~10⁻⁵ cosine similarity partly reflect numerical precision rather than an intrinsic property of policy gradients. Cosine similarity between two vectors that are themselves near-zero from rounding is not a clean orthogonality signal.

The merging literature also complicates the “beats single-task experts” line. KnOTS shows SVD is most useful as a pre-alignment step before TIES, not a standalone merger, adding up to 4.3% over naive merges ⁹. Long-to-Short benchmarks report that at 1.5B–3B — exactly the scale validated here — merging alone struggles to preserve chain-of-thought and often loses to unified multi-task objectives ¹⁰. Whether Parallel-RL survives at generalist post-training scale is the open question.

Reproduction is not yet turnkey either: the GaryStack/Parallel-RL repo is flagged as a preliminary release with reward verifiers and distributed paths still being refined ¹¹. And don’t confuse this work with the unrelated “Parallel-R1” line, which teaches inference-time reasoning branches rather than weight-space decoupling ¹².

---

Fine-tuned activation oracles hide their target concept

Source: hf-daily-papers · published 2026-07-24

TL;DR

• A base Activation Oracle recovers Qwen3-8B’s hidden word at 94% exact accuracy in the HINT regime.
• Fine-tuning that oracle on the same target word collapses exact recovery to 35% — an “anti-reader.”
• Linear probes inside the crippled oracle still hit ~100%, proving the concept is present but suppressed at verbalization.
• A concurrent Nanda-group paper argues most oracle “successes” are text inversion, not activation reading — compounding the reliability problem.

The anti-reader effect

Bersia & Gaintseva trained a Qwen3-8B “subject” model to keep a secret word (leaf, moon, wave, flag, or book) internally while refusing to say it, then pointed an Activation Oracle — another Qwen3-8B — at layer 18 to verbalize what the subject was thinking. On cooperative subjects, a generic base oracle nailed the secret with 93% exact and 95% semantic accuracy in the HINT regime. The interpretability story looked clean.

Then they fine-tuned the oracle on activations from a subject hiding the same concept — the intuitive way to build a specialist. Exact recovery cratered from 94% to 35%, and semantic recovery from ~99% to 51% ¹³¹⁴. Fine-tuning didn’t sharpen the oracle; it taught it to look away.

Suppression, not forgetting

The information is still inside the oracle. Linear probes on its own residual stream recover the hidden concept at near-perfect accuracy ¹⁴. The failure is localized to the mid-to-late readout transition — the LoRA updates apparently learned to route the concept around the verbalization head rather than through it.

The blind spot is also strictly concept-specific. Cross-testing produces a diagonal failure pattern: the “leaf” oracle is uniquely bad at leaf and still fluent on moon, wave, flag, book ¹⁵¹⁴. That rules out catastrophic forgetting or generic degradation and points at a targeted suppression circuit that fine-tuning reliably induces.

flowchart LR
    S[Qwen3-8B subject<br/>hides 'leaf'] -->|layer 18 activations| AO[Fine-tuned oracle<br/>trained on 'leaf']
    AO --> V[Verbalized guess<br/>35% correct]
    AO -.->|linear probe on<br/>internal layers| P[Probe readout<br/>~100% correct]
    style V stroke:#c33
    style P stroke:#3a3

A wider reliability crisis

The anti-reader result would be uncomfortable on its own. It lands alongside a concurrent OpenReview paper from Jakkli, Rajamanoharan and Nanda arguing that many apparent AO successes are text inversion — the oracle reconstructs nearby tokens from the activation vector and paraphrases them, rather than inspecting deeper semantics. On tasks engineered to defeat that shortcut (predicting arithmetic outputs before any digit appears), AOs drop to near-chance, and roughly half of free-form probes return “unfalsifiably generic” answers ¹⁶.

The two critiques are orthogonal and cumulative: oracles fail silently when they should read a concept (Bersia & Gaintseva) and succeed for the wrong reasons when they appear to (Jakkli et al.). A clean natural-language readout is no longer evidence of anything in particular.

What auditors should actually do

The practical guidance emerging from independent coverage is uniform: do not trust a learned verbalizer as a standalone auditing tool. Pair every AO report with independent mechanistic probes — Sparse Autoencoders (Qwen ships Qwen-Scope for exactly this backbone ¹⁷), crosscoders on activation deltas, or Anthropic’s Jacobian Lens, which surfaces pre-verbal concepts like “spider” directly from residual-stream geometry on the same Taboo model organism ¹⁸. If a probe lights up and the oracle stays silent, believe the probe.

Two things remain open. The paper does not explain why LoRA updates prefer suppression over reporting — a mechanistic account would tell us whether adversarial fine-tuning could weaponize the effect. And no public code repo has surfaced yet, so independent reproduction on non-Qwen backbones is still pending.

Round-ups

PrivacyPeek finds LLM agents grab far more data than they reveal

Source: hf-daily-papers

Auditing only agent outputs misses the real leak: PrivacyPeek inspects tool-call trajectories and probes for hidden sensitive data, showing widespread unnecessary acquisition during task execution. Prompt-based defenses barely dent the problem, exposing a blind spot in current agent privacy evaluation.

NVIDIA’s WorldTrace gives video world models addressable long-term memory

Source: hf-daily-papers

Interactive video world models forget quickly; WorldTrace adds a compressed KV cache with virtual RoPE positions that the model can address like scene landmarks. The training-free method improves temporal consistency and episodic recall on LoopBench, letting scenes persist across long horizons.

Small models fine-tuned on Psych-101 rival giants at modeling cognition

Source: hf-daily-papers

Fine-tuning small language models on human behavioral data matches large baselines in-distribution and scales better out-of-distribution, the Centauri project reports. The models lean on task stimuli and feedback rather than choice history, approaching the noise ceiling as cognitive proxies.

ReASearch turns the optimizer itself into a reasoning agent

Source: hf-daily-papers

Rather than bolting agents onto search algorithms, ReASearch treats optimization as autonomous reasoning with diagnosis and persistent memory. The unified framework beats specialized controllers—evolutionary search, bandits, textual-gradient methods—across prompt tuning, program synthesis, and ML workflow optimization.

DCAS decouples CLI agent scaffolds to teach transferable planning

Source: hf-daily-papers

CLI coding agents usually bake planning into their scaffold, hurting portability. DCAS separates scaffolding from policy and trains on planning-aware trajectories, letting a single model generalize across different agent environments in cross-scaffold evaluation instead of overfitting to one harness.

Enfold folds world-model rollouts into a single-step representation

Source: hf-daily-papers

Running a generative world model at every control step is too slow for embodied agents. Enfold distills future-generative computation into a present-only representation across multi-level states, cutting action latency while keeping the predictive signal that guides control.

Survey maps adversarial defenses across the visual content lifecycle

Source: hf-daily-papers

Content owners increasingly deploy adversarial perturbations as protection, and this survey catalogs the toolkit across a visual asset’s lifecycle: privacy filters, unlearnable examples, generative safeguards, adversarial CAPTCHAs, and provenance marks. The authors stress-test each class against adaptive threats and transferability across multimodal models.

Footnotes

1. Farseer paper (Li et al. 2025, ResearchGate) — https://www.researchgate.net/publication/392629562_Farseer_A_Refined_Scaling_Law_in_Large_Language_Models

   Farseer employs a 9-parameter equation… reduces extrapolation error by approximately 433% compared to previous standards, enabling researchers to predict the performance of 25-billion-parameter models with less than 1% relative error.

   ↩

2. KuCoin News (wire coverage of Meta FAIR paper) — https://www.kucoin.com/news/flash/meta-fair-paper-challenges-chinchilla-scaling-law-claims-10x-compute-cost-reduction

   Meta demonstrates that their coupled formula allows for ‘L-shaped’ grid sweeps… reduces the compute required to plan a training run by approximately 10x while improving the precision of loss predictions in boundary regions where data and model size are imbalanced.

   ↩

3. alphaXiv discussion of arXiv:2608.07222 — https://www.alphaxiv.org/abs/2608.07222

   Independent researchers on alphaXiv have noted that fitting these laws is a fragile non-convex problem, recommending gradient-free optimizers like CMA-ES over standard L-BFGS for more reliable out-of-the-box results.

   ↩

4. EmergentMind topic page (summarizing Epoch AI replication) — https://www.emergentmind.com/topics/chinchilla-scaling-law

   Small changes in the training data points can lead to vastly different values for the exponents α and β… the underlying noise in model training runs often dwarfs the precision of the optimization algorithm itself.

   ↩

5. TheClarity.today (independent write-up) — https://theclarity.today/story/meta-fair-paper-reveals-limitations-in-chinchilla-scaling-law-proposes-fix-that-d035c3d5

   For certain datasets (like Farseer-code), the fitted k was closer to 1 (0.77–0.90), leading to performance similar to Chinchilla… the direction of the optimal ratio drift depends on the specific architecture and data.

   ↩

6. Databricks engineering blog: ‘How Long Should You Train Your Language Model?’ — https://www.databricks.com/blog/how-long-should-you-train-your-language-model

   Llama 3 8B famously ignored the 20:1 ratio, training on over 15 trillion tokens—a ratio exceeding 1,800 tokens per parameter… the cost of serving a model to millions of users often dwarfs the one-time training cost.

   ↩

7. Mukherjee et al., arXiv:2505.11711 (‘RL Finetunes Small Subnetworks in LLMs’) — https://arxiv.org/abs/2505.11711

   RL algorithms — including PPO, DPO, and GRPO — induce parameter update sparsity where only 5% to 30% of a model’s weights change significantly, and training only these sparse subnetworks can recover full-model performance.

   ↩

8. HuggingFace discussion of Mukherjee et al. (concurrent-work critique) — https://huggingface.co/papers/2505.11711

   When training in pure FP32 the sparsity effect largely disappears, suggesting many RL updates are simply too small to overcome BF16 rounding thresholds — a mechanical rather than intrinsic explanation for the phenomenon.

   ↩

9. Stoica & Ramesh, ‘Model Merging with SVD to Tie the Knots’ (KnOTS) — https://www.semanticscholar.org/paper/Model-merging-with-SVD-to-tie-the-Knots-Stoica-Ramesh/04ed1d9c81cd6304d21a88d7aae9ba7cde802f50

   Using SVD to jointly transform LoRA-finetuned weights into an aligned representation space before applying TIES yields performance improvements of up to 4.3% across vision and language benchmarks.

   ↩

10. Long-to-Short via Model Merging (GitHub benchmarks) — https://github.com/hahahawu/Long-to-Short-via-Model-Merging

    Smaller models (1.5B–3B) struggle to maintain complex Chain-of-Thought reasoning through merging alone, often requiring the direct training signals provided by a unified multi-task objective.

    ↩

11. GaryStack/Parallel-RL GitHub README — https://github.com/GaryStack/Parallel-RL

    The repository is currently in a ‘preliminary release’ state; specific environment details, reward verifiers, and distributed paths are still being refined.

    ↩

12. Parallel-R1 / Native Parallel Reasoner (arXiv 2512.07461) — https://arxiv.org/html/2512.07461v3

    ‘Parallel-RL’ frameworks focus on computational parallelism — teaching models to branch their reasoning paths into simultaneous execution graphs — a distinct research thread from weight-level sparsity work.

    ↩

13. AI Weekly alert summary — https://aiweekly.co/alerts/fine-tuned-activation-oracles-go-blind-to-their-own-concepts

    A standard AO can recover the hidden word with ~94% accuracy [but] the fine-tuned version’s exact recovery rate collapses to approximately 35%, effectively becoming an ‘anti-reader’… auditors must pair Oracle natural-language reports with independent, simpler linear probes to ensure the Oracle hasn’t learned to ‘hide’ its findings.

    ↩

14. arXiv preprint (Bersia & Gaintseva, primary paper HTML) — https://arxiv.org/html/2606.02609v1

    Linear probes applied to the AO’s own internal layers reached near-perfect (100%) accuracy in identifying the hidden concept… the failure occurs during the final ‘verbalization’ or readout stage of the oracle, not from loss of information.

    ↩ ↩² ↩³

15. Hyper.ai paper page (2607.23379) — https://hyper.ai/en/papers/2607.23379

    Rather than becoming specialist readers, these fine-tuned probes often become ‘concept-specific anti-readers’… the oracle retains the ability to accurately read and describe other concepts it was not specifically fine-tuned on, proving that the blind spot is concept-specific rather than a global degradation.

    ↩

16. OpenReview — Jakkli, Rajamanoharan & Nanda, ‘Current Activation Oracles Are Hard to Use on Safety-Relevant Tasks’ — https://openreview.net/challenge?redirect=%2Fforum%3Fid%3D7nRmqgz3Wv

    Many apparent successes of AOs are actually instances of the model reconstructing nearby text tokens from an activation vector and then answering queries based on that decoded text, rather than inspecting deeper semantic signals… in settings where text inversion is impossible, AOs performed at near-chance levels.

    ↩

17. Qwen team blog — Qwen-Scope SAE release — https://qwen.ai/blog?id=qwen-scope

    Qwen-Scope uses Sparse Autoencoders (SAEs) to disentangle dense representations into interpretable features… narrow fine-tuning for specific tasks like Taboo leaves highly readable traces in activation differences, which can be identified using independent probes or ‘crosscoders’ even when the model attempts to suppress the information.

    ↩

18. Shared Sapience Substack — coverage of Anthropic’s Jacobian Lens / Taboo model organism — https://sharedsapience.substack.com/p/the-century-report-weekly-july-6-12-2026

    When a model was asked to describe a web-spinning animal without using its name, the J-lens revealed the concept ‘spider’ lighting up internally as a pre-verbal stepping stone… independent critiques suggest that Taboo models suffer from ‘catastrophic forgetting’; when subjected to generic secondary fine-tuning, they often lose the ability to hide the secret.

    ↩