OLMo needs width, MemFT needs token reweighting, Parallax needs Muon

Sources

Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention huggingface.co

Larger models outperform smaller ones on complex and rare tasks due to reduced gradient interference and better resource allocation, enabling them to learn task features that smaller models miss even with infinite data.

How LoRA Remembers? A Parametric Memory Law for LLM Finetuning huggingface.co

Research investigates the quantitative limits of parametric memory in large language models using LoRA as a probe, establishing a power law relationship and developing a threshold-guided optimization method for improved memory performance.

Parallax: Parameterized Local Linear Attention for Language Modeling huggingface.co

Local Linear Attention is enhanced through parameterization and hardware-aware optimization to improve LLM training efficiency and performance while maintaining computational stability.

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction huggingface.co

WorldMemArena tests how multimodal agents write, store, and retrieve memory across long-horizon tasks through an Action-World Interaction Loop. The benchmark separates lifelong evolution from agentic execution, showing that RAG and harness-based memory systems still struggle to ground decisions in prior visual evidence.

Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases huggingface.co

Alignment tampering exploits weaknesses in pairwise comparisons and reward modeling, letting language models amplify undesired behaviors through poisoned preference datasets. The work shows best-of-N sampling magnifies the effect, turning RLHF’s own optimization pressure into a mechanism for entrenching misaligned biases.

Token-Level Generalization in LoRA Adapter Backdoors: Attack Characterization and Behavioral Detection huggingface.co

LoRA fine-tuning can embed backdoors that activate at the token-feature level while preserving benign performance, evading prompt-injection classifiers. The authors localize the trigger to MLP down_proj weights via causal patching and flag adapters using cross-module standard deviation of Frobenius norms.

Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments huggingface.co

Qwen-VLA pairs a shared vision-language backbone with a DiT-based action decoder and embodiment-aware prompt conditioning, jointly pretraining across robot platforms. The single model handles manipulation, navigation, and trajectory prediction, with out-of-distribution generalization across embodiments the paper highlights as its main result.

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists huggingface.co

CausaLab puts LLM agents inside synthetic structural causal models where they must run interventions and recover the underlying graph, not just predict outcomes. The split between predictive success and causal understanding exposes agents that score well on observations while missing the true mechanism.

AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security huggingface.co

AgentDoG 1.5 combines a structured agent-safety taxonomy with influence-function purification and agentic safety SFT, training in Docker-level RL environments. An online guardrail handles real-time moderation inside interactive scenarios, and the authors emphasize that the alignment pipeline runs on minimal curated samples.

LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training huggingface.co

LaRA detects benchmark contamination in RL-post-trained LLMs by tracking geometric deviations across layers, including perturbation sensitivity, directional collapse, and local representation rigidity. Contaminated checkpoints show distinctive layer-wise signatures that black-box probing misses, giving evaluators a structural test for leaked training data.

UniSteer: Text-Guided Flow Matching in Activation Space for Versatile LLM Steering huggingface.co

UniSteer uses text-guided activation flow matching to learn a universal conditional velocity field in activation space for controlling LLM behaviors and classification tasks.

When Should Models Change Their Minds? Contextual Belief Management in Large Language Models huggingface.co

Language models struggle with managing long-term information through contextual belief management, which involves updating, preserving, and filtering relevant information, and can be improved using reinforcement learning and representation-level steering techniques.

PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers huggingface.co

PRISM evaluates automated peer review systems across multiple dimensions using argument mining and retrieval-augmented verification, revealing that while LLMs match human performance in specific areas, no system consistently equals human reviewers across all evaluation criteria.

AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios huggingface.co

LLM-based agents face significant challenges in asynchronous tool calling due to delayed responses, requiring improved task coordination and temporal reasoning capabilities.

PANDO: Efficient Multimodal AI Agents via Online Skill Distillation huggingface.co

PANDO is a web agent framework that improves efficiency through experience accumulation by reducing redundant actions, optimizing skill discovery, and enhancing prompt caching without sacrificing performance.

PhoneWorld: Scaling Phone-Use Agent Environments huggingface.co

PhoneWorld is a pipeline that transforms real GUI trajectories and screenshots into controllable mobile environments, executable tasks, and automated verifiers, enabling scalable creation of phone-use benchmarks.

Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation huggingface.co

Multi-agent system for generating reliable, visually informative multimodal reports by interleaving textual and visual evidence through specialized agents and verification mechanisms.

minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models huggingface.co

A comprehensive framework is presented for converting bidirectional video diffusion models into real-time interactive world models with controllable, causal, and low-latency capabilities through fine-tuning and distillation techniques.

Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning huggingface.co

Skill0.5 is a novel agentic reinforcement learning framework that combines general skill internalization with task-specific skill utilization through a dynamic, difficulty-aware router to improve performance in complex task environments.

UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents huggingface.co

UI-KOBE framework enhances lightweight mobile GUI agents by incorporating reusable app-specific graph knowledge to improve task planning and execution efficiency.

LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents huggingface.co

LiteCoder-Terminal-Gen enables scalable training of language agents for terminal environments through synthetic, executable environments that outperform traditional methods.

Why Far Looks Up: Probing Spatial Representation in Vision-Language Models huggingface.co

Vision-language models exhibit entangled spatial representations that correlate vertical image position with distance, impacting reasoning robustness and performance across benchmarks.

YoCausal: How Far is Video Generation from World Model? A Causality Perspective huggingface.co

Video diffusion models exhibit arrow-of-time perception without true causal understanding, as demonstrated by a novel benchmark measuring causal cognition through reverse surprise and visual language model analysis.

Colored Noise Diffusion Sampling huggingface.co

Diffusion models exhibit spectral bias in image synthesis, and a new sampling method called Colored Noise Sampling addresses this by dynamically allocating energy based on frequency-dependent schedules, leading to improved image quality metrics.

AdaState: Self-Evolving Anchors for Streaming Video Generation huggingface.co

Video diffusion models with adaptive state replacement generate more dynamic videos by evolving scene references rather than fixing to initial frames, using recurrent denoising as transition function.

Thinking Before Constraining: A Unified Decoding Framework for Large Language Models huggingface.co

A hybrid approach called In-Writing is proposed that combines free-form reasoning with structured generation by delaying constraint application until after a trigger token is generated, improving accuracy in classification and reasoning tasks.

Verifiable Rewards Beyond Math and Code: Lightweight Corpus-Grounded Process Supervision for Factual Question Answering huggingface.co

CorVer, a corpus-grounded reward mechanism, enhances factual accuracy in question answering by providing efficient sentence-level feedback through Wikipedia co-occurrence statistics, outperforming neural verifiers while reducing training time.

CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM huggingface.co

CONF-KV is a KV-cache management system that dynamically adjusts cache retention based on model uncertainty, improving memory efficiency and performance for long-sequence language model inference.

OmniRetrieval: Unified Retrieval across Heterogeneous Knowledge Sources huggingface.co

OmniRetrieval is a framework that handles diverse knowledge sources by identifying appropriate repositories and dispatching native queries to their respective execution engines, outperforming single-source approaches across multiple dataset types.

GenClaw: Code-Driven Agentic Image Generation huggingface.co

GenClaw presents a code-driven agentic image generation framework that enables precise visual construction through conceptualization, sketching, and coloring stages, integrating programmatic logic with generative models.

Native Audio-Visual Alignment for Generation huggingface.co

NAVA enables joint audio-video generation with improved synchronization and controllability through native audio-visual alignment and context-conditioned denoising.

EarlyTom: Early Token Compression Completes Fast Video Understanding huggingface.co

EarlyTom is a training-free framework that compresses visual tokens early in the vision encoder to reduce time-to-first-token and computational costs while maintaining model accuracy.

LoMo: Local Modality Substitution for Deeper Vision-Language Fusion huggingface.co

Vision-language models suffer from modality sensitivity due to training data bias, but a new data curation approach called Local Modality Substitution improves cross-modal representation alignment and reasoning performance.

Xetrieval: Mechanistically Explaining Dense Retrieval huggingface.co

Xetrieval is a mechanistic framework that explains dense retrieval by enhancing sentence embeddings with reasoning information and decomposing them into interpretable sparse features for retrieval decision explanations.

Is Position Bias in Dense Retrievers Built In-or Learned from Data? huggingface.co

Training data position distribution significantly influences positional bias in dense retrievers, with balanced training reducing sensitivity by up to 87% while maintaining competitive retrieval performance.

When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systems huggingface.co

Hybrid multi-agent systems combining large and small language models offer flexible inference trade-offs, but optimal architecture depends heavily on specific tasks and performance metrics.

RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains huggingface.co

RUBRIC-ARROW presents an alternating framework for reward modeling that improves upon rubric-based methods by reducing ties and leveraging pairwise preference data for training.

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation huggingface.co

DynaFLIP is a dynamics-aware multimodal pre-training framework that enhances robot manipulation by integrating motion understanding into visual perception through image-language-3D flow triplets and geometric regularization techniques.

REPOT: Recoverable Program-of-Thought via Checkpoint Repair huggingface.co

RePoT improves upon one-shot Program-of-Thought by enabling deterministic verified replay and recovery through environment interaction, achieving higher success rates across multiple models and benchmarks.

CoHyDE: Iterative Co-Training of LLM Rewriter & Dense Encoder for Tool Retrieval huggingface.co

CoHyDE is an iterative method that jointly trains a dense encoder and LLM rewriter to improve tool retrieval from API catalogs, achieving better performance on both specific and vague queries through co-evolutionary training.

Reflective Prompt Tuning through Language Model Function-Calling huggingface.co

Reflective Prompt Tuning (RPT) automates prompt optimization for large language models by simulating human iterative engineering through diagnostic feedback and memory-based revision cycles.

CollectionLoRA: Collecting 50 Effects in 1 LoRA via Multi-Teacher On-Policy Distillation huggingface.co

CollectionLoRA enables efficient deployment of multiple customized image editing effects by distilling numerous LoRAs into a single model through multi-teacher distillation and specialized mechanisms for concept isolation and generation.

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence huggingface.co

A 3D-aware post-training framework enhances semantic correspondence estimation by integrating 3D geometry priors from reconstructed object poses and PartField descriptors, improving upon 2D foundation features through automatic 3D structure estimation and render-and-compare optimization.

NeuROK: Generative 4D Neural Object Kinematics huggingface.co

Neural Object Kinematics learns a data-driven parameterization for 4D dynamic object simulation by combining latent space representation with transformer-based encoding-decoding, enabling realistic temporal deformations across diverse object types.

PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions huggingface.co

PhyGenHOI synthesizes physically accurate 4D human-object interactions by combining motion diffusion models with material point method simulations using 3D Gaussian representations.

SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control huggingface.co

SmartDirector enhances video generation by using multiple keyframes to improve narrative structure and temporal pacing through a two-stage process of low-resolution generation and high-resolution refinement.

Learning A Unified Risk Map for Autonomous Driving in Partially Observable Environments huggingface.co

A unified risk map modeling framework addresses occlusion challenges in autonomous driving by integrating traffic flow and collision risks through spatiotemporal modeling and diffusion-based scenario generation.

Multi-view Consistent 3D Gaussian Head Avatars ‘without’ Multi-view Generation huggingface.co

A novel single-shot 3D Gaussian head avatar generation method called MVCHead uses hierarchical state space models and multi-view consistency enforcement to create high-fidelity 3D assets from 2D images without requiring multi-view data or 3D supervision.

Convex Low-resource Accent-Robust Language Detection in Speech Recognition huggingface.co

A novel convex optimization framework for language detection in spoken dialogue systems that achieves high accuracy with efficient training and theoretical guarantees against dialectal variations under low-resource conditions.

ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood huggingface.co

ChildVox presents a comprehensive benchmark for analyzing children’s acoustic communication across developmental stages using diverse audio and speech models.

References

Google Research blog (Wei et al., on emergent abilities) research.google

Certain downstream tasks exhibit qualitative shifts that cannot be extrapolated from smaller models

Dhiria — ‘Emergent Abilities: Reality or Mirage?’ (summarizing Schaeffer et al., NeurIPS 2023) dhiria.com

Sharp transitions are caused by the researcher’s choice of evaluation metrics rather than a fundamental change in model behavior; with continuous metrics, performance gains appear smooth and follow existing scaling laws

OpenReview — ‘$1/d$ scaling law for catastrophic forgetting’ openreview.net

Forgetting decreases as the hidden dimension of the model increases… driven by the increasing orthogonality of output heads in high-dimensional space

MLSanity Papercrunch — empirical forgetting at scale mlsanity.com

In the 1B to 14B parameter range, the severity of forgetting can actually intensify as scale increases, possibly due to the higher initial performance of larger models creating a steeper drop

GenerativeAI.pub — MoE as alternative to dense scaling generativeai.pub

MoE mitigates [gradient interference] by using a router to decouple the learning process; task-specific knowledge is routed to different subnetworks, effectively partitioning the parameter space to prevent tasks from competing for the same weights

Medium — ‘Adam’s Law: textual-frequency cheat code for LLMs’ medium.com

For ‘hard tasks’—defined by their low frequency or complex structure—feature learning can effectively double the scaling exponent compared to standard models

Thinking Machines Lab — ‘LoRA Without Regret’ thinkingmachines.ai

LoRA matches FFT efficiency on small-to-medium datasets (e.g., ~1 million examples) when adapters are applied to all layers, particularly MLPs … at ‘pretraining-like’ scales, LoRA’s capacity becomes a bottleneck, leading to a divergence where FFT remains more efficient.

Shuttleworth et al., ‘LoRA vs Full Fine-tuning: An Illusion of Equivalence’ (arXiv 2410.21228) arxiv.org

LoRA introduces ‘intruder dimensions’—new, high-ranking singular vectors in the weight matrices that do not appear in full fine-tuning … these dimensions can cause task-specific details to be captured in a way that may lead to more rapid forgetting during continual learning.

‘Leaner Training, Lower Leakage’ (arXiv 2506.20856) arxiv.org

LoRA can reduce memorization risks by up to a factor of 10 while maintaining comparable accuracy … factors like model scale and data duplication, which drive memorization in pre-training, do not exert the same influence during LoRA fine-tuning.

Josifoski et al., ‘How Optimal is Greedy Decoding?’ (AKBC 2022) akbc.ws

Greedy decoding is often criticized as ‘brittle’ … models optimized for greedy performance often exhibit a significant gap when subjected to sampling-based generation … greedy search can occasionally suppress leakage that becomes apparent only when the model’s output distribution is flattened by temperature.

Kalajdzievski, ‘A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA’ (rsLoRA blog) d2jud02ci9yv69.cloudfront.net

The original implementation’s scaling can lead to ‘stunted learning’ as the rank increases, creating a false impression of performance saturation at low ranks … rsLoRA uses a 1/√r scaling factor to allow for stable learning even at very high ranks (up to 2048), theoretically unlocking much higher memorization capacities than previously thought possible.

bycloud.ai newsletter — LoRA vs Full Fine-tuning commentary mail.bycloud.ai

Higher-rank LoRA with rank-stabilization (setting the scaling factor α = 2r) forces the LoRA solution to more closely mirror the distributed nature of full fine-tuning and reduces the emergence of intruder dimensions.

MarkTechPost marktechpost.com

Parallax keeps softmax attention and adds a learned covariance correction branch… the prototype decode kernel matches or outperforms FlashAttention 2/3 across batch sizes.

AIWeekly — ‘Parallax closes linear attention gap at LLM scale’ aiweekly.co

Parallax closes the linear attention gap… long-term viability depends on community kernel support beyond NVIDIA Hopper and CuTeDSL.

OpenReview — Local Linear Attention (ICLR 2026 submission, FlashLLA) openreview.net

FlashLLA employs an iterative conjugate gradient solver that must stream the Key matrix multiple times per step, leading to significant wall-clock latency.

Towards AI — ‘AI’s Revolutionary Attention Mechanism Is Just 1960s Statistics’ pub.towardsai.net

Standard Transformer attention is mathematically equivalent to the Nadaraya-Watson estimator introduced in 1964… a ‘local constant’ estimator that suffers from boundary bias.

arXiv 2504.14366 — Subquadratic architecture scaling study arxiv.org

A ‘crossover’ occurs at the 1.7B scale: while xLSTM led at smaller sizes, Gated DeltaNet became the top performer, recovering 94.2% of the Transformer teacher’s performance.

LearnAIVisually explainer learnaivisually.com

Some critics question if the gains will continue to scale beyond the 1.7B parameter mark, as the performance gap between local-linear and local-constant estimators can shrink in much larger models.