Sources

DiScoFormer: One transformer for density and score, across distributions huggingface.co

A Verifiable Search Is Not a Learnable Chain-of-Thought huggingface.co

Training models on chain-of-thought demonstrations fails for tasks requiring backtracking search because the forward derivation cannot be faithfully imitated, demonstrating a fundamental limitation in learning search procedures through demonstration.

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention huggingface.co

Grouped Query Experts (GQE) improves Transformer efficiency by selectively activating query heads based on token content while maintaining key-value cache benefits of grouped-query attention.

Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs huggingface.co

McGill researchers propose releasing open-weight LLMs whose computation graph swaps in a private parameter subset only when unlocked with a secret key. Public weights keep instruction-following intact, while sensitive factual knowledge stays inaccessible without authorization, offering a middle path between full open release and closed APIs.

Tapered Language Models huggingface.co

Allocating more MLP capacity to early layers and tapering later ones improves perplexity at matched total parameters and compute. The depth-aware schedule is architecture-agnostic across transformer, recurrent and memory-based variants, challenging the long-standing convention of identical layers stacked uniformly through a network.

Unlimited OCR Works huggingface.co

Reference Sliding Window Attention caps KV-cache growth during long-sequence OCR, letting a single forward pass transcribe multi-page documents without ballooning memory. Baidu reports the parsing-attention mechanism generalizes beyond OCR to ASR and translation, with code released on GitHub.

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization huggingface.co

Rather than alternating layers, HydraHead assigns Full or Linear attention to individual heads using interpretability-driven selection and scale-normalized fusion. The hybrid reuses pretrained parameters via distillation, beating uniform architectures on long-context benchmarks while cutting training overhead.

Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding huggingface.co

Qwen researchers find the final transformer layer is not always the most reliable token predictor. An entropy-guided backward search dynamically stops at a more confident intermediate layer, lifting scores on GPQA-Diamond, Omni-MATH and HLE with minimal overhead by sidestepping alignment-induced projection noise.

FastMix: Fast Data Mixture Optimization via Gradient Descent huggingface.co

FastMix treats data-mixture selection as a bilevel optimization, jointly updating mixture coefficients and model parameters through differentiable inner and outer loops. The approach replaces expensive grid search or proxy models with direct gradient-based discovery of optimal domain weights during a single training run.

Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining huggingface.co

For data-constrained pretraining, simple training-time augmentations — token-level noise, sequence permutations and target-offset prediction — push back the epoch where validation loss turns up. Final model quality improves on fixed datasets trained for many epochs, with all recipes released on GitHub.

Self-Compacting Language Model Agents huggingface.co

SelfCompact is a scaffolding approach that enables models to autonomously determine optimal compaction timing and methods for managing long agent traces, achieving better performance with reduced token costs compared to fixed-interval methods.

Comparing Linear Probes with Mahalanobis Cosine Similarity huggingface.co

The Mahalanobis cosine similarity provides a theoretically grounded method for comparing linear probes that correlates strongly with out-of-distribution performance metrics.

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning huggingface.co

Data-centric approach using curated datasets and minimal GRPO setup significantly improves long-context reasoning in large language models, outperforming prior reinforcement learning methods.

KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking huggingface.co

KaLM-Reranker-V1 is a fast reranker that decouples query and passage computation using encoder-decoder architecture with Matryoshka embedding pooling and cross-attention for efficient relevance modeling.

Safe Few-Step Generation via Velocity Editing huggingface.co

VESFlow is a training-free safety method for flow matching-based text-to-image generation that edits velocity fields to ensure safe output while maintaining prompt integrity.

Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation huggingface.co

Trajectory-Augmented Policy Optimization (TAPO) enhances large language model reasoning by creating explicit corrective trajectories that preserve erroneous reasoning while incorporating natural-language diagnoses and corrections, outperforming traditional self-distillation methods through improved error-correction capabilities.

Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning huggingface.co

Large language models can be trained through reinforcement learning to develop a meta-capability enabling continuous learning and adaptation across long sequences of tasks in dynamic environments.

MeshFlow: Mesh Generation with Equivariant Flow Matching huggingface.co

MeshFlow generates triangle meshes directly using equivariant optimal-transport flow matching models with improved inference speed over autoregressive methods.

Exploring the Design Space of Reward Backpropagation for Flow Matching huggingface.co

FlowBP addresses limitations in flow matching model alignment by using a surrogate trajectory framework that reduces memory usage and gradient chaining while maintaining performance across multiple text-to-image models.

World Action Models: A Survey huggingface.co

World Action Models are predictive-action systems that generate future states for decision-making, with designs balancing representational richness against computational constraints.

Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity? huggingface.co

Computer-use agents frequently expose inappropriate information across applications, prompting the creation of AgentCIBench to evaluate and mitigate privacy risks in cross-application contexts.

Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents huggingface.co

A failure detection framework for long-horizon robotic tasks uses action-conditioned world models and functional conformal prediction to monitor manipulation trajectories with only final task labels.

PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models huggingface.co

PolicyTrim is a reinforcement learning-based framework that enhances VLA model efficiency by extending reliable action chunk lengths and reducing redundant physical steps through dynamic exploration and redundancy-aware rewards.

When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agents huggingface.co

Pre premature commitment in long-horizon LLM agents leads to silent failures where agents defend early interpretations without considering alternatives, and hidden-state convergence serves as an early diagnostic for trajectory consistency.

EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory huggingface.co

EvoEmbedding is a dynamic embedding model that generates adaptive representations by maintaining a continuously updated latent memory, enabling improved retrieval performance in long-context scenarios.

CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents huggingface.co

A principled synthesis engine generates high-quality terminal-agent tasks through multi-dimensional capability taxonomy and evidence-guided research, creating a distilled dataset that enables significant performance gains in LLM training.

Tmax: A simple recipe for terminal agents huggingface.co

A novel RL training approach for terminal agents achieves superior performance using a simplified recipe and expanded dataset, enabling effective training with fewer parameters than previous methods.

CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks huggingface.co

Calibrated verifier telemetry enhances LLM agents in knowledge-intensive question answering by providing confidence scores and grounding verification, reducing both over-retrieval and unsupported answers.

Training Open Models for Agentic Phone Use huggingface.co

PhoneBuddy combines real and mock app environments to improve training of open models for phone use, demonstrating enhanced task success rates through mixed reinforcement learning approaches.

PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning huggingface.co

PoLAR introduces a geometrically structured latent action representation in hyperbolic space that separates transition extent from transition mode, improving robotic policy learning performance.

Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models huggingface.co

Reinforcement learning approaches for improving LLM reasoning capabilities are enhanced by a Bayesian Manifold Curriculum framework that structures problem sampling based on task manifold relationships and endogenous non-stationarity.

OpenRath: Session-Centered Runtime State for Agent Systems huggingface.co

OpenRath introduces a PyTorch-like programming model for multi-agent systems using Session as a central runtime abstraction that enables explicit fork, merge, and replay operations while recording comprehensive execution state.

BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language huggingface.co

BioMatrix is a novel multimodal foundation model that integrates molecular sequences, structures, and natural language into a unified decoder-only architecture for diverse biological tasks.

Go-with-the-Track: Video Compositing and Motion Control with Point Tracking huggingface.co

Go-with-the-Track unifies motion control and reference image compositing in video generation by using point-track embeddings with spatial-aware encoding and video diffusion transformers.

Vera: A Layered Diffusion Model for Content-Preserving Video Editing huggingface.co

Vera is a layered diffusion framework that preserves video content during editing by generating edit layers and alpha mattes through a Mixture-of-Transformers architecture.

DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams huggingface.co

Agentic Data Tailoring paradigm uses learnable data processing to structure high-entropy multimodal streams, with DataClaw_0-9B model achieving robust alignment through SFT and GRPO on a novel benchmark.

SkillHarness: Harnessing Safe Skills for Computer-Use Agents huggingface.co

SkillHarness is a framework that enables computer-use agents to safely learn and execute skills in dynamic environments by incorporating safety constraints and adaptive skill selection mechanisms.

Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild huggingface.co

Lift4D presents a test-time optimization framework that combines temporal consistency from single-view 3D reconstruction with deformable 3D Gaussian Splatting and view-conditioned diffusion priors to reconstruct dynamic non-rigid objects from monocular video.

Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views huggingface.co

DR-MV3D presents a map-grounded learning framework with dense rewards to improve multi-view 3D visual question answering through global map construction, view-trajectory planning, and egocentric grounding.

Arbor: Explicit Geometric Conditioning for Controllable 3D Asset Generation huggingface.co

Arbor enables explicit 3D spatial control in text-conditioned latent generation through constraint meshes that define occupancy, avoidance, and contact regions, maintaining object quality while improving constraint adherence.

Causal Discovery in the Era of Agents huggingface.co

Language models should assist causal discovery workflows by providing contextual support and explanations rather than generating causal conclusions, as demonstrated through a platform that integrates data analysis and expert knowledge.

UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation huggingface.co

UniverSat introduces a Universal Patch Encoder for Vision Transformers that enables robust, sensor-agnostic spatial feature extraction across diverse Earth Observation data types.

Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills huggingface.co

Notes2Skills framework converts laboratory notes into verifiable skills for AI agents while maintaining author uncertainty levels, addressing gaps in scientific AI development.

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions huggingface.co

EnterpriseClawBench presents a benchmark for enterprise agents based on real-world sessions with 852 reproducible tasks, emphasizing comprehensive evaluation metrics beyond single performance scores.

PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems huggingface.co

PlanBench-XL evaluates large language model agents’ ability to plan and adapt in complex tool-rich environments with limited visibility and dynamic disruptions.

Counsel: A Meta-Evaluation Dataset for Agentic Tasks huggingface.co

A large-scale dataset of human-metaevaluations of LLM critiques for agentic tasks is introduced to improve the calibration and reliability of automated evaluation methods.

Deep Research in Physical Sciences: A Multi-Agent Framework and Comprehensive Benchmark huggingface.co

PhySciBench benchmark reveals limited performance of current LLM agents in physical science research, leading to development of DelveAgent framework that improves accuracy through modular design and physics-grounded mechanisms.

HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions huggingface.co

HAKARI-Bench provides a lightweight benchmark for comparing retrieval methods across multiple configurations and languages, enabling efficient model selection and performance analysis.

Libretto: Giving LLM Agents a Sense of Musical Structure huggingface.co

Libretto provides a structured framework for symbolic music generation and revision using LLM-native grammar and statistical evaluation across musical dimensions.

Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City huggingface.co

Research examines how self-driving car systems and humans perform on visual question answering tasks across different geographic locations, revealing that both human and AI responses diverge based on question types but show similar performance regardless of location.

DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks huggingface.co

Search agents face challenges in real-world evaluation due to limited benchmarks and coarse metrics, necessitating more nuanced assessment approaches.

References

Kempner Institute (Harvard) — ‘To Backtrack or Not to Backtrack’ kempnerinstitute.harvard.edu

Sequential backtracking underperformed parallel sampling in CountDown but excelled in Sudoku… training models on fixed search traces ‘locks’ them into rigid, suboptimal strategies.

LogicPuzzleRL (ResearchGate) researchgate.net

A model fine-tuned specifically on cryptarithms achieved an accuracy of 44.98%, a +1.06% gain over the 44.51% baseline.

ppc.land — ‘Small Models Can’t Learn Search Tasks Despite More Data’ ppc.land

Small Transformers successfully learned search procedures for small graphs but failed systematically as graph size increased… even with effectively limitless training data, models rely on shortcuts that do not generalize.

Can Transformers Learn to Verify During Backtracking Search (ResearchGate) researchgate.net

Decoder-only Transformers trained on cumulative traces often emit different decisions for the same state if the prior history differs — ‘history entanglement’ rather than state-based decisions.

Cobus Greyling — ‘Chain-of-Thought Reasoning Is Not Always Faithful’ (Medium) cobusgreyling.medium.com

Even advanced models like Claude 3.7 or DeepSeek R1 exhibit implicit post-hoc rationalization, where the model arrives at a biased answer and then constructs a logical-sounding justification after the fact.

themoonlight.io review of ‘A Verifiable Search Is Not a Learnable Chain-of-Thought’ themoonlight.io

During free-running generation, the model may perform perfect arithmetic on 97–100% of its lines yet emit incorrect elimination verdicts 43–84% of the time, as the verdict has become a high-probability ‘filler’ phrase.

ICML 2026 Spotlight Posters listing icml.cc

DiScoFormer was accepted as a Spotlight (top 2.2%) and Oral presentation at ICML 2026.

OpenReview discussion (reviewers + rebuttal) openreview.net

Reviewer 8TYy questioned boundary effects on compactly supported densities; authors acknowledged boundary bias ‘might still mirror KDE-like behavior due to the unbounded support of the training GMMs.’

OpenReview rebuttal thread openreview.net

Reviewer NWbe pointed to a potential flaw in Proposition 3.2, which appeared to rely on a strong assumption of L2-normalized input vectors; authors refined their theoretical positioning in response.

Sudjianto, ‘Self-Attention as a Kernel Machine’ (Substack) agussudjianto.substack.com

Dot-product attention is a learned version of the Nadaraya-Watson estimator (1964); softmax attention effectively implements a Gaussian RBF kernel — prior frameworks like KDEformer already made this connection explicit.

Song & Ermon, Sliced Score Matching (PubMed-indexed) pubmed.ncbi.nlm.nih.gov

Sliced Score Matching scales linearly with dimensionality and accurately recovers parameters of 100-dimensional Gaussian mixtures where standard score matching fails — an existing high-D baseline DiScoFormer does not directly benchmark against.

MoH: Multi-Head Attention as Mixture-of-Head Attention (Jin et al., arXiv:2410.11842) arxiv.org

MoH-LLaMA3-8B achieves a 2.4% accuracy improvement across 14 benchmarks while utilizing only 75% of the attention heads, and supports continue-tuning from pre-trained dense models like LLaMA-3.

SwitchHead (Csordás et al., arXiv:2312.07987) arxiv.org

SwitchHead applies MoE to the value and output projections of attention, using 4–8× fewer attention matrices than standard Transformers while matching performance.

Mixture of Attentive Experts (MAE), Peng et al., ACL 2020 aclanthology.org

MAE reallocates heads to different inputs using block coordinate descent, treating attention heads as experts rather than statically pruning them.

Step-3 / Stepfun technical report chat.stepfun.com

Commercial-scale systems such as DeepSeek-V3 and GLM-class models adopt content-dependent sparse attention computing only the top ~2,048 tokens per head, yielding ~2.9× per-token FLOPs reduction at a 1M-token window.

MoH Hugging Face papers page (community thread) huggingface.co

Traditional MoH implementations often retain a full KV cache per head to maintain flexibility, which can be prohibitively expensive as sequence lengths grow.

CSDN practitioner write-up on GQE blog.csdn.net

Reviewers note the design is interesting but experiments at 250M may be ‘insufficient’ to prove generalization, and compare it unfavourably with mathematically identical compression methods like Slim Attention that avoid MoE training-stability risks.

Sources

References

Jack Sun, writing.