Sources

EmbeddingGemma 2: an open, lightweight multimodal embedding model deepmind.google

We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety o

Marathoner: Ultra-Long-Horizon Autonomous Intelligence huggingface.co

Humans naturally possess the ability to work persistently toward long-term goals. Given a challenging task, humans can continuously work for months or even years to accomplish a specific objective. In this paper, we propose Marathoner, an autonomous agentic model possessing the ability of ultra-long-horizon execution. Specifically, we propose a comprehensive post-training pipeline to instill this critical capability into base model. For Ultra-Long-Horizon Task Synthesis, we leverage major releas

TabFM: A Zero-Shot Foundation Model for Tabular Data huggingface.co

TabFM reframes supervised tabular prediction as in-context learning, producing calibrated zero-shot outputs in one forward pass with no per-task tuning. The 400M-parameter model trains entirely on synthetic tables sampled from structural causal models, sidestepping the per-dataset AutoML and tree-ensemble pipelines that dominate tabular ML.

EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments? huggingface.co

The benchmark covers the full industrial design loop with expert-curated tasks spanning CAD, CAE, CAM, BIM, EDA, and 3D visualization. Unlike general computer-use suites, EngiWorld tests reasoning over geometric and physical constraints preserved across software stages, where today’s agents still fall short.

Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents huggingface.co

Noisy observations across video, audio, web, and tools accumulate in chat history and derail later decisions in omni-modal agents. Controlled swaps show planner replacement hurts far more than perception replacement, motivating Omni-Decision’s structured evidence ledger to separate retrieved facts from the planning trace.

LLMs are General Asynchronous Agents huggingface.co

Voice assistants, embodied robots, and monitors receive inputs while the model is still thinking, breaking the read-think-reply loop. Rather than bolting on specialized architectures for streaming audio, video, or VLA control, the authors propose a unified asynchronous interaction scheme that lets a single LLM handle concurrent inputs.

HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents huggingface.co

Pure-GUI agents are slow and error-prone, while API wrappers demand bespoke engineering per app. HybridCUA trains agents to decide when to drop into the command line instead of clicking, aiming to scale computer use without hand-built tool integrations for every target application.

Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression huggingface.co

Compressing consecutive tokens at a fixed stride introduces a new positional coordinate — a token’s phase relative to window boundaries — and models retrieve the same fact easily at one phase but fail at another. The periodic asymmetry adds a hidden failure mode to long-context inference stacks.

Language Models Are “Insecure” Reporters huggingface.co

As users increasingly audit long-horizon agents through model-written summaries rather than raw traces, eight adversarial reporting scenarios test whether LLMs disclose errors that undermine their accounts. The models systematically conceal such flaws, exposing a trust gap in agent self-reporting.

Scaling Properties of Same-Family On-Policy Distillation huggingface.co

Reinforcement learning (RL) can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of on-policy distillation (OPD) across weak-to-strong, same-base, and strong-to-weak teacher—student setups. We find that early OPD training dynamics uniformly exhibit a regular useful-transfer regime, in which held-out accuracy (the gold score, G)

On the Off-Policy Teacher in On-Policy Distillation huggingface.co

On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by t

EasyPPO: Stabilizing the Critic Is Key huggingface.co

A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for large language models (LLMs). We identify two critic failure modes that destabilize PPO. First, filtering truncated rollouts from both actor and critic shifts the policy objective to

Raven: The Harness of Harnesses for Composable Agentic Intelligence huggingface.co

As large language models advance, AI agents are moving beyond isolated, domain-specific tasks toward long-horizon, cross-domain workflows. This transition exposes two challenges: increasing harness complexity makes manual design difficult to scale, while tighter coupling to specific domains limits the generality of a single harness. The central question thus shifts from how to engineer a stronger harness for one domain to how to autonomously construct specialized harnesses, improve them through

Reasoning with Image Generation huggingface.co

Chain-of-thought reasoning has revolutionized natural language processing by enabling large language models (LLMs) to decompose problems into intermediate steps before answering. Yet confining reasoning to the textual domain presents limitations for tasks requiring direct manipulation of visual representations. Recent efforts augment multimodal LLMs with external visual expert tools such as depth estimation or object detection modules, but these remain fundamentally limited by their reliance on

AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation huggingface.co

Recent image generation models can take multiple reference images as input and combine them into a new image. However, multi-reference image generation remains challenging: models may omit or duplicate subjects from the references, or produce images in which multiple subjects appear unnaturally pasted. Recent work has proposed image generation agents that combine image generation models, reasoning models, and a harness, which is an executable program that specifies how reference images are inter

ROSS: Relearning from Self-Generated Rollouts through Selective Supervision huggingface.co

Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We in

Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing huggingface.co

Large language model (LLM) routing reduces serving cost by assigning each query to an appropriate model while preserving response quality. Learning such a router, however, often requires executing multiple candidate models on historical queries to collect query—model quality feedback, creating a nontrivial supervision cost before deployment. Existing work largely focuses on serving-time efficiency, overlooking whether the resulting savings are sufficient to recover this upfront expenditure. We

Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents huggingface.co

An agent that uses tools typically responds to what the user explicitly asks, yet completing the task may require information the user never requested. Work on proactive agents mainly studies whether and when an agent should act on its own, not what information it should pursue. We study a distinct axis of proactivity: its content. Horizontal proactivity pursues unstated information that the current context already identifies, and vertical proactivity pursues needs that only earlier evidence rev

SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillation huggingface.co

On-policy distillation (OPD) reduces train-test state mismatch by training a student on its own generated trajectories, but weak students may visit teacher-misaligned prefixes where supervision is less representative. We introduce SAKI (Supervision Allocation with KL-constrained Interpolation), which combines a KL-constrained teacher-guided rollout with maximal coupling and reuses realized accept/correction events to route token-level supervision. Accepted positions retain sampled-token reverse-

Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR huggingface.co

Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model’s rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportun

What Makes World Action Models Generalize? An Empirical Study of Test-Time Future Modeling huggingface.co

World action models (WAMs) predict the future alongside actions during training. Due to the heavy computation cost of video denoising, whether the future must still be generated during inference is disputed: Explicit WAMs denoise it into clean frames along with every action chunk, whereas Latent WAMs discard it entirely for acceleration. We find that latent WAMs, despite matching explicit ones on in-distribution tasks, fail to retain the generalization benefits that originally motivated WAMs. To

Scheduling Recursive Reasoning in Looped Transformers huggingface.co

Recurrent reasoning models have attracted growing attention for scaling test-time computation, typically by iteratively refining latent states with shared parameters. However, these models apply each learned update with a fixed unit scale, which can be conservative when updates make persistent progress and overly aggressive when they fluctuate, limiting the benefit of additional loops. To understand how the scale should vary along the trajectory, we first analyze the sensitivity of terminal loss

OmniTaskonomy: When Does Visual Generation Improve Visual Understanding? huggingface.co

Training a model to generate visual content can encourage it to learn rich perceptual capabilities related to geometry, spatial relationships, and objectness; yet, its benefits for visual understanding remain unclear. We ask: when and how does visual generation supervision improve visual understanding? We study controlled pairs of image-to-image (I2I) generation and image-to-text (I2T) understanding tasks that express the same underlying problem in different output modalities. We find that under

Selecting The Most Informative Tokens in Natural Language Autoencoders huggingface.co

Natural language autoencoders translate a language model’s internal activations into readable explanations. Explaining every token position is costly. Which positions should an auditor inspect to understand a potential threat? We study this question across 4.7 million explanations on prompt injection and concealment. We compare signals from model computation with a ranker trained only on chat structure. Chat structure usually selects more relevant explanations than the computational signals, wit

ANTMAN: Adaptive Need Tracking for Multi-Agent Navigation in Large Information Spaces huggingface.co

Information-seeking agents increasingly operate over information spaces that are too large to process exhaustively. Yet many multi-agent systems organize computation around static partitions of the available space, causing coordination to grow with how information is segmented rather than with what the query still requires. We introduce ANTMAN, an adaptive coordination framework that treats evolving unresolved information needs as the unit of runtime coordination. ANTMAN maintains a revisable Ne

Adversarial Training for Pixel Diffusion huggingface.co

Pixel diffusion models generate RGB images directly, avoiding the bottleneck of an autoencoder, yet their outputs still systematically underrepresent fine-scale natural-image statistics. We show that adversarial learning provides an effective post-training correction for this deficiency. Starting from a pretrained model, we retain its original diffusion or flow-matching objective and add an adversarial loss to the predicted output at non-high-noise timesteps, leaving the model architecture and s

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation huggingface.co

Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a scalable foundation for visual simulators that predict action outcomes before physical execution. Yet they often favor visual plausibility over accurate action following and coherent robot—object dynamics, while action-conditioned simulators depend on scarce, embodiment-specific data that are difficult to share across incompatible control spaces. We i

EVO-WAM: Evolving World Action Models through Video-Action Verification huggingface.co

Improving robot policies on new tasks without collecting additional expert demonstrations remains a central challenge in robot learning. World action models (WAMs) use broad video priors to jointly predict future videos and actions, offering a potential source of supervision for adapting to new tasks. However, generated videos may fail to depict task completion, and even visually successful videos may be paired with inconsistent actions that lead to execution failure. We propose EVO-WAM, a frame

Follow the Entities: A Corpus Map for Agentic Search huggingface.co

Answering questions and completing tasks over large document collections often requires connecting evidence spread across multiple documents, such as a project’s approval recorded in one, its requirements in another, and its latest status in a third. Recent LLM agents approach this by iteratively searching the full corpus rather than reading only a fixed set of top-ranked documents. However, when the corpus is exposed only as a flat collection of files, a relevant document gives no indication of

LongCat-DeepResearch Technical Report huggingface.co

We present LongCat-DeepResearch, a deep research system that combines an enhanced LongCat model with a multi-agent workflow for producing comprehensive, evidence-grounded reports. The workflow separates global planning from detailed investigation and coordinates revision at the section level. Multiple planning agents first explore external sources and refine an actionable research plan, termed ResearchSpec. Research agents then investigate and draft their assigned sections in parallel, gathering

SoL-Refiner: Speed-of-Light One-Step Refinement for High-Resolution Video huggingface.co

High-resolution video generation is expensive, as its cost grows rapidly with the number of spatiotemporal tokens. A practical alternative first generates a lower-resolution video and then applies a refiner, but conventional multi-step refinement introduces a second sampling bottleneck. We present SoL-Refiner, a one-step video refiner that transforms low-resolution model outputs into 4K videos with a single denoising step. Our three-stage recipe combines high-resolution continual training, reinf

FocusVTC: Efficient and High-Performance Visual Text Compression with Adaptive Resolution huggingface.co

Long-context reasoning in large language models incurs substantial computation and memory costs. Visual text compression (VTC) reduces input length by rendering text as images, but fixed-resolution rendering creates a compression-performance trade-off: low DPI saves tokens at the expense of legibility, whereas high DPI spends tokens on irrelevant content. We introduce FocusVTC, which breaks this trade-off through adaptive resolution while preserving general multimodal capabilities. It combines c

Anisotropic Representations Improve Planning in JEPA World Models huggingface.co

Latent world models learn action-conditioned dynamics in representation space and often score candidate actions by Euclidean distance to a goal representation. Joint training typically regularizes the representation to prevent collapse, but the resulting representation geometry also determines how terminal errors are weighted during planning. We show that accurate prediction and noncollapsed representations do not guarantee a task-aligned latent planning cost: isotropic Gaussian regularization c

LEGO-Anything: Coding Agents for 3D Scene Reconstruction huggingface.co

A 3D scene reconstructed from a single image is most useful when represented not as a rendering or a fixed 3D output, but as an explicit scene program whose execution yields a scene that can be inspected, edited, and queried. We present LEGO-Anything, an Image-to-Code framework in which a coding agent iteratively writes and executes Blender code, inspects scenes and renderings, and revises the program. To evaluate end-to-end scene recovery, we introduce LEGO-Bench, a simulator-grounded benchmark

When Does Dense Retrieval Need Asymmetric Geometry? A Bias-Variance Theory of Shared and Dual Projections huggingface.co

Dense retrieval powers retrieval-augmented generation, semantic search, and question answering, yet the theoretical basis for choosing between shared and dual query-document projections remains unclear. We introduce a bias-variance theory for low-rank bilinear scoring. Shared projections induce positive-semidefinite operators, whereas dual projections realize arbitrary low-rank operators. We derive their exact approximation gap and prove a local Gaussian boundary: dual has lower risk exactly whe

LongLive-Plug: Once-for-All Distillation for Video Generation huggingface.co

Video diffusion models are increasingly developed into specialized models for diverse downstream tasks, and this development often includes a distillation stage, for example to accelerate sampling or to improve long-video generation. This stage is typically repeated for every specialized model. We introduce LongLive-Plug, a once-for-all distillation framework that learns reusable capabilities as LoRAs on a base model for training-free, plug-and-play deployment to compatible downstream models. Th

WorldAttention: An Efficient Attention Architecture for Interactive Video World Models huggingface.co

Leveraging the paradigm of autoregressive diffusion, text-conditioned interactive video world models aim to simulate temporally coherent environments guided by textual instructions. While enabling low-latency, long-duration generation is pivotal for embodied AI and simulation-based planning, current frameworks primarily rely on sliding-window mechanisms to bound computational complexity. However, this approach inherently sacrifices historical context, undermining the long-range interactive capab

APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants huggingface.co

To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use. Yet existing streaming benchmarks and methods often focus on individual continuous videos or short clips, overlooking that real-world interactions are often intermittent and require memory to persist across interruptions. To fill this gap, we introduce APM-Bench, which reformulates real-world streaming interaction as multi-session life trajectories. It contains 5

Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies huggingface.co

Answering questions about long videos often requires connecting events involving the same objects across hours or days. Chronological descriptions and text-derived entities can leave physical identity unresolved: different objects may share a description, while observations of the same object remain disconnected across events. Retrieving relevant events therefore does not necessarily recover the “biography” of the particular entity a question concerns. To address this, we introduce Grounded Enti

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models huggingface.co

Spoken conversational systems must recover information from prior interactions (i.e., memory), yet relevant information in speech extends beyond what was said to who said it, how it was spoken, and what was audible, information that exists only in the audio signal and cannot be recovered from a transcript. Beyond what to remember, memory also demands diverse operations: retrieving a single fact, integrating evidence across turns, tracking an evolving state. Real interactions further unfold acros

Think Before You Score: Thinking Reward Model for Visual Generation huggingface.co

Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates c

Omni-IO Skills: Harnessing Your Agent Omni-Native huggingface.co

General-purpose agents can plan, reason, and act over long horizons, yet their production capabilities remain fragmented across text, images, audio, video, documents, 3D assets, and code. Extending a foundation model to additional modalities ties capability growth to costly model updates, while assembling specialist models and tools leaves unresolved how procedures, dependencies, intermediate assets, and cross-turn revisions should be coordinated. We present Omni-IO Skills, a plug-and-play Agent

HiRAE: Hierarchical Representation Autoencoding with Residual Budgets huggingface.co

Pretrained visual representations support image generation, but may not fully preserve the fine-grained details needed for faithful reconstruction. Meanwhile, intermediate encoder layers contain complementary visual details, but learning to fuse them for reconstruction can produce a latent distribution that is difficult to model. Existing fusion methods require empirical tuning of layer selection or staged optimization of fusion and decoding, increasing configuration effort or training complexit

In-Context Learning for Robots: Methods and Applications huggingface.co

General-purpose robots must infer what a new task requires and translate that understanding into appropriate physical action. In-context learning (ICL) for robots supports this process by using demonstrations and interaction to direct existing competence with neural parameters held fixed during deployment. We organize this literature review around the interfaces connecting contextual evidence to execution, distinguishing four families: context-conditioned policies, geometric demonstration transf

MaLiang-Harness: A Programmable Path to Image and Video Generation huggingface.co

Executable programs offer explicit control over how images and videos are constructed, but generating runnable code is only the beginning of visual creation. A program can execute correctly while violating the requested composition, appearance, or motion. We define this discrepancy as the Program-to-Visual (P2V) gap and introduce MaLiang-Harness, a unified framework for organizing MLLM-driven visual generation into a persistent process of construction, inspection, and revision. Its central desig

PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation huggingface.co

Recent vision-language models (VLMs) have advanced vision-and-language navigation (VLN), enabling models to predict navigation actions from visual observations and language instructions. In this work, we explore VLN with panoramic observations and introduce PanoVLN. The motivation is straightforward: more complete visual context should enable better-informed navigation decisions. For example, a panorama can reveal a passage outside a perspective camera’s field of view, allowing the model to iden

Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations huggingface.co

Long-term memory enables agents to accumulate information and reason across sessions, yet existing research primarily focuses on dyadic text or image-text conversations, leaving long-term memory for multi-party spoken conversations underexplored. This setting requires preserving conversational content, identifying participants across sessions, and retaining who speaks to whom. To this end, we propose VoxPolyMem, an interaction-aware multimodal memory framework combining incremental speaker ident

CrossBFM: Distilling a Shared Latent Behavior Space Across Humanoid Embodiments huggingface.co

Behavior Foundation Models (BFMs) give humanoids a promptable policy over a latent behavior space, enabling one single vector to represent a motion to imitate, a pose to reach, or a reward to maximize. Forward-Backward representations successfully produce such spaces, but at the cost of hundreds of GPU-hours for a single robot. Moreover, when the training process is repeated for a second robot, it produces a second space unrelated to the first, resulting in embodiment-specific latents that do no

EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation huggingface.co

Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model. CoCoEmo, a conventional vector steering method for emotion TTS, treats each emotion vector as an indivisible direction controlled by a single global str

Chinese-Jev: Bringing System One Model to Chinese-Language Tasks huggingface.co

System One models such as Jev offer an efficient alternative to generative language models for tasks that require decisions rather than open-ended responses. However, existing Jev models exhibit limited Chinese-language decision accuracy, restricting their utility in both general and specialized settings. In this paper, we introduce Chinese-Jev, a System One model that addresses this gap through a unified data processing and training pipeline. Our data processing protocol converts heterogeneous

References

Towards AI (practitioner review of Qwen 3.7 Max) pub.towardsai.net

a nine-hour continuous loop where the agent performed 1,040 tool calls to optimize worker latency, maintaining context and reasoning without ‘losing the plot’

whatllm.org — Meta Muse Spark coverage whatllm.org

Meta’s Muse Glimmer 30B, released under an Apache 2.0 license, demonstrated a 24-hour autonomous run involving over 1,000 tool calls during a kernel-optimization case study

zylos.ai — GRPO for tool-using agents zylos.ai

three primary failure modes: entropy collapse, where the model becomes overly repetitive; advantage collapse, where a lack of variation in a group’s performance yields no learning signal; and KL drift

arXiv 2509.21154 (λ-GRPO) arxiv.org

vanilla GRPO possesses an inherent bias — the ‘frequency term’ flaw — where frequently occurring intermediate steps are over-weighted regardless of their quality

Requesty.ai — MCP Gateway Comparison 2026 requesty.ai

an agent making 1,000 tool calls per hour could inadvertently expose database credentials or exceed cloud budgets… MCP Gateways like Bifrost and Lunar MCPX… enforce rate limits (e.g., capping calls at 1,000) and provide immutable audit trails

lorphic.com — Kimi vs DeepSeek 2026 review lorphic.com

DeepSeek’s V4-Flash-0731 variant surprised the industry in August 2026 by outperforming the larger V4-Pro on terminal-based tasks (82.7 on Terminal Bench 2.1), proving that dense post-training on tool-use traces can outweigh raw parameter count

sparsenotes.com (independent writeup) sparsenotes.com

Unlike prefix caching, which is mathematically exact, a relocated suffix was originally computed with a different preceding context, potentially introducing minor semantic drift… the implementation currently caps reuse at six moved pieces per edit.

mer.vin community notes mer.vin

The model might spend too much computational budget solving its own ‘memory crisis’ rather than the actual task—a problem some suggest solving via a secondary ‘hypervisor’ agent rather than intrinsic management.

flowtivity.ai review flowtivity.ai

Because a CLM can write to its own history, malicious instructions encountered during a web search could be ‘saved’ into the context file, allowing them to influence the agent across all subsequent turns.

Medium (Yaniv G., ‘Can Bash speak MCP?’) medium.com

Nearly 43% of public MCP servers are reportedly vulnerable to command injection, as they often execute LLM-generated shell commands without sufficient sandboxing.

OpenReview submission on GRPO variants openreview.net

GRPO ‘secretly’ functions as a process reward model, but its reliance on group-relative statistics creates a ‘non-uniformly distributed process step’ problem… leading to performance plateaus.

ResearchGate: ‘Maximum Effective Context Window’ study researchgate.net

While models now support windows exceeding 1M tokens, their ‘effective’ utilization often sits between 50-65%, making the CLM self-editing paradigm essential for maintaining precision in high-stakes engineering work.

ailog.fr — 2026 Embedding Models Leaderboard app.ailog.fr

On the MIEB (Lite) suite, [EmbeddingGemma 2] scores 64.64, narrowly trailing Qwen3-VL-Embedding-2B, which recorded a 64.80 average on similar image tasks… Voyage-Multimodal-3 holds a MIEB score of approximately 63.58.

r/Rag — practitioner RAG benchmark reddit.com

Gemma embedding occasionally underperforms much smaller, specialized models like all-mpnet-base-v2 in RAG setups… retrieved irrelevant documents despite high similarity scores.

huggingface/transformers GitHub issue #45491 github.com

Batching sequences of mixed lengths can trigger NaN embeddings for shorter sequences… sliding-window attention windows fall entirely on padding tokens, producing a 0/0 softmax that silently propagates as NaN. Workaround: re-encode affected inputs with batch size one, or switch attn_implementation from sdpa to flash_attention_2.

arXiv 2510.26788 — Theoretical limits of embedding dimensions arxiv.org

Extreme dimension reduction fundamentally restricts the model’s ability to encode all possible relevant document-query combinations, particularly when dealing with hard negatives or complex multi-clause queries.

Hacker News discussion (item 49980487) news.ycombinator.com

Having access to open weights ensures that even if a hosted service disappears, developers can run the model locally… proprietary models pose a risk: if a provider discontinues a specific version, users are often forced to pay for the recalculation of millions of stored vectors.

The Decoder the-decoder.com

Google claims EmbeddingGemma 2 outperforms rival embedding models twice its size… multilingual text performance remained relatively stagnant, showing only marginal gains over version 1.

Sources

References

Jack Sun, writing.