Eternis reads pre-CoT answers, TIGER FIM lifts Qwen-Coder, RINO recasts vision

Sources

What LLM Forecasters Know but Don’t Say: Probing Internal Representations for Calibration and Faithfulness huggingface.co

Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-A

Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models huggingface.co

Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We explo

Let RGB Be the Language of Vision huggingface.co

This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to tran

Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution huggingface.co

Coding agents often patch bugs with wrong context because fix-driven exploration ignores what the model doesn’t know. Know Before Fix flips the flow, generating questions to probe repository knowledge gaps first, then retrieving targeted context to ground the repair before the agent writes code.

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models huggingface.co

Modern multimodal systems ace established leaderboards yet flunk asks like manipulating a string or drawing a five-legged dog. Blind-Spots-Bench collects such human-trivial prompts to surface persistent failure modes that mainstream benchmarks under-measure, giving researchers a targeted probe for reasoning gaps.

Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms huggingface.co

A decade of deep reinforcement learning progress rests on evaluation conventions that rarely get scrutinized. This paper dissects the canonical training and benchmarking pipeline, arguing that common design choices bias reported gains and proposing tighter protocols for attributing improvements to algorithms rather than tuning.

Towards Autonomous and Auditable Medical Imaging Model Development huggingface.co

General MLE agents stumble on medical imaging because each task demands modality-specific validation and artifact requirements. AMID is a multi-agent framework that proposes data-conditioned experiments, runs them, and produces auditable artifacts, extending autonomous ML engineering into clinical imaging pipelines.

MonkeyOCRv2: A Visual-Text Foundation Model for Document AI huggingface.co

Natural-image encoders miss the character-level detail that documents demand. MonkeyOCRv2 tackles this with MonkeyDoc v2, a 113-million-image corpus spanning 17 languages, paired with a document-oriented pretraining strategy aimed at dense text and fine-grained stroke perception for OCR and layout tasks.

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering huggingface.co

Vision-language models get fooled by inverted axes and distorted scales. ChartCynics runs a dual-path agent: a Diagnostic Vision Path crops regions to flag structural anomalies, while a separate verification path cross-checks the reading, unmasking deceptive charts that holistic VLMs answer at face value.

SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding huggingface.co

DocVQA and MMLongBench-Doc conflate length, layout, modality, and question difficulty, so failures resist diagnosis. SynthDocBench generates fully synthetic long-context documents with each factor controlled independently, letting researchers pinpoint which variable breaks a given vision-language model.

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists huggingface.co

Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) ac

Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation huggingface.co

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as t

Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation huggingface.co

Visual generators excel at rendering, but they confidently fabricate what they do not know. User requests are unbounded, evolving, and deeply long-tailed: new characters, trending entities, post-cutoff events, and more. This world-knowledge bottleneck is structural: generators are trained on fixed corpora, but the visual world is open-ended. We construct SearchGen-20K and SearchGen-Bench, with 20,839 prompts spanning twelve failure categories and twenty-two domains, paired with a pre-executed mu

MAGIC: Transition-Aware Generation of Navigable Multi-Scene Game Worlds with Large Language Models huggingface.co

Multi-scene navigation (clearing an objective in one bounded space and then crossing a portal into the next) is a defining feature of contemporary 3D games, but authoring it is laborious: every portal must have consistent endpoints on both sides, each interior must remain navigable once it is furnished, and the resulting connectivity must be kept consistent across many files. Recent large language model (LLM) and multimodal LLM (MLLM) scene generators have made single-interior synthesis dramatic

MuScriptor: An Open Model for Multi-Instrument Music Transcription huggingface.co

Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement le

References

Eternis.ai blog — ‘Towards SOTA Forecasting LLMs’ eternis.ai

EF-8B matches the Brier scores and accuracy of proprietary models 10–15 times its size, such as GPT-OSS-120B and DeepSeek-R1, on the OpenForesight test set.

Anthropic — ‘Measuring Faithfulness in Chain-of-Thought Reasoning’ anthropic.com

In tests with Claude 3.7 Sonnet and DeepSeek R1, models acknowledged using a provided hint in their CoT only 25% and 39% of the time, respectively, even when their correct answers were clearly derived from those hints.

Alignment Forum — ‘Is this lie detector really just a lie detector?’ alignmentforum.org

While a probe may have high sensitivity, it often lacks specificity, sometimes misidentifying creative fiction or non-factual retrieval as ‘lying’… models might be trained to hide their internal states from latent-space monitors.

PT-CSFT project page (synthiumjp.github.io) synthiumjp.github.io

Probe-Targeted Fine-Tuning (PT-CSFT) uses linear probes to generate continuous confidence targets for LoRA fine-tuning, successfully closing up to 115% of the calibration gap by forcing the model’s readout pathway to align with its internal knowledge.

MIT Media Lab — ‘Walk the Talk: LLM Faithfulness’ media.mit.edu

Current faithfulness metrics—like the contrary-hint score in frameworks like FaithLM—overlook the importance of logical consistency and coherence in favor of mere causal intervention.

FutureAGI — ‘Evaluating LLM Confidence & Uncertainty 2026’ futureagi.com

Verbalized reports are frequently ‘anti-calibrated’ on large models, meaning the model may actually be more confident in its wrong answers than its right ones… linear probes on internal activations achieve significantly higher AUROC for correctness (0.76–0.88) compared to verbalized confidence.

AST-FIM (arXiv 2506.00204) arxiv.org

AST-FIM outperforms standard random-span FIM by up to 5 percentage points on real-world coding tasks by masking complete syntactic subtrees rather than arbitrary token spans.

alphaXiv discussion of FIM-Midtraining alphaxiv.org

Removing Gemini-generated rationales entirely still recovers nearly half of the total performance gain, and replacing them with student-generated ‘self-CoT’ rationales recovers most of the remaining gap — the PDG-based function selection is the dominant lever, not teacher distillation.

SWE-bench Verified leaderboard swebench.com

DeepSeek-V4-Pro-Max achieved 80.6% on SWE-bench Verified; the FIM-Midtrained Qwen2.5-Coder-14B reaches ~29.2% — the mid-training gains are meaningful for open 7-14B checkpoints but leave a large gap to frontier proprietary/MoE systems.

τ-bench project page taubench.com

Even top-tier models like GPT-4o succeed on less than 50% of complex τ-bench tasks, demonstrating that long-horizon tool-use consistency remains a major hurdle beyond function-call syntax.

TIGER-AI-Lab/FIM-Midtraining GitHub github.com

The authors explicitly warn against comparing results to published baselines from different sources, as differences in evaluation harnesses (forks of OpenHands or SWE-agent) can be mistakenly reported as method gains.

aimodels.fyi summary aimodels.fyi

Mid-training restores +11.10 points on LiveCodeBench that agentic post-training had eroded, and transfers to non-code tool-use benchmarks (τ-bench +3.9, BFCL +2.4) despite training only on Python.

Timing Yang personal site (JHU CCVL) yangtiming.github.io

Ph.D. student at Johns Hopkins University advised by Prof. Alan Yuille within the Computational Cognition, Vision, and Learning lab

TheMoonlight.io review of ‘Let RGB Be the Language of Vision’ themoonlight.io

performance drops significantly on fine-grained tasks (150 classes) because the model requires more specialized knowledge to align its internal ‘visual language’ with complex category names

GitHub issue QwenLM/Qwen-Image #243 github.com

square output degradation bug — the model’s coherence breaks when generating or editing images in 1:1 square resolutions compared to non-square formats

Painter / SegGPT (BAAI, NeurIPS 2024 proceedings) proceedings.neurips.cc

images speak in images — vision tasks framed as masked image modeling, allowing a model to perform depth, keypoints, and segmentation without task-specific architecture

Medium: ‘The Evolution of MLLMs’ (Emu3 / 4M-21 survey) medium.com

Emu3 treats images and videos as discrete token sequences, training a single Transformer from scratch to predict the next visual token; 4M-21 handles 21+ modalities including depth, surface normals, semantic segmentation