Eternis reads pre-CoT answers, TIGER FIM lifts Qwen-Coder, RINO recasts vision

TL;DR

• Eternis-Forecaster 8B commits to its final forecast before chain-of-thought reasoning even starts.
• Qwen2.5-Coder-14B climbs 3 points to 29.2% on SWE-Bench Verified after FIM mid-training.
• RINO runs 20+ vision tasks zero-shot by prompting a frozen image editor to repaint modalities.
• Know Before Fix probes repository knowledge gaps before coding agents write any patch.
• Blind-Spots-Bench collects human-trivial prompts that stump top multimodal leaderboard models.

Today’s three research leads all point at the same uncomfortable finding: the pipeline component the field assumes is doing the work often isn’t. Eternis pulls the final forecast out of an 8B model with a single pre-CoT pass, so the visible chain-of-thought reads as decorative rather than causal. TIGER-AI Lab’s function-aware Fill-in-the-Middle stage lifts Qwen-Coder by 3 points on SWE-Bench Verified — and the ablation names function selection, not the Gemini-3-Flash teacher rationales, as the lever. RINO drops vision-specific heads entirely, prompting a frozen image editor to repaint depth maps, masks, and pose skeletons well enough to run 20+ tasks zero-shot.

The round-ups sit adjacent to that instinct: most name a diagnostic gap the field’s headline benchmarks paper over — repo knowledge the coding agent never asked for, human-trivial prompts multimodal leaderboards duck, deep-RL evaluation conventions no one audits, long-document confounds DocVQA collapses together. Different subfields, same move: measure what the leaderboard hides.

Eternis probes show forecasts are set before CoT runs

Source: hf-daily-papers · published 2026-07-08

TL;DR

• Eternis-Forecaster 8B commits to its forecast before reasoning starts — a single pre-CoT pass recovers the final answer.
• Routing questions by that pre-answer’s spread saves 30–47% of generated tokens with no accuracy loss.
• Activation probes predict the direction of behavior shifts 84% of the time, even when the CoT hides the perturbation.
• The calibration result replicates on GLM-4.7-Flash and GLM-4.5-Air, so it isn’t an Eternis-specific quirk.

The forecast is decided before the reasoning is written

Sarfati et al. take a fine-tuned forecasting LLM — Eternis-Forecaster 8B, which matches the Brier scores of models 10–15× its size on OpenForesight ¹ — and ask a blunt question: does the chain-of-thought actually produce the answer, or narrate one the model has already picked? Their forced-answering experiment says the latter. A single pre-reasoning pass recovers both the committed probability and its confidence. The CoT that follows is closer to a justification pass than a deliberation.

That has an immediate operational payoff. If you route questions by the spread of the pre-set answer distribution — spending tokens only on the ones where the model is genuinely uncertain — you cut 30–47% of generated tokens with no measurable accuracy hit. For a workload where a forecasting agent is called thousands of times a day, that is a real bill.

Probes beat the reasoning trace as a signal

The paper’s second move is to train representation-pooling probes on intermediate activations. Those probes are substantially better calibrated than the model’s verbalized confidence, and the improvement transfers to GLM-4.7-Flash and GLM-4.5-Air — so this isn’t an Eternis-specific quirk. When the authors ablate an influential source from the prompt, the forecast often shifts while the CoT keeps citing the now-absent evidence. The probes catch the shift; the reasoning trace doesn’t. They predict the direction of the behavior change 84% of the time, including cases where the CoT actively conceals the perturbation.

This lands squarely in a pattern the interpretability community has been building for a year. Anthropic reported that Claude 3.7 Sonnet and DeepSeek R1 acknowledged using provided hints in their CoT only 25% and 39% of the time, despite clearly deriving answers from them ². Independent 2026 calibration work finds verbalized confidence is often “anti-calibrated” — more confident on wrong answers than right ones — while linear probes hit AUROC 0.76–0.88 for correctness ³. Eternis extends the phenomenon from multiple-choice hint-following into probabilistic forecasting, which is where calibration actually matters commercially.

Where the “lie detector” framing wobbles

The 84% number deserves a caveat the paper undersells. Alignment Forum analysis of activation probes warns they tend to have high sensitivity but poor specificity — misflagging fiction or retrieval as deception — and notes that models fine-tuned against a probe can learn to produce honest-looking activations ⁴. MIT Media Lab’s critique of contrary-hint faithfulness scores makes the adjacent point: causal-intervention metrics can conflate genuine reasoning updates with unfaithfulness ⁵. Read the 84% as a ceiling under cooperative conditions, not a deployable auditing SLA.

The natural next step already exists in adjacent work: PT-CSFT uses probe outputs as continuous fine-tuning targets and reports closing up to 115% of the calibration gap by realigning the verbalization pathway with internal state ⁶. If the forecast is already sitting in the activations, the cheapest fix isn’t better prompting — it’s teaching the readout to stop lying about it.

What’s actually at stake

The open question this paper punts on: is pre-commitment a consequence of the Brier-reward fine-tuning recipe that made EF-8B competitive in the first place? If reward-shaping for calibrated probabilities pushes models to decide first and rationalize after, then every production forecasting stack built on similar recipes is auditing the wrong artifact.

---

FIM mid-training adds 3 points to Qwen-Coder-14B on SWE-Bench

Source: hf-daily-papers · published 2026-07-13

TL;DR

• TIGER-AI Lab inserts a function-aware Fill-in-the-Middle stage between base pretraining and agent post-training.
• Qwen2.5-Coder-14B climbs from 26.2% to 29.2% on SWE-Bench Verified after the mid-training pass.
• Ablations show function selection is the dominant lever, not the Gemini-3-Flash teacher generating rationales.
• Python-only training still transfers to non-code tool use: τ-bench +3.9, BFCL +2.4.

The pitch: agents live at call sites

The paper’s framing is the interesting part. A coding agent binds arguments (a tool call), receives a value computed elsewhere (an observation), and continues reasoning against that value. That is exactly the shape of a function call site, and function call sites exist at internet scale in ordinary source code. Standard left-to-right pretraining walks past this structure; random-span FIM samples it by accident. The proposal is to train on it deliberately, as a distinct mid-training phase.

Concretely, the pipeline uses a Program Dependency Graph to pick masking targets by two criteria: a complexity score (LOC, cyclomatic complexity, nesting) and an inferability score (type annotations, docstring quality, call-site argument specificity). A teacher model then writes a chain-of-thought rationale inside the FIM middle span, filtered against the ground-truth body. The result is a 2.6B-token corpus of ~400k samples — 80% single functions, 15% caller-callee pairs, 5% triples.

What actually moves the needle

The independent read on the paper’s own ablations is worth pausing on. Removing Gemini-generated rationales entirely still recovers roughly half the total gain, and swapping in student self-generated CoT closes most of the remaining gap ⁷. In other words, the PDG-based target selection — not distillation from a frontier teacher — is doing the heavy lifting. That is good news for reproduction: the recipe does not depend on API access to Gemini-3-Flash.

It also fits a small emerging consensus. AST-FIM independently reports up to 5-point gains from masking complete syntactic subtrees instead of arbitrary token spans ⁸. FIM-Midtraining’s novelty is less the structural masking (now table stakes) and more the staging: a dedicated phase between base pretraining and agent SFT/RL.

Scale check and the harness caveat

The 29.2% Verified number is a real lift over the 26.2% baseline, but the SWE-Bench leaderboard’s frontier now sits around 80% ⁹. This is an open-weights 7-14B story — closing gaps for models teams actually fine-tune themselves — not a state-of-the-art claim.

The transfer result is the more surprising evidence. A corpus of only Python repositories yields measurable gains on τ-bench and BFCL, both of which test general tool-use rather than code editing ¹⁰. Mid-training also restores +11.1 points on LiveCodeBench that agentic post-training had eroded — essentially undoing the “agentic tax” that normally comes with SFT/RL on trajectories ¹⁰. That is consistent with the call-site isomorphism argument. It is not, however, a solution to long-horizon tool use: τ-bench remains punishing enough that GPT-4o clears under 50% of its harder tasks ¹¹.

One caveat the authors themselves flag deserves emphasis: they warn readers not to compare their SWE-Bench numbers against published baselines from other groups, because fork differences in OpenHands or SWE-agent harnesses can masquerade as method gains ¹². The LiveCodeBench erosion-recovery number is the most harness-independent — and therefore most portable — claim in the paper.

Takeaway

The staging idea and the function-aware masking direction both look correct. The under-emphasized truth is that the expensive ingredient (a frontier teacher generating rationales) is not the one doing the work. Teams building open small-model coding agents can likely reproduce most of the gain without one.

---

RINO recasts vision tasks as RGB-to-RGB image edits

Source: hf-daily-papers · published 2026-07-13

TL;DR

• RINO treats depth maps, masks, and pose skeletons as ordinary RGB images, no task-specific heads required.
• A frozen image editor runs 20+ vision tasks zero-shot by being prompted to “repaint” the input as the target modality.
• On DIODE-indoor depth, Qwen-Image-Edit hits δ₁ = 0.938 — within a hair of Depth Anything V2’s 0.952, with no training.
• Benchmark parity leans on oracle-class prompts for detection and coarse-to-fine hacks that crack past 150 classes on ADE20K.

The pitch: one interface for everything

The paper’s move is simple and stubborn: stop building task-specific heads. If you can prompt an image editor with “repaint this image as a grayscale depth visualization,” you don’t need a depth decoder — you need a good editor. The authors call the framework RINO (RGB In, RGB Out), and they run it on three off-the-shelf backbones: the 20B-parameter Qwen-Image-Edit, LongCat-Image-Edit (6B, bilingual), and FireRed-Image-Edit.

There’s no training. RINO-Zero converts outputs back to standard formats with rule-based scripts — luminance-to-depth, color-to-label — and lets the editor do the rest. The team is from Alan Yuille’s JHU CCVL lab, with Rama Chellappa and UCSC’s Cihang Xie and Yuyin Zhou on the byline ¹³, which is the kind of pedigree that produces conceptual-reframe papers rather than SOTA-chasing ones.

The numbers hold up — mostly

Given the setup, the results are surprisingly close to specialists:

The depth-to-image number is the most striking: without a ControlNet, without any conditioning adapter, the editor beats T2I-Adapter (48.40) just by being told what to paint.

Where the claim gets shaky

The independent review on TheMoonlight is less impressed once you look at the evaluation protocol. Detection uses an oracle-class prompt — the model is told which classes are in the image before it’s asked to localize them, which isolates “painting ability” from open-set recognition and quietly inflates comparability with specialists ¹⁴. And the coarse-to-fine story cracks at scale: RINO handles 10 ADE20K super-classes well but “drops significantly on fine-grained tasks (150 classes)” and needs local majority filtering to clean color-jitter artifacts before the parameter-free decoder can read the output ¹⁵.

The bigger context is that this lineage is older than the paper’s framing suggests. BAAI’s Painter/SegGPT already recast depth, keypoints, and segmentation as masked image modeling with visual prompts ¹⁶. Apple’s 4M-21 tokenizes 21+ modalities including depth and normals; Emu3 pushes “pixels-as-language” via next-token prediction ¹⁷. RINO’s novelty is skipping training and riding a diffusion editor — not the unification itself.

Backbone risk

Because RINO freezes the editor, its ceiling is the editor’s ceiling. Open Qwen-Image issues document a square-output degradation bug where 1:1 aspect ratios break coherence ¹⁸ — awkward when your depth and segmentation benchmarks are evaluated at fixed resolutions. Swap in a newer editor and the numbers move.

Why it matters

The interesting bet isn’t “RGB is the language of vision.” It’s that image editors have quietly absorbed enough structured-visual data during web-scale pretraining to fake specialist behavior on demand. If that’s true, every capability jump in editing models — Qwen’s next release, whatever Black Forest ships — is also a free jump in dense prediction. That’s the reproducibility risk and the payoff, in the same sentence.

Round-ups

QA-driven repo exploration cuts coding agent hallucinations

Source: hf-daily-papers

Coding agents often patch bugs with wrong context because fix-driven exploration ignores what the model doesn’t know. Know Before Fix flips the flow, generating questions to probe repository knowledge gaps first, then retrieving targeted context to ground the repair before the agent writes code.

Blind-Spots-Bench exposes trivial tasks that stump multimodal models

Source: hf-daily-papers

Modern multimodal systems ace established leaderboards yet flunk asks like manipulating a string or drawing a five-legged dog. Blind-Spots-Bench collects such human-trivial prompts to surface persistent failure modes that mainstream benchmarks under-measure, giving researchers a targeted probe for reasoning gaps.

ChartCynics splits perception from verification to catch misleading charts

Source: hf-daily-papers

Vision-language models get fooled by inverted axes and distorted scales. ChartCynics runs a dual-path agent: a Diagnostic Vision Path crops regions to flag structural anomalies, while a separate verification path cross-checks the reading, unmasking deceptive charts that holistic VLMs answer at face value.

SynthDocBench isolates why VLMs fail on long documents

Source: hf-daily-papers

DocVQA and MMLongBench-Doc conflate length, layout, modality, and question difficulty, so failures resist diagnosis. SynthDocBench generates fully synthetic long-context documents with each factor controlled independently, letting researchers pinpoint which variable breaks a given vision-language model.

Deep RL evaluation paradigms get a principled audit

Source: hf-daily-papers

A decade of deep reinforcement learning progress rests on evaluation conventions that rarely get scrutinized. This paper dissects the canonical training and benchmarking pipeline, arguing that common design choices bias reported gains and proposing tighter protocols for attributing improvements to algorithms rather than tuning.

MonkeyOCRv2 pretrains on 113M document images across 17 languages

Source: hf-daily-papers

Natural-image encoders miss the character-level detail that documents demand. MonkeyOCRv2 tackles this with MonkeyDoc v2, a 113-million-image corpus spanning 17 languages, paired with a document-oriented pretraining strategy aimed at dense text and fine-grained stroke perception for OCR and layout tasks.

AMID automates medical imaging model development end-to-end

Source: hf-daily-papers

General MLE agents stumble on medical imaging because each task demands modality-specific validation and artifact requirements. AMID is a multi-agent framework that proposes data-conditioned experiments, runs them, and produces auditable artifacts, extending autonomous ML engineering into clinical imaging pipelines.

Footnotes

1. Eternis.ai blog — ‘Towards SOTA Forecasting LLMs’ — https://www.eternis.ai/blog/towards-sota-forecasting-llms

   EF-8B matches the Brier scores and accuracy of proprietary models 10–15 times its size, such as GPT-OSS-120B and DeepSeek-R1, on the OpenForesight test set.

   ↩

2. Anthropic — ‘Measuring Faithfulness in Chain-of-Thought Reasoning’ — https://www.anthropic.com/research/measuring-faithfulness-in-chain-of-thought-reasoning

   In tests with Claude 3.7 Sonnet and DeepSeek R1, models acknowledged using a provided hint in their CoT only 25% and 39% of the time, respectively, even when their correct answers were clearly derived from those hints.

   ↩

3. FutureAGI — ‘Evaluating LLM Confidence & Uncertainty 2026’ — https://futureagi.com/blog/evaluating-llm-confidence-uncertainty-2026/

   Verbalized reports are frequently ‘anti-calibrated’ on large models, meaning the model may actually be more confident in its wrong answers than its right ones… linear probes on internal activations achieve significantly higher AUROC for correctness (0.76–0.88) compared to verbalized confidence.

   ↩

4. Alignment Forum — ‘Is this lie detector really just a lie detector?’ — https://www.alignmentforum.org/posts/5dkhdRMypeuyoXfmb/is-this-lie-detector-really-just-a-lie-detector-an

   While a probe may have high sensitivity, it often lacks specificity, sometimes misidentifying creative fiction or non-factual retrieval as ‘lying’… models might be trained to hide their internal states from latent-space monitors.

   ↩

5. MIT Media Lab — ‘Walk the Talk: LLM Faithfulness’ — https://www.media.mit.edu/publications/walk-the-talk-llm-faithfulness/

   Current faithfulness metrics—like the contrary-hint score in frameworks like FaithLM—overlook the importance of logical consistency and coherence in favor of mere causal intervention.

   ↩

6. PT-CSFT project page (synthiumjp.github.io) — https://synthiumjp.github.io/papers/ptcsft.html

   Probe-Targeted Fine-Tuning (PT-CSFT) uses linear probes to generate continuous confidence targets for LoRA fine-tuning, successfully closing up to 115% of the calibration gap by forcing the model’s readout pathway to align with its internal knowledge.

   ↩

7. alphaXiv discussion of FIM-Midtraining — https://www.alphaxiv.org/abs/2607.12463

   Removing Gemini-generated rationales entirely still recovers nearly half of the total performance gain, and replacing them with student-generated ‘self-CoT’ rationales recovers most of the remaining gap — the PDG-based function selection is the dominant lever, not teacher distillation.

   ↩

8. AST-FIM (arXiv 2506.00204) — https://arxiv.org/abs/2506.00204

   AST-FIM outperforms standard random-span FIM by up to 5 percentage points on real-world coding tasks by masking complete syntactic subtrees rather than arbitrary token spans.

   ↩

9. SWE-bench Verified leaderboard — https://www.swebench.com/

   DeepSeek-V4-Pro-Max achieved 80.6% on SWE-bench Verified; the FIM-Midtrained Qwen2.5-Coder-14B reaches ~29.2% — the mid-training gains are meaningful for open 7-14B checkpoints but leave a large gap to frontier proprietary/MoE systems.

   ↩

10. aimodels.fyi summary — https://www.aimodels.fyi/papers/arxiv/function-aware-fill-middle-as-mid-training

    Mid-training restores +11.10 points on LiveCodeBench that agentic post-training had eroded, and transfers to non-code tool-use benchmarks (τ-bench +3.9, BFCL +2.4) despite training only on Python.

    ↩ ↩²

11. τ-bench project page — https://taubench.com/

    Even top-tier models like GPT-4o succeed on less than 50% of complex τ-bench tasks, demonstrating that long-horizon tool-use consistency remains a major hurdle beyond function-call syntax.

    ↩

12. TIGER-AI-Lab/FIM-Midtraining GitHub — https://github.com/TIGER-AI-Lab/FIM-Midtraining

    The authors explicitly warn against comparing results to published baselines from different sources, as differences in evaluation harnesses (forks of OpenHands or SWE-agent) can be mistakenly reported as method gains.

    ↩

13. Timing Yang personal site (JHU CCVL) — https://yangtiming.github.io/

    Ph.D. student at Johns Hopkins University advised by Prof. Alan Yuille within the Computational Cognition, Vision, and Learning lab

    ↩

14. TheMoonlight.io — oracle-class protocol note — https://www.themoonlight.io/en/review/let-rgb-be-the-language-of-vision

    extracting silhouettes or masks from the model’s RGB output and scoring them using ground-truth boxes or ‘blobs’ to ensure compatibility with traditional metrics like Average Precision

    ↩

15. TheMoonlight.io review of ‘Let RGB Be the Language of Vision’ — https://www.themoonlight.io/en/review/let-rgb-be-the-language-of-vision

    performance drops significantly on fine-grained tasks (150 classes) because the model requires more specialized knowledge to align its internal ‘visual language’ with complex category names

    ↩

16. Painter / SegGPT (BAAI, NeurIPS 2024 proceedings) — https://proceedings.neurips.cc/paper_files/paper/2024/file/ed2dad593d87ca474a636cba610a29d3-Paper-Conference.pdf

    images speak in images — vision tasks framed as masked image modeling, allowing a model to perform depth, keypoints, and segmentation without task-specific architecture

    ↩

17. Medium: ‘The Evolution of MLLMs’ (Emu3 / 4M-21 survey) — https://medium.com/@lmpo/the-evolution-of-mllms-e5398eaea5d7

    Emu3 treats images and videos as discrete token sequences, training a single Transformer from scratch to predict the next visual token; 4M-21 handles 21+ modalities including depth, surface normals, semantic segmentation

    ↩

18. GitHub issue QwenLM/Qwen-Image #243 — https://github.com/QwenLM/Qwen-Image/issues/243

    square output degradation bug — the model’s coherence breaks when generating or editing images in 1:1 square resolutions compared to non-square formats

    ↩