Images 2.5 trails Nano Banana, Multiverse cuts refusal to 5% on saturated XSTest

TL;DR

• Images 2.5 splits into Flare and Sunburst at flat $8/$30 per M tokens with 50% latency cut.
• Sunburst and Flare took #1 and #2 on LMSYS Image Arena within days of launch.
• Nano Banana Pro still leads independent fidelity benchmarks, making 2.5 a catch-up release.
• Multiverse cuts XSTest over-refusal from 74% to 5.20% on Qwen3-8B via self-distillation.
• XSTest is now considered saturated after ICML’s 80,000-prompt OR-Bench release.

Two tech ships today, one pattern: the marquee number lands on a scoreboard the vendor picked, and the harder measure tells a quieter story. OpenAI’s Images 2.5 takes the top two slots on LMSYS Image Arena within days of launch, but Google’s Nano Banana Pro still leads the independent fidelity benchmarks — and the shipped multi-turn consistency still drifts on character across sequential edits. Multiverse Computing posts a headline collapse of XSTest over-refusal from 74% to 5.20% on Qwen3-8B, but XSTest is the benchmark practitioners now treat as saturated after ICML’s 80,000-prompt OR-Bench release.

Neither result is fake; both are real engineering. The frame worth carrying into the features is that the scoreboard chosen at launch keeps flattering the ship in ways the next-generation benchmark won’t.

OpenAI Images 2.5 speeds up, but edits still cascade globally

Source: openai-blog · published 2026-09-08

TL;DR

• Images 2.5 splits into Flare and Sunburst — flat $8/$30 per M tokens, 50% latency cut on Flare.
• Sunburst and Flare took #1 and #2 on LMSYS Image Arena within days of launch.
• Google’s Nano Banana Pro still leads independent fidelity benchmarks, making 2.5 a catch-up release.
• Character drift persists across sequential edits despite the marquee “multi-turn consistency” claim.

A catch-up release dressed as a leap

The Images 2.5 drop — a primary launch post plus a Verge write-up centered on the new Sketch canvas — is the first OpenAI image release framed almost entirely around workflow, not pixels. Sketch lets you doodle a rough layout as a structural guide, Templates ship preset flyers and social formats, and on-image commenting lets you pin edits to specific regions. The API splits into GPT-Image-2.5 Flare (fast, 2–4× speedup for iteration) and Sunburst (high-fidelity, marketing-asset tier). Both landed at #1 and #2 on the LMSYS Image Arena within days ¹.

Read against the competitive field, though, this is a catch-up. Artificial Analysis’s editing leaderboard still puts Microsoft’s MAI-Image-2.6 above OpenAI on targeted edits, and Google’s Nano Banana Pro (Gemini 3 Pro Image) remains the reference point for 4K photorealism and long-form in-image text ². The Sketch-plus-templates push is OpenAI competing on ergonomics because it can’t yet compete on raw fidelity.

What breaks under real use

Character drift persists where OpenAI’s headline “multi-turn consistency” claim is loudest. Testers report that button colors, hair tone, and small UI glyphs still shift across sequential edits, and one round of tech coverage called the Sketch canvas “MS Paint from 1995” ³. Reddit threads around the launch also flag over-blocking on benign prompts as the content filter tightened ³. Sunburst likely delivers on the marketing-asset use case it’s tuned for; the general “just keep editing” workflow OpenAI implies is still fragile.

Provenance theater

The safety section leans on C2PA metadata plus invisible watermarking. Independent forensic work published alongside the launch documents an “Integrity Clash”: C2PA manifests can be stripped and re-signed while a SynthID-style pixel watermark persists — or vice versa — producing “authenticated fakes” that pass provenance checks ⁴. Social platforms already strip C2PA on re-encode, so the deployed guarantee is thinner than the post reads.

The artist-facing story is no cleaner. The Sketch launch reopened the Ghibli fight — Toshio Suzuki called the outputs “meaningless and not fun” — and clarified that OpenAI blocks individual living artists’ names but explicitly permits “studio styles,” a carve-out illustrators treat as a laundering mechanism ⁵. That distinction is heading for a court test amid an unusually hot copyright docket.

Why it ships anyway

None of this dents distribution. ChatGPT sits at ~1 billion weekly actives with 92% of the Fortune 500 integrated ⁶, and Images 2.5 is already generating 3B images a week. Even a second-place image model with a shaky provenance story reaches a captive audience most competitors can’t touch. The question the launch actually poses isn’t whether Sunburst beats Nano Banana Pro — it’s whether default placement inside ChatGPT matters more than being best.

Further reading

• ChatGPT Sketch turns your bad drawings into detailed AI images — the-verge-ai

---

Multiverse cuts XSTest over-refusal from 74% to 5%

Source: huggingface-blog · published 2026-09-08

TL;DR

• Multiverse Computing cuts XSTest over-refusal from 74% to 5.20% on Qwen3-8B via boundary-aware self-distillation.
• The trick: 1,539 boundary pairs of same-topic, opposite-intent prompts teach the model where “harmful” actually starts.
• Headline gains rest on XSTest, a benchmark practitioners now consider saturated after ICML’s 80,000-prompt OR-Bench release.
• The pitch is deployment-specific safety: one base model, different refusal boundaries per use case, with FakeHarm gated on release.

The pitch: sharper lines, not bigger walls

Most safety tuning treats “politics” or “medicine” as a topic to swerve around. Multiverse’s Boundary-Aware Self-Distillation reframes the target as a subset of each topic — the harmful prompts — and explicitly trains the model to answer the benign complement. On Qwen3-8B, that pushed refusal of manipulation-flavoured political prompts from 9.47% to 84.75%, while the general unsafe-response rate across HarmBench, StrongREJECT and WildJailbreak collapsed from 26.26% to 0.14%.

The interesting engineering isn’t the outcome, it’s the data recipe. Standard self-distillation pipelines throw away any prompt the base model refuses to refuse cleanly; Multiverse’s escalating-steering retry loop rescues almost all of them, taking discarded examples from 8,009 down to 79. Then 1,539 held-out boundary pairs — two prompts about the same topic, opposite intent — teach the model where the line actually is rather than letting it memorise keywords. A separate 11,955-prompt pool of “surface-dangerous” benign examples reinforces the distinction.

Why the XSTest numbers deserve an asterisk

The blog’s headline over-refusal win runs 74% → 5.20% on XSTest ⁷. (A separate, tighter 32.94% → 4.16% figure applies to the held-out benign side of Multiverse’s own boundary pairs, not XSTest — worth keeping straight.) The XSTest move is real, but XSTest is the benchmark AI researchers spent 2025 warning about. ICML’s OR-Bench paper introduced 80,000 fresh prompts precisely because frontier models saturate XSTest while still over-refusing in the wild: Claude 3.5 Sonnet posts 43.8% over-refusal on OR-Bench’s hard subset despite near-perfect XSTest scores ⁸. Sebastian Raschka’s 2025 review names this pattern “benchmaxxing” and warns that XSTest wins rarely survive production traffic ⁹.

Multiverse reports no OR-Bench, FalseReject, or WildGuard numbers. For a paper pitched as a general solution to over-refusal, that’s the gap the next reviewer will circle.

Who decides which politics is “harmful”?

The worked example — “targeted political manipulation” as harmful, “factual civics questions” as benign — is doing more work than the paper admits. Independent political-bias testing shows refusal behaviour is highly framing-dependent: Anomify found Claude Opus 4.6 refuses ~32% of political questions when given an explicit opt-out but 0% under forced choice ¹⁰. Whatever LlamaGuard-3 scored as “refused” in Multiverse’s pipeline is measuring surface compliance, not substantive neutrality, and the choice of which subset counts as manipulation sits entirely with the data curator.

The arXiv version is more candid about this than the blog: the same base model can power a civics tutor and a public-sector assistant with different boundaries, and the FakeHarm dataset will be released “for safety research purposes only” because the harmful split contains genuinely persuasive prompts ¹¹. That’s a governance stance, not just a training trick.

Net take

Boundary pairs plus coverage repair plus in-distribution benign data is a reproducible contribution, and the framing — safety as a per-deployment boundary rather than a topic-wide wall — is the right direction. But this is also Multiverse Computing’s second research-marketing beat of the year, after $215M raised on compression claims some practitioners read as repackaged low-rank decomposition ¹². Treat the headline percentages as the ceiling, not the floor, until someone reruns the recipe against OR-Bench.

Footnotes

1. Unite.ai — https://www.unite.ai/openai-releases-chatgpt-images-2-5-with-sketch-and-two-new-api-models/

   OpenAI has standardized pricing for both models at $8.00 per million input tokens and $30.00 per million output tokens… On the LMSYS Image Arena, Sunburst and Flare quickly secured the #1 and #2 rankings, respectively.

   ↩

2. Feisworld (designer hands-on) — https://www.feisworld.com/blog/adobe-firefly-chatgpt-campaign-visuals

   Independent benchmarks from Artificial Analysis indicate that while OpenAI’s models lead in general text-to-image quality, specialized models like Microsoft’s MAI-Image-2.6 often top editing leaderboards… Nano Banana Pro has been called the ‘world’s highest-rated’ for fidelity.

   ↩

3. Economic Times — https://economictimes.indiatimes.com/ai/ai-insights/chatgpt-images-2-5-whats-new-in-openais-latest-image-model/articleshow/133956278.cms?from=mdr

   Some tech journalists likened the drawing canvas to ‘MS Paint from 1995’… users report that facial features and specific details (such as the color of a button or the grayness of a character’s hair) often change across multiple turns, even with the improved ‘multi-turn consistency’ promised in 2.5.

   ↩ ↩²

4. C2PA Viewer — https://c2paviewer.com/articles/openai-google-c2pa-synthid-2026

   Independent researchers have identified a vulnerability termed the ‘Integrity Clash.’ This condition occurs when a digital asset carries a valid C2PA manifest claiming human authorship while its pixels simultaneously contain an AI watermark… metadata can be ‘washed’ and replaced through standard editing pipelines without compromising the cryptographic signatures, leading to ‘authenticated fakes’.

   ↩

5. eesel.ai — https://www.eesel.ai/blog/chatgpt-images-2-5-alternatives

   Studio Ghibli president Toshio Suzuki dismissed the outputs as ‘meaningless and not fun’… While OpenAI maintains that it blocks the names of individual living artists, it explicitly permits ‘studio styles,’ a distinction that many artists find legally and ethically porous.

   ↩

6. Master of Code (ChatGPT stats) — https://masterofcode.com/blog/chatgpt-statistics

   ChatGPT reached 1 billion weekly active users… OpenAI currently serves over 1 million paying business customers, with 92% of Fortune 500 companies having integrated the platform.

   ↩

7. Röttger et al., XSTest paper (ResearchGate) — https://www.researchgate.net/publication/382627021_XSTest_A_Test_Suite_for_Identifying_Exaggerated_Safety_Behaviours_in_Large_Language_Models

   A test suite for identifying exaggerated safety behaviours in large language models — the same XSTest that Multiverse’s paper uses as its primary over-refusal yardstick.

   ↩

8. ICML 2025 poster — OR-Bench: An Over-Refusal Benchmark for LLMs — https://icml.cc/virtual/2025/poster/45159

   OR-Bench… 80,000 prompts across ten categories… Claude 3.5 Sonnet shows a high over-refusal rate of 43.8% on OR-Bench despite its reputation for safety, while models saturate XSTest at near-100%.

   ↩

9. Sebastian Raschka — State of LLMs 2025 — https://magazine.sebastianraschka.com/p/state-of-llms-2025

   A growing concern among AI developers in 2025 is ‘benchmaxxing’—labs over-optimize models for public leaderboards at the expense of real-world utility… high scores on academic benchmarks like XSTest often fail to reflect performance in production environments.

   ↩

10. Anomify 2025 political bias benchmarks — https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHczwn0tQOoGLDKXBT69JIXyzQDTNK9maUHfSFSTqI9HXtir7cXn4ApRjtvnONNcPk5m1QRcXD7IFgnjFoSjaZLimwzeTwQw5qjPqIAvCgkscJksGSgTiDj

    Claude Opus 4.6 refused approximately 32% of questions when given a formal ‘prefer not to answer’ option, compared to zero refusals in forced-choice settings — refusals are strategic avoidance of subjective bias rather than hard-coded censorship.

    ↩

11. arXiv preprint (Multiverse Computing) — Boundary-Aware Self-Distillation — https://arxiv.org/html/2609.04482v1

    A civics tutor and a public-sector assistant may share the same base model while requiring distinct refusal boundaries… datasets [including politics and religion] and generation code will be released for safety research purposes only, as the harmful split contains persuasive and manipulative prompts.

    ↩

12. Quantum Zeitgeist — Multiverse Computing funding coverage — https://quantumzeitgeist.com/multiverse-computing-secures-215m-to-scale-ai-model-compression-technology/

    Multiverse Computing secures $215M to scale AI model compression technology… some technical observers argue the technique closely resembles classical low-rank decomposition or SVD and question whether the ‘quantum’ terminology is primarily marketing.

    ↩