arxiv:2610.23987

What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

Published on Oct 14

· Submitted by

Tengju_Ru on Oct 23

Upvote

Authors:

Tengju Ru ,

Zecong Tang ,

Kaixuan Wang ,

Abstract

GAFU leverages native generation-side representations to improve fine-grained visual understanding in unified multimodal models by compressing them into compact tokens for task-conditioned retrieval.

Generated by thinkingmachines/Inkling-Small

Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding. However, existing generation-assisted understanding methods mainly rely on generative supervision or explicit generation and have not systematically explored direct use of native generation-side representations. To address this gap, we systematically analyze what generation-side representations offer for fine-grained understanding, finding that they are more sensitive to fine-grained semantic changes, capture task-relevant changes missed by understanding-side representations, and depend on correct spatial organization for this advantage. Guided by these findings, we design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. It compresses generation-side visual representations into compact \(G\) tokens that preserve fine-grained visual details, enabling task-conditioned retrieval by the understanding pathway. Experiments show that GAFU consistently outperforms SFT baselines trained on the same data across diverse visual understanding tasks and UMM architectures, with maximum relative improvements of 20.00\%, 17.94\%, and 8.62\% on VStar-Bench, OmniSpatial, and VSR-Bench, respectively. Overall, GAFU converts complementary information from the generation pathway into gains in fine-grained visual understanding, providing an effective path toward synergy between generation and understanding in UMMs.

View arXiv page View PDF GitHub 28 Add to collection

Community

Lruca

Paper author Paper submitter 2 days ago

🚀 We are excited to share our latest work: "What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models"!

Unified multimodal models (UMMs) can generate images with rich visual details. But can the internal representations underlying their generation capabilities also help improve fine-grained multimodal understanding?

Existing generation-assisted understanding methods primarily rely on generative supervision or explicitly generated intermediate outputs, while the potential of native generation-side representations remains largely unexplored. We aim to investigate whether these representations can be leveraged to enhance fine-grained visual understanding.

To this end, we systematically investigate the visual representations of generation and understanding pathways in UMMs. Building on our findings, we introduce GAFU (Generation-Assisted Fine-grained Understanding), a post-training framework that directly leverages native generation-side representations to enhance fine-grained visual understanding.

🔑 Key Highlights

🔍 Systematic Investigation: We systematically compare generation-side and understanding-side representations across six UMMs, revealing key properties in fine-grained semantic sensitivity, representational complementarity, and spatial dependence. Our findings show that generation-side representations are more sensitive to fine-grained semantic changes and capture visual variations that are less effectively reflected in understanding-side representations, with these advantages closely tied to spatial organization.
🧩 GAFU Framework: We propose a post-training framework that leverages native generation-side representations without requiring explicit image generation at inference. GAFU compresses and aligns generation-side features into compact G tokens, enables task-conditioned retrieval and injection of complementary information, and incorporates pairwise preference learning to encourage effective utilization of generation-side representations.
📈 Extensive Experimental Validation: We evaluate GAFU across three unified multimodal architectures (BAGEL, Janus-Pro, and Lance) and five benchmarks covering general, spatial, and domain-specific understanding. GAFU consistently outperforms matched-data SFT baselines across all 15 model–benchmark combinations, achieving up to 20.00%, 17.94%, and 8.62% relative improvements on VStar-Bench, OmniSpatial, and VSR-Bench, respectively.

Our findings demonstrate that the generation capabilities of UMMs are valuable not only for producing visual content but also for providing complementary representations that enhance fine-grained visual understanding, opening up new opportunities for synergy between understanding and generation.

📄 Paper: https://arxiv.org/abs/2610.23987

💻 Code: https://github.com/LRuca/GAFU

We welcome feedback and discussions!