ReMoA：基于稀疏潜在精炼的多模态扩散模型递归模块化自适应框架

View PDF HTML (experimental)

Abstract:Diffusion models have achieved success in high-fidelity data synthesis, yet their capacity for more complex, structured reasoning such as text following remains constrained. While advances in language models have leveraged strategies such as latent reasoning and recursion to enhance text understanding capabilities, extending these to multimodal text-to-image generation tasks is challenging due to the continuous and non-discrete nature of visual tokens. To tackle this problem, we draw inspiration from modular human cognition and propose a Recursive Modular Adaptation (ReMoA) framework integrated into joint attention layers of conventional diffusion models, iteratively refining visual tokens over multiple recursion steps while efficiently sharing parameters via sparse selection of neural modules. Comprehensive evaluations on both class-conditioned ImageNet generation and text-to-image generation using the GenEval and DPG benchmarks demonstrate that recursive visual latent refinement contributes to richer object details and better-aligned outputs. Additionally, visual latent trajectory and routing analyses reveal that neural modules learn to specialize across diffusion timesteps, with each module's trajectories diverging at later, low-noise steps to support context-adaptive refinement. ReMoA increases recall for DiT-XL from 0.57 to 0.62 on ImageNet, and improves the overall GenEval score for SD3-medium from 68.33% to 71.18%, without introducing additional parameters through recursion.

Subjects:	Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2604.25299 [cs.CV]
	(or arXiv:2604.25299v2 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2604.25299 arXiv-issued DOI via DataCite

Submission history

From: Yuwei Sun [view email]
[v1] Tue, 28 Apr 2026 07:09:18 UTC (5,791 KB)
[v2] Wed, 4 Nov 2026 04:13:44 UTC (7,737 KB)