跳到正文
arXiv cs.AI· Xiaoyu Wu, Yifei Wang, Weihang Guo, Chen Wei, Lydia E Kavraki, Niloofar Mireshghallah, Steven Wu·· 2 小时前AI 评分38

分层扩散 Transformer HiT 实现端到端图像生成

Hierarchical Diffusion Transformer for End-to-End Image Generation

AI 导读

论文提出分层扩散 Transformer(HiT),通过逐步分组 token 并用压缩预算调节压缩率,在单阶段像素空间训练中学习视觉层次结构。

正文 · 原文

View PDF HTML (experimental)

Abstract:Pixel-space diffusion learns visual representations and image generation jointly in a single end-to-end training stage, without a separately pretrained visual encoder. However, plain pixel-space Transformers still trail leading two-stage latent diffusion models in generation quality and training efficiency. We hypothesize that part of this gap arises from retaining one token per patch throughout the backbone, which does not explicitly encourage a visual hierarchy that summarizes related content at progressively coarser levels. To address this, we introduce Hierarchical Diffusion Transformers (HiT), which progressively group related tokens and represent each group with a single token. The main Transformer bottleneck processes these compact group tokens, and ungrouping restores features to their original image positions. A compression budget regulates the average compression ratio across the batch, allowing group memberships and counts to vary across images and diffusion timesteps during both training and generation. The model exhibits emergent semantic structure, learning to group coherent objects and object parts without segmentation labels. In text-to-image generation, HiT further learns to group caption words with image patches depicting the same concepts, without region annotations. HiT achieves 1.48 FID on ImageNet $256\times256$ and reaches comparable generation quality to JiT using approximately 40\% of its cumulative training FLOPs. For text-to-image generation, HiT exceeds UniLiP's average score across five benchmarks with roughly $6\times$ fewer denoiser parameters and $3\times$ fewer forward FLOPs. These results suggest that learning a visual hierarchy within the generator breaks down the barrier to stronger image generation without external visual pretraining.
Comments: Under Review
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.22962 [cs.CV]
  (or arXiv:2611.22962v1 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.22962

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xiaoyu Wu [view email]
[v1] Tue, 17 Nov 2026 15:47:56 UTC (23,409 KB)

来源:arXiv cs.AI · arxiv.org