arXiv cs.AI· Xinhang Zhang, Boning Zhang, Chengchun Liu, Chunpu Li, Lei Sun, Weifeng Zhang, Limin Xiao·· 2 小时前AI 评分36
CHARM+：跨硬件注意力机制结合重合并多流机制
CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism
AI 导读
CHARM+ 是一种针对序列并行注意力（如 Ring-Attention）通信瓶颈的优化方法，通过分级重组通信实现并行与计算重叠，并将跨硬件通信与硬件内通信解耦，消除带宽不均衡带来的同步开销。
正文 · 原文
Abstract:As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of 1.3x and 1.7x on a dual-NUMA PCIe architecture, and 2.8x and 2.9x on a dual-node NVLink architecture, respectively.
| Comments: | Accepted to NeurIPS 2026 (Poster) |
| Subjects: | Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2611.05906 [cs.AI] |
| (or arXiv:2611.05906v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05906 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Boning Zhang [view email]
[v1]
Wed, 4 Nov 2026 03:57:56 UTC (758 KB)
来源：arXiv cs.AI · arxiv.org