arXiv cs.AI· Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro, Po-An Tsai, Shuyi Pei, Yangwook Kang, Yufei Ding·· 4 小时前AI 评分42
AMMA：面向低延迟 1M 上下文注意力推理的多芯粒内存中心架构
AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
AI 导读
研究者提出 AMMA，一种面向长上下文注意力推理的多芯粒、内存中心架构，用 HBM-PNM 立方体取代 GPU 计算核心，将可用内存带宽提升约一倍。架构还包括逻辑核心微架构、两级混合并行和重排序集合通信流等设计。评测显示，与 NVIDIA H100 相比，AMMA 的注意力延迟降低 15.5X，能耗降低 6.9X。
正文 · 原文
Authors:Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro, Po-An Tsai, Shuyi Pei, Yangwook Kang, Yufei Ding
Abstract:All current LLM serving systems place the GPU at the center, from production-level attention-FFN disaggregation to NVIDIA's Rubin GPU-LPU heterogeneous platform. Even academic PIM/PNM proposals still treat the GPU as the central hub for cross-device communication. Yet the GPU's compute-rich architecture is fundamentally mismatched with the memory-bound nature of decode-phase attention, inflating serving latency while wasting power and die area on idle compute units. The problem is compounded as reasoning and agentic workloads push context lengths toward one million tokens, making attention latency the primary user-facing bottleneck.
To address these inefficiencies, we present AMMA, a multi-chiplet, memory-centric architecture for low-latency long-context attention. AMMA replaces GPU compute dies with HBM-PNM cubes, roughly doubling the available memory bandwidth to better serve memory-bound attention workloads. To translate this bandwidth into proportional performance gains, we introduce (i) a logic-die microarchitecture that fully exploits per-cube internal bandwidth for decode attention under a minimal power and area budget, (ii) a two-level hybrid parallelism scheme, and (iii) a reordered collective flow that reduces intra-chip die-to-die communication overhead. We further conduct a design-space exploration over per-cube compute power and intra-chip D2D link bandwidth, providing actionable guidance for hardware designers. Evaluations show that AMMA achieves 15.5X lower attention latency and 6.9X lower energy consumption compared with the NVIDIA H100.
| Comments: | Accepted to the 59th IEEE/ACM International Symposium on Microarchitecture (MICRO 2026) |
| Subjects: | Hardware Architecture (cs.AR); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.26103 [cs.AR] |
| (or arXiv:2604.26103v3 [cs.AR] for this version) | |
| https://doi.org/10.48550/arXiv.2604.26103 arXiv-issued DOI via DataCite |
Submission history
From: Zhongkai Yu [view email]
[v1]
Tue, 28 Apr 2026 20:36:50 UTC (1,071 KB)
[v2]
Thu, 30 Apr 2026 09:10:31 UTC (1,071 KB)
[v3]
Fri, 30 Oct 2026 06:58:33 UTC (1,188 KB)
来源：arXiv cs.AI · arxiv.org