跳到正文
arXiv cs.AI· Tomas Ruiz, Zhen Qin, Yifan Zhang, Xuyang Shen, Yiran Zhong, Mengdi Wang·· 4 小时前AI 评分39

FlashSampling:快速且内存高效的精确采样

FlashSampling: Fast and Memory-Efficient Exact Sampling

AI 导读

FlashSampling 是一种精确采样原语,将采样融合进 LM head 矩阵乘法,无需在 HBM 中生成 logits 张量,通过逐块计算 logits 并加 Gumbel 噪声、保留每行最大值实现精确采样。

正文 · 原文

View PDF HTML (experimental)

Abstract:Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We present FlashSampling, an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in HBM. The method is simple: compute logits tile-by-tile on chip, add Gumbel noise, keep only one maximizer per row and per vocabulary tile, and finish with a small reduction over tiles. In tensor-parallel decoding, FlashSampling replaces the all-gather of logits with streaming peer-to-peer writes: This overlaps GPU-to-GPU communication with computation and HBM loads across up to 8 GPUs, with near-ideal scaling at large batch sizes. Our kernel is exact because argmax decomposes over partitions; grouped variants for online and tensor-parallel settings are exact by hierarchical factorization of the categorical distribution. FlashSampling demonstrates kernel-level speedups on decode workloads across 4 different datacenter GPUs (H100, H200, B200, B300), and in end-to-end vLLM experiments, it reduces time per output token by up to $10%$ on the models we test. These results show that exact sampling, with no approximation, can be integrated into the matmul itself, consolidating the bandwidth-bound sampling step in an efficient epilogue.
Comments: Project Page: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2603.15854 [cs.LG]
  (or arXiv:2603.15854v4 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2603.15854

arXiv-issued DOI via DataCite

Submission history

From: Yifan Zhang [view email]
[v1] Mon, 16 Mar 2026 19:37:08 UTC (242 KB)
[v2] Tue, 12 May 2026 21:02:06 UTC (252 KB)
[v3] Fri, 25 Sep 2026 23:50:47 UTC (252 KB)
[v4] Thu, 12 Nov 2026 22:27:21 UTC (260 KB)

来源:arXiv cs.AI · arxiv.org