arXiv cs.AI· Vedant Jhaveri, Arashpreet Singh Mor, Satyam Kumar, Alex Tsun, Qing Lan, Shaunak Mishra, Sriram Vasudevan·· 56 分钟前AI 评分38
Trie-Head:基于 Trie 解码与共享上下文注意力的高效宽束生成式检索
Efficient Wide-Beam Generative Retrieval with Model-Native Trie Decoding and Shared-Context Attention
AI 导读
提出生成式检索方法 Trie-Head,将标识符 trie 存为 GPU 张量的模型原生输出接口,避免 Python 约束回调与全词表候选缓冲,无需重训练即可保持原始模型有效 token 分数;配合共享上下文注意力在多个 beam 间复用 prompt 的 key/value 块。
正文 · 原文
Abstract:Generative retrieval (GR) predicts catalog entries as short semantic identifiers (SIDs). Trie-constrained beam search keeps these identifiers valid, but wide beams expose two serving costs: vocabulary-wide score tensors that are mostly discarded, and repeated attention over the same prompt. We present Trie-Head, a model-native output interface that stores the identifier trie as validated GPU tensors. Compared with a Python custom logit processor (CLP), it reduces CPU--GPU coordination and memory traffic by avoiding Python constraint callbacks and vocabulary-wide candidate buffers. It scores compact codebook candidates and streams the full-vocabulary normalizer, preserving the original model's valid-token scores in real arithmetic without retraining. Shared-context attention complements this output path by reusing prompt key/value blocks across beams while preserving their private decode tails. We implement both mechanisms in SGLang and evaluate candidate sourcing for LinkedIn Hiring Assistant and qualification retrieval for the Qualification Retrieval Platform. The combined system improves closed-loop throughput by up to 5.57x over the CLP baseline and up to 2.63x over BF16 FlashRec at a common beam width of 200 on the Qualification Retrieval Platform workload.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22309 [cs.AI] |
| (or arXiv:2611.22309v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22309 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shaunak Mishra [view email]
[v1]
Tue, 17 Nov 2026 07:07:00 UTC (2,354 KB)
来源:arXiv cs.AI · arxiv.org