SpecSearch:面向硬件感知的高效投机解码方法
SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding
AI 导读
一项 arXiv 论文提出 SpecSearch,一种硬件感知的搜索引擎,可跨工作负载与 GPU 组合联合优化解码策略、草稿长度和精度。该方法基于解析成本模型捕捉内存与算力约束,并用 LoRA 适配器微调冻结的 Qwen3-4B 训练 Jev 预测器,免除部署 GPU 的校准。
正文 · 原文
Abstract:Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX~5090, the state-of-the-art speculative drafter DFlash achieves $3.50\times$ the throughput of autoregressive (AR) decoding at low concurrency, but only $0.76\times$ at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload--GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX~5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of $2.01\times$ the throughput of AR and $1.36\times$ the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with $1.30\times$ throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains $97.36\%$ of the throughput achieved by SpecSearch that utilizes full calibration.
| Subjects: | Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2611.22296 [cs.AI] |
| (or arXiv:2611.22296v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22296 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Selin Yildirim [view email]
[v1]
Tue, 17 Nov 2026 06:53:30 UTC (110 KB)
来源:arXiv cs.AI · arxiv.org