TreeWY：面向 Gated DeltaNet 混合模型的投机验证方法

View PDF HTML (experimental)

Abstract:Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key-value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today's systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at essentially identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win.

Comments:	10 pages, 3 figures
Subjects:	Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Performance (cs.PF)
Cite as:	arXiv:2608.20961 [cs.AI]
	(or arXiv:2608.20961v2 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2608.20961 arXiv-issued DOI via DataCite

Submission history

From: Sneha Murthy Ghantasala [view email]
[v1] Fri, 21 Aug 2026 10:31:15 UTC (65 KB)
[v2] Thu, 22 Oct 2026 18:32:01 UTC (57 KB)