跳到正文
arXiv cs.AI· Aritra Das, Francesco Cagnetta, Darshil Doshi, Victor V. Albert, Maissam Barkeshli, Andrey Gromov·· 2 小时前AI 评分34

tree-grams:层级 n-gram 模型作为语言的可控近似器

What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

AI 导读

一项新研究提出 tree-grams,一种自回归生成式 n-gram 模型,通过 order-2 与 order-3 张量的层级组合构建条件概率张量,其中 n 控制最大上下文窗口,秩超参数 d 控制模型容量。

正文 · 原文

View PDF HTML (experimental)

Abstract:Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $\mu$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.
Comments: 7+13 figures, 10+30 pages
Subjects: Computation and Language (cs.CL); Disordered Systems and Neural Networks (cond-mat.dis-nn); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.23093 [cs.CL]
  (or arXiv:2611.23093v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.23093

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Aritra Das [view email]
[v1] Tue, 17 Nov 2026 17:14:01 UTC (1,613 KB)

来源:arXiv cs.AI · arxiv.org