GPT-2 架构 Transformer 在有根树算术序列上的可学习性研究

View PDF HTML (experimental)

Abstract:We study whether a transformer network can learn the deterministic sequence of trees generated by the iterated prime factorization of the natural numbers. Each integer is mapped into a rooted planar tree and the resulting sequence $\mathbb{N}\mathscr{T}$ defines an arithmetic text with measurable statistical structure. A transformer network (the GPT-2 architecture) is trained from scratch on the first $10^{11}$ elements and evaluated on next-token prediction and masked-token prediction tasks, with a Hidden Markov Model as baseline and a scaling analysis over context window, dataset size, vocabulary size and model size. The model reaches a word accuracy of about $0.4$, well above the baseline, and its performance remains stable on test blocks located at $10^{13}$-$10^{15}$, far beyond the training interval. Moreover, the likelihood assigned by the trained model separates the arithmetic text from two controls: synthetic token sequences that preserve the marginal token frequencies but not their order, with a separation that widens with the context, and sequences containing more than three consecutive square-free integers, a configuration that arithmetic forbids. These results indicate that the transformer captures regularities of the arithmetic text that go beyond its frequency profile.

Comments:	42 pages, 18 figures
Subjects:	Artificial Intelligence (cs.AI); Disordered Systems and Neural Networks (cond-mat.dis-nn); Mathematical Physics (math-ph); Number Theory (math.NT)
Cite as:	arXiv:2512.01870 [cs.AI]
	(or arXiv:2512.01870v3 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2512.01870 arXiv-issued DOI via DataCite

Submission history

From: Gabriele Sicuro [view email]
[v1] Mon, 1 Dec 2025 16:51:38 UTC (423 KB)
[v2] Sun, 2 Aug 2026 14:50:25 UTC (847 KB)
[v3] Fri, 23 Oct 2026 13:41:47 UTC (1,346 KB)