Where Position Matters：基于剖析的 Transformer 逐层 RoPE 分配方法
Where Position Matters: Profiling-Guided Layer-Wise RoPE Allocation in Transformers
AI 导读
一篇 arXiv 论文提出基于剖析引导的 Transformer 逐层 RoPE 分配方法，在固定预算下为各层分配高低覆盖率，取代传统的统一覆盖。该框架支持 Multi-head Latent Attention（MLA）中的固定宽度 query/key 表示与物理旋转维度分配，并通过归一化深度映射将小模型的剖析结果迁移到更大的模型。
正文 · 原文
Abstract:Rotary Position Embedding (RoPE) typically uses uniform coverage across Transformer layers despite heterogeneous positional requirements. Phase-off interventions reveal non-uniform, model-specific rotary-phase dependence, while coverage interventions relate it to coverage-upgrade benefits. We propose profiling-guided layer-wise RoPE allocation, assigning high/low coverage under a fixed budget using early coverage-benefit profiles. The framework supports fixed-width query/key representations and physical rotary-dimension allocation in Multi-head Latent Attention (MLA). Normalized-depth mapping transfers profiles within model families, enabling small-model measurements to guide larger-model allocations before pretraining. A brief analysis relates selection and transfer to score calibration and inter-layer interactions. From-scratch experiments demonstrate validation-loss improvements over available budget-matched baselines, while analytical resource accounting identifies favorable quality--storage--compute trade-offs for physical MLA. These results connect positional demand to transferable, efficient architecture design.
| Comments: | 28 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.39223 [cs.LG] |
| (or arXiv:2610.39223v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39223 arXiv-issued DOI via DataCite |
Submission history
From: Boao Kong [view email]
[v1]
Thu, 29 Oct 2026 02:42:58 UTC (94 KB)
来源：arXiv cs.AI · arxiv.org