arXiv cs.AI· Anshu Raina, Peyman Razaghi, Yuankai Chen, Cheng Yao, Yao Fu, Vidushi Goyal, Devang Patel, Wen Xie, Zhenyu Gu, Emad Barsoum·· 3 小时前AI 评分54
Primus Projection 论文提出分布式 LLM 训练配置的投影与自动调优方法
Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training
AI 导读
论文提出 Primus Projection，用混合投影引擎与 Primus Tuning Agent 替代多节点试错来配置分布式 LLM 训练。
正文 · 原文
Abstract:Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, and overlap, a combinatorial space where memory legality can hinge on a single byte of activation accounting. Today, teams navigate it through expensive multi-node trial and error, burning GPU-hours on crashes and suboptimal recipes. We present Primus Projection, which closes this loop with two coupled components. First, a hybrid projection engine that measures what it can and simulates what it cannot: GPU microbenchmarks on as few as a single device capture compute kernels and any in-scope communication; a sub-node downscale-measure-upscale workflow analytically restores out-of-scope collectives, FSDP overlap, and the full pipeline schedule at the multi-node target. A pure simulation mode runs entirely on CPU, enabling pre-silicon planning without any GPU. Second, the Primus Tuning Agent treats the projection engine as a fast scoring oracle (seconds per candidate instead of tens of minutes) and uses it to drive an intelligent search over the configuration space. A deterministic seed planner sweeps single axes; an LLM investigation loop proposes cross-axis combinations the planner cannot reach, with an analytic memory pre-filter that rejects infeasible candidates before any GPU time is spent. Validated on Llama 3.1 (dense) and Mixtral 8x22B (MoE) across two GPU generations, all projections stay within ~10% of measured multi-node throughput, and fidelity holds at production scale: 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs. In a Mixtral 8x22B case study, the Primus Tuning Agent discovers a configuration delivering +27% throughput over the published 4-node BF16 reference at the same scale, in under 30 minutes of single-node exploration.
| Comments: | Accepted at NeurIPS 2026. 22 pages, 7 figures, 25 tables |
| Subjects: | Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2610.19423 [cs.AI] |
| (or arXiv:2610.19423v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19423 arXiv-issued DOI via DataCite |
Submission history
From: Anshu Raina [view email]
[v1]
Wed, 14 Oct 2026 02:15:28 UTC (357 KB)
[v2]
Thu, 12 Nov 2026 02:49:44 UTC (326 KB)
来源：arXiv cs.AI · arxiv.org