FLASHRT-PAI:面向物理 AI 的低延迟推理引擎论文发布
FlashRT-PAI: Fast Kernels Are Not Fast Pipelines for Physical AI
AI 导读
FLASHRT-PAI 论文提出以模型-硬件管线协同设计替代孤立 kernel 优化,解决 VLA、VA 与世界动作模型实时单批次推理延迟问题。系统通过深度 kernel 融合与启动重叠、执行契约和执行状态胶囊实现跨决策状态复用与快照恢复,并支持无改动 PyTorch 模型的 drop-in 加速。
正文 · 原文
Abstract:Vision-language-action (VLA), video-action (VA), and world action models (WAMs) form the backbone of modern physical AI systems, yet their real-time single-batch inference remains a defining bottleneck. Latency is dominated not by individual kernel computation, but by system-level overhead: thousands of short kernel launches, intermediate data movement across operator boundaries, and redundant recomputation across consecutive decisions. General-purpose graph compilers cannot address these costs, as they optimize individual calls in isolation and lack visibility into cross-call state semantics and global pipeline structure. We present FLASHRT-PAI, a low-latency inference engine built around model-hardware pipeline co-design rather than isolated kernel optimization. FLASHRT-PAI combines three core mechanisms: deep kernel fusion with launch overlap eliminates inter-operator boundaries and launch gaps; an execution contract formalizes the pipeline's stage DAG, update rates, and named buffers to enable cross-decision state reuse and selective refresh; and execution-state capsules enable fast snapshot, restore, and fork of persistent pipeline state. The system also supports drop-in acceleration of unmodified PyTorch models via parity- and latency-gated kernel replacement. Evaluation covers six policies on RTX 5090 and four on Jetson AGX Thor. At matched 16-bit precision, FLASHRT-PAI is 1.3-3.7x faster than official compiled paths. Combining pipeline co-design with reduced precision delivers 3.38-12.66x speedup over official eager implementations. In LIBERO-10, four-bit pi0.5 achieves 94.04% success on Thor and 93.32% on RTX 5090, compared with 92.4% for the measured official FP16 reference. FLASHRT-PAI is open source, distributed via Hugging Face Kernel Hub, and its opt-in pi0.5 kernels have been adopted into the LeRobot framework.
| Comments: | source code this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.24349 [cs.AI] |
| (or arXiv:2610.24349v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.24349 arXiv-issued DOI via DataCite |
Submission history
From: Jason Li [view email]
[v1]
Sat, 17 Oct 2026 00:35:58 UTC (71 KB)
[v2]
Thu, 12 Nov 2026 23:30:40 UTC (63 KB)
来源:arXiv cs.AI · arxiv.org