arXiv cs.AI· Zhengxiang Huang, Yujie Sun, Shengheng Chen, Zhaode Wang, Chaoyue Niu, Zeyu Zhao, Chengfei Lv, Fan Wu, Guihai Chen·· 3 小时前AI 评分43
EngineAgent：具备推理栈层级感知与硬件能力感知的端侧 LLM 推理引擎智能体化方案
EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities
AI 导读
EngineAgent 是一个自动化端侧 LLM 模型适配的智能体系统，其核心思路是将可复用的引擎与硬件知识从模型特定实现中分离，构建具备层级感知的引擎索引，并用 builder 和 auditor 两个角色协调跨层改动与设备端验证调试。
正文 · 原文
Abstract:Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine--backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.
| Subjects: | Operating Systems (cs.OS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.14356 [cs.OS] |
| (or arXiv:2611.14356v1 [cs.OS] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14356 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhengxiang Huang [view email]
[v1]
Tue, 10 Nov 2026 17:16:47 UTC (7,831 KB)
来源：arXiv cs.AI · arxiv.org