arXiv cs.AI· Jingyan Jiang, Akhilesh Bondapalli, Krishna Rauniyar, Murat Keceli, Yasaman Ghadar, Haritha Siddabathuni Som, Venkatram Vishwanath, Michael E. Papka, Rong Ge, Huihuo Zheng·· 3 小时前AI 评分41
AskALCF：面向 HPC 智能用户支持的自改进智能体系统
AskALCF: A Self-Improving Agentic System for Intelligent HPC User Support
AI 导读
ALCF 团队提出 AskALCF，一个用于 HPC 用户支持的智能体式日志分析与改进流水线，可细粒度评估生产环境问答质量，并将失败归因到知识缺失、检索、提示词、模型行为或工具执行，再路由到可执行的修复。
正文 · 原文
Abstract:AI-based question-answering systems are increasingly used for user support, including at high-performance computing (HPC) facilities. However, existing systems lack a systematic, deployment-oriented approach for evaluating final-answer quality and attributing failures to specific components. They also remain largely static after deployment, apart from periodic knowledge-base updates.
We present AskALCF, an agentic log-analysis and improvement pipeline for HPC user support. AskALCF evaluates production interactions at a fine-grained level, identifies whether failures arise from missing knowledge, retrieval, prompting, model behavior, or tool execution, and routes each failure to an actionable repair. Its diagnostic core combines answer-quality assessment with a two-scope sufficiency test that distinguishes information missing from the documentation corpus from information present but not retrieved. We further introduce a controlled harness-training process that uses expert references, null edits, and explicit ship, revert, or decline decisions to improve the diagnostic pipeline reliably.
Across 2,174 support turns, AskALCF reduced unattributable failures from 41.4% to 14.2%, while harness training improved expert agreement from 108/150 to 123/150 turns. These results demonstrate a governed form of self-evolution in which production evidence drives targeted system improvements. By turning production interactions into fine-grained diagnoses and actionable repairs, AskALCF provides a systematic foundation for continuously improving deployed AI support systems beyond knowledge-base updates alone.
| Comments: | 12 pages, 5 figures. Accepted at HUST 2026 (13th International Workshop on HPC User Support Tools), co-located with SC26, Chicago, IL, USA, November 15, 2026. Camera-ready version |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7, H.3.3, C.2.4 |
| Cite as: | arXiv:2611.15700 [cs.SE] |
| (or arXiv:2611.15700v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15700 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jingyan Jiang [view email]
[v1]
Wed, 11 Nov 2026 16:38:54 UTC (648 KB)
来源：arXiv cs.AI · arxiv.org