HPC 用户支持研究：基于文档与历史工单的 Agentic LLM 助手（RAG + Llama-3-8B QLoRA 微调）
Before the Ticket: Grounding HPC User Support in Documentation and Resolved Tickets with an Agentic LLM Assistant
AI 导读
研究者提出一个面向 HPC 帮助台的 Agentic LLM 助手，通过 MCP 协议调用两个工具：对实时文档的 RAG 检索，以及用从历史工单挖掘的 11,244 组问答对经 QLoRA 微调的 Llama-3-8B，并合并两者答案。
正文 · 原文
Abstract:Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search. We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers. Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution. On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average. The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers. Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.31703 [cs.SE] |
| (or arXiv:2610.31703v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31703 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Elham J. Barezi Ms [view email]
[v1]
Wed, 21 Oct 2026 19:01:41 UTC (383 KB)
来源：arXiv cs.AI · arxiv.org