双智能体 LLM 流水线实现可扩展的对话式考试评估
Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline
AI 导读
研究者提出可扩展对话式评估系统，用 LLM 与学生进行交互式文字对话以考察论证能力。系统采用双智能体架构：面试官直接与学生互动，隐藏评估者依据评分细则追踪进度并指导追问，以降低答案泄露风险。系统先在模拟学生（含对抗型 persona）上测试，随后在一门小型 AI 素养选修课中用于形成性与总结性评估。
正文 · 原文
Abstract:Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral exams scale poorly. In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed.
| Comments: | Version submitted to SIGCSE TS 2027 (not accepted). A revised version is in preparation |
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI) |
| ACM classes: | K.3.2 |
| Cite as: | arXiv:2611.17220 [cs.CY] |
| (or arXiv:2611.17220v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17220 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Geoffrey Challen [view email]
[v1]
Thu, 12 Nov 2026 18:16:37 UTC (90 KB)
来源：arXiv cs.AI · arxiv.org