研究评估 Claude Code 作为手写大学试卷 AI 评分系统的可靠性
Reliability of Claude Code as an Agentic AI Grading System for Handwritten University Exams
AI 导读
一项arXiv研究考察了基于 Claude Opus 4.6 的智能体部署 Claude Code 为手写大学试卷评分的可靠性。在沙特一门计算机图形学课程中，它对 60 名学生的 20 分期中卷进行评分，总分与专家评分高度一致（r = 0.97，MAE = 0.85 分），重测一致性 ICC = 0.96，反馈准确率达 92.8% 诊断性和 97.8% 确认性。
正文 · 原文
Abstract:This study examines the reliability of Claude Code, an agentic deployment of the Claude Opus 4.6 large language model, as a grading system for handwritten university exams. Four research questions addressed agreement with expert human grading on total scores (RQ1), agreement across five question types from multiple choice to evaluative judgement (RQ2), consistency across two independent grading sessions one week apart (RQ3), and the accuracy of AI-generated feedback (RQ4). Claude Code graded the 20-mark mid-term papers of 60 consenting students in a Saudi computer graphics course in a single agentic session, guided by three rubric files. Reliability was estimated on disjoint samples: 20 papers for RQ1, RQ2 and RQ4, and 15 papers for RQ3. Analyses used Pearson correlation, mean absolute error, Bland-Altman analysis, weighted Cohen kappa, intraclass correlation, and a four-category coding scheme separating diagnostic from confirmation feedback. The author, as the sole subject-matter expert for the course, provided the human benchmark, so the design assesses agreement with a domain-expert reference rather than inter-rater reliability. AI grading agreed strongly with human grading on totals (r = 0.97; MAE = 0.85 marks) with a small but significant AI-positive bias of 0.68 marks, showed excellent test-retest consistency (ICC = 0.96), and produced accurate feedback (92.8% diagnostic; 97.8% confirmation). Drawing items were the outlier on three of four measures. Three failure modes recurred: keyword matching without conceptual depth, selection-state ambiguity on corrected multiple-choice answers, and region-of-attention drift on hand-drawn diagrams. The findings suggest that AI grading at this capability level can assist instructors on objective and short-form items under structural safeguards but is not yet ready to replace instructor judgement at boundary cases or on drawing items.
| Subjects: | Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.13217 [cs.AI] |
| (or arXiv:2611.13217v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13217 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rayed Alghamdi [view email]
[v1]
Mon, 9 Nov 2026 20:39:02 UTC (3,911 KB)
来源：arXiv cs.AI · arxiv.org