TrussTutor-Bench：用 200 道桁架题评测 LLM 的解题准确率、谄媚与错误检测
TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics
AI 导读
TrussTutor-Bench 是一个包含 200 道经核验二维桁架题及自动生成学生解答的基准，用于从解题准确率、谄媚度和错误检测三方面评测 13 个大语言模型。
正文 · 原文
Abstract:Large language models (LLMs) have rapidly become a default source of help for students in science and engineering courses. In foundational engineering mechanics courses such as statics, students routinely use these tools to check answers, work through partial solutions, and ask for explanations. Effective tutoring in this setting requires more than a correct final answer. A model must also confirm correct work when a student doubts it, and identify errors in flawed work. Existing benchmarks evaluate LLMs primarily on final-answer accuracy, so it is unclear whether models that can solve mechanics problems also have the other technical capabilities that tutoring depends on. Here we introduce TrussTutor-Bench, a benchmark of 200 verified two-dimensional truss problems paired with automatically generated student solutions, and use it to evaluate thirteen LLMs on solution accuracy, sycophancy, and error detection. Baseline accuracy ranged from 64.5% to 97.0%. Every model detected nearly all injected errors in synthetic flawed student work (90.4% to 100%), so error detection did not distinguish models under these clean conditions. Sycophancy did. When a student insisted that correct work was wrong, models rejected correct solutions to problems they had themselves solved in 1.6% to 50.0% of cases. These results show that performance varies widely across models: the leading models solve, check, and defend student work nearly perfectly, while others are far less reliable, particularly when a student doubts correct work. TrussTutor-Bench provides a verified, extensible standard for determining which models are technically reliable enough to put in front of students. Follow-up work must now focus on teaching rather than technical capability, drawing on learning science to determine whether LLM help leads students to understand the material or simply to reach the right answer.
| Comments: | 25 pages, 7 figures. Code: this https URL |
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.38917 [cs.CY] |
| (or arXiv:2610.38917v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38917 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Evangelos Apostolakis [view email]
[v1]
Wed, 28 Oct 2026 21:07:58 UTC (2,263 KB)
来源：arXiv cs.AI · arxiv.org