PINK 指标识别多行手写数学 OCR 中 VLM 的"过度纠正"问题
When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCR
AI 导读
一项 EMNLP 2026 口头报告研究首次系统评估多行手写数学 OCR,发现视觉语言模型存在"过度纠正"问题——把学生错误"改对",掩盖教育评估要检测的错误。
正文 · 原文
Abstract:Accurate transcription of handwritten mathematics is crucial for educational AI systems, yet current benchmarks fail to evaluate this capability properly. Most prior studies focus on single-line expressions and rely on lexical metrics such as BLEU, which fail to assess the semantic reasoning across multi-line student solutions. In this paper, we present the first systematic study of multi-line handwritten math Optical Character Recognition (OCR), revealing a critical failure mode of Vision-Language Models (VLMs): over-correction. Instead of faithfully transcribing a student's work, these models often "fix" errors, thereby hiding the very mistakes an educational assessment aims to detect. To address this, we propose PINK (Penalized INK-based score), a semantic evaluation metric that leverages a Large Language Model (LLM) for rubric-based grading and explicitly penalizes over-correction. Our comprehensive evaluation of 15 state-of-the-art VLMs on the FERMAT dataset reveals substantial ranking reversals compared to BLEU: models like GPT-4o are heavily penalized for aggressive over-correction, whereas Gemini 2.5 Flash emerges as the most faithful transcriber. Furthermore, human expert studies show that PINK aligns significantly better with human judgment (55.0% preference over BLEU's 39.5%), providing a more reliable evaluation framework for handwritten math OCR in educational settings. Code is available at this https URL.
| Comments: | Accepted to EMNLP 2026 Main Conference (Oral). 22 pages. Code: this https URL |
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.22774 [cs.CY] |
| (or arXiv:2604.22774v3 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2604.22774 arXiv-issued DOI via DataCite |
|
| Journal reference: | EMNLP 2026, pages 10932-10953 |
| Related DOI: | https://doi.org/10.18653/2026.emnlp-main.476
DOI(s) linking to related resources |
Submission history
From: Jin Seong [view email]
[v1]
Wed, 1 Apr 2026 05:27:52 UTC (2,486 KB)
[v2]
Tue, 26 May 2026 06:37:15 UTC (2,494 KB)
[v3]
Tue, 17 Nov 2026 06:11:15 UTC (2,115 KB)
来源:arXiv cs.AI · arxiv.org