研究显示人类与大型推理模型对内容审核难题的难度反应相似
Humans and AI respond similarly to task difficulty when reasoning about content moderation dilemmas
AI 导读
一项对比 6 个大型推理模型与 1,854 名人类受试者的内容审核实验发现,模型在人类耗时更久的难题上也会投入更多推理。当两帖攻击性程度持平时,人类判断耗时增加 31%,模型的 CoT 长度增加 51%–306%;按攻击性语言类型分组后平均反应时间与推理努力相关性达 r = 0.85–0.93,而个体决策仅 r = 0.15–0.19。
正文 · 原文
Abstract:Large reasoning models (LRMs) generate intermediate reasoning steps before producing answers, improving performance on challenging problems by iteratively developing solutions. There is evidence that LRMs expend more effort on the problems that take people longer to solve, but it is unclear whether these relationships hold on more subjective tasks without verifiable answers. This study addresses this issue through an experiment using a content moderation task, where task difficulty is manipulated by varying the relative offensiveness of two different options. Comparing six large reasoning models with 1,854 human subjects, we find that models expend more reasoning effort, measured using the length of the chain-of-thought (CoT), on the kinds of cases that take people longer to adjudicate. People took 31\% longer and models produced 51 to 306\% longer CoTs when offensiveness was held constant across the two posts. The correlation is relatively low for individual decisions ($r = 0.15$--$0.19$), but average response times and average reasoning effort are closely correlated when tasks are grouped by the types of offensive language ($r = 0.85$--$0.93$). Qualitative analyses of the CoT summaries from Gemini 2.5 Pro corroborate these findings, showing that secondary factors were mentioned more often when offensiveness was fixed. The results point to the potential for using reasoning traces to prioritize difficult cases and enhance interpretability.
| Subjects: | Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2508.20262 [cs.AI] |
| (or arXiv:2508.20262v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2508.20262 arXiv-issued DOI via DataCite |
Submission history
From: Thomas Davidson [view email]
[v1]
Wed, 27 Aug 2025 20:36:16 UTC (574 KB)
[v2]
Sat, 20 Dec 2025 18:56:18 UTC (1,801 KB)
[v3]
Tue, 17 Nov 2026 00:25:38 UTC (5,970 KB)
来源:arXiv cs.AI · arxiv.org