人类能分辨吗？LLM 生成新闻感知度的双轴研究

View PDF HTML (experimental)

Abstract:Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.

Comments:	6 pages, 6 figures, 1 table. Author version of the paper published in WebSci Companion '26, pp. 34-39
Subjects:	Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC)
ACM classes:	H.5.2; H.1.2; K.4.2; I.2.7
Cite as:	arXiv:2604.03755 [cs.CY]
	(or arXiv:2604.03755v2 [cs.CY] for this version)
	https://doi.org/10.48550/arXiv.2604.03755 arXiv-issued DOI via DataCite
Journal reference:	Companion Publication of the 2026 18th ACM Web Science Conference (WebSci Companion '26), pp. 34-39
Related DOI:	https://doi.org/10.1145/3795513.3807431 DOI(s) linking to related resources

Submission history

From: Alexander Loth [view email]
[v1] Sat, 4 Apr 2026 15:03:42 UTC (1,197 KB)
[v2] Thu, 29 Oct 2026 22:00:44 UTC (1,178 KB)