研究发现假新闻检测器对大语言模型生成文本存在偏见
Fake News Detectors are Biased against Texts Generated by Large Language Models
AI 导读
一项研究评估了假新闻检测器在人类撰写与 LLM 生成虚假信息场景下的表现,发现许多现有检测器存在显著偏见:更容易将 LLM 生成内容判为假新闻,却常把人类撰写的假新闻误判为真。研究者提出用 LLM 改写的真实新闻进行对抗训练的缓解策略,提升了对两类新闻的检测准确率,并发布了 GossipCop++ 和 PolitiFact++ 两个数据集。
正文 · 原文
Abstract:The spread of fake news has emerged as a critical challenge, undermining trust and posing threats to society. In the era of Large Language Models (LLMs), the capability to generate believable fake content has intensified these concerns. In this study, we present a novel paradigm to evaluate fake news detectors in scenarios involving both human-written and LLM-generated misinformation. Intriguingly, our findings reveal a significant bias in many existing detectors: they are more prone to flagging LLM-generated content as fake news while often misclassifying human-written fake news as genuine. This unexpected bias appears to arise from distinct linguistic patterns inherent to LLM outputs. To address this, we introduce a mitigation strategy that leverages adversarial training with LLM-paraphrased genuine news. The resulting model yielded marked improvements in detection accuracy for both human and LLM-generated news. To further catalyze research in this domain, we release two comprehensive datasets, \texttt{GossipCop++} and \texttt{PolitiFact++}, thus amalgamating human-validated articles with LLM-generated fake and real news.
| Comments: | The first two authors contributed equally |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2309.08674 [cs.CL] |
| (or arXiv:2309.08674v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2309.08674 arXiv-issued DOI via DataCite |
Submission history
From: Terry Yue Zhuo [view email]
[v1]
Fri, 15 Sep 2023 18:04:40 UTC (9,845 KB)
[v2]
Fri, 13 Nov 2026 11:31:26 UTC (6,685 KB)
来源:arXiv cs.AI · arxiv.org