LLM4SCREENLIT：评估大语言模型在系统综述文献筛选中表现的建议
LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews
AI 导读
研究者提出 LLM4SCREENLIT 建议，指导如何评估用于系统综述文献筛选的大语言模型。基于 29 篇论文的回顾发现，仅 10% 报告 MCC、仅 24% 报告完整混淆矩阵，且没有论文对假阴性成本定价。
正文 · 原文
Abstract:Context: Large language models (LLMs) are increasingly used to screen literature for systematic reviews (SRs), but the standard confusion-matrix metrics used to evaluate them can mislead under the imbalanced, cost-asymmetric conditions of screening.
Objective: We develop and justify LLM4SCREENLIT-practical recommendations for researchers conducting LLM-screening evaluations and for editors and reviewers assessing such studies-differentiated by study type (retrospective benchmarking vs deployment for a specific SR).
Method: Using Delgado-Chaves et al. (2025), an 18-LLM benchmark across three biomedical SRs, as a motivating example, we reviewed 28 additional papers and extracted their reported metrics. We propose a Weighted Matthews Correlation Coefficient (WMCC) that integrates MCC's chance-correction with asymmetric misclassification costs, and validated it on three software-engineering (SE) reanalyses, the largest covering 9 LLMs x 24 SE secondary studies (34,528 articles).
Results: Across the 29 papers, only 10% reported MCC, only 24% reported full confusion matrices, and none of the five papers claiming workload savings priced false-negative cost. In the largest SE reanalysis, MCC and WMCC disagree on the best LLM in 55% of evaluable studies; in the most striking 9,695-article SE study, the Accuracy-best LLM loses 63.3% of relevant evidence (Lost Evidence), the MCC-best 43.9%, but the WMCC-best only 5.8%. Sensitivity analysis (median crossover at w~=2.7, all <7) supports w=10 as a conservative default.
Conclusions: SR-screening evaluations should prioritize Lost Evidence and use cost-sensitive WMCC alongside MCC for ranking. Reporting must include the full confusion matrix and treat unclassifiable outputs as positives requiring human review. Designs should be leakage-aware, with non-LLM baselines when the study aims to inform SR practice and labels are available.
| Comments: | Please cite the published version: Lech Madeyski, Barbara Kitchenham, Martin Shepperd, Information and Software Technology 198 (2026) 108204, this https URL | v3 replaces the preprint with the version of record (open access, CC BY 4.0). 20 pages, 6 figures |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| ACM classes: | I.2; D.2 |
| Cite as: | arXiv:2511.12635 [cs.SE] |
| (or arXiv:2511.12635v3 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2511.12635 arXiv-issued DOI via DataCite |
|
| Journal reference: | Information and Software Technology 198 (2026) 108204 |
| Related DOI: | https://doi.org/10.1016/j.infsof.2026.108204
DOI(s) linking to related resources |
Submission history
From: Lech Madeyski [view email]
[v1]
Sun, 16 Nov 2025 15:04:50 UTC (305 KB)
[v2]
Sat, 25 Apr 2026 21:51:20 UTC (325 KB)
[v3]
Wed, 11 Nov 2026 16:35:31 UTC (2,618 KB)
来源：arXiv cs.AI · arxiv.org