基于 LLM 的推荐系统中位置偏差的统计检测方法与实证研究
Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence
AI 导读
一项 arXiv 研究提出两种检测 LLM 推荐系统中位置偏差的方法：基于列联矩阵与卡方独立性检验的 POB-ICM，以及具备统计保证的 POB-UNBP。实验用基于真实商品目录的 RAG 测试平台、2000 条模板生成查询各 20 次排列共 40,000 个实例，发现所有受测 SOTA 模型都存在位置偏差，且该偏差是结构性的、依赖模型的，作者呼吁对已部署系统进行例行审计与治理控制，源代码已公开。
正文 · 原文
Abstract:Large Language Models (LLMs) are increasingly used as recommendation generators in conversational recommender systems (RS) via retrieval-augmented generation (RAG). This flexible design introduces new sources of structural bias. This paper studies positional bias in top-1 and single-turn LLM-based RS: systematic dependence of the recommended item on the order in which candidates are presented in the prompt. We propose two complementary detection methodologies: (i) POB-ICM, based on contingency matrix analysis and Chi-square independence testing, and (ii) POB-UNBP, a sample-efficient approach derived from a uniformity null model under permutation-based evaluation, enabling stochastic decoding with formal statistical guarantees. Using a controlled RAG testbed for product recommendation, built on a real product catalog, a fixed prompt template and 2000 template-generated user queries with 20 permutations each (40,000 instances), both methodologies reject the null hypothesis of positional independence and uniformity in all evaluated state-of-the-art models. We introduce three complementary metrics -- permutation sensitivity, positional bias, and strong positional bias -- to quantify this effect, the latter targeting cases where the position displaces the recommendation towards clearly less relevant items rather than only choosing among comparably relevant ones. A robustness analysis across candidate-set size and decoding temperature shows that the bias is structural and model-dependent, not an artifact of deterministic decoding or candidate-set size. These results establish positional bias as a measurable, reproducible property of prompt-mediated recommendation pipelines in the evaluated setting and motivate routine auditing and governance controls (e.g., order randomization, standardized prompting) for deployed LLM-based RS. Source code is publicly released.
| Comments: | 34 pages, 8 figures, 18 tables |
| Subjects: | Information Retrieval (cs.IR); Artificial Intelligence (cs.AI) |
| ACM classes: | H.3.3; I.2.7; G.3 |
| Cite as: | arXiv:2610.37962 [cs.IR] |
| (or arXiv:2610.37962v1 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37962 arXiv-issued DOI via DataCite |
Submission history
From: Carlos Peláez-González [view email]
[v1]
Tue, 27 Oct 2026 20:48:30 UTC (963 KB)
来源：arXiv cs.AI · arxiv.org