量化语言模型心理测量中的提示词伪影
The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics
AI 导读
一项研究提出诊断框架，量化语言模型心理测量中的提示词伪影，用 1-Wasserstein 距离区分目标效应与伪影。该框架应用于 13 个 0.6B 到 14B 的开源权重小模型，在 Big Five Inventory 和 Short Dark Triad 上测试五类提示组件。结果发现多数模型中任务指令和选项符号造成的分布偏移大于人格描述或条目改述，相当比例条目的伪影解释变异超过 50%。
正文 · 原文
Abstract:When prompting language models for psychometric assessment, researchers assume that the responses reflect the injected persona and the meaning of the survey item. We test this premise using a diagnostic design that crosses five semantically distinct baseline personas with five semantically equivalent variants of each of four prompt components (persona wording, task instruction, item wording, option symbol). Measuring the 1-Wasserstein distance between the resulting response distributions and partitioning the variation among the five components allows for the separation of target effects from prompt artifacts. We apply the framework to 13 open-weight small language models (0.6B to 14B) on the Big Five Inventory and the Short Dark Triad. We find that in most models, the task instruction and option symbol displace response distributions further than paraphrasing the persona description or the item itself. For a substantial share of items, the artifact share of explained variation exceeds 50%; non-semantic changes of the prompt account for more response variation than the baseline personas. Our framework lets researchers quantify these prompt artifacts before interpreting psychometric output.
| Comments: | 12 pages, 5 figures, 3 tables. Accepted to the Findings of EMNLP 2026, Budapest, Hungary |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.03357 [cs.CL] |
| (or arXiv:2606.03357v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.03357 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: EMNLP 2026 |
| Related DOI: | https://doi.org/10.18653/2026.findings-emnlp.1009
DOI(s) linking to related resources |
Submission history
From: Nils Schwager [view email]
[v1]
Tue, 2 Jun 2026 09:05:25 UTC (184 KB)
[v2]
Mon, 31 Aug 2026 13:56:58 UTC (166 KB)
[v3]
Thu, 5 Nov 2026 09:54:09 UTC (166 KB)
来源：arXiv cs.AI · arxiv.org