研究评估 GPT-4、Gemma、Llama 3.1、Mistral 能否通过 BFI-2-60 人格问卷扮演数字孪生
Large Language Models as Digital Twins: a Distribution-Aware Questionnaire-Based Evaluation
AI 导读
一项研究测试 GPT-4、Gemma、Llama 3.1 和 Mistral 能否根据用户社交媒体帖子，以该用户的身份作答 BFI-2-60 大五人格问卷，并在 MyPersonality 数据集上与真实得分对比。
正文 · 原文
Abstract:This paper investigates whether Large Language Models can answer a personality questionnaire as a specific individual would. Given a user's social-media posts, the model completes the BFI-2-60 Big Five questionnaire as that user. The predicted responses are converted into trait scores and compared with the user's actual scores. We evaluate GPT-4, Gemma, Llama 3.1, and Mistral on the MyPersonality dataset. To determine whether predictions reflect information from the user's writing, we compare them with answers produced without user text and with values derived from general population tendencies. We measure prediction error across all users and separately for individuals with low, intermediate, and high trait scores. We also test whether an LLM can express an explicit personality profile when given only the target Big Five scores, without user text, and examine how predictions change as more text is provided. The results show that user text does not consistently improve performance. Models perform better near the middle of the trait distribution, while at the lower and upper ends they often remain close to central values or perform well at one end but poorly at the other. When given an explicit profile, GPT-4 most clearly translates trait information into consistent questionnaire responses, particularly for Extraversion and Neuroticism. Increasing context does not systematically reduce error: open-source models fluctuate more, whereas GPT-4 remains comparatively stable. Overall, the evaluated models do not consistently reproduce the questionnaire behavior of specific individuals.
| Comments: | This work was partially funded by the European Union within the National Programme 'Research, Innovation and Competitiveness' 2021-2027, project SINTESI (Nr. MI_0000962). F. Di Cursi's, C. Boldrini's, A. Passarella's, and M. Conti's work was also partly funded by the PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - "FAIR" |
| Subjects: | Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.16586 [cs.AI] |
| (or arXiv:2611.16586v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16586 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Francesco Di Cursi [view email]
[v1]
Thu, 12 Nov 2026 10:18:16 UTC (761 KB)
来源：arXiv cs.AI · arxiv.org