arXiv cs.AI· Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz·· 2 小时前AI 评分50
HorizonBench:面向偏好演化长周期个性化的基准测试
HorizonBench: Long-Horizon Personalization with Evolving Preferences
AI 导读
HorizonBench 及其数据生成器通过采样用户偏好随生活事件演变的序列,生成六个月对话来评测长周期个性化,包含 360 名用户的 4,245 道题,并配套发布 160 人 13 周共 83,024 条评分的人类纵向偏好数据集。
正文 · 原文
Authors:Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz
Abstract:Personal assistants now talk with the same user for months, and users' preferences change as their lives change; an assistant that keeps serving a preference the user has outgrown fails that user. Real conversations do not record when or why a preference changed, and existing benchmarks either hold preferences fixed or have the user state the change, so they cannot tell whether a model forgot what the user said or failed to update its belief about the user. We introduce HorizonBench and its data generator, which samples how a user's preferences change and generates six months of conversation from that sequence; each change follows a life event that the user mentions without stating the new preference. A human longitudinal preference dataset that we collect and release, 83,024 ratings from 160 people over 13 weeks, sets how often and how much life events change preferences. HorizonBench contains 4,245 questions from 360 users, and when a preference has changed, its earlier value is one of the answer options. All 25 frontier models we evaluate choose the outdated preference more often than evenly spread errors would predict, and the best reaches 52.8% accuracy. Two of the newest models score no higher when given only the relevant conversations than with the full history, yet one added sentence stating the change raises their accuracy by 11 to 22 points, and retrieval and memory methods do not recover the lost accuracy. Remembering what a user said and knowing what the user prefers now are therefore separate capabilities, and HorizonBench, its generator, and the human dataset support research on both.
| Comments: | 26 pages, 6 figures, 19 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.17283 [cs.CL] |
| (or arXiv:2604.17283v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.17283 arXiv-issued DOI via DataCite |
Submission history
From: Shuyue Stella Li [view email]
[v1]
Sun, 19 Apr 2026 06:55:10 UTC (804 KB)
[v2]
Tue, 17 Nov 2026 22:59:02 UTC (735 KB)
来源:arXiv cs.AI · arxiv.org