arXiv cs.AI· Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, Michael Galarnyk, Sudheer Chava, Chao Zhang·· 4 小时前AI 评分47
Forecast-Dojo：用于评测与训练 LLM 预测智能体的可回放环境
Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
AI 导读
研究者推出 Forecast-Dojo，一个可回放环境，将已结算的 Polymarket 事件重建为按日期排列的预测步骤序列，包含 1,568 个事件和 1880 万篇新闻文章，支持可复现评测与智能体训练。
正文 · 原文
Authors:Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, Michael Galarnyk, Sudheer Chava, Chao Zhang
Abstract:Forecasting requires large language model (LLM) agents to gather evolving evidence, revise their beliefs, and make calibrated predictions over time. Yet evaluating and improving these capabilities is challenging because real-world events unfold continuously and outcomes may take months to resolve. We introduce Forecast-Dojo, a replayable environment that reconstructs resolved prediction-market events as sequences of dated forecasting steps, enabling reproducible evaluation, longitudinal analysis, and training from agent interactions. It comprises 1,568 Polymarket events split temporally into training and evaluation sets, alongside 18.8 million dated news articles. Across 12 models, research tools consistently improve forecasting performance, and greater capture of new evidence is associated with larger improvements over time. Nevertheless, all models trail historical market forecasts, while carrying a belief notebook across dates reduces research costs without consistently improving forecast quality. We supervise fine-tuned Qwen3-30B-A3B-Thinking-2507 on collected trajectories from Qwen3-235B-A22B-Thinking-2507 to demonstrate the environment's training utility. The student's accuracy is improved from 34.8% to 42.8% on the evaluation set.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.28876 [cs.AI] |
| (or arXiv:2609.28876v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.28876 arXiv-issued DOI via DataCite |
Submission history
From: Liqin Ye [view email]
[v1]
Thu, 24 Sep 2026 00:52:53 UTC (339 KB)
[v2]
Mon, 26 Oct 2026 01:51:48 UTC (523 KB)
来源：arXiv cs.AI · arxiv.org