arXiv cs.AI· Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, Takashi Ishida·· 4 小时前AI 评分39
CoffeeBench:在异构多智能体经济中评测长程 LLM 智能体
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
AI 导读
CoffeeBench 是一个基于咖啡供应链仿真的基准,用 6 个自主 LLM 编码智能体分别扮演农户、烘焙商和零售商,在 90 个模拟天内谈判交易以最大化净收入,已入选 NeurIPS 2026(Evaluations and Datasets Track)。
正文 · 原文
Abstract:LLM agents are increasingly capable of handling complex, long-horizon tasks and may eventually take on business management, a role traditionally performed by humans. Deploying them safely requires monitoring their capabilities and behavior in environments that reflect real economies. We introduce CoffeeBench, a benchmark built on a compact simulation of a coffee supply chain with realistic business mechanics. Six autonomous LLM coding agents run its farmers, roasters, and retailers, and negotiate and trade with one another to maximize their own net income over 90 simulated days. We evaluate eight recent proprietary and open-weight models by letting each run one roaster while a fixed background model runs the other five firms, and scoring that roaster's net income. Claude Sonnet 5.5 earns the highest net income, closely followed by GPT-6 Astra. The leaders sell at higher prices, and the best model also buys its inputs far more cheaply, whereas weaker models trade at thin margins. Since performance targets are a common cause of accounting fraud in human firms, we also test whether agents under KPI pressure inflate their numbers. Under a revenue target, circular trades occur for some pairs of models: one agent proposes that the two rival firms sell goods back and forth at inflated prices so that both book revenue, and the rival accepts. We hope CoffeeBench serves as an early step toward a future in which LLM agents manage firms without misconduct. We release our code and agent trajectories.
| Comments: | Accepted to NeurIPS 2026 (Evaluations and Datasets Track) |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.16613 [cs.AI] |
| (or arXiv:2606.16613v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.16613 arXiv-issued DOI via DataCite |
Submission history
From: Issa Sugiura [view email]
[v1]
Mon, 15 Jun 2026 12:04:44 UTC (1,311 KB)
[v2]
Fri, 13 Nov 2026 13:22:21 UTC (448 KB)
来源:arXiv cs.AI · arxiv.org