arXiv cs.AI· Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis·· 2 小时前AI 评分51
SlackBench 论文发布：基于真实代码仓库评测 Agent 的协作工作能力
SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories
AI 导读
研究者发布 SlackBench 基准，评测 Agent 在需要联合推理职场沟通与代码仓库状态的真实研究与工程工作流中的表现。基准基于真实学术项目的隐私保护重现，提供含 GitHub 仓库与 Google Docs 的可复现环境，包含 140 个查询，覆盖多源智能体搜索、对不可答或错误前提问题的弃答、目标导向摘要与长对话情境推断。
正文 · 原文
Abstract:AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions. Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources.
We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state. Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs. It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations. These tasks are not reducible to static corpus question answering: agents must determine which conversations matter, traverse threads, inspect repository state, and synthesize evidence across communication and code artifacts. Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%, highlighting substantial headroom for improvement. SlackBench thus provides an extensible framework for evaluating agents in collaborative work environments. The SlackBench dataset is available at this https URL .
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05512 [cs.AI] |
| (or arXiv:2611.05512v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05512 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mingxuan Li [view email]
[v1]
Tue, 3 Nov 2026 19:10:50 UTC (8,048 KB)
来源：arXiv cs.AI · arxiv.org