arXiv cs.AI· Zidi Xiu, David Q. Sun, Kevin Cheng, Maitrik Patel, Josh Date, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, Samy Bengio·· 4 小时前AI 评分25
ASTRA-bench：评估个人上下文中工具使用智能体的推理与行动规划
ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context
AI 导读
ASTRA-bench 是一个统一时间演化个人上下文、交互工具箱与复杂用户意图的工具使用智能体评测基准，通过事件驱动流水线生成覆盖四位主角的 2,413 个场景。对 Claude-4.5-Opus、DeepSeek-V3.2 等 SOTA 模型的评测显示，高复杂度条件下性能显著下降，参数生成是主要瓶颈。该基准连同完整执行环境和评测脚本已开源发布。
正文 · 原文
Authors:Zidi Xiu, David Q. Sun, Kevin Cheng, Maitrik Patel, Josh Date, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, Samy Bengio
Abstract:Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.01357 [cs.AI] |
| (or arXiv:2603.01357v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2603.01357 arXiv-issued DOI via DataCite |
Submission history
From: Zidi Xiu [view email]
[v1]
Mon, 2 Mar 2026 01:34:48 UTC (5,955 KB)
[v2]
Thu, 5 Nov 2026 06:20:21 UTC (5,955 KB)
来源：arXiv cs.AI · arxiv.org