arXiv cs.AI· Wei Chen, Peilun Zhou, Zhaoyu Hu, Jiajun Chai, Derong Xu, Zhongni Hou, Yufei Zhang, Guojun Yin, Wei Lin, Zhi Zheng, Tong Xu·· 5 小时前AI 评分36
ATLAS：美团小团工具使用智能体的双视野诊断评估框架
ATLAS: Dual-Horizon Diagnostic Evaluation for Tool-Use Agents in Meituan Xiaotua
AI 导读
美团提出 ATLAS 框架，对百万级 DAU 本地生活助手“小团”的工具使用智能体进行轮内与跨轮双视野诊断评估，通过经业务日志校准的 LLM 判官并蒸馏为紧凑诊断模型。14 天在线 A/B 测试（每组近 200 万用户）显示，ATLAS 指导的智能体使答案读完率提升 2.84%、付费订单量提升 6.13%，已全量部署上线。
正文 · 原文
Authors:Wei Chen, Peilun Zhou, Zhaoyu Hu, Jiajun Chai, Derong Xu, Zhongni Hou, Yufei Zhang, Guojun Yin, Wei Lin, Zhi Zheng, Tong Xu
Abstract:LLM-based assistants on web-scale service platforms increasingly address local-service needs through multi-step planning and tool use. Under dynamic business conditions and evolving user needs, outcome-level evaluation offers limited evidence of where execution fails or how service drifts across continued interaction. We present ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. Within-turn evaluation uses trajectory-wise signals across thinking and reflection, tool and skill execution, and response generation to identify execution failures and unmet capability requirements. Across-turn evaluation uses user-wise signals spanning multiple lifecycles to diagnose failures in maintaining or updating user intent, constraints, and corrections. Fine-grained signals use rule-based checks and LLM judgments with explicit evidence scopes and decision boundaries. To support reliable diagnosis at production scale, LLM judges are calibrated against dynamically maintained references from business logs, and selected judges are distilled into compact diagnostic models. The resulting diagnostic profiles help developers attribute failures, prioritize targeted interventions, and reassess improvements. Applied in Meituan Xiaotuan, a million-scale DAU local-services assistant, ATLAS achieves high diagnostic fidelity across calibrated judges and distilled models. Blinded human auditing and a 14-day online A/B test with nearly two million users per arm confirm that the ATLAS-guided agent improves recommendation relevance, factual reliability, user engagement (+2.84% answer read-through rate), and order volume (+6.13% paid orders), leading to full production deployment.
| Comments: | 19 pages; The project's homepage is this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.30685 [cs.AI] |
| (or arXiv:2608.30685v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.30685 arXiv-issued DOI via DataCite |
Submission history
From: Wei Chen [view email]
[v1]
Mon, 31 Aug 2026 12:25:24 UTC (7,470 KB)
[v2]
Thu, 29 Oct 2026 08:57:19 UTC (1,961 KB)
来源：arXiv cs.AI · arxiv.org