arXiv cs.AI· Edesio Alcoba\c{c}a, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Concei\c{c}\~ao Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath·· 4 小时前精选AI 评分64
arXiv 论文提出在 1.4 亿规模生产部署前用仿真筛选客服 AI 智能体
Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
AI 导读
Nubank 团队发表 arXiv:2609.30137，提出在部署前用假设驱动的仿真工作流筛选客服 CX 智能体，合成客户与模拟工具输出支持多步智能体流程，无需调用生产后端。
推荐理由
论文给出生产环境可复用的仿真筛选工作流，用 Nubank 真实部署数据说明模拟评估与线上效果的关联程度。
正文 · 原文
Authors:Edesio Alcobaça, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Conceição Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath
Abstract:Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust.
We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation. Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test. We also screened open-weight configurations in over 16,000 simulated conversations. In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS. Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure, enabling production improvements that would have been impractical to pursue through live experimentation alone.
| Comments: | 17 pages, 11 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.30137 [cs.AI] |
| (or arXiv:2609.30137v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.30137 arXiv-issued DOI via DataCite |
Submission history
From: Aman Gupta [view email]
[v1]
Thu, 24 Sep 2026 17:07:38 UTC (2,728 KB)
[v2]
Thu, 5 Nov 2026 18:38:42 UTC (2,728 KB)
来源：arXiv cs.AI · arxiv.org