arXiv cs.AI· Darek Kleczek, Fuheng Zhao, Alexander W. Lee, Julien Tissier, Pawel Liskowski, Ugur Cetintemel, Anupam Datta·· 4 小时前AI 评分47
AvalancheBench：通过潜在世界恢复评估企业级智能体语义查询处理
AvalancheBench: Evaluating Enterprise Agent-Driven Semantic Query Processing Through Latent World Recovery
AI 导读
AvalancheBench 是一个多模态企业分析基准，从潜在世界规格生成观测数据，评估数据智能体端到端恢复该世界的能力，含 3 个验证任务、58 个跨文本、图像与文档的问题，并使用 Snowflake Cortex AISQL 作为引擎。
正文 · 原文
Abstract:As LLMs improve, data agents are emerging as a new interface for analytics, while semantic query processing engines use LLMs to operate over structured and unstructured data. Existing benchmarks largely evaluate agents and semantic query engines in isolation, leaving it unclear whether agents can decide when and how to use semantic operators in end-to-end analysis.
We introduce AvalancheBench, a multimodal enterprise analytics benchmark that generates observed data from latent world specifications and evaluates data agents on their ability to recover that world. The benchmark's 3 validated tasks contain 58 questions spanning text, images, and documents. Questions are provided together as a multi-part analytical brief and may share intermediate results, exposing the avalanche effect: errors can propagate through legitimate dependencies, and inappropriate reuse can cause otherwise unrelated questions to fail. AvalancheBench measures analytical fidelity, monetary cost, and capability-adjusted token cost (CATC), a new metric that weights token use by model capability to separate technical efficiency from provider pricing.
We evaluate 5 model-harness configurations across 3 agent harnesses using Snowflake Cortex AISQL as the engine, with the best configuration reaching macro analytical fidelity of 0.82. Agents fail primarily in constructing the analysis: failing to recover facts that require semantic computation costs 45.1% of lost points, yet incorrect operator outputs account for only 2.5%, versus 36.3% from agents misusing or avoiding semantic operators. Misinterpreting requirements costs the most, at 51.5%. Errors also spread between questions: 29% of lost points come from reusing another question's results. Finally, monetary cost and CATC produce different Pareto frontiers, distinguishing economically attractive configurations from those that are technically resource efficient.
| Subjects: | Databases (cs.DB); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.24183 [cs.DB] |
| (or arXiv:2605.24183v2 [cs.DB] for this version) | |
| https://doi.org/10.48550/arXiv.2605.24183 arXiv-issued DOI via DataCite |
Submission history
From: Dariusz Kleczek [view email]
[v1]
Fri, 22 May 2026 20:16:41 UTC (110 KB)
[v2]
Wed, 4 Nov 2026 18:20:29 UTC (9,533 KB)
来源：arXiv cs.AI · arxiv.org