arXiv cs.AI· Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy·· 1 小时前AI 评分36
SynthCorp：一个面向数据智能体基准测试的平台
SynthCorp: A Platform for Creating Benchmarks for Data Agents
AI 导读
SynthCorp 是一个通过模拟业务事件随时间生成和修改企业数据来创建可定制数据资产与基准测试的平台，用于解决现有 LLM 数据智能体基准快速饱和、难以针对特定能力的问题。其初始基准包含七个数据资产（最多 1K 运营表）和覆盖五个领域的 256 个任务，流水线检查错误率接近零。跨三种模型、四种数据上下文设置的评估显示基准能有效区分数据上下文与模型层级，基准即将发布。
正文 · 原文
Abstract:Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass deterministic pipeline checks (such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures) with a near-zero error rate, introduce cross-asset complexities, and cover all requested dimensions. Evaluating three models across four data-context settings shows that the benchmarks separate data contexts and model tiers. Auditing candidate queries against causal provenance in the simulation log verifies gold programs without manual annotation, including on multi-event composite tasks that expose reasoning failures in state-of-the-art agents. We hope the community uses SynthCorp to generate more challenging and useful data estates and benchmarks that foster the development of enterprise data agents.
| Comments: | SynthCorp v1.0. We will release the benchmarks soon |
| Subjects: | Artificial Intelligence (cs.AI); Databases (cs.DB) |
| Cite as: | arXiv:2611.14765 [cs.AI] |
| (or arXiv:2611.14765v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14765 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiani Zhang [view email]
[v1]
Tue, 10 Nov 2026 21:43:12 UTC (123 KB)
来源：arXiv cs.AI · arxiv.org