arXiv cs.AI· Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica Piskac·· 2 小时前AI 评分43
DeoGen：面向策略推理的自动基准生成方法
Deogen: Automatic Benchmark Generation for Reasoning over Policies
AI 导读
arXiv 论文提出 DeoGen，一个可扩展、可参数化的道义推理（deontic reasoning）自动基准生成器。它将策略规则自动形式化为一阶逻辑并构建决策树，树中每条路径生成一条基准条目：叶子路径对应判定，非叶子路径测试模型在信息不足时的弃权能力。评估显示前沿模型正确判定率为 89-92%，但在信息不足的条目上仅 22-40% 正确，即使提示词明确要求弃权。
正文 · 原文
Abstract:Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict. Benchmarking this capability, known as deontic reasoning, remains a costly and manual process, with existing datasets largely hand-crafted. In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. Each path down a tree becomes a benchmark entry, with each tree producing combinatorially many entries relative to tree depth. Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information. Evaluations show frontier models return the correct verdict 89-92% of the time. Notably, performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. In all, DeoGen presents an automatic, scalable, and parametric method for generating deontic reasoning benchmarks.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.38687 [cs.AI] |
| (or arXiv:2610.38687v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38687 arXiv-issued DOI via DataCite |
Submission history
From: Sam Kouteili [view email]
[v1]
Wed, 28 Oct 2026 18:56:59 UTC (1,173 KB)
来源：arXiv cs.AI · arxiv.org