arXiv 论文：Agent 脚手架会改变固定 LLM 对银行行为规则的合规表现
The Harness Is the Model: Agent Scaffolding Changes a Fixed LLM's Compliance with Banking Conduct Rules, Unevenly Across Models
AI 导读
一篇 arXiv 预注册论文测试 4 个开源权重模型各运行 5 种 Agent 脚手架，在含 11 类规则、156 个案例的模拟零售银行中共评 3120 个回合。
正文 · 原文
Abstract:Banks validate language models, but they deploy systems: a model inside a harness that decides what it sees, which tools it can call and who checks its actions. We test whether that harness changes compliance with banking conduct rules when the model is held fixed. Four open-weight models each ran five harnesses (a pipeline that makes one structured decision, a ReAct-style tool loop, a loop in which the policy is reachable only through search, a loop with a long benign customer memory, and a loop with a same-model compliance reviewer) on 156 pre-registered cases in a simulated retail bank whose 11 rule families paraphrase EU consumer-credit, payments, investor-protection, data-protection and anti-money-laundering provisions. Every one of the 3120 two-turn episodes was scored deterministically from the executed tool trace, with violations and over-refusals counted separately. Pooled across models the harness moved the correct-case rate only between 87.3% and 92.1%, but within a model it mattered a great deal: for one model the spread between its best and worst harness was 11.5 percentage points, enough to move it from first to last place among the four models, while for another the harness made no detectable difference; the model x harness interaction itself was suggestive rather than conclusive. The same-model reviewer cut violations from 9.6% to 6.4% without raising over-refusals, contrary to our pre-registered prediction. Rules that require date arithmetic failed in every harness. In an exploratory analysis with two LLM judges and a blind human check, 26% of violations were accompanied by a customer message that denied the action the agent had just executed, which a transcript-only review would record as a correct refusal. For model-risk management the unit of validation should be the model-harness pair, and conduct testing should read the action log, not the conversation.
| Comments: | 16 pages, 4 tables. Pre-registered. Code, data and all 3120 episode traces: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.06116 [cs.CL] |
| (or arXiv:2611.06116v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06116 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mehmet Murat Huyut PhD [view email]
[v1]
Wed, 4 Nov 2026 07:59:59 UTC (21 KB)
来源：arXiv cs.AI · arxiv.org