低成本测量方法：跨厂商与版本追踪大语言模型行为
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases
AI 导读
一篇 arXiv 论文提出简单、廉价、可复现的大语言模型行为测量方法：用固定的公开刺激在跨厂商模型上运行，每个模型花费几美元以内，并通过精确匹配、LLM judge 代码簿或记录智能体实际行为的监测环境三种方式读取转录。
正文 · 原文
Abstract:Language models advise people, keep them company, and write software while they sleep. Measuring what they do is hard: behavior has to be sampled repeatedly across models, prompts and releases, most of it lives in unstructured text that has to be coded before it can be counted, and the result has to be legible and rigorous enough to meaningfully compare models and vendors. To address these constraints, we present a simple, cheap, scalable, and replicable model for studying model behavior. Each study is a frozen, public stimulus run identically on a cross-vendor panel, at a few dollars per model or less. Each reads its transcripts one of three ways, chosen by how much interpretation the behavior needs: exact match on a clamped reply, a codebook applied by LLM judges whose agreement with a human coder is reported per code, and an instrumented environment that records what an agent did independently of what it said. Run across four years of model releases from both frontier and open-source labs, these instruments find four things. Convergence: asked to pick a word, 27 of 44 models answer serendipity at least once in four tries. Resistance: a trailing "right?" moves endorsement by up to 32 points, and the sign flips from sycophantic to resistant as generations advance, keyed to the tag's surface form. House: whether a model holds a position under pressure tracks its generation, and how it holds tracks the lab that built it. Account: told to do something the documentation in their repository contradicts, some coding agents never went along silently and others always did, and the same model can change with the harness it runs in. Re-run on every release, batteries like these track how behavior is changing across vendors and over time.
| Comments: | Corrects the author list and title of the reference to Alnasser et al. (arXiv:2609.07662). No other changes |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.30012 [cs.CL] |
| (or arXiv:2609.30012v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.30012 arXiv-issued DOI via DataCite |
Submission history
From: Tapan Parikh [view email]
[v1]
Thu, 24 Sep 2026 15:51:17 UTC (8 KB)
[v2]
Wed, 11 Nov 2026 00:22:21 UTC (8 KB)
来源：arXiv cs.AI · arxiv.org