LexTemporis 用韩国上市公司 SME 状态记录评估可执行法律规则
Grading Executable Legal Rules Against Independent Records: Transitional Provisions and Korean SME Status
AI 导读
arXiv 论文提出 LexTemporis,用韩国上市公司披露的 10,901 个公司年度 SME 状态记录及规则各版本,评估 LLM 撰写的可执行法律程序,而非只对照阅法者的答案。一项过渡性条款使与披露的吻合率从 83-86% 升至 89-90%;四个 LLM 的预注册对比未发现任何条件在全部模型上更优,而按阅法者评级的优势在按记录评级时对三个模型缩小,表明对照评阅者自身解读可能高估程序效果。
正文 · 原文
Abstract:Laws are increasingly turned into programs that decide cases, such as whether a firm was a small or medium enterprise (SME) on a given date, and large language models (LLMs) now write them. The programs are usually graded against answers written by someone who read the same law, so a program can score well by sharing the grader's reading rather than by getting the law right. LexTemporis grades them instead against records of what actually happened: 10,901 firm-years of SME status disclosed by Korean listed companies, with every version of the governing rules. An appendix adds comparison cases from US income tax and German hospital minimum volumes. The date changes outcomes mainly through provisions outside the main statutory text: one transitional provision raises agreement with the disclosures from 83-86% to 89-90%. The disclosures also test the author's reading of a contested provision, favouring it in 130 of 189 firm-years where readings diverge. Four LLMs wrote the rules, given different selections of the law. A pre-registered comparison finds no condition better across all four models. In exploratory analyses, supplying the right provisions looks better graded against the author's reading than against the records: the advantage shrinks for three models (for one, to between a quarter and three-fifths of its size), and per firm-year it is several times larger where the author's reading is wrong, partly by construction, since there reproducing that reading more faithfully must lose. Grading legal programs against the grader's own reading can overstate how well they work.
| Comments: | 51 pages, 2 figures (plus 1 in the appendix). Submitted to Artificial Intelligence and Law. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24055 [cs.CL] |
| (or arXiv:2611.24055v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24055 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinhyung Bae [view email]
[v1]
Wed, 18 Nov 2026 08:46:52 UTC (367 KB)
来源:arXiv cs.AI · arxiv.org