GPT 5.4 能否进行有法律意义的推理？基于欧洲人权法院案例的小规模研究
Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
AI 导读
一项小规模研究以欧洲人权法院（ECtHR）案例为测试床，考察 OpenAI GPT 5.4 在法律案例预测中的推理表现，比较不同提示词策略，并结合人工与 LLM 评估。结果显示该模型法律推理远不理想，分析结构完整但实质浅薄；LLM-as-a-Judge 评估内部一致但与训练过的标注者仅弱对齐，不能替代人工评估。专家设计的提示词带来更全面的推理，却未提升预测准确率。
正文 · 原文
Abstract:Reasoning has become a standard technique and feature for contemporary LLMs; however, its application and quality in the context of demanding legal-oriented tasks, such as legal case forecasting, remain under explored. We investigate how LLMs reason in the context of legal case forecasting, using legal cases from the European Court of Human Rights (ECtHR) as a testbed. We evaluate OpenAI GPT 5.4, a recent top-tier LLM, by exploring alternative prompting strategies that are more or less suggestive of what counts as legally meaningful reasoning in the context of ECtHR jurisprudence. We present our findings derived from assessing the model's responses with both human and LLM evaluation. We find that the examined model scores far from ideal in legal reasoning, the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet align only weakly with our trained annotators, i.e., reliable but not a valid substitute for human evaluation. Overall, the expert-curated prompt leads to more comprehensive reasoning, which does not result in more accurate predictions compared to the other examined settings. Based on our findings, we urge the community not to rely solely on automated LLM-based evaluation and to avoid using task accuracy as an appropriate proxy for reasoning quality.
| Comments: | 24 pages, 4 figures, 4 tables, Submitted to AI4LAW Workshop at ICML 2026, github link attached |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.17168 [cs.CL] |
| (or arXiv:2608.17168v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.17168 arXiv-issued DOI via DataCite |
Submission history
From: Amogh Raina [view email]
[v1]
Mon, 17 Aug 2026 22:11:22 UTC (1,874 KB)
[v2]
Sat, 19 Sep 2026 17:07:12 UTC (1,874 KB)
[v3]
Thu, 5 Nov 2026 15:54:26 UTC (1,874 KB)
来源：arXiv cs.AI · arxiv.org