LGMT：一款测试大语言模型逻辑推理可靠性的开源工具
LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs
AI 导读
研究者发布开源工具 LGMT，通过基于一阶逻辑的蜕变关系检测大语言模型的推理不一致性，无需真值标签即可比对等价测试用例间的预测是否一致。工具内置 20 种蜕变关系，支持经 Z3 定理证明器验证的自定义关系，提供 CLI 与网页界面及无需 API key 的离线模式。在 3,091 组蜕变关系基准上，六种常用 LLM 的不一致率介于 20.84% 至 29.44%，代码已在 GitHub 公开。
正文 · 原文
Abstract:Large Language Models (LLMs) increasingly serve as reasoning components in modern software systems, yet a model that answers a logical query correctly can contradict itself when the same problem is restated in an equivalent form. Static benchmarks evaluate queries in isolation and therefore fail to expose such inconsistencies. To address this limitation, we present LGMT, an open-source tool that detects reasoning inconsistencies through metamorphic relations grounded in first-order logic. Given a source test case in premise-conclusion form, LGMT generates logically equivalent follow-up cases and checks whether the model's entailment predictions agree across the pair, without ground-truth labels. The tool provides 20 built-in metamorphic relations, supports custom ones verified by the Z3 theorem prover, and runs from a CLI or a web studio, including an offline mock mode requiring no API key. Across six widely used LLMs on a bundled benchmark of 3,091 metamorphic groups, LGMT reveals inconsistency rates ranging from 20.84% to 29.44%. LGMT is publicly available on GitHub at this https URL, with a demonstration video at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32627 [cs.AI] |
| (or arXiv:2610.32627v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32627 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zenghui Zhou [view email]
[v1]
Fri, 23 Oct 2026 13:41:43 UTC (766 KB)
来源：arXiv cs.AI · arxiv.org