MirrorCraft：在 Minecraft 隐藏规则变化下评估 LLM 智能体的配对基准
MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraft
AI 导读
MirrorCraft 是一个在 Minecraft 中评估 LLM 智能体应对隐藏规则变化的配对基准，每个 Mirror 世界与 Vanilla 世界地形、资源、任务和操作预算完全匹配，仅通过数据包静默修改部分服务器端规则。
正文 · 原文
Abstract:With the rapid progress of large language models (LLMs), an interesting question has emerged: how do LLM-based agents perform in Minecraft? To date, most Minecraft benchmarks evaluate them under fixed game mechanics, which are extensively documented online. High performance in these settings does not show whether an agent can continue making progress when familiar recipes, drops, and other rules change. In this paper, we introduce MirrorCraft, a paired benchmark for evaluating agents under hidden rule changes in Minecraft. Each Mirror world is a copy of its paired Vanilla world, with selected server-side rules silently modified by the datapack of one rule suite. Terrain, spawn, resource placement, task, interface, and action budget remain matched within every Vanilla-Mirror pair. MirrorCraft includes 10 Vanilla worlds in five controlled biomes, six rule suites, and three progression tasks, and we evaluate two LLMs and six agent configurations under a shared Mineflayer interface. We measure progress with server-verified advancement milestones and completion with success rate, and define the Rule Intervention Effect (RIE) as the performance change between paired Vanilla and Mirror worlds. Experiments with 8,640 episodes show that hidden rule changes have sharply different effects across suites. Among the configurations evaluated without rule descriptions, ReAct, Reflexion, and Voyager achieve the highest pooled Mirror Scores, but ReAct loses the most under compressed resource drops. Providing ReAct with the exact rules yields modest gains in average progress and completion across all three tasks, yet most of the difference between suites remains once the rules are known. MirrorCraft extends Minecraft evaluation beyond fixed mechanics and provides a controlled setting for studying how agents use gameplay outcomes when the rules of the current world differ from familiar ones.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.29218 [cs.AI] |
| (or arXiv:2607.29218v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.29218 arXiv-issued DOI via DataCite |
Submission history
From: Jianxin Gao [view email]
[v1]
Fri, 31 Jul 2026 09:43:54 UTC (9,777 KB)
[v2]
Fri, 23 Oct 2026 15:19:06 UTC (1,410 KB)
来源：arXiv cs.AI · arxiv.org