跳到正文
arXiv cs.AI· Kehang Li, Xinlan Xu, Yulin Jin, Heng Cao, Xuchen Li, Houxian Ren, Zihao Ji, Qiang Luo, Yadong Mao, Qilin He, Jian Zhang, Jian Zhao, Zihao Wang·· 4 小时前AI 评分41

OCTAVE:面向端到端 HTML5 游戏生成的能力分解基准

OCTAVE: A Capability-Resolved Benchmark for End-to-End HTML5 Game Generation

AI 导读

OCTAVE 是一个面向完整 HTML5 游戏生成的基准,围绕 8 个能力维度组织,包含 180 个任务和 1,800 条评估需求,覆盖 PC Web、Mobile Web、2D 和 3D。

正文 · 原文

Authors:Kehang Li, Xinlan Xu, Yulin Jin, Heng Cao, Xuchen Li, Houxian Ren, Zihao Ji, Qiang Luo, Yadong Mao, Qilin He, Jian Zhang, Jian Zhao, Zihao Wang

View PDF HTML (experimental)

Abstract:Complete game generation requires agents to coordinate rules, interactions, states, and presentation over a long development process. A single aggregate score obscures which capabilities are implemented and which requirements remain unmet. We present OCTAVE, a benchmark for complete HTML5 game generation organized around eight Capability Dimensions, with 180 tasks and 1,800 Evaluation Requirements drawn from real player requests, commercial-mechanics reconstruction, and human expert design. It covers PC Web, Mobile Web, 2D, and 3D. Requirements distinguish Completeness, assessing publicly specified core implementation, from Richness, assessing hidden, deeper system relationships. Static source-code judgments provide requirement-level diagnosis across capabilities and task conditions. Evaluation of 11 model-harness combinations yields 33.2% Completeness and 32.5% Richness; 19.9% of generation records are static gate failures or missing artifacts. Excluding same-model self-evaluation, even the best combination remains below 50% on both public and hidden requirements. Richness is not harder than Completeness under every task condition. Systems perform relatively better on Core Rules and Numerical Design, while AI/NPC and Feedback and Game Feel show more prominent gaps. Requirement difficulty offers coarse stratification but is only weakly negatively correlated with pass rates. A limited human cross-check finds that requirements judged satisfied by humans but failed by the automated evaluator are the main direction of disagreement. OCTAVE extends evaluation beyond aggregate scores to diagnosis localized by requirement, capability, and task condition.
Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Cite as: arXiv:2611.17681 [cs.AI]
  (or arXiv:2611.17681v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.17681

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Kehang Li [view email]
[v1] Fri, 13 Nov 2026 02:48:04 UTC (285 KB)

来源:arXiv cs.AI · arxiv.org