Gym-Anything-RT：面向视觉语言智能体的实时环境基准
Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
AI 导读
Pranjal Aggarwal 与 Sean Welleck 发布 Gym-Anything-RT，用智能体流程基于六项实时交互能力自动生成实时小游戏，构建 100 个环境并开源。评测发现实时交互能力差距显著，GPT-6 Astra (xHigh) 实时模式通过率仅 9.0%，暂停环境后升至 75.0%；最强智能体在可写代码时会绕开实时感知与行动。
正文 · 原文
Abstract:AI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. Gym-Anything-RT uses an agentic pipeline to automatically generate such environments based on a taxonomy of six capabilities required for real-time interaction, including the ability to perceive the environment and act under an evolving environment state. A key feature of Gym-Anything-RT is that it allows for controlled comparison of real-time interaction capabilities: we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. We find large capability gaps in real-time interaction. For example, GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. We also identify intriguing behaviors by the most powerful frontier agents: for instance, when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. We publicly release Gym-Anything-RT and the generated minigames to spur further research on AI agents in real-time interactive settings.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37567 [cs.LG] |
| (or arXiv:2610.37567v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37567 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pranjal Aggarwal [view email]
[v1]
Tue, 27 Oct 2026 17:13:05 UTC (1,973 KB)
来源：arXiv cs.AI · arxiv.org