RoboBuilder 基准:让 AI 智能体结合真实世界视频反馈协同设计机器人
RoboBuilder: Agentic Robot Co-Design with Real-world Video Feedback in the Loop
AI 导读
RoboBuilder 是一项让编程智能体协同设计机器人硬件与程序的基准,通过仿真的设计会被实体搭建并实测。研究在四个前沿编程智能体上开展,共进行 16,427 次仿真、构建 12,774 个不同机器人与 135 台实体机器人,智能体几乎能解决全部简单障碍,但真实世界中仅 66% 的仿真通过有效。引入真实失败视频反馈后,智能体可重新设计并消除 72% 的一、二级失败。
正文 · 原文
Abstract:Can AI agents design robots that work in the real world? We introduce RoboBuilder, a benchmark in which coding agents co-design a robot's hardware and program from standard parts, motors and a custom 3D-printed part, and every design that passes in simulation is built and tested on a physical course. Simulation alone is not enough: under public part data, none of 15 initial passing designs crossed the real course, so an agent learns the simulator from measurements it requests and videos of real runs. Across four frontier coding agents, ten obstacles at three difficulty levels, 16,427 simulations of 12,774 distinct robots and 135 physical robots built in an estimated 240 hours, agents solve nearly all easy obstacles and invent mechanisms our search baselines do not reach, but falter at harder levels, when one robot must cross every obstacle, and in the real world, where 66% of simulated passes hold across all levels. Closing the loop helps: shown a video of its real failure, an agent redesigns the robot, sometimes correcting its simulator, and removes 72% of level-1 and level-2 failures. RoboBuilder measures progress toward agents that engineer real machines. Project Page: this https URL
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO) |
| Cite as: | arXiv:2611.23855 [cs.AI] |
| (or arXiv:2611.23855v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23855 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ghadi Nehme [view email]
[v1]
Wed, 18 Nov 2026 05:02:07 UTC (24,219 KB)
来源:arXiv cs.AI · arxiv.org