arXiv cs.AI· Shengtao Wu, Zichao Dong, Weixuan Qiao, Xiaolu Liu, Xin Zhan, Junbo Chen, Jianke Zhu·· 3 小时前AI 评分39
从基础 VLM 到多能驾驶 VLM：Triplet Incremental Training 三流增量训练框架
From Foundation VLM to Versatile Driving VLM: A Triplet Incremental Training Approach
AI 导读
一项 arXiv 论文提出 Triplet Incremental Training Framework，通过自动构建（图像、深度、QA 对）三元组，将 3D 几何感知注入通用 VLM，同时保留其开放世界能力。
正文 · 原文
Abstract:Adapting general Vision-Language Models (VLMs) to Autonomous Driving (AD) is thought to require heavy 3D spatial priors or intrusive architectural overhauls to compensate for a lack of spatial grounding. In this work, we present a counterintuitive pilot study: off-the-shelf general VLMs already exhibit remarkable adaptability across diverse driving tasks, while precise metric depth remains the primary missing link, a bottleneck readily addressable via onboard sensors and depth foundation models. To this end, we introduce the Triplet Incremental Training Framework, a non-intrusive paradigm that seamlessly instills 3D geometric awareness into a general VLM backbone while fully preserving its open-world capabilities. Driven by automatically constructed (image, depth, QA pair) triplets, our framework operates via a unified tri-stream training pipeline: an unfrozen primary VLM stream to adapt to driving tasks, an auxiliary depth stream that guides the visual backbone to implicitly imagine 3D scene geometry via explicit geometric constraints, and a frozen twin VLM stream employing logit-level KL distillation to prevent semantic drift. Crucially, the auxiliary depth branch is discarded during deployment, ensuring the enhanced geometric awareness comes with zero inference overhead. We evaluate our framework across four major benchmarks targeting four diverse capabilities spanning 3D spatial perception, driving-oriented language reasoning, end-to-end planning, and general multimodal understanding. Our approach outperforms state-of-the-art driving VLMs, rivals domain-specific expert models in 3D perception, and successfully retains its original general reasoning capabilities. This demonstrates that Triplet Incremental Training offers a unified, deployment-ready paradigm for transforming general foundation models into robust driving intelligences.
| Comments: | 13 pages, 3 figures, 7 tables. Accepted at the 10th Conference on Robot Learning (CoRL 2026) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16727 [cs.CV] |
| (or arXiv:2611.16727v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16727 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shengtao Wu [view email]
[v1]
Thu, 12 Nov 2026 12:39:25 UTC (21,173 KB)
来源：arXiv cs.AI · arxiv.org