DEX：让视觉基础模型从透视图像泛化到鱼眼深度估计与开放词汇分割
From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation
AI 导读
arXiv 论文提出可学习参数模块 Distortion Extenders（DEX），用于让视觉基础模型泛化到鱼眼相机图像。DEX 通过自监督对齐损失将鱼眼图像的潜在嵌入变换为类似透视图像的表示，建模径向畸变系数及两种图像间的分布偏移。
正文 · 原文
Abstract:Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.27860 [cs.CV] |
| (or arXiv:2608.27860v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2608.27860 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.1007/978-3-032-37627-5_14
DOI(s) linking to related resources |
Submission history
From: Suchisrit Gangopadhyay [view email]
[v1]
Fri, 28 Aug 2026 03:10:18 UTC (30,430 KB)
[v2]
Mon, 26 Oct 2026 21:50:37 UTC (15,270 KB)
来源：arXiv cs.AI · arxiv.org