OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation

• Main Authors: Yilmaz, Gonca, Peng, Songyou, Pollefeys, Marc, Engelmann, Francis, Blum, Hermann
• Format: Journal Article
• Language: English
• Published: 30-05-2024
• Subjects: Computer Science - Computer Vision and Pattern Recognition
• DOI: 10.48550/arxiv.2405.20141

Recently, Vision-Language Models (VLMs) have advanced segmentation techniques
by shifting from the traditional segmentation of a closed-set of predefined
object classes to open-vocabulary segmentation (OVS), allowing users to segment
novel classes and concepts unseen during training of the segmentation model.
However, this flexibility comes with a trade-off: fully-supervised closed-set
methods still outperform OVS methods on base classes, that is on classes on
which they have been explicitly trained. This is due to the lack of
pixel-aligned training masks for VLMs (which are trained on image-caption
pairs), and the absence of domain-specific knowledge, such as autonomous
driving. Therefore, we propose the task of open-vocabulary domain adaptation to
infuse domain-specific knowledge into VLMs while preserving their
open-vocabulary nature. By doing so, we achieve improved performance in base
and novel classes. Existing VLM adaptation methods improve performance on base
(training) queries, but fail to fully preserve the open-set capabilities of
VLMs on novel queries. To address this shortcoming, we combine
parameter-efficient prompt tuning with a triplet-loss-based training strategy
that uses auxiliary negative queries. Notably, our approach is the only
parameter-efficient method that consistently surpasses the original VLM on
novel classes. Our adapted VLMs can seamlessly be integrated into existing OVS
pipelines, e.g., improving OVSeg by +6.0% mIoU on ADE20K for open-vocabulary 2D
segmentation, and OpenMask3D by +4.1% AP on ScanNet++ Offices for
open-vocabulary 3D instance segmentation without other changes. The project
page is available at https://open-das.github.io/.