Scaling Up Multi-domain Semantic Segmentation with Sentence Embeddings

• Published in: International journal of computer vision Vol. 132; no. 9; pp. 4036 - 4051
• Main Authors: Yin, Wei, Liu, Yifan, Shen, Chunhua, Sun, Baichuan, van den Hengel, Anton
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-09-2024; Springer Nature B.V
• Subjects: Artificial Intelligence; Computer Imaging; Computer Science; Computer vision; Datasets; Embedding; Image annotation; Image Processing and Computer Vision; Image segmentation; Instance segmentation; Labels; Pattern Recognition; Pattern Recognition and Graphics; Scaling up; Semantic segmentation; Semantics; Sentences; Taxonomy; Vision; Multi domain; Semantic segmentation; Sentence embedding; Open world
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-024-02060-4

The state-of-the-art semantic segmentation methods have achieved impressive performance on predefined close-set individual datasets, but their generalization to zero-shot domains and unseen categories is limited. Labeling a large-scale dataset is challenging and expensive, Training a robust semantic segmentation model on multi-domains has drawn much attention. However, inconsistent taxonomies hinder the naive merging of current publicly available annotations. To address this, we propose a simple solution to scale up the multi-domain semantic segmentation dataset with less human effort. We replace each class label with a sentence embedding, which is a vector-valued embedding of a sentence describing the class. This approach enables the merging of multiple datasets from different domains, each with varying class labels and semantics. We merged publicly available noisy and weak annotations with the most finely annotated data, over 2 million images, which enables training a model that achieves performance equal to that of state-of-the-art supervised methods on 7 benchmark datasets, despite not using any images therefrom. Instead of manually tuning a consistent label space, we utilized a vector-valued embedding of short paragraphs to describe the classes. By fine-tuning the model on standard semantic segmentation datasets, we also achieve a significant improvement over the state-of-the-art supervised segmentation on NYUD-V2 (Silberman et al., in: European conference on computer vision, Springer, pp 746–760, 2012) and PASCAL-context (Everingham et al. in Int J Comput Visi 111(1):98–136, 2015) at
60
%
and
65
%
mIoU, respectively. Our method can segment unseen labels based on the closeness of language embeddings, showing strong generalization to unseen image domains and labels. Additionally, it enables impressive performance improvements in some adaptation applications, such as depth estimation and instance segmentation. Code is available at
https://github.com/YvanYin/SSIW
.