Semantic-conditioned Dual Adaptation for Cross-domain Query-based Visual Segmentation (2023)

Abstract: Visual segmentation from language queries has attracted significant research interest. Despite the effectiveness, existing works require expensive labeling and suffer severe degradation when deployed to an unseen domain. In this paper, we investigate a novel task Cross-domain Query-based Visual Segmentation (CQVS), aiming to adapt the segmentation model from a labeled domain to a new unlabeled domain. The challenges of CQVS stem from three domain discrepancies: (1) multi-modal content shift, (2) uni-modal feature gap and (3) cross-modal relation bias. Existing domain adaptation methods fail to address them comprehensively and precisely (e.g. at pixel level), thus being suboptimal for CQVS. To overcome this limitation, we propose Semantic-conditioned Dual Adaptation (SDA), a novel framework to achieve precise feature- and relation-invariant across domains via a universal semantic structure. The SDA consists of two key components: Content-aware Semantic Modeling (CSM) and Dual Adaptive Branches (DAB). First, CSM introduces a common semantic space across domains to provide uniform guidance. Then, DAB seamlessly leverages this semantic information to develop a contrastive feature branch for category-wise pixel alignment, and design a reciprocal relation branch for relation enhancement via two complementary masks. Extensive experiments on three video benchmarks and three image benchmarks evidence the superiority of our approach over the state-of-the-arts.

## introduction vision-language understanding @xcite @xcite @xcite is a fundamental problem in deep learning. recently, in this field, query-based visual segmentation @xcite has received considerable critical attention, which aims at localizing the visual region at the pixellevel that semantically corresponds to the language query. though existing works have made tremendous progress, the manual collection of pixel-wise annotations is expensive and tedious in practice, raising a significant requirement for the application in unseen circumstances. however, varying construction conditions on different domains inevitably degrades the adaptation performance, which is formally known as the domain shift problem. in this paper, to breakthrough the constraint, we propose a novel task cross-domain query-based visual segmentation (cqvs). as shown in figure 1 , given a source domain with pixel-wise annotations and an unlabeled target domain, cqvs aims to adapt the segmentation model and recognize the query-aligned pixels on the target domain. to achieve effective adaptation for this crossmodal grounding task, we have to deal with three domain discrepancies as shown in figure 1 (c): (1) multi-modal content shift. the free-form query and aligned visual region describe open and diverse contents, leading to arbitrary semantic shift between domains, e.g. one domain mainly describes humans while the other focus more on animals. (2) uni-modal feature gap. even describing the same content, each modality may have huge feature gap between domains caused by varying conditions, e.g. visual light and linguistic syntax. (3) cross-modal relation bias. the relation between modalities are easily learned to be biased by domain-specific factors, especially when only source annotations is available. as illustrated, it cannot be guaranteed to be aligned across domains by separately aligning each modality and requires dedicated solutions. to mitigate domain discrepancies, domain adaptation (da) methods align the distributions across domains @xcite or learn domain-invariant representations @xcite . while promising, they are confined for cqvs by two critical limitations. (1) traditional da methods study the uni-modal tasks, e.g. image segmentation @xcite ) and text classification @xcite , which are insufficient without consideration of multi-modal content and cross-modal relation. (2) though recent works @xcite investigate the multi-modal tasks, they (i) are limited to image-level retrieval that is imprecise compared with pixel-level grounding, and (ii) only consider partial discrepancies (feature and relation) regardless of internal correlation between domain contents. motivated by the fact that humans can leverage abstract concepts to guide thinking about concrete details @xcite , we aim to model the high-level semantic structure in multimodal contents to harmonize the low-level pixel adaptation for feature and relation discrepancies. grounded on the above discussions, we propose semantic-conditioned dual adaptation (sda), a novel framework to achieve precise feature-and relation-level adaptation via a universal semantic structure. our proposed sda consists of two key components, content-aware semantic modeling (csm) and dual adaptive branches (dab). (1) csm builds a sharable semantic space across domains based on the multi-modal content. first, to discover the consistent contents from visual and textual modalities, we extract informative words with visual-guided attention. then we establish the semantic structure upon the contents, where we apply unsupervised clustering on the source domain and measure the cross-domain semantic similarity to identify common parts on the target domain. through this module, we implicitly encode the language gap and introduce semantic category infor-mation as common guidance to regularize adaptation. (2) next, dab seamlessly integrates dual branches to separately address feature and relation discrepancies. on the one hand, a contrastive feature branch leverages the semantic information to learn category-wise alignment for foreground pixels. to preserve pixel-level discrimination during aligning, we adopt contrastive learning @xcite to highlight relevant foreground pixels between domains and suppress diverse background pixels. on the other hand, a reciprocal relation branch mitigates the cross-modal relation bias via two reciprocal masks. concretely, the domain-based masks induced by source knowledge can provide precise but biased results and the semantic-based masks induced by semantic information can provide comprehensive but inaccurate results. with the complementary signals, we indirectly enhance relation via segmentation training on both domains and also directly maximize the mutual information between vision and language. in summary, our main contributions are listed as follows: (1) we propose a new task cqvs to explore domain adaptation for query-based visual segmentation. (2) we introduce a novel framework sda, which develops a content-aware semantic modeling module to model the multimodal contents and designs a dual adaptive branches module to mitigate the feature and relation discrepancies. ## conclusion in this work, we first study the task of cross-domain query-based visual segmentation. to address this problem, we propose semantic-conditioned dual adaptation, a novel framework that achieves the feature-and relation-level adaptation via a universal semantic structure. experiments shows that our framework performs consistently well on both query-based video and image benchmarks.