Towards Noise-Tolerant Speech-Referring Video Object Segmentation: Bridging Speech and Text (2023)

Abstract: Linguistic communication is prevalent in Human-Computer Interaction (HCI). Speech (spoken language) serves as a convenient yet potentially ambiguous form due to noise and accents, exposing a gap compared to text. In this study, we investigate the prominent HCI task, Referring Video Object Segmentation (R-VOS), which aims to segment and track objects using linguistic references. While text input is well-investigated, speech input is under-explored. Our objective is to bridge the gap between speech and text, enabling the adaptation of existing text-input R-VOS models to accommodate noisy speech input effectively. Specifically, we propose a method to align the semantic spaces between speech and text by incorporating two key modules: 1) Noise-Aware Semantic Adjustment (NSA) for clear semantics extraction from noisy speech; and 2) Semantic Jitter Suppression (SJS) enabling R-VOS models to tolerate noisy queries. Comprehensive experiments conducted on the challenging AVOS benchmarks reveal that our proposed method outperforms state-of-the-art approaches.

## introduction recent advances in vision-language learning have significantly advanced @xcite . a demanding task within hci is referring video object segmentation (r-vos), which involves segmenting and tracking objects in videos based on textual references. the successful development of r-vos techniques has paved the way for diverse real-world applications such as video editing @xcite and augmented reality @xcite . notably, recent r-vos methods @xcite have shown unprecedented progress, propelled by the rapid advancement of multimodal foundation models such as clip @xcite . these r-vos models enable various text-referred scenarios, allowing referring segmentation for generalized textual expressions even in complex visual scenes. "a pink cat on the grass" text: r-vos mask video: noisy speech: stbridge (a) (b) "a pink ca☐ on t☐e grass" figure 1: (a) referring video object segmentation (r-vos) employs text as a query for segmentation. (b) speech-referring video object segmentation. compared to written language (text), spoken language (speech) is a more noisy form, potentially involving greater information loss and disturbance due to background noises. we propose a plug-and-play stbridge module, which seamlessly extends a frozen text-conditioned r-vos model to accommodate noisy speech inputs. however, a more challenging scenario arises in the prevalent speech dialogue system, where the aim is to refer to specific targets using spoken language, i.e., speech. the inherent nature of speech introduces vulnerabilities to disturbances from background noises (sound except for the referring speech). consequently, crucial information within the spoken content can be distorted or even lost, which poses extra challenges in maintaining effective segmentation when referring to targets verbally. though previous r-vos methods achieve remarkable performance with textual queries, their performance with real-world spoken language is rarely discussed. hence, it is crucial to develop an effective approach to bridge the text and speech to adapt well-trained r-vos methods (frozen) for speech inputs. yet, bridging speech to r-vos methods introduces new challenges. a straightforward solution involves utilzing automatic speech recognition (asr) @xcite to convert speech to text, followed by text-conditioned referring segmentation. however, this can result in suboptimal performance for two primary reasons. (1) noise in language queries of r-vos. existing r-vos models rely on clean text-video pairs, wherein the textual expression unambiguously identifies the target object. nonetheless, information extracted from speech may be incomplete or distorted due to background noise and asr errors, leading to inadequate references to the target object. to maintain robust segmentation quality, it is crucial to adapt r-vos models to handle perturbed referring queries. (2) noise in speech understanding. in practice, speech and background noise are closely intertwined, making it difficult to accurately comprehend semantic information from speech. considering the diverse types of noise, an effective noise-tolerant speech understanding approach is vital for achieving robust speech-referring video object segmentation. in this paper, we present stbridge, a novel approach that enables r-vos models trained on clean text-video pairs to adapt to noisy speech as referring guidance, maintaining robust performance even amidst background noises. as illustrated in fig. 1 , the proposed stbridge links the well-trained r-vos model with speech input, incorporating two core considerations to improve the model's robustness: (1) enhancing the welltrained r-vos model to accept incomplete guidance, and (2) providing the noise-tolerant capability for speech understanding. on the one hand, we introduce a semantic-jitter suppression (sjs) module to help the r-vos model understand noisy information from referring guidance. the sjs module generates object queries with randomly jittered textual features, allowing the model to learn from incomplete referring guidance under proper supervision. on the other hand, we introduce a noise-aware semantic adjustment (nsa) module, which generates noise-adaptive filters to enhance the speech representation. this differs from traditional speech enhancement, as it focuses solely on encoded semantics during speech understanding, while discarding low-level information, i.e., waveforms. we further introduce a slack semantic alignment to align text and speech queries, enabling the integration of speech input with well-trained r-vos models. notably, our method incorporates additional modules without any retraining of r-vos models, which is essential for numerous real-world applications. in summary, our contributions are as follows: figure 2 : overview of stbridge. we extend frozen r-vos model (in blue) with trainable stbridge (in red) modules to adapt to noisy speech as input. (i) training: a speech encoder is utilized to extract noisy speech g ns and noise g n embeddings from noisy speech s. on one hand, a noise-aware semantic adjustment (nsa) module is utilized to mitigate the noise influence, which derives the cleaner speech embedding g s . on the other hand, to enhance the noise-tolerant capability of r-vos model, we first generate perturbation to the text embedding g t and then equip a semantic jitter suppression (sjs) module to suppress the noises. moreover, semantic alignment constraints are introduced to align the text g t and speech g s embeddings. (ii) inference: after aligning the text and speech embeddings during training, we can directly discard the text branch and leverage speech embedding g s as the input to the sjs module. onlinerefer @xcite employs the query propagation module to enable the online r-vos. spoken language understanding. spoken language, i.e., speech, enables a more natural way for humans to refer to a certain object than using text-based language. thanks to the emergence of datasets with paired images and speech, e.g., flicker8k @xcite and avos @xcite , more works @xcite @xcite started to research on the representation of speech and explore the synergy between speech and other modalities, e.g., image and video. for example, lavish @xcite incorporates a small set of latent tokens to align the visual and audio representation, and visu-alvoice @xcite conducts speech separation with the speaker's facial appearance as a conditional prior. later, research on speech has also moved towards finer granularity tasks. some works @xcite focus on the mono-modal impact of speech to study the subtle semantic information of spoken language to better understand human speech, while others @xcite study how to introduce the knowledge of speech understanding to create more natural human-computer interaction applications, e.g. talking head @xcite @xcite . ## method to ground objects verbally, we start from a frozen text-referring video object segmentation (r-vos) model (shown as blue modules in fig. 2 ), including frozen video-text encoders and a mask decoder. we introduce a speech encoder, a semantic jitter suppression (sjs) module, a noise-aware semantic adjustment (nsa) module, and a semantic alignment constraint to bridge text and speech (shown as pink modules in fig. 2 ). during training, stbridge leverages video v , text t , and noisy speech s triplets to align the query spaces between text and speech. thereafter, we can discard the text branch and directly query the objects with speech during inference. ## conclusion in conclusion, this paper presents stbridge, a novel approach that enables r-vos models trained on clean text-video pairs to adapt to noisy speech as referring guidance, maintaining robust performance. the approach incorporates semantic jitter suppression (sjs) and noise-aware semantic adjustment (nsa) modules to enhance noise tolerance in speech queries. experimental results demonstrate the effectiveness of stbridge, outperforming previous methods on three benchmarks. stbridge expands the applicability of r-vos models, enabling robust speech-referred video object segmentation in real-world scenarios. limitation. in spite of stbrdge's high performance on existing benchmarks, we only consider the scenario that text and speech queries are in the same language. bridging text and speech in different languages can impose more challenges as the semantic spaces may suffer more divergence, which will be our future focus.