Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances (2024)

Abstract: Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions. Existing methods manifest limitations in leveraging nonverbal information for discerning complex semantics in unsupervised scenarios. This paper introduces a novel unsupervised multimodal clustering method (UMC), making a pioneering contribution to this field. UMC introduces a unique approach to constructing augmentation views for multimodal data, which are then used to perform pre-training to establish well-initialized representations for subsequent clustering. An innovative strategy is proposed to dynamically select high-quality samples as guidance for representation learning, gauged by the density of each sample’s nearest neighbors. Besides, it is equipped to automatically determine the optimal value for the top- K parameter in each cluster to refine sample selection. Finally, both high- and low-quality samples are used to learn representations conducive to effective clustering. We build baselines on benchmark multimodal intent and dialogue act datasets. UMC shows remarkable improvements of 2-6% scores in clustering metrics over state-of-the-art methods, marking the first successful endeavor in this domain. The complete code and data are available at https://github.com/thuiar/UMC.

## introduction discovering the semantics of dialogue utterances in unsupervised multimodal data requires integrating various modalities (i.e., text, video, and audio) to effectively mine the complicated semantics inherent in multimodal language. conventional methods for semantics discovery typically focus solely on the text modality with clustering algorithms @xcite , failing to leverage the rich multimodal information in the real world (e.g., body language, facial expressions, and tones). understanding semantics in multimodal utterances has attracted much attention with the boom in multimodal language analysis @xcite @xcite . for example, @xcite annotated multimodal dialogue act (da) labels on two popular multimodal multi-party conversational datasets @xcite and performed da recognition using attention sub-networks build upon modality encoders. @xcite pioneered multimodal intent analysis, introducing a new dataset with multimodal intent labels and establishing baselines with three multimodal fusion methods @xcite @xcite . however, these works remain restricted within supervised tasks, i.e., the training target for each piece of data is known, which is not applicable in unsupervised scenarios. in contrast, semantics discovery is an emerging field in nlp. it fundamentally operates as a clustering task and has seen the development of many unsupervised @xcite @xcite and semi-supervised @xcite @xcite methods. however, these methods are primarily designed for the text-only modality and lack proficiency in handling the diverse modalities encountered in real-world scenarios. thus, there is a lack of multimodal clustering methods for discovering utterance semantics, posing two challenges: (1) determining how to leverage information from nonverbal modalities to complement the text modality in clustering and (2) devising ways to fully exploit multimodal unlabeled data to learn clusteringfriendly representations. to address these challenges, we introduce umc, a novel unsupervised multimodal clustering algorithm for semantics discovery, as shown in figure 2 . we utilize the capabilities of the pre-trained language model @xcite to process text data. for the video and audio modalities, deep features are initially extracted using powerful backbones from computer vision and speech signal processing. two transformer encoders are then employed to capture the deep semantics of these features. the text modality is designated as the anchor, guiding the learning of the other modalities. for this purpose, we concatenate features from all three modalities and mask the video or audio features with zero vectors, creating two sets of positive augmentation views. these multimodal representations and their augmentations are applied to an unsupervised contrastive loss, yielding wellinitialized representations for subsequent process. to fully mine the semantic similarities among unsupervised multimodal data, we introduce a novel strategy that initially selects high-quality samples. this strategy employs a dynamic sample selection threshold t, aiming to select the highest-quality t percent of samples in each iteration for training. this selection is based on a unique mechanism that calculates the density of each sample within its respective cluster and ranks them accordingly. besides, an evaluation process is designed to automatically determine the optimal parameters for the top-k nearest neighbors from a set of candidates. after selecting high-quality samples, we propose a sequential process for multimodal representation learning. this process begins by learning from high-quality samples using supervised contrastive loss and then refines the remaining low-quality samples using unsupervised contrastive loss. this twostep approach promotes beneficial intra-class and inter-class relations among high-quality samples while pushing apart low-quality samples, thereby generating representations conducive to clustering. the entire process is repeated until the sample selection threshold t is met. we summarize our contributions as follows: in this work, we make a pioneering contribution by formulating the challenging multimodal semantics discovery task. to solve this problem, we first introduce a novel method for constructing positive augmentations for multimodal data, effectively leveraging non-verbal modalities for unsupervised pre-training, which provides a good initialization for unsupervised clustering. then, we propose a new clustering algorithm, umc, which features an innovative high-quality sample selection strategy and a sequential representation learning method between high-and lowquality samples, resulting in excellent performance across both single and multimodal modalities. finally, we establish baselines using benchmark multimodal intent and dialogue datasets. extensive experiments show that the proposed umc outperforms state-of-the-art clustering algorithms by a notable margin of 2-6% scores in standard clustering metrics. to the best of our knowledge, this is the first successful attempt at leveraging multiple modalities for unsupervised clustering, marking a substantial advancement in this area. a-transformer plm encoder v-transformer linear linear linear z t 0 z a cluster 𝐶1 cluster 𝐶3 cluster 𝐶2 figure 2: overview of our proposed unsupervised multimodal clustering algorithm umc. clustering @xcite iteratively assign clusters until convergence based on features. deep clustering methods, like @xcite and dcn @xcite , enhance this process by jointly clustering and feature learning, employing stacked autoencoders @xcite . deepcluster @xcite uses cluster assignments as guidance for feature learning. recent methods using contrastive learning @xcite have achieved the state-of-the-art performance. for instance, sccl @xcite combines instance-level contrastive learning with cluster refinement from target distributions. cc @xcite optimizes contrastive losses at both instance and cluster levels to generate clustering-friendly representations. however, these methods focus on merely the single text or image modality and fall short with multimodal data. mcn @xcite is tailored for multimodal clustering, learning a unified representation from all modalities and applying cross-modal contrastive losses during clustering. however, mcn struggles with complex utterance semantics. ## problem formulation for the task of multimodal semantics discovery, we are provided with a multimodal intent or dialogue act dataset d @xmath0, s a i , s v i )|y @xmath1, @xmath2, ..., n }, where each i th instance s i contains multimodal utterances, including s t i , audio s a i , and video s v i . here, n represents the total number of instances. the ground-truth label y i , belonging to the set of intent or dialogue act classes @xmath3, remains unseen during training and validation and is only available during testing. the number of classes is denoted by k y . the objective is to learn a multimodal neural network f(•) capable of obtaining multimodal representations z conducive to clustering. these representations are subsequently employed to divide the set {s i } n @xmath4. ## methodologies 4.1 multimodal representation to obtain multimodal representations, we first extract deep features from text, video, and audio modalities. for text, we employ the pre-trained language model (plm), bert @xcite as the encoder, fine-tuning it on the text inputs s t . the initial [cls] token embedding, x @xmath5, serves as the sentence-level representation, where d t is the feature dimension of 768. we then incorporate a linear layer, represented as f t (•), yielding z @xmath6. here, h indicates a dimensionally reduced space, enhancing computational efficiency and accentuating primary features. for non-verbal modalities, we use semantically rich features as inputs as suggested in @xcite . for video, we employ the swin transformer @xcite to extract video feature representations x @xmath7. here, l v represents the video length, and d v is the feature dimension of 1024. for audio s a , we first extract audio waveforms as in @xcite and then use the wavlm @xcite to obtain features x @xmath8. here, l a and d a denote the audio length and feature dimension of 768, respectively. unsupervised multimodal clustering can benefit from these two powerful non-verbal features extracted from the swin transformer and wavlm models. a comparison between them and other multimodal features is shown in appendix d. for both audio and video modalities, initially introduce a linear layer f m (•) in alignment with the text modality. subsequently, we apply the multi-headed attention mechanism with the transformer @xcite encoder, adeptly capturing intricate semantic relationships and temporal nuances. eventually, in line with @xcite , the last sequence elements are employed to derive the sentence-level representation z m : here, σ gelu represents the gelu activation function, and w 1 and b 1 are the corresponding weight and bias matrices, respectively. subsequently, we employ z tav and its augmentations for further clustering and representation learning. ## conclusions this paper introduces the multimodal semantics discovery task and proposes a novel unsupervised multimodal clustering (umc) method to address this critical challenge. umc effectively utilizes non-verbal modalities for semantics discovery by constructing positive multimodal data augmentations. besides, it proposes a novel high-quality sample selection mechanism and a two-step representation learning strategy. we conduct extensive experiments on both multimodal intent and dialogue act benchmark datasets. umc achieves remarkable improvements of 2-6% in standard clustering metrics compared to stateof-the-art clustering algorithms. further analyses demonstrate the effectiveness of each component and the robustness of the learned representations conducive to clustering. we believe this work makes significant progress in this area and provide a solid foundation for related research.