CLASP : Cross-modal Alignment Using Pre-trained Unimodal Models (2024)

Abstract: Recent advancements in joint speech-text pre-training have significantly advanced the processing of natural language. However, a key limitation is their reliance on parallel speech-text data, posing challenges due to data accessibility. Addressing this, our paper introduces an innovative framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. Utilizing pre-trained unimodal models, we extract distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. A unique swap reconstruction mechanism enhances the alignment and is followed by fusion via a multi-head mechanism, seamlessly merging modality-invariant and modality-specific representations. Testing for emotion recognition (SLU task) and idiom usage detection (NLU task) demonstrates robust performance, with commendable robustness to noise in text or speech data.

## introduction in recent years, advancements in speech-text pretraining for learning universal feature representations from large training corpora have been significant @xcite @xcite , leading to notable success in various unimodal @xcite and multimodal downstream tasks @xcite @xcite . these approaches leverage multimodal self-supervised learning objectives, including cross-modal masked data modeling @xcite and cross-modal contrastive learning @xcite , aimed at aligning speech representations with their corresponding text representations. however, a primary drawback of these methods is their reliance on parallel speech-text data during the pre-training phase. unlike non-parallel unimodal corpora collections, acquiring parallel corpora (e.g., datasets like librispeech @xcite ) often requires extensive manual filtering and annotation efforts. this necessity significantly limits the amount of parallel corpora compared to their unimodal counterparts, particularly in modality-scarce scenarios, such as when data is available in only one modality (e.g., languages that are primarily spoken). our paper addresses this limitation. our approach involves leveraging unimodal models pre-trained on non-parallel corpora to obtain speech and text representations, subsequently aligning them using a small sample of task-specific parallel data. this strategy not only reduces the dependence on large parallel corpora during pretraining but also enables the utilization of extensive and diverse repositories of unimodal data, expanding the effectiveness and applicability of speech and text pre-training methods. our study introduces a framework (clasp) for processing multimodal information in speech and text by learning both factorized and shared representations for each modality, followed by fusing the learned representations with only parallel data from downstream tasks. inspired by previous attempts to disentangle representations from different modalities into modality-specific and modalityinvariant components @xcite , and innovating beyond solely using contrastive learning at the modality level; we incorporate a multilevel contrastive learning mechanism for representation disentanglement. additionally, we propose a novel swap reconstruction mechanism based on the intuition that ideal modality-invariant and modality-specific representations should be capable of reconstructing their original unimodal representations when combined. this mechanism promotes tighter cross-modal alignment by aligning modality-invariant representations between text and speech within each batch of data. acknowledg-ing the significance of modality-specific representations for certain tasks, such as emotion recognition (e.g., prosodic elements for emotion @xcite ), and the varied roles of different heads in pretrained models of different modalities, we integrate a multi-head fusion mechanism to merge modality-specific and modality-invariant representations. clasp's efficacy is validated through extensive experiments on different language understanding tasks that benefit from combining the speech and text modalities. we experiment with two spoken language understanding (slu) tasks with parallel text input (emotion recognition @xcite and intent classification @xcite ) and an natural language understanding (nlu) task with parallel speech input. the latter, which we call multimodal idiom usage detection, is novel (innovating over text-only idiomatic/literal disambiguation previosuly studied @xcite and aims to differentiate between literal and figurative uses of idioms using both text and speech data. this task operates at a more granular level than emotion detection and is significant because many emotions are expressed through abstract feelings conveyed using figurative expressions @xcite @xcite . our approach results in consistent gains over several competent multimodal methods for both tasks. overall, the main contributions are as follows: multimodal pre-training with speech and text. compared to the extensive explorations in the realm of multimodal pre-training for vision-andtext @xcite , the area of speech-text pre-training remains underexplored. a few notable works in this domain include speechbert @xcite , ctal @xcite , slam @xcite , st-bert @xcite , calm @xcite , maestro @xcite and stpt @xcite , which focus on the joint training of speech and text with aligned data in the pretraining corpus, including those in low-resource data environments @xcite and conversational settings that improve modeling the contextual information @xcite . all these works rely on aligning the modalities during large-scale pre-training utilizing parallel data from speech and text. in contrast, our work utilizes unimodal pretrained models for speech and text, but relies on aligning the modalities using task-specific parallel data (potentially in smaller numbers compared to the large pre-training data), thereby achieving the multimodal alignment in a task-specific finetuning setting. ## framework our framework combines unimodal representations from pre-trained speech and text models into a multimodal representation in a fine-tuning stage for downstream tasks. this eliminates the necessity for pre-training on parallel speech-text data, and that of extensive parallel data. the functioning of our proposed framework can be divided into three main stages: (1) disentangling multimodal representation using a multi-level contrastive learning mechanism as previous works only perform contrastive learning on the modality level but ignoring the semantic level whereas both levels are important; (2) cross-modal alignment through a swap reconstruction mechanism which reconstructs the unimodal encoding from the shared modality-invariant encoding and modality-specific encoding from opposite modalities because the ideal modality-invariant and modality-specific representations should be capable of reconstructing their original unimodal representations when combined; and (3) fusing multimodal representation via a multi-head fusion mechanism to pay different attentions to different heads for different modalities. figure 1 illustrates the workflow of our proposed framework and its details follow. unimodal representations. we utilize unimodal pre-trained models to extract representations-roberta-base @xcite ) for text and wav2vec2-base @xcite for speech: here we briefly define notations used in the following model formulation. for the representation, the subscripts denote the modality (a for speech and t for text) and whether the representation is invariant (inv) or specific (spe) to the modality. thus, h i a,inv , stands for a representation in the speech modality in the modality-invariant setting. ## idiom usage detection when individuals articulate their emotions verbally, they delve into a somewhat abstract realm, often prompting the use of figurative language @xcite . thus, a precursor to detecting emotions is to detect the occurrence of figurative language, typically done using idiomatic expressions. here, we study this in the specific context of the use of idioms, which can be in their literal sense or idiomatic sense in a context-dependent manner @xcite . our key insight is that detecting whether an idiom is used in a literal or figurative sense from text (see ## results from the results presented in ## conclusion and future work we introduced a novel approach, clasp, that uses unimodal pre-trained models for multimodal tasks, eliminating the need for parallel speech-text corpora during pre-training. our framework includes a disentanglement mechanism, a swap reconstruction mechanism, and a multimodal representation fusion module. empirical evaluations on the slu and nlu tasks justify the efficacy of clasp and its broad applicability. notably, it demonstrated robustness against noisy multimodal data, highlighting its real-world utility and adaptability. additionally, given the substantial size of clasp, future efforts could focus on developing parameter-efficient methods for its training.