Multi-Level Cross-Modal Alignment for Speech Relation Extraction (2024)

Abstract: Speech Relation Extraction (SpeechRE) aims to extract relation triplets from speech data. However, existing studies usually use synthetic speech to train and evaluate SpeechRE models, hindering the further development of SpeechRE due to the disparity between synthetic and real speech. Meanwhile, the modality gap issue, unexplored in SpeechRE, limits the performance of existing models. In this paper, we construct two real SpeechRE datasets to facilitate subsequent researches and propose a Multi-level Cross-modal Alignment Model (MCAM) for SpeechRE. Our model consists of three components: 1) a speech encoder, extracting speech features from the input speech; 2) an alignment adapter, mapping these speech features into a suitable semantic space for the text decoder; and 3) a text decoder, autoregressively generating relation triplets based on the speech features. During training, we first additionally introduce a text encoder to serve as a semantic bridge between the speech encoder and the text decoder, and then train the alignment adapter to align the output features of speech and text encoders at multiple levels. In this way, we can effectively train the alignment adapter to bridge the modality gap between the speech encoder and the text decoder. Experimental results and in-depth analysis on our datasets strongly demonstrate the efficacy of our method.

## introduction relation extraction (re) aims to extract structured knowledge from unstructured data in the form of relation triplet. since such structured knowledge can benefit various downstream applications, many efforts have been devoted to this task. however, this work is done when liang zhang was interning at pattern recognition center, wechat ai, tencent inc, china. meanwhile, @xcite propose a baseline model for speechre, where a cnn-based length adapter is used to connect a speech encoder with a text decoder (see figure 1(a) ). since the speech encoder and text decoder are pre-trained on corpora of different modalities, a significant modality gap exists between them, thus limiting the performance of this model. moreover, speechre models are usually required to comprehensively understand the input speech at multiple levels to effectively extract relation triplets from it. particularly, the speechre model first recognizes entities in the input speech based on its token/entity-level information, and then predict the relations between these entities according to its overall semantics at the sentenc level. thus, it is essential to bridge the modality gap between the speech encoder and text decoder at multiple levels for better speechre model. although the modality gap issue has been widely studied in other speech-related tasks @xcite @xcite , it has not yet been explored in the context of speechre. in this paper, we first conduct a preliminary study to investigate the efficacy of advanced cross-modal alignment methods in other tasks for speechre. we observe that token-level and sentence-level alignment methods can enhance the entity recognition and relation extraction performance of the speechre model, respectively. meanwhile, we notice some obvious problems with these methods: 1) the ctc-based token-level alignment method tends to overfit high-frequency tokens, leading to the collapse of speech features. 2) the compression-thenalignment-based sentence-level alignment method loses considerable fine-grained information in the input speech, significantly limiting the model's entity recognition capability. based on the above analysis, we propose a multilevel cross-modal alignment model (mcam) for speechre, which consists of a speech encoder, an alignment adapter, and a text decoder. we first use the speech encoder to extract speech features from the input speech. then, we employ the alignment adapter to map the speech features into a suitable semantic space for the text decoder. finally, the text decoder is utilized to autoregressively generate relation triplets based on these speech features. the alignment adapter is designed to bridge the modality gap between our speech encoder and text decoder from multiple levels. to do this, during training, we introduce an additional text encoder to extract the text features from the input text, which will be removed during inference. then, we train the alignment adapter to align the feature sequences produced by the speech and text encoders at three levels: 1) token-level alignment. we first concatenate the text features in the current batch to create a token feature matrix, and then use it to calculate alignment scores from speech features to token features. lastly, we compute a ctc loss based on these scores and achieve the token-level alignment by minimizing the loss. since a token has distinct features in various contexts, our method can effectively avoid the overfitting issue for high-frequency tokens. 2) entity-level alignment. here, we construct a mixed feature sequence by replacing the text features of entities in the text feature sequence with their corresponding speech features. then, we minimize the kl divergence between the output distributions generated by the text decoder based on the original text feature sequence and the mixed one. 3) sentence-level alignment. we first separately compress the speech and text feature sequences into r global feature vectors using r relation-specific learnable vectors, serving as their soft prompts. then, we use a contrastive loss to align the final representations of soft prompts from different modalities. through the soft prompt strategy, we can prevent the loss of fine-grained information during the compression process. to evaluate the performance of speechre models in real-world scenarios and facilitate future studies, we annotate two real speechre datasets: conll04 and retacred. experimental results on the two datasets show that our model consistently outperforms all baselines. extensive ablation studies further demonstrate the effectiveness of various components in our model. notably, we conduct extensive analysis experiments on our dataset with the aim of inspiring subsequent research. ## preliminary study in this section, we investigate the effectiveness of advanced cross-modal alignment methods from other speech-related tasks for speechre. here, we consider two alignment methods: ctc-based token-level alignment (see figure 1(b) ) @xcite and compression-thenalignment-based sentence-level alignment (see figure 1(c )) @xcite . for the former, a ctc loss is employed to monotonically align speech features with the embedding vectors of their corresponding tokens. meanwhile, these speech features are projected into the embedding space of the text encoder and serve as its input. for the latter, an attention-based semantic compressor is utilized to compress the feature se in these results suggest that naively applying crossmodal alignment methods from other tasks to speechre is suboptimal. therefore, we customize a more effective cross-modal alignment method for speechre, which avoids the defects of the above methods and inherits their advantages. ## speechre datasets in this section, we provide a detailed description to the construction process of our speechre datasets. given a textre instance (x, y), where x and y respectively denote the input text and output relation triplets, we create a speechre instance (s, x, y) by converting x into its corresponding speech s in a human-read manner. specifically, we invite 12 native english speakers to transform the input texts in the conll04 @xcite and retacred @xcite datasets into fluent and clear speeches, thus obtaining two benchmark datasets for speechre. we totally annotate 7,246 instances over one month and each speaker is responsible for annotating 242 instances per day (about 2 hours) to prevent fatigue. the statistics of these two datasets are presented in to ensure the quality of speeches, we employ the whisper @xcite model to transcribe each speech into text, and then compare the resulting text with its true input text. meanwhile, we hire a professional english speaker to sample and check the annotated instances for further quality control. notably, for the retacred dataset, we only annotate its test set while retaining the training and validation sets synthesized by @xcite . using this dataset, we can effectively evaluate the model's robustness to the noise in synthetic data. during training, we introduce a text encoder to serve as a semantic bridge between the speech encoder and text decoder, which is removed during model inference. then, the alignment adapter is trained to align the output features of the speech and text encoders at three levels. in this way, the trained alignment adapter can effectively bridge the modality gap between the speech encoder and text decoder. in the following sections, we first individually detail each component involved in our model training (see sections 3.1-3.3). additionally, we introduce our model training in section 3.4. ## conclusion and future work in this paper, we first annotate two real speechre datasets to facilitate future research in the field of speechre. then, we propose a new speechre model mcam, which consists of a speech encoder, an alignment adapter, and a text decoder. the alignment adapter aims to project the speech features, extracted by the speech encoder from the input speech, into a suitable semantic space for the text decoder, enabling it to effectively generate relation triplets based on these speech features. by doing so, we successfully bridge the modality gap between the speech encoder and text decoder. to efficiently train our adapter, we introduce an additional text encoder and train the adapter to align the output features of the speech and text encoders at three levels. experiments on our two speechre datasets demonstrate the effectiveness of our model. in future, we plan to apply our model to other speech-related tasks, such as st and asr, so as to verify its generality. moreover, exploring methods to train a zero-shot speechre model using only vast asr and textre data is an interesting future research direction.