Towards relation extraction from speech (2022)

Abstract: Relation extraction typically aims to extract semantic relationships between entities from the unstructured text.One of the most essential data sources for relation extraction is the spoken language, such as interviews and dialogues.However, the error propagation introduced in automatic speech recognition (ASR) has been ignored in relation extraction, and the end-to-end speech-based relation extraction method has been rarely explored.In this paper, we propose a new listening information extraction task, i.e., speech relation extraction.We construct the training dataset for speech relation extraction via text-to-speech systems, and we construct the testing dataset via crowd-sourcing with native English speakers.We explore speech relation extraction via two approaches: the pipeline approach conducting text-based extraction with a pretrained ASR module, and the end2end approach via a new proposed encoder-decoder model, or what we called SpeechRE.We conduct comprehensive experiments to distinguish the challenges in speech relation extraction, which may shed light on future explorations. We share the code and data on https://github.com/wutong8023/SpeechRE.

## introduction relation extraction (re) @xcite is an important information extraction task, which aims at extracting structured semantic relations between entities from unstructured data, typically text. besides text, there is also a plethora of speech data that is being continually produced. these include news reports, interviews, meetings and dialogues, to name a few. extracting relations from speech is an important but under-explored problem. in this work, we take the first step towards addressing relation extraction (re) from speech, introducing a new information extraction task, speech relation extraction (speechre). the input for this task is raw audio and the output is one or more triplets, each of which representing a relation between a pair of two entities appearing in the speech, e.g., @xcite . speechre and text-based re (textre) both involve content understanding. the former is more challenging than the latter, mainly due to the characteristics of speech. (i) speech carries much richer information beyond linguistic content (unlike text), for instance, emotion, speaker style and background noise; and it is non-trivial to disentangle the content element @xcite , which is needed for speechre. (ii) speech is continuous without sequence/word boundaries, implying the difficulty of determining the exact audio spans for target words (entity and relation). (iii) audio signals are orders of magnitude longer than the corresponding transcripts, which makes speech encoding for long-span extraction more challenging due to more demanding hardware requirements, especially with transformer @xcite . in the absence of speechre training data, we construct three benchmark datasets for this task by converting two commonly used textre datasets (i.e., conll04 @xcite and retacred @xcite to speech with a sota text-to-speech (tts) system. we then pair the synthetic speech with the corresponding target relation triplets as instances. to better evaluate model performance on real speeches, we also compile a human-read test set. we approach speechre with a pipeline method, speechre pipe , and an end-to-end (e2e) method, speechre e2e . in speechre pipe , we train our pipeline model with an automatic speech recognition (asr) module that converts speech to text, followed by a re module that extracts triplets. in speechre e2e , we build a single speech-to-text model that extracts triplets directly from speech. we use a sota pretrained speech encoder, wav2vec2 (w2v2) @xcite and bart @xcite decoder. inspired by @xcite , we attach a length adaptor on top of the encoder to bridge the length mismatch between speech representation and text representation. the end-to-end approaches in speech processing tasks often face more severe data scarcity issues than the pipeline approaches @xcite , as the latter can essentially leverage massive asr data and labelled data for the downstream text-based tasks. to tackle this challenge, we further propose two data augmentation techniques: upsampling via generating speech with different voices, and pseudo-labelling @xcite by leveraging abundant asr data and a sota textre system. our contributions can be summarized as follows. ## speech relation extraction we define speechre as a joint entity and relation extraction task that takes a speech utterance as the input and generates a set of relational triplets in the form of [entity1, relation, entity2] as the output. in this section, we first describe the data construction method ( §3.1). next, we present our two approaches to the task ( §3.2). last, we describe our two data augmentation techniques ( §3.3) to improve end-to-end speechre performance. ## discussion based on our analysis, we discuss the following question: for a new speechre task, should we choose a pipeline or an end-to-end approach? while raw performance is largely attributed to data resources, to answer this question, other factors need to be taken into account in addition to the availability of data resources. these include compute power and latency. pipeline method is suitable in the low-resource scenarios. as shown in figure 3 , prior to 100%, speechre pipe requires less training data to train than speechre e2e , while exhibiting reasonably good performance. therefore, the general asr method based on pre-training provides a reasonable performance lower bound for low-resource speech extraction. the major concern is errors contained in entities, as shown in §4.8. as a future direction, we conjecture that this issue could be potentially alleviated by the mixed extraction method from both transcript and speech. yet, the pipeline approach may be limited by fundamental issues (e.g. error propagation and high latency) that cannot be solved easily. the end-to-end method is preferred when labelled training data size is sufficient or external data is accessible. according to figure 3 , with the increasing volumes in training set, the performance of speechre e2e on extracting correct entities and relations steadily improves. extracting meaning from speech directly avoids the risk of information loss and error propagation, unlike in the pipeline setting. because of this, the e2e approach can potentially solve the extraction task in a principled manner. the data scarcity issue that it faces can be eased through data augmentation, for its effectiveness on both machine-generated speech and realistic speech. exploring more sophisticated augmentation and filtering techniques is thus a fruitful future direction. further, it is of importance to improve data efficiency and enhance entity prediction performance. particularly, enforcing constraints on entities in the decoding process and the inclusion of memory banks are promising directions. ## conclusion we propose a new spoken language understanding task, speech relation extraction (speechre), and present two synthetic datasets and a human-read test set. we approach speechre with two methods, the pipeline and e2e approaches. through extensive experiments, quantitative and qualitative analyses, we identify data scarcity and spoken entity recognition as two main challenges for this task. we then present two augmentation techniques that are effective in addressing these challenges. lastly, being the first working on the task, we outline key directions for future research.