Semantic Role Labeling from C hinese Speech via End-to-End Learning (2024)

Abstract: Semantic Role Labeling (SRL), crucial for understanding semantic relationships in sentences, has traditionally focused on text-based input. However, the increasing use of voice assistants and the need for hands-free interaction have highlighted the importance of SRL from speech.SRL from speech can be accomplished via a two-step pipeline directly: transcribing speech to text via Automatic Speech Recognition (ASR) and then applying text-based SRL, which could lead to error propagation and loss of useful acoustic features.Addressing these challenges, we present the first end-to-end approach for SRL from speech, integrating ASR and SRL in a joint-learning framework, focusing on the Chinese language. By employing a Stright-Through Gumbel-Softmax module for connecting ASR and SRL models, it enables gradient back-propagation and joint optimization, enhancing robustness and effectiveness.Experiments on the Chinese Proposition Bank 1.0 (CPB1.0) and a newly annotated dataset AS-SRL based on AISHELL-1 demonstrate the superiority of the end-to-end model over traditional pipelines, with significantly improved performance.

## introduction semantic role labeling (srl), as one of the core tasks in natural language processing (nlp, @xcite , concentrates on analyzing semantic relationships between predicates and their corresponding arguments in sentences, such as who did what to whom, as well as when, where and how it occurred @xcite . semantic information has been exploited in a variety of complex downstream nlp tasks including information extraction @xcite , question answering @xcite 2015; * authors contributed equally. † corresponding author: meishan zhang. 他[说] 高 兴 的 时 候 不[能][喝 水] asr 他[说] 搞 笑 的 时 候 不[能]和 谁 srl he said when you are funny you can not be with anyone he said when you are happy you can not drink water ta shuo gao xiao de shi hou bu neng he shui ta shuo gao xing de shi hou bu neng he shui •: stresses. space: pauses. →: undulations. under the pipeline architecture, error propagation can influence srl greatly (e.g., predicate "喝水(drink water)" would be lost ) and acoustic features such as stresses, pauses and undulations are unable to be employed by srl. @xcite , and robot command parsing @xcite . previous srl typically assumes input as text @xcite @xcite . however, in many real-world scenarios, such as for the elderly who may find typing challenging, or in situations requiring remote control of machines, speech often emerges as the more direct medium for conveying information @xcite . as voice assistants @xcite , social bots @xcite , and intelligent home devices @xcite become increasingly prevalent, the demand for language understanding capabilities that can process speech inputs grows correspondingly. thus, srl from speech directly would exhibit increasing interest. srl from speech can be accomplished via a twostep pipeline directly: (1) automatic speech recognition (asr) that transcribes the speech into text first, and then (2) text-based srl. however, such a pipeline could be insufficient for speech-based srl @xcite , as illustrated in figure 1 . on the one hand, the sequential pipeline might suffer from the commonly-recognized error propagation problem, i.e., the asr errors would propagate to srl @xcite , since traditional text-based srl always assume well-written text as input. on the other hand, after asr the acoustic features are completely isolated to the second step @xcite , while part of them might be useful for srl, e.g., the phonological features such as stress, pauses and undulations. to address the above issues, end-to-end learning of the two tasks is one natural solution. in this work, we present the first work to tackle srl from speech in an end-to-end way, uniting asr and srl in a joint-learning paradigm. we focus on the chinese language as the pioneer study here. we start from the state-of-the-art asr and srl models, and then connect the two models by a stright-through gumbel-softmax (gs) module @xcite . the gs enables gradient backpropagation over the intermediate discrete transcribed texts, therefore a joint optimization towards the asr and srl objectives can be achieved, and the end-to-end learning is formed naturally. additionally, as the transcribed text might be deformed into a noisy one, which makes gold-standard annotations difficult to be explored during srl learning, we design an alignment mechanism to reach a reasonable srl objective. this alignment also enables more robust evaluation for speech-based srl and akin. we conduct experiments on chinese proposition bank 1.0 (cpb1.0) @xcite mainly to verify the effectiveness of our end-to-end model. specifically, we recruit mandarin-qualified annotators to obtain authentic speech for the test set for evaluation, and the speech for the remaining part is annotated by a text-to-speech (tts) tool, foot_0 since the manual annotation is of high cost. in addition, we also build another evaluation dataset as-srl based on the open-source mandarin speech corpus aishell-1 @xcite , annotating it with srl by experts following the guideline of cpb1.0. results show that our end-to-end method is highly effective, achieving averaged improvements of 0.47% and 1.20% in the f 1 score com-pared with the standard pipeline counterpart. we also design a series of other strong baselines for comparisons, and the results show that our final model outperform all these systems. further, we perform in-depth analysis to provide a comprehensive understanding of our methodology. in summary, our contributions can be summarized as follows: ## method a straightforward solution for srl from speech is through a pipeline strategy, first executing asr and then performing srl. since the pipeline architecture induces error propagation inevitably and meanwhile cannot use acoustic clues for srl, we suggest end-to-end learning to unite the two tasks, as shown in figure 2 . this section first introduces a state-of-the-art pipeline, and then describes our end-to-end learning approach. ## related work semantic role labeling was originally conceptualized as a task for shallow natural language understanding by @xcite . the conventional approach to srl involves two subtasks: recently, there has been an increasing interest in consolidating the two subtasks by a single model @xcite . the line of work completes srl fully with sota performance, and meanwhile exhibits significant efficiency advantages. therefore, we follow this line of work, selecting @xcite as the srl baseline. asr has been concerned since very early @xcite @xcite . deep learning has advanced this task greatly, with models like las employing rnns to capture the contextual nuances of speech signals, leading to notable enhancements @xcite @xcite . moreover, the breakthrough success of the transformer architecture in various nlp tasks has inspired the integration of attention mechanisms into asr, further advancing its capabilities @xcite . whisper employs a transformer-based architecture, trained on large-scale corpora @xcite . it has essentially achieved sota performance on asr tasks. thus, whisper is selected as the model for our exploration of asr. the adventure asr technologies @xcite and the increasing availability of large-scale speech datasets @xcite @xcite have prompted a transition towards integrating speech directly into nlp tasks. many studies, addressing fundamental nlp tasks such as sentiment analysis @xcite , machine translation @xcite , and spoken language understanding @xcite , have adeptly leveraged speech, employing end-to-end approaches to achieve remarkable performance by exploiting intrinsic features of speech. however, for more intricate nlp tasks like parsing @xcite @xcite @xcite , pipeline models that convert speech to text before applying conventional nlp strategies are commonly used. our study pioneers srl from speech, proposing an end-to-end framework, marking a novel approach in this field. ## conclusion in this paper, we proposed a novel approach to srl from speech through an end-to-end methodology, integrating asr and srl into a unified learning framework. this work, pioneering in its focus on the chinese language, leverages state-of-the-art asr and srl models, interconnected via a stright-through gumbel-softmax module. the gs module facilitates gradient back-propagation across intermediate, discrete transcriptions, enabling joint optimization of asr and srl objectives and the natural formation of end-to-end learning. to address the challenge of transcriptions potentially becoming noisy, an alignment tool is devised, ensuring that srl learning can effectively utilize goldstandard annotations and allowing for a more robust evaluation of speech-based srl. we constructed two chinese speech-based srl datasets and performed experiments on them to evaluate our end-to-end method. our method not only demonstrates significant performance improvements over traditional pipeline methods but also outperforms a series of strong baseline models. further in-depth analysis provides a comprehensive understanding of the approach, illustrating its potential for broader application and setting a new benchmark in the field.