Generative Error Correction for Emotion-aware Speech-to-text Translation (2025)

Abstract: This paper explores emotion-aware speech-to-text translation (ST) using generative error correction (GER) by large language models (LLMs). Despite recent advancements in ST, the impact of the emotional content has been overlooked. First, we enhance the translation of emotional speech by adopting the GER paradigm: Finetuned an LLM to generate the translation based on the decoded N-best hypotheses. Moreover, we combine the emotion and sentiment labels into the LLM finetuning process to enable the model to consider the emotion content. In addition, we project the ST model’s latent representation into the LLM embedding space to further improve emotion recognition and translation. Experiments on an English-Chinese dataset show the effectiveness of the combination of GER, emotion and sentiment labels, and the projector for emotion-aware ST. Our code is available at https://github.com/N-Orien/EmoST.

## introduction speech-to-text translation (st) is a task where the model takes speech in one language as input and translates it into text in another language. st performance has greatly improved over the recent years with significant efforts on datasets @xcite @xcite @xcite and models @xcite @xcite . however, an essential aspect often overlooked in speech translation is the emotion of speech. human speech naturally includes emotions. in real-life conversations, a listener often uses cues from the speaker's voice tone to grasp what is being said. therefore, emotion can significantly influence the results of translating speech. as the instance shown in figure 1 , the phrase "i can't believe this" can convey a range of emotions, from surprise and shock to awe and excitement, which can alter its translation in another language. emotion has been studied in machine translation (or text-to-text translation) studies @xcite and other tasks in natural language processing, such as sentiment analysis and recognizing emotions in conversations @xcite . however, there has been little effort focusing on emotion in st. seamless expressive @xcite examines the preservation of emotional states in speech-to-speech translation, without addressing the influence of emotions on the semantic aspects of translation. some datasets @xcite are constructed for emotion-aware st, but further community effort investigating the methodology for this task is required. meanwhile, recent advancements in large language models (llms) leads to growing interest in leveraging their capabilities in modalities beyond text including speech. training end-to-end st models often face challenges due to insufficient speechtext parallel data. however, llms are trained on vast amounts of textual data and obtain powerful textual generation abilities, which can enhance the st performance. this has been proven by recent studies that use llms as decoders for st systems @xcite or as generative error correction (ger) models to improve st qualities @xcite . speech-text parallel data is scarce, and it is even scarcer when it includes emotion annotations. therefore, leveraging external models like llms to help the system understand the correlation between emotion and language can be greatly beneficial. however, to the best of our knowledge, there have not been studies on utilizing llms for emotion-aware st. therefore, this study pioneers the exploration of the effectiveness of emotion-aware st by: (a) adopting the llm ger paradigm, (b) adding emotion and sentiment labels into the ger finetuning process, (c) injecting acoustic representation from the st model into ger finetuning with a projector. ## conclusion in this paper, we pioneered the investigation of emotion-aware st using llms. we proposed adopting the ger method, integrating emotion and sentiment labels, and injecting acoustic information from the speech into the ger finetuning process. the experimental results showed its effectiveness. future works include verifying the effectiveness of our method with other llms and datasets, as well as increasing the diversity of the n -best list and exploring semi-supervised strategies for the task.