MELD - ST : An Emotion-aware Speech Translation Dataset (2024)

Abstract: Emotion plays a crucial role in human conversation. This paper underscores the significance of considering emotion in speech translation. We present the MELD-ST dataset for the emotion-aware speech translation task, comprising English-to-Japanese and English-to-German language pairs. Each language pair includes about 10,000 utterances annotated with emotion labels from the MELD dataset. Baseline experiments using the SeamlessM4T model on the dataset indicate that fine-tuning with emotion labels can enhance translation performance in some settings, highlighting the need for further research in emotion-aware speech translation systems.

## introduction human conversation naturally involves emotion. an addressee relies on the speaker's multimodal cues, such as vocal tones and facial expressions, to understand the meaning of an utterance. handling emotions in machine learning systems is therefore considered an important task, as exemplified by nlp tasks such as sentiment analysis and emotion recognition in conversation @xcite . considering emotion in translation is also important. for example, the phrase "oh my god!" can express a wide range of emotions or reactions, including surprise, shock, awe, excitement, distress, etc., to convey a strong emotional response to a situation, whether positive or negative. because the literal translation of this wouldn't make sense in a different culture, such emotional phrases need to be translated differently depending on the emotion. for instance, the japanese translation of the phrase showing surprise could be "マ ジ か! (majika)," whereas it could be "やった! (yatta)" when it shows excitement. ## related work st performance has significantly improved in recent years owing to the development of datasets, including s2tt datasets such as @xcite or covost 2 @xcite datasets, as well as s2st datasets such as cvss @xcite or gigas2s @xcite @xcite datasets, inter alia. there are also st datasets focusing on specific aspects of st, such as gender @xcite or dialects @xcite , as well as specific settings such as subtitles @xcite or crosslanguage dialogue @xcite , inter alia. a recent study by seamless @xcite investigates emotion in st, presenting the seamlessexpressive model that captures prosody and preserves vocal style. they created mexpresso and mdral corpora as extensions of existing datasets @xcite , as well as automatically aligned and synthetic corpora. the meld-st dataset is based on the meld dataset @xcite , an emotion recognition dataset of multimodal multi-party conversation based on the tv series friends. it contains videos with english speech and is annotated with english text, sentiment foot_0 and emotion foot_1 labels, speaker information, and timestamps based on audio for each utterance. the key differences between the meld-st dataset and existing expressive st datasets include: 1) inclusion of emotion labels for each utterance, which can be useful for experiments and analyses. 2) origin from a tv series in an emotionally rich environment, with translations and acted speech by professionals, making it suitable for a pilot study of emotion-aware st research. 3) coverage of the en-ja language pair, introducing unique challenges such as the need for translation content adjustments, as described in section 1. ## meld-st dataset the meld-st dataset is constructed from translations obtained from a blu-ray disk and emotion labels from the meld dataset. this section describes the construction process. the dataset statistics are summarized in ## experimental settings we conducted s2tt and s2st experiments using seamlssm4t. this section provides the details of the experiments and evaluation settings. ## conclusion in this study, we presented the meld-st dataset, an st dataset in an emotionally rich situation, which contains both en-ja and en-de language pairs. we conducted baseline s2tt and s2st experiments with and without utilizing emotion labels, which showed that emotion labels can boost performance in some settings. for improving the translation performance, several approaches can be considered, such as training a multitask model of speech emotion recognition and st, and utilizing dialogue context in translation.