MSCTD : A Multimodal Sentiment Chat Translation Dataset (2022)

Abstract: Multimodal machine translation and textual chat translation have received considerable attention in recent years. Although the conversation in its natural form is usually multimodal, there still lacks work on multimodal machine translation in conversations. In this work, we introduce a new task named Multimodal Chat Translation (MCT), aiming to generate more accurate translations with the help of the associated dialogue history and visual context. To this end, we firstly construct a Multimodal Sentiment Chat Translation Dataset (MSCTD) containing 142,871 English-Chinese utterance pairs in 14,762 bilingual dialogues. Each utterance pair, corresponding to the visual context that reflects the current conversational scene, is annotated with a sentiment label. Then, we benchmark the task by establishing multiple baseline systems that incorporate multimodal and sentiment features for MCT. Preliminary experiments on two language directions (English-Chinese) verify the potential of contextual and multimodal information fusion and the positive impact of sentiment on the MCT task. Additionally, we provide a new benchmark on multimodal dialogue sentiment analysis with the constructed MSCTD. Our work can facilitate researches on both multimodal chat translation and multimodal dialogue sentiment analysis.

## introduction multimodal machine translation @xcite and textual chat translation @xcite @xcite ) mainly focus on investigating the potential visual features and dialogue context, respectively. both of them have received much attention. although plenty of studies on them have been carried out based on either image captions @xcite @xcite @xcite or textual dialogues @xcite @xcite , to our knowledge, little research work has been devoted to multimodal machine translation in conversations. one important reason is the lack of multimodal bilingual conversational datasets. generally, conversation in its natural form is multimodal @xcite . when humans converse, what a speaker would say next depends largely on what he/she sees. that is, the visual information plays a key role in (i) supplementing some crucial scene information (e.g., the specific locations or objects, or facial expressions), (ii) resolving ambiguous multi-sense words (e.g., bank), and (iii) addressing pronominal anaphora issues (e.g., it/this). for instance, as shown in fig. 1 (a), the image obviously points out the current location "on the sea", which may help disambiguate the meaning of "course" in the utterance x 5 . specifically, the dialogue history (i.e., talking about maritime affairs) and the corresponding visual context (i.e., on the sea/boat) assist us to determine that the word "course" means "route/direction" instead of "curriculum". in fig. 1 (b) , the visual context indicates object information, i.e., the "defibrillator" in x 1 , which may help with translation. in fig. 1 (c ), the image of the utterance x 1 also demonstrates that it can provide appropriate candidates (i.e., the jeans) when translating the pronoun "these". besides, the image offers some clues to judge the sentiment when it is hard to judge the polarity based only on the utterance (e.g., y 2 in fig. 1 (b ) and x 3 in fig. 1 (c )). all of the above call for a real-life multimodal bilingual conversational data resource that can encourage further research in chat translation. in this work, we propose a new task named multimodal chat translation (mct), with the goal to produce more accurate translations by taking the dialogue history and visual context into consideration. to this end, we firstly construct a multimodal sentiment chat translation dataset (msctd). the msctd includes over 17k multimodal bilingual conversations (more than 142k english-chinese and 30k english-german utterance pairs), where each utterance pair corresponds with the associated visual context indicating where it happens. in addition, each utterance is annotated with one sentiment label (i.e., positive/neutral/negative). based on the constructed msctd, we benchmark the mct task by establishing multiple transformer-based @xcite systems adapted from several advanced representative multimodal machine translation models @xcite and textual chat translation models @xcite . specifically, we incorporate multimodal features and sentiment features into these mod-els for a suitable translation under the current conversational scene. extensive experiments on four language directions (english↔chinese and english↔german) in terms of bleu @xcite , meteor @xcite and ter @xcite , demonstrate the effectiveness of contextual and multimodal information fusion, and the positive impact of sentiment on mct. furthermore, experiments on the multimodal dialogue sentiment analysis task of the three languages show the added value of the proposed msctd. in summary, our main contributions are: ## tasks in this section, we firstly clarify the symbol definition, and then define the proposed multimodal chat translation task and the existing multimodal dialogue sentiment analysis task. in a multimodal bilingual conversation (e.g., fig. 1 (a )), we assume the two speakers have alternatively given utterances in different languages for u turns, resulting in x 1 , x 2 , x 3 , x 4 , ..., x u and y 1 , y 2 , y 3 , y 4 , ..., y u on the source and target sides, respectively, along with the corresponding visual context representing where it happens: z 1 , z 2 , z 3 , z 4 , ..., z u . among these utterances, x 1 , x 3 , x 5 , ..., x u are originally spoken by the first speaker and y 1 , y 3 , y 5 , ..., y u are the corresponding translations in the target language. similarly, y 2 , y 4 , y 6 , ..., y u-1 are originally spoken by the second speaker and x 2 , x 4 , x 6 , ..., x u-1 are the translated utterances in the source language. according to languages and modalities, we define three types of context: (1) the dialogue history context of x u on the source side as c @xmath0, x 2 , x 3 , ..., x u-1 }, and (2) that of y u on the target side as ## dataset in this section, we mainly introduce our msctd in five aspects: data source § 3.1, annotation procedure § 3.2, annotation quality assessment § 3.3, dataset statistics § 3.4, and the introduction of related datasets § 3.5. ## image features following previous work @xcite @xcite , we focus on two types of image representation, namely the coarsegrained spatial visual feature maps and the finegrained object-based visual features. ## baseline models to provide convincing benchmarks for the msctd, we perform experiments with multiple transformerbased @xcite models for the multimodal chat translation task. additionally, we provide several baselines for the multimodal dialogue sentiment analysis task. ## conclusion and future work in this paper, we introduce a new multimodal machine translation task in conversations. then, we construct a multimodal sentiment chat translation dataset named msctd. finally, we establish multiple baseline systems and demonstrate the importance of dialogue history and multimodal information for mct task. additionally, we conduct multimodal dialogue sentiment analysis task on three languages of the msctd to show its added value. mct is a challenging task due to the complex scene in the msctd, leaving much room for further improvements. this work mainly focuses on introducing the new task and dataset, and we provide multiple models to benchmark the task. in the future, the following issues may be worth exploring to promote the performance of mct: