Investigating Effectiveness of Multi-Encoder for Conversational Neural Machine Translation (2022)

Abstract: Multilingual chatbots are the need of the hour for modern business. There is increasing demand for such systems all over the world. A multilingual chatbot can help to connect distant parts of the world together, without sharing a common language. We participated in WMT22 Chat Translation Shared Task. In this paper, we report descriptions of methodologies used for participation. We submit outputs from multi-encoder based transformer model, where one encoder is for context and another for source utterance. We consider one previous utterance as context. We obtain COMET scores of 0.768 and 0.907 on English-to-German and German-to-English directions, respectively. We submitted outputs without using context at all, which generated worse results in English-to-German direction. While for German-to-English, the model achieved a lower COMET score but slightly higher chrF and BLEU scores. Further, to understand the effectiveness of the context encoder, we submitted a run after removing the context encoder during testing and we obtain similar results.

## introduction translation of dialogues is a crucial part of building multilingual chatbots. with easier access to the internet than ever, we have the opportunity to connect with different people with different languages. however, language remains a barrier to smooth communication. using automated machine translation systems can alleviate such issues. however, most of the general mt systems are not very suitable for conversations. this is due to additional challenges chat translation possesses that general domains do not have. this includes the presence of noisy utterances. compared to other domains, chat is more prone to contain noisy sentences. this comes from multiple sources, as follows. a) keyboard typos: spelling mistakes that occurred due to quick typing. in this case, often, some characters are replaced by nearby characters on the keyboard. further, the insertion of extra characters or the absence of some characters is also common. b) intentional shortening of words: users often use short forms of words by removing certain characters (primarily vowels) while keeping the pronunciation similar to the correct word (for example, 'hw' instead of 'how'). c) grammatical errors: conversations usually occur in an informal setting, and grammar is mostly ignored as long as the meaning is understood correctly. however, this makes it difficult to translate. further, there are other challenges, like context dependency. that is, the utterances can be ambiguous, and the correct meaning of an utterance can not be understood without referring to its dialogue history. in this paper, we use a multi-encoder transformer to translate chat utterances. we use six encoder layers for source text and one encoder layer for context. for better comparison, we have submitted translations from two other models. to test the effectiveness of context, we did not provide context during the testing phase as described in section 3.3.2. further, we train another model without using any context at all as described in 3.3.3. we achieved very competitive results for the agent subset (english-to-german), where we obtained 0.551 bleu, 0.730 chrf, and 0.768 comet scores, where the best result among primary submissions of the participants are 0.555, 0.735, and 0.810 bleu, chrf and comet score respectively. for german-to-english, our method produced 0.907, 0.729, and 0.587 comet, chrf, and bleu scores, respectively. ## related work the area of chat translation mostly remained unexplored until recent years. this is in part due to the unavailability of suitable dialogue datasets. farajian et al. (2020) introduced an german-english parallel conversational corpus. @xcite proposed a method that replaced rare characters with a special '<@xmath0, which helps the model to learn when to copy the tokens from source to target. further, they used methods like inline casing, tagged back-translation (bt) @xcite , byte-pair-encoding (bpe) @xcite , and ensemble of models using domainspecific adaptive layers, etc. ensemble model with a domain-specific adaptor layer generated the best translation on wmt20 chat data. @xcite used fine-tuned pre-trained models @xcite on the pseudo-in-domain and indomain data. wang et al. (2020) 2022a ) observed visual features helps to generate better quality translation on multi-modal dialogue. apart from chat translation, context is commonly used in other translation tasks as well. this include document translation @xcite @xcite where other sentences from the document is used as context, multimodal translation @xcite where image features are used as context, etc. @xcite proposed a method where context is concatenated with source on both source and target side, requiring the model to translate context also, thus avoiding ignorance of context in question-answer translation. ## results we obtain a comet @xcite ) score of 0.768 and 0.907 on en-de and de-en directions. further, we obtain chrf @xcite scores of 0.730 and 0.729 for en-de and de-en. we obtain bleu scores of 0.551 and 0.587 for agent and customer subsets. with contrastive-1 submission, we obtain similar results. for agent subset, comet score improved by 0.001 whereas, decreased by 0.002 for customer subset. similarly for contrastive-2 submission, comet decreased by 0.003 whereas chrf and bleu score decreased by 0.001 and 0.006 respectively for agent subset. without context method generated better results for customer subset, improving bleu and chrf by 0.005 and 0.002 respectively, whereas we observe a decrease of 0.005 on comet metric. thus, our experiment suggests that the usage of context played very limited role in the submitted systems. we suggest this is due to a lower context window in our experimental setting. we use only one previous sentence as a context. while it has been observed that using one context is usually sufficient on conversational or document-level datasets, wmt22 chat task data contain very shorter and repetitive sentences. this includes one or two word utterances ( thanks, #email#, #name#, good bye, etc), app navigational information ( tap settings, tap device information, etc), etc. these utterances has very limited information to be useful as a context. further, appearance of duplicate utterances is a challenge during training process. however, unlike general mt, conversational datasets can not be de-duplicated easily. this is because removal of some utterance from a conversation will break its structure and might not be as meaningful. ## conclusion task translation is a challenging and important task for our society. one of the major challenges in chat translation is context-dependency. we participated in wmt22 chat translation task, where we submit results obtained from multi-encoder based transformer model. we obtain comet scores of 0.768 and 0.907 on english-to-german and german-to-english directions, respectively. we found that role of context in our experimental setting is limited. in future, we would like to explore these methods with larger window size. further, we would like to explore data de-duplication strategies for conversations.