“Hi, how can I help you?” Improving Machine Translation of Conversational Content in a Business Context (2022)

Abstract: This paper addresses the automatic translation of conversational content in a business context, for example support chat dialogues. While such use cases share characteristics with other informal machine translation scenarios, translation requirements with respect to technical and business-related expressions are high. To succeed in such scenarios, we experimented with curating dedicated training and test data, injecting noise to improve robustness, and applying sentence weighting schemes to carefully manage the influence of the different corpora. We show that our approach improves the performance of our models on conversational content for all 18 investigated language pairs while preserving translation quality on other domains - an indispensable requirement to integrate these developments into our MT engines at SAP.

## introduction at sap we build machine translation systems to cope with a huge translation volume, covering product localization and translation of documentation, training materials or support instructions for up to 85 languages. we usually train mixed-domain neural machine translation (mt) engines, whose training input consists of a multitude of data sources including the contents of the company-internal translation memories from various domains. the resulting mt systems produce high-quality technical translations but have difficulties generating appropriate output for conversational content, required for multilingual chatbots or product support chats. for an enhanced user experience, it becomes more and more important that our customers can communicate in the language of their choice. therefore, we started a project to improve translation quality for business-related conversations. this includes better translations, but also more robustness towards typos and poor input quality. we focus on methods that allow for improvements in one area without degrading quality in others, since we aim to deliver a single model per language pair. the remainder of this paper is structured as follows. section 2 introduces the particular challenges to be addressed when training an mt system for corporate conversational content. the methods we explored are presented in section 3 and experimental results on selected language pairs can be found in section 4. section 5 contains results on 18 language pairs of major interest at sap using the final experimental configuration. we point out open research questions in section 6. related problems and approaches are highlighted in section 7 before we summarize our findings in section 8. ## challenges our baseline mt system is built on large amounts of technical documents and ui-style training data with the consequence that it performs unsatisfactorily in business conversations, where the content is technical, but style is conversational. an example conversation is given in agent original conversation in english baseline mt improved mt user hello anyone bonjour à tous bonjour à tous user why my gdm is not working pourquoi mon gdm ne fonctionne pas pourquoi mon gdm ne marche pas ? user it says .service file is not there cela indique que le fichier .service n'existe pas. c ¸a dit que le fichier .service n'est pas là support what are you doing exactly que faites-vous exactement que faites-vous exactement ? support where is it exactly o ù c'est exactement o ù est-ce exactement ? support i didnt evn see the question.. i didnt voir la question... je n'ai même pas vu la question. user i have dsl connection my isp provides me and i got device not manaed so i edit.conf file of nm i a dsl connexion mon isp me fournit et i get device not manaed so i edit .conf file of nm j'ai une connexion dsl que mon isp me fournit et je n'ai pas géré l'appareil donc je modifie le fichier .conf de nm user yeah it is oui, oui, c'est c ¸a support id leave it alone. et laissez-le tranquille. je le laisse tranquille. user i thinking to filter that port i penser à filtrer ce port je pense filtrer ce port. support i never noticed it being open to the network. just to localhost i n'a jamais remarqué qu'elle était ouverte au réseau. il s'agit simplement d'un hôte local je n'ai jamais remarqué qu'il était ouvert au réseau. juste à localhost user ok ok ok support so you m ay be worried about nothing vous vous inquiétez donc de ne rien voir donc tu es toujours inquiet pour rien. support seems its not really an security issue and it makes loookups quicker semble qu'il ne s'agit pas vraiment d'un problème de sécurité et qu'il accélère les loookups n'est pas vraiment un problème de sécurité et accélère les recherches user thanks remerciements merci ## methods in this section, we describe the methods we investigated to address some of these challenges. ## experiments we now empirically evaluate the methods introduced in section 3, with the goal of improving mt quality on conversational content. we focus on conducting detailed experiments and presenting results for two language pairs per method, one being rather close languages, the other rather distant. these are english to french and japanese (en-fr, en-ja) for up-weighting and noise injection, and italian and japanese to english (it-en, ja-en) for back-translation. in section 5 we will demonstrate that our main findings generalize to other language pairs. ## from experiments to production the experimental results from section 4 motivated us to use the same data assembling techniques and configurations for other language pairs that had not been previously tested. for the translation directions with english source, the results of adding the back-translated udc data with error-sensitive weight factors for systems translating into english are shown in to illustrate the differences, we refer back to ## outlook although we see nice improvements, the translation quality in technical business conversations could be further improved. we point out the main open issues in this section, leaving them for future work and calling for new methods to address them. ## conclusion we have shown that an mt model specialized in the it and business domains can be enhanced to also cover conversational content well. this balancing act is highly relevant in scenarios such as product support chats or multilingual chatbots. we have achieved that by curating high-quality parallel data to address phenomena where the model exhibited the most devastating shortcomings. we further add back-translated data from the dialogue domain, inject typos, punctuation and capitalization variants to make the model more robust, and carefully manage the influence of the different corpora using a sentence weighting scheme. we have demonstrated that promising results from experiments involving only a few language pairs generalize well to the main languages in our production scenario at sap, achieving an improvement of 2.4 chrf2 / 3.1 bleu on average for language pairs from english and 1.2 chrf2 / 1.5 bleu for language pairs to english on our conversational test sets, while the performance on other domains and test sets remains stable.