Low Resource Chat Translation: A Benchmark for H indi– E nglish Language Pair (2022)

Abstract: Chatbots or conversational systems are used in various sectors such as banking, healthcare, e-commerce, customer support, etc. These chatbots are mainly available for resource-rich languages like English, often limiting their widespread usage to multilingual users. Therefore, making these services or agents available in non-English languages has become essential for their broader applicability. Machine Translation (MT) could be an effective way to develop multilingual chatbots. Further, to help users be confident about a product, feedback and recommendation from the end-user community are essential. However, these question-answers (QnA) can be in a different language than the users. The use of MT systems can reduce these issues to a large extent. In this paper, we provide a benchmark setup for Chat and QnA translation for English-Hindi, a relatively low-resource language pair. We first create the English-Hindi parallel corpus comprising of synthetic and gold standard parallel sentences. Thereafter, we develop several sentence-level and context-level neural machine translation (NMT) models, and measure their effectiveness on the newly created datasets. We achieve a BLEU score of 58.7 and 62.6 on the English-Hindi and Hindi-English subset of the gold-standard version of the WMT20 Chat dataset. Further, we achieve BLEU scores of 52.9 and 76.9 on the gold-standard Multi-modal Dialogue Dataset (MMD) English-Hindi and Hindi-English datasets. For QnA, we achieve a BLEU score of 49.9. Further, we achieve BLEU scores of 50.3 and 50.4 on question and answers subsets, respectively. We also perform thorough qualitative analysis of the outputs by the real users.

## introduction chatbots or conversational systems serve a crucial role in various sectors such as banking, healthcare, e-commerce, customer support, etc. while chatbots are convenient and fast, most are available only for resource-rich languages like english, limiting their widespread usage to users from languages other than the chatbot's language. it is essential to make these services available in non-english languages for their broader applicability. machine translation (mt) can be an effective technology for developing multilingual chatbots to meet the need of multilingual societies. in recent years, there has been significant progress in neural machine translation (nmt) with applications in a variety of domains, such as document @xcite , biomedical @xcite , news @xcite etc. however, nmt requires a huge amount of data @xcite , and its manual creation requires significantly a lot of human effort. chat translation poses more challenges than sentencelevel translation due to the following: (i). translation of the current utterance may depend on the contextual sentences. to translate the current sentence properly, we may need to refer to the previous sentences in the chat to grasp the meaning and generate appropriate translation; (ii). chat often contains informal sentences, urban slang, stretched words (e.g., niceeee), code-mixed phrases, etc. further, communicating with chatbots or customer service may not be enough to make an informed decision regarding a product. queries regarding a product can effectively be answered by the community of customers who use a similar product, leading to the requirement of a qna system. hence, it is also vital to perform qna translation. this paper provides a benchmark setup for chat and qna translation for hindi-english language pair, which has very little to no resources for such tasks. firstly, we create a synthetic and gold-standard parallel corpus for english-hindi chat and qna translation. we use the existing mt systems described in section 5 to generate synthetic translation. with this model, we get a bleu score of 47.8, indicating the translation is of good quality. further, we create gold-standard parallel corpus by professional translators. we employ the transfer-learning technique by initializing the transformer architecture using the model trained on similar domain data @xcite . further, we report the results on using same-speaker context to generate context-aware translation, which was useful for better translations for english-german @xcite . for qna, we combine the qna pair to generate consistent translation. we used to question and answer tags during training to help the model learn qna-specific properties. further, we compare the results with models trained on only question corpus or only answer corpus. we report bleu @xcite and ter @xcite scores. to our knowledge, there is no publicly available dialogue dataset for the english-hindi chat or qna translation. we introduce two datasets containing a total of 68.7k synthetic sentences, as well as 3,037 sentences of goldstandard data for chat translation. further, we provide about 2.1 million qna pairs (4.2 million sentences), their corresponding translation (synthetic), and 1,000 gold-standard sentences for each validation and test set, respectively. we make all the data and codes publicly available 1 . to summarize, our work has the following attributes: (i). introduce three english-hindi parallel corpora for chat translation in service and e-commerce domains; (ii). use several context-aware and context-agnostic methods and observe their effectiveness on chat translation in (extremely) low-resource language settings, especially for the task at hand; @xcite . propose a method to generate consistent translation performance in question-answer settings, where question-answers can be treated as a short chat with only two utterances; (iv). mt outputs have been quality checked by the actual users recruited by the well-known e-commerce company. ## related work there are several approaches to applying chatbots in e-commerce. @xcite proposed a system that asks aspect-based questions to the user in a sequence, and recommendations are provided when the system is confident. @xcite proposed a personalized recommendation model which considers past ratings of products rated by the user and queries in the current conversation session to generate product recommendations. @xcite integrated the recommender system and dialog system, where the dialog system enhanced the recommendation system by introducing knowledge-grounded information about users' preferences. further, the recommender system improved the dialog generation system by providing recommendation-aware vocabulary bias. @xcite showed that a simple transfer learning technique from an existing large-scale community question answering helped to generate 10% more accurate answers. @xcite proposed positional history answer embedding method to encode conversation history with position information. further, they proposed a method that attends to conversational utterances with different weights based on their helpfulness in answering the current question. @xcite generated opinion-aware answers by jointly learning answer generation and opinion mining with a unified model. however, it is to be noted that all of the systems are monolingual. despite its demand, the field of dialogue translation remains mostly unexplored due to the lack of publicly available chat corpus. @xcite introduced an english-german parallel conversational corpus. @xcite adopted several methods including replacement of rare characters with a special '<@xmath0, inline casing, tagged back-translation (bt) @xcite , byte-pair-encoding (bpe) @xcite , dropout @xcite , tagged synthetic noise, and ensemble of models using domain-specific adaptive layers, etc. while the largest single contributor to the improvement of translation quality was fine-tuning, the ensemble method with a domain-specific adaptor layer generated the best translation on wmt20 chat data. @xcite used the pre-trained models @xcite and finetuned them on the pseudo-in-domain and in-domain data. @xcite adapted cross-lingual language model pre-training @xcite objectives into document-level nmt by using three previous contexts along with the current sentence. @xcite used the transformer architecture, modified with an additional encoder to process one previous context. additional encoder failed to generate better overall translation in terms of bleu score. @xcite proposed a rule-based context selection technique where previous sentences by the same user are used to enhance the translation quality. @xcite introduced an english-chinese dialogue dataset named bmeld, which is an automatically translated and manually post-edited version of the meld dataset @xcite . further, they introduced a conditional variational auto-encoder (cvae) model that captures role preference, dialogue coherence, and translation consistency. @xcite proposed a multi-tasking system performing monolingual response generation, cross-lingual response generation, subsequent utterance discrimination, and speaker identification along with nmt. here, the context-aware multi-tasking methods could generate better translation than context-agnostic models. @xcite extended the same by introducing an additional objective, cross-lingual subsequent utterance discrimination, and evaluated the models with bmeld (english-chinese) and bcontrast (english-german) datasets. @xcite proposed a multi-task learning-based nmt system to identify missing pronouns and typos and utilize context to translate dialogue utterances for english-chinese language pairs. liang et al. (2022a) observed visual features helps to generate better quality translation on multi-modal dialogue. our task is different from others in the following aspects: a). we focus on dialogue or chat translation in low-resource language settings and the e-commerce domain; b). the existing works primarily focus on dialogues or formal conversations. in contrast, we focus on noisy and informal conversations or chat (specifically for qna); c). we perform experiments on the sentence-level system (transformer) with domain adaptation and transfer learning. we also report the evaluation results on a context-based system exploiting source-side context. ## corpus creation we create the english-hindi parallel corpus from the existing english dialogue corpora. specifically, we choose multimodal dialogue (mmd) corpus @xcite and english part of german-english corpus from wmt20 chat translation task @xcite , which is based on taskmaster-1 corpus @xcite . further, we translate the flipkart qna corpus, consisting of 2.1m qna pairs. this dataset contains the queries about a product asked by users of the website, which have been answered by either of (a). other customers or (b). seller of the product. the qna covers queries of a large range of products, including electronics, clothing, appliances, etc., on the e-commerce website, while the aspect is about the quality of the product, its features, compatibility, durability, and others. we create two types of parallel corpora, synthetic corpus and gold standard corpus. the synthetic corpus is created through forward-translation or backward-translation (depending upon the translation direction of the task) of english corpora into hindi with the existing nmt models. the gold standard corpus is created by manually translating english into hindi. ## description of corpora this section describes the wmt20 chat, mmd, and qna corpus. ## methodology and experimental setup we use samanantar @xcite corpus as a general domain corpus that contains 10m sentence pairs for english-hindi. we train the transformer @xcite model on the samanantar corpus and consider it a baseline model. ## conclusion dialogue translation is different from sentence translation due to several additional challenges. the particular field could not be adequately explored in indian languages due to a lack of datasets. multilingual chatbots are in high demand as many of the world's population are not fluent in english and other major languages. we introduce two english-hindi parallel datasets for dialogue translation and one large-scale english-hindi parallel corpus for qna translation. the datasets contain various domains, including fashion, making reservations, ordering foods, e-commerce products, etc., and contain sentences from different roles and languages. demands for online chatbot services involving the domains mentioned above are tremendous. the multilingual property of the datasets will be beneficial in building multilingual chatbots and qna translators. we report a baseline result on some sentence-level and a contextlevel model exploiting context. in the future, we would like to introduce better ways to utilize context and use other chat-specific characteristics, including informality, codemixing, etc. further, we would like to extend the work into other indian languages, including tamil, telugu, malayalam, etc.