Product Review Translation: Parallel Corpus Creation and Robustness towards User-generated Noisy Text (2021)

Abstract: Reviews written by the users for a particular product or service play an influencing role for the customers to make an informative decision. Although online e-commerce portals have immensely impacted our lives, available contents predominantly are in English language- often limiting its widespread usage. There is an exponential growth in the number of e-commerce users who are not proficient in English. Hence, there is a necessity to make these services available in non-English languages, especially in a multilingual country like India. This can be achieved by an in-domain robust machine translation (MT) system. However, the reviews written by the users pose unique challenges to MT, such as misspelled words, ungrammatical constructions, presence of colloquial terms, lack of resources such as in-domain parallel corpus etc. We address the above challenges by presenting an English–Hindi review domain parallel corpus. We train an English–to–Hindi neural machine translation (NMT) system to translate the product reviews available on e-commerce websites. By training the Transformer based NMT model over the generated data, we achieve a score of 33.26 BLEU points for English–to–Hindi translation. In order to make our NMT model robust enough to handle the noisy tokens in the reviews, we integrate a character based language model to generate word vectors and map the noisy tokens with their correct forms. Experiments on four language pairs, viz. English-Hindi, English-German, English-French, and English-Czech show the BLUE scores of 35.09, 28.91, 34.68 and 14.52 which are the improvements of 1.61, 1.05, 1.63 and 1.94, respectively, over the baseline.

## introduction in the era of exponentially rising internet users, contents over social media, e-commerce portals are increasing rapidly. in recent times, there has been a phenomenal growth in the number of e-commerce users, especially during this covid pandemic situation. however, the contents in such e-commerce portals are mostly in english, limiting the scope of these services to only a section of the society who can read and/or write in english. india is a multilingual country with 22 officially spoken languages. the number of internet users in india has increased dramatically in the last few years with the widespread usage of low-cost android phones. users find it very difficult to understand the english contents written in these service portals. hence, there is a great demand to translate these contents from english to indian languages. as the manual translation is both time-consuming and cost-sensitive, building an automated machine translation (mt) system that could translate these enormous amounts of reviews will be of great interest. however, there are several challenges for this, such as the non-availability of in-domain parallel training corpus, noisy nature of the text, ungrammatical constructions, and the mixing of more than one language (i.e. code-mixed contents) (ref . in our current work, we take up english-to-hindi translation as there are 57.09% hindi to handle the noisy tokens as source input, we integrate a similarity based token replacement model before word segmentation at inference time where the word vectors of noisy input tokens are matched with the correct and seen tokens in the source vocabulary, and replaced with the highest similar token. we use a character based language model to generate the word vectors and map the noisy and correct version of tokens in vector space. the generation of the word vectors depends on the characters present in that word. the remainder of the paper is organized as follows. in section 2, we discuss the related work. section 5 presents the approaches of training the character language model, word vector generation model, and handling of noisy source input tokens at inference time. section 6 presents the details regarding the dataset used and the experimental setup. results and analysis of our approach are discussed in section 7. finally, section 8 concludes the work with future research directions. ## related work machine translation with noisy text is, itself, a very challenging task. noisy tokens (misspelled words) pose great challenges to develop the neural machine translation (nmt) models (c.f. sr. english sentence (crawled) hindi sentence (corrected) 1. rounded corners make griping the phone very well . राउं डे ड कोनर् र फोन को बहु त ह अच्छ पकड़ दे ते हैं । (raunded kornar phon ko bahut hee achchhee pakad dete hain) 2. one of the best phone ever at this price . इस क मत में अब तक के सबसे अच्छे फोन में से एक । (is keemat mein ab tak ke sabase achchhe phon mein se ek .) 3. but this is apple and apple is like that only ले कन यह ऐप्पल है और ऐप्पल ऐसा ह होता है (lekin yah aippal hai aur aippal aisa hee hota hai) 4. at first i want to say thank u flipkart. सबसे पहले मैं थैं क यू िफ्लपकाटर् कहना चाहता हू ं । (sabase pahale main thaink yoo phlipakaart kahana chaahata hoon.) ## training nmt over human corrected corpus we train an english-to-hindi baseline model using the human corrected corpus as mentioned in after training over the human corrected cor-13 https://www.iitp.ac.in/~ai-nlp-ml/ resources/data/review-corpus.zip pus, we perform testing over the review domain test set and achieves 33.26 bleu points and name it as 'base'-the baseline model. in addition to it, we also add the back-translated synthetic corpus into human corrected corpus, and train the nmt model over it. we call it as the 'base+bt' model that yields 37.79 bleu points. ## handling noisy tokens in this section, we describe the methodology used in our work. figure 2 presents the overall process of our proposed method. it consists of various steps like character language model (lm) training, word vector (embedding) generation, and finally noisy token replacement at inference time. section 5.1 and section 5.2 describe the steps in details. ## dataset and experimental setup in this section, we present the details of the datasets used in our experiments and the various setups. ## result and analysis we evaluate the models using bleu, and these results are shown in also perform experiments using the word segmentation approaches, viz. bpe @xcite and subword regularization @xcite . for english-to-hindi review domain translation, proposed method yields 35.09 bleu points which significantly outperforms synthetic noise, sr and bpe with a difference of 0.82, 1.61 and 1.83 bleu points, respectively. we also perform experiments for en-to-hi translation using benchmark new-stest2014 as the test set. we achieve 0.96 and 1.29 bleu improvements over subword regularization (sr) @xcite and bytepair-encoding bpe @xcite , respectively. we also evaluate the performance for en-to-de translation and achieve 1.05 and 1.07 bleu improvements over sr and bpe, respectively. for en-to-cs, we use the iwslt17 testset, and the evaluation yields 1.94 and 2.48 bleu improvements over sr and bpe, respectively. for en-to-fr, we evaluate over two datasets, iwslt17 and mtnt18. the mtnt is a noisy testset for en-fr translation. for the mtnt testset, our model yields 1.55 and 2.18 bleu improvements over sr and bpe, respectively. for iwslt testset, en-to-fr translation using our approach achieves the 1.63 and 1.57 bleu improvements over sr and bpe, respectively. we also perform experiments by adding synthetic noise in the training corpus @xcite which is a noise handling technique. for en-to-hi, en-to-de, en-to-fr and en-to-cs, our proposed method achieves 0.96, 1.05, 1.63 and 1.94 bleu improvement, respectively, over the synthetic noise model @xcite . we perform statistical significance tests 20 (koehn, 2004), and found that the proposed model attains significant performance gain with 95% confidence level (with @xmath0.013 which @xmath1.05). for en-to-fr over mtnt18 testset, we achieve only 0.14 bleu improvement over the synthetic noise model @xcite which is not a en-to-hi (newstest2014) 20% 25% 30% 40% subword regularization (sr) 14.37 13.68 10.31 9.81 proposed 14.84 14.54 12.86 11.27 en-to-de (newstest2014) subword regularization (sr) 27.24 26.18 24.83 23.48 proposed 28.53 27.34 26.08 25.22 en-to-fr (iwslt17) -subword regularization (sr) 33.14 32.26 29.24 28.37 proposed 34.45 33.29 31.07 30.35 ## conclusion in this paper, we have developed a robust nmt model for product review translation that can handle noisy input text. because of the absence of an in-domain parallel corpus, we introduce a parallel english-hindi corpus for product review domain. we crawl the user reviews of electronic gadgets from e-commerce sites written into english language. these are pre-processed; passed through an in-house judicial domain nmt system; and a part of this dataset is post-edited by the language experts. it is also observed that product reviews which are user generated content contain noisy tokens which are a challenge to handle in any mt system. due to the limitation of fixed vocabulary size at training time, it is not possible for the nmt models to see all the noisy variants of input tokens. we have integrated a token replacement approach during the inference time. we trained a character based language model which generates the vector representation of the tokens present in the source vocabulary based on the characters present in that word. the token replacement approach finds the most similar token from the source vocabulary for each noisy input token at inference time to replace it with the correct token. we perform experiments over a variety of language pairs, such as en-to-hi, en-to-de, en-to-fr and en-to-cs. and using the proposed approach, we achieve 35.09, 28.91, 34.68 and 14.52 bleu points respectively. we also observe the behaviour of the proposed method by varying the % (20, 25, 30 and 40%) of noisy tokens at the input side. the proposed method significantly outperforms the baseline in the presence of different quantities of noisy tokens. human evaluation shows that our model achieves good fluency and adequacy levels.