On Machine Translation of User Reviews (2021)

Abstract: This work investigates neural machine translation (NMT) systems for translating English user reviews into Croatian and Serbian, two similar morphologically complex languages. Two types of reviews are used for testing the systems: IMDb movie reviews and Amazon product reviews. Two types of training data are explored: large out-of-domain bilingual parallel corpora, as well as small synthetic in-domain parallel corpus obtained by machine translation of monolingual English Amazon reviews into the target languages. Both automatic scores and human evaluation show that using the synthetic in-domain corpus together with a selected sub-set of out-of-domain data is the best option. Separated results on IMDb and Amazon reviews indicate that MT systems perform differently on different review types so that user reviews generally should not be considered as a homogeneous genre. Nevertheless, more detailed research on larger amount of different reviews covering different domains/topics is needed to fully understand these differences.

## introduction machine translation (mt) has evolved very rapidly since the emergence of neural approaches in 2015, and it is being used for different genres and domains. every year, evaluation campaigns which include both human and automatic evaluation are carried out with the goal of advancing the state of the art. the most well-known is the wmt shared task foot_0 which focuses on news articles and (since 2016) on biomedical texts, and both can be considered as instances of "formal written text". the iwslt evaluation campaign foot_1 , on the other hand, focuses on the translation of ted talks, and some european projects @xcite investigated the translation of online lectures. in both cases, the text can be considered to be "formal speech", with the challenges of dealing with characteristics of spoken language and speech recognition output. recently, interest in the translation of usergenerated content in the form of "informal written text" has been increasing. for example, jsalt 2019 workshop foot_2 focused on translation of very noisy text content originating from sources like whatsapp, twitter and reddit. in this work, we focus on a different type of written user-generated content, namely user reviews. while the style is not as colloquial and noisy as that of twitter or of other similar sources, it certainly is much less formal than news texts or other sources that have been investigated traditionally in the mt community. there are also important applications for focusing on this kind of data, both from commercial and from user perspective. more and more companies are expanding into multinational markets, and user reviews of products have become an important asset for online transactions and a feature that many customers expect to find. and in the era of always-available internet connectivity, many individuals rely on experiences of other people not only for guiding purchasing decisions, but also for entertainment options like choosing movies, books, restaurants, etc. in this work, we focus on both kinds of user reviews, namely product reviews from amazon and movie reviews from imdb. translating user reviews can increase and improve its reach and utility. the main issue for human translation is the fact that there is way too much content to be translated. therefore, mt is very helpful for this kind of content. however, the genre introduces several important challenges, such as informal language, spelling errors, a large number of domains/topics, and lack of in-domain parallel (bilingual) data. in this work, we compare two approaches for building mt systems for translating user reviews: training on large parallel out-of-domain data and training on small synthetic in-domain data. we also compare mt performance on two types of user reviews: imdb movies and amazon products. we investigate croatian and serbian as target languages, as a case involving mid-size less-resourced morphologically rich european languages. for these languages, a reasonable amount of out-ofdomain parallel data is publicly available to train an nmt system, however still much lower than for "major" european languages (such as german, french, spanish). all our experiments were carried out on publicly available data sets. we used opus foot_3 parallel data for out-of-domain training and a selected set of amazon reviews foot_4 for in-domain training. for development, we used the publicly available texts foot_5 consisting of a selected set of english imdb reviews foot_6 and their croatian and serbian human translations. for testing, we used another selected set of imdb reviews as well as a selected set of amazon reviews. neither of the test reviews has been investigated yet, and they will also be made publicly available. ## building nmt systems all our systems are based on the transformer architecture @xcite and built using the sockeye implementation @xcite . previous work on the given two target languages @xcite reported that multilingual sys-tem which translates into both languages performs better than two separated bilingual systems. therefore, all our systems are multilingual, built using the same technique as @xcite , namely adding a target language label "sr" or "hr" to each source sentence. the amount of croatian and serbian data is balanced in all set-ups in order to achieve optimal performance for both target languages. the systems operate on sub-word units generated by byte-pair encoding (bpe) @xcite with 32000 bpe merge operations both for the source and for the target language texts. we do not use shared vocabularies between the source and the target languages because they are distinct. on the other hand, we built a joint vocabulary for the two target languages because they are very similar. all the systems have transformer architecture with 6 layers for both the encoder and decoder, model size of 512, feed forward size of 2048, and 8 attention heads. for training, we use adam optimiser (kingma and ba, 2015), initial learning rate of 0.0002, and batch size of 4096 (sub)words. validation perplexity is calculated after every 4000 batches (at so-called "checkpoints"), and if this perplexity does not improve after 20 checkpoints, the training stops. in our set-up, knowledge distillation is used for domain adaptation: the teacher model is the system trained on a large amount of out-of-domain parallel data. this system is used to create a small synthetic in-domain corpus, which is then used to train the student model. the details about all data sets and data selection are presented in the next section. ## experimental set-up in order to systematically explore influence of different sizes and natures of training data, we built the following mt systems: ## summary and outlook this work investigates machine translation of two types of user reviews, imdb movie reviews and amazon product reviews, from english into serbian and croatian. since one of the main challenges for mt of user reviews is lack of parallel in-domain train-ing data, we explored a possibility to make use of large out-of-domain bilingual parallel corpora as well as monolingual in-domain english corpora. we trained a general "teacher" system on all outof-domain data and then used this system to create a small synthetic in-domain parallel corpus by translating english amazon reviews into the target languages. both automatic scores and human evaluation show that using this synthetic in-domain corpus together with a selected sub-set of out-ofdomain data is the best option. the results on separated imdb and amazon reviews indicate that mt systems perform differently on different review types so that user reviews generally should not be considered as a homogeneous genre. however, evaluating and training on larger amount of different reviews covering different domains/topics is needed to identify the nature of differences between different types of reviews, and also influence of different topics. another direction of future work should include using more in-domain data, as well as other techniques for domain adaptation.