Translation Quality Estimation for I ndian Languages (2018)

Abstract: Translation Quality Estimation (QE) aims to estimate the quality of an automated machine translation (MT) output without any human intervention or reference translation. With the increasing use of MT systems in various cross-lingual applications, the need and applicability of QE systems is increasing. We study existing approaches and propose multiple neural network approaches for sentence-level QE, with a focus on MT outputs in Indian languages. For this, we also introduce five new datasets for four language pairs: two for English–Gujarati, and one each for English–Hindi, English–Telugu and English–Bengali, which includes one manually post-edited dataset for English– Gujarati. These Indian languages are spoken by around 689M speakers world-wide. We compare results obtained using our proposed models with multiple state-of-the-art systems including the winning system in the WMT17 shared task on QE and show that our proposed neural model which combines the discriminative power of carefully chosen features with Siamese Convolutional Neural Networks (CNNs) works best for all Indian language datasets.

## introduction in recent years, machine translation (mt) systems have seen significant improvements. however, the quality of the output obtained from these c ⃝ 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. * the author is also a principal applied scientist at microsoft. mt systems is neither perfect nor consistent across multiple test cases. the task of translation quality estimation (qe) aims to estimate the quality of an mt output without any reference translation. qe is now critically important with the increasing deployment of mt systems in practical environments. qe has been shown to be extremely useful and is widely used in computer aided translation (cat) environments @xcite . qe can also be useful in various applications and systems such as crosslingual summarization, cross-lingual information retrieval, etc., which rely on high quality translations. with the help of qe, such systems can automatically pick the best translation out of several proposed translations by multiple mt systems. if the estimated quality is still unsatisfactory the system can alert the user about the poor quality, or fall-back to some alternate way to find a better translation. word, phrase, sentence or document level qe has been studied extensively by various researchers. wmt12-17 (the 7 th to 10 th workshops on statistical machine translation and the 1 st and 2 nd conferences on machine translation) held a shared task on qe @xcite @xcite @xcite . the shared task has explored qe on several datasets and settings for english-spanish and english-german language pairs over years. little work has been done to study qe for indian languages. in this work, we focus on four indian languages: telugu, hindi, gujarati and bengali. according to a 2007 estimate 1 , there are 366 million hindi speakers (across five countries), 207 million bengali speakers (across four countries), 69.7 million telugu speakers (across four countries), and 46.1 million gujarati speakers (across eight countries) worldwide, denoting the importance of our choice of these datasets. while english is a west germanic language that originated from anglo-frisian dialects, hindi, bengali and gujarati are indo-aryan languages foot_2 , and telugu is a dravidian language 3 . indian languages are relatively free word order languages and morphologically richer when compared to english. additionally, some indian languages, for example telugu, are highly agglutinative. in comparison with english, hindi has approximately twice as many vowels and consonants. although hindi has tenses similar to those used in english, there is a lack of correspondence in their use to express various meanings. gender and status relations between speakers causes morphological changes in hindi words, unlike english. compared to english, bengali uses onomatopoeia extensively, and so one has to convey that through particular adjectives and adverbs. besides these differences, there are some phrases, idioms and compound words in english which do not have equivalents in indian languages due to significant cultural differences. because of the differences in the characteristics of the languages involved, existing methods for qe may or may not be effective for all language pairs. we experiment with multiple datasets in different language pairs, each involving english and an indian language, to study the effectiveness of various models on these datasets. in addition to the different characteristics of indian languages, many of these languages are resource-scarce, from a computational linguistics perspective. linguistic resources like dependency parsers or semantic role labelers are not available for most languages we use in this paper. additionally, large amount of manually annotated data, such as parallel corpora are also difficult and costly to obtain. hence, in this work, we try to minimize dependency on external large datasets, especially ones which require manual annotation. we hope that the qe accuracy can be further improved using such extra information, and plan to explore it as future work. to study qe for indian languages, we also introduce five datasets, for four different language pairs. one dataset, news.gu, described in section 3.2 has been prepared by manually postediting mt outputs. the other four datasets, described in section 3.3 make use of existing parallel corpora to create datasets for qe. all datasets are prepared using neural machine translation (nmt) api provided by google translate foot_4 . to the best of our knowledge, we are the first to study qe when using the nmt system. in this paper, we evaluate the effectiveness of various state-of-the-art systems (proposed for other language pairs) including the winning system of the wmt17 shared task on various indian language datasets. we also propose and evaluate multiple neural network models for qe. finally we show that one of our proposed models cnn.combined, described in section 4.2.2, gives best results on most indian language datasets. our major contributions through this paper are as follows. the rest of the paper is organized as follows. we describe related work in section 2. section 3 describes the datasets used for the experiments. section 4 describes different methods and proposed models used for our experiments. section 5 contains a few notes about the experimental settings. section 6 provides analysis and related discussions. finally, we conclude with a brief summary in section 7. ## related work related previous work on translation quality estimation can be organized into two broad kinds of approaches: manual feature engineering based approaches, and neural networks based approaches. wmt12-17 shared task on qe @xcite @xcite @xcite has recorded the overview and progress of the field over years. ## datasets we used six different datasets for five different language pairs for our experiments. source language is english for all the datasets. all datasets are split into the typical train, development and test sets. ## models for translation quality estimation this section describes various models used for experiments and evaluation. we first discuss the baseline models in section 4.1 and then the proposed models in section 4.2. ## experimental settings the code used for experiments has been made publicly available at https://goo.gl/gg9j6f . svr.baseline model is trained using scikit-learn library @xcite . keras @xcite , with theano (theano development team, 2016) is used to implement all the neural network models, including the baselines. development set was used for parameter tuning for svr.baseline for each dataset. for neural models, development data was used as validation data while training models, to early stop the training to prevent overfitting. gru cells @xcite , with 500 hidden units, are used in rnns in all the neural network models. sentences are clipped to length of 100 words and padded with masking. vocabulary size is limited to 40,000 words for all the experiments. word embedding size is set to 300. for all proposed cnn based models, 200 filters of sizes 3, 4 and 5 each were used in the sentence encoders. sentence vector size was set to 500. ## evaluation and results two types of evaluation are performed for all experiments: 1) using pearson's correlation coefficient between the predicted quality scores and the actual quality scores, to evaluate scoring. 2) using spearman's correlation coefficient to evaluate the ranking of sentences according to quality. we also report statistical significance of the results considering postech.two-step as baseline, over ten different runs. we find that postech.two-step model works best for wmt17 en-de dataset for both the tasks, but fails to give best results for any other dataset, in the low-resource settings explored in this paper. we also find that the proposed cnn-based models generally work better for indian language datasets. the better performance of cnn-based models over rnn-based models for indian languages might be because of the free word order property of indian languages. cnn does not directly rely on entire sequence and order of words, rather it picks best phrases depending on filter sizes from the sentence without explicitly looking at the order. our final model cnn.combined, with or without the use of fasttext embeddings works best for four out of five indian language datasets for the scoring task. for news.gu dataset, our combined cnn model, without engineered features, cnn.combined.no-features+fasttext, gives the best results. on investigating the relatively low results of the two variants of cnn.combined model on news.gu, we found that due to some engineered features and the relatively small size of train set, the combined cnn model with features was rapidly overfitting. the similar model without engineered features, cnn.combined.nofeatures works as expected and yields the best results on news.gu. similarly, for the ranking task, the two variants of cnn.combined model outperform all the models for four out five datasets. for the remaining indian language dataset, news.gu, cnn.siamese+fasttext model yields the best result. we also notice that using fasttext embeddings in cnn based models generally works better comthe rnn based models work comparably or better for all indian language datasets, but are much simpler and have much lower number of trainable parameters compared to postech models. however, the difference between the two rnn based models, rnn and rnn.summary-attention, across datasets is inconclusive. in ## conclusions in this paper, we study the effectiveness of different neural network architectures for qe for indian languages. we also introduce multiple datasets for the task, which can be used as benchmark for future work in the area. we observe that our proposed cnn.combined model beats the state-ofthe-art methods by a significant margin.