N o D ee L e: A Novel Deep Learning Schema for Evaluating Neural Machine Translation Systems (2021)

Abstract: Due to the wide-spread development of Machine Translation (MT) systems –especially Neural Machine Translation (NMT) systems– MT evaluation, both automatic and human, has become more and more important as it helps us establish how MT systems perform. Yet, automatic evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU, METEOR and ROUGE) may correlate poorly with human judgments. This paper seeks to put to the test an evaluation model based on a novel deep learning schema (NoDeeLe) used to compare two NMT systems on four different text genres, i.e. medical, legal, marketing and literary in the English-Greek language pair. The model utilizes information from the source segments, the MT outputs and the reference translation, as well as the automatic metrics BLEU, METEOR and WER. The proposed schema achieves a strong correlation with human judgment (78% average accuracy for the four texts with the highest accuracy, i.e. 85%, observed in the case of the marketing text), while it outperforms classic machine learning algorithms and automatic metrics.

## introduction recently, studies in natural language proccessing (nlp) have been using neural networks @xcite . neural networks have made significant progress in several nlp tasks including mt @xcite , summarization @xcite , dialogue generation @xcite and image captioning @xcite . the evaluation of mt systems is a crucial field of research, as has been highlighted by a number of researchers @xcite @xcite , given that it is used to compare different systems but also to identify a system's weaknesses and help improve it. various methods have been suggested for the evaluation of mt-both automatic and human @xcite . although, human evaluation is considered to be the best indicator of a system's quality, it is an expensive and time-consuming process, so it cannot be readily used for the development of the mt system. as a result, mt researchers and developers mostly use automatic evaluation metrics which constitute an acceptable estimation quality and they are easy and cheap to compute. some of them rely on score-based metrics, such as bilingual evaluation understudy (bleu) @xcite , national institute of standards and technology (nist) @xcite and word error rate (wer) @xcite , metrics using external resources, like meteor @xcite , and neural metrics such as reval @xcite and regressor using sentence embeddings (ruse) @xcite , while some others use machine learning schemata @xcite @xcite . automatic evaluation methods must be evaluated with specific criteria. according to banerjee and lavie @xcite , a satisfactory automated evaluation system should meet the following conditions: high correlation with human judgments quantified in relation to translation quality, sensitivity to nuances in quality among systems or outputs of the same system in different stages of its development, result consistency, reliability, a great range of fields and speed and usability. the most important condition is considered to be correlation with human judgment @xcite . yet, the automatic evaluation metrics mentioned above have lagged behind, as they do not correlate well with human judgments @xcite . this paper seeks to put to the test an evaluation model based on a novel deep learning schema developed by mouratidis et al. @xcite used to compare two nmt systems on four different text genres, i.e. medical, legal, marketing and literary in the english-greek language pair. the model, nodeele, utilizes information from the source segments, the mt outputs and the reference translation, as well as the automatic metrics bleu, meteor and wer. ## related work deep learning (dl) is one of the fastest-growing fields of information technology (it) today being used among others for mt evaluation. duh @xcite decomposes rankings into parallel decisions, with the best translation for each candidate pair predicted, using a ranking-specific feature set, bleu score and the support vector machine (svm) classifier. a similar pairwise approach was proposed by mouratidis and kermanidis @xcite , using a random forest (rf) classifier. cho et al. @xcite proposed a score-based schema to learn the translation probability of a source phrase to a target phrase (mt output) with a recurrent neural network (rnn) encoder-decoder. they showed that this learning schema has improved the translation performance. the schema proposed by sutskever et al. @xcite is similar to the work by cho et al. @xcite , but sutskever et al. chose the top 1000 best candidate translations produced by a statistical machine translation (smt) system with a long short-term memory (lstm) sequence-to-sequence model. wu et al. @xcite also trained a deep lstm network to optimize bleu scores focusing on german-english and german-french language pairs, but they found that the improvement in bleu scores did not reflect the human evaluation of translation quality. mouratidis et al. @xcite used lstm layers in a learning framework for evaluating pairwise mt outputs using vector representations, in order to show that the linguistic features of the source text (st) can affect mt evaluation. gehring et al. @xcite proposed an architecture for sequence to sequence modeling based on a convolutional neural network (cnn). the model is equipped with linear units @xcite and residual connections @xcite . ## results according to the annotators, and as it emerges from figure 2 , deepl performed better than google translate for all texts. it should be noted that an almost perfect agreement between the annotators for all four texts (κ=0.83 for t1, κ=1.0 for t2, κ=0.92 for t3 and κ=0.85 for t4 ) was observed. in the few cases of disagreement, the mediating annotator's decision was used. unequal values between the classes were observed with the class belonging to google translate being the minority class. the smote supervised filter @xcite was applied to the minority class. figure 3 presents the classification results (classification accuracy) for the two mt outputs over the four different datasets. it emerges that the classification accuracy level is higher in the case of the marketing text followed by the legal and the literary text. the lowest accuracy level is observed in the case of the medical text, most probably due to its rich and highly-specialised terminology. we also applied a smote filter with a view to improving the model accuracy. indeed, an increase of 2% of the classification accuracy for the medical text, 5% for the legal and the marketing text, and 3% for the literary text is observed. the above accuracy results are in accordance with the annotators' results (see figure 2 ). better accuracy results (f1 score) are observed for deepl (s1 ) compared to google translate (s2 ) for all texts (figure 4 ). the bleu and meteor scores for the mt outputs of the four texts are given in figure 5 . in particular, the medical text received the highest score for both metrics followed by the legal text, while the literary text received the lowest score. interestingly, the scores are not in line with the results of nodeele, i.e. the proposed deep learning schema, according to which the medical text received the worst classification accuracy and the marketing text received the best classification accuracy. in addition, although deepl (s1 ) performed better in all cases according to nodeele, google translate (s2 ) performed better in the case of the literary and marketing text according to bleu, and in the case of the medical text according to meteor. as far as the legal text is concerned, no difference was observed between the automatic metrics and nodeele. apart from bleu and meteor, nodeele was also compared to other methods. for that reason, additional experiments were carried out using different classifiers e.g. svm and rf using the weka framework as backend. the svm and the rf classifiers were trained on the same data and feature set as nodelee. as depicted in figure 6 , nodeele achieves stronger correlation with the human judgments (78% average accuracy for the four texts), compared to the rf classifier (74% average accuracy for the four texts) and the svm classifier (67% average accuracy for the four texts). unlike bleu and meteor scores, nodeele as well as the rf and svm classifiers indicate that the marketing text has the best classification accuracy, followed by the legal text and the literary text, while the medical text has the worst classification accuracy. in addition, nodeele as well as the rf and svm classifiers reveal that deepl (s1) performed better in all text genres in contrast with bleu and meteor, with the former showing that google translate (s2 ) performed better in the case of the literary and marketing text, and the latter showing that it performed better in the case of the medical text. ## conclusions and future work in this paper, a deep learning novel schema for evaluating nmt systems and outputs is presented and discussed. the schema, i.e. nodeele, used information from the source segments, the mt outputs and the reference translations as well as automatic metrics, and was applied in four different text genres: medical, legal, literary and marketing. experimental results showed that nodeele achieves stronger correlation with the human judgments compared to the rf classifier and the svm classifier. unlike bleu and meteor, nodeele, as well as the rf and svm classifiers, indicate i. that the marketing text has the best classification accuracy and the medical text the worst classification accuracy and ii. deepl (s1 ) performed better in all text genres. these findings suggest that the bleu and meteor automatic metrics may not be appropriate for the evaluation of nmt output, as has been also indicated by other studies @xcite . to complement and expand this study, we aim to explore if pre-trained embeddings e.g. fasttext, could improve classification accuracy, especially concerning texts with specialised terminology. in addition, we are planning to test: i. another neural network structure and ii. a learned evaluation metric, the bleurt metric @xcite , on the same datasets. finally, in order to further explore the observed difference between the bleu and meteor automatic metrics and nodeele, we are planning to carry out a refined human error analysis to evaluate the linguistic quality of the mt outputs. linguistics (coling 2008). pp. 1121-1128. coling 2008 organizing committee, manchester, uk (aug 2008), https://www.aclweb.org/anthology/c08-1141 35. ziganshina, l.e., yudina, e.v., gabdrakhmanov, a.i., ried, j.: assessing human post-editing efforts to compare the performance of three machine translation engines for english to russian translation of cochrane plain language health information: results of a randomised comparison. in: informatics. vol. 8, p. 9. multidisciplinary digital publishing institute (2021)