Automatic evaluation of Machine Translation ( MT ) quality is essential to developing highquality MT systems . Various evaluation metrics have been proposed , and BLEU is now used as the de facto standard metric . However , when we consider translation between distant language pairs such as Japanese and English , most popular metrics ( e.g. , BLEU , NIST , PER , and TER ) do not work well . It is well known that Japanese and English have completely different word orders , and special care must be paid to word order in translation . Otherwise , translations with wrong word order often lead to misunderstanding and incomprehensibility . For instance , SMT-based Japanese-to-English translators tend to translate ' A because B ' as ' B because A. ' Thus , word order is the most important problem for distant language translation . However , conventional evaluation metrics do not significantly penalize such word order mistakes . Therefore , locally optimizing these metrics leads to inadequate translations . In this paper , we propose an automatic evaluation metric based on rank correlation coefficients modified with precision . Our meta-evaluation of the NTCIR-7 PATMT JE task data shows that this metric outperforms conventional metrics . SMT systems thus fail to find ( R0 ) . Consequently , the global word order is essential for translation between distant language pairs , and wrong word order can easily lead to misunderstanding or incomprehensibility . Perhaps , some readers do not understand why we emphasize word order from this example alone . A few more examples will clarify what happens when SMT is applied to Japanese-to-English translation . Even the most famous SMT service available on the web failed to translate the following very simple sentence at the time of writing this paper . Japanese : meari wa jon wo koroshita . Reference : Mary killed John . SMT output : John killed Mary . Since it can not translate such a simple sentence , it obviously can not translate more complex sentences correctly . Japanese : bobu ga katta hon wo jon wa yonda . Reference : John read a book that Bob bought . SMT output : Bob read the book John bought . Japanese : bobu wa meari ni yubiwa wo kau tameni , jon no mise ni itta . Reference : Bob went to John 's store to buy a ring for Mary . SMT output : Bob Mary to buy the ring , John went to the store . Automatic evaluation of machine translation ( MT ) quality is essential to developing high-quality machine translation systems because human evaluation is time consuming , expensive , and irreproducible . If we have a perfect automatic evaluation metric , we can tune our translation system for the metric . BLEU ( Papineni et al . , 2002b ; Papineni et al . , 2002a ) showed high correlation with human judgments and is still used as the de facto standard automatic evaluation metric . However , Callison-Burch et al . ( 2006 ) argued that the MT community is overly reliant on BLEU by showing examples of poor performance . For Japanese-to-English ( JE ) translation , Echizen-ya et al . ( 2009 ) showed that the popular BLEU and NIST do not work well by using the system outputs of the NTCIR-7 PATMT ( patent translation ) JE task ( Fujii et al . , 2008 ) . On the other hand , ROUGE-L ( Lin and Hovy , 2003 ) , Word Error Rate ( WER ) , and IMPACT ( Echizen-ya and Araki , 2007 ) worked better . In these studies , Pearson 's correlation coefficient and Spearman 's rank correlation ρ with human evaluation scores are used to measure how closely an automatic evaluation method correlates with human evaluation . This evaluation of automatic evaluation methods is called meta-evaluation . In human evaluation , people judge the adequacy and the fluency of each translation . Denoual and Lepage ( 2005 ) pointed out that BLEU assumes word boundaries , which is ambiguous in Japanese and Chinese . Here , we assume the word boundaries given by ChaSen , one of the standard morphological analyzers ( http://chasenlegacy.sourceforge.jp/ ) following Fujii et al . ( 2008 ) In JE translation , most Statistical Machine Translation ( SMT ) systems translate the Japanese sentence ( J0 ) kare wa sono hon wo yonda node sekaishi ni kyoumi ga atta which means ( R0 ) he was interested in world history because he read the book into an English sentence such as ( H0 ) he read the book because he was interested in world history in which the cause and the effect are swapped . Why does this happen ? The former half of ( J0 ) means " He read the book , " and the latter half means " ( he ) was interested in world history . " The middle word " node " between them corresponds to " because . " Therefore , SMT systems output sentences like ( H0 ) . On the other hand , Rule-based Machine Translation ( RBMT ) systems correctly give ( R0 ) . In order to find ( R0 ) , SMT systems have to search a very large space because we can not restrict its search space with a small distortion limit . Most In this way , this SMT service usually gives incomprehensible or misleading translations , and thus people prefer RBMT services . Other SMT systems also tend to make similar word order mistakes , and special care should be paid to the translation between distant language pairs such as Japanese and English . Even Japanese people can not solve this word order problem easily : It is well known that Japanese people are not good at speaking English . From this point of view , conventional automatic evaluation metrics of translation quality disregard word order mistakes too much . Single-reference BLEU is defined by a geometrical mean of n-gram precisions p n and is modified by Brevity Penalty ( BP ) min(1 , exp(1 -r / h ) ) , where r is the length of the reference and h is the length of the hypothesis . BLEU = BP × ( p 1 p 2 p 3 p 4 ) 1/4 . Its range is [ 0 , 1 ] . The BLEU score of ( H0 ) with reference ( R0 ) is 1.0×(11/11×9/10×6/9×4/8 ) 1/4 = 0.740 . Therefore , BLEU gives a very good score to this inadequate translation because it checks only ngrams and does not regard global word order . Since ( R0 ) and ( H0 ) look similar in terms of fluency , adequacy is more important than fluency in the translation between distant language pairs . Similarly , other popular scores such as NIST , PER , and TER ( Snover et al . , 2006 ) also give relatively good scores to this translation . NIST also considers only local word orders ( n-grams ) . PER ( Position-Independent Word Error Rate ) was designed to disregard word order completely . TER ( Snover et al . , 2006 ) was designed to allow phrase movements without large penalties . Therefore , these standard metrics are not optimal for evaluating translation between distant language pairs . In this paper , we propose an alternative automatic evaluation metric appropriate for distant language pairs . Our method is based on rank correlation coefficients . We use them to compare the word ranks in the reference with those in the hypothesis . There are two popular rank correlation coefficients : Spearman 's ρ and Kendall 's τ ( Kendall , 1975 ) . In Isozaki et al . ( 2010 ) , we used Kendall 's τ to measure the effectiveness of our Head Finalization rule as a preprocessor for English-to-Japanese translation , but we measured the quality of translation by using conventional metrics . It is not clear how well τ works as an automatic evaluation metric of translation quality . Moreover , Spearman 's ρ might work better than Kendall 's τ . As we discuss later , τ considers only the direction of the rank change , whereas ρ considers the distance of the change . The first objective of this paper is to examine which is the better metric for distant language pairs . The second objective is to find improvements of these rank correlation-metrics . Spearman 's ρ is based on Pearson 's correlation coefficients . Suppose we have two lists of numbers x = [ 0.1 , 0.4 , 0.2 , 0.6 ] , y = [ 0.9 , 0.6 , 0.2 , 0.7 ] . To obtain Pearson 's coefficients between x and y , we use the raw values in these lists . If we substitute their ranks for their raw values , we get x = [ 1 , 3 , 2 , 4 ] and y = [ 4 , 2 , 1 , 3 ] . Then , Spearman 's ρ between x and y is given by Pearson 's coefficients between x and y . This ρ can be rewritten as follows when there is no tie : ρ = 1 -i d 2 i n+1 C 3 . Here , d i indicates the difference in the ranks of the i-th element . Rank distances are squared in this formula . Because of this square , we expect that ρ decreases drastically when there is an element that significantly changes in rank . But we are also afraid that ρ may be too severe for alternative good translations . Since Pearson 's correlation metric assumes linearity , nonlinear monotonic functions can change its score . On the other hand , Spearman 's ρ and Kendall 's τ uses ranks instead of raw evaluation scores , and simple application of monotonic functions can not change them ( use of other operations such as averaging sentence scores can change them ) . When Statistical Machine Translation is applied to distant language pairs such as Japanese and English , word order becomes an important problem . SMT systems often fail to find an appropriate translation because of a large search space . Therefore , they often output misleading or incomprehensible sentences such as " A because B " vs. " B because A. " To penalize such inadequate translations , we presented an automatic evaluation method based on rank correlation . There were two questions for this approach . First , which correlation coefficient should we use : Spearman 's ρ or Kendall 's τ ? Second , how should we solve the overestimation problem caused by the nature of one-to-one correspondence ? We answered these questions through our experiments using the NTCIR-7 PATMT JE translation data . For the first question , τ was slightly better than ρ , but ρ was improved by precision . For the second question , it turned out that BLEU 's Brevity Penalty was counter-productive . A precision-based penalty gave a better solution . With this precisionbased penalty , both ρ and τ worked well and they outperformed conventional methods for NTCIR-7 data . For similar language pairs , our method was comparable to conventional evaluation methods . Fu-

Challenge: Existing automatic evaluation metrics do not perform well on two distant language pairs where word order is important.
Approach: They propose to use rank correlation coefficients for the evaluation of distant language pairs which can penalize wrongly ordered outputs.
Outcome: The proposed methods outperform existing methods for distant language pairs and are comparable to similar language pairs.