Accurate , automatic evaluation of machine translation is critical for system tuning , and evaluating progress in the field . We proposed a simple unsupervised metric , and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences . We find that these models rival or surpass all existing metrics in the WMT 2017 sentence-level and systemlevel tracks , and our trained model has a substantially higher correlation with human judgements than all existing metrics on the WMT 2017 to-English sentence level dataset . Evaluation metrics are a fundamental component of machine translation ( MT ) and other language generation tasks . The problem of assessing whether a translation is both adequate and coherent is a challenging text analysis problem , which is still unsolved , despite many years of effort by the research community . Shallow surfacelevel metrics , such as BLEU and TER ( Papineni et al . , 2002 ; Snover et al . , 2006 ) , still predominate in practice , due in part to their reasonable correlation to human judgements , and their being parameter free , making them easily portable to new languages . In contrast , trained metrics ( Song and Cohn , 2011 ; Stanojevic and Sima'an , 2014 ; Ma et al . , 2017 ; Shimanaka et al . , 2018 ) , which are learned to match human evaluation data , have been shown to result in a large boost in performance . This paper aims to improve over existing MT evaluation methods , through developing a series of new metrics based on contextual word embeddings ( Peters et al . , 2018 ; Devlin et al . , 2019 ) , a technique which captures rich and portable representations of words in context , which have been shown to provide important signal to many other NLP tasks ( Rajpurkar et al . , 2018 ) . We propose a simple untrained model that uses off-the-shelf contextual embeddings to compute approximate recall , when comparing a reference to an automatic translation , as well as trained models , including : a recurrent model over reference and translation sequences , incorporating attention ; and the adaptation of an NLI method ( Chen et al . , 2017 ) to MT evaluation . These approaches , though simple in formulation , are highly effective , and rival or surpass the best approaches from WMT 2017 . Moreover , we show further improvements in performance when our trained models are learned using noisy crowd-sourced data , i.e. , having single annotations for more instances is better than collecting and aggregating multiple annotations for single instances . The net result is an approach that is more data efficient than existing methods , while producing substantially better human correlations.1 We show that contextual embeddings are very useful for evaluation , even in simple untrained models , as well as in deeper attention based methods . When trained on a larger , much noisier range of instances , we demonstrate a substantial improvement over the state of the art . In future work , we plan to extend these models by using cross-lingual embeddings , and combine information from translation-source interactions as well as translation-reference interactions . There are also direct applications to Quality Estimation , by using the source instead of the reference . B System-level results for WMT 17 news and WMT 2016 IT domain

Challenge: While evaluation metrics are fundamental for machine translation systems, the problem of assessing whether a translation is both adequate and coherent remains unsolved.
Approach: They propose untrained contextual embeddings and also trained models such as a recurrent model to compute approximate recall between a reference and an automatic translation.
Outcome: The proposed models rival or surpass existing metrics on WMT 2017 sentence and system-level tracks while having higher correlations with human judgements.