State-of-the-art statistical machine translation ( MT ) systems have made significant progress towards producing user-acceptable translation output . However , there is still no efficient way for MT systems to inform users which words are likely translated correctly and how confident it is about the whole sentence . We propose a novel framework to predict wordlevel and sentence-level MT errors with a large number of novel features . Experimental results show that the MT error prediction accuracy is increased from 69.1 to 72.2 in F-score . The Pearson correlation between the proposed confidence measure and the human-targeted translation edit rate ( HTER ) is 0.6 . Improvements between 0.4 and 0.9 TER reduction are obtained with the n-best list reranking task using the proposed confidence measure . Also , we present a visualization prototype of MT errors at the word and sentence levels with the objective to improve post-editor productivity . State-of-the-art Machine Translation ( MT ) systems are making progress to generate more usable translation outputs . In particular , statistical machine translation systems ( Koehn et al . , 2007 ; Bach et al . , 2007 ; Shen et al . , 2008 ) have advanced to a state that the translation quality for certain language pairs ( e.g. Spanish-English , French-English , Iraqi-English ) in certain domains ( e.g. broadcasting news , force-protection , travel ) is acceptable to users . However , a remaining open question is how to predict confidence scores for machine translated words and sentences . An MT system typically returns the best translation candidate from its search space , but still has no reliable way to inform users which word is likely to be correctly translated and how confident it is about the whole sentence . Such information is vital to realize the utility of machine translation in many areas . For example , a post-editor would like to quickly identify which sentences might be incorrectly translated and in need of correction . Other areas , such as cross-lingual question-answering , information extraction and retrieval , can also benefit from the confidence scores of MT output . Finally , even MT systems can leverage such information to do n-best list reranking , discriminative phrase table and rule filtering , and constraint decoding ( Hildebrand and Vogel , 2008 ) . Numerous attempts have been made to tackle the confidence estimation problem . The work of Blatz et al . ( 2004 ) is perhaps the best known study of sentence and word level features and their impact on translation error prediction . Along this line of research , improvements can be obtained by incorporating more features as shown in ( Quirk , 2004 ; Sanchis et al . , 2007 ; Raybaud et al . , 2009 ; Specia et al . , 2009 ) . Soricut and Echihabi ( 2010 ) developed regression models which are used to predict the expected BLEU score of a given translation hypothesis . Improvement also can be obtained by using target part-of-speech and null dependency link in a MaxEnt classifier ( Xiong et al . , 2010 ) . Ueffing and Ney ( 2007 ) introduced word posterior probabilities ( WPP ) features and applied them in the n-best list reranking . From the usability point of view , back-translation is a tool to help users to assess the accuracy level of MT output ( Bach et al . , 2007 ) . Literally , it translates backward the MT output into the source language to see whether the output of backward translation matches the original source sentence . However , previous studies had a few shortcomings . First , source-side features were not extensively investigated . Blatz et al.(2004 ) only investigated source ngram frequency statistics and source language model features , while other work mainly focused on target side features . Second , previous work attempted to incorporate more features but faced scalability issues , i.e. , to train many features we need many training examples and to train discriminatively we need to search through all possible translations of each training example . Another issue of previous work was that they are all trained with BLEU / TER score computing against the translation references which is different from predicting the human-targeted translation edit rate ( HTER ) which is crucial in post-editing applications ( Snover et al . , 2006 ; Papineni et al . , 2002 ) . Finally , the backtranslation approach faces a serious issue when forward and backward translation models are symmetric . In this case , back-translation will not be very informative to indicate forward translation quality . In this paper , we predict error types of each word in the MT output with a confidence score , extend it to the sentence level , then apply it to n-best list reranking task to improve MT quality , and finally design a visualization prototype . We try to answer the following questions : • Can we use a rich feature set such as sourceside information , alignment context , and dependency structures to improve error prediction performance ? • Can we predict more translation error types i.e substitution , insertion , deletion and shift ? • How good do our prediction methods correlate with human correction ? • Do confidence measures help the MT system to select a better translation ? • How confidence score can be presented to improve end-user perception ? In Section 2 , we describe the models and training method for the classifier . We describe novel features including source-side , alignment context , and dependency structures in Section 3 . Experimental results and analysis are reported in Section 4 . Section 5 and 6 present applications of confidence scores . In this paper we proposed a method to predict confidence scores for machine translated words and sentences based on a feature-rich classifier using linguistic and context features . Our major contributions are three novel feature sets including source side information , alignment context , and dependency structures . Experimental results show that by combining the source side information , alignment context , and dependency structure features with word posterior probability and target POS context ( Ueffing & Ney 2007 ; Xiong et al . , 2010 ) , the MT error prediction accuracy is increased from 69.1 to 72.2 in F-score . Our framework is able to predict error types namely insertion , substitution and shift . The Pearson correlation with human judgement increases from 0.52 to 0.6 . Furthermore , we show that the proposed confidence scores can help the MT system to select better translations and as a result improvements between 0.4 and 0.9 TER reduction are obtained . Finally , we demonstrate a prototype to visualize translation errors . This work can be expanded in several directions . First , we plan to apply confidence estimation to perform a second-pass constraint decoding . After the first pass decoding , our confidence estimation model can label which word is likely to be correctly translated . The second-pass decoding utilizes the confidence informa-tion to constrain the search space and hopefully can find a better hypothesis than in the first pass . This idea is very similar to the multi-pass decoding strategy employed by speech recognition engines . Moreover , we also intend to perform a user study on our visualization prototype to see if it increases the productivity of post-editors .

Challenge: Methods for computing confidence scores and predicting errors about a whole sentence for statistical machine translation systems is under explored.
Approach: They propose a framework to predict word and sentence-level errors with linguistic and context features and apply it to n-best list reranking for better translations.
Outcome: The proposed method improves error prediction by predicting insertion, substitution and shift error types and also increases the Pearson correlation with human judgements.