Several attempts have been made to learn phrase translation probabilities for phrasebased statistical machine translation that go beyond pure counting of phrases in word-aligned training data . Most approaches report problems with overfitting . We describe a novel leavingone-out approach to prevent over-fitting that allows us to train phrase models that show improved translation performance on the WMT08 Europarl German-English task . In contrast to most previous work where phrase models were trained separately from other models used in translation , we include all components such as single word lexica and reordering models in training . Using this consistent training of phrase models we are able to achieve improvements of up to 1.4 points in BLEU . As a side effect , the phrase table size is reduced by more than 80 % . A phrase-based SMT system takes a source sentence and produces a translation by segmenting the sentence into phrases and translating those phrases separately ( Koehn et al . , 2003 ) . The phrase translation table , which contains the bilingual phrase pairs and the corresponding translation probabilities , is one of the main components of an SMT system . The most common method for obtaining the phrase table is heuristic extraction from automatically word-aligned bilingual training data ( Och et al . , 1999 ) . In this method , all phrases of the sentence pair that match constraints given by the alignment are extracted . This includes overlapping phrases . At extraction time it does not matter , whether the phrases are extracted from a highly probable phrase alignment or from an unlikely one . Phrase model probabilities are typically defined as relative frequencies of phrases extracted from word-aligned parallel training data . The joint counts C ( f , ẽ ) of the source phrase f and the target phrase ẽ in the entire training data are normalized by the marginal counts of source and target phrase to obtain a conditional probability EQUATION The translation process is implemented as a weighted log-linear combination of several models h m ( e I 1 , s K 1 , f J 1 ) including the logarithm of the phrase probability in source-to-target as well as in target-to-source direction . The phrase model is combined with a language model , word lexicon models , word and phrase penalty , and many others . ( Och and Ney , 2004 ) The best translation êÎ 1 as defined by the models then can be written as EQUATION In this work , we propose to directly train our phrase models by applying a forced alignment procedure where we use the decoder to find a phrase alignment between source and target sentences of the training data and then updating phrase translation probabilities based on this alignment . In contrast to heuristic extraction , the proposed method provides a way of consistently training and using phrase models in translation . We use a modified version of a phrase-based decoder to perform the forced alignment . This way we ensure that all models used in training are identical to the ones used at decoding time . An illustration of the basic idea can be seen in Figure 1 . In the literature this method by itself has been shown to be problematic because it suffers from over-fitting ( DeNero et al . , 2006 ) , ( Liang et al . , 2006 ) . Since our initial phrases are extracted from the same training data , that we want to align , very long phrases can be found for segmentation . As these long phrases tend to occur in only a few training sentences , the EM algorithm generally overestimates their probability and neglects shorter phrases , which better generalize to unseen data and thus are more useful for translation . In order to counteract these effects , our training procedure applies leaving-one-out on the sentence level . Our results show , that this leads to a better translation quality . Ideally , we would produce all possible segmentations and alignments during training . However , this has been shown to be infeasible for real-world data ( DeNero and Klein , 2008 ) . As training uses a modified version of the translation decoder , it is straightforward to apply pruning as in regular decoding . Additionally , we consider three ways of approximating the full search space : 1 . the single-best Viterbi alignment , 2 . the n-best alignments , 3 . all alignments remaining in the search space after pruning . The performance of the different approaches is measured and compared on the German-English Europarl task from the ACL 2008 Workshop on Statistical Machine Translation ( WMT08 ) . Our results show that the proposed phrase model training improves translation quality on the test set by 0.9 BLEU points over our baseline . We find that by interpolation with the heuristically extracted phrases translation performance can reach up to 1.4 BLEU improvement over the baseline on the test set . After reviewing the related work in the following section , we give a detailed description of phrasal alignment and leaving-one-out in Section 3 . Section 4 explains the estimation of phrase models . The empirical evaluation of the different approaches is done in Section 5 . We have shown that training phrase models can improve translation performance on a state-ofthe-art phrase-based translation model . This is achieved by training phrase translation probabilities in a way that they are consistent with their use in translation . A crucial aspect here is the use of leaving-one-out to avoid over-fitting . We have shown that the technique is superior to limiting phrase lengths and smoothing with lexical probabilities alone . While models trained from Viterbi alignments already lead to good results , we have demonstrated that considering the 100-best alignments allows to better model the ambiguities in phrase segmentation . The proposed techniques are shown to be superior to previous approaches that only used lexical probabilities to smooth phrase tables or imposed limits on the phrase lengths . On the WMT08 Europarl task we show improvements of 0.9 BLEU points with the trained phrase table and 1.4 BLEU points when interpolating the newly trained model with the original , heuristically extracted phrase table . In TER , improvements are 0.4 and 1.7 points . In addition to the improved performance , the trained models are smaller leading to faster and smaller translation systems .

Challenge: Existing approaches that learn phrase translation probabilities for phrase-based statistical machine translation beyond pure phrase counting in word-aligned training data suffer from over-fitting.
Approach: They propose a leave-one-out approach for training phrase models jointly with other translation components to be consistent with the training data.
Outcome: The proposed model outperforms baseline models with smoothing or phrase length limitations on Europarl German-English and TER datasets and keeps the system faster and smaller.