In this work , the problem of extracting phrase translation is formulated as an information retrieval process implemented with a log-linear model aiming for a balanced precision and recall . We present a generic phrase training algorithm which is parameterized with feature functions and can be optimized jointly with the translation engine to directly maximize the end-to-end system performance . Multiple data-driven feature functions are proposed to capture the quality and confidence of phrases and phrase pairs . Experimental results demonstrate consistent and significant improvement over the widely used method that is based on word alignment matrix only . Phrase has become the standard basic translation unit in Statistical Machine Translation ( SMT ) since it naturally captures context dependency and models internal word reordering . In a phrase-based SMT system , the phrase translation table is the defining component which specifies alternative translations and their probabilities for a given source phrase . In learning such a table from parallel corpus , two related issues need to be addressed ( either separately or jointly ): which pairs are considered valid translations and how to assign weights , such as probabilities , to them . The first problem is referred to as phrase pair extraction , which identifies phrase pairs that are supposed to be translations of each other . Methods have been proposed , based on syntax , that take advantage of linguistic constraints and alignment of grammatical structure , such as in Yamada and Knight ( 2001 ) and Wu ( 1995 ) . The most widely used approach derives phrase pairs from word alignment matrix ( Och and Ney , 2003 ; Koehn et al . , 2003 ) . Other methods do not depend on word alignments only , such as directly modeling phrase alignment in a joint generative way ( Marcu and Wong , 2002 ) , pursuing information extraction perspective ( Venugopal et al . , 2003 ) , or augmenting with modelbased phrase pair posterior ( Deng and Byrne , 2005 ) . Using relative frequency as translation probability is a common practice to measure goodness of a phrase pair . Since most phrases appear only a few times in training data , a phrase pair translation is also evaluated by lexical weights ( Koehn et al . , 2003 ) or term weighting ( Zhao et al . , 2004 ) as additional features to avoid overestimation . The translation probability can also be discriminatively trained such as in Tillmann and Zhang ( 2006 ) . The focus of this paper is the phrase pair extraction problem . As in information retrieval , precision and recall issues need to be addressed with a right balance for building a phrase translation table . High precision requires that identified translation candidates are accurate , while high recall wants as much valid phrase pairs as possible to be extracted , which is important and necessary for online translation that requires coverage . In the word-alignment derived phrase extraction approach , precision can be improved by filtering out most of the entries by using a statistical significance test ( Johnson et al . , 2007 ) . On the other hand , there are valid translation pairs in the training corpus that are not learned due to word alignment errors as shown in Deng and Byrne ( 2005 ) . We would like to improve phrase translation accuracy and at the same time extract as many as possible valid phrase pairs that are missed due to incorrect word alignments . One approach is to leverage underlying word alignment quality such as in Ayan and Dorr ( 2006 ) . In this work , we present a generic discriminative phrase pair extraction framework that can integrate multiple features aiming to identify correct phrase translation candidates . A significant deviation from most other approaches is that the framework is parameterized and can be optimized jointly with the decoder to maximize translation performance on a development set . Within the general framework , the main work is on investigating useful metrics . We employ features based on word alignment models and alignment matrix . We also propose information metrics that are derived from both bilingual and monolingual perspectives . All these features are data-driven and independent of languages . The proposed phrase extraction framework is general to apply linguistic features such as semantic , POS tags and syntactic dependency . In this paper , the problem of extracting phrase translation is formulated as an information retrieval process implemented with a log-linear model aiming for a balanced precision and recall . We have presented a generic phrase translation extraction procedure which is parameterized with feature functions . It can be optimized jointly with the translation engine to directly maximize the end-to-end translation performance . Multiple feature functions were investigated . Our experimental results on IWSLT Chinese-English corpus have demonstrated consistent and significant improvement over the widely used word alignment matrix based extraction method . 3

Challenge: Existing approaches for the phrase pair extraction apply a filter to extracted pairs to obtain high precision but lose recall due to word alignment errors.
Approach: They propose an algorithm parameterized with feature functions which can be jointly optimized to maximize final translation quality to obtain a balanced precision and recall.
Outcome: The proposed algorithm shows consistent improvements over a common existing method which is only based on a word alignment matrix on the IWSLT Chinese-English corpus.