Although discriminative training guarantees to improve statistical machine translation by incorporating a large amount of overlapping features , it is hard to scale up to large data due to decoding complexity . We propose a new algorithm to generate translation forest of training data in linear time with the help of word alignment . Our algorithm also alleviates the oracle selection problem by ensuring that a forest always contains derivations that exactly yield the reference translation . With millions of features trained on 519 K sentences in 0.03 second per sentence , our system achieves significant improvement by 0.84 BLEU over the baseline system on the NIST Chinese-English test sets . Discriminative model ( Och and Ney , 2002 ) can easily incorporate non-independent and overlapping features , and has been dominating the research field of statistical machine translation ( SMT ) in the last decade . Recent work have shown that SMT benefits a lot from exploiting large amount of features ( Liang et al . , 2006 ; Tillmann and Zhang , 2006 ; Watanabe et al . , 2007 ; Blunsom et al . , 2008 ; Chiang et al . , 2009 ) . However , the training of the large number of features was always restricted in fairly small data sets . Some systems limit the number of training examples , while others use short sentences to maintain efficiency . Overfitting problem often comes when training many features on a small data ( Watanabe et al . , 2007 ; Chiang et al . , 2009 ) . Obviously , using much more data can alleviate such problem . Furthermore , large data also enables us to globally train millions of sparse lexical features which offer accurate clues for SMT . Despite these advantages , to the best of our knowledge , no previous discriminative training paradigms scale up to use a large amount of training data . The main obstacle comes from the complexity of packed forests or n-best lists generation which requires to search through all possible translations of each training example , which is computationally prohibitive in practice for SMT . To make normalization efficient , contrastive estimation ( Smith and Eisner , 2005 ; Poon et al . , 2009 ) introduce neighborhood for unsupervised log-linear model , and has presented positive results in various tasks . Motivated by these work , we use a translation forest ( Section 3 ) which contains both " reference " derivations that potentially yield the reference translation and also neighboring " non-reference " derivations that fail to produce the reference translation.1 However , the complexity of generating this translation forest is up to O(n 6 ) , because we still need biparsing to create the reference derivations . Consequently , we propose a method to fast generate a subset of the forest . The key idea ( Section 4 ) is to initialize a reference derivation tree with maximum score by the help of word alignment , and then traverse the tree to generate the subset forest in linear time . Besides the efficiency improvement , such a forest allows us to train the model without resort- r 1 X ⇒ ⟨X 1 bei X 2 , X 1 was X 2 ⟩ e 2 r 2 X ⇒ ⟨qiangshou bei X 1 , the gunman was X 1 ⟩ e 3 r 3 X ⇒ ⟨jingfang X 1 , X 1 by the police⟩ e 4 r 4 X ⇒ ⟨jingfang X 1 , police X 1 ⟩ e 5 r 5 X ⇒ ⟨qiangshou , the gunman⟩ e 6 r 6 X ⇒ ⟨jibi , shot dead⟩ Figure 1 : A translation forest which is the running example throughout this paper . The reference translation is " the gunman was killed by the police " . ( 1 ) Solid hyperedges denote a " reference " derivation tree t 1 which exactly yields the reference translation . ( 2 ) Replacing e 3 in t 1 with e 4 results a competing non-reference derivation t 2 , which fails to swap the order of X 3,4 . ( 3 ) Removing e 1 and e 5 in t 1 and adding e 2 leads to another reference derivation t 3 . Generally , this is done by deleting a node X 0,1 . ing to constructing the oracle reference ( Liang et al . , 2006 ; Watanabe et al . , 2007 ; Chiang et al . , 2009 ) , which is non-trivial for SMT and needs to be determined experimentally . Given such forests , we globally learn a log-linear model using stochastic gradient descend ( Section 5 ) . Overall , both the generation of forests and the training algorithm are scalable , enabling us to train millions of features on large-scale data . To show the effect of our framework , we globally train millions of word level context features motivated by word sense disambiguation ( Chan et al . , 2007 ) together with the features used in traditional SMT system ( Section 6 ) . Training on 519 K sentence pairs in 0.03 seconds per sentence , we achieve significantly improvement over the traditional pipeline by 0.84 BLEU . We have presented a fast generation algorithm for translation forest which contains both reference derivations and neighboring non-reference derivations for large-scale SMT discriminative training . We have achieved significantly improvement of 0.84 BLEU by incorporate 13.9 M feature trained on 519 K data in 0.03 second per sentence . In this paper , we define the forest based on competing derivations which only differ in one rule . There may be better classes of forest that can produce a better performance . It 's interesting to modify the definition of forest , and use more local operators to increase the size of forest . Furthermore , since the generation of forests is quite general , it 's straight to apply our forest on other learning algorithms . Finally , we hope to exploit more features such as reordering features and syntactic features so as to further improve the performance .

Challenge: Applying discriminative models to statistical machine translation systems is expensive, however; reducing training set size leads to overfitting.
Approach: They propose a faster algorithm for translation forest generation which runs in linear time.
Outcome: The proposed forest generation method can process a sentence in 0.03 seconds using a large feature space and also significantly outperforms the existing pipeline system.