Which Words Matter in Defining Phrase Reordering Behavior in Statistical Machine Translation? (2016)

Abstract: Lexicalized and hierarchical reordering models use relative frequencies of fully lexicalized phrase pairs to learn phrase reordering distributions. This results in unreliable estimation for infrequent phrase pairs which also tend to be longer phrases. There are some smoothing techniques used to smooth the distributions in these models. But these techniques are unable to address the similarities between phrase pairs and their reordering distributions. We propose two models to use shorter sub-phrase pairs of an original phrase pair to smooth the phrase reordering distributions. In the first model we follow the classic idea of backing off to shorter histories commonly used in language model smoothing. In the second model, we use syntactic dependencies to identify the most relevant words in a phrase to back off to. We show how these models can be easily applied to existing lexicalized and hierarchical reordering models. Our models achieve improvements of up to 0.40 BLEU points in Chinese-English translation compared to a baseline which uses a regular lexicalized reordering model and a hierarchical reordering model. The results show that not all the words inside a phrase pair are equally important in defining phrase reordering behavior and shortening towards important words will decrease the sparsity problem for long phrase pairs.

## introduction the introduction of lexicalized reordering models (lrms) @xcite was a significant step towards better reordering by modeling the orientation of the current phrase pair with respect to the previously translated phrase. lrms score the order in which phrases are translated by using a distribution of distinguished orientations conditioned on phrase pairs. typically, the set of orientations consists of: monotone (m), swap (s) and discontinuous (d). however, lrms are limited to reorderings of neighboring phrases only. @xcite proposed a hierarchical phrase reordering model (hrm) for more global reorderings. lrms and hrms both use relative frequencies observed in a parallel corpus to estimate the distribution of orientations conditioned on phrase pairs. as a result, they both suffer from the same problem of estimating reliable distributions for cases that occur rarely during training and therefore have to resort to smoothing methods to alleviate sparsity issues. @xcite builds on top of hrms and proposes a sparse feature approach which uses word clusters instead of fully lexicalized forms for infrequent words to decrease the effect of sparsity on the estimated model. in this paper, we propose two types of approaches to use the most influential words from inside the original phrase pairs to estimate better orientation distributions for infrequent phrase pairs that takes phrase pair similarity more into account. in the first approach, we define a backoff model to shorten towards important words inside the original phrase pairs following the idea our second contribution is a deeper analysis showing that orientation distributions conditioned on long phrase pairs typically depend on a few words within phrase pairs and not the whole lexicalized form. this supports and adds to the sparse reordering features @xcite . ## problem definition in order to smooth the original maximum likelihood estimation, lrms originally back off to the general distribution over orientations: to better understand what kind of information is ignored by both of the aforementioned smoothing methods, consider the phrase pairs and their corresponding distributions given in in this paper, we argue that in order to obtain smoother reordering distributions for phrasepairs such as the ones in ## related work the problem of data sparsity of training lrms has first been addressed by @xcite who propose to use pos tags and word clustering methods and distinguish the first or last word of a phrase, based on the language, as the head of a phrase. somewhat complementary to our work, @xcite introduced hierarchical reordering models that group phrases occurring next to the current phrase into blocks, ignoring the internal derivation within a block, which biases orientations more towards monotone and swap. at the same time, orientations are still conditioned on entire phrase pairs, which means that their approach suffers from the same sparsity problems as lrms. this problem has been more directly addressed by @xcite who uses unsupervised word classes for infrequent words of the phrase pairs in the form of sparse features. like @xcite , the first and last words of phrase pair are used as features in his model. unfortunately, this approach also introduces thousands of additional sparse features, many of which have to be extracted during decoding, requiring changes to the decoder as well as a sizable tuning set. @xcite investigate the effect of generalized word forms on reordering in an n-gram-based operation sequence model, where they use different generalized representations including pos tags and unsupervised word classes to generalize reordering rules to similar cases with unobserved lexical operations. while the approaches above use discrete representations, @xcite propose a discriminative model using continuous space representations of phrase pairs to address data sparsity problems. they train a neural network classifier based on recursive auto-encoders to generate vector space representations of phrase pairs and base reordering decision on those representations. they apply their model as an additional hypergraph reranking step since direct integration into the decoder would make hypothesis recombination more complex and substantially increase the size of the search space. in addition to reordering models, several approaches have used word classes to improve other models within a statistical machine translation system, including translation @xcite and language models, where the problem of data sparsity is particularly exacerbated for morphologically rich target languages @xcite . ## model definition in this section, we propose two different models which use different words as source of information to better estimate reordering distributions of sparse phrase pairs. each model uses a different generalization scheme to obtain less sparse but still informative representations. ## experiments we evaluate our models for chinese-to-english translation. our training data consists of the parallel and monolingual data released by nist's openmt campaign, with mt04 used for tuning and news data from mt05 and mt06 for testing, see ## conclusions we have introduced a novel method that builds on the established idea of backing off to shorter histories, commonly used in language model smoothing, and shown that it can be successfully applied to smoothing of lexicalized and hierarchical reordering models in statistical machine translation. furthermore, we have shown that not all sub-phrase pairs are influential in that regard. the sub-phrase pair consisting of just exposed heads of a phrase pair tends to be the most important one and most other words inside a phrase pair have negligible influence on reordering behavior. earlier approaches, such as @xcite and @xcite , often assume that the last and the first word of a phrase pair are important, but our experiments indicate that exhgposed heads tend to be stronger predictors. we showed that generalized representations of phrase pairs based on exposed heads can help decrease sparsity and result in more reliable reordering distributions. considering the analysis of the length of infrequent phrase pairs used during translation, we also conclude that a smoothing model that would be able to further improve the distribution of single word phrase pairs is crucial for achieving higher improvements during translation. for future work, we plan to investigate the effect surface word forms from outside of a phrase pair can have on reordering. this could further help improve reordering distributions of infrequent single word phrase pairs more effectively as they constitute a large portion of the phrases used during decoding.