Using W ordnet to Improve Reordering in Hierarchical Phrase-Based Statistical Machine Translation (2016)

Abstract: We propose the use of WordNet synsets in a syntax-based reordering model for hierarchical statistical machine translation (HPB-SMT) to enable the model to generalize to phrases not seen in the training data but that have equivalent meaning. We detail our methodology to incorporate synsets’ knowledge in the reordering model and evaluate the resulting WordNet-enhanced SMT systems on the English-to-Farsi language direction. The inclusion of synsets leads to the best BLEU score, outperforming the baseline (standard HPB-SMT) by 0.6 points absolute.

## introduction statistical machine translation (smt) is a data driven approach for translating from one natural language into another. natural languages vary in their vocabularies and also in the manner that they arrange words in the sentence. accordingly, smt systems should address two interrelated problems: finding the appropriate words in the translation ("lexical choice") and predicting their order in the sentence ("reordering"). reordering is one of the hardest problems in smt and has a significant impact on the quality of the translation, especially between languages with major differences in word order. although smt systems deliver state-of-theart performance in machine translation nowadays, they perform relatively weakly at addressing the reordering problem. phrased-based smt (pb-smt) is arguably the most widely used approach to smt to date. in this model, the translation operates on phrases, i.e. sequences of words whose length is between 1 and a maximum upper limit. in pb-smt, reordering is generally captured by distance-based models @xcite and lexical phrase-based models @xcite , which are able to perform local reordering but they cannot capture non-local (long-distance) reordering. the weakness of pb-smt systems on handling long-distance reordering led to proposing the hierarchical phrase-based smt (hpb-smt) model @xcite , in which the translation operates on tree structures (either derived from a syntactic parser or unsupervised). despite the relatively good performance offered by hpb-smt in medium-range reordering, they are still weak on long-distance reordering @xcite . a great deal of work has been carried out to address the reordering problem by incorporating reordering models (rm) into smt systems. a rm tries to capture the differences in word order in a probabilistic framework and assigns a probability to each possible order of words in the target sentence. most of the reordering models can perform reordering of common words or phrases relatively well, but they can not be generalized to unseen words or phrases with the same meaning ("semantic generalization") or the same syntactic structure ("syntactic generalization"). for example, if in the source language the object follows the verb and in the target language it precedes the verb, these models still need to see particular instances of verbs and objects in the training data to be able to perform required reordering between them. likewise, if two words in the source language follow a specific reordering pattern in the target language, these models can not generalize to unseen words with equivalent meaning in the same context. in order to improve syntactic and semantic generalization of the rm, it is necessary to incorporate syntactic and semantic features into the model. while there has been some encouraging work on integrating syntactic features into the rm, to the best of our knowledge, there has been no previous work on integrating semantic reordering model features types features zens and ney (2006) lexical surface forms of the source and target words unsupervised class of the source and target words cherry (2013) lexical surface forms of frequent source and target words unsupervised class of rare source and target words green et al. (2010) lexical syntactic surface forms of the source words, pos tags of the source words, relative position of the source words sentence length bisazza and federico (2013) and goto et al. (2013) lexical syntactic surface forms and pos tags of the source words surface forms and pos tags of the source context words gao et al. (2011) and kazemi et al. (2015) lexical syntactic surface forms of the source words dependency relation the proposed method lexical syntactic semantic surface forms of the source words dependency relation synset of the source words ## related work many different approaches have been proposed to capture long-distance reordering by incorporating a rm into pb-smt or hpb-smt systems. a rm should be able to perform the required reorderings not only for common words or phrases, but also for phrases unseen in the training data that hold the same syntactic and semantic structure. in other words, a rm should be able to make syntactic and semantic generalizations. to this end, rather than conditioning on actual phrases, state-of-theart rms generally make use of features extracted from the phrases of the training data. one useful way to categorize previous rms is by the features that they use to generalize. these features can be divided into three groups: (i) lexical features (ii) syntactic features and (iii) semantic features. ## method following @xcite we implement a syntax-based rm for hpb-smt based on the dependency tree of the source sentence. the dependency tree of a sentence shows the grammatical relation between pairs of head and dependent words in the sentence. as an example, figure 1 shows the dependency tree of an english sentence. in this figure, the arrow with label "nsubj" from "fox" to "jumped" indicates that the dependent word "fox" is the subject of the head word "jumped". given the assumption that constituents move as a whole during translation @xcite , we take the dependency tree of the source sentence and try to find the ordering of each dependent word with respect to its head (head-dep) and also with respect to the other dependants of that head (dep-dep). for example, for the english sentence in figure 1 , we try to predict the orientation between (head-dep) and (dep-dep) pairs as shown in we consider two orientation types between the constituents: monotone and swap. if the order of two constituents in the source sentence is the same as the order of their translation in the target sentence, the orientation is monotone and otherwise it is swap. to be more formal, for two source words (s 1 ,s 2 ) and their aligned target words (t 1 ,t 2 ), with the alignment points (p s1 ,p s2 ) and (p t 1 ,p t 2 ), we find the orientation type between s 1 and s 2 as shown in equation 1 @xcite . for example, for the sentence in figure 1 , the orientation between the source words "brown" and "quick" is monotone, while the orientation between "brown" and "fox" is swap. we use a classifier to predict the probability of the orientation between each pair of constituents to be monotone or swap. this probability is used as one feature in the log-linear framework of the hpb-smt model. using a classifier enables us to incorporate fine-grained information in the form of features into our rm. as on top of this syntactic generalization, the rm should be able to make semantic generalizations. to this end, we use wordnet synsets as an additional feature in our rm. wordnet is a lexical database of english which groups words into sets of cognitive synonyms. in other words, in word-net a set of synonym words belong to the same synset. for example, the words "baby", "babe" and "infant" are in the same synset in wordnet. the use of synsets enables our rm to be generalized from words seen in the training data to any of their synonyms present in wordnet. ## conclusions and future work in this paper we have extended a syntax-based rm for hpb-smt with semantic features (wordnet synsets), in order to enable the model to generalize to phrases not seen in the training data but that have equivalent meaning. the inclusion of synsets has led to the best bleu score in our experiments, outperforming the baseline (standard hpb-smt) by 0.6 points absolute. as for future work, we propose to work mainly along the following two directions. first, an investigation of the extent to which using a wordnetinformed approach to classify the words into semantic classes (as proposed in this work) outperforms an unsupervised approach via word clustering. second, an in-depth human evaluation to gain further insights of the exact contribution of word-net to the translation output.