SRL for low resource languages isn’t needed for semantic SMT (2018)

Abstract: Previous attempts at injecting semantic frame biases into SMT training for low resource languages failed because either (a) no semantic parser is available for the low resource input language; or (b) the output English language semantic parses excise relevant parts of the alignment space too aggressively. We present the first semantic SMT model to succeed in significantly improving translation quality across many low resource input languages for which no automatic SRL is available —consistently and across all common MT metrics. The results we report are the best by far to date for this type of approach; our analyses suggest that in general, easier approaches toward including semantics in training SMT models may be more feasible than generally assumed even for low resource languages where semantic parsers remain scarce. While recent proposals to use the crosslingual evaluation metric XMEANT during inversion transduction grammar (ITG) induction are inapplicable to low resource languages that lack semantic parsers, we break the bottleneck via a vastly improved method of biasing ITG induction toward learning more semantically correct alignments using the monolingual semantic evaluation metric MEANT. Unlike XMEANT, MEANT requires only a readily-available English (output language) semantic parser. The advances we report here exploit the novel realization that MEANT represents an excellent way to semantically bias expectationmaximization induction even for low resource languages. We test our systems on challenging languages including Amharic, Uyghur, Tigrinya and Oromo. Results show that our model influences the learning towards more semantically correct alignments, leading to better translation quality than both the standard ITG or GIZA++ based SMT training models on different datasets.

## introduction statistical machine translation (smt) for low resource languages has been a difficult task due to the unavailability of large parallel corpora. it becomes imperative to make learning from small data more efficient by adding additional constraints to create stronger inductive biases-especially linguistically well-motivated constraints, such as the shallow semantic parses of the training sentences. however, while automatic semantic role labeling (srl) is readily available to produce shallow semantic parses for a high-resource output language (typically english), the problem is that srl is usually not available for low resource input languages such as tigrinya, oromo, uyghur or uzbek. in this paper, we propose a new method which adopts the monolingual semantic evaluation metric meant as a confidence-weighting measure to assess the degree of goodness of training instances, giving a newer strategy than @xcite who used the degree of compatibility or similarity between the semantic role labeling of the input and output sentences. their approach might outperform ours for high-resource languages, but is completely inapplicable to low resource languages because xmeant requires both the input and output semantic parses -whereas meant does not require an srl parse for the low resource input language. additionally, we also introduce a notion of semantic role labeling coverage as a second english monolingual confidence-weighting measure. an srl coverage score roughly quantifies what proportion of a sentence is accounted for by a shallow semantic parse. the variety of approaches proposed here belong to a family of semantic smt methods that has recently been advanced, wherein srl constraints or biases are injected very early in the smt training pipeline so as to maximize their influence on what translation model is learned. we test our models on multiple difficult low resource translation tasks: amharic, somali, tigrinya, oromo, uzbek and uyghur always translating into english. despite having srls only on the english side, we show that our models influence the learning toward more semantically correct alignments. our results show that this way of inducing itgs gives a better translation quality than the conventional itg @xcite and the traditional giza++ @xcite alignments. ## core model the approaches proposed in this work inject a form of semantic parse bias into early stage word alignment using itg @xcite training, which (as shown in the results section) outperforms conventional giza++ @xcite based intersection/union-of-bidirectional-ibm-word-alignment strategies. specifically, our defined approaches assume a token based bitg (bracketing itg) @xcite system, a choice based on previous works showing that: (a) bitg based alignments outperform giza++ alignments @xcite ; (b) itg alignments have been empirically shown to cover almost 100% of semantic frame alternations, while ruling out the majority of incorrect alignments @xcite . the bitg model used in this work is initialized with uniform structural probabilities, setting aside half of the probability mass for lexical rules. the lexical probability mass is distributed among the lexical rules according to co-occurrence counts from the training data, assuming each sentence contains one empty token to account for singletons. these initial probabilities are refined with 10 iterations of em, where the expectation step is calculated using beam pruned parsing @xcite with a beam width of 100. in the last iteration, the alignments imposed by the viterbi parses are extracted as the final word alignments. @xcite showed how to compute expectations for em re-estimation with outside probabilities as follows: ## meant as a training objective function 4.1 injecting meant a more robust way to assess the degree of goodness of training instances has been shown to be the crosslingual evaluation metric @xcite . unfortunately, this is not applicable in low resource settings since xmeant assesses the compatibility between the english output and the input foreign language-for which the semantic parse is unavailable. instead of computing the crosslingual compatibility between the input and the output semantic parses, we adopt the monolingual semantic frame evaluation metric meant as a confidence measure. the evaluation metric meant computes the semantic frame coverage between the input and the mt reference. we propose to use meant as a confidence-weight measure by computing the semantic frame coverage in the english sentence. we obtain the srl coverage of a sentence by computing the meant score between the input english sentence and the same sentence as a reference. we do not take into account the chunks that have no semantic parse (backoff was mentioned in figure 2 ). figure 2 illustrates two out of three possible situations for applying meant as a confidenceweight measure. the sentences that are fully semantically parsed like [arg0 i][target ate][arg1 an apple]. have a meant score equal to 1.0. if the sentence is partially srled, the meant score is less than 1.0. for instance, the meant score for the parse where do [arg0 i][target get][arg2 off] to go to union square? is less than 1, but higher than 0. furthermore, we note that a few sentences have a 0 meant score. in fact, we have experimented with three automatic srls: assert @xcite , mate @xcite and mateplus (roth and woodsend, 2014); we have observed that these srl systems completely fail to parse sentences containing the verb to be; sentences like the light was red are ignored. however, we show that even while ignoring sentences containing to be, our systems are still outperforming conventional models on multiple challenging low resource languages. ## results adopting meant for confidence-weighting gives the best results for translating low resource languages. we compare the performance of the meant and the monolingual english srl coverage based bitg alignments against the conventional bitg and the traditional giza++ alignments. to efficiently assess the quality of our different systems, we evaluate using surface based metrics such as bleu @xcite , editdistance based metrics such as cder @xcite , wer @xcite , per @xcite , ter @xcite and the semantic evaluation metric meant @xcite . ## conclusion we have shown that adopting the monolingual semantic evaluation metric meant as an objective function for driving itg induction yields a high improvement compared to the conventional alignment methods on many challenging low resource languages. we have also proposed another heuristic for evaluating how good an english semantic parse is, then used it to induce itgs. we have experimented with several challenging low resource languages from different language families and have demonstrated that using a monolingual semantic frame based objective function during the actual learning of the translation model helps learn good bilingual correlations with a relatively small dataset in contrast to conventional smt systems. the promising results we report in this new line of research make it seem that learning more semantically motivated translation models might be less challenging than generally assumed and is worth exploring.