Machine translation systems require semantic knowledge and grammatical understanding. Neural machine translation (NMT) systems often assume this information is captured by an attention mechanism and a decoder that ensures fluency. Recent work has shown that incorporating explicit syntax alleviates the burden of modeling both types of knowledge. However, requiring parses is expensive and does not explore the question of what syntax a model needs during translation. To address both of these issues we introduce a model that simultaneously translates while inducing dependency trees. In this way, we leverage the benefits of structure while investigating what syntax NMT must induce to maximize performance. We show that our dependency trees are 1. language pair dependent and 2. improve translation quality. Language has syntactic structure and translation models need to understand grammatical dependencies to resolve the semantics of a sentence and preserve agreement (e.g., number, gender, etc). Many current approaches to MT have been able to avoid explicitly providing structural information by relying on advances in sequence to sequence (seq2seq) models. The most famous advances include attention mechanisms (Bahdanau et al., 2015) and gating in Long Short-Term Memory (LSTM) cells (Hochreiter and Schmidhuber, 1997) . In this work we aim to benefit from syntactic structure, without providing it to the model, and to disentangle the semantic and syntactic compoThe boy sitting next to the girls ordered a coffee Figure 1 : Our model aims to capture both: syntactic (verb ordered → subj/obj boy, coffee) alignment (noun girls → determiner the) attention. nents of translation, by introducing a gating mechanism which controls when syntax should be used. Consider the process of translating the sentence "The boy sitting next to the girls ordered a coffee." (Figure 1 ) from English to German. In German, translating ordered, requires knowledge of its subject boy to correctly predict the verb's number bestellte instead of bestellten. This is a case where syntactic agreement requires long-distance information. On the other hand, next can be translated in isolation. The model should uncover these relationships and decide when and which aspects of syntax are necessary. While in principle decoders can utilize previously predicted words (e.g., the translation of boy) to reason about subject-verb agreement, in practice LSTMs still struggle with long-distance dependencies. Moreover, Belinkov et al. (2017) showed that using attention reduces the decoder's capacity to learn target side syntax. In addition to demonstrating improvements in translation quality, we are also interested in analyzing the predicted dependency trees discovered by our models. Recent work has begun analyzing task-specific latent trees (Williams et al., 2018) . We present the first results on learning latent trees with a joint syntactic-semantic objective. We do this in the service of machine translation which inherently requires access to both aspects of a sentence. Further, our results indicate that language pairs with rich morphology require and therefore
REF propose an NMT model that uses a self-attention encoder to induce latent dependency trees on the source side and report that the induced trees are task-specific and do not conform to the conventional definition of syntax.
Inducing Grammars with and for Neural Machine Translation
ArXiv
ArXiv
Computer Science