Encodings of Source Syntax: Similarities in NMT Representations Across Target Languages (2020)

Abstract: We train neural machine translation (NMT) models from English to six target languages, using NMT encoder representations to predict ancestor constituent labels of source language words. We find that NMT encoders learn similar source syntax regardless of NMT target language, relying on explicit morphosyntactic cues to extract syntactic features from source sentences. Furthermore, the NMT encoders outperform RNNs trained directly on several of the constituent label prediction tasks, suggesting that NMT encoder representations can be used effectively for natural language tasks involving syntax. However, both the NMT encoders and the directly-trained RNNs learn substantially different syntactic information from a probabilistic context-free grammar (PCFG) parser. Despite lower overall accuracy scores, the PCFG often performs well on sentences for which the RNN-based models perform poorly, suggesting that RNN architectures are constrained in the types of syntax they can learn.

## introduction neural machine translation (nmt) encoder representations have been used successfully for crosstask and cross-lingual transfer learning in a variety of natural language contexts @xcite @xcite . previous work has investigated whether these representations encode syntactic information @xcite , as syntactic information is useful in many natural language tasks @xcite . the deep recurrent neural network (rnn) architectures used by many nmt encoders can learn syntactic features, even without explicit supervision @xcite ; nmt encoders specifically have been found to encode information about ancestor constituent labels for words @xcite and even full syntactic parses of source language sentences @xcite . cross-linguistically, there is mixed evidence for how target language impacts the encoding of information in nmt encoder representations. @xcite found that representations clustered based on target language family when sentence representations were aligned in a shared space. however, @xcite found only small effects of target language on the ability of nmt encoder states to predict part-of-speech (pos) tags. because pos tags are typically reliant only on local features within sentences, these contrasting results could suggest that (1) localized encoded information is independent of nmt target language, or (2) encoded syntactic information in general is independent of nmt target language. in this work, we address the second possibility. to evaluate more global syntactic information in nmt encoder representations, we assess the ability of nmt encoder states to predict ancestor constituent labels of words; this task is adopted from @xcite . extending @xcite , we train nmt models towards multiple target languages and evaluate performance on individual constituent labels (e.g. noun phrases). we find that significant syntactic information is encoded regardless of target language, and target language has little impact on the syntactic information learned by nmt encoders. furthermore, we find that nmt encoders rely on explicit morphosyntactic cues to extract syntactic information from sentences. finally, by training deep rnns directly on the constituent label prediction task, we find that rnns with explicit syntactic training data learn similar syntax to the nmt encoders. in contrast, a probabilistic context-free grammar (pcfg) parser performs significantly differently from both rnnbased models, suggesting that rnns may be constrained by their reliance on explicit syntactic cues. ## methodology we trained nmt models from english to six different target languages, assessing the ability of nmt encoder states to predict pos, parent, grandparent, and great-grandparent constituent labels of words. ## nmt syntax vs. other models the similarity of syntactic information in nmt encoder representations across target languages could suggest that regardless of target language, a similar amount of syntactic information is helpful for translation. however, it is also possible that the structure of the constituent label task limited the syntactic information the encoders could represent, as predicting a label based only on a partial sentence is an inherently ambiguous task. a third alternative is that the rnn encoder architectures limited the information preserved in each representation. to further explore how well the nmt encoders extracted syntactic information from raw sentences, we compared their constituent label prediction performance to two alternative models: an rnn trained directly for the constituent label task, and a probabilistic context-free grammar (pcfg) parser. in contrast to the nmt encoders, the rnn can learn representations that are best suited for retaining syntax; like the nmt encoders, it sees one word at a time. the pcfg is trained with complete syntactic information for partial sentences, and its prediction task is an entire hierarchical structure, rather than a single type of label. these comparisons can show whether there are syntactic features that are predictable but systematically missed by the nmt encoder representations. ## conclusion in this work, we found that nmt encoder representations across target languages encode similar source syntax, and this syntax is comparable to the syntax learned by rnns trained directly on syntactic tasks. however, explicit syntactic architectures may be necessary for tasks requiring fine-tuned syntactic parses. our results have many implications in transfer learning and multilingual sentence representations: a better understanding of the information contained in sentence representations provides necessary insight into the tasks these representations can be used for.