Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks (2021)

Abstract: It is now established that modern neural language models can be successfully trained on multiple languages simultaneously without changes to the underlying architecture, providing an easy way to adapt a variety of NLP models to low-resource languages. But what kind of knowledge is really shared among languages within these models? Does multilingual training mostly lead to an alignment of the lexical representation spaces or does it also enable the sharing of purely grammatical knowledge? In this paper we dissect different forms of cross-lingual transfer and look for its most determining factors, using a variety of models and probing tasks. We find that exposing our LMs to a related language does not always increase grammatical knowledge in the target language, and that optimal conditions for lexical-semantic transfer may not be optimal for syntactic transfer.

## introduction one of the most important nlp discoveries of the past few years has been that a single neural network can be successfully trained to perform a given nlp task in multiple languages without architectural changes compared to monolingual models @xcite . besides important practical advantages (fewer parameters and models to maintain), such multilingual neural networks (mnns) provide an easy but powerful way to transfer taskspecific knowledge from high-to low-resource languages @xcite @xcite @xcite @xcite . these success stories have led to a need for understanding how exactly cross-lingual transfer works within these models. figure 1 illustrates different possible characterizations of a trained mnn: while the no-transfer scenario is rather easy to rule out, understanding which linguistic categories are shared, and to what extent, is more challenging. in this work, we focus on the transfer of syntactic knowledge among languages and look for evidence that mnns induce a shared syntactic representation space while not receiving any direct cross-lingual supervision. to be clear, if we measure transfer among languages x and y, every training sentence for language modeling will be either in language x or y, while for machine translation every sentence pair will be either in language pair (x, z) or (y, z). thus, the only pressure to share linguistic representations is given by the sharing of the hidden layer parameters (as well as, possibly, some of the word embeddings). no transfer shallow transfer deep transfer compression of separate language understanding/ generation systems compression of (mostly) separate systems + some "information leakage" figure 1: possible characterizations of a trained mnn in terms of cross-lingual transfer levels. neural language models have been shown to implicitly capture non-trivial structure-sensitive phenomena like long-range number agreement @xcite @xcite . however most of these studies have been confined to monolingual models. we then investigate the following questions: our research questions are reminiscent of wellknown questions in the fields of psycholinguistic and second language acquisition, where work has shown that both lexical and syntactic representations are shared in the mind of bilinguals @xcite . taking inspiration from this body of work, we investigate what factors are needed for mnns to successfully transfer linguistic knowledge, including vocabulary overlap, language relatedness, number of training languages, training regime (joint vs sequential) and training objective (next word prediction vs translation to a third language). in contrast to the current mainstream focus on bert-like models @xcite , we evaluate more classical lstm-based models trained for next word prediction or translation over a moderate number of languages (2 or 9). we choose this setup because (i) it allows for more controlled and easy-to-replicate experiments in terms of both training data and model configuration and (ii) lstms trained on a standard sequence prediction objective are more cognitively plausible and directly applicable to our main probing task, namely agreement prediction. in this setup, we find limited and rather inconsistent evidence for the transfer of implicit grammatical knowledge when semantic cues are removed @xcite . while moderate pos category transfer occurs, truly language-agnostic syntactic categories (such as noun or subject) do not seem to emerge in our mnn representations. finally, we find that optimal conditions for lexical-semantic transfer may not be optimal for syntactic transfer. ## previous work multilingual machine translation early work on multilingual nmt focused on building dedicated architectures @xcite @xcite . starting from @xcite , m-nmt models are mostly built with the same architecture as their monolingual counterparts, by simply adding language identifying tags to the training sentences. using a small set of english sentences and their japanese and korean translations, @xcite showed that semantically equivalent sentences form welldefined clusters in the high-dimensional space induced by a nmt encoder trained on large-scale proprietary datasets. @xcite analyze the similarity of encoder representations of different languages within a massively m-nmt model. they find that representation similarity correlates strongly with linguistic similarity and that encoder representations diverge based on the target language. however they do not disentangle the syntactic aspect from other types of transfer. ## probing tasks to answer our rq1 (are mnns capable of implicitly transferring syntactic knowledge between languages?) we choose the task of number agreement. for our rq2 (are mnns able to induce a common representation with shared syntactic categories?) we look at less complex syntactic tasks such as pos tag classification and dependency relation classification, and contrast them with a lexical-semantic task (word translation retrieval). we choose these tasks because they can be framed as simple classification (or ranking) problems and have a direct linguistic interpretation. we do not consider parsing because it is a complex task with a highly structured prediction space requiring dedicated model components. the probed models are lstm-based language models and translation models, trained at the word-level. more details are provided below. ## does exposure to l1 improve implicit syntactic knowledge on a related l2? to answer rq1 we use the number agreement task, which is explained in detail in sect. 3.1. we choose italian (it) and russian (ru) from the g18 dataset as our target languages l2. as helper languages, l1, we choose french (fr) and spanish (es) for l2 it, and fr and ukrainian (uk) for l2 ru, which allows us to study the impact of language relatedness. accuracy is calculated as follows: for each sentence in the l2 benchmark, if the probability of the correct verb form is higher than the incorrect form, the agreement is said to be correct, and incorrect otherwise. ## conclusions we have presented an in-depth analysis of various factors affecting cross-lingual syntactic transfer within multilingually trained lstm-based language (and translation) models. our main result is a negative one: transfer of purely grammatical knowledge (specifically long-range agreement in nonce sentences) is very limited in general -confirming recent findings by @xcite -and strongly dependent on the specific choice of source-target languages. namely, small gains were only reported on @xmath0, while a considerable drop was reported on @xmath1. when semantic cues were not removed (original sentences), transfer levels were overall higher with a peak of +7% absolute in @xmath2, but @xmath3. while es is arguably closer to it than fr, we cannot yet find a convincing linguistic explanation for the large differences observed. our second set of experiments shows that pos categories are shared to a moderate extent, but dependency categories are not shared at all in our models. this suggests that syntactic knowledge transfer within our multilingual models is rather shallow, and may explain the lack of agreement transfer. our experiments with different training objectives and number of input languages show that what are optimal conditions for the alignment of word embedding spaces (lexical-semantic transfer) may not be optimal for syntactic transfer, and vice versa. language relatedness is by far the most determining factor for both word embedding alignment and pos transfer. and finally, scaling from two languages to a mix of nine languages from three different families results in better pos transfer between related languages but considerably worse between unrelated ones. together with the findings by @xcite , our results suggest that scaling to highly multilingual models may improve syntactic transfer among the most related languages by decreasing the per-language capacity, but may also exacerbate the divergence among less related ones. thus modern multilingual nns appear still far from acquiring a true interlingua.