Are There Any Limits to E nglish- S wedish Language Transfer? A Fine-grained Analysis Using Natural Language Inference (2023)

Abstract: The developments of deep learning in natural language processing (NLP) in recent years have resulted in an unprecedented amount of computational power and data required to train state-of-the-art NLP models. This makes lower-resource languages, such as Swedish, increasingly more reliant on language transfer effects from English since they do not have enough data to train separate monolingual models. In this study, we investigate whether there is any potential loss in English-Swedish language transfer by evaluating two types of language transfer on the GLUE/SweDiagnostics datasets and comparing between different linguistic phenomena. The results show that for an approach using machine translation for training there is no considerable loss in overall performance nor by any particular linguistic phenomena, while relying on pre-training of a multilingual model results in considerable loss in performance. This raises questions about the role of machine translation and the use of natural language inference (NLI) as well as parallel corpora for measuring English-Swedish language transfer.

## introduction the leveraging of knowledge transfer has been a pivotal development in deep learning and in natural language processing (nlp). the core idea is that the knowledge of one source domain can be transferred to another target domain. in the age of deep learning, this usually means training models on large quantities of data on some generic or widely-applied task, such as language modeling or machine translation, and fine-tuning the model on new data or another downstream task. this does not only lead to an immediate performance boost on the new dataset or downstream task, but also saves computational resources of the user as well as simplifies the implementation and problem solving procedure. a crucial type of knowledge transfer in nlp is that of language transfer, which refers to the leveraging of high-resource languages, most notably english, to solve tasks for lower-resource languages. it has been studied in multiple different architectures and applications of nlp, such as in word embedding architectures and machine translation. the most used type of language transfer, however, is arguably that of pretrained language models, which generic pre-training allows for finetuning a wide range of down-stream tasks (both in the areas of text generation and text classification). as deep learning has scaled up in recent years as a result of more computationally efficient hardware and model architectures, most notably with the release of the transformer @xcite , so has the data needed for training. for lower-resource languages these developments come in the shape of a double-edged sword: on the one hand, the breadth of data and contextual information encoded in these models enables a high level of knowledge transfer, on the other hand, they become even more reliant on high-resource languages like english to use the most recent stateof-the-art language models. this leaves lowerresource languages the options of either opting for smaller models and, thus, missing out on the latest progress or to rely heavily on language transfer. seeing that the latter option seems most plausible, this raises the question to what limits there are to language transfer from a linguistic perspective, but also in terms of potential dangers of political, gender and cultural biases @xcite . in this study, we focus on english-swedish language transfer. swedish, a mid-resource language, is an interesting example of a language becoming more reliant on language transfer. this is because swedish has had enough available pre-training data to train monolingual language models, like bert @xcite , but not enough for the most recent gpt-3 model @xcite . also, a significant amount of down-stream tasks in english are not trainable in a supervisable way in swedish because corresponding annotated data do not exist for the language. in order to test the potential limits of english-swedish language transfer, we use the glue/swediagnostics parallel corpora to compare two different types of language transfer. in short, we aim to answer the following research questions: q1: is there a loss in performance in english-swedish language transfer in the context of natural language inference (nli)? q2: which type of language transfer works best (machine translation or multilingual pre-training)? q3: are there linguistic phenomena for which language transfer works less effectively? with this work, we aim to shed light on the english-swedish language transfer capabilities of pretrained language models and, thus, provide direction for applying and evaluating language transfer in the future. ## related work this work focuses on the english-swedish language transfer of pretrained language models in which nli is used as the measurement. as such, this section will focus on work relating to language transfer and nli. ## datasets & models for the following study, we use a collection of different nli datasets and pretrained language models to assess language transfer capabilities from english to swedish. we use the datasets mnli and snli for training, which due to their unmatched size are most suitable for fine-tuning large pretrained language models. for testing, we use the glue/swediagnostic parallel corpora, which we use to both evaluate performance as well as to make fine-grained analysis of specific language phenomena. when it comes to choosing models, we select them based on the criteria that their architectures are 1) available in both languages and 2) directly comparable in terms of architectural size. for these purposes, the bert-model is naturally favored since its base-version is available in both swedish and english (see section 3.2). the following subsections give a closer overview of the datasets (section 3.1) and models (section 3.2). ## method our methodology consists of firstly training different bert models using two types of language transfer (section 4.1) and, secondly, evaluating their performance on the glue/swediagnostics parallel corpora (section 4.1). 2 ## discussion the results show that even for a high-level reasoning task such as nli, english-swedish language transfer can be made without any considerable loss in performance (q1). this finding is further solidified when comparing by different linguistic phenomena where the performance is comparable to an equivalent english baseline (q3). when comparing different types of language transfer, we see that that fine-tuning on english-swedish machine translated data performs markedly better than relying only on language transfer from multilingual pre-training. thus, it confirms the findings of @xcite that leveraging machine translation between english and swedish is an effective method of language transfer (q2). the high similarity in model performance and prediction agreement suggests that the information used by the bert model to predict inference, is not lost after english-swedish machine translation. conversely, this also suggests that whatever language information is lost after machine translation, it is not essential for predicting inference. this further highlights, as previous studies have, the brittleness of nli as a measurement of natural language understanding. if we were to assume that (a) the model uses logical reasoning to solve nli and (b) important entailment information is lost after machine translation, then the performance after language transfer should vary considerably. however, since that is not the case, it seems more plausible that the model uses other textual artifacts to predict inference. alternatively, the entailment relation between the hypothesis and premise does not change considerably after machine translation. since we do not know, however, the extent to which entailment relations hold after machine translation, we cannot know for certain and, thereby, could be a question for future studies to explore. given the observations above, it is also important to take into account that the results of this study are only as generalizeable as the peculiarities of the glue/swediagnostic dataset. its relatively small size (1106 sentence pairs), choice of genres, linguistic phenomena and annotation procedure might not generalize to all cases of nli language transfer. in particular, since swediagnostics was manually translated into swedish from english, it most likely has translationese (gellerstam, 1986) elements in it and might, thus, naturally be biased towards machine translation output. furthermore, translating premise and hypothesis independently, has been shown to reduces lexical overlap between the sentences @xcite , which could help the model not overfit on spurious annotation artefacts. until a unique nli dataset for swedish is created, which samples are taken from naturally occurring spoken or written swedish, we cannot know the extent to which this impacts the results. ## conclusion in this study, we show that for the high-level reasoning task of nli, english-swedish language transfer can be done without any considerable loss in performance. we also see that for a model which uses machine translation for training, there is no considerable loss by any specific linguistic phenomenon. meanwhile, a multilingual model which only relies on pre-training for language transfer does not see the same level of language transfer. given the increasing reliance on english-swedish language transfer as a result of the development towards larger models with need for more training data in nlp, we see a need for further studies into the potential effects of language transfer on swedish. in this effort, understanding the role of english-swedish translation as well as comparing these results to datasets that are based on naturally occurring written or spoken swedish, will be essential to understand the true impact of english linguistic and cultural influences on english-swedish language transfer. finally, applying similar studies to newer and larger pretrained language models, such as gpt-sw3, will become even more important as they will be used more broadly in the future.