Role of L anguage R elatedness in M ultilingual F ine-tuning of L anguage M odels: A C ase S tudy in I ndo- A ryan L anguages (2021)

Abstract: We explore the impact of leveraging the relatedness of languages that belong to the same family in NLP models using multilingual fine-tuning. We hypothesize and validate that multilingual fine-tuning of pre-trained language models can yield better performance on downstream NLP applications, compared to models fine-tuned on individual languages. A first of its kind detailed study is presented to track performance change as languages are added to a base language in a graded and greedy (in the sense of best boost of performance) manner; which reveals that careful selection of subset of related languages can significantly improve performance than utilizing all related languages. The Indo-Aryan (IA) language family is chosen for the study, the exact languages being Bengali, Gujarati, Hindi, Marathi, Oriya, Punjabi and Urdu. The script barrier is crossed by simple rule-based transliteration of the text of all languages to Devanagari. Experiments are performed on mBERT, IndicBERT, MuRIL and two RoBERTa-based LMs, the last two being pre-trained by us. Low resource languages, such as Oriya and Punjabi, are found to be the largest beneficiaries of multilingual fine-tuning. Textual Entailment, Entity Classification, Section Title Prediction, tasks of IndicGLUE and POS tagging form our test bed. Compared to monolingual fine tuning we get relative performance improvement of up to 150% in the downstream tasks. The surprise take-away is that for any language there is a particular combination of other languages which yields the best performance, and any additional language is in fact detrimental.

## introduction 1 transformer-based @xcite language models (lm) have been proven to be extremely useful in variety of natural language processing (nlp) tasks. some of the most notable models are gpt @xcite , gpt-2 @xcite , @xcite , bert @xcite , roberta @xcite , xlnet @xcite , and bart @xcite . to fine-tune a pre-trained language model for downstream tasks has become a de facto approach in recent literature. we empirically study whether (and to what extent) do the related languages accentuate the performance of models in downstream tasks with multilingual fine-tuning foot_0 in comparison to monolingual fine-tuning. to understand the quantitative advantage of including languages gradually, we explore the gradation of multilinguality by incrementally adding in new languages added one-by-one building up to an all-language multilingual fine-tuning. a good approximation for language relatedness is their membership to the same language family as languages of a family often share properties such as grammar, vocabulary, etymology, and writing systems. we choose the indo-aryan (ia) family for the study, since constituent languages 1) include low-resource languages, 2) have similar abugida writing system, 3) are relatively understudied, and 4) are covered in a well-defined nlp benchmark indicglue @xcite . further, the fact that all constituent languages except one use similar abugida writing systems (rooted in the ancient brahmi script foot_1 ) presents an opportunity for script normalization via transliteration. overall, although the general notion of language relatedness is explored, and the multilingual fine-tuning is explored in literature, the detailed linguistic understanding of role of language relatedness in multilingual fine-tuning remains understudied; even more so for ia family. further, in this context the script-conversion aspect is not explored for multilingual fine-tuning. to summarize, in this paper we seek to answer the following research questions (employing the indo-aryan language family as the experimental test-bed). these inquiries are critical to understanding the right balance between per-language fine-tuning and massively multilingual fine-tuning, as the viable way forward. additionally, we also explore the role of common script representation in multilingual ft of related languages. to facilitate these inquiries, we utilize existing pre-trained models, namely indicbert, mbert, and muril, and also pre-train two language models for indo-aryan language family from scratch. we utilize various tasks of indicglue @xcite as our test-beds. ## related work multilinguality aspect has been explored in context of pre-training language models, for effective transfer from one language to other, and in multilingual fine-tuning, to an extent. ## methodology traditionally, a pre-trained lm (such as mbert) is used as base model, which is fine-tuned for a downstream task for a specific language (monolingual). in this work, we aim to evaluate the role of script and language relatedness in multilingual fine-tuning by employing the indo-aryan language family. therefore, we include the following components to the approach: (1) multilingual fine-tuning, (2) transliteration, and (3) language models. next, we discuss these in detail. ## experimental setup in this section, we describe the datasets used in our experiments, their pre-processing, and implementation details. ## experiments to answer the research questions, we experiment on a variety of tasks suitable for multilingual fine-tuning and analyse the results. to investigate rq1, in §5.1, the first set of experiments are aimed to understand the utility of multilingual ft with related languages. to investigate rq2, in §5.2, the second set of experiments are designed to track gradual performance variation with addition of assisting languages. with last set of analysis, in §5.3, we investigate the role of transliteration. ## conclusion we show that multilingual fine-tuning efficiently leverages language relatedness leading to improvements over monolingual approach. we substantiate this claim on the indo-aryan language family with experiments on five language models. multilingual fine-tuning is particularly effective for low-resource languages (e.g., oriya and punjabi show improvement up to 150% on relative scale). also, we show that careful selection of subset of related languages, can further improve performance. devising automatic approaches for finding optimal subset of related languages is a promising future direction. additionally, in multilingual fine-tuning, we see some benefits of transliteration to common script.