Cross-lingual Transfer Learning with P ersian (2023)

Abstract: The success of cross-lingual transfer learning for POS tagging has been shown to be strongly dependent, among other factors, on the (typological and/or genetic) similarity of the low-resource language used for testing and the language(s) used in pre-training or to fine-tune the model. We further unpack this finding in two directions by zooming in on a single language, namely Persian. First, still focusing on POS tagging we run an in-depth analysis of the behaviour of Persian with respect to closely related languages and languages that appear to benefit from cross-lingual transfer with Persian. To do so, we also use the World Atlas of Language Structures to determine which properties are shared between Persian and other languages included in the experiments. Based on our results, Persian seems to be a reasonable potential language for Kurmanji and Tagalog low-resource languages for other tasks as well. Second, we test whether previous findings also hold on a task other than POS tagging to pull apart the benefit of language similarity and the specific task for which such benefit has been shown to hold. We gather sentiment analysis datasets for 31 target languages and through a series of cross-lingual experiments analyse which languages most benefit from Persian as the source. The set of languages that benefit from Persian had very little overlap across the two tasks, suggesting a strong task-dependent component in the usefulness of language similarity in cross-lingual transfer.

## introduction and background cross-lingual transfer learning consists in using a (usually high resource) language for fine-tuning a pre-trained model for a given task, but then using such model to obtain predictions for a different (usually low-resourced) language. this is advantageous if the lesser-resourced language lacks enough resources for training. while in early work on transfer learning english has often been used as source language, due to its high availability, more recent research has shown that this might not be the optimal choice. for example, @xcite show that for pos tagging language similarity has a great impact on the success of transfer learning, and even with a small amount of data, one can achieve high accuracy. @xcite expands this study by doing cross-lingual transfer learning between over 100 languages, in search of good combinations of source and target languages. they find that there is no single language that is a good source language for cross-lingual transfer learning with all other languages. besides, the target language being included in the model pre-training is the most effective factor on performance of the model which does not play a role in low-resource settings. the next best predictor found for finding a good performing source-target language pair is the ldnd distance @xcite between them, considered as the language similarity measure. this measure is based on the levenshtein distance between a set of selected words in two languages. as a contribution to a better understanding of the properties of source and target languages towards successful transfer learning, and towards better processing for low-resource languages, we investigate cross-lingual transfer learning with a focus on persian. we analyze the results of @xcite experiments that include persian as either the source or the target language to find the languages that are a good match with persian for pos tagging. to explain the potential reasons for the results, we use the linguistic features from world atlas of language structures (wals). we also examine the performance of parsbert, the pre-trained monolingual persian model, in comparison to xlm-roberta, a pre-trained multilingual model, for the pos tagging task. finally, we investigate whether the language pairs with persian in the pos tagging are generalizable to other nlp tasks or not. we perform cross-lingual transfer learning for sentiment analysis as there is persian dataset available for this task and this task is a high-level nlp task compared to pos tagging as a low-level nlp task. this combination of tasks has been of interest for cross-lingual transfer learning in other studies as well @xcite . we gather sentiment analysis datasets from various resources and carry out experiments using the pre-trained multilingual xlm-roberta language model. we fine-tune this model using persian data and then test it with other languages. in the end, we compare the best target languages with persian as source in sentiment analysis and pos tagging. persian language is the official language of iran, afghanistan and tajikestan. the variety of persian in these countries is iranian persian (main and official variety of persian), dari, and tajik. the writing system of iranian persian and dari are the same, using persian alphabet, whereas, the tajik variety has a different writing system. figure 1 shows the geographical location of people whose mother tongue is persian. persian is an indo-european language, with a subject-object-verb word order, and it has words borrowed from french and english. additionally, its grammar is similar to many indo-european languages. but also it has many words in common with arabic, as iran has iraq as one of its neighbour countries and the official religious book for both countries is in arabic. considering iran's population of 85 million people, the number of persian speakers is considerably large. according to figure 2, persian speakers are widely spread around the world. these observations show the importance of research with persian language as it is used by a lot of people around the world, and it can result in applications benefiting a large group of people. 2 experimental setup 2.1 datasets for the pos tagging analysis, we a subset of the universal dependencies (ud) dataset as de vries et al. ( 2022 ) foot_0 that has a tag set of 17 tags. there are 105 languages in this dataset in total, all having at least 10 samples as test data that we consider as target language in our analysis. among them, 65 languages also have at least 25 samples as train data which we consider as source languages. we also obtained the accuracy scores of these 6825 different source-target language pairs using the xlm-roberta model from de @xcite . we use the ldnd distance measure between 90 different languages from the asjp database foot_1 (wichmann et al., 2022) as a measure of language similarity. in addition, we use the wals dataset (wal, 2013) foot_2 including 192 different phonological, grammatical, and lexical properties of 2 676 unique languages. the number of common linguistics features is the second language similarity measure that we use in our analysis. a multilingual sentiment analysis dataset containing all the languages that exist in ud dataset for pos tagging does not exist. the largest one that we found contains negative and positive tagged data including 23 languages foot_3 as follows: algerian, arabic, basque, bulgarian, cantonese, chinese, croatian, english, finnish, german, greek, hebrew, indonesian, japanese, korean, maltese, norwegian, russian, slovak, spanish, thai, turkish, and vietnamese. in addition, we gather data for 8 languages namely persian, urdu, hindi, welsh, polish, romanian, bambara, and uyghur from multiple resources, resulting in 31 languages in total. details about the datasets is provided in appendix a. we converted all of them to the same structure and only kept the positive and negative data entries foot_4 . 2022 ) fine-tuned the xlm-roberta model with a source language and then tested it with a target language. we focus on the result of experiments that have persian as the source or target language and attempt to find languages that result in a high score with persian in each scenario. we find the target languages that have persian as one of their top 10 source languages based on accuracy score. then, we consider persian as the target language, and find the source languages that have persian as one of their top 10 target languages. ## conclusion all in all, we analyse the result of previous experiments for pos tagging and investigate whether having persian as source or target language in cross-lingual transfer learning would be beneficial for persian and other languages. we observe that persian is the best source for itself as target and achieves a score of 91.43% for pos tagging. besides, it can serve as a good source for 6 lowresource languages. we use ldnd distance measure and linguistic features from wals to reason that persian can be a potential good source for kurmanji and tagalog for other tasks than pos tagging as well. lastly for pos tagging, we observe that parsbert outperforms xlm-robeta only for monolingual persian experiment and achieves a score of 96%. then, we gather data and perform sentiment analysis to investigate whether the same target languages found for pos tagging would also benefit from persian as the source language for sentiment analysis. we observe different target languages from the pos tagging results and only two languages: polish and bulgarian appear for both tasks. in addition, monolingual persian experiment does not achieve the highest accuracy and slovak is the best performing target. therefore, we conclude that cross-lingual transfer learning with persian is task dependent.