Investigating Zero-shot Cross-lingual Language Understanding for A rabic (2023)

Abstract: Numerous languages exhibit shared characteristics, especially in morphological features. For instance, Arabic and Russian both belong to the fusional language category. The question arises: Do such common traits influence language comprehension across diverse linguistic backgrounds? This study explores the possibility of transferring comprehension skills across languages to Arabic in a zero-shot scenario. Specifically, we demonstrate that training language models on other languages can enhance comprehension of Arabic, as evidenced by our evaluations in three key tasks: natural language inference, question answering, and named entity recognition. Our experiments reveal that certain morphologically rich languages (MRLs), such as Russian, display similarities to Arabic when assessed in a zero-shot context, particularly in tasks like question answering and natural language inference. However, this similarity is less pronounced in tasks like named entity recognition.

## introduction language models have been mainly utilized by training on a large corpus using a monolingual approach i.e. on a single language like bert @xcite , gpt @xcite , and roberta @xcite . on the other hand, there were some attempts to train such language models on multiple languages like multilingual bert (mbert) @xcite by combining text from different languages. a tokenizer such as wordpiece, is trained on the joined text from different languages to be able to recognize the scripts from such languages. this makes the vocabulary size of such models huge. for example, mbert has a shared vocabulary size of 110k across the 104 languages that were used for training compared to the 30k vocabulary size that was used to train the monolingual bert. with such a huge * corresponding author vocabulary and the number of languages, it is not clear how knowledge or language understanding is shared across such languages. more importantly, it is important to investigate how such knowledge is shared among similar languages, especially in terms of morphological features. we mainly focus on languages that exhibit rich morphology like arabic. in this study, our primary objective is to explore the integration of knowledge into arabic by fine-tuning mbert across various tasks, including question answering, natural language inference, and named entity recognition in multiple languages, followed by a zero-shot evaluation specifically on arabic. this paper is organized as follows. in section 2, we discuss the related studies to our work. in section 3, we focus on discussing the scope of our work. in sections 4 and 5, we discuss morphology in general and how it's an intrinsic property of arabic. in section 6, we investigate mbert and why it's an important model to evaluate such properties on. in section 7, we detail the datasets and tasks used for evaluating our study. finally, in section 8, we detail our experiments and discuss our results. ## related work multilinguality focuses on training language models with shared vocabulary for multiple languages. over the past few years, many models have adopted this strategy like multi-lingual bert (mbert) for 104 languages @xcite , xlm-r for 100 languages @xcite , and mt5 for 101 languages @xcite . the advantage of using such models is the simplicity of creating a shared vocabulary using a uniform linear mapping between the different multilingual embeddings. interestingly, mbert demonstrates proficiency in zero-shot cross-lingual model transfer, as observed in prior research @xcite . this capability aids in comprehending a given language in a universal context. however such models are required to be trained on a multilingual objective in order to generalize more for distant languages with different typography @xcite . not to mention how to transfer knowledge to low-resource languages. lately, there has been growing interest in utilizing more sophisticated architectures to enhance knowledge optimization in low-resource scenarios. one of the most interesting approaches is using adapter modules to avoid catastrophic forgetting 1 when training multilingual models on different languages. @xcite focused on creating a framework for multi-task adapter-based cross-lingual transfer. @xcite created a benchmark of the evaluation of cross-lingual transfer for 40 languages xtreme. in the literature, there were some limited efforts to apply zero-shot understanding for arabic. @xcite used self-training of pretrained language models for zero-and few-shot multi-dialectal arabic sequence labeling by first fine-tuning on modern standard arabic (msa). some studies focused on applying these techniques for arabic like gigabert which can achieve bilingual zero-shot understanding from english to arabic @xcite . they apply these methods 1 happens when the weights are fine-tuned on new datasets. the models usually forget the previous knowledge. for information extraction tasks (ie) like part of speech tagging (pos), named entity recognition (ner), relation extraction (re), and argument role labeling (arl) tasks. @xcite studied cross-lingual understanding from english and french to arabic. they show strong performance in a zero-shot setting despite the differences between the source and target languages in terms of morphology and grammar. there were many efforts also to benchmark chatgpt models in a zero-shot fashion on multiple tasks for arabic without fine-tuning @xcite , @xcite , @xcite , and @xcite . models like chatgpt which was trained on a large mixture of scripts for hundreds of languages were able to attain strong performance on multiple tasks in a zero-shot fashion. ## zero-shot evaluation the default approach of evaluating a language model on a given task is by training the model on that dataset and then evaluating on the unseen split of the dataset. we assume that both training and test splits belong to the same language/task. however, we can also argue that we can train the language model on a given task say t1 then evaluate on another task, say, t2. similarly, we can train a language model on a given language l1 and evaluate on another language l2 (see figure 1 ). however, in order to do that, the language model has to be able to predict or generate tokens in that language. hence, multilingual models have been utilized to evaluate cross-lingual understanding. such language models like mbert (discussed in section 6) are trained on a corpus that contains multiple languages. as a result, we can hypothesize that such language models have attained some kind of relationship across different languages either in terms of script or topology. as an example, arabic and persian have the same script and share many common words. more interestingly, using zero-shot evaluation we can test whether a given language is closer to other languages in terms of more complex features like morphology. for example, both arabic and russian are rich in terms of morphology and they are both inflectional languages. in this paper, we mainly focus on zero-shot evaluation on the same set of tasks but in different languages. to summarize, given a language l1 we train it on a given task t1 and zero-shot evaluate on l2 on the same task t1. in this paper, l2 is arabic, and l1 could be any language. ## morphology (arabic and beyond) tackling morphology is a very important step toward improving language modeling for languages like arabic. in the literature, @xcite showed slightly better results by pre-splitting words using the farasa segmentation tool @xcite on multiple tasks. the morphological seg-mentation results in better performance in text classification tasks while worse results in question answering and named entity recognition tasks. similarly, @xcite showed that we can get some improvement when we employ different morphological analyzers on top of neural and statistical models for machine translation. @xcite showed that by utilizing a combination of character-and word-level representations they achieved better results on the diacritization task. (alkaoud and syed, 2020) modified the tokenization algorithm for multilingual bert to achieve better results than monolingual bert on two different datasets. morphology also exists in other languages but with different levels of complexity depending on the language as shown in ## arabic's vocabulary sparsity vocabulary sparsity is the problem of having a very large vocabulary set with many different inflections. this presents a challenge for language modeling because typically, when designing a language model, we aim to acquire vocabulary embeddings. in this section, we analyze arabic morphology through vocabulary counting of parallel datasets. to conduct this experiment, we utilized the parallel dataset sourced from the united nations @xcite . this dataset comprises united nations general assembly resolutions in six distinct languages: arabic, chinese, english, ## multilingual bert bert @xcite is a transformer-based model that was trained on a large corpus using unsupervised learning. the main architecture of the model is based on the transformer model from @xcite which leverages attention to design an efficient encoder-decoder model that beats the existing machine translation models at that time. bert is trained using a concatenation of two objectives: using the concatenation of such objectives, the model can learn efficient text representation and can be fine-tuned on multiple tasks by attaching some uninitialized weights at the end of the model. the multilingual version of bert trains the model on a multilingual corpus that contains 104 languages. the initial training corpus was extracted from wikipedia with the top 100 languages then thai and mongolian were later added. this results in some languages which are under-represented. to mediate that, the authors used sampling to reduce the probability of training on high-resource languages like english and increase the probability of sampling from low-resource languages like icelandic. the base model used a shared vocabulary size of 110k which was extracted using the word-piece tokenization algorithm. similar to sampling, the word counts are multiplied with the sampling factor as in the training to reduce the effect of variation in the existence of different languages. ## tasks in this paper, we mainly focus on three tasks which are natural language inference, question answering, and named entity recognition. we use the datasets that have parallel sentences i.e. the same sentences are used for training the language models but in different languages. the reason for that choice is 1) we want the same amount of data in terms of height (number of samples) and roughly the same depth (number of tokens per sentence) and 2) we don't want to infuse any types of bias due to using different sentences for different languages i.e we want to force the model to use the knowledge in a similar setting to machine translation. we chose three datasets which are xnli for natural language inference, xquad for question answering, and massive for named entity recognition. here is a detailed explanation of each dataset. in ## results and discussions we fine-tune mbert foot_1 using the trainer class foot_2 which provides a simple way for training and finetuning transformer-based models. all the experiments were run using google colab foot_3 with the default virtual machine that contains a t4 nvidia card with 16 gb memory size. we use the py-torch examples from the transformer repository on github foot_4 with the following parameters for each task: in figure 4a , we show the results for the zeroshot evaluation on the natural language inference dataset xnli. english and russian achieve very similar results which approach the baseline for arabic. the german language also achieves somewhat close results to the baselines. urdu and thai both achieve the worst results for natural language inference which are close to 50 % accuracy which is much lower than chinese. although chinese and thai are similar in pronunciation and other grammatical features, they belong to different lan-guage families. note that this is just based on the 50,000 samples used for training. increasing the training samples might result in different results, especially for the languages that are close in results. furthermore, the results approach the baseline for fine-tuning and evaluating on arabic. this might be the effect of using a machine-translated dataset for evaluation. in figure 4b , we show the results for the zeroshot evaluation on the question-answering dataset xquad. the dashed lines show the results of the baselines after training and evaluating on arabic for exact match and f1 scores. we notice that the russian language achieves the best scores for both the exact match and f1 scores. these results correlate with the initial experiments in section 5. followed by the romanian language which seems quite close to russian in terms of structure. the thai language achieves the worst scores across all metrics which might be related to the structure of the language which is quite close to chinese which does not use white-space tokenization. in figure 4c , we present the outcomes of the zero-shot evaluation conducted on the named entity recognition task. overall, it is discernible that the outcomes, particularly the f1 scores, exhibit notable decrements when compared to the baseline, specifically within the arabic language context. this discrepancy could be attributed to the dataset's substantial entity count, exceeding 100. consequently, this abundance of entities introduces a degree of stochasticity into the cross-lingual comprehension process, complicating the derivation of definitive insights from the results. ## conclusion in this research, we delved into the realm of crosslingual zero-shot transfer, where we explored the application of knowledge from various languages to arabic through the evaluation of multiple tasks. these tasks encompassed named entity recognition, natural language inference, and question answering. initially, we employed an unsupervised approach to scrutinize the distinctions in morphology between arabic and other languages. subsequently, by employing supervised methods, we revealed certain connections between arabic and other languages concerning their structure and writing systems. our investigation demonstrated that superior results can be achieved by training models on languages other than arabic and subsequently assessing their performance on arabic, as opposed to direct training on arabic. this phenomenon may be attributed to several factors, including the simplicity of the language, the resemblance of these languages to arabic, and the distribution of the initial training data used for unsupervised learning. as a future direction, it could be interesting to look into more diverse tasks and more advanced transformer-based architectures.