Assessing the Linguistic Knowledge in A rabic Pre-trained Language Models Using Minimal Pairs (2022)

Abstract: Despite the noticeable progress that we recently witnessed in Arabic pre-trained language models (PLMs), the linguistic knowledge captured by these models remains unclear. In this paper, we conducted a study to evaluate available Arabic PLMs in terms of their linguistic knowledge. BERT-based language models (LMs) are evaluated using Minimum Pairs (MP), where each pair represents a grammatical sentence and its contradictory counterpart. MPs isolate specific linguistic knowledge to test the model’s sensitivity in understanding a specific linguistic phenomenon. We cover nine major Arabic phenomena: Verbal sentences, Nominal sentences, Adjective Modification, and Idafa construction. The experiments compared the results of fifteen Arabic BERT-based PLMs. Overall, among all tested models, CAMeL-CA outperformed the other PLMs by achieving the highest overall accuracy.

## introduction recently, tremendous pre-trained neural network models existed and are used effectively in different natural language processing (nlp) tasks. this renaissance began roughly when google launched the transformers architecture in 2017 @xcite . furthermore, different models are developed after the transformers, such as generative pre-training (gpt) @xcite , gpt-2 @xcite , gpt-3 @xcite , and bidirectional encoder representations from transformers (bert) @xcite . these models have proved their strength in many nlp tasks, such as machine translation, summarization, and sentiment analysis. in 2019, several attempts appeared to train bert models, specifically for the arabic language. arabert was one of the first arabic models that aimed to contribute to arabic nlp in three different tasks: sentiment analysis (sa), named entity recognition (ner), and question answering (qa) @xcite . furthermore, the end of 2020 has witnessed a race where several arabic models were published, namely: arabic-bert @xcite , gigabert @xcite , arbert, and marbert @xcite . also, 2021 was no less intense; several versions of arabert models @xcite , as well as araelectra @xcite , and qarib @xcite , were published. despite these developments, the vision remains blurred in terms of how these models analyze the language in regard to various linguistic phenomena such as syntax, semantics, and grammar, which is an open area for research. evaluating the linguistic knowledge of plms has gained popularity recently. therefore, numerous methods were developed to test the model's linguistic competence and the acquisition of different linguistic phenomena. humans develop the ability to distinguish between grammatical and ungrammatical sentences while they are growing. thus, studies showed that plms could mimic human ability, whereas, despite having no formal grammar training, the models can distinguish between grammatical and ungrammatical sentences @xcite . many of these studies are specified for exploring the models' linguistic knowledge in the english language @xcite and chinese language @xcite . nevertheless, to the best of our knowledge, no study has been devoted to understanding the linguistic knowledge of arabic pre-trained language models. plms, such as bert, assign a probability/score to a sequence of words @xcite . many studies have used these scores to rank a sentence's correctness and evaluate the models' knowledge @xcite . a common method to evaluate the model's linguistic knowledge is minimal pairs (mp). mp is a set of two-sentence pairs (grammatical and ungrammatical) that is used to test the model's preferences among them. assigning a higher score for the grammatical sentence from the mp pair verifies the model's understanding of a specific phenomenon. each pair of sentences provided by mp minimally differs by changing one word only. this change should ensure that the grammatical rule is contrasted, whereas, the grammatical and ungrammatical sentences are balanced. example 1 illustrates a pair of mp sentences where we provided two verbal sentences; the first one is based on correct arabic language grammar where the verb agrees with the subject in gender. the second sentence of example 1 presents a contrast for the rule, as the verb does not agree in gender with the subject. furthermore, another example of arabic language grammar is presented in example 2. the first sentence in example 2 (correct) provides a verb that does not agree with the subject in number, while the second sentence contrasts the rule. as we noticed from the examples, mps are used to prompt the analysis and subsequent improvements of plms @xcite @xcite . in addition, each pair isolates a specific phenomenon, allowing the plm to be tested separately for each linguistic phenomenon. example 1: ‫ﯾ‬ ‫ﺰ‬ ‫ر‬ ‫ع‬ ‫ا‬ ‫ﻟ‬ ‫ﻔ‬ ‫ﻼ‬ ‫ح‬ ‫ا‬ ‫ﻟ‬ ‫ﺸ‬ ‫ﺠ‬ ‫ﺮ‬ ‫ة‬ ) ‫ﺟ‬ ‫ﻤ‬ ‫ﻠ‬ ‫ﺔ‬ ‫ﺻ‬ ‫ﺤ‬ ‫ﯿ‬ ‫ﺤ‬ ‫ﺔ‬ ( yzre alflah al@xmath0jrp the farmer (male) plants (feminine verb) the tree (ungrammatical) example 2: ‫ذ‬ ‫ھ‬ ‫ﺐ‬ ‫ا‬ ‫ﻟ‬ ‫ﻮ‬ ‫ﻟ‬ ‫ﺪ‬ ‫ا‬ ‫ن‬ ‫ﺑ‬ ‫ﺮ‬ ‫ﺣ‬ ‫ﻠ‬ ‫ﺔ‬ ) ‫ﺟ‬ ‫ﻤ‬ ‫ﻠ‬ ‫ﺔ‬ ‫ﺻ‬ ‫ﺤ‬ ‫ﯿ‬ ‫ﺤ‬ ‫ﺔ‬ ( *hb alwldan brhlp the boys went (single) on a trip (grammatical) ‫ذ‬ ‫ھ‬ ‫ﺒ‬ ‫ﺎ‬ ‫ا‬ ‫ﻟ‬ ‫ﻮ‬ ‫ﻟ‬ ‫ﺪ‬ ‫ا‬ ‫ن‬ ‫ﺑ‬ ‫ﺮ‬ ‫ﺣ‬ ‫ﻠ‬ ‫ﺔ‬ ) ‫ﺟ‬ ‫ﻤ‬ ‫ﻠ‬ ‫ﺔ‬ ‫ﺧ‬ ‫ﺎ‬ ‫ط‬ ‫ﺌ‬ ‫ﺔ‬ ( *hba alwldan brhlp 1 https://github.com/wafa7d/assessingarab icberts the boys went (dual) on a trip (ungrammatical) in this study, we introduce a handcrafted arabic minimal pair mps consisting of around 3000 sentences 1 . as each mp contains both grammatical and ungrammatical sentences, the dataset is balanced and written in modern standard arabic (msa). moreover, since the arabic language is extensive and complex, we limited this study to cover nine basic arabic syntactic, semantic, and grammatical phenomena, including: verbal sentence, nominal sentence, adjective modification, and idafa construction. fifteen bert-based lms were tested using the models' sensitivity to detect the grammatical contrast. therefore, our contributions in this paper can be listed as follows: ## arabic linguistics arabic is a distinctive language with unique characteristics, rich morphology, and free word ordering @xcite . the arabic sentence is divided into two types: the verbal sentence and the nominal sentence. for each type, there are several forms that the sentence can take and remain linguistically correct. the following subsections cover a summary of these primary forms. additionally, the relationship between nouns, case assignment, gender, and number agreement in the sentence structure are also covered. ## the evolution of arabic plms chronologically, the first multilingual bert model that supported the arabic language appeared in 2019; it was mxbert @xcite . it was followed by the first monolingual arabic model, i.e., arabert @xcite , which appeared in early 2020. figure 1 illustrates the evolution of arabic plms, showing their parameter sizes and existence order. moreover, ## method in the following subsections, we precisely describe the data coverage and the conducted experiment. ## results and discussion we evaluated each model using the accuracy metric. as shown in equation 1, the accuracy is the fraction of examples for which the model assigns a relatively higher probability for the correct sentence. ## conclusion this paper aims to comprehend the linguistic abilities conferred by arabic plms. we present a study to understand the basic grammar concepts obtained by the current bert-based arabic plms using mps. each mp represents a distinct phenomenon; hence, it can reflect the model understanding to that phenomenon. therefore, utilizing the grammatical/ungrammatical pairs of mps, it is feasible to assess how well the model comprehends a particular phenomenon by assigning it a higher probability to the grammatical sentence. the experiments include evaluating nine basic arabic phenomena on fifteen bert-based arabic plms. the findings indicate a clear lack of plms' understanding of most of the evaluated arabic phenomena. however, the highest average accuracy was achieved by camel-ca and gigabert reaching 55.1%, with camel-ca outperforming in three linguistic phenomena. it is worth mentioning that camel-ca has used classical arabic in its pre-training process, which justifies its high scores in our evaluation. finally, the capacities targeted by our experiments are not exhaustive. future research can build on this paper's findings to study other linguistic aspects of arabic plms in depth and include other models.