Automatic Difficulty Classification of A rabic Sentences (2021)

Abstract: In this paper, we present a Modern Standard Arabic (MSA) Sentence difficulty classifier, which predicts the difficulty of sentences for language learners using either the CEFR proficiency levels or the binary classification as simple or complex. We compare the use of sentence embeddings of different kinds (fastText, mBERT , XLM-R and Arabic-BERT), as well as traditional language features such as POS tags, dependency trees, readability scores and frequency lists for language learners. Our best results have been achieved using fined-tuned Arabic-BERT. The accuracy of our 3-way CEFR classification is F-1 of 0.80 and 0.75 for Arabic-Bert and XLM-R classification respectively and 0.71 Spearman correlation for regression. Our binary difficulty classifier reaches F-1 0.94 and F-1 0.98 for sentence-pair semantic similarity classifier.

## introduction in the last century, measuring text readability (tr) has been undertaken in education, psychology, and linguistics. there appears to be some agreement that tr is the quality of a given text to be easy to comprehend by its readers in adequate time with reasonable effort @xcite . research to date has tended to focus on assigning readability levels to whole text rather than to individual sentences, despite the fact that any text is composed of a number of sentences, which vary in their difficulty @xcite . assigning readability levels for a text is a challenging task and it is even more challenging on the sentence level as much less information is available. also, the sentence difficulty is influenced by many parameters, such as, genre or topics, as well grammatical structures, which need to be combined in a single classifier. difficulty assessment at the sentence level is a more challenging task in comparison to the better researched text level task, but the availability of a readability sentence classifier for arabic is vital, since this is a prerequisite for research on automatic text simplification (ats), i.e. the process of reducing text-linguistic complexity, while maintaining its meaning @xcite . we focus here on experiments aimed at measuring to what extent a sentence is understandable by a reader, such as a learner of arabic as a foreign language, and at exploring different methods for readability assessment. the main aim of this paper lies in developing and testing different sentence representation methodologies, which range from using linguistic knowledge via feature-based machine learning to modern neural methods. in summary, the contributions of this paper are: ## features and extraction methods we work with following groups of features in ## experiments cefr language proficiency levels can be presented as labels or as a continuous scale. the former is solved as a classification task with macro-averaged f-1 as the main measure for accuracy. the latter is solved as a regression task @xcite . at first we decided to work with the three main levels (a,b,and c) because it was quite difficult to determine the boundary between the inner sub levels as in the boundary between b1 and b2.yet, the other binary classification is either simple (a+b) or complex (c). here there is a problem for evaluation, since the gold standard labels are represented as integers 1, 2, 3 (for the a, b and c levels respectively), which leads to a large number of ties. out of the standard correlation measures, kendall's tau-b is designed to handle ties, so in addition to pearson's ρ this is our measure for regression @xcite . ## related work on arabic the last two decades have seen enormous efforts (especially for the english language) to develop readability measurement ranging from the traditional readability formulae to ml algorithms. english language researchers have introduced more than 200 readability formulae @xcite as well as hundreds of models @xcite . in contrast, less research has addressed arabic language issues and their challenges for robust readability formulae. some attempts to formulate statistical formulae for the arabic language reflected traditional english formulae such as the flesch-kincaid grade. the simplest formulae included the average word length, the average sentence and other surface features. according to cavalli-sforza et al. (2018) these simple formulae are dawood formula (1977), al-heeti formula (1984), and the formula presented by daud et al. (2013) based on a corpus. the more sophisticated formulae represent the syllables and more insights the arabic sentences, such as aari base by al tamimi et al. (2014) and osman by el @xcite . other studies were conducted to measure text readability by targeting either first or second language learners for arabic language modelled using different ml algorithms. most of these studies used the previously traditional features along with varying lists of part of speech features (pos) representing the words in each document as in studies by @xcite @xcite . forsyth (2014) used the word frequency dictionary by @xcite to classify the words' level against this dictionary frequencies. the dictionary was used later by @xcite along with 133 pos features to achieve an accuracy of 100% with 3-classes. the 'al-kitaab' textbook has a word list introduced at the beginning of each chapter in the book. these lists were used by @xcite for comparing the words appeared in a text against this list and labelling them by (target, known, unknown). @xcite , highlight adding new syntactic features to their features targeting more in-depth analysis. they used two different datasets for both first and second arabic language learning. this yielded an accuracy of 94.8%, 72.4% for first language learners and second language learners respectively. ## conclusions we present the first attempt to build a methodology for arabic difficulty classification on the sentence level. we have found that while linguistic features, such as pos tags, syntax or frequency lists are useful for prediction, deep learning is the most important contribution to performance, but the traditional features can help in interpreting the black box of deep learning alone. for this specific task and for the arabic language, fine-tuned arabic-bert offers better performance than other sentence embedding methods. also, application of the classifiers trained on one dataset to a very different evaluation corpus shows that the classifiers learn some important properties of what is difficult in arabic, but the transfer is more successful for the feature-based models than for the bert-based ones. in the end, our best classifier is reasonably reliable in detecting complex sentences; however, it is less successful in separating between the lower learner levels. still the binary classifier provides the functionality for filtering out really difficult sentences, not suitable for the learners. if we are thinking of arabic learners especially in higher education, we are expecting learners to graduate with a ba degree in the case of arabic as a complex language with confidence in reading b2 texts, which implies that the tool for separating a+b vs c level texts is really useful for undergraduate teaching. through our tool providing computational assessment of difficulty, we will be able: i) to select the appropriate texts for students; ii) to access everlarger volumes of information to find educational material of the right difficulty online; iii) to explore curriculum-based assessment to find what is most effective in finding gaps in a curriculum that can be filled according to students' needs. our future work involves building a parallel simple/complex arabic corpus for sentence simplification. the corpus will be classified on the basis of how difficult the sentences are in a common crawl snapshot of arabic web pages. using the text difficulty classifier, we can split the corpus into two groups for complex and simple sentences. we also consider the semantic similarity detection on "saaq al-bambuu" as a benchmark, which could be used in the corpus compilation. in this study we only performed some ablation analysis, but because bert-like models are more useful as the classifiers, we want to investigate their performance via probing for linguistic features following the bertology framework @xcite . we also want to explore the link between the difficulty assessment on the document vs sentences levels @xcite .