Broad Linguistic Complexity Analysis for G reek Readability Classification (2021)

Abstract: This paper explores the linguistic complexity of Greek textbooks as a readability classification task. We analyze textbook corpora for different school subjects and textbooks for Greek as a Second Language, covering a very wide spectrum of school age groups and proficiency levels. A broad range of quantifiable linguistic complexity features (lexical, morphological and syntactic) are extracted and calculated. Conducting experiments with different feature subsets, we show that the different linguistic dimensions contribute orthogonal information, each contributing towards the highest result achieved using all linguistic feature subsets. A readability classifier trained on this basis reaches a classification accuracy of 88.16% for the Greek as a Second Language corpus. To investigate the generalizability of the classification models, we also perform cross-corpus evaluations. We show that the model trained on the most varied text collection (for Greek as a school subject) generalizes best. In addition to advancing the state of the art for Greek readability analysis, the paper also contributes insights on the role of different feature sets and training setups for generalizable readability classification.

## introduction automatic readability classification systems are intended to assess whether a text is appropriate for a given group of readers. the groups of readers differ in terms of their level of education in their first language (l1), their second language proficiency (l2), or in terms of their special needs (e.g., due to cognitive disabilities). textbooks play a major role in fostering language development in school, and educational policy documents such as the us common core state standards (ccsso, 2010) make explicit that the texts should offer a "staircase of increasing complexity so that all students are ready for the demands of college-and career-level reading no later than the end of high school." foot_0 correspondingly, in the us context there are a number of research and commercial approaches to measuring text difficulty for english @xcite . research continues in this domain, e.g., investigating features from second language acquisition @xcite or psycholinguistic research @xcite , training neural networks (azpiazu and pera, 2019), broadening the corpus basis @xcite , and externally grounding the ratings @xcite . for second language learners, texts at an appropriate level of proficiency are of particular importance, with a substantial strand of computational linguistic research focusing on this target group, e.g., @xcite @xcite , and parallel automated proficiency evaluation approaches analyzing l2 writing, e.g., @xcite @xcite . while much of the initial computational linguistic research on readability focused on english, research targeting other languages is also increasingly emerging, e.g., @xcite @xcite . some recent approaches employ neural networks and deep learning for classification-related tasks @xcite . for instance, @xcite employed neural classifiers on english and slovenian corpora using unsupervised and supervised techniques to assess readability, outperforming some previous state-of-the-art approaches. in this paper, we focus on the greek language and investigate readability classification using a broad range of linguistic complexity features on a comprehensive collection of five textbook corpora covering various subjects, and greek as a second language. we first explore which feature groups and combinations thereof best predict the school grade or proficiency level assigned by the publisher. going beyond the typical within-corpus setup, we then conduct cross-corpus tests to study how well the models trained on different feature sets and corpora generalize across data sets. the paper is organized as follows: section 2 presents the textbook corpora and section 3 the linguistic complexity feature sets used in the readability classification experiments. section 4 then spells out the experimental setup and the results, before concluding in section 5. ## data our data consists of five corpora of greek texts used as educational material for different school subjects and educational contexts. we use four corpora from previous work on greek @xcite , in addition to a greek as a second language (gsl) textbooks corpus compiled for the needs of the research at hand. ## linguistic complexity analysis we used a state-of-the-art greek dependency parser (prokopidis and papageorgiou, 2017) of the institute for language and speech processing (ilsp) to annotate the corpora. the conll-format files produced by the parser were used as input to our code performing the complexity features extraction, returning a vector of feature values for each text. greek is an inflected language with rich morphology, allowing relative flexible word order in sentence construction. we computed a broad set of linguistic complexity features capturing lexical, morphological, and syntactic characteristics. we combined measures from greek readability research @xcite @xcite with features originally introduced for german @xcite and added cognitivelymotivated features from psycholinguistic research on english @xcite @xcite used in research on l2 and academic language development @xcite . together with some traditional readability formulas, for each text we computed an overall set of 215 features foot_3 : the classification experiments were conducted using weka @xcite . we initially compared several machine learning algorithms, including logistic regression, multilayer perceptron and sequential minimal optimization (smo). the smo classifier systematically outperformed other options, so we here focus on the results discussion of the smo classifier foot_4 . for all our experiments, we report accuracy using 10-fold cross validation and the f-score. to study the impact of the groups of complexity features (lexical, morphological, syntactic), we ran machine learning experiments with each individual linguistic feature subset, then with binary combinations thereof @xcite , with all 212 linguistic features, and finally with all 215 features. to investigate the informativity of individual features, we ranked features using weka's info gain (ig) attribute evaluation @xcite and ran experiments with the top 30 features as well as with all features with an @xmath0.1. the results indicate a strikingly similar pattern in history and science corpora. for each of the three linguistic feature groups (lexical, morphological, syntactic), the classifier reaches around 75% accuracy (74.97%-76.88%), which, compared to the 33% random baseline, clearly demonstrates that the history and science texts in different school levels systematically differ in each of these linguistic dimensions.