Explainable AI in Language Learning: Linking Empirical Evidence and Theoretical Concepts in Proficiency and Readability Modeling of P ortuguese (2024)

Abstract: While machine learning methods have supported significantly improved results in education research, a common deficiency lies in the explainability of the result. Explainable AI (XAI) aims to fill that gap by providing transparent, conceptually understandable explanations for the classification decisions, enhancing human comprehension and trust in the outcomes. This paper explores an XAI approach to proficiency and readability assessment employing a comprehensive set of 465 linguistic complexity measures. We identify theoretical descriptions associating such measures with varying levels of proficiency and readability and validate them using cross-corpus experiments employing supervised machine learning and Shapley Additive Explanations. The results not only highlight the utility of a diverse set of complexity measures in effectively modeling proficiency and readability in Portuguese, achieving a state-of-the-art accuracy of 0.70 in the proficiency classification task and of 0.84 in the readability classification task, but they largely corroborate the theoretical research assumptions, especially in the lexical domain.

## introduction as technology evolves at a rapid pace, the field of education undergoes continuous adaptation. particularly in language learning, numerous tools are being developed with the goal of facilitating the practice of a second language and providing tailored materials. in order to effectively model natural language, it's crucial to identify and empirically validate the relevant linguistic properties to use. linguistic modelling with complexity measures has been proven to be highly effective in providing evidence-based insight into the assessment of both proficiency and readability @xcite . second language proficiency and text readability are often associated concepts in language learn-ing. proficiency is usually equated to the notions of mastery and ability of understanding and producing another language @xcite . readability, in turn, encompasses the degree of reading difficulty which a text may exert on a reader @xcite . while widely acknowledged as multidimensional and dynamic constructs, proficiency and readability are commonly assessed using standardized scales. the common european framework of reference for languages (cefr, council of europe) stands out as one of the most prominent scales for measuring proficiency, while readability is usually estimated according to different education, proficiency, and literacy levels. in this context, it is essential to note the limited empirical evidence supporting the categorization of the mentioned constructs into levels and the precise definition of each level @xcite . the english profile programme (egp, @xcite is a notable effort to clarify proficiency levels by identifying linguistic features whose presence or absence corresponds to specific english cefr levels. however, the success of such an initiative heavily relies on the availability of abundant data and specialized manpower for data annotation and analysis, which may not be readily accessible for languages other than english. in this paper, our objective is to propose an automatic method that comprehensively captures the nuanced characteristics defining language proficiency and text readability in portuguese, as well as to provide a robust multilingual text analysis platform which will be made freely available online. by performing linguistic modeling and applying explanatory methods, we seek to validate theoretical postulations and enhance our understanding of the linguistic properties which are crucial for language learning, by answering the following research questions: to address these questions, we use two distinct datasets: one consisting of european portuguese learner productions and another consisting of brazilian school materials. the constructs under investigation will be modeled as classification tasks. by leveraging such measures, language learning tools and generative models may better capture the nuances of written language, leading to a deeper understanding of the intricacies of readability and proficiency assessment @xcite . ## related work the extraction and analysis of linguistic complexity measures have been extensively explored in research. while many studies on automatic proficiency and readability assessment have primarily investigated the english language (e.g. @xcite @xcite , advancements in natural language processing (nlp) have facilitated the extension of research to other languages. in european portuguese, del río (2019b) employed a supervised-learning approach using a learner corpus, achieving a 0.72 accuracy and 0.71 f-score by combining 39 linguistic complexity measures with other types of features, such as n-grams and readability formulas. similarly, in brazilian portuguese, @xcite used a corpus from a brazilian portuguese proficiency exam, extracting 48 linguistic measures for a binary classification task distinguishing between beginner and advanced learners, achieving an accuracy of 0.70 with a j48 classifier. automatic readability assessment has also been explored in the context of the portuguese language. for instance, @xcite analyzed a corpus of l2 portuguese texts, extracting 52 linguistic complexity measures. their experiments achieved accuracy scores of 0.86 and 0.79 for three-level and five-level classification tasks, respectively. additionally, @xcite extracted 489 linguistic complexity measures using the platform herein presented with machine learning algorithms for readability classification. this study demonstrated that models which incorporated informative features exhibited the highest generalization rate across various samples. regarding the use of explainable ai in portuguese studies, @xcite explores the estimation of textual cohesion across essays in both portuguese and english. the study found that although a deep learning-based model demonstrated superior performance, conventional machine learning models showed stronger potential in explainability. the mentioned studies represent a crucial advancement in the automatic classification of proficiency and readability in portuguese; however, they have limitations. except for a few, most of the studies in portuguese readability and proficiency assessment suffer from either a lack of a comprehensive set of measures, which might not fully capture the complexity and nuances of proficiency and readability, or from the absence of interpretability and detailed insight into feature importance. as a result, the depth of understanding regarding the constructs themselves and their categorization into separate levels may be limited. ## data two corpora were selected for our experiments and analyses: nli-pt @xcite and corpus de complexidade textual para estágios escolares do sistema educacional brasileiro @xcite . the former comprises 3069 l2 portuguese learner texts, categorized into three general levels: a (consisting of the cefr levels a1 and a2), b (b1 and b2), and c (c1). the distributions of the texts in this corpus can be found in it is important to note that the distribution of texts into the separate categories in both corpora is imbalanced. that means that some levels are better represented than others, possibly influencing the classification results. ## methods our experiments consist of three main steps: first, extracting linguistic complexity measures from the chosen corpora; then conducting two types of classification experiments, one for proficiency and one for readability; and finally, analyzing the results using an explainable ai method to understand feature importance. ## proficiency classification results while all classifiers showed similar performance, the svm classifier exhibited slightly better results compared to the others, as shown in in the confusion matrix ( figure 1 shows the mean shap values for the top 20 features with the most impact on the model's output for each proficiency level, listed in descending order. among these features, 10 are related to the lexical domain. the most impactful feature is complex word frequency, particularly influential a b c a 219 41 3 b 51 178 19 c 19 32 29 phrasal and clausal elaboration significantly influenced the model's output. the measures of relative clauses per clause and per t-unit were influential for distinguishing levels a and c, while the mean length of noun phrases is most impactful for predicting level b. the nominative case inflection emerges as the sole highly discriminative morphological measure. additionally, word frequencybased features, clausal elaborateness, and lexical sophistication measures contribute to the list. ## readability classification results consistently with the proficiency classification experiments, logistic regression, random forest, support vector machine and multy-layer perceptron classifiers were implemented. the results achieved with 10-fold cross-validation and held-out test sets for each classifier are displayed in figure 2 displays the mean shap values for the top twenty influential features. thirteen of these features pertain to the lexical domain, four to morphology, two to surface features, and one to syntax. the imageability of lexical word types had the strongest impact on the model's output. familiarity, age of acquisition, the lexical density of articles and determiners and frequency-based measures were also highly discriminative. additionally, superficial measures like the standard deviation of token length in syllables and letters were predictive. morphological measures were also influential, with inflections in case, mood, person, and number showing strong impacts, particularly in differentiating the first and last levels. notably, phrasal and clausal elaborateness seemed less significant in predicting school levels compared to proficiency classification. ## feature selection during the analysis of the measures, we found that some linguistic features were highly correlated with each other, aligning closely with expectations, for example, the correlation between the number of letters and the number of syllables, or the number of determiners and the number of articles. however, other correlations were less anticipated, such as those between subordinate clauses and corrected type-token ratio (ttr) of verbs. although removing correlated features is important for enhancing a model's performance, appreciating their interactions remains crucial for interpretation. thus, a trade-off between interpretability, model complexity and performance emerges as a central consideration. to maintain model interpretability, we refrained from employing feature engineering or dimensionality reduction techniques, opting instead for the cfssubseteval and infogain methods implemented by the waikato environment for knowledge analysis (weka, @xcite . cfssubseteval identifies an informative yet uncorrelated subset of features. similarly, infogain evaluates each feature's contribution to reducing entropy, aiding in the selection of the most informative features. ## discussion in addition to obtaining the mean shap values of the most discriminative features, the shap values associated with each level were also inspected, foot_2 the measures referring to the superficial and lexical aspects exhibited the strongest discriminative power. advanced learners produced more and longer words and sentences than beginners. a uniform increase was present concerning most of these features. the same is true for the school materials corpus herein utilized. texts from the highest education levels demonstrated a higher incidence of words considered complex, abstract, infrequent and generally unfamiliar when compared to the lower ones. the same pattern was identified in terms of lexical variation. this is in line with the postulations in the simplification manual. regarding the syntactic domain, our data also corroborates most of the remarks. in the productions of l2 portuguese learners, it was verified that clausal subjects, passive sentences, subordinate and relative clauses, as well as asyndetic coordinated clauses are indicative of more advanced levels. these grammatical constructions only arise after the general level b in the analyzed data. comparatively, texts from the different educational levels demonstrated more homogeneity regarding syntactic measures. although sentences and clauses are shorter in the early levels, constructions like subordinate and relative clauses as well as clausal subjects remained relatively constant across the levels. more pronounced contrasts regarding this domain were only found in terms of passive sentences and left embeddedness, which is in line with both the portuguese rld and the simplification manual. morphological measures also demonstrated contributions in differentiating the levels. for instance, it was observed that l2 learners placed at proficiency level a produced a distinctively higher amount of nominative case inflections, and, on the other hand, they exhibited low amounts of the accusative case inflection. in terms of verbal mood, it was observed that beginners also produce high amounts of indicative mood. this corroborates suggestions from the portuguese rld which suggests most verb tenses in the subjunctive mood are learned at levels b1 and b2. the same trend regarding the high use of the nominative case was observed in the brazilian corpus. the elementary school texts contained a distinctively higher amount of this inflection when compared to high school or university ones; however the incidence of accusative case inflection was not as pronounced. concerning discursive measures, it has been suggested that as l2 learners progress, they tend to use fewer explicit cohesive devices @xcite . this trend was observed specifically for causal connectives: their absence indi-cated advanced l2 proficiency. on the other hand, regarding the school texts, this was the case for temporal connectives, which may very well point to a diminished presence of narrative discourse and higher amounts of expository discourse. finally, in terms of referential cohesion, its low values were decisive for the prediction of the beginning proficiency level, but no impact was identified for the prediction of other levels or for the school texts. ## conclusion in this paper, we explored portuguese broad linguistic modeling in relation to l2 proficiency and text readability. employing an elaborate nlp pipeline, we extracted 465 measures of linguistic complexity from two corpora. our ultimate objective was to understand which measures exerted the most impact in each level's prediction and assess the extent to which these measures support the concept of holistic, static, ascending categories of proficiency and readability by implementing classification experiments and applying explainable ai methods. our results show that the consistent performance across different evaluation metrics suggests that the svm classifier, trained on a broad set of linguistic complexity measures, provides a robust framework for modeling proficiency and readability levels in portuguese texts. in particular, lexical features were found to have strong discriminative capabilities between different proficiency and readability levels. these findings provide evidence as to validate these measures and confirm the feasibility of modeling natural language using a diverse range of linguistic features. it also shows that xai methods can be applied to linguistic complexity analysis. in line with the portuguese rld and the simplification manual, the texts herein analyzed exhibited a uniform increase in the use of longer, more abstract, less familiar and less frequent words across both proficiency and readability levels. moreover, an increase in sentence embeddedness and coordination, as well as tense and voice inflection was also positively confirmed in our findings. additionally, trends in discursive measures suggest shifts in cohesive device usage as proficiency progresses, with possible implications for different discourse types. these findings offer valuable insights for the refinement of language learning tools and assessment techniques. specifically, they emphasize the significance of certain linguistic characteristics, such as vocabulary type, morphological and syntactic complexity, in modeling learner language and assessing proficiency and readability. additionally, our intention to make ctap's portuguese analysis feature openly accessible online aims to support the development of more linguistically informed analyses through an accessible platform. this initiative is expected to facilitate the integration of linguistic insights into educational technologies.