Exploring a New Grammatico-functional Type of Measure as Part of a Language Learning Expert System (2023)

Abstract: This paper explores the use of L2-specific grammatical microsystems as elements of the domain knowledge of an Intelligent Computer-assisted Language Learning (ICALL) system. We report on the design of new grammatico-functional measures and their association with proficiency. We illustrate the approach with the design of the IT, THIS, THAT proform microsystem. The measures rely on the paradigmatic relations between words of the same linguistic functions. They are operationalised with one frequency-based and two probabilistic methods, i.e., the relative proportions of the forms and their likelihood of occurrence. Ordinal regression models show that the measures are significant in terms of association with CEFR levels, paving the way for their introduction in a specific proform microsystem expert model.

## introduction this paper explores the use of l2-specific grammatical systems as elements of the domain knowledge of an intelligent computer-assisted language learning (icall) system. such systems rely on natural language processing approaches that conduct several high-end tasks such as grammatical error detection (ged), automatic reformulation or proficiency level prediction. as part of the intelligent tutoring system (its) category, they rely on models that have have an expertise, which is language use in their case. expert models encapsulate the domain knowledge which is required to describe the learner's language skills involved in tasks such as writing production. in itss, there are several possible strategies used to acquire and represent domain knowledge @xcite . among those are rule-based cognitive models, describing learning strategies, and constraint-based models (cbm) describing principles that rely on correct solutions to a problem. in the case of icall, representing the knowledge of learners has traditionally been done within the constraint-based model (cbm) framework thanks to correct-usage principles derived from native language use. for instance, some grammatical error detection tasks are processed on the basis of target hypotheses (th) @xcite , i.e. the correct version of what is meant by a learner in a specific segment. in this type of tasks, correct versions of erroneous segments or patterns are compared with the th @xcite to identify incorrect uses. as useful as it has proved to be, this type of approach tends to reduce the knowledge about l2 language production to what native speakers would say or write by focusing on error correction. in doing so, it overlooks the meta-knowledge that language learning experts possess regarding acquisition processes. experts' evaluations of learner language not only rely on th, but also on what they know of the grammatical, lexical, semantic and pragmatic features in l2 writings of different proficiency levels, be they negative or positive features @xcite . this allows them to position the learner's productions in terms of level and to provide feedback. we argue that an expert icall system should not be reduced to error identification and correction on the basis of native language production, but include comprehensive knowledge about the range of l2 linguistic profiles at different stages of language learning. we intend to use such profiling as part of a learning-analytics system providing in-formation to teachers on their learners' linguistic developmental stages. modelling the domain knowledge with such profiles linked to proficiency levels is necessary. in order to do so, we draw inspiration from rulebased cognitive models. the role of rule-based cognitive models is to describe the knowledge involved in "student performance in a given task domain, including strategies, problem-solving principles, and knowledge of how to apply problemsolving principles in the context of specific problems" @xcite . when applied to language learning, this approach complies well with describing the strategies used to elaborate language patterns, including idiosyncrasies. this makes rule-based cognitive models quite comprehensive in describing learner language characteristics. our proposal follows this approach, as it considers an expert model as a cognitive entity that knows positive and negative characteristics of an l2 set of writings at various stages of proficiency. the expert model should not simply "know" the rules that operate for native speakers, it should also include the probability of patterns that govern specific levels. many grammatico-functional microsystems (ms) exist that describe a part of the grammatical reasoning at work in production. they are convenient to describe the psychological reality of the learner and may be linked to proficiency as in the english grammar profile (o' @xcite . as an illustration of the broad process, we present the design and implementation of a specific linguistic microsystem as a rule-based cognitive model, namely the this, that and it proform microsystem. our working hypothesis revolves around the idea that different proficiency levels prompt different linguistic contexts around the use of the microsystem, which leads to different odds of using the forms in the microsystem. therefore, by observing the probability of using a given form in the microsystem as a function of the context, we could capture aspects of the learner's grammatical reasoning that points to a given proficiency level. this approach raises two research questions: to answer these questions, we propose to use a model to describe the probabilities of use of this, that and it proforms depending on context, as a first step toward modelling linguistic profiles. in other words, this model predicts the likelihood of a learner using either this, that or it given the linguistic context of this proform, while being agnostic to whether the choice of such proform was correct or not. to assess the relevance of using the likelihood of microsystem forms as linguistic profiling, a second model predicts proficiency levels using only the probabilities of using this, that and it output by the microsystem prediction model. if this second classification model can discriminate proficiency levels using only the predicted probabilities for the forms of the microsystem, then the microsystem likelihood model is a coherent way to build domain knowledge indicators for profiling. in section 2 we present the theoretical background underlying our research. section 3 presents the data and the microsystem extraction methods used to exploit it. in section 4, we present how the microsystems are implemented and evaluated in terms of extraction and predictability. section 5 covers the results obtained with different modelling approaches to validate the associations between microsystem and proficiency. expert models in general itss. intelligent tutoting systems require expert models which fall into three main categories, i.e. black-box, glassbox (or rule-based) and cognitive models @xcite . following aleven (2010), we place rule-based models alongside cbms as part of the cognitive category. black box models are said to be inexplicit in their representations as they only provide the final results @xcite and show correct inputoutput behaviour with very little use for their internal computation @xcite . cognitive models show different degrees of interpretability, which is useful for instruction delivery. their decision making processes lend themselves well to giving feedback to learners. in the subcategory of cbms the requirements that all solutions should satisfy are set in advance rather than having to map all possible errors and correct solutions. this simplifies the search space by narrowing down the possible solutions and avoiding breaking the domain principles. conversely, rule-based models rely on an comprehensive set of rules that can be deterministic or probabilistic. the rules mirror the way an expert analyses a problem by taking into account positive or negative observations. a good expert model seems to revolve around several principles. not only does it have to produce correct results, but it also needs to have high cognitive fidelity, i.e., the compliance of its decision making features with those that are used by learners. in addition, the expert model must filter out the feature space "according to the same restrictions as a human does" @xcite . expert models in icall. the aforementioned distinctions can be used to understand the different types of expert models that exist in icall systems. are they black boxes or cognitive models? rule-based or cbm? depending on the tasks and the adopted nlp approaches, they may fall into one of the categories, showing or not their cognitive inclination. as far as we know, most second language models employ supervised learning methods which rely on very different types of features. neural approaches with text embeddings and transformer models provide very accurate results in different tasks such as grammar error detection (ged) @xcite or automatic essay scoring (aes) @xcite . however, the rules and features they rely on are difficult to interpret, turning them into black boxes and leading to poor cognitive fidelity. a number of ged tasks have relied on supervised learning approaches based on error coded datasets including corrected statements as target hypotheses @xcite . these hypotheses may be seen as the cbm principle, i.e. reference points with which learner language is compared @xcite . by way of editdistance metrics, the models can be used to provide error identification in context. however, they cannot explain the reasons for the errors. their decision making process does not rely on information that is cognitively meaningful for learners. other supervised-learning models are based on probabilistic rules relying on explicit linguistic features. in proficiency prediction tasks, a number of experiments were conducted with models relying on morphosyntactic and lexical features @xcite @xcite . these linguistic features make up intelligible rules that have a degree of cognitive fidelity. however, in spite of their linguistic characterisation, some are not very actionable by teachers. this is due to the complexity of their design in terms of variables @xcite . for instance, the automated readability index (ari; @xcite ), a measure of difficulty in reading, is composed of two variables (average sentence length (asl), the average word length (awl)) whose combination in a formula foot_0 is hard to interpret. because they are not designed to provide any other feedback than the result, these models do not have high cognitive fidelity. some advanced automated writing evaluation (awe) systems show greater cognitive fidelity as they try to match their feedback with interpretable linguistic information. based on linguistic features used in supervised learning methods, the systems can contextualise the errors with grammatical justifications @xcite . some automatic essay scoring systems, which rely on semantic and discourse complexity metrics, feed from their expert models' features to elaborate feedback messages on cohesion for learners @xcite . in these cases, the connection between the models' rules and the wording of the feedback messages shows a focus for high cognitive fidelity. while elaborating the messages, these systems rely on expert models that filter out irrelevant knowledge that could impair the cognitive reception by learners. one important aspect is that @xcite add specificity as an extra dimension. they use two specific models for two specific tasks, i.e. a view of cohesive links in discourse and a view of stance variation in discourse. our proposal follows the same principle applied to the grammatical rather than the cohesive dimension. the objective is to design expert models that capture the hesitations that learner may have on specific syntactic paradigms. for instance, learners may hesitate between different determiners, or they may have confusions in the use of demonstrative pronouns. we intend to design several expert models for microsystems specifically linked to linguistic functions. their goal is to provide fine-grained knowledge of the variations between forms of the same function. microsystems reflect the learners' hesitations that are part of the competition model in which learners constantly resolve conflicts while choosing forms @xcite . these hesitations create microsystem instability as learners unexpectedly group forms that are not necessarily mapped to the same functional paradigm @xcite . due to this instability in the mappings, the microsystems are transitional in nature @xcite . they include erroneous mappings which later are removed, leading the learner to better proficiency. following gaillat et al. (2021) , we focus on the referential proform microsystem made up of this, that and it. the purpose is to compute how each of these forms, mapped to the same referential function, is likely to occur in relation to its two other competitors. ## conclusion in this paper, we have reported on the design of new grammatico-functional metrics which are to be used in the expert module of an icall system. the metrics rely on paradigmatic syntactic relations between words of specific functions. we have illustrated the approach with the design of the it, this, that proform microsystem. the measures rely on the relative proportions of the forms and their likelihood of occurrence. they show sig-nificance in terms of association with cefr levels, paving the way for their introduction in a specific proform microsystem expert model.