Analyse exploratoire des traces numériques clavier pour la prédiction des niveaux d’apprenants (2025)

Abstract: Cet article présente une typologie des métriques des traces numériques clavier en vue d’une analyse des stratégies d’écriture des différents profils d’apprenants appliquée à une tâche de prédiction du niveau CECRL.

## introduction in this paper, we discuss the possibility of predicting cefr levels on the basis of keylogging metrics. this set of metrics has been developed to characterize the behavior of learners, the type of writing processes by learners, to possibly associate those patterns to proficiency skills in english. the aim is to characterize learner writing behaviours and keystroke dynamics as captured by metrics based on bursts and linguistic units, as opposed to more biometry-centered metrics @xcite that operate at character level. the rest of the paper is structured as follows: section 2 summarizes previous research on keylogging analysis of learner data for second language acquisition modeling. section 3 presents our experimental design. we explain how keylogs can be used as one of the microservices analyzing learner data in a wider, bigger cms based moodle infrastructure. section 4 presents our preliminary results on a prototype dataset. section 5 discusses our results and concludes. ## previous research on (learner) keylogging data within the complexity accuracy fluency paradigm @xcite , many studies have addressed classification tasks based on spoken @xcite or written learner productions @xcite , where learner proficiency is usually expressed in terms of the cefr reference framework @xcite . to the best of our knowledge, no systematic review exists on the use of keylogs for l2 second acquisition modeling. previous research has mostly identified the location and the types of pauses as the main concepts or the main construct to operationalize planning, time management, and other skills. other studies followed bayesian models to investigate writing processes and typing behaviour. @xcite suggested modeling writing and copy tasks with the use of bayesian approach where she presented a process-based model of typographic error revisions which provided a systematic way to analyze how typists detect errors and correct them in real time. the bayesian modelling showed a clear difference of writing patterns for the copy task and the writing task. in addition, @xcite examined typing disfluencies using bayesian mixture models where they attempted to understand keystroke variability. furthermore, @xcite computed fluency patterns using regression techniques and bayesian modeling where she tried to fill the gap between linguistic properties and typing dynamics. she utilized analysis tools in r and stan and the iki dataset structure (interkey interval). as a follow-up to previous investigations on learner writing @xcite investigating constructions in learner essays and, more generally, the writing process in a corpus-based perspective @xcite . @xcite examines the employment of keylogging to explore l2 writing processes. a key application examined in her study is the multiword unit (mwu) processing, where keylogging data shows that mwus with pauses at their boundaries and not within them are probably processed systemically. (garcés manzanera, 2024) investigated the relationship between pauses and revision behaviour using keystroke logging software. he detected the writing process of 22 elementary students aged between 10-11 with a1-a2 cefr levels while completing a task of picture description. his findings presented a negative correlation between pauses (p-bursts) frequency and text quality, while a positive correlation was found between revision bursts (r-bursts) with the quality of the text, which suggests the great importance of revision behaviour in boosting writing quality. while many studies have used keystroke logging in order to better comprehend writing behaviour, an integration between process-based metrics and cefr-level classification is still underexplored. most existing models do not systematically link dynamic typing features (e.g., burst size, backspaces, pause lengths) to english proficiency levels. we expected revision bursts to be of great importance for the cefr classification, and our preliminary results support these expectations. ## preliminary results for exploratory purposes, we report our preliminary analysis on our dataset of 232 essays. we created a 80/20 training/test split ensuring proportional representation of the cefr variable via stratification. we then used elastic net regularization (combining l1/lasso and l2/ridge penalties) through glmnet. the model optimizes the trade-off between bias and variance by minimizing a penalized cost function. findings regarding pause patterns and cefr levels revealed that long pre-burst pauses were linked with low proficiency levels, proposing a high cognitive load. furthermore, logistic regression analysis of pos tagging indicates that the usage of collocations and formulaic expressions was a powerful proficiency predictor, aligning with @xcite findings, where advanced writers present better fluency in noun-verb sequence. our keystroke analysis provides preliminary results of systematic differences in typing behaviour across proficiency levels. short pauses between sentences and phrases were exhibited by higher-proficiency participants, aligning with @xcite findings. in addition, regression models showed that higher-proficiency writers demonstrated more structured revision behaviour at the sentence level, in comparison to lower-proficiency writers who were involved in within-word revisions and frequent backspacing. since our preliminary analysis was performed on a limited sample, further work is suggested with a broader dataset and extending the sample size for additional validation. in our confusion matrix (figure 3 ), the best predicted classes are unsurprisingly the most numerous ones. 5 discussion and conclusion the overall accuracy (0.356) of the classification of the learner levels is a bit disappointing, but the dataset is rather limited (232 essays) to build a completely reliable model. we plan to extend our methodology to bigger datasets like the kupa-keys dataset and its 1,006 essays @xcite . the keylogging pipeline is integrated as a microservice for a moodle-based analysis of learner data as part of the a4ll project foot_1 . this infrastructure is meant to collect more data points and could be used to improve our modelling. the general aim of the project is to provide feedback to learners using a server on the huma-num infrastructure. for the time being, cefr levels and feedback recommendations are based on the computation of complexity metrics but we aim to include the outputs of keylogging metrics to guide learners. if we extrapolate the writing behaviour from the metrics, in this dataset, revision bursts seem to be of paramount importance for the cefr classification. the importance of the metric ratio_backspace_seq_shorter_than_or_equal_3_for_typo suggests learners apparently edit more for typos than for revision. some construction patterns could be detected by the fact that revision bursts tend to be observed for sequences beginning with verbs and ending with proper nouns. at keystroke level, backspace is the crucial 'behavioural metric' for learners, then comes the use of '?' and '!' in learner texts. in our data, word-based metrics and the size of p-bursts (captured by the number of keystrokes) did not seem to be relevant (even mean length of essays, when included in the model), but the size of r-bursts was, so we still believe that it is relevant to try to apprehend learner behaviour by varying the constituent scope of the metrics, which sounds like a promising avenue for research. in this paper, we have illustrated the interest of investigating keylogging data with metrics. by varying the scope of the metrics with the size of the reference constituent, we manage a complementary approach between static computations and the dynamics of writing to characterise learner writing.