Towards a Data Analytics Pipeline for the Visualisation of Complexity Metrics in L 2 writings (2021)

Abstract: We present the design of a tool for the visualisation of linguistic complexity in second language (L2) learner writings. We show how metrics can be exploited to visualise complexity in L2 writings in relation to CEFR levels.

## introduction the analysis of educational data has been a growing field in the last decade. learning content management software (lcms) platforms in education have provided the opportunity to collect and process large quantities of educational data supporting both data mining and analytics @xcite . as far as we know, the field of foreign language learning has not availed yet of projects with data analytics at their core. the proceedings from the visualisation and digital humanities workshop series 1 and those of the learning analytics and knowledge conference foot_1 fall short of studies focused on the automatic exploitation of linguistic data for learners of a language. this problem may be linked to the complexity of apprehending learner writings due to the multidimensional nature of this type of language (errors, usage, phraseology ...) one way to approach the problem is to use data analytics methods in order to bridge the gap between the collection of learner productions and their automatic analysis resulting in meaningful feedback. to this end, it is necessary to identify quantifiable features of learner writings. data could be sourced in one of the three dimensions of language proficiency, i.e., complexity, accuracy and fluency (caf) @xcite . a data analytics framework could rely on measures that operationalise these three theoretical constructs. a selection of caf measures could be the source of automatically generated linguistic profile reports of l2 writings. as part of caf, linguistic complexity is one of the constructs that lends itself well to computational methods. at theoretical level it informs on the elaborateness of the learner language. at operational level there are a number of statistical measures in the form of frequencies, ratios and indices @xcite . the construct is already used in combination with corpora to achieve different tasks such as automatic proficiency level prediction. in these tasks, complexity metrics are exploited with supervised learning methods to predict levels @xcite @xcite .among all the metrics that have been tested (see @xcite for a review), several have been reported as predictive of proficiency @xcite @xcite . some readability metrics have also been used in l2 studies @xcite . a number of text analysis tools exist in education but are not focused on l2 learning. they provide environments for reading or writing assessment and training @xcite @xcite @xcite . they focus on providing quantified results in relation to internal scales for first language (l1) learning. in addition, and to the best of our knowledge, the tools do not provide visualisations for the textual measurements. in the field of l2 learning a tool called feedbook @xcite provides visualisations of linguistic features as part of the feedback given to students. one need that remains to be addressed is the ability for learners to position the linguistic properties of their productions with regard to proficiency levels. our proposal is to exploit state-of-the-art linguistic complexity metrics in the automatic analysis of l2 writings. nlp tools are used to annotate, compute metrics and display visualisations of learner productions compared with writings classified according to the cefr foot_2 levels @xcite . section 2 describes the method. section 3 covers visualisation interpretation. learner engagement is presented in section 4. we discuss issues and perspectives in section 5. ## methodology 2.1 a cefr-based reference data set to compare new texts with existing texts, we exploit a learner corpus of written productions. texts from english for specific purposes (esp) university students are used. this corpus includes 274 third-level education writings. two language certification experts assessed the writings in terms of cefr proficiency levels. the first production task for learners consisted in describing an experiment/discovery/invention/technology of their choice and the second task was to give their opinion on the impact of the previously described item. learners had 45 minutes to complete both tasks. cefr annotation was evaluated with a measurement of inter-rater agreement (cohen's weighted @xmath0.71). complexity metrics were computed and six subsets or cohorts were created according to the six cefr levels. a comparative data set of metrics and cefr levels was thus created foot_3 . ## interpreting visualisations in this section, we conduct an illustrative analysis of a sample text and compare some of its features with the visualised metrics. it was written by a french learner of english as part of the french national language certification proficiency exam 10 . it was classified as b2 or higher. for reasons of space, we only provide the following exerpt. with the development of new technologies such as smartphones, new questions are being caused about how to evaluate students. indeed, using cellphones to cheat is common in highschools. the first question we have to ask ourselves is wether we should authorize students to access their phone or not. arguments against are well-know: ... but we also have to consider arguments in favor of it, in order to do what is best for our students. first, they will be working with these technologies in their professional lives, and we should be preparing them for that, by teaching them a proper use of smartphones and computers... in figure 2 the metrics are compared with those of the cohort of b2 learners (see section 2.1). the learner's individual report is divided in two parts. on the left, a radar chart displays ratio-based metrics and, on the right, raw frequencies are reported. in addition to the metric acronym a categorisation label is provided in order to indicate the word, sentence or text scope (anonymised reference 2019b) of the metric. for instance the number of different words (ndw) is labeled text.size.type and can be interpreted at text level in terms of types as a unit. the radar chart displays the cohort in a greyshaded area representing the two central quartiles. the dark line represents the median of the cohort. the boxplots show the position of the learner's metric in relation to the full b2 cohort. the indicators in the radar chart show that the learner's ratios globally correspond to those of the b2 cohort. for instance, the learner makes use of complex nominals in her text. this includes 10 see cles certification at https://www. certification-cles.fr/en/ figure 2: individual report of a learner in comparison with a cohort of b2 learners the adverbial clause "with the development of new technologies such as smartphones" which is used in apposition to the main clause. it also includes the nominal clause "the first question we have to ask ourselves" used in subject position of be. the use of adjective + noun as in "proper use" is another more simple example of nominal complexity. the three different cases are all accounted for by the system. it appears that the learner's level of use is slightly under the b2 median (c1 and c2 radar charts show even higher values for the two central quartiles). the teacher and learner can analyse such structures more into details. the teacher could in turn note the lack of use of compounds and genitives in the text. in short, the metric helps learners and teachers identify an issue related to an objective criterion of linguistic complexity. specific feedback and actions can then be undertaken. ## learner engagement the efficacy of the tool needs to be evaluated. learner engagement remains to be assessed thor-oughly but a preliminary qualitative assessment was conducted in a class setting environment. we show results about the impact of the tool on learners' engagement. fifty-four first year higher-level students were given five individual writing tasks in five weekly waves. after each wave they were provided feedback within 24 hours. notwithstanding the results, we measured the number of submitted writings (figure 3 ) and the frequency of consultation of feedback reports (figure 4 ). we use these measurements as a rough proxy to measuring learner engagement, i.e. how they respond to the feedback they receive @xcite . the statistics are assumed to tap into the intensity of learners' interest in the reports. over time the number of submitted papers did not decrease in spite of the lockdown imposed on students in the midst of the covid crisis. following detailed explanations from their teacher to ensure comprehension, a majority of students consulted their reports three times or more, showing continuous interest. figure 4: consultation of feedback reports 5 discussion and perspectives in this paper we have presented a linguistic complexity visualisation tool. it displays learner writings according to several criteria and positions them in relation to cohorts of specific cefr levels. more work remains to be done. firstly, the visualizations used may be difficult to understand for learners who are not used to such types as radar charts. the tool aims primarily at helping trained language teachers analyse their students' writings in order to give them objective and specific feedback @xcite . by gaining access to these features, teachers can give specific answers regarding the mastery of certain concepts. they also become aware of features of language use that need to be addressed. teachers can then provide evidence-based advice. secondly, the collected data shows limitations. the metrics on which the visualisations rely need to be evaluated on the data in terms of proficiency predictive power. correlation analysis remains to be conducted in order to validate significant metrics to be displayed. the reference corpus is small and at the same time lacks diversity. all the texts belong to university students of specific fields, which may impact vocabulary and syntactic structures. more data needs to be collected in each field in order to support finer-grained analysis of third level education writings. one last limitation is that some metrics remain difficult to interpret linguistically as argued by @xcite . for instance, readability formulas combine different features such as morphology and most common words. specific advice on one of the features is therefore near impossible. nevertheless, by interpreting the linguistic scopes (whether the measures apply at word, sentence or text level), it is possible to provide a certain degree of feedback. the tool could be exploited in the learner module of an intelligent tutoring system dedicated to language learning. because linguistic complexity measurements keep track of the evolution of systemic syntactic and lexical complexity, these data constitute part of the knowledge representation of the learner. the tool gives access to learning analytics at linguistic level. in a context of distance learning, teachers are empowered with a rapid diagnostics tool that gives them an objective, although reduced, view of some of the features of their learners' language. further developments will focus on identifying and evaluating more significant metrics in terms of proficiency and meta-linguistic influence on learners. other types of charts could also be explored, and an aggregation functionality could provide group visualisations to reveal linguistic class patterns for teachers.