Towards automatically extracting morphosyntactical error patterns from L 1- L 2 parallel dependency treebanks (2023)

Abstract: L1-L2 parallel dependency treebanks are UD-annotated corpora of learner sentences paired with correction hypotheses. Automatic morphosyntactical annotation has the potential to remove the need for explicit manual error tagging and improve interoperability, but makes it more challenging to locate grammatical errors in the resulting datasets. We therefore propose a novel method for automatically extracting morphosyntactical error patterns and perform a preliminary bilingual evaluation of its first implementation through a similar example retrieval task. The resulting pipeline is also available as a prototype CALL application.

## introduction l1-l2 parallel dependency treebanks are corpora where sentences produced by learners of a second language (l2), paired with native-like (l1) correction hypotheses, are annotated following the universal dependencies (ud) standard @xcite . this data format, proposed by @xcite , has interoperability as its main goal: ud provides a uniform annotation layer across different languages and its fine-grained morphosyntactical analysis is meant to make explicit error tagging unnecessary, preventing the incompatibilities that arise from the use of project-specific taxonomies. in addition, the availability of increasingly reliable dependency parsers can significantly speed up, if not completely automate, the annotation process. putting l1-l2 treebanks into use, however, requires effective ways to extract information from them. errors, explicitly marked in most learner corpora, are for instance not straightforward to identify in such datasets. in this paper, we report on ongoing work on this problem, focusing on morphosyntax. in particular, we propose a novel approach to locate error-correction pairs and convert them into machine-readable error patterns, which can serve as a starting point for a variety of tasks, includ-ing explainable automatic error classification and controlled feedback comment generation. we put a first implementation of this method to the test through an example retrieval task where patterns extracted from a set of example sentencecorrection pairs are used to find similar errors in an l1-l2 treebank. an interactive version of the resulting system is also made available as a prototype computer-assisted language learning (call) application, similar to @xcite 's corpus search tool for l2 japanese learners. foot_0 ## related work standardizing and automating the annotation of learner corpora is desirable for a variety of purposes. notable in this sense is errant @xcite , an automatic error annotation toolkit for learner english whose principal aim is allowing finer-grained evaluation of grammatical error correction (gec) and detection (ged) systems. errant extracts edit operations from learner sentence-correction pairs. each edit is later labelled following an error taxonomy relying solely on dataset-agnostic information such as the pos (part of speech) tag of the tokens involved. with l1-l2 parallel ud treebanks, there is no explicit error annotation step: the idea is that morphosyntactical annotation should suffice, as error can be described by means of tree patterns pairs, comparing the original learner attempt with its target l1 counterpart @xcite . when it comes to retrieving instances of specific patterns of error, a query engine was developed by @xcite . @xcite , on the other hand, used ud-annotated parallel data to automatically derive sercl, a new taxonomy of syntactic errors for automatic classification, later combined with errant's under the name of ser-rant @xcite . sercl error types i write slowly pron verb adv jag skriver långsamt ? nsubj ? advmod ? root i write slow pron verb adj jag skriver långsam* ? nsubj ? amod ? root figure 1: a correct-incorrect ud sentence pair both in english and swedish, with discrepancies highlighted in bold. ## methodology we see error pattern extraction as a two-stage process. given a learner sentence and the corresponding correction, the first step, discussed in section 3.1, is locating its problematic portions to extract error-correction pairs. as per section 3.2, the latter are then converted into machine-readable patterns. ## conclusions and future work we presented a novel approach for extracting morphosyntactical error patterns from l1-l2 parallel ud treebanks and put it to the test through an example retrieval task. while performed on datasets for linguistic acceptability judgments rather than authentic learner data, our preliminary evaluation gave promising results and provided helpful insights for the further development of the tool. future work on the extraction method itself will focus on handling nonexistent word forms and dealing with the complexity of actual l2 data. realworld l2 texts come with two main challenges: handling non-morphosyntactical errors, such as spelling mistakes and incorrect lexical choices, and isolating each of the grammatical errors occurring in the same sentence. we mentioned that our system extracts patterns of different sizes and at varying degrees of simplification, whose usefulness depends on the use case. this drives us to also investigate pattern selection and ranking. the latter, together with a more user-friendly interface, could contribute to the improvement the example retrieval pipeline to better suit the learners' needs. further improvements will require addressing the l2 parsing issues identified through the our preliminary evaluation, for instance by fine-tuning a udpipe model on l2 data, and possibly intervening on the alignment step.