A query engine for L 1- L 2 parallel dependency treebanks (2023)

Abstract: L1-L2 parallel dependency treebanks are learner corpora with interoperability as their main design goal. They consist of sentences produced by learners of a second language (L2) paired with native-like (L1) correction hypotheses. Rather than explicitly labelled for errors, these are annotated following the Universal Dependencies standard. This implies relying on tree queries for error retrieval. Work in this direction is, however, limited. We present a query engine for L1-L2 treebanks and evaluate it on two corpora, one manually validated and one automatically parsed.

## introduction l1-l2 parallel dependency treebanks are learner corpora where sentences produced by learners of a second language (l2) are paired with correction hypotheses, assumed to be native-like and therefore referred to as l1 sentences. both the learner originals and the corresponding corrections are annotated following the cross-lingual universal dependencies (ud) standard (nivre et al., 2020). the idea is that such morphosyntactical information makes explicit error labelling unnecessary and allows errors to instead be retrieved via tree queries. this format, proposed by lee et al. (2017a), was in fact designed to address the interoperability issues arising from the coexistence of the different markup styles and error taxonomies normally employed for the annotation of learner corpora. these tend not only to be languagespecific, but also to vary widely across different same-language projects. an additional advantage of using ud is the availability of several increasingly fast and reliable parsers (straka, 2018; qi et al., 2020). while not yet very robust to learner errors (huang et al., 2018), they can already speed up the annotation process significantly. l1-l2 ud treebanks exist for english (berzak et al., 2016), chinese (lee et al., 2017b) and italian (di nuovo et al., 2022). work on error retrieval tools, on the other hand, has been limited. only one of these corpora, the esl (english as a second language) treebank, is equipped with a query engine. @xcite this tool, however, presents several limitations, a major one being its reliance on a pre-existing error taxonomy, in contrast with lee et al. (2017a)'s idea. @xcite closer in spirit to the latter, choshen et al. (2020) have developed a method to automatically derive dynamic syntactical error taxonomies from l1-l2 treebanks, but do not provide a way to look for specific error patterns. in this paper, we present a language-and error taxonomy-agnostic query engine for l1-l2 parallel dependency treebanks. the tool allows searching for morphosyntactical errors by describing them in a pre-existing pattern matching language for ud trees, which we extend to facilitate comparing l2 sentences to their corrections, resulting in what we call l1-l2 patterns. our main contribution is a sentence retrieval algorithm that matches the l1 and l2 portions of a query pattern on the corresponding treebanks in parallel, ensuring that correspondences are found between segments that align with each other. addressing another limitation of the existing tools, we also make it possible to extract the specific portions of an l1-l2 sentence that match a given pattern. the engine is part of l2-ud, a larger open source toolkit for ud-annotated l2 data, available for download at github.com/harisont/l2-ud. @xcite sono andato da entrambi aux verb adp pron sing sing masc plur masc ? aux ? case ' @xmath0 ? obl ? ## related work as mentioned in the introduction, learner corpora exist in a variety of formats. lee et al. (2017a)'s proposal to use l1-l2 parallel dependency treebanks is not the only one aimed at overcoming the interoperability issues that follow. bryant et al. (2017), for instance, introduced errant, an er-ror annotation toolkit operating in the framework of a taxonomy that exclusively relies on dataset-agnostic information such as the pos (part of speech) tag and morphological features of the tokens involved. in a sense, this can be seen as an attempt to solve the problem by developing a "universal" error taxonomy. while errant has become dominant in grammatical error correction research, it still coexists with several other tagsets which differ significantly both in their underlying assumptions, often language-specific, and in the granularity of the annotation, which varies according to the intended use of each individual corpus. lee et al.'s idea, while only concerned with morphosyntactical errors, is more radical, as udannotated parallel treebanks have the potential to remove the need for any explicit error precategorization and instead allow to infer error taxonomies automatically and dynamically. choshen et al.'s work on syntactical error classification, showing promising results even on automatically parsed l1-l2 treebanks, goes in this direction. while there is a wide variety of tools and language to choose from for extracting information from (monolingual) ud treebanks, 4 not many options are available when it comes to retrieving example sentences matching specific patterns of error from l1-l2 treebanks. to the best of our knowledge, the above mentioned esl treebank query engine is the only tool specifically meant for this task. while it is reasonable to assume that the latter could easily be generalized to work 4 pml-tq (pajas and štěpánek, 2009), grew-match (guillaume, 2021), sets (luotolahti et al., 2015) and tün-dra (martens, 2013), just to name a few. with any l1-l2 treebank, it presents several limitations from the perspective of the tree queries envisioned by lee et al. (2017a). first and foremost, searching for errors is primarily done by selecting an error label from a pre-defined set. a simple query language is also available, but it only allows searching for sequences of word forms, pos tags and dependency labels. in other words, ud sentences are treated as lists of tokens rather than trees. this can be restrictive since, for the purposes of grammatical error retrieval, dependency structure is often more relevant than linear order. furthermore, there is no coupling, other than sentence-level alignment, between the l1 and l2 parts of the treebank. patterns are therefore only matched against l2 sentences, making it impossible to search for errors whose description requires a comparison with the correction (cf. figure 1 for an example) or locate the relevant portions of their l1 counterparts. a final, related limitation is that the tool always returns complete sentences, while it is sometimes useful to isolate the segments that match the query. ## design and implementation addressing these limitations, we aim for a query engine with the following characteristics: därför vill jag inte flytta adv aux pron part verb ' @xmath1 ? aux # ? nsubj ? advmod ? root därför jag vill inte flytta* adv pron aux part verb ' @xmath2 ? nsubj # ? aux ? advmod ? root figure 2: ud trees for the l1-l2 swedish sentence därför {vill @xmath3, discrepancies highlighted in bold. the l2 sentence, on the right, violates v2 word order. example adapted from the dalaj corpus. ## evaluation aiming at assessing the performance of the query engine, we tested it on two l1-l2 error-tagged corpora in two different languages, one that comes with manually validated ud annotation and one that was only parsed automatically. in both cases, we randomly selected 100 sentences to be used during development and set the rest aside for testing. carrying out a systematic evaluation was not possible: more often than not, an error tag maps not to a single l1-l2 query, but to a potentially rather large set of queries whose exhaustiveness is hard to verify. as a consequence, we opted for computing the sentence-level precision and recall obtained upon running, for each corpus, a single-token, a tree and a sequence example query, all chosen to be descriptive of an error typical of the language at hand. to automate evaluation as much as possible, we also tried to make each query match one of the error labels of the dataset at hand. in this way, performance for a given query can be assessed by simply comparing the sentence ids returned by the engine with those of the sentences marked with the corresponding error label. finding exact correspondences was feasible for single-token queries, but challenging for tree and sequence patterns, which tend to be finer-grained. in such cases, we formulated queries describing a subset of the error cases denoted by a certain label and manually inspected sentences marked with it to select the relevant items. by comparing the results obtained on the two corpora, we also aim at getting insights about the ways in which automatic annotation affects the performance of the engine, even though we cannot quantify its impact. ## conclusions and future work we presented a query engine for l1-l2 parallel ud treebanks, the first in a larger collection of tools for l2 ud treebanks. the tool, which does not rely on an underlying error taxonomy, allows to search for error-correction pairs via l1-l2 patterns, i.e. pairs of morphosyntactical structures expressed in a pattern matching language for dependency trees, which we extended in order to simplify its use on parallel treebanks. our novel retrieval algorithm allows searching for full sentences as well as extracting their specific querymatching portions. our first, small-scale evaluation of the tool gives promising results, but also shows that the alignment component is often the bottleneck. this propmts us to investigate alignment techniques specifically meant for parallel learner corpora, such as felice et al. (2016)'s for l2 english. the fact that, for automatically annotated data, many alignment issues derive from parse errors also seems to confirm the scarce robustness of standard tools to learner errors, pointing to a need to train ad hoc models or explore new, more specific approaches. in future versions of the tool, we plan to optimize variables and generalize them to all ud fields, thus increasing expressive power of the query language. furthermore, while the tool is designed with l1-l2 treebanks in mind, nothing prevents us from testing it on multilingual parallel ud treebanks, for example to find instances of known translation divergences. as for the future of l2-ud, our efforts in the near future will be focused on extracting error patterns from l1-l2 treebanks. eventually, we hope it will also be possible to integrate the two and enable using error-correction pairs to retrieve similar examples. this would help making the engine more user-friendly, replacing explicit queries, but could also be a strategy to provide l2 learners with feedback, along the lines of arai et al. (2019).