Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in Grammatical Error Detection (2024)

Abstract: Grammatical Error Detection (GED) methods rely heavily on human annotated error corpora. However, these annotations are unavailable in many low-resource languages. In this paper, we investigate GED in this context. Leveraging the zero-shot cross-lingual transfer capabilities of multilingual pre-trained language models, we train a model using data from a diverse set of languages to generate synthetic errors in other languages. These synthetic error corpora are then used to train a GED model. Specifically we propose a two-stage fine-tuning pipeline where the GED model is first fine-tuned on multilingual synthetic data from target languages followed by fine-tuning on human-annotated GED corpora from source languages. This approach outperforms current state-of-the-art annotation-free GED methods. We also analyse the errors produced by our method and other strong baselines, finding that our approach produces errors that are more diverse and more similar to human errors.

## introduction grammatical error detection (ged) refers to the automated process of detecting errors in text. it is often framed as a binary sequence labeling task where each token is classified as either correct or erroneous @xcite . ged is widely used in language learning applications and contributes to the performance of grammatical error correction (gec) systems @xcite @xcite . prior research in multilingual ged has primarily operated in supervised settings @xcite @xcite , relying on human annotated data for training. despite recent efforts to obtain annotated corpora @xcite many languages still lack these resources, motivating research on methods operating without ged annotations. to overcome the absence of human annotations, researchers have explored two primary approaches. the first involves language-agnostic artificial error generation (aeg). this is achieved using rules @xcite , non-autoregressive translation @xcite , or round-trip translation @xcite . these methods are not trained to replicate human errors and compare unfavorably to supervised techniques like backtranslation @xcite @xcite which train models to learn to generate human errors. the second approach leverages the cross-lingual transfer (clt) capabilities of bert-like @xcite multilingual pre-trained language models (mplms). this involves fine-tuning a ged model on languages with abundant human annotations (termed as source languages) and evaluating their performance on languages devoid of human annotations (referred to as target languages). while certain languages exhibit unique error types, most adhere to shared linguistic rules, which mplms can exploit to detect errors across languages. in this paper, we hypothesize that error generation also share linguistic similarities across languages. we propose a novel approach to zeroshot clt in ged by combining back-translation with the clt capabilities of mplms to perform aeg in various target languages. our methodology involves a two-stage fine-tuning pipeline: first, a ged model is fine-tuned on multilingual synthetic data produced by our language-agnostic back-translation approach; second, the model undergoes further fine-tuning on human-annotated ged corpora from the source languages. we experiment on 6 source and 5 target languages and show that our technique surpasses previous state-of-the-art annotation-free ged methods. in addition, we provide a detailed error analysis comparing several aeg methods to ours. the contributions of this paper are as follows: historically, most research in ged has been concentrated on the english language. however, recently, @xcite organised the first shared task on multilingual ged in which @xcite set state-of-the-art in all non-english datasets by fine-tuning a xlm-roberta large model on human annotated data in a monolingual setting. while we follow their methodology to train our ged model, we complement prior research by exploring ged for languages lacking annotations. artificial error generation current methods for aeg can be broadly categorized into language-agnostic and language-specific approaches. language-specific methods focus on replicating the error patterns found in a specific gec corpora. this can involve heuristic approaches tailored to mimic the linguistic errors identified in gec corpora @xcite @xcite , or employing techniques such as back-translation @xcite @xcite . while effective for languages with annotated corpora, these methods are not suitable for languages lacking such resources. our work advances existing synthetic data generation methods by exploring a language-agnostic variant of back-translation. unsupervised gec unlike ged, gec without human annotations has been explored in several studies @xcite @xcite . state-of-the-art unsupervised gec systems @xcite typically begin with the development of a ged model trained on erroneous sentences generated through rule-based methods @xcite or masked language models @xcite . this ged model is subsequently used with the break-it-fix-it (bifi) method to create an unsupervised gec system. however, the methods used by @xcite for creating the ged model are not language-agnostic, as they rely on a thorough analysis of language-specific error patterns, making them difficult to apply to languages lacking such annotations. cross-lingual transfer previous studies have shown the capacity of mplms to generalize to languages unseen during fine-tuning for both nlu @xcite @xcite and generative tasks @xcite @xcite . close to our work, @xcite explored cross-lingual transfer in gec, a closely related topic. their findings indicate that pre-training with masked language modeling and translation language modeling enhances cross-lingual transfer. additionally, they show that fine-tuning on a combination of a high and a low-resource language improves the performance of gec models on the low-resource language. legend : step 3 step 4 step 2 step 1 maeg fine-tuning ged data generation ged fine-tuning on artificial data ged fine-tuning on human-annotated data target languages source languages gec unannotated raw text aeg model (generative) synthetic ged synthetic ged ged model (bert-like) ged model (bert-like) ged (from gec) in contrast to @xcite our research focuses on zero-shot cross-lingual transfer, specifically for ged and aeg, without relying on target language annotations. additionally, we advance previous work on zero-shot cross-lingual transfer by demonstrating its effectiveness in improving downstream task performance. investigating zero-shot clt in ged is particularly significant because the "translate-train" baseline @xcite , which involves training a ged model on a translated dataset, is infeasible. this arises because machine translation systems tend to correct the errors that the ged model is intended to detect. ## method our proposed ged method is developed through a four-step process, as illustrated in figure 1 . initially, we train a multilingual aeg model using gec datasets from the source languages. this aeg model is subsequently employed to produce a ged dataset encompassing both target and source languages. in the third step, we fine-tune a ged model on this multilingual artificially generated dataset. finally, we perform an additional finetuning of the ged model using human-annotated ged data from the source languages. the resultant ged model is capable of detecting errors across any target language. data our method necessitates three types of corpora. first, the aeg model is trained using gec datasets in a collection of source languages, d s , which include pairs of ungrammatical sentences and their corrected versions. additionally, monolingual corpora in the source languages ds and in the target low-resource languages dt , consisting of raw sentences, are required. step 1: maeg fine-tuning a generative mplm is fine-tuned to generate errors from a dataset d s combining all source languages. the model learns to generate errors by using corrected text as input and ungrammatical text as output. we refer to the resulting model as our multilingual artificial error generator (maeg). post-training, the maeg can introduce errors in any language supported by the mplm, leveraging the inherent zero-shot crosslingual transfer capabilities of generative mplms. step 2: ged data generation using our maeg system we obtain a multilingual dataset d synth of raw sentences and their corresponding synthetically generated ungrammatical versions by corrupting sentences from ds and dt . we obtain ged token-level annotation from d synth by tokenizing using language-specific tokenizers, and aligning both sentence versions using levenshtein distance with minimal alignment following @xcite . we follow the labeling methodology of @xcite ; @xcite . we designate tokens that are not aligned with themselves or tokens following a gap as incorrect, while remaining tokens are labeled as correct. step 3: ged fine-tuning on artificial data ged model training begins with the fine-tuning of an mplm such as xlm-r @xcite on our synthetically generated multilingual ged datasets created in step 2. ## analysis of synthetic errors we compare the errors produced by the aeg methods. we first study czech using a czech extension @xcite of the errant @xcite error annotation tool and an artificial vs human error discriminator. we then extend our analysis to many languages using @xcite to classify error types. ## conclusion we introduced a novel zero-shot approach for ged with low-resource languages. our method combines back-translation with the clt capabilities of mplms to perform aeg across various target languages. then, we fine-tune the ged model in two steps: first on multilingual synthetic data from source and target languages, then on humanannotated source language corpora. this method achieves state-of-the-art performance in annotationfree ged. our error analysis shows that we produce errors that are more diverse and human-like than the baselines. in future work, we intend to explore the potential of our ged models to enhance unsupervised gec methods.