Generation of Synthetic Error Data of Verb Order Errors for S wedish (2022)

Abstract: We report on our work-in-progress to generate a synthetic error dataset for Swedish by replicating errors observed in the authentic error annotated dataset. We analyze a small subset of authentic errors, capture regular patterns based on parts of speech, and design a set of rules to corrupt new data. We explore the approach and identify its capabilities, advantages and limitations as a way to enrich the existing collection of error-annotated data. This work focuses on word order errors, specifically those involving the placement of finite verbs in a sentence.

## introduction the lack of sufficient data to train algorithms capable of detecting, labeling and correcting grammatical errors calls for the need to generate synthetic (i.e. machine-made, not human-produced) error datasets to enrich the existing resources. as mentioned by @xcite , the need for synthetic datasets (aka corrupt or artificial datasets) exists not only for low-resource languages, but also for high-resource languages like english. this is due to the fact that data for error detection and correction is far more sparse than required for most tasks in nlp, as grammatical errors are found in different frequencies and distributed unevenly across written language. moreover, the appearance of grammatical errors in student essays depend notably on the speaker's particularities, such as their proficiency level, native language(s) and age. the need is especially acute for languages that are on the low-resource end in this respect, as is the case for swedish. in this paper, we present a pilot study to generate artificial error data for swedish by mimicking error patterns present in authentic error datasets, namely, in the swell learner corpus @xcite and its one-error-per-sentence dalaj derivative @xcite . we create a corruption pipeline to insert artificial errors into the sentences from coctaill, a corpus of textbooks used for teaching swedish @xcite . we expect the artificially produced error data to be a valuable resource for such tasks as grammatical error detection / labeling (ged) and grammatical error correction (gec) for swedish, which at the moment are dormant fields. in this pilot, we focus on word order errors involving placement of finite verbs (tagged s-finv). the final dataset comprises 31,788 corrupted sentences each containing one error of the syntactical error type "s-finv", paired with their correct counterparts. the code and the generated data can be found on github foot_0 . ## related work recently much attention has been given to practical and theoretical aspects of artificial error data generation as a way to enhance performance of grammatical error correction systems, both with respect to methods of generation, source (aka seed) corpora used for corruption and the ways pseudodata is used in system architectures (e.g. @xcite . @xcite give probably the most nuanced introduction to the problem. approaches to generation of synthetic error datasets can be roughly divided into rule-based and model-based ones, which further exhibit variation with regards to presence or absense of error labels. advantages of model-based approaches (e.g. @xcite ) is that they capture the variety of error types present in the authentic data and the artificial data is fast to generate. however, training a model for replicating errors requires access to large amounts of such data, which often is a problem to start with. it has also been observed that models may show biases towards the data they have been trained on, with a consequence that they are not general enough for unseen contexts or domains (e.g. @xcite . advantages of rule-based approaches (e.g. @xcite are directly opposite, namely, that they can be created with zero or minimal access to gold data and can better generalize since they are kept on an abstract level. some known rules include simple random operations, e.g. deletion of a word, randomly swapping neighbouring words, exchange of one inflected form with another or use of so-called confusion pairs, i.e. incorrect segment/@xmath0.g. @xcite . a more linguistic approach to error rules, e.g. through abstracting to part of speech (pos) patterns or patterns including morpho-syntactic information, requires more time for designing corruption rules, but has one obvious advantage: using such rules allows control over the generated data and, importantly, it is possible to add error labels to corrupted sentences, which makes the pseudo-data applicable both to error correction and to error classification tasks. it also has an advantage of inserting realistic errors typical of learners, and has been shown to increase performance of gec systems, compared to random error types @xcite . given the scarcity of the swedish authentic data, we experiment with rule-based approaches using linguistic analysis to extract typical error patterns and to generate synthetic errors based on those. ## data and resources the overview of the pseudo-data generation is shown in figure 1 . on the left, the top level shows steps for working with learner data, starting from the swell data, including preparation of one-errorper-sentence dalaj dataset, analysis of a 100 cleaned dalaj samples for identification of error patterns, and production of corruption rules. the bottom level on the left shows a parallel work involving selection of a seed corpus with correct language and preselection of sentences for corruption. on the right, the graph shows the process for corruption of the seed data and its subsequent cleaning and evaluation. to perform the task at hand, thus, two main sources of textual data are needed: a corpus of tagged errors and a corpus of clean (i.e., error-free) texts. additionally, a part of speech (pos) tagger to extract grammatical information is required. ## results a total of 31,788 sentences were corrupted from the 20,307 usable sentences available. the distribution of error sub-types is shown in similarly to the frequency distribution in student essays, pronoun-dependent verb order errors are the most frequent ones in the corrupted data, with 41.05% of synthetic errors being of this type. the second most productive rules are the ones involving adverbs, with 31,21% of errors, followed by nouns at 25,3%. finally, as expected, the corruption pipeline produced a considerably lesser quantity of errors involving proper names at 2,44%. the distribution in the corrupted data, thus, reflects the observed tendency in the authentic data. to assess the quality of the corruption method, we carried out a small-scale evaluation. two people have independently checked 100 randomly selected corrupted sentences in terms of how similar they are to hypothetical learner-made errors (i.e. to make sure they are high quality). following @xcite , we used a three level scale of assessment: good, acceptable and bad. for acceptable and bad, a reason could be indicated for further analysis. the evaluation shows that 76% (67%) of sentences are good, 14% (25%) are acceptable and 10% (8%) are bad. the numbers in brackets come from the second annotator. some observed problems had to do with more complex phrase shifts that were missed. in others, the problem comes from the source data, incl. unfinished sentences with an uncertain sentence type (affirmative vs interrogative), which then sounds correct even if the verb and noun change places. it should be noted that the main purpose of this evaluation was to see whether humans think that the synthetic data will be useful for training algorithms, and the result where on average 90% sentences are either good or acceptable is very encouraging. it has been earlier claimed that even unrealistic errors are useful for pre-training gec models (e.g. @xcite . given our results, therefore, we consider the produced dataset appropriate for the task. we have run the first experiments exploring effects of pseudo-data on the model performance for the task of error detection and classification, where classification is limited to the top error categories (orthographic, lexical, morphological, punctuation, syntactical). detailed description of that experiment is the topic of another publication, however, we can shortly name here that we have observed a tangible improvement of the classification results when 500 fakedalaj sentences of s-finv nature were added to the training data. when more sentences were added, the models seemed to learn to classify syntactical errors disadvantaging other error types. a sample of the results obtained, measured with the f0.5 score, are shown in ## conclusions and future work this paper introduces a process for generation of synthetic error datasets with corresponding error labels based on linguistic analysis of real-life learner errors in the context of limited error-annotated learner data. this process could be replicated for other error tags, or extended and adapted to other low-resource languages. manually studying and designing corruption rules is time-consuming and can be inaccurate due to human error and language biases. therefore, an alternative to optimize time and avoid human mistakes could be to rely on guided models as suggested by @xcite or @xcite . however, we have to adhere to rule-based approaches due to the lack of sufficient amount of gold data. yet, we foresee considerable benefits of generating realistic errors. the resulting fakedalaj (s-finv) dataset is released for public use. 6 currently, we are testing this dataset in a task for error detection and classifi-cation. in the near future, we will also release a set of cleaned 100 dalaj sentences per each error tag in the swell-gold data, so that the community of interested researchers and developers can use them for generation of synthetic datasets for other error types.