Detecting Annotation Errors in Morphological Data with the Transformer (2022)

Abstract: Annotation errors that stem from various sources are usually unavoidable when performing large-scale annotation of linguistic data. In this paper, we evaluate the feasibility of using the Transformer model to detect various types of annotator errors in morphological data sets that contain inflected word forms. We evaluate our error detection model on four languages by introducing three different types of artificial errors in the data: (1) typographic errors, where single characters in the data are inserted, replaced, or deleted; (2) linguistic confusion errors where two inflected forms are systematically swapped; and (3) self-adversarial errors where the Transformer model itself is used to generate plausible-looking, but erroneous forms by retrieving high-scoring predictions from the search beam. Results show that the Transformer model can with perfect, or near-perfect recall detect errors in all three scenarios, even when significant amounts of the annotated data (5%-30%) are corrupted on all languages tested. Precision varies across the languages and types of errors, but is high enough that the model can be very effectively used to flag suspicious entries in large data sets for further scrutiny by human annotators.

## introduction deep learning models have been responsible for state-of-the-art performance in many tasks involving morphological generation and analysis @xcite @xcite . however, to reach adequate performance, large amounts of labeled examples are usually required for training @xcite @xcite . annotation of morphological data is particularly expensive since it requires both domain and language expertise @xcite . manual correction and quality control of annotated data adds to the cost @xcite . in light of this, we evaluate the feasibility of using a deep learning model to automatically detect annotation errors with the goal of reducing the cost of annotation correction and quality control. earlier work on annotation error detection has largely been non-neural and focused on other types of annotation, such as part-of-speech (pos) tagging @xcite @xcite , syntactic parsing @xcite , or semantic labeling @xcite . a neural model error detector-an lstm-based tagger-has been used by @xcite to detect pos tagging errors. in this paper, we propose a method to apply a transformer model @xcite to detect annotation errors in morphological data. in order to evaluate the method, we simulate errors by introducing artificial perturbations to our annotated data, which are generated in three different ways to simulate different types of annotation errors. experimental results show that the transformer model can detect annotation errors in morphological data very effectively, even when the datasets contain a high percentage of erroneous forms. ## results and discussion figure 2 provides a summary of the experiment results, plotting the accuracy, precision, recall, and f1-score for each pos of each language, averaged across the m models after adding artificial errors i, ii, iii at different amounts, respectively. detailed numbers are provided in we observe that the accuracy of the model decreases as more word erroneous forms are added, but is still high overall. this indicates that the leaven-out training strategy is robust to noise in the data. for every type of artificial error, the recall is 1.0 or very close to 1.0 after varying amounts of noise is injected. in other words, the model can identify all, or nearly all the artificial errors we introduce, even when a large amount of noise is mixed into the gold data. the precision increases (from a low of 0.11 to a high of 0.95) as more errors are added, indicating that a reasonably small amount of false positives would be produced by the model. (see as such, if an annotator were to manually correct the forms flagged by the model, all erroneous annotations would be corrected and the annotator should not be frustrated by vetting a large number of already-correct annotations. to illustrate this, consider the average precision (0.43) for all six datasets with artificial error type i (typos) where 1% of the forms are corrupted-a plausible scenario in an annotation project. under such assumptions, our model would present flagged forms in a dataset for vetting to an annotator, and, indeed, nearly half of these flagged forms would be true errors, and no errors would be undetected (since the recall is 1.0). however, we observe that the worst case (e.g. lowest f1 scores on average) where the annotation error detection model performs is the second type of artificial error. in this type of error, we consistently switched a portion of slots. the worst error detection model performance on this type of error points to the limitation of the annotation error detection method we propose: it cannot detect consistent errors if the errors in question are present in a large portion of the data; for example, in the extreme case that all the forms in the paradigm carry the same error, it is impossible for the inflection model to learn the ground-truth inflection. another shortcoming of our proposed approach is that it requires relatively complete inflection tables, which are expensive to annotate as to expertise and effort. future work is needed to evaluate whether the method works when there are slots missing in most inflection tables. ## conclusion in this work, we propose a method to leverage the transformer model architecture for annotation error detection in morphological data. we propose to systematically leave out one slot in each morphological inflection table as the data to be detected and use such subsets of annotated data to train individual transformer inflection models-one for each group of missing slots-and then apply the inflection models to make predictions for the heldout slots. if the predicted form disagrees with the actual annotation (a form the predicting model has not seen), the model flags that form as erroneous. to check efficiency, we evaluate the model under three different scenarios where we inject artificial errors into gold data, simulating noisy data resulting from an annotation process: typographic errors generated by inserting, replacing or deleting a single character in an inflected word form; errors resulting from annotator confusion where two slots in an inflection table are swapped; and selfadversarial errors where erroneous but plausible predictions generated by the transformer inflection model are introduced. our experiments on four languages with different morphological characteristics and levels of irregularity indicate that the proposed method can detect every type of error in morphological datasets very effectively. even when large portions of the data (5% to 30%) have been replaced with corrupted forms, our model retains perfect, or near-perfect, recall and also shows increasingly higher precision as more erroneous forms are present. the results show that the transformer model can detect various kinds of errors without producing excessive false positive predictions. we believe such a model can directly be incorporated into the correction and quality control process of morphological data annotation projects, specifically for low-resource language where datasets are in the early stages of development and few annotators are available. further research should investigate how well this basic method of error detection works in other linguistic annotation domains.