An Investigation of Noise in Morphological Inflection (2023)

Abstract: With a growing focus on morphological inflection systems for languages where high-quality data is scarce, training data noise is a serious but so far largely ignored concern. We aim at closing this gap by investigating the types of noise encountered within a pipeline for truly unsupervised morphological paradigm completion and its impact on morphological inflection systems: First, we propose an error taxonomy and annotation pipeline for inflection training data. Then, we compare the effect of different types of noise on multiple state-of-the- art inflection models. Finally, we propose a novel character-level masked language modeling (CMLM) pretraining objective and explore its impact on the models’ resistance to noise. Our experiments show that various architectures are impacted differently by separate types of noise, but encoder-decoders tend to be more robust to noise than models trained with a copy bias. CMLM pretraining helps transformers, but has lower impact on LSTMs.

## introduction neural morphological inflection has shown impressive results for a huge variety of languages @xcite . performance is impressive even for languages with very little supervised inflection data, and often generalizes to unseen lemmas. however, the language settings that arguably stand to benefit the most from these tools, those with extremely sparse normalized texts, are less likely to have clean, goldstandard data. despite this, inflection training data noise is rarely addressed nor evaluated in popular benchmarks. noise, like incorrect annotations, or mixed dialects or orthographies, can arise in inflection data from web-scraping issues @xcite , human error or changes in writing standards in field data @xcite , or system errors when bootstrapping silver-standard data in an unsupervised fashion @xcite @xcite . unsupervised systems are also prone to over-regularization, where the dataset contains few or no irregular samples. datasets derived from fsts or textbook examples can also display over-regularization @xcite . in this work, we investigate the impact of noise on inflection generation systems. we build an automatic pipeline for annotating inflection noise and explore the noise distribution that arises in an unsupervised system for bootstrapping inflection data. we measure the impact of noise on several stateof-the-art neural inflection generation systems that we benchmark on the sigmorphon 2017 shared task development set @xcite . finally, we explore a novel character-level masked language modeling (cmlm) pretraining objective to mitigate the impact of noise during training. by this, we aim to shed light on which architectures and training methods are how robust to noise, which types of inflection noise should be targeted in filtering approaches, and how conservatively unsupervised systems should sample inflection pairs. we find that noise related to slot alignment issues is more common, but also less impactful than noise related to paradigm induction issues. architectures with an inductive bias towards copying from the lemma are more effective on datasets that lack sample diversity, but more typical encoderdecoder models are more robust to noise. standard encoder-decoders display better performance on noisy data when pretrained with cmlm, which is especially effective for the transformer. our code and data are publicly available. 1 2 noisy training data our data comprises four languages: icelandic, german, swedish, and russian. we describe the source of training data below. finally, tumpc aligns similarly inflected forms across paradigms that belong to the same pos. the result is a dataset of paradigms for which each type is marked with its inflectional slot. we take all possible pairs of words from the tumpc paradigms to form inflection training data. this dataset is likely to contain noise due to errors in the learning process. in §4, we discuss this noise in detail. unimorph tumpc relies on frequency thresholds to find productive inflection transformations, which makes it likely that morpho-phonological variations and irregular inflections may not be wellattested in the data. to control for the potential lack of sample diversity in our training data, we create a second training dataset wherein all samples marked as correct are replaced by a pair sampled from unimorph @xcite , a database of morphological paradigms covering hundreds of languages. we sample pairs from the same msds as the tumpc correct samples. following recent results @xcite @xcite , we also ensure that the lemma overlap with the evaluation set is the same as in the original tumpc data. the resulting dataset has higher lemma diversity, without introducing missing msds, and maintains lemma overlap with the evaluation set. we combine this data with the noisy training data from tumpc to investigate the impact of noise when sample diversity is less pervasive. slot mapping tumpc slots are arbitrary identifiers with no grammatical meaning. in order to compare tumpc to unimorph, and because we evaluate on a benchmark sampled from unimorph, we need to map each tumpc slot to a unique uni-morph slot. unimorph slots are bundles of morphological tags that express categories like person or tense. we refer to these bundles as morphosyntactic descriptions (msds). since the tumpc training pairs have low overlap with unimorph, we require a morphological analyzer to get ground truth morphological tags of tumpc types. here, we use morphological analyzers from apertium @xcite , which we map to unimorph msds. then, we apply the mapped apertium to all tumpc types, resulting in ground truth msds for every training sample in tumpc. we also want the slots predicted by tumpc expressed as unimorph msds during training. for this, we compare the unsupervised slot and gold msd of all tumpc types and find the alignment that maximizes their overlap following the evaluation metric from @xcite . every training sample can then be assigned the (noisy) msd from this alignment. for more details see section a.1 in the appendix. many inflection pairs created by tumpc could not be reliably annotated by our pipeline and are thus filtered out. learning from noisy data several approaches have been proposed for mitigating the impact of noise in machine learning, for example: confidence weighting @xcite , loss correction @xcite , and noise-contrastive estimation @xcite . for a recent survey on noise robust neural networks, see @xcite . most approaches consider classification, framing noise as label corruption. however, there is also work exploring noise in tasks like machine translation @xcite , as well as morphological disambiguation @xcite . we focus on morphological inflection, a conditional generation problem. ## noise taxonomy we develop an automated pipeline to annotate each inflection pair from a taxonomy of noise, primarily relying on rule-based morphological analyzers for each language from apertium. here we describe each type of noise in our taxonomy, which we organize into three categories: lemma errors, paired errors, and msd errors. for a description of how each type of noise is detected, see section appendix a.2 in the appendix. ## discussion we find that low sample diversity has a strong impact on performance of all models. the tumpc training setup favors architectures with a copy bias, and demonstrates that models can learn from noisy training samples when the dataset is not diverse. we find that the low lstm and trm performance is largely explained by low sample diversity. on the other hand, they seem to be more robust to noisy data, particularly lstm, which has stable performance as noise is added to the unimorph dataset. pretraining with cmlm leads to further gains in performance for lstm, but the largest gains from cmlm are for trm. on average, trm pretrained with cmlm is the best performing model under noise, when we have sufficient sample diversity. when we look at specific noise in the data, we find that slot alignment issues in tumpc tend to have low impact on every model. this could be for several reasons: slot noise should be irrelevant under high amounts of syncretism, which is abundant in german, icelandic, and swedish. additionally, as models become more corrupted by noise, they may rely on a bias towards copying from the source form -which is unaffected by the slot. we find that certain noise that results from errors in paradigm induction are particularly impactful for m&c. this is especially true for pos noise, which may motivate better pos induction in unsupervised morphology systems. under greater sample diversity, trm is similarly impacted by paradigm induction errors. this suggests that trm begins to learn an inductive bias similar to m&c on noisy data. the addition of almost any single noise annotation leads to an increase in accuracy in ptrgen, and removing noise from the training data often impacts ptrgen negatively. however, experiment 2 suggests that ptrgen is not as robust to increasing amounts of noise as other models. a lot of reduction in accuracy may come from combinations of different noise types, which is not captured by experiment 3. future work could investigate noise distributions by type with particular focus on ptrgen behavior. we additionally find that m&c is negatively impacted by cmlm pretraining. we believe this may be due to overfitting its copy bias and learning spurious transductions from the masking objective. however, future work should consider alternate pretraining strategies for m&c. overall this implies that, although copy models are preferred for training on low sample diversity, classic encoder-decoders are a good choice for noisy datasets with more diversity. our results indicate that pos and paradigm induction components are more important for training data quality than slot alignment in unsupervised systems and that bootstrapping inflection pairs should prioritize lemma diversity, even if it may induce noise. ## conclusion we have investigated the impact of noise on stateof-the-art neural morphological inflection models. we find that the noise that arises in an unsupervised system for bootstrapping inflection pairs is frequently related to slot alignment errors, but that those also have less impact on the models. we have also compared two inflection architectures with a copy bias to two typical encoder-decoder models. we find that, though copy bias is helpful under low sample diversity, the encoder-decoders are more robust to noise. finally, we find that a simple masked pretraining objective makes encoderdecoders, and especially transformers, more accurate under noise.