Ensemble Self-Training for Low-Resource Languages: Grapheme-to-Phoneme Conversion and Morphological Inflection (2020)

Abstract: We present an iterative data augmentation framework, which trains and searches for an optimal ensemble and simultaneously annotates new training data in a self-training style. We apply this framework on two SIGMORPHON 2020 shared tasks: grapheme-to-phoneme conversion and morphological inflection. With very simple base models in the ensemble, we rank the first and the fourth in these two tasks. We show in the analysis that our system works especially well on low-resource languages.

## introduction the vast majority of languages in the world have very few annotated dataset available for training natural language processing models, if at all. dealing with the low-resource languages has sparked much interest in the nlp community @xcite @xcite . when annotation is difficult to obtain, data augmentation is a common practice to increase training data size with reasonable quality to feed to powerful models @xcite @xcite . for example, the data hallucination method by @xcite automatically creates non-existing "words" to augment morphological inflection data, which alleviates the label bias problem in the generation model. however, the data created by such method can only help regularize the model, but cannot be viewed as valid words of a language. orthogonal to the data augmentation approach, another commonly used method to boost model performance without changing the architecture is ensembling, i.e., by training several models of the same kind and selecting the output by majority voting. it has been shown that a key to the success of ensembling is the diversity of the base models @xcite , since models with different inductive biases are less likely to make the same mistake. in this work, we pursue a combination of both directions, by developing a framework to search for the optimal ensemble and simultaneously annotate unlabeled data. the proposed method is an iterative process, which uses an ensemble of heterogeneous models to select and annotate unlabeled data based on the agreement of the ensemble, and use the annotated data to train new models, which are in turn potential members of the new ensemble. the ensemble is a subset of all trained models that maximizes the accuracy on the development set, and we use a genetic algorithm to find such combination of models. this approach can be viewed as a type of selftraining @xcite , but instead of using the confidence of one model, we use the agreement of many models to annotate new data. the key difference is that the model diversity in the ensemble can alleviate the confirmation bias of typical self-training approaches. we apply the framework on two of the sigmor-phon 2020 shared tasks: grapheme-to-phoneme conversion @xcite and morphological inflection @xcite . our system rank the first in the former and the fourth in the latter. while analyzing the contribution of each component of our framework, we found that the data augmentation method does not significantly improve the results for languages with medium or large training data in the shared tasks, i.e., the advantage of our system mainly comes from the massive ensemble of a variety of base models. however, when we simulate the low-resource scenario or consider only the low-resource languages, the benefit of data augmentation becomes prominent. ## conclusion we present an ensemble self-training framework and apply it on two sequence-to-sequence generation tasks: grapheme-to-phoneme conversion and morphological inflection. our framework includes an improved self-training method by optimizing and utilizing the ensemble to obtain more reliable training data, which shows clear advantage on lowresource languages. the optimal ensemble search method with the genetic algorithm easily accommodates the inductive biases of different model architectures for different languages. as a potential future direction, we could incorporate the framework into the scenario of active learning to reduce annotator workload, i.e., by suggesting plausible predictions to minimize the need of correction.