Deep Active Learning for Morphophonological Processing (2023)

Abstract: Building a system for morphological processing is a challenging task in morphologically complex languages like Arabic. Although there are some deep learning based models that achieve successful results, these models rely on a large amount of annotated data. Building such datasets, specially for some of the lower-resource Arabic dialects, is very difficult, time-consuming, and expensive. In addition, some parts of the annotated data do not contain useful information for training machine learning models. Active learning strategies allow the learner algorithm to select the most informative samples for annotation. There has been little research that focuses on applying active learning for morphological inflection and morphophonological processing. In this paper, we have proposed a deep active learning method for this task. Our experiments on Egyptian Arabic show that with only about 30% of annotated data, we achieve the same results as does the state-of-the-art model on the whole dataset.

## introduction recently, there has been lots of interest in morphological (re-)inflection processing @xcite @xcite . having an acceptable model for morphological processing will help improve the performance of different natural language processing (nlp) tasks like speech synthesis @xcite , morphological disambiguation @xcite , and machine translation @xcite @xcite . despite recent progress in this field of study, there are lots of challenges for low-resource languages. especially for utilizing successful but data-hungry deep learning (dl) models, the need for annotated data is vital. however, data annotation is a hard, expensive, and time-consuming task. in addition, lots of annotated data does not contain useful information for improving the quality of a learning algorithm. in this paper, we propose a deep active learning (dal) algorithm for morphophonological processing that is able to decrease the need for annotated data, by using only informative samples. in our experiments, we have chosen arabic, a morphologically rich language. in addition, lots of arabic dialects are very low-resource and the results from this study can help in building required datasets in a smarter way. among arabic dialects, cairene egyptian arabic has been selected, because it is well-studied and has many resources and it is appropriate for our dal simulation experiments. it should be noted that the proposed method is not specific to arabic and it can be utilized on other languages or dialects. as our baseline, we have chosen a very successful transformer model for character-level transduction tasks @xcite . we propose a pool-based dal method in this study. to find the most uncertain (informative) samples, we combine an entropy strategy with a clustering method to keep an acceptable balance between the uncertainty and diversity of the chosen samples. the results of our experiments on the selected dataset show the success of the proposed dal method. ## previous work in this section, we give a brief review of morphological inflection processing methods. some recent dal methods will also be reviewed. there are several approaches for applying dl models for the morphological inflection problem @xcite @xcite @xcite , which achieve successful results on different languages. most of these models use character-level neural transducers using transformers, data augmentation, and recurrent neural network (rnn)s. in arabic, there is also much non-neural research on morphological modeling, including finite state technology @xcite , precompiled tabular morphological analyzers @xcite @xcite , and allomorphy modeling through linguistically descriptive rules @xcite . in recent years, dal has been used in some sub-fields of nlp. @xcite and @xcite achieved satisfactory results in text classification using active learning (al) with convolutional neural networks and adversarial uncertainty sampling, respectively. different acquisition functions using the conditional random field model have been applied in named entity recognition @xcite @xcite . in neural machine translation, @xcite used an attention-based function, and @xcite applied a reinforcement learning method. in another study, @xcite proposed a word frequency-based acquisition function to train neural machine translation actively. more recently, @xcite introduced a dal method for lemma inflection in written modern standard arabic and some other languages. their method uses entropy at the word level, while we find that the max of character-level entropy for a word performs best. ## dataset we use the arabic morphophonology dataset created by @xcite . it uses a broad phonemic transcription. they generated urs from the calima egy morphological analyzer @xcite for every sf extracted from the ecal dataset @xcite . they also added the analyzer's segmentation to the ur part, delimiting word boundaries with #, prefixes with -, and suffixes with =. the dataset contains pairs of (ur, sf) as shown in ## proposed method in this section, we give a brief description of the baseline network. then, the proposed dal method will be explained in more detail. ## experimental results the details of the model's parameters and other variables in our experiments can be seen in appendix a. to evaluate our proposed dal method, we have run all experiments 5 times and the average and standard deviation of accuracy are visualized in figure 1 . we have reported random training (passive learning), al with entropy, and al with combined entropy and clustering method for each cycle of training. as can be seen in figure 1 , the proposed dal method (with and without clustering) grows much faster than the random curve and presents an asymptotical shape which shows that it has extracted all the useful information present in u when it reaches the asymptote using only 4,000 samples (i.e., about 30% of the training set). in contrast, the random (passive) learner requires the entire training set to achieve maximum performance. this is true for all evaluation sets, except for eval-oov, where the random (passive) learner reaches maximum performance after 6,000 samples. we have no explanation for this unusual behavior of eval-oov, but we do observe that the overall top performance on eval-oov is 2 percentage points lower than on dev-oov, while eval and dev have similar results. this supports the hypothesis that eval-oov contains some very difficult cases which the underlying system (with active or passive learning) cannot learn, so that active and passive learning converge earlier. unfortunately, we could not obtain evidence from this dataset that the proposed hybrid method (combining entropy and clustering) is effective. our model is designed to be language and model independent. it can be applied on different languages and it can be used on top of different character-based dl models. however, the proposed method should be applied on new languages and dialects in further studies. we expect that further studies will show that clustering can be effective, especially if many diverse phenomena are present in the data. ## error analysis we performed an error analysis on 100 randomly chosen errors from the dev-oov set, generated by models trained on 1,900 samples using al with entropy and random selection methods. on this sample size, the random selection approach achieves an error rate of 12.9%, which reduces to 7.8% through al (a 40% error reduction). we distinguish three basic types of errors: the system deletes a letter (i.e., phoneme) it should not; the system changes a letter it should not; the system adds a letter it should not, or does not delete a letter it should. we summarize results in