Observing the Learning Curve of NMT Systems With Regard to Linguistic Phenomena (2021)

Abstract: In this paper we present our observations and evaluations by observing the linguistic performance of the system on several steps on the training process of various English-to-German Neural Machine Translation models. The linguistic performance is measured through a semi-automatic process using a test suite. Among several linguistic observations, we find that the translation quality of some linguistic categories decreased within the recorded iterations. Additionally, we notice some drops of the translation quality of certain categories when using a larger corpus.

## introduction during the last years, neural machine translation (nmt) has seen immense progress and achieved high performance. as most machine learning methods, nmt is based on an iterative training process that learns to translate given big amounts of parallel corpora. despite the remarkable achievements of the training process in terms of producing a model able to translate, it is used as a black box. this is also due to the fact that it is training a neural network, one of the least interpretable machine learning algorithms. thus, little effort has been done in order to investigate how the training process evolves with regards to measurable factors of translation quality, such as the rules of linguistic correctness (grammar, syntax, semantics). in particular, the training process performs several iterations through which the neural network weights are gradually adjusted to achieve the optimal performance for the training data seen at the moment. after several iterations, the performance of the model, with its current weights, is typically validated against a development set, using some automatic metrics (cross entropy or bleu score; @xcite , which may also define whether the optimal conditions have been reached and training should stop. although these automatic metrics have been proven useful for the training process itself, they provide a single number for a generic notion of the translation quality. as specified, we are interested in observing the training process from a more fine-grained perspective and particularly how it proceeds with learning specific linguistic phenomena. this work is intended to provide nmt researchers and engineers with additional guidance on what to look for when evaluating and designing machine translation systems. this is a preliminary work towards this direction, aiming to investigate how the training process evolves with regards to linguistic performance for several phenomena. we do this by selecting snapshots of particular training epochs and evaluating these snapshots with test suites, which probe the translation of specific linguistic phenomena. as a result, we can observe the learning curve of those linguistic aspects, along with strengths and weaknesses. we find that as the training ends and the bleu score reaches the maximum value, some linguistic categories experience a drop in their accuracy. additionally, we notice further drops of the translation quality of certain categories when using a larger corpus. finally, we provide further observations on particular linguistic phenomena, by focusing on certain test items. our experiment is focusing on the language direction @xmath0. in the next section (section 2) we review related work. section 3 presents our used methods, while in section 4 the experiment setup is further discussed. we present our results in section 5 and compare the different models in section 6, followed by a short conclusion and notes on further work in section 7. ## methods we are interested in observing the learning curve of neural machine translation with regards to linguistic phenomena. particularly, the aim is to examine how the linguistic performance of a translation model improves along the iterations of the training process. in order to do that, we perform the following steps: by comparing the statistics from the fine-grained evaluation for various snapshots, we intend to get insights with a linguistic perspective in the machine learning process. we can only evaluate particular snapshots, since the functioning of the test suite tool allows semi-automatic error annotation and there is still need to manually evaluate some uncertain decisions and edge cases. to decide which snapshots to pick, we relied on the use of bleu score as a first indicator, despite its limitations. additionally, we build several systems with different architectures and corpus sizes to allow further comparisons. this being a student experiment, the computational and time restrictions allowed a limited number of models trained with an amount of data that is smaller than the state-of-the-art. however, that should serve as proof of concept. despite the models not being state-of-the-art, our focus remains on the evolution of the linguistic performance, starting from the early steps of the training process. in our experiments we will have three systems: a small rnn model trained on a small amount of corpora, a bigger rnn model with more data than the former, and a transformer model. technical details for these models are given further in section 4.3. ## comparison between iterations and models as figure 7 shows, there is a clear difference between the two rnn trainings regarding the resilience of the bleu score over time. while there is a lot of jittering in the rnn model with a small amount of data, a nearly constant increase is given in the model with a bigger amount of data, showing no huge peaks or valleys. regarding bleu scores, the system with the larger corpus performed a little bit better (∼16) than the one smaller one (∼14), but this was not reflected in the test suite comparison, where there was no big difference in terms of test suite accuracy, even though the used corpus has more than doubled in the big rnn model. additionally, it can be observed that some categories in the bigger rnn perform worse than what was achieved in the smaller one. the inability of the bigger model to take advantage of the additional data may be addressed to the rather shallow architecture of the encoder and the decoder. with the current range of experiments, there are some open questions regarding further comparisons between rnn-small and rnn-big models. future experiments could investigate the reasons for the fact that rnn-small and rnn-big systems perform comparably on the test suite, e.g. whether this can be attributed to the shallow architecture, to a subtle domain mismatch between europarl and dgt or to the domain mismatch between the training data and the test suite. the transformer model development takes some more iterations until it reaches a competitive bleu score but then clearly outperforms both rnn systems by more than 60%, although the comparison with the rnn models is not direct, since the transformer is built with more layers and a not directly comparable architecture. there is no generalizable development over all examined categories; some performed better than others, while some of the categories had no development at all. scoring with the test suite was difficult for many sentences, because of insufficient vocabulary and wrong lexical choices. the system had trouble with punctuation, such as quotation marks. names were often translated with fragments or as mixtures of different fragments, clearly coming from the europarl proceedings. ## conclusions and further work we performed a fine-grained evaluation on several training stages of three different nmt models. the most interesting observation is that although the training process stops when the best scores of the automatic metrics are achieved (early stopping), the accuracy of some linguistic phenomena is dropping, as compared to previous epochs. for this reason, the contribution of the scoring metric and the stopping criterion should be further investigated, while it might be also depend on whether the development sets contain these phenomena. the fact that some linguistic categories have a steeper curve than the others may also signalise the difficulty of these categories from a machine learning perspective. since this is a preliminary study, the amount of items per linguistic category is small and does not allow for statistically significant conclusions. this could be improved in the future with further annotation effort. finally, the systems examined are taken as random samples in terms of settings and parameters. we should repeat the measurements on state of the art systems, allowing fair comparisons among different architectures and design decisions.