Understanding the effects of word-level linguistic annotations in under-resourced neural machine translation (2020)

Abstract: This paper studies the effects of word-level linguistic annotations in under-resourced neural machine translation, for which there is incomplete evidence in the literature. The study covers eight language pairs, different training corpus sizes, two architectures, and three types of annotation: dummy tags (with no linguistic information at all), part-of-speech tags, and morpho-syntactic description tags, which consist of part of speech and morphological features. These linguistic annotations are interleaved in the input or output streams as a single tag placed before each word. In order to measure the performance under each scenario, we use automatic evaluation metrics and perform automatic error classification. Our experiments show that, in general, source-language annotations are helpful and morpho-syntactic descriptions outperform part of speech for some language pairs. On the contrary, when words are annotated in the target language, part-of-speech tags systematically outperform morpho-syntactic description tags in terms of automatic evaluation metrics, even though the use of morpho-syntactic description tags improves the grammaticality of the output. We provide a detailed analysis of the reasons behind this result.

## introduction training neural machine translation (nmt) systems for under-resourced language pairs, for which the amount of parallel corpora is orders of magnitude smaller than those available for prevailing language pairs, may be challenging. recently, @xcite have shown that even under these circumstances, nmt surpasses classical approaches such as phrase-based statistical mt @xcite . in these under-resourced scenarios, the use of relevant linguistic word-level annotations has proved to improve translation performance @xcite . linguistic annotations can be used to label source-language (sl) or target-language (tl) words. the former lead to more accurate representations of the sl sentence @xcite and usually require changes in the encoder but not in the training loss @xcite . the latter involve producing probability distributions for both tl words and tl linguistic annotations, which can be seen as a form of multi-task learning. multi-task learning architectures explored in the literature include: independent decoders for words and linguistic annotations @xcite @xcite @xcite ; independent output layers in the same decoder @xcite @xcite ; and even sharing the same network for both tasks @xcite @xcite and alternatively produce linguistic annotations and words. the latter approach is usually referred to as interleaving. we can also classify the approaches according to the type of linguistic annotations used: part-of-speech tags @xcite ; morpho-syntactic description tags, which comprise part of speech and morphological inflection information @xcite ; and syntactic structure information @xcite @xcite @xcite . using morpho-syntactic description tags as tl annotations allows us to train the network to produce tl lemmas instead of surface forms (words as they appear in running texts). this strategy can reduce data sparseness but requires the use of an external morphological generator as a post-processing step @xcite . despite the body of work published about this topic, no strategy has clearly emerged as the most appropriate method for integrating linguistic annotations into nmt. the literature mainly contains incomplete evidence. for instance, @xcite conclude that tl part-of-speech annotations boost translation quality with an ad-hoc architecture, but @xcite claims that tl morpho-syntactic description tags degrade translation quality when they are interleaved: it is not clear whether the difference between both results is caused by the type of linguistic annotations or by the approach followed to integrate them. there are also contradictory results, such as those reported by @xcite , who claim that tl annotations are only useful when they are combined with lemmatisation, and @xcite , who report positive results without lemmatisation. in addition, the influence of factors such as the size of the available training parallel corpus and the language typology have not been properly evaluated. in this paper, we aim at clarifying how linguistic annotations help nmt by carrying out systematic experiments with eight language pairs. we focus on an under-resourced scenario where linguistic annotations are likely to provide information that cannot be inferred from scarce training data. we analyse multiple factors, namely, language typology, side which is annotated with linguistic information (sl, tl or both), architecture of the nmt system, training corpus size, and type of information encoded in the tags. for the latter factor, we focus only on part-of-speech tags and morpho-syntactic description tags, since other type of annotations, such as ccg supertags @xcite , are unlikely to be available for under-resourced languages. we train systems on linguistically annotated surface forms via interleaving, which does not require modifications to the neural network and allows us to easily compare different types of linguistic annotations and nmt architectures. in addition, an automatic error classification allows us to qualitatively compare them. a qualitative analysis has only been previously performed by @xcite , but it covered only two language pairs with english as tl, and a single type of annotation. the rest of the paper is organised as follows. next section overviews the process of interleaving linguistic annotations in the training data. section 3 then describes the experimental settings whereas section 4 reports and discusses the results obtained. section 5 presents automatic error classification results for all the systems evaluated, while section 6 studies the reasons behind the poor performance of systems with tl morpho-syntactic description tags. the paper ends with some concluding remarks. ## interleaving in neural machine translation the interleaving approach for integrating linguistic annotations into nmt @xcite annotates each word with a single tag which is interleaved in the sentence before the word, i.e. introduced in the sentence as if it were another word. in our experiments, tags can represent either the part of speech (pos) of the word or its morpho-syntactic description (msd). as corpora are pre-processed with bpe @xcite , the tag is introduced just once, before the first sub-word unit. to study if the effect of using tags is related to the fact that input and output sequences get longer and word boundaries are explicitly defined, and not to the information provided by tags, we also tried with a dummy tag (dum) conveying no linguistic information at all, and used the same dummy tag for every word @xcite . interleaved tl tags are removed from the final translation generated by the system before computing the automatic evaluation metrics. the example below shows the result of interleaving msd tags in the english sentence it has happened before. the sentence contains a pronoun (pron) followed by an auxiliary verb (aux), a main verb (verb), an adverb (adv) and a punctuation mark (punct). the analysis of the pronoun tells us that it is personal, nominative, neuter, singular, and 3rd-person. the symbol @@ acts as a sub-word unit separator. ## experimental settings we conducted experiments for the translation of english text into four languages, and vice-versa. these languages -czech (cs), german (de), spanish (es) and turkish (tr)-belong to different language families and differ at the syntactic and morphological levels. german, czech and spanish are indo-european languages: they are, respectively, germanic, slavic and romance. both german and czech have declension and svo sentence structure, except for the subordinate sentences in german, which are sov. spanish has no declension and its sentence structure is svo. turkish is an agglutinative turkic language with declension and sov sentence structure. the morphological differences between these languages are reflected in the sparsity of the msd tags: the number of unique tags in the interleaved training corpora ranges from a few hundreds for english and spanish to a few thousands for czech, german and turkish. 1 we simulated an under-resourced scenario by downsampling available parallel corpora for the selected language pairs. downsampling has important advantages over using truly under-resourced language pairs: (i) we can choose languages from different families and evaluate them on standard, high-quality test sets; (ii) we can confirm whether conclusions hold for richer-resource scenarios by training on larger datasets for the same language pairs; and (iii) linguistic annotations can be obtained with the same state-of-the-art morphological analyser, minimising the potential distortions introduced by differences in the morphological analyser technology and in performance between the languages. the pos and msd tags were obtained by means of the standfordnlp tagger @xcite . in any case, the approaches described in this paper could be applied to truly under-resourced language pairs as transfer learning allows to obtain morphological analysers even from scarce morphologically annotated data @xcite . corpora. the training, development and test sets used belong to the news domain. for training, we used texts from the news commentary v14 corpus, 2 except for turkish, for which we used texts from the setimes corpus @xcite . for development and testing we used evaluation sets from the wmt 2019 conference on machine translation, each of which contains around 3,000 parallel sentences. 3 to see if the conclusions drawn on the under-resourced settings hold in a richer-resourced scenario, we trained english-german systems (in both directions) on the concatenation of the parallel data made available for the wmt 2017 shared task on news translation 4 plus the synthetic parallel data obtained through back-translation released by @xcite . all corpora were tokenised and truecased with the moses scripts 5 and parallel sentences longer than 100 words in either side were discarded. translation models. we tested the performance of the recurrent-neural-network encoder-decoder with attention (hereafter, recurrent; @xcite and the transformer @xcite architectures when the different types of tags introduced in section 2 are interleaved in the sl input sequence, in the tl output sequence, and in both of them. for each architecture, we also trained a baseline using no tags at all. to keep the experiments to a manageable size, the systems that included tags in both languages were not trained on the nine possible combinations of tag types (three in the sl and three in the tl). instead, they were trained only on sl msd and tl pos tags, which were those with the best general performance when used in isolation in the sl and in the tl, respectively. for the same reason, we only explored the sl msd/tl pos tag combination for systems trained on large-scale wmt data. in order to determine the appropriate values for training hyper-parameters, a grid search over the number of bpe operations and the neural network sizes was carried out. the optimum hyper-parameter values for each language pair, training corpus size and architecture were obtained after training the baseline systems. these hyper-parameters were also used with the systems integrating linguistic annotations. appendix a provides a detailed description of the training process. error classification. we followed the automatic error analysis strategy by @xcite , who used the tool hjerson @xcite to classify word errors into five categories: inflection, reordering, missing words, extra words and incorrect lexical choices. as it is difficult to automatically distinguish between the latter three categories @xcite , we grouped them into a unique category named lexical errors. hjerson works on the surface form and lemma of the words in the reference translations and mt outputs. the lemmas were obtained again with the standfordnlp tagger. ## results and discussion as the four languages paired with english are morphologically richer than english, we split the analysis of the results we describe next into two groups: translation into a tl morphologically richer than the sl (pairs with english as sl), and translation from a morphologically richer sl (pairs with english as tl). it is also worth mentioning that, in all the scenarios evaluated, when a system was trained with interleaved tl tags, the decoder alternately produced tl tags and surface forms at test time as expected. translation into a morphologically rich language. when the tl is morphologically richer than the sl, interleaved tags lead to higher bleu scores, although the impact changes depending on the information encoded in the tag and the language where they are used (sl or tl). sl dum tags are not very effective: they bring a statistically significant increase in bleu only to 2 out of the 8 systems evaluated with the recurrent architecture, 7 and to none of the 8 transformer systems. sl pos and msd tags generally outperform dum tags, as they contain information that helps to obtain a better representation of the sl sentence and break the grammatical ambiguity of english @xcite . there is a statistically significant difference between sl pos and sl msd tags in 6 out of the 16 systems evaluated, 8 and in 5 out of these 6 systems msd tags outperform pos tags. for some language 6 those systems trained with both sl msd and tl pos tags could not be compared with systems with both sl dum and tl dum tags because the latter were not included in the experimental set-up in order to keep the experiments to a manageable size. hence, their scores do not contain any bullet. 7 four language pairs and two training corpus sizes. 8 four language pairs, two training corpus sizes and two architectures. translation from a morphologically rich language. the effects of using sl tags when the sl is morphologically richer than the tl are similar to those observed in the opposite scenario: pos and msd tags often outperform dum tags. when statistically significant differences between pos and msd tags are found, they favour msd tags. concerning tl tags, the systematic degradation observed for msd tags is less frequent than in the opposite direction, and it is mainly concentrated in the smallest corpus size. an explanation could be that morphological information in english is less complex and easier to predict from the sl sentence. finally, combining sl msd tags and tl pos tags also leads to the highest scores. large-scale training data. the results for the english-german wmt large-scale training data, also depicted in main findings. in line with previous works @xcite , the results analysed so far suggest that interleaved linguistic annotations are helpful both in the sl and the tl and they should be included in both languages in order to maximize performance. while morphological features can be useful in the sl, they should be avoided in the tl if it is morphologically rich. even when large corpora are available, linguistic annotations can help to boost translation quality. ## error analysis to better understand the results obtained, we computed the relative difference in the number of hjerson errors between the systems with interleaved tags and the baseline; 10 a positive value means that the system made more errors than the baseline. as we did before, we split the results into two groups of language pairs: those with english as sl, depicted in figure 1 , and those with english as tl, depicted in figure 2 . in the remainder of this section, we analyse the results obtained and illustrate them with examples. sl tags. sl pos and sl msd tags systematically reduce lexical errors (green, empty squares and triangles are below the horizontal line). reordering errors are also reduced with the exception of english-czech, in which the tl has a relatively flexible word order. 11 concerning inflection errors, there is not 9 in order to confirm that the differences observed were not caused by the random initialisation of the network weights, we trained the systems on wmt data two more times. the bleu scores obtained for the additional training runs confirm that linguistic annotations only help in recurrent systems. we did not perform additional training runs for the under-resourced scenario because of time constraints and because the conclusions drawn for that scenario are based on the results obtained for different language pairs (and hence training runs), which we think makes them solid enough. 10 computed as (#errors interleaved -#errors baseline)/#errors baseline. 11 improvements in reordering quality are not captured by the automatic error classification if translation hypotheses with an acceptable word order that differs from that in the reference are produced by the system. figure 2 : for language pairs with english as tl, relative changes in the number of errors for each error category, training corpus size and type of interleaved tag. clear trend. as aforementioned, a possible explanation could be that sl tags help to obtain more accurate representations of the sl sentences; since inflection errors are related to modeling tl grammar rather than to representing the sl sentence, they are not reduced by interleaving sl tags. all these results are compatible with the evaluation metrics, which showed that sl tags generally improve translation quality. in the first example in tl tags. different error distributions can be observed depending on the information encoded in the tl tags. tl msd tags systematically reduce inflection errors in both architectures (the blue, filled triangle is usually among the lowest points in the figure). the largest inflection error reductions occur with highly inflected tls such as czech and turkish. tl pos tags, on the contrary, do not systematically reduce inflection errors. hence, the system using tl msd tags is using the morphological features they encode (tense, number, etc.) for producing the correct inflected form according to the reference. in the second example in however, the prediction of msd tags with complex morphology (see figure 1 ) also leads to an increase in lexical errors in comparison with the prediction of pos tags. it can be observed that tl msd tags bring an increase in lexical errors over the baseline (note the green, filled triangles at the top of the figures), while the impact in lexical errors of the pos tags is less clear. similarly to inflection errors, the difference between the increases of lexical errors brought by msd and pos tags is larger for turkish and czech, which are the two languages with the most sparse msd tags. turkish is a turkic agglutinative language and czech is a slavic fusional language with seven cases and four genders. this is compatible with the automatic evaluation metrics: although using tl msd tags leads to a more grammatical output, the increase in lexical errors makes the system produce translations that are overall less similar to the reference. note that lexical errors are the most frequent ones. 12 the third example in when sl msd tags and tl pos tags are both interleaved, there is a general reduction in the three error categories as compared with the systems using only tags in one of the languages. this confirms that the advantages of sl and tl tags are complementary. ## analysing the effect of target language morphology we compare the output of the systems interleaving tl pos and tl msd tags in order to ascertain whether the increase in lexical errors is caused by the difficulty of predicting the more complex and sparse msd tags, or by the conditioning of the prediction of surface forms on tl msd tags. in the latter case, there is the risk that the system learns to strongly condition on tags and avoids generating new words @xcite . this problem could be exacerbated by the sparsity of tl msd tags because some of them may co-occur only with a few surface forms in the training corpus. we tried to answer this question by independently evaluating the prediction of tags and surface forms, and comparing the systems interleaving pos and msd tags. the prediction of surface forms was evaluated by re-decoding the test set and forcing the system to choose the tags from the reference during beam search, whereas the prediction of tags was evaluated by forcing the surface forms from the reference. if the system really learned to strongly condition on tags, when a tag was observed together with only a few surface forms in the training corpus, low-frequency words would not be generated when translating the test set. to test this hypothesis we studied the surface form prediction accuracy for two subsets: infrequent words (@xmath1. figure 3 shows the results for those language pairs with english as sl. it can be observed that there is a trade-off between tag and surface form prediction accuracy: msd tags are more difficult to predict, but conditioning on them leads to better surface form prediction. on low-frequency and oov words, msd tags still outperform pos tags in terms of surface form prediction accuracy, although the difference between them is smaller. for a fair comparison of both types of tags, we computed the part-of-speech prediction accuracy when predicting msd tags. the results are depicted in the rows labelled as pos in figure 4 . for highly inflected tls, those systems that predict msd tags have consistently lower pos prediction accuracy than those that predict only pos tags. the difference is larger for recurrent systems. these results suggest that the difficulty of predicting together the part of speech and its morphological features is indeed one of the reasons behind the lexical degradation of systems using msd tags. sparseness of msd tags seems to play an important role in this degradation: highly inflected tls present the largest degradation. to evaluate the impact of predicting morphological features regardless of the low part-of-speech accuracy of msd tags, we re-computed surface form prediction accuracy by letting the beam search algorithm choose among those msd tags with the part of speech in the reference. the results, depicted in the rows labelled as s.f. in figure 4 , show that, if the systems with interleaved msd tags correctly predicted the part of speech, the surface form predictions would not be worse than those of systems with interleaved pos tags, neither in general nor for low-frequency and out-of-vocabulary words. hence, errors in surface form prediction arising from strongly conditioning on sparse msd tags do not seem to be the main cause behind the degradation of translation quality brought by tl msd tags. actually, when tags with the part of speech of the reference are chosen, conditioning on msd tags outperforms conditioning on pos tags in terms of overall surface form prediction accuracy for language pairs with english as tl. for the other language pairs, the gain introduced by msd tags is less clear. one possible reason could be that bpe segmentation does not allow the system to learn a general mapping between tags and word endings from the training data. another explanation could be related to the fact that predicting the morphological gender for german, czech and spanish forces the tag prediction task to be aware of tl lexical information, preventing an optimum division of labour between tag and surface form predictions. in conclusion, the prediction of tl morphological information needs to be factorised differently in order not to harm part-of-speech prediction. for instance, in the morphological analysis field, @xcite and @xcite predict part of speech and each morphological attribute independently. ## concluding remarks in this paper, we have studied the effects of using linguistic annotations of sl and tl words in underresourced nmt by interleaving linguistic tags for different language pairs, architectures, training data sizes and types of linguistic information (part of speech and morpho-syntactic descriptions). we have shown that both sl and tl linguistic annotations are useful, in line with previous works in the literature @xcite . sl linguistic annotations lead to more accurate sl sentence representations, and for some language pairs, the use of morpho-syntactic descriptions (consisting of part of speech and morphological features) improves the representation obtained when only part-of-speech tags are used. surprisingly, for highly inflected tls, tl linguistic annotations are more useful if they simply consist of part-of-speech information. using morpho-syntactic descriptions leads to an overall translation quality degradation in terms of automatic evaluation metrics, even though it improves the grammaticality of the output. we have also shown that predicting tl morpho-syntactic descriptions frequently results in wrong part-of-speech predictions. hence, to optimize the use of tl morphological information in nmt, it is advisable to avoid the prediction of part-of-speech and morphological features together as monolithic tags. the gain introduced by linguistic information encoded as interleaved tags scales to large data availability scenarios only for the recurrent architecture. this result, together with the conclusions of the automatic error analysis, suggest that adding extra tokens to the tl stream is not the optimum way of introducing additional linguistic information in self-attention-based nmt systems. in summary, the use of morpho-syntactic descriptions in the sl and part-of-speech information in the tl, which can be easily obtained even in under-resourced scenarios, systematically improves translation quality when they are simply interleaved in training data as linguistic tags, even without using a morphological generator @xcite , which could be error-prone for under-resourced languages, and without any kind of information about syntactic structures.