Distilling E stonian Text Domains for Production-Oriented Machine Translation (2023)

Abstract: This paper explores knowledge distillation for multi-domain neural machine translation (NMT). We focus on the Estonian-English translation direction and experiment with distilling the knowledge of multiple domain-specific teacher models into a single student model that is tiny and efficient. Our experiments use a large parallel dataset of 18 million sentence pairs, consisting of 10 corpora, divided into 6 domain groups based on source similarity, and incorporate forward-translated monolingual data. Results show that tiny student models can cope with multiple domains even in case of large corpora, with different approaches benefiting frequent and low-resource domains.

## introduction the quality of neural machine translation (nmt, @xcite systems heavily depends on training data and the text domains covered in it. large-scale nmt transformer models are usually trained on multiple corpora representing different domains @xcite , which in turn requires training models with higher representation capacity and an exceedingly large number of parameters, sometimes in the tens of billions for the largest models @xcite . however, using such models for inference in a production setting becomes more costly and cumbersome with increasing size. in parallel to the challenge of using more representational and learning power, a constraint from the practical side is to have models be as small and fast as possible for efficient deployment in production. an additional challenge arises from the variability of natural language and different text domains and styles. while methods for training an nmt model to perform well on a particular type of text are relatively straightforward, the requirement of having a single nmt model translate multiple varieties of input text without a significant loss of quality on any of them due to interference between the domains in the training data remains more difficult. in this paper, we aim to bridge the gap between previous research and systems applicable in production by experimenting with multi-domain knowledge distillation for nmt on the example of estonian-english translation. we show that even for tiny nmt student-models and large-scale training data, it is efficient to train a single student model on data distilled by multiple fine-tuned domain-specific teacher models. our contributions are: in this way, the student model is trained with the goal of imitating the teacher model's probability distribution over the translations, thus constraining the task from the full space of natural language translations to the much smaller space of translations generated by the teacher, and making it more easily achievable for the small student model. we follow the sequence-level knowledge distillation procedure proposed by @xcite in our knowledge distillation experiments. recent advances in efficient mt have extended the limits of small and fast nmt models @xcite @xcite , using knowledge distillation, increasingly lightweight architectures and cpu optimization for faster inference, while suffering increasingly small quality decrease compared to full-scale models. however, these experiments are typically focused on single-domain or generaldomain translation; we build on the findings of mt efficiency research and use them in a multidomain scenario. ## methods in our experiments, we aim to create neural machine translation models which 1) perform well on several data domains, and 2) are small and efficient. to distill nmt models, we follow the sequence-level knowledge distillation framework initially proposed by @xcite , where a smaller student model is trained using the synthetic target-side data produced by a larger teacher model, and experiment with distilling multiple domain-specific teacher models into a single student model. we follow @xcite in employing a straightforward strategy: however, our student models are much smaller than the student models used by @xcite , and we use significantly more data, bringing our setup closer to a full-scale real-world scenario. ## results it seems that fine-tuning the teacher on forwardtranslated monolingual data yields no positive effect. bleu score on the wmt18 test set remains the same as for the general teacher, while scores on the held-out test set drop. this is not unexpected, as, while the teacher stops encountering data from other domains during fine-tuning, it also only encounters the data from the news domain that it forward-translated itself, and most likely cannot learn to exhibit any new behaviour on these data. while the behaviour of fine-tuned teachers is rather straightforward, the performance of student models is more varied. comparing s0 and s-ft, we observe relatively similar performance: the difference on various test sets ranges from none (opensubtitles) to 1.8 (infopankki) bleu points. the bleu score averaged over all test sets is better for s-ft, but not by a very large margin. on the external wmt18 test set, s-ft performs best, although it only outperforms s0 by 0.2 bleu points. on 6 test sets out of 11, s-ft is better than s0, although only on 3 of those the difference is statistically significant @xcite , and on one more test set (opensubtitles) their result is the same. the extremely small gnome and ubuntu corpora obviously benefit from balancing the data, and the scores on their test sets improve significantly compared to the unbalanced s-ft. performance on the emea corpus, which comprises the second smallest data group, also noticeably benefits from upsampling. at the same time, if we compare the results obtained by s-ft and s-ftbal on other corpora, we notice drops of 0.1-7.0 bleu points. the best average bleu score is achieved by s-ft-bal, the student trained on data which is forward-translated by the fine-tuned teacher models and balanced. ## qualitative analysis in example 1, which comes from the ubuntu corpus, only the model trained on balanced data manages to translate "ruutu soldat" as "jack of diamonds", while both s0 and s-ft translate "ruutu" literally ("squares"), and s0 translates "soldat" incorrectly ("solder" instead of the direct translation "soldier", which is likely due to subword interaction). in example 2, the s-ft-bal model shows signs of overfitting: the content part of the sentence is identical to the reference, and the number "63" is generated at the start of the text, where the reference sentence has "53". however, there is no num-ber in the source sentence. sentences produced by the s0 and s-ft models are, in fact, more exact translations of the estonian source sentence ("you have the feeling"/"you feel" vs. "you think" and "the effect of vivanza"/"vivanza's effect" vs. "vivanza"). in the infopankki example (3), all models manage to convey the original meaning of the source sentence, but s-ft-bal does so in a more informal style and with simpler grammar that the reference and the translations by s0 and s-ft (e.g. "work and business office" vs. "employment and economic development office", and "helps" vs. "help is available"/"you can get help"). in the example from the opensubtitles corpus (4), all models use "his" instead of "her" (the estonian pronoun "ta" does not have grammatical gender, so the correct english pronoun can only be inferred from wider context). the s-ft model uses the more informal contraction "it's", which is appropriate for the domain. the s-ft-bal model fails to translate a part of the compound word "kõnepost" and generates "voice" instead of the correct "voicemail". in example 5 (europarl), the s-ft-bal model is the only one not to use contractions ("we have" vs. "we've"), which do not typically occur in the formal style of parliament proceedings. however, all three model hypotheses are faithful. the dgt example (6) sees the s-ft model translate very similarly to the reference, while both s0 and s-ft-bal overgenerate repetitively ("and ovens and ovens" and "non-electric nonelectric non-electric"). finally, in the wmt18 example, all models fail to use the specific correct word "minesweeper", and instead translate the compound word "miinijahtija" more literally as "mine hunter". otherwise, the s-ft hypothesis is the only one to convey the full meaning of the source correctly. ## discussion we observe that the fine-tuned teacher models predictably suffer from forgetting the general teacher's knowledge on domains other than the one the particular teacher is fine-tuned to. the extent of this forgetting varies, e.g. the teacher finetuned to web-crawled text performs 2.7 bleu points worse than the mixed-domain teacher on the europarl test set, while the teacher fine-tuned to medical documents is 27.5 points worse on the this half-year, the estonian minesweeper sakala is also part of the international unit. 7 s0 this half-year is also part of the international unit sakala, a mine hunter of the estonian navy. 34.3 s-ft this half-year the international unit also includes the estonian naval mine hunter sakala. 73.7 s- @xcite in this half, the estonian navy mine hunter also includes the estonian navy mine hunter. 63.7 opensubtitles test set than the original teacher, its score dropping to 4.4 bleu, which suggests that these translations are not too far from random. the different levels of forgetting could potentially serve as a clue to domain similarity and guide the choice of manual data groupings. there is a sizeable gap between the average performance of teachers fine-tuned to each data group and the best student model average. while the difference in capacity becomes even more drastic when we compare not one, but several large finetuned models to a single small student model, this gap remains large, and suggests the possibility of pushing the limits of student models' performance further. distilling the data clearly benefits the training of small models, the s-orig model lagging behind other student models. the best average bleu is achieved by the student model trained on data distilled by multiple fine-tuned teachers and balanced between groups. however, in a real-world scenario trade-offs may still need to be made between the performance on specific domains, with average bleu score not being representative enough for fine-grained evaluation. ## future work in our experiments, we used 10 corpora and, for fine-tuning, split them manually into 6 data groups based on the assumed similarity of their sources and topics. however, as demonstrated by @xcite , the known domain labels may be suboptimal, and assigning the domains automatically can improve the multi-domain mt performance. generating automatic domain labels using the general-domain model's internal data representations has been shown to further improve indomain translation quality @xcite . in future work, we would like to explore these methods for automatic domain discovery in conjunction with multi-domain knowledge distillation. aiming to bring our experiments closer to a production scenario, we tried incorporating forwardtranslated monolingual data into our multi-domain distillation setup. however, large-scale systems typically use back-translation and round-trip translation to increase the amount of training data. it currently remains unclear how to best incorporate monolingual data into the multi-domain knowledge distillation framework effectively, given the suboptimal results we achieved when fine-tuning a mixed-domain teacher model to a forwardtranslated news corpus. we hypothesize that the teacher model cannot learn to exhibit any new behaviours when it is fine-tuned on data generated by itself. thus, adding monolingual domains to distilled multi-domain systems is a potential topic for future exploration. another important direction for future work is extending our research to other language pairs and translation directions. while we perform our experiments on the @xmath0, which, to the best of our knowledge, has not been experimented with in a similar setting before, using other languages, especially lowresource ones, might lead to different results and insights. ## conclusion in this work, we explored distilling multiple domain-specific neural machine translation teacher models into a single student model. while following procedures proposed in previous work, we incorporated research findings on model efficiency and focused on obtaining very lightweight student models. we used a training corpus of 18m estonian-english sentence pairs, comprised of 10 unbalanced domains. we separated the domains into groups based on their perceived similarity, explored the effects of balancing, and incorporated monolingual forward-translated data into training of multi-domain students. our experiments show that the knowledge of several fine-tuned teachers models can be distilled into a very small student model, with balanced representation of domains further improving the average result. the massive total capacity of several fine-tuned teacher models has a huge average gain over the untuned teacher (almost 10 bleu points) and the student models with their limited capacity achieve a much more modest increase in translation quality. still, the increase in translation quality compared to the baseline student is stable and noticeable (+1.5 bleu points).