The Effects of Language Token Prefixing for Multilingual Machine Translation (2022)

Abstract: Machine translation traditionally refers to translating from a single source language into a single target language. In recent years, the field has moved towards large neural models either translating from or into many languages. The model must be correctly cued to translate into the correct target language. This is typically done by prefixing language tokens onto the source or target sequence. The location and content of the prefix can vary and many use different approaches without much justification towards one approach or another. As a guidance to future researchers and directions for future work, we present a series of experiments that show how the positioning and type of a target language prefix token effects translation performance. We show that source side prefixes improve performance. Further, we find that the best language information to denote via tokens depends on the supported language set.

## introduction machine translation (mt) started as a basic sequence-to-sequence problem. confined to a single input and output language, the model was only responsible for learning the mapping between these two languages. multilingual neural machine translation (mnmt) shifted the paradigm to consider many input and output languages @xcite . language tokens, or tokens that signify the source language and the desired target language, became common prefixes on source and target sequences. in considerations for the placement of token may be convenience-prefixing on the source makes off-the-shelf training pipelines quickly deployable. source-side prefixing obviously affects encodings, and there has been recent interest in making the encodings of a multilingual model language agnostic with evidence to suggest it makes the model more robust in zero-shot settings @xcite . we focus on supervised directions-language pairs seen during training-which has not been thoroughly evaluated to the best of our knowledge. we find differences in conclusions in supervised directions over previous results on zero-shot @xcite . in this work, we show that source side prefixing is preferable to target side prefixing, but the best token-type varies on language set. adding source language information is beneficial for many language pairs-contrary to zero-shot conclusions. we also vary encoder and decoder dpeths to determine if the source-side tokens are successful as result of strong encodings and find similar results in both source and target side prefixing. investigation in these techniques has been limited to studying the effects on zero-shot translation. @xcite considered combinations of these techniques to target zero-shot translation but ultimately found that constraining the decoding by filtering for the target language is more productive. conversely, @xcite has investigated zero-shot translation and found that " t | ∅" outperforms other approaches. n elnokrashy et al. (2022) find that "s t | t" can beat "t | ∅" in zeroshot settings. the preferred prefixing technique may be dependent on use-case and the set of sup-family script iso sentences task 1 indo-european latin en 107m hr 23.7m cyrillic mk 1.4m sr 11.3m uralic latin et 20.4m hu 50.1m task 2 indo-european latin en 18.0m malayo-polynesian latin id 12.7m jv 1.4k ms 3.3m tl 1.1m dravidian tamil ta 879k token prefixing pitfalls can be mitigated by having multiple decoders responsible for a subset of languages. shallow decoders have been shown to be ineffective in mnmt compared to bilingual equivalents but multiple shallow decoders can compensate for these differences @xcite . we use a single unified decoder. ## experimental design language tokens are typically additional vocabulary items where the content designates the source language, the target language or a combination of the two (i.e., <src>, <tgt>, @xmath0, respectively). designating the target language is necessary and many choose to add source information as well as an additional signal to the encoder. these tokens can be prepended onto either the source or target--directly affecting the encodings of either the encoder or decoder. in order to compare across these techniques, we train models with seven prefixing strategies outlined in ## conclusion prefixing strategies are wide and varied. previous work focused on zero-shot settings while our work complements that by investigating supervised performance. source-side prefixing performs better than target-side irrespective of encoder/decoder depth. further, researchers should consider the number of languages in their set as the quantity, diversity, and balance of pairs may make some prefixes more beneficial than others. future work should consider more forceful prompting methodologies and experiment with how prefixes function with respect to language set scaling.