Towards Modeling the Style of Translators in Neural Machine Translation (2021)

Abstract: One key ingredient of neural machine translation is the use of large datasets from different domains and resources (e.g. Europarl, TED talks). These datasets contain documents translated by professional translators using different but consistent translation styles. Despite that, the model is usually trained in a way that neither explicitly captures the variety of translation styles present in the data nor translates new data in different and controllable styles. In this work, we investigate methods to augment the state of the art Transformer model with translator information that is available in part of the training data. We show that our style-augmented translation models are able to capture the style variations of translators and to generate translations with different styles on new data. Indeed, the generated variations differ significantly, up to +4.5 BLEU score difference. Despite that, human evaluation confirms that the translations are of the same quality.

## introduction translators often translate the original content with provided guidelines for styles. 1 however, guidelines are supposed to be high level and not comprehensive. personal stylistic choices are thus welcome as creative part of the translator's job, as long as their translation style consistency is ensured to the task. by contrast, although neural machine translation (nmt) models @xcite are trained from these human translations (e.g. europarl, ted talks), the models do not explicitly learn to capture the rich variety of translators' styles from the data. this limits their capability to creatively translate new data with different and consistent styles as translators do. we believe that modeling the style of translators is an * y. wang carried out this work during an internship with amazon ai. in particular, our work investigates ways to integrate translator information into nmt, with an emphasis on mimicking the translator's style. our study uses the ted talk dataset, with four language pairs with translator annotations. we present and compare a set of different methods of using a discrete translator token to model and control translator-related stylistic variations in translation. note that using a discrete token is a common approach to model and control not only specific traits in translation such as verbosity, politeness and speaker-related variances @xcite ) but also other aspects in nmt such as language ids @xcite . however, our study is the first to use such a discrete token to model the style of translators. it also provides several insights regarding translation style modeling as follows. first, we show that the state-of-the-art transformer model implicitly learns the style of translators only to a limited extent. moreover, methods that add translator information to the decoder surprisingly result in nmt that fully ignores the additional knowledge. this is regardless of whether the token is added to the bottom (i.e. the embedding layer) or to the top (i.e. the softmax layer) of the decoder. meanwhile, methods that add the information to the encoder seem to model the translator's style effectively. second, we show that our best style-augmented nmt method is able to control the generation of translation in a way that mimics the translator's style, e.g. lexical and grammatical preferences, verbosity. while output produced by the styleaugmented nmt can vary significantly with the translator-token values, with bleu score variations up to +4.5, a human evaluation confirms that observed differences are all about style and not translation quality. finally, we show that the translator information has more impact on nmt than the speaker information, which was investigated by @xcite . ## related work style itself is a broad concept @xcite . it includes both simple high-level stylistic aspects of language such as verbosity @xcite @xcite , formality @xcite , politeness @xcite and complex aspects such as demography @xcite @xcite and personal traits @xcite @xcite . our study focuses on capturing the personal style of translators. the closest work to our study is thus the work of @xcite , where they study instead the effects of using the speaker information in nmt. in our results, we show that the translator information has indeed more impact to nmt than the speaker information. finally, another distantly related research line tries to improve the diversity in the top rank translations of an input @xcite @xcite . in fact, adding the translator information to nmt also provides means to generate translations with significantly different stylistic variations. ## nmt with translator information nmt reads an input sequence @xmath0, ..., x n in the source language with an encoder and then produces an output sequence @xmath1, ..., y m in the target language. the generation process is performed in a token-by-token manner and its probability can be factored as m @xmath2, x), where @xmath3. the prediction for each token over the vocabulary v is based on a softmax function as follows: here, o @xmath4.g. 512 or 1024), encoding both the context from the encoder and the state of the decoder at time j. meanwhile, w @xmath5. we adjust nmt in different ways as below to let it mimic and control the translator's style. source token. in our first approach, we insert the translator token t as the beginning of each input sentence. the translator token is thus assigned with an embedding vector like any other source token. hence, the embedding sequence e enc for the mt encoder becomes: token embedding. we also consider adding the embedded translator token e(t ) to every token embedding in the encoder and/or decoder as follows: our motivation is to reinforce the influence of the translator token in mt. output bias. following michel and neubig (2018), we add the translator token information to the output bias at the final layer of the decoder (full-bias variant). specifically, the method directly modulates the word probability over vocabulary v as follows: with parameters @xmath6. note that while the above methods digest the translator token at an earlier stage, this one consumes translator signals in a late fusion manner. ## conclusion we designed various ways of incorporating translator information into nmt, in order to model and control the generation of translation with different translator styles. we show that resulting styleaugmented nmt produces significantly different stylistic variations, mimicking professional translators. human evaluation confirms that the generated variations are all of same translation quality.