Towards Formality-Aware Neural Machine Translation by Leveraging Context Information (2023)

Abstract: Formality is one of the most important linguistic properties to determine the naturalness of translation. Although a target-side context contains formality-related tokens, the sparsity within the context makes it difficult for context-aware neural machine translation (NMT) models to properly discern them. In this paper, we introduce a novel training method to explicitly inform the NMT model by pinpointing key informative tokens using a formality classifier. Given a target context, the formality classifier guides the model to concentrate on the formality-related tokens within the context. Additionally, we modify the standard cross-entropy loss, especially toward the formality-related tokens obtained from the classifier. Experimental results show that our approaches not only improve overall translation quality but also reflect the appropriate formality from the target context.

## introduction translation quality is determined not only by preserving semantics across languages but also by conforming to an appropriate syntax. formality is one of the important syntactic properties, including honorifics. for example, as shown in ## related work as nmt models have reached a considerable level, handling of discourse phenomena @xcite @xcite has become crucial to improve translation quality. formality is an important linguistic feature addressed by context-aware nmt models for coherent and fluent discourse. since a few tokens within a sentence reveal the formality, context-aware nmt models have difficulty to training and evaluating consistent stylistic translation. first, due to this sparsity @xcite that also occurs within the context, a training alternative is needed to amplify the weak training signal. in addition, bleu @xcite , a standard translation quality metric, often fails to capture improvements, which bring subtle differences. to better perceive these differences in translation, human evaluation is sometimes performed in parallel @xcite @xcite . our approach directs the model to reach the heart of minor but decisive formality tokens. as a result, our experiments include the performance in terms of bleu and human evaluation. ## methods given a source document @xmath0, x 2 , ..., x n } as a sequence of n sentences, context-aware nmt models translate a target document @xmath1: instance-level context attention regularization mechanism. we obtain a continuous distribution of the target context by making perturbations (ỹ k-1 ) to the target context y k-1 . we then regularize two attention heads of the top decoder layer. however, to generate consistent translation reflecting the sophisticated formality of the target context y k-1 , training and inference strategies are modified. when training with teacher forcing, the cross-entropy loss is not applied to the target context. during the model inference, the target context is not generated but provided as a condition to avoid exposure bias. ## analysis attention visualization figure 2 illustrates how attention weights from regularized and nonregularized heads are distributed when generating formality-related tokens, which is marked in bold. regularized heads (ours) in each example are visualized with the averaged attention weights over two regularized heads in attnreg. non-regularized heads (concat) and non-regularized heads (ours) are highlighted using the averaged attention weights over all non-regular heads in concat-level and at-tnreg, respectively. we observe that regularized heads pay proper attention to formality-related tokens within the target context, such as endings ("예 요") or nominative markers ("가"); on the other hand, non-regularized heads seem to focus mostly on the previous few tokens. this experiment shows that the model is able to learn various end expressions via the classifier, and the classifier can control the formality beyond rule-based learning. alignment between regularized heads and human perception attention regularization makes our model explicitly refer to key syntactic tokens that determine the formality of contextual sentences. we investigate how the attention weights of regularized heads are aligned with human perception. the attention weights of regularized heads in our model are shown to be adequately aligned with human perception showing the highest similar-ity score of 0.669 foot_1 , while non-regularized heads hardly align with human perception: 0.033 (nonregularized heads for our model) and 0.029 (nonregularized heads for concat-level baseline). ## conclusion to generate consistent translations reflecting the formality of the target-side context, we propose a new training approach to focus on formalityinvolved tokens by leveraging the formality classifier. our approach results in convincing improvements in translation quality as measured by bleu and human evaluation. as a result, we suggest a lesson that the formality classifier is easily applicable to nmt model as a guider.