Focus on the Target’s Vocabulary: Masked Label Smoothing for Machine Translation (2022)

Abstract: Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that simply applying both techniques can be conflicting and even leads to sub-optimal performance. When allocating smoothed probability, original label smoothing treats the source-side words that would never appear in the target language equally to the real target-side words, which could bias the translation model. To address this issue, we propose Masked Label Smoothing (MLS), a new mechanism that masks the soft label probability of source-side words to zero. Simple yet effective, MLS manages to better integrate label smoothing with vocabulary sharing. Our extensive experiments show that MLS consistently yields improvement over original label smoothing on different datasets, including bilingual and multilingual translation from both translation quality and model’s calibration. Our code is released at https://github.com/PKUnlp-icler/MLS

## introduction the vocabulary of both source and target language into a whole vocabulary, and therefore the vocabulary is shared. it enhances the semantic correlation between the two languages and reduces the number of total parameters of the embedding matrices. however, in this paper, we argue that jointly adopting both label smoothing and vocabulary sharing techniques can be conflicting, and leads to suboptimal performance. specifically, with vocabulary sharing, the shared vocabulary can be divided into three parts as shown in figure 1 . but with label smoothing, the soft label still considers the words at the source side that are impossible to appear at the target side. this would mislead the translation model and exerts a negative effect on the translation performance. as shown in to address the conflict of label smoothing and vocabulary sharing, we first propose a new mechanism named weighted label smoothing (wls) to control the smoothed probability distribution and its parameter-free version masked label smoothing (mls). simple yet effective, mls constrains the soft label not to assign soft probability to the words only belonging to the source side. in this way, we not only keeps the benefits of both label smoothing and vocabulary sharing, but also address the conflict of these two techniques to improve the quality of the translation. according to our experiments, mls leads to a better translation not only in scores like bleu but also reports improvement in model's calibration. compared with original label smoothing with vocabulary sharing, mls outperforms in wmt'14 en-de(+0.47 bleu), wmt'16 en-ro (+0.33 bleu) and other 7 language pairs including de,ro-en multilingual translation task. k denotes the number of classes, α is the label smoothing parameter, α/k is the soft label, ŷ is a vector where the correct label equals to 1 and others equal to zero and ŷls is the modified targets. label smoothing is first introduced to image classification @xcite task. pereyra et al. (2017); edunov et al. (2018) explore label smoothing's application in sequence generation from token level and norouzi et al. (2016) propose sentence level's label smoothing. theoretically, müller et al. (2019); meister et al. (2020) all point out the relation between label smoothing and entropy regularization. gao et al. (2020) explores the best recipe when applying label smoothing to machine translation. to generate more reliable soft labels, lukasik et al. (2020) takes semantically similar n-grams overlap into consideration level label smoothing. wang et al. (2020) proposes graduate label smoothing that generate soft label according to the different confidence scores of model. to the best of our knowledge, we are the first to investigate label smoothing's influence on machine translation from the perspective of languages. category de->en ro->en vi->en source 39% 50% 36% common 20% 8% 11% target 41% 42% 53% vocabulary sharing vocabulary sharing is widely applied in most neural machine translation studies @xcite @xcite . researchers have conducted in-depth studies in vocabulary sharing. @xcite propose shared-private bilingual word embeddings, which give a closer relationship between the source and target embeddings. while @xcite point out that there is an vocabulary mismatch between parent and child languages in shared multilingual word embedding. ## conflict between label smoothing and vocabulary sharing words or subwords in a language pair's joint dictionary can be categorized into three classes: source, common and target using venn diagram according to their belonging to certain language as depicted in figure 1 . this can be achieved by checking whether one token in the joint vocabulary also belongs to the source/target vocabulary. we formalized the categorization algorithm in appendix a. then we compute the tokens' distribution in different translation directions as shown in ## conclusion we reveal the conflict between label smoothing and vocabulary sharing techniques in nmt that jointly adopting the two techniques can lead to suboptimal performance. to address this issue, we introduce masked label smoothing to eliminate the conflict by reallocating the smoothed probabilities according to the languages' differences. simple yet effective, mls shows improvement over original label smoothing from both translation quality and model's calibration on a wide range of tasks. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n. gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in proceedings of the 31st international conference on neural information processing systems, nips'17, page 6000-6010, red hook, ny, usa. curran associates inc. shuo wang, zhaopeng tu, shuming shi, and yang liu. 2020. on the inference calibration of neural machine translation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 3070-3079, online. association for computational linguistics. yingce xia, tianyu he, xu tan, fei tian, di he, and tao qin. 2019. tied transformers: neural machine translation with shared encoder and decoder. in aaai.