Contrastive Token Learning with Similarity Decay for Repetition Suppression in Machine Translation (2024)

Abstract: For crosslingual conversation and trade, Neural Machine Translation (NMT) is pivotal yet faces persistent challenges with monotony and repetition in generated content. Traditional solutions that rely on penalizing text redundancy or token reoccurrence have shown limited efficacy, particularly for lengthy article and e-commerce descriptions with inherent redundancy, even with the advent of Large Language Models (LLMs). This paper investigates the underlying causes of textual repetition through the lens of information entropy, attributing the phenomenon to the elevated uncertainty within the input text. To address this, a novel algorithm named Contrastive Token Learning with Similarity Decay (CTSD) is introduced, which modulates the suppression of tokens dynamically, informed by varying attention weights and inter-token distances. Furthermore, an e-commerce dataset comprised of title texts of online real items is compiled and released susceptible to hallucination translations to benchmark the algorithm. Extensive evaluations demonstrate that CTSD significantly outperforms existing approaches in precision and generalizability. Additional online A/B testing underscores its practical value, showing marked improvements in user engagement and conversion. Notably, this method has been implemented with full traffic on eight multilingual sites of alibaba.com, the largest B2B e-commerce platform in the world.

## introduction in recent years, the synergy of neural networks coupled with the increasing scale of parallel corpora has significantly propelled neural machine translation (nmt) forward @xcite . notably, the sophisticated reasoning abilities and specialized knowledge acquired by large language models (llms) @xcite ., * equal contribution. † corresponding author. previous methods mainly employed two strategies to suppress repetition generation. the first is the direct strategy interventions during the inference stage, such as n-gram not repeat, contrastive search (cs) @xcite , and penalized sampling (ps) @xcite . these techniques focus on preventing repeated tokens to eliminate oscillations. however, they would disrupt the token distribution of output, leading to other errors. consequently, recent methods focus on designing training objectives during the model training stage to better address hallucination problems @xcite @xcite . yet, these training objectives do not adequately explore the root of oscillations in transformer-based models, often using a direct intervention way to suppress tokens that have appeared. although they can effectively suppress word or sentence repetition, they also lead to a lower coherence and accuracy in translation @xcite . in this paper, we conduct an in-depth exploration of the fundamental reasons underlying textual repetition in machine translation, primarily utilizing the concept of information entropy. our research reveals that this phenomenon largely stems from increased levels of uncertainty present within the input text. repetitive token generation occurs because the information from previously generated tokens does not provide additional value (information entropy). to effectively deal with this issue, we propose an innovative algorithm called "contrastive token learning with similarity decay" (ctsd). this innovative approach aims to dynamically adjust the suppression of tokens by analyzing the attention differences between different output tokens' embeddings and the distance between the inner tokens, thereby enhancing the accuracy and stability of the output. meanwhile, our method can be applied to both specialized translation models and llm without additional data preparation. the results show that our method can effectively improve the performance of models in translation tasks and prevent oscillation hallucinations. compared with contrastive token learning (ct), ctsd achieves improvements by 1% to 10% in translation quality on both the flores-200 and our proprietary e-commerce datasets. in addition, a comprehensive evaluation by experts reveals that ctsd exceeds current methods in both precision and generalizability. online a/b tests further highlight its practicality, as evidenced by substantial gains in user engagement with higher click-through and conversion rates and final gross merchandise volume. importantly, this method has been successfully deployed across eight multilingual websites with full traffic of alibaba.com, the world's largest b2b e-commerce platform. ## conclusion in conclusion, this study addresses the critical challenge of repetition generation in nmt. by analyzing and visualizing the underlying causes from the lens of information entropy, we propose one novel method, which can dynamically modulate token suppression to reduce the redundancy of some generated words. extensive experiments on offline general and e-commerce datasets and rigorous online a/b tests have verified its performance in improving translation quality and handling oscillation hallucinations.