Entropy– and Distance-Regularized Attention Improves Low-Resource Neural Machine Translation (2024)

Abstract: Transformer-based models in Neural Machine Translation (NMT) rely heavily on multi-head attention for capturing dependencies within and across source and target sequences. In Transformers, attention mechanisms dynamically determine which parts of the sentence to focus on in the encoder and decoder through self-attention and cross-attention. Our experiments show that high-resource NMT systems often exhibit a specific peaked attention distribution, indicating a focus on key elements. However, in low-resource NMT, attention tends to be dispersed throughout the sentence, lacking the focus demonstrated by high-resource models. To tackle this issue, we present EaDRA (Entropy– and Distance-Regularized Attention), which introduces an inductive bias to prioritize essential elements and guide the attention mechanism accordingly. Extensive experiments using EaDRA on diverse low-resource language pairs demonstrate significant improvements in translation quality, while incurring negligible computational cost.

## introduction neural networks have revolutionized machine translation (mt), as evidenced by the significant progress made in recent years @xcite . the transformer architecture @xcite has garnered substantial attention and achieved remarkable advancements across various downstream tasks @xcite @xcite , including its application to neural machine translation (nmt). however, the performance of the transformer architecture heavily relies on the effectiveness and reliability of its attention mechanism. our observations from well-performing models suggest that attention should prioritize important elements, resulting in a peaked distribution of attention weights. by emphasizing crucial information, the attention mechanism enables more accurate predictions. this selective attention allows the model to effectively capture and utilize relevant information, leading to improved performance. there-fore, optimizing the attention mechanism is critical for harnessing the full potential of the transformer architecture and enhancing its performance across tasks, including nmt. however, achieving focused attention behavior poses a significant challenge for nmt systems @xcite , especially in low-resource settings. our preliminary experiments show that as the amount of available training data decreases, nmt systems tend to exhibit a lack of the desired focused attention behavior. in such lowresource scenarios, where training data is limited, the attention distribution becomes more dispersed and less selective. consequently, the model's ability to effectively capture and utilize crucial information is hindered, leading to reduced translation performance. therefore, as the amount of available data diminishes, it becomes crucial to develop techniques that can guide the attention mechanism towards relevant and informative elements of the source sentence. in order to address this issue, prior research has suggested hard-coded or fixed attention patterns for self-attention heads to improve translation qual-ity @xcite . however, the complexity and diversity of language necessitate the consideration of varied attention patterns for different sentences in the context of translation. by constraining the attention weights to fixed values, the model may encounter difficulties in accommodating diverse sentence structures and capturing long-range dependencies with accuracy. consequently, this constraint may result in diminished performance @xcite , particularly for sentences that do not align harmoniously with the predetermined attention patterns. in this paper, after identifying a significant difference in the entropies of attention heads between high-resource and low-resource trained models (section 5.1), we introduce an inductive bias through the proposition of entropy and distance regularization (section 3.3). our approach aims to induce selective attention by regularizing the distance and entropy in the distribution of attention heads. specifically, we introduce a novel term into the loss function to guide the learning process, which encourages the low-resource nmt model to emulate the patterns observed in the attention of higherresource models. this additional bias is incorporated to improve the low-resource nmt model's capability to capture intricate language patterns and enhance translation performance. experimental results demonstrate the effectiveness of our approach and underscore the importance of inductive bias in narrowing the performance disparity between lowand high-resource nmt systems. ## related work prior work has explored various approaches to improve low-resource performance by leveraging highresource language pairs. this includes initializing model parameters from a large-scale trained model @xcite , as well as techniques such as multilingual neural machine translation @xcite , cross-lingual knowledge distillation @xcite and large pre-trained models that aim for universal language understanding @xcite @xcite . while these methods have significantly improved low-resource nmt, they rely on the availability of a large amount of additional data. however, it is crucial to explore techniques that facilitate the more efficient utilization of the model. inductive bias plays a fundamental role in machine learning as it allows for the incorporation of prior knowledge or assumptions into learning systems @xcite . different regularization techniques and architectural choices can introduce specific biases to shape the behavior of models. for example, regularization biases models towards relying less on a few influential features, convolutional neural networks bias models to capture local relationships between input, and attention mechanisms @xcite bias models to capture longrange dependencies. additionally, in the context of attention mechanisms, specific biases can be introduced to shape the behavior of models and improve their performance. @xcite encourage the attention to pay more focus on the content words rather than functions words. in the context of summarization, @xcite propose an attention mechanism that proactively generates tokens in the decoder that are similar or topical to the input. @xcite introduce an attention mechanism that is encouraged to assign similar attention weights to consecutive words. structured attention networks @xcite incorporate graphical models to generalize simple attention, while the training time significantly (5×) increases. more similar to our motivation, lp-sparsemap (niculae and martins, 2020) models attention distance between consecutive words for a classification task by introducing trainable parameters, but its scalability to large-scale experiments is limited. in contrast, our approach, based on applying a regularizer, is faster, less complex, and can be efficiently executed on gpus, making it scalable for large-scale training and fine-tuning setups. the closest work to our method is fixedattention @xcite , which enforces fixed (untrainable) attention patterns. however, they focus solely on encoder self-attention, overlooking the importance of cross-attention heads in neural machine translation @xcite . similarly, @xcite introduce hard-coded gaussian attention that replaces the attention distribution computation, i.e., scaled dot product of queries and keys, with a fixed gaussian distribution, leading to a negative impact on translation quality. given the concept of entropy that has been used in machine translation @xcite , in the next section, we propose our method that can be applied to all different attention components, i.e., encoder self-attention, decoder self-attention, and encoder-decoder (cross) attention, while consistently yielding significant improvements across various experimental setups. ## results in this section, we start with a comprehensive analysis of multi-head attention entropy across various data setups. subsequently, we demonstrate the striking effectiveness of eadra when compared to both the transformer model and the most closely related approach, fixed attention. additionally, we delve into the influence of eadra's hyperparameters. moreover, we present results involving large pre-trained fine-tuning, a method widely recognized as a strong baseline. ## discussion by introducing regularization techniques that target distance and entropy in attention heads, we achieve substantial improvements over various language pairs. extensive experiments demonstrate the effectiveness of these methods in low-resource nmt scenarios. the flexibility offered by eadra enables the nmt model to selectively allocate attention during training. conversely, fixed and unlearnable attention patterns prove to be more beneficial in the case of extremely low-resource languages with fewer than 50k training samples. in such scenarios, fixing the attention mechanism provides a more reliable approach, as the model's capacity to learn from a small dataset is limited. in this work, we mitigate the challenge of improving low-resource nmt by introducing a form of regularized attentions. we introduce eadra, which promotes focused attention by prioritizing key elements. extensive experiments on diverse lowresource language pairs demonstrate significant improvements in translation quality, validating the effectiveness of eadra. our findings highlight the importance of attention regularization techniques in enhancing nmt performance, particularly in low-resource settings. eadra offers a practical and scalable solution with negligible computational overhead and a few lines of code. ## limitations we only focus on improving low-resource nmt. however, higher-resource settings might also gain from regularized attentions facilitated by eadra and it may contribute to faster convergence as well. additionally, we demonstrate the effectiveness of our proposed method using multiple low-resource language pairs, whereas there are many other language pairs with limited data. furthermore, the encouragement of focused attention rather than dispersed attention through eadra leads us to hypothesize that our method may exhibit higher generalizability to sentence perturbations. this, in turn, could result in less volatile behavior of the nmt system @xcite . we leave these investigations to future work. the results demonstrate that eadra sustains a convergence speed comparable to the baseline. this observation underscores the efficiency of eadra in terms of convergence, further solidifying its viability in practical applications. this suggests that the incorporation of eadra does not come at the cost of prolonged training times, making it a practical choice for low-resource nmt tasks