On Biasing Transformer Attention Towards Monotonicity (2021)

Abstract: Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has facilitated or enforced learning of monotonic attention behavior via specialized attention functions or pretraining. In this work, we introduce a monotonicity loss function that is compatible with standard attention mechanisms and test it on several sequence-to-sequence tasks: grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization. Experiments show that we can achieve largely monotonic behavior. Performance is mixed, with larger gains on top of RNN baselines. General monotonicity does not benefit transformer multihead attention, however, we see isolated improvements when only a subset of heads is biased towards monotonic behavior.

## introduction many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has focused on learning monotonic attention behavior either through specialized attention functions @xcite @xcite or pretraining @xcite . however, it is non-trivial to port specialized attention functions to different models, and recently, @xcite ; @xcite found that a transformer model @xcite outperforms previous work on monotone tasks such as grapheme-to-phoneme conversion, despite having no mechanism that biases the model towards monotonicity. in the transformer, it is less straightforward to what extent individual encoder states, especially in deeper layers, still represent distinct source inputs after passing through several self-attention layers. consequently, it is unclear whether enforcing monotonicity in the transformer is as beneficial as for recurrent neural networks (rnns). in this paper, we investigate the following research questions: specifically, we want to incorporate a monotonicity bias in a way that is agnostic of the task and model architecture, allowing for its application to different sequence-to-sequence models and tasks. to this end, we introduce a loss function that measures and rewards monotonic behavior of the attention mechanism. 1 we perform experiments and analysis on a variety of sequence-to-sequence tasks where we expect the alignment between source and target to be highly monotonic, such as grapheme-to-phoneme conversion, transliteration, morphological inflection, and dialect normalization and compare our results to previous work that successfully applied hard monotonic attention to recurrent sequence-tosequence models for these tasks @xcite . our results show that a monotonicity bias learned through a loss function is capable of making the soft attention between source and target highly monotonic both in rnns and the transformer. we find that this leads to a similar improvement to previous works on hard monotonic attention for rnns, whereas for transformer models, the results are mixed: biasing all attention heads towards monotonicity may limit the representation power of multihead attention in a way that is harmful even for monotonic sequence-tosequence tasks. however, for some tasks, we see small improvements when limiting monotonicity to only a subset of heads. ## related work attention models @xcite @xcite are a very powerful and flexible mechanism to learn the relationship between source and target sequences, but the flexibility might come at the cost of making the relationship harder to learn. previous work has shown that their performance can be improved by introducing inductive biases. @xcite introduce various structural alignment biases into a neural machine translation model, including a positional bias. while this bias is motivated by the fact that a given token in the source often aligns with a target token at a similar relative position, it does not explicitly encourage monotonicity. in contrast, @xcite propose to modify the attention mechanism to learn hard monotonic alignments instead of computing soft attention over the whole source sequence. several extensions have been proposed: having a pointer monotonically move over the source sequence and computing soft attention on a local window @xcite or from the beginning of the sequence up to the pointer @xcite . for tasks like simultaneous translation and automatic speech recognition, the main benefit from hard monotonic attention is that decoding becomes faster and can be done in an online setting. however, many sequence-to-sequence tasks behave roughly monotonic and biasing the attention towards monotonicity can improve performance; especially in low-resource settings. @xcite show that hard monotonic attention works well for morphological inflection if it mimics an external alignment. @xcite propose a probabilistic latentvariable model for hard but non-monotonic attention which @xcite later extend to exact hard monotonic attention. in contrast to aharoni and goldberg (2017), the alignment is learned jointly with the model. their approach outperforms several other models on grapheme-to-phoneme conversion, transliteration, and morphological inflection. monotonic attention has also improved tasks such as summarization @xcite and morphological analysis @xcite . recently, the transformer architecture @xcite has outperformed rnns in lowresource settings for character-level transduction tasks @xcite and neural machine translation @xcite . while there has been some work on extending the methods of @xcite ; @xcite ; @xcite to multihead attention @xcite , we are not aware of any work that studied monotonicity in transformers for monotonic tasks, such as grapheme-to-phoneme conversion, transliteration, or morphological inflection. to this end, we propose a model-agnostic monotonicity loss that can seamlessly be integrated into rnns as well as the transformer. our monotonicity loss captures how monotone the soft attention behaves during training, while two hyperparameters allow us to control how much monotonicity is enforced. by encouraging monotonicity through a loss instead of a modification of the attention mechanism, our implementation still brings all the benefits of soft attention to tasks where fast, online inference is not paramount and allows us to explore various trade-offs between unconstrained and fully monotonic attention. ## monotonicity loss we now introduce our monotonicity loss function. the loss function is differentiable and compatible with standard soft attention mechanisms and is thus easy to integrate into popular encoder-decoder architectures such as the transformer. on a high level, we compare the attention distribution between decoder time steps in a pairwise fashion and measure whether the mean attended position increases for each pair. let us denote the input sequence as @xmath0, ..., x |x| ), and the output sequence as @xmath1, ..., y |y | ). the interface between the encoder and decoder is one or several attention mechanisms. in its general form, the attention mechanism computes some energy e ij between a decoder state at time step i and an encoder state j. while this energy function varies, with popular choices being a feedforward network @xcite or (scaled) dot-product @xcite , they are typically normalized to a vector of attention weights α using the softmax function: these attention weights are then applied to obtain a weighted average c i of a vector of value states v : for our monotonicity loss, we also compute the mean attended position āi : we can then define the monotonicity loss in a pairwise fashion, comparing the mean attended position at time steps i and i + 1: we might want to bias the model towards strictly monotonic behavior, penalizing it if ā remains unchanged over several time steps. we can achieve this by incurring a loss if ā does not increase by some margin, controlled by δ. at the most extreme, with δ = 1, the loss is minimized if the mean attended position follows the main diagonal of the alignment matrix, increasing by |x| |y | at each time step. figure 1 shows how the margin δ can influence the monotonicity loss with some examples. in equation 4, costs are later summed over the target sequence. in practice, we normalize the cost by the number of tokens in a batch for training stability, as is typically done for the cross-entropy loss. if a model has multiple attention mechanisms, e.g. attention in multiple layers, or multihead attention, we separately compute the loss for each attention mechanism, then average the losses. we can also just apply the loss to a subset of attention mechanisms, allowing different attention heads to learn specialized behavior @xcite . ## analysis overall, our results show that the proposed monotonicity loss succeeds in making attention more monotonic, but effects on quality are more positive for rnns than for transformers. we now analyze the proposed loss function in more detail. ## conclusion we propose a model-agnostic loss function that measures and rewards monotonicity and can easily be integrated into various attention mechanisms. to achieve this, we track how monotonically the average position of the attention shifts over the source sequence across time steps. we show that this loss function can be seamlessly integrated into rnns as well as transformers. models trained with our monotonicity loss learn largely monotonic behavior without any specific changes to the attention mechanism. while we see some performance gains in rnns, our results show that biasing all attention heads in transformers towards monotonic behavior is undesirable. however, a bias towards monotonicity may be helpful if applied to only a subset of figure 6 : transformer attention maps for the sentence shown in figure 3 ; "but it was meant as compliment" and "we delete this and get to work". left: baseline (λ=0), right: with monotonicity loss on all heads (λ=0.1). for the future, we are interested in more sophisticated schedules for the monotonicity loss, possibly reducing λ over the course of training. this would help to learn monotonic behavior in the early training stages but gives the model more flexibility to deviate from such an attention pattern if needed. in this context, our loss function could also be used as an additional pretraining objective for transfer to very low-resource tasks. we would also like to test our loss function on tasks where the alignment may be harder to learn, for example in multimodal models or for long sequences. finally, using our loss function as a way to measure monotonicity could be an interesting tool for interpretability research.