Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation (2022)

Abstract: The multi-head self-attention mechanism of the transformer model has been thoroughly investigated recently. In one vein of study, researchers are interested in understanding why and how transformers work. In another vein, researchers propose new attention augmentation methods to make transformers more accurate, efficient and interpretable. In this paper, we combine these two lines of research in a human-in-the-loop pipeline to first discover important task-specific attention patterns. Then those patterns are injected, not only to smaller models, but also to the original model. The benefits of our pipeline and discovered patterns are demonstrated in two case studies with extractive summarization and topic segmentation. After discovering interpretable patterns in BERT-based models fine-tuned for the two downstream tasks, experiments indicate that when we inject the patterns into attention heads, the models show considerable improvements in accuracy and efficiency.

## introduction with transformer-based models @xcite dominating the leaderboard for many key nlp tasks such as summarization @xcite , topic segmentation @xcite , and sentiment analysis @xcite , their core multi-head self-attention mechanism has also been thoroughly investigated. in particular, to explain why and how transformers work, researchers have analyzed the learnt self-attention matrices of trained transformer models (e.g., @xcite ; @xcite ), with @xcite for instance, exploring the attention patterns in bert @xcite and gpt-2 @xcite , as well as analyzing their alignment with syntax. meanwhile, a parallel line of research has explored injecting predefined patterns into attention * corresponding author. matrices of transformers in an attempt to reduce the run-time complexity of self-attention while maintaining competitive accuracy. this can be done by either replacing the attention weights with a fixed matrix @xcite @xcite ; or alternatively by guiding the attention weights through more flexible masking strategies @xcite @xcite @xcite @xcite . in this work, we propose and test a novel humanin-the-loop pipeline that to the best of our knowledge is the first attempt to combine research on analyzing self-attention with work on injecting patterns into attention matrices. to start, human users visually explore the attention matrices of transformers to identify task-specific patterns that could be formalized as a predicate. after quantitatively evaluating the patterns on the validation set, they can be injected into attention heads of transformer models to simultaneously improve task accuracy and make the model more efficient by sparsifying the attention matrices foot_0 . this is in contrast to previous work that mostly focuses on the trade-off between two metrics. in both scenarios, we argue the interpretability of the resulting model is improved. we provide a justification of our claim based on the predictive, descriptive, and relevant (pdr) framework proposed by @xcite . specifically, by injecting human-interpretable patterns into the model, we increase the model's descriptive accuracy by explicitly encoding useful relationships between input tokens in the attention weights while simultaneously improving the predictive accuracy in task performance. further, the patterns are relevant for the problem since they are discovered in the human-in-the-loop process and are verified to be important for the task. in order to test the feasibility and potential benefits of our approach, we run two case studies on the tasks of extractive summarization and topic segmentation using bert-based models, and we find that: (i) some of the important heads do have patterns with interpretable meaning, either lexical, local or positional. for instance, the matching token (i.e. the trend of attending to other tokens with the same id) is an important clue for the summarization model. (ii) we show that when the discovered patterns are injected to the attention heads of transformer models, both the task accuracy and efficiency of the model can be significantly improved. (iii) additionally, we also propose a strategy to improve the performance of pretrained transformer models by injecting patterns through pals. ## related work in §2.1 and §2.2, we describe the two lines of research that our work aims to combine. §2.3 summarizes recent trends on enhancing the interpretability of neural nlp models, while §2.4 introduces the two nlp tasks used for our case studies. ## proposed generic pipeline as an overview, we first briefly describe the proposed pipeline (figure 1 ). specifically, given a trained model, users are asked to first discover important patterns using the visual interface @xcite by following three steps: step 1 ( §3.1.1): estimate the importance scores for all the heads on the validation set, and find important heads that stand out. step 2 ( §3.1.2): discover relevant patterns in the important heads, using criteria described in §3.1.2. step 3 ( §3.1.3): evaluate and validate the patterns to confirm their global relevance. once the important patterns are identified, there are two common approaches (i.e. fixing and masking) to inject them as constraints to the attention matrices in the transformer-based neural models (see §3.2). the pipeline also enables two scenarios, in which injecting the patterns can be beneficial: the first one is to train a new model with the patterns injected, while the second one is to enhance the original model. ## case studies in this section, we demonstrate the effectiveness of our pipeline in two nlp tasks (extractive summarization and topic segmentation) and discuss our findings in detail. ## conclusion and future work in this paper, we propose a generic human-in-theloop pipeline, which combines two popular research directions, where the findings from an analysis of the multi-head self-attention mechanism in transformers can be utilized to create more accurate and interpretable transformer models. a human analyzes the attention heads of a task-specific model, discovers and verifies potentially meaningful patterns, and injects them into the attention heads of 7 an illustrative example is shown in appendix c.2 models. by running a case study on two nlp tasks, we show the effectiveness of our pipeline. we do discover meaningful patterns in some important heads, and the relationships encoded in the patterns help us understand the features used by the model for both tasks. furthermore, by injecting the patterns into the smaller models and the original model, the performance and interpretability get improved in both cases. as future work, we plan to apply our pipeline to other nlp tasks (e.g. language modeling, abstractive summarization) and explore and verify whether the important patterns from one task can be transferable to another task. similarly, we also plan to apply our pipeline to different model variants to examine and compare the patterns encoded in the attention weights. in the long term, our pipeline could be naturally automated by replacing the pattern discovery step with evaluating predefined linguistic patterns. however, assessing the efficiency gains from injecting such patterns (requiring ground-truth annotations) would require more in-depth studies beyond the scope of this paper. finally, since human factors are an important aspect of interpretability, we plan to conduct extensive user studies across different nlp tasks and model sizes to examine the trade-off between human-cost and the coverage of discovered patterns.