Hierarchical Attention Adapter for Abstractive Dialogue Summarization (2025)

Abstract: Dialogue summarization is still a very challenging task even for large language models (LLMs). On the one hand, some previous approaches have pre-trained language models specifically for dialogue understanding and summarization, but they have been limited to relatively small models. On the other hand, other works have tried to directly exploit the dialogue semantics and discourse structures in their modeling effort, but by construction, they require access to those structures, which is in itself a largely unsolved problem. In this paper, we synergistically combine these two ideas in an approach that can be seamlessly integrated into the decoder-only architecture adopted by the most state-of-the-art LLMs. In particular, our novel solution leverages the parameter-efficient fine-tuning (PEFT) paradigm to model the hierarchical structure of dialogues, where input sequences are naturally segmented into dialogue turns, and then fine-tune the model for abstractive summarization. From experiments on two datasets, we find that Hierarchical Attention Adapter outperforms all baseline adapter methods on SummScreen, where our approach can also be combined with LoRA to achieve the best performance on SamSum.

## introduction the explosion in real-time messaging, consultation forums, and online meetings has resulted in a vast amount of conversational data, necessitating more efficient methods for understanding and extracting key information. dialogue summarization, which aims to automatically distill salient information from dialogues, has been widely applied across various scenarios in different domains. these include task-oriented dialogues such as customer service @xcite , law @xcite , medical care @xcite ), and open-ended dialogues like chit-chat @xcite , screen plays @xcite , and forum discussions @xcite . conventional dialogue summarization models typically approach the task as a sequence-tosequence problem and fine-tune encoder-decoder models such as bart @xcite and t5 @xcite . although these methods show promising results @xcite @xcite , they are limited to relatively small transformer architecture models and cannot be easily employed on large decoder-only models. another line of research examines dialogue specific features such as speaker marker, discourse structure, topic changes, and coreference information @xcite @xcite and incorporates dialogue semantics and structural information into model pre-training, oftentimes involving an intermediate stage for dialogue-specific information extraction. however, these methods require additional effort and the results constructed from such information may not be accurate. on the other hand, recent advances in large language models (llms) have revolutionized the field of nlp and have become an essential building block in various intelligent user-facing applications @xcite , resulting in a shift in focus from relatively small encoderdecoder models to large-scale decoder-only models. the remarkable achievements of these llms can largely be attributed to research on model scaling @xcite @xcite , where increasing the number of model parameters and the volume of pretraining data can lead to significant enhancements of their capabilities to understand and generate human language. however, despite the success of sophisticated prompting @xcite and demonstration selection @xcite strategies, there remains a noticeable performance gap compared to fine-tuning @xcite , especially for tasks such as dialogue summarization where the input sequences can be long and possess hierarchical structures. in this paper, we synergistically combine the ideas of using modern llms with dialogue structural information in dialogue summarization task, without explicitly injecting rigid linguistic structures. to this end, we select gpt-style decoderonly architecture as our llm backbone. this simplified architecture allows for more efficient pretraining through the language modeling objective, where the model can quickly process and generate tokens without first transforming the input sequence into an abstract representation by the encoder. however, an innate drawback of such architecture is the token-level unidirectional flow, which limits the ability to model the full context of the dialogue, especially the nuances and dependencies that emerge from future turn back to past ones. by contrast, encoder-decoder architecture encodes the input sequence using bidirectional conditioning and results in inherently stronger representations compared to the causal conditioning representation. in order to fine-tune decoder-only models more effectively, we propose a novel parameter-efficient fine-tuning (peft) architecture that encodes the input with bidirectional contextualization. another great challenge in dialogue summarization is the length of text, which can sometimes exceed the model's input limit. while recent efforts have managed to increase the context window of the model by scaling the positional embeddings @xcite and reducing the complexity of attention through sparsity @xcite , the theoretical support for long context are often measured with the language modeling loss and synthetic tasks, which do not comprehensively demonstrate their effectiveness in practical applications. inspired by previous studies @xcite @xcite demonstrating that hierarchical structures of input can significantly enhance downstream performance, especially for tasks with naturally segmented input sequences, we propose encoding dialogues at both the speech turn level and the dialogue level. this is achieved through two attention layers in our adapter module. our proposed adapter, called hierarchical attention adapter, can incorporate the interactions of speech turns in dialogues naturally without the need for external structural integration. to summarize, our contributions are threefold: ## conclusions and future work in this work, we present hierarchical attention adapter, a novel peft architecture that uses bidi- summary type generated summary samsum reference kate broke her arm and she's going to the hospital. she'd like to know whether her medical insurance covers hospital costs. greg suggests her to call linda or ask someone at the reception about it. lora kate broke her arm and is going to the hospital. ours kate broke her arm and is being sent to the hospital. she wants to know if their medical insurance covers hospital costs. reference casey got a new nail polish and did her nails herself. it took her nearly 4 hours, so she won't do her friends' nails, as it takes too long. lora casey did her nails herself. it took her 4 hours. ours casey did her nails herself and they took her 4 hours. amelia and kristen want her to do their nails too but she refuses. summscreen reference adam bails krystal out of jail and makes her an "offer" in exchange for her telling him where their mutual grandson is. she denies that she knows he's alive and refuses his offer. but jr gets bianca to admit that he is alive. . . lora jonathan and greenlee continue to argue about ryan's actions. jonathan wants to know if greenlee is going to stand by ryan. greenlee tells jonathan that she will stand by ryan. jonathan tells greenlee that he wants to know if she is going to stand by him. . . ours greenlee and jonathan argue about ryan and kendall. maggie comes in and tells them that she just saw bianca. greenlee and jonathan argue about kendall. maggie comes in and tells them that she just saw bianca. . . reference kendall's trial begins. bianca urges kendall to listen to what she learned in her dream about something she needs to find in order to help kendall get acquitted. jack proposed to erica and she accepts. edmund suddenly regains consciousness. . . lora kendall is nervous about her trial. bianca tells her that she had a dream that she was at michael's condo the night he was killed. kendall tells her that it was impossible. bianca tells her that she is worried about her. kendall tells her that she is worried about her. . . ours kendall is in court for her trial. bianca tells kendall that she had a dream that she was at michael's condo the night that he died. kendall tells bianca that she cannot believe that she slept the whole day. . . for future work, we wish to perform additional analysis to study the usefulness of the different components in our proposed architecture and perform further experiments on additional datasets. we will also perform further evaluation to compare the faithfulness and factuality of summaries generated by different models. lastly, while our current proposal requires segment annotation of the input sequence (i.e., dialogue turns), we intend to extend our approach to implicitly learn the segment boundaries during training and generalize to other summarization tasks such as scientific paper summarization.