Trainable Hard Negative Examples in Contrastive Learning for Unsupervised Abstractive Summarization (2024)

Abstract: Contrastive learning has demonstrated promising results in unsupervised abstractive summarization. However, existing methods rely on manually crafted negative examples, demanding substantial human effort and domain knowledge. Moreover, these human-generated negative examples may be poor in quality and lack adaptability during model training. To address these issues, we propose a novel approach that learns trainable negative examples for contrastive learning in unsupervised abstractive summarization, which eliminates the need for manual negative example design. Our framework introduces an adversarial optimization process between a negative example network and a representation network (including the summarizer and encoders). The negative example network is trained to synthesize hard negative examples that are close to the positive examples, driving the representation network to improve the quality of the generated summaries. We evaluate our method on two benchmark datasets for unsupervised abstractive summarization and observe significant performance improvements compared to strong baseline models.

## introduction abstractive summarization is the task of generating concise summaries that potentially contain new phrases or sentences while preserving the core information of the source documents @xcite @xcite . abstractive summarization systems could be deployed in various applications such as news headline generation. due to the challenge of collecting massive and high-quality parallel data (i.e., document-summary pairs) for training, it is increasingly important to study unsupervised abstractive summarization, which is especially valuable to uncommon domains and languages without sufficient labeled data @xcite . document ... a new meme was born last night, once again at the expense of miami heat star forward lebron james. the meme, #lebroning, is flooding social media in response to james being carried off of the court in the waning minutes of the first game of the nba finals...jordan famously played a game in the 1997 nba finals while suffering from influenza, winning the game ... ## discussion and conclusion in the era of llms (large language models), llms could generate high-quality summaries that are significantly preferred by humans @xcite . why do we still study unsupervised summarization? we believe that llms (e.g. chatgpt) are not suitable for all scenarios (e.g., confidential/sensitive data or domain, minority languages) @xcite @xcite , and thus it is still important to conduct research on training models for summarization tasks. in this paper, we have provided an unsupervised training strategy for summarization. researchers or engineers could utilize our method to train the models on their own data (e.g. company confidential data, personal private data), domains (e.g. medical texts, legal documents), languages (e.g. minority language), where llms could not be used or might not be good enough. since our method is unsupervised, there is no need for human-written summaries as references, thus significantly reducing human labor and costs in training. besides, researchers could fine-tune their own llms or pretrained models (e.g. pretrained language models) using our method for better summarization performance. our method could also be applied in some semi-supervised scenarios where limited human-written references are available. to conclude our work, we explore and study the problem of trainable hard negative examples in contrastive learning for unsupervised abstractive summarization, and propose to train a negative example network and a representation network in an adversarial manner. the negative example network is optimized to generate high-quality and diverse hard negative examples for the representation network to generate better summaries and representations. extensive experiments and analysis on two benchmark datasets demonstrate the effectiveness of our proposed method, as well as the significant advantages over the strong baseline models.