Improving Aspect-Based Summarization via Contrastive Learning with Anchored Negative Examples (2025)

Abstract: Text summarization helps users manage information overload, but traditional methods can be cumbersome when seeking specific details within a document. Aspect-based text summarization addresses this by using a query to guide which information should be summarized. However, distinguishing relevant from irrelevant information for a given aspect remains challenging in LLM-based summarization models. In this work, we propose utilizing contrastive learning to encourage LLMs to focus on aspect-related signals during training. We further design two variants of the learning algorithm, aspect-anchored and summary-anchored, corresponding to the strategies used in constructing negative examples. Evaluation with two representative LLM families (Llama 2 and Pythia) and two benchmark datasets (AnyAspect and CovidET) demonstrates the proposed methods’ strong performance compared to their supervised fine-tuning and zero-shot counterparts, highlighting contrastive learning as a promising direction for aspect-based text summarization.

## introduction aspect-based text summarization is a crucial task within natural language processing that addresses the limitations of conventional summarization methods. traditional models generate concise summaries of documents, aiming to save users time and effort. however, the varying lengths and diverse content of documents often render general summaries inadequate. consider a news article covering an event from multiple perspectives -location, key figures, and the event itself. a user might be interested in a summary focused on the event's location rather than the individuals involved, as shown in figure 1 . aspect-based summarization tackles this issue by providing both a document and figure 1 : an illustration of aspect-based text summarization created based on an example from the anyaspect dataset @xcite . the first summary is related to the given aspect england, while the second one is not. a specific aspect, guiding the model to generate a summary tailored to that particular focus. the objective is to parse the document and selectively use the information only relevant to the given aspect. a significant challenge in aspect-based text summarization lies in the model's ability to isolate and highlight aspect-specific information while effectively distinguishing it from the rest of the text. without this crucial capability, the model risks generating a generic summary that fails to address the intended aspect. for example, a model presented with a news article about an earthquake @xcite and tasked with generating summaries for two aspects (geography and re-covery) might produce identical summaries detailing the earthquake's magnitude and recovery effort, neglecting the distinct nuances of each aspect. this underscores the need for a mechanism that enables the model to cluster similar information and separate dissimilar information within the latent space. while prior work has established a foundation for understanding the capabilities of large language models (llms) in aspect-based text summarization @xcite and explored the impact of contrastive elements @xcite , there remains a gap in informing aspect-based signals in llms. furthermore, it lacks dedicated learning algorithms designed to teach llms to distinguish aspect-related signals during training (e.g., via comparing the aspects and their associated summaries). while adding instructions in prompts (e.g., "summarize this article with respect to aspect within one short sentence.") has shown promise @xcite , a training-level contrastive loss function is necessary to effectively incorporate aspect information and emphasize crucial signals within the data. to address this gap, we propose a novel approach that integrates contrastive learning into the finetuning of open-source llms to enhance aspectbased text summarization. we augment existing aspect-based summarization datasets to create two types of anchored negative examples: aspect anchored negative examples (same aspect with different summaries) and summary anchored negative examples (same summary with different aspects). during fine-tuning, the model is trained to differentiate either summaries or aspects with an anchor presented. by learning to discriminate between these pairs, the model gains additional context and generates summaries that are more closely aligned with the given aspect. this work, to the best of our knowledge, is the first to integrate contrastive learning into llm finetuning specifically for aspect-based text summarization. our contributions are threefold: (1) we design a novel contrastive learning algorithm with anchored negative examples for llms fine-tuning on aspect-based text summarization; (2) we perform an empirical study evaluating the performance of two prominent llms on two benchmark datasets; ## related work the research work in this paper is related to prior work at least in the following two dimensions: aspect-based text summarization and contrastive learning. aspect-based summarization. as previously described, the motivation behind aspect-based sum-marization is to extract and summarize information relevant to a given aspect from a document. for instance, one early work by @xcite aimed to summarize contrasting opinions to enhance the comprehensiveness of the final summary. recent research in aspect-based summarization can be broadly categorized into two areas: developing novel learning strategies and constructing new datasets. regarding learning strategies, @xcite al. (2023b) 2023 ) introduced an unsupervised method for extracting opinions from source documents for summarization. unlike these previous studies, this paper focuses on improving the summary accuracy of aspect-based summarization. specifically, for a given aspect, the proposed approach ensures that the generated summary is relevant to that particular aspect and not to other aspects. in addition, recent research has shown that llms cannot handle aspect-based summarization as well as traditional summarization tasks @xcite , which is echoed in our baseline experiment (section 5) and further motivated to develop new learning strategies. concurrent with the development of new learning strategies, significant effort has been dedicated to creating benchmark datasets for aspect-based summarization. these datasets span various domains, including scientific publications @xcite , climate change @xcite , social media (zhan et al., 2022a), legal decisions (t.y.s.s. et al., 2024), disordered texts @xcite , and news articles @xcite . while many datasets are limited in the number of aspects or specific domains to focus on, some works have explored a larger range of aspects @xcite or not targeted on specific domains @xcite . in this work, we utilize the anyaspect dataset @xcite and the covidet dataset @xcite to ensure evaluation across diverse domains and document lengths. contrastive learning for summarization. while contrastive learning has been surveyed generally within the domain of natural language processing @xcite and extensively employed to enhance representation learning in text summarization @xcite , its application to aspect-based summarization remains largely unexplored. specifically, contrastive learning has been shown to improve the alignment of generated summaries with source documents, reducing factual inconsistencies and hallucinations @xcite . it has also been used to address the issue of exposure bias @xcite . @xcite further explored different strategies for constructing contrastive examples, such as masking, swapping, and replacing words or sentences, to improve learning performance in text summarization. @xcite proposed an automated method of constructing "hard" negative examples for contrastive learning. this method is applied within standard text summarization-our method similarly creates "hard" negative examples by leveraging negative aspects and summaries from the same source document as the positive example, thus reducing potential noise. @xcite proposed a novel approach that leverages contrastive learning to formulate summarization generation as a reference-free evaluation problem, where the model is trained to distinguish between high-quality and low-quality summaries without relying on reference summaries. @xcite utilize llms as evaluators in contrastive learning for smaller models such as bart. @xcite create a new evaluation method for text summarization without using gold standard summaries. they utilize linguistic and semantic aspects to perturb text summaries for negative samples and train their evaluator using contrastive learning. @xcite mitigate hallucinations and non-factual information in text summaries by implementing a contrastive reward learning framework that generates candidate summaries from a pretrained sequence-to-sequence model, which are then ranked using factuality metrics. @xcite utilize contrastive preference optimization to mitigate hallucinations made by llms in news summarization. among existing works, wang and wan (2021) presents the most closely related study, employing contrastive learning to inform the (dis)similarity of aspects. however, the present work differs sig-nificantly in both loss function design and training strategy. specifically, the proposed method defines the contrastive loss directly on the generated aspectbased summaries, compelling large language models to capture the crucial aspect-related information from long input documents. ## contrastive fine-tuning in aspect-based text summarization, the length and content variability of documents create a difficult learning environment for models. a successful model must identify and extract information specifically related to the given aspect and synthesize a summary. if the model cannot effectively distinguish between informative sentences, it risks generating a generic summary that omits aspect-specific information, thus failing to fulfill its objective. to facilitate the model's learning of pertinent aspectrelated information, we employ contrastive learning through a particular design on loss function and example construction. as with other applications of contrastive learning in text summarization, two key components are essential: the definition of contrastive loss and the construction of contrastive examples. in this work, to mitigate hallucinations, we utilize only the reference summary of each example as the positive example. consequently, our focus in example construction is solely on generating negative examples. for each training example consisting of a document d, aspect a, and summary s, the formulation of contrastive learning for aspect-based text summarization can be expressed as: unlike losses defined in previous work @xcite , the contrastive loss in equation 1 can be interpreted as the geometric mean of the likelihoods of the negative examples, log( n @xmath0. given the document and the aspect, a contrastive example is constructed by selecting a summary from the same source document with a different aspect. variants of the loss. another variant of the loss function focuses on the aspect dimension rather than the summary. the corresponding loss function is defined as: a further variant could combine l s and l a , applying negative examples to both summary and aspect variables. however, this poses additional training challenges, so we chose not to explore this direction further. quality of negative examples. in this work, for a given document and aspect, we select the summary from another aspect within the same document as the negative example in l s . similarly, for l a , we choose the aspects available in the same document as the negative examples. by augmenting the summaries and aspects from within the same data point, we ensure that these negative examples are on-topic and semantically sound. we noticed that prior work on contrastive learning often struggles with the quality and various issues when constructing negative examples, as discussed in @xcite . for example, noise introduced into the negative examples may cause further issues in a text summarization system, such as hallucinations @xcite . on the other hand, the proposed example construction method helps the model learn which information is similar and dissimilar from the same input document without the risk of noise from automatically perturbed samples. ## experimental setup this section describes the experimental setup, including datasets, baseline and competitive models, evaluation metrics, and additional implementation details. ## experimental results our experiments demonstrate the effectiveness of contrastive learning for aspect-based text summarization, as shown with the anyaspect dataset in across both the anyaspect and covidet datasets, our contrastive fine-tuning approach outperformed baseline models, standard fine-tuned llms, and zero-shot llms on most of the evaluation dimensions. compared to standard finetuning, we notice that our method has a stronger performance on the anyaspect dataset compared to covidet. this indicates that contrastive finetuning may be a more viable approach depending on the data. the performance improvement on anyaspect is nearly 2% on rouge-1 score, 0.5% improvements on rouge-2, over 1% improvement in rouge-l, and over 2% improvement in bleu score with the llama 2 (13b) model when trained with our contrastive loss compared to its non-contrastive counterpart. we validate our findings with significance testing between llama 2 (13b) contrastive fine-tuned and llama 2 (13b) supervised fine-tuned for anyaspect and find a p-value of 0.003. a table of p-values of the top-performing contrastive fine-tuned models can be referenced at we also confirmed the significant advantage of fine-tuning over zero-shot inference, with finetuned models exhibiting substantial gains. while encoder-decoder models like flan-t5 and bart provided strong baselines, our contrastive finetuned llms achieved superior performance. finally, we observed a performance difference across datasets, which we attribute to variations in domain and aspect granularity. in the remainder of this section, we will dive into different perspectives of the results. most of the discussion will be based on rouge and bleu, while the results from llm-based evaluation will be discussed in the end of the section. an analysis of negative example construction will be presented in section 6 with other hyper-parameters. performance across different models. among the contrastively fine-tuned llms, llama 2 consistently outperformed pythia, reinforcing previous findings on llm performance hierarchies in aspectbased summarization @xcite . this superior performance of llama 2 may be attributed to its larger context size and extensive pre-training data. furthermore, the substantial performance gap between fine-tuned and zero-shot llms (over 10% in rouge-1) confirms the benefits of data-specific adaptation. regarding model size, we see that llama 2 13b outperforms its smaller counterpart for the anyaspect dataset. the same is not observed for pythia, where pythia 1b outperforms its larger counterpart. for covidet, we see llama 2 7b and 13b's results are commensurate, whereas the larger pythia model outperforms the smaller. while flan-t5, bart, and gpt-4o provided competitive baselines, their performance trailed that of the contrastively fine-tuned llama 2, suggesting that the combination of llm scale and contrastive learning provides the best performance. the similar performance of flan-t5 and bart can be attributed to their shared encoder-decoder architecture, commonly recognized for its effectiveness in text summarization tasks. further investigation is warranted to explore the relative strengths of decoder-only versus encoder-decoder architectures for text summarization. performance difference between datasets. we observed a performance difference between the datasets, with models generally achieving higher scores on anyaspect compared to covidet. this discrepancy likely stems from domain differences and aspect granularity. anyaspect, derived from cnn/daily mail news articles, aligns more closely with the training data of models like bart (which was instruction-tuned on cnn/daily mail). covidet, focusing on sentiment analysis, presents a different domain. additionally, the nature of the aspects themselves may play a role. the aspects from the anyaspect dataset (e.g., spain vs. u.s.) are more distinct than those in covidet (e.g., joy vs. anticipation), potentially making the latter a more challenging classification task. more nuanced aspects could increase the difficulty of creating distinct summaries, leading to lower evaluation scores. ## algorithm analysis to gain a comprehensive understanding of the proposed algorithms, we used llama 2 7b as the base model and anyaspect as the dataset. we considered the following hyper-parameters and their effect on performance: learning rate, strength of the negative example via contrastive coefficient, and cutoff length (as shown in figure 3 ). based on the results in learning rate. the following learning rates were evaluated: 1e-3, 1e-4, 1e-5, 5e-4, and 5e-5. we can see that llama 2 peaks in performance at 1e-4 with diminishing returns for both large and smaller learning rates. this indicates that 1e-4 strikes a balance between performance, convergence, and stability when performing contrastive fine-tuning. contrastive coefficient λ. we evaluate the following values of λ in our contrastive loss ranging between (0,1): 0.1, 0.25, 0.5, and 0.75. our results indicate a linear decrease in rouge and bleu scores as λ value increases. the smallest λ value (0.1) yielded the highest rouge-1 score among all evaluated hyper-parameters in our algorithm analysis. this indicates that a positive example supplemented by a weaker impact from negative examples was more helpful in the model's learning as opposed to more dominant negative examples during training. cutoff length. we evaluate two different cutoff lengths during training and inference: 300 tokens vs. 500 tokens -the upper limit is capped by the hardware resource we used in the experiment. we see performance increase as cutoff length increases. due to the longer length of the input document for a news domain, some content related to the aspects will be located closer to the end of the document. a instruction generate a summary based on aspect: facebook ## conclusion in this study, we perform the first systematic analysis of llm performance in aspect-based text summarization with a formulated contrastive loss function. our method sees an improvement in evaluation scores compared to standard fine-tuning and baseline models. we contribute additional analyses in an ablation study which evaluates the effects of negative example construction on performance, as well as hyper-parameters such as cutoff length, learning rate, and strength of the contrastive component through a coefficient value.