Learning to Rank Salient Content for Query-focused Summarization (2024)

Abstract: This study examines the potential of integrating Learning-to-Rank (LTR) with Query-focused Summarization (QFS) to enhance the summary relevance via content prioritization. Using a shared secondary decoder with the summarization decoder, we carry out the LTR task at the segment level. Compared to the state-of-the-art, our model outperforms on QMSum benchmark (all metrics) and matches on SQuALITY benchmark (2 metrics) as measured by Rouge and BertScore while offering a lower training overhead. Specifically, on the QMSum benchmark, our proposed system achieves improvements, particularly in Rouge-L (+0.42) and BertScore (+0.34), indicating enhanced understanding and relevance. While facing minor challenges in Rouge-1 and Rouge-2 scores on the SQuALITY benchmark, the model significantly excels in Rouge-L (+1.47), underscoring its capability to generate coherent summaries. Human evaluations emphasize the efficacy of our method in terms of relevance and faithfulness of the generated summaries, without sacrificing fluency. A deeper analysis reveals our model’s superiority over the state-of-the-art for broad queries, as opposed to specific ones, from a qualitative standpoint. We further present an error analysis of our model, pinpointing challenges faced and suggesting potential directions for future research in this field.

## introduction query-focused summarization (qfs) is gaining prominence in research community. unlike conventional summarization tasks that aim to capture the overall essence of a document or a set of documents, qfs focuses on generating concise summaries in response to posed queries. this specialization enables a more targeted information retrieval process, offering summaries that directly address the informational needs rather than providing a broad overview of the source material. the advancements in qfs have been notably driven by the introduction of invaluable datasets of long documents such as qmsum with an average of 9k tokens @xcite and squality with an average of 5.2k tokens @xcite , which have facilitated deeper exploration and innovation in this field. these datasets have laid the groundwork for the development of transformersbased models which have shown strong potential in generating summaries that respond accurately to queries @xcite @xcite @xcite . however, despite this proficiency, their ability to effectively prioritize information-assessing its importance relative to a query to enhance summary relevance-remains an area for improvement. this study seeks to address this limitation, with a particular focus on long-input qfs, where summarizing multiple segments foot_0 for a given query presents unique challenges in capturing and prioritizing relevant content. particularly, in this study, we present a novel enhancement to qfs through the incorporation of learning-to-rank (ltr), a technique with established efficacy in information retrieval. our approach aims to refine the system's capability to discern and prioritize content segments not only by their relevance but also by their relative importance. this methodological advancement ensures that the produced summaries more accurately reflect the query's intent and hierarchically organize information by its significance. central to our approach is the augmentation of use of the decoder that shares parameters with the summarization decoder 2 , specifically designed for executing the ltr task at the segment level. while learningto-rank (ltr) is a classic approach, our innova-tion lies in adapting ltr principles specifically for query-focused summarization (qfs) at the segment level, which has been less explored in the literature. this strategy, inspired by the work of @xcite in adapting the t5 @xcite encoder-decoder framework for text ranking in query-document scenarios, is tailored to address the nuances of segment ranking within the qfs context. through the joint fine-tuning of summarization with cross-entropy loss, and ltr task-utilizing listwise cross-entropy softmax loss, our system not only aims to elevate the relevance of generated summaries but also to introduce a nuanced understanding and representation of information importance, which can aid the summarization system at attending to the source content given their relative importance. in short, our contributions are threefold: ## related work the field of query-focused summarization (qfs) @xcite has evolved significantly over the years, transitioning from early unsupervised extractive models @xcite @xcite @xcite to recent approaches leveraging transformer-based models @xcite @xcite . this evolution has been marked by the introduction of various techniques aimed at improving the relevance of summaries. passage retrieval techniques @xcite @xcite @xcite , transfer learning from the qa task @xcite @xcite , query modeling @xcite @xcite , segment encoding @xcite , and attention mechanisms to capture query-utterance relations @xcite have all played a pivotal role in this advancement. furthermore, the adoption of question-driven pretraining @xcite and contrastive learning @xcite has introduced new dimensions to the task, simplifying the identification and summarization of salient content. more recently, @xcite introduced the ranker-generator framework, which ranks utterances by learning from pairwise comparisons and global ordering. the top-ranked utterances are then used as input for the generator in summary generation. while these methods have advanced qfs, existing techniques tend to treat all content segments equally without explicitly considering their relative importance within long-input qfs tasks, which require processing large amounts of text and identifying key segments. hence, the comprehensive modeling of segment importance within the long qfs task remains a less explored area. our approach introduces a novel application of learningto-rank (ltr) @xcite mechanism to address this challenge, drawing inspiration from the successful application of ltr in broader information retrieval contexts @xcite . ## background: segment summarizer (segenc) the backbone of current state-of-the-art systems for query-focused long summarization are built upon the segment encoding (segenc) approach @xcite . segenc operates by encoding fixed-length, overlapping segments of the source text, which are then integrated into a cohesive summary in an end-to-end manner, leveraging the decoder's ability to simultaneously attend to all encoded segments. to specifically adapt to query-focused summarization framework, segenc embeds the query within each segment of the source text. this is achieved through a particular input framing technique, where the query is encapsulated by special markers and placed adjacent to each segment, adhering to the format: <@xmath0. this incorporation of the query into the summarization process is designed to enhance the focus on the query, offering a tailored approach to generating query-focused summaries. ## model: ltr-assisted summarization this study introduces an extension to the segenc summarizer by integrating the learning-to-rank (ltr) principles, a notable ranking technique from the realm of information retrieval. this integration enables the summarizer to effectively learn the ranking of the gold segments. the segments' relevance labels are determined using a span probability-based heuristic (details in section 5.1) during the preprocessing step. an auxiliary ltr task is then formulated to instruct the summarizer in ranking source segments while performing the summarization task. figure 1 shows the overview of our proposed system. in particular, we exploit a shared decoder to perform two tasks including summarization and learning-to-rank. this shared decoder operates by executing two forward passes, one for each task. for the ltr task, following encoding of each segment (denoted as enc(s i )), dec ltr takes in the segment encoder representations (i.e., the encoder representations associated @xmath1, followed by an ltr head (i.e., a feed-forward neural network (ffnn)) that is applied to the decoder outputs: having obtained the ranking outputs for all segments with the above procedure, a listwise softmax cross-entropy function is used to compute the softmax loss as follows: in which ℓ generation is a cross-entropy loss computed for generation task, and λ is a balancing parameter that should be tuned. ## experimental results in this section, we present the automatic and human study results, followed by relevant analyses over query type impact, and segment retrieval. ## error analysis two sources of underperformance were identified in response to our rq4: imbalanced labels. we discovered that in approximately 48% of the underperformed cases, the model exhibited a tendency to misidentify gold segments when generating summaries. upon further investigation, we observed that these cases were commonly characterized by a label imbalance issue, wherein the number of gold segments was significantly smaller than non-gold segments. in such cases, the model selected segments that contained partially relevant information but were not the actual gold segments. as shown in the example within ## conclusion our method combines learning-to-rank with longinput qfs, ensuring content relevance via prioritization. the experimental results demonstrated that our proposed method matches or exceeds sota at reduced training costs. human evaluations highlight improved relevance and faithfulness without compromising fluency. further analysis suggests that the system outperforms on broad queries while lagging on specific ones, with errors linked to imbalanced labels and segment summarizer challenges.