IRS um: One Model to Rule Summarization and Retrieval (2025)

Abstract: Applications that store a large number of documents often have summarization and retrieval functionalities to help users digest large amounts of information efficiently. Currently, such systems need to run two task-specific models, for summarization and retrieval, redundantly on the same set of documents. An efficient approach to amend this redundancy would be to reuse hidden representations produced during the summary generation for retrieval. However, our experiment shows that existing models, including recent large language models, do not produce retrieval-friendly embeddings during summarization due to a lack of a contrastive objective during their training. To this end, we introduce a simple, cost-effective training strategy which integrates a contrastive objective into standard summarization training without requiring additional annotations. We empirically show that our model can perform on par or even outperform in some cases compared to the combination of two task-specific models while improving throughput and FLOPs by up to 17% and 20%, respectively.

## introduction an increase in textual information has been observed in various domains, posing challenges in content discovery and driving extensive efforts in the development of summarization and information retrieval systems. the former aims to produce a shorter version of a given document which encapsulates its essential information @xcite , and in the context of the latter, a number of text encoders have been introduced which output document embeddings that can match the query embedding to retrieve relevant documents @xcite @xcite . while the output format from each approach differs, i.e., a summarization model generates a text and a text encoder produces a vector, due to the shared motivation, systems with a large number of documents often apply these two models to the same set of documents. for instance, papersearching platforms apply both summarization and encoder models to their collection of scientific @xcite or news documents @xcite . however, with existing methods, such systems need to run two models for each document -one for summarizing and one for encoding. this is an inefficient and expensive process, especially with the current trend of increasing model sizes @xcite . one possible solution for this issue would be a model that generates a summary as well as a text embedding for the retrieval of an input document at the same time. however, regardless of its practical value, there is no work that targets this setup. to fill this gap, we define a new task in which a single model needs to solve summarization and retrieval within the same forward pass, dubbed irsum. in irsum, a model must produce hidden representations suitable for retrieval during the summary generation, as summarized in figure 1 . in order to evaluate the effectiveness of our approach, we extend three existing summarization datasets to enable retrieval evaluation using the same set of documents. using these newly constructed datasets, we benchmark a pre-trained language model (plm), t5, introduced by @xcite , as well as two large language models (llms), namely llama 2 7b @xcite and mistral 7b @xcite . while these models produce high-quality summaries, retrieval performance achieved by the embeddings obtained during the summary generation is well below par with reference baselines, calling for additional learning to unlock the retrieval ability of these models' embeddings. to this end, we propose a simple multitask training strategy that combines a contrastive objective with a summarization objective. our method only requires standard summarization datasets for training, and only a small change is needed for its implementation. our experimental results show that our approach retains both summarization and retrieval abilities close to the combination of two specialized models. our model can achieve 90% performance for each task while requiring 20% fewer flops and can process 17% more documents per second compared to the existing approach. our contributions are as follows. (1) we define a new task, irsum, that evaluates a model's ability to produce a summary and embedding for retrieval with only one forward pass, coupled with extensions of three datasets to achieve its evaluation. (2) we benchmark strong baseline models, including llm-based summarization models and show that, in contrast to their high-performing summarization ability, their text embeddings are far from being satisfactory for retrieval. (3) we propose a simple and efficient multitask training strategy and show our model achieves comparable performance to the two specialized models with various efficiency improvements. ## irsum in this section, we first formalize the evaluation of irsum, then describe how we extend existing summarization datasets for its operationalization, and finally benchmark existing models. ## multitask model for irsum previously, we showed that even llm-based summarization models fail at the retrieval part of irsum. now, we propose a novel multitask train-ing strategy where a model optimizes for summarization and contrastive objectives simultaneously. we design our training strategy following two principles. (1) only requiring summarization datasets for training: our method does not require any additional annotations other than pairs of source documents and reference summaries from standard summarization datasets. (2) simple training: our method is a simple add-on to the standard finetuning for summarization without complex additional implementation. ## conclusion in this paper, we first define a new multi-object task setup which asks a model to summarize and encode a document for retrieval within a single forward pass. we extend three existing summarization datasets so that we can use the same set of documents to evaluate on the two tasks. by using them, we find that existing summarization models based on a plm and recent llms cannot achieve satisfactory performance in this setup. given this result, we propose a new multitask training strategy which cheaply integrates a contrastive objective into the standard summarization training loop and show that our models often achieve performance comparable to a combination of two specialized models or even sometimes outperform them while being much more computationally efficient.