Non-Parametric Memory Guidance for Multi-Document Summarization (2023)

Abstract: Multi-document summarization (MDS) is a difficult task in Natural Language Processing, aiming to summarize information from several documents. However, the source documents are often insufficient to obtain a qualitative summary. We propose a retriever-guided model combined with non-parametric memory for summary generation. This model retrieves relevant candidates from a database and then generates the summary considering the candidates with a copy mechanism and the source documents. The retriever is implemented with Approximate Nearest Neighbor Search (ANN) to search large databases. Our method is evaluated on the MultiXScience dataset which includes scientific articles. Finally, we discuss our results and possible directions for future work.

## introduction multi-document summarization is performed using two methods: extractive @xcite or abstractive @xcite . so-called extractive methods rank sentences from source documents that best summarize them. these methods reuse important information well to construct a good summary but they lack coherence between sentences. to overcome this issue, abstractive methods are studied to imitate human writing behavior. they show great performance in human writing style but they often miss key information. to make abstractive models aware of essential information, @xcite guides their model with additional information like a set of keywords, graph triples, highlighted sentences of source documents, or retrieved similar summaries. their method, which uses every guidance previously mentioned, improves summary quality and controllability compared with unguided models. however, guidances require specific training data, especially for keywords, graph triples, and highlighted sentences. our proposal is that by guiding with pre-existing summaries, the model can draw inspiration from the summary as a whole. but also be able to extract keywords and phrases using a copy mechanism. consequently, this work focuses on guidance by similar summaries extracted from a knowledge base using a similarity metric between source documents and pre-existing summaries. the model, inspired by rag @xcite , is fully differentiable. in addition, the model generator uses a copy mechanism on the candidates returned from the knowledge base, inspired by @xcite . the findings of these two studies motivated the development of our model for the multi-document text summarization task. we demonstrate the potential of our method on multixscience @xcite . this dataset gathers scientific articles where we have to generate the "related work" part with the "abstract" of the source article and the "abstracts" of the citations. in the case of scientific articles, we believe that the source documents are insufficient to generate the "related work" part because external knowledge is necessary to write such a paragraph. in this work, we investigate a sequence-tosequence model guided by a memory retriever of similar summaries. specifically, source documents are the input of the memory retriever, which returns the top k similar summaries from a potentially large database using an approximate nearest neighbor search. then, the decoder generates the summary taking into account the source and retrieved summaries and is trained to identify interesting texts for the targeted summary. the code of our work is available on github foot_0 . our contribution is twofold: firstly, we integrate a retriever to retrieve candidates for the generation of the summary, and secondly, we make use of a copy mechanism to incorporate these candidates into the generation procedure. ## related work we start with a brief review of related work. @xcite proposes to capture the structure of the document to better represent the information of the source document. their method is applied to scientific articles from arvix and pubmed which are long documents. for the same purpose, @xcite propose to generate a summary from the articles that cite the article to be summarised. the disadvantage of these methods is that they cannot be used when writing an article. in this work, we use references and not the papers that cite the documents to be summarised. more recently, @xcite proposed a pre-training strategy dedicated to multi-document text summarisation, their masking strategy showed significant improvement for the mds task. they applied their method to the multixscience dataset. the models using guidances are close to our work, indeed @xcite use retrieved summaries to better control the summary generation. however, they use information retrieval systems such as elasticsearch to find candidates for summary generation. also, @xcite has introduced dense search systems for text summarization, but they do not train the retriever with the summary generator. in our case, the re-triever is dense and trainable to find the most relevant candidates for the generation of the summary. in addition, retrieval-augmented models share commonalities with our work. rag, @xcite which introduced this type of model, is used for the question-answering task, where a context is given to answer the question. the model retrieves several contexts with a retriever and then answers the question using each of the retrieved candidates. these types of models are also used in the translation task, where @xcite translates a sentence with a pre-established translation base. their model searches this base for translations close to the sentence to be translated and then incorporates them into the generation of the translation through a copy mechanism. this approach shares some similar intuition with our proposed approach because our architecture is based on an augmented retriever that incorporates the memory by means of a copy mechanism. it is interesting to investigate whether the encouraging success of the copy mechanism recently obtained in translation carries over to the mds task. ## proposed method inspired by @xcite , we propose a model composed of a memory retriever and a copy generator. figure 1 illustrates our framework, where we start by encoding the entire knowledge base. after an arbitrary number of steps during the training, the encoded knowledge base is updated. then, the forward pass encodes source documents and finds similar documents. retrieved documents are encoded and fed to the generator with the source documents. our memory retriever has multiple encoders, one for encoding the query, one for the knowledge base, one for the sources documents, and one for the retrieved candidates. our copy generator is a decoder with a cross-attention mechanism on source document embeddings and a copy mechanism on retrieved candidates, which is placed at the top of the decoder. we begin describing the retriever and then show how our generator works. ## experiments in this section, we report on the experiments performed on the multixscience dataset to evaluate our model. training the full model is more difficult due to its size but also due to the cold start problem. the latter corresponds to the fact that the similar summaries retrieved are not sufficiently relevant to help the model. in addition, we have trained two other methods adapted to text summarisation as a comparison, bart @xcite and t5 @xcite . we detail the training procedure for each of them. all models use the beam search method to generate summaries. we chose a beam size of 4, a length penalty of 1.0, and limited the repetition of tri-grams. the rouge scores @xcite on the multixscience dataset are reported in ## conclusion and future work this paper presents an architecture for multidocument text summarization inspired by retrievalaugmented models. this architecture includes a retriever that searches a knowledge base to find relevant documents for the generation of a summary. these documents are integrated in the generation by means of a copy mechanism. a reduced version of the model was evaluated on the multixscience dataset. the preliminary results are already com-petitive with the state of the art however we expect to improve our results further by: 1) properly fixing the cold start problem, and 2) training the full model. in the future, we also plan to increase the size of the knowledge base with new data and apply our method to other mds benchmark datasets.