Discriminative Marginalized Probabilistic Neural Method for Multi-Document Summarization of Medical Literature (2022)

Abstract: Although current state-of-the-art Transformer-based solutions succeeded in a wide range for single-document NLP tasks, they still struggle to address multi-input tasks such as multi-document summarization. Many solutions truncate the inputs, thus ignoring potential summary-relevant contents, which is unacceptable in the medical domain where each information can be vital. Others leverage linear model approximations to apply multi-input concatenation, worsening the results because all information is considered, even if it is conflicting or noisy with respect to a shared background. Despite the importance and social impact of medicine, there are no ad-hoc solutions for multi-document summarization. For this reason, we propose a novel discriminative marginalized probabilistic method (DAMEN) trained to discriminate critical information from a cluster of topic-related medical documents and generate a multi-document summary via token probability marginalization. Results prove we outperform the previous state-of-the-art on a biomedical dataset for multi-document summarization of systematic literature reviews. Moreover, we perform extensive ablation studies to motivate the design choices and prove the importance of each module of our method.

## introduction the task of multi-document summarization aims to generate a compact and informative summary from a cluster of topic-related documents, which represents a very challenging natural language processing (nlp) application due to the presence of redundant and sometimes conflicting information among documents @xcite . in the medical domain, in which machine learning plays an increasingly significant role @xcite , multi-document summarization finds application in the generation of systematic literature reviews, a biomedical paper that summarizes results across many studies @xcite . @xcite are the first that address this task, showing the related issues. state-of-the-art approaches leverage two leading solutions: hierarchical networks that capture crossdocument relations via graph encodings @xcite @xcite or hidden states aggregation @xcite @xcite , and long-range neural models that apply multi-input concatenation @xcite . while effective, these solutions struggle to process clusters of many topic-related documents in low computational resource scenarios @xcite because they need to truncate the inputs. moreover, pre-trained state-of-the-art transformers are not leveraged despite showing strong performance when fine-tuned in downstream tasks such as single-document summarization @xcite @xcite @xcite . multi-document summarization requires models to have more robust capabilities for analyzing the cluster to discriminate the correct information from noise and merge it consistently. in this work, we propose a discriminative marginalized probabilistic neural method (damen) that selects worthy documents in the cluster with respect to a shared background and generates the summary via token probability marginalization. the marginalization of the probability has been successfully applied in past nlp models such as plsa @xcite to learn the word probability distribution in documents by maximizing the likelihood. recently, new deep neural models that use probability marginalization approaches have been proposed for the question-answering task, where, however, each input/output sentence is generally several orders of magnitude shorter than sets of documents in multi-document summarization @xcite . to the best of our knowledge, we are the first that propose such a method for multi-document summarization. to this aim, we conduct experiments on the only medical dataset for multi-document summarization of systematic literature reviews (ms2). besides, we perform extensive ablation studies to motivate the design choices and prove the importance of each component of our method. to sum up, our contributions are as follows: ## related work we describe related works on multi-document summarization categorized on model architectures. ## method we introduce damen, a discriminative marginalized probabilistic neural method for the multidocument summarization of medical literature based on three components: in this phase, we index each document in the cluster with an embedding generated by a bert-based model. such a pre-trained language model is the state-of-the-art in semantic modeling from textual data thanks to the vast knowledge learned during pre-training @xcite , achieving groundbreaking results across an extensive range of nlp downstream tasks even without fine-tuning. for this reason, we use it to create a dense latent representation of each document, called document em-bedding, which is a vector of continuous numbers that indicates a point in a latent semantic space. the technique we use is known as dense passage retriever (dpr) @xcite , and it is widely adopted in the information retrieval domain (e.g., @xcite . we choose the dpr method because it does not interrupt the backpropagation, differently from other solutions, e.g., bm25, tf-idf @xcite or lsa @xcite . we formalize this step as b β (c i ) = e, where b is a bert-based model, β represents its parameters, and e is a matrix of shape (len(c i ), 768), where each row j of the matrix is the latent representation of the document c j . ## conclusion we proposed a novel probabilistic method based on the combination of three language models to tackle multi-document summarization in the medical domain. this task is characterized by redundant information, noise, and the possible presence of vital information in each sentence that makes arbitrary input truncation unacceptable. for this reason, we proposed a multi-document summarization method able to discriminate salient contents from irrelevant before summarizing. in detail, the solution first leverages a bert-based model (indexer) for creating dense indices for each chunk of each document in the cluster. then, a second bert-based model (discriminator) is used to process the shared background and select only the most relevant chunks. the final bart model is trained to perform a probability marginalization over each token prediction for each selected chunk. in this way, our solution reads all document information and selects just the most relevant chunks, discarding noise before feeding the generator. the discriminator and generator are trained end-to-end, backpropagating the probability distribution as explained in §3. the indexer is frozen; training would lead to some