WSL - DS : Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization (2020)

Abstract: In the Query Focused Multi-Document Summarization (QF-MDS) task, a set of documents and a query are given where the goal is to generate a summary from these documents based on the given query. However, one major challenge for this task is the lack of availability of labeled training datasets. To overcome this issue, in this paper, we propose a novel weakly supervised learning approach via utilizing distant supervision. In particular, we use datasets similar to the target dataset as the training data where we leverage pre-trained sentence similarity models to generate the weak reference summary of each individual document in a document set from the multi-document gold reference summaries. Then, we iteratively train our summarization model on each single-document to alleviate the computational complexity issue that occurs while training neural summarization models in multiple documents (i.e., long sequences) at once. Experimental results on the Document Understanding Conferences (DUC) datasets show that our proposed approach sets a new state-of-the-art result in terms of various evaluation metrics.

## introduction with the rapid growth of textual documents on the internet, accessing information from the web has become a challenging issue @xcite . often users want the summary of a topic from various sources to fulfill their information needs @xcite . the qf-mds task deals with such problems where the goal is to summarize a set of documents to answer a given query. in the qf-mds task, the summaries generated by the summarizer can be either extractive or abstractive @xcite ). an extractive summarizer extracts relevant text spans from the source document(s), whereas an abstractive summarizer generates a summary in natural language that may contain some words which did not appear in the source document(s) @xcite @xcite . with the rising popularity of virtual assistants in recent years, there is a growing interest to integrate abstractive summarization capabilities in these systems for natural response generation @xcite . one major challenge for the qf-mds task is that the datasets used for such tasks do not contain any labeled training data @xcite . therefore, neural summarization models that leverage supervised training cannot be used in these datasets. note that for other related tasks @xcite @xcite , how to reduce the demands for labeling the data and how to leverage unlabeled data were also identified as a major challenge. while using datasets similar to the target dataset as the training data for the qf-mds task, we find that these datasets only contain multi-document gold summaries. however, the state-of-the-art transformer-based @xcite summarization models @xcite cannot be used in long documents due to computational complexities @xcite . to tackle these issues, we propose a novel weakly supervised approach by utilizing distant supervision to generate weak reference summary of each single-document from multi-document gold reference summaries. we train our model on each document with weak supervision and find that our proposed approach that generates abstractive summaries is very effective for the qf-mds task. more concretely, we make the following contributions: early work on multi-document summarization was mostly focused on generic summarization @xcite , whereas the amount of work for qf-mds had been very limited @xcite . due to the lack of training data for the qf-mds task, most previous works were based on various unsupervised approaches that could only generate extractive summaries @xcite @xcite @xcite @xcite @xcite . to generate the abstractive summaries for the qf-mds task, @xcite proposed a transfer learning technique to tackle the issue of no training data. they adopted the pointer generation network (pgn) @xcite pre-trained for the generic abstractive summarization task in a large dataset to predict the query focused summaries in the target dataset via modifying the attention mechanism of the pgn model. however, their model failed to outperform different extractive approaches in terms of various rouge scores @xcite . identifying sentences which are relevant to the query is an important step for the qf-mds task. for this purpose, various approaches were utilized such as counting word overlaps @xcite or the cross-entropy method @xcite . though neural models based on supervised training have significantly outperformed various non-neural models for the answer selection task in recent years @xcite , such neural models have not been effectively used for the qf-mds task yet due to the absence of labeled data for the relevant sentences in the qf-mds datasets. recently, @xcite showed that neural models pre-trained in a large question answering (qa) dataset could effectively select answers in other qa datasets. more recently, such pre-trained answer selection models were used by @xcite for the qf-mds task. in their work, they utilized distant supervision from various qa datasets using the fine-tuned bert @xcite model to filter out the irrelevant sentences from the documents. however, @xcite showed that filtering sentences as an early step could lead to performance deterioration for the qf-mds task. thus, instead of applying distant supervision to filter out some sentences from the document, we apply it to generate the weak reference summary of each unlabeled document in our training datasets. our proposed weakly supervised learning approach not only allows us to leverage the advantage of finetuning pre-trained generic summarization models @xcite , but also allows us to overcome the limitation of training neural models in long documents @xcite . ## our proposed approach suppose, we have a query @xmath0, q 2 , ..., q k containing k words and a set of n documents @xmath1, d 2 , ..., d n . for the qf-mds task, the goal is to generate a summary @xmath2, s 2 , ...s n containing n words from the document set d for the given query q. in figure 1, we show the overall architecture of our proposed approach. since there is no training data available for the qf-mds task, we provide supervised training to our target dataset by using other qf-mds datasets as the training data. however, the available qf-mds datasets @xcite only contain the gold summaries generated by human experts from multiple documents and do not contain the gold summary of each individual document. due to the limitations of using neural models in long documents @xcite , we propose an iterative approach to train our model on each document of a document set. for this purpose, we generate the weak reference summary of each document from the multi-document gold summaries using distant supervision to train our model for the qf-mds task. finally, we rank the generated query focused summaries via an answer selection model @xcite . in the following, we give a detailed description of our proposed approach. ## experimental setup we now describe the datasets used in this paper, followed by the details of our implementation. datasets: we use the @xcite @xcite datasets for the qf-mds task. the number of document sets were 50, 50, and 45 where the number of documents in each document set were @xcite and 2007 datasets respectively @xcite . each document set is associated with a query and the objective is to generate a summary containing at most 250 words from the document set based on the given query. given the absence of the training data, to evaluate our model in each year's dataset we use the datasets from the other two years for training. from each year's training data, we randomly selected 20% of the document sets for validation while we used the rest for training. implementation: for the roberta model, we used its large version @xcite and implemented using huggingface's transformer @xcite . for fine-tuning the summarization model, we used the bertsum ext-abs foot_2 model pre-trained on the cnn/dailymail dataset @xcite . while selecting the most relevant sentences as the final query focused summary, we used the trigram blocking to reduce redundancy @xcite . to fine-tune the bertsum model, we kept most parameters similar to the original work @xcite and ran 50 steps with batch size equal to 200. among these 50 steps, we selected the step for evaluation that performed the best on the validation set. all of our models were run in multi-gpu settings using 4 nvidia v100 gpus. we report the results based on both recall and f1 scores in terms of rouge-1, rouge-2, and rouge-su4 metrics @xcite . from now on, we denote rouge as r. model recall f1 statistically significant pqsum wsl-ds 42.73 42.84 without distant supervision 41.77 (-2.25%) 41.88 (-2.24%) no, based on paired t-test (@xmath3.05) without trigram blocking 40.92 (-4.24%) 41.01 (-4.27%) no, based on paired t-test (@xmath4.05) without weakly supervised learning 40.01 (-6.37%) 40.12 (-6.35%) yes, based on paired t-test (@xmath5.05) ## results and discussions we now analyze the performance of our proposed model by comparing with other models (see ## conclusions and future work in this paper, we propose a novel weakly supervised approach for the query focused multi-document abstractive summarization task to tackle the issue of no available labeled training data for such tasks. we also propose an iterative approach to address the computational problem that occurs while training neural models in long documents @xcite @xcite . experimental results on three datasets show that our proposed approach sets a new state-of-the-art result in various evaluation metrics. in the future, we will apply our models on more tasks, such as information retrieval applications @xcite @xcite , sentiment analysis @xcite , learning from imbalanced or unlabeled datasets @xcite @xcite , and automatic chart question answering @xcite .