Towards Generating Query to Perform Query Focused Abstractive Summarization using Pre-trained Model (2020)

Abstract: Query Focused Abstractive Summarization (QFAS) represents an abstractive summary from the source document based on a given query. To measure the performance of abstractive summarization tasks, different datasets have been broadly used. However, for QFAS tasks, only a limited number of datasets have been used, which are comparatively small and provide single sentence summaries. This paper presents a query generation approach, where we considered most similar words between documents and summaries for generating queries. By implementing our query generation approach, we prepared two relatively large datasets, namely CNN/DailyMail and Newsroom which contain multiple sentence summaries and can be used for future QFAS tasks. We also implemented a pre-processing approach to perform QFAS tasks using a pretrained language model, BERTSUM. In our pre-processing approach, we sorted the sentences of the documents from the most query-related sentences to the less query-related sentences. Then, we fine-tuned the BERTSUM model for generating the abstractive summaries. We also experimented on one of the largely used datasets, Debatepedia, to compare our QFAS approach with other models. The experimental results show that our approach outperforms the state-of-the-art models on three ROUGE scores.

## introduction text summarization has two major types: extractive summarization and abstractive summarization. the extractive summarization approach only selects important sentences for generating extractive summaries and may lose the main context of the documents. in contrast, the abstractive summarization approach considers all the sentences of the document to hold the actual context of the document and paraphrase sentences to generate ab-stractive summaries. query focused abstractive summarization (qfas) emphasizes those sentences relevant to the given query and generates abstractive summaries based on the query. for example, a user may need to know the summary of the tourist places located in vancouver rather than all the tourist places of the entire canada. then the qfas approach will focus on the query keywords 'tourist places''vancouver'and generate an abstractive summary. with the advancement of the neural network, modern approaches of text summarization have focused on abstractive summarization, which paraphrases the words in the sentences by using encoder-decoder architecture @xcite @xcite @xcite . with the rnn encoder-decoder model, @xcite introduced a dataset for chinese text summarization. to solve the problem of recurring words in encoderdecoder models, @xcite have given an attention model to minimize the repetition of the same words and phrases. in other works, the transformer model has been used to get better summaries @xcite . a pretrained language model, bidirectional encoder representations from transformers (bert) @xcite model can combine the word and sentence representations in a single substantial transformer @xcite , which can be fine-tuned for next sentence prediction tasks. recently, the bert has been used on the bertsum model for summarization tasks and showed state-of-the-art results @xcite . however, all these research works were focused only on generating better abstractive summaries and did not consider the relevance of query for abstractive summarization. query focused summarization highlights those sentences which are relevant to the context of a given query. still, only few works have been done on qfas @xcite @xcite . here, @xcite , aryal and chali (2020) independently used debatepedia foot_0 @xcite dataset and hasselqvist et al. (2017) used cnn/dailymail foot_1 (hermann et al., 2015) dataset for qfas tasks. debatepedia dataset is a small dataset which consists of single sentence summaries. hence, we intended to investigate whether relatively large datasets with multiple sentence summaries perform better on qfas tasks. we prepared and used two large datasets; cnn/dailymail and newsroom foot_2 @xcite for our qfas task, which have multiple sentence summaries. using cnn/dailymail dataset, @xcite generated queries for qfas task and conducted their research experiments. in their query generation approach, the authors considered only summaries and did not focus on the relevant documents which may have an impact on the performance of their proposed model. therefore, we developed our new query generation approach, considering the relevant documents and summaries for our qfas task. in our qfas approach, we emphasized on the input representation and implemented our idea of sorting the sentences of the documents according to the corresponding queries. then we used our pre-processed input to fine-tune the bert-sum model for generating abstractive summaries. for cnn/dailymail our approach achieved better rouge scores than the work of @xcite . as there is no previous work which performs qfas tasks on newsroom dataset, we present our results for future research comparison. we also implemented our query generation and qfas approaches on debatepedia dataset and found that our approaches work well on debatepedia dataset in comparison with the existing qfas based research works. ## related work the research work of @xcite implemented the attention model for both queries and documents on debatepedia dataset. their model succeeded in solving the problem of repeating phrases in summaries. they proposed a model with two key addition to the encode attend decode ## dataset preparation in this work, we used three datasets; cnn/dailymail, newsroom and debatepedia for our experiments. for our qfas task, we prepared these three datasets with our new query generation approach. the cnn/dailymail dataset comprises of 287k news articles with 3-4 lines related highlights. in our work, we collected the stories as the text documents and the highlights as the corresponding summaries. in this way, the summaries contain more than one sentence which made the dataset more useful for the pretrained language models. the newsroom dataset has been developed from 38 major news publications. the authors collected words and phrases from articles to generate summaries by combining the abstractive and extractive approaches. in newsroom, there are three types of datasets: abstractive, extractive, and mixed. for our qfas task, we used abstractive and mixed datasets of newsroom where we eliminated those data which had single sentence summaries. the debatepedia dataset corpus has 663 debates under 53 categories. though the dataset contains single sentence summaries, some qfas models used this dataset for their experiments. therefore, we experimented our query generation and qfas approaches using the debatepedia dataset to compare whether our new approach outperforms the state-of-the art result or not. ## our summarization framework we used the source document and query as the system input and generated summary as system output. our summarization framework has two parts, at first we pre-processed the source document according to the query by which we incorporated the query relevance to our qfas task. then, we used the bertsum model to generate abstractive summaries, where we fine-tuned the model with our pre-processed source documents. our summarization approach is shown in figure 2 . ## experimental setups we implemented our query generation and qfas approaches on cnn/dailymail, newsroom and debatepedia datasets using the same experimental setup. ## result ## conclusion in this research, one of our aim was to incorporate query and prepare two datasets which contain multiple sentence summaries for qfas task. our another goal was to pre-process the source documents with a new document sorting approach and then fed to the pretrained model. we targeted to fine-tune the bertsum model for our qfas task. we compared our results and investigated that our qfas approach successfully achieved new state-ofthe-art results for debatepedia and cnn/dailymail datasets. as no previous research used newsroom dataset for qfas task, we provided our results of newsroom dataset for future comparison of the related research work.