L ong S umm 2021: Session based automatic summarization model for scientific document (2021)

Abstract: Most summarization task focuses on generating relatively short summaries. Such a length constraint might not be appropriate when summarizing scientific work. The LongSumm task needs participants generate long summary for scientific document. This task usual can be solved by language model. But an important problem is that model like BERT is limit to memory, and can not deal with a long input like a document. Also generate a long output is hard. In this paper, we propose a session based automatic summarization model(SBAS) which using a session and ensemble mechanism to generate long summary. And our model achieves the best performance in the LongSumm task.

## introduction most of the document summarization tasks focus on generate a short summary that keeps the core idea of the original document. for long scientific papers, a short abstract is not long enough to cover all the salient information. researchers often summarize scientific articles by writing a blog, which requires specialized knowledge and a deep understanding of the scientific domain. the longsumm, a shared task of sdp 2021( https://sdproc.org/2021/ sharedtasks.html ), opts to leverage blog posts created by researchers that summarize scientific articles and extractive summaries based on video talks from associated conferences @xcite to address the problem mentioned above. most of the previous methods divide the document according to section, and use the extraction or abstraction model to predict the summary for each part respectively, and combine the results as the final summary of the document. section based method may drop some important information among the sections. generally, only uses one type of model for prediction can not make good use of the advantages of different models. combined with the later models and solutions, we propose an ensemble method based on session like figure1. we split the task into four steps: session generation, extraction, abstraction, and merging the results at the end. first, we split an document into several sessions with a certain size, and use a rouge metric to match the ground truth (sentences from given document's summary). then, we train two different types of model. one is the abstraction-based model. specifically, we use the bigbird @xcite , a sparse attention mechanism that reduces this quadratic dependency to linear, and pegasus @xcite , a pretrained model specially designed for summarization. the other one is based on extraction method. we test the performance of textrank @xcite , dgcnn(dilate gated convolutional neural network) @xcite and bertsumm @xcite . in the end, for each type of model, we generate the summary from the one which has the best performance, and use an ensemble method to merge the summaries together. the result show that our method is effective and beats the state-of-art models in this task. ## related work the common automatic summarization is mainly divided into the extraction-based summarization and the abstraction-based summarization. the extraction-based model extracts several sentences and words from the original article by the semantic analysis and sentence importance analysis to form the abstract of the article. typical models include textrank @xcite algorithm which based on sentence importance and the extraction method based on pre-training model @xcite . the abstracts obtained by the extraction model can better reflect the focus of the article, but because the extracted sentences are scattered in different parts of the article, the coherence of the abstracts is a problem to be challenged. the abstraction-based models are based on the structure of seq2seq, and the pretraining model is used to achieve better generation effect like bart @xcite , t5 @xcite . recently, pegasus @xcite , a pre-training model released by google, specially designed the pre-training mode for the summarization task, and achieved the state-of-art performance on all 12 downstream datasets. this task focuses on the the solution of the long summary. the input and ouput text of the traditional model is limited due to the memory and time-consuming. however, this task requires the model to summarize scientific papers and generate very long summaries. to solve this problem, most of the solutions in the the previous are based on sections @xcite . they divide scientific papers into sections, generate abstracts for each seciton, and finally combine them to get the final results. resently, google's new model bigbird @xcite , using sparse attention mechanism to enable the model fit long text, is suitable for this task scenario. ## method the pre-training model plays a significant role in the field of automatic summarization, but due to its huge amount of parameters, most of the models can only be used for short text tasks. for long articles, there are two common ways to do. one is to directly truncate the long articles, the other is to predict the articles according to the section. this paper proposes a text segmentation method based on session, and use an ensemble method with the extraction model and the abstraction model to generate the final summary. ## experiment we extract the text from the pdf of paper by using science parse( https://github.com/ allenai/science-parse ). there is a lot of dirty text in the data, which will make the model hard to converge during training. so we clean the text as follows: (1) replace the url link in the text with @xcite (2) remove special characters from the text and keep only some common symbols. (3) merge the broken words and remove some words that is not in the word list. we spilt the text of each paper into sessions,and the best session size by testing should be 1024 words. the buffer size is 128 which we think is enought to keep the context. each sentence of ground truth is set as the target summary of one of the sessions according to the location of the most similar sentence in the original paper. we use the nltk to count words of the session. as for pretrained model, all input session are truncated to a maximum of 1024 words, and their target summary are truncated to a maximum of 128 words. based on the test results, the best generation model is built as follows: the model is fine-tuned on the pegasus-arxiv pre-trained model released by google which has about 570 million parameters for 20 epochs with a learning rate of 2e-5. the batch size is 8 and the model is trained on four v100(32g) gpus for about 20 hours. as for building dgcnn-extraction model, all input papers are truncated to a maximum of 400 sentences(1024d) and 7 dgcnn-layers (with 1,2,4,8,16,1,1 dilation rate) are added to the model. then we compile the model with adam optimizer(learning @xmath0.001). the model is trained for 20 epochs on training set and the batch size is set to 32. dgcnn is a lightweight model that only takes 30 minutes to train. follow the method mentioned above, we ensemble the summaries obtained from the best generation model and extraction model. ## result we test three different models on the test set: (1)sbas extract : the model only include the dgcnn-extraction model for summary. the result show that both sbas abstract and sbas extract model are competitive. as for the result of sbas abstract , its recall-score is much lower than f1-score, this might be caused by the summary generated by sbas abstract is shorter than the ground truth. we limit the length of summary extracted by sbas extract to 900 words, and get an excellent result compared with other teams. the result of sbas ensemble is far superior to the others models, we believe this is because our ensemble method not only remove the redundant sentences in the combined summary, but also make the output of sbas extract well supplement for the result of sbas abstract . we extract some of the abstract for manual evaluation, and find that the abstract generated by our method can generate sentences with high readability and cover a lot of important information of the paper, but sentence to sentence is not coherent, the fluency of the abstract is insufficient. and we will try to improve the fluency of the summary in future work. ## conclusion pre-train models such as bert and gpt have obvious effects in all nlp fields, but they can't deal with long text due to their huge amount of parameters and computation. in this paper, we propose an ensemble model based on session for the long-summ task. in our method, the document is firstly segmented according to the session, and some context semantics are reserved. then, the labels corresponding to each session are matched by a specific algorithm to generate a new dataset. the extraction and abstraction models are trained on the new dataset, and the final summary is obtained by merging the results of different models through the ensemble method. the method proposed in this paper considers the context of the text as much as possible while limiting the memory growth, so that the summary predicted by the model is more coherent. and the method of merging two different types of summary models is proposed for the first time. the prediction results of different models are dropped and combined for the second time, so as to make the results closer to the real summary. our model has achieved the best performance in all metrics of this task, but there for improvement. the current approach is to compress the input and output to make the task adapt to the model, but the best design idea is to make the model fit the task. one of the biggest problems is how to reduce the resource consumption of the transformers structure model. bigbird model proposed by google alleviates this problem through sparse attention mechanism, but after our test, because of the decoding part of the model still uses full attention, bigbird does not solve the problem of long text output, and it is difficult to directly generate a complete long summary from scientific documents in this task. therefore, future research can focus on how to decode longer text, so that the language model can adapt to more nlp scenarios.