CIST @ CL - S ci S umm 2020, L ong S umm 2020: Automatic Scientific Document Summarization (2020)

Abstract: Our system participates in two shared tasks, CL-SciSumm 2020 and LongSumm 2020. In the CL-SciSumm shared task, based on our previous work, we apply more machine learning methods on position features and content features for facet classification in Task1B. And GCN is introduced in Task2 to perform extractive summarization. In the LongSumm shared task, we integrate both the extractive and abstractive summarization ways. Three methods were tested which are T5 Fine-tuning, DPPs Sampling, and GRU-GCN/GAT.

## introduction the increasing scientific documents published on the internet allow researchers to find more and more documents of interest. however, how to quickly and efficiently obtain the most important facts or ideas of a document is a big challenge. summarization of scientific documents can mitigate this issue by presenting a brief summary of the whole document to researchers. this year, we participate in two shared tasks of sdp 2020 @xcite . the cl-scisumm shared task is the first medium-scale shared task on scientific document summarization in the field of computational linguistics and aims to generate a structured summary for the rp (reference paper) with the utilization of 10 or more cps (citing papers). the longsumm shared task opted to leverage blogs created by researchers in the nlp (natural language processing) and machine learning communities and use these summaries as reference summaries to generate the abstractive and extractive summaries for scientific papers. in this paper, we will introduce our methods, experiments, and results of two shared tasks. for the cl-scisumm shared task, based on our previous work @xcite , we continue to leverage similarity calculation on multiple features to perform citation linkage in task1a. in task1b, we first extract position features and content features of rt (reference text) and ct (citation text), then apply different machine learning methods to classify the facet. in task2, we apply dpps (determinantal point processes) and gcn (graph convolutional network) to perform extractive summarization this time. as for the longsumm shared task, we retain those extractive methods in the task2 of the cl-scisumm shared task as the basis for our summarization system. furthermore, we also introduce the gat (graph attention network) and apply an abstractive summarization method based on finetuning. ## related work the task1a of cl-scisumm is a citation linkage task. the most intuitive method is to calculate and compare the similarity between the cts (citation text spans) and every text span in rp (reference paper), and select the rt with the highest similarity as the result. there are many ways to calculate the similarity, not only traditional idf and jaccard similarity, but also levenshtein distance @xcite . the basic characteristics of words often play an important role in similarity calculating. as the size of the data set continues to grow, neural network language models such as word2vec (goldberg and levy, 2014) and bert @xcite that contain the semantic similarity information in word-level can make a huge improvement. but these word embedding methods will gradually smooth the difference between keywords in the process of calculating, so wmd @xcite was proposed to pay attention to the feature mapping between words. in addition to improvement in feature extraction, researchers have also proposed many new algorithms to process features, such as introducing cnn @xcite (dos santos and gatti, 2014) into the nlp field to make more complex judgments on feature vectors, or using matchpyramid @xcite to process the similarity comparison focusing on the similarity between words. task1b of cl-scisumm is essentially a classification task. classification methods are mainly divided into two parts: rule-based methods and supervised machine learning methods. traditional supervised machine learning methods like lr (logistic regression) @xcite , adaboost @xcite and xgboost (chen and guestrin, 2016) can be easily applied for this task. besides, the neural networks, such as textcnn @xcite , textrnn @xcite , textrcnn @xcite , fasttext @xcite and char-cnn @xcite , can work directly on text, and generate dense vectors for classification. the task2 of cl-scisumm and the longsumm shared task are both summarization task. recently, the research on automatic summarization tasks has mainly focused on two ways: extractive summarization and abstractive summarization. in the field of extractive summarization, we studied the sampling process used in dpps @xcite where we calculated the kernel matrix using wmd sentence similarity for further sampling @xcite . @xcite explored how to make the system generate higher quality summaries. they selected three metrics: network architecture, knowledge transfer, and learning mode, and analyzed the impact of the three metrics on the quality of summary generation through experiments. gcn is a powerful neural network framework processing graph structural data. @xcite extended the traditional cnn to non-euclidean space and introduce local spectral filtering to optimize the propagation process during the training of the standard graph neural network. @xcite further studied the application of gcn in semi-supervised classification. gat @xcite allocates different weights on different node neighbors to aggregate information. a document can also be converted into a graph. @xcite introduced gcn in multi-document summarization. the clusters of documents were fed into rnn to obtain intermediate representations. then gcn continued to extracting features considering the connections of documents clusters. at last, each sentence was scored based on its cluster-aware representations, and sentences with high score were chosen as summaries. as for abstractive summarization, @xcite introduced an attention mechanism to the seq2seq model, which enables the model to focus on words in specific positions in the original text via the weight matrix when generating abstracts, thus avoiding the problem of losing too much information due to long sentences. since bert @xcite has achieved great success in the field of nlp, the method of pre-training and fine-tuning has become a new paradigm. researchers began to explore how to apply pre-trained models to natural language generation. at first, researchers tried to replace the encoder with a pre-trained bert @xcite , then more and more pre-training target functions for the seq2seq model were explored like masked generation @xcite , denoising @xcite ), text-to-text @xcite . some specially designed tasks for summarization have also been proposed, such as extracting gap-sentences @xcite . we use the gap-sentence method in @xcite to combine and transform all the data, then utilize the t5 model @xcite to fine-tune and generate the summary. as shown in figure 1 , the citation linkage task, task1a of cl-scisumm, contains two steps: feature extraction and content linkage. in the feature extraction step, we perform similarity calculation based on different feature extraction ways for each rt and every ct (citation text) in cts (citation text spans), where some traditional features will be used, such as idf similarity and jaccard similarity. additionally sentence context information is used on the basis of these simple features in order to more comprehensively reflect the similarity information of the sentence. besides, we also use the lin and jcn features of wordnet, word-cos, word vector, and lda-jaccard @xcite . lda-jaccard performs better than lda on sparse topics, and it pays more attention to the union set of the same topic that both two sentences have. in the content linkage step, we add all the scores that each ct belonging to the same cts, then sort all rts by the final scores, and take the first n results as the final answer of task1a. we use four multi-feature fusion methods: voting-1.2, voting-2.1, jaccard-focused, and jaccard-cascade based on our last year work @xcite by increasing the training set and adjusting the hyper-parameters. ## conclusion and future work in the cl-scisumm shared task, jaccard-focused performs better than other methods in task1a. in future work, we will try to use the knowledge graph and gnn for better expression of semantic and structure information. in task1b, pos-xgb performs the best, which shows that the position fea-tures contributes more than the content features. in the future, more information can be extracted and fused to obtain richer features, or combined with some hand-craft rules to assist the classification. in task 2, gcn shows great potential to perform the summarization task. we expect the neural network language models to make contributions to obtain more meaningful semantic representation for sentences against statistical features. in the longsumm shared task, model t5 and extractive summarization model based on gcn perform well on the official data set, and dpps still has great potential, we expect to provide more features or modify the sampling processes so as to improve the performance of our models. what's more, in this paper we mainly focus on how to extract/generate section-wise summaries with high quality and diversity, but how to pick and combine these summaries is also an interesting work to be done.