Zero-Shot Aspect-Based Scientific Document Summarization using Self-Supervised Pre-training (2022)

Abstract: We study the zero-shot setting for the aspect-based scientific document summarization task. Summarizing scientific documents with respect to an aspect can remarkably improve document assistance systems and readers experience. However, existing large-scale datasets contain a limited variety of aspects, causing summarization models to over-fit to a small set of aspects and a specific domain. We establish baseline results in zero-shot performance (over unseen aspects and the presence of domain shift), paraphrasing, leave-one-out, and limited supervised samples experimental setups. We propose a self-supervised pre-training approach to enhance the zero-shot performance. We leverage the PubMed structured abstracts to create a biomedical aspect-based summarization dataset. Experimental results on the PubMed and FacetSum aspect-based datasets show promising performance when the model is pre-trained using unlabelled in-domain data.

## introduction scientific document summarization aims to summarize research papers, and it is usually considered as generating paper abstracts @xcite . compared to the news summarization datasets like cnn/daily mail @xcite and xsum @xcite , scientific papers are significantly longer, follow a standard structure, and contain more technical terms and complex concepts @xcite . recently, there have been remarkable improvements in the area of scientific document summarization due to the availability of large-scale datasets such as arxiv, pubmed @xcite , and sumpubmed @xcite introduction cancer is the second leading cause of mortality and morbidity in both developed .... ## methods in this section, we first present how we formulate the aspect-based summarization problem relying on bart pre-trained model. then, we propose a method to use unlabelled data for an additional self-supervised pre-training step to improve the zero-shot performance. ## datasets for our experiments, we consider facetsum, an aspect-based summarization benchmark built on emerald articles. in addition, we process pubmed and convert into a large aspect-based scientific document summarization dataset. we scraped the pubmed website to collect the structured abstracts corresponding to the papers in the pubmed summarization dataset. we match papers to their web-page using their article id. we use beautiful-soup library 3 and leverage the html structure of abstracts on their web-page to extract five aspects: introduction, objectives, methods, results, and conclusion. we manually checked the aspects and their summary and set rules to convert different spellings and typos (e.g., @xmath0, @xmath1. for papers text and sections, we stick to the pubmed dataset. for self-supervised pre-training we create two self-supervised datasets: pubmed ⋆ and facetsum ⋆ , from pubmed and facetsum aspect-based summarization datasets as described in section 3.2. pubmed ⋆ and facetsum ⋆ contain 658k and 279k samples and 150k and 96k unique aspects, respectively. additional dataset pubmed ⋆ -nooverlap and bart @xcite 45.49 18.10 42.74 bart-facet @xcite 49.29 19.60 45.76 bart 49.98 19.89 46.68 facetsum greedy extractive (oracle) 51.87 32.09 41.55 bart @xcite 23.27 10.31 20.29 bart-facet @xcite facetsum ⋆ -nooverlap are the variants in which we exclude aspects that overlap with the main aspects (shown by red in figure 2 ). we only exclude aspects containing the main aspects but not semantically equivalent words. these datasets would allow assessing to what extent the model can perform semantic connection with new aspects. ## experiments and results in this section, we first explain model hyperparameters. then, we assess models' ability to make a semantic connection between aspects and summaries in different experimental setups and understand to what extend pre-training helps. we rely on bart base available through hug-gingface's transformers library @xcite . it is trained for each dataset we tackle. fine-tuning is done on 1 gpu (nvidia v100), with a batch size of 64 (8 gradient accumulation steps). we train the model for 10 epochs (2 epochs for selfsupervised pre-training) with a learning rate of 3e-4 and 500 warm-up steps and set the maximum input length to 1024, the bart official length (see appendix a for a full list of hyper-parameters). ## conclusion in this paper, we studied the problem of zeroshot aspect-based summarization of scientific documents. we established various experimental setups to investigate the effect of additional pre-training and intermediate training on the zero-shot performance with respect to domain shift from biomedical to management and unseen aspects. we proposed a self-supervised approach to pre-train the model using unlabelled target datasets. results indicate that additional pre-training on the target dataset followed by intermediate training results in the best zero-shot performance. we established leave-one-out and paraphrasing experimental setups to simulate the practical case of facing unseen aspects and showed the promising effect of additional self-supervised pre-training. our proposed pre-training step improves the performance in the few-shot settings. investigating the effect of pre-training in terms of semantics and factuality evaluation scores can be done in the future.