Automatic Generation of Abstracts for Research Papers (2022)

Abstract: Summarizing has always been an important utility for reading long documents. Research papers are unique in this regard, as they have a compulsory summary in the form of the abstract in the beginning of the document which gives the gist of the entire study often within a set upper limit for the word count. Writing the abstract to be sufficiently succinct while being descriptive enough is a hard task even for native English speakers. This study is the first step in generating abstracts for research papers in the computational linguistics domain automatically using the domain-specific abstractive summarization power of the GPT-Neo model.

## introduction the abstract of a research paper provides a quick summery of the entire paper: from the problem to the proposed solution to the result. thus by definition, this section is expected to be concise and informative @xcite . text summarization is one of the main domains in natural language processing (nlp) which has numerous use cases. there are two broad categories for this: extraction and abstraction. in extractive methods it uses existing words, phrases or sentences to form a summary. in contrast, abstractive methods follow a more complex mechanisms. first, a semanatic representation of the content is built. then natural language generation mechanisms are used to create the summary using the aforementioned representation. this research proposes a hybrid mechanism of text summarization to generate the abstract scientific papers with evaluating several paths for the proposed solution. the objective of this research is to reduce the burden on researchers by automatically generating the abstract section by using the sections of the paper that follows it. the researchers then may do minor adjustments to the generated section and publish. considering existing summarization techniques, abstractive solutions have domain specific limitations. on the other hand, domain specific implementations perform better in the perspective of precise representation of the subject matter. abstractive solutions gain domain specificity from the process of models being built upon and information extracted from the training documents. despite the loss of generalization, this improves the accuracy of the solution within the selected domain. thus, we propose to build and test our solution for research paper abstract generation with the scope limited to the domain of computational linguistics. as future work, it may then be extended to other research domains. ## methodology in this section, we discuss the data set generation as well as the methods used for comparative analysis. ## results separately fine tuned gpt-neo models were evaluated for each pre-summerizer as shown in it was then decided to analyse the configurations given in the legal domain by samarawickrama et al. (2020). even though there is no definitive meta-study on the content in the research papers of the computational linguistics domain to conclude such a bias towards precision or recall, it was deemed prudent to report these values. for the ease of reading and comparison, the f1 values of table are also brought forward. average vector method takes the average of encoded vectors of the chunks divided from the text of the paper before passing it into gtp-neo for training or predicting. while average vector model seems to be too trivial for this task at a glance, recent prior work in the nlp domain have proved its usefulness at establishing a baseline for even complex tasks such as sentiment analysis @xcite . results of this method are shown in since the advent of pagerank algorithm @xcite , using graph-based methods to rank text documents has been a popular solution for document level analysis @xcite . textrank @xcite is also a graph based sentence extraction method which creates a graph for each sentence and rank the 34th conference on computational linguistics and speech processing (rocling 2022) taipei, taiwan, november 21-22, 2022. the association for computational linguistics and chinese language processing them based on the similarity. in their legal document retrieval system, @xcite showed how textrank can be utilized in representing documents in a semantically consistent manner. pre-summarization based on this method has scored as shown in the lsa (latent semantic analysis) @xcite ) method is extracting and representing the contextual-usage meaning of words by statistical computations applied to the text. we have calculated the rouge scores of this method as a pre-summarizer with gpt-neo and the results are shown in luhn algorithm @xcite calculates the significance of a sentence by considering frequency of word occurrence in the text and the relative position within a sentence. gpt-neo model trained luhn algorithm as a pre-summarizer gave the results shown in lsa based pre-summarization method has been scored the highest on rouge-1 and rouge-l while luhn based pre-summarization method is scoring higher on rouge-2. all extractive presummarizations has been scored more than the twice of the score of the baseline, vector average method, in rouge-2. ## conclusion we have used transfer learning with gpt-neo for generating abstracts of research papers automatically. gpt-neo model provides a language model that can be utilized for many tasks but we have to face the token limitation. we managed this limited token size with two main approaches which are, an average-pooling of the body context vectors and an extractive summarization. observations have shown that extractive pre-summarization with gpt-neo has better results compared to average pooling. we intend to extend the findings to generate the introduction as well.