SC hu BERT : Scholarly Document Chunks with BERT -encoding boost Citation Count Prediction. (2020)

Abstract: Predicting the number of citations of scholarly documents is an upcoming task in scholarly document processing. Besides the intrinsic merit of this information, it also has a wider use as an imperfect proxy for quality which has the advantage of being cheaply available for large volumes of scholarly documents. Previous work has dealt with number of citations prediction with relatively small training data sets, or larger datasets but with short, incomplete input text. In this work we leverage the open access ACL Anthology collection in combination with the Semantic Scholar bibliometric database to create a large corpus of scholarly documents with associated citation information and we propose a new citation prediction model called SChuBERT. In our experiments we compare SChuBERT with several state-of-the-art citation prediction models and show that it outperforms previous methods by a large margin. We also show the merit of using more training data and longer input for number of citations prediction.

## introduction predicting the quality of scientific articles is a novel task in the field of deep learning. there are many indicators of quality such as whether a paper was accepted or rejected, meta-information such as the author's h-index(es), and the number of citations. the number of citations, while not a perfect indicator of quality, is available for any paper which makes it suitable for constructing a large dataset. in this work we propose acl-bibliometry, a new dataset consisting of 30000 papers with citation information. we also test several state-of-the deep learning models and propose a new model called schubert which outperforms all other methods. using the full text of scholarly documents has the potential to substantially improve the performance of the citation count prediction task. but prohibitive memory costs of applying advanced deep learning models on the full text can be a roadblock. in particular, bert @xcite and its variants have been very successful as building blocks for state-of-the-art natural language processing models for many tasks. citation count prediction for scholarly documents is a task where bert has clear potential as well. however, scholarly documents are particularly long texts in general. since bert has a time complexity that is quadratic with respect to the input length, it is limited to 512 tokens by default, a limit which can not be increased by much without causing prohibitive computational cost. recent models including the reformer @xcite and longformer @xcite have sought to overcome the quadratic computational cost of the transformer model @xcite underlying bert. while these models are very promising, they do not offer the unsupervised pre-training on large amounts of data that makes bert so powerful as of yet. although in principle these models could be applied as a dropin replacement for bert, it requires more research to show if and how unsupervised pre-training as done in bert can be made to work well with very long context. for these reasons, in this work we use bert as our base building block and find effective ways to overcome its input length limit, leaving experimentation with the aforementioned models for future research. for dealing with large amounts of training examples containing very long input text we need an approach that: 1) is able to fit the encoding of the long text into memory, 2) can efficiently process the large amount of training examples when training over many epochs. both requirements can be fulfilled by chunking the long input text of our examples into parts, and pre-computing bert embeddings for each of these parts using a pre-trained bert model. the core of the final model is a sequence-model, in particular a gated recurrent unit (gru) @xcite , which directly uses the pre-computed chunk embeddings as inputs. this approach simultaneously overcomes the memory problems associated with dealing with very long input texts, as well as achieves high computational efficiency by performing the expensive step of computing bert embeddings for chunks only once. while the task of citation count prediction using the contents of a scholarly document is not new, and goes back at least to the work of @xcite , work up until now has been limited in: a) the size of the training data, b) the size of the input text. the rest of the paper is organized as follows: in section 2 we discuss related work, in section 3 we describe the models used for citation count prediction, in section 4 we discuss the dataset construction, in section 5 we present our experiments, in section 6 we show our results and in section 7 we end with conclusions. ## related work recently, multiple datasets have been released which are useful for the scientific quality prediction problem. the s2orc dataset @xcite has abstract information for 81.1m papers and full-text for 8.1m papers, both with citations. other large datasets exist such as unarxive and pubmed central open access subset, but these datasets span various domains. given the difficulty of the citation prediction task, we made a new dataset for just the computational linguistics and natural language processing domain, to be used as a benchmark for citation prediction models. the peerread dataset @xcite is another useful dataset that has accept/reject decisions for 14.7k full-text papers. this is a useful dataset on which more research has been performed @xcite , but the amount of papers in it is fairly limited. for this reason, we propose our new dataset which contains full-text and citations for a large number of papers. a number of methods have been proposed for the citation prediction problem. @xcite try to predict future citations of a paper by using web usage statistics, e.g. the number of times the paper was downloaded. (abrishami and aliakbary, 2019) use deep learning techniques to predict longterm citations using short-term citations. @xcite use a measure called paper potential index (ppi) which is based on a combination of features such as the impact of the authors and early citations. the problem with these methods is that information such as short-term citations and web usage statistics are only available after the paper is published. furthermore, these methods disregard any of the actual papers' content. because of this reason, our work focuses on predicting the citations using only the textual content. limited research is available on this topic. one of the first papers which focused on predicting citation count by only using information available at publication is by @xcite . they use the paper title, abstract and keywords as well as bibliometric information as input data for an svm. they then predict a binary label (positive or negative) based on whether the paper received at least a set number of citations within 10 years. this work was expanded upon by @xcite . they predict a discrete value (few, some or many citations) using multiple classification models, outperforming the baseline set by (fu and aliferis, 2008) using both naive bayes as well as logistic regression. both papers use a fairly small dataset (3788 papers for (fu and aliferis, 2008) and 2246 papers for @xcite ). ## models in this section we briefly describe our two baseline models: bilstm and hierarchical attention networks (hans). this is followed by a description of the bert-based schubert model, to the best of our knowledge first applied to the task of citation count prediction in this work. ## dataset construction citation count prediction relies on sufficiently large labeled data, of good quality and preferably with full document text. to obtain such data, we need: 1. a large set of good quality scholarly documents, preferably in the same domain, or a way to collect such a set from the internet. ## experiments in our experiments, we want to assess the usability of the acl data for number of citations prediction, and generally larger training data for citation prediction, as enabled by the automatic number of citation labeling framework contributed in this work. we also want to test two hypotheses: 1. longer input text improves performance: using then entire paper text (title + abstract + body text) is substantially better than using only the paper title + abstract. to test our second hypothesis, we take 10% of our dataset to get a dataset which is approximately the size of peerread cl ( 3000 papers). we then compare this to half our data and full data to show the importance of larger datasets. lastly, we also test schubert on a portion of the chunks to ensure a fair comparison with bilstm and han which were capped at 20k characters. statistics about the number of characters per example in the different settings are shown in ## results our results show that schubert is able to outperform both bilstm as well as han for the citation prediction problem by a significant margin. in as a final comparison of the systems, we show the number of trainable parameters in ## conclusion in this work, we showed the importance of larger and better curated data for the citation prediction problem. we proposed acl-bibliometry, a new large dataset created with the algorithms we provide in this work. we also proposed schubert, a new model for the citation prediction problem which can deal with large inputs and gets significantly better results than several state-of-the-art models. the model shows the strength of modern language models and contextualized word embeddings and their appliance to the citation prediction problem. our results showed that both the length of the input as well as the amount of data are important for achieving better results. the current work takes a step forward by using a larger training set of full text examples and leveraging this data with stronger models, in particular the schu-bert model, without considering the historical publication context and other factors. we leave experimentation with further extended context for the predictive models, as well as other language models and even larger datasets for future work.