Realistic Citation Count Prediction Task for Newly Published Papers (2023)

Abstract: Citation count prediction is the task of predicting the future citation counts of academic papers, which is particularly useful for estimating the future impacts of an ever-growing number of academic papers. Although there have been many studies on citation count prediction, they are not applicable to predicting the citation counts of newly published papers, because they assume the availability of future citation counts for papers that have not had enough time pass since publication. In this paper, we first identify problems in the settings of existing studies and introduce a realistic citation count prediction task that strictly uses information available at the time of a target paper’s publication. For realistic citation count prediction, we then propose two methods to leverage the citation counts of papers shortly after publication. Through experiments using papers collected from arXiv and bioRxiv, we demonstrate that our methods considerably improve the performance of citation count prediction for newly published papers in a realistic setting.

## introduction in recent years, the number of academic papers in various fields has increased drastically. accordingly, the demand for techniques for predicting papers that will become influential in the future is growing to help readers identify those papers and support efficient knowledge acquisition. in this study, we adopt the citation count as a measure of future impact, following several previous studies (e.g., @xcite , and we address the citation count prediction task, which entails predicting how many times a target paper will be cited in the future. there have been many studies on citation count prediction (e.g., @xcite . however, none of those settings is strictly applicable to predicting the citation count of newly published papers, because they assume citation count one year after publication papers published more than 12 months ago papers published 6 months ago papers published 3 months ago target paper figure 1 : comparison of a realistic citation count prediction setting with existing research settings. each bar (■+□) represents the citation count one year after publication, which existing studies assume to be available, while the gray part (■) represents the citation count that is actually available at the time of a target paper's publication. the availability of future citation counts for papers shortly after publication. for example, consider the case of predicting the citation counts one year after publication. for training and testing, the correct citation count of the target paper one year after publication must be known; hence, only papers published more than one year ago are used in the experiments. consequently, even for papers published less than one year before the target paper, the number of citations one year after publication is available, and these citation counts are commonly used to train the prediction model. the bars (■+□) in figure 1 represent the citation count information used in such settings. however, in actually predicting the future citation count of newly published papers, the correct citation counts one year after the publication of papers published less than one year ago are not available; what is actually available is the gray part of each bar (■) in the figure. the unrealistic assumption in previous studies might appear to have a limited impact on the performance of a prediction model. however, information on the future citation counts of recently published papers could cause leakage of research trends in the near future, which turns out to have a non-negligible impact on performance. hence, in this study, we first show that the settings of existing studies leak future information that contributes significantly to the prediction performance. we then introduce a realistic citation count prediction task that strictly uses information available at the time of a target paper's publication. furthermore, we propose two methods to capture research trends in the near future that are applicable even in our realistic setting. the first method is citation count complementation, which uses papers published less than one year ago as training data by estimating the citation count one year after publication from the current citation count. the second method leverages the degree of early adoption by using the property that papers that cite more recent papers and papers that cite more frequently cited papers tend to receive more attention in the future. ## datasets for the experiments here, we used two datasets: a cl dataset, consisting of papers in the field of computational linguistics, and a bio dataset, consisting of papers in the field of biology. to construct the cl dataset, we collected 16,940 papers submitted to arxiv in the computation and language (cs.cl) category foot_0 from june 2014 to june 2020. we considered preprints suitable for this study because they include papers that have not been peer-reviewed and are expected to have a large variance in their future impact. we then obtained the publication dates of papers that cited the collected papers from semantic scholar foot_1 to calculate the citation count for each elapsed month after the publication of each paper in the dataset. we created 13 subsets, each of which consists of papers published in one of the months from june 2019 to june 2020 and papers published in the five years prior to that month. within each subset, the papers published in the latest month were used for evaluation, and the remainder was used for training. for example, one subset consists 3 task formulation ## citation count complementation we propose a method to estimate the citation count one year after the publication of papers that were published less than one year ago. our method uses the citation counts of those papers at the time the target paper was published to estimate the counts one year after they were published. specifically, we estimate the citation counts of a paper m months after publication with a citation count c m by the following two methods: case-based extract all papers in the training set that have a citation counts c m at m months after publication, and use the median of those papers' counts one year after publication as the estimate. ratio-based for the training set, calculate the ratio of the average citation count m months after publication to the average count one year after publication, and multiply it by the citation count c m to obtain the estimate. while case-based estimation is expected to be accurate for less-cited papers, where there are many other papers with the same citation count, it is not suitable for highly-cited papers that have no or few other papers with the same citation count. thus, if the citation count c m is associated with a paper in the list of top 10% papers, it is estimated using the ratio-based method. otherwise, it is estimated using the case-based method. the rank order of c m is calculated from the distribution of citation counts m months after publication for the papers in the training set. to confirm the appropriateness of this citation count complementation, we calculated spearman's rank correlation between the correct citation counts one year after publication against the predicted citation count before and after complementation (c m and complemented citation count). for this investigation, we used the training portion of the 12 subsets to train and evaluate the model on the cl dataset, and we compared the average spearman's rank correlations for each subset. as a result, we found that the correlation improved from 0.88 to 0.92, which demonstrates that the citation count complementation is appropriate. ## degree of early adoption in realistic citation count prediction, the full citation counts of papers published less than one year after publication cannot be used for training, yet papers that are frequently cited in such a short term are likely to be impactful. in addition, papers that top 0-1% 1-2.5% 2.5-5% 5-10% 10-25% 25-100% no citation within 3 months 15.5 (4.6%) 14.3 (3.8%) 10.6 (3.6%) 8.8 (5.0%) 7.5 (6.6%) 6.5 (4.9%) 5.0 (71.5%) within 6 months 14.3 (9.6%) 12.6 (7.3%) 9.8 (6.2%) 7.6 (8.8%) 6.3 (10.7%) 4.6 (9.4%) 3.7 (48.1%) within 9 months 13.8 (15.4%) 11.2 (10.3%) 7.7 (7.8%) 6.6 (10.1%) 5.3 (12.9%) 3.5 (12.4%) 2.5 (31.0%) within 12 months 12.7 (21.6%) 10.1 (12.0%) 6.4 (9.1%) 5.7 (10.5%) 4.4 (13.5%) 2.5 (12.9%) 2.0 (20.6%) specifically, we examined the average citation count one year after publication for those papers that cite at least one paper with the top k 1 % to k 2 % citation counts published within m months. for this investigation, we used 15,962 papers published in arxiv's cs.cl category between june 2015 and may 2020, which form the training portion of the subset described in section 2. in the case of multiple citations of papers published within m months, we used the highest rank order of the citation counts among them. for (k 1 , k 2 ), we used 6 pairs: (0, 1), (1, 2.5), (2.5, 5), (5, 10), (10, 25), and (25, 100). for m, we used four values: 3, 6, 9, and 12. we then calculated the average citation count for each combination of (k 1 , k 2 ) and m. ## experiments we conducted experiments on the datasets described in section 2 to validate the effectiveness of using citation count complementation and the degree of early adoption in realistic citation count prediction. ## related work early works on citation count prediction formulated the task and explored effective features. @xcite formulated citation count pre-diction as a regression problem and used author reputation to predict the citation count. @xcite formulated citation count prediction as a classification problem and investigated several features that are effective for such prediction, including a paper's title, abstract, and author information. other studies have sought to improve the prediction performance by using various features. one such feature is a citation graph constructed from citation relationships among papers. @xcite proposed a method to use the graph's temporal and topological features. @xcite achieved high prediction performance by mining frequent graph patterns. @xcite proposed a method to use the citation context, which is the text in a paper that mentions other cited papers. @xcite found that the interdisciplinarity of authors is effective in predicting citation counts. @xcite proposed a method to use peer-reviewed text from multiple aspects. several studies have focused on aspects other than features. @xcite and te @xcite found several patterns in the growth of citation counts by analyzing a large number of papers, and they proposed a two-step prediction method, first classifying papers into each pattern and then predicting counts for each pattern. @xcite proposed a method to predict the citation count at an arbitrary point in time from the publication of a paper, with the aim of predicting its future potential impact. in recent years, there has been research on the use of deep learning techniques to predict citation counts. @xcite proposed an rnn-based method to predict a paper's future citation count by using the citation counts for each elapsed year since its publication. @xcite proposed a method to predict the citation count by dividing a paper's text into chunks and encoding the paper's entire body with bert. @xcite proposed a method to predict the citation count by extracting semantic features from a paper's title and abstract via doc2vec and bi-lstm with an attention mechanism. lication of a target paper for training. we further proposed two methods to use papers published less than one year after publication for citation count prediction, as these papers cannot be directly used for training because their citation counts one year after publication are unknown. the first method is citation count complementation, which uses recent papers for training by estimating their citation counts one year after publication. the second method is to leverage the degree of early adoption, which incorporates the tendency for papers that cite highly cited papers earlier to have higher average citation counts. through experiments using papers collected from arxiv and biorxiv, we demonstrated that the use of papers published less than one year after publication improves the performance of realistic citation count prediction. for future work, we intend to build models that incorporate information from papers that was not used in this study, such as the body, figures, tables, and author information.