Comparing Edge-based and Node-based Methods on a Citation Prediction Task (2024)

Abstract: Citation Prediction, estimating whether paper a cites paper b, is particularly interesting in a forecasting setting where the model is trained on papers published before time t, and evaluated on papers published after h, where h is the forecast horizon. Performance improves with t (larger training sets) and degrades with h (longer forecast horizons). The trade-off between edge-based methods and node-based methods depends on t. Because edges grow faster than nodes, larger training sets favor edge-based methods.We introduce a new forecast-based Citation Prediction benchmark of 3 million papers to quantify these trends.Our benchmark shows that desirable policies for combining edge- and node-based methods depend on h and t.We release our benchmark, evaluation scripts, and embeddings.

## introduction citation prediction is the task of predicting whether a given paper cites a target paper (färber and jatowt, 2020). imagine the scenario where an author is writing a paper and is open to suggestions about what to cite. consider a recommender system which will suggest papers in semantic scholar (s2) @xcite (ammar et al., 2018), a collection of 200 million academic papers from many fields. @xcite recommendations can be based on whatever is available in the input draft, including both text and references. in order to make progress toward this ambitious goal, we introduce a new citation prediction task with an emphasis on the time dimension, and evaluate both a node-based model and an edge-based model on this task. the node-based model focuses 1 https://www.semanticscholar.org/product/api 2 medicine (45m), chemistry (13m), computer science (13m), biology (13m), materials science (10m), engineering (8m), physics (7m), psychology (7m), mathematics (5m), political science (4m), business (4m), sociology (3m), geography (3m), economics (3m), environmental science (3m), geology (3m), history (2m), art (2m), philosophy (1m). 1 . train split in green, test in blue. the bottom plot with the train-test cut-off in 2010 gives a temporally consistent split. on titles and abstracts, and the edge-based model focuses on citations. these models have not been previously compared with one another on graphs of different sizes, especially in a forecasting scenario. standard benchmarks such as open graph benchmark (ogb) @xcite and scirepeval (singh et al., 2022b) evaluate models such as graph neural networks (gnns) @xcite (scarselli et al., 2009; zhou et al., 2018; wu et al., 2019) and specter (cohan et al., 2020) on various academic document modelling tasks including citation prediction. unfortunately, most benchmarks are too small to see the region where edge-based methods overtake node-based methods. we expect citations (edges) to outperform text (nodes) when the graph is large enough because of network effects. these scaling and forecasting issues are important for citation tasks because the literature is growing exponentially, doubling every 9 years @xcite (wade, 2022; kinney et al., 2023). this growth rate is shown in figure 2 for a collection of more than 200 million papers in semantic scholar (s2). the scaling properties mentioned above are somewhat similar to metcalfe's law (metcalfe, 2013). metcalfe's law applies when benefits scale with edges (n 2 ) and costs scale with vertices (n). in a telephone network, costs scale with subscribers (n), and benefits scale with connections between subscribers (n 2 ). these network effects are often cited @xcite for the success of businesses such as telephones (at&t), web search (google) and social media (facebook). our contributions are as follows: following the demonstration of the effectiveness edge-based methods in this paper, we implemented such a recommendation system over semantic scholar. it is available at https:// recommendpapers.xyz . ## related work there is a huge body of work on recommender systems (resnick and varian, 1997). the literature on citations is not as large, though still considerable: ## results and analysis figure 5 shows the performance of prone-s 50 (labeled '5') on every tenth evaluation bin. accuracy is better over the training set (left of the verticle line). accuracy suddenly decreases moving into the first forecasting bin (@xmath0. these results confirm our forecast dynamics assumption, that as time into the future increases, the model's performance degrades. we plot the specter @xcite (labeled 's') for comparison. despite having more recent training data, specter underperforms prone-s @xcite . figure 6 is like figure 5 , but figure 6 shows performance of three prone-s models, trained on prone-s @xcite (blue/4), prone-s (50) (black/5) and prone-s @xcite (red/6), respectively. note that the red line is consistently above the black line, and the black line is consistently above the blue line (prone-s (60) > prone-s (50) > prone-s @xcite ) because training on more bins is better than training on fewer bins. figures 5 6 show that accuracy is better when tested on the training set, and declines the more we predict into the future. figures 5-6 are based on for every additional training bin, accuracy improves by 0.0009 (coefficient: 0.0009, t-value: 12.280, p-value: <0.0001). this supports the use of very large training graphs. appendix b shows the full results for all cumulative 100 prone-s models and results on each 100 evaluation bins. ## conclusion we produced a link-prediction benchmark based on semantic scholar (s2) to measure forecasting capability of document models at scale. we then evaluated 100 prone-s models and specter on data binned by time. forecasting is important because science evolves over time. we found: our results confirmed our central thesis: the time-evolving nature of science requires that academic papers be modelled in a forecasting setting. metcalfe's law suggests edge-based methods such as prone-s should be relatively effective for larger graphs. we found that was correct, with a cross-over point around 82 million papers, much larger than the training set for specter. since prone is transductive (and does not generate embeddings for novel papers), we introduced the centroid assumption, so prone-s can generate embeddings for papers that cite known papers. this extended version of prone-s performed well on our forecasting benchmark, especially when trained on larger graphs. we can interpret the forecasting horizon as a statement about how quickly prone embeddings go stale. semantic scholar creates specter embeddings as papers are ingested. this is good practice because the text does not change after publication. it often takes time for the community to decide what is important. this means that for link-based methods like prone-s, citations will accrue on impactful papers over time, and therefore it is important to update prone embeddings frequently. whilst this may be compute-expensive, our tests show that it leads to valuable increases in representational accuracy. we further investigated the relationship of text and context models through an analysis of model combination strategies. first, we found that simply using specter as a fallback for missing values in prone-s boosts long-horizon predictions significantly. secondly, through evaluation over 100 cumulative prone-s models, we found the optimal model combination policy for combining specter and prone-s depends on forecast horizon, h. in practical applications, h ≫ 0, since we typi-cally train models once, and then use them much later for inference. retraining more often will reduce h, and improve predictions at inference time. if we plan to combine models such as specter and prone-s, the combination should be reevaluated more often since combinations also depend on h. finally, we will make our benchmark, embeddings and scripts available for further research and experimentation. bob metcalfe. 2013. metcalfe's law after 40 years of ethernet. computer, 46(12):26-31. david mimno and andrew mccallum. 2007. expertise modeling for matching papers with reviewers. in proceedings of the 13th acm sigkdd international conference on knowledge discovery and data mining, pages 500-509. malte ostendorff, nils rethmeier, isabelle augenstein, bela gipp, and georg rehm. 2022. neighborhood contrastive learning for scientific document representations with citation embeddings. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 11670-11688, abu dhabi, united arab emirates. association for computational linguistics. reshma s pillai and lr deepthi. 2022. a survey on citation recommendation system. in 2022 third international conference on intelligent computing instrumentation and control technologies (icicict), pages 423-429. ieee. paul resnick and hal r varian. 1997. recommender systems. communications of the acm, 40(3):56-58. dwaipayan roy. 2017. an improved test collection and baselines for bibliographic citation recommendation. in proceedings of the 2017 acm on conference on information and knowledge management, pages 2271-2274. f. scarselli, m. gori, a. tsoi, m. hagenbuchner, and g. monfardini. 2009. the graph neural network model. ieee transactions on neural networks, 20:61-80. amanpreet singh, mike d ' arcy, arman cohan, doug downey, and sergey feldman. 2022a. scirepeval: a multi-format benchmark for scientific document representations. amanpreet singh, mike d'arcy, arman cohan, doug downey, and sergey feldman. 2022b. scirepeval: a multi-format benchmark for scientific document representations. arxiv, abs/2211.13308. laura steinert. 2017. beyond similarity and accuracya new take on automating scientific paper recommendations. pages 1-161. lynn a streeter and karen e lochbaum. 1988. who knows: a system based on automatic representation of semantic structure. in user-oriented content-based text and image handling, pages 380-388. trevor strohman, w. bruce croft, and david d. jensen. 2007. recommending citations for academic papers. pages 705-706. kazunari sugiyama and min-yen kan. 2013. exploiting potential citation papers in scholarly paper recommendation. pages 153-162. yuancheng tu, nikhil johri, dan roth, and julia hockenmaier. 2010. citation author topic model in expert search. in coling 2010: posters, pages 1265-1273, beijing, china. coling 2010 organizing committee. francisco j. valverde-albacete and carmen peláez-moreno. 2014. 100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox. plos one, 9(1):1-10. alex d wade. 2022. the semantic scholar academic graph (s2ag). companion proceedings of the web conference 2022. zonghan wu, shirui pan, fengwen chen, guodong long, chengqi zhang, and philip s. yu. 2019. a comprehensive survey on graph neural networks. ieee transactions on neural networks and learning systems, 32:4-24. david yarowsky and radu florian. 1999. taking the load off the conference chairs-towards a digital paperrouting assistant. in 1999 joint sigdat conference on empirical methods in natural language processing and very large corpora. michihiro yasunaga, jure leskovec, and percy liang. 2022. linkbert: pretraining language models with document links. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 8003-8016, dublin, ireland. association for computational linguistics. dawit yimam-seid and alfred kobsa. 2003. expertfinding systems for organizations: problem and domain analysis and the demoir approach. journal of organizational computing and electronic commerce, 13(1):1-24. jie zhang, yuxiao dong, yan wang, jie tang, and ming ding. 2019. prone: fast and scalable network representation learning. in ijcai, volume 19, pages 4278-4284. yu zhang, yan-jun shen, xiusi chen, bowen jin, and jiawei han. 2023. "why should i review this paper?" unifying semantic, topic, and citation factors for paper-reviewer matching. arxiv, abs/2310.14483. jie zhou, ganqu cui, zhengyan zhang, cheng yang, zhiyuan liu, and maosong sun. 2018. graph neural networks: a review of methods and applications. arxiv, abs/1812.08434. jiang zhuoren, z. xiaozhong liu, and liangcai gao. 2014. dynamic topic/citation influence modeling for chronological citation recommendation. international conference on information and knowledge management, proceedings, 2014-november:15-18.