C ausal C ite: A Causal Formulation of Paper Citations (2024)

Abstract: Citation count of a paper is a commonly used proxy for evaluating the significance of a paper in the scientific community. Yet citation measures are widely criticized for failing to accurately reflect the true impact of a paper. Thus, we propose CausalCite, a new way to measure the significance of a paper by assessing the causal impact of the paper on its follow-up papers. CausalCite is based on a novel causal inference method, TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings. TextMatch encodes each paper using text embeddings from large language models (LLMs), extracts similar samples by cosine similarity, and synthesizes a counterfactual sample as the weighted average of similar papers according to their similarity values. We demonstrate the effectiveness of CausalCite on various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, (test-of-time) awards for past papers, and its stability across various subfields of AI. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of the quality of a paper. Our code is available at https://github.com/causalNLP/causal-cite.

## introduction recent years have seen explosive growth in the number of scientific publications, making it increasingly challenging for scientists to navigate the vast landscape of scientific literature. therefore, identifying a good paper has become a crucial challenge for the scientific community, not only for technical research purposes, but also for making decisions, such as funding allocation @xcite , research evaluation @xcite , recruitment @xcite , and university ranking and evaluation @xcite . a traditional approach to recognize paper quality is peer review, a mechanism that requires large efforts, what would the counterfactual success metric y' be? and yet has inherent randomness and flaws @xcite @xcite @xcite . moreover, the number of papers after peer review is still overwhelmingly large for researchers to read, leaving the challenge of identifying truly impactful research unaddressed. another commonly used metric is citations. however, this metric faces criticism for biases, such as a preference for survey, toolkit, and dataset papers @xcite . together with altmetrics @xcite , which incorporates social media attention to a paper, both metrics also bias towards papers from major publishing countries @xcite , with extensive publicity and promotion, and authored by established figures. to provide a more equitable assessment of paper quality, we employ the causal inference framework @xcite to quantify a paper's impact by how much of the academic success in the follow-up papers should be causally attributed to this paper. we introduce causalcite, an enhanced citation based metric that poses the following counterfactual question (also shown in figure 1 ): "had this paper never been published, what would have happened to its follow-up studies?" to compute the causal attribution of each follow-up paper, we contrast its citations (the treatment group) with citations of papers that address a similar topic, but are not built on the paper of interest (the control group). traditionally, this problem is solved by using the match-ing method @xcite in causal inference, which discretizes the value of the confounder variable, and compares the treatment and control groups with regard to each discretized value of the confounder variable. however, this approach does not apply when the confounder variable is high-dimensional, e.g., text data, such as the content of the paper. thus, we improve the matching method to adapt for textual confounders, by marrying recent advancement of large language models (llms) with traditional causal inference. specifically, we propose textmatch, which uses llms to encode an academic paper as a high-dimensional text embedding to represent the confounders, and then, instead of iterating over discretized values of the confounder, we match each paper in the treatment group with papers from the control group with high cosine similarity by the text embeddings. textmatch makes contributions in three different aspects: (1) it relaxes the previous constraint that the confounder variable should be binned into a limited set of intervals, and makes the matching method applicable for high-dimensional continuous variable type for the confounder; (2) since there are millions of papers, we enable efficient matching via a matching-and-reranking approach, first using information retrieval (ir) @xcite to extract a small set of candidates, and then applying semantic textual similarity (sts) @xcite for fine-grained reranking; and (3) we enable a more stable causal effect estimation by leveraging all the close matches to synthesize the counterfactual citation score by a weighted average according to the similarity scores of the matched papers. causalcite quantifies scientific impact via a causal lens, offering an alternative understanding of a paper's impact within the academic community. to test its effectiveness, we conduct extensive experiments using the semantic scholar corpus @xcite , comprising of 206m papers and 2.4b citation links. we empirically validate causalcite by showing higher predictive accuracy of paper impact (as judged by scientific experts on a past dataset of 1k papers @xcite ) compared to citations and other previous impact assessment metrics. we further show a stronger correlation of the metric with the test-of-time (tot) paper awards. we find that, unlike citation counts, our metric exhibits a greater balance across various research domains in ai, e.g., general ai, nlp, and computer vision (cv). while citation numbers for papers in these domains vary significantly -for example, while an average cv paper has many more citations than an average nlp paper, causalcite scores papers across ai sub-fields more similarly. after demonstrating the desirable properties of our metric, we also present several case studies of its applications. our findings reveal that the quality of conference best papers is noisier on average than that of tot papers (section 5.1). we then showcase and present causal-cite for several well-known papers (section 5.3) and utilize causalcite to identify high-quality papers that are less recognized by citation counts (section 5.4). in conclusion, our contributions are as follows: ## problem formulation our problem formulation involves a citation graph and a causal graph. we use lowercase letters for specific papers and uppercase for an arbitrary paper treated as a random variable. citation graph in the citation graph g := (p, l), p is a set of papers, and each edge ℓ i,@xmath0.e., is cited by) a follow-up paper p j . to obtain the citation graph, we use the semantic scholar academic graph dataset @xcite with 206m papers and 2.4b citation edges. treatment t building paper b on paper a effect y success of paper b what is the causal effect size? confounders x title+abstract incl., topic, research question year treatment t building paper b on paper a effect y success of paper b mediators performance e.g., "90%" venue e.g., "acl" colliders post-hoc award ## the textmatch method as illustrated in figure 1 , the objective of our study is to quantify the causal effect of the treatment t (i.e., whether paper b is built on paper a) on the effect y (i.e., the outcome of paper b). to approach this, we envision a counterfactual scenario: what if paper a had never been published, yet certain key characteristics of paper b remain unchanged? the critical question then becomes: which key characteristics of paper b should be controlled for in this hypothetical situation? ## performance evaluation the contribution of a paper is inherently multidimensional, making it infeasible to encapsulate its richness fully through a scalar. yet the demand for a single, comprehensible metric for research impact persists, fueling the continued use of traditional citations despite their known limitations. in this section, we show how our new metrics significantly improve upon traditional citations by providing quantitative evaluations comparing the effectiveness of citations, semantic scholar's highly influential (sshi) citations @xcite , and our causalcite metric. ## findings having demonstrated the effectiveness of our metrics, we now explore some open-ended questions: (1) do best papers have high causal impact? (section 5.1) (2) how does the causalcite value distribute across papers? (section 5.2) (3) what is the impact of some famous papers evaluated by causalcite? (section 5.3) (4) can we use this metric to correct for citations? (section 5.4). ## related work the quantification of scientific impact has a rich history and continuously evolves with technology. bibliometric analysis has been largely influenced by early methods that relied on citation counts @xcite @xcite . @xcite investigate the evolution of citation analysis, employing reference publication year spectroscopy ## conclusion in this study, we propose causalcite, a novel causal formulation for paper citations. our method combines traditional causal inference methods with the recent advancement of nlp in llms to provide a new causal outlook on paper impact by answering the causal question: "had this paper never been published, what would be the impact on this paper's current follow-up studies?". with extensive experiments and analyses using expert ratings and test-of-time papers as criteria for impact, our new causalcite metric demonstrates clear improvements over the traditional citation metrics. finally, we use this metric to investigate several open-ended questions like "do best papers have high causal impact?", conduct a case study of famous papers, and suggest future usage of our metric for discovering good papers less recognized by citations for the scientific community.