Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics (2021)

Abstract: Quantitatively measuring the impact-related aspects of scientific, engineering, and technological (SET) innovations is a fundamental problem with broad applications. Traditional citation-based measures for assessing the impact of innovations and related entities do not take into account the content of the publications. This limits their ability to provide rigorous quality-related metrics because they cannot account for the reasons that led to a citation. We present approaches to estimate content-aware bibliometrics to quantitatively measure the scholarly impact of a publication. Our approaches assess the impact of a cited publication by the extent to which the cited publication informs the citing publication. We introduce a new metric, called “Content Informed Index” (CII), that uses the content of the paper as a source of distant-supervision, to quantify how much the cited-node informs the citing-node. We evaluate the weights estimated by our approach on three manually annotated datasets, where the annotations quantify the extent of information in the citation. Particularly, we evaluate how well the ranking imposed by our approach associates with the ranking imposed by the manual annotations. CII achieves up to 103% improvement in performance as compared to the second-best performing approach.

## introduction scientific, engineering, and technological (set) innovations have been the drivers behind many of the significant positive advances in our modern economy, society, and life. to measure various impact-related aspects of these innovations various quantitative metrics have been developed and deployed. these metrics play an important role as they are used to influence how resources are allocated, assess the performance of personnel, identify intellectual property (ip)-related takeover targets, value a company's intangible assets, and identify strategic and/or emerging competitors. citation networks of peered-reviewed scholarly publications (e.g., journal/conference articles and patents) have widely been used and studied in order to derive such metrics for the various entities @xcite ). however, most of these traditional metrics, such as citation counts and h-index, treat all citations and publications equally and do not take into account the content of the publications and the context in which a prior scholarly work was cited. another related line of work, such as pagerank @xcite and hits @xcite considers the node centrality (as a proxy for influence) but still operate in a content-agnostic manner. content-agnostic metrics fail to precisely size up the scholarly impact of an article as they do not differentiate between the possible reasons that a scholarly work is being cited. in addition, they can be easily manipulated by the presence of malicious entities, such as publication venues indulging in self-citations, which leads to high impact factor, or a group of scholars citing each others' work. for example, journal citation reports (jcr) foot_0 routinely suppresses many journals that indulge in citation stacking, a practice where the reviewers and journal editors pressure authors to cite papers that either they wrote or that are published in their journal. thus, there is a need to establish content-aware metrics to accurately measure various innovationrelated aspects such as their significance, novelty, impact, and market value. such metrics are essential for ensuring that set-driven innovations will play an ever more significant role in the future. a straightforward solution to develop contentaware metrics is to manually annotate the citations, where the annotations describe the reasons for the citations. these annotations can then be used to train a machine-learning system that takes the content of the publications as input and predicts the reasons for the citation. along this direction, there has been considerable effort to identify important citations @xcite @xcite . however, generating labeled data for such supervised approaches is difficult and time-consuming, especially when the meaning of the labels is user-defined. in this work, we present approaches to estimate content-aware bibliometrics to quantitatively measure the scholarly impact of a publication. our approaches are distant supervised, that require no manual annotation. the proposed approaches leverage the readily available content of the papers as a source of distant supervision. our approaches assess the impact of a cited publication by the extent to which it informs the citing publication. they automatically estimate the weights of the edges in the citation network, such that higher-weighted edges correspond to higher-impact citations. we use these weights to introduce a new metric, called content informed index (cii). we evaluate cii on three manually annotated datasets, where the annotations tell us the citation importance, thus, quantify the extent of information in the citation. particularly, we evaluate how well the ranking imposed by cii associates with the ranking imposed by the manual annotations. the proposed approach achieves up to 103% improvement in performance as compared to the second-best performing approach. ## related work the research areas relevant to the work present in this paper belong to citation indexing, citation recommendation, link prediction approaches, and distant-supervised credit attribution approaches, and citation-intent classification approaches. we briefly discuss these below: ## content-informed index (cii) to address the disadvantages of content-agnostic bibliometrics, we present approaches that use machine-learning to estimate content-aware bibliometrics to measure the scholarly impact of a publication. our approaches are distant supervised, requiring no manual annotation. they automatically estimate the weights of the edges in the citation network, such that edges with higher weights correspond to higher-impact citations. we use these weights to come up with a new metric, called content informed index (cii). next, we discuss the assumptions behind cii and provide deeper details. ## discussion and conclusions in this paper, we presented approaches to estimate content-aware bibliometrics to accurately quantitatively measure the scholarly impact of a publication. our distant-supervised approaches use the content of the publications to weight the edges of a citation network, where the weights quantify the extent to which the cited-publication informs the citing-publication. experiments on the three manually annotated datasets show the advantage of using the proposed method on the competing approaches. the code is available on github 16 . our work makes a step towards developing content-aware bibliometrics, and envision that the proposed method will serve as a motivation to develop other rigorous quality-related metrics.