Citations are ubiquitous in evaluating research, but how exactly they relate to what they are thought to measure (quality and intellectual impact) is unclear . We investigate the relationships between citations , quality, and impact using a survey with an embedded experiment in which 12,670 authors in 15 academic fieldsdescribe about 25K specific referencing decisions. Results suggest that citation counts, when equated with quality and impact, are biased in opposite directions. First, experimentally exposing papers' actual citation counts during the survey causes respondents to perceive all but the top 10\% cited papers as of lower quality. Because perceptions of quality are a key factor in citing decisions, citation counts are likely to endogenously cause more citing of top papers and equating them with quality overestimates the actual quality of those papers. Conversely, 54\% of references had either zero or minor influence on authors who cite them, but references to highly citedpapers were about 200\% more likely to denote substantial impact. Equating citations with impact thus underestimates the impact of highly cited papers . Real citation practices thus reveal that citations are biased measures of quality and impact .
Researchers cite works for a variety of reasons, including some having nothing to do with acknowledging influence. The distribution of different citation types in the literature, and which papers attract which types, is poorly understood . We investigate high-influence and low-influence citations and the mechanisms producing them using 17,154 ground-truth citation types provided via survey by 9,380 authors systematically sampled across academic fields. Overall, 54\% of citations denote little-to-no influence and these citations are concentrated among low status (lightly cited) papers. In contrast, high-influence citations are concentrated among high status (highly cited) papers through a number of steps that resemble a pipeline. Authors discover highly cited papers earlier in their projects, more often through social contacts, and read them more closely. Papers' status, above and beyond any quality differences, directly helps determine their pipeline: experimentally revealing or hiding citation counts during the survey shows that low counts cause lowered perceptions of quality . Accounting for citation types thus reveals a "double status effect": in addition to affecting how often a work is cited, status affects how meaningfully it is cited. Consequently, highly cited papers are even more influential than their raw citation counts suggest .
R
Citations are ubiquitous in evaluating research, but how exactly they relate to what they are thought to measure (quality and intellectual impact) is unclear
Researchers cite works for a variety of reasons, including some having nothing to do with acknowledging influence. The distribution of different citation types in the literature, and which papers attract which types, is poorly understood
R
the relationships between citations , quality, and impact using a survey with an embedded experiment in which 12,670 authors in 15 academic fieldsdescribe about 25K specific referencing decisions. Results suggest that citation counts, when equated with quality and impact, are biased in opposite directions. First, experimentally exposing papers' actual
high-influence and low-influence citations and the mechanisms producing them using 17,154 ground-truth citation types provided via survey by 9,380 authors systematically sampled across academic fields. Overall, 54\% of citations denote little-to-no influence and these citations are concentrated among low status (lightly cited) papers. In contrast, high-influence citations are concentrated among high status (highly cited) papers through a number of steps that resemble a pipeline. Authors discover highly cited papers earlier in their projects, more often through social contacts, and read them more closely. Papers' status, above and beyond any quality differences, directly helps determine their pipeline: experimentally revealing or hiding
R
causes respondents to perceive all but the top 10\% cited papers as of lower quality. Because
shows that low counts cause lowered
R
are a key factor in citing decisions, citation counts are likely to endogenously cause more citing of top papers and equating them with quality overestimates the actual quality of those papers. Conversely, 54\% of references had either zero or minor influence on authors who cite them, but references to highly citedpapers were about 200\% more likely to denote substantial impact. Equating citations with impact thus underestimates the impact of
. Accounting for citation types thus reveals a "double status effect": in addition to affecting how often a work is cited, status affects how meaningfully it is cited. Consequently,
R
. Real citation practices thus reveal that citations are biased measures of quality and impact
are even more influential than their raw citation counts suggest