A Sleeping Beauty (SB) in science refers to a paper whose importance is not recognized for several years after publication. Its citation history exhibits a long hibernation period followed by a sudden spike of popularity. Previous studies suggest a relative scarcity of SBs. The reliability of this conclusion is, however, heavily dependent on identification methods based on arbitrary threshold parameters for sleeping time and number of citations, applied to small or monodisciplinary bibliographic datasets. Here we present a systematic, large-scale, and multidisciplinary analysis of the SB phenomenon in science. We introduce a parameter-free measure that quantifies the extent to which a specific paper can be considered an SB. We apply our method to 22 million scientific papers published in all disciplines of natural and social sciences over a time span longer than a century. Our results reveal that the SB phenomenon is not exceptional. There is a continuous spectrum of delayed recognition where both the hibernation period and the awakening intensity are taken into account. Although many cases of SBs can be identified by looking at monodisciplinary bibliographic data, the SB phenomenon becomes much more apparent with the analysis of multidisciplinary datasets, where we can observe many examples of papers achieving delayed yet exceptional importance in disciplines different from those where they were originally published. Our analysis emphasizes a complex feature of citation dynamics that so far has received little attention, and also provides empirical evidence against the use of short-term citation metrics in the quantification of scientific impact. Significance-Scientific papers have typically a finite lifetime: their rate to attract citations achieves its maximum a few years after publication, and then steadily declines. Previous studies pointed out the existence of a few blatant exceptions: papers whose relevance has not been recognized for decades, but then suddenly become highly influential and cited. The Einstein, Podolsky, and Rosen "paradox" paper is an exemplar Sleeping Beauty. We study how common Sleeping Beauties are in science. We introduce a quantity that captures both the recognition intensity and the duration of the "sleeping" period, and show that Sleeping Beauties are far from exceptional. The distribution of such quantity is continuous and has power-law behavior, suggesting a common mechanism behind delayed but intense recognition at all scales. There is an increasing interest in understanding the dynamics underlying scientific production and the evolution of science [1]. Seminal studies focused on scientific collaboration networks [2], evolution of disciplines [3], team science [4][5][6][7], and citation-based scientific impact [8][9][10]. An important issue at the core of many research efforts in science of science is characterizing how papers attract citations during their lifetime. Citations can be regarded as the credit units that the scientific community attributes to its research products. As such, they are at the basis of several quantitative measures aimed at evaluating career trajectories of scholars [11] and research performance of institutions [12, 13] . They are also increasingly used as evaluation criteria in very important contexts, such as hiring, promotion, and tenure, funding decisions, or department and university rankings [14, 15] . Several factors can potentially affect the amount of citations accumulated by a paper over time, including its quality, timeliness, and potential to trigger further inquiries [9], the reputation of its authors [16, 17] , as well as its topic and age [8]. Studies about fundamental mechanisms that drive citation dynamics started already in the 1960s, when de Solla Price introduced the cumulative advantage (CA) model to explain the emergence of power-law citation distributions [18] . CA essentially provisions that the probability of a publication to attract a new citation is proportional to the number of citations it already has. The criterion, now widely referred to as preferential attachment, was 2 recently popularized by Barabási and Albert [19], who proposed it as a general mechanism that yields heterogeneous connectivity patterns in networks describing systems in various domains [20, 21] . Other processes that effectively incorporate the CA mechanism have been proposed to explain power-law citation distributions. Krapivsky and Redner, for example, considered a redirection mechanism, where new papers copy with a certain probability the citations of other papers [22] . An important effect not included in the CA mechanism is the fact that the probability of receiving citations is time dependent. In the CA model, papers continue to acquire citations independently of their age so that, on average, older papers accumulate higher number of citations [19, 22, 23] . However, it has been empirically observed that the rate at which a paper accumulates citations decreases after an initial growth period [24] [25] [26] [27] . Recent studies about growing network models include the aging of nodes as a key feature [24, [27] [28] [29] [30] . More recently, Wang et al. developed a model that includes, in addition to the CA and aging, an intuitive yet fundamental ingredient: a fitness or quality parameter that accounts for the perceived novelty and importance of individual papers [9]. In this work, we focus on the citation history of papers receiving an intense but late recognition. Note that delayed recognition cannot be predicted by current models for citation dynamics. All models, regardless of the number of ingredients used, naturally lead to the socalled first-mover advantage, according to which either papers start to accumulate citations in the early stages of their lifetime or they will never be able to accumulate a significant number of citations [23] . Back in the 1980s, Garfield provided examples of articles with delayed recognition and suggested to use citation data to identify them [31] [32] [33] [34] . Through a broad literature search, Glänzel et al. gave an estimate for the occurrence of delayed recognition, and highlighted a few shared features among lately recognized papers [35] . The coinage of the term "Sleeping Beauty" (SB) in reference to papers with delayed recognition is due to van Raan [36]. He proposed three dimensions along which delayed recognition can be measured: (i ) length of sleep, i.e., the duration of the "sleeping period;" (ii ) depth of sleep, i.e., the average number of citations during the sleeping period; and (iii ) awake intensity, i.e., the number of citations accumulated during 4 years after the sleeping period. By combining these measures, he identified a few SB examples occurred between 1980 and 2000. These seminal studies suffer from two main limitations: (i ) the analyzed datasets are very small, especially if compared to the size of the bibliographic databases currently 3 available; and (ii ) the definition and the consequent identification of SBs are to the same extent arbitrary, and strongly depend on the rules adopted. More recently, Redner analyzed a very large dataset covering 110 years of publications in physics [37] . Redner proposed a definition of revived classic (or SB) for articles satisfying the three following criteria: (i ) publication date antecedent 1961; (ii ) number of citations larger than 250; and (iii ) ratio of the average citation age to publication age greater than 0.7. Whereas Redner was able to overcome the first limitation mentioned above, his study is still affected by an arbitrary selection choice of top SBs, justified by the principle that SBs represent exceptional events in science. In addition, Redner's analysis has the limitation to be field specific, covering only publications and citations within the realm of physics. Here we perform an analysis on the SB phenomenon in science. We propose a parameterfree approach to quantify how much a given paper can be considered as an SB. We call this index "beauty coefficient," denoted as B. By measuring B for tens of millions of publications in multiple scientific disciplines over an observation window longer than a century, we show that B is characterized by a heterogeneous but continuous distribution, with no natural separation between papers with low, high, or even extreme values of B. Also, we demonstrate that the empirical distributions of B cannot be easily reconciled with obvious baseline models for citation accumulation that are based solely on CA or the reshuffling of citations. We introduce a simple method to identify the awakening time of SBs, i.e., the year when their citations burst. The results indicate that many SBs become highly influential more than 50 years after their publication, far longer than typical time windows for measuring citation impact, corroborating recent studies on understanding the use of short time windows to approximate long-term citations [38] [39] [40] . We further show that the majority of papers exhibit a sudden decay of popularity after reaching the maximum number of yearly citations, independently of their B values. Our study points out that the SB phenomenon has two important multidisciplinary components. First, particular disciplines, such as physics, chemistry, and mathematics, are able to produce top SBs at higher rates than other scientific fields. Second, top SBs achieve delayed exceptional importance in disciplines different from those where they were originally published. Based on these results, we believe that our study may pave the way to the identification of the complex dynamics that trigger the awakening mechanisms, shedding light on highly cited papers that follow nontraditional popularity trajectories. The beauty coefficient value B for a given paper is based on the comparison between its citation history and a reference line that is determined only by its publication year, the maximum number of citations received in a year (within a multi-year observation period), and the year when such maximum is achieved. Given a paper, let us define c t as the number of citations received in the t-th year after its publication; t indicates the age of the paper. Let us also assume that our index B is measured at time t = T , and that the paper receives its maximum number c tm of yearly citations at time t m ∈ [0, T ]. Consider the straight line t that connects the points (0, c 0 ) and (t m , c tm ) in the timecitation plane ( Fig. 1 ). This line is described by the equation where (c tm − c 0 ) /t m is the slope of the line, and c 0 the number of citations received by the paper in the year of its publication. For each t ≤ t m , we then compute the ratio between t − c t and max{1, c t }. Summing up the ratios from t = 0 to t = t m , the beauty coefficient B is defined as By definition, B = 0 for papers with t m = 0. Papers with citations growing linearly with time (c t = t ) have B = 0. B is non-positive for papers whose citation trajectory c t is a concave function of time. Our index B has a number of desirable properties: (i ) B can be computed for any paper and does not rely on arbitrary thresholds on the sleeping period or the awakening intensity, paving the way to treat the SB phenomenon not as just an exception; (ii ) B increases with both the length of the sleeping period and the awakening intensity; (iii ) B takes into account the entire citation history in the time window 0 ≤ t ≤ t m ; and (iv ) The denominator of Eq. 2 penalizes early citations so that, at parity of total citations received, the later those citations are accumulated the higher is the value of B. (Eq. 3) of a paper. The blue curve represents the number of citations c t received by the paper at age t (i.e., t represents the number of years since its publication). The black dotted line connecting the points (0, c 0 ) and (t m , c tm ) is the reference line t (Eq. 1) against which the citation history of the paper is compared. The awakening time t a ≤ t m is defined as the age that maximizes the distance from (t, c t ) to the line t (Eq.  We now give a plausible definition of awakening time-the year when the abrupt change in the accumulation of citations of SBs occurs. Being able to pinpoint the awakening time may help identifying possible general trigger mechanisms behind said change. For example, in SI Appendix we show that around the awakening time, the SBs co-citation dynamics exhibit clear topical patterns (SI Appendix, Fig. S11 ) [37] . We define the awakening time t a as the time t at which the distance d t between the point (t, c t ) and the reference line t 6 reaches its maximum: where d t is given by As we shall show, the above definition works well for limit cases where there are no citations until the spike, and seems to well capture the qualitative notion of awakening time when a strong SB-like behavior is present. We use two datasets in the following empirical analysis, the American Physical Society (APS) and the Web of Science (WoS) dataset (SI Appendix, section S1). The APS journals are the major publication outlets in physics. WoS includes papers in both sciences and social sciences. We focus on the 384, 649 papers in the APS and 22, 379, 244 papers in the WoS that received at least one citation. Those papers span more than a century, and thus allow us to investigate the SB phenomenon for a long observation period. Whereas the APS dataset can be viewed as a perfect proxy to characterize citation dynamics within the monodisciplinary research field of physics and is used to compare our analysis with a previous study [37] , the WoS dataset allows us to underpin multidisciplinary features of the SB phenomenon. II. RESULTS First, we qualitatively demonstrate the resolution power of B for four papers with radically different citation trajectories. Fig. 2A shows a paper with a very high B value. Published in 1951, this paper collected a small number of yearly citations until 1994, when it suddenly started to receive many citations until reaching its maximum in 2000. curve reaches its peak, but does not consider how it decreases after that. The paper in To this end, we provide a statistical description of the distribution of beauty coefficients across all papers in each of the two datasets. Fig. 3 The result of the previous section implicitly suggests that the SB phenomenon could be in principle described via a simple mechanism that works essentially at all scales. This leads naturally to the question whether the observed distributions of B can be accounted for by idealized network evolution models. To address this question, we first consider a citation network randomization (NR) process where citations are randomly reshuffled, preserving time order (SI Appendix, section S4). SI Appendix, Fig. S2 compares the citation history of the top nine SBs in the APS dataset and the corresponding ones obtained through the NR process. They typically show opposite trends, with NR histories exhibiting a rapid decline. This is not surprising: As later papers are considered, the probability for an existing paper to receive a citation from one of such late papers decreases simply because there is a larger number of papers that could potentially receive the citation. This leads to typically smaller beauty coefficients, as evident in the sharp decrease of the NR distribution in Fig. 3 , and the associated small maximum value B = 30. Next, we consider the preferential attachment (PA) mechanism as another baseline model, as it is one of the most fundamental ingredients used in most modeling efforts aimed at describing citation histories of papers. In the PA baseline, references of progressively added The occurrence of extreme cases of SBs is not limited to physics. Table I lists How are SBs distributed among different (sub-)disciplines? To further investigate the multidisciplinary character of the SB phenomenon, we took advantage of journal classifications provided by Journal Citation Reports (JCR) (thomsonreuters.com/en/productsservices/scholarly-scientific-research/research-managementand-evaluation/journal-citationreports.html), which classify scientific journals into one or more subject categories (e.g. physics, multidisciplinary; mathematics; medicine, general and internal). We first consider only papers published in journals belonging to at least one JCR subject category, and focus on the top 0.1% of papers with highest B values. Then, we compute the fraction of 12 those papers that belong to a given subject category. Fig. 4 (ii ) journals in the multidisciplinary sciences subject category are really more fit to attract publications that become field-defining even decades after their appearance. A full answer to this question would require a case-by-case examination, but it can be addressed in a systematic way by studying the papers that cite the SB before and after its awakening. To illustrate this strategy, it is worth to examine two paradigmatic examples of indicators in research evaluation [46] . The change in contextual importance of the 1955 paper by Garfield is further revealed by the frequency of keywords appearing in the titles of its citing papers before and after year 2000 ( Fig. 5B and C ) , with the notion of "impact factor" becoming the main recognizable difference. With a similar motivation, the 1977 paper by Zachary also tops the ranking of SBs coming from the social sciences [47] . This paper was essentially unnoticed for about 30 years, but then became suddenly important in network science research after the publication of the seminal paper by Girvan and Newman, which adopts the social network described in the Zachary paper as a paradigmatic benchmark to validate community detection methods on graphs [48] (SI Appendix, Fig. S12 ). The examples above suggest that a partial explanation behind the sudden awakening of top SBs may lie in the fact that the paper in question is suddenly "discovered" as relevant by an entire community in another discipline. To support this hypothesis, in Fig 6 we 16 divide the papers in the WoS dataset in three disjoint subsets with high, medium, and low values of B. For each subset we compute the cumulative distribution for the fraction of citations received by a paper from publications in a discipline (as inferred by the journal of publication) different from that of the cited paper. Top SBs are clearly different from the other two categories and are characterized by a typically very high fraction of citations from other disciplines: for about 80% of the top SBs, as much as 75% or more of citations are of interdisciplinary nature. The main purpose of this work was to introduce a parameter-free method to quantify to what extent a paper is an SB. Through a systematic analysis carried out on large-scale bibliographic databases and over observation windows longer than a century, we have shown that our method correctly identifies cases that meet the intuitive notion of SBs. We noticed that our measure is not entirely free of biases: Comparing the degree of beauty between papers in different disciplines or ages may be problematic due to differences in the overall citation patterns. Despite this limitation, we found that papers whose citation histories are characterized by long dormant periods followed by fast growths are not exceptional outliers, but simply the extreme cases in very heterogeneous but otherwise continuous distributions. Simple models based on cumulative advantage, although consistent with overall citation distributions, are not easily reconciled with the observed distributions of beauty coefficients. Further work is needed to uncover the general mechanisms that may be held responsible for the awakening of SBs. ACKNOWLEDGMENTS
Ke et al. REF proposed methods to systematically identify them from citation time series.
Defining and identifying Sleeping Beauties in science
Proceedings of the National Academy of Sciences of the United States of America
Proceedings of the National Academy of Sciences of the United States of America
Computer Science Physics Medicine