The impact of preprint servers in the formation of novel ideas (2020)

Abstract: We study whether novel ideas in biomedical literature appear first in preprints or traditional journals. We develop a Bayesian method to estimate the time of appearance for a phrase in the literature, and apply it to a number of phrases, both automatically extracted and suggested by experts. We see that presently most phrases appear first in the traditional journals, but there is a number of phrases with the first appearance on preprint servers. A comparison of the general composition of texts from bioRxiv and traditional journals shows a growing trend of bioRxiv being predictive of traditional journals. We discuss the application of the method for related problems.

## introduction a paper submitted to a journal goes through several stages: peer review, editorial work, copyediting, publication. this leads to a long waiting time between submission and the final publication @xcite . the situation is especially bad in the life sciences, where the waiting time between submission and publication approaches the duration of a traditional phd study, creating serious difficulties for young scientists @xcite . while this is frustrating for scientists whose recognition and promotion often depend on the publication record, it is also bad for science itself, significantly slowing down its progress @xcite . preprint servers were offered as a means of accelerating science @xcite @xcite @xcite , especially in the wake of covid-19 epidemics @xcite . the discussion about the benefits (and dangers) of preprint is no longer confined to the scientific literature, coming to the pages of popular newspapers (eisen and tibshirani, 2020). ## related works novel ideas and breakthroughs are among the central concepts for the science of science. a number of studies propose different ideas to quantify originality in science and technology @xcite @xcite @xcite @xcite or their impact on the other works @xcite @xcite @xcite @xcite @xcite . the prediction of breakthroughs, scientific impact and citation counts is a well developed area @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, the question asked in these works is different from the one we ask. most of the researchers tried to determine what makes a work original or impactful, and how to predict originality or impact. our question is the following: suppose we know a certain idea is novel (or impactful). can we pinpoint a moment in time when this idea appeared, and where did it appear? sentence level novelty detection was a topic of novelty tracks of text retrieval conferences (trec) from 2002 to 2004 @xcite @xcite . the goal of these tracks was to highlight the relevant sentences that contain novel information, given a topic and an ordered list of relevant documents. at the document level, @xcite computed novelty score based on the inverse document frequency scoring function. another work by @xcite presents a comparison study of different novelty detection methods evaluated on news articles where language model based methods perform better than the cosine similarity based ones. @xcite conducted experiments with information entropy measure to calculate novelty of a document. again, the work that we present here significantly differs from the existing novelty detection methods since we use the novelty detection as a starting point rather than a goal. we first get candidate phrases from the documents with the most appropriate key phrases extraction method, and then determine the appearance timing of these phrases. another field that is relevant to our research is the detection of change points in a stream of events. change point detection (or cpd) detects abrupt shifts in time series trends that can be easily identified via the human eye, but are harder to pinpoint using traditional statistical approaches. the research in cpd is applicable across an array of industries, including finance, manufacturing quality control, energy, medical diagnostics, and human activity analysis. there are many representative methods of cpd. binary segmentation @xcite is a sequential approach: first, one change point is detected in the complete input signal, then series is split around this change point, then the operation is repeated on the two resulting sub-signals. as opposite to binary segmentation, which is a greedy procedure, bottom-up segmentation (fryzlewicz, 2007) is generous: it starts with many change points and successively deletes the less significant ones. first, the signal is divided in many sub-signals along a regular grid. then contiguous segments are successively merged according to a measure of how similar they are. because the enumeration of all possible partitions is impossible, pelt @xcite relies on a pruning rule. many indexes are discarded, greatly reducing the computational cost while retaining the ability to find the optimal segmentation. window-based change point detection @xcite uses two windows which slide along the data stream. dynamic programming was also used for this task @xcite . in this work we propose a simple bayesian approach to the detection of change points, which seems to give intuitively reasonable results for our purpose. ## bayesian approach to novelty detection our proposed bayesian approach to novelty detection (band) finds the time interval τ that maximizes the observed series of publication frequency for a phrase (figure 1 ). paper publications are events, so it is reasonable to assume that the number of publications n in a unit interval at the time t containing the given phrase z follows a poisson distribution. the joint density function for publication frequency is given by the equation prior to τ , we expect the number of publications containing the given phrase to be small, ideally zero. the parameter α > 0 controls for the noise (misattributed papers, improper or ambiguous usage of the phrase, etc.). after the moment τ , we expect the steady grow of phrase popularity with the rate β. in other words, τ is the point in time ehen the phrase begins to be adopted. we consider each phrase independently and use bayesian modeling to find the most probable parameters α, β, and τ given the observed data using the standard bayesian approach our motivation for finding τ using band is to compare the impact of different publication venues (i.e. preprint servers and peer-reviewed journals) that cover overlapping research topics. for a single phrase z, we use the observed data from two sources and estimate the posterior probability p (α, β, τ ) for each source separately. then we run a simulation to find the 95% confidence interval of τ using the following procedure: in order to compare two sources, we create two sets of tuples, one for each source. then we randomly draw a tuple from the first set and a tuple from the second one. for each pair i we compute δ i , the expected difference between the two sources where τ (s) i is the i-th sample in source s: the conclusion about the priority is based on distribution of δ around zero. if δ > 0 for the majority of the pairs, then the phrase gains traction first on source s 0 . otherwise the second source wins. ## experimental setup in this section we describe our data collection procedure and background on methods of text processing and assessing data quality. our code for running experiments is publicly available. foot_0 ## results and analysis in this section we discuss the following hypotheses and research questions: below we answer each of these questions. ## future directions our work serves as a first attempt for discovering research impact of publication venues using full text analysis. one notable assumption we make is that all phrase mentions are treated equally. in the future, we may want to distinguish how a phrase is being used. for example, if the phrase of interest is z, then a research paper may write about works extending z, but alternatively it may simply discuss techniques similar to z. similar analysis has provided useful for citations @xcite . furthermore, our approach leverages textrank to find many relevant phrases, but we do not cluster phrases, so two or more phrases with similar meaning will be treated separately (i.e. facs and fluorescence activated cell sorting). a simple alias table or string similarity extension @xcite would be a clear improvement. leveraging high precision concept extraction systems @xcite might improve clustering even more. another approach would be to use a better proxy for ideas than textual phrase, like concepts or topics extracted from the corpus. ## conclusions we introduce a bayesian model for novelty detection (band), and use this model to investigate how quickly new ideas form on pre-print servers compared to peer-reviewed journals. our findings indicate that novel phrases, which we use as a proxy for new ideas, in most cases appear on pre-print servers and in peer reviewed journals. in some cases, novel phrases appear on pre-print servers first. in many cases the content of preprints is a predictor of the content of peer reviewed journals. as the preprint servers mature, this feature becomes more prominent. when a fast review time is in high demand (such as during epidemic outbreaks), pre-print servers have a high utility, and the related novel phrases appear on pre-print servers first.