  The reproducibility of published research has become an important topic in
science policy. A number of large-scale replication projects have been
conducted to gauge the overall reproducibility in specific academic fields.
Here, we present an analysis of data from four studies which sought to forecast
the outcomes of replication projects in the social and behavioural sciences,
using human experts who participated in prediction markets and answered
surveys. Because the number of findings replicated and predicted in each
individual study was small, pooling the data offers an opportunity to evaluate
hypotheses regarding the performance of prediction markets and surveys at a
higher power. In total, peer beliefs were elicited for the replication outcomes
of 103 published findings. We find there is information within the scientific
community about the replicability of scientific findings, and that both surveys
and prediction markets can be used to elicit and aggregate this information.
Our results show prediction markets can determine the outcomes of direct
replications with 73% accuracy (n=103). Both the prediction market prices and
the average survey responses are correlated with outcomes (0.581 and 0.564
respectively, both p < .001). We also found a significant relationship between
p-values of the original findings and replication outcomes. The dataset is made
available through the R package pooledmaRket and can be used to further study
community beliefs towards replications outcomes as elicited in the surveys and
prediction markets.
