Research Replication Prediction Using Weakly Supervised Learning (2020)

Abstract: Knowing whether a published research result can be replicated is important. Carrying out direct replication of published research incurs a high cost. There are efforts tried to use machine learning aided methods to predict scientific claims’ replicability. However, existing machine learning aided approaches use only hand-extracted statistics features such as p-value, sample size, etc. without utilizing research papers’ text information and train only on a very small size of annotated data without making the most use of a large number of unlabeled articles. Therefore, it is desirable to develop effective machine learning aided automatic methods which can automatically extract text information as features so that we can benefit from Natural Language Processing techniques. Besides, we aim for an approach that benefits from both labeled and the large number of unlabeled data. In this paper, we propose two weakly supervised learning approaches that use automatically extracted text information of research papers to improve the prediction accuracy of research replication using both labeled and unlabeled datasets. Our experiments over real-world datasets show that our approaches obtain much better prediction performance compared to the supervised models utilizing only statistic features and a small size of labeled dataset. Further, we are able to achieve an accuracy of 75.76% for predicting the replicability of research.

## introduction non-reproducible scientific results will mislead the progress of science and undermine the trustworthiness of the research community. in recent years, we saw the emergence of systematic large-scale replication projects which are based on the concerns of research credibility in the social and behavioral sciences @xcite @xcite @xcite . researchers conducted preregistered replications of hundreds of classic and contemporary published findings in the social and behavioral sciences. unfortunately, the reported replication rates only range from 39% to 62%. therefore it is important to develop a confidence scoring system for the following question: to what extend can a research result be reproduced? the answer to the above question will help facilitate the policymakers as well as the general public to better understand and digest a published claim. as a response, for example, defense advanced research projects agency (darpa) has announced a systematic confidence checking of published claims @xcite . alongside the above encouraging movement, the downside is that the average replication expense of each research project (which often consists a number of research studies) can go up to @xmath0500,000 to us$2,000,000 investment" ular scientific claim. nonetheless, existing machine learning works on replication prediction face a couple of outstanding challenges: we aim for a method that is fully automatic in feature generation, and that can leverage the existence of the large corpus of unlabeled (checked) articles for boosting up the performance in predicting replications. to tackle the first challenge, we will resort to natural language processing (nlp) tools to process the research articles to obtain meaningful text features. text information of research papers is important and intuitive resource for training machine learning models. the rich amount of structural text information looks promising to us to help improve the predictive performance of replication. further, a good understanding of text information from different components of an article (e.g., abstract, introduction, methods, experimental results, etc.) will also be helpful for highlighting suspicious sections of the articles for a more targeted check. however, the training of the state-of-the-art nlp models aligns with our second challenge that it often relies on a massive volume of annotated training data. due to the severely limited ground truth annotation we have, we desire a method that leverages large amounts of unlabeled research articles. these unlabeled examples, although possibly noisy, can provide informative features. to make the most use of the unlabeled data, we explore the possibility of using a weakly supervised approach to perform replication prediction. the particular type of weakly supervised learning method that we will focus on utilizes techniques from the literature on learning from noisy labels @xcite @xcite @xcite . our high-level idea is to bootstrap the small set of labeled data to train a set of weak predictors which will help us generate "artificial" and noisy labels for the unlabeled articles. then we will apply tools from learning with noisy labels to improve the training with these artificially supervised examples. we focus on two approaches to address the above problem of learning with artificial labels. the first approach uses efficient variational inference methods @xcite to estimate the error rates of the noisy labels. the above knowledge of error rates allows us to perform loss correction @xcite to improve the performance with the help of an unlabeled dataset. the second approach is inspired by a recent work @xcite ) that proposed a family of peer loss functions which can perform learning with noisy labels without knowing noise rates and without conducting intermediate error rates' estimation step. we utilized both labeled and unlabeled datasets to carry out the study of replication prediction. the labeled dataset containing 399 research articles are obtained from summarizing eight research replication projects (details will be given later). as for the unlabeled dataset, a python crawler is implemented to obtain the pdf files of 2,170 research papers from the websites of corresponding journals. we preprocess the files to extract text information. then bert @xcite is used for tokenization and for obtaining word embeddings to serve as the input features for training. the experimental results demonstrate that i) using text information as features can improve the performance than utilizing only pre-and handextracted statistics features. the combination of models trained on text features and statistics features separately can obtain better performance than separate models; and ii) our weakly supervised methods that take advantage of unlabeled data can significantly improve the prediction performance. the best of our proposed methods can achieve a prediction accuracy of 75.76%, as well as a 72.50% precision, a 88.24% recall, and a 78.95% f1 score. we summarize our contributions as follows: ## related work replication crisis has spurred systematic largescale direct replication projects in the social and behavioral sciences @xcite @xcite @xcite . data is collected by individual volunteers, volunteer teams, or amazon mechanical turk (amt). however, direct research replication is expensive and time-consuming @xcite . machine learning serves as a much more efficient method to conduct replication prediction. @xcite applied ml methods on the data from four large-scale replication projects in experimental psychology and economics and studied which variables drive predictable replication. but they used only statistics features such as p-value, sample size, etc. and train only on a small labeled dataset. we hold the hypothesis that text features contain rich information to potentially improve the performance of replication prediction. in nlp, many research works have been proposed for text processing to make use of text features @xcite @xcite . weakly supervised learning approaches have been proposed to utilize both labeled and unlabeled data @xcite @xcite . our weakly supervised learning approaches tie close to learning with the inaccurate supervision @xcite @xcite @xcite . particularly relevant to us, a surrogate loss function is proposed in @xcite to achieve an unbiased estimation of the true training loss using only noisy labels. @xcite introduced a new family of loss functions, peer loss functions, to empirical risk minimization (erm), for a broad class of learning with noisy labels problems, without requiring estimating the error rates of the noisy labels. the question of how we treat an article/claim as replicable is an active research question itself @xcite . to include as many annotated data points as possible, we adopt the most basic binary model that defines replication success as a "statistically significant (p-@xmath1.05) effect in the same direction as in the original study." @xcite the annotated dataset comes from eight research replication projects which are the registered replication report (rrr) @xcite , many labs 1 @xcite , many labs 2 @xcite , many labs 3 @xcite , social sciences replication project (ssrp) @xcite , psychfiledrawer @xcite , experimental economics replication project @xcite , and reproducibility project: psychology (rpp) @xcite . 2006 2007 2008 2009 2010 2011 2012 we list the average length (# of words contained), minimum length, and maximum length information of different datasets in ## weakly supervised research replication prediction we introduce the pipeline of our weakly supervised research prediction framework. ## method in this section we present two weakly supervised methods. the first approach is based on the error correction proxy loss function @xcite and the variational inference approaches (mean field) @xcite to estimate the error rates. the two techniques jointly provide us a bias-corrected training process to improve the model's robustness against noises in labels. we name this solution as variational inference aided weakly supervised learning. the second approach is built on the peer loss approach @xcite . this approach is particularly suitable for our application when the label noises are unclear. in this paper, we will apply peer loss function in the weakly supervised learning scenario for the research replication prediction problem. we name this solution as peer loss aided weakly supervised learning. ## experiments in this section, we present our experimental results and findings and offer discussions. ## discussion in this paper, we used two fields of corpus ("economic review" and "psychological science") to train our model together because both of them are social sciences that rely heavily on quantitative methodologies (e.g., survey, experiments) and draw conclusions based on statistics. thus, they share the same definition of replicability such that whether the same statistical findings (e.g., effect size, p-value) can be reproduced in replications following the same methodological procedure with different samples. the same methodologies are also widely used in empirical sciences (e.g., lab experiments in biology and medicine) which demand replicability in the same sense and also follow the same format in reporting their procedures and findings. thus, our proposed methods should also work in the contexts mentioned above. ## conclusion the paper studies the possibilities of using weakly supervised learning methods based on text information of research papers to improve the prediction accuracy of research replication using a small amount of labeled data and a large amount of unlabeled data. our experiments show that our ap-proaches successfully improved prediction performance compared to the supervised models utilizing only statistic features and a small size of labeled dataset. our approach can also be generically extended to other weakly supervised nlp. our study has limitations. first of all, our sampling of the unsupervised articles is not ideal. as a next step, we will include a more diverse and bigger pool of representative articles into our study. our method replied on bert for feature extraction, which remains largely as a "blackbox" processor. in the future, we plan to explore other advanced nlp techniques such as named entity recognition, relation extraction, etc. to help us identify more explainable features. this information will help facilitate the human evaluation of a research claim's replicability.