Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking (2022)

Abstract: Research Replication Prediction (RRP) is the task of predicting whether a published research result can be replicated or not. Building an interpretable neural text classifier for RRP promotes the understanding of why a research paper is predicted as replicable or non-replicable and therefore makes its real-world application more reliable and trustworthy. However, the prior works on model interpretation mainly focused on improving the model interpretability at the word/phrase level, which are insufficient especially for long research papers in RRP. Furthermore, the existing methods cannot utilize a large size of unlabeled dataset to further improve the model interpretability. To address these limitations, we aim to build an interpretable neural model which can provide sentence-level explanations and apply weakly supervised approach to further leverage the large corpus of unlabeled datasets to boost the interpretability in addition to improving prediction performance as existing works have done. In this work, we propose the Variational Contextual Consistency Sentence Masking (VCCSM) method to automatically extract key sentences based on the context in the classifier, using both labeled and unlabeled datasets. Results of our experiments on RRP along with European Convention of Human Rights (ECHR) datasets demonstrate that VCCSM is able to improve the model interpretability for the long document classification tasks using the area over the perturbation curve and post-hoc accuracy as evaluation metrics.

## introduction scientific research results that cannot be reproduced are unreliable and negatively impact the development of science. therefore, it is important to know whether a published research result can be replicated or not. to this end, domain researchers have conducted several direct replication projects in contemporary published social science studies the mean error rates appear in the lower part of ## problem statement in this paper, our main goal is to improve the intepretibility of neural textual classifier for research replication prediction (rrp). first we introduce the rrp task. ## method the details of our proposed variational contextual consistency sentence masking (vccsm) method are described in this section. ## experimental setup the proposed vccsm method is evaluated with two typical neural network models commonly used on text classification tasks, lstm @xcite and bert @xcite on two datasets. ## results we tested our proposed models on two text classification datasets (rrp along with echr), and the details about prediction accuracy and interpretability are described in this section. ## concluding remarks in this paper, we proposed vccsm to improve both interpretability and prediction accuracy on rrp along with echr datasets, using largely unlabeled datasets. we tested vccsm with two different neural text classifiers (lstm and bert) and evaluated both prediction accuracy and interpretability metrics. as future work, we plan to explore other advanced interpretable models and weakly supervised methods to further improve the prediction performance and interpretability of long document classification tasks. in addition, rrp dataset also contains 2,170 research articles as the unlabeled dataset. @xcite observed that most papers in the labeled dataset in the rrp dataset are economical and psychology related. among those papers, they are mainly from american economic review and psychological science journals. therefore, a python crawler is written by @xcite to get 2,170 published research articles on the american economic review (jan 2011 -dec 2014) and psychological science websites @xcite . the number of articles crawled from american economic review and psychological science websites are 981 and 1,189 respectively.