Reproducibility Signals in Science: A preliminary analysis (2022)

Abstract: Reproducibility is an important feature of science; experiments are retested, and analyses are repeated. Trust in the findings increases when consistent results are achieved. Despite the importance of reproducibility, significant work is often involved in these efforts, and some published findings may not be reproducible due to oversights or errors. In this paper, we examine a myriad of features in scholarly articles published in computer science conferences and journals and test how they correlate with reproducibility. We collected data from three different sources that labeled publications as either reproducible or irreproducible and employed statistical significance tests to identify features of those publications that hold clues about reproducibility. We found the readability of the scholarly article and accessibility of the software artifacts through hyperlinks to be strong signals noticeable amongst reproducible scholarly articles.

## introduction transparency in the scientific process accelerates scientific discovery and strengthens public opinions on scientifically driven matters. reproducibility plays a crucial role in aiding this transparency, and it is encouraging to have a consensus in the scientific community to address the problem of reproducibility in science. policymakers, government entities, open source communities, peer-reviewed journals, conferences, and the academic community at large have a shared responsibility to promote reproducible research. effective dissemination of science cannot happen without trust and integrity in the scientific process. practically, reproducible science has a first-hand impact in notable places such as research labs, classrooms, industries, and academia. lack of reproducible research could restrict attaining a deeper understanding of the original researcher's thought process and, therefore, severely impact people involved in the communities mentioned earlier. the concept of reproducibility is intricate and stratified with different but complementary issues. before we attempt to understand how to approach the problem of reproducibility, we must first provide some definition of what we mean by this term in this context. studies such as @xcite @xcite highlight how the definition of reproducibility varies across different studies and disciplines and how differing definitions can result in confusion. for that reason, the flexible definition presented in @xcite is appealing: "the ability of an independent research team to produce the same results using the same method based on the documentation made by the original research team." collective efforts from various players of the research community such as publishers, conference organizers, and journals in promoting good practices for ensuring reproducibility in the experimentation process is refreshing, but there is still a lack of agreement on what exactly constitutes a "good practice" which is a concern. in this study, we attempt to understand the relationship between the structure of science @xcite and the concept of reproducibility by using statistical significance tests. in doing so, our emphasis is to examine epistemic opacity @xcite of linguistic features and structural features concerning reproducibility. we achieve this by running numerous hypothesis tests and identifying the significant factors affecting the reproducibility of scholarly articles. our goal is to utilize statistical tests to pick signals that could help identify articles requiring more (or less) effort to reproduce. ## related work reproducibility is an important concept that affects large communities in general @xcite . the breadth of literature on re-producibility spanning different disciplines @xcite @xcite has broadly focused on either performing large meta-analyses that reproduce a large set of scholarly articles or qualitative studies that encourage researchers to adopt a certain methodology. our study falls in line with the studies that attempts to quantify the factors important for reproducibility, e.g. @xcite . identifying such important factors would also be helpful in building machine learning models that can estimate the degree of reproducibility in scholarly articles @xcite . ## data while scientific publications often follow similar structures, there is significant freedom in how ideas are communicated and expressed. this lack of rigidity allows authors to weave stories around fundamental ideas, and the absorption of particular ideas can sometimes be related to how they are presented. we are interested in whether the structure of a publication reveals anything about its potential for (ir-)reproducibility. to examine this, we compiled a collection of scholarly articles that have been evaluated as either reproducible or irreproducible from three different sources. for each article, we gathered comprehensive metadata and extracted structural and linguistic features. these collections of articles include: the data collected provides significant detail about the effort involved in reproducing the studies in the original publications. the current repository provides results for 207 papers; 142 are classified as reproducible and 65 as non-reproducible. from each of the three sources, we used the available metadata to locate each article. in some cases, we searched by article and authors' names to obtain a doi or, in some cases, a url for an article. if we were unable to unambiguously determine this information, the article was dropped from the dataset. using the doi, we were able to obtain further metadata and the full text of the article, usually in pdf format. after filling out the metadata and obtaining the full text, we had 305 papers in total; 206 were classified as reproducible, and 99 were classified as non-reproducible. data and code will be made available as supplementary information upon publishing. ## results we computed correlations and performed statistical significance tests on the combined data sources to identify features that played a significant role in indicating the reproducibility of scholarly articles. the point biserial correlations as shown in tables 1 and 2 suggested that only mean readability and number of hyperlinks significantly correlate with reproducibility. the results of the mann-whitney u and chisquared tests show that mean readability, number of hyperlinks, number of algorithms, average word length, and yule's measure of lexical diversity to be statistically significant features that align and signal scholarly work that is reproducible with reasonable certainty. more significantly, the readability of a scholarly article and accessibility of software artifacts, either as code repositories, psuedo-code, or algorithms, could be considered strong indicators for reproducibility. it is important to note that these signals do not quantify or assure the reproducibility of a scholarly article but rather help identify articles that require more (or less) effort to reproduce. ## discussion the structure of science involves a well-formed process that begins with factual and valid data, continues through detailed descriptions of experimental procedures, and follows on to clearly presented results. the scientific process has many tenets, but these represent some. they have been promulgated over the years to allow the scientific process to flourish with checks and balances in the form of peer reviews. contextually, factors such as discipline, year, type of scientific study, etc., play a major role in identifying the effort required to reproduce articles. therefore, the dataset we built is an essential factor to consider while interpreting our findings that the readability of the scholarly article and accessibility of the software artifacts through hyperlinks are significant features among reproducible scholarly articles. our motivation is to discover additional latent variables that consider these contextual factors while identifying the effort required to reproduce articles. ## conclusions and future work in this study, our pursuit of identifying features that can signal reproducible science involved correlations and significance tests. we found the readability of the scholarly article and accessibility of the software artifacts through hyperlinks to be significant features among reproducible scholarly articles. our code repository with data and experiments will be available post-publishing. in the future, we plan on expanding the scope of our study by 1) gathering more badged data from acm; 2) testing the validity of our findings against adversarial examples; and 3) observing the effects