Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research (2024)

Abstract: We analysed a sample of NLP research papers archived in ACL Anthology as an attempt to quantify the degree of openness and the benefit of such an open culture in the NLP community. We observe that papers published in different NLP venues show different patterns related to artefact reuse. We also note that more than 30% of the papers we analysed do not release their artefacts publicly. Further, we observe a wide language-wise disparity in publicly available NLP-related artefacts.

## introduction the advancement of the computer science research field heavily depends on publicly available code, software, and tools. its sub-fields machine learning and natural language processing (nlp) have the additional requirement of datasets -to train and evaluate computational models. lack of access to these research artefacts has been identified as a major reason for the difficulty in reproducing works of others @xcite . the data requirement is particularly challenging in nlp -a dataset available for one language usually cannot be used in the context of another language foot_0 . therefore, the nlp community is highly encouraged to make their research artefacts publicly available. however, as far as we are aware, there is no quantifiable evidence on (1) the degree of openness in the nlp community or (2) the benefit of openness to the community. since "what we do not measure, we cannot improve" @xcite , in this paper, we quantify both these aspects. to this end, we semi-automatically analyse a sample of nlp research papers published in acl anthology (aa) and corpora/ language models (lms) released in hugging face foot_1 , and answer the following questions: to answer the first question, we record the number of papers that reuse the artefacts released by past research. since there is a language-wise disparity in nlp research @xcite , this analysis is conducted while separating low-and high-resource languages. to answer the second question, we record the papers that indicate they would release the newly produced artefacts. we also record whether they have provided a repository url. we do further analysis to find out whether these repositories have the artefacts they are supposed to have. finally, we record the number of datasets and lms available for different language classes on hugging face. we observe that papers published in different venues show different patterns in artefact reuse. we also observe that a worrying percentage of papers that produced an artefact have not publicly released those artefacts. to a lesser degree, broken repository links and empty resource repositories were also noted. finally, it is noted that the languagewise disparity in lm/data availability @xcite @xcite is still staggering. ## data extraction we use aa as the research paper repository. while aa is the largest nlp-related paper repository, ranathunga and de silva (2022) note that many papers related to low-resource languages also get published in other venues such as ieee conferences or regional journals. however, the popularly used google scholar does not have a free api to extract data, and the coverage of semantic scholar is rather poor foot_2 . moreover, some conference and journal publications are hidden behind paywalls. while archives such as arxiv are a possible option, they do not contain the meta data for us to carry out a conference/journal-specific analysis. considering all these factors, we selected aa to extract papers for our analysis. aa has been the common choice for many research related to diversity analysis in nlp research @xcite @xcite . when collecting data from aa, we reuse data and code from ranathunga and de silva (2022) who in turn had used code and data from @xcite and rohatgi (2022) (respectively). however, we had to collect data post 2022 by ourselves. we use the urls of papers from the acl anthology bibliography to extract the title and abstract of each paper. we then allocate the papers to different languages, following the language list (of 6419 languages) given by ranathunga and de silva (2022). for each language name, we check for matches in both the title and abstract and download the matched papers using their respective urls (where a url to the pdf is available). of these, 130 languages are ignored due to the high count of false positives caused by matches with existing words and author names foot_4 . next, we convert each paper to its text format. then we further group these language-wise papers according to language category. the commonly used language category definition that is based on language resources is joshi et al. (2020) (see ## conclusion we hope our findings would help the nlp community to better appreciate the benefit of openness and to commit to releasing the artefacts they produce. we further hope these statistics will be useful to acl in making informed decisions. it would be interesting to run this same experiment 5 or 10 years down the line, to see if there are any changes in releasing and reusing artefacts. in hopes to assist in such efforts, our code is publicly released foot_16 .