Resource papers as registered reports: a proposal (2023)

Abstract: This is a proposal for publishing resource papers as registered reports in the Northern European Journal of Language Technology. The idea is that authors write a data collection plan with a full data statement, to the extent that it can be written before data collection starts. Once the proposal is approved, publication of the final resource paper is guaranteed, as long as the data collection plan is followed (modulo reasonable changes due to unforeseen circumstances). This proposal changes the reviewing process from an antagonistic to a collaborative enterprise, and hopefully encourages NLP resources to develop and publish more high-quality datasets. The key advantage of this proposal is that it helps to promote responsible resource development (through constructive peer review) and to avoid research waste.

## introduction a common sentiment in nlp is that the creation of corpora and benchmarks is under-appreciated @xcite , even though resources are one of the driving factors of progress in our field. moreover, the measurement of progress critically depends on having solid benchmarks. if authors are weary of producing new resources, we all suffer the consequences. how can we avoid this? ## process outline the general writing process for registered reports has been described elsewhere (e.g., @xcite . what would the process look like for resource papers? here is a brief sketch of what this process could look like if nejlt would accept registered reports. ## conclusion this letter proposed to offer potential resource authors the opportunity to publish their resources as registered reports, as an addition to the existing paper types. (the resource category would not need be removed.) the proposal outlined here is more modest than the one put forth by @xcite , who suggest that all types of nlp papers (except position papers) could in theory be published as registered reports. this modesty is not for a lack of ambition; instead, this proposal is offered as a first step, to see if registered reports could actually work for nlp research. and what better way to start, than to support the creation of fundamental resources? registered-reports-policy.html on 13 july 2023.