Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond (2022)

Abstract: The shift towards publicly available text sources has enabled language processing at unprecedented scale, yet leaves under-serviced the domains where public and openly licensed data is scarce. Proactively collecting text data for research is a viable strategy to address this scarcity, but lacks systematic methodology taking into account the many ethical, legal and confidentiality-related aspects of data collection. Our work presents a case study on proactive data collection in peer review – a challenging and under-resourced NLP domain. We outline ethical and legal desiderata for proactive data collection and introduce “Yes-Yes-Yes”, the first donation-based peer reviewing data collection workflow that meets these requirements. We report on the implementation of Yes-Yes-Yes at ACL Rolling Review and empirically study the implications of proactive data collection for the dataset size and the biases induced by the donation behavior on the peer reviewing platform.

## introduction empirical nlp is shaped by its data sources. while early work mostly considered canonical sources like newswire @xcite , the last decade has been marked by the transition to fortuitous @xcite , found data openly available for collection. the rise of crowdsourcing platforms also made it possible to generate massive amounts of textual data on demand @xcite , feeding novel nlp developments in both task-specific problem engineering and general-purpose representation learning. yet problems persist with mainstream approaches to data collection. canonical data is not representative. not every language variety can be "found", and, by far, not every found text can be used for research @xcite . generating 1 code available at: https://github.com/ukplab/ openreview-licensing-workflow n -reviewers acceptance authors -y ? ? 1y y y ? 2y stop y y y 3y outcome 3y workflow 2 3 protected public 1 figure 1 : the yes-yes-yes workflow: data collection is conditioned on participants' consent (1, 3) and on the confidentiality status of the publication (2). only data that fulfills all conditions is added to public datasets. given the raising attention to the development of specialized nlp applications @xcite , the ability to acquire data from the domains of interest on-demand becomes crucial. to that end, a promising but not systematically explored alternative to canonical, found, and generated source data is proactive, targeted data collection, where texts are harvested from a previously closed process in the domain of interest, and made available for research. this requires careful consideration of the text production process in the target domain, as well as many ethical, legal, confidentiality-related and other aspects of data collection. in line with the recent work in responsible data handling for nlp @xcite and previous discussions on data collection in the digital mental health domain @xcite , our work explores practical implications of proactive data collection from a previously under-resourced domain: peer review. peer review is the cornerstone of academic quality control. the ever-increasing submission rates expose the weaknesses of this process, motivating the first generation of computational studies in peer review within and beyond nlp @xcite @xcite @xcite . such studies crucially depend on the availability of peer reviewing data. yet this data is hard to come by and is associated with a range of ethical, confidentiality and copyright issues, and while current methodological advances in peer review processing show great promise, a solid data foundation for the study of peer review is still lacking. this makes peer review an excellent target for proactive data collection. in this work we: ## problem dimensions we use peer reviewing data as an umbrella term for the artifacts produced during peer review, and limit our discussion here to submission drafts and review reports.we distinguish between metadata (numerical scores, track, paper format, etc.) and textual data, and focus our discussion on the latter. textual data falls under the eu general data protection regulation (gdpr) definition of personal data as "any information relating to an identified or identifiable natural person" foot_3 -the identities of the reviewers and authors are known to the editors, and remain potentially identifiable based on the writers' professional expertise and via author profiling. although peer reviewers and authors rarely sign formal non-disclosure agreements, peer reviewing data is confidential. finally, most of the peer reviewing data is anonymous, and only the editors know the identities of the participants. a. collection strategy. personal data requires consent or other explicit grounds for processing. two main approaches to obtaining consent are terms of service (tos) and donation. tos is a one-size-fits-all approach that requires the users to agree to the terms in order to use a platform. establishing universal tos is a challenge that involves balancing interests of many stakeholder groups, at the risk of losing participants who disagree with data collection. in a donation system, the decision to contribute data is made individually. although technically more intricate, donation-based collection allows participants who do not wish to contribute to still use the platform. donation-based approach might introduce participation bias into the data @xcite . b. stakeholder involvement. who has the authority to consent for what data? this question is not trivial: for example, in peer reviewing data, while a reviewer might agree to publishing their reports, the authors might object, not only due to potential negative reviews, but also due to the risk of leaking unpublished ideas and results prepublication. ideally one would want all involved stakeholders to consent; yet, increasing the number of involved parties can substantially reduce the amount of collected data and augment the bias. c. licensing. liberal data licensing allows the community to build upon prior data and ensures replicability. creative commons (cc) is a popular licensing choice for nlp datasets that supports additional restrictions on data sharing, adaption and commercial use. most cc licenses require attribution -specifying the title, authorship and source of the data. yet as most of the peer reviewing data is anonymous, it cannot be directly attributed to its authors, and declaring the work public domain (cc0) leaves the data reuse entirely unregulated, incl. commercialization and claiming copyright and attaching restrictive license to data derivatives. ## analysis with the implementation of the proposed workflow at arr, we can study the effects of donationbased proactive data collection on the resulting dataset composition, as well as the donation behavior across the users of the platform. for our analysis, we focus on the publications that were accepted at the 60th annual meeting of the acl (acl-2022) -the first major conference to employ arr as its main reviewing platform. as over 98% of the papers accepted at acl-2022 were submitted to september, october and november 2021 cycles of arr, we consider those as the full dataset (arr-all). we then focus the analysis on the metadata of the donated peer review reports from the subsets arr-1y (reviewers agree), -2y (paper accepted) and -3y (authors agree). for context, we bring in two prior datasets in the nlp domain. the acl-2018 @xcite was collected during the 56th annual meeting of the acl; consent collection during the peer reveiwing process and the lack of license transfer prevented the public release of the full dataset. the acl-2017 portion of the peerread corpus @xcite includes reviews of submissions to the 55th annual meeting of acl, for which both authors and reviewers agreed to share the review reports. ## discussion based on our experience, we determine four main directions for the future studies in proactive data collection. developing better approaches to stimulate participation is crucial for fast dataset growth: this includes ensuring high visibility of the data collection effort, high transparency of the process, and deeper integration of data collection into reviewing process without interfering with the process itself. our pilot investigation of bias in donation-based review collection revealed that some steps of the workflow indeed introduce polarity bias -yet it remains unclear whether it affects nlp applications and what other biases are present. the study of bias requires access to the full reviewing data -while we used anonymous numerical data from 1y and 2y as a proxy, deeper analysis would require access to review texts. while the 3y-workflow allows collection of protected data in theory (e.g. consented-to peer review texts blocked by the paper authors), ensuring fair access to protected data presents a technical, administrative and legal challenge: who provides and maintains the infrastructure for safe storage of the protected data? what can be done with this data? who can access the data and how is the access regulated? what are the consequences of protected data misuse and who would enforce them? a further community discussion is necessary to address those and other challenges and make protected data available for research. finally, from the legal perspective the replicability of research based on 3y data is ensured via license transfer to a data controller -in our case, the acl. this approach has many advantages over publishing unlicensed data, but requires an external license holder and does not have a formal mechanism to ensure data withdrawal from derivative datasets without compromising replicability. the search for alternative legal and technical frameworks for publishing peer reviewing data constitutes another promising avenue for future studies. double-blind pre-publication peer review, as done at arr, is an especially hard case for proactive data collection due to the confidentiality and anonymity. not all proposed measures will be relevant for any proactive data collection campaign, even in peer review -for example, other venues hosted by openreview make both accepted and rejected publications available, invalidating the confidentiality concern and rendering the 2y step of our workflow redundant. yet many of our findings in peer reviewing data collection at arr point at the potential gaps in the proactive data collection methodology in general. unlike canonical, found and generated data, proactive data collection demands systematic study of contribution behavior by the content creators. as most of the data to be collected proactively is user-generated personal data, solid ethical and legal frameworks are required to support its research use. as some of the data can never be made public, technical solutions for secure access to protected data are urgently needed to enable the study of bias in the new domains. ## conclusion given that most nlp tasks are either prone to domain shift, or are specific to a particular domain, or both, the future of nlp crucially depends on the field's ability to acquire new sources of textual data. we have presented "yes-yes-yes", the first ethically sound, consent-driven data collection process for peer reviewing data and reported on its implementation at the acl rolling review. we have further analysed the effects of ethical data collection strategies on dataset composition and found that different steps of our proposed data collection workflow indeed have quantifiable, systematic effects on the data. yet, many questions remain open, from strategies for better community engagement, to the technicalities related to replicability and archival of the data. we hope that our study sparks the systematic discussion on proactive data collection strategies as a viable alternative to canonical and found data in nlp, in the domain of peer reviewing, and beyond.