Program Chairs’ Report on Peer Review at ACL 2023 (2023)

Abstract: We present a summary of the efforts to improve conference peer review that were implemented at ACL’23. This includes work with the goal of improving review quality, clearer workflow and decision support for the area chairs, as well as our efforts to improve paper-reviewer matching for various kinds of non- mainstream NLP work, and improve the overall incentives for all participants of the peer review process. We present analysis of the factors affecting peer review, identify the most problematic issues that the authors complained about, and provide suggestions for the future chairs. We hope that publishing such reports would (a) improve transparency in decision-making, (b) help the people new to the field to understand how the *ACL conferences work, (c) provide useful data for the future chairs and workshop organizers, and also academic work on peer review, and (d) provide useful context for the final program, as a source of information for meta-research on the structure and trajectory of the field of NLP.

## introduction with the continued growth of our field and the rising number of conference submissions, peer review draws more and more attention from the community-as an application area @xcite @xcite , in meta-research @xcite , in initiatives to organize and release peer review data @xcite @xcite , and, of course, in the regular heated social media discussions during submission deadlines, review release dates, and acceptance notifications. it is unlikely that peer review will ever be perfect -it remains 'the least bad system' we have for ensuring the quality of scientific publications @xcite . still, with each iteration we should learn a little more about what works better for organizing peer review at such scale, and in a community so diverse in expertise and experience. as a step in that direction, acl'23 makes its peer review report public and an official part of the conference proceedings, complementing the introduction and other administrative materials. the goal is to increase the visibility of the results of the conference process, as well as any incidental findings from conference organizations and the lessons learned the hard way that may be useful to the future chairs and workshop organizers. such publications also provide extra incentives for the future program chairs to invest more effort in the analysis of their process, and they provide a useful background to the composition of the final program that may be useful for meta-science research (since they essentially document the selection process for that program). last but not least, such publications will improve the transparency of the *acl conference process, which may be useful to the researchers who are new to the field. we present the core statistics per track ( §2), analysis of resubmissions ( §3) and core demographics ( §4), our efforts for improving peer review quality ( §5), improving decision support for the chairs ( §6), out analysis of various factors contributing to review scores and final decisions ( §7), ethics review and best paper selection ( §8), and our efforts towards improving incentives for the authors, reviewers and chairs ( §9). we conclude with overall recommendations for future conference organizers ( §10). the materials we developed will be available at a dedicated repository 1 . the results presented here are based on the analysis of internal data of acl'23, as well as exit surveys that we sent to the chairs, authors and reviewers. we received responses from 25 senior area chairs (sacs) (35.7% response rate), 134 area chairs (acs) (30.5% response rate), 510 reviewers (11.4% response rate), and 556 authors (4.07% response rate of all authors foot_1 ). ## tracks and acceptance statistics acl'23 had 26 tracks, most of which have also been offered at other recent nlp conferences. at the suggestion of emnlp 2022 chairs, we kept their separation of "large language models" 3 track from "machine learning for nlp" track. at community requests we added the following tracks: "linguistic diversity" and "multilingualism and cross-lingual nlp". each track had at least two senior area chairs (sacs), who then recruited area chairs (acs) for that track. the full list of senior chairs per track is available at the conference website. 4 internally, in the start system there were also two special tracks: "ethics review" track (which handled the reviews of papers that were flagged for ethical issues), and "conflicts of interest" (coi) track, which handled the papers with which the sacs of the relevant tracks had a coi. acl'23 implemented a hybrid process, in which it was possible to submit papers either directly to the start system (to be reviewed through acl'23 internal peer review process to be described in this report), or commit it through acl rolling review (arr) with reviews already performed at arr. most submissions to acl'23 were direct submissions (4559), and 305 more came through acl rolling review (arr). ## resubmissions among the 4559 direct submissions to acl'23, 754 indicated that they were resubmissions (see fig. 1a ). the biggest "donors" were eacl foot_3 (296), emnlp (258), iclr (103), aaai (52), and acl rolling review foot_4 (39). although the selectivity of top-tier conferences means that the majority of papers are * all information is self-reported, not independently verified, and does not correspond to any specific definition of affiliation, gender, or country (e.g., some authors from edinburgh may elect to list their country as "scotland" rather than "uk".) moreover, acl'23 authors had the option to submit previous reviews as an attachment, but only 243 submissions used this option, which suggests that most resubmitters preferred to have a completely new set of reviewers. arr allows that option within arr, but the arr submissions themselves did not have a high rate of revise-and-resubmit (only 8/305), as shown in fig. 1b . intuitively, one could expect that resubmissions have a higher chance of acceptance, since these are the papers that have received feedback and had a chance to revise. but fig. 1c suggests otherwise. see more analysis in §7.3. ## authors and reviewers at acl'23 we received a record 4864 submissions (4559 direct, 305 from arr) from the total of 13,658 authors, reviewed by 4490 reviewers. this section reviews our recruitment process and the three demographic variables (country, affiliation type, and gender) to which we had access in the global start profiles of all participants of acl peer review process. reviewer recruitment. we initially sent review invitations to the reviewer list which we had received from the organizers of previous conferences. we also required the authors of all submissions to nominate at least one experienced reviewer, whom we also sent invitations. as we elicited reviewer data, we found that for a quarter of our reviewers 7 there is no reliable semantic scholar publication history data that can be used for paper-reviewer matching. for conferences that fully rely on automated paper-reviewer matching based on publication history, this factor obviously sets a bound on their possible performance. often the author pages exist because semantic scholar automatically created them, but the authors did not claim them and did not clean them up, which may result in the addition of publications by namesake authors (e.g. the automatically created profile for "anna rogers" originally had contributions from at least three researchers with that name.) this is particularly worrying because at this point many venues have used this information for paper-reviewer matching, and urged the nlp community to maintain their semantic scholar profiles. we also specifically reminded about this, but still a quarter of our sign-up pool stated that their publication history is not accurate. in addition to this problem, matching based on publication history has the issue with establishing expertise of different authors on on multi-author publications. hence, we developed an alternative matching approach described in §5.2. affiliation types. figure 2a presents the overall distribution of the affiliations of our authors and reviewers (as stated in start profiles). the biggest group of authors, reviewers, and chairs are academic faculty. the second biggest group (by absolute numbers) in all three categories is industry, which is relevant to the recent concerns about the influence of industry on academic nlp research @xcite . furthermore, students form at least 26% of reviewer pool (ph.d. 22.7%, m.sc. 3.3%). this was also our experience as area chairs at other recent conferences, and it highlights the need to continue the reviewer training efforts. ## efforts towards improving review quality this section describes the following steps that acl'23 proposed and implemented within its peer review process to improve review quality: review tutorials ( §5.1), area-contribution-language paper-reviewer matching ( §5.2), flagging of review issues by the authors ( §5.3). the efforts to improve the overall incentives are decribed in §9.2 and §9.3. ## improving decision support for the chairs in addition to the efforts for improving the quality of peer review ( §5), we implemented the following steps for facilitating the decision support by acs and sacs: revised sac and ac guidelines ( §6.1), guidance for assignment checking ( §6.2), match rationales ( §6.3), soundness/excitement scores ( §6.4). ## what factors contribute to acl peer review outcome? here we present the results of statistical analysis of acl'23 data, with the goal of explicating what factors contributed to the final decisions and to the quality of individual reviews. we hope that this process both improves the transparency around chair decision-making, and highlights the potential biases and points of improvement for future conferences. for the new authors, we should explain the general process for the acceptance decisions at acl'23. first, the reviewers contribute their reviews. at the author response the authors see the reviews and have an opportunity to respond: a process mostly intended to clarify any misunderstandings (we disallowed submitting new results). then the acs initiate the reviewer discussion, with the goal to clarify misunderstandings and improve the quality of the reviews. based on the final reviews and their own expertise, they write the meta-reviews and make recommendations for acceptance (main track or findings) or rejection. they are not concerned with the acceptance quotas. their recommendations and meta-reviews (as well as reviews and author response if necessary) are then considered by the sacs, who have the constraint of the target acceptance quota (which we set at about 22% for the main track and 35% for findings). their decisions are based on three main factors: meta-reviews, quotas, and editorial priorities (with case-by-case consideration as needed). if they run out of their quota, they may additionally rank more papers by priority that may be accepted to main/track findings if there is space (e.g., because some tracks did not use their quota fully). the final step is that the program chairs confirm the sac decisions, and try to fit in as many papers of the ranked "maybes" as possible. in our case, that resulted in accepting more findings papers than we originally planned based on prior conferences. ## improving the incentives 9.1 improving reporting incentives for the authors: responsible nlp checklist following the effort started by naacl 2022 and continuted at acl rolling review @xcite , we used the responsible nlp checklist as a way to ensure that all submissions conform to a certain minimum standard of reporting on their reproducibility efforts, data collection principles, and consideration of broader impacts. however, at naacl 2022 and acl rolling review, these checklists are only used internally during peer review. to improve the transparency of nlp research and create a stronger incentive to invest effort in this work, we made the responsible nlp checklists an official part of all published papers. the authors filled out the checklist information in a special form, and we later used that form to generate pdf versions of the checklist, which was appended to every paper pdf for the acl anthology. this change was announced in our call for papers, and we additionally communicated it to the authors. the authors had the opportunity to update the checklist form during the preparation of the camera-ready version of their papers. one modification to the checklist was introducing a mandatory question about ai writing assistance. this was motivated by the introduction of openai's chatgpt (openai, 2022), the precedent of aiassisted scientific paper writing of meta's galactica @xcite , and, more importantly, a massive wave of promotion for ai "writing assistants" shortly before our direct submission deadline. we did not aim to completely ban ai-assisted writing (which does have legitimate use cases such as assistance to non-native english speakers), but to improve transparency: just like with the other ethics-related questions in the checklists, our posted policy required authors to explicitly state what they did. our question and policy were subsequently adopted by acl rolling review. @xcite have reported that the higher rate of "yes" responses to the reproducibility checklist at 4 nlp conferences. given that our checklist includes reproducibility questions, and reproducibility positively correlates with both soundness and excitement, we would expect the responsible nlp checklist to perform the same role. the reviewers themselves were predominantly positive about it: 66.99% rated it as "somewhat useful", 18.13% as "very useful", and only 14.35% -as "not useful". an interesting case is the "catch question" a3 (does your abstract accurately summarize your work?). it drew some criticism as "meaningless bureaucracy", since all submissions should respond "yes" to it. it was actually intended to see that the responders were not just clicking through the checklist. most authors did respond 'yes', but those 2.24% that didn't saw a -25.4 decrease in acceptance rate. we interpret this as suggesting that the sloppiness in filling out the checklist correlates with sloppiness elsewhere in the work. finally, our new question about the use of writing assistants is the only one where the response 'yes' is associated with a decrease in acceptance rate, although not very large. ## recommendations improving logistics. there are several sources of papers to the acl main conference that the program chairs have no control over: tacl, cl, industry track papers, srw papers. this means that the pcs need to ingest four different sources of information with potentially little means of interacting with the relevant authors (in contrast to direct submissions). arr is in a liminal space between direct submissions and these other papers. the timing and format of how the papers enter acl should be standardized.