What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2023)

Abstract: We present the results of the NLP Community Metasurvey. Run from May to June 2022, it elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split in half on the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us to uncover false sociological beliefs where the community’s predictions don’t match reality. Among other results, we find that the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.

## introduction what do nlp researchers think about nlp? are we devoting too many resources to scaling up? do language models understand language? is the traditional paradigm of model benchmarking still tenable? what models are ethical for researchers to build and release? these questions and many more are actively debated in the research community, and views on them are a major factor in deciding what work gets done. understanding the prevalence of different views on these issues is valuable for understanding the trajectory of nlp research and the structure of the field. in addition, communication among researchers often rests on sociological beliefs about these questions: what people think people think. getting these sociological beliefs wrong can slow * work done while at nyu. down communication and lead to wasted effort, missed opportunities, and needless fights. for example, researchers might spend time and effort defending a widely-believed position which they mistakenly think is controversial, or might fail to argue effectively if they appeal to premises which they believe are well-established but are actually contentious or unpopular. many of the ways that the nlp research community gets to know itself-invited talks, panel discussions, social media, etc.-are biased, for example through self-selection towards similar people and amplification of already-prominent and controversial voices. this makes it difficult to get a sense of the community's beliefs as a whole. for these reasons, we believe it is worth trying to objectively assess nlp researchers' views on controversial issues. so from may to june 2022, we conducted the nlp community metasurvey. we present stances, such as currently, the field focuses too much on scaling up machine learning models (q5-1), and ask respondents whether they agree or disagree. then we ask them to predict what percent of respondents who will agree. this gives us insight into the community's object-level beliefs as well as its sociological beliefs, and allows us to identify where the two may be misaligned. this work is directly inspired by the philpapers surveys @xcite , an initiative by philosophers to assess the philosophy community's beliefs about current topics in their field and their sociological beliefs about their professional community. the rest of this document reports the methodology and results of the nlp community metasurvey. our results are descriptive, not prescriptive, as these issues cannot be resolved by majority vote. by necessity, we are covering a subjectively chosen set of questions and reducing many complex issues into simplified scales, but we hope that these results can create common ground for fruitful discussion among nlp researchers. ## methodology survey construction the survey questions are shown in figures 3 and 4 . we present them in thematic sections (e.g., state of the field), phrased as statements expressing a certain opinion (e.g., q1-1. private firms have too much influence in guiding the trajectory of the field). respondents answer on a 4-point scale of agree, weakly agree, weakly disagree, and disagree. we don't include a neutral middle option because our intent is to push respondents to consider which side they stand on; we instruct them to choose weakly agree or weakly disagree if they have even slight preferences for one side or the other (e.g., "depends, leaning negative"). for cases where they truly cannot make a judgment, we include three ## results all agree/disagree questions and their responses are shown in figures 3 and 4 . an extended version of this discussion addressing every question is in appendix d. in the rest of this section, we will discuss some highlights. 4 belief in scaling maximalism is rare and greatly overestimated, but concerns about agi are not uncommon. only a small minority (17%) of respondents agree with the hypothesis that scaling up current systems and methods could solve "practically any important problem" in nlp (figure 3b , q2-1), but this view is perceived as much more popular (at 47% predicted agreement). similarly, a large majority believes that nlp research is focusing too much on scaling up (71% > 58% predicted, figure 4a , q5-1 ). this suggests that the popular discourse around recent developments in scaling up @xcite may not be reflective of the views of the nlp research community as a whole, which may not have bought into the bitter lesson @xcite foot_3 as much as it thinks it has. yet, narrow majorities of respondents regard recent progress in large-scale modeling as progress towards artificial general intelligence (agi; 57%, figure 3c , q3-2), and think agi should be a concern for nlp researchers (58%, q3-1). so while 61% 77% 1-1. private firms have too much influence private firms have too much influence in guiding the trajectory of the field. ## correlation analysis this survey provides us the opportunity to examine the relationships between opinions: which beliefs tend to come together, and which don't? to get a sense of this, we compute pairwise spearman (rank-order) correlations between pairs of questions ( §5.1), and demographic characteristics ( §5.2), and perform a clustering analysis on our results using pca ( §5.3). ## discussion with the nlp community metasurvey, we have put concrete numbers to many contentious issues in nlp: the necessity of expert-designed inductive bias, the importance of agi, whether language models understand language, and more. perhaps more interestingly, we have also made concrete numbers of the community's impressions of these controversies, in some cases confirming what we already believe and in others producing surprises. for example, the idea that mere scale will solve most of nlp is much less controversial (and much less believed) than it is thought to be, and nlp researchers unexpectedly agree that we should do more to incorporate insights and methods from domain sciences, and that we should prioritize problem formulation and task design. interestingly, very few of the issues we ask about (only the necessity of linguistic structure and expert-designed inductive bias, q2-2 and q2-3) are noticeably more controversial than respondents expected them to be. this could be due to biases from the amplification of controversy (e.g., in social media), or it could just reflect mundane biases in respondent predictions, e.g., regression towards the middle of the range (∼50%) under uncertainty. there are other biases to keep in mind when interpreting our results: the united states is overrepresented, and senior researchers and academics probably are as well ( §3, appendix c), not to mention unmeasured population biases based in the personal networks of the authors. many of our questions have multiple possible interpretations, many rely on vague terms like "plausible" or "major concern," and some rely on comparisons to reference points such as the industrial revolution (q3-3) or "specific, non-trivial results from... linguistics or cognitive science" (q2-4) which may carry different implications for different readers. given these issues, it is probably best to view the answers to the questions on this survey as reflecting something between objective beliefs and signaling behavior. agreement or disagreement with a particular statement may indicate where a respondent believes they stand relative to "received wisdom," which would determine what statements would be worth asserting in the context of the status quo; or, a response could be driven by identification with (or rejection of) an already-known ideological camp that the statement is taken to refer to. while these issues affect the way we should interpret the absolute numbers in our results, they should apply equally to the meta-questions, so we believe it is meaningful to compare the survey's actual and predicted results as a way of discovering false sociological beliefs. we hope the results of the nlp community metasurvey can help us update our sociological beliefs to closer match reality, creating common ground for fruitful and productive discourse among nlp researchers as we confront these issues in the course of our work. about the survey's results, and the anonymous reviewers for useful feedback about its limitations. this project has benefited from financial support to sb by eric and wendy schmidt (made by recommendation of the schmidt futures program) and apple. this material is based upon work supported by the national science foundation under grant nos. 1746891, 1922658 and 2046556. any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the national science foundation.