Changing the World by Changing the Data (2021)

Abstract: NLP community is currently investing a lot more research and resources into development of deep learning models than training data. While we have made a lot of progress, it is now clear that our models learn all kinds of spurious patterns, social biases, and annotation artifacts. Algorithmic solutions have so far had limited success. An alternative that is being actively discussed is more careful design of datasets so as to deliver specific signals. This position paper maps out the arguments for and against data curation, and argues that fundamentally the point is moot: curation already is and will be happening, and it is changing the world. The question is only how much thought we want to invest into that process.

## introduction the key ingredient behind the recent successes in nlp is transformer-based language models. the paradigm of pre-training followed by fine-tuning on downstream tasks was popularized by bert @xcite , and is actively developed @xcite . in december 2020 the human performance baselines on superglue @xcite were surpassed twice, making the community wonder if it is possible to formulate benchmarks not solvable in this paradigm. however, the successes are not the full story. it is becoming increasingly clear that much of the remarkable performance is down to benchmarks that do not actually require sophisticated verbal reasoning skills due to annotation artifacts and spurious patterns correlating with the target labels @xcite @xcite .the social biases in nlp models are also attracting more attention @xcite @xcite . the "garbage in, garbage out" principle suggests that the situation will not change without a dramatic reappraisal of how nlp data is collected, both for pre-training and task-specific resources. but that seemingly uncontroversial conclusion is at the core of the interdisciplinary tension between nlp understood as a deep learning (dl) application area, and the more qualitative approaches of computational linguistics and ai ethics. how deep that tension goes is illustrated by the recent heated (and sometimes less than professional foot_0 ) debate around "on the dangers of stochastic parrots: can language models be too big? this position paper brings together the arguments for and against curating data foot_1 from linguistic and ethical perspectives ( §2). it makes the case that curation is unavoidable and already happening, and that any data choices that we make, explicitly or implicitly, will affect the real world ( §3). thus the debate is only about how much thought we should put into this process. if we are to at least try to steer it, we have to overcome the interdisciplinary tension and reconsider what counts as "nlp work" ( §4). §5 outlines some policies that could help. ## why curation is inevitable in cognitive and sociolinguistics, one of the methods of studying the linguistic and conceptual reper-toire of a certain individual or a demographic is through collecting a representative corpus of their speech (synchronic or diachronic). that corpus inevitably reflects a particular world view 5 . the differences in these world views are expressed as variation in what kinds of linguistic structures people are likely to use, what they are likely to talk about, what are their presuppositions and social context and stereotypes, to what extent any of that is verbally expressed, etc. some of that variation is idiosyncratic, some attributable to social groups, but even a cursory look at all the variation strongly suggests that there is no "language in general". it is still possible to talk about language at a certain level of abstraction (e.g. "british english" vs the myriad of uk dialects), but only with a good sample representing all the necessary subsets. for example, it would be wrong to construct a "british english" resource based only on london samples, because they do not represent the rest of the country (either linguistically or socioeconomically). a major achievement of corpus linguists are the "national corpora" such as bnc @xcite , painstakingly created to represent a diverse sample of written and spoken genres in a certain geographical region in a certain timeframe, so as to enable studies of that specific variety of language. creating such corpora involves careful sampling, detailed documentation of the domains and speakers that were represented, and much negotiation with publishers for copyright exceptions. a typical corpus for training language models, or really any nlp dataset, is likewise a sample of speech of a certain group of people, who have their linguistic preferences and sets of values. consequently, that sample, whether it is coherent or not, and whether it was collected with any specific intentions, represents a certain "picture of the world". moreover, the purpose of using this data for training is to create a system that would encode that view of the world and make predictions consistent with it. but a typical nlp dataset 6 currently has few specifications of the demographics, dialects, or the range of domains and linguistic phenomena it covers. unfortunately, it does not mean that the 5 this is a key concept in the works of neo-humboldtian scholars: "world image" (weltbild) of weisgerber, "naive picture of the world" (naivnaja kartina mira) of apresyan (1995), and many others. why does it matter? the linguistic and conceptual repertoire of humans is dynamic. our vocabulary, grammar, style, cultural competence change as we go on with our lives, encounter new concepts, forget some things and reinforce others. a key part of that change is the linguistic signals we encounter in communication: on the nativist account children have innate constraints that guide foot_4 their learning from the data they encounter @xcite , and on the usage-based accounts @xcite that process is entirely datadriven. humans can learn the meaning of words from a single exposure @xcite , but there is also robust evidence of frequency effects in language acquisition @xcite , may 09). it is not by accident that the frequency of the vocabulary to be learned is a key variable in language pedagogy @xcite . in short, humans, like dl models, learn from the patterns in the speech that they encounter. and those patterns do not have to come from human speakers anymore: much speech that we will encounter in the future is likely to be synthetic. according to @xcite , gpt-3 is already generating 4.5b words per day in applications such as question answering, summarization, interactive games, and customer support. this cannot but have impact back on the human speakers foot_5 in the following ways: the point to take from all of this is that any mismatch of linguistic and social feature distributions between nlp systems and their users will have some impact on the world, and for the commercial, widely used nlp systems that impact may be significant. so the debate is not about whether we should change the world by making choices about the data: this is happening either way, because even our convenience samples still reflect numerous implicit choices. the debate is only about how much thinking we want to invest into changing our world. this thought is somewhat scary (in what way will children growing up with alexa be different?), but also exciting: the educational opportunities alone could be breathtaking, reaching far beyond the students who are already in a good position to do well in school. we could also create something simplistic, uninspiring, mindlessly entertaining, and/or not-inclusive. that choice is ours. ## what does it mean to "do nlp"? to sum up the above discussion: there are no "neutral", one-size-fits-all textual corpora. there is also no manual that would provide foolproof instructions for collecting a "correct" corpus for any given context. and all of these complications are not even the main problem, right? after all, data only serves the task of creating a model, which is the real contribution of an nlp paper? in theory, the field of nlp is interdisciplinary. in practice, it became something closer to "one of the applied areas of machine learning" rather than "computational linguistics". furthermore, at least as far as graduate students are concerned, it is something performed as an academic exercise, and as such it does not really have to concern itself with its possible effects on the world. the students can hardly be blamed: keeping up with the latest frameworks and architectures is al-ready hard enough. most dl practitioners have neither the training nor time to also do the data work at the level that the linguists and ethicists are calling for. the publication system does not provide the right incentives for that either: modeling nlp work is prestigious and welcomed at top conferences, while data work is "janitorial", less well paid, "under-valued and de-glamorised" foot_6 @xcite . it does not help that there seems to be a systematic miscommunication between the fields. when linguists or ethicists talk about the issues with the current solutions, the practitioners may take it as an accusation that they are not doing a good job, rather than as an invitation to improve things together. likewise, when the practitioners propose new systems, the linguists and ethicists may be frustrated: not by the incremental improvements on leaderboards as such, but by lack of accompanying discussion of what the proposed methods are supposed to do better, and for whom. if anything is to change, we need to overcome this antagonism. here are a few suggestions for how that could be achieved. ## moving forward step 1. understand each other better. the fact is, the ai ethics people are not really out to "cancel" everybody. it is easy to see why they would be frustrated that the social justice issues have never been a priority, terrified at what "move fast & break things" has already done with the social world, and dubious that they just need to wait and change would come. the linguists are not completely useless. chances are, many problems that the dl engineers are having could be fixed if someone was just around to realize that the tokenizer didn't handle the suffixes well. and the engineers are not inherently evil. they just need resources, training, collaborators, time, and better research incentives. instead, they have to churn out papers in 2 months just to stay in the publication race, with no time to dive deeper into what their systems are actually doing. step 2. improve the incentive structure. one way to change the incentive structure that led to the current situation is through conferences. there will be a lot more interest in data work if it becomes more publishable. as of now, the "resources and evaluation" track is something of a poor relative to the "machine learning" track, which in acl 2020 foot_7 attracted nearly 3 times more submissions. most task-specific tracks (question answering, summarization, dialogue etc.) are supposed to receive both engineering and data submissions, but in that setting the interdisciplinary tension may lead to resource papers voted down simply for being resource papers @xcite . @xcite cites an acl 2019 reviewer who complained that "the paper is mostly a description of the corpus and its collection and contains little scientific contribution". we really need to take the type of contribution foot_8 into account in reviewer assignment, into review form design, and into reviewer training programs. we also need to make sure that the resource tracks are consistently offered 12 , with dedicated best paper awards to raise the prestige of this work in the community. some conferences already started to provide reviewer mentoring, double down on ethics, consider what signal they send to companies and students by their best paper awards. we can all help by lobbying program chairs whenever we have a chance, offline and online. a helpful factor is that the ever-increasing size of models is making the state-of-the-art leaderboard chase financially untenable for even well-resourced labs, and they are looking for other outlets. this is a chance for the nlp community to engage more deeply with the phenomena that we are modeling. step 3. educate. the idea that "nlp" means "deep learning" may well arise if it is taught as a one-semester course focusing on the engineering. if the coursework is fully powered by existing resources, it creates the impression that data is not a part of the job. the result is that the students learn that it is entirely possible to just run off-the-shelf parsers without knowing anything about syntax, or do sentiment analysis without knowing anything about pragmatics. and if it is possible to not do more work, why would anyone bother? we need to provide our students with the skills to stress-test their systems and critically examine their data, so as to be able to spot potential issues early on. for that, they will need the basic linguistic theory, the fundamentals of sociolinguistics and pragmatics. likewise, some aspects of psychology (dual processing theories, memory and attention span, cognitive biases, "nudging") are a pre-requisite for designing interfaces not only for annotation projects, but for any kind of interactive nlp systems. and some awareness of the social power structures would help in not propagating the harmful stereotypes. some strategies for building nlp curricula have been discussed at the teach-ingnlp workshop @xcite @xcite @xcite . most importantly, nlp courses need to combat the idea that all the knowledge about the human world is just irrelevant in the age of big data and dl. the "garbage in, garbage out" principle is still relevant. we may be able to sort the garbage and learn from it anyway, but only if we have at least some idea about what kind of garbage we have. step 4. collaborate. large companies and universities provide a significant competitive edge to their authors just in virtue of the in-house collaboration networks they could offer. but it is becoming increasingly easy for everyone to find external collaborations, especially in the world in pandemic lockdown. one opportunity is twitter, used by estimated 40% of emnlp 2020 authors 13 . what would it mean to "collaborate"? at the bare minimum, in an engineering project the linguists and social scientists could help to at least try to characterize the data that was used with something like data statements @xcite . a more ambitious goal would be to involve them early on in the data selection, preparation, and iterative development. ideally, there would be joint formulation of research goals, thinking together about what kind of world we are building. finding collaborators is much easier for established researchers, not only because they are a known quantity, but also because they are already aware of what could be done in an interdisciplinary project. they probably even already know the people who they could ask to join. but the students could use some help, especially those from the less well-connected institutions. they could bene-fit from establishing some kind of skill exchange network, where the students with engineering background could help out in data projects and students with linguistics/social science background could help out in engineering projects. this would probably the best way to ease the interdisciplinary tension, instill respect for each other's expertise, as well as the awareness that nlp is a huge problem that we do not even understand that well, and for which we need all the help we can get. step 5. estimate. the goal of all the above data work is ultimately to enable informed decisions by the public, the ceos, and the policy makers about what kind of world we would live in. one takeaway from the heated debate around @xcite is that if one side in an interdisciplinary debate focuses mostly on the potential benefits of something, and the other mostly on its harms, the stance is likely to become adversarial, and we do not give each other the benefit of the doubt 14 . nevertheless, the people on both sides of the debate are researchers, and they want to make informed decisions. that is only possible through cost-benefit analysis. it is clear that the first step has to be thorough documentation of the data @xcite : this lets us compare the represented population and the population of the target users, and think through the possible harms. however, it is not clear how to weigh the harms against the benefits. at the very least, to make informed decisions we would probably need to know the following 15 : unfortunately, the world is volatile and business plans change all the time. there is so much uncertainty for each of these points that it is not clear how to even start. yet we have to try to come up with a process for working these things out, and eventually develop templates and calculators that developers could use to make estimates for best-, worst-and realistic scenarios. this is an area in which nlp is desperately in need of collaboration with economics, governance and law. in that, again, nlp conferences could take the lead. there could be regular tracks that would incentivize joint publications with experts from these fields. the search for solutions is already going on, but this way nlp community would participate in it rather than just meet with regulation post-factum. to be able to provide meaningful peer review for such work, we would need a mechanism of recruiting external reviewers with the required expertise on as-need basis. ## conclusion our data is already changing the world, and will keep doing so whether we are being intentional about it or not. we might as well at least try: we do want more robust and linguistically capable models, and we do want models that do not leak sensitive data or propagate harmful stereotypes. whether those goals would be ultimately achieved by curating large corpora or by more algorithmic solutions, in both cases we need to do a lot more data work. the current dynamic suggests that this won't happen, unless we overcome the interdisciplinary tensions and turn our conferences into truly shared spaces.