Square One Bias in NLP : Towards a Multi-Dimensional Exploration of the Research Manifold (2022)

Abstract: The prototypical NLP experiment trains a standard architecture on labeled English data and optimizes for accuracy, without accounting for other dimensions such as fairness, interpretability, or computational efficiency. We show through a manual classification of recent NLP research papers that this is indeed the case and refer to it as the square one experimental setup. We observe that NLP research often goes beyond the square one setup, e.g, focusing not only on accuracy, but also on fairness or interpretability, but typically only along a single dimension. Most work targeting multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English; and so on. Such one-dimensionality of most research means we are only exploring a fraction of the NLP research search space. We provide historical and recent examples of how the square one bias has led researchers to draw false conclusions or make unwise choices, point to promising yet unexplored directions on the research manifold, and make practical recommendations to enable more multi-dimensional research. We open-source the results of our annotations to enable further analysis.

## introduction our categorization of objects, say screwdrivers or nlp experiments, is heavily biased by early prototypes @xcite . if the first 10 screwdrivers we see are red and for hexagon socket screws, this will bias what features we learn to associate with screwdrivers. likewise, if the first 10 nlp experiments we see or conduct are in sentiment analysis, this will likely also bias how we think of nlp experiments in the future. in this position paper, we postulate that we can meaningfully talk about the prototypical nlp experiment, and that the existence of such an experimental prototype steers and biases the research dynamics in our community. we will refer to this prototype as nlp's square one-and to the bias that follows from it, as the square one bias. we argue this bias manifests in a particular way: since research is a creative endeavor, and researchers aim to push the research horizon, most research papers in nlp go beyond this prototype, but only along a single dimension at a time. such dimensions might include multilinguality, efficiency, fairness, and interpretability, among others. the effect of the square one bias is to baseline novel research contributions, rewarding work that differs from the prototype in a concise, one-dimensional way. we present several examples of this effect in practice. for instance, analyzing the contributions of acl 2021 papers along 4 dimensions, we observe that most work is either clustered around the square one or makes a contribution along a single dimension (see figure 1 ). multilingual work typically disregards efficiency, fairness, and interpretability. work on efficient nlp typically only performs evaluations on english datasets, and disregards fairness and interpretability. fairness and interpretability work is also mostly limited to english, and tends to disregard efficiency concerns. we argue that the square one bias has several negative effects, most of which amount to the study of one of the above dimensions being biased by ignoring the others. specifically, by focusing only on exploring the edges of the manifold, we are not able to identify the non-linear interactions between different research dimensions. we highlight several examples of such interactions in section 3. overall, we encourage a focus on combining multiple dimensions on the research manifold in future nlp research, and delve deeper into studying their (linear and non-linear) interactions. contributions. we first establish that we can meaningfully talk about the prototypical nlp experiment, through a series of annotation experiments and surveys. this prototype amounts to applying a standard architecture to an english dataset and optimizing for accuracy or f1. we discuss the impact of this prototype on our research community, and the bias it introduces. we then discuss the negative effects of this bias. we also list work that has taken steps to overcome the bias. finally, we highlight blind spots and unexplored research directions and make practical recommendations, aiming to inspire the community towards conducting more 'multi-dimensional' research (see figure 1 ). ## finding the square one in order to determine the existence and nature of a square one, we assess contemporary research in nlp along a number of different dimensions. ## counter-examples most of the exceptions to our thesis about the 'onedimensionality' of nlp research, in our classification of acl 2021 oral papers, came from studies of efficiency in a multilingual context. another example of this is @xcite , who show that for low-resource languages, weight pruning hurts performance on tail phenomena, but improves robustness to out-of-distribution shifts-this is not observed in the square one (high-resource) regime. there are also studies of fairness in a multilingual context. @xcite , for example, show significant differences in social bias for multilingual hate speech systems across different languages. @xcite study gender bias in multilingual word embeddings and cross-lingual transfer. gonzález et al. (2020) also study gender bias, but by relying on reflexive pronominal constructions that do not exist in the english language; this is a good example of research that would not have been possible taking square one as our point of departure. @xcite study adversarial debiasing in the context of a multilingual corpus and show some mitigation methods are more effective for some languages rather than others. @xcite studies multilingual toxicity classification and finds that models misinterpret nonhateful language-specific taboo interjections as hate speech in some languages. there has been much less work on other combinations of these dimensions, e.g., fairness and efficiency. hansen and @xcite show that weight pruning has disparate effects on performance across demographics and that the min-max difference in group disparities is negatively correlated with model size. @xcite observe that techniques to make inference more efficient, e.g., greedy search, quantization, or shallow decoder models, have a small impact on performance, but dramatically amplify gender bias. in a rare study of fairness and interpretability, @xcite propose a methodology to interpret which parts of a model are causally implicated in its behavior. they apply this methodology to analyze gender bias in pre-trained transformers, finding that gender bias effects are sparse and concentrated in small parts of the network. ## blind spots we identified several under-explored areas on the research manifold. the common theme is a lack of studies of how dimensions such as multilinguality, fairness, efficiency, and interpretability interact. we now summarize some open problems that we believe are particularly important to address: (i) while recent work has begun to study the trade-off between efficiency and fairness, this interaction remains largely unexplored, especially outside of the empirical risk minimization regime; (ii) fairness and interpretability interact in potentially many ways, i.e., interpretability techniques may affect the fairness of the underlying models @xcite , but rationales may also, for example, be biased toward certain demographics in how they are presented @xcite ; (iii) finally, multilinguality and interpretability seem heavily underexplored. while there exists resources for english for evaluating interpretability methods against gold-standard human annotations, there are, to the best of our knowledge, no such resources for other languages. foot_9 ## contributing factors we finally highlight possible factors that may contribute to the square one bias. biases in nlp education. we hypothesize that early exposure to predominantly english-centric experiment settings and tasks using a single performance metric may potentially propagate further to more advanced nlp research. to investigate to what extent this may be the case, we created a short questionnaire, which we sent to a geographically diverse set of teachers, including first authors from the last teaching nlp workshop @xcite , asking about the first experiment that they presented in their nlp 101 course. we received 71 responses in total. our first question was: the last time you taught an introductory nlp course, what was the first task you introduced the students to, or that they had to implement a model for? the relative majority of respondents (31.9%) said sentiment analysis, while 10.1% indicated topic classification. 12 more importantly, we also asked them about the language of the data used in the year book language task 1999 manning and schütze (1999) english-french alignment 2009 jurafsky and martin (2009) english lm 2009 bird et al. (2009) english name cl. 2013 søgaard (2013) english doc.cl. 2019 eisenstein (2019) english doc.cl. experiment, and what metric they optimized for. more than three quarters of respondents reported that they used english language training and evaluation data and more than three quarters of the respondents asked the students to optimize for accuracy or f1. the choice of using english language datasets is particularly interesting in contrast to the native languages of the teachers and their students: in around two thirds of the classes, most students shared an l1 language that was not english; and less than a quarter of the teachers were l1 english speakers themselves. we extend this analysis to prototypical nlp experiments in undergraduate and graduate research based on five exemplary nlp textbooks, spanning 20 years (see commercial factors. for commercially focused nlp, there is an incentive to focus on settings with many users, such as major languages with many speakers. similarly, as long as users do not mind using highly accurate black-box systems, researchers working on real-world applications can often afford to ignore dimensions such as interpretability and fairness. ## discussion is square one bias not the flipside of scientific protocol? one potential argument for a community-wide square one bias is that when studying the impact of some technique t, say a novel regularization term, we want to compare some system with and without t, i.e., control for all other factors. to maximize impact and ease workload, it makes sense at first sight to stick to a system and experimental protocol that is familiar or wellstudied. always returning to the square one is a way to control for all other factors and relating new findings to known territory. the reason why this is only seemingly a good idea, however, is that the factors we study in nlp research, may be nonlinearly related. the fact that t makes for a positive net contribution under one set of circumstances, does not imply that it would do so under different circumstances. this is illustrated most clearly by the research surveyed in §3. ideally, we thus want to study the impact of t under as many circumstances as possible, but in the absence of resources to do so, it is a better (collective) search strategy to apply t to a random set of circumstances (within the space of relevant circumstances, of course). comment on meta-research. this paper can be seen in the line of other meta-research @xcite @xcite ) that seeks to analyze research practices and whether a scientific field is heading in the right direction. within the nlp community, much of such recent discussion has focused on the nature of leaderboards and the practice of benchmarking @xcite . ## conclusion we identified the prototypical nlp experiment through annotation experiments and surveys. we highlighted the associated square one bias, which encourages research to go beyond the prototype in a single dimension. we discussed the problems resulting from this bias, by studying the area statistics of a recent nlp conference as well as by discussing historic and recent examples. we finally pointed to under-explored research directions and made practical recommendations to inspire more multi-dimensional research in nlp.