Variationist: Exploring Multifaceted Variation and Bias in Written Language Data (2024)

Abstract: Exploring and understanding language data is a fundamental stage in all areas dealing with human language. It allows NLP practitioners to uncover quality concerns and harmful biases in data before training, and helps linguists and social scientists to gain insight into language use and human behavior. Yet, there is currently a lack of a unified, customizable tool to seamlessly inspect and visualize language variation and bias across multiple variables, language units, and diverse metrics that go beyond descriptive statistics. In this paper, we introduce Variationist, a highly-modular, extensible, and task-agnostic tool that fills this gap. Variationist handles at once a potentially unlimited combination of variable types and semantics across diversity and association metrics with regards to the language unit of choice, and orchestrates the creation of up to five-dimensional interactive charts for over 30 variable type-semantics combinations. Through our case studies on computational dialectology, human label variation, and text generation, we show how Variationist enables researchers from different disciplines to effortlessly answer specific research questions or unveil undesired associations in language data. A Python library, code, documentation, and tutorials are made publicly available to the research community.

## introduction language data is at the core of a large body of work in many research fields and at their intersections. language data is used to train large language models (llms) by natural language processing (nlp) practitioners, but also by linguists and social scientists to analyze human language and behavior. with a tendency in the nlp community to overlook what actually is in the training data of models @xcite , especially at the level of * these authors contributed equally to this work. figure 1 : a high-level overview of the core elements and functionalities of variationist. the tool computes association metrics between any unit in language and potentially unlimited variable type-semantics combinations, orchestrates the creation of interactive charts, and also supports user-defined custom components (⋆). while some language data exploration tools already exist, especially english-centric corpus linguistics tools @xcite , these cannot typically handle different types of textual units (e.g., tokens, bigrams, characters, and more) and multiple variables or combinations thereof, only offering surface-level metrics that are not easily customizable, and providing low-dimensional visualization. on the other hand, modern analysis tools in nlp mainly focus on interpreting model outputs @xcite rather than exploring the language data in itself. variationist aims to fill this gap, offering the chance to researchers from diverse disciplines to easily explore the intersections between variables in textual corpora in a plethora of different configurations in a unified manner (cf. figure 1 ). addition-ally, variationist allows users to plug in their own custom tokenization functions and metrics in a seamless way, opening up an unlimited number of analysis configurations in just a few lines of code, and going beyond english-centric assumptions on what the definition of a unit in language actually is. we demonstrate the flexibility of variation-ist through a set of case studies spanning research questions pertaining to diverse areas of research: computational dialectology, human-label variation @xcite , and text generation. contributions we propose variationist, a highly flexible and customizable tool for allowing researchers from many fields to seamlessly explore and visualize language variation and bias in textual corpora across many dimensions. we release our code, 1 a python library, 2 a detailed documentation, 3 a video presentation, 4 and a set of tutorials. 5 ## tool design in this section, we present the overall design and aim of variationist. in section 2.1 we detail the guiding design principles, whereas in section 2.2 we summarize the core elements and functionalities around which variationist is designed. ## implementation and usage in this section, we present implementation details (section 3.1 and section 3.2) and an example usage of our python library (section 3.3). ## case studies in the following, we scratch the surface of varia-tionist's capabilities by presenting case studies on diverse topics, from computational dialectology (section 4.1) to human label variation (section 4.2) and text generation (section 4.3). three personas with different backgrounds and aims exemplify our case studies: alice, bob, and carol. we then provide ideas for further applications (section 4.4). code for case studies is available in our repository. ## related work there exist many tools for data exploration in literature, especially in the field of corpus linguistics (see @xcite for an overview). however, there is currently a lack of a unified tool to serve diverse research communities that goes beyond descriptive statistics and basic charts, and that handles many variables at once in a simple fashion. the closest work to variationist is hugging face's data measurements tool (dmt; @xcite . however, it does not consider multiple texts and variables in the analysis, and it does not provide customization and flexibility in terms of units, metrics, tokenizers, and charts. varia-tionist serves to fill this gap in literature. ## conclusion we introduced variationist, a modular, customizable, and easy-to-use analysis and visualization tool that aims at helping researchers in understanding language variation and unveiling potential biases in written language corpora across many dimensions. through the case studies of alice, bob, and carol, we showed the potential of our tool in answering different research questions across disciplines. we hope that our work will also be useful to dave, a fourth fictional character who unfortunately looks at the data very rarely before using it, to begin to reconsider the pivotal role of exploring data before using it for training language models.