A State-Vector Framework for Dataset Effects (2023)

Abstract: The impressive success of recent deep neural network (DNN)-based systems is significantly influenced by the high-quality datasets used in training. However, the effects of the datasets, especially how they interact with each other, remain underexplored. We propose a state-vector framework to enable rigorous studies in this direction. This framework uses idealized probing test results as the bases of a vector space. This framework allows us to quantify the effects of both standalone and interacting datasets. We show that the significant effects of some commonly-used language understanding datasets are characteristic and are concentrated on a few linguistic dimensions. Additionally, we observe some “spill-over” effects: the datasets could impact the models along dimensions that may seem unrelated to the intended tasks. Our state-vector framework paves the way for a systematic understanding of the dataset effects, a crucial component in responsible and robust model development.

## introduction in recent years, data-driven systems have shown impressive performance on a wide variety of tasks and massive, high-quality data is a crucial component for their success @xcite . currently, the availability of language data grows much more slowly than the computation power (approximately at moore's law), raising the concern of "data exhaustion" in the near future @xcite . this impending limitation calls for more attention to study the quality and the effects of data. the data-driven systems gain linguistic abilities on multiple levels ranging from syntax, semantics, and even some discourse-related abilities during the training procedures @xcite . the training procedures almost always include multiple datasets -usually there is a "pre-training" phase and a "fine-tuning" phase, where the model devel-figure 1 : an example of the individual dataset effect. this figure shows how two linguistic abilities of roberta, as characterized by the accuracies of two probing tasks: syntactic tree depth and tense, are changed by fine-tuning the sst2 dataset. probing provides convenient frameworks to study the linguistic abilities of dnn systems from multiple perspectives. probing analyses show how the components of dnns demonstrate linguistic abilities @xcite @xcite . the probing results are relevant to the model's ability to the extent that the probing accuracies can predict the model's downstream performance @xcite . these findings make probing classification a promising candidate for setting up a framework to describe the effects of datasets. in this paper, we set up a state-vector framework for describing the dataset effects. we formalize idealized probes, which give the true linguistic ability of the dnn model in a state of training. then, we derive two terms, individual and interaction effects, that describe the dataset effects along multiple dimensions of linguistic abilities. a benefit of our state-vector framework is that it allows a convenient setup of statistical tests, supporting a rigorous interpretation of how the datasets affect the models. the state framework allows us to frame transitions and set up experiments efficiently. many frequently-used datasets have "spill-over" effects besides the purposes they are originally collected to achieve. additionally, the interaction effects are concentrated and characteristic. our framework provides a systematic approach to studying these effects of the datasets, shedding light on an aspect of model developments that deserves more attentions. all scripts, data, and analyses are available at our github repository. ## related works understanding the datasets recent work has investigated various properties of datasets. @xcite used the signals of the training dynamics to map individual data samples onto "easy to learn", "hard to learn", and "ambiguous" regions. @xcite used an aggregate score, predictive v-information @xcite , to describe the difficulty of datasets. some datasets can train models with complex decision boundaries. from this perspective, the complexity can be described by the extent to which the data samples are clustered by the labels, which can be quantified by the calinski-habasz index. recently, @xcite generalized this score to multiple datasets. we also consider datasets in an aggregate manner but with a unique perspective. with the help of probing analyses, we are able to evaluate the effects of the datasets along multiple dimensions, rather than as a single difficulty score. @xcite studied how fine-tuning affects the linguistic knowledge encoded in the representations. our results echo their finding that fine-tuning can either enhance or remove some knowledge, as measured by probing accuracies. our proposed dataset effect framework formalizes the study in this direction. aroca-ouellette and rudzicz (2020) studied the effects of different losses on the dnn models and used downstream performance to evaluate the multidimensional effects. we operate from a dataset perspective and use the probing performance to evaluate the multidimensional effects. @xcite compared two multitask fine-tuning settings, sequential and joint training. the former can reach higher transfer performance when the target dataset is larger in size. we use the same dataset size on all tasks, so either multitask setting is acceptable. ## a state-vector framework this section describes the procedure for formulating a state-vector framework for analyzing dataset effects in multitask fine-tuning. an abstract view of probes there have been many papers on the theme of probing deep neural networks. the term "probing" contains multiple senses. a narrow sense refers specifically to applying post-hoc predictions to the intermediate representations of dnns. a broader sense refers to examination methods that aim at understanding the intrinsics of the dnns. we consider an abstract view, treating a probing test as a map from multidimensional representations to a scalar-valued test result describing a designated aspect of the status of the dnn. ideally, the test result faithfully and reliably reflects the designated aspect. we refer to the probes as idealized probes henceforth. idealized probes vectorize model states training dnn models is a complex process consisting of a sequence of states. in each state s, we can apply a battery of k idealized probes to the dnn and obtain a collection of results describing the linguistic capabilities of the model's state at timestep @xmath0, t 2 , ...t k ]. in this way, the state of a dnn model during multitask training can be described by a state vector @xmath1. without loss of generality, we define the range of each probing result in r. empirically, many scores are used as probing results, including correlation scores @xcite , usable v-information @xcite , minimum description length @xcite , and combinations thereof @xcite . currently, the most popular probing results are written in accuracy values, ranging from 0 to 1. we do not require the actual choice of probing metric as long as all probing tasks adopt the same metric. from model state to dataset state now that we have the vectorized readout values of the states, how is the state s determined? in general, there are three factors: the previous state, the dataset, and the training procedure applied to the model since the previous state. we can factor out the effects of the previous state and the training procedure. to factor out the effect of the previous state, we introduce the concept of an "initial state". the initial state is specified by the structures and the parameters of the dnns. if we study the effects of data in multitask fine-tuning bert, then the initial state is the bert model before any fine-tuning. let us write the initial state as s 0 . based on this initial state, the dataset x specifies a training task that leads the model from the initial state s 0 to the current state s x . to factor out the effect of the training procedure, we assume the dataset x, together with the training objective, defines a unique global optimum. additionally, we consider the training procedure can eventually reach this optimum, which we write as the dataset state, s x . practically, various factors in the training procedures and the randomness in the sampled data may lead the models towards some local optima, but the framework considers the global optimum when specifying the dataset state. ## discussion checking the effects before using datasets considering that the datasets can show spill-over effects when used independently and interaction effects when used jointly, we call for more careful scrutiny of datasets. while the model developers already have a busy working pipeline, we call for model developers to at least be aware of spill-over effects of the datasets. considering the possibly negative effects to the models' linguistic abilities, adding datasets to the model's training might not always be beneficial. ## conclusion we propose a state-vector framework to study dataset effects. the framework uses probing classifiers to describe the effects of datasets on the resulting models along multiple linguistic ability dimensions. this framework allows us to identify the individual effects and the interaction effects of a number of datasets. with extensive experiments, we find that the dataset effects are concentrated and characteristic. additionally, we discuss how the state-vector framework to study dataset effects can improve the dataset curation practice and responsible model development workflow.