Cross-functional Analysis of Generalization in Behavioral Learning (2023)

Abstract: In behavioral testing, system functionalities underrepresented in the standard evaluation setting (with a held-out test set) are validated through controlled input-output pairs. Optimizing performance on the behavioral tests during training (behavioral learning) would improve coverage of phenomena not sufficiently represented in the i.i.d. data and could lead to seemingly more robust models. However, there is the risk that the model narrowly captures spurious correlations from the behavioral test suite, leading to overestimation and misrepresentation of model performance—one of the original pitfalls of traditional evaluation. In this work, we introduce BeLUGA, an analysis method for evaluating behavioral learning considering generalization across dimensions of different granularity levels. We optimize behavior-specific loss functions and evaluate models on several partitions of the behavioral test suite controlled to leave out specific phenomena. An aggregate score measures generalization to unseen functionalities (or overfitting). We use BeLUGA to examine three representative NLP tasks (sentiment analysis, paraphrase identification, and reading comprehension) and compare the impact of a diverse set of regularization and domain generalization methods on generalization performance.1

## introduction the standard paradigm for evaluating natural language processing (nlp) models is to compute correctness metrics on a held-out test set from the same distribution as the training set @xcite . if the test set is large and diverse, this may be a good measure of average performance, but it fails to account for the worst-case performance @xcite . by exploiting correlations in the training data, models work well in most cases but fail in those where the correlations do not hold @xcite @xcite , leading to overestimation of model performance in the wild @xcite . furthermore, standard evaluation does not indicate the sources of model failure @xcite and disregards important model properties such as fairness @xcite . behavioral testing @xcite has been proposed as a complementary evaluation framework, where model capabilities are systematically validated by examining its responses to specific stimuli. this is done through test suites composed of input-output pairs where the input addresses specific linguistic or social phenomena and the output is the expected behavior given the input. the suites can be seen as controlled challenge datasets @xcite aligned with human intuitions about how the agent should perform the task @xcite . in this work, we understand test suites as a hierarchy of functionality classes, functionalities, and test cases @xcite . functionality classes stand at the highest level, capturing system capabilities like fairness, robustness and negation. they are composed of functionalities that target finer-grained facets of the capability. for example, a test suite for sentiment analysis can include the functionality ''negation of positive statement should be negative'' inside the negation class. finally, each functionality is composed of test cases, the input-output pairs used to validate model behavior. for the functionality above, an example test case could be the input ''the movie was not good'' and the expected output ''negative'', under the assumption that the non-negated sentence is positive. though behavioral test suites identify model weaknesses, the question of what to do with such feedback is not trivial. while test suite creators argue that these tools can aid the development of better models @xcite and lead to improvements in the tested tasks @xcite , how to act on the feedback concretely is not discussed. one common approach is fine-tuning on data targeting the failure cases, which previous work has shown can improve performance in these same cases @xcite @xcite . but this practice overlooks the possibility of models overfitting to the covered tests and consequently overestimates model performance. even if one takes care to split the behavioral test cases into disjoint sets for training and testing, models can still leverage data artifacts such as word-label co-occurrences to achieve seemingly good performance that is overoptimistic and does not align with out-ofdistribution (ood) performance. this creates the following dilemma: either one does not use the feedback from test suites for model development and loses the chance to improve model trustworthiness; or one uses it to address model shortcomings (e.g., by training on similar data)-and run the risk of overfitting to the covered cases. prior work (luz de @xcite has addressed this in part by employing structured crossvalidation, where a model is trained and evaluated on different sets of phenomena. however, the analyses have been so far restricted to limited settings where only one task, training configuration and test type is examined. moreover, these studies have not examined how different regularization and generalization mechanisms influence generalization. in this paper, we introduce beluga, a general method for behavioral learning unified generalization analysis. by training and evaluating on several partitions of test suite and i.i.d. data, we measure model performance on unseen phenomena, such as held-out functionality and functionality classes. this structured cross-validation approach yields scores that better characterize model performance on uncovered behavioral tests than the ones obtained by over-optimistic i.i.d. evaluation. our main contributions are: we extend previous work on behavioral learning by exploring two training configurations in addition to fine-tuning on suite data (luz de araujo and roth, 2022; liu et al., 2019): training on a mixture of i.i.d. and suite data; and training on i.i.d. data followed by fine-tuning on the data mixture. this work is not a recommendation to train on behavioral test data, but an exploration of what happens if data targeting the same set of phenomena as the tests is used for model training. we find that naive optimization and evaluation do yield over-optimistic scenarios: fine-tuning on suite data results in large improvements for seen functionalities, though at the same time i.i.d. data and unseen functionalities performance can degrade, with some models adopting degenerate solutions that pass the tests but lead to catastrophic i.i.d. performance. including i.i.d. as well as test suite samples was found to prevent this, mitigating i.i.d. performance degradation-with even improvements in particular cases-and yielding higher scores for unseen functionalities as well. ## beluga beluga is an analysis method to estimate how training on test suite data impacts generalization to seen and unseen phenomena. given an i.i.d. dataset d, a test suite t , and a training configuration χ ( § 3.1), beluga trains on several controlled splits of suite data and outputs scores that use performance on unseen phenomena as a proxy measure ( § 3.3) for generalization. that is, beluga can be formalized as a function f parametrized by d, t , and χ that returns a set of metrics m : @xmath0, t , χ) . by including measures of performance on i.i.d. data and on seen and unseen sets of phenomena, these metrics offer a more comprehensive and realistic view of how the training data affected model capabilities and shed light on failure cases that would be obfuscated by other evaluation schemes. below we describe the examined training configurations ( § 3.1), how beluga optimizes the expected behaviors encoded in t ( § 3.2), how it estimates generalization ( § 3.3), and the metrics it outputs ( § 3.4). ## conclusion we have presented beluga, a framework for cross-functional analysis of generalization in nlp systems that both makes explicit the desired system traits and allows for quantifying and examining several axes of generalization. while in this work we have used beluga to analyze data from behavioral suites, it can be applied in any setting where one has access to data structured into meaningful groups (e.g., demographic data, linguistic phenomena, domains). we have shown that, while model performance for seen phenomena greatly improves after fine-tuning on test suite data, the generalization scores reveal a more nuanced view, in which the actual benefit is less pronounced and depends on the task and training configuration-method combination. we have found the @xmath1. conversely, some methods struggle in the @xmath2.i.d. data, respectively. in these cases, a model both practically aces all tests and fails badly for i.i.d. data, which reinforces the importance of considering both i.i.d. and test suite performance when comparing systems, which is accounted for by beluga's aggregate scores. these results show that naive behavioral learning has unintended consequences, which the @xmath3. there is still much room for improvement, though, especially if generalization to unseen types of behavior is desired. through beluga, progress in that direction is measurable, and further algorithmic improvements might make behavioral learning an option to ensure desirable behaviors and preserve general performance and generalizability of the resulting models. we do not recommend training on behavioral tests in the current technological state. instead, we show a way to improve research on reconciling the qualitative guidance of behavioral tests with desired generalization in nlp models.