Perturbed examples reveal invariances shared by language models (2024)

Abstract: The rapid growth in natural language processing (NLP) research has led to numerous new models, outpacing our understanding of how they compare to established ones. One major reason for this difficulty is saturating benchmarks, which may not well reflect differences in model performance in the wild. In this work, we introduce a novel framework to compare two NLP models by revealing their shared invariance to interpretable input perturbations targeting a specific linguistic capability. Via experiments on models from the same and different architecture families, this framework offers insights about how changes in models (e.g., distillation, size increase) affect linguistic capabilities. Furthermore, our framework enables evaluation of invariances between commercial black-box models (e.g., InstructGPT family) and models that are better understood (e.g., GPT-2). Across experiments, we observe that large language models share many invariances encoded by models of various sizes, whereas the invariances by large models are only shared by other large models. Possessing a wide variety of invariances may be key to the recent successes of large language models, and our framework can shed light on the types of invariances retained or emerging in new models. We make the code publicly available.

## introduction a key reason for the tremendous progress and adoption of natural language processing (nlp) models has been the ready availability of models that can be adapted to diverse downstream tasks and datasets @xcite . however, with the increasing number of new models, it is difficult to know how new models compare to betterunderstood ones. this is complicated by the fact that standard benchmark datasets are saturating @xcite , and small differences on these may in fact correspond to large differences in model performance in the wild @xcite @xcite . to enable more comprehensive model comparisons, we propose a novel framework for comparing two nlp models by investigating their shared invariance to specific input perturbations. we illustrate the idea of shared invariances with an example. consider a scenario involving social media content moderation, in which a model is tasked with classifying whether a sentence is offensive. now take two sentences that differ only by a synonym, e.g. "this game is a killer, totally blew my mind." and "this game is a slayer, totally blew my mind.". a well-understood model, m 1 , trained for general social media usage, classifies this pair of sentences and other similarly perturbed pairs as non-offensive, and is thus invariant to synonymbased perturbations. we would now like to evaluate a new model, m 2 , which is specifically finetuned for children's content moderation. our framework allows us to determine to what degree m 2 maintains a similar synonym invariance to m 1 , despite possible differences in the predictions of m 1 and m 2 -such as m 1 classifying the sentences as nonoffensive and m 2 as offensive. to systematically measure these shared invariances across various perturbations, we introduce an invariant sample generation approach (see section 3) and propose novel metrics: hard-scope and soft-scope (see figure 1 ). these metrics assess shared invariances independently of the agreement between model predictions, i.e., hard-scope and soft-scope can be high even if two models consistently disagree in their predictions, as long as they remain invariant to the same perturbations. such scenarios present an interesting opportunity for investigation, as they highlight how significant design changes, like finetuning, may not necessarily alter the features a model treats invariant (and irrelevant), despite changing its specific predictions. while evaluating shared invariance is important, not all invariances are created equal: some may be desirable (e.g., invariance to synonym substitution for content moderation) while others may be undesirable (e.g., invariance to word order of image captioning). we enable the evaluation of specific shared invariances via interpretable input perturbations designed to target a specific linguistic capability (e.g., synonym-invariance, typo-invariance). a linguistic capability evaluates a model's competence on a particular aspect of knowledge and understanding required to solve an nlp task by validating its input-output behavior under the corresponding scenario. for instance, the linguistic capability 'synonym-invariance' evaluates whether a sentiment analysis model changes its prediction if the positive verb is replaced by its synonym. hence, the generated perturbations along specific linguistic capabilities enable us to measure shared model invariances along different linguistic capabilities. we demonstrate our proposed framework's utility in deriving novel insights on how changes in models such as distillation and increase/decrease in size affect shared invariances along multiple well-defined linguistic capabilities. we also show how our framework can be used to compare how invariances along different linguistic capabilities evolve over the course of pre-training for a particular model. additionally, we also demonstrate how our framework can enable evaluation of the invariances shared between models that are available as commercial black-box apis (e.g., instructgpt family) and models that are relatively better understood (e.g., gpt-2). across several experiments, we find that while larger language models share many of the invariances encoded by models of varying scale, invariances encoded by large language models are only shared by other large models of similar sizes. our main contributions can be summarized as follows: (1) we propose a novel framework for defining linguistic capabilities w.r.t a reference nlp model to generate interpretable invariant perturbations. (2) we propose two novel measures: hard-scope and soft-scope to measure the degree of shared (behavioral) invariances between two models along a particular linguistic capability. ## related works similarity measures between two neural networks usually operate at two levels of abstraction: output behavior and intermediate layer representations. behavior: many works compare the behavioral similarity between two models (trained for a given task) by evaluating the difference between their average performances on the held-out "test-set" (e.g., iid accuracy, perplexity, etc). for example, previous work has used iid accuracy to evaluate the effect of well-defined design choices such as model architecture and training scheme @xcite , training time constraints (geiping and goldstein, 2022), and latency and memory @xcite . however, recently many researchers have highlighted the limitations of iid test-sets in identifying different failure modes @xcite and have consequently proposed alter-native approaches for rigorous evaluation @xcite @xcite . most relevant to our work, ribeiro et al. (2020) proposed checklist-a methodology for evaluating the behavior of nlp models along general linguistic capabilities that are applicable for many nlp tasks. more recently, la malfa and kwiatkowska (2022) defined linguistic capabilities as symbolic perturbations of an input sentence for a particular task, and evaluated whether a model's predictions for this sentence align with human annotators. while the above approaches can highlight differences between the two models' ability to generalize under the perturbations introduced by a linguistic capability, they perform an indirect behavioral comparison via the human annotators. in this work, we provide a complementary approach that directly evaluates shared behavioral invariances between two models by defining linguistic capabilities with respect to an nlp model instead of a human annotator. representations: numerous works have also proposed methods for analyzing and comparing nlp models based on their internal representations @xcite @xcite . @xcite investigate the representational similarity of nlp models at both neuron and layer-level output to quantify the effects of different design choices across models from both across and within architectural families. @xcite explore the effects of fine-tuning a neural language encoder by comparing representations of a fine-tuned language encoder with its pre-trained counterpart across layers. @xcite proposed a novel measure (stir) to quantify the similarity between representations of two models via measuring their shared invariances. they achieve this by first generating a set of perturbations that don't change the representations of one model and consequently measuring the extent to which the other model's representations are invariant on them. however, this setup is not directly applicable to nlp due to the discrete nature of language input, where representation inversion would lead to perturbations along arbitrary directions in the input space and consequently linguistically inconsistent samples @xcite . we address this by generating invariant perturbations (for a particular model) along well-characterized and interpretable linguistic capabilities by using discrete optimization. finally, while a central theme of our work is also comparing the similarities and differences between two nlp models, we present an orthogonal approach that focuses on behavior. ## methodology measuring shared invariances along an interpretable linguistic capability necessitates the generation of controlled, meaningful perturbations to the input text. however, creating such perturbations end-to-end using gradient descent is impractical, due to the discrete nature of language inputs. to effectively generate these perturbations, one must first define a transformation class of broadly allowed perturbations (e.g., replacing words with their synonyms) based on the linguistic capability (e.g., synonym-invariance) under investigation, along with the constraints to outline specific exclusion criteria (e.g., ignoring stop words). moreover, to automate the generation of these perturbations, we also need a search method to effectively traverse the space of all possible perturbations, and choose the one that maximizes a goal function. therefore, each linguistic capability can be decomposed to four key components: transformation, constraints, goal function, and search method. this characterization is connected to insights in nlp adversarial robustness literature that explore generating perturbations that fools a particular model instead. next, we discuss each component in detail. ## effect of model design choices on shared-invariances in this section, we investigate the effect of different design choices on the invariances shared by two models. thus, we experiment with a range of nlp models varying in training objective (bert @xcite , distilbert @xcite ), and size (bert-tiny, mini, small, medium, base). unless otherwise stated, we finetune all architectures for 5 epochs on stanford sentiment treebank (sst2 a binary sentiment classification dataset) @xcite . sst2 has a train/test split of 67.3k and 872, respectively. we present additional results for different datasets (ag-news) and tasks (language modeling) in the supplementary sec. b and sec. g. we build upon the "textattack" library @xcite to implement several linguistic capabilities based on our requirements. for each capability and reference-model pair we generate the perturbed examples three times and report the average results with standard errors. in this section, we aim to investigate whether a design choice (i.e., distillation) that has a nominal impact on the iid accuracy, preserves shared invariances along different linguistic capabilities. specifically, we fix bert as the reference model & distilbert as the target model and compare shared capabilities along synonym-invariance and typo-invariance. ## relationship of familiar models with black box apis in the previous sections we discussed specialist models that are tuned to perform well on a specific task (e.g., sentiment analysis on sst2). however, in recent years the nlp community has shifted focus towards building more generalist models that can perform a diverse set of tasks when prompted with appropriate instructions and exemplars. however, the state-of-the-art of these models are primarily available in the form of black-box apis, with little information available about their training. we explore how one can quantify the behavioral similarity between models released via black-box apis (instructgpt family) and models that are widely adopted in practice (e.g. gpt-2). we follow the methodology in @xcite for estimating output probability distribution over the task-labels (positive and negative sentiment) from instructgpt models. specifically, we sample the output multiple times for each input and take the mode as the final prediction and its frequency as the probability score for that particular label. we perform all experiments in a zero-shot manner. to reduce costs, we perform experiments on 100 randomly selected samples from the sst2 test-set. the finetuned gpt-2 achieves 96% accuracy on this subset. it cost us ≈ $55 to compute all the results for this section. larger instructgpt models share more invariances with gpt-2: we use instructgpt models text-ada-001, text-babbage-001, text-curie-001, text-davinci-001 that roughly correspond to model sizes: 350m, 1.3b, 6.7b, and 175b respectively @xcite . we note that there's a substantial difference in iid performance between the smaller models (text-ada-001) and larger models (text-curie-001, text-davinci-001). moreover, textada-001 is not only less agreeable to the gpt-2 model, but also seems to be substantially less invariant to perturbations that gpt-2 is invariant on i.e., low hard-scope and soft-scope. interestingly, even though text-curie-001 and text-davinci-001 achieve similar iid accuracy (i.e., 94%) there's substantial differences in their shared-invariances. thus, even though both models seem equivalent based on iid performance, using text-davinci-001 would ensure higher behavioral similarity from the perspective of shared-invariances. also, this result ties back to our previous observations in sec. 4.2 about larger models sharing more invariances. rl based finetuning may retain more invariances: we explore the effect of different finetuning methods for instruction following on sharedinvariances in fig. 4 (right). for this, we perform experiments on text-davinci-001, text-davinci-002, and text-davinci-003 models released by ope-nai. text-davinci-001 is finetuned using supervised learning on human and selected model written demonstrations. while, text-davinci-002 utilizes the same objective function, it's pretrained on a mix of text and code. text-davinci-003 differs from text-davinci-002 by using reinforcement learning for finetuning instead of supervised learning. we note that both text-davinci-002 and text-davinci-003 have similar performances across iid accuracy, agreement rates, and hard-scope. interestingly, there's a substantial gap in their soft-scope values, indicating that even though both models remain invariant on an equivalent number of samples (similar hard-scope), text-davinci-003's output probability distribution is more invariant to perturbations generated along synonym-invariance. ## conclusion we propose a framework for evaluating interpretable shared-invariances between two nlp models by evaluating the degree to which a target model shares behavioral similarity on a linguistic capability defined with respect to a reference model. we conduct extensive experiments to highlight the implications of different design choices (e.g. distillation) and find that shared-invariances tend to be affected more along certain linguistic capabilities than others. furthermore, we also analyze models of different sizes and find that larger target models in general tend to share more invariances. lastly, we demonstrate the use of our framework in analyzing relationships between black-box apis and familiar models.