Assessing Combinational Generalization of Language Models in Biased Scenarios (2022)

Abstract: In light of the prominence of Pre-trained Language Models (PLMs) across numerous downstream tasks, shedding light on what they learn is an important endeavor. Whereas previous work focuses on assessing in-domain knowledge, we evaluate the generalization ability in biased scenarios through component combinations where it could be easy for the PLMs to learn shortcuts from the training corpus. This would lead to poor performance on the testing corpus, which is combinationally reconstructed from the training components. The results show that PLMs are able to overcome such distribution shifts for specific tasks and with sufficient data. We further find that overfitting can lead the models to depend more on biases for prediction, thus hurting the combinational generalization ability of PLMs.

## introduction transformer-based @xcite pretrained language models (plms) have enabled substantial performance gains across numerous downstream tasks @xcite . to evaluate plms, existing work largely follows the scheme of sampling training and test data from the same distribution. in reality, given the productivity of human language, humans are widely assumed to interpret new linguistic utterances based on some notion of compositionality @xcite . in this paper, we want to investigate to what extent models are prone to using biases for prediction and how this may affect their performance on unseen instances requiring combinational inference. this relates to their generalization ability, which is regarded as a key challenge in building human-like models @xcite . we propose a figure 1 : example of data induction for 3-way sentiment classification. the top shows two training instances. the below part shows three generated instances (id: in-distribution, cood: combinational out-of-distribution, ood: out-of-distribution). in the table, labels in red are the combinations used for training and id, whereas black and blue labels are used in the test set. blue ones (cood) can be inferred from the training data, while black ones (ood with label 3 in template) cannot be combinationally inferred because the training set does not include such template. method to assess a plm's generalization capacity in classification tasks that require combinational generalization to overcome biases in the training data. specifically, we modify an original dataset by recombining components of training data points to form unseen test data. figure 1 provides a brief illustration of the principle. based on the training data, the model can easily classify the id instances. however, since we introduce a scenario with special hidden biases, a plm that only picks up such training data biases would fail on the cood instances in the test set. to handle those correctly, the combinational inference is required, i.e., drawing conclusions based on smaller fragments of text observed during training. finally, there are also truly challenging genuine ood instances that are not easily combinationally inferrable. details of the hidden bias scenario are given in the bottom part of figure 1 as well as later on in section 3. overall, our results suggest that plms possess excellent generalization abilities and avoid succumbing to the risky form of bias introduced in the training data. however, the performance depends on the task and data size. most prior work assesses plms based on the setting of test and training data stemming from the same distribution. this yields insights on standard in-task or in-domain learning, while in our work we are interested in the type of more generalizable knowledge acquired from the in-task training data. this relates to the robustness of plms, as the model can only do well on our test data if it pays attention to all components of the data rather than falling prey to biases in the training data. combinational generalization. the combinational generalization here refers to the model's ability to properly handle unseen data samples consisting of fragments observed during training, and regard combinational generalization as a part of compositional generalization. some studies investigate the compositional features and inductive biases of neural net models for sequence-to-sequence and generation tasks @xcite , mostly at the phrase level, while we consider encoder models for classification tasks and focus on compositional inference connecting entire sentences. one similar work is r&r @xcite , which also constructs data from fragments of training data. the major difference is that they incorporate the constructed data into training, while we use it for cood evaluation. prompt-based tuning. prompt tuning has been proposed to reduce the gap between pre-training and fine-tuning on downstream tasks @xcite . it often involves adding templates to the data and predicting label names at the position of the [mask] token @xcite . inspired by prompt engineering, our work also involves the use of template engineering. however, we do not invoke them to elicit a plm's prior knowledge, but as a core part of the input semantics. additionally, finding the best templates and label names is not our focus, so we have not investigated automated prompt identification techniques @xcite ), but we demonstrate that our results are coherent across different templates and label names. ## conclusion in this paper, we present a new method to probe the robustness of plms when subjected to biased data. our findings include that (1) plms exhibit combinational generalization; (2) the combinational generalization is affected by the training data and parameter count;