Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers (2023)

Abstract: Neural networks have revolutionized language modeling and excelled in various downstream tasks. However, the extent to which these models achieve compositional generalization comparable to human cognitive abilities remains a topic of debate. While existing approaches in the field have mainly focused on novel architectures and alternative learning paradigms, we introduce a pioneering method harnessing the power of dataset cartography (Swayamdipta et al., 2020). By strategically identifying a subset of compositional generalization data using this approach, we achieve a remarkable improvement in model accuracy, yielding enhancements of up to 10% on CFQ and COGS datasets. Notably, our technique incorporates dataset cartography as a curriculum learning criterion, eliminating the need for hyperparameter tuning while consistently achieving superior performance. Our findings highlight the untapped potential of dataset cartography in unleashing the full capabilities of compositional generalization within Transformer models.

## introduction in recent years, deep learning methods and machine learning infrastructure have made remarkable progress, enabling models to surpass humanlevel performance in numerous tasks. natural language processing (nlp) is at the forefront of this progress. models based on transformers @xcite such as bert @xcite and benchmarks like superglue @xcite led to significant advancements in language modeling and various downstream tasks. however, there is an ongoing debate on whether these models exhibit compositional generalization @xcite @xcite . compositional generalization refers to the ability of a model to combine known parts of a sentence, such as primitive tokens, to generate novel compositions of these primitive elements. it is considered a fundamental aspect of human cognition and linguistics @xcite . in addition to its human aspect, compositional generalization is also crucial for enhancing the robustness and practical use of deep learning models. efforts to understand and improve the compositional generalization abilities of models have gained significant attention lately. researchers have recently explored techniques such as compositional data augmentation @xcite , metalearning (lake, 2019), and structural priors @xcite . additionally, the importance of architectural modifications to capture compositional structures more effectively, such as attention mechanisms @xcite and hierarchical structures @xcite have been investigated recently. in another direction, there is also an increasing interest in studying the compositional generalization abilities of transformers @xcite @xcite . in this study, we take a distinct approach and utilize dataset cartography @xcite to explore how training dynamics can improve the compositional generalization abilities of transformers. dataset cartography is a recently proposed technique that quantifies the variability and confidence associated with instances during training, capturing their ambiguity and difficulty, thereby representing the informational value of each training sample. @xcite demonstrated that it could be used to improve outof-distribution (ood) generalization in models for classification-based natural language inference (nli) tasks. as compositional generalization is inherently an ood task, we hypothesize that harnessing dataset cartography in compositional generalization can provide new insights. diverging from the original cartography setup, we focus on language generation tasks for the systematic generalization problem and propose an experimental setting to apply dataset cartography to a generative task. initially, we train a sequenceto-sequence (seq2seq) transformer model using the complete training set for only a few epochs. throughout the training, the dynamics of each instance are observed and recorded separately. next, we utilize these stored training dynamics to build a curriculum and create a reduced training set by selecting specific samples to fully train the model. our experimental setup has notable challenges beyond the compositional generalization setting, distinguishing it from the setup originally used in @xcite . instead of relying on crowdsourced datasets that are prone to errors and heavily reliant on data quality, we utilize synthetically generated datasets, namely cfq @xcite and cogs @xcite , which are free from such limitations. moreover, these datasets are relatively smaller in size, making it challenging to achieve performances on par with the 100% train set when using smaller subsets. lastly, unlike @xcite , we tackle the complexity of learning the task directly without using pre-trained models. this becomes even more pronounced as the datasets contain non-natural language, rendering pre-training less applicable and learning much harder. finally, and more importantly, as we are dealing with language generation tasks, quantifying how hard a sequence is, is not straightforward. to address this, we base our evaluation by utilizing inverse perplexity (inv ppl), chia @xcite , and bleu @xcite as confidence measures, avoiding the overly strict exact matching strategy. in summary, our paper makes the following key contributions: first, we introduce the novel use of dataset cartography as both a curriculum learning criterion and a sample selection strategy for enhancing compositional generalization. by leveraging dataset cartography, we enable models to deal effectively with the complexities of compositional tasks. second, we thoroughly investigate the effectiveness of various confidence measures for sequences in extracting dataset cartography within the compositional generalization setting. this analysis provides insights into quantifying the difficulty of sequences and leads to the development of robust training strategies. third, through extensive analyses, we demonstrate the significant impact of leveraging training dynamics through dataset cartography on the compositional generalization capabilities of transformer models. our approach yields significant improvements of up to 10% on challenging cfq and cogs datasets, highlighting the effectiveness of our proposed method. @xcite propose a visualization tool named data maps with two dimensions, confidence and variability, which characterizes the informativeness of training instances of a dataset with respect to a model. confidence is calculated as the mean probability of the true label across epochs, whereas variability corresponds to the spread of confidence across epochs, using the standard deviation. therefore, confidence (μ i ) and variability (σ i ) is denoted as follows: ## conclusion transformers are great at language modeling and various downstream tasks, but their ability to achieve compositional generalization compared to humans remains debatable. in this study, we addressed this challenge by demonstrating that selecting a subset of the training dataset using dataset cartography and training models on this subset can enhance model accuracy by up to 10%. we showed that our setup can generalize to different model architectures and natural datasets. moreover, we achieved improved performance by employing a dataset cartography-based curriculum learning without the need for hyperparameter tuning. looking ahead, we anticipate that this research direction promises insights into the necessary syntax, semantics, and structure for informative data instances, informing the development of novel data augmentation strategies and advancing our understanding of deep models' generalization capabilities.