ASPIRE : Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations (2024)

Abstract: Neural image classifiers can often learn to make predictions by overly relying on non-predictive features that are spuriously correlated with the class labels in the training data. This leads to poor performance in real-world atypical scenarios where such features are absent. This paper presents ASPIRE (Language-guided Data Augmentation for SPurIous correlation REmoval), a simple yet effective solution for supplementing the training dataset with images without spurious features, for robust learning against spurious correlations via better generalization. ASPIRE, guided by language at various steps, can generate non-spurious images without requiring any group labeling or existing non-spurious images in the training set. Precisely, we employ LLMs to first extract foreground and background features from textual descriptions of an image, followed by advanced language-guided image editing to discover the features that are spuriously correlated with the class label. Finally, we personalize a text-to-image generation model using the edited images to generate diverse in-domain images without spurious features. ASPIRE is complementary to all prior robust training methods in literature, and we demonstrate its effectiveness across 4 datasets and 9 baselines and show that ASPIRE improves the worst-group classification accuracy of prior methods by 1% - 38%. We also contribute a novel test set for the challenging Hard ImageNet dataset.

## introduction spurious correlations are unintended associations or biases learned by models between the input image and the target label, often resulting from factors like data selection biases @xcite . the repeated co-occurrence of ## methodology preliminaries. in this section, we provide an overview of our proposed approach. fig. 2 pictorially describes the various steps in aspire. background: snow alternate background: desert foreground: man, dog sled, dogs image captioning a man and his dogs ride a snow wave on a dog sled. change background (to desert) remove foreground object (dogs) llm if language grounding segmentation background foreground identifying spurious foreground objects identifying spurious background non spurious augmentation generation dog sled other class 5 4a non-spurious image from test 3 2 4b label: dog sled generated synthetic augmentations label: dog sled text text2image diffusion image decoder unet step 1 unet step 2 unet step 3 (personalizing text2image generation) base erm 1 retraining of base erm 6 inference extract spurious images 1 we first train a base classifier e using erm on the entire training set and extract images with features that are spuriously correlated to construct d hold . 2 we caption each image in d hold . 3 we feed the caption to a llm and extract the foreground objects and background for each image. 4a we remove one foreground object at a time and predict the class of the edited image e. if e predicts incorrectly, we consider the object as a plausible spurious correlation learned by e for that class. @xcite we edit the image to change its original background with an alternative background suggested by the llm and follow the process to similar to 4a . 5 we personalize a text-to-image diffusion model using edited images from the previous step for the top-k unique items leading to the highest number of wrong predictions. 6 we re-train e using the generated augmentations to obtain ê. let's assume we have a training dataset d @xmath0,y i }, where every group of images belonging to a particular class predominantly has images with co-occurring spurious features, also known as the majority group. d train might have a much smaller number of non-spurious images, or might not, which is also known as the minority group. we do not assume our training dataset to have any group labeling or additional supervision, like labeling for spurious objects. we also have a test dataset d @xmath1,y i } where both groups are represented equally. additionally, we have a model e trained on d train , using naive empirical risk minimization (erm). thus, e would already identify the majority group images in d test with remarkable accuracy; our primary objective is to improve the performance of the classifier on the minority group without hurting the model's overall performance. the next subsections describe each step in detail. the primary objective of this step is to identify one or several unique features per class that are plausible spurious correlations. to achieve this, we build on recent advancements in language-guided image in-painting to remove one object at a time identified in f i followed by allowing e to predict the class of the edited image. if e predicts the image correctly, we do not do anything with that image. if e predicts an image incorrectly, we identify the removed object as a plausible spurious correlation for the class c in the dataset and add the image to a set d synth (which we later use to personalize textto-image generation). additionally, we add the text phrase of the spurious object to another set t synth . precisely, for every fore-ground object f in f i , we first localize the object using grounding dino @xcite , which takes as input the text phrase of f identified from the caption and outputs a bounding box bb for f . this is followed by extracting the segmentation map m for f using segment anything @xcite , which accepts bb as the segmentation prompt. m is then used to remove f from x i using lama image inpainting @xcite . for detailed information on the workings of grounding dino, segment anything, and lama, we request our readers refer to the original paper. to overcome the aforementioned problems and generate in-domain images with the desired nonspurious features, we resort to personalizing a textto-image generation model. specifically, we train stable diffusion using textual-inversion @xcite with samples from top-k phrases in t synth , and their corresponding images in d synth . textualinversion effectively learns concepts and style from a small set of images for each class in d synth by just fine-tuning a single token in the embedding layer (which in our case is just the original class label) without over-fitting the generation model. d synth is the perfect candidate for extracting this small set as it contains non-spurious images, i.e., images without spurious features and concepts. finally, we prompt the fine-tuned model to generate n× diverse samples for d aug . ## experimental setup datasets. to evaluate the effectiveness of aspire, we experiment on 4 benchmark datasets, including waterbirds @xcite , celeba @xcite , spuco dogs @xcite and hard imagenet @xcite . the waterbirds dataset, generated synthetically by combining images of birds from the cub dataset @xcite and backgrounds from the places dataset @xcite , has 4 groups of images in training and testing datasets including waterbirds on water background, waterbirds on land background, landbirds on water background and landbirds on land background. the minority groups for the dataset (groups with the least number of samples in the training set) are waterbirds on land and landbirds on water. the main challenge is correctly identifying the minority groups in the test. for celeba, we perform the hair color prediction task, which has 4 groups of images, including blond and non-blond males and blond and non-blond females. the minority group is blond males. spuco dogs has 4 groups of images, including big dogs in indoor and outdoor settings and small dogs in indoor and outdoor settings. the minority groups are big dogs indoors and small dogs outdoors. the hard imagenet dataset has images from 15 ima-genet synsets and is more complex than the other 3 datasets, does not have group labeling, and has multiple spurious correlations for each class. for more details, we request our readers to refer to @xcite . since the dataset does not have a test set, we contribute a novel expert-annotated test dataset with 25 spurious and 25 non-spurious images per class. the spurious and non-spurious features for each class were inspired by the original paper. more details about dataset statistics and annotation can be found in appendix a.1. baselines. to prove the efficacy of aspire augmentations, we add aspire augmentations to the original training pipeline for various robust training methods proposed in literature. precisely, we employ group dro @xcite , subg @xcite , just train twice (jtt), learning from failure (lff) @xcite , correct-n-contrast (cnc) @xcite , deep feature reweighting (dfr). @xcite and masktune @xcite , to this list, we add the standard empirical risk minimization (erm) baseline, trained using sgd without any additional modifications. additionally, we compare aspire augmentations with augmentations generated using the methods proposed by @xcite and @xcite . pire is only meant to complement methods proposed on the standard framework of fine-tuning vision encoders for image classification hyper-parameters. for training the base erm model, we train the model for 100 epochs with a learning rate of 1e -3 using the sgd optimizer with a weight decay of 1e -4 . for training all other baselines, we use the original hyper-parameter settings proposed by the authors in their original paper. this includes the seed settings and the number of runs for every model. we use just 1 × augmentations of non-spurious images. though this is possible for us as all our current datasets are also annotated with group labels, the number of aspire augmentations to be added can be decided using hyper-parameter search, and we noticed no signs of over-fitting till 3× augmentations (see appendix 1). for top-k, we resort to @xmath2,2,3,4,5}. @xmath3. examples of extracted top-k can be found in figure 4 and appendix 1. for prompting instructpix2pix, we use text @xmath4.5 and image @xmath5.5. prompt in appendix a.6. ## results and analysis ## conclusion in this paper, we present aspire, a novel data augmentation methodology to augment existing datasets with non-spurious minority group images to build robust and de-biased image classifiers. we evaluate aspire on 4 benchmark datasets with 9 baselines and show that aspire augmentations improve the worst-group accuracy of all baselines while maintaining average accuracies.