Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Abstract
Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or
Flamingo, but little is known about the dataset creation processes.
In this work, we introduce a testbed of six publicly available data sources—YFCC, LAION,
Conceptual Captions, WIT, RedCaps, Shutterstock—to investigate how pre-training distributions induce robustness in CLIP.
We find that the performance of the pretraining data varies substantially across distribution shifts, with no single data source dominating.
Moreover, we systematically study the interactions between these data sources and find that combining multiple sources does not necessarily yield better models, but rather dilutes the robustness of the best individual data source.
We complement our empirical findings with theoretical insights from a simple setting, where combining the training data also results in diluted robustness.
In addition, our theoretical model provides a candidate explanation for the success of the CLIP-based data filtering technique recently employed in the LAION dataset.
Overall our results demonstrate that simply gathering a large amount of data from the web is not the most effective way to build a pre-training dataset for robust generalization, necessitating further study into dataset design. Code is available at https://github.com/mlfoundations/clip_quality_not_quantity.
Introduction
Large models pre-trained on web-scale datasets are becoming a cornerstone of machine learning.
For instance, the past two years have witnessed the arrival of several new models such as GPT-3 [12], Chinchilla [33], and PaLM [15] for natural language processing, or CLIP [48], BASIC [46] and
Flamingo [3] for computer vision.
These models exhibit unprecedented generalization capabilities in zero-shot inference, in-context learning, and robustness to distribution shift.
A key ingredient enabling their generalization performance are the large and diverse pre-training corpora that exceed previous datasets by multiple orders of magnitude.
For instance, the training set of BASIC [46] contains 6.6 billion images, which is more than 1,000 times larger than the widely used ImageNet [18] training set from the 2012 competition containing 1.2 million images.
Despite the central role datasets play for pre-trained models, little is known about them, especially for image-text models.
The aforementioned CLIP [48], BASIC [46], and Flamingo [3] all rely on datasets internal to the respective organizations, which is also the case for other models such as DALL-E [49], Florence [63], and ALIGN [35].
In addition, research publications often provide little details on the data collection processes, e.g., the data sources or data filtering mechanisms.
Beyond clear issues such as reproducibility and the potential presence of harmful content, the opaque dataset creation practices also make it hard to identify effective methods for assembling pre-training datasets.
As a result, researchers cannot build on each other’s dataset innovations, which obstructs the incremental research process that has successfully accumulated algorithm and architecture improvements in machine learning models. A more principled understanding of dataset creation will likely enable further progress in the generalization capabilities of pre-trained models.
A basic approach to dataset creation would be to simply train on all available data of a given type. While scaling up training sets has indeed been integral to the recent progress in large models, advances in weak and self-supervision [14, 29, 30, 12, 20] have led to an abundance of potential training data. For instance, the large LAION-5B dataset [2] of 5 billion image-text pairs is itself a subset of about 50 billion images from Common Crawl. This abundance is already exceeding the amount of data models can currently be trained on within a reasonable time, 1 making the aforementioned baseline of using all available data infeasible.
Consequently, deciding what data to train on is becoming increasingly important. In this paper, we take a step towards a better understanding of pre-training data and investigate the impact of dataset design on the generalization capabilities of image-text models.
Specifically, we focus on CLIP, the first large model of this kind that demonstrated remarkable robustness to multiple challenging distribution shifts.
The main questions we ask are: ( i ) How much do different web data sources vary in their induced robustness?
( ii ) Do dataset combinations lead to better robustness?
( iii ) Can filtering with an existing image-text model improve data quality?
We address these questions from both an experimental and theoretical perspective.
On the experimental side, we begin by assembling a corpus of six different datasets from the web, spanning a variety of sources including Flickr, Shutterstock, Wikipedia, Common Crawl, and Reddit.
We then measure the robustness of CLIP models trained on each dataset.
In particular, we compare the zero-shot accuracy of these models on ImageNet and a set of canonical ImageNet distribution shifts.
We find that the robustness induced by each pre-training dataset varies widely, and that sources with careful curation such as Wikipedia do not necessarily outperform those with minimal filtering.
In addition, the datasets cannot be compared along a single dimension: different datasets help with robustness to different distribution shifts.
This in turn motivates studying how combining datasets affects robustness.
We find that a model trained on two datasets does not inherit the robustness properties of both.
Rather, while the model is exposed to both distributions, its robustness interpolates between that of the individual datasets.
This indicates that dataset designers must carefully combine the pre-training data in order to preserve and enhance the robustness of the resulting models.
Building on our empirical results, we introduce a theoretical model to better understand our experimental findings.
Our model is simple enough to allow for mathematical analysis and can still capture some phenomena of real-world, web-crawled datasets.
Specifically, our theoretical model also shows that combining multiple datasets dilutes the robustness of the better data distribution.
Moreover, our model provides a candidate explanation for another interesting phenomenon in the curation of pre-training data: the LAION-400M experiment [52] and follow-up CLIP reproductions [2] demonstrated that filtering a noisy source (Common Crawl) with a CLIP model results in a dataset on which newly trained CLIP models exhibit higher robustness to some distribution shifts.
Paper outline.
We first briefly review related work and the relevant background on robustness to distribution shift (Section 2), before discussing our experimental setup in Section 3.
Sections 4 andthen measure the robustness of CLIP induced by individual data sources and their combinations.
To support these empirical results, Section 6 presents our theoretical analysis.
We conclude with future research directions in Section 7.
Background & Related Work
Vision-language models. Large vision-language models like CLIP and ALIGN have become an active area of research owing to their success on various computer vision tasks [48, 35]. Existing work expands their capabilities by either increasing model and dataset size [46], or by using additional supervision as in DeCLIP [40], SLIP [44], and FILIP [62]. In contrast, we study the effect of pre-training data composition on task performance with a focus on robustness.
In a recent work, Fang et al.
[22] showed that CLIP’s robustness primarily stems from its diverse pre-training distribution and not training set size, language supervision, or contrastive loss functions. However, Fang et al. [22] only conducted experiments with two datasets (YFCC-15M and ImageNetCaptions), one of which contains less than one million images. We take the insights of Fang et al.
[22] as our starting point and expand the range of pre-training datasets to six different sources, each containing at least five million images. This enables us to study the robustness induced by various pre-training sources, and how dataset combinations affect robustness. [Figure]
Distribution shift. Robustness to distribution shift is a long-standing issue in machine learning [58, 47] and has recently received renewed attention as researchers scrutinize the generalization performance of neural networks in greater detail [55, 8, 9, 5, 51, 27, 16, 38]. Similar to CLIP [48], we focus on robustness to natural distribution shifts, where the corresponding test sets contain only unmodified images and are not intentionally perturbed by adversarial examples or synthetic corruptions (such as Gaussian noise or blur patterns) Specifically, we test robustness on ImageNetV2 [50], ImageNet-R [32], ImageNet-Sketch [59], and ObjectNet [6] because prior work has established many baselines for these distribution shifts [56].
Moreover, models robust to these shifts also show improved robustness on other out-of-distribution benchmarks such as WILDS [36, 60].
The robustness literature usually discusses distribution shift in terms of “in-distribution” and “out-of-distribution” data.
This terminology is natural when data from the same distribution as the “in-distribution” test set is used for training or fine-tuning. However, it is unclear what counts as “in-distribution” when models are trained on large-scale, generic pre-training datasets that aim to improve performance on a wide variety of tasks.
To address this issue, we follow the more flexible definition of distribution shift employed in prior work [56, 43] and measure robustness as accuracy difference between two related but distinct test distributions (e.g., ImageNet and ImageNet-Sketch).
The expectation is that the shift between the two test distributions should not affect an ideal robust model, for instance because the shift does not affect the accuracy of humans labelers [53].
Taori et al.
[56] defined this notion of robustness as effective robustness . Radford et al. [48] adopted the effective robustness framework in the evaluations of their CLIP model.
Effective robustness is illustrated in Figure 1 and measures movement towards a perfectly robust classifier which is not affected by the shift between two test distributions.
Prior work [43] has evaluated several hundred models and demonstrated experimentally that changes to model architecture, training set size, training algorithm, and other model-related factors do not change effective robustness in most cases.
In contrast, changes to the pre-training distribution can improve the effective robustness of a model.
This makes effective robustness a useful metric for evaluating the influence of pre-training data sources on robust generalization, since it removes confounders stemming from model hyperparameters and the number of training samples.
The effect of the pre-training distribution on effective robustness is particularly pronounced when models are evaluated in a zero-shot setting: while training on a target distribution can improve accuracy on that distribution [60, 25, 64, 65, 61], this process can deteriorate robustness to distribution shift [48, 60, 4, 46].
The existence of a universal linear trend for accuracies on a pair of test sets has been analytically studied in [43].
For a simple binary classification model similar to our Assumption 1 (see Section 6), convergence to a linear trend is shown with the deviation from the line scaling as O (1 / √ d ) .
However, the analysis in [43] is restricted to a specific stochastic distribution shift for the out-of-distribution test data and a fixed classifier independent of the training data. Hence the dependence of the slope on the training set size, variations in the training methods, and properties of the training distribution cannot be described in their model.
We provide significantly more fine-grained analyses (Theorem 1) that captures all such tradeoffs, which allows us to draw novel insights into how data mixing (Theorem 2) and filtering (Theorem 3) affect model robustness.
Experiment Setup
Model.
We focus on the CLIP model [48] which has demonstrated unprecedented zero-shot performance on a wide range of downstream tasks, as well as robustness to various distribution shifts [43].
Given an image-text pair, CLIP is trained to maximize the cosine similarity between the embeddings of the text and that of the image, relative to the similarity of unconnected image-text pairs.
We use the CLIP implementation from the OpenCLIP GitHub repository [34], with ResNet-50 [28] as the image encoder model.
We vary the pre-training set size and hyperparameters such as number of epochs to obtain different accuracies on each data distribution. Due to compute constraints, the total training set size is at most 15M samples for most of our experiments.
Appendix B contains more training details.
Data.
To study the effects of training distributions on robust generalization, we collect several datasets from publicly available sources. Most of these have been studied in the context of various vision and language tasks in previous work:
• YFCC: We experiment with the 15M subset of the YFCC100M dataset [57] that the original CLIP paper [48] used for dataset ablation studies.
The images and captions are collected from Flickr.
• LAION [52]: The images and corresponding alt-texts come from web pages collected by Common Crawl [1] between 2014 and 2021.
We randomly select a subset of 15M samples to experiment with, and ensure that the accompanying NSFW tags of all chosen images are ‘UNLIKELY’.
• Conceptual Captions [13]: We use CC-12M for our experiments, which consists of images and HTML alt-text from an unspecified set of web pages.
• RedCaps [19]: This dataset contains 12M examples, obtained from 350 manually curated subreddits between 2008 and 2020.
The subreddits are selected to contain a large number of image posts that are mostly photographs and not images of people.
• Shutterstock: 15M images and captions were crawled from the Shutterstock website in 2021.
• WIT [54]: Image-text pairs come from Wikipedia pages.
We use reference description as the source of text data and obtain 5M examples after filtering to include only English language examples.
Appendix A.1 contains an analysis of image and text statistics, as well as randomly selected data samples from each source.
Evaluation.
We pick ImageNet as the reference distribution and evaluate CLIP on four natural distribution shifts derived from ImageNet: (i) ImageNet-V2 [50]—a reproduction of ImageNet test set using the original dataset creation process, (ii) ImageNet-R [32]—renditions (e.g., sculptures, paintings) for 200 ImageNet classes, (iii) ImageNet-Sketch [59]—sketches of ImageNet class objects, and (iv) ObjectNet [6]—a test set of objects in novel backgrounds, rotations and viewpoints withclasses overlapping with ImageNet.
Visualizations of random samples from each distribution shift can be found in Appendix A.2.
Individual Pre-training Data Sources
We first investigate how well a CLIP model trained on each data source would perform under different distribution shifts of interest.
As seen from Figure 2, while all sources yield the same linear trend on ImageNet-V2, some display clear advantages when evaluated on other test distributions.
For example, Shutterstock offers the best out-of-distribution performance on ImageNet-Sketch, while
RedCaps displays the most effective robustness on ObjectNet.
On ImageNet-R, LAION, CC12M and Shutterstock seem to do much better than the rest.
Overall no pre-training data distribution is consistently the most robust across all evaluation settings.
We also measure the data efficiency of each source, i.e., how much the performance would change with more samples from the same source, see Appendix C. Similar to the previous observation, the six pre-training sources display vastly different data efficiency depending on the distribution shifts of interest.
Although LAION and CC-12M exhibit similar effective robustness in all evaluation settings in Figure 2, this analysis reveals subtle differences between these two training distributions in the low-data regimes.
Combining Data Sources
In the previous section, we demonstrate the variability in behavior of different data sources based on the distribution shift at test time.
A natural question then arises from this observation: does combining [Figure] [Figure] multiple sources help improve robustness across some, if not all, test distributions of interest?
We investigate two common approaches of aggregating information from different training sets—input mixing and output ensembling.
In the subsequent discussion, we focus on distribution shifts that bring out significant differences in behavior across the pre-training data sources (e.g., ImageNet-R and ImageNet-Sketch); the full results on all distribution shifts can be found in Appendices D and E.
5.1 Input Mixing
In input mixing, we randomly select and combine samples from multiple data sources to build the pre-training dataset.
Our findings indicate that this exposure to more training distributions doesn’t help CLIP take advantage of the complimentary strengths of each source.
Rather, the effective robustness of the resulting model is less than that of training on the best individual source for each distribution shift.
We start with combining data from two largest data sources in the testbed—YFCC and LAION.
To remove dataset size as a confounder, we fix the total amount of training data at 15M samples, and sample 7.5M image-text pairs from each source.
This data mixture yields a linear trend that lies in between the trends obtained from training on 15M YFCC and 15M LAION datapoints separately [Figure] [Figure]
[Figure] (Figure 3). Even when we remove the constraint on the training set size and use all the data available from these sources (i.e., 30M samples), the same observation on robustness holds, and the resulting linear trend is highly correlated with that of training on the YFCC-7.5M + LAION-7.5M mixture.
The previous set of experiments combines YFCC and LAION data with a 50:50 ratio.
In Figure 4, we find that when this ratio is varied within a fixed budget of 15M datapoints, the robustness linear trends of the corresponding mixtures form a smooth interpolation between the linear trends of training on 15M YFCC and 15M LAION samples separately.
Experiments with different combinations of sources, as well as mixtures of a larger number of sources, can be found in Appendix D.
There we also include results for combining data from CIFAR-10 and CINIC-10 distributions to train ResNets on image classification tasks, where we observe the same interpolation pattern.
We provide a theoretical justification for this phenomenon in Section 6.2.
5.2 Output Mixing
Another common approach to take advantage of different training sets is to combine them at model output level (i.e., ensemble) [21, 39, 26, 23, 7, 11, 24, 45].
Here, we train a CLIP model on each pre-training distribution of interest and combine the logit predictions of all the resulting models with equal weights.
In experiments with mixing YFCC and LAION (Figure 5), the ensembles ofsingle-source-trained CLIP models taken from the same epoch of training, lie on the linear trend of a single CLIP model trained on the combined data.
This holds across different distribution shifts that we consider.
The same observation also applies when we ensemble 6 CLIP models trained separately on the 6 data sources collected, with the contribution of each model being weighted equally.
Refer to Appendix E for more details. [Figure]
We next show that this phenomenon extends to more complex mixtures. The fact that output mixing (i [Figure]
e., ensembling the predictions of CLIPs trained on individual sources) is predictive of the linear trend produced by input mixing (i [Figure] e., training on all the data from these sources) presents an opportunity to estimate model robustness given new data sources, without having to train the model from scratch on the new, potentially much larger, combined dataset.
For example, in Figure 6, assuming access to a CLIP model trained on the YFCC-5M + LAION-5M mixture, and another one trained on CC-5M, the ensembles of these two models across different stages of training share the same linear trend as a CLIP model trained on the YFCC-5M + LAION-5M + CC-5M mixture.
In
Appendix E, we show the results of ensembling 6 CLIP models trained with the 6 data sources we collected, as well as the CINIC-10 + CIFAR-10 ensemble, a uni-modal image classification setting where output mixing accuracies are also predictive of the linear trend of input mixing.
Analysis under Simple Binary Classification Models
Empirically we observe, for e.g., in Figure 2, that for a pair of test datasets ( D 1 , D 2 ) , the corresponding test accuracies after probit transform achieved by various trained models (including different architectures and training algorithms) on the same training dataset D all lie on the same line.
Furthermore, this line includes models trained on only a subset of D [43].
In other words, there is a universal line that is determined only by the training data distribution and the two test distributions, and is independent of which architecture we use, how we train the model, or how many samples we use from the training set.
To explain this phenomenon, we study the following class of trained models parametrized by ( θ ∈ R d , ρ ∈ R + ) representing the training distribution, with n representing the training data size, and ξ ∈ R + representing the variations due to the training algorithm.
6.1 Universality of Accuracy on the Line for Binary Classification
We analyze a simple but canonical binary classification example where each training data is parametrized by its ground-truth linear classifier θ ∈ R d and its Signal-to-Noise Ratio (SNR) ρ 2 ∈ R + .
Concretely, a training dataset D n,θ,ρ = { ( x i ∈ R d , y i ∈ ± 1) } ni =1 is a set of n i.i.d.
paired samples from a joint distribution P θ,ρ defined as follows.
Assumption 1 (Data distribution) .
We define a joint distribution ( x i , y i ) ∼ P θ,ρ as ( i ) y
i = ±uniformly at random, and ( ii ) x i = y i θ + ( ∥ θ ∥ /ρ )
z i where the noise z i is zero-mean and has independent entries with variance one.
For an observation ( x i , y i ) , we refer to ∥ θ ∥ 2 as the signal power and ∥ θ ∥ 2 /ρ 2 as the corresponding noise power with SNR ρ 2 .
Assumption 2 (Trained model distribution) .
We consider a (random) linear model parameter ˆ θ n,ξ ∈ R d that predicts a binary label sign( ⟨ x, ˆ θ n,ξ ⟩ ) for a test example x ∈ R d .
We assume that the random model ˆ θ n,ξ = θ + ( ξ ∥ θ ∥ / ( ρ √ n ))
z ∈ R d , trained on D n,θ,ρ is unbiased, E [ˆ θ n,ξ ] = θ , and that z has independent entries with variance one each.
The randomness comes from the training data as well as any internal randomness in the training algorithm.
The parameter ξ ∈ R + captures the variations in the resulting model distribution due to changes the training algorithm.
Concretely, one canonical example of a trained model is ˆ θ n,ξ = (1 /n )
(cid:80) ni =
y i x i .
It follows that ˆ θ n = θ +( ∥ θ ∥ / ( ρ √ n ))
z with ξ = 1 .
Hence, ξ measures the randomness of the trained model relative to this simple training algorithm.
We analyze the resulting accuracy when evaluated on two test distributions P θ 1 ,ρ 1 and P θ 2 ,ρ 2 as defined above, with Acc θ 1 ,ρ 1 := P θ 1 ,ρ 1 { sign( ⟨ X, ˆ θ n,ξ ⟩ ) =
Y } , and Acc θ 2 ,ρ 2 defined similarly.
In particular, we are interested in how the accuracy pair (Φ − 1 (Acc θ 1 ,ρ 1 ) , Φ − 1 (Acc θ 2 ,ρ 2 )) behaves as we vary the sample size n and as we vary the training algorithm represented by ξ .
Here, Φ − 1 : [0 , 1] → R is the inverse of the CDF of a standard Gaussian distribution, defined as Φ( t ) =
P ( z ≤ t ) where z ∼ N (0 , 1) .
Φ − 1 is also called the probit function.
This choice of mapping the accuracy with the probit function is critical in getting the linear relation, which we will explain in Remark 1.
Theorem 1 (Universality of accuracy on the line) .
Under Assumptions 1 and 2, asymptotically as the dimension d grows linearly in the sample size n such that lim d →∞ n/d = α 2 , we have
[Equation]
and lim
[Equation]
Further, under Assumption 5 in the Appendix, for some universal constant c > 0 ,
[Equation]
We provide a proof in Appendix F.1.
This analysis implies that for any training sample size n and any (variation due to the) training algorithm ξ , the resulting accuracy pair after probit transform lies on a universal line determined by Eq.
(2), and the slope only depends on the two test distributions ( P θ 1 ,ρ 1 , P θ 2 ,ρ 2 ) and the training distribution P θ,ρ .
The similarity between the training data and each test dataset is captured by the angles: cos( θ 1 , θ ) and cos( θ 2 , θ ) .
More similar training data achieves a larger test accuracy.
The hardness of the test distribution is captured by the SNR of each test dataset: ρ 1 and ρ 2 .
We emphasize that this line is universal in the sense that the slope does not depend on the sample size n and training algorithm parameter ξ .
We are interested in the regime where n scales linearly in d and both are large such that the second term in the above equation is negligible.
Remark 1.
Why do we get the accuracy-on-the-line phenomenon?
The prediction of the trained (random) linear model on a random test data point X involves the inner product, which performs a natural spatial averaging over the d coordinates.
Since the noise across the coordinates is independent and has bounded variance, the central limit theorem applies.
The resulting error has a Gaussian tail.
Hence, the probit mapping, Φ − 1 (Acc θ 1 ρ 1 ) , is critical in translating accuracy into the relevant mean to standard deviation ratio: cos( θ 1 , θ ) ρ 1 ρ √ n/ ( ξ √ d ) .
This is consequently important for getting the universal linear relation, because irrelevant parameters, n and ξ , cancel out in the slope of Φ − 1 (Acc θ 2 ρ 2 ) /
Φ − 1 (Acc θ 1 ρ 1 ) . Refer to the proof of Theorem 1 (Appendix F.1) for more details.
Remark 2.
Intersection at the random guess: Previous work [43] that considered models with a wide range of accuracies has found that all lines intersect at a point corresponding to “random guess", which in this binary example is (Φ − 1 (1 / 2) , Φ − 1 (1 / 2))
= (0 , 0) .
Our theoretical analysis is consistent with this empirical observation: all universal lines intersect at (0 , 0) up to a small additive error scaling as O (1 / √ n ) .
6.2 Input Mixing Yields an Intermediate Slope
Figures 3 and 4 show that when a model is trained on samples combined from two datasets, the resulting robustness trend lies in between the trends achieved by individual datasets.
We show that this finding is universally true under our current setup in Theorem 2.
Note that for the linear models we consider, input mixing (combining training sets) and output mixing (combining model outputs) are equivalent.
Assumption 3.
Consider two training datasets D n 1 , ˜ θ 1 , ˜ ρ 1 and D n 2 , ˜ θ 2 , ˜ ρ 2 of sizes n 1 and n 2 from distributions P ˜ θ 1 , ˜ ρ 1 and P ˜ θ 2 , ˜ ρ 2 as defined in Assumption 1.
Separately training on individual datasets gives two models ˆ θ n 1 ,ξ 1 ( D n 1 , ˜ θ 1 , ˜ ρ 1 ) and ˆ θ n 2 ,ξ 2 ( D n 2 , ˜ θ 2 , ˜ ρ 2 ) as defined in Assumption 2.
A model trained on a combined (and possibly subsampled) training dataset D n ′ 1 , ˜ θ 1 , ˜ [-⇢ 
 ′ 2 , ˜ θ 2 , ˜ ρ 2 is represented by ˆ θ n ′ 1 + n ′ 2 ,ξ ( D n ′ 1 , ˜ θ 2 , ˜ ρ 2 ∪ D n ′ 2 ˜ θ 2 , ˜ ρ 2 ) =
¯ θ + ( ξ ∥ ¯ θ ∥ / (¯ ρ (cid:112) n ′ 1 + n ′ 2 ))
z where ¯ θ = ( n ′ 1 ˜ θ 1 + n ′ 2 ˜ θ 2 ) / ( n ′ 1 + n ′ 2 ) , ¯ ρ = ∥ ¯ θ ∥ / (cid:113) ( n ′ 1 ∥
˜ θ 1 ∥ 2 /ρ 21 ) + ( n ′ 2 ∥
˜ θ 2 ∥ 2 /ρ 22 ) , and z ∈ R d is a zero-mean random vector with independent entries each with variance one.
Again, a canonical example of a trained model is ˆ
[Equation]
We then have ˆ θ n
[Equation]
=
[Equation]
It follows from applying Theorem 1 to the model trained on the combined data (Assumption 3) that the resulting linear trend achieves
[Equation]
We show that the slope obtained from training on the combined dataset lies between those obtained from training on the two datasets separately.
The proof could be found in Appendix F.2.
Let Slope 1 :=
Slope(ˆ θ n 1 ( D n 1 , ˜ θ 1 , ˜ ρ 1 )) and Slope 2 be defined similarly.
Theorem 2.
Under Assumption 3, Slope 1 ≤ Slope (cid:0)
[Equation]
6.3 Filtering Data to Improve Robustness
The input mixing analysis in Section 6.2 suggests a filtering strategy to improve robustness.
Assumption 4 (Gaussian distributions for filtering) .
Consider a training dataset D n,θ train ,ρ of size n draw i.i.d.
from ( X, Y ) ∼ P θ train ,ρ where Y = ± 1 uniformly at random and X | Y ∼
N ( Y θ train , ( ∥ θ train ∥ /ρ ) 2 I ) .
A model trained on an unfiltered D n,θ train ,ρ is denoted by ˆ θ unfiltered =
(1 /n ) (cid:80) ( x i ,y i ) ∈D n,θ train ,ρ x i y i .
Note that ˆ θ unfiltered ∼ N ( θ train , ( ∥ θ train ∥ / ( ρ √ n )) 2 I ) .
The model is evaluated on two test datasets: in-distribution (ID) and out-of-distribution (OOD), each with an isotropic Gaussian noise: ( X, Y ) ∼ P θ ID ,ρ ID , where Y = ± 1 uniformly at random and X | Y ∼ N ( Y θ ID , ( ∥ θ ID ∥ /ρ ID ) 2 I ) , and P θ OOD ,ρ OOD is defined similarly.
Let Slope( θ ) :
= cos( θ
OOD ,θ ) ρ OOD cos( θ ID ,θ ) ρ ID and assume that all models achieve better ID accuracy than OOD accuracy, i.e.,
Slope ( θ ) < 1 .
A model is more robust if the slope is closer to one.
Suppose we have access to a pre-trained model ˆ θ pretrained that achieves better robustness than the model ˆ θ unfiltered trained on the unfiltered data, i.e., Slope(ˆ θ unfiltered ) <
Slope(ˆ θ pretrained ) ≤ 1 .
We want to use the pre-trained model to filter the data D n,θ train ,ρ and achieve better robustness.
We consider a generic family of filtering strategies that subsamples each data point ( x i , y i ) ∈ D n,θ train ,ρ with probability that is monotonically non-decreasing with its correlation to the pre-trained model, i.e., P (( x i , y i ) passes the filter )
∝ h ( ⟨ x i y
i , ˆ θ pretrained ⟩ ) for some monotonic scalar function h : R →
R + .
We analyze the model ˆ θ filtered = (1 /m ) (cid:80) ( x i ,y i ) ∈ filtered dataset x i y i .
The next theorem shows that any such filtering strategy improves robustness, and the full proof can be found in Appendix F.3.
Theorem 3.
Under Assumption 4, if Slope(ˆ θ unfiltered ) <
Slope(ˆ θ pretrained ) ≤ 1 then any model ˆ θ filtered that is trained on the above family of filtered datasets achieves better robustness than the model trained on unfiltered data: Slope(ˆ θ unfiltered ) <
Slope( E [ˆ θ filtered ]) ≤
Slope(ˆ θ pretrained ) , where Slope( E [ˆ θ filtered ]) denotes the slope of the linear trend of a model trained on the filtered data.
Remark 3.
Relevance to empirical practice: The data filtering approach currently employed for the
LAION dataset [52] motivates our theoretical setup.
The authors of LAION first retrieved a large number of image-text pairs from Common Crawl [1] and then computed the cosine similarity between the image and text embeddings using one of the CLIP models originally introduced by Radford et al.
Next, the authors removed all pairs with similarity less than 0.3 from the dataset.
While it was initially unclear if filtering the noisy Common Crawl source with an existing CLIP model would lead to a dataset with good generalization properties, experiments with medium-scale models on LAION so far demonstrate that the resulting models are comparable to the original CLIP models [2].
Our theorem 3 does not provide a full justification for the filtering process described above, but it offers theoretical evidence that filtering a noisy data source with an existing robust model can indeed improve the effective robustness of the resulting dataset.
It remains to be shown empirically that the slope of the linear trend exhibited by the CLIP model from [48] is larger than that of a model trained on the Common Crawl data pool without filtering.
If this assumption holds, we can expect that training on data with better text-image alignment according to the original CLIP model, will improve robustness compared to training on unfiltered data from Common Crawl.
Conclusion
Motivated by CLIP’s robustness to distribution shift, we introduced a testbed of six image-text datasets to study the influence of pre-training data on the robustness of multimodal models such as CLIP.
We analyzed the interactions of these datasets through both experiments and theoretical analyses, finding that simply combining multiple datasets dilutes the robustness of the best-performing one.
Moreover, we offer an explanation for the potential benefits of the data filtering process currently used for LAION.
Our findings suggest that training on a carefully selected subset of data may provide more robustness than training on simply more data, e.g. obtained by combining multiple data sources. To summarize, our results lead to the following recommendations for dataset design: • On the evaluation front: building a pre-training dataset requires multiple robustness test sets, as different pre-training datasets exhibit vastly different behaviors across distribution shifts. • On the dataset design front: each candidate data source for the overall pre-training dataset should be evaluated separately for its generalization and robustness properties.
The final pre-training dataset should then contain more samples from the higher-quality pre-training sources. Creating a more “diverse” pre-training dataset by randomly sampling from all candidate sources does not result in a better dataset.
Assuming access to models separately trained on each source of interest, output ensemble is a good predictor of the effective robustness obtained from input mixing, and hence can significantly reduce the time to search for the right mixing strategy. • Filtering a noisy data source with an existing robust model can improve the generalization properties of the resulting dataset. The broad societal implications of image-text models and web-crawled multimodal datasets have been extensively discussed by Radford et al. [48] and Birhane et al.
[10] respectively. As these models continue to grow in size, full-scale experiments are becoming prohibitively expensive.
This necessitates small, systematic experiments that allow reliable extrapolation to larger scales.
Since effective robustness is independent of model size and quantity of training data, we hope that our testbed and results can serve as a first step towards building multimodal pre-training datasets in a principled manner.
There remain interesting open questions about finding better ways to filter and combine samples from multiple data distributions, or how to build the most robust dataset given a fixed set of data sources.
We look forward to addressing these in our future work. Limitations.
The pre-training datasets we experiment with range from 5M to 15M samples.
While this lags behind current state-of-the-art settings that large image-text models are often trained on, we believe our findings also hold for models with higher accuracy for the following reasons:
• The original CLIP paper [48] offers evidence showing that CLIP’s performance exhibits reliable linear trends across compute scales (Figure 9 of [48], which includes the ResNet50-encoder that our paper works with) and performance on natural distribution shifts (Figure 14 of [48]). In particular, the linear trends in this Figure 14 are analogous to the linear trends in our scatter plots. • In the OpenCLIP GitHub repository [34] that seeks to match the performance of the original CLIP paper, the authors there also note that their models that were trained on 15M samples or fewer share the same effective robustness trend as OpenAI’s CLIP models (trained on 400M samples). Refer to their Why are low-accuracy CLIP models interesting?
remark for more details.
• Previous work [43] has experimented with training on randomly sampled subsets of the training dataset to produce models covering a wide range of accuracies, and found that these models lie on the same effective robustness trend.
This includes models trained on only 1% of the original data, which matches the scale of our experiments (5M-15M samples) relative to the scale of the original CLIP paper (400M samples).
The focus of our work is to study the behavior of a diverse set of data sources, and performing full-scale experiments on 6 datasets (and their combinations) would be extremely expensive.
This prompts us to leverage publicly available web-crawled datasets, and study the behavior of our trained models through the effective robustness framework, which has been shown to be scale-invariant.
Acknowledgments and Disclosure of Funding
We would like to thank Tiffany Cai, Nicholas Carlini, Tatsunori Hashimoto, Jong Wook Kim,
Hongseok Namkoong, Alec Radford, Rohan Taori, and Thomas Wolf for helpful discussions while working on this paper.
We also thank Tiffany Cai and Jack Gallagher for their help with the dataset collection.
HuggingFace has provided generous assistance in the form of compute resources that made many of our experiments possible.
In particular, we would like to thank Leandro von Werra and
Victor Sanh for their help with setting up the compute infrastructure and their constructive feedback on the dataset testbed.
This work is supported in part by Open Philanthropy, the Allen Institute for
AI and NSF grants CNS-2002664, IIS-1929955, DMS- and CCF-2019844 as a part of NSF
Institute for Foundations of Machine Learning (IFML)
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] see Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] we study publicly available data sources (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] see Appendix F.1, F.2, F. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] , our models were trained using code from open_clip GitHub repository, which is publicly available [34] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] see Appendix B (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] see Appendix C (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] see Appendix B 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] for datasets that have been published, we refer readers to the corresponding publications.
For Shutterstock, the license follows what is stated on the website (see Section 3).
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating? [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Dataset Details A.1 Pre-training Datasets
[Table]
ShutterStock
[Table]
To get a better understanding of the diversity of different data sources
[Figure]
we analyze the distributions of caption lengths, image sizes and image aspect ratios for a set of 10,000 samples randomly selected from each source: [Figure] [Figure] [Figure]
Below we also show some examples of image-caption pairs randomly selected from each data source:
Cody This picture is #5 in my 100 strangers project.
Find out more about the project and see pictures taken by other photographers at www.100Strangers.com.
Dan with the Man-Purse Dan Budiac, metrosexual
Cuffless To link or not to link, that is the question...
Dixie Union Chapel, 1836 Replaced a wooden structure that was built in
ATWS Slide presentation from Sandra Carvao WTO_0288 Slide presentation from Sandra
Carvao Deputy Chief Market Intelligence and Promotion Department, World Tourism...
Web 2.0 Expo 2010 - San Francisco Please feel free to use this picture in your blog, website or presentation, in accordance with the stated Creative Commons and c...
Look out, King Vidor Jesse behind the lens.
Jodhpurs and bullhorn not included.
Liverpool, England, United Kingdom Albert Dock - Liverpool, England, United Kingdom
Mariposa Butterfly
THATCamp Computational Archaeology August 2012, University of Virginia
{+Harry Potter and the Half Blood Prince Release Party / September 
+}
They had to release this thing on my anniversary?
Follow the flag to get your book.
Auntie's ...
Nicholas Yeager Co. C, 1st
Arkansas Infantry
John Paul the Great St Patrick's Cathedral Charlotte NC May 20, 2007.
We were there for the 33rd Annual Rosary Rally.
100 words for snow and ice A plethora of hues and textures
GNP: St. Mary Prom
We had an employee prom in August.
Great times in St. Mary :)
Ice at One Mile Low of 22 degrees on Tuesday, December 8th, 2009.
Leather Pencil Case - Common Room PH
[Table]
pants
[Table]
Liderazgo
[Table]
Andrea
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Wholesale toy candy machine for sale - Group buy The same Mini grabbing music clip candy machine small egg twisting machine grabbing childrens intellectual toys Tug Boat Model available on Turbo Squid, the world's leading provider of digital models for visualization, films, television, and games. Tug Boats, Motor Boats, B...
Real wedding RAC Epsom on the English Wedding Blog with Murray Clarke Photography (62)
Cat smoking relax it's just the bud not corona virus vintage s Hoodie
The North Hill <PERSON> is a made in France t-shirt
Cake on a white plate, presentation.
The swimming pool at or near Maca Villas and Spa
Weighing bananas in the supermarket.
Man weighing bananas on the scales in the supermarket, close-up view with no face stock images
The Legend of Zelda Breath of The Wild Link Costume
The Green Elite Wolf Hoodie
Farmer with basket vegetables, isolated in a round frame, contour drawing, icon, logo, coloring, black and white vector.
Illustration, outline cartoon drawing sto...
Students in waders stand in ocean water with a large floating net.
The Girls Rooms logo design type logo layout branding design
Each page explains an aspect of the product, with easy navigation and animation to help drive the point across.
[Figure]
Every Day Is Another Chance Sun Women's Cotton Modal Jersey Tank Dress california dogface butterfly!
decent knife at a truck stop?
maybe if it's from buccee's!
thought i lost my columbian spotted pleco days ago.
that is till i saw this.
here is my small collection as of today carbon snoozy, sunny, snuggly saucissons.
these naturally forming feathers of ice.
chiquita biting more than she can chew the living room of an industrial loft, located in an old building from 1928, in budapest, hungary. designed by golovach tatiana and andrey kot.
tapas - empanadas, potato croquettes, patatas bravas, breadsticks & melon wrapped in serrano ham, ciabatta bread with garlic butter with two types of spanish saus... what is this mold in my basil. appeared overnight much smaller than the rest of your creations
but i decorated this cake earlier.
a newspaper i found on thanksgiving vacation.
can you spot the error to?
sunrise in eastern north carolina.
those aren't mountains, they are clouds i rolled my truck last night, and found this penny while cleaning it out this morning.
the stage in the auditorium of an old high school.
detroit, mi.
Agathosma apiculata
Eight pence note (1778), engraved and printed by Paul Revere
Anachronous[a] map of the Dutch colonial Empire Light green: territories administered by or originating from territories administered by the Dutch East
India Com...
An experiment from William Harvey's de Motu Cordis,
British Army's counter-insurgency campaign in the British controlled territories of
South Arabia,
Deepa Shree Niraula in the left
Aerial view of Iguaçu falls.
Teodor Jeske-Choi ski
Seymour at Wolfgang's nightclub, San Francisco, April
Koch at the commissioning of USS Lake Champlain in New York City, August 12,
An 1862 greenback five-dollar bill
While a member of the Cleveland Indians in 1920, Elmer Smith became the first player to hit a grand slam in the World Series.
He later moved down to play two seas...
HLA-A
One of the violins in the Stradivarius collection of the Palacio Real, Madrid,
Spain
Monte Cimone
Nominees Harrison and Tyler
Modern art.
Colorful contemporary artwork.
Color strokes of paint.
Brushstrokes on abstract background.
Brush painting.
Vector hand-drawn eucalyptus plant isolated on a white background
Abstract kaleidoscope background.
Beautiful multicolor kaleidoscope texture.
Unique kaleidoscope design.
New Year red background with Christmas balls.
Vector illustration.
Greece Santorini island in Cyclades, traditional detail sights of colorful and white washed traditional houses and caldera sea in background
Happy birthday golden text on the background of red and silver gifts on a bokeh background
Friends taking a selfie at a party
Payment Failure Icon
In Trendy Style Isolated Background
Summer Orchid Care
Reflections on a car parked in the street
Austrian Museum of Applied Arts in Vienna, the MAK
Set of round colorful vector shapes.
Abstract vector banners.
Design elements.
Vector illustration.
A panoramic view with village houses and cows grazing on a river bank
Abstract color background, illustration paint colors pallette with brush
Jerusalem,Israel-May 11,2018:
The palmers on Holy Sepulchre Church.
Anointing Unction
Stone Where Jesus Body Was Wrapped Church of the Holy Sepulcher Jerusalem Isra...
A.2 Test Distributions
Figure 16 illustrates the four distribution shifts that we use for evaluating the quality of CLIP features after pre-training on different data sources.
[Figure]
al
[Figure]
Hendrycks et al.
[Figure]
B Training Details
Our implementation closely follows the training code from OpenCLIP GitHub repository [34].
When training CLIP from scratch on each of the pre-training datasets, unless otherwise mentioned, we use AdamW optimizer [42] with default PyTorch parameters β 1 = 0 .
9 , β 2 = 0 .
999 , ϵ = 10 − 8 , (per
GPU) batch size 128 and weight decay of 0.1.
For learning rate, we start with a learning rate of 10 −and apply a cosine-annealing learning rate schedule [41] with 5,000 steps.
We use the same data augmentations as in [48].
Models then undergo distributed training on 8 A40 or A100 GPUs forepochs.
C Behavior of Individual Data Sources
[Figure]
I m a g e N e t % e rr o r ( l o g s c a l e ) yfcclaionredcapswitcc12mshutter  5 7 9 11 13 15 Number of samples (millions
[Figure]
{+I m a g e N e t V 2 % e rr o r ( l o g s c a l e ) yfcclaionredcapswitcc12mshutter  5 7 9 11 
+}
[Figure]
I m a g e N e t R %
e rr o r ( l o g s c a l e ) yfcclaionredcapswitcc12mshutter
[Figure]
I m a g e N e t S k e t
c h %
[Figure]
rr o r ( l o g s c a l e ) yfcclaionredcapswit
[Figure]
[Figure]
[Figure]
N
[Figure]
[Figure]
D Input Mixing D.1 More Experiments with CLIP Pre-training Data Sources
[Figure]
[Figure]
t
[Figure]
t (
[Figure]
o p
[Figure]
[Figure]
[Figure]
t
N
[Figure]
t ( t o p - 1 , % )
[Figure]
[Figure]
D.2 Experiments on CIFAR-10 & CINIC-
We also investigate the phenomenon that mixing data sources resulting in diluted robustness (Section 5.1) in smaller-scale, uni-modal classification settings.
Here, we experiment with mixing CIFAR-
[37] and CINIC-10 [17] sources, each having 50K samples in total.
CINIC-10 is itself a mixture of
CIFAR-10 images and images selected and downsampled from the ImageNet database (for the same 10 classes).
We use three architectures—ResNet-18, ResNet-34 and ResNet-50 [28]—and vary the number of epochs of training to obtain models of different accuracies.
Models are evaluated on both
CIFAR-10 and CINIC-10 standard test sets, and their performances are plotted along the axes of a scatter plot.
Similar to previous input mixing results for CLIP, we observe in Figure 22 that ResNets trained on a 50K-sample dataset made up of both CIFAR-10 and CINIC-10 data, produce a linear trend that lies in between the trends of training models separately on just 50K CIFAR-10 images and just 50K CINIC-10 images.
[Figure]
[Figure]
[Figure]
j
[Figure]
c t
N
[Figure]
t ( t o p -
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
E.2 Experiments on CIFAR-10 & CINIC-
[Figure]
F Proofs of the Analyses
We provide proofs of main theoretical claims in Section 6.
F.1 Proof of Theorem
Assumption 5.
{+Under the hypotheses of Theorem 1, suppose there exists a positive constant c such that the third moments are bounded by E ( X,Y ) ∼ P θ 1 ,ρ 
+}
[( X i Y − θ 1 ,i ) 3 ] ≤ c E ( X,Y ) ∼ P θ 1 ,ρ 1 [( X i Y − θ 1 ,i ) 2 ] 3 / 2 , E
{+( X,Y ) ∼ P θ 2 ,ρ 
+}
[( X i Y − θ 1 ,i ) 3 ] ≤ c E ( X,Y ) ∼ P θ 2 ,ρ 2 [( X i Y − θ 1 ,i ) 2 ] 3 / 2 , and E [(ˆ θ n,i − θ i ) 3 ] ≤ c E
[(ˆ θ n,i − θ i ) 2 ] 3 / 2 for all i ∈ [ d ] . Under this assumption, we show that [Equation]
and
[Equation]
as it will make the first and second claims straightforward.
For ( X 1 , Y 1 ) ∼ P θ 1 ,ρ 1 , the first error event is { sign( [Equation] [Equation] [Equation]
[Equation]
where we used the fact that ˆ θ n,ξ = θ + ( ξ ∥ θ ∥ /ρ √ n ) z and XY d = θ 1 + ( ∥ θ 1 ∥ /
[Equation]
1 ) z 1 . Since the third moments are bounded, applying Berry-Esseen theorem, we get that the probability of error is bounded by Φ( − ( ⟨ θ, θ 1 ⟩ ρ 1 ρ √ n/ ( ξ ∥ θ 1 ∥∥ θ ∥√ d ))) + O (1 / √ d ) . This gives Acc θ 1 ,ρ 1 = Φ( ⟨ θ, θ 1 ⟩ ρ 1 ρ √ n/ ( ξ ∥ θ 1 ∥∥ θ ∥√ d ))
+ O (1 / √ d ) , and consequently [Equation] This proves the desired claim. F.2 Proof of Theorem
Recall that Slope(ˆ θ n ( D n,θ,ρ ))
= c ⟨ θ 2 , θ ⟩ / ⟨ θ 1 , θ ⟩ for a positive constant c > 0 that does not depend on the training data. Although the slope only depends on the training data and algorithm through θ , we write all the parameters including the sample size n and the training SNR ρ to make it explicit that the results hold for all variations of the sample size and the training algorithm within the class that we assume. It is sufficient to show that this is a monotonic function over θ when we linearly traverse from ˜ θ 1 to ˜ θ 2 , i.e. θ ( α ) =
α ˜ θ
+ (1 − α ) ˜ θ 2 for α ∈ [0 , 1] . Note that f ( α ) = c ⟨ θ 2 , θ ( α ) ⟩ / ⟨ θ 1 , θ ( α ) ⟩ = c 1 + c 2 / ⟨ θ 1 , θ ( α ) ⟩ for some c 1 and c 2 that do not depend on α .
The monotonicity follows from the fact that the derivative is
[Equation]
whose sign does not change for any α .
F.3 Proof of Theorem
The train data distribution satisfies x i y i ∼ N ( θ train , ( ∥ θ train ∥ /ρ ) 2 I ) .
Note that filtering does not change the distribution in d − 1 dimensional subspace orthogonal to ˆ θ pretrained , due to rotation invariance of a spherical Gaussian distribution.
This implies that E [ P ⊥ (ˆ θ filtered )]
= P ⊥ ( θ train ) , where P ⊥ denotes the projection operator to the d − 1 dimensional subspace orthogonal to ˆ θ pretrained . On the other hand, on the direction of ˆ θ pretrained , the filtering increases the correlation in expectation: | E [ P ∥ (ˆ θ filtered )] −P ∥ (ˆ θ pretrained ) | ≤ |P ∥ ( θ train )
−P ∥ (ˆ θ pretrained ) | , where P ∥ denotes the projection operator to the one dimensional subspace spanned by ˆ θ pretrained . This implies that E [ˆ θ filtered ] = θ train + c ˆ θ pretrained for some positive c . It follows that E [ˆ θ filtered ] is a convex interpolation between two vectors, each in the direction of θ trian and ˆ θ pretrained , respectively.
We can apply Theoremwhich gives that [Equation] , when Slope(ˆ θ unfiltered ) < Slope(ˆ θ pretrained ) .