Original: S YN B ENCH : T ASK -A GNOSTIC B ENCHMARKING OF P RETRAINED R EPRESENTATIONS USING S YNTHETIC D ATA
Revised: S YN B ENCH : T ASK -A GNOSTIC B ENCHMARKING OF P RETRAINED R EPRESENTATIONS USING S YNTHETIC D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent success in fine-tuning large models, that are pretrained on broad data at scale , on downstream tasks has led to a significant paradigm shift in deep learn ing from task-centric model design to task-agnostic representation learning and task-specific fine-tuning.
Revised: Recent success in fine-tuning large models, that are pretrained on broad data at scale, on downstream tasks has led to a significant paradigm shift in deep learning, from task-centric model design to task-agnostic representation learning and task-specific fine-tuning.

Original: As the representations of pretrained models are used as a foundation for different downstream tasks, this paper proposes a new task-agnostic framework, SynBench , to measure the quality of pretrained representations using synthetic data.
Revised: As the representations of pretrained models are used as a foundation for different downstream tasks, this paper proposes a new task-agnostic framework, SynBench , to measure the quality of pretrained representations using synthetic data.

Original: We set up a reference by a theoretically-derived robustnessaccuracy tradeoff of the  class conditional Gaussian mixture.
Revised: To address the challenge of task-agnostic data-free evaluation, we design synthetic binary classification proxy tasks with class conditional Gaussian mixtures to probe and compare model’s robustness-accuracy performance on input synthetic data and their representations. Since the synthetic tasks spare access to real-life data, SynBench offers a holistic evaluation and informs the model designers of the intrinsic robustness level of the model given a user-specified threshold accuracy.

Original: Given a pretrained model, the representations of data synthesized from the Gaussian mixture are used to compare with our reference to infer the quality.
Revised: Moreover, the use of class conditional Gaussian mixture allows us to derive a theoretically optimal robustness-accuracy tradeoff, which serves as a reference when evaluating the tradeoff on representations.

Original: By comparing the ratio of area-under-curve between the raw data and their representations, SynBench offers a quantifiable score for robustness-accuracy performance benchmarking.
Revised: By comparing the ratio of area-under-curve between the raw data and their representations, SynBench offers a quantifiable score for robustness-accuracy performance benchmarking.

Original: Our framework applies to a wide range of pretrained models taking continuous data inputs and is independent of the downstream tasks and datasets.
Revised: Our framework applies to a wide range of pretrained models taking continuous data inputs and is independent of the downstream tasks and datasets.

Original: Evaluated with several pretrained vision transformer models, the experimental results show that our SynBench score well matches the actual linear probing performance of the pre-trained model when fine-tuned on downstream tasks.
Revised: Evaluated with several pretrained vision transformer models, the experimental results show that our SynBench score well matches the actual linear probing performance of the pre-trained model when fine-tuned on downstream tasks.

Original: Moreover, our framework can be used to inform the design of robust linear probing on pretrained representations to mitigate the robustness-accuracy tradeoff in downstream tasks
Revised: Moreover, our framework can be used to inform the design of robust linear probing on pretrained representations to mitigate the robustness-accuracy tradeoff in downstream tasks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, the use of large pretrained neural networks for efficient fine-tuning on downstream tasks has prevailed in many application domains such as vision, language, and speech.
Revised: In recent years, the use of large pretrained neural networks for efficient fine-tuning on downstream tasks has prevailed in many application domains such as vision, language, and speech.

Original: Instead of designing task-dependent neural network architectures for different downstream tasks, the current methodology focuses on the principle of task-agnostic pretraining and task-specific finetuning, which uses a neural network pretrained on a large-scale dataset (often in a self-supervised or unsupervised manner) to extract generic representations of the input data, which we call pretrained representations for simplicity.
Revised: Instead of designing task-dependent neural network architectures for different downstream tasks, the current methodology focuses on the principle of task-agnostic pretraining and task-specific finetuning, which uses a neural network pretrained on a large-scale dataset (often in a self-supervised or unsupervised manner) to extract generic representations of the input data, which we call pretrained representations for simplicity.

Original: The pretrained representations are then used as a foundation (Bommasani et al., 2021) to solve downstream tasks by training a linear head (i.e., linear probing) on the data representations with the labels provided by a downstream dataset, or by simply employing zero-shot inference.
Revised: The pretrained representations are then used as a foundation (Bommasani et al., 2021) to solve downstream tasks by training a linear head (i.e., linear probing) on the data representations with the labels provided by a downstream dataset, or by simply employing zero-shot inference.

Original: Moreover, to handle multi-modal data, one can use a similar neural network architecture (e.g., transformer) for multi-modal data representation learning and alignment.
Revised: Moreover, to handle multi-modal data, one can use a similar neural network architecture (e.g., transformer) for multi-modal data representation learning and alignment.

Original: Successful examples following this new machine learning paradigm include the GPT-3 language model (Brown et al., 2020), the vision transformer (Arnab et al., 2021), and the CLIP image-text model (Radford et al., 2021), to name a few.
Revised: Successful examples following this new machine learning paradigm include the GPT-3 language model (Brown et al., 2020), the vision transformer (Arnab et al., 2021), and the CLIP image-text model (Radford et al., 2021), to name a few.

Original: As large pretrained models are shown to achieve state-of-the-art performance on a variety of downstream tasks with minimal fine-tuning, there is an intensified demand for using pretrained representations from a large model for efficient finetuning.
Revised: As large pretrained models are shown to achieve state-of-the-art performance on a variety of downstream tasks with minimal fine-tuning, there is an intensified demand for using pretrained representations from a large model for efficient finetuning.

Revised: When gauging the usefulness of a pretrained model, it is a convention to compare the accuracy of selected real-life tasks.

Original: However,  if the underlying pretrained model is at risk, such as lacking robustness to adversarial examples, this trending practice of pretraining and fine-tuning also signifiesthe immediate damage to all downstream tasks. 
Revised: However, this ap- proach has two possible drawbacks: (1) if the underlying pretrained model has hidden risks, such as lacking robustness to adversarial examples, the standard accuracy cannot inform the risk as it does not correlate well (even worse, sometimes has negative correlation) with adversarial robustness (Su et al., 2018). Therefore, the trending practice of pretraining and fine-tuning also signifies immediate damage to all downstream tasks. (2) the implications suggested by any “better” results on specific datasets are subjective to the datasets used for evaluation and could be inconclusive when the evaluation datasets change (e.g. ViT-L/16 is reportedly performing worse than ViT-B/16 on 4 out oflinear probing tasks according to Radford et al. (2021), and is incompetent to ViT-B/16 on finetuned medical tasks (Okolo et al., 2022;

Revised: Tummala et al., 2022)).

Revised: Consequently, an ideal pretrained model should entail both good accuracy and adversarial robustness, and the level of goodness can be measured in a task-agnostic manner.

Original: To address this emerging challenge, we propose a novel framework named SynBench to evaluate the quality of pretrained representations, in terms of quantifying the tradeoff between standard accuracy and adversarial robustness to input perturbations.
Revised: To address this emerging challenge, we propose a novel framework named SynBench to evaluate the quality of pretrained representations, in terms of quantifying the tradeoff between standard accuracy and adversarial robustness to input perturbations.

Original: Specifically, SynBench uses synthetic data generated from a conditional Gaussian distribution to establish a reference characterizing the robustness-accuracy tradeoff based on the Bayes optimal linear classifiers.
Revised: Specifically, SynBench uses synthetic data generated from a conditional Gaussian distribution to establish a reference characterizing the robustness-accuracy tradeoff based on the Bayes optimal linear classifiers.

Original: Then, SynBench obtains the representations of the same synthetic data from the pretrained model and compare them to the reference for performance benchmarking.
Revised: Then, SynBench obtains the representations of the same synthetic data from the pretrained model and compares them to the reference for performance benchmarking.

Original: Finally, we define the ratio of area-under-curves in robustness-accuracy characterization as a quantifiable metric of the quality of pretrained representations.
Revised: Finally, we define the ratio of area-under-curves in robustness-accuracy characterization as a quantifiable metric of the quality of pretrained representations.

Original: The entire procedure of SynBench is illustrated in Figure 1.
Revised: The entire procedure of SynBench is illustrated in Figure 1.

Original: Our SynBench framework features the following key advantages.
Revised: Our SynBench framework features the following key advantages.

Original: Soundness : We formalize the fundamental tradeoff in robustness and accuracy of the considered conditional Gaussian model and use this characterization as a reference to benchmark the quality of pretrained representations.
Revised: Soundness : We formalize the fundamental tradeoff in robustness and accuracy of the considered conditional Gaussian model and use this characterization as a reference to benchmark the quality of pretrained representations.

Original: Task-independence : Since the pretraining of large models is independent of the downstream datasets and tasks (e g., through self-supervised or unsupervised training on broad data at scale), the use of synthetic data in SynBench provides a task-agnostic approach to evaluating pretrained representations without the knowledge of downstream tasks and datasets.
Revised: Task-independence : Since the pretraining of large models is independent of the downstream datasets and tasks (e.g., through self-supervised or unsupervised training on broad data at scale), the use of synthetic data in SynBench provides a task-agnostic approach to evaluating pretrained representations without the knowledge of downstream tasks and datasets.

Original: Completeness and privacy : The flexibility of generating synthetic data (e g., by adopting a different data sampling procedure) offers a good proxy towards a more comprehensive evaluation of pretrained representations when fine-tuned on different downstream datasets, especially in the scenario when the available datasets are not representative of the entire downstream datasets.
Revised: Completeness and privacy : The flexibility of generating synthetic data (e.g., by adopting a different data sampling procedure) offers a good proxy towards a more comprehensive evaluation of pretrained representations when fine-tuned on different downstream datasets, especially in the scenario when the available datasets are not representative of the entire downstream datasets.

Original: Moreover, the use of synthetic data enables full control and simulation over data size and distribution, protects data privacy, and can facilitate model auditing and governance.
Revised: Moreover, the use of synthetic data enables full control and simulation over data size and distribution, protects data privacy, and can facilitate model auditing and governance.

Original: We highlight our main contributions as follows.
Revised: We highlight our main contributions as follows.

Original: • We propose SynBench, a novel task-agnostic framework that uses synthetic data to evaluate the quality of pretrained representations.
Revised: • We propose SynBench, a novel task-agnostic framework that uses synthetic data to evaluate the quality of pretrained representations.

Original: The evaluation process of SynBench is independent of the downstream datasets and tasks and it applies to any model taking continuous data inputs.
Revised: The evaluation process of SynBench is independent of the downstream datasets and tasks and it applies to any model taking continuous data inputs.

Original: • Evaluated with several pretrained vision transformers, our experimental results show that the metric provided by SynBench well matches the model performance in terms of adversarial robustness and standard accuracy when finetuned on several downstream datasets.
Revised: • Evaluated with several pretrained vision transformers, our experimental results show that the metric provided by SynBench well matches the model performance in terms of adversarial robustness and standard accuracy when finetuned on several downstream datasets.

Original: For example, SynBenchScore suggests that the Imagenet21k pretrained network ( ViT-B/16-in21k ) improves with finetuning on Imagenet1k ( ViT-B/16 ), echoing with the higher CIFAR10 and CIFAR10-c linear probing accuracy of ViT-B/16 .
Revised: For example, SynBenchScore suggests that the Imagenet21k pretrained network ( ViT-B/16-in21k ) improves with finetuning on Imagenet1k ( ViT-B/16 ), echoing with the higher CIFAR10 and CIFAR10-c linear probing accuracy of ViT-B/16 .

Original: • We show that SynBench can be used to inform the design and selection of the hyperparameters in robust linear probing to mitigate the robustness-accuracy tradeoff when fine-tuned on downstream datasets.
Revised: • We show that SynBench can be used to inform the design and selection of the hyperparameters in robust linear probing to mitigate the robustness-accuracy tradeoff when fine-tuned on downstream datasets.

Original: For example, conducting ϵ -robust linear probing with ϵ selected by SynBench-Score gives ViT-L/16 0 .
Revised: For example, conducting ϵ -robust linear probing with ϵ selected by SynBench-Score gives ViT-B/16 0 .

Original: 2% increase in CIFAR10 accuracy and 1 . 1% increase in CIFAR10-c accuracy.
Revised: 6% increase in CIFAR10 accuracy and 1 . 3% increase in CIFAR10-c accuracy.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Pretrained models in vision.
Revised: Pretrained models in vision.

Original: In the past few years, much focus in the machine learning community has been shift to train representation networks capable of extracting features for a variety of down- stream tasks with minimal fine-tuning.
Revised: In the past few years, much focus in the machine learning community has been shift to train representation networks capable of extracting features for a variety of downstream tasks with minimal fine-tuning.

Original: Nowadays, many common vision tasks are achieved with the assistant of good backbones, e.g. classifications (Yu et al., 2022; Wortsman et al., 2022;
Revised: Nowadays, many common vision tasks are achieved with the assistant of good backbones, e.g. classifications (Yu et al., 2022; Wortsman et al., 2022;

Original: Foret et al., 2020; Xie et al., 2020; Dosovitskiy et al., 2020;
Revised: Foret et al., 2020; Xie et al., 2020; Dosovitskiy et al., 2020;

Original: Chen et al., 2020a), object detection (Redmon& Farhadi, 2017; Liu et al., 2016), segmentation (Chen et al., 2017; Xie et al., 2021), etc.
Revised: Chen et al., 2020a), object detection (Redmon Farhadi, 2017; Liu et al., 2016), segmentation (Chen et al., 2017; Xie et al., 2021), etc.

Original: Among the popular backbones, vision transformers (ViT)
Revised: Among the popular backbones, vision transformers (ViT)

Original: (Dosovitskiy et al., 2020) have attracted enormous interest.
Revised: (Dosovitskiy et al., 2020) have attracted enormous interest.

Original: ViTs stem from Transformers (Vaswani et al., 2017) and split an image into patches, which are then treated as tokens as for the original Transformers.
Revised: ViTs stem from Transformers (Vaswani et al., 2017) and split an image into patches, which are then treated as tokens as for the original Transformers.

Original: We will exemplify the use of SynBench using several pretrained ViTs.
Revised: We will exemplify the use of SynBench using several pretrained ViTs.

Original: Benchmarking pretrained models.
Revised: Benchmarking pretrained models.

Original: Since pretrained models are used as a foundation for different downstream tasks, it is central to transfer learning (Neyshabur et al., 2020; Pruksachatkun et al., 2020), and also tightly related to model generalization (Qiao et al., 2020;
Revised: Since pretrained models are used as a foundation for different downstream tasks, it is central to transfer learning (Neyshabur et al., 2020; Pruksachatkun et al., 2020), and also tightly related to model generalization (Qiao et al., 2020;

Original: Carlucci et al., 2019).
Revised: Carlucci et al., 2019).

Original: To benchmark the performance of a pretrained model, it is a convention to apply the pretrained model for a number of popular tasks and conduct linear probing on the representations (Chen et al., 2020b;
Revised: To benchmark the performance of a pretrained model, it is a convention to apply the pretrained model for a number of popular tasks and conduct linear probing on the representations (Chen et al., 2020b;

Original: Dosovitskiy et al., 2020; Chen et al., 2020a; 2021).
Revised: Dosovitskiy et al., 2020; Chen et al., 2020a; 2021).

Original: Besides linear probing, evaluation frameworks have been proposed based on mutual information (Bachman et al., 2019) and minimum description length (MDL) (Blier & Ollivier, 2018; Voita & Titov, 2020), which are reliant on the label information of the downstream tasks and are hence task-specific.
Revised: Besides linear probing, evaluation frameworks have been proposed based on mutual information (Bachman et al., 2019) and minimum description length (MDL) (Blier & Ollivier, 2018; Voita & Titov, 2020), which are reliant on the label information of the downstream tasks and are hence task-specific.

Original: Moreover, recent work (Whitney et al., 2020) also discussed the sensitivity of validation accuracy (nonlinear probes) and MDL to evaluation dataset size, and proposed a variant of MDL and a sample complexity based quantifier that depends on the data distribution.
Revised: Moreover, recent work (Whitney et al., 2020) also discussed the sensitivity of validation accuracy (nonlinear probes) and MDL to evaluation dataset size, and proposed a variant of MDL and a sample complexity based quantifier that depends on the data distribution.

Original: It was not until recently that more fundamental questions are brought up related to the pretrained models (Bommasani et al., 2021; Tran et al., 2022; Zhang & R ´ e, 2022).
Revised: It was not until recently that more fundamental questions are brought up related to the pretrained models (Bommasani et al., 2021; Tran et al., 2022; Zhang & R ´ e, 2022).

Original: Lately, Bommasani et al.
Revised: Lately, Bommasani et al.

Original: (2021) raised practical concerns about the homogenization incentivized by the scale of the pretraining.
Revised: (2021) raised practical concerns about the homogenization incentivized by the scale of the pretraining.

Original: Although the homogenization might help in achieving competitive performance for some downstream tasks, the defects are also inherited by all these downstreams.
Revised: Although the homogenization might help in achieving competitive performance for some downstream tasks, the defects are also inherited by all these downstreams.

Original: On that account, a more careful study of the fundamentals of pretrained models is of paramount importance.
Revised: On that account, a more careful study of the fundamentals of pretrained models is of paramount importance.

Original: Tran et al.
Revised: Tran et al.

Original: (2022) was dedicated to explore the reliability of pretrained models by devising 10 types of tasks on 40 datasets in evaluating the desired aspect of reliability.
Revised: (2022) was dedicated to explore the reliability of pretrained models by devising 10 types of tasks on 40 datasets in evaluating the desired aspect of reliability.

Original: Furthermore, it is pointed out by Zhang & R ´ e (2022) that pretrained models may not be robust to subpopulation or group shift as shown in 9 benchmarks.
Revised: Furthermore, it is pointed out by Zhang & R´e (2022) that pretrained models may not be robust to subpopulation or group shift as shown in 9 benchmarks.

Original: The adversarial robustness is benchmarked by authors of (Shao et al., 2021), where authors conduct white-box and transfer attacks on Imagenet and CIFAR10.
Revised: The adversarial robustness is benchmarked by authors of (Shao et al., 2021; Paul & Chen, 2022), where Paul & Chen (2022) demonstrated the superior robustness of ViTs through Imagenet and Shao et al. (2021) conducted white-box and transfer attacks on Imagenet and CIFAR10.

Original: Optimal representations.
Revised: Optimal representations.

Original: In the seminal work of deep representation theory, Achille & Soatto (2018) depicted the desired optimal representations in supervised learning to be sufficient for downstream task, invariant to the effect of nuisances, maximally disentangled, and has minimal mutual information between representations and inputs.
Revised: In the seminal work of deep representation theory, Achille & Soatto (2018) depicted the desired optimal representations in supervised learning to be sufficient for downstream task, invariant to the effect of nuisances, maximally disentangled, and has minimal mutual information between representations and inputs.

Original: Focusing more on generalization than compression, Dubois et al.
Revised: Focusing more on generalization than compression, Dubois et al.

Original: (2020) gave the optimal representation based on V -information (Xu et al., 2019) and probed generalization in deep learning.
Revised: (2020) gave the optimal representation based on V -information (Xu et al., 2019) and probed generalization in deep learning.

Original: More recently, Ruan et al.
Revised: More recently, Ruan et al.

Original: (2021) defined the optimal representations for domain generalization.
Revised: (2021) defined the optimal representations for domain generalization.

Original: In (Dubois et al., 2022), authors characterize the idealized representation properties for invariant self-supervised representation learning.
Revised: In (Dubois et al., 2022), authors characterize the idealized representation properties for invariant self-supervised representation learning.

Original: Specifically, idealized representation should be well-distinguished by the desired family of probes for potential invariant tasks, have sufficiently large dimension, and be invariant to input augmentations.
Revised: Specifically, idealized representation should be well-distinguished by the desired family of probes for potential invariant tasks, have sufficiently large dimension, and be invariant to input augmentations.

Original: SynBench differs from the above quantifiers as it does not need knowledge of any downstream data and has controls over the evaluation set size since we could draw arbitrary number of synthetic data.
Revised: SynBench differs from the above quantifiers as it does not need knowledge of any downstream data and has controls over the evaluation set size since we could draw arbitrary number of synthetic data.

Original: With the assumed synthetic data distribution, we could theoretically characterize the robustnessaccuracy tradeoff that is independent to the downstream tasks.
Revised: With the assumed synthetic data distribution, we could theoretically characterize the robustnessaccuracy tradeoff that is independent to the downstream tasks.

Original: Therefore, SynBench provides a predefined standard of the tradeoff, which serves as the reference for representations induced by pretrained models.
Revised: Therefore, SynBench provides a predefined standard of the tradeoff, which serves as the reference for representations induced by pretrained models.

Original: It should be also mentioned that, recently sim-to-real transfer paradigm has been leveraged to test the quality of real data, by projecting those onto the space of a model trained on large-scale synthetic data generated from a set of pre-defined grammar rules (Marzoev et al., 2020).
Revised: It should be also mentioned that, recently sim-to-real transfer paradigm has been leveraged to test the quality of real data, by projecting those onto the space of a model trained on large-scale synthetic data generated from a set of pre-defined grammar rules (Marzoev et al., 2020).

Original: SynBench, though conceptually similar at a very high level, is different from that line of work – as the focus of this work is to quantify the accuracy-robustness tradeoff of pretrained representations using synthetic data from conditional distributions.
Revised: SynBench, though conceptually similar at a very high level, is different from that line of work – as the focus of this work is to quantify the accuracy-robustness tradeoff of pretrained representations using synthetic data from conditional distributions.

Original:  S YN B ENCH : M ETHODOLOGY AND E VALUATION
Revised:  S YN B ENCH : M ETHODOLOGY AND E VALUATION

Original: Without the knowledge of the downstream tasks and data, we aim to develop a task-agnostic framework to evaluate some fundamental behaviors of the representation network.
Revised: Without the knowledge of the downstream tasks and data, we aim to develop a task-agnostic framework to evaluate some fundamental behaviors of the representation network.

Original: As robustness is a key desired property, we probe the network to check how representation networks are preserving robustness in the original data.
Revised: As robustness is a key desired property, we probe the network to check how representation networks are preserving robustness in the original data.

Original: It is crucial to note that the probing method developed herein specifies the robustness-accuracy tradeoff in the pretrained representations, can be used for understanding (and possible ranking) different pretrained networks.
Revised: It is crucial to note that the probing method developed herein specifies the robustness-accuracy tradeoff in the pretrained representations, can be used for understanding (and possible ranking) different pretrained networks.

Original: On the whole, we want to measure the idealized robustness-accuracy tradeoff using synthetic data.
Revised: On the whole, we want to measure the idealized robustness-accuracy tradeoff using synthetic data.

Original: By propagating the Gaussian realizations through different representation networks, we can also measure the robustness-accuracy tradeoff for representations.
Revised: By propagating the Gaussian realizations through different representation networks, we can also measure the robustness-accuracy tradeoff for representations.

Original: We start this section by giving the synthetic data  of interest.
Revised: We start this section by giving the synthetic data and the corresponding optimal linear classifier of interest.

Original: 3.  L INEAR C LASSIFIER
Revised: 3. S YNTHETIC D ATA AND O PTIMAL L INEAR C LASSIFIER

Original: We consider  binary classification problems with data pair ( x, y ) generated from the mixture of two 
Revised: We consider imbalanced) binary classification problems with data pair ( x, y ) generated from the mixture of two Gaussian distributions P µ 1 ,µ 2 , Σ , such that [Equation] , or equivalently,

Original: Gaussian distributions
Revised: x − µ [Equation] + µ [Equation]

Original: [Equation]
Revised: [Equation]

Original: where y ∈ C
Revised: where y ∈ C

Original: = { +1 , − 1 } , P ( y = +1) 
Revised: = { +1 , − 1 } , P ( y = +1) =

Original: = p ,and P ( y = − 1) = 1 − p .
Revised: τ , P ( y = − 1) = 1 − τ , and ˜ µ = µ 1 − µ 2.

Original:  When sampling from this idealized distribution, we eliminate the factor of data bias and can benchmark the robustness degradation in an ideal setting.
Revised: We focus on the class-balanced case ( τ = 12 ) and defer the imbalanced case to the Appendix D. When sampling from this idealized distribution, we eliminate the factor of data bias and can benchmark the robustness degradation in an ideal setting.

Revised: Let ∥ · ∥ p denote the ℓ p norm of a vector for any p ≥ 1 .

Original: For a given classifier f and input x with f ( x ) =
Revised: For a given classifier f and input x with f ( x ) =

Original: y , where y is the predicted label, it is not rational for the classifier to respond differently to x + δ than to x for a small perturbation level measured by ∥ δ ∥ p , i.e. inconsistent top-1 prediction (Szegedy et al., 2013; Goodfellow et al., 2014).
Revised: y , where y is the predicted label, it is not rational for the classifier to respond differently to x + δ than to x for a small perturbation level measured by ∥ δ ∥ p , i.e. inconsistent top-1 prediction (Szegedy et al., 2013; Goodfellow et al., 2014).

Original: Therefore, the level of (adversarial) robustness for a classifier can be measured by the minimum magnitude of perturbation that causes misclassification, i.e.  min δ : f ( x + δ ) ̸
Revised: Therefore, the level of (adversarial) robustness for a classifier can be measured by the minimum magnitude of perturbation that causes misclassification, i.e. ∥ ∆ ∥ p := min δ : f ( x + δ ) ̸

Original: = f ( x ) ∥ δ ∥ p .
Revised: = f ( x ) ∥ δ ∥ p .

Original: For a generic function f , solving the optimization problem exactly is hard (Katz et al., 2017; Sinha et al., 2018).
Revised: For a generic function f , solving the optimization problem exactly is hard (Katz et al., 2017; Sinha et al., 2018).

Original: Luckily, one can readily solve for the optimization if f is affine (Moosavi-Dezfooli et al., 2016).
Revised: Luckily, one can readily solve for the optimization if f is affine (Moosavi-Dezfooli et al., 2016).

Original: In the following, we will exploit this point and consider the linear classifier that minimizes the robust classification error.
Revised: In the following, we will exploit this point and consider the linear classifier that minimizes the robust classification error.

Original: An ideal candidate classifier for the class conditional Gaussian (equation 1) is specified by the robust Bayes optimal classifier (Bhagoji et al., 2019; Dobriban et al., 2020).
Revised: An ideal candidate classifier for the class conditional Gaussian (equation 1) is specified by the robust Bayes optimal classifier (Bhagoji et al., 2019; Dobriban et al., 2020).

Original: Specifically, it is stated that the optimal robust classifier (with a robust margin ϵ ) for data generated from equation 1 is a linear [-classifier f ( x ) = sign ( w T 0 x ) , where w 
-]
Revised: Specifically, it is stated that the optimal robust classifier (with a robust margin ϵ ) for data generated from equation 1 is a linear [-classifier f ( x ) = sign ( w T 0 x ) , where w 
-]classifier.

Original: = Σ − 1 ( µ − z Σ ( µ )) , z Σ ( µ ) = arg min ∥ z ∥ 2 ≤ ϵ ( µ − z ) T Σ − 1 ( µ − z ) , and sign ( · ) is the typical sign function.
Revised: We derive the following result as a direct application of the fact.

Original: To simplify the exposition, we focus on  ℓp with p = 2  in the remainder of this paper 1 .
Revised: To simplify the exposition, we focus on the ℓ 2 norm in the remainder of this paper.

Original: We derive the following result as a direct application of the fact:
Revised: We refer the readers to Appendix C for general ℓ p -norm results. We use “bound” to denote the minimal perturbation of a sample.

Original: Result 3.1.
Revised: Theorem 1.

Original: For samples x following the conditional Gaussian in equation 1 with Σ 
Revised: For any sample x , the optimal robust classifier f ϵ for P µ 1 ,µ 2 , Σ gives

Original: = I d ( d by d identity matrix) , given an ℓ 2 adversarial budget ϵ ≤ ∥ µ ∥ 2 , the robust Bayes optimal classifier has the decision margin δ lower bounded by | q/ 2 − x  T  µ(1 − ϵ/ ∥  µ  ∥ 2 ) | (1 −ϵ/ ∥ µ ∥ 2 ) ∥ µ ∥, where q = ln { (1 − p ) /p } .
Revised: (i) the bound (decision margin) ∥ ∆ ∥ 2 = | ( x − µ 1+ µ 2) T Σ − 1 (˜ µ − z Σ (˜ µ )) | ∥ Σ − 1 (˜ µ − z Σ (˜ µ )) ∥, (ii) the scaled bound ∥ ¯∆ ∥ 2 = | ( x − µ 1+ µ 2) T Σ − 1 (˜ µ − z Σ (˜ µ )) | | ˜ µ T Σ − 1 (˜ µ − z Σ (˜ µ )) | .

Original: With p = 12 , the lower bounds become | x T µ | ∥ µ ∥ 2 .
Revised: For a sample x ∼ P µ 1 ,µ 2 , Σ , it further gives

Original: Since the bound is subject to the distance between two Gaussians, we scale the bound by ∥ µ ∥ and obtain the minimal scaled perturbation ¯ δ as
Revised: [Figure]

Original: [Equation]
Revised: [Figure]

Revised: where z Σ is the solution of the convex problem arg min

Revised: ∥ z

Original:  We note thatwhen the classes are balanced, for samples generated from  1  and Σ = σ 2 I d , all ϵ -robust Bayes optimal classifier overlap with each other.However, for data generated from conditional
Revised: ∥ 2 ≤ ϵ (˜ µ − z ) T Σ − 1 (˜ µ − z ) and Φ denotes the CDF of the standard normal distribution. We note that for samples drawn from P µ 1 ,µ 2 , Σ and Σ = σ 2 I d , all ϵ -robust Bayes optimal classifier overlap with each other.

Original: Gaussian with general covariance matrices, the ϵ of a ϵ -robust Bayes classifier specifies the desired size of margin and demonstrates the robustness accuracy tradeoff (see Figure 2).
Revised: For a general covariance matrix Σ , the ϵ of a ϵ -robust Bayes classifier specifies the desired size of margin and demonstrates the robustness accuracy tradeoff. We give an illustrative 2D class conditional Gaussian example in Figure 2(a), where different ϵ -robust Bayes classifiers give different overall margins at the cost of accuracy.

Original: [Figure]
Revised: Concretely, as ϵ increases, the robust Bayes optimal classifier rotates counterclockwise, leading to increased misclassifications, but also overall enlarged margins.

Original: 3. O BJECTIVE
Revised: 3. O BJECTIVE

Original: For a given representation network [Equation] we are interested in evaluating the expected bounds  under a thresholding accuracy a t , i.e.E [Equation]
Revised: For a given representation network parametrized by θ , we are interested in evaluating the ex- pected bounds on synthetic data and their representations, under a thresholding accuracy a t , i.e.

Original: ∼ [Equation] ∼
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ,x − ¯ µ | y ∼N ( yµ, Σ) [Equation] ∥ ¯ δ ∥ 2 | ˆ y ∗ ( x ) = 
Revised: | f ϵ ( x ) = y, a > a t (cid:3) for ¯∆ =

Original: y, a > a t (cid:3) , where P µ and P Σ characterize the probability density function of the synthetic data manifold of interest [Equation] and ¯ µ is a translation vector allowing non-symmetric class conditional Gaussian.
Revised: ¯∆ x and ¯∆ z respectively, where P µ and P Σ characterize the probability density function of the synthetic data manifold of interest, and ¯ µ is a translation vector allowing non-symmetric class conditional Gaussian.

Original: Here, without the prior of applications, we assume µ = s · 1 d / √ d , where s denotes a random variable that follows uniform distribution and 1 d / √ d is the normalized all-ones vector.
Revised: Here, without the prior of applications, we assume µ = s · 1 d / √ d , where s denotes a random variable that follows uniform distribution and 1 d / √ d is the normalized all-ones vector.

Original: For simplicity, we let Σ = I d .
Revised: For simplicity, we let Σ = I d .

Original: Formally, we define E θ,ϵ ( a t ) as
Revised: Formally, we define E θ,ϵ ( a t ) as

Original: [Equation]
Revised: [Equation]

Original: s [Equation] where 1 a(s i· 1 d / √ d , I d ,ϵ ) > a t is the indicator function specifying the s i , ϵ -dependent a that surpasses the threshold accuracy a t . 
Revised: where 1 a(s i ,ϵ ) > a t is the indicator function specifying the s i , ϵ -dependent a that surpasses the threshold accuracy a t . We put the detailed derivation in Appendix A.

Original: In the following sections, we will illustrate how to calculate the inner expectation term E x− ¯ µ | y ∼N ( ys i · 1 d / √ d,I d ) (cid:2) ∥
Revised: In the following sections, we will illustrate how to calculate the inner expectation term E x (cid:2) ∥

Original: ¯ δ ∥ 2 | ˆ y ∗ ( x ) =
Revised: ¯∆ ∥ 2 | f ϵ ( x ) =

Original: y (cid:3) for both the raw data and representations.
Revised: y (cid:3) for both the raw data and representations.

Original: 3.2. R AW DATA
Revised: 3.2. R AW DATA

Original: Denote the CDF of the standard normal distribution as Φ , we rely on the below two theorems to obtain the expected bounds for the generated synthetic data: Result 3.2. Assume a balanced dataset ( p = 12 , q = 0 )
Revised: For raw data synthesized from P µ 1 ,µ 2 , Σ according to equation 1, the inner expectation term is given by Theorem 1(iv) E (cid:2) ∥

Original: where samples x follow the general conditional Gaussian x  | y = 1 ∼ N (µ 1 , Σ) , x| y = − 1 ∼ N ( µ 2 , Σ) , given an ℓ 2 adversarial budget ϵ , the robust Bayes optimal classifier gives a standard accuracy of Φ( ˜ µ T Λ − 1 (˜ µ − z Λ (˜ µ )) ∥ Λ − 1 (˜ µ − z Λ (˜ µ )) ∥ Λ ), where ˜ µ = F T µ 1 − µ 2, F Λ F T =
Revised: ¯∆ x ∥ 2 | f ϵ ( x ) =

Original: Σ is the economy-size (thin) decomposition with nonzero eigenvalues, and z Λ is the solution of the convex problem arg min ∥ z ∥ 2 ≤ ϵ ( µ − z ) T Λ − 1 ( µ − z ) . While Result 3.2 gives us the theoretical classification accuracy as a function of synthetic conditional Gaussian parameters, the following result establishes a direct link between the expected scaled bound and accuracy (i.e. robustness-accuracy tradeoff).
Revised: y (cid:3) = 1 √ 2 π 1 a Φ − 1 ( a ) e − 12 ( Φ − 1 ( a ) ) 2 + 1 , where a denotes the standard accuracy.

Original: Result 3.3. Assume a balanced dataset ( p = 12 , q = 0 ) where samples x follow the general conditional Gaussian x | y = 1 ∼ N ( µ 1 , Σ) , x | y = − 1 ∼ N ( µ 2 , Σ) , given an ℓ 2 adversarial budget ϵ , therobust Bayes optimal classifier gives an expected scaled boundof E (cid:2) ∥
Revised: The subscript x in the expected scaled bound E (cid:2) ∥

Original: ¯ δ x ∥ 2 | ˆ y ∗ ( x ) =
Revised: ¯∆ x ∥ 2 | f ϵ ( x ) =

Original: y (cid:3) ≥ 1 √ 2 π 1 a Φ − 1 ( a ) e − 12 ( Φ − 1 ( a ) ) 2 + 1 , where a denotes the standard accuracy.

Original: The subscript x in the expected scaled bound E (cid:2) ∥ ¯ δ x ∥ 2 | ˆ y ∗ ( x ) =

Original: y (cid:3) indicates the raw data space, to distinguish from the scaled bound to be derived for representations.
Revised: y (cid:3) indicates the raw data space, to distinguish from the scaled bound to be derived for representations.

Original: We highlight that Result 3. directly gives a robustness-accuracy tradeoff.
Revised: We highlight that Theorem 1(iv) directly gives a robustness-accuracy tradeoff.

Original: We plot the expected scaled bound as a function of accuracy in Figure 2(b).
Revised: We plot the expected scaled bound as a function of accuracy in Figure 2(b).

Original: This tradeoff holds true when the data follow the conditional Gaussian exactly.
Revised: This tradeoff holds true when the data follow the conditional Gaussian exactly.

Original: In the proposed SynBench framework, we treat this theoretically-derived robustness-accuracy tradeoff as the reference, enabling a fair comparison among representations induced by different pretrained models.
Revised: In the proposed SynBench framework, we treat this theoretically-derived robustness-accuracy tradeoff as the reference, enabling a fair comparison among representations induced by different pretrained models.

Original: We give an illustrative 2D class conditional Gaussian example in Figure 2(a), where different ϵ -robust Bayes classifiers give different overall margins at the cost of accuracy.

Original: Concretely, as the ϵ increases, robust Bayes optimal classifier rotates counterclockwise, leading to misclassifications, but overall bigger margins.

Original: By now, with Result 3.3, we can already calculate the inner expectation term in equation 2 for the raw data and provide a theoretically-sounded characterization of robustness-accuracy tradeoff of

Original: Bayes optimal classifiers on raw data.

Original: 3.2. R EPRESENTATIONS
Revised: 3.2. R EPRESENTATIONS

Original: Given a pretrained network , we gather the representations of the Gaussian realizations and quantify the desired bound induced by robust Bayes optimal classifier in the representation space.
Revised: Given a pretrained network , we gather the representations of the Gaussian realizations and quantify the desired bound induced by robust Bayes optimal classifier in the representation space.

Original: When deriving the robust Bayes optimal classifier, we model the representations by a general conditional
Revised: When deriving the robust Bayes optimal classifier, we model the representations by a general conditional

Original: Gaussian z | y = 1 ∼ N ( µ 1 , Σ) , z | y =
Revised: Gaussian z | y = 1 ∼ N ( µ 1 , Σ) , z | y =

Original: − 1 ∼ N ( µ 2 , Σ) .
Revised: − 1 ∼ N ( µ 2 , Σ) .

Original: It is worthwhile to note that now the Bayes optimal classifier does not necessarily coincide with robust Bayes optimal classifier even when the dataset is class balanced (see Figure 2(a)).

Original: The following result is essential to the development of the robustness-accuracy quantification of representations.

Original: Result 3.4.

Original: For representations z following the general class-balanced conditional Gaussian z | y = 1 ∼ N ( µ 1 , Σ) , z | y = − 1 ∼ N ( µ 2 , Σ) , given an ℓ 2 adversarial budget ϵ , the robust Bayes optimal classifier has the decision margin δ z lower bounded by | ( z − [Equation]
Revised: By Theorem 1(ii), we consider the optimal robust classifier for the modeled conditional Gaussian in the representation space to calculate the scaled bound

Original: [Equation]

Original: , and a scaled bound of ∥ ¯ δ z ∥ 2 ≥ | ( z − µ 1+ µ 2)

Original: [Equation]
Revised: [Equation]

Original: (˜ [Equation] z Λ (˜ µ )) | | ˜ µ T Λ [Equation]
Revised: for correctly-classified samples and the inner expectation is estimated empirically.

Original: 1 (˜

Original: [Equation] − z Λ (˜ µ )) | , where ˜ µ = F T µ 1 − µ 2, F Λ F T = Σ is  the economysize (thin) decomposition with nonzero eigenvalues, and z Λ is the solution of the convex problem arg min ∥ z ∥ 2 ≤ ϵ ( µ − z ) T Λ − 1 ( µ − z ) .
Revised: It is worthwhile to note that now the Bayes optimal classifier does not necessarily coincide with robust Bayes optimal classifier even when we synthesized the dataset with an identity matrix covariance in the input space.

Original: 3. R OBUSTNESS -A CCURACY Q UANTIFICATION OF R EPRESENTATIONS
Revised: 3. R OBUSTNESS -A CCURACY Q UANTIFICATION OF R EPRESENTATIONS

Original: Recall that we want to calculate
Revised: Recall that we aim to calculate

Original: [Equation]
Revised: [Equation]

Original: 1 a(s i · 1 d / √ d , I d ,ϵ ) > a t for both raw data and the representations (i.e. ∥ ¯ δ x ∥ and ∥ ¯ δ z ∥ ).
Revised: ,ϵ ) > a t for both raw data and the representations (i.e. ∥ ¯∆ x ∥ and ∥ ¯∆ z ∥ ).

Original: We treat the expected bounds of the raw data under a threshold accuracy as the reference.
Revised: We treat the expected bounds of the raw data under a threshold accuracy as the reference.

Original: Given a representation network, we compare the expected bounds of the representations rendered by representation networks with the reference.
Revised: Given a representation network, we compare the expected bounds of the representations rendered by representation networks with the reference.

Original: We take s ∼ U{ 0 . 1 , 5 } under the guidance of Result 3.2.
Revised: We take s ∼ U{ 0 . 1 , 5 } under the guidance of Theorem 1(iii).

Original: Specifically, as Results 3.2 gives an analytical expected accuracy for class conditional Gaussian, we can obtain the desired range of s by giving the accuracy.
Revised: Specifically, as Theorem 1(iii) gives an analytical expected accuracy for class conditional Gaussian, we can obtain the desired range of s by giving the accuracy.

Original: Now since we are interested in having the reference as a class conditional
Revised: Now since we are interested in having the reference as a class conditional

Original: Gaussian that yields accuracy from 55% to almost 100%, we set the starting and ending s by the fact that Φ(0 . 1) ∼ 0 .
Revised: Gaussian that yields accuracy from 55% to almost 100%, we set the starting and ending s by the fact that Φ(0 . 1) ∼ 0 .

Original: 55 and Φ(5) ∼ 1 . 0 .
Revised: 55 and Φ(5) ∼ 1 . 0 .

Original: We reiterate that with more accurate modelling of the data manifold of interest, SynBench can give more precise capture of the pretrained representation performance.[Table] Arch [Table]
Revised: We reiterate that with more accurate modelling of the data manifold of interest, SynBench can give more precise capture of the pretrained representation performance.

Original: No

Original: [Table]

Original: [Table] When the data is perfect Gaussian (e [Table] g. raw data)
Revised: When the data is perfect Gaussian (e.g. input synthetic data), we calculate E θ,ϵ ( a t ) with the help of Section 3.2.1.

Original: [Table] we calculate E θ raw ,ϵ ( a t ) with the help of Result 3.3. Since E θ raw ,ϵ ( a t ) of the raw data is defined for a specific representation network parameter θraw , and all the ϵ -robust classifiers  overlap with each other, we further denote it by E ( a t )  to differentiate it from that of the representations.
Revised: We note that ¯∆ x is independent of pretrained network parameters θ , and all the ϵ -robust classifiers f ϵ in the input space overlap with each other when Σ = I d . We hereby denote the desired metric on the input synthetic data by E ( a t ) , to distinguish from that on the representations E θ,ϵ ( a t ) .

Original: For representations, we calculate E θ,ϵ ( a t ) with the help of Result 3.4 and the expectation is estimated empirically.
Revised: For representations, we calculate E θ,ϵ ( a t ) with the help of Section 3.2.2 and the expectation is estimated empirically.

Original: We show an example of the probing results in Figure 3.
Revised: We show an example of the probing results in Figure 3.

Original: To integrate over all the desired threshold accuracy, we use the area under the curve (AUC) and give the ratio to the reference by
Revised: To integrate over all the desired threshold accuracy, we use the area under the curve (AUC) and give the ratio to the reference by

Original: [Equation]
Revised: [Equation]

Original: which correspond to [Equation] area A + area B in Figure 3.
Revised: which correspond to area B area A + area B in Figure 3.

Original: Larger value of SynBench-Score implies better probing performance on pretrained representations.
Revised: Larger value of SynBench-Score implies better probing performance on pretrained representations.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this experiment, we exemplify the use of SynBench given a pretrained representation network.
Revised: In this experiment, we exemplify the use of SynBench given a pretrained representation network.

Original: In order to compare among network attributes, it is desirable to control the variates.
Revised: In order to compare among network attributes, it is desirable to control the variates.

Original: In Table 1, we list severeal pretrained vision transformers (ViTs) 

Revised: In Table 1, we list severeal pretrained vision transformers (ViTs)

Original: (Dosovitskiy et al., 2020; Chen et al., 2021) and
Revised: (Dosovitskiy et al., 2020; Chen et al., 2021; Caron

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  make comparisons to our best knowledge.
Revised: et al., 2021) and ResNets 2 (Chen et al., 2020c), and make comparisons to our best knowledge.

Original: We note that the performance of these models might be nuanced by scheduler, curriculum, and training episodes, which are not captured in the above table.
Revised: We note that the performance of these models might be nuanced by scheduler, curriculum, and training episodes, which are not captured in the above table.

Original: To provide a comprehensive evaluation, we give SynBench-Score ( θ, ϵ, a t ) with a t ranging from 0. to 0.9, and ϵ from 0 to 0.8.
Revised: To provide a comprehensive evaluation, we give SynBench-Score ( θ, ϵ, a t ) with a t ranging from 0.7 to 0.9, and ϵ from 0 to 0.8.

Original: Due to space limit, some a t results are deferred to the appendix.
Revised: Due to space limit, some a t results are deferred to the appendix.

Revised: The runtime of SynBench depends on the number of outcomes of the discrete uniform distribution U{ 0 .

Revised: For one s ∼ U{ 0 . 1 , 5 } , it costs 59 seconds to generate 2048 Gaussian samples, 37 and 81 seconds to obtain the SynBench-Score for ViT-B/ and ViT-L/16 on one GeForce RTX 2080 super.

Original:  Apart from the task-agnostic metrics SynBench-Score developed in this paper, we also report linear probing accuracy on CIFAR10 and CIFAR10-c
Revised: SynBench-Score offers a quantifiable score for robustness-accuracy performance benchmarking and is intrinsically a task-agnostic evaluation that characterizes general behaviors of the pretrained representations without the knowledge and use of any downstream data. Apart from the task-agnostic metrics SynBench-Score developed in this paper, we also report linear probing accuracy on CIFAR10/ImageNet and CIFAR10-c/ImageNet-c (Hendrycks & Dietterich, 2019) to validate the standard and transfer accuracy (use the probing layer trained on CIFAR10/ImageNet to probe CIFAR10-c/ImageNet-c). We note that evaluating pretrained representations on real-life tasks is sensitive to the choice of tasks and the results may be inconclusive.

Original: (Hendrycks & Dietterich, 2019) to validate the standard and transfer accuracy (use the probing layer trained on CIFAR10 to probe CIFAR10-c).
Revised: For example, CIFAR10/ ImageNet suggest that ViT-L performs better than ViT-B, wheras KITTI/SST (Geiger et al., 2012; Socher et al., 2013)

Original: We emphasize that SynBench-Score offers a quantifiable score for robustness-accuracy performance benchmarking and is intrinsically a task-agnostic evaluation of the pretrained model.
Revised: (Radford et al., 2021, Table 10), FoodSeg103 (Wu et al., 2021, Table 8), X-ray images (Okolo et al., 2022, Table 4-8), magnetic resonance [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: Therefore, although SynBench-Score may share trends with empirical real-life tasks (e.g. CIFAR10), it is meant to characterize a general behavior of the pretrained representations.
Revised: [Table] [Table] imaging (Tummala et al., 2022, Table 2-3) suggest the opposite. In contrast, because SynBenchScore is intrinsically a task-agnostic evaluation of the pretrained model, its result is independent of the choice of tasks.

Original: Fine-tuned pretraining representation.
Revised: Fine-tuned pretraining representation.

Original: When applying a pretrained representation network to the desired task, one can either only train a linear head on top of a fixed pretrained model, or perform fine-tuning of both the representation network and the linear head.
Revised: When applying a pretrained representation network to the desired task, one can either only train a linear head on top of a fixed pretrained model, or perform fine-tuning of both the representation network and the linear head.

Original: Thus, in Table 2, we investigate how the fine-tuning process is affecting the representation networks.
Revised: Thus, in Table 2, we investigate how the fine-tuning process is affecting the representation networks.

Original: Specifically, both networks in
Revised: Specifically, both networks in

Original: Table 2 is pretrained on Imagenet 21k with supervision.
Revised: Table 2 is pretrained on Imagenet 21k with supervision.

Original: After the pretraining, ViT-B/16 is further finetuned on Imagenet 1k.
Revised: After the pretraining, ViT-B/16 is further finetuned on Imagenet 1k.

Original: Interestingly, SynBench-Score shows that this finetuning is beneficial as improvements are witnessed across all ϵ with SynBench-Score, which well match the empirical observation give by CIFAR10 and CIFAR10-c and prior results (Kumar et al., 2021).
Revised: Interestingly, SynBench-Score shows that this finetuning is beneficial as improvements are witnessed across all ϵ with SynBench-Score, which well match the empirical observation give by CIFAR10 and CIFAR10-c and prior results (Kumar et al., 2021).

Original: Model size.
Revised: Model size.

Original: In Table 3, we compare ViTs of different sizes.
Revised: In Table 3, we compare ViTs of different sizes.

Original: Specifically, we perform SynBench on ViT-Tiny, ViT-Base, and ViT-Large with patch size being 16.
Revised: Specifically, we perform SynBench on ViT-Ti, ViT-B, and ViT-L with patch size being 16.

Original: The model parameter θ is provided by the pretrained model.
Revised: The model parameter θ is provided by the pretrained model.

Original: It is noticeable that ViT-B/16 is generally on par with ViT-L/16.
Revised: It is noticeable that ViT-B/16 is generally on par with ViT-L/16.

Original: When we set the threshold accuracy to be higher values, ViT-L/16 starts to give slightly better evaluations especially with larger ϵ .
Revised: When we set the threshold accuracy to be higher values, ViT-L/16 starts to give slightly better evaluations especially with larger ϵ .

Original: One interesting observation is that for each model, SynBench score is not necessarily monotonic in ϵ , which indicates standard linear probing (i.e., ϵ = 0 ) may not be the most effective way to probing pretrained representations in terms of robustness-accuracy performance, which is consistent with recent findings (Fan et al., 2021).
Revised: One interesting observation is that for each model, SynBench-score is not necessarily monotonic in ϵ , which indicates standard linear probing (i.e., ϵ = 0 ) may not be the most effective way to probing pretrained representations in terms of robustness-accuracy performance, which is consistent with recent findings (Fan et al., 2021).

Original: See the “Robust linear probing” paragraph below for detailed analysis.
Revised: See the “Robust linear probing” paragraph below for detailed analysis.

Original: We also observe that larger models exhibit better resilience (slower reduction in SynBench score) as ϵ increases.
Revised: We also observe that larger models exhibit better resilience (slower reduction in SynBench-score) as ϵ increases.

Revised: On Self-supervised pretrained representations (Table 4), we observe that bigger models have higher SynBench-scores – ViT-B/16-DINO, ViT-B/16-DINO, and

Revised: Res101-SimCLRv2 have bigger SynBench-scores compared to ViT-S/16-DINO, ViT-S/16-DINO, and Res50-SimCLRv2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ViT patch size.
Revised: ViT patch size.

Original: We also compare vision transformer patch sizes in Table 4.
Revised: We also compare vision transformer patch sizes in Table 5.

Original: Specifically, we give
Revised: Specifically, we give

Original: ViT-Base with patch size being 16 and 32, individually.
Revised: ViT-B with patch size being 16 and 32, individually.

Original: SynBench-Scores show an consistent trend as the model performance on CIFAR10 and CIFAR10-c. 
Revised: SynBench-Scores show an consistent trend as the model performance on CIFAR10 and CIFAR10-c. From Table

Revised: [Table]

Revised: we see that the SynBenchscore of ViT-S/16-DINO is on par with that of ViT-S/8-DINO

Revised: [Table]

Original: Robust linear probing 
Revised: and ViT-B/16-DINO has higher SynBench-score than ViT-B/8-DINO. Although linear probing on CIFAR10 and ImageNet do not share the trend, bigger patch size model (ViT-S/16-DINO) does perform better than smaller ones (ViT-S/8-DINO) on the PASCAL VOC (Everingham et al., 2010) segmentation task (Caron et al., 2021, Figure 4 bottom table). Robust linear probing.

Original: According to Table 3, 0 .
Revised: According to Table 3, 0 .

Original: 2 -robust Bayes classifiers consistently give better scores compared to 0 -robust (standard) Bayes classifiers with ViT-B/16 and ViT-L/16.
Revised: 2 -robust Bayes classifiers consistently give better scores compared to 0 -robust (standard) Bayes classifiers with ViT-B/16 and ViT-L/16.

Original: This offers us a quick way of gauging the suitable downstream robust probing parameter for the given pretrained model.
Revised: This offers us a quick way of gauging the suitable downstream robust probing parameter for the given pretrained model.

Original: We stipulate that observing a 0 .
Revised: We stipulate that observing a 0 .

Original: 2 -robust Bayes classifier to yield better SynBench-Score than a 0 -robust Bayes classifier may suggest the pretrained network to produce representations that have better overall performance with linear classifiers trained by 0 .
Revised: 2 -robust Bayes classifier to yield better SynBench-Score than a 0 -robust Bayes classifier may suggest the pretrained network to produce representations that have better overall performance with linear classifiers trained by 0 .

Original: 2 -robust linear probing.
Revised: 2 -robust linear probing.

Original: We validate this by performing robust linear probing on representations rendered by ViTs for CIFAR10 classifications.
Revised: We validate this by performing robust linear probing on representations rendered by ViTs for CIFAR10 classifications.

Original: Results are shown in Table 5.
Revised: Results are shown in Table 6.

Original: For a given pretrained model, let f
Revised: For a given pretrained model, let f

Original: and g be the pretrained network and linear probing layer, we solve the optimization problem min g max ∥ δ ∥≤ ϵ L ( g ( f ( x + δ )) , y ) using the PyTorch library Torchattacks 3 and 10-step PGDL2 attacks (Madry et al., 2018) for adversarial training.
Revised: and g be the pretrained network and linear probing layer, we solve the optimization problem min g max ∥ δ ∥≤ ϵ L ( g ( f ( x + δ )) , y ) using the PyTorch library Torchattacks 3 and 10-step PGDL2 attacks (Madry et al., 2018) for adversarial training.

Original: From Table 5, we see that robust linear probing with ϵ = arg max ϵ SynBench-Score generally gives a decent robustness-accuracy tradeoff.
Revised: From Table 6, we see that robust linear probing with ϵ = arg max ϵ SynBench-Score generally gives a decent robustness-accuracy tradeoff.

Original: For example, with robust linear probing, we obtain a1 . 2% increase in CIFAR10-c transfer accuracy at the cost of 0 . 
Revised: For example, with robust linear probing, we obtain a 0 . 6% and 1 .

Original: 4% drop in CIFAR10  accuracy with ViT-B/16 (as in Table 8).
Revised: 3% increase in CIFAR10 standard and CIFAR10-c transfer accuracy with ViTB/16 (as in Table 9).

Original: A more complete table on ϵ -robust linear probing results with different ϵ is given in the appendix.
Revised: A more complete table on ϵ -robust linear probing results with different ϵ is given in the appendix.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this paper, we propose a new task-agnostic framework SynBench for benchmarking the robustness-accuracy performance of pretrained representations.
Revised: In this paper, we propose a new task-agnostic framework SynBench for benchmarking the robustness-accuracy performance of pretrained representations.

Original: SynBench is fundamentally taskindependent and provides a quantifiable score that does not reply on any real-life data.
Revised: SynBench is fundamentally taskindependent and provides a quantifiable score that does not reply on any real-life data.

Original: SynBench exploits an idealized data distribution, class conditional Gaussian mixture, to establish a theoreticallyderived robustness-accuracy tradeoff, which serves as the reference for pretrained representations.
Revised: SynBench exploits an idealized data distribution, class conditional Gaussian mixture, to establish a theoreticallyderived robustness-accuracy tradeoff, which serves as the reference for pretrained representations.

Original: Finally, a quantifiable score SynBench-Score is provided that compares the ratio of area-under-curve between the reference and the pretrained representations.
Revised: Finally, a quantifiable score SynBench-Score is provided that compares the ratio of area-under-curve between the reference and the pretrained representations.

Original: We validate the usefulness of SynBench on several pretrained vision transformers in giving insightful comparisons on different model attributes (e.g. model size, fine-tuned pretraining representations, ViT patch size, linear probing).
Revised: We validate the usefulness of SynBench on several pretrained vision transformers in giving insightful comparisons on different model attributes (e.g. model size, fine-tuned pretraining representations, ViT patch size, linear probing).

Original: While we delved into the robustness-accuracy performance of pretrained representations of vision transformers, we envision the SynBench framework to be further extended to other trustworthiness dimensions such as privacy, fairness, etc.
Revised: While we delved into the robustness-accuracy performance of pretrained representations of vision transformers, we envision the SynBench framework to be further extended to other trustworthiness dimensions such as privacy, fairness, etc.

Original: Moreover, as the popularization of pretrained representations in various domains (e.g. vision, language, speech), we foresee SynBench to be generalized to more domains, and shed light on task-agnostic benchmarking designs. 
Revised: Moreover, as the popularization of pretrained representations in various domains (e.g. vision, language, speech), we foresee SynBench to be generalized to more domains, and shed light on task-agnostic benchmarking designs. A O BJECTIVE [Equation]

Original: A P ROOFS
Revised: B P ROOFS

Original: Result A.1.
Revised: Theorem 1.

Original: For samples x following the conditional Gaussian in equation 1 with Σ 
Revised: For any sample x [Equation] the optimal robust classifier f ϵ for P µ 1 ,µ 2 , Σ gives

Original: = I d ( d by d identity matrix) , given an ℓ 2 adversarial budget ϵ ≤ ∥ µ ∥ 2 , the robust Bayes optimal classifier has the decision margin δ lower bounded by | q/ 2 − x T µ (1 − ϵ/ ∥ µ ∥2 ) | (1 − ϵ/ ∥ µ ∥ 2 ) ∥ µ ∥, where q = ln { (1 − p ) /p } .
Revised: (i) the bound (decision margin) ∥

Original: With p =12 , the lower bounds become |  x T µ | ∥ µ∥ 2 .
Revised: ∆ ∥ 2 = | ( x − µ

Original: Proof.
Revised: [Equation]

Original: Consider the Bayes optimal ℓ 2 ϵ -robust classifier (Dobriban et al., 2020, Theorem 4.1)
Revised: (ii) the scaled bound ∥ ¯∆ ∥ 2 = | ( x − µ 1+

Original: [Equation]
Revised: [Equation]

Original: , where q = ln { (1 − p ) /p } . For a realization x , we give the lower bound on the decision margin δ ( x + δ ) T µ (1 − ϵ/ ∥ µ ∥ 2 ) − q/ 2 =⇔ δ T µ (1 − ϵ/ ∥ µ ∥ 2 ) =
Revised: ˜

Original: q/ 2 − x T µ (1 − ϵ/ ∥ µ ∥ 2 ) ⇒ ∥
Revised: [Equation]

Original: δ ∥ 2 ≥ | q/ 2 − x T µ (1 − ϵ/ ∥ µ ∥ 2 ) | (1 − ϵ/ ∥ µ ∥ 2 ) ∥ µ ∥. Result A.2.
Revised: [Equation] For a sample x ∼ P µ 1 ,µ 2 , Σ , it further gives (iii) the standard accuracy a =

Original: Assume a balanced dataset ( p = 12 , q = 0 )
Revised: Φ( ˜ µ T Σ [Equation] 1 (˜ µ − z Σ (˜ µ ))

Original: where
Revised: ∥ Σ

Original: samples x follow the general conditional Gaussian x | y = 1 ∼ N ( µ 1 , Σ) , x | y = − 1 ∼ N ( µ 2 , Σ) , given an ℓ 2 adversarial budget ϵ , the robust Bayes optimal classifier gives a standard accuracy of Φ( ˜ µ T Λ − 1 (˜ µ − z Λ (˜ µ ))
Revised: [Equation]

Original: ∥ Λ − 1 (˜ µ − z Λ (˜ µ ))
Revised: ˜ µ − z Σ [Equation] ˜ [Equation]

Original: ∥ Λ ) , where ˜ µ = F T µ 1 − µ 2, F Λ F T =
Revised: ∥ [Equation] (iv) the expected scaled bound E (cid:2) ∥ ¯∆

Original: Σ is the economy-size (thin) decomposition with nonzero eigenvalues, and z Λ is the solution of the convex problem arg min
Revised: [Equation] where z Σ is the solution of the convex problem arg min

Original: ∥ z
Revised: ∥ z

Original: ∥ 2 ≤ ϵ ( µ − z ) T Λ − 1 ( µ − z ) [Equation]
Revised: ∥ 2 ≤ ϵ (˜ µ − z ) T Σ − 1 (˜ µ − z ) and Φ denotes the CDF of the standard normal distribution.

Original: Proof.
Revised: Proof.

Original: With a general non-symmetric conditional Gaussians
Revised: (i) Following Bhagoji et al. Dan et al.

Original: [Equation]
Revised: [Equation]

Original: we apply proper translation to symmetric conditional Gaussians 
Revised: the Bayes optimal robust classifier for the general non-symmetric conditional Gaussians P µ 1 ,µ 2 , Σ specified in equation 1 is

Original: [Equation]
Revised: [Equation]

Original: where ˜ µ = F T µ 1 − µ 2. Then, following Bhagoji et al. Dan et al.
Revised: where sign ( · ) is the typical sign function and z Σ is the solution of the convex problem

Original: [Equation]
Revised: [Equation]

Original: we have the Bayes optimal robust classifier
Revised: The corresponding decision boundary is at

Original: [Equation]
Revised: [Equation]

Original: where z Λ is the solution of the convex problem arg min

Original: ∥ z
Revised: = ⇒

Original: ∥ 2 ≤ ϵ ( µ − z ) T Λ − 1 ( µ − z ) .
Revised: [Equation] (ii)

Original: With this classifier
Revised: Since the bound ∥ ∆ ∥ 2 is subject to the positions of two Gaussians, we scale the bound by the distance from Gaussian centers to the classifier,

Original: [Equation]
Revised: [Equation]

Original: we can calculate the analytical standard accuracy by
Revised: z Σ (˜ µ )) ∥ 2 and obtain ∥ ¯∆ ∥ 2 = | ( x − µ 1 + µ 2) T Σ − 1 (˜ µ − z Σ

Original: [Equation]
Revised: (˜ µ ))

Original: Result A.3.
Revised: | ∥ Σ − 1 (˜ µ − z Σ (˜ µ )) ∥∥ Σ − 1 (˜ µ − z Σ (˜ µ )) ∥|

Original: Assume a balanced dataset ( p = 12 , q = 0 )
Revised: ˜ µ T Σ − 1 (˜ µ − z Σ (˜ µ )) | = | ( x − µ 1 + µ 2) T Σ − 1 (˜ µ − z Σ (˜ µ )) | | ˜ µ T Σ − 1 (˜ µ − z Σ (˜ µ )) | .

Original: where
Revised: (iii)

Original: samples xfollow the general conditional Gaussian x | y = 1 ∼ N ( µ 1 , Σ) , x | y = − 1 ∼ N ( µ 2 , Σ) , given an ℓ 2 adversarial budget ϵ ,  therobust Bayes optimal  classifier gives an expected scaled bound of E (cid:2) ∥
Revised: For sample x ∼ P µ 1 ,µ 2 , Σ , consider the Bayes optimal robust classifier in equation 4, we can calculate the analytical standard accuracy by [Equation]

Original: ¯ δ x ∥ 2 | ˆ y ∗ ( x ) =
Revised: (iv) For sample x ∼ P [Equation]

Original: y (cid:3) ≥ 1 √ 2 π 1 a Φ − 1 ( a ) e − 12 ( Φ − 1 ( a ) ) 2 + 1 , where a denotes the standard accuracy.
Revised: let a denote the accuracy

Original: Proof.
Revised: [Equation]

Original: Let a denote the accuracy, tdenote F T x − F T µ 1 + µ 2, and w denote Λ − 1 (˜ µ − z Λ (˜ µ )) . From Result 3.2, we have that the standard accuracy for the Bayes optimal (robust) classifier is Φ( ˜ µ T w ∥ w ∥ Λ ) , so
Revised: t

Original: [Equation]
Revised: [Equation]

Original: √
Revised: Σ

Original: [Equation]
Revised: [Equation]

Original: i = Φ
Revised: z Σ (˜ µ )) .

Original: [Equation] 1 ( a ) .
Revised: From (iii), we have that the standard accuracy of conditional Gaussian samples with the Bayes optimal (robust) classifier is Φ( ˜ µ T w ∥ w ∥ Σ ) , so ˜ µ T w ∥ w ∥ Σ = Φ − 1 ( a ) .

Original: Since for binary classification, we only care about accuracy from 0.5 to 1, so we should have(cid:80) d ˜ µ i wi > 0 .
Revised: Since for binary classification, we only care about accuracy from 0.5 to 1, so we should have ˜ µ T w > 0 .

Original: Now consider the classifier in equation 4, the correspondinglower bound and scaledlower bound can be given as [Equation]
Revised: Now consider the classifier in equation 4 and the corresponding scaled bound from (ii),

Original: Λ (˜ µ )) | | ˜ µ T Λ − 1 (˜ µ − z Λ (˜ µ )) | .

Original: When t = F T x − F T µ 1 + µ 2, and w = Λ − 1 (˜ µ − z Λ (˜ µ )) ,

Original: [Equation]
Revised: [Equation]

Original: µ i

Original: [Equation]

Original: i .

Original: Since t | y ∼ N ( y ˜ µ, Λ) , we have t i wi | y ∼ N ( y ˜ µ i w i , λ i w2 i ) and [Equation] N [Equation]
Revised: Since t | y ∼ N ( y ˜ µ, Σ) , we have t T w | y ∼ N ( y ˜ µ T w, w T Σ T w ) .

Original: When we only want to get the expected scaled bound of the correctly-classified samples, we have that
Revised: When we only want to get the expected scaled bound of the correctly-classified samples, we have that

Original: [Equation]
Revised: [Equation]

Original: Recall that(cid:80) d t i wi | y ∼ N ((cid:80) d y ˜ µ i w i , (cid:80) d λ i w2 i ) , then by the mean of truncated normal distribution, it is true that
Revised: Recall that t T w | y ∼ N ( y ˜ µ T w, w T Σ T w ) , then by the mean of truncated normal distribution, it is true that

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Therefore

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: By replacing

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: , we got

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Therefore E (cid:2) ∥ ¯ δ ∥ 2 | ˆ y ∗ ( x ) =
Revised: We note that our results in Appendix B can be straightforwardly generalized to ℓ p . Given an ℓ p adversarial budget ϵ :

Original: y (cid:3) ≥
Revised: Theorem 2. For any sample x , the optimal robust classifier f ϵ for P µ 1 ,µ 2 , Σ gives (i) the bound (decision margin) ∥ ∆ ∥ p = | ( x − µ 1+ µ

Original: [Equation]
Revised: [Equation]

Original: i
Revised: [Equation] (ii) the scaled bound ∥ ¯∆ ∥ p = | ( x − µ 1+

Original: [Equation]
Revised: [Equation]

Original:   = 1 + (cid:113)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: T Σ

Original: [Equation]
Revised: [Equation]

Original: w

Original: [Equation]
Revised: [Equation]

Original: e
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: (cid:32)
Revised: z Σ (˜

Original: [Equation]
Revised: [Equation]

Original:  ˜ µiwi √
Revised: ) | | ˜ µ T Σ

Original: [Equation]
Revised: [Equation]

Original: λiw 2 i
Revised: 1 (˜ µ −

Original: [Equation]
Revised: [Equation]

Original: By replacing (cid:80) d ˜ µ i w i √ (cid:80) d λ i w 2 i by Φ − 1 ( a ) , we got
Revised: Σ (˜ µ )) | . For sample x ∼ P µ 1 ,µ 2 , Σ , it further gives (iii) the standard accuracy a = Φ( ˜ µ T Σ − 1 (˜ µ − z Σ (˜ µ ))

Original: E (cid:2) ∥ ¯ δ ∥ 2 | ˆ y ∗ ( x ) =
Revised: ∥ Σ − 1 (˜ µ − z Σ (˜ µ ))

Original: y (cid:3) ≥
Revised: ∥ Σ ) , (iv) the expected scaled bound

Original: [Equation]
Revised: [Equation]

Original: Result A.4. For representations z following the general class-balanced conditional Gaussian z | y = 1 ∼ N ( µ 1 , Σ) , z | y = − 1 ∼ N ( µ 2 , Σ) , given an ℓ 2 adversarial budget ϵ , the robust Bayes optimal classifier has the decision margin δ z lower bounded by | ( z − µ 1+ µ 2) T F Λ − 1 (˜ µ − z Λ (˜ µ ))
Revised: where z Σ is the solution of the convex problem arg min

Original: | ∥ F Λ − 1 (˜ µ − z Λ (˜ µ )) ∥, and a scaled bound of ∥ ¯ δ z ∥ 2 ≥ | ( z − µ 1+ µ 2) T F Λ − 1 (˜ µ − z Λ (˜ µ )) | | ˜ µ T Λ − 1 (˜ µ − z Λ (˜ µ )) | , where ˜ µ = F T µ 1 − µ 2, F Λ F T = Σ is the economysize (thin) decomposition with nonzero eigenvalues, and z Λ is the solution of the convex problem arg min

Original: ∥ z 
Revised: ∥ z ∥

Original: ∥ 2 ≤ ϵ ( µ − z ) T Λ − 1 ( µ − z ) .
Revised: p ≤ ϵ (˜ µ − z ) T Σ − 1 (˜ µ − z ) and Φ denotes the CDF of the standard normal distribution.

Original: Proof.
Revised: Proof.

Original: The proof follows similarly as before and is the intermediate result in the proof of Result 3.3.
Revised: We follow the proof of Theorem 1 and consider the classifier in equation 4.

Original: With a general non-symmetric conditional Gaussians
Revised: By H ¨ older’s inequality [Equation] we now have the corresponding lower bound and scaled lower bound as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: where 1 p + 1 q = 1 . The remainder of the proof will then follows as in Theorem 1. Remark.

Original: where ˜ µ = F  T µ 1 − µ 2.
Revised: In general, in the case that Σ is singular, we can apply the economy-size (thin) decomposition with nonzero eigenvalues Σ = F Λ F T .

Original: Following Bhagoji et al. Dan et al.
Revised: Then, with a general non-symmetric conditional Gaussians

Original: [Equation]
Revised: [Equation]

Original: we have the Bayes optimal robust classifier
Revised: we apply proper translation to symmetric conditional Gaussians

Original: [Equation]
Revised: [Equation]

Original: where z Λ is the solution of the convex problem arg min
Revised: where ˜ µ = F T µ 1 − µ 2. D C LASS IMBALANCE

Original:  ∥ z ∥ 2 ≤ ϵ ( [Equation] z [Equation]
Revised: Given an ℓ 2 adversarial budget ϵ ≤ ∥ µ ∥ 2 , consider the conditional Gaussian in equation 1 with Σ

Original: Λ − [Equation] ( µ − z ) . The corresponding lower bounds is
Revised: = I d ( d by d identity matrix) and general class prior τ , then the following theorem holds.

Original: [Equation]
Revised: Theorem 3.

Original: Given the original data and an ℓ ∞ adversarial budget ϵ 
Revised: For any sample x , the optimal robust classifier f ϵ for P µ 1 ,µ 2 ,I d gives (i) the bound (decision margin) ∥

Original: [Equation]
Revised: [Equation]

Revised: (ii) the scaled bound ∥

Revised: [Equation]

Revised: For a sample x ∼ P µ 1 ,µ 2 ,I d , it further gives

Revised: (iii) the standard accuracy a =

Revised: τ

Revised: [Equation]

Revised: [Equation]

Revised: (iv) the expected scaled bound E (cid:2) ∥

Revised: ¯∆ ∥

Revised: [Equation]

Revised: ϵ

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: y

Revised: [Equation]

Revised: where q = ln { (1 − τ ) /τ } , w = ˜ µ (1 − ϵ/ ∥

Revised: ˜

Revised: µ ∥ 2 ) , ϕ and Φ denotes the PDF and CDF of the standard normal distribution.

Original: we consider the Bayes optimal ℓ ∞ robust classifier (Theorem 6.3 Dobriban et al.
Revised: Proof. (i ) Consider the Bayes optimal ℓ 2 ϵ -robust classifier (Dobriban et al., 2020, Theorem 4.1)

Original: [Equation]
Revised: [Equation]

Revised: where q = ln { (1 − τ ) /τ }

Original: } , (6) where [ sign ( µ ) min( | µ | , ϵ )]
Revised: [Equation] For any x , [Equation] (ii)

Revised: Since the bound ∥ ∆ ∥ 2 is subject to the positions of two Gaussians, we scale the bound by the distance from Gaussian centers to the classifier, | ˜

Original: i = sign ( µ i ) min( | µ i | , ϵ ) and q = ln { (1 − p ) /p } .
Revised: µ [Equation] and obtain [Equation]

Original: Now, to give a lower bound on the margin ( ∥ δ ∥ ∞ ) given a realization, ( x + δ ) T ( µ  − sign ( µ  )min( | µ | , ϵ )) − q/ 2 =⇒ ∥
Revised: (iii) For sample x ∼ P µ 1 ,µ 2 ,I d , consider the Bayes optimal robust classifier in equation 4, we can calculate the analytical standard accuracy by [Equation] Let w denote ˜ µ (1 − ϵ/ ∥ ˜ µ ∥ 2 ) , the we got the accuracy

Original: δ ∥ ∞ ≥ | q/ 2 − x T ( µ  − sign ( µ  ) min( | µ | , ϵ ))
Revised: [Equation] (iv) For sample x ∼ P µ 1 ,µ 2 ,I d , let t denote x − µ 1 + µ 2, and w denote ˜ µ (1 − ϵ/ ∥ ˜ µ ∥ 2 ) .

Revised: According to

Original: | ∥ µ − sign ( µ ) min( | µ | , ϵ ) ∥.
Revised: Theorem 3(iii), when ˜ µ T ˜ µ (1 − ϵ/ ∥ ˜

Original: C A DDITIONAL TABLES
Revised: µ ∥ 2 ) − q/ 2 > 0 , the accuracy would be higher than 0.5. Therefore we consider ˜ µ T w − q/ 2 > 0 . Now consider the classifier in equation 5 and the corresponding scaled bound from (ii), ∥ [Equation]

Revised: Since t | y ∼ N ( y ˜ µ, I d ) , we have t T w − q/ 2 | y ∼ N ( y ˜ µ T w − q/ 2 , w T w ) .

Original: a t 
Revised: When we only want to get the expected scaled bound of the correctly-classified samples, we have that [Equation] Recall that t T w − q/ 2 | y ∼ N ( y ˜ µ T w − q/ 2 , w T w ) , then by the mean of truncated normal distribution, it is true that [Equation]

Original: Model ϵ =
Revised: Therefore [Equation] 

Original: ϵ = 0 .ϵ
Revised:  E A DDITIONAL TABLES [Table] [Table]

Original: = 0 .ϵ
Revised: a r g m a x ϵ li n e a r p r ob i ng 0 . 1 -r obu s t li n ea r p r ob i ng 0 . 2 -r obu s t li n ea r p r ob i ng 0 . 3 -r obu s t li n

Original: = 0 .ϵ
Revised: [Table] r p r ob i ng 0 . 4 -r obu s t li n ea r p r ob i ng

Original: = 0 .
Revised: S yn B e n c hS c o r e C I F

Original: CIFAR
Revised: A R C I F

Revised: A

Original: CIFAR10-c 0 .
Revised: R 10 - c C I F A

Revised: R

Revised: C

Revised: I F

Revised: A

Revised: [Table]

Revised: 10 - c

Revised: C

Revised: I F

Revised: A

Original: ViT-B/ 0.
Revised: [Table] [Table] I [Table]

Revised: A

Revised: [Table]

Revised: 10 - c

Revised: C I

Revised: [Table]

Revised: [Table]

Revised: I

Revised: [Table]

Revised: 10 - c

Revised: [Table]

Revised: A

Revised: [Table]

Revised: [Table]

Revised: V i

Revised: [Table]

Revised: [Table]

Revised: i /

Revised: [Table]

Revised: [Table]

Original: 81.9 ViT-B/16-in21k 0.
Revised: [Table] [Table] [Table] V i T - B / 16 - i n21k

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: ViT-B/ 0.
Revised: [Table] T a b l e 9 : F u ll t a b l e o f T a b l e 6 . ‘ / ’ m ea n s t h e nu m b e r s a r e s i m il a r .

Revised: F S YNTHETIC D ATASET COMPLEXITY

Revised: SynBench can adjust the synthetic task complexity by statistically modeling the structure of the covariance matrix P Σ .

Revised: In our previous experiments, we have considered an identity covariance matrix, here we assume a channel-wise band matrix covariance Σ (R,G,B channel entries are externally independent, and internally Gaussians with a band matrix covariance).

Revised: Essentially, the assembled Σ is a block-diagonal matrix with each block being a band matrix in the size of the image.

Revised: For ViT with

Revised: ImageNet size inputs, the 3 blocks are 224 2 × 224 2 .

Revised: We let the 3 band matrices be Pentadiagonal matrices (only main, first two upper, and two lower diagonals are nonzero).

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: G O THER BASELINES

Revised: For completeness, we report several baseline metrics for the synthetic conditional Gaussian classification task.

Revised: We follow the implementation of Whitney et al.

Revised: (2020) 4 and set ns (the training set size) to be the length of the synthetic dataset to compute canonical results.

Original: 81.9 ViT-B/16-in21k 0.
Revised: In Table 15, We report validation loss (val loss), minimum description length (MDL) (Voita & Titov, 2020), surplus description length (SDL) and ϵ -sample complexity ( ϵ -SC) (Whitney et al., 2020). As a reference, obtaining the metrics for ViT-B/16 costs 6807 seconds and ViT-L/16 costs 7373 seconds on one Tesla V100.

Original: ViT-B/ 0.
Revised: [Table] N a m e V i T - T i/ V i T - B /

Original: 81.9 ViT-B/16-in21k 0.
Revised: V i T - L / V i T - B / 16 - i n21k V i T - B / V i T - S / 16 - D

Original: ViT-B/ 0.
Revised: I NO V i T - S / 8 - D I NO

Original: 81.9 ViT-B/16-in21k 0.
Revised: V i T - B / 16 - D I NO V i T - B / 8 - D I

Original: ViT-B/ 0.
Revised: NO R e s n e t 50 - S i m C L R v

Original: [Table]
Revised: R e s n e t- S i m C L R v

Revised: V a l l o ss 4 .0 .1 .0 .2 .1 .0 .0 .0 .0 .0 .

Revised: M D L 3.9939 .

Original: [Table]{+S D L , ε => .3479 . > > . > .2056 .
+}
Revised: {+S D L , ε => .3479 . > > . > .2056 .
+}

Revised: .3700 .776 .ε

Original: [Table]
Revised: S C , ε => 7372 . > 7372 . > 7372 . > 7372 .

Original: [Table]
Revised: T a b l e 15 : B a s e li n e m e t r i c s e v a l u a ti ng t

Revised: h e r e p r e s e n t a ti on qu a lit y on t h e c ond iti on a l G a u ss i a n s yn t

Revised: h e ti c

Revised: d a t a .