Original: Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks
Revised: Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Unrestricted color attacks, which manipulate semantically meaningful color of animage, have shown their stealthiness and success in fooling both human eyes anddeep neural networks.
Revised: Unrestricted color attacks, which manipulate semantically meaningful color of animage, have shown their stealthiness and success in fooling both human eyes anddeep neural networks.

Original: However, current works usually sacrifice the flexibility of theuncontrolled setting to ensure the naturalness of adversarial examples.
Revised: However, current works usually sacrifice the flexibility of theuncontrolled setting to ensure the naturalness of adversarial examples.

Original: As a result,the black-box attack performance of these methods is limited.
Revised: As a result,the black-box attack performance of these methods is limited.

Original: To boost transferability of adversarial examples without damaging image quality, we propose a novel
Revised: To boost transferability of adversarial examples without damaging image quality, we propose a novel

Original: Natural Color Fool (NCF) which is guided by realistic color distributions sampledfrom a publicly available dataset and optimized by our neighborhood search andinitialization reset.
Revised: Natural Color Fool (NCF) which is guided by realistic color distributions sampledfrom a publicly available dataset and optimized by our neighborhood search andinitialization reset.

Original: By conducting extensive experiments and visualizations, weconvincingly demonstrate the effectiveness of our proposed method.
Revised: By conducting extensive experiments and visualizations, weconvincingly demonstrate the effectiveness of our proposed method.

Original: Notably, onaverage, results show that our NCF can outperform state-of-the-art approachesby 15.0% ∼ 32.9% for fooling normally trained models and 10.0% ∼ 25.3% forevading defense methods.
Revised: Notably, onaverage, results show that our NCF can outperform state-of-the-art approachesby 15.0% ∼ 32.9% for fooling normally trained models and 10.0% ∼ 25.3% forevading defense methods.

Original: Our code is available at Appendix.
Revised: Our code is available at Appendix.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks (DNNs) have achieved excellent performance on a large number of tasks, suchas image recognition [1, 2], medical diagnostics [3, 4] and autonomous driving [5].
Revised: Deep neural networks (DNNs) have achieved excellent performance on a large number of tasks, suchas image recognition [1, 2], medical diagnostics [3, 4] and autonomous driving [5].

Original: However, with therise of the field of adversarial attack [6, 7, 8, 9], the robustness of state-of-the-art DNNs [2, 10, 11, 12]is put in question.
Revised: However, with therise of the field of adversarial attack [6, 7, 8, 9], the robustness of state-of-the-art DNNs [2, 10, 11, 12]is put in question.

Original: Adversarial examples thus reveal the vulnerability of DNNs and inevitably raiseconcerns about widely deployed models.
Revised: Adversarial examples thus reveal the vulnerability of DNNs and inevitably raiseconcerns about widely deployed models.

Original: To avoid potential risks, it is crucial to expose as many ofthe model’s “blind spots” as possible at this stage.
Revised: To avoid potential risks, it is crucial to expose as many ofthe model’s “blind spots” as possible at this stage.

Original: Generally, scenarios of adversarial attacks can divide into white-box and black-box.
Revised: Generally, scenarios of adversarial attacks can divide into white-box and black-box.

Original: For the white-boxscenario [13, 14, 15, 16], all information about the target model is publicly available, such as thearchitecture and parameters.
Revised: For the whitebox scenario [13, 14, 15], all information about the target model is publicly available, such as thearchitecture and parameters.

Original: Although the attacker can easily fool them with human-imperceptibleperturbations in this case, it is not practical in the real world since deployed models are usuallyopaque to unauthorized users.
Revised: Although the attacker can easily fool them with human-imperceptibleperturbations in this case, it is not practical in the real world since deployed models are usually opaqueto unauthorized users.

Original: To overcome this limitation, numerous works begin to study theblack-box attack [7,  17, 18, 19, 20, 21], which relies on the cross-model transferability of adversarialexamples.
Revised: To overcome this limitation, numerous works begin to study the black-boxattack [7, 16, 17, 18, 19, 20], which relies on the cross-model transferability of adversarial examples.

Original: Typically, most existing black-box attacks maintain image quality by constraining the 
Revised: Typically, most existing black-box attacks maintain image quality by constraining the L p -norm ofperturbation.

Original: L p -norm of perturbation [22, 19, 8]. However, the L p -norm constraint in RGB color space isnot an ideal indicator for measuring the perceptual distance between two images since adversarialexamples with a small perturbation size may still look unnatural to human eyes [15].
Revised: However, the L p -norm constraint in RGB color space is not an ideal indicator formeasuring the perceptual distance between two images since adversarial examples with a smallperturbation size may still look unnatural to human eyes [21, 18, 8].

Original: Thus theunrestricted attack [23, 24, 25, 26, 27, 28, 29] is proposed — replacing the traditional small L p -normperturbations with uncontrolled yet human-imperceptible ones, which is more practically meaningful.
Revised: Thus the unrestricted attack [22,23, 24, 25, 26, 27, 28] is proposed — replacing the traditional small L p -norm perturbations withuncontrolled yet human-imperceptible ones, which is more practically meaningful.

Original: Among existing unrestricted attacks, the unrestricted color branch is a very promising direction.
Revised: Among existingunrestricted attacks, the unrestricted color branch is a very promising direction.

Original: As demonstrated in previous works [25, 27, 24], only manipulating the color of an image is notnoticeable to human eyes but can easily fool DNNs.
Revised: As demonstratedin previous works [24, 26, 23], only manipulating the color of an image is not noticeable to humaneyes but can easily fool DNNs.

Original: This is mainly because global uniform changesin images are usually imperceptible, and DNNs are vulnerable to certain large-scale patterns.
Revised: This is mainly because global uniform changes in images are usuallyimperceptible, and DNNs are vulnerable to certain large-scale patterns.

Original: However, current unrestricted color attacks usually sacrifice the flexibility of the uncontrolled settingto ensure the naturalness of adversarial examples: either rely on intuition [23, 24] and objectivemetric [27] or use relatively small changes [25, 26].
Revised: However, current unrestricted color attacks usually sacrifice the flexibility of the uncontrolled settingto ensure the naturalness of adversarial examples: either rely on intuition [22, 23] and objectivemetric [26] or use relatively small changes [24, 25].

Original: Consequently, the transferability of adversarialexamples generated by these methods is limited.
Revised: Consequently, the transferability of adversarialexamples generated by these methods is limited.

Original: To remedy this, we propose a Natural Color Fool (NCF) based on a more flexible perturbationspace.
Revised: To remedy this, we propose a Natural Color Fool (NCF) based on a more flexible perturbationspace.

Original: Specifically, we build a color distribution library that contains diverse yet realistic colordistributions sampled from the ADE20K dataset [30] for each semantic class.
Revised: Specifically, we build a color distribution library that contains diverse yet realistic colordistributions sampled from the ADE20K dataset [29] for each semantic class.

Original: By borrowing the colormapping technique of [31], we utilize the library to generate a set of image variants with realistic colordistribution for each clean image.
Revised: By borrowing the colormapping technique of [30], we utilize the library to generate a set of image variants with realistic colordistribution for each clean image.

Original: Then we use the substitute model to select the most adversarialvariant.
Revised: Then we use the substitute model to select the most adversarialvariant.

Original: To boost its transferability, we further introduce neighborhood search and initializationreset to optimize, thus obtaining the resulting adversarial example.
Revised: To boost its transferability, we further introduce neighborhood search and initializationreset to optimize, thus obtaining the resulting adversarial example.

Original: Figure 1 illustrates the simplifiedpipeline for our method.
Revised: Figure 1 illustrates the simplifiedpipeline for our method.

Original: In summary, our main contributions are:
Revised: In summary, our main contributions are:

Original: • We observe that current unrestricted color attacks lack flexibility, which results in limited
Revised: • We observe that current unrestricted color attacks lack flexibility, which results in limited

Original: transferability of the adversarial examples.
Revised: transferability of the adversarial examples.

Original: • To alleviate this issue, we propose a Natural Color Fool (NCF), which fully exploits colordistributions of semantic classes in an image to craft human-imperceptible, flexible, andhighly transferable adversarial examples.
Revised: • To alleviate this issue, we propose a Natural Color Fool (NCF), which fully exploits colordistributions of semantic classes in an image to craft human-imperceptible, flexible, andhighly transferable adversarial examples.

Original: • Extensive experiments and visualizations demonstrate the effectiveness of our proposed
Revised: • Extensive experiments and visualizations demonstrate the effectiveness of our proposed

Original: method.
Revised: method.

Original: Significantly, with high NIMA scores [32], NCF averagely outperforms state-of-
Revised: Significantly, with high NIMA scores [31], NCF averagely outperforms state-of-

Original: the-art methods by 15.0% ∼ 32.9% for fooling normally trained models, and 10.0% ∼ 25.3%for evading defense models.
Revised: the-art methods by 15.0% ∼ 32.9% for fooling normally trained models, and 10.0% ∼ 25.3% for evading defense models.

Original:  Related Works
Revised:  Related Works

Original: 2.1 Unrestricted Attacks
Revised: 2.1 Unrestricted Attacks

Original: Unrestricted attacks [25, 33, 34, 27], which usually significantly modify the image but can yieldhuman-imperceptible adversarial examples, have been widely studied in recent years.
Revised: Unrestricted attacks [24, 32, 33, 26], which usually significantly modify the image but can yieldhuman-imperceptible adversarial examples, have been widely studied in recent years.

Original: For example,some works modify domain-specific structural attributes of the image.
Revised: For example,some works modify domain-specific structural attributes of the image.

Original: SemanticAdv [33] uses anattribute-conditioned image editing model to modify the attributes of the face (e.g., modifying thedegree of mouth opening).
Revised: SemanticAdv [32] uses anattribute-conditioned image editing model to modify the attributes of the face (e.g., modifying thedegree of mouth opening).

Original: advCam [34] hides adversarial perturbations by using the style transfertechnique (e.g., changing a traffic sign to a rust style).
Revised: advCam [33] hides adversarial perturbations by using the style transfertechnique (e.g., changing a traffic sign to a rust style).

Original: Of course, there are also many works modifygeneric attributes of images, such as texture and color.
Revised: Of course, there are also many works modifygeneric attributes of images, such as texture and color.

Original: Specifically, tAdv [27] infuses the texture ofanother image to generate adversarial examples.
Revised: Specifically, tAdv [26] infuses the texture ofanother image to generate adversarial examples.

Original: EdgeFool [28] craft adversarial perturbations byenhancing edge details of an image.
Revised: EdgeFool [27] craft adversarial perturbations byenhancing edge details of an image.

Original: Compared with texture-based attacks, color-based ones usually yield more natural results sincemanipulating groups of pixels along dimensions is less perceptible to human eyes.
Revised: Compared with texture-based attacks, color-based ones usually yield more natural results sincemanipulating groups of pixels along dimensions is less perceptible to human eyes.

Original: To our knowledge,
Revised: To our knowledge,

Original: Hosseini&Poovendran [23] first proposed the Semantic Adversarial Examples (SAE) in this branch.
Revised: Hosseini&Poovendran [22] first proposed the Semantic Adversarial Examples (SAE) in this branch.

Original: Specifically, they convert the image from the RGB color space to the HSV one, then uniformly andrandomly perturbed the H (Hue) and S (Saturation) channels of the entire image.
Revised: Specifically, they convert the image from the RGB color space to the HSV one, then uniformly andrandomly perturbed the H (Hue) and S (Saturation) channels of the entire image.

Original: Laidlaw&Feizi
Revised: Laidlaw&Feizi

Original: [25]also notice that RGB color space is not a perceptually uniform space.
Revised: [24]also notice that RGB color space is not a perceptually uniform space.

Original: Thus, they operate in the
Revised: Thus, they operate in the

Original: CIELUV color space and propose a ReColorAdv that uses a specific function to uniformly transformall pixels belonging to the same color.
Revised: CIELUV color space and propose a ReColorAdv that uses a specific function to uniformly transformall pixels belonging to the same color.

Original: Similar to ReColorAdv, ACE [26] proposes a simple piecewiselinear color adjustment filter to manipulate the image.
Revised: Similar to ReColorAdv, ACE [25] proposes a simple piecewiselinear color adjustment filter to manipulate the image.

Original: The main difference is that ACE treats threechannels of an image independent of each other when transformed.
Revised: The main difference is that ACE treats threechannels of an image independent of each other when transformed.

Original: AGV [35] increases the attackeffectiveness by filter composition.

Original: Unlike them, cAdv [27] performs color transformation using apretrained colorization network and jointly varies input hints and masks to manipulate adversarialexamples.
Revised: Unlike them, cAdv [26] performscolor transformation using a pretrained colorization network and jointly varies input hints and masksto manipulate adversarial examples.

Original: Among existing unrestricted color attacks, ColorFool [24], which exploits image semantics to modifycolors selectively, is the most related to ours.
Revised: Among existing unrestricted color attacks, ColorFool [23], which exploits image semantics to modifycolors selectively, is the most related to ours.

Original: Nevertheless, there are several significant differencesbetween our work and ColorFool.
Revised: Nevertheless, there are several significant differencesbetween our work and ColorFool.

Original: First, ColorFool requires manually selecting several humansensitive semantic classes (i.e., sky, water, plants, and humans), while our approach does not need.
Revised: First, ColorFool requires manually selecting several humansensitive semantic classes (i.e., sky, water, plants, and humans), while our approach does not need.

Original: Second, ColorFool adds uncontrolled perturbations on human-insensitive semantic classes, whileour approach utilizes a color distribution library as a guide.
Revised: Second, ColorFool adds uncontrolled perturbations on human-insensitive semantic classes, whileour approach utilizes a color distribution library as a guide.

Original: Third, ColorFool optimizes adversarialexamples only via multiple random trials, while our approach also utilizes the gradient of the substitutemodel to fine-tune results.
Revised: Third, ColorFool optimizes adversarialexamples only via multiple random trials, while our approach also utilizes the gradient of the substitutemodel to fine-tune results.

Original: 2.2 Adversarial Defenses
Revised: 2.2 Adversarial Defenses

Original: To mitigate the threat of adversarial examples, various defense methods have been proposed forthe past few years.
Revised: To mitigate the threat of adversarial examples, various defense methods have been proposed forthe past few years.

Original: Generally, existing defense techniques can fall into two categories: adversarialtraining [36, 37, 38] and input pre-process [39, 40, 41].
Revised: Generally, existing defense techniques can fall into two categories: adversarialtraining [34, 35, 36] and input pre-process [37, 38, 39].

Original: The former gains immunity to attacks byleveraging adversarial examples during the training phase.
Revised: The former gains immunity to attacks byleveraging adversarial examples during the training phase.

Original: For example, Tram ` e r et al.
Revised: For example, Tram ` e r et al.

Original: [37] improvethe black-box robustness by considering adversarial examples generated by other irrelevant models.
Revised: [35] improvethe black-box robustness by considering adversarial examples generated by other irrelevant models.

Original: Xie et al.
Revised: Xie et al.

Original: [38] introduce feature denoising modules and train them on adversarial examples to buildwhite-box robust models.
Revised: [36] introduce feature denoising modules and train them on adversarial examples to buildwhite-box robust models.

Original: Although adversarial training is usually the most robust strategy, it suffersfrom expensive training costs.
Revised: Although adversarial training is usually the most robust strategy, it suffersfrom expensive training costs.

Original: To overcome this limitation, the input pre-process branch is designedto cure the infection of adversarial examples before feeding to DNNs.
Revised: To overcome this limitation, the input pre-process branch is designedto cure the infection of adversarial examples before feeding to DNNs.

Original: For example, Guo et al.
Revised: For example, Guo et al.

Original: [39]introduce several input transformation techniques (e.g., JPEG compression [39]) to recover fromthe adversarial perturbations.
Revised: [37]introduce several input transformation techniques (e.g., JPEG compression [37]) to recover fromthe adversarial perturbations.

Original: Xie et al.
Revised: Xie et al.

Original: [42] propose R&P which mitigates the adversarial influencethrough random resizing and padding.
Revised: [40] propose R&P which mitigates the adversarial influencethrough random resizing and padding.

Original: Liao et al.
Revised: Liao et al.

Original: [43] come up with a high-level representationguided denoiser (HGD) to reduce the effect of adversarial perturbations.
Revised: [41] come up with a high-level representationguided denoiser (HGD) to reduce the effect of adversarial perturbations.

Original: Jia et al.
Revised: Jia et al.

Original: [40] propose anend-to-end image compression model called ComDefend against adversarial examples.
Revised: [38] propose anend-to-end image compression model called ComDefend against adversarial examples.

Original: Naseer etal.
Revised: Naseer etal.

Original: [41] train a neural representation purifier model (NRP) that cleans adversarial perturbation basedon the automatically derived supervision.
Revised: [39] train a neural representation purifier model (NRP) that cleans adversarial perturbation basedon the automatically derived supervision.

Original:  Methodology
Revised:  Methodology

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: Let x be a clean image with true label y and F θ ( · ) denote the deep learning classifier with parametersθ .
Revised: Let x be a clean image with true label y and F θ ( · ) denote the deep learning classifier with parametersθ .

Original: Formally, unrestricted attacks aim to craft human-imperceptible adversarial perturbations (may bevery large in RGB color space) for x so that the resulting adversarial examples x ′ can induce F θ ( · )to misclassify:
Revised: Formally, unrestricted attacks aim to craft human-imperceptible adversarial perturbations (may bevery large in RGB color space) for x so that the resulting adversarial examples x ′ can induce F θ ( · )to misclassify:

Original: [Equation]
Revised: [Equation]

Original: In this paper, we focus on a more challenging black-box scenario (compared to the white-boxscenario), where the target model’s information (e.g., parameters and structure) is inaccessible.
Revised: In this paper, we focus on a more challenging black-box scenario (compared to the white-boxscenario), where the target model’s information (e.g., parameters and structure) is inaccessible.

Original: Therefore, adversarial examples can only be crafted via the accessible substitute model F ϕ ( · ) andrely upon their transferability to fool target models.
Revised: Therefore, adversarial examples can only be crafted via the accessible substitute model F ϕ ( · ) andrely upon their transferability to fool target models.

Original: 3.2 Color Distribution Library
Revised: 3.2 Color Distribution Library

Original: Existing unrestricted color attacks tend to sacrifice their flexibility so that they can generate humanimperceptible adversarial examples.
Revised: Existing unrestricted color attacks tend to sacrifice their flexibility so that they can generate humanimperceptible adversarial examples.

Original: For example, ReColorAdv [25] requires constraining the perturbation to a relatively small range, which cannot take full advantage of the “unrestricted” setting;cAdv [27] enforces the color belonging to the low-entropy cluster to remain unchanged, whichinevitably reduces the attack space;
Revised: For example, ReColorAdv [24] requires constraining the perturbation to a relatively small range, which cannot take full advantage of the “unrestricted” setting;cAdv [26] enforces the color belonging to the low-entropy cluster to remain unchanged, whichinevitably reduces the attack space;

Original: ColorFool [24] manually splits an image into two parts and addscontrolled noises on the human-sensitive part, which largely depends on the authors’ intuition (but itvaries from person to person).
Revised: ColorFool [23] manually splits an image into two parts and addscontrolled noises on the human-sensitive part, which largely depends on the authors’ intuition (but itvaries from person to person).

Original: To remedy the above limitation, we borrow the idea of [44] to construct a “distribution of colordistributions” (DoD) for different semantic classes based on the ADE20K [30] dataset.
Revised: To remedy the above limitation, we borrow the idea of [42] to construct a “distribution of colordistributions” (DoD) for different semantic classes based on the ADE20K [29] dataset.

Original: Specifically,for each class, DoD provides 20 different dominant distribution sets (represented by the color distributions), which are obtained by a hierarchical clustering based on color palettes 1 of correspondingsegmentation class in the dataset.
Revised: Specifically,for each class, DoD provides 20 different dominant distribution sets (represented by the color distributions), which are obtained by a hierarchical clustering based on color palettes 1 of correspondingsegmentation class in the dataset.

Original: Since the color distributions in each distribution set are similar and averaging all color distributionsof a specific distribution set may not yield a natural representation, we randomly select a colordistribution to represent the overall color characteristics for simplicity.
Revised: Since the color distributions in each distribution set are similar and averaging all color distributionsof a specific distribution set may not yield a natural representation, we randomly select a colordistribution to represent the overall color characteristics for simplicity.

Original: Therefore, for a semanticclass ˜ y , its realistic color distribution space can be expressed as:
Revised: Therefore, for a semanticclass ˜ y , its realistic color distribution space can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: where c m ˜ y is a randomly sampled distribution from the m -th set, and our color distribution library Ccan be denoted as:
Revised: where c m ˜ y is a randomly sampled distribution from the m -th set, and our color distribution library Ccan be denoted as:

Original: [Equation]
Revised: [Equation]

Original: ℒ 𝑎𝑎𝑎𝑎𝑎𝑎 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐 … 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐
Revised: ℒ 𝑎𝑎𝑎𝑎𝑎𝑎 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐 … 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐

Original: Car ( ℂ 𝑐𝑐𝑎𝑎𝑐𝑐 ) 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚 … 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚
Revised: Car ( ℂ 𝑐𝑐𝑎𝑎𝑐𝑐 ) 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚 … 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚

Original: Mountain ( ℂ 𝑚𝑚𝑚𝑚𝑚𝑚 )
Revised: Mountain ( ℂ 𝑚𝑚𝑚𝑚𝑚𝑚 )

Original: Transfer Matrix 𝑻𝑻 … …
Revised: Transfer Matrix 𝑻𝑻 … …

Original: Color Distribution Library ℂ
Revised: Color Distribution Library ℂ

Original: Segm
Revised: Segm

Original: MK Solution
Revised: MK Solution

Original: Color Mapping [Figure] 𝒙𝒙 𝑯𝑯′ 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐
Revised: Color Mapping 𝒙𝒙 𝒙𝒙 𝑯𝑯′ 𝒄𝒄 𝑚𝑚𝑚𝑚𝑚𝑚 𝒄𝒄 𝑐𝑐𝑎𝑎𝑐𝑐

Original: Semantic Mask 𝒙𝒙 𝑯𝑯 …
Revised: Semantic Mask 𝒙𝒙 𝑯𝑯 …

Original: +
Revised: +

Original: Forward (Clean image)
Revised: Forward (Clean image)

Original: Forward (Image variant)
Revised: Forward (Image variant)

Original: Backward (Neighborhood search) 
Revised: Backward (Neighborhood search) Query

Revised: Randomly

Original: [Figure]
Revised: [Figure]

Original: 𝑯𝑯 ℱ 𝜙𝜙
Revised: 𝑯𝑯 ℱ 𝜙𝜙

Original: For the input image x , we first obtain its mask by a segmentation model (Segm).
Revised: For the input image x , we first obtain its mask by a segmentation model (Segm).

Original: Then we randomly assign a realistic color distribution c for each class via our color distribution library C and fuse them into a target H based on the proportion of corresponding area w in the image.
Revised: Then we randomly assign a realistic color distribution c for each class via our color distribution library C and fuse them into a target H based on the proportion of corresponding area w in the image.

Original: Here x H is an image (without spatial information) directly converted by H so that we can adopt Monge-Kantorovitch (MK) solution to obtain the transfer matrix T .
Revised: Here x H is an image (without spatial information) directly converted by H so that we can adopt Monge-Kantorovitch (MK) solution to obtain the transfer matrix T .

Original: With it and Eq. 4, we can get the color mapping image variant x ′ H .
Revised: With it and Eq. 4, we can get the color mapping image variant x ′ H .

Original: Finally, we fine-tune T (i.e., neighborhood search) according to the loss L adv so that more threatening adversarial examples can be mapped.
Revised: Finally, we fine-tune T (i.e., neighborhood search) according to the loss L adv so that more threatening adversarial examples can be mapped.

Original: where M denotes the total number of semantic classes contained in our library (see Appendix A forthe pipeline of building color distribution library).
Revised: where M denotes the total number of semantic classes contained in our library (see Appendix A forthe pipeline of building color distribution library).

Original: In the following, we adopt this color distributionlibrary to ensure the colors of adversarial examples are natural and coordinated, thus making ourattack more flexible.
Revised: In the following, we adopt this color distributionlibrary to ensure the colors of adversarial examples are natural and coordinated, thus making ourattack more flexible.

Original: 3.3 Natural Color Fool
Revised: 3.3 Natural Color Fool

Original: To generate natural adversarial examples, precisely controlling the color perception is necessary.
Revised: To generate natural adversarial examples, precisely controlling the color perception is necessary.

Original: Inthis paper, we craft adversarial perturbations in CIELab color space where the perception is moreuniform than RGB color space.
Revised: Inthis paper, we craft adversarial perturbations in CIELab color space where the perception is moreuniform than RGB color space.

Original: The framework of our proposed Natural Color Fool (NCF) isillustrated in Figure 1 (refer to Appendix B for Algorithm of NCF).
Revised: The framework of our proposed Natural Color Fool (NCF) isillustrated in Figure 1 (refer to Appendix B for Algorithm of NCF).

Original: Formally, for a given image x ,we first obtain a semantic mask through a semantic segmentation model.
Revised: Formally, for a given image x ,we first obtain a semantic mask through a semantic segmentation model.

Original: Then, for each semanticclass ˜ y in x , we randomly select a color distribution c i ˜ y from C ˜ y
Revised: Then, for each semanticclass ˜ y in x , we randomly select a color distribution c i ˜ y from C ˜ y

Original: in the color distribution library.
Revised: in the color distribution library.

Original: After that, we fuse the color distributions of all semantic classes based on the proportion of imagesoccupied by each class (e.g., w car , w mou in Figure 1), thus generating a target color distribution H .
Revised: After that, we fuse the color distributions of all semantic classes based on the proportion of imagesoccupied by each class (e.g., w car , w mou in Figure 1), thus generating a target color distribution H .

Original: With it, we can use the following Eq.
Revised: With it, we can use the following Eq.

Original: 4 to generate a target image variant x ′ H (refer to Figure 1)which approximately replaces the color distribution of each semantic class of x with correspondingone in target H [31]:
Revised: 4 to generate a target image variant x ′ H (refer to Figure 1)which approximately replaces the color distribution of each semantic class of x with correspondingone in target H [30]:

Original: [Equation]
Revised: [Equation]

Original: where T ∈ R 3 × 3 is the transfer matrix, x H is an image reconstructed by H but without spatialinformation 2 (see Figure 1), and µ x , µ x H ( ∈ R 3 ) , Σ x , Σ x H ( ∈ R 3 × 3 ) are channel means of x ,channel means of x H , the covariance matrix of x , and the covariance matrix of x H , respectively.
Revised: where T ∈ R 3 × 3 is the transfer matrix, x H is an image reconstructed by H but without spatialinformation 2 (see Figure 1), and µ x , µ x H ( ∈ R 3 ) , Σ x , Σ x H ( ∈ R 3 × 3 ) are channel means of x ,channel means of x H , the covariance matrix of x , and the covariance matrix of x H , respectively.

Original: Formally, there are numerous solutions [45, 46, 47, 48] for T .
Revised: Formally, there are numerous solutions [43, 44, 45, 46] for T .

Original: However, most of them may not obtainan intended color mapping, i.e., only color proportions are expected.
Revised: However, most of them may not obtainan intended color mapping, i.e., only color proportions are expected.

Original: To tackle this issue, we follow
Revised: To tackle this issue, we follow

Original: Piti ´e et al.
Revised: Piti ´e et al.

Original: [31] and convert color mapping into a Monge–Kantorovich transportation problem, thusgetting the solution for T :
Revised: [30] and convert color mapping into a Monge–Kantorovich transportation problem, thusgetting the solution for T :

Original: [Equation]
Revised: [Equation]

Original: Intuitively, relying solely on randomly picking color distribution may limit the attack success rate.
Revised: Intuitively, relying solely on randomly picking color distribution may limit the attack success rate.

Original: Since our color distribution library contains a wide variety of color combinations and we haveaccessible substitute models, there is no reason not to take advantage of them.
Revised: Since our color distribution library contains a wide variety of color combinations and we haveaccessible substitute models, there is no reason not to take advantage of them.

Original: Thus, we utilize thesubstitute model F ϕ ( · ) to select a most adversarial distribution from η different color distributionssampled from our color distribution library:
Revised: Thus, we utilize thesubstitute model F ϕ ( · ) to select a most adversarial distribution from η different color distributionssampled from our color distribution library:

Original: [Equation]
Revised: [Equation]

Original: where H is the set containing η different color distributions and L adv is the C&W loss [14]:
Revised: where H is the set containing η different color distributions and L adv is the C&W loss [14]:

Original: [Equation]
Revised: [Equation]

Original: where z j denotes the logit concerning the j -th class.
Revised: where z j denotes the logit concerning the j -th class.

Original: To further boost our attack, we propose twotechniques: neighborhood search and initialization reset .
Revised: To further boost our attack, we propose twotechniques: neighborhood search and initialization reset .

Original: Specifically, the former aims to slightlyadjust T (derived from Eq. 6) to improve the attack success rate while ensuring a natural mappingresult (see the comparison of adversarial examples with or without adopting neighborhood searchin Appendix E).
Revised: Specifically, the former aims to slightlyadjust T (derived from Eq. 6) to improve the attack success rate while ensuring a natural mappingresult (see the comparison of adversarial examples with or without adopting neighborhood searchin Appendix E).

Original: Thus, the neighborhood search can be formulated as the following optimizationproblem:
Revised: Thus, the neighborhood search can be formulated as the following optimizationproblem:

Original: [Equation]
Revised: [Equation]

Original: where ϵ denotes the maximal perturbation size for T .
Revised: where ϵ denotes the maximal perturbation size for T .

Original: In our paper, we adopt MI-FGSM [17] with Niterations to optimize Eq. 9.
Revised: In our paper, we adopt MI-FGSM [16] with Niterations to optimize Eq. 9.

Original: Nonetheless, an adversarial transfer matrix T ′ found in the neighborhoodof a specific T may not be effective, since the search space is indeed limited.
Revised: Nonetheless, an adversarial transfer matrix T ′ found in the neighborhoodof a specific T may not be effective, since the search space is indeed limited.

Original: To alleviate this issue, theinitialization reset repeats the operation of selecting one from η randomly sampled color distributions(i.e.
Revised: To alleviate this issue, theinitialization reset repeats the operation of selecting one from η randomly sampled color distributions(i.e.

Original: , Eq.
Revised: , Eq.

Original: K times, thus obtaining K ideal initial color distributions for the neighborhood search.
Revised: K times, thus obtaining K ideal initial color distributions for the neighborhood search.

Original: Then we choose the best one among them:
Revised: Then we choose the best one among them:

Original: [Equation]
Revised: [Equation]

Original: where H ∗ i and T ′ i denote the resulting color distribution and corresponding transfer matrix of thei -th reset.
Revised: where H ∗ i and T ′ i denote the resulting color distribution and corresponding transfer matrix of thei -th reset.

Original: Consequently, resulting adversarial examples can be obtained by:
Revised: Consequently, resulting adversarial examples can be obtained by:

Original: [Equation]
Revised: [Equation]

Original: In Figure 2, we show resulting adversarial examples generated by our proposed method and recentworks, i.e., cAdv [27], ColorFool [24], and ACE [26] (see Appendix G for more comparisons).
Revised: In Figure 2, we show resulting adversarial examples generated by our proposed method and recentworks, i.e., cAdv [26], ColorFool [23], and ACE [25] (see Appendix G for more comparisons).

Original: Notably, despite our NCF implements significant modifications to the color distribution of the originalimages, the adversarial examples still look very natural.
Revised: Notably, despite our NCF implements significant modifications to the color distribution of the originalimages, the adversarial examples still look very natural.

Original:  Experiments
Revised:  Experiments

Original: In this section, we first introduce the setup for our experiments in Section 4.1.
Revised: In this section, we first introduce the setup for our experiments in Section 4.1.

Original: Then we report attacksuccess rates on normally trained and defense models in Section 4.2.
Revised: Then we report attacksuccess rates on normally trained and defense models in Section 4.2.

Original: After that, we compare theimage quality of adversarial examples in Section 4.3.
Revised: After that, we compare theimage quality of adversarial examples in Section 4.3.

Original: Finally, we conduct a series of ablation studiesin Section 4.4.
Revised: Finally, we conduct a series of ablation studiesin Section 4.4.

Original: To better analyze the adversarial example, we also provide an illustration of attentionshift [49] in Appendix F.
Revised: To better analyze the adversarial example, we also provide an illustration of attentionshift [47] in Appendix F.

Original: 4.1 Setup
Revised: 4.1 Setup

Original: Dataset.
Revised: Dataset.

Original: We conduct our experiments on the ImageNet-compatible Dataset 3 .
Revised: We conduct our experiments on the ImageNet-compatible Dataset 3 .

Original: This dataset iscomprised of 1,000 images and is widely used in recent transfer-based attacks [22, 18, 19, 50, 51, 52].
Revised: This dataset iscomprised of 1,000 images and is widely used in recent transfer-based attacks [21, 17, 18, 48, 49, 50].

Original: [Figure]
Revised: [Figure]

Original: Models.
Revised: Models.

Original: In this paper, we evaluate the vulnerability of various convolutional neural networks(CNNs) and vision transformers (ViTs).
Revised: In this paper, we evaluate the vulnerability of various convolutional neural networks(CNNs) and vision transformers (ViTs).

Original: For CNNs, we consider both normally trained modelsand defense models.
Revised: For CNNs, we consider both normally trained modelsand defense models.

Original: Normally trained models include VGG-19 [53], ResNet-18 (Res-18), ResNet50 (Res-50) [2], Inception-v3 (Inc-v3) [54], DenseNet-121 (Dense-121) [10], and MobileNet-v2(Mobile-v2) [55].
Revised: Normally trained models include VGG-19 [51], ResNet-18 (Res-18), ResNet50 (Res-50) [2], Inception-v3 (Inc-v3) [52], DenseNet-121 (Dense-121) [10], and MobileNet-v2(Mobile-v2) [53].

Original: Defense methods contain JPEG [39], Gray (gray-scale conversion), HGD [43],
Revised: Defense methods contain JPEG [37], Gray (gray-scale conversion), HGD [41],

Original: R&P[42], ComDefend [40],  NRP [41], 4 Inc- v3 ens3 , IncRes- v2 ens [37], Res152 B , Res152 D and
Revised: R&P [40], ComDefend [38], NRP [39], 4 Inc- v3 ens3 , IncRes- v2 ens [35], Res152 B , Res152 D and

Original: ResNeXt DA [38].
Revised: ResNeXt DA [36].

Original: For ViT, we consider normally trained ViT-S/16 (ViT-S) [56], XCiT-N12/16(XCiT-N12) [57], and DeiT-S [56].
Revised: For ViT, we consider normally trained ViT-S/16 (ViT-S) [54], XCiT-N12/16(XCiT-N12) [55], and DeiT-S [54].

Original: Implementation Details.
Revised: Implementation Details.

Original: In all experiments, the semantic segmentation model is Swin-T [58](the influence of different segmentation models on the results can be found in Appendix H) andthe color distribution library size M = 150 , the number of random searches η = 50 , the iterationof neighborhood search N = 15 , the maximum perturbation of transfer matrix ϵ = 0 .
Revised: In all experiments, the semantic segmentation model is Swin-T [56] andthe color distribution library size M = 150 , the number of random searches η = 50 , the iterationof neighborhood search N = 15 , the maximum perturbation of transfer matrix ϵ = 0 .

Original: 2 , the stepsize α
Revised: 2 , the stepsize α

Original: = ϵ/N ≈ 0 .
Revised: = ϵ/N ≈ 0 .

Original: 013 , the momentum u = 0 .
Revised: 013 , the momentum u = 0 .

Original: 6 (for MI-FGSM), and the reset number K = 10(see Appendix C for ablation of K ).
Revised: 6 (for MI-FGSM), and the reset number K = 10(see Appendix C for ablation of K ).

Original: We compare our proposed method with Semantic Adversarial
Revised: We compare our proposed method with Semantic Adversarial

Original: Examples (SAE) [23], ReColorAdv [25], cAdv [27], ColorFool [24], and ACE [26].
Revised: Examples (SAE) [22], ReColorAdv [24], cAdv [26], ColorFool [23], and ACE [25].

Original: The parametersof each competitor follow the corresponding default setting.
Revised: The parametersof each competitor follow the corresponding default setting.

Original: Our experiments are run on an NVIDIA
Revised: Our experiments are run on an NVIDIA

Original: TITAN
Revised: TITAN

Original: Xp GPU with 12GB of memory.
Revised: Xp GPU with 12GB of memory.

Original: 4.2 Transferability Comparison
Revised: 4.2 Transferability Comparison4.2.1 Results on Normally Trained Models

Original: 4.2.1 Results on Normally Trained Models

Original: In this section, we compare our proposed method with SA [23], ReColorAdv [25], cAdv [27],
Revised: In this section, we compare our proposed method with SA [22], ReColorAdv [24], cAdv [26],

Original: ColorFool [24] and ACE [26] on a variety of normally trained CNNs and Vision Transformers (ViTs).
Revised: ColorFool [23] and ACE [25] on a variety of normally trained CNNs and Vision Transformers (ViTs).

Original: Adversarial examples are crafted via Res-18, VGG-19, Mobile-V2, and Inc-V3, respectively.
Revised: Adversarial examples are crafted via Res-18, VGG-19, Mobile-V2, and Inc-V3, respectively.

Original: For theresults of ensemble attack [17] and using ViTs as the substitute model, please refer to AppendixI and D.
Revised: Forresults with ViTs as the substitute model, please refer to Appendix D.

Original: The attack success rates are presented in Table 1.
Revised: The attack success rates are presented in Table 1.

Original: We can observe that our resulting adversarialexamples usually achieve the highest transferability compared to those generated via state-of-the-artcompetitors.
Revised: We can observe that our resulting adversarialexamples usually achieve the highest transferability compared to those generated via state-of-the-artcompetitors.

Original: For example, when the substitute model is Res-18, only 46.9%, 39.9%, and 40.8%of adversarial examples crafted by SA, ReColorAdv, and cAdv can successfully fool VGG-19.
Revised: For example, when the substitute model is Res-18, only 46.9%, 39.9%, and 40.8%of adversarial examples crafted by SA, ReColorAdv, and cAdv can successfully fool VGG-19.

Original: Incomparison, our NCF can achieve a much higher transferability of 72.1% .
Revised: Incomparison, our NCF can achieve a much higher transferability of 72.1% .

Original: Besides, we also considerthe transferability from CNNs to ViTs, as their structures are quite different.
Revised: Besides, we also considerthe transferability from CNNs to ViTs, as their structures are quite different.

Original: As shown in Table 1,when transferring adversarial examples from Mobile-V2 to XCiT-N12, recent ColorFool and ACEonly obtain success rates of 22.8% and 22.6%, respectively, while our method can get 56.4% successrate.
Revised: As shown in Table 1,when transferring adversarial examples from Mobile-V2 to XCiT-N12, recent ColorFool and ACEonly obtain success rates of 22.8% and 22.6%, respectively, while our method can get 56.4% successrate.

Original: On average, adversarial examples crafted by our method are capable of achieving a 54.5% success rate, which significantly outperforms state-of-the-art approaches by 15.0% ∼ 32.9% .
Revised: On average, adversarial examples crafted by our method are capable of achieving a 54.5%success rate, which significantly outperforms state-of-the-art approaches by 15.0% ∼ 32.9% .

Original: Thisresult convincingly demonstrates the effectiveness of our method in fooling normally trained models.
Revised: Thisresult convincingly demonstrates the effectiveness of our method in fooling normally trained models.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.2.2 Results on Defense Models
Revised: 4.2.2 Results on Defense Models

Original: Threats posed by L p -norm adversarial attacks [7, 17, 19, 51] promote the emergence of adversarialdefenses [36, 38, 41].
Revised: Threats posed by L p -norm adversarial attacks [7, 16, 18, 49] promote the emergence of adversarialdefenses [34, 36, 39].

Original: However, whether these defense mechanisms are robust against unrestrictedattacks has not been extensively explored.
Revised: However, whether these defense mechanisms are robust against unrestrictedattacks has not been extensively explored.

Original: Therefore, in this section, we consider both input preprocess defenses (i.e., JPEG [39], Gray, R&P [42], HGD [43], ComDefend [40], and NRP [41]) andadversarially trained models (i.e., Inc-v3 ens3 , IncRes-v2 ens , Res B [37], Res D , and ResNext DA [38])to fully investigate the performance of our method.
Revised: Therefore, in this section, we consider both input preprocess defenses (i.e., JPEG [37], Gray, R&P [40], HGD [41], ComDefend [38], and NRP [39]) andadversarially trained models (i.e., Inc-v3 ens3 , IncRes-v2 ens , Res B [35], Res D , and ResNext DA [36])to fully investigate the performance of our method.

Original: The black-box success rates of unrestricted attacks are reported in Table 2.
Revised: The black-box success rates of unrestricted attacks are reported in Table 2.

Original: Similar to Table 1, ourapproach still consistently outperforms existing advanced methods by a large margin.
Revised: Similar to Table 1, ourapproach still consistently outperforms existing advanced methods by a large margin.

Original: On average,our proposed NCF is capable of making 48.1% adversarial examples fool these defenses, while SA,
Revised: On average,our proposed NCF is capable of making 48.1% adversarial examples fool these defenses, while SA,

Original: ReColorAdv, cAdv, ColorFool, and ACE only obtain success rates of 38.1%, 22.8%, 27.9%, 34.8%,and 24.6%, respectively.
Revised: ReColorAdv, cAdv, ColorFool, and ACE only obtain success rates of 38.1%, 22.8%, 27.9%, 34.8%,and 24.6%, respectively.

Original: Another interesting observation from Table 2 is that advanced defensealgorithms are not necessarily effective when faced with unrestricted adversarial examples.
Revised: Another interesting observation from Table 2 is that advanced defensealgorithms are not necessarily effective when faced with unrestricted adversarial examples.

Original: Forexample, even white-box robust feature denoising models [38] are vulnerable, e.g., 60.1% adversarial examples can fool ResNeXt DA .
Revised: Forexample, even white-box robust feature denoising models [36] are vulnerable, e.g., 60.1% adversarialexamples can fool ResNeXt DA .

Original: We speculate the main reason is that the defense mechanism isintermediate feature denoising, while unrestricted color attacks mainly manipulate the color and donot generate significant high-frequency noise.
Revised: We speculate the main reason is that the defense mechanism isintermediate feature denoising, while unrestricted color attacks mainly manipulate the color and donot generate significant high-frequency noise.

Original: In addition, we notice that many input pre-process defenses with target model VGG-19 cannot defendagainst our proposed NCF, but instead boost the transferability of adversarial examples (compareto the VGG-19 results of Table 1).
Revised: In addition, we notice that many input pre-process defenses with target model VGG-19 cannot defendagainst our proposed NCF, but instead boost the transferability of adversarial examples (compareto the VGG-19 results of Table 1).

Original: To better analyze this abnormal phenomenon, we visualize thepurified images generated by JPEG, ComDefend, and NRP in Figure 3.
Revised: To better analyze this abnormal phenomenon, we visualize thepurified images generated by JPEG, ComDefend, and NRP in Figure 3.

Original: Interestingly, we observethat all defenses have little effect on our unrestricted color perturbations.
Revised: Interestingly, we observethat all defenses have little effect on our unrestricted color perturbations.

Original: It is mainly because ouradversarial examples are crafted using realistic colors as a guide and, therefore,
Revised: It is mainly because ouradversarial examples are crafted using realistic colors as a guide and, therefore,

Original: can circumvent thesedefenses designed for common perturbation characteristics.
Revised: can circumvent thesedefenses designed for common perturbation characteristics.

Original: Besides, we observe that JPEG and NRPsuppress key features of “Great egret”, and ComDefend generates many irrelevant repeating patterns.
Revised: Besides, we observe that JPEG and NRPsuppress key features of “Great egret”, and ComDefend generates many irrelevant repeating patterns.

Original: As discussed in [59], these side effects of defense methods can reduce the confidence of the modelconcerning the true class, resulting in our adversarial examples instead of boosting the transferabilityafter being processed by these approaches.
Revised: As discussed in [57], these side effects of defense methods can reduce the confidence of the modelconcerning the true class, resulting in our adversarial examples instead of boosting the transferabilityafter being processed by these approaches.

Original: This result also reminds us that the robustness evaluationof defenses needs to be more comprehensive.
Revised: This result also reminds us that the robustness evaluationof defenses needs to be more comprehensive.

Original: [Figure]
Revised: [Figure]

Original: 4.3 Image Quality
Revised: 4.3 Image Quality

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: Clean

Original: [Table]
Revised: [Table]

Original: cAdv

Original: [Table] [Figure] [Table] Following ColorFool [24], we quantitatively evaluate image quality by a non-reference perceptual image qualitymeasure called neural image assessment (NIMA)
Revised: Following ColorFool [23], we quantitatively evaluate image quality by a non-reference perceptual image qualitymeasure called neural image assessment (NIMA)

Original: [32]which is highly correlated with human perception.
Revised: [31]which is highly correlated with human perception.

Original: Inour paper, NIMA is based on the architecture of MobileNet [60] and is a composite of two models: one trainson AVA [61] for image technical assessment and anothertrains on TID2013 [62] for image aesthetics assessment.
Revised: Inour paper, NIMA is based on the architecture of MobileNet [58] and is a composite of two models: one trainson AVA [59] for image technical assessment and anothertrains on TID2013 [60] for image aesthetics assessment.

Original: As reported in Table 3, image quality assessment results ofour NCF are on par with those of state-of-the-art competitors.
Revised: As reported in Table 3, image quality assessment results ofour NCF are on par with those of state-of-the-art competitors.

Original: For NIMA (technical), both ACE and our NCF obtaina high score of over 4.939 , which is just 0.094 lower thanclean images.
Revised: For NIMA (technical), both ACE and our NCF obtaina high score of over 4.939 , which is just 0.094 lower thanclean images.

Original: The case of NIMA (aesthetic) is similar,where aesthetic scores of our adversarial examples and clean images differ by only 0.097 , indicatingthat our NCF does not destroy the perceptual quality of images.
Revised: The case of NIMA (aesthetic) is similar,where aesthetic scores of our adversarial examples and clean images differ by only 0.097 , indicatingthat our NCF does not destroy the perceptual quality of images.

Original: 4.4 Ablation Studies
Revised: 4.4 Ablation Studies

Original: About neighborhood search and initialization reset.
Revised: About neighborhood search and initialization reset.

Original: In this section, we investigate the effect ofneighborhood search and initialization reset.
Revised: In this section, we investigate the effect ofneighborhood search and initialization reset.

Original: Specifically, we use Inc-v3 to craft adversarial examples,and then evaluate the transferability towards Res-18, VGG-19, Mobile-v2, Dense-121, Res-50, ViT-S,
Revised: Specifically, we use Inc-v3 to craft adversarial examples,and then evaluate the transferability towards Res-18, VGG-19, Mobile-v2, Dense-121, Res-50, ViT-S,

Original: XCiT-N12, and DeiT-S.
Revised: XCiT-N12, and DeiT-S.

Original: The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: In terms of the white-box attack, NCFoutperforms NCF-NS (NCF without NS), NCF-IR (NCF without IR), and NCF-IR-NS (NCF without
Revised: In terms of the white-box attack, NCFoutperforms NCF-NS (NCF without NS), NCF-IR (NCF without IR), and NCF-IR-NS (NCF without

Original: IR and NS) by 17.7% , 13.3% , and 32.2% , respectively.
Revised: IR and NS) by 17.7% , 13.3% , and 32.2% , respectively.

Original: On the black-box attacks, neighborhoodsearch and initialization reset bring an average performance improvement of 8.3% , and 6.5% .
Revised: On the black-box attacks, neighborhoodsearch and initialization reset bring an average performance improvement of 8.3% , and 6.5% .

Original: Whenboth neighborhood search and initialization reset techniques are used, the transferability towardsblack-box models can be improved by an average of 12.5% .
Revised: Whenboth neighborhood search and initialization reset techniques are used, the transferability towardsblack-box models can be improved by an average of 12.5% .

Original: The significant performance gain confirms the effectiveness of neighborhood search and initialization reset.
Revised: The significant performance gainconfirms the effectiveness of neighborhood search and initialization reset.

Original: In addition, we notice thateven if neither proposed technique is used, our method still surpasses state-of-the-art competitors,except for SAE.
Revised: In addition, we notice thateven if neither proposed technique is used, our method still surpasses state-of-the-art competitors,except for SAE.

Original: We attribute this phenomenon to the flexibility of our approach, which can naturallymodify the whole image without restriction.
Revised: We attribute this phenomenon to the flexibility of our approach, which can naturallymodify the whole image without restriction.

Original: Note that NCF-IR-NS does not mean selecting randomcolors to attack.

Original: The difference of them is discussed in Appendix J.

Original: [Table]
Revised: [Table]

Original: About color distribution library size.
Revised: About color distribution library size.

Original: In this section, we investigate the influence of color distribution library size M on attack success rates.
Revised: In this section, we investigate the influence of color distribution library size M on attack success rates.

Original: Specifically, we tune M from 0 to 150 with an intervalof 15 and freeze other hyper-parameters, and adversarial examples are crafted via Inc-v3.
Revised: Specifically, we tune M from 0 to 150 with an intervalof 15 and freeze other hyper-parameters, and adversarial examples are crafted via Inc-v3.

Original: Note that
Revised: Note that

Original: M = 0 means that our proposed library is not used, thus T is an identity matrix.
Revised: M = 0 means that our proposed library is not used, thus T is an identity matrix.

Original: The attack successrate is reported in Figure 4, where the dashed line indicates the white-box attack success rate andsolid lines refer to the black-box ones.
Revised: The attack successrate is reported in Figure 4, where the dashed line indicates the white-box attack success rate andsolid lines refer to the black-box ones.

Original: It can be observed that the library size M plays a key role the number of semantic classes M .
Revised: It can be observed that the library size M plays a key role in attack success rates.

Original: The substitute model is Inc-v3.

Original: in attack success rates. Concretely, the attacksuccess rates increase rapidly at first, then gradually stabilize when the library size exceeds30, and reach the maximum at M = 150 inmost cases.
Revised: Concretely, the attacksuccess rates increase rapidly at first, then gradually stabilize when the library size exceeds30, and reach the maximum at M = 150 inmost cases.

Original: For instance, when M = 150 ,
Revised: For instance, when M = 150 ,

Original: NCF has success rates of 57.7%, 57.6%, 56.8%,40.1%, and 47.6% on Res-18, VGG-19, Mobilev2, Dense-121, and Res-50, respectively.
Revised: NCF has success rates of 57.7%, 57.6%, 56.8%,40.1%, and 47.6% on Res-18, VGG-19, Mobilev2, Dense-121, and Res-50, respectively.

Original: When
Revised: When

Original: M = 0 , it significantly degrades to 24.6%,18.6%, 21.1%, 14.8%, and 14.2%, respectively.
Revised: M = 0 , it significantly degrades to 24.6%,18.6%, 21.1%, 14.8%, and 14.2%, respectively.

Original: Intuitively, this is because the target color distribution trivially turns to be the correspondingdistribution of the original image, and thus thespace we perturb on the original image will bereduced.
Revised: Intuitively, this is because the target color distribution trivially turns to be the correspondingdistribution of the original image, and thus thespace we perturb on the original image will bereduced.

Original: This convincingly demonstrates theeffectiveness of building a class-rich color distribution library.
Revised: This convincingly demonstrates theeffectiveness of building a class-rich color distribution library.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a black-box unrestricted color attack called Natural Color Fool (NCF).
Revised: In this paper, we propose a black-box unrestricted color attack called Natural Color Fool (NCF).

Original: Different from existing works which either rely on intuition and objective metrics or use relativelysmall changes to yield natural results, our method instead exploits color distribution to craft humanimperceptible, flexible, and highly transferable adversarial examples.
Revised: Different from existing works which either rely on intuition and objective metrics or use relativelysmall changes to yield natural results, our method instead exploits color distribution to craft humanimperceptible, flexible, and highly transferable adversarial examples.

Original: Extensive experiments andvisualizations demonstrate the effectiveness of our NCF.
Revised: Extensive experiments andvisualizations demonstrate the effectiveness of our NCF.

Original: Notably, our adversarial examples caneasily evade current L p robust defense techniques.
Revised: Notably, our adversarial examples caneasily evade current L p robust defense techniques.

Original: Furthermore, the transferability of our adversarialexamples is even boosted after being processed by several input pre-process defenses.
Revised: Furthermore, the transferability of our adversarialexamples is even boosted after being processed by several input pre-process defenses.

Original: We hope ourwork can draw researchers’ attention to unrestricted color attacks.
Revised: We hope ourwork can draw researchers’ attention to unrestricted color attacks.

Original: Limitation.
Revised: Limitation.

Original: In our paper, for each semantic class, we provide 20 different color distributions forchoosing.
Revised: In our paper, for each semantic class, we provide 20 different color distributions forchoosing.

Original: Therefore, this library is discrete, which may be insufficient to model the whole colorspace.
Revised: Therefore, this library is discrete, which may be insufficient to model the whole colorspace.

Original: In the future, we will try to build a continuous color space to expand the spatial range andmore accurately simulate the color range of objects.
Revised: In the future, we will try to build a continuous color space to expand the spatial range andmore accurately simulate the color range of objects.

Original: Negative Societal Impacts.
Revised: Negative Societal Impacts.

Original: Adversarial examples crafted by our proposed unrestricted color attacklook very natural but are with strong black-box transferability even towards defenses.
Revised: Adversarial examples crafted by our proposed unrestricted color attacklook very natural but are with strong black-box transferability even towards defenses.

Original: Therefore,unscrupulous people may adopt our method to undermine real-world applications, which inevitablyraises new concerns about AI safety.
Revised: Therefore,unscrupulous people may adopt our method to undermine real-world applications, which inevitablyraises new concerns about AI safety.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] SeeSection 5. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] The code and
Revised: [Yes] The code and

Original: instructions have been submitted as the supplemental material.
Revised: instructions have been submitted as the supplemental material.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Our experiments did not involve any model training, but all thealgorithm details were specified.
Revised: [Yes] Our experiments did not involve any model training, but all thealgorithm details were specified.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No] We fixed the random seed for all the experiments.
Revised: [No] We fixed the random seed for all the experiments.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g.,type of GPUs, internal cluster, or cloud provider)?
Revised: g.,type of GPUs, internal cluster, or cloud provider)?

Original: [No] We did not mention the runtime of the attacks due to the space limitation, but our hardware setting is available in
Revised: [No] We did not mention the run time of the attacks due to the space limitation, but our hardware setting is available in

Original: Section 4.1.
Revised: Section 4.1.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] see Section 4.1.
Revised: [Yes] see Section 4.1.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: In a footnote of Section 4.1.
Revised: In a footnote of Section 4.1.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: [Table]

Original: We did not use any new assets.
Revised: We did not use any new assets.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: In a footnote of Section 4.1.
Revised: In a footnote of Section 4.1.

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [No]
Revised: [No]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]