Original: D OMAIN G ENERALIZATION WITH M IX S TYLE
Revised: D OMAIN G ENERALIZATION WITH M IX S TYLE

Original: A BSTRACT
Revised: BSTRACT

Original: Though convolutional neural networks (CNNs) have demonstrated remarkable ability in learning discriminative features, they often generalize poorly to unseen domains.
Revised: Though convolutional neural networks (CNNs) have demonstrated remarkable ability in learning discriminative features, they often generalize poorly to unseen domains.

Original: Domain generalization aims to address this problem by learning from a set of source domains a model that is generalizable to any unseen domain.
Revised: Domain generalization aims to address this problem by learning from a set of source domains a model that is generalizable to any unseen domain.

Original: In this paper, a novel approach is proposed based on probabilistically mixing instancelevel feature statistics of training samples across source domains.
Revised: In this paper, a novel approach is proposed based on probabilistically mixing instancelevel feature statistics of training samples across source domains.

Original: Our method, termed MixStyle, is motivated by the observation that visual domain is closely related to image style (e.g., photo vs. sketch images).
Revised: Our method, termed MixStyle, is motivated by the observation that visual domain is closely related to image style (e.g., photo vs. sketch images).

Original: Such style information is captured by the bottom layers of a CNN where our proposed style-mixing takes place.
Revised: Such style information is captured by the bottom layers of a CNN where our proposed style-mixing takes place.

Original: Mixing styles of training instances results in novel domains being synthesized implicitly, which increase the domain diversity of the source domains, and hence the generalizability of the trained model.
Revised: Mixing styles of training instances results in novel domains being synthesized implicitly, which increase the domain diversity of the source domains, and hence the generalizability of the trained model.

Original: MixStyle ﬁts into mini-batch training perfectly and is extremely easy to implement.
Revised: MixStyle ﬁts into mini-batch training perfectly and is extremely easy to implement.

Original: The effectiveness of MixStyle is demonstrated on a wide range of tasks including category classiﬁcation, instance retrieval and reinforcement learning.
Revised: The effectiveness of MixStyle is demonstrated on a wide range of tasks including category classiﬁcation, instance retrieval and reinforcement learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Key to automated understanding of digital images is to compute a compact and informative feature representation.
Revised: Key to automated understanding of digital images is to compute a compact and informative feature representation.

Original: Deep convolutional neural networks (CNNs) have demonstrated remarkable ability in representation learning, proven to be effective in many visual recognition tasks, such as classifying photo images into 1,000 categories from ImageNet (Krizhevsky et al., 2012) and playing Atari games with reinforcement learning (Mnih et al., 2013).
Revised: Deep convolutional neural networks (CNNs) have demonstrated remarkable ability in representation learning, proven to be effective in many visual recognition tasks, such as classifying photo images into 1,000 categories from ImageNet (Krizhevsky et al., 2012) and playing Atari games with reinforcement learning (Mnih et al., 2013).

Original: However, it has long been discovered that the success of CNNs heavily relies on the i.i.d.
Revised: However, it has long been discovered that the success of CNNs heavily relies on the i.i.d.

Original: assumption, i.e. training and test data should be drawn from the same distribution; when such an assumption is violated even just slightly, as in most realworld application scenarios, severe performance degradation is expected (Hendrycks & Dietterich, 2019; Recht et al., 2019).
Revised: assumption, i.e. training and test data should be drawn from the same distribution; when such an assumption is violated even just slightly, as in most realworld application scenarios, severe performance degradation is expected (Hendrycks & Dietterich, 2019; Recht et al., 2019).

Original: Domain generalization (DG) aims to address such a problem (Zhou et al., 2021;
Revised: Domain generalization (DG) aims to address such a problem (Zhou et al., 2021;

Original: Blanchard et al., 2011; Muandet et al., 2013; Li et al., 2018a;
Revised: Blanchard et al., 2011; Muandet et al., 2013; Li et al., 2018a;

Original: Zhou et al., 2020b; Balaji et al., 2018; Dou et al., 2019;
Revised: Zhou et al., 2020b; Balaji et al., 2018; Dou et al., 2019;

Original: Carlucci et al., 2019).
Revised: Carlucci et al., 2019).

Original: In particular, assuming that multiple source domains containing the same visual classes are available for model training, the goal of DG is to learn models that are robust against data distribution changes across domains, known as domain shift, so that the trained model can generalize well to any unseen domains.
Revised: In particular, assuming that multiple source domains containing the same visual classes are available for model training, the goal of DG is to learn models that are robust against data distribution changes across domains, known as domain shift, so that the trained model can generalize well to any unseen domains.

Original: Compared to the closely related and more widely studied domain adaptation (DA) problem, DG is much harder in that no target domain data is available for the model to analyze the distribution shift in order to overcome the negative effects.
Revised: Compared to the closely related and more widely studied domain adaptation (DA) problem, DG is much harder in that no target domain data is available for the model to analyze the distribution shift in order to overcome the negative effects.

Original: Instead, a
Revised: Instead, a

Original: DG model must rely on the source domains and focus on learning domain-invariant feature representation in the hope that it would remain discriminative given target domain data.
Revised: DG model must rely on the source domains and focus on learning domain-invariant feature representation in the hope that it would remain discriminative given target domain data.

Original: A straightforward solution to DG is to expose a model with a large variety of source domains.
Revised: A straightforward solution to DG is to expose a model with a large variety of source domains.

Original: Specifically, the task of learning domain-invariant and thus generalizable feature representation becomes easier when data from more diverse source domains are available for the model.
Revised: Specifically, the task of learning domain-invariant and thus generalizable feature representation becomes easier when data from more diverse source domains are available for the model.

Original: This would reduce the burden on designing special models or learning algorithms for DG.
Revised: This would reduce the burden on designing special models or learning algorithms for DG.

Original: Indeed, model training with[Figure] large-scale data of diverse domains is behind the success of existing commercial face recognition or vision-based autonomous driving systems.
Revised: Indeed, model training with large-scale data of diverse domains is behind the success of existing commercial face recognition or vision-based autonomous driving systems.

Original: A recent work by Xu et al.
Revised: A recent work by Xu et al.

Original: (2021) also emphasizes the  importance of diverse training distributions for out-of-distribution generalization.
Revised: (2021) also emphasizes the [Figure] importance of diverse training distributions for out-of-distribution generalization.

Original: However, collecting data of a large variety of domains is often costly or even impossible.
Revised: However, collecting data of a large variety of domains is often costly or even impossible.

Original: It thus cannot be a general solution to DG.
Revised: It thus cannot be a general solution to DG.

Original: In this paper, a novel approach is proposed based on probabilistically mixing instance-level feature statistics of training samples across source domains.
Revised: In this paper, a novel approach is proposed based on probabilistically mixing instance-level feature statistics of training samples across source domains.

Original: Our model, termed MixStyle , is motivated by the observation that visual domain is closely related to image style.
Revised: Our model, termed MixStyle , is motivated by the observation that visual domain is closely related to image style.

Original: An example is shown in
Revised: An example is shown in

Original: Fig.
Revised: Fig.

Original: 1: the four images from four different domains depict the same semantic concept, i.e. dog, but with distinctive styles (e.g., characteristics in color and texture).
Revised: 1: the four images from four different domains depict the same semantic concept, i.e. dog, but with distinctive styles (e.g., characteristics in color and texture).

Original: When these images are fed into a deep CNN, which maps the raw pixel values into category labels, such style information is removed at the output.
Revised: When these images are fed into a deep CNN, which maps the raw pixel values into category labels, such style information is removed at the output.

Original: However, recent style transfer studies (Huang & Belongie, 2017; Dumoulin et al., 2017) suggest that such style information is preserved at the bottom layers of the CNN through the instance-level feature statistics, as shown clearly in Fig.
Revised: However, recent style transfer studies (Huang & Belongie, 2017; Dumoulin et al., 2017) suggest that such style information is preserved at the bottom layers of the CNN through the instance-level feature statistics, as shown clearly in Fig.

Original: Importantly, since replacing such statistics would lead to replaced style while preserving the semantic content of the image, it is reasonable to assume that mixing styles from images of different domains would result in images of (mixed) new styles.
Revised: Importantly, since replacing such statistics would lead to replaced style while preserving the semantic content of the image, it is reasonable to assume that mixing styles from images of different domains would result in images of (mixed) new styles.

Original: That is, more diverse domains/styles can be made available for training a more domain-generalizable model.
Revised: That is, more diverse domains/styles can be made available for training a more domain-generalizable model.

Original: Concretely, our MixStyle randomly selects two instances of different domains and adopts a probabilistic convex combination between instance-level feature statistics of bottom CNN layers.
Revised: Concretely, our MixStyle randomly selects two instances of different domains and adopts a probabilistic convex combination between instance-level feature statistics of bottom CNN layers.

Original: In contrast to style transfer work (Huang & Belongie, 2017; Dumoulin et al., 2017), no explicit image synthesis is necessary meaning much simpler model design.
Revised: In contrast to style transfer work (Huang & Belongie, 2017; Dumoulin et al., 2017), no explicit image synthesis is necessary meaning much simpler model design.

Original: Moreover, MixStyle perfectly ﬁts into modern mini-batch training.
Revised: Moreover, MixStyle perfectly ﬁts into modern mini-batch training.

Original: Overall, it is very easy to implement with only few lines of code.
Revised: Overall, it is very easy to implement with only few lines of code.

Original: To evaluate the effectiveness as well as the general applicability of MixStyle, we conduct extensive experiments on a wide spectrum of datasets covering category classiﬁcation (Sec. 3.1), instance retrieval (Sec. 3.2), and reinforcement learning (Sec. 3.3).
Revised: To evaluate the effectiveness as well as the general applicability of MixStyle, we conduct extensive experiments on a wide spectrum of datasets covering category classiﬁcation (Sec. 3.1), instance retrieval (Sec. 3.2), and reinforcement learning (Sec. 3.3).

Original: The results demonstrate that MixStyle can signiﬁcantly improve CNNs’ cross-domain generalization performance.
Revised: The results demonstrate that MixStyle can signiﬁcantly improve CNNs’ cross-domain generalization performance.

Original:  M ETHODOLOGY 2. B ACKGROUND
Revised:  M ETHODOLOGY 2. B ACKGROUND

Original: Normalizing feature tensors with instance-speciﬁc mean and standard deviation has been found effective for removing image style in style transfer models (Ulyanov et al., 2016; Huang & Belongie, 2017; Dumoulin et al., 2017).
Revised: Normalizing feature tensors with instance-speciﬁc mean and standard deviation has been found effective for removing image style in style transfer models (Ulyanov et al., 2016; Huang & Belongie, 2017; Dumoulin et al., 2017).

Original: Such an operation is widely known as instance normalization (IN, Ulyanov et al.
Revised: Such an operation is widely known as instance normalization (IN, Ulyanov et al.

Original: Let x ∈ R B × C × H × W be a batch of tensors, with B , C , H and W denoting the dimension of batch, channel, height and width, respectively, IN is formulated as
Revised: Let x ∈ R B × C × H × W be a batch of tensors, with B , C , H and W denoting the dimension of batch, channel, height and width, respectively, IN is formulated as

Original: [Equation]
Revised: [Equation]

Original:  where γ, β ∈ R C are learnable afﬁne transformation parameters, and µ ( x ) , σ ( x ) ∈ R B × C are mean and standard deviation computed across the spatial dimension within each channel of each instance (tensor), i.e.
Revised: 1) where γ, β ∈ R C are learnable afﬁne transformation parameters, and µ ( x ) , σ ( x ) ∈ R B × C are mean and standard deviation computed across the spatial dimension within each channel of each instance (tensor), i.e.

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Huang & Belongie (2017) introduced adaptive instance normalization (AdaIN), which simply replaces the scale and shift parameters in Eq.
Revised: Huang & Belongie (2017) introduced adaptive instance normalization (AdaIN), which simply replaces the scale and shift parameters in Eq.

Original: (1) with the feature statistics of style input y to achieve arbitrary style transfer:
Revised: (1) with the feature statistics of style input y to achieve arbitrary style transfer:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: 2. M IX S TYLE
Revised: M IX S TYLE

Original: Our method, MixStyle, draws inspiration from AdaIN.
Revised: Our method, MixStyle, draws inspiration from AdaIN.

Original: However, rather than attaching a decoder for image generation, MixStyle is designed for the purpose of regularizing CNN training by perturbing the style information of source domain training instances.
Revised: However, rather than attaching a decoder for image generation, MixStyle is designed for the purpose of regularizing CNN training by perturbing the style information of source domain training instances.

Original: It can be implemented as a plug-andplay module inserted between CNN layers of, e.g., a supervised CNN classiﬁer, without the need to explicitly generate an image of new style.
Revised: It can be implemented as a plug-andplay module inserted between CNN layers of, e.g., a supervised CNN classiﬁer, without the need to explicitly generate an image of new style.

Original: [Equation]
Revised: [Equation]

Original: x 1 x 2 xx 1 x 2 x 3 x 4 x 5 x 6 x = [ ] ] ˜ x

Original: = [ xx 2 xx 4 x 5 x

Original: [Figure]
Revised: [Figure]

Original: More speciﬁcally, MixStyle mixes the feature statistics of two instances with a random convex weight to simulate new styles.
Revised: More speciﬁcally, MixStyle mixes the feature statistics of two instances with a random convex weight to simulate new styles.

Original: In terms of implementation, MixStyle can be easily integrated into mini-batch training.
Revised: In terms of implementation, MixStyle can be easily integrated into mini-batch training.

Original: Given an input batch x , MixStyle ﬁrst generates a reference batch ˜ x from x .
Revised: Given an input batch x , MixStyle ﬁrst generates a reference batch ˜ x from x .

Original: When domain labels are given, x is sampled from two different domains i and j , e.g., x = [ x i , x j ] ( x i and x j have the same batch size).
Revised: When domain labels are given, x is sampled from two different domains i and j , e.g., x = [ x i , x j ] ( x i and x j have the same batch size).

Original: Then, ˜ x is obtained by swapping the position of x i and x j , followed by a shufﬂing operation along the batch dimension applied to each batch, i.e. ˜ x =
Revised: Then, ˜ x is obtained by swapping the position of x i and x j , followed by a shufﬂing operation along the batch dimension applied to each batch, i.e. ˜ x =

Original: [Shuﬄe( x j ) , Shuﬄe( x i )] .
Revised: [Shuﬄe( x j ) , Shuﬄe( x i )] .

Original: See Fig. 2(a) for an illustration.
Revised: See Fig. 2(a) for an illustration.

Original: In cases where domain labels are unknown, x is randomly sampled from the training data, and ˜ x is simply obtained by ˜ x
Revised: In cases where domain labels are unknown, x is randomly sampled from the training data, and ˜ x is simply obtained by ˜ x

Original: = Shuﬄe( x ) (see Fig. 2(b)).
Revised: = Shuﬄe( x ) (see Fig. 2(b)).

Original: Fig.
Revised: Fig.

Original: 4 shows that sub-domains exist within each domain, so even if two instances of the same domain are sampled, new domain could be synthesized.
Revised: 4 shows that sub-domains exist within each domain, so even if two instances of the same domain are sampled, new domain could be synthesized.

Original: After shufﬂing, MixStyle computes the mixed feature statistics by
Revised: After shufﬂing, MixStyle computes the mixed feature statistics by

Original: [Equation]
Revised: [Equation]

Original: 6) where λ ∈ R B are instance-wise weights sampled from the Beta distribution, λ ∼ Beta ( α, α ) with α ∈ (0 , ∞ ) being a hyper-parameter.
Revised: [Equation] (6) where λ ∈ R B are instance-wise weights sampled from the Beta distribution, λ ∼ Beta ( α, α ) with α ∈ (0 , ∞ ) being a hyper-parameter.

Original: Unless speciﬁed otherwise, we set α to 0.1 throughout this paper.
Revised: Unless speciﬁed otherwise, we set α to 0.1 throughout this paper.

Original: Finally, the mixed feature statistics are applied to the style-normalized x ,MixStyle
Revised: Finally, the mixed feature statistics are applied to the style-normalized x ,

Original: [Equation]
Revised: [Equation]

Original: In practice, we use a probability of 0.5 to decide if MixStyle is activated or not in the forward pass.
Revised: In practice, we use a probability of 0.5 to decide if MixStyle is activated or not in the forward pass.

Original: At test time, no MixStyle is applied.
Revised: At test time, no MixStyle is applied.

Original: Note that gradients are blocked in the computational graph of µ ( · ) and σ ( · ) .
Revised: Note that gradients are blocked in the computational graph of µ ( · ) and σ ( · ) .

Original: MixStyle can be implemented with only few lines of code.
Revised: MixStyle can be implemented with only few lines of code.

Original: See Algorithm 1 in
Revised: See Algorithm 1 in

Original: Appendix A.1 for the PyTorch-like pseudo-code.
Revised: Appendix A.1 for the PyTorch-like pseudo-code.

Original: [Table]
Revised: [Table]

Original: + MixStyle w/ domain label
Revised: + MixStyle w/ domain label

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS 3. G ENERALIZATION IN C ATEGORY C LASSIFICATION
Revised:  E XPERIMENTS 3. G ENERALIZATION IN C ATEGORY C LASSIFICATION

Original: Dataset and implementation details.
Revised: Dataset and implementation details.

Original: We choose the PACS dataset (Li et al., 2017), a commonly used domain generalization (DG) benchmark concerned with domain shift in image classiﬁcation.
Revised: We choose the PACS dataset (Li et al., 2017), a commonly used domain generalization (DG) benchmark concerned with domain shift in image classiﬁcation.

Original: PACS consists of four domains, i.e. Art Painting, Cartoon, Photo and Sketch, with totally 9,991 images of 7 classes.
Revised: PACS consists of four domains, i.e. Art Painting, Cartoon, Photo and Sketch, with totally 9,991 images of 7 classes.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, the domain shift mainly corresponds to image style changes.
Revised: 1, the domain shift mainly corresponds to image style changes.

Original: For evaluation, a model is trained on three domains and tested on the remaining one.
Revised: For evaluation, a model is trained on three domains and tested on the remaining one.

Original: Following prior work (Li et al., 2019; Zhou et al., 2020a), we use ResNet-18 (He et al., 2016) as the classiﬁer where MixStyle is inserted after the 1st, 2nd and 3rd residual blocks.
Revised: Following prior work (Li et al., 2019; Zhou et al., 2020a), we use ResNet-18 (He et al., 2016) as the classiﬁer where MixStyle is inserted after the 1st, 2nd and 3rd residual blocks.

Original: Our code is based on
Revised: Our code is based on

Original: Dassl.pytorch
Revised: Dassl.pytorch

Original: (Zhou et al., 2020c).
Revised: (Zhou et al., 2020c).

Original: Baselines.
Revised: Baselines.

Original: Our main baselines are general-purpose regularization methods including Mixup (Zhang et al., 2018b), Manifold Mixup (Verma et al., 2019), DropBlock (Ghiasi et al., 2018), CutMix (Yun et al., 2019) and Cutout (DeVries & Taylor, 2017), which are trained using the same training parameters as MixStyle and the optimal hyper-parameter setup as reported in their papers.
Revised: Our main baselines are general-purpose regularization methods including Mixup (Zhang et al., 2018b), Manifold Mixup (Verma et al., 2019), DropBlock (Ghiasi et al., 2018), CutMix (Yun et al., 2019) and Cutout (DeVries & Taylor, 2017), which are trained using the same training parameters as MixStyle and the optimal hyper-parameter setup as reported in their papers.

Original: We also compare with the existing DG methods which reported state-of-the-art performance on PACS.
Revised: We also compare with the existing DG methods which reported state-of-the-art performance on PACS.

Original: These include domain alignment-based CCSA (Motiian et al., 2017) and MMD-AAE (Li et al., 2018b), Jigsaw puzzle-based JiGen (Carlucci et al., 2019), adversarial gradient-based CrossGrad (Shankar et al., 2018), meta-learning-based Metareg (Balaji et al., 2018) and Epi-FCR (Li et al., 2019), and
Revised: These include domain alignment-based CCSA (Motiian et al., 2017) and MMD-AAE (Li et al., 2018b), Jigsaw puzzle-based JiGen (Carlucci et al., 2019), adversarial gradient-based CrossGrad (Shankar et al., 2018), meta-learning-based Metareg (Balaji et al., 2018) and Epi-FCR (Li et al., 2019), and

Original: data augmentation-based L2A-OT
Revised: data augmentation-based L2A-OT

Original: (Zhou et al., 2020a).
Revised: (Zhou et al., 2020a).

Original: Comparison with general-purpose regularization methods.
Revised: Comparison with general-purpose regularization methods.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: Overall, we observe that the general-purpose regularization methods do not offer any clear advantage over the vanilla ResNet-18 in this DG task, while MixStyle improves upon the vanilla ResNet18 with a signiﬁcant margin.
Revised: Overall, we observe that the general-purpose regularization methods do not offer any clear advantage over the vanilla ResNet-18 in this DG task, while MixStyle improves upon the vanilla ResNet18 with a signiﬁcant margin.

Original: Compared with Mixup, MixStyle is 5.2% better on average.
Revised: Compared with Mixup, MixStyle is 5.2% better on average.

Original: Recall that Mixup also interpolates the output space, we further compare with a variant of Mixup in order to demonstrate the advantage of mixing style statistics at the feature level over mixing images at the pixel level for DG—following Sohn et al.
Revised: Recall that Mixup also interpolates the output space, we further compare with a variant of Mixup in order to demonstrate the advantage of mixing style statistics at the feature level over mixing images at the pixel level for DG—following Sohn et al.

Original: (2020), we remove the label interpolation in Mixup and sample the mixing weights from a uniform distribution of [0 , 1] .
Revised: (2020), we remove the label interpolation in Mixup and sample the mixing weights from a uniform distribution of [0 , 1] .

Original: Still, MixStyle outperforms this new baseline with a large margin, which justiﬁes our claim.
Revised: Still, MixStyle outperforms this new baseline with a large margin, which justiﬁes our claim.

Original: MixStyle and DropBlock share some commonalities in that they are both applied to feature maps at multiple layers, but MixStyle signiﬁcantly outperforms DropBlock in all test domains.
Revised: MixStyle and DropBlock share some commonalities in that they are both applied to feature maps at multiple layers, but MixStyle signiﬁcantly outperforms DropBlock in all test domains.

Original: The reason why DropBlock is ineffective here is because dropping out activations mainly encourages a network to mine discriminative patterns, but does not reinforce the ability to cope with unseen styles, which is exactly what MixStyle aims to achieve: by synthesizing “new” styles (domains) MixStyle regularizes the network to become more robust to domain shift.
Revised: The reason why DropBlock is ineffective here is because dropping out activations mainly encourages a network to mine discriminative patterns, but does not reinforce the ability to cope with unseen styles, which is exactly what MixStyle aims to achieve: by synthesizing “new” styles (domains) MixStyle regularizes the network to become more robust to domain shift.

Original: In addition, it is interesting to see that on Cartoon and Photo, MixStyle w/ random shufﬂe obtains slightly better results.
Revised: In addition, it is interesting to see that on Cartoon and Photo, MixStyle w/ random shufﬂe obtains slightly better results.

Original: The reason might be because there exist sub-domains in a source domain (see Fig.
Revised: The reason might be because there exist sub-domains in a source domain (see Fig.

Original: 4(a-c)), which allow random shufﬂing to produce more diverse “new” domains that lead to a more domain-generalizable model.
Revised: 4(a-c)), which allow random shufﬂing to produce more diverse “new” domains that lead to a more domain-generalizable model.

Original: Comparison with state-of-the-art DG methods.
Revised: Comparison with state-of-the-art DG methods.

Original: Overall, MixStyle outperforms most DG methods by a clear margin, despite being a much simpler method.
Revised: Overall, MixStyle outperforms most DG methods by a clear margin, despite being a much simpler method.

Original: The performance of MixStyle w/ domain label is nearly 1% better on average than the recently introduced L2A-OT.
Revised: The performance of MixStyle w/ domain label is nearly 1% better on average than the recently introduced L2A-OT.

Original: From a data augmentation perspective, MixStyle and L2A-OT share a similar goal—to synthesize data from pseudo-novel domains.
Revised: From a data augmentation perspective, MixStyle and L2A-OT share a similar goal—to synthesize data from pseudo-novel domains.

Original: MixStyle accomplishes this goal through mixing style statistics at the feature level.
Revised: MixStyle accomplishes this goal through mixing style statistics at the feature level.

Original: Whereas L2A-OT works at the pixel level: it trains an image generator by maximizing the domain difference (measured by optimal transport) between the original and the generated images, which introduces much heavier computational overhead than MixStyle in terms of GPU memory and training time.
Revised: Whereas L2A-OT works at the pixel level: it trains an image generator by maximizing the domain difference (measured by optimal transport) between the original and the generated images, which introduces much heavier computational overhead than MixStyle in terms of GPU memory and training time.

Original: It is worth noting that MixStyle’s domain label-free version is highly competitive: its 82.8% accuracy is on par with L2A-OT’s.
Revised: It is worth noting that MixStyle’s domain label-free version is highly competitive: its 82.8% accuracy is on par with L2A-OT’s.

Original: 3. G ENERALIZATION IN I NSTANCE R ETRIEVAL
Revised: 3. G ENERALIZATION IN I NSTANCE R ETRIEVAL

Original: Dataset and implementation details.
Revised: Dataset and implementation details.

Original: We evaluate MixStyle on the person re-identiﬁcation (re-ID) problem, which aims to match people across disjoint camera views.
Revised: We evaluate MixStyle on the person re-identiﬁcation (re-ID) problem, which aims to match people across disjoint camera views.

Original: As each camera view is itself a distinct domain, person re-ID is essentially a cross-domain image matching problem.
Revised: As each camera view is itself a distinct domain, person re-ID is essentially a cross-domain image matching problem.

Original: Instead of using the standard protocol where training and test data come from the same camera views, we adopt the cross-dataset setting so test camera views are never seen during training.
Revised: Instead of using the standard protocol where training and test data come from the same camera views, we adopt the cross-dataset setting so test camera views are never seen during training.

Original: Speciﬁcally, we train a model on one dataset and then test its performance on the other dataset.
Revised: Speciﬁcally, we train a model on one dataset and then test its performance on the other dataset.

Original: Two commonly used re-ID datasets are adopted: Market1501 (Zheng et al., 2015) and Duke (Ristani et al., 2016; Zheng et al., 2017).
Revised: Two commonly used re-ID datasets are adopted: Market1501 (Zheng et al., 2015) and Duke (Ristani et al., 2016; Zheng et al., 2017).

Original: Ranking accuracy and mean average precision (mAP) are used as the performance measures (displayed in percentage).
Revised: Ranking accuracy and mean average precision (mAP) are used as the performance measures (displayed in percentage).

Original: We test MixStyle on two CNN architectures: ResNet-50 (He et al., 2016) and OSNet (Zhou et al., 2019).
Revised: We test MixStyle on two CNN architectures: ResNet-50 (He et al., 2016) and OSNet (Zhou et al., 2019).

Original: The latter was designed speciﬁcally for re-ID.
Revised: The latter was designed speciﬁcally for re-ID.

Original: In both architectures,
Revised: In both architectures,

Original: MixStyle is inserted after the 1st and 2nd residual blocks.
Revised: MixStyle is inserted after the 1st and 2nd residual blocks.

Original: Our code is based on Torchreid (Zhou &
Revised: Our code is based on Torchreid (Zhou &

Original: Xiang, 2019).
Revised: Xiang, 2019).

Original: Baselines.
Revised: Baselines.

Original: We compare with three baseline methods: 1) The vanilla model, which serves as a strong baseline; 2) DropBlock, which was the top-performing competitor in Table 1; 3)
Revised: We compare with three baseline methods: 1) The vanilla model, which serves as a strong baseline; 2) DropBlock, which was the top-performing competitor in Table 1; 3)

Original: RandomErase (Zhong et al., 2020), a widely used regularization method in the re-ID literature (similar to Cutout).
Revised: RandomErase (Zhong et al., 2020), a widely used regularization method in the re-ID literature (similar to Cutout).

Original: Results.
Revised: Results.

Original: The results are reported in Table 2.
Revised: The results are reported in Table 2.

Original: It is clear that only MixStyle consistently outperforms the strong vanilla model under both settings with considerable margins, while DropBlock and RandomErase are unable to show any beneﬁt.
Revised: It is clear that only MixStyle consistently outperforms the strong vanilla model under both settings with considerable margins, while DropBlock and RandomErase are unable to show any beneﬁt.

Original: Notably, RandomErase, which simulates occlusion by erasing pixels in random rectangular regions with random values, has been used as a default trick when training re-ID CNNs.
Revised: Notably, RandomErase, which simulates occlusion by erasing pixels in random rectangular regions with random values, has been used as a default trick when training re-ID CNNs.

Original: However, RandomErase shows a detrimental effect in the cross-dataset re-ID setting.
Revised: However, RandomErase shows a detrimental effect in the cross-dataset re-ID setting.

Original: Indeed, similar to DropBlock, randomly erasing pixels offers no guarantee to improve the robustness when it comes to domain shift.
Revised: Indeed, similar to DropBlock, randomly erasing pixels offers no guarantee to improve the robustness when it comes to domain shift.

Original: 3. G ENERALIZATION IN R EINFORCEMENT L EARNING
Revised: 3. G ENERALIZATION IN R EINFORCEMENT L EARNING

Original: Though RL has been greatly advanced by using CNNs for feature learning in raw pixels (Mnih et al., 2013), it has been widely acknowledged that RL agents often overﬁt training environments while generalize poorly to unseen environments (Cobbe et al., 2019; Igl et al., 2019).
Revised: Though RL has been greatly advanced by using CNNs for feature learning in raw pixels (Mnih et al., 2013), it has been widely acknowledged that RL agents often overﬁt training environments while generalize poorly to unseen environments (Cobbe et al., 2019; Igl et al., 2019).

Original: Dataset and implementation details.
Revised: Dataset and implementation details.

Original: We conduct experiments on Coinrun (Cobbe et al., 2019), a recently introduced RL benchmark for evaluating the generalization performance of RL agents.
Revised: We conduct experiments on Coinrun (Cobbe et al., 2019), a recently introduced RL benchmark for evaluating the generalization performance of RL agents.

Original: As shown in Fig. 3(a), the goal in Coinrun is to control a character to collect golden coins while avoiding both stationary and non-stationary obstacles.
Revised: As shown in Fig. 3(a), the goal in Coinrun is to control a character to collect golden coins while avoiding both stationary and non-stationary obstacles.

Original: We follow Igl et al.
Revised: We follow Igl et al.

Original: (2019) to construct and train our RL agent: the CNN architecture used in IMPALA (Espeholt et al., 2018) is adopted as the policy network, and is trained by the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017).
Revised: (2019) to construct and train our RL agent: the CNN architecture used in IMPALA (Espeholt et al., 2018) is adopted as the policy network, and is trained by the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017).

Original: Please refer to Igl et al.
Revised: Please refer to Igl et al.

Original: for further implementation details.
Revised: for further implementation details.

Original: MixStyle is inserted after the 1st and 2nd convolutional sequences.
Revised: MixStyle is inserted after the 1st and 2nd convolutional sequences.

Original: Training data are sampled from 500 levels while test
Revised: Training data are sampled from 500 levels while test

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: data are drawn from new levels of only the highest difﬁculty.
Revised: data are drawn from new levels of only the highest difﬁculty.

Original: As domain labels are difﬁcult to deﬁne, we use the random shufﬂe version of MixStyle.
Revised: As domain labels are difﬁcult to deﬁne, we use the random shufﬂe version of MixStyle.

Original: Our code is built on top of Igl et al.
Revised: Our code is built on top of Igl et al.

Original: Baselines.
Revised: Baselines.

Original: Following Igl et al.
Revised: Following Igl et al.

Original: (2019), we train strong baseline models and add MixStyle on top of them to see whether MixStyle can bring further improvements.
Revised: (2019), we train strong baseline models and add MixStyle on top of them to see whether MixStyle can bring further improvements.

Original: To this end, we train two baseline models: 1) Baseline, which combines weight decay and data augmentation; 5 2) IBAC-SNI (the λ = 0 .
Revised: To this end, we train two baseline models: 1) Baseline, which combines weight decay and data augmentation; 5 2) IBAC-SNI (the λ = 0 .

Original: 5 version), the best-performing model in Igl et al.
Revised: 5 version), the best-performing model in Igl et al.

Original: (2019) which is based on selective noise injection.
Revised: (2019) which is based on selective noise injection.

Original: Results.
Revised: Results.

Original: The test performance is shown in Fig. 3(b).
Revised: The test performance is shown in Fig. 3(b).

Original: Comparing Baseline (blue) with Baseline+MixStyle (orange), we can see that MixStyle brings a signiﬁcant improvement.
Revised: Comparing Baseline (blue) with Baseline+MixStyle (orange), we can see that MixStyle brings a signiﬁcant improvement.

Original: Interestingly, the variance is also signiﬁcantly reduced by using MixStyle, as indicated by the smaller shaded areas (for both orange and red lines).
Revised: Interestingly, the variance is also signiﬁcantly reduced by using MixStyle, as indicated by the smaller shaded areas (for both orange and red lines).

Original: These results strongly demonstrate the effectiveness of MixStyle in enhancing generalization for RL agents.
Revised: These results strongly demonstrate the effectiveness of MixStyle in enhancing generalization for RL agents.

Original: When it comes to the stronger baseline IBAC-SNI (green),
Revised: When it comes to the stronger baseline IBAC-SNI (green),

Original: MixStyle (red) is able to bring further performance gain, suggesting that MixStyle is complementary to IBAC-SNI.
Revised: MixStyle (red) is able to bring further performance gain, suggesting that MixStyle is complementary to IBAC-SNI.

Original: This result also shows the potential of MixStyle as a plug-and-play component to be combined with other advanced RL methods.
Revised: This result also shows the potential of MixStyle as a plug-and-play component to be combined with other advanced RL methods.

Original: It is worth noting that Baseline+MixStyle itself is already highly competitive with IBAC-SNI.
Revised: It is worth noting that Baseline+MixStyle itself is already highly competitive with IBAC-SNI.

Original: Fig. 3(c) shows the generalization gap from which it can been seen that the models trained with MixStyle (orange & red) clearly generalize faster and better than those without using MixStyle (blue & green).
Revised: Fig. 3(c) shows the generalization gap from which it can been seen that the models trained with MixStyle (orange & red) clearly generalize faster and better than those without using MixStyle (blue & green).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: A NALYSIS
Revised: A NALYSIS

Original: Where to apply MixStyle?
Revised: Where to apply MixStyle?

Original: We repeat the experiments on PACS (category classiﬁcation) and the re-ID datasets (instance retrieval) using the ResNet architecture.
Revised: We repeat the experiments on PACS (category classiﬁcation) and the re-ID datasets (instance retrieval) using the ResNet architecture.

Original: Given that a standard ResNet model has four residual blocks denoted by res1-4 , we train different models with MixStyle applied to different layers.
Revised: Given that a standard ResNet model has four residual blocks denoted by res1-4 , we train different models with MixStyle applied to different layers.

Original: For notation, res1 means MixStyle is applied after the ﬁrst residual block; res means MixStyle is applied after both the ﬁrst and second residual blocks; and so forth.
Revised: For notation, res1 means MixStyle is applied after the ﬁrst residual block; res means MixStyle is applied after both the ﬁrst and second residual blocks; and so forth.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: We have the following observations.
Revised: We have the following observations.

Original: Applying MixStyle to multiple lowerlevel layers generally achieves a better performance—for instance, res12 is better than res1 on both tasks.
Revised: Applying MixStyle to multiple lowerlevel layers generally achieves a better performance—for instance, res12 is better than res1 on both tasks.

Original: Different tasks favor different combinations— res123 achieves the best performance on PACS, while on the re-ID datasets res12 is the best. 3)
Revised: Different tasks favor different combinations— res123 achieves the best performance on PACS, while on the re-ID datasets res12 is the best. 3)

Original: On both tasks, the performance plunges when applying MixStyle to the last residual block.
Revised: On both tasks, the performance plunges when applying MixStyle to the last residual block.

Original: This makes sense because res4 is the closest to the prediction layer and tends to capture semantic content (i.e. label-sensitive) information rather than style.
Revised: This makes sense because res4 is the closest to the prediction layer and tends to capture semantic content (i.e. label-sensitive) information rather than style.

Original: In particular, res4 is followed by an average-pooling layer, which essentially forwards the mean vector to the prediction layer and thus forces the mean vector to capture label-related information.
Revised: In particular, res4 is followed by an average-pooling layer, which essentially forwards the mean vector to the prediction layer and thus forces the mean vector to capture label-related information.

Original: As a consequence, mixing the statistics at res4 breaks the inherent label space.
Revised: As a consequence, mixing the statistics at res4 breaks the inherent label space.

Original: This is clearer in Fig.
Revised: This is clearer in Fig.

Original: 4: the features and style statistics in res1-3 exhibit clustering patterns based on domains while those in res4 have a high correlation with class labels.
Revised: 4: the features and style statistics in res1-3 exhibit clustering patterns based on domains while those in res4 have a high correlation with class labels.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Mixing vs. replacing.
Revised: Mixing vs. replacing.

Original: Unlike the AdaIN formulation, which completely replaces one style with the other, MixStyle mixes two styles via a convex combination.
Revised: Unlike the AdaIN formulation, which completely replaces one style with the other, MixStyle mixes two styles via a convex combination.

Original: Table 4 shows that mixing is better than replacing.
Revised: Table 4 shows that mixing is better than replacing.

Original: This is easy to understand: mixing diversiﬁes the styles (imagine an interpolation between two data points).
Revised: This is easy to understand: mixing diversiﬁes the styles (imagine an interpolation between two data points).

Original: [Table]
Revised: [Table]

Original: Random vs. ﬁxed shufﬂe at multiple layers.
Revised: Random vs. ﬁxed shufﬂe at multiple layers.

Original: Applying MixStyle to multiple layers, which has been shown advantageous in Table 3, raises another question of whether to shufﬂe the mini-batch at different layers or use the same shufﬂed order for all layers.
Revised: Applying MixStyle to multiple layers, which has been shown advantageous in Table 3, raises another question of whether to shufﬂe the mini-batch at different layers or use the same shufﬂed order for all layers.

Original: Tablesuggests that using random shufﬂe at different layers gives a better performance, which may be attributed to the increased noise level that gives a better regularization effect.
Revised: Tablesuggests that using random shufﬂe at different layers gives a better performance, which may be attributed to the increased noise level that gives a better regularization effect.

Original: Sensitivity of hyper-parameter.
Revised: Sensitivity of hyper-parameter.

Original: Recall that α is used to control the shape of Beta distribution, which has a direct effect on how the convex weights λ are sampled.
Revised: Recall that α is used to control the shape of Beta distribution, which has a direct effect on how the convex weights λ are sampled.

Original: The smaller α is, the more likely the value in λ is close to the extreme value of 0 or 1.
Revised: The smaller α is, the more likely the value in λ is close to the extreme value of 0 or 1.

Original: In other words, a smaller α favors the style statistics in Eqs.
Revised: In other words, a smaller α favors the style statistics in Eqs.

Original: (5) & (6) to be dominated by one side.
Revised: (5) & (6) to be dominated by one side.

Original: We ﬁrst evaluate α on PACS.
Revised: We ﬁrst evaluate α on PACS.

Original: Fig. 5(a) shows that with α increasing from 0.1 to 0.4, the accuracy slides from 82.8% to 81.7%.
Revised: Fig. 5(a) shows that with α increasing from 0.1 to 0.4, the accuracy slides from 82.8% to 81.7%.

Original: However, further increasing α does not impact on the accuracy.
Revised: However, further increasing α does not impact on the accuracy.

Original: Therefore, the results suggest that the performance is not too sensitive to α ; and selecting α from { 0 . 1 , 0 . 2 , 0 .
Revised: Therefore, the results suggest that the performance is not too sensitive to α ; and selecting α from { 0 . 1 , 0 . 2 , 0 .

Original: 3 } seems to be a good starting point.
Revised: 3 } seems to be a good starting point.

Original: We further experiment with α ∈ { 0 . 1 , 0 . 2 , 0 . 3 } on the re-ID datasets and the Coinrun benchmark.
Revised: We further experiment with α ∈ { 0 . 1 , 0 . 2 , 0 . 3 } on the re-ID datasets and the Coinrun benchmark.

Original: Figs. 5(b) & (c) show that in general the variance for the results of different values is small.
Revised: Figs. 5(b) & (c) show that in general the variance for the results of different values is small.

Original: Therefore, we suggest practitioners to choose α from { 0 . 1 , 0 . 2 , 0 . 3 } , with α = 0 .
Revised: Therefore, we suggest practitioners to choose α from { 0 . 1 , 0 . 2 , 0 . 3 } , with α = 0 .

Original: 1 being a good default setting.
Revised: 1 being a good default setting.

Original: For more analyses and discussions, please see Appendix A.2.
Revised: For more analyses and discussions, please see Appendix A.2.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Domain generalization, or DG, studies out-of-distribution (OOD) generalization given only source data typically composed of multiple related but distinct domains.
Revised: Domain generalization, or DG, studies out-of-distribution (OOD) generalization given only source data typically composed of multiple related but distinct domains.

Original: We refer readers to Zhou et al.
Revised: We refer readers to Zhou et al.

Original: (2021) for a comprehensive survey in this topic.
Revised: (2021) for a comprehensive survey in this topic.

Original: Many DG methods are based on the idea of aligning features between different sources, with a hope that the model can be invariant to domain shift given unseen data.
Revised: Many DG methods are based on the idea of aligning features between different sources, with a hope that the model can be invariant to domain shift given unseen data.

Original: For instance, Li et al.
Revised: For instance, Li et al.

Original: (2018b) achieved distribution alignment in the hidden representation of an autoencoder using maximum mean discrepancy; Li et al.
Revised: (2018b) achieved distribution alignment in the hidden representation of an autoencoder using maximum mean discrepancy; Li et al.

Original: (2018c) resorted to adversarial learning with auxiliary domain classiﬁers to learn features that are domain-agnostic.
Revised: (2018c) resorted to adversarial learning with auxiliary domain classiﬁers to learn features that are domain-agnostic.

Original: Some works explored domain-speciﬁc parameterization, such as domain-speciﬁc weight matrices (Li et al., 2017) and domain-speciﬁc BN (Seo et al., 2020).
Revised: Some works explored domain-speciﬁc parameterization, such as domain-speciﬁc weight matrices (Li et al., 2017) and domain-speciﬁc BN (Seo et al., 2020).

Original: Recently, meta-learning has drawn increasing attention from the DG community (Li et al., 2018a;
Revised: Recently, meta-learning has drawn increasing attention from the DG community (Li et al., 2018a;

Original: Balaji et al., 2018; Dou et al., 2019).
Revised: Balaji et al., 2018; Dou et al., 2019).

Original: The main idea is to expose a model to domain shift during training by using pseudo-train and pseudo-test domains, both drawn from source domains.
Revised: The main idea is to expose a model to domain shift during training by using pseudo-train and pseudo-test domains, both drawn from source domains.

Original: Data augmentation has also been investigated for learning domain-invariant models.
Revised: Data augmentation has also been investigated for learning domain-invariant models.

Original: Shankar et al.
Revised: Shankar et al.

Original: (2018) introduced a cross-gradient training method (CrossGrad) where source data are augmented by adversarial gradients obtained from a domain classiﬁer.
Revised: (2018) introduced a cross-gradient training method (CrossGrad) where source data are augmented by adversarial gradients obtained from a domain classiﬁer.

Original: Gong et al.
Revised: Gong et al.

Original: (2019) proposed DLOW (for the DA problem), which models intermediate domains between source and target via a domainness factor and learns an image translation model to generate intermediate-domain images.
Revised: (2019) proposed DLOW (for the DA problem), which models intermediate domains between source and target via a domainness factor and learns an image translation model to generate intermediate-domain images.

Original: Very recently, Zhou et al.
Revised: Very recently, Zhou et al.

Original: (2020a)
Revised: (2020a)

Original: introduced L2A-OT to learn a neural network to map source data to pseudo-novel domains by maximizing an optimal transport-based distance measure.
Revised: introduced L2A-OT to learn a neural network to map source data to pseudo-novel domains by maximizing an optimal transport-based distance measure.

Original: Our MixStyle is related to DLOW and L2A-OT in its efforts to synthesizing novel domains.
Revised: Our MixStyle is related to DLOW and L2A-OT in its efforts to synthesizing novel domains.

Original: However, MixStyle differs in the fact that it is done implicitly with a much simpler formulation leveraging the feature-level style statistics and only few lines of extra code on top of a standard supervised classiﬁer while being more effective.
Revised: However, MixStyle differs in the fact that it is done implicitly with a much simpler formulation leveraging the feature-level style statistics and only few lines of extra code on top of a standard supervised classiﬁer while being more effective.

Original: Essentially, MixStyle can be seen as feature -level augmentation, which is clearly different from the image -level augmentation-based DLOW and L2A-OT.
Revised: Essentially, MixStyle can be seen as feature -level augmentation, which is clearly different from the image -level augmentation-based DLOW and L2A-OT.

Original: Generalization in deep RL has been a challenging problem where RL agents often overﬁt training environments, and as a result, perform poorly in unseen environments with different visual patterns or levels (Zhang et al., 2018a).
Revised: Generalization in deep RL has been a challenging problem where RL agents often overﬁt training environments, and as a result, perform poorly in unseen environments with different visual patterns or levels (Zhang et al., 2018a).

Original: A natural way to improve generalization, which has been shown effective in (Cobbe et al., 2019; Farebrother et al., 2018), is to use regularization, e.g., weight decay.
Revised: A natural way to improve generalization, which has been shown effective in (Cobbe et al., 2019; Farebrother et al., 2018), is to use regularization, e.g., weight decay.

Original: However, Igl et al.
Revised: However, Igl et al.

Original: (2019) suggested that stochastic regularization methods like dropout and batch normalization (which uses estimated population statistics) have adverse effect as the training data in
Revised: (2019) suggested that stochastic regularization methods like dropout and batch normalization (which uses estimated population statistics) have adverse effect as the training data in

Original: RL are essentially model-dependent.
Revised: RL are essentially model-dependent.

Original: As such, they proposed selective noise injection (SNI), which basically combines a stochastic regularization technique with its deterministic counterpart.
Revised: As such, they proposed selective noise injection (SNI), which basically combines a stochastic regularization technique with its deterministic counterpart.

Original: They further integrated SNI with information bottleneck actor critic (IBAC-SNI) to reduce the variance in gradients.
Revised: They further integrated SNI with information bottleneck actor critic (IBAC-SNI) to reduce the variance in gradients.

Original: Curriculum learning has been investigated in (Justesen et al., 2018) where the level of training episodes progresses from easy to difﬁcult over the course of training.
Revised: Curriculum learning has been investigated in (Justesen et al., 2018) where the level of training episodes progresses from easy to difﬁcult over the course of training.

Original: Gamrian & Goldberg (2019) leveraged the advances in GAN-based image-to-image translation (Liu et al., 2017) to map target data to the source domain which the agent was trained on.
Revised: Gamrian & Goldberg (2019) leveraged the advances in GAN-based image-to-image translation (Liu et al., 2017) to map target data to the source domain which the agent was trained on.

Original: Tobin et al.
Revised: Tobin et al.

Original: (2017) introduced domain randomization, which diversiﬁes training data by rendering images with different visual effects via a programmable simulator.
Revised: (2017) introduced domain randomization, which diversiﬁes training data by rendering images with different visual effects via a programmable simulator.

Original: With a similar goal of data augmentation, Lee et al.
Revised: With a similar goal of data augmentation, Lee et al.

Original: (2020) pre-processed input images with a randomly initialized network.
Revised: (2020) pre-processed input images with a randomly initialized network.

Original: Very recent studies (Laskin et al., 2020;
Revised: Very recent studies (Laskin et al., 2020;

Original: Kostrikov et al., 2020) have shown that it is useful to combine a diverse set of label-preserving transformations, such as rotation, shifting and Cutout.
Revised: Kostrikov et al., 2020) have shown that it is useful to combine a diverse set of label-preserving transformations, such as rotation, shifting and Cutout.

Original: Different from the aforementioned methods, our MixStyle works at the feature level and is orthogonal to most existing methods.
Revised: Different from the aforementioned methods, our MixStyle works at the feature level and is orthogonal to most existing methods.

Original: For instance, we have shown in Sec.
Revised: For instance, we have shown in Sec.

Original: 3.3 that MixStyle signiﬁcantly improves upon IBAC-SNI.
Revised: 3.3 that MixStyle signiﬁcantly improves upon IBAC-SNI.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a simple yet effective domain generalization method, termed MixStyle.
Revised: We presented a simple yet effective domain generalization method, termed MixStyle.

Original: MixStyle mixes the feature statistics of two instances to synthesize novel domains, which is inspired by the observation in style transfer work that the feature statistics encode style/domain-related information.
Revised: MixStyle mixes the feature statistics of two instances to synthesize novel domains, which is inspired by the observation in style transfer work that the feature statistics encode style/domain-related information.

Original: Extensive experiments covering a wide range of tasks were conducted to demonstrate that MixStyle yields new state-of-the-art on three different tasks.
Revised: Extensive experiments covering a wide range of tasks were conducted to demonstrate that MixStyle yields new state-of-the-art on three different tasks.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was supported in part by the Shanghai Committee of Science and Technology,
Revised: This work was supported in part by the Shanghai Committee of Science and Technology,

Original: China (Grant No. 20DZ1100800), in part by the National Natural Science Foundation of China under
Revised: China (Grant No. 20DZ1100800), in part by the National Natural Science Foundation of China under

Original: Grant (61876176, U1713208), the National Key Research and Development Program of China (No. 2020YFC2004800), and in part by the Guangzhou Research Program (201803010066).
Revised: Grant (61876176, U1713208), the National Key Research and Development Program of China (No. 2020YFC2004800), and in part by the Guangzhou Research Program (201803010066).

Original: A A PPENDIX A. P SEUDO -C ODE OF M IX S TYLE
Revised: A A PPENDIX A. P SEUDO -C ODE OF M IX S TYLE

Original: Algorithm 1 provides a PyTorch-like pseudo-code.
Revised: Algorithm 1 provides a PyTorch-like pseudo-code.

Original: Algorithm
Revised: Algorithm

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A. F URTHER A NALYSIS
Revised: A. F URTHER A NALYSIS

Original: [Table]
Revised: [Table]

Original: MixStyle between same-domain instances.
Revised: MixStyle between same-domain instances.

Original: We are interested in knowing if mixing styles between same-domain instances helps performance.
Revised: We are interested in knowing if mixing styles between same-domain instances helps performance.

Original: To this end, we sample each mini-batch from a single domain during training when using MixStyle.
Revised: To this end, we sample each mini-batch from a single domain during training when using MixStyle.

Original: The results are shown in Tablewhere we observe that mixing styles between same-domain instances is about 1% better than the baseline model.
Revised: The results are shown in Tablewhere we observe that mixing styles between same-domain instances is about 1% better than the baseline model.

Original: This suggests that instancespeciﬁc style exists.
Revised: This suggests that instancespeciﬁc style exists.

Original: Nonetheless, the performance is clearly worse than mixing styles between instances of different domains.
Revised: Nonetheless, the performance is clearly worse than mixing styles between instances of different domains.

Original: Performance on source domains.
Revised: Performance on source domains.

Original: To prove that MixStyle does not sacriﬁce the performance on seen domains in exchange for gains on unseen domains, we report the test accuracy on the held-out validation set of the source domains on PACS in Table 7.
Revised: To prove that MixStyle does not sacriﬁce the performance on seen domains in exchange for gains on unseen domains, we report the test accuracy on the held-out validation set of the source domains on PACS in Table 7.

Original: Results on Digits-DG and Ofﬁce-Home.
Revised: Results on Digits-DG and Ofﬁce-Home.

Original: In addition to the experiments on PACS (in Sec. 3.1), we further evaluate MixStyle’s effectiveness on two DG datasets, namely Digits-DG (Zhou et al., 2020a) and Ofﬁce-Home (Venkateswara et al., 2017).
Revised: In addition to the experiments on PACS (in Sec. 3.1), we further evaluate MixStyle’s effectiveness on two DG datasets, namely Digits-DG (Zhou et al., 2020a) and Ofﬁce-Home (Venkateswara et al., 2017).

Original: Digits-DG contains four digit datasets (domains) including MNIST (LeCun et al., 1998), MNIST-M (Ganin & Lempitsky, 2015), SVHN (Netzer et al., 2011) and SYN (Ganin & Lempitsky, 2015).
Revised: Digits-DG contains four digit datasets (domains) including MNIST (LeCun et al., 1998), MNIST-M (Ganin & Lempitsky, 2015), SVHN (Netzer et al., 2011) and SYN (Ganin & Lempitsky, 2015).

Original: Images from different digit datasets differ drastically in font style, stroke color and background.
Revised: Images from different digit datasets differ drastically in font style, stroke color and background.

Original: Ofﬁce-Home is composed of four domains (Artistic,
Revised: Ofﬁce-Home is composed of four domains (Artistic,

Original: Clipart, Product and Real World) with around 15,500 images of 65 classes for home and ofﬁce object recognition.
Revised: Clipart, Product and Real World) with around 15,500 images of 65 classes for home and ofﬁce object recognition.

Original: The results are shown in Tables 8 and 9.
Revised: The results are shown in Tables 8 and 9.

Original: Similar to the results on PACS, here we observe that MixStyle also brings clear improvements to the baseline CNN model and outperforms all general-purpose regularization methods on both Digits-DG and Ofﬁce-Home.
Revised: Similar to the results on PACS, here we observe that MixStyle also brings clear improvements to the baseline CNN model and outperforms all general-purpose regularization methods on both Digits-DG and Ofﬁce-Home.

Original: Compared with more sophisticated DG methods like L2A-OT, MixStyle’s performance is comparable, despite being much simpler to train and consuming much less computing resources.
Revised: Compared with more sophisticated DG methods like L2A-OT, MixStyle’s performance is comparable, despite being much simpler to train and consuming much less computing resources.

Original: [Table]
Revised: [Table]