Original: I NVARIANT F EATURE L EARNING BY A TTRIBUTE P ERCEPTION M ATCHING
Revised: I NVARIANT F EATURE L EARNING BY A TTRIBUTE P ERCEPTION M ATCHING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: An adversarial feature learning (AFL) is a powerful framework to learn representations invariant to a nuisance attribute, which uses an adversarial game between a feature extractor and a categorical attribute classiﬁer.
Revised: An adversarial feature learning (AFL) is a powerful framework to learn representations invariant to a nuisance attribute, which uses an adversarial game between a feature extractor and a categorical attribute classiﬁer.

Original: It theoretically sounds in term of it maximize conditional entropy between attribute and representation.
Revised: It theoretically sounds in term of it maximize conditional entropy between attribute and representation.

Original: However, as shown in this paper, the AFL often causes unstable behavior that slows down the convergence.
Revised: However, as shown in this paper, the AFL often causes unstable behavior that slows down the convergence.

Original: We propose an attribute perception matching as an alternative approach, based on the reformulation of conditional entropy maximization as pair-wise distribution matching .
Revised: We propose an attribute perception matching as an alternative approach, based on the reformulation of conditional entropy maximization as pair-wise distribution matching .

Original: Although the naive approach for realizing the pair-wise distribution matching requires the signiﬁcantly large number of parameters, the proposed method requires the same number of parameters with AFL but has a better convergence property.
Revised: Although the naive approach for realizing the pair-wise distribution matching requires the signiﬁcantly large number of parameters, the proposed method requires the same number of parameters with AFL but has a better convergence property.

Original: Experiments on both toy and real-world dataset prove that our proposed method converges to better invariant representation signiﬁcantly faster than AFL.
Revised: Experiments on both toy and real-world dataset prove that our proposed method converges to better invariant representation signiﬁcantly faster than AFL.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: How to learn representations invariant to nuisance attribute a is technical challenges raised in domain generalizaton(Blanchard et al., 2011; Muandet et al., 2013; Ghifary et al., 2015; Motiian et al., 2017), fair classiﬁcation,and privacy-protection (Edwards & Storkey, 2016; Iwasawa et al., 2017), and many other area.
Revised: How to learn representations invariant to nuisance attribute a is technical challenges raised in domain generalizaton (Blanchard et al., 2011; Muandet et al., 2013; Ghifary et al., 2015; Motiian et al., 2017), fair classiﬁcation, privacy-protection (Edwards & Storkey, 2016; Iwasawa et al., 2017), and many other area.

Original: Assume that we are given a training dataset made of pairs S = (cid:8) ( x i , y i , a i ) (cid:9) n i =1 , where x is an observation, y is a target of x , and a is a corresponding intrinsic attribute of K -way categorical variable A .
Revised: Assume that we are given a training dataset made of pairs S = (cid:8) ( x i , y i , a i ) (cid:9) n i =1 , where x is an observation, y is a target of x , and a is a corresponding intrinsic attribute of K -way categorical variable A .

Original: The goal of invariant representation learning is to obtain an encoder E that reduces information about attribute a while maintaining information about y .
Revised: The goal of invariant representation learning is to obtain an encoder E that reduces information about attribute a while maintaining information about y .

Original: An adversarial game between a feature extractor and an attribute classiﬁer, called adversarial feature learning (Xie et al., 2017), is a powerful framework for this purpose.
Revised: An adversarial game between a feature extractor and an attribute classiﬁer, called adversarial feature learning (Xie et al., 2017), is a powerful framework for this purpose.

Original: The key of AFL is to measure the invariance by leveraging the discriminative power of neural network beyond the pre-deﬁned metric such as l 2 distance or maximum mean discrepancy.
Revised: The key of AFL is to measure the invariance by leveraging the discriminative power of neural network beyond the pre-deﬁned metric such as l 2 distance or maximum mean discrepancy.

Original: That is, if the external network (also referred to as a discriminator) can predict a from z = E ( x ) , AFL regards z to have considerable information about a .
Revised: That is, if the external network (also referred to as a discriminator) can predict a from z = E ( x ) , AFL regards z to have considerable information about a .

Original: Formally, the AFL solves the following optimization problem:
Revised: Formally, the AFL solves the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where q M and q D is the conditional probability that M and D gives a correct estimation of y and a respectively.
Revised: where q M and q D is the conditional probability that M and D gives a correct estimation of y and a respectively.

Original: As Xie et al.
Revised: As Xie et al.

Original: (2017) explained, this alternating procedure can be regarded as a way to maximize the conditional entropy H ( A | Z )
Revised: (2017) explained, this alternating procedure can be regarded as a way to maximize the conditional entropy H ( A | Z )

Original: =
Revised: =

Original: (cid:80) a ∈A ,z ∈Z − p ( a, z ) log p ( a | z ) , where A and Z is a support set of a and z .
Revised: (cid:80) a ∈A ,z ∈Z − p ( a, z ) log p ( a | z ) , where A and Z is a support set of a and z .

Original: Xie et al.
Revised: Xie et al.

Original: (2017) also showed that the min–max game has an equilibrium, in which E maximize the conditional entropy H
Revised: (2017) also showed that the min–max game has an equilibrium, in which E maximize the conditional entropy H

Original: ( A | Z ) .
Revised: ( A | Z ) .

Original: It has been show superior performance in fairclassiﬁcation, privacy-protection, and domain generalization tasks (Ganin et al., 2016;
Revised: It has been show superior performance in fairclassiﬁcation, privacy-protection, and domain generalization tasks (Ganin et al., 2016;

Original: Edwards &
Revised: Edwards &

Original: Storkey, 2016; Xie et al., 2017; Iwasawa et al., 2017), compared to the prediﬁned metric approaches (Zemel et al., 2013; Louizos et al., 2016; Motiian et al., 2017).
Revised: Storkey, 2016; Xie et al., 2017; Iwasawa et al., 2017), compared to the prediﬁned metric approaches (Zemel et al., 2013; Louizos et al., 2016; Motiian et al., 2017).

Original: Despite the theoretical justiﬁcations, the above min–max formulation is suspicious for several practical issues.
Revised: Despite the theoretical justiﬁcations, the above min–max formulation is suspicious for several practical issues.

Original: Namely, the gradient from the discriminator vanishes if the discriminator sufﬁciently trained since E
Revised: Namely, the gradient from the discriminator vanishes if the discriminator sufﬁciently trained since E

Original: [log q D ( a | z = E ( x ))] is small then.
Revised: [log q D ( a | z = E ( x ))] is small then.

Original: Besides, in mathematical level, it only keeps away representations from the non-desired point where we can predict a label correctly, but not ensure that it approaches the desired invariant point.
Revised: Besides, in mathematical level, it only keeps away representations from the non-desired point where we can predict a label correctly, but not ensure that it approaches the desired invariant point.

Original: Please also refer Fig.
Revised: Please also refer Fig.

Original: 1 for visualization of the instability.
Revised: 1 for visualization of the instability.

Original: Note that, GAN community evade similar issues by the incorporating alternative objectives, such as the Wasserstein distance (Arjovsky et al., 2017).
Revised: Note that, Generative Adversarial Networks community, which utilize similar formulation to generate realistic images, evade similar issues by the incorporating alternative objectives, such as the Wasserstein distance (Arjovsky et al., 2017).

Original: However, the Wasserstein distance is deﬁned over two distributions and applying to our setting (consisting of multiple distributions) is not trivial.
Revised: However, the Wasserstein distance is deﬁned over two distributions and applying to our setting (consisting of multiple distributions) is not trivial.

Original: This paper holds the following contributions to the invariant feature learning problem.
Revised: This paper holds the following contributions to the invariant feature learning problem.

Original: First, we empirically show that AFL is suffered from practical issues that signiﬁcantly slow down the convergence.
Revised: First, we empirically show that AFL is suffered from practical issues that signiﬁcantly slow down the convergence.

Original: We then reformulate the optimization problem of AFL as pair-wise distribution matching and derive parameter practical realization of pairwise distribution matching while inheriting the merit of
Revised: We then reformulate the optimization problem of AFL as pair-wise distribution matching and derive parameter practical realization of pairwise distribution matching while inheriting the merit of

Original: AFL that leveraging the discriminative power to measure the invariance.
Revised: AFL that leveraging the discriminative power to measure the invariance.

Original: It is worth mentioning that the reformulation enable us to use Wasserstein metric in theory, however, it is still computationally infeasible in practice because a naive way to calculate the Wasserstein distance between all the pair of the distributions requires O ( K 2 ) discriminators, where K = |A| , which raise computational issues both in terms of parameter size and forward/backward time.
Revised: It is worth mentioning that the reformulation enable us to use Wasserstein metric in theory, however, it is still computationally infeasible in practice because a naive way to calculate the Wasserstein distance between all the pair of the distributions requires O ( K 2 ) discriminators, where K = |A| , which raise computational issues both in terms of parameter size and forward/backward time.

Original: Finally, we empirically validate the superior performance of our proposed method on both artiﬁcial dataset and real-world datasets.
Revised: Finally, we empirically validate the superior performance of our proposed method on both artiﬁcial dataset and real-world datasets.

Original:  C ONVERGENCE ISSUES OF AFL
Revised:  C ONVERGENCE ISSUES OF AFL

Original:  a post-hoc classiﬁer D eval that have the same architecture as D .
Revised: with 100 hidden units followed by a ReLU activation). Similarly, a ﬂoat value in parenthesis on the top of ﬁgures is an NLL of a post-hoc classiﬁer D eval that have the same architecture as D .

Original: To be more speciﬁc, we ﬁrst train the discriminator 100 times with 128 batch size and train D and E iteratively with stochastic gradient descent with learning rate=0.1.
Revised: To be more speciﬁc, we ﬁrst train the discriminator 100 times with 128 batch size and train D and E iteratively with stochastic gradient descent with learning rate=0.1.

Original: Figure 1-(f,g) shows the gradient vector ﬁelds of different time steps for a = blue , where the arrow represents the direction of the gradient, and the norm represents its magnitude.
Revised: Figure 1-(f,g) shows the gradient vector ﬁelds of different time steps for a = blue , where the arrow represents the direction of the gradient, and the norm represents its magnitude.

Original: For simplicity, we only show the vector ﬁelds of a = blue , but the observations are quite similar for the other a .
Revised: For simplicity, we only show the vector ﬁelds of a = blue , but the observations are quite similar for the other a .

Original: The ﬁgure reveals two practical issues in AFL optimization.
Revised: The ﬁgure reveals two practical issues in AFL optimization.

Original: (1) The distribution alignment is initially quite slow (compare with the behavior of the proposed method shown in Figure 2).
Revised: (1) The distribution alignment is initially quite slow (compare with the behavior of the proposed method shown in Figure 2).

Original: This is because the gradient is small when the discriminator correctly distinguishes a a .
Revised: This is because the gradient is small when the discriminator correctly distinguishes a a .

Original: (2) AFL behavior is unstable.
Revised: (2) AFL behavior is unstable.

Original: The distributions somewhat align after 38 steps (given 0 .
Revised: The distributions somewhat align after 40 steps (given 0 .

Original: 630 NLL with the post-hoc classiﬁer), but it is catastrophically failed three steps later because the discriminator did not capture the true conditional entropy (implied by the mostly similar counterplot of D ) and therefore gave a false gradient as shown in (f) and (g).
Revised: 683 NLL with the post-hoc classiﬁer), but it is catastrophically failed ﬁve steps later because the discriminator did not capture the true conditional entropy (implied by the mostly similar counterplot of D ) and therefore gave a false gradient as shown in (f) and (g).

Original: The intuitive reason for this phenomenon is that AFLs loss essentially tries to pull a distribution apart from the non-desired point, i.e., the point where we can correctly predict the label.
Revised: The intuitive reason for this phenomenon is that AFLs loss essentially tries to pull a distribution apart from the non-desired point, i.e., the point where we can correctly predict the label.

Original: The problem of AFL is that it only keeps away a distribution from the non-desired point, but not ensure it approaches the desired invariant point.
Revised: The problem of AFL is that it only keeps away a distribution from the non-desired point, but not ensure it approaches the desired invariant point.

Original: After several steps, D starts to follow the change of the distribution (as shown in Figure 1-e).
Revised: After several steps, D starts to follow the change of the distribution (as shown in Figure 1-e).

Original: The instability of the AFL also appears in the signiﬁcant gap between the NLL of the D and D eval .
Revised: The instability of the AFL also appears in the signiﬁcant gap between the NLL of the D and D eval .

Original: Note that the second issue may be alleviated if D has a sufﬁciently large capacity and is trained many times at each iteration.
Revised: Note that the second issue may be alleviated if D has a sufﬁciently large capacity and is trained many times at each iteration.

Original: However, this is not a realistic assumption since it is fair to say that real datasets are more complicated than this toy situations, making it more challenging to ﬁnd the supremum.
Revised: However, this is not a realistic assumption since it is fair to say that real datasets are more complicated than this toy situations, making it more challenging to ﬁnd the supremum.

Original:  E FFICIENT P AIWRISE D ISTRIBUTION M ATCHING
Revised:  E FFICIENT P AIWRISE D ISTRIBUTION M ATCHING

Original: The next question we must answer is how to maximize conditional entropy while avoiding the issues mentioned above.
Revised: The next question we must answer is how to maximize conditional entropy while avoiding the issues mentioned above.

Original: Assume that A is a categorical random variable drawn from a uniform discrete distribution, and denote that support set of a and z as A and Z ; then, the following theorem holds:
Revised: Assume that A is a categorical random variable drawn from a uniform discrete distribution, and denote that support set of a and z as A and Z ; then, the following theorem holds:

Original: Theorem 1.
Revised: Theorem 1.

Original: The maximum conditional entropy H ( A | Z ) is − log 1 K , and H ( A | Z ) is maximized if and only if p ( z | a = i ) =
Revised: The maximum conditional entropy H ( A | Z ) is − log 1 K , and H ( A | Z ) is maximized if and only if p ( z | a = i ) =

Original: p ( z | a = j ) for all i (cid:54) =
Revised: p ( z | a = j ) for all i (cid:54) =

Original: j ∈ 1 , · · · , K and z ∈ Z .
Revised: j ∈ 1 , · · · , K and z ∈ Z .

Original: The proof is followed in the appendix.
Revised: The proof is followed in the appendix.

Original: One implication of the theorem is, the actual maximum conditional entropy should bound the approximated conditional entropy.
Revised: One implication of the theorem is, the empirical measurements of conditional entropy should be boudned.

Original: It suggests another problematic point of AFL objective since it is not lower bounded.
Revised: It suggests another problematic point of AFL objective since it is not lower bounded.

Original: Another implication is that this theorem permits us to rethink the problem of the conditional entropy maximization as a problem of aligning all pairs of  distributions p ( z | a = i ) and p ( z | a = j ) .
Revised: Another implication is that this theorem permits us to rethink the problem of the conditional entropy maximization as a problem of aligning all pairs of [Figure] [Figure] distributions p ( z | a = i ) and p ( z | a = j ) .

Original: This reformulation is signiﬁcant since we can now use any conventional measurement of two distributions.
Revised: This reformulation is signiﬁcant since we can now use any conventional measurement of two distributions.

Original: However, the naive approach requires to match all pairs of the distributions
Revised: However, the naive approach requires to match all pairs of the distributions

Original: (= K C 2 ), which would be cumbersome to compute.
Revised: (= K C 2 ), which would be cumbersome to compute.

Original: [Figure] [Figure] We, therefore, propose a computationally efﬁcient way to realize pairwise distribution matching, moving attribute perception matching .
Revised: We, therefore, propose a computationally efﬁcient way to realize pairwise distribution matching, moving attribute perception matching .

Original: As with AFL, APM is based on the alternating training of attribute classiﬁer D and feature extractor E , but APM deceives D differently.
Revised: As with AFL, APM is based on the alternating training of attribute classiﬁer D and feature extractor E , but APM deceives D differently.

Original: Formally, we propose the following APM matching objective:
Revised: Formally, we propose the following APM matching objective:

Original: [Equation]
Revised: [Equation]

Original: where k D is some distance function deﬁned over either a hidden representation of the discriminator
Revised: where k D is some distance function deﬁned over either a hidden representation of the discriminator

Original: D or output probability q D ( a | E ( x )) itself and C a j is the moving average of the centroid for attribute a j in the attribute perception.
Revised: D or output probability q D ( a | E ( x )) itself and C a j is the moving average of the centroid for attribute a j in the attribute perception.

Original: Although there are many valid choices for k D ( ., . ) , including the simple l 2 norm, we primarily use the well-known Kullback–Leibler (KL) divergence.
Revised: Although there are many valid choices for k D ( ., . ) , including the simple l 2 norm, we primarily use the well-known Kullback–Leibler (KL) divergence.

Original: We initialized
Revised: We initialized

Original: C 0 a j via computing the centroids using all training data points.
Revised: C 0 a j via computing the centroids using all training data points.

Original: The key of the proposal is that it only requires the same number of parameters  AFL, but it ensures that representations of an attribute approach to the representations of the other attributes.
Revised: The key of the proposal is that it only requires the same number of parameters as AFL, but it ensures that representations of an attribute approach to the representations of the other attributes.

Original: Also, it inherits the merit of AFL that leveraging the discriminative power to measure the invariance Figure 2-(a–c) shows the behavior of the APM under the exact same experimental settings shown in Figure 1.
Revised: Also, it inherits the merit of AFL that leveraging the discriminative power to measure the invariance. Figure 2-(a–c) shows the behavior of the APM under the exact same experimental settings shown in Figure 1.

Original: The proposed method maximizes the conditional entropy signiﬁcantly faster than AFL: 1 .after only ﬁve iterations, but AFL gives 0 .
Revised: The proposed method maximizes the conditional entropy signiﬁcantly faster than AFL: 1 .after only ﬁve iterations, but AFL gives 0 .

Original: 667 after 16 iterations.
Revised: 683 after 40 iterations.

Original: Note that the value match the theoretical maximum value of the conditional entropy ( log 1 3 ≈ 1 . 061 ).
Revised: Note that the value match the theoretical maximum value of the conditional entropy ( log 13 ≈ 1 . 061 ).

Original: The comparison of the full plot of NLL (Figure 2-(d,e) also shows the faster convergence and stable behavior of the APM.
Revised: The comparison of the full plot of NLL (Figure 2-(d,e) also shows the faster convergence and stable behavior of the APM.

Original: Another merit of the proposed method is that we can enforce semantic alignment with a simple modiﬁcation.
Revised: Another merit of the proposed method is that we can enforce semantic alignment with a simple modiﬁcation.

Original: Individually, semantic alignment can be carried out by merely computing the centroids for each (attribute, label) tuple and aligning the perceptions of { x, y, a } between only centroids of the same label y (cid:48)
Revised: Individually, semantic alignment can be carried out by merely computing the centroids for each (attribute, label) tuple and aligning the perceptions of { x, y, a } between only centroids of the same label y (cid:48)

Original: =
Revised: =

Original: y but different attributes a (cid:48)
Revised: y but different attributes a (cid:48)

Original: (cid:54) = a .
Revised: (cid:54) = a .

Original: Although this modiﬁcation does not help to maximize the conditional entropy, it prevents performance degradation of L y .
Revised: Although this modiﬁcation does not help to maximize the conditional entropy, it prevents performance degradation of predicting y .

Original: Since most applications of invariant feature learning require that L yis also minimized, we use this modiﬁcation for all the later-described experiments.
Revised: Since most applications of invariant feature learning require to keep a information about y we use this modiﬁcation for all the later-described experiments.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We use three domain generalization tasks and two user-anonymization tasks.
Revised: We use three domain generalization tasks and two user-anonymization tasks.

Original: MNISTR (Ghifary et al., 2015) and PACS (Krizhevsky et al., 2012) is a well known image-based datasets designed for the purpose of DG.
Revised: MNISTR (Ghifary et al., 2015) and PACS (Krizhevsky et al., 2012) is a well known image-based datasets designed for the purpose of domain generalization.

Original: We also test the methods of noise-robust speech recognition scenarios using Google Speech Command Dataset(Speech).
Revised: We also test the methods of noise-robust speech recognition [Figure] [Figure]

Revised: [Table]

Revised: scenarios using Google Speech Command Dataset (Speech).

Original: Regarding user-anonymization, we use two useranonymization tasks on the data of wearables, OppG and USC (Iwasawa et al., 2017).
Revised: Regarding user-anonymization, we use two user-anonymization tasks on the data of wearables, OppG and USC (Iwasawa et al., 2017).

Original: The neural networks require to learn representations that help activity classiﬁcation and at the same time, prevent to access the information about users (userID).
Revised: The neural networks require to learn representations that help activity classiﬁcation and at the same time, prevent to access the information about users (userID).

Original: As baselines, we use (1) A CNN trained on the [Figure] [Figure] [Table]
Revised: As baselines, we use (1) A CNN trained on the aggregation of data from all source domains.

Original: aggregation of data from all source domains.

Original: (2) AFL (Xie et al., 2017), which was explained in
Revised: (2) AFL (Xie et al., 2017), which was explained in

Original: Section 3.1.
Revised: Section 3.1.

Original: (3) RevGrad (Ganin et al., 2016) is a slightly modiﬁed version of AFL, which uses the gradient reversal layer to train all the networks.
Revised: (3) RevGrad (Ganin et al., 2016) is a slightly modiﬁed version of AFL, which uses the gradient reversal layer to train all the networks.

Original: (4) CrossGrad (Shankar et al., 2018) is regarded as a state-of-the-art method in domain generalization tasks.
Revised: (4) CrossGrad (Shankar et al., 2018) is regarded as a state-of-the-art method in domain generalization tasks.

Original: (5) Activation Matching (AM) , which trains the encoder with the regularization of the l 2 distance on a feature space.
Revised: (5) Activation Matching (AM) , which trains the encoder with the regularization of the l 2 distance on a feature space.

Original: (6) APM is our proposal.
Revised: (6) APM is our proposal.

Original: For all datasets and methods, we used RMSprop for each optimization.
Revised: For all datasets and methods, we used RMSprop for each optimization.

Original: For all datasets except
Revised: For all datasets except

Original: PACS, we set the learning rate to 0 .
Revised: PACS, we set the learning rate to 0 .

Original: 001 and the batch size to 128 .
Revised: 001 and the batch size to 128 .

Original: For PACS, we set the learning rate to 5 e − 5 and the batch size to 64 .
Revised: For PACS, we set the learning rate to 5 e − 5 and the batch size to 64 .

Original: For a fair comparison, hyperparameters were tuned on a validation set for each baseline.
Revised: For a fair comparison, hyperparameters were tuned on a validation set for each baseline.

Original: For the adversarial-training-based method, we optimized weighting parameter λ from { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 } , except for MNISTR, for which it was optimized from { 0 . 01 , 0 . 1 , 1 . 0 , 2 . 0 } .
Revised: For the adversarial-training-based method, we optimized weighting parameter λ from { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 } , except for MNISTR, for which it was optimized from { 0 . 01 , 0 . 1 , 1 . 0 , 2 . 0 } .

Original: The value of α for CrossGrad was selected from { 0 . 1 , 0 . 25 , 0 . 5 , 0 . 75 , 0 . 9 } .
Revised: The value of α for CrossGrad was selected from { 0 . 1 , 0 . 25 , 0 . 5 , 0 . 75 , 0 . 9 } .

Original: We set the decay rate γ to 0 .
Revised: We set the decay rate γ to 0 .

Original: 7 for all experiments.
Revised: 7 for all experiments.

Original: In all the experiments, we selected the data of one or several domains for the test set and used the data of a disjoint domain as the training/validation data.
Revised: In all the experiments, we selected the data of one or several domains for the test set and used the data of a disjoint domain as the training/validation data.

Original: Speciﬁcally, we split the data of the disjoint domain into groupings of 80% and 20% .
Revised: Speciﬁcally, we split the data of the disjoint domain into groupings of 80% and 20% .

Original: We denote the test domain by a sufﬁx (e.g., MNISTR-M0).
Revised: We denote the test domain by a sufﬁx (e.g., MNISTR-M0).

Original: We measured the label classiﬁcation accuracy and the level of invariance.
Revised: We measured the label classiﬁcation accuracy and the level of invariance.

Original: We measured the level of invariance by training a post-hoc classiﬁer f eva following previous studies Xie et al.
Revised: We measured the level of invariance by training a post-hoc classiﬁer f eva following previous studies Xie et al.

Original: ; Iwasawa et al.
Revised: ; Iwasawa et al.

Original: Figure 3 compares the performance the invariance of representations.
Revised: Figure 3 compares the performance the invariance of representations.

Original: For each method, we used the largest weighting parameter λ on the condition that label classiﬁcation accuracy did not signiﬁcantly decrease.
Revised: For each method, we used the largest weighting parameter λ on the condition that label classiﬁcation accuracy did not signiﬁcantly decrease.

Original: The results show that the proposed method stably achieves better invariant representations except for speech dataset.
Revised: The results show that the proposed method stably achieves better invariant representations except for speech dataset.

Original: For example, APM achieved 20% of the A-Acc in MNISTR, which is nearly perfect invariance as there are ﬁve domains in the validation data, while RevGrad and AFL achieved 30% at best.
Revised: For example, APM achieved 20% of the A-Acc in MNISTR, which is nearly perfect invariance as there are ﬁve domains in the validation data, while RevGrad and AFL achieved 30% at best.

Original: On speech dataset, the performance of AFL and APM is mostly similar.
Revised: On speech dataset, the performance of AFL and APM is mostly similar.

Original: These results conﬁrm that the proposed method stably achieves more invariant representation compared with AFL.
Revised: These results conﬁrm that the proposed method stably achieves more invariant representation compared with AFL.

Original: Table 1 summarizes the methods’ classiﬁcation performance on three different datasets:
Revised: Table 1 summarizes the methods’ classiﬁcation performance on three different datasets:

Original: MNISTR, Speech, and PACS.
Revised: MNISTR, Speech, and PACS.

Original: The leftmost column of each table represents the test domain.
Revised: The leftmost column of each table represents the test domain.

Original: We report the mean accuracy.
Revised: We report the mean accuracy.

Original: We can make the following observations.
Revised: We can make the following observations.

Original: (1) APM demonstrates the best or comparable performance on all three datasets, including CrossGrad, which is regarded as the state-of-the-art method in domain generalization tasks.
Revised: (1) APM demonstrates the best or comparable performance on all three datasets, including CrossGrad, which is regarded as the state-of-the-art method in domain generalization tasks.

Original: (2) RevGrad and AFL often fail to improve performance even when compared with a standard CNN.
Revised: (2) RevGrad and AFL often fail to improve performance even when compared with a standard CNN.

Original: These results suggest that the previous adversarial-training-based method suffered from the lack of semantic alignment when applied to domain generalization.
Revised: These results suggest that the previous adversarial-training-based method suffered from the lack of semantic alignment when applied to domain generalization.

Original: The Wilcoxon rank sum test shows that APM is statistically better than
Revised: The Wilcoxon rank sum test shows that APM is statistically better than

Original: CNN, RevGrad, and AFL with p < 0 .
Revised: CNN, RevGrad, and AFL with p < 0 .

Original: 01 , and than CrossGrad with p < 0 .
Revised: 01 , and than CrossGrad with p < 0 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes a new approach to incorporating desired invariance to representations learning, based on the observations that the current state-of-the-art AFL has practical issues.
Revised: This paper proposes a new approach to incorporating desired invariance to representations learning, based on the observations that the current state-of-the-art AFL has practical issues.

Original: Empirical results on both toy and real-world datasets support the stable performance of the proposed method to learn invariant features and superior performance on domain generalization tasks.
Revised: Empirical results on both toy and real-world datasets support the stable performance of the proposed method to learn invariant features and superior performance on domain generalization tasks

Original: A P ROOF OF THE THEOREM
Revised: A P ROOF OF THE THEOREM

Original: Proof.
Revised: Proof.

Original: Using the Lagrange multiplier method the derivative of
Revised: Using the Lagrange multiplier method, the derivative of

Original: [Equation]
Revised: [Equation]

Original: is equal to zero for the maximum entropy H ( A | Z ) .
Revised: is equal to zero for the maximum entropy H ( A | Z ) .

Original: Solving the simultaneous equations, we can say p ( a =1 | z ) =
Revised: Solving the simultaneous equations, we can say p ( a =1 | z ) =

Original: p ( a =2 | z ) = · · · = p ( a = K | z )
Revised: p ( a =2 | z ) = · · · = p ( a = K | z )

Original: = 1 K for all z ∈ Z when the conditional entropy is maximized and based on the deﬁnition, the conditional entropy become − log 1 K .
Revised: = 1 K for all z ∈ Z when the conditional entropy is maximized, and based on the deﬁnition, the conditional entropy become − log 1 K .

Original: From Bayes’ law,
Revised: From Bayes’ law,

Original: [Equation]
Revised: [Equation]

Original: (4) holds ∀ i (cid:54) =
Revised: holds ∀ i (cid:54) =

Original: j ∈ A and z ∈ Z .
Revised: j ∈ A and z ∈ Z .