L EARNING R OBUST R EPRESENTATIONS BY P ROJECTING S UPERFICIAL S TATISTICS O UT
A BSTRACT
Despite impressive performance as evaluated on i.i.d.
holdout data, deep neural networks depend heavily on superﬁcial statistics of the training data and are liable to break under distribution shift.
For example, subtle changes to the background or texture of an image can break a seemingly powerful classiﬁer.
Building on previous work on domain generalization, we hope to produce a classiﬁer that will generalize to previously unseen domains, even when domain identiﬁers are not available during training.
This setting is challenging because the model may extract many distribution-speciﬁc (superﬁcial) signals together with distribution-agnostic (semantic) signals.
To overcome this challenge, we incorporate the gray-level cooccurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superﬁcial: they are sensitive to texture but unable to capture the gestalt of an image.
Then we introduce two techniques for improving our networks’ outof-sample performance.
The ﬁrst method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable.
The second method is built on the independence introduced by projecting the model’s representation onto the subspace orthogonal to
GLCM representation’s.
We test our method on battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.
I NTRODUCTION
Imagine training an image classiﬁer to recognize facial expressions.
In the training data, while all images labeled “smile” may actually depict smiling people, the “smile” label might also be correlated with other aspects of the image.
For example, people might tend to smile more often while outdoors, and to frown more in airports.
In the future, we might encounter photographs with previously unseen backgrounds, and thus we prefer models that rely as little as possible on the superﬁcial signal.
The problem of learning classiﬁers robust to distribution shift, commonly called Domain Adaptation (DA) , has a rich history.
Under restrictive assumptions, such as covariate shift (Shimodaira, 2000;
Gretton et al., 2009), and label shift (also known as target shift or prior probability shift ) (Storkey, 2009; Sch¨olkopf et al., 2012; Zhang et al., 2013; Lipton et al., 2018), principled methods exist for estimating the shifts and retraining under the importance-weighted ERM framework.
Other papers bound worst-case performance under bounded shifts as measured by divergence measures on the train v.s. test distributions (Ben-David et al., 2010a; Mansour et al., 2009; Hu et al., 2016).
While many impossibility results for DA have been proven (Ben-David et al., 2010b), humans nevertheless exhibit a remarkable ability to function out-of-sample, even when confronting dramatic
[Figure]
distribution shift.
Few would doubt that given photographs of smiling and frowning astronauts on the Martian plains, we could (mostly) agree upon the correct labels.
While we lack a mathematical description of how precisely humans are able to generalize so easily out-of-sample, we can often point to certain classes of perturbations that should not effect the semantics of an image.
For example for many tasks, we know that the background should not inﬂuence the predictions made about an image.
Similarly, other superﬁcial statistics of the data, such as textures or subtle coloring changes should not matter.
The essential assumption of this paper is that by making our model depend less on known superﬁcial aspects, we can push the model to rely more on the difference that makes a difference .
This paper focuses on visual applications, and we focus on high-frequency textural information as the relevant notion of superﬁcial statistics that we do not want our model to depend upon.
The contribution of this paper can be summarized as follows.
• We propose a new differentiable neural network building block (neural gray-level cooccurrence matrix) that captures textural information only from images without modeling the lower-frequency semantic information that we care about (Section 3.1).
• We propose an architecture-agnostic, parameter-free method that is designed to discard this superﬁcial information, (Section 3.2).
• We introduce two synthetic datasets for DA/DG studies that are more challenging than regular DA/DG scenario in the sense that the domain-speciﬁc information is correlated with semantic information.
Figure 1 is a toy example (Section 4).
R ELATED W ORK IN D OMAIN A DAPTATION AND D OMAIN G ENERALIZATION
Domain generalization (DG) (Muandet et al., 2013) is a variation on DA, where samples from the target domain are not available during training.
In reality, data-sets may contain data cobbled together from many sources but where those sources are not labeled.
For example, a common assumption used to be that there is one and only one distribution for each dataset collected, but Wang et al.
(2016) noticed that in video sentiment analysis, the data sources varied considerably even within the same dataset due to heterogeneous data sources and collection practices.
Domain adaptation (Bridle & Cox, 1991; Ben-David et al., 2010a), and (more broadly) transfer learning have been studied for decades, with antecedents in the classic econometrics work on sample selection bias Heckman (1977) and choice models Manski & Lerman (1977).
For a general primer, we refer the reader to these extensive reviews (Weiss et al., 2016; Csurka, 2017).
Domain generalization (Muandet et al., 2013) is relatively new, but has also been studied extensively: covering a wide spectrum of techniques from kernel methods (Muandet et al., 2013; Niu et al., 2015;
Erfani et al., 2016; Li et al., 2017c) to more recent deep learning end-to-end methods, where the methods mostly fall into two categories: reducing the inter-domain differences of representations through adversarial (or similar) techniques (Ghifary et al., 2015;
Wang et al., 2016; Motiian et al., 2017; Li et al., 2018; Carlucci et al., 2018), or building an ensemble of
one-for-each-domain deep
[Figure]
T s o f t m a x
CNN
NGLCM
HEX
F A
F G
F L
M ETHOD
In this section, we introduce our main technical contributions.
We will ﬁrst introduce the our new differentiable neural building block, NGLCM that is designed to capture textural but not semantic information from images, and then introduce our technique for excluding the textural information.
3. N EURAL G RAY -L EVEL C O - OCCURRENCE M ATRIX FOR S UPERFICIAL I NFORMATION
Our goal is to design a neural building block that 1) has enough capacity to extract the textural information from an image, 2) is not capable of extracting semantic information.
We consulted some classic computer vision techniques for inspiration and extensive experimental evidence (Appendix A1), suggested that gray-level co-occurrence matrix (GLCM) (Haralick et al., 1973; Lam, 1996) may suit our goal.
The idea of GLCM is to count the number of pixel pairs under a certain direction (common direction choices are 0 ◦ , 45 ◦ , 90 ◦ , and 135 ◦ ).
For example, for an image
A ∈ N m × m , where N denotes the set of all possible pixel values.
The GLCM of A under the direction to 0 ◦ (horizontally right) will be a | N |×| N | matrix (denoted by G ) deﬁned as following:
[Equation]
where | N | stands for the cardinality of N , I ( · ) is an identity function, i, j are indices of A , and k, l are pixel values of A as well as indices of G .
We design a new neural network building block that resembles GLCM but whose parameters are differentiable, having (sub)gradient everywhere, and thus are tunable through backpropagation.
We ﬁrst ﬂatten A into a row vector a ∈ N 1 × m 2 .
The ﬁrst observation we made is that the counting of pixel pairs ( p k , p l ) in Equation 1 is equivalent to counting the pairs ( p k , ∆ p ), where ∆ p = p k − p l .
Therefore, we ﬁrst generate a vector d by multiplying a with a matrix D , where D is designed according to the direction of GLCM.
For example, D in the 0 ◦ case will be a m 2 × m 2 matrix D such that D i,i = 1 , D i,i +1 = − 1 , and 0 elsewhere.
To count the elements in a and d with a differentiable operation, we introduce two sets of parameters φ a ∈ R | N |× 1 and φ b ∈ R | N |× 1 as the tunable parameter for this building block, so that:
[Equation]
where s () is a thresholding function deﬁned as:
[Equation]
where (cid:9) denotes the minus operation with the broadcasting mechanism, yielding both s ( a ; φ a ) and s ( d ; φ b ) as | N | × m 2 matrices.
As a result, G is a | N | × | N | matrix.
The design rationale is that, with an extra constrain that requires φ to have only unique values in the set of { n − (cid:15) | n ∈ N } , where (cid:15) is a small number, G in Equation 2 will be equivalent to the GLCM extracted with old counting techniques, subject to permutation and scale.
Also, all the operations used in the construction of G have (sub)gradient and therefore all the parameters are tunable with backpropagation.
In practice, we drop the extra constraint on φ for simplicity in computation.
Our preliminary experiments suggested that for our purposes it is sufﬁcient to ﬁrst map standard images with 256 pixel levels to images with 16 pixel levels, which can reduce to the number of parameters of NGLCM ( | N | = 16).
3. HEX
We ﬁrst introduce notation to represent the neural network.
We use (cid:104) X, y (cid:105) to denote a dataset of inputs X and corresponding labels y .
We use h ( · ; θ ) and f ( · ; ξ ) to the bottom and top components of a neural network.
A conventional neural network architecture will use f ( h ( X ; θ ); ξ ) to generate a corresponding result F i and then calculate the argmax to yield the prediction label.
Besides conventional f ( h ( X ; θ ); ξ ) , we introduce another architecture
[Equation] [Equation]
s
[Equation]
a
[Equation]
a
[Equation]
) where φ = { φ a , φ b , W m , b m } , s ( a ; φ a ) s T ( d ; φ b ) is introduced in previous section, { W m , b m , σ m } (weights, biases, and activation function) form a standard MLP.
With the introduction of g ( · ; φ ) , the ﬁnal classiﬁcation layer turns into f [ h ( X ; θ ) , g ( X ; φ )]; ξ ) (where we use [ · , · ] to denote concatenation).
Now, with the representation learned through raw data by h ( · ; θ ) and textural representation learned by g ( · ; φ ) , the next question is to force f ( · ; ξ ) to predict with transformed representation from h ( · ; θ ) that in some sense independent of the superﬁcial representation captured by g ( · ; φ ) .
To illustrate following ideas, we ﬁrst introduce three different outputs from the ﬁnal layer:
[Equation]
where F A , F G , and F P stands for the results from both representations (concatenated), only the textural information (prepended with the 0 vector), and only the raw data (concatenated wit hthevecotr), respectively.
0 stands for a padding matrix with all the zeros, whose shape can be inferred by context.
Several heuristics have been proposed to force a network to “forget” some part of a representation, such as adversarial training (Ganin et al., 2016) or information-theoretic regularization (Moyer et al., 2018).
In similar spirit, our ﬁrst proposed solution is to adopt the reverse gradient idea (Ganin et al., 2016) to train F P to be predictive for the semantic labels y while forcing the F P to be invariant to
F G .
Later, we refer to this method as ADV .
When we use a multilayer perceptron (MLP) to try to predict g ( X ; φ ) from h ( X ; θ ) and update the primary model to fool the MLP via reverse gradient, we refer to the model as ADVE .
Additionally, we introduce a simple alternative.
Our idea lies in the fact that, in an afﬁne space, to ﬁnd a transformation of representation A that is least explainable by some other representation B , a straightforward method will be projecting A with a projection matrix constructed by B (sometimes referred as residual maker matrix.).
To utilize this linear property, we choose to work on the space of F generated by f ( · ; ξ ) right before the ﬁnal argmax function.
Projecting F A with
[Equation]
will yield F L for parameter tuning.
All the parameters ξ, φ, θ can be trained simultaneously (more relevant discussions in Section 5).
In testing time, F P is used.
Due to limited space, we leave the following topics to the Appendix: 1) rationales of this approach (A2.1) 2) what to do in cases when F TG F G is not invertible (A2.2).
This method is referred as HEX .
Two alternative forms of our algorithm are also worth mentioning: 1) During training, one can tune an extra hyperparameter ( λ ) through
[Equation]
to ensure that the NGLCM component is learning superﬁcial representations that are related to the present task where l ( · , · ) is a generic loss function.
2) During testing, one can use F L , although this requires evaluating the NGLCM component at prediction time and thus is slightly slower.
We experimented with these three forms with our synthetic datasets and did not observe signiﬁcant differences in performance and thus we adopt the fastest method as the main one.
Empirically, we also notice that it is helpful to make sure the textural representation g ( X ; φ ) and raw data representation h ( X ; θ ) are of the same scale for HEX to work, so we column-wise normalize these two representations in every minibatch.
E XPERIMENTS
To show the effectiveness of our proposed method, we conduct range of experiments, evaluating
HEX’s resilience against dataset shift.
To form intuition, we ﬁrst examine the NGLCM and HEX separately with two basic testings, then we evaluate on two synthetic datasets, on in which dataset shift is introduced at the semantic level and another at the raw feature level, respectively.
We ﬁnally evaluate other two standard domain generalization datasets to compare with the state-of-the-art.
All these models are trained with ADAM (Kingma & Ba, 2014).
We conducted ablation tests on our two synthetic datasets with two cases 1) replacing NGLCM with one-layer MLP (denoted as M ), 2) not using HEX/ADV (training the network with F A (Equation 3) instead of F L (Equation 4))
(denoted as N ).
We also experimented with the two alternative forms of
HEX: 1) with F G in the loss and λ = 1 (referred as HEX-ADV), 2) predicting with F L (referred as HEX-ALL).
We also compare with the popular DG methods (DANN (Ganin et al., 2016)) and another method called information-dropout (Achille & Soatto, 2018).
4. S YNTHETIC E XPERIMENTS FOR B ASIC P ERFORMANCE T ESTS 4.1. NGLCM O NLY E XTRACTS T EXTURAL I NFORMATION
To show that the NGLCM only extracts textural information, we trained the network with a mixture of four digit recognition data sets: MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011),
MNIST-M (Ganin & Lempitsky, 2014), and USPS (Denker et al., 1989).
We compared NGLCM with a single layer of MLP.
The parameters are trained to minimize prediction risk of digits (instead of domain ).
We extracted the representations of NGLCM and MLP and used these representations as features to test the ﬁve-fold cross-validated Na¨ıve Bayes classiﬁer’s accuracy of predicting digit and domain.
With two choices of learning rates, we repeated this for every epoch through 100 epochs of training and reported the mean and standard deviation over 100 epochs in Table 1: while MLP and
NGLCM perform comparably well in extracting textural information, NGLCM is signiﬁcantly less useful for recognizing the semantic label.
4.1. HEX PROJECTION
To test the effectiveness of HEX, we used the extracted SURF (Bay et al., 2006) features (dimension) and GLCM (Lam, 1996) features ( 256 dimension) from ofﬁce data set (Saenko et al.,
[Table] [Table]
2010) ( 31 classes).
We built a two-layer MLP ( 800 × 256 , and 256 × 31 ) as baseline that only predicts with SURF features.
This architecture and corresponding learning rate are picked to make sure the baseline can converge to a relatively high prediction performance.
Then we plugged in the GLCM part with an extra ﬁrst-layer network 256 × 32 and the second layer of the baseline is extended to 288 × 31 to take in the information from GLCM.
Then we train the network again with HEX with the same learning rate.
The Ofﬁce data set has three different subsets: Webcam ( W ), Amazon ( A ), and DSLR ( D ).
We trained and validated the model on a mixture of two and tested on the third one.
We ran ﬁve experiments and reported the averaged accuracy with standard deviation in Table 2.
These performances are not comparable to the state-of-the-art because they are based on features.
At ﬁrst glance, one may frown upon on the performance of HEX because out of three conﬁgurations, HEX only outperforms the baseline in the setting { W , D } → A .
However, a closer look into the datasets gives some promising indications for HEX: we notice W and D are distributed similarly in the sense that objects have similar backgrounds, while A is distributed distinctly (Appendix A3.1).
Therefore, if we assume that there are two classiﬁers C 1 and C 2 : C 1 can classify objects based on object feature and background feature while C 2 can only classify objects based on object feature ignoring background feature.
C 2 will only perform better than C 1 in { W , D } →
A case, and will perform worse than Cin the other two cases, which is exactly what we observe with HEX.
4. F ACIAL E XPRESSION C LASSIFICATION WITH N UISANCE B ACKGROUND
We generated a synthetic data set extending the Facial Expression Research Group Database (Aneja et al., 2016), which is a dataset of six animated individuals expressing seven different sentiments.
For each pair of individual and sentiment, there are over 1000 images.
To introduce the data shift, we attach seven different backgrounds to these images.
In the training set (50% of the data) and validation set (30% of the data), the background is correlated with the sentiment label with a correlation of ρ ; in testing set (the rest 20% of the data), the background is independent of the sentiment label.
A simpler toy example of the data set is shown in Figure 1.
In the experiment, we format the resulting images to 28 × 28 grayscale images.
We run the experiments ﬁrst with the baseline CNN (two convolutional layers and two fully connected layers) to tune for hyperparameters.
We chose to run 100 epochs with learning rate 5ebecause this is when the CNN can converge for all these 10 synthetic datasets.
We then tested other methods with the same learning rate.
The results are shown in Figure 3 with testing accuracy and standard deviation from ﬁve repeated experiments.
Testing accuracy is reported by the model with the highest validation score.
In the ﬁgure, we compare baseline CNN ( B ), Ablation Tests ( M and N ),
ADV ( A ), HEX ( H ), DANN ( G ), and InfoDropout ( I ).
Most these methods perform well when ρ is small (when testing distributions are relatively similar to training distribution).
As ρ increases, most methods’ performances decrease, but Adv and HEX behave relatively stable across these ten correlation settings.
We also notice that, as the correlation becomes stronger, M deteriorates at a faster pace than other methods.
Intuitively, we believe this is because the MLP learns both from the seman-
B M N E A H V L D
I 0.
A cc u r a c y ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
B M N E A H V L D
I 0.
A cc u r a c y ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
B M N E A H V L D
I 0.
ρ : 0.
( G ), and InfoDropout ( I ).
BMN E
A H V L D
I 0.
A cc u r a c y original independent
BMN E
A H V L D
I 0.
random independent
BMN E
A H V L D
I 0.
radial independent
BMN E
A H V L D
I 0.
originaldependent
BMN E
A H V L D
I 0.
randomdependent
BMN E
A H V L D
I 0.
radial dependent tic signal together with superﬁcial signal, leading to inferior performance when HEX projects this signal out.
We also notice that ADV and HEX improve the speed of convergence (Appendix A3.2).
4. M ITIGATING THE T ENDENCY OF S URFACE S TATISTICAL R EGULARITIES IN MNIST
As Jo & Bengio (2017) observed, CNNs have a tendency to learn the surface statistical regularities: the generalization of CNNs is partially due to the abstraction of high level semantics of an image, and partially due to surface statistical regularities.
Here, we demonstrate the ability of HEX to overcome such tendencies.
We followed the radial and random Fourier ﬁltering introduced in (Jo &
Bengio, 2017) to attach the surface statistical regularities into the images in MNIST.
There are three different regularities altogether (radial kernel, random kernel, and original image).
We attached two of these into training and validation images and the remaining one into testing images.
We also adopted two strategies in attaching surface patterns to training/validation images: 1) independently : the pattern is independent of the digit, and 2) dependently : images of digit 0-4 have one pattern while images of digit 5-9 have the other pattern.
Some examples of this synthetic data are shown in
Appendix A3.3.
We used the same learning rate scheduling strategy as in the previous experiment.
The results are shown in Figure 4.
Figure legends are the same as previous.
Interestingly, NGLCM and HEX contribute differently across these cases.
When the patterns are attached independently, M performs the best overall, but when the patterns are attached dependently, N and HEX perform the best overall.
In the most challenging case of these experiments (random kerneled as testing, pattern attached dependently), HEX shows a clear advantage.
Also, HEX behaves relatively more stable overall.
[Table]
We continue to compare HEX with other state-of-the-art DG methods (that use distribution labels) on popular DG data sets.
We experimented with the MNIST-rotation data set
[Table]
on which many DG methods have been tested.
The images are rotated with different degrees to create different domains.
We followed the approach introduced by Ghifary et al.
To reiterate: we randomly sampled a set M of 1000 images out of MNIST (100 for each label).
Then we rotated the images in M counter-clockwise with different degrees to create data in other domains, denoted by M 15 ◦ , M 30 ◦ , M 45 ◦ , M 60 ◦ , M 75 ◦ .
With the original set, denoted by M 0 ◦ , there are six domains altogether.
We compared the performance of HEX/ADV with several methods tested on this data including
CAE (Rifai et al., 2011), MTAE (Ghifary et al., 2015), CCSA (Motiian et al., 2017), DANN (Ganin et al., 2016), Fusion (Mancini et al., 2018), LabelGrad, and CrossGrad (Shankar et al., 2018).
The results are shown in Table 3: HEX is only inferior to previous methods in one case and leads the average performance overall.
4. PACS: G ENERALIZATION IN P HOTO , A RT , C ARTOON , AND S KETCH
Finally, we tested on the PACS data set (Li et al., 2017a), which consists of collections of images of seven different objects over four domains, including photo, art painting, cartoon, and sketch.
Following (Li et al., 2017a), we used AlexNet as baseline method and built HEX upon it.
We met some optimization difﬁculties in directly training AlexNet on PACS data set with HEX, so we used a heuristic training approach: we ﬁrst ﬁne-tuned the AlexNet pretrained on ImageNet with PACS data of training domains without plugging in NGLCM and HEX, then we used HEX and NGLCM to further train the top classiﬁer of AlexNet while the weights of the bottom layer are ﬁxed.
Our heuristic training procedure allows us to tune the AlexNet with only 10 epoches and train the toplayer classiﬁer 100 epochs (roughly only 600 seconds on our server for each testing case).
We compared HEX/ADV with the following methods that have been tested on PACS: AlexNet (directly ﬁne-tuning pretrained AlexNet on PACS training data (Li et al., 2017a)), DSN (Bousmalis et al., 2016), L-CNN (Li et al., 2017a),
MLDG (Li et al., 2017b), Fusion (Mancini et al., 2018).
Notice that most of the competing methods (DSN, L-CNN, MLDG, and Fusion) have explicit knowledge about the domain identiﬁcation of the training images.
The results are shown in
Table 4.
Impressively, HEX is only slightly shy of Fusion in terms of overall performance.
Fusion is a method that involves three different AlexNets, one for each training domain, and a fusion layer to combine the representation for prediction.
The Fusion model is roughly three times bigger than
HEX since the extra NGLCM component used by HEX is negligible in comparison to AlexNet in terms of model complexity.
Interestingly, HEX achieves impressively high performance when the testing domain is Art painting and Cartoon, while Fusion is good at prediction for Photo and Sketch.
D ISCUSSION AND C ONCLUSION
We introduced two novel components: NGLCM that only extracts textural information from an image, and HEX that projects the textural information out and forces the model to focus on semantic information.
Limitations still exist.
For example, NGLCM cannot be completely free of semantic information of an image.
As a result, if we apply our method on standard MNIST data set
, we will
[Table]
see slight drop of performance because NGLCM also learns some semantic information, which is then projected out.
Also, training all the model parameters simultaneously may lead into a trivial solution where F G (in Equation 3) learns garbage information and HEX degenerates to the baseline model.
To overcome these limitations, we invented several training heuristics, such as optimizing
F P and F G sequentially and then ﬁx some weights.
However, we did not report results with training heuristics (expect for PACS experiment) because we hope to simplify the methods.
Another limitation we observe is that sometimes the training performance of HEX ﬂuctuates dramatically during training, but fortunately, the model picked up by highest validation accuracy generally performs better than competing methods.
Despite these limitations, we still achieved impressive performance on both synthetic and popular DG data sets.
A PPENDIX A R EASONS TO C HOOSE GLCM
In order to search the old computer vision techniques for a method that can extract more textural information and less semantic information, we experimented with three classifcal computer vision techniques: SURF (Bay et al., 2006), LBP (He & Wang, 1990), and GLCM (Haralick et al., 1973) on several different data sets: 1) a mixture of four digit data sets (MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011), MNIST-M (Ganin & Lempitsky, 2014), and USPS (Denker et al., 1989)) where the semantic task is to recognize the digit and the textural task is to classify the which data set the image is from; 2) a rotated MNIST data set with 10 different rotations where the semantic task is to recognize the digit and the textural task is to classify the degrees of rotation; 3) a MNIST data randomly attached one of 10 different types of radial kernel, for which the semantic task is to recognize digits and the textural task is to classify the different kernels.
[Table]
From results in Table A1, we can see that GLCM suits our goal very well: GLCM outperforms other methods in most cases in classifying textural patterns while predicts least well in the semantic tasks.
A E XPLANATION OF HEX A2. M ATHEMATICAL R ATIONALE
With F A and F G calculated in Equation 3, we need to transform the representation of F A so that it is least explainable by F G .
Directly adopting subtraction maybe problematic because the F A − F G can still be correlated with F G .
A straightforward way is to regress the information of F G out of
F A .
Since both F A and F G are in the same space and the only operation left in the network is the argmax operation, which is linear, we can safely use linear operations.
To form a standard linear regression problem, we ﬁrst consider the column k of F A , denoted by
F ( k ) A .
To solve a standard linear regression problem is to solve:
[Equation]
This function has a closed form solution when the minibatch size is greater than the number of classes of the problem ( i.e. when the number of rows of F G is greater than number of columns of
F G ), and the closed form solution is:
[Equation]
Therefore, for k th column of F A , what cannot be explained by F G is (denoted by F ( k ) L ):
[Equation]
Repeat this for every column of F A will lead to:
[Equation]
which is Equation 4.
A2. W HEN F TG F G IS NOT INVERTIBLE
As we mentioned above, Equation 4 can only be derived when the minibatch size is greater than the number of classes to predict because F TG F G is only non-singular (invertible) when this condition is met.
Therefore, a simple technique to always guarantee a solution with HEX is to use a minibatch size that is greater than the number of classes.
We believe this is a realistic requirement because in the real-world application, we always know the number of classes to classify, and it is usually a number much smaller than the maximum minibatch size a modern computer can deal with.
However, to complete this paper, we also introduce a more robust method that is always applicable independent of the choices of minibatch sizes.
We start with the simple intuition that to make sure F TG F G is always invertible, the simplest conduct will be adding a smaller number to the diagonal, leading to F TG F G + λI , where we can end the discussion by simply treating λ as a tunable hyperparameter.
However, we prefer that our algorithm not require tuning additional hyperparameters.
We write
F TG F G + λI back to the previous equation,
[Equation]
With the Kailath Variant (Bishop et al., 1995), we can have:
[Equation]
where β ( k ) λ is a result of a heteroscadestic regression method where λ can be estimated through maximum likelihood estimation (MLE) (Wang et al., 2017), which completes the story of a hyperparameter-free method even when F TG F G is not invertible.
However, in practice, we notice that the MLE procedure is very slow and the estimation is usually sensitive to noise.
As a result, we recommend users to simply choose a larger minibatch size to avoid the problem.
Nonetheless, we still release these steps here to 1) make the paper more complete, 2) offer a solution when in rase cases a model is asked to predict over hundreds or thousands of classes.
Also, we name our main method “HEX” as short of heteroscadestic regression.
A E XTRA E XPERIMENT R ESULTS A3. A CLOSER LOOK INTO O FFICE D ATA SET
We visualize some images of the ofﬁce data set in Figure A1, where we can see that the background of images for DSLR and Webcam are very similar while the background of images in Amazon are distinctly different from these two.
A3. HEX CONVERGES MUCH FASTER
We plotted the testing accuracy of each method in the facial expression classiﬁcation in Figure A2.
From the ﬁgure, we can see that HEX and related ablation methods converge signiﬁcantly faster than baseline methods.
A3. E XAMPLES OF P ATTERN - ATTACHED MNIST DATA SET
Examples of MNIST images when attached with different kernelled patterns following (Jo & Bengio, 2017), as shown in Figure A3.
[Figure] [Figure]