MetaMask: Revisiting Dimensional Confounder for Self-Supervised Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
As a successful approach to self-supervised learning, contrastive learning aims tolearn invariant information shared among distortions of the input sample.
Whilecontrastive learning has yielded continuous advancements in sampling strategyand architecture design, it still remains two persistent defects: the interference oftask-irrelevant information and sample inefficiency, which are related to the recurring existence of trivial constant solutions.
From the perspective of dimensionalanalysis, we find out that the dimensional redundancy and dimensional confounderare the intrinsic issues behind the phenomena, and provide experimental evidenceto support our viewpoint.
We further propose a simple yet effective approach
MetaMask , short for the dimensional Mask learned by Meta -learning , to learnrepresentations against dimensional redundancy and confounder.
MetaMask adoptsthe redundancy-reduction technique to tackle the dimensional redundancy issueand innovatively introduces a dimensional mask to reduce the gradient effects ofspecific dimensions containing the confounder, which is trained by employinga meta-learning paradigm with the objective of improving the performance ofmasked representations on a typical self-supervised task.
We provide solid theoretical analyses to prove MetaMask can obtain tighter risk bounds for downstreamclassification compared to typical contrastive methods.
Empirically, our methodachieves state-of-the-art performance on various benchmarks.
Introduction
A fundamental idea behind self-supervised learning is to learn discriminative representations fromthe input data without relying on human annotations.
Recent advances in visual self-supervisedlearning [1, 2, 3, 4] demonstrate that unsupervised approaches can achieve competitive performanceover supervised approaches by introducing sophisticated self-supervised tasks.
A representativelearning paradigm is contrastive learning [5, 6, 7, 8, 9], which aims to learn invariant information fromdifferent views (generated by data augmentations) by performing instance-level contrast, i e., pullingviews of the same sample together while pushing views of different samples away.
Various[10, 11, 12]techniques have been explored to address the problem of trivial solutions to self-supervised contrastivelearning, e.g., constant representations.
However, state-of-the-art methods still suffer from two crucialdefects, including the interference of task-irrelevant information and sample inefficiency.
The trivialsolution problem is related to such defects, e.g., a small number of negative samples or a largeproportion of task-irrelevant information may lead the model to learn constant representations.
Werevisit the current self-supervised learning paradigm from the perspective of dimensional analysis
[Figure]
e t
[Figure]
and argue that the intrinsic issues behind the defects of self-supervised learning are the dimensionalredundancy and dimensional confounder 1 .
From the perspective of information theory, each dimension holds a subset of the representation’sinformation entropy.
The dimensional redundancy denotes that multiple dimensions hold overlappinginformation entropy.
To tackle this issue, Barlow Twins [14] takes advantage of a neuroscienceapproach, i.e., redundancy-reduction [15], in self-supervised learning.
It makes the cross-correlationmatrix of twin embeddings close to the identity matrix, which is conceptually simple yet effectiveto address the dimensional redundancy issue.
Figure 1 (a) supports the superiority of redundancyreduction in addressing the dimensional redundancy problem.
Since Barlow Twins encouragesthe dimensions of learned representations to model decoupled information, it naturally avoids thecollapsed trivial solution that outputs the same vector for all images.
However, only reducing dimensional redundancy still suffers from the problem of dimensionalconfounder.
The dimensional confounder indicates a set of dimensions that solely contain “harmful”information and further degenerates the performance of the model, e.g., the dimensions simplycapturing chaotic and worthless background information from the input.
To prove the existence ofsuch dimensional confounders, we conduct motivating experiments from two perspectives.
It is statedin [14] that Barlow Twins keeps improving as the dimensionality of the projection head increases,since the backbone network acts as a dimensionality bottleneck to constrain the dimensionality of therepresentation so that the learning paradigm of Barlow Twins can promote the representation capturemore task-irrelevant information with the same dimensionality.
Counterintuitively, our observationfrom the motivating experiments is in stark contrast with the conclusion in [14].
As shown in Figure1 (b), the performance of Barlow twins as a function of the dimensionality presents a convex patternon either CIFAR-10 or ImageNet.
After reaching the peak, the performance gradually decreasesas the dimensionality increases until it collapses.
We reckon that since the task-relevant semanticinformation is limited, as the information captured in different dimensions is gradually decoupled,
[Figure]
ByondBaseline
Baseline
BelowBaseline
[Figure]
I n f e r e n ce acc u r ac y
CIFAR73.
[Figure]
ByondBaseline
Baseline
BelowBaseline
[Figure]
ImageNet
Following the experimental protocol of [14, 12], we evaluate the performance on CIFAR-10 and
ImageNet by adopting KNN prediction and linear probing, respectively, which further proves the general existence of dimensional confounders. more task-irrelevant noisy information is encoded into the representation, i.e., the dimensionalconfounder increases, so the performance of the learned representation is getting worse until collapse.
To further validate our inference, we randomly mask several dimensions by frankly setting the maskedvalues to 0 , and then we evaluate the performance of the masked representation.
The experimentalresults are demonstrated in Figure 2, where every single point denotes the result of a sampled mask.
The representations with some specific masked dimensions can indeed achieve better performancethan the unmasked representation, and such an observation is consistent on a small-scaled dataset,e.g., CIFAR-10, and a large-scale dataset, e.g., ImageNet.
The dimensional mask rate that canimprove the performance of the model on ImageNet is on average lower than that on CIFAR-10.
Weunderstand that this phenomenon depends on an ingredient: the representation learned on CIFAR-10models discriminative information for 10 categories in 512 dimensions, while the representation uses2048 dimensions to model information for 1000 categories on ImageNet, so the dimension-to-classratio on CIFAR-10 ( 51 . 2 ) is much higher than that on ImageNet ( 2 . 048 ).
Therefore, the presenceof dimensional confounders naturally remains at a low level on ImageNet since supporting 1000category classification requires a large amount of heterogeneous discriminative information.
Thisempirical evidence further supports our assumption of the dimensional confounder.
To this end, we intuitively propose a simple yet effective approach MetaMask , short for the Metalearned dimensional Mask , to adjust the impacts of different dimensions in a self-paced manner.
In order to learn representations against dimensional redundancy and confounder, we regard thecross-correlation matrix constraint of Barlow Twins as a dimensional regularization term and furtherintroduce a dimensional mask, which simultaneously enhances the gradient effect of the dimensionscontaining discriminative information and reduces that of the dimensions containing the confounderduring training.
We employ the meta-learning paradigm to train the learnable dimensional mask withthe optimization objective of improving the performance of masked representations on typical selfsupervised tasks, e.g., contrastive learning.
Consequently, the dimensional mask incessantly adjuststhe weights of different dimensions with respect to their gradient contribution to the optimization, so asto make the model focus on the acquisition of discriminative information, which can address the trivialsolution caused by treating all dimensions equally in optimization, especially for high-dimensionalrepresentations.
Empirically, we conduct head-to-head comparisons on various benchmark datasets,which prove the effectiveness of MetaMask.
Concretely, the contributions of this paper are four-fold:
• We present a timely study on the adaptation of the state-of-the-art self-supervised modelson downstream applications and identify two critical issues associated with the learned
representation, i.e., dimensional redundancy and dimensional confounder.• We propose a novel approach to tune the gradient effects of dimensions in a self-paced manner during optimization according to their improvements on a self-supervised task.
• Theoretically, we provide theorems and proofs to support that MetaMask achieves relatively
tighter risk bounds for downstream classification compared to typical contrastive methods.
• We conduct extensive experiments to demonstrate the superiority of MetaMask over bench-
mark methods on downstream tasks.
Related Work
Self-supervised learning achieved impressive success in the field of representation learning without human annotation by constructing auxiliary self-supervised tasks.
Typically, self-supervisedapproaches can be categorized as either generative or discriminative [16, 6, 10].
Conventional generative approaches bridge the input data and latent embedding by building a distribution over them, andthe learned embeddings are treated as image representations [17, 18, 19, 20, 21, 22].
However, sucha learning paradigm learns the distribution at the pixel-level, which is computationally expensive.
The detailed information required for image generation may be redundant for representation learning.
For discriminative methods, contrastive learning achieves state-of-the-art performance on variousdownstream tasks [4, 23, 24, 25, 26, 27, 28, 29, 11].
Deep InfoMax [1] explores to maximize themutual information between the input and output of an encoder.
CPC [3] adopts noise-contrastiveestimation [30] as the contrastive loss to train the model by maximizing the mutual informationof multiple views, which is deduced by using the Kullback-Leibler divergence [31].
CMC [5] and
AMDIM [2] employs contrastive learning on the multi-view data.
SimCLR [6] and MoCo [7, 32]use large batch or memory bank to enlarge the amount of available negative features.
To sampleinformative negatives, DebiasedCL[8] and HardCL [9] design sampling strategies by using positiveunlabeled learning methods [33, 34].
Motivated by [35], [36] provides an information theoreticalframework for self-supervised learning and proposes an information bottleneck-based approach.
However, these methods rely on a large number of negative samples to learn valuable representationswhile avoiding trivial solutions, i.e., constant representation.
SimSiam [12] and BYOL [10] jointlywaive negatives and avoid training degradation by using the techniques of asymmetric siamesenetwork and stop-gradient.
SSL-HSIC [37] provides a unified framework for negative-free contrastivelearning based on Hilbert Schmidt Independence Criterion [38].
The representation learning paradigmemployed by current contrastive methods is contrary to the coding criteria proposed by [15] and[39], respectively.
Therefore, Barlow Twins [14] proposes to constrain the cross-correlation matrixbetween the features to be close to the identity matrix, which can effectively reduce the dimensionalredundancy.
Yet, such an approach still suffers from the dimensional confounder issue, and we thuspropose a new meta-learning-based method to attenuate the over-learning of dimensional confoundersby learning a mask to filter out irrelevant information.
Preliminaries
Formally, we suppose the input multi-view dataset as X = (cid:110) x ji (cid:12)(cid:12)(cid:12) i ∈ (cid:74) 1 , N ∗ (cid:75) , j ∈ (cid:74) 1 , M (cid:75) (cid:111) , where
N ∗ denotes the number of samples, and M denotes the number of views.
Therefore, x ji representsthe j -th view of the i -th sample following x ji ∼ X .
The multiple views are built by performingidentically distributed data augmentations [5, 6, 14], i.e., { t j } Mj =1 ∼ T .
3.1 Contrastive Learning
We recap the preliminaries of contrastive learning [6], which aims to learn an embedding that jointlymaximizes agreement between views of the same sample and separates the views of different samplesin the latent space.
Given a minibatch of N examples sampled from the multi-view dataset X ,positives represent the pairs { x ji , x j ′ i } where j, j ′ ∈ (cid:74) 1 , M (cid:75) , and negatives represent { x ji , x j ′ i ′ } wherei ̸ = i ′ .
Conventional contrastive methods feed the input x ji into a view-shared encoder f θ ( · ) to learnthe representation h ji , which is mapped into a feature z ji by a projection head g ϑ ( · ) , where θ and ϑare the network parameters.
f θ ( · ) and g ϑ ( · ) are trained by using the contrastive loss [3, 6]:
[Equation]
where d ( · ) is a discriminating function to measure the similarity of the feature pairs, 1 [ i ̸
= i ′ ∨ j ̸ = j ′ ]denotes an indicator function equalling to 1 if i ̸
= i ′ or j ̸ = j ′ , and τ is a temperature parameter
[Figure]
valued by following [6].
In inference, the projection head g ϑ ( · ) is discarded, and the representationh ji is directly used for downstream tasks.
3.2 Dimensional Redundant Reduction
Following the principle proposed by [15], [14] introduces a redundancy-reduction objective function:
[Equation]
where λ is a positive hyper-parameter valued by following [14], and C is the cross-correlation matrixcomputed between the outputs of the encoder along the batch dimension:
[Equation]
where k and k ′ index the dimension of features.
C is a square matrix with the size of D , whichdenotes the dimensionality of the projected feature z .
The value range of the elements in C is [ − 1 , 1] ,i.e., complete anti-correlation to complete correlation.
Methodology
Our goal is to learn discriminative representations against dimensional redundancy and dimensionalconfounder from multiple views by performing self-supervised learning.
To this end, we propose
MetaMask to automatically determine the optimal dimensionality of the representation by decouplinginformation into different dimensions and filtering out those dimensions with irrelevant information.
MetaMask addresses the dimensional redundancy issue by adopting a redundancy-reduction regularization.
To tackle the dimensional confounder problem, MetaMask learns a dimensional mask toadjust the gradient weights of dimensions according to their contributions to improving a specificself-supervised task during optimization, which is achieved in a meta-learning manner.
MetaMask’s architecture is shown in Figure 3.
Specifically, the sample x ji is first fed into the encoderto obtain its representation h ji .
Then, we build a learnable dimensional mask M = (cid:8) ω
k (cid:12)(cid:12) k ∈ (cid:74) 1 , D (cid:75) (cid:9) ,which is introduced to assign a weight to each dimension of the representation h ji by
[Equation]
where ˜ h ji denotes the masked representation, and · ⊗ · is an element-wise Hadamard product function.˜
h ji is then fed into two separate projection heads g drr ( · ) and g clϑ ( · ) , respectively.
g clϑ ( · ) plays the samerole as in conventional contrastive learning methods, i.e., the contrastive loss L contrast in Eq.
(1) isapplied to the output features of g clϑ ( · ) .
For the projection head g drr ( · ) , the redundancy-reductionloss L drr in Eq.
(2) is performed to its output features to tackle the dimensional redundancy.
In training, the encoder f θ ( · ) , the projection heads
g drr ( · ) and g clϑ ( · ) , as well as the dimensional mask
M are trained in a meta-learning manner consisting of two steps.
In the first regular training step, wetrain f θ ( · ) , g drr ( · ) and g clϑ ( · ) by jointly minimizing the redundancy-reduction and contrastive losses,which is formalized by
[Equation]
where α is a coefficient that controls the balance between L drr and L contrast , and we evaluate theinfluence of this hyper-parameter in Section 6.
In the second meta-learning-based step, we update M by using the second-derivative technique [40]to solve a bi-level optimization problem.
We encourage M to mask (reduce the gradient weights of)the specific dimensions containing task-irrelevant information, which are regarded as dimensionalconfounders, so that the encoders can better explore the discriminative information in training.
Tothis end, M can be updated by computing its gradients with respect to the performance of f θ ( · ) andg clϑ ( · ) .
The corresponding performance is measured by using the gradients of f θ ( · ) and g clϑ ( · ) duringthe back-propagation of contrastive loss.
Formally, we update the dimensional mask M by
[Equation]
where X ′ represents a minibatch sampled from the training dataset X , ϑ trial and θ trial represent thetrial weights of the encoders and projection heads after one gradient update using the contrastive lossdefined in Equation 1, respectively.
We formulate the updating of such trial weights as follows:
[Equation]
where ℓ θ and ℓ ϑ are learning rates.
The intuition behind such a behavior is that we perform aderivative over the derivative (Hessian matrix) of the combination { θ, ϑ } to update M , i.e., thesecond-derivative trick.
Specifically, we compute the derivative with respect to M by using a retainedcomputational graph of { θ, ϑ } and then update M by back-propagating the derivative as Equation 6.
We train M only based on the performance of contrastive learning for two reasons: 1) the dimensionalredundancy reduction is a regularization term with strong constraints and generally stable convergence,and it is independent of downstream tasks; 2) such a meta-learning-based training approach thatpromotes contrastive learning performance empowers our method to constrain the upper and lowerbounds of the cross-entropy loss on downstream tasks, which ensures that M can partially mask thegradient contributions of dimensions containing task-irrelevant information and further promote theencoder to focus on learning task-relevant information, which is demonstrated in Section 5.
The two steps for updating f θ ( · ) , g drr ( · ) , and g clϑ ( · ) and updating M are iteratively repeated untilconvergence.
The training pipeline is detailed by Algorithm 1.
Theoretical Analyses
Current contrastive methods follow the fundamental assumption of multi-view learning (Assumption1 in [35], Assumption 1 in [36], and Assumption 4.1 in [41]), i.e., views (or distortions) of a same
Input: Multi-view dataset X , minibatch size N , and a hyper-parameter α .
Initialize The neural network parameters: θ for f θ ( · ) , ϑ for g clϑ ( · ) , ϑ drr for g drr ( · ) , and M = (cid:8) ω
k (cid:12)(cid:12) k ∈ (cid:74) 1 , D (cid:75) (cid:9) .
The learning rates: ℓ θ and ℓ ϑ , etc. repeatfor t -th training iteration do
Iteratively sample a minibatch X ′ with N examples from X .
# regular training step, fix M
[Equation]
# compute trial weights and retain computational graph , fix θ and ϑ
[Equation]
# meta training step using second derivative
[Equation]
end for until θ , ϑ , ϑ drr , and M converge.
sample holds the invariant discriminative label-relevant information.
Theorem 4.2 in [41] furtherproposes the guarantees for general encoders in the learning paradigm of regular contrastive learning,which proves that the contrastive loss in the self-supervised learning stage can constrain the upperand lower bounds of the cross-entropy loss in the supervised learning stage for downstream tasks.
Toelaborate the behavior of our MetaMask, we extend this Theorem to build a connection between theproposed masked representation and downstream performance by
Theorem 5.1.
(Connecting Masked Representations to Downstream Cross-Entropy Loss).
Underthe general assumption of self-supervised multi-view learning, when M is optimal, for any coupledh , ˜ h ∈ R , L crossentropy ( h ) for downstream classification can be bounded by L contrast (cid:16) g clϑ
[Equation]
˜ h
(cid:17)(cid:17)in self-supervised learning:
[Equation]
where M denotes the quantity of negative samples, D denotes the dimensionality of the representation,h denotes the unmasked representation, ˜
h denotes the masked representation obtained by Equation4, y denotes the target label, Φ ( ·| y ) =
E P ( y ) (cid:2) E P ( ·| y ) δ (cid:0) · , E P ( ·| y )
(cid:1)(cid:3) represents the conditionalvariance function where δ ( · , · ) is a discriminating function to measure the difference between theterms, e.g., δ ( · , · ) =
∥ [ · ] − [ · ] ∥ 2 in low-dimensional embedding space or δ ( · , · ) = − log (cos ( · , · ))
=− log [ · ] × [ · ] ||·||×||·|| in high-dimensional embedding space, ⌊·⌉ k is a function acquiring k -th dimensionvector, log (cid:0) MD (cid:1) is a constant, and O (cid:16) M − 12 (cid:17) is the approximation error’s order.
Theorem 5.1 states that reducing the risk of contrastive learning can improve the performance ondownstream tasks, which further supports our intuition to make the model focus on the acquisitionof discriminative information by learning a dimensional mask with the objective of improving theperformance of masked representations on contrastive learning is theoretically sound.
Refer to
Appendix A.1 for the elaboration of MetaMask’s behavior to mask the gradient contribution ofspecific dimensions containing confounder information.
Hence, we derive
Theorem 5.2.
(Guarantees for Reduced Conditional Variance of Masked Representations).
When
M is optimal, for any coupled h , ˜ h ∈ R , given label y , the conditional variance of ˜ h is reduced:
[Equation]
Theorem 5.2 states that given the label y , the masked representation ˜ h has smaller conditional variancethan h in contrastive learning.
We bring Theorem 5.2 into Theorem 5.1 to derive a conclusion:
our
[Table]
approach can better bound the downstream classification risk, i
[Table]
e., the upper and lower bounds ofsupervised cross-entropy loss obtained by MetaMask are tighter than typical contrastive learningmethods.
Please refer to Appendix A.2 for the corresponding proofs.
Experiments
6.1 Benchmarking MetaMask with Various Backbones
Implementations.
For the comparisons demonstrated in Table 1, we uniformly set the batch size as
[Table]
and we adopt a network with the 5 convolutional layers in AlexNet [44] as conv and a network with2 additional fully connected layers as fc .
We straightforwardly adopt the same data augmentations in[5] and [45], and the memory bank [46] is adopted to facilitate calculations by retrieving 4,096 pastnegative features during training.
Following the experimental principle of linear probing, we removethe projection head and re-train an MLP as the classification head for the learned representation.
For the experiments in Table 2, the batch size is valued by 512, and ResNet-18 [47] is used as thebackbone encoder.
We adopt the data augmentation and other experimental settings following [14].
Note that the KNN classifier is adopted for evaluation.
Discussion of the comparisons using conv and fc encoders.
Table 1 shows the comparisons on fourbenchmark datasets, and the corresponding positive or negative margins are reported.
We observethat MetaMask beats the best prior methods by a significant margin on most downstream tasks, whichdemonstrates that MetaMask can efficiently improve the performance of the learned representationwhen the self-supervision is insufficient, i.e., the limited batch size and weak encoder.
Discussion of the comparisons using the ResNet encoder.
We further perform classificationcomparisons using a strong encoder, i.e., ResNet.
Table 2 shows that MetaMask can consistentlyimprove the state-of-the-art methods, which indicates that the proposed method has strong adaptability to different encoders.
The experimental evidence proves our assumption of the dimensionalconfounder.
Thus, given a fixed dimensionality of the projection head, MetaMask can indeed improvethe discriminativeness of the learned representation by masking the dimensional confounder.
6.2 Validation of the Robustness of MetaMask against Dimensional Confounder
We conduct experiments to prove that MetaMask is able to mitigate the negative impact of thedimensional confounder on the discriminativeness of the learned representations, and the results areshown in Figure 4.
The intriguing outcome demonstrates two merits of the proposed method: 1)
[Table]
[Figure]
ResNet18.
∗ denotes MetaMask using a trick of fixed learning rate instead of the cosine annealing strategy.
even if the dimensionality of the projection head changes, MetaMask is consistently robust to thedimensional confounder, i.e., the model maintains relatively consistent performance; 2) MetaMaskcan alleviate the negative impact of curse of dimensionality [48, 49], e.g., given a fixed number ofexamples, as the dimensionality of the representation increases during training, the performance of themodel on downstream tasks instead degenerates.
Concretely, Figure 2 demonstrates that regardless ofwhether representation learning suffers from the curse of dimensionality, dimensionality confoundingfactors are widely present in learned representations, and hence, the defined dimensional confounderis a more generalized issue of the curse of dimensionality.
Refer Appendix A.3 for the furtherdiscussion.
Moreover, we observe a surprising result that MetaMask ∗ achieves better performance.
We provide an understandable explanation: the essence of the curse of dimensionality is that anetwork with a large number of parameters is prone to over-fitting to the training data.
Therefore,compared with the annealing strategy, a fixed and relatively large learning rate can prevent the modelfrom falling into a local optimum (over-fitting) during optimization.
Our further experiments showthat MetaMask ∗ collapses abruptly at larger dimensional settings, e.g., 16384, which demonstratesthat the trick of using a fixed learning rate can only unsteadily improve the performance of MetaMask,but this behavior is not essential.
Figure 4 shows that under the different experimental settings, vanilla
MetaMask consistently counteracts the negative effects of the dimensional confounder.
Conclusion and Future Discussion
From the dimensional perspective, we first clarify that the dimensional redundancy and confounderissues are the essence of the current self-supervised learning paradigm’s defects.
Multiple motivatingexperiments are conducted to support our viewpoint, and the intriguing results are opposite tothe traditional conclusions.
To jointly tackle the dimensional redundancy and confounder issues,we propose a heuristic learning approach, called MetaMask, which leverages a second-derivativetechnique to learn a dimensional mask in order to adjust the gradient weights of different dimensionsduring training.
The proposed theoretical analyses demonstrate that MetaMask obtains relativelytighter risk bounds for downstream classification compared to conventional contrastive methods.
From the experiments, we observe a consistent improvement in the performance of our method overstate-of-the-art methods on benchmark datasets.
One of our critical contributions is proposing a heuristic learning paradigm for self-supervised learning: regarding the dimensional redundancy reduction as a regularization term and further exploringto mask dimensions containing the confounder.
In the current methodology, we train the learnabledimensional mask with the objective of improving the performance of masked representations ona specific self-supervised task , i.e., contrastive learning, and provide the theoretical analysis toprove that such a learning approach can indeed prompt the model focus on acquiring discriminativeinformation.
However, contrastive learning is immutable for the self-supervised task in MetaMask.
It is commonly acknowledged that masked image modeling (MIM) methods [51, 52, 53] achieveimpressive performance in self-supervised learning.
Although the theoretical contribution bridgingthe gap between MIM loss and cross-entropy loss on downstream tasks has not yet emerged, we stilldeem combining MIM approaches with our proposed paradigm is attractive to the community.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
c) Did you discuss any potential negative societal impacts of your
work? [No] Since
this work presents a general technique to tackle the dimensional confounder for self-
supervised learning , we did not see any particular foreseeable ethical consequence.
d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 5.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. and Appendix A.2.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplementary files.
(b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?
[Yes] See Section 6, Appendix A.4, and the supplementary files.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] The experimental results provided by benchmark methods
do not contain such error bars.
We also follow this principle, and we reckon the reason
behind such a behavior is that the comparisons are conducted in balanced datasets, so
the difference between results is not excessively large.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 1 and Appendix A.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 1, Section 2, Section 5, and Section 6.
(b) Did you mention the license of the assets?
[No] The assets we adopted are totally
established and public to the community.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the supplementary files. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [No] The datasets are all public to the community.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] The datasets have been discussed, validated, and used by many researchers.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A] A AppendixA.1 Elaborating MetaMask’s Behavior of Masking Gradient Contribution
For better understanding the intuition behind the behavior of the proposed MetaMask, we elaboratethe procedure of our method masking gradient contributions of dimensions.
We restate Equation 1 ofcontrastive loss by converting the original projected features z into the masked feature ˜ z :
[Equation]
which is derived by considering Equation 4.
To simplify the equation, we hold
[Equation]
[Equation]
Therefore, the indicator function 1 [ i ̸
= i ′ ∨ j ̸ = j ′ ] can be substituted as follows:
[Equation]
where g ( · ) denotes g clϑ ( · ) [Equation] and h − is neither equal to h nor equal to h + [Equation]
Proposition A.1.
The mask M exists on each partial differential equation of the gradient of L i,j,j + : ∇ h L i,j
[Equation]
j + =
[Equation]
[Equation]
where (cid:8) w k (cid:12)(cid:12) k ∈ (cid:74) 1 , N g (cid:75) (cid:9) denotes the weights of the projection head g ( · ) , N g denotes the numberof corresponding projection layers, and h −′ is neither equal to h nor equal to h + (note that h − andh −′ are two independent variables).
M is the dimensional mask.
Proposition A.1 states that the dimensional mask M consistently exists on each partial differentialequation of the gradient of the contrastive loss L i,j,j + .
We further observe that there exists 7types of irreducible terms containing M : g ( h ⊗ M ) , g ( h + ⊗ M ) , g ( h − ⊗ M ) , g (cid:16) h −′ ⊗ M (cid:17) ,∇ h h ⊗ M , ∇ h + h ⊗ M , ∇ h − h ⊗ M , where the first 4 types are in the regular computing mannerso that M can directly change the values of different dimensions by assigning dimension-specificweights, and the last 3 types are the first-order derivatives.
Note that the scalar derivation, e.g., theloss, of a matrix can be thought of as derivation of each element of the matrix, and such a derivationis derived and then putting the element-wise results in the order of the matrix to get a gradient matrixof the same dimension.
Following such a law, we utilize Chain Rules to derive the derivative of the loss L i,j,j + with respectto each element in the batch matrix containing h , h + , h − and further obtain the derivative matrix:∂ L i,j
[Equation]
j + [Equation] p
[Equation]
q (cid:21)
[Equation]
 [Equation] D − 1 ∂ x − N −ω D
∂
⌊ f θ ( x − N − 2 ) ⌉ D ∂ x − N −  (16)
where B = (cid:2) h , h + , h − 1 , h − 2 , . . , h − N − 3 , h − N − 2 (cid:3) is a batch of features with N examples, and X ′ isthe corresponding input data matrix. h , h + , and h − are feature vectors containing D dimensions.ˆ M = [ M , M , . . .
, M , M ] is an extended matrix containing N copies of M .
Intriguingly, weobserve that M generally assign the dimension-specific weights for each samples, which empowers
MetaMask to naturally reduce the gradient effect of specific dimensions containing the confounder byemploying the meta-learning paradigm with the objective of improving the performance of maskedrepresentations on a typical self-supervised task.
After clarifying the dimension-specific weightingmechanism from the perspective of gradient analysis, we further demonstrate the effectiveness of
MetaMask’s training paradigm as follows.
A.2 Proofs
In order to prove Theorem 5.2 and the conclusion that the bounds of supervised cross-entropy lossobtained by MetaMask are tighter than typical contrastive learning methods, we provide separatedproofs for the corresponding two parts of Theorem 5.2 and the tighter bounds on downstreamcross-entropy loss.
A.2.1 Proof for the Equality Part
To prove
[Equation]
(17) we observe that the only variable that differs on both sides of the Equation 17 is the representation,which is masked on the left hand side of the equation while unmasked on the right hand side of the equation.
Then, we substitute the definition of Φ ( · ) into Equation 17 and derive
[Equation]
In our approach
[Equation] the learned representation generally exists in high-dimensional embedding space,e [Equation] g., 1024, 2048, 8192, etc.
Therefore, in high-dimensional embedding space, the left hand side ofthe equation can be extended to
[Equation]
19) which is derived by following the Bayes Rule, the definition of δ ( · , · ) in Theorem 5.1, and the factthat g clϑ ( · ) is an MLP without amounts of deep layers so that the embedding space shift is limited.
Note that µ y denotes the center of categories’ feature or the corresponding classification weight of theclassifier for a specific category, and y = (cid:8) y l (cid:12)(cid:12) l ∈ (cid:74) 1 , N L (cid:75) (cid:9) , where N L is the number of categories.
Then, we bring Equation 19 into Equation 18 and get
[Equation]
To prove Equation 20, we demonstrate an evidence example in Figure 5.
As shown in the subfigure(a), the cosine error risk of the target vector µ y and the candidate vector, e.g., ⌊ h ⌉ k , ⌊ ˜ h
⌉ k , etc, isinsensitive to the specific value of vectors, i.e., the cosine error risk keeps generally consistent as thevalues of vectors fall or vice versa.
Such an assumption is supported by comparing plot (1 with plot(2 or comparing plot 3) with plot 4).
Therefore, we derive
[Equation]
so that
[Equation]
[Equation]
A.2.2 Proof for
the Inequality Part
To prove
[Equation]
[Equation]
[Figure]
[Figure]
we follow the formula transformation of Equation 19 and Equation 20 to derive
[Equation]
As shown in the subfigure (a) of Figure 5, all of ⌊ h ⌉ k , ⌊ ˜ h
⌉ k , ⌊ h
⌉ k +1 , and ⌊ ˜ h ⌉ k +1 denote a specificdimension of the corresponding representations. Then, we reconstruct the complete representationsfrom the dimensions, which is achieved in the subfigure (b) of Figure 5.
The combinations of (2,3) and (1, 4) are hypothetical and fake, the combination (1, 3) denotes the original and unmaskedrepresentation h , and (2, 4) denotes the masked representation ˜ h .
We observe that although thespecific value of the vector of each dimension has a thin effect on the corresponding cosine errorrisk, the combined representation vector is largely sensitive to such values.
As demonstrated in(1, 3) and (2, 4) in the subfigure (b), the cosine error risk of ˜ h and µ y is apparently less than hand µ y , which supports that, given a target y , the masked representation can indeed derive smallerconditional variance than the unmasked representation.
The reason behind such a phenomenonis that, following Theorem 5.1, the self-paced dimensional mask jointly enhances the gradienteffect of the dimensions containing discriminative information and reduces that of the dimensionscontaining the confounder by adjusting the weights of different dimensions with respect to theirgradient contribution to the optimization of a specific self-supervised task, e.g., contrastive learning, during training.
Therefore, the dimensions containing discriminative information have relatively largeweights, while the dimensions containing confounder information are partially masked .
A.2.3 Proof for the Tighter Bounds
Being aware of proofs in Section A.2.1 and Section A.2.2, we confirm the validation of Theorem
Then, we bring Theorem 5.2 into Theorem 5.1 to derive the comparison of the lower bounds ofsupervised cross-entropy loss that are separately obtained by the masked representation ˜ h and theunmasked representation h :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Therefore, the lower bound obtained by the masked representation, i.e., MetaMask, is larger than theunmasked representation, i.e., typical contrastive learning methods.
We further analyze the upperbound of supervised cross-entropy loss on downstream tasks and derive
[Equation]
where the upper bound obtained by MetaMask is smaller than the typical contrastive learning methods.
Concretely, we conclude that our approach can better bound the downstream classification risk, i.e.,the upper and lower bounds of supervised cross-entropy loss obtained by MetaMask are tighter thantypical contrastive learning methods.
A.3 Discussion on the Difference between Dimensional Confounder and Curse ofDimensionality
The restricted definition of the curse of dimensionality is that given a fixed number of examples,as the dimensionality of the representation increases during training, the performance of the modelon downstream tasks instead degenerates, e.g., the Hughes phenomenon .
Recent researchers statea generalized definition of the curse of dimensionality from the perspective of over-fitting: given afixed number of examples, the performance of the model degrades as the amount of computationin the network increases, e.g., the increase of representations’ dimensionality or the increase of thelayers of a deep network.
Concretely, the curse of dimensionality is a theory to explain a specificphenomenon of over-fitting from the perspective of sample size.
However, our dimensional confounder is defined as a negative factor that may lead to model degradation, which is proposed from the dimensional perspective.
The motivating example in Figure 2demonstrates that even if the deep network architecture and the dimension of the representation areconsistent, the dimensional confounder still causes the performance of the model to degenerate.
Thisexperimental analysis proves that the proposed dimensional confounder and the curse of dimensionality are two different terminologies.
Further, to conduct the experiments in Figure 1, we fix thedimensionality of the representation and only increase the dimensionality of the features generatedby the projection head, which is under the restriction of the redundancy-reduction technique [14].
The intuition behind such a behavior is that as the dimensions of the projected features increase, theinformation entropy contained in each dimension of the representation becomes more decoupled, andthe total amount of task-relevant information entropy is limited, such that the dimensional confounder(i.e.
, the dimensions only containing task-irrelevant information) is gradually increasing.
The experimental results demonstrate that as the dimensional confounder in the learned representation increases,the performance of the model is weakened.
Such a phenomenon is consistent with the generalizeddefinition of the curse of dimensionality, since the dimensionality of the representation is constantwhile the parameter (computation) of the deep network is increasing.
Therefore, we conclude thatthe dimensional confounder can lead to the curse of dimensionality, and compared with the curse
[Figure] [Figure] of dimensionality, the proposed dimensional confounder is a generalized issue. The comparisons in
Table 1 and Table 2 prove that our MetaMask can alleviate the dimensional confounder issue, and theexperiments in Figure 4 further support the superiority of MetaMask over current self-supervisedmethods to alleviate the curse of dimensionality issue.
[Figure]
A.4 Extended Experiments
[Figure]
We conduct several experiments to study the intrinsic property of the proposed MetaMask.
[Figure]
A.4.1 The Impact of the Hyper-Parameter α
As demonstrated in Figure 6, we conduct comparisons of MetaMask using ResNet-18 on CIFAR-10.
We follow the experimental principle in Section 6 and use KNN prediction as the evaluation approach.
The comparison results show that an elaborate assignment of α can indeed improve the performanceof MetaMask, and when α = 100 , the accuracy of MetaMask achieves the peak value.
The reasonsbehind this curve are as follows: 1) L drr is naturally excessively larger than L contrast so that whenα is not large enough, L contrast ’s impact with respect to the gradient is diminished; 2) when αis over-large, the impact of L drr is weakened, and the dimensional redundancy issue cannot besufficiently addressed so that the representation may be collapsed to a trivial constant.
A.4.2 Further Exploration of the Robustness of MetaMask The experimental analysis demonstrated in Section 6.2 proves that MetaMask can indeed alleviatethe negative impact caused by the dimensional confounder. However, MetaMask ∗ achieves betterperformance than compared methods, including vanilla MetaMask. We provide a further explorationshown in Figure 7, and we observe that MetaMask ∗ collapses abruptly at larger dimensional settings,e.g., 16384, which demonstrates our explanation of such a phenomenon in Section 6.2, i.e., the fixedlearning rate trick can temporarily improve the performance of MetaMask, but this improvementis inconsistent.
The results in Figure 7 further show that with or without this trick, the proposed
MetaMask is robust against the dimensional confounder.
[Figure]
Note that for fair comparisons, this experiment is based on
[Table]
A.4.3 Impacts of Encoder’s Ability towards MetaMask
From the experimental results reported in Section 6, we observe that MetaMask improves the benchmark methods with weak encoders by excessively significant margins, but the improvement withstrong encoders is relatively limited.
Our consideration behind this phenomenon is that although representations learned by both weak and strong encoders (without our method) may contain dimensionalconfounders, the strong encoder can better capture semantic information so that the dimensionalconfounder of the learned representation is naturally less, and the useful discriminative informationis much more than the representation learned by the weak encoder.
[41] provide the theorem andcorresponding proof to demonstrate that the contrastive loss can bound the cross-entropy loss indownstream tasks.
Strong encoders better minimize the contrastive loss, and thus the representationslearned by strong encoders contain more semantic information, and accordingly, fewer dimensionalconfounders.
However, from the results reported in Table 2, our method can still improve thebenchmark methods.
We further conduct experiments by imposing random dimensional masks on the learned representations for weak and strong encoders.
The results are reported in Figure 8.
The comparisonsdemonstrate that under the consistent setting of 5% random dimensional mask rate, the results of theweak encoder (conv) range from 52.79 to 54.13, and the results of the strong encoder (ResNet-18)range from 60.71 to 61.06.
Note that we conduct 10 trials for each experiment to achieve unbiasedresults.
The quality of the strong encoder’s representation is apparently better, and the performanceof the strong encoder is more stable (the variance of the strong encoder’s results is smaller), whichproves that the representation learned by the strong encoder contains fewer dimensional confounders,and each dimension contains more discriminative information.
Therefore, our method improves weakencoders more than strong encoders.
A.4.4 Training Complexity of MetaMask
To compare the training complexity of MetaMask and benchmark methods, we conduct experimentson CIFAR10 by using ResNet-18 as the backbone network.
The results are demonstrated in Table3, which shows that the parameter number used by MetaMask is close to the ablation model, i.e.,
SimCLR + Barlow Twins, and Barlow Twins.
Compared with Barlow Twins, SimCLR + Barlow
Twins only adds a MLP with several layers as the projection head for the contrasting learning of
SimCLR.
For MetaMask, we only add a learnable dimensional mask M in the network of MetaMask.
Note that to decrease the time complexity of MetaMask, we create an additional parameter space tosave the temporary parameters for computing second-derivatives, but such cloned parameters are notincluded in the computation of the network during training.
[Table]
[Table]
For the time complexity, due to the learning paradigm of meta-learning (second-derivatives technique),
MetaMask has larger time complexity than benchmark methods (including the ablation model
SimCLR + Barlow Twins).
We further conduct experiments to evaluate the performance of thecompared methods using similar total training time costs.
The results are reported in Table 4, whichdemonstrates that MetaMask still achieves the best performance, and MetaMask can also beat theablation model SimCLR + Barlow Twins.
This proves that although the time complexity of MetaMaskis a little bit high, the improvement of MetaMask is consistent and solid.
We also include these resultsand the corresponding analysis in Appendix of the rebuttal revised version.
A.4.5 Exploration of Barlow Twins using the Dropout Trick
We apply dropout to randomly set the features generated by the backbone network to 0 with a givenprobability.
We use two different settings: 1) features from different views are randomly set to 0independently with 6 probabilities, including 0.01, 0.05, 0.1, 0.2, and 0.5, which is shown as thesetting of dropout shared between views is “No”; 2) for the same sample, we set the same channels ofthe features from different views to 0.
In detail, we let the features from the first view pass the dropoutlayer, and then record the channels which are set to 0.
Finally, we set the same feature channels of thesecond view to 0 and multiply the features with 1/(1-p), where “p” refers to the probability of dropout.
In this case, we use three probabilities: 0.05, 0.1, and 0.5.
This setting is shown as “Yes” for thedropout shared between views.
The results are reported in Table 5.
We observe that models trainedby following the second setting are collapsed, and these models far underperform MetaMask.
For thefirst setting, the performance trend is in accordance with our expectation, using several well-chosendropout ratios can improve the performance of Barlow Twins to a certain extent, but the improvementis inconsistent.
Furthermore, even the best model of Barlow Twins using the dropout trick cannotachieve comparable performance to MetaMask, since as shown in Table 2, MetaMask achieves 87.53on CIFAR10 with ResNet-18.
A.4.6 Performing MetaMask on ImageNet with ResNet-50
Training one trial of MetaMask on ImageNet is excessively time-consuming for the adopted server.
Itis hard to impose sufficient hyperparameter tuning experiments, because tuning parameters on thevalidation set and then retraining is time-consuming.
For the current version, we principally adopt thehyperparameter of MetaMask on ImageNet-200 with ResNet-18 for the experiments on ImageNetwith ResNet-50.
We report the results on Table 6, which demonstrates that MetaMask achieves the
[Table]
[Figure]
top-3 best performance.
Note that the major results are reported by [14], and the implementationof MetaMask is based on SimCLR and Barlow Twins.
For comparisons with the main baselines,
MetaMask beats SimCLR by 4.6 on top-1 accuracy and 2.4 on top-5 accuracy, and MetaMask beats
Barlow Twins by 0.7 on top-1 accuracy and 0.4 on top-5 accuracy.
The improvement of MetaMask isin accordance with our observation in Appendix A.4.3.
Furthermore, ImageNet-200 is a truncateddataset of ImageNet, so the domain shift between these datasets is slight.
The comparisons in Table1, and Table 2 demonstrate that the proposed MetaMask can still improve the benchmark methods inlarge-scale datasets.
A.4.7 Exploration of Confounders Contained by Masked Dimensions
MetaMask trains M by adopting a meta-learning-based training approach, which ensures that Mcan partially mask the “gradient contributions” of dimensions containing task-irrelevant informationand further promote the encoder to focus on learning task-relevant information.
So, MetaMask onlyperforms the gradient mask during training instead physically masking dimensions in the test.
Weprovide theoretical explanation and proofs in Appendix A.1 and Appendix A.2.
The reason behindour behavior is that even dimensions that contain dimensional confounders are also possible to containdiscriminative information so that lowering the gradient contribution of such dimensions can notonly prevent the over-interference of the dimensional confounders on the representation learning butalso preserve the acquisition of the information of these dimensions.
Accordingly, the foundationalidea behind self-supervised learning is to learn “general” representation that can be generalizedto various tasks.
MetaMask introduces a meta-learning-based approach to train the dimensionalmask M with respect to improving the performance of contrastive learning.
However, the theorems, proposed by [41], only prove that the contrastive learning objective is associated with the downstreamclassification task, while there is no theoretical evidence to demonstrate the connection betweencontrastive learning objective and other downstream tasks.
Therefore, we consider not directlymasking the dimensions containing dimensional confounders in the test.
Furthermore, we conduct experiments to explore the performance of the variant that directly masksthese dimensions in the test, which is demonstrated in Figure 9.
For the exploration of our maskingscheme and its variants, we conduct experiments as follows: after training, we collect the finaldimensional weight matrix M and then choose dimensions with weights below average as themasked dimensions.
These dimensions are considered to be associated with dimensional confounders.
To prove whether these dimensions have confounders, we perform random dimensional masking tothese dimensions, and when the masking rate is 100%, the model turns to the variant that directlymasks all these dimensions in the test.
The experiments are based on SimCLR + MetaMask.
Notethat we conduct 10 trials per mask rate (except for 0% and 100% mask rates) for fair comparisons.
We observe that the original MetaMask (i.e., mask rate is 0%) achieves the best performance onaverage, and MetaMask outperforms the variant masking the dimensions with confounders by asignificant margin, which proves that our proposed approach, i.e., masking the “gradient contributions”of dimensions in the training is more effective than the compared approach, i.e., directly maskingdimensions in the test.
While, several trials with specific mask rates demonstrate better performancethan MetaMask, which proves that the dimensions filtered by MetaMask indeed contain dimensionalconfounders.
Additionally, observing the results reported in Figure 2 and the results in Figure9, we find that the results achieved by the proposed variants are better than Barlow Twins withrandom dimensional masks on average, which can further prove the filtered dimensions containingconfounders and that MetaMask indeed assigns lower gradient weights to the dimensions containingconfounders.
Additionally, we conduct experiments to directly impose a linear probe on the whole maskedor unmasked features.
Note that the definition of masked features is mentioned above, and theexperiments are conducted on CIFAR10 with ResNet-18.
As shown in Figure 9, a linear probeis solely imposed on the whole “unmasked” features (without “masked” features), which is thesame as the 100% mask rate variant for the “masked” features above, and the result is 79.09.
Weprovide the corresponding reasons and analyses above.
Additionally, such results may be due tothe masking scheme for these experiments, i.e., collecting the final dimensional weight matrix Mand then masking the dimensions with weights below average, which is dramatically different from
MetaMask’s behavior, and this scheme is only used for this exploration.
The result achieved by a linearprobe on the whole “masked” features is 56.30, which demonstrates our statement that dimensionscontaining dimensional confounders are also possible to contain discriminative information, becausethe achieved accuracy is not under 10.
The result also proves that MetaMask can indeed assign lowergradient weights to the dimensions containing confounders, since such a model far underperforms
MetaMask (86.01).
A.4.8 Further Ablation Study of MetaMask
Barlow Twins handles dimensional redundancy but suffers dimensional confounders.
MetaMaskmitigates dimensional confounders by learning and applying a dimensional mask.
We conduct experiments to demonstrate our statement, and the results are reported in Figure 10.
For the experimentsshown in Figure 10 (a), we demonstrate the effectiveness of the proposed dimensional mask M andthe corresponding meta-learning-based training paradigm by directly removing such approaches.
The results show that the sole w/o ML only improves Barlow Twins by 0.04, while MetaMask canimprove BT by 0.29, which proves the proposed M is pivotal to the performance promotion.
In
Figure 10 (b), to verify whether there would be performance gain only from alleviating dimensionalconfounders without L drr , we evaluate the performance of w/o drr and MetaMask.
We observethat for the experiments with ResNet-18, w/o drr (without Barlow Twins) improves SimCLR by0.2 but cannot reach the performance of BT, and MetaMask can improve both SimCLR and BT by4.28 and 0.29, respectively.
For the experiments with conv, w/o drr improves SimCLR by 1.03 andalso outperforms BT by 4.08, and MetaMask improves both SimCLR and BT by 2.27 and 6.22,respectively.
Concretely, the performance of w/o drr is related to the performance of SimCLR, and itcan always improve SimCLR but underperform the complete MetaMask.
MetaMask has consistentbest performance by using different encoders.
[Figure]
ResNetconv
BT SimCLR w/o drr MetaMask
[Figure]
experiments
We consider that L drr (Barlow Twins loss) could exacerbate dimensional confounders, but as ourdiscussion in Appendix A.4.7, i.e., dimensions containing confounders are also possible to containdiscriminative information, more dimensions with confounders due to L drr may also carry morediscriminative information.
Likewise, the model without L drr may generate representations with overredundant dimensions so that the total amount of available discriminative information will decrease.
However, roughly using the representations with complex dimensional information (without L drr )may result in insufficient discriminative information mining, e.g., w/o ML can only outperform
SimCLR by a limited margin.
Our proposed MetaMask effectively avoids the appearance of suchan undesired phenomenon by leveraging M and the corresponding meta-learning-based trainingparadigm, which is empirical