Adversarial Auto-Augment with Label Preservation: A Representation Learning Principle Guided Approach
Anonymous Author(s)
AffiliationAddress email
Abstract
Data augmentation is a critical contributing factor to the success of deep learningbut heavily relies on prior domain knowledge which is not always available.
Recentworks on automatic data augmentation learn a policy to form a sequence of augmentation operations, which are still pre-defined and restricted to limited options.
In this paper, we show that a prior-free autonomous data augmentation’s objectivecan be derived from a representation learning principle that aims to preserve theminimum sufficient information of the labels.
Given an example, the objectiveaims at creating a distant “hard positive example” as the augmentation, while stillpreserving the original label.
We then propose a practical surrogate to the objectivethat can be optimized efficiently and integrated seamlessly into existing methodsfor a broad class of machine learning tasks, e.g., supervised, semi-supervised,and noisy-label learning.
Unlike previous works, our method does not requiretraining an extra generative model but instead leverages the intermediate layerrepresentations of the end-task model for generating data augmentations.
In experiments, we show that our method consistently brings non-trivial improvements tothe three aforementioned learning tasks from both efficiency and final performance,either or not combined with pre-defined augmentations, e.g., on medical imageswhen domain knowledge is unavailable and the existing augmentation techniquesperform poorly.
Code will be released publicly.
Introduction
Data augmentation has emerged as an effective data pre-processing or data transformation step tomitigate overfitting [30], to encourage local smoothness [53] and to improve generalization [6] inmachine learning pipelines such as deep neural networks.
Notably, effective data augmentation,which incorporates class-related data invariance and enriches the in-class sample, is one of the keycontributing factors for representation learning with weak or self supervision [9, 22].
Given a task, we aim to generate “good” augmentations efficiently.
As part of the machine learningmodel pipeline, an autonomous domain-agnostic but task-informed data augmentation mechanism isdesirable.
However, a number of challenges exist.
(1) Existing augmentation operators are usuallyhand-crafted based on domain expert knowledge, which is not always available in some domain [47].
For example, widely used augmentations for natural images are not effective on medical images.
Moreover, the performance of those machine learning pipelines drastically varies with differentchoices of data augmentations.
Existing few autonomous augmentation approaches developedlately are neither fully autonomous nor universally applicable to varying domains.
Although afew autonomous data augmentation approaches have been developed in recent years [14, 12], theytrain policies to produce a sequence of pre-defined augmentation operations and thus are not fullyautomated and are limited to a few domains.
(3) Existing augmentations usually do not fully utilize the task feedback (i.e., task-agnostic) and may be sub-optimal for the targeted task.
A class of automateddata augmentation methods train an extra data generative model to generate new augmentations fromscratch given a real-world example [2].
However, they require training a generative model, which is anon-trivial task in practice that may either relies on strong prior knowledge or a substantial increasednumber of training examples.
In this paper, we first investigate the conditions required to generate domain-agnostic but taskinformed data augmentations.
Consider a representation learning pipeline, we started from a probabilistic graphical model that describes the relations among the label Y , the nuisance N , the example
X , its augmentation X ′ , and the latent representations Z .
We argue that a minimum-sufficient representation for the task preserves the label information but excludes other distractive information fromthe nuisance.
We then investigate the conditions for an augmentation X ′ that results in learning suchpreferred representations.
These conditions motivate an optimization objective that can be used toproduce automated domain-agnostic but task-informed data augmentations for each example, withoutreplying on pre-defined augmentation operators or specific domain knowledge.
Consequently, ourproposed optimization objective addresses all aforementioned challenges.
For practicality, we further propose a surrogate of the derived objective that can be efficientlycomputed from the intermediate-layer representations of the model-in-training.
The surrogate is builtupon the data likelihood estimation through perceptual distance [23] defined on the intermediatelayers’ representations.
Specifically, our proposed surrogate objective maximizes the perceptualdistance between X and X ′ , under a label preserving constraint on the model prediction of X ′ .
Thisproblem can be efficiently solved by optimizing its Lagaragian relaxation.
Thereby, given X and itslabel Y , the solution to our surrogate objective generates “hard positive examples” for X withoutloosing its label information.
Once generated, X ′ is used to train the model towards producing theminimum-sufficient representation Z for the targeted task.
Our proposed method, named LabelPreserving Adversarial Auto-Augment ( LP-A3 ) , does not require any extra generative models suchas Generative Adversarial Networks, unlike previous automated augmentation methods [39].
Wefurther propose a sharpness-aware criterion selecting only the most informative examples to applyour auto-augmentation on so it does not cause expensive extra computation.
Our proposed LP-A3 is a general and autonomous data augmentation technique applicable to a varietyof machine learning tasks, such as supervised, semi-supervised and noisy-label learning.
Moreover,we demonstrate that it can be seamlessly integrated with existing algorithms for these tasks andconsistently improve their performance.
In experiments on the three learning tasks, we equip LP-A3with existing methods and obtain significant improvement on both the learning efficiency and thefinal performance.
The generated augmentations are optimized for the model-in-training in a targettask-aware manner and thus notably accelerate the slow convergence in computationally intensivetasks such as semi-supervised learning.
It is worth noting that our augmentation can consistentlybring improvement to tasks without domain knowledge or pre-defined augmentations such as medicalimage classification, on which the traditional image augmentations lead to performance degeneration.
Related work
Hand-crafted vs. Autonomous Data Augmentations.
Most of the existing widely used dataaugmentations are hand-crafted based on domain expert knowledge [29, 38, 28, 9, 13].
For example,
MoCo [16] and InstDis [46] create augmentations by applying a stochastic but pre-defined data augmentation function to the input.
CMC [38] splits images across color channels.
PIRL [28] generatesdata augmentations through random JigSaw Shuffling.
CPC [29] renders strong data augmentationsby utilizing RandAugment [12], which learns a policy producing a sequence of pre-defined augmentation operations selected from a pool [14].
AdvAA [56] designs a adversarial objective to learn theaugmentation policy.
[12, 14, 56] are all based on pre-defined operations which is not available incertain domains, and their objective cannot guarantee the label-preserving of the generated data whichmay lead to suboptimal performance.
“InfoMin” principle of data augmentation is proposed [39] tominimize the mutual information between different views (equivalent to min I ( X , X ′ ) ).
However,their theory depends on access to a minimal sufficient encoder which may be difficult to obtain.
Incontrast, we not only consider how to generate optimal views or augmentations, but also considergenerating the minimal sufficient representation.
The algorithm [39] deploys a generator to renderaugmentation (which may be costly to train especially on non-natural image domains), while wedirectly learn the augmentation through gradient descent w.r.t.
the input.
Information Theory for Representation Learning.
Information theory is introduced in deep learning to measure the quality of representations [41, 1].
The key idea is to use information bottleneckmethods [40, 41] to encourage the learned representation being minimal sufficient.
Mutual information objectives are commonly used in self-supervised learning.
For example, InfoMax principle [26]used by many works aims to maximize the mutual information between the representation and theinput [38, 4, 45].
But simply maximizing the mutual information does not always lead to a betterrepresentation in practice [42].
In contrast, InfoMin principle [39] minimizes the mutual informationbetween different views.
Both InfoMax and the InfoMin principles can be associated with ourproposed representation learning criteria in Section 4, as they lead to sufficiency and minimality ofthe learned representation, respectively.
Augmentation in Self-supervised Contrastive Learning.
Self-supervised Contrastive Representation Learning [29, 18, 46, 38, 34, 9] learn representation through optimization of a contrastive losswhich pulls similar pairs of examples closer while pushing dissimilar example pairs apart.
Creatingmultiple views of each example is crucial for the success of self-supervised contrastive learning.
However, most of the data augmentation methods used in generating views, although sophisticated, arehand-crafted or not learning-based.
Some use luminance and chrominance decomposition [38], whileothers use random augmentation from a pool of augmentation operators [46, 9, 4, 16, 49, 36, 57, 60].
Augmentation in Semi-supervised Learning.
Data augmentation plays an important role in semisupervised learning, e.g., (1) consistency regularization [32, 33] enforces the model to produce similaroutputs for a sample and its augmentations; (2) pseudo labeling [24] trains a model using confidentpredictions produced by itself [37] for unlabeled data.
Data augmentations are critical [22, 7] becausethey determine both the output targets and input signals: (1) accurate pseudo labels are achieved byaveraging the predictions over multiple augmentations; (2) weak augmentations (e.g., flip-and-shift)are important to produce confident pseudo labels, while strong augmentations [14, 12]) are used totrain the model and expand the confidence regions (so more confident pseudo labels can be collectedlater).
Data selection [51, 58] for high-quality pseudo labels is also critical and its criterion isestimated on augmentations, e.g., the confidence [8] or time-consistency [58] of each sample.
Augmentation in Noisy-label Learning Two primary challenges in noisy-label learning is cleanlabel detection [27, 15, 19] and noisy label correction by pseudo labels [31, 3, 25].
Both significantlydepend on the choices of data augmentations since the former usually relies on confidence thresholding and augmentations help rule out the overconfident samples, while the latter relies on the qualityof semi-supervised learning.
Moreover, as shown in previous works [25, 59], removing strong augmentations such as RandAugment can considerably degenerate the noisy label learning performance.
Preliminaries
Notations and Problem Setup.
In this paper, we use bold capital letters (e.g., X , Y ) to denoterandom variables, lowercase letters (e.g., x, y ) to denote their realizations, and curly capital letters(e.g.
, X , Y ) to denote the corresponding sample spaces.
Since we mainly consider supervised and semi-supervised problems, we define Let P X
Y be the jointdistribution of data observation X and label Y , where X is a random vector taking values on a finiteobservation space X (e.g., images) and Y is a discrete random variable taking values on the labelspace
Y
(e.g., classes).
Our goal is to learn a classifier to predict y ∈ Y from an observation x ∈ X .
Task-nuisance Decomposition.
To advance the analysis, we decouple the randomness in X into twoparts, one pertaining to the label and another independent to the label.
Concretely, we define a randomvariable nuisance N such that 1) the nuisance N is independent to the label Y , i.e., N ⊥⊥ Y ; and 2)the observation X is a deterministic function of the nuisance N and the label Y , i.e., X =
g
( Y , N )for some g .
Lemma 3.1 demonstrates that such a random variable always exists.
Lemma 3.1 (Task-nuisance Decomposition [1]) .
Given a joint distribution P X , Y , where Y is adiscrete random variable, we can always find a random variable N independent of Y such that
X =
d ( Y , N ) , for some deterministic function d .
Remarks.
We can rewrite the conditions of task-nuisance decomposition in terms of informationtheory.
1) Since the nuisance N is independent to the label Y , we have I ( Y ∧ N ) = 0 ; and 2) Sincethe nuisance N and the label
Y determines the observation X , we have H ( X | Y , N ) = 0 .
[Figure]
Principles of Representation Learning: Theoretical Interpretation
[Figure]
4.1 What Is A Good Representation?
[Figure] In real-world applications, the observation X is usually complex in a high-dimensional space X ,making it hard to directly learn a good classifier for Y .
To remedy this curse of dimensionality,it is important to learn a good representation of X , i.e., learn an encoder E ( · ) that maps the highdimensional observation X into a low-dimensional representation Z .
We illustrate the process of datageneration and representation learning by a probabilistic graphical model as shown in Figure 1a.
An ideal encoder should keep the important information from X (e.g. label-relevant information) andmaximally discard the noise or nuisance of X , such that it is much easier to learn a classifier from Zthan from X .
Based on the above intuition, we define an ϵ -optimal representation of X , which hassufficient information for classifying w.r.t.
Y , while remaining little information of the nuisance.
Definition 4.0.1 ( ϵ -Minimal Sufficient Representation ( ϵ -Optimal Representation)) .
For a Markovchain Y → X → Z , we say that a representation Z of X is sufficient for Y if I ( Z ∧ Y ) = I ( X ∧ Y ) ,and Z is ϵ -minimal sufficient for Y if Z is sufficient and I ( Z ∧ X ) ≤ I (˜ Z ∧ X )+ ϵ for all ˜ Z satisfying
I (˜ Z ∧ Y ) = I ( X ∧ Y ) .
Remark.
Due to the property of mutual information, we have 0 ≤ ϵ ≤ H ( X ) .
The lower ϵ is, themore “minimal” the representation is.
When ϵ = 0 , the representation is minimal sufficient, which isa desirable property as characterized by many prior works [41, 1].
Definition 4.0.1 characterizes how good a sufficient representation is, based on how much redundantinformation is remained.
Recall that X comes from a deterministic function of label Y and nuisance
N .
The redundancy of Z can also be measured by the mutual information between Z and N .
Achille et al.
[1] prove that if a representation Z is sufficient and is invariant to nuisance N , i.e.,
I ( Z ∧ N ) = 0 , then Z is also minimal.
However, since N is not known, it is hard to directlyencourage the representation to be invariant to N .
Can we learn an ϵ -minimal sufficient representation in a principled way?
Inspired by the recentsuccess of data augmentation techniques in self-supervised learning and semi-supervised learning, wefind that data augmentation can implicitly encourage the representation to be invariant to the nuisance
N .
However, most augmentation methods are driven by pre-defined transformations, which do notnecessarily render a minimal sufficient representation.
In the next section, we will analyze the effectsof data augmentation in representation learning in details.
4.2 Proper Data Augmentation Leads to (Near-)Optimal Representation
In this section, we investigate the role of data augmentation for learning good representations.
Wefirst make the following mild assumption on the underlying relationship between X and Y .
Assumption 4.1.
There exists a deterministic function π : X → Y , i.e., H ( Y | X ) = 0 .
Assumption 4.1 requires that there exists a “perfect classifier” that identifies the label y of theobservation x with no error, which is common in practice.
Note that for data with ambiguity, a tiebreaker can be used to map each observation to a unique label.
Therefore, Assumption 4.1 is realistic.
Let g be a deterministic augmentation function such that X ′ :
= g ( X , A ) is the augmented data,where A is a random variable denoting the augmentation selection.
For example, if X = x is animage sample, A = a is the augmentation “rotate by 90 degree”, then X ′
=
x ′ is the correspondingrotated image sample.
We learn an encoder E ( · ) that maps the augmented data X ′ to a representation
Z ′ .
With this augmentation processes, the graphical model in Figure 1a is updated to Figure 1b.
We show in the theorem below that if the augmentation process preserves the information of Y , Z ′can be sufficient for Y .
Furthermore, if the augmented data X ′ contains no information of the originalnuisance N , Z ′ will be invariant to N and thus will become a minimal sufficient representation.
Theorem 4.2.
Consider label variable Y , observation variable X and nuisance variable N satisfying
Assumption 4.1.
Let A be the augmentation variable, X ′ be the augmented data, and Z ∗ be thesolution to
[Equation]
subject to
[Equation]
[Equation] Then, Z ∗ is a ϵ -minimal sufficient representation of X for label Y if the following conditions hold:
Condition (a) : I ( X ′ ∧ Y ) =
I ( X ∧ Y ) ( X ′ is an in-class augmentation) and
Condition (b) : I ( X ′ ∧ N ) ≤
ϵ
( X ′ does not remain much information about N ).
Remarks.
(1) The objective of learning Z ∗ can be either task-independent (maximizing I ( Z ′
∧ X ′ ) ),or task-dependent (maximizing I ( Z ′ ∧ Y ) ).
The former matches the “InfoMax” principle commonlyused in self-supervised learning works [26, 18], while the latter can be achieved by supervised training(e.g.
, learning a classifier of Z for Y with cross-entropy loss).(2) When Condition (b) holds for ϵ = 0 , representation Z ′ is optimal (minimal sufficient).
Theorem 4.2, proved in Appendix C.1, shows that if we have a good augmentation that maximallyperturbs the label-irrelevant information while keeps the label-relevant information, then the representation learned on the augmented data can be minimal sufficient.
Theorem 4.2 serves as a principleof constructing augmentation.
Based on this principle, we propose an auto-augment algorithm in
Section 5, and verify the algorithm in a wide range tasks in Section 6.
Proposed Methods
[Figure]
In this section, we introduce our data augmentation and how to obtainthe augmentation using the representation learning network F ( · ) .
Then we show how to plug our augmentation into the representationlearning procedure of F ( · ) .
5.1 Label-Preserving Adversarial Auto-Augment ( LP-A3 )
As illustrated in the previous section, an ideal data augmentation X ′ for representation learning shouldcontain as little information about nuisance N as possible while still keeping all the information aboutclass Y .
Since N is not observed, we transfer the objective min X
′ I
( X ′ ∧ N ) into min X
′ I
( X ′ ∧ X )since I ( X ′ ∧ X )
=
I ( X ′ ∧ N ) + I
( X ′ ∧ Y )
and I ( X ′ ∧ Y ) is a constant under the constraint
I ( X ′ ∧ Y ) =
I ( X ∧ Y ) .
Thus the optimization problem is:
[Equation]
Implementation of Mutual Information.
To solve Equation (2), computing the mutual informationterms I ( X ′ ∧ X ) , I ( X ′ ∧ Y ) and I ( X ∧ Y ) is required.
Next, we will show how to compute theseterms using a neural net classifier F ( · ; θ ) , parameterized by θ , that consists of two components:a representation encoder E ( · ) and a predictor M ( · ) .
Specifically, F ( · ; θ ) =
M ( E ( · )) , where therepresentation encoder E ( · ) maps input X into representation Z , and the predictor M ( · ) predicts theclass of Z .
This is demonstrated in Figure 2.
Constraint implementation.
Since I ( X ′ ∧ Y ) =
H ( Y ) − H
( Y | X ′ )
and I ( X ∧ Y ) =
H ( Y ) − H
( Y | X ) , we can remove the H ( Y ) term in both sides and turn the constraint into
H ( Y | X ) =
H ( Y | X ′ ) .
Thus we only need to compute the conditional entropy of Y given X or X ′ ,which can be approximated through the neural net classifier: H ( Y | X ) =
E X , Y [ − log P ( y | x )] ≈
E X , Y [ − log( F ( x ; θ )
[ y ])] , where we use softmax class probability F ( x ; θ )
[ y ] to approximate thelikelihood P ( y | x ) .
And H ( Y | X ′ ) can be computed similarly.
Objective implementation.
Then we show how to compute the objective I ( X ′ ∧ X ) .
Since I ( X ′ ∧
X )
=
H ( X ) − H
( X | X ′ )
where H ( X ) is not related to X ′ and thus can be neglected, we onlyneed to compute H ( X | X ′ )
=
E X , X ′
[ − log P ( x | x ′ )] .
We use the Learned Perceptual Image Patch
Similarity (LPIPS) [55] between x and x ′ to compute the data likelihood P ( x | x ′ ) since LPIPSdistance is a widely used metric to measure the data similarity in data generative model field [20, 54] and many previous work has shown that LPIPS distance is the best surrogate for human comparisonsof similarity [55, 23], compared with any other distance including ℓ 2 and ℓ ∞ distance.
Although suchsurrogate may have error, it worth noting that Theorem 4.2 allows the surrogate to have ϵ error.
The
LPIPS distance is defined by the ℓ 2 distance of stacked feature maps from a neural network.
Herewe use F ( · ; θ ) to compute the LPIPS distance.
Let F ( · ; θ ) has L layers and (cid:98) F l ( · ; θ ) denotes thesechannel-normalized activations at the l -th layer of the network.
Next, the activations are normalizedagain by layer size and flattened into a single vector ϕ ( x ) ≜ ( (cid:99) F 1 ( x ; θ ) √ w 1 h 1 , ..., (cid:99) F L ( x ; θ ) √ w L h L ) , where w l and h lare the width and height of the activations in layer l , respectively.
The LPIPS distance between inputx and the augmentation x ′ is then defined as:
[Equation]
Constraint Relaxation for Efficiency.
Now, given an input x , its data augmentation x ′ can becomputed by solving the following optimization problem using the neural network F ( · ; θ ) in practice:
[Equation]
The equality constraint in Equation (4) is too strict to solve since it may be inefficient to search for anx ′ that exactly satisfies log F ( x ′ ; θ )
[ y ] = log F ( x ; θ )
[ y ] .
Thus we relax the constraint with a smallσ and change the constaint into: log F ( x ′ ; θ )
[ y ] ≥ log F ( x ; θ )
[ y ] − σ .
It’s worth noting that if σ issufficiently small, the label is still well preserved.
There is a trade-off to the value of σ , we search σto find a sweet spot where the problem is practical to solve and meanwhile the label is well preserved.
There are many off-the-shelf methods that solve Equation (4), and here we apply the Fast Lagrangian
Attack Method [23] as a demonstration.
We initialize x ′ by x plus a uniform noise.
And we find theoptimal x
′ by solving the following the Lagrangian multiplier function and gradually scheduling thevalue of the multiplier λ : [Equation]
[Equation]
The detailed procedure of the algorithm can be found in Appendix 2.
The algorithm has a similarform as adversarial attack in that they both find an optimal augmentation x ′ by adding perturbationsto the original image x .
However, the difference is that we aim to generate hard augmentation thatpreserves the label, while adversarial attack aims to change the class label.
5.2 Plugging LP-A3 into a Representation Learning Task
One primary advantage of LP-A3 is that it only requires a neural net F ( · ; θ ) to produce the augmentation and F ( · ; θ ) can be the current representation learning model, so we can plug LP-A3 intoany representation learning procedure requiring no additional parameters, which is plug-and-playand parameter-free.
At each step, we first fix F ( · ; θ ) to generate the augmentation x ′ by solving
Equation (5) using Algorithm 2.
And then we train F ( · ; θ ) by running the original representationlearning algorithm using our augmentation x ′ .
Data
selection.
It is not necessary to find hard positives for every sample.
To save more computation, we can apply a sharpness-aware criterion, i.e., time-consistency (TCS) [58], to select the mostinformative data ( τ % data with the lowest TCS in Algorithm 1) that have sharp loss landscapes, whichindicate the existence of nearby hard positives, and we only apply LP-A3 to them.
It reduces thecomputational cost without degrading the performance because (1) the improvement brought by augmentations is limited for examples whose loss already reaches a flat minimum, while the model doesnot generalize well near examples with a sharp loss landscapes; and (2) the hard positives for exampleswith flat loss landscape are distant from the original ones and might introduce extra bias to the training.
LP-A3 is compatible with any representation learning task minimizing a loss L : X ×Y ×W → R + ,which takes in a data batch and a model to output a loss value.
Y here denotes the groundtruth labelfor labeled data and pseudo label for unlabeled data.
The pseudo-code of plugging LP-A3 into therepresentation learning procedure with TCS-based data selection is provided in Algorithm 1.
Experiments
In this section, we apply LP-A3 as a data augmentation method to several popular methods forthree different learning tasks, i.e., (1) semi-supervised classification; (2) noisy-label learning and
Input: neural network
Output do 2:
Sample batch B b y b )
∼
Y
Data
: S ← τ % data lowest TCS
B ; 4: LP A3 :
Freeze θ and solve Equation (5) using Algorithm 2 for every sample in S , resulting in an augmented set
A = { ( x ′ 1 , y 1 ) , ... ( x ′ m y m ) of size m S LP
A3 augmented end while (3) medical image classification.
In all the experiments, LP-A3 can (1) consistently improve theconvergence and test accuracy of existing methods and (2) autonomously produce augmentationsthat bring non-trivial improvement even without any domain knowledge available.
A walk-clock timecomparison is given in the Appendix, showing LP-A3 effectively reduces the computational cost.
Inaddition, we conduct a thorough sensitivity study of LP-A3 by changing (1) label-preserving marginand (2) data selection ratio on the three tasks.
More experimental details can be found in the Appendix.
[Figure]
6.1 Medical Image Augmentations produced by LP-A3 vs. RandAugment
We visualize data augmentations generated by LP-A3 and RandAugment [12] on the testset of
DermaMnist [48] with a ResNet-18 classifier and its confidence on the groundtruth class in Fig.
We also use ScoreCAM [43] as an interpretation method to highlight the area in each image thatthe classifier relies on to make the prediction.
we find that LP-A3 preserves relevant derma areashighlighted by ScoreCAM and they are consistent with those in the original image.
On the contrary, RandAugment changes the color or occludes those derma areas, resulting in highly different
ScoreCAM heatmaps and hence wrong predictions (red bounding box in Fig. 3).
Instead, LP-A3 canpreserve the class information and mainly perturb the class-unrelated area in the original image.
6.2 Applying LP-A3 to Three Different Representation Learning Tasks
Here we apply LP-A3 to three different tasks by pluging LP-A3 to existing baselines of each task.
Fig.
4 shows that LP-A3 greatly speeds up the convergence of each baseline.
[Figure]
Semi-supervised learning
To evaluate how LP-A3 improves the learning without sufficient labeleddata, we conduct experiments on semi-supervised classification on standard benchmarks including CIFAR [21] and STL-10 [10] where only a very small amount of labels are revealed.
We apply LP-A3 in
FixMatch [35] and compare it with the original FixMatch and InfoMin [39], a learnable augmentationmethod for semi-supervised learning.
Their results are reported in Table 1, where LP-A3 consistentlyimproves FixMatch and the improvement becomes more significant if reducing the labeled data.
It’s worth noting that the original FixMatch already employs a carefully designed set of pre-definedaugmentations [13] that have been tuned to achieve the best performance, indicating that LP-A3 iscomplementary to existing data augmentations.
Moreover, LP-A3 also outperforms InfoMin by alarge margin ( > 5% ), which indicates that LP-A3 is also superior to existing learnable augmentations.
[Table]
Noisy-label Learning
Data augmentation is critical to noisy-label learning by providing different [Table] views of data to prevent neural nets from overfitting to noisy labels.
We apply LP-A3 to two state-ofthe-art methods DivideMix [25] and PES [5] on CIFAR with different ratios of noise labels.
LP-A3can consistently improve the performance of these two SoTA methods and the improvement is moresignificant in more challenging cases with higher noise ratios, e.g., on CIFAR100 with 90% of labelsto be noisy, LP-A3 improves PES by ≥ 15% (Table 2)
[Table] [Table]
Medical Image Classification To evaluate the performance in specific areas without domainknowledge or pre-defined augmentations, we compare LP-A3 with existing data augmentations onmedical image classification tasks from MedMNIST [48], which is composed of several sub-datasetwith various styles of medical images.
We compare our LP-A3 with RandAugment [13] on training
ResNet-18 and ResNet-50 [17].
We report the results in Table 3, where RandAugment designedfor natural images fails to improve the performance in this scenario.
In contrast, LP-A3 does notrely on any domain knowledge brings improvement to all the datasets except for BloodMnist where
[Table]
the base ResNet has already achieves > 95% accuracy so there is little improvement space for dataaugmentations.
The results indicate that hand-crafted data augmentations do not generalize to alldomains but LP-A3 can autonomously produce augmentations guided by our representation learningprinciple without relying on any domain knowledge.
[Figure]
6.3 Sensitivity Analysis of Hyperparameters
[Figure]
Label preserving margin σ : We evaluate how LP-A3 performs with different label preservingmargin σ on the three tasks.
The results are presented in Fig.
5, where a reverse U-shape is observed.
And LP-A3 using all the evaluated σ outperforms baselines, which indicates LP-A3 is robust to σ .
Data selection ratio: We evaluate the performance of LP-A3 with different amount of data selectedon the three tasks.
As shown in Fig.
5, selecting all the data does not perform the best since somedata’ augmentations are useless to apply data augmentation.
Moreover, selecting only 30% data toapply LP-A3 can outperform all baselines by a large margin, especially on MedMNIST where theimprovement is ≥ 2% , which verifies the effectiveness of LP-A3 and our data selection method.
Conclusion
In this paper, we study how to automatically generate domain-agnostic but task-informed data augmentations.
We first investigate the conditions required for augmentations leading to representationsthat preserves the task (label) information and then derive an optimization objective for the augmentations.
For practicality, we further propose a surrogate of the derived objective that can be efficientlycomputed from the intermediate-layer representations of the model-in-training.
The surrogate is builtupon the data likelihood estimation through perceptual distance.
This leads to LP-A3 , a general andautonomous data augmentation technique applicable to a variety of machine learning tasks, such assupervised, semi-supervised and noisy-label learning.
In experiments, we demonstrate that LP-A3 canconsistently bring improvement to SoTA methods for different tasks even without domain knowledge.
In future work, we will extend LP-A3 to more learning tasks and further improve its efficiency.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec.4(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix C3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Will open source later
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Aappendix D(c)
Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[No]4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable? [N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]