R EGULARIZATION N EURAL N ETWORKS VIA C ONSTRAINED V IRTUAL M OVEMENT F IELD
Anonymous authors
Paper under double-blind review
A BSTRACT
We provide a novel thinking of regularization neural networks.
We smooth the objective of neural networks w.r.t small adversarial perturbations of the inputs.
Different from previous works, we assume the adversarial perturbations are caused by the movement ﬁeld.
When the magnitude of movement ﬁeld approaches 0, we call it virtual movement ﬁeld.
By introducing the movement ﬁeld, we cast the problem of ﬁnding adversarial perturbations into the problem of ﬁnding adversarial movement ﬁeld.
By adding proper geometrical constraints to the movement ﬁeld, such smoothness can be approximated in closed-form by solving a min-max problem and its geometric meaning is clear.
We deﬁne the approximated smoothness as the regularization term.
We derive three regularization terms as running examples which measure the smoothness w.r.t shift, rotation and scale respectively by adding different constraints.
We evaluate our methods on synthetic data, MNIST and CIFAR-10.
Experimental results show that our proposed method can significantly improve the baseline neural networks.
Compared with the state of the art regularization methods, proposed method achieves a tradeoff between accuracy and geometrical interpretability as well as computational cost.
I NTRODUCTION
Deep neural networks have achieved great success in recent years
Lecun et al.
By improving the depth of computational graphs and the accounts of trainable parameters, neural networks can ﬁt the training dataset better.
However, overﬁtting becomes a serious problem in supervised training especially when the free parameters are numerous.
One of the most effective ways to against overﬁtting is adding regularization terms into the original supervised objective function.
Many regularization methods have been proposed for training neural networks, such as dropout Srivastava et al.
(2014) and its variants Wang & Manning (2013); Kingma et al.
From a Bayesian perspective, dropout regularizes neural networks by introducing randomness into the parameters.
Another regularization way to against overﬁtting is generating new data by transform or perturbation the existing data.
The objective of the generated data or the smoothness w.r.t the small perturbations can be regarded as a regularization term.
Bachman et al.
(2014) assume the perturbations are fully random.
They use the model’s sensitivity to those random perturbations in their construction of the regularization function.
However, Goodfellow et al.
; Szegedy et al.
(2014) found the robustness of neural networks can’t be improved sufﬁciently with random noise.
Instead of using random perturbations, adversarial training (AT) Goodfellow et al. (2015) and virtual adversarial training (VAT)
Miyato et al.
ﬁnd the so-called adversarial perturbations by optimizing some objectives under simple constraints, such as L ∞ norm and L 2 norm.
Speciﬁcally, AT selects the adversarial perturbation direction which maximizes the objective of neural networks.
This lead to set the direction of perturbation the same as the gradients of objective w.r.t the inputs.
Once the optimal perturbation is obtained, they apply it to the inputs and get the perturbated inputs.
Then AT minimizes the objective of both the original inputs and the perturbated inputs.
VAT follows similar spirits of AT.
The key difference is that VAT obtains the optimal perturbation by maximizing Kullback-Leibler divergence (KLD) between the outputs of models.
This makes VAT applicable for semi-supervised learning.
VAT also designs an iterative algorithm to approximate the optimal perturbation.
Experimental results demonstrate that AT and VAT against the adversarial perturbation as well as improve the generalization ability of neural networks.
However, there are two drawbacks of AT and VAT.
• For a single batch of data, both AT and VAT need to run at least two forward-backward loops to complete the training process, which is time-consuming for big models.
• The obtained optimal perturbation lacks geometrical interpretability.
It is hard to fully understand why those adversarial perturbations are most likely to fool neural networks.
We try to overcome the above drawbacks by introducing constraints into the space of perturbations.
In this work, we assume perturbations are caused by the movement ﬁeld of the lattice structured data , such as speech signals, images and videos.
Movement ﬁeld represents the motion vector of each pixel in the lattice.
We call it virtual movement ﬁeld when the magnitude of the movement ﬁeld is sufﬁciently small.
We smooth the objective of neural networks when the virtual movement ﬁeld is applied to the inputs.
Inspired by VAT, we ﬁrst ﬁnd the so-called adversarial movement ﬁeld which maximizes above smoothness under a set of constraints. Then we minimize the corresponding smoothness under the adversarial virtual movement ﬁeld. That is to obtain the smoothness, we need to solve a min-max problem with constraints.
This can be done in closed-form if the set of constraints are carefully designed.
Once this smoothness term is obtained, we minimize the objective of neural networks together with it. Because the movement ﬁeld is virtual, it is unnecessary to get the perturbated inputs.
Instead, the smoothness is expressed by the derivative of the objective w.r.t the movement ﬁeld.
Thus, the training process of each batch is completed in a single forwardbackward loop which yields lower computational costs.
Moreover, the obtained perturbations or smoothness term are much more interpretable because of the constraints.
For example, we can see which direction of movement of an image is most likely to fool neural networks.
We call our method as virtual movement training (VMT).
We summarize the novelties of VMT as follows:
• The assumption of “small perturbations are caused by the virtual movement ﬁeld” is a completely new idea in the literature of adversarial training or adversarial examples. By this assumption, we introduce data dependent constraints into the space of perturbations.
And we cast the problem of ﬁnding perturbations into the problem of ﬁnding movement ﬁeld. • We develop a general framework to design regularization terms for neural networks trained with lattice structured data, i.e. solving a min-max problem associated with the movement ﬁeld.
closed-form terms are obtained by introducing proper geometrical constraints to the movement ﬁeld.
In this work, we focus more on computational efﬁciency and geometrical interpretability of our method instead of against adversarial examples Szegedy et al.
We derive three simple regularization terms as running examples based on introducing different constraints (shift, rotation and scale) into movement ﬁelds.
These regularization terms measure how sensitive of neural networks under virtual (extremely small) shift, rotation and scale perturbations respectively.
We evaluate our method in a 1D synthetic data and two benchmark image classiﬁcation datasets:
MNIST and CIFAR-10.
Experimental results demonstrate that our method remarkably improves the baseline neural networks.
Compared with AT and VAT, VMT achieves a tradeoff between accuracy and geometrical interpretability as well as computational cost.
M ETHODS
We ﬁrst formally deﬁne the movement ﬁeld and the virtual movement ﬁeld.
Then we formulate our method.
Finally, we provide three running examples.
2. V IRTUAL M OVEMENT F IELD
For data I ∈ R d 1 × d 2 ···× d n , i.e. n dimension lattice structure and the length of i th dimension is d i , we deﬁne the movement ﬁeld V of as an n + 1 dimension tensor, that is V ∈ R d 1 × d 2 ···× d n × n .
Denote p ∈
Z n as the position vector of I
and I p is the value in that position.
Then V p ∈ R n is the movement of location p , i.e. its new position would be p + V p .
Note that for 2 dimension lattice data such as images, their movement ﬁeld is somewhat similar to the concept of optical ﬂow.
However, throughout this paper, we still use the word of “movement ﬁeld” because it is generalized to any dimension of lattice data.
If we assume data I is sampled from an underlying continues space or the ﬁrst order derivatives of I exists, we can approximate the value of the new position with the ﬁrst order Taylor series of the value of the original position (when the movement is small).
Formally
[Equation]
For V p , there are two factors: the length and the direction.
In some cases, it is necessary to decompose those two factors.
So we normalize it as follows:
[Equation]
Then the averege square length of (cid:101) V is equal to one.
Denote ε (cid:101) V as the actual movement ﬁeld.
We call ε the degree of the movement ﬁeld.
When ε approaches 0 , we call (cid:101) V the virtual movement ﬁeld .
Based on (1) and (2), if (cid:101) V is given, we have:
[Equation]
2. P ROBLEM F ORMULATION
Given a dataset D
= { ( I n , y n ) | n = 1 , 2 , . . .
, N } , where I n and y n are i th pair of input and label in D .
Denote f θ as a function which is parameterized by θ .
f θ maps the input space into the output space.
For each pair of { I n , y n } , we minimize the predeﬁned loss function between the predicted output and the label w.r.t θ .
[Equation]
We expect that L is stable for some particular kind of movements of I , e.g. rotation for images.
We can apply a small movement ε (cid:101) V to I n
and we get the new input I n ( ε (cid:101) V ) .
Intuitively, since (cid:101) V is normalized we can measure the smoothness of L under the movement ﬁeld (cid:101) V as follows:
[Equation]
That is the proportion between the change of objective and the degree of inputs movement.
When training neural networks, in order to obtain the above smoothness, we need to run the forward computational graph two times: the one for L ( I n , y n ) , the other for L ( I n ( ε (cid:101) V ) , y n ) .
However, in this work, we are interested in the extreme situation of (5): what if we apply a virtual movement ﬁeld to I p ?
Or how sensitive the objective w.r.t ε when ε → 0 .
Note that L can be reparameterized as a function of ε by ﬁxing other variables.
Thus when ε → 0 , (5) is equivalent to
[Equation]
By the chain rule for differentiation
[Equation]
Substitute (3) into it, we have
[Equation]
The ﬁrst term on the right side of the above equation is the gradient of objective w.r.t the input which is easily obtained by back propagation.
The second term is the gradient of input w.r.t its coordinates which is also called the directional derivative of signals.
For discrete signal, it is approximated by ﬁnite difference operator in practice.
Thus, once (cid:101) V is given, we can obtain the smoothness of objective w.r.t ε when the corresponding virtual movement ﬁeld applied to the input in a single forward-backward loop. 12 14       (a) Shift      12 14       (b) Rotation      12 14       (c) Scale [Table]
[Equation] However, the degree of freedom of (cid:101) V is extremely high for real-world data.
Search for such high dimension is inefﬁcient.
Thus it is necessary to introduce constraints into (cid:101) V .
Those constraints should embody the priors of data and physical mechanisms of how it is generated.
For example, the movement ﬁeld of natural images should enjoy the properties of local smoothness and isotropy.
We denote the set of constraints as C ( (cid:101) V ) .
Note that the normalization constraint in (2) is included in
C ( (cid:101) V ) .
We can expect that the degree of freedom of (cid:101) V is sufﬁciently reduced under those constraints.
If there are still freedom of (cid:101) V , we can randomly draw samples over those freedom.
However, inspired by the adversarial training, we ﬁrst ﬁnd the adversarial movement ﬁeld (cid:101) V ∗ which maximizes the smoothness of neural networks then we minimize the obtained smoothness term plus the original objective w.r.t θ under (cid:101) V ∗ .
Similar to the generative adversarial networks (GAN) Goodfellow et al.
(2014), above problem can be formulated as a min-max game under constraints:
[Equation]
Once (cid:101) V ∗ is obtained by solving the above max problem, the second term in (9) is determined.
We call it as the corresponding regularization term of (cid:101) V ∗ .
Then (9) is reduced to
[Equation]
Generally, solving the max problem in (9) is not an easy task.
However, we will show that (cid:101) V ∗ can be obtained in closed-form if the constraint set is carefully designed.
And in this paper, we just focus on this simple case because we hope to train each batch of data in a single forward-backward loop.
2. D ESIGN THE M OVEMENT F IELD
Now we provide three sets of constraints for (cid:101) V which make the corresponding (cid:101) V ∗ solved in closedform.
All these movement ﬁelds are designed for 2D lattice data since image is one of the most important types of data in real world.
The ﬁrst one is called Shift ﬁeld.
That is all pixels in 2D lattice are shifted by the same vector.
Because (cid:101) V is normalized, the only freedom is the direction of the vector in 2D space.
Formally
[Equation]
Combination (11) and (8), the max problem in (9) is equivalent to
[Equation]
12) where p 1 and p 2 are the ﬁrst coordinate and the second coordinate of an image respectively.
Then the optimal value of φ is obtained easily.
And the corresponding maximum value of | ∂ L /∂ε | in this case is
[Equation]
[Equation]
See Appendix C for derivation.
The second movement ﬁeld is rotation ﬁeld.
In this work, we simply assume the center of rotation is the center of the image, i.e. ( c 1 , c 2 ) .
[Equation]
(14) where Z is the normalization constant described in (2).
Thus the degree of freedom is 0 (The direction of rotation doesn’t matter because we care about the absolute value of ∂ L /∂ε ) .
Then
R rotation is obtained straightforwardly.
The third movement ﬁeld is scale ﬁeld which scales an image by different factors along two coordinates.
Thus the degree of freedom is 1.
We parameterize it by
[Equation]
Then R scale is obtained in a similar way as R shift .
We summary these three movement ﬁelds and their corresponding derived regularization terms in Tab 1.
Although other kinds of movement ﬁelds are possible to be designed, we just use them to evaluate our method because they are simple, easy to implementation and geometrically meaningful.
2. P RACTICAL C ONSIDERATIONS
As mentioned in section 2.2, for lattice data, the directional gradients are approximated by ﬁnite difference operator.
In this work, we choose the simplest one:
[Equation]
where x is an arbitrary coordinate in p .
If the local smooth property is not well satisﬁed, above approximation is not accurate.
This suggests that our method is more suitable for smooth data.
Another problem is that we ﬁnd the magnitudes of ∂ L /∂I change rapidly with the network conﬁgurations.
This makes the values of the corresponding regularization terms change rapidly with the network conﬁgurations.
To keep the values of R in a stable range, we normalize ∂ L /∂I into a unit tensor.
D ISCUSSIONS AND RELATED WORKS
Our work was mainly motivated by the adversarial training Goodfellow et al.
and was related to the virtual adversarial training Miyato et al.
Adversarial training can be reformulated as follows:
[Equation]
[Equation] and δI is approximated by sign ( ∂ L /∂I ) in their paper because the only constraint of δI is L ∞ norm.
However, in our work, we assume δI is caused by the movement ﬁeld (cid:101) V .
[Equation]
That is the perturbation δI is constrained by both the movement ﬁeld (cid:101) V and the directional gradients of I , instead of the simple norm constraint.
Another key difference between AT and our method is ε → 0 in our work, thus it is unnecessary to generate I + εδI and run additional forward-backward loop.
By setting ε sufﬁciently small, (17) is equivalent to
[Equation]
This formulation is similar to ours which suggests our method is an extreme case of AT if we ignore the difference between constraints of perturbations.
Denote f θ as the forward function of neural networks parameterized by θ .
Then virtual adversarial training can be reformulated as follows:
[Equation]
The core difference between VAT and AT is that VAT minimizes KLD of the outputs of neural networks under adversarial perturbations.
This property makes VAT applicable for semi-supervised learning.
However, the KLD term makes it difﬁcult to ﬁnd the optimal perturbation.
Thus Miyato et al.
(2016) developed an iterative algorithm for approximation.
The idea of smooth the objective w.r.t small perturbations also has appeared in earlier works, such as tangent propagation Simard et al. and inﬂuence function Koh & Liang (2017).
The derivative of the objective w.r.t perturbations is obtained by chain rule.
However, the transformations are predeﬁned in Simard et al.
(1998) while the transformations in VMT are obtained by solving a constrained min-max problem though the freedom of those transformations is low currently. For inﬂuence function, there are no constraints in the space of perturbations.
And the smoothness w.r.t small perturbations is mainly used to analyze the behaviors of a trained model instead of regularizing it during training.
In summary, we focus more on geometrical interpretability and computational efﬁciency of our method.
Thus our method is not required to be better than VT and VAT.
However, we still compare them in next section.
E XPERIMENTAL R ESULTS
We evaluate our proposed method for supervised classiﬁcations in three datasets: 1D synthetic dataset, MNIST and CIFAR-10.
We compare it with the baseline, adversarial training with Lconstraint and virtual adversarial training.
For each dataset, the shared hyper-parameters keep same for all methods.
While separate hyper-parameters are tuned by cross-validation using grid-search or copied from literature if they are provided.
ε in (17) and (20) is searched over [0 . 01 , 10] and λ in (10) is searched over [0 . 005 , 5] .
All neural networks are implemented in Tensorﬂow Abadi et al.
We call our method VMT-shift, VMT-rotation and VMT-scale when we use the corresponding regularization terms.
4. T HE B INARY C LASSIFICATION OF 1D S YNTHETIC D ATASET
We create a 1D synthetic dataset with two classes using the following random process:
[Equation]
where t ∈ R 100 is uniformly sampled from [ − 2 π, 2 π ] .
Thus x ∈ R 100 is a 1D lattice signal.
Based on (21), we generate 5000 positive samples by setting ω = 0 .
99 and 5000 negative samples by setting ω
= 1 . 01 .
We randomly select 1000 samples as training set and the rest as test set.
We train neural networks with two hidden layers each of which has 128 units and is followed by batch normalization Ioffe & Szegedy (2015) and ReLU activation Glorot et al.
We set batchsize to 20 and run 100 epochs using ADAM optimizer Kingma & Ba (2015).
We run each method 5 times and report the average test errors.
We summarize the results in Tab 2.
VMT-shift is signiﬁcantly better than the baseline and is slightly better than AT.
We show values of R shift on test set over epochs in
100 t 1.
x negativepositive (a) Samples 10 20 30 40 50 60 70 80 90 100 Epoch 0. R s h
i f t o n t e s t s e t
BaselineATVATVMT-shift (b) R shift 10 20 30 40 50 60 70 80 90 100 Epoch 0. G e n e r a li z a t i
o n g a p BaselineATVATVMT-shift
[Figure]
are consistent with the test errors.
That is the smaller the values of R shift at the end of training, the smaller the test errors.
We also empirically compare the generalization abilities of those methods.
The generalization ability is measured by the so-called generalization gap, i.e. the difference between the empirical risks on test set and training set.
[Equation]
It is suggested by Fig 2(c) that R shift is an appropriate regularization term which can improve the generalization ability of baseline on our hand-created data.
We ﬁnd that VMT-scale is inferior to baseline on our hand-created data.
This is not surprising because each sample is labeled by its angular frequency, i.e. ω .
When we scale a sample, its angular frequency will be changed.
Thus the assumption of the smoothness under small scale perturbations is not satisﬁed in this case.
In fact, the priors of data can be encoded in the constraints of movement ﬁeld.
Thus, for VMT, it is possible to targeted design regularization terms based on the properties of data.
4. T HE C LASSIFICATION OF MNIST D ATASET
We tested the performance of our regularization method on the MNIST dataset, which consists of handwritten digits with size 28 × 28 and their corresponding labels from 0 to 9.
We train our models using the whole 60000 training samples.
For network structures, we follow the setting used in Miyato et al.
Speciﬁcally, we train NNs with 4 hidden dense layers with nodes (1200 , 600 , 300 , 150) respectively.
Each hidden layer is followed by batch normalization and ReLU activation.
We apply all of the three regularization terms in Tab 1 and compare them with the baseline, dropout, AT and VAT.
We run each method 5 times and report the average test errors.
The results are summarized in Tab 2 which show our methods are inferior to AT and VAT but are signiﬁcantly better than baseline.
The training time on Synthetic dataset and MNIST is summarized in Appendix B.
4. T HE CLASSIFICATION OF CIFAR-10 DATASET
We also conducted studies on the CIFAR-10 dataset (without data argument) which consists of 50000 training images and 10000 testing images in 10 classes, each image with size 32 × 32 × 3 .
Our focus is on the behaviors of different regularization methods, but not on pushing the stateof-the-art results, so we use a relatively small neural network for evaluation and comparison of those regularization methods.
Speciﬁcally, we conﬁgure the neural networks as the “conv-small” used in Salimans et al.
(2016) which contains 9 convolutional layers.
See appendix A for detailed architecture.
All neural networks are trained by SGD with momentum with 80 epochs.
We evaluate the test errors and the average training time of each epoch.
We deﬁne R all as the linear combination of the regularization terms in Tab 1. Formally
[Equation] [Table] [Table]
[Figure] where w i ∼ U (0 , 1) . We randomly sample w i for each batch. VMT-all means we use R all as the regularization term.
Results are summarized in Tab 3.
All our regularization terms are signiﬁcantly better than the baseline and AT.
And they are competitive compared with VAT.
When we linearly combine R shift , R rotation and R scale , the performance is further improved by a relatively large margin.
Such combination is cheap in practice (See running time in Tab 3).
We believe that the performance can be improved to a higher level if we design and combine more regularization terms.
And our method is faster than AT and VAT.
We show the values of the regularization terms in Fig 3.
All these values of our methods are smaller than the values of baseline.
C ONCLUSIONS
In this paper, we have provided a novel thinking of regularization neural networks.
We smooth the objective function of neural networks when the virtual moment ﬁeld is applied to lattice data.
By carefully introducing constraints into the movement ﬁeld, we have derived the smoothness in closed-form by solving a min-max problem.
We have provided three regularization terms which measure the smoothness w.r.t the transformations of shift, rotation and scale respectively.
Experimental results demonstrate that our method remarkably improves the baseline neural networks on 1D synthetic data, MNIST and CIFAR-10.
Compared with AT and VAT, VMT achieves a tradeoff between accuracy and geometrical interpretability as well as computational cost.
Unlike AT and VAT, the training process of each batch is completed in a single forward-backward loop.
Moreover, by control the movement ﬁeld, we can understand the geometric meaning of perturbations and what kind of smoothness the regularization term is measured. A D ETAILED ARCHITECTURE OF “ CONV - SMALL ” [Table]
B T RAINING TIME ON S YNTHETIC DATA AND MNIST
[Table]
C D ERIVATION OF R shift
By setting a =
[Equation]
p
[Equation]
p
[Equation]
L ∂I p ∂I p ∂p(12) is equivalent to
[Equation]
where α = tan − 1 ba
Thus the maximum value of (12) is √ a 2 + b 2 when φ = α .