T RAIN N EURAL N ETWORK B Y E MBEDDING S PACE P ROBABILISTIC C ONSTRAINT
A BSTRACT
Using higher order knowledge to reduce training data has become a popular research topic.
However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels.
Based on this observation, we consider constraining output probability distribution as higher order domain knowledge.
We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.
While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution.
We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.
I NTRODUCTION
Probability is an abstract measure on how a certain event occurs independent of features of the events.
Knowing how likely a certain event occurs, people leverages such prior knowledge to their decision making.
For example, doctors know certain diseases are rare, even if they are told in terms of probabilities instead of ”training examples”.
Based on this knowledge, they make less predictions on these diseases than those common ones.
Do neural networks behave in a similar way?
Unfortunately, the answer is no.
When we train a multi-layer perceptron(MLP) for MNIST classiﬁer (LeCun et al.
(1998)) with limited labelled examples, the output distribution can be extremely biased in favor of some of the labels.
In Figure 1a, we compare the predicted number of labels with ground truth.
While the training accuracy is 1.0, the model clearly overﬁts to those training examples and leave labels between training data points undeﬁned in high dimensional feature space.
As we plot the last hidden layer of a MLP trained with 50 labelled MNIST data as shown in Figure 1b, we ﬁnd neural networks fail to learn the decision boundary correctly from a limited number of examples.
Thus, it is natural to consider introducing output label probability distribution as higher order knowledge when we train neural networks.
Different from traditional logical constraints (Xu et al.
or functional constraints (Stewart & Ermon (2016), we propose a novel embedding space probabilistic constraint.
Because of the sparsity of high dimensional feature space with only a few labeled examples, we perform our probabilistic constraint on neural network’s embedding space, which is constructed unsupervisedly by projecting data into low dimensional space through autoencoder.
Based on observation by Xie et al.
(2016), Zhang et al.
(2016), embedding space preserves information of separations of different label clusters.
In the embedding space, we pool softmax activation
[Figure]
R ELATED W ORKS
Weak Supervision Current supervised representation learning algorithms have gained great success on various tasks in computer vision (He et al.
(2017), Kostrikov et al.
(2018)), natural language processing (Trivedi et al.
(2018), Athiwaratkun et al.
with little domain knowledge,
but they require large quantity and high-quality labels for training.
Thus, there is a growing trend of research that address this problem by transferring knowledge learned from different datasets (Azizzadenesheli et al.
(2018), Shen et al.
or introducing higher level knowledge.
In this work, we consider incomplete weak supervision problem (Zhou (2017)).
A typical incomplete supervision problem (Chapelle et al.
(2006)) is formulated as following: with a dataset { X , Y } that consists of labeled dataset X 1 =
{ X 1 , y 1 } and unlabeled dataset X 2 =
{ X 2 , y 2 } , where { y 2 } is not visible during training.
| X 1 | (cid:28) | X 2 | .
This problem can usually be tackled by state-of-art semi-supervised learning algorithms like AtlasRBF (Pitelis et al.
, Neural Rendering Model (Ho et al.
(2018))or LadderNet (Rasmus et al.
or using novel approaches such as logical constraints (Xu et al.
While they still rely on certain amount of high quality labeled data, while in this work, we further decrease the number of labeled data needed for convergence.
Learning With Constraints Learning with constraints takes various higher order domain knowledge into the optimization of neural networks.
Based on domain knowledge, different constraints are effective on different tasks.
For example, Pathak et al.
(2015) uses linear constraints on the output space and optimizes the training objective as a biconvex optimization for linear models to perform dense pixelwise semantic segmentation.
Frameworks such as semantic loss by Xu et al.
and logical loss by Hu et al.
(2016) specify logic rules when training neural networks.
Stewart & Ermon (2016) propose a novel framework that one can learn physical or causal relationship without labels.
In this work, we consider the case where limited labeled examples lead to biased output distribution.
Different from these arithmetic or logical constraints, we consider placing an output probability constraint.
E MBEDDING S PACE P ROBABILISTIC C ONSTRAINT
In this section, we state our problem formulation and describe the proposed algorithm and architecture for this problem.
Higher-order Knowledge Formulation Based on the incomplete weak supervision deﬁned in
Section 2, we specify our introduced higher order knowledge.
We assume from our domain knowledge of output probability distribution, the model can acquire a set Q = { ( k, P ( Y = k )
+ (cid:15) } k ∈{ Y } ) .
One thing to note is that domain knowledge distribution Q does not necessarily cover all k ∈ Y .
We use (cid:15) ∈ R drawn from Gaussian distribution such that (cid:15) ∼ N (0 , σ 2 ) to reﬂect the variance of human domain knowledge from true Y .
We use σ = 0 .
05 throughout this paper.
We need a training algorithm A
( { X 1 , y 1 } , X 2 , Q } ) such that it trains a multi-layer perceptron f ( x ) =
σ ( W m ...
Φ( W 2 Φ( W 1 x )) , where W i is model’s weights, Φ is an nonlinear function like ReLU, and σ is the softmax function.
This algorithm minimizes the loss function (cid:96) : { Y, f ( X ) } −→ R .
Batchwise Probability Constraint
Following from this problem formulation, we deﬁne a loss term (cid:96) c : R 2 −→ R to regulate the output distribution.
We regard a single update batch { X (cid:48) , Y (cid:48) } ∈ { X, Y } with size c as the unit of output probability distribution.
Inspired by Liang et al.
(2017) and
Hendrycks & Gimpel (2016), the activation of ﬁnal softmax layer of classiﬁer σ ( · ) can reﬂect neural network’s conﬁdence towards a certain label.
Instead of performing counting the arguments of the maxima for all labels, which is not inefﬁcient, we consider calculating the mean pooling of all the activation outputs(Wang et al.
It can be written mathematically as 1 c (cid:80) ci {+=
+} f
( X (cid:48) i ) .
Here we use f k to denote the softmax activation of the k th label.
This potentially improves the accuracy of detecting low conﬁdence or out-of-distribution examples.
A basic ﬂowchart of our mechanism can be found in Figure 2.
It is natural to use Kullback-Leibler (KL) divergence as a metric of our output distribution different from the reference domain knowledge probability distribution Q , that is,
[Equation]
One may notice the probability of labels in the batch does not always reﬂect the domain knowledge distribution Q .
That is, | P ( Y (cid:48) =
k ) − Q ( k ) | > (cid:15) for some (cid:15) > 0 .
For the simplicity of this work, we assume additional but very weak supervision on identifying some of those batches and using different but noisy batch probability distribution.
However, this supervision can be easily done through at-a-glance (abriel Ryan (2018)) supervision or auto-regressive algorithms similar to Reed et al.
Our proposed algorithm and its convergence analysis can be found in Appendix A.
Constraint on Embedding Space In order to use existing unlabeled data to draw decision boundaries, we propose to jointly optimize this probability constrained classiﬁer with an embedding space regularizer.
Embedding is a lower dimensional form that structurally preserves data from original hyper-dimensional space.
In our case, we treat a hyperparameter i th hidden layer of perceptron E ( x ) as our embedding space, where E ( x ) =
W i Φ( ... Φ( W 2 Φ( W 1 x ))) and f ( x )
= σ ( W m (Φ( W m − 1 ( ... Φ( W i +1 )))
◦ E ( x ) , where the dimension of E ( x ) should be much smaller than dimension of input x .
Zhang et al.
(2016) propose using unsupervised loss can preserve information of separations between different label clusters.
Thus, we adopt the structure of decoder of autoencoder and deﬁne a multi-layer neural network D ( · ) as a decoder of our embedding space.
For a single batch { X (cid:48) , Y (cid:48) } , our loss function for training a separation-preserving embedding space by reconstructing the original input, that is,
[Equation]
General Framework Our proposed method uses unsupervised loss (cid:96) r to construct an embedding in low dimensional space, uses limited labeled data to identify the cluster location in the embedding space by original classiﬁcation loss (cid:96) original , and uses domain knowledge of output probability distribution to determine the actual decision boundaries.
Then our updating loss function is
[Equation]
, where λ 1 , λ 2 and λ 3 are hyperparameter constants.
E VALUATION
Experiment setup We evaluate our proposed embedding space probabilistic constraint in semisupervised learning setting.
Using the similar base multilayer perceptron model as in Rasmus et al.
(2015) and Xu et al.
All the experiments are repeated ﬁve times with different seeds.
We add an additional embedding layer with width 40, and the decoder has a symmetric architecture as the feed forward neural network.
Model Description To guarantee that our comparison focuses on output probability distribution instead of one single instance’s label, we train our models with batch size 128.
We experiment our model under different level of constraints.
Datasetwise probability constraints assumes the target output should be all 10% , and the noisy datasetwise probability constraints adds a random noise drawn from N (0 , 0 . 3) to simulate user’s knowledge.
Also, we use batchwise probability constraint, which assumes we know the probability of labels in every batch, as an upper bound for our algorithm.
We compare our model with other state-of-art semi-supervised learning models (Pitelis et al.
Rasmus et al.
(2015)), and logical constraint model (Xu et al.
Since we require more human supervision than other semi-supervised learning models, we use their results to demonstrate our model can converge to high accuracy with much less high quality labeled examples.
We also choose other baselines models without both losses and without embedding loss to show the beneﬁt of our architecture.
[Table]
[Table]
D ISCUSSION
In this paper, we present an algorithm that constraints output probability distribution in an embedding space.
Our result shows with a little more supervision than normal semi-supervised learning [Table] algorithms, we need far fewer high quality training examples to reach high accuracy.
Thus, we conclude jointly optimizing the output constraint with hypothesis can draw a decision boundary with smaller labelled training data than other state-of-art methods.
Our focus is to show the power of a very weak labelling method without high quality labelling technique.
We leave it as a future research direction to design an auto-regressive algorithm that requires less supervisions.
In addition, since our formulation sums all the activation functions together as a measure of conﬁdence instead of counting-based probabilities this allows us to use it as a future direction on conﬁdence of classiﬁcation in semi-supervised learning setting
A A LGORITHM
In Bootstrap phase, we require users to provide an entry supervision mark { Θ } .
It marks entries that has unusually high probabilities of occurrence from target probability distribution.
We need this extra supervision because reliance on model’s own probabilistic judgement might converge to a wrong model.
Feeding additional but weak supervision can give model enough information to judge if current probability distribution has such unusual batches.
We also claim that we can bound it by concentration inequalities, as shown in Theorem 1.
The probability of this mark happens can be bounded by Theorem 1.
Theorem 1 For a single batch { X, y } with size c , on a label k with domain knowledge output probability Q ( k ) .
For all (cid:15) > 0 , we have
[Equation]
Proof of Theorem 1.
This claim directly follows from Hoeffding inequality, which we state as following:
For independent bounded random variables X i ∈
[ a i , b i ] and for all t > 0 , which has a ≤ X ≤ b , for all (cid:15) , we have
[Equation]
The proof can be found in Shalev-Shwartz & Ben-David (2014).
In our case, our random variable is P ( | Y = k ) , ranging from [0 , 1] , while the target distribution, by higher knowledge, is E
[ P ( | Y = k )]
=
Q ( k ) .
With batch size c, we have
[Equation]
We state our bootstrap algorithm in Algorithm 1.
When we found a marked batch with higher probability, we redistribute our new target distribution based on its original target distribution.
When an output probability is marked, we rescale the target distribution of the label q marked to adapt the higher-than-usual probability by Equation 2.
A detailed derivation can be found in Appendix B.2.
[Equation]
Data: Training batch { X , y } ∈ R c × m × R n , entry supervision mark { Θ } , error threshold (cid:15) ,
Domain knowledge output distribution Q ∈ R n , neural network f
Feed Forward f ( X ) and pool output activations by 1 c (cid:80) ci =
f
( X i ) ; for i ← 0 to m do if this batch is marked then
[Equation]
else
[Equation]
end end
Train the network f with { X, y, q } by Equation 3;
The boundary between bootstrap phase and auto-regressive phase is a hyper-parameter.
In this work, we use validation set to observe the accuracy of current model.
When the accuracy on validation set is larger than 70% , we enter auto-regressive phase.
A convergence plot example can be found in
Figure 3.
Data: Training batch { X , y } ∈ R c × m × R n , error threshold (cid:15) , Domain knowledge output distribution Q ∈ R n , neural network f
Feed Forward f ( X ) ; Ω ← pool output activations by 1 c (cid:80) ci =
f
( X i ) ; Θ ← ∅ ; for i ← 0 to m do if | Ω i − Q i | > (cid:15) then Θ ← Θ ∪ Ω ; else end q ← Q ∧ ¬ Θ ;
Train the network f with { X, y, q } by loss function from Equation 3;
B D ERIVATIONS B. D ERIVATIONS FOR E QUATION
Lemma 1 Let f be a feed forward neural network with softmax as the last layer’s activation function.
Then given a batch { X, y } ∈ R c × m × R n , the mean-pooling Ω = 1 c (cid:80) ci =
f
( X i ) is a probability distribution.
Proof of Lemma 1.1 Without loss of generality, we write function f ( x ) as σ ( · ) ◦ g ( x ) for some function g ( x ) .
Then for all x ∈ X , f ( x ) is in range of σ ( · ) , that is, ∃ y ∈ R n , f ( x ) = σ ( y ) .
Then for a vector z ∈ R n , for any label i , σ ( z ) i =
e z i (cid:80) nk =1 e zi
Then we perform mean-pooling on σ ( y ) i , we have Ω i =
1 c c (cid:88) i =
f
( X i ) =
1 c c (cid:88) i =
σ ( y i )
= 1 c c (cid:88) i =
e i (cid:80) nk =1 e yi
The pooled output activation Ω i is trivially larger than 0, and since σ ( z ) i < 1 for all z , mean ( σ ( z ) i ) < 1 .
Since (cid:80) i Ω i = 1 , 1 c (cid:80) ci =0 f
( X i ) =
1 c c = 1 .
Then we conclude it fullﬁlls all the axiom of a probability distribution.
Since Ω is a probability distribution by lemma 1, we can apply KL divergence to Ω from our target distribution Q. With speciﬁed constraints set K , with f k ( x ) for the speciﬁc dimension of k ∈ K , we have
[Equation]
B. D ERIVATIONS FOR E QUATION
For a single training example x j ∈ X , the features for x j = { x j 1 , x j 2 ...x jn } are dependent, so we cannot apply statistical bounds by entries.
However, when we compare training examples in the same batch, that is, x j 1 and x j +1, they are independent.
As a result, we can still use the Proof of
Theorem 1.1.
When we would like to bound
[Equation]
for some error d , we rearrange
[Equation]
that is,
[Equation]
In this case, we ﬁnd a safe margin d that controls the tradeoff between human supervision and training batch accuracy.
Directly applying the upper bound is empirically ﬁne when we perform our experiments.
C I MPLEMENTATION D ETAILS
This section includes main implementation details not included in main text.
C. D ATASET D ESCRIPTION
[Equation] dataset (LeCun et al.
(1998)) is a dataset of handwritten digits, which hastraining images and 10000 testing images.
For each of the image, it is a grey-scale 28 × 28 matrix that belongs to 10 classes from 1 to 10.
FASHION a dataset of clothes that possess similar structure as MNIST, which has 60000 training images and 10000 testing images.
For each of the image, it is a grey-scale 28 × 28 matrix that belongs to 10 classes for different types of clothes.
CIFAR-10 CIFAR-10 (Krizhevsky (2009)) is a dataset that contains colored images with size 32 × 32 .
Each image belongs to one of ten classes like dog, cat, car.
The training set hasimages and the testing set has 10000 images.
C. D ATASET P REPARATION
Selection
In order to make the probability the same for all classes, we keep the same number of images among 10 classes.
We choose the number as the minimum number of examples out of all classes.
For example, for MNIST dataset, it has numbers of examples 5923, 6742, 5958, 6131, 5842, 5421, 5918, 6265, 5851 respectively from 0 to 1.
Since 5421 is the minimum for all labels, we choose 5421 as the number of examples we use.
Based on this threshold, we choose the training examples randomly from the dataset.
To make examples from different classes balanced, we choose randomly (number of labeled examples/number of classes) number of examples.
Processing We ﬁrst normalize all images scale from 0 to 1.
Then we have 50% of chance to perform two operations on training examples to prevent overﬁtting of the training set: adding Gaussian noise and cropping.
For Gaussian noise, we add to the image another matrix of random noise drawn from Gaussian distribution with mean 0 and standard deviation 0.3.
We crop the image by three pixels.
For example, if the original image has pixel 28 × 28 , our cropped image has 25 × 25 .
C. A RCHITECTURE
Multi-layer Perceptron We evaluate our proposed embedding space probabilistic constraint in semi-supervised learning setting.
Using the similar base multilayer perceptron model as in Rasmus et al.
(2015) and Xu et al.
(2018) with layer of the size 784-1000-500-250-250-250-40-10, except adding a layer with width 40 as embedding layer, we feed the output of the embedding layer to another multilayer deocoder with size 40-250-250-1000-784.
We also perform batch normalization and dropout with a dropout rate 50% to increase robustivity of our model.
Convolution Neural Network To compare fairly with other state-of-art models, we adopt a basic 10-layer architecture similar to Xu et al.
(2018) for classiﬁcation.
For every three layers, we use one convolution layer with activation function ReLU, one 2 × 2 max-pool layer with stride 2.
We insert one layer of dropout with a dropout rate 0.5 and one batch normalization layer as we described in multi-layer perceptron.
For the decoder, we use a symmetric decoder.
For every three convolution layers, we use a upsampling layer.
We insert batch normalization layer and dropout layer between them.
The embedding space has a vector with dimension 50.
C. T RAINING
Detailed hyperparameter is as following: we use Stochastic Gradient Descent(SGD) to update our neural network with a learning rate 10 e − 4 .
For simplicity, we choose λ 1 , λ
and λ 3 to be 1.
A rule of thumb is to make λ 1 (cid:96) original ≈ λ 2 (cid:96) c .
When they possess similar values, they can converge to the solution quickly.
The convergence of our algorithm can be shown in Figure 3.
In the cold start session, with batch size 128, we choose to ﬁnd batch output probabilities p / ∈ [0 . 02 , 0 . 25] .
Empirically this covers 98% of data, we can safely set them with probability 0 .
1 , while the rest but obvious data, we set them a manual probability, in our case, we set them 0 .
02 and 0 . 28 .