Semi-supervised classiﬁcation by reaching consensus among modalities
Abstract
Deep learning has demonstrated abilities to learn complex structures, but they can be restricted by available data.
Recently, Consensus Networks (CNs)[25] were proposed to alleviate data sparsity by utilizing features from multiple modalities, but they too have been limited by the size of labeled data.
In this paper, we extend
CN to Transductive Consensus Networks (TCNs), suitable for semi-supervised learning.
In TCNs, different modalities of input are compressed into latent representations, which we encourage to become indistinguishable during iterative adversarial training.
To understand TCNs two mechanisms, consensus and classiﬁcation, we put forward its three variants in ablation studies on these mechanisms.
To further investigate TCN models, we treat the latent representations as probability distributions and measure their similarities as the negative relative Jensen-Shannon divergences.
We show that a consensus state beneﬁcial for classiﬁcation desires a stable but imperfect similarity between the representations.
Overall, TCNs outperform or align with the best benchmark algorithms given 20 to 200 labeled samples on the Bank Marketing and the DementiaBank datasets.
Introduction
Deep learning has demonstrated impressive capacities to learn complicated structures from massive data sets.
However, acquiring sufﬁcient labeled data can be expensive or difﬁcult (e.g., for speciﬁc pathological populations [11]).
Transductive learning (a set of semi-supervised algorithms) uses intrinsic structures among unlabeled data to boost classiﬁer performance.
In the real world, data can spread across multiple modalities (e g., visual, acoustic, and text) in typical tasks, although many existing transductive algorithms do not exploit the structure across these modalities.
Co-training [3] and tri-training [24] use one classiﬁer per modality to supervise each other, but they can only apply to two and three modalities respectively.
Recently, Consensus Networks (CNs) [25] incorporated the idea of co-training.
Not limited by the number of modalities, CNs showed promising results on detecting cognitive impairments from multi-modal datasets of speech.
A consensus network contains several interpreters (one per modality), a discriminator, and a classiﬁer.
The interpreters try to produce low-dimensional representations of input data that are indistinguishable by the discriminator.
The classiﬁer makes predictions based on these representation vectors.
Despite promising results, CN is limited by the amount of available training data.
This motivates our extension into semi-supervised learning with our Transductive Consensus Network (TCN).
TCNs operate in two mechanisms: as consensus or classiﬁer.
The consensus mechanism urges the modality representations to resemble each other (trained on the whole dataset without using labels), and the classiﬁer mechanism optimizes the networks to retain information useful for classiﬁcation (trained on the labeled dataset).
To illustrate the importance of these two mechanisms in an ablation study, we also put forward its three variants: TCN-embed, TCN-svm, and TCN-AE in §3.
By this ablation study, we show that both mechanisms should function together via iterative training.
To further reveal the mechanisms of TCN, we formulate in §3.5 the similarity between latent representations using negative Jensen-Shannon divergences.
By monitoring their similarities, we show that a meaningful consensus state prefers representations to have suboptimal similarities.
In experiments (§4), we compare TCN to its three variants, TCN’s multimodal supervised learning counterpart (CN), and several other semi-supervised learning benchmark algorithms on two datasets:
Bank Marketing (from the UCI repository) and DementiaBank (a dataset of pathological speech in multiple modalities).
On both datasets, the F-scores of TCN align with the best benchmark models when there are more labeled data available, and outperform benchmarks (including tri-training) given as few as 20 labeled points.
Related work
Transductive SVMs [9] were an early attempt in transductive semi-supervised learning.
In addition to the SVM objective, TSVMs minimize the hinge loss on unlabeled data.
TSVMs have yielded good performance on our datasets, so we include them for completeness.
Later, many semi-supervised learning algorithms took either autoencoding or GAN approaches.
In autoencoding, a model learns a low-dimensional representation and a reconstruction for each data sample.
Usually, noise is added in generating the low-dimensional representation.
By trying to minimize the difference between reconstructed and original data, the model learns (i) an encoder capturing low-dimensional hidden information and (ii) a decoder, which is a generative model able to recover data.
This is the approach of the denoising autoencoder [23, 12].
An extension is Ladder network [20], which stacks denoising autoencoders and adds layer-wise reconstruction losses.
Ladder networks are often more computationally efﬁcient than stacked denoise autoencoders.
In generative adversarial networks (GANs) [7], a generator tries to produce data that are indistinguishable from true data, given a discriminator which itself learns to tell them apart.
This adversarial training procedure could proceed with few labeled data points.
For example, Feature-matching GANs [21] add generated (“synthetic”) samples into the training data of the discriminator as an additional class.
Another example is Categorical GANs [22] which optimize uncertainty (measured by entropy of predictions) in the absence of labels.
Noticeably, [4] showed that a discriminator performing well on a training set might not beneﬁt the whole dataset.
CNs and TCNs, despite not containing generative components, are built with the adversarial principles inspired from GANs.
The idea to make multiple components in the network agree with each other has been adopted by several previous models.
For example, [2] proposed Parallel Consensus Networks, where multiple networks classify by majority voting.
Each of the networks is trained on features after a unique transform.
[14] proposed consensus optimization in GAN in which a term is added to the utility functions of both the generator and discriminator to alleviate the adversity between generator and discriminator.
However, neither they nor semi-supervised learning utilized the multiple modalities.
Multi-modal learning is also referred to as multi-view learning.
[19] computed multiple viewpoints from speech samples and classiﬁed cognitive impairments.
By contrast, our multi-view learning is semi-supervised, and can involve non-overlapping subsets of features.
In domain adaptation, some work has been applied to ﬁnd a uniﬁed representation between domains, for example, by applying domain invariant training [6] and semantic similarity loss [16].
However, our approach does not involve multiple domains – we only handle data from one domain.
Here, the term ‘domain’ refers to how the data are naturally generated, whereas the term ‘modality’ refers to how different aspects of data are observed. [Figure]
Methodology
In previous work, Consensus Networks [25] was proposed for multimodal supervised learning.
We extend the model to be suitable for semi-supervised learning, resulting in Transductive Consensus
Networks (TCNs).
This section also presents three variants: TCN-embed, TCN-svm, and TCN-AE.
3.1 Problem setting
Given labeled data, { x ( i ) , y ( i ) } ( x ( i ) ∈
X L ), and unlabeled data, { x ( i ) } (where x ( i ) ∈
X U ), we want to learn a model that reaches high accuracies in predicting labels in unlabeled data.
In the semisupervised learning setting, there are many more unlabeled data points than labeled: |X U | (cid:29)
|X L | , where the whole dataset is X = X L ∪ X U .
Each data point x contains feature values from multiple modalities (i.e., ‘views’).
If M be the total number of modalities, then x = [ x 1 , x 2 , ..., x M ] .
For m = 1 , ..., M , x m could have different dimensions, but the dimension of x ( i ) m is consistent throughout the dataset.
E.g., there may be(100) acoustic (semantic) features for each data point.
3.2 Consensus Networks
Here we brieﬂy review the structures of CNs.
In a CN model, M interpreter networks I 1 ,...,M ( . ) each compress the corresponding modality for a data sample into a low-dimensional vector v .
[Equation]
We call these networks interpreters , because they interpret the feature spaces with representations.
In TCNs, a consensus is expected to be reached given representations from multiple views of the same data sample.
A discriminator network D ( . ) tries to identify the originating modality of each representation vector.
If we write the m th modality of the dataset as a set M m of vectors, then the discriminator function D ( . ) can be expressed as:
[Equation]
To prevent the discriminator from looking at only superﬁcial aspects for each data sample in the forward pass, Consensus Networks [25] include an additional ‘noise modality’ representation sampled from a normal distribution, with mean and variance determined by the ‘non-noise’ representations:
[Equation]
The model’s discrimination loss L D is therefore deﬁned as the cross entropy loss across all modalities (plus the noise modality), averaged across both labeled and unlabeled datasets X :
[Equation]
Finally, a classiﬁer network C ( . ) predicts the probability of class assignment ( y ) from the combined representation vectors, given model parameters:
[Equation]
The classiﬁcation loss L C is just the cross entropy loss across the labeled data:
[Equation]
The overall optimization goals for CN can therefore be described as:
[Equation]
3.3 Transductive Consensus Network
Consensus Networks (CNs), as a supervised learning framework, are limited by the amount of labeled data.
This motivates us to generalize the approach to semi-supervised scenarios.
There are two mechanisms in the CN training procedures, namely the classiﬁer mechanism and consensus mechanism.
The classiﬁer mechanism requires labeled data, but the consensus mechanism does not explicitly require these labels.
We let the consensus mechanism handle both labeled and unlabeled data.
This results in Transductive Consensus Networks.
Formally, the loss functions are rewritten as:
[Equation]
where X consists of both labeled data X L and unlabeled data X U .
Overall, the optimization goal can still be written as:
[Equation]
These goals set up a complex nonlinear optimization problem.
To ﬁgure out a solution, we break down the goals into three iterative steps, similar to GAN[7]:
• The ‘I’ step encourages interpreters to produce indistinguishable representations: max I 1 ..M L D .
• The ‘D’ step encourages discriminators to recognize modal-speciﬁc information retained in representations: min D L D .
• The ‘CI’ step trains the networks to make a correct decision: min C,I 1 ..M L C .
3.4 TCN variants
The consensus mechanism builds a low-dimensional latent representation of each (labeled and unlabeled) data sample containing common knowledge across different modalities, and the classiﬁer mechanism tries to make these representations meaningful.
Three modiﬁcations are made to our base
TCN model, resulting in the following models:
TCN-embed consists of the same networks as TCN but is trained slightly differently.
Before the I-D-CI optimization cycle, we add a pretraining phase with I-D iterations, which emphasizes the consensus mechanism.
TCN-svm removes the classiﬁer network from TCN-embed.
After the pretraining phase across the whole dataset, we extract the representations of those labeled data samples to train a supervised learning classiﬁer (i e., an SVM).
TCN-svm discards the classiﬁer mechanism, which results in deteriorations to model performance (§5).
TCN-AE provides insights from another perspective.
In contrast to TCN, TCN-AE contains several additional reconstructor networks, R 1 ..M ( . )
(one per modality).
Each reconstructor network tries to recover the input modality from the corresponding low-dimensional representations (plus a small noise (cid:15) ):
[Equation]
Deﬁning reconstruction loss as L R =
E
x ∈X E m | ˆx m − x m | 2 , the optimization target in TCN-AE can be expressed as:
[Equation]
and [Equation] TCN-AE is inspired by denoising autoencoder[23], where the existence of reconstructor networks encourage the latent variables to preserve realistic information.
This somehow works against the consensus mechanism, which according to [25] tries to agree on simple representations.
TCN-AE therefore weakens the consensus mechanism.
We will show in §5 that an inhibited consensus mechanism results in inferior model performance.
3.5 The similarity of representations
We want to quantitatively measure the effects of the consensus and the classiﬁcation mechanisms.
To evaluate the similarities of representations, we treat the hidden dimensions of each representation v m =
[ v m, 1 , v m, 2 , .., v m,j , ... ] (after normalization) as discrete values of a probability mass function 1 , which we write as p m .
The M modalities for each data point are therefore approximated by M probability distributions.
Now we can measure the relative JS divergences between each pair of representations v m and v n derived from the same data sample ( ˆ D ( p m || p n ) ).
To acquire the relative value, we normalize the JS divergence by the total entropy in p m and p n : [Equation] where D KL
[Equation]
where v m,j and v n,j are the j th component of v m and v n respectively.
In total, for each data sample with M modalities, M ( M − 1)pairs of relative divergences are calculated.
We average the negative of these divergences to get the similarity:
[Equation]
Note that by our deﬁnition the maximum value of the “similarity” value is 0 (where there is no JS divergence between any pair of the representation vectors), and it has no theoretical lower bound.
Figure 4 shows several 2D visualizations of representation vectors drawn from an arbitrary run.
In
Figure 5, we illustrate how the similarities between modality representations evolve during training.
Experiments 4.1 Datasets
We compare TCN and its variants on two benchmark datasets: Bank Marketing, and DementiaBank.
The full list of features, by modalities, are provided in the Supplementary Material.
The Bank Marketing dataset is from the UCI machine learning repository[5].
used for predicting whether the customer will subscribe a term deposit in a bank marketing campaign via telephone[15].
There are originally 4,640 positive samples (subscribed) and 36,548 negative ones (did not subscribe).
Since consensus network models do not work well on imbalanced datasets, we randomly sample 5,000 negative samples to create an (almost) balanced dataset.
We also convert the categorical raw features 2 into one-hot representations.
We then divide the features into three modalities: basic information, statistical data, and employment-related features.
DementiaBank 3 contains 473 spoken picture descriptions of the clinical “cookie-theft picture”[1], containing 240 positive samples (the Dementia class) and 233 negative samples (the Control class).
We extract 413 linguistic features from each speech sample and their transcriptions, including acoustic (e.g., pause durations), lexical & semantic (e.g., average cosine similarities between words in sentences) and syntactic (e.g., complexity of the syntactic parse structures) modalities.
[Table]
4.2 Benchmarks
We evaluate TCN and its variants against several benchmarks, including:
Multimodal semi-supervised learning benchmark: Tri-training [24] 4 . 2.
TCN’s supervised counterpart: Consensus Network (CN).
Unimodal semi-supervised learning: TSVM [9], ladder network [20], CatGAN [22].
4.3 Implementation
For simplicity, we use fully connected networks for all of I 1 ..
M , D , C , and R 1 ..
M in this paper.
To enable faster convergence, all fully connected networks have a batch normalization layer [8].
For training, the batch size is set to 10.
The neural network models are implemented using PyTorch [17], and supervised learning benchmark algorithms (SVM, MLP) use scikit-learn [18].
We use the Adam optimizer [10] with an initial learning rate of 0.001.
In training TCN, TCN-embed, and TCN-AE, optimizations are stopped when the classiﬁcation loss does not change by more than 10 − 5 in comparison to the previous step, or when the step count reaches 100.
In the pre-training phase of TCN-embed and TCN-svm, training is stopped when the discrimination loss changes by less than 10 − 5 , or when pretraining step count reaches 20.
Sometimes, the iterative optimization (i.e., the I-D-CI cycle for TCN / TCN-embed, and the I-D-RI-CI cycle for the TCN-AE variant) is trapped in local saddle points – the training classiﬁcation loss does not change while the training classiﬁcation loss is higher than log 2 ≈ 0 .
This is the expected loss of a binary classiﬁer with zero knowledge.
If the training classiﬁcation loss is higher than log 2 , the model is re-initialized with a new random seed and the training is restarted.
Empirically, this re-initialization happens no more than once per ten runs, but the underlying cause needs to be examined further.
Results and discussion 5.1 TCN performances versus the benchmarks
As shown in Figure 2, TCN outperforms or matches the best benchmarks.
On the Bank Marketing dataset, TCN, CN, and TSVM clearly outperform the rest.
On DementiaBank, Tri-train, TCN, and
TSVM form the “ﬁrst tier”.
Also, semi-supervised learning does not always outperform those supervised algorithms.
For example, on the Bank Marketing dataset, CN (i.e., TCN’s supervised learning counterpart) holds the second best performance.
5.2 What can we learn from the TCN variants?
As shown in Figure TCN aligns with or outperforms TCN-embed.
Both of these approaches signiﬁcantly outperform TCN-AE.
On the other hand TCN-svm produces almost trivial classiﬁers.
There are several points worth noting.
• Both the consensus and the classiﬁcation mechanisms are beneﬁcial to classiﬁer performance.
The classiﬁcation mechanism can be beneﬁcial even with as few as 20 labeled data samples.
• Iterative optimization is crucial.
Without classiﬁcation mechanisms, the consensus mechanism by itself fails to derive good reprensentations.
Without consensus mechanisms (i
e., when the reconstructors hinder the consensus mechanisms), accuracies drop signiﬁcantly.
5.3 Understanding the similarity of representations
To understand more about TCN, we visualize them with T-SNE[13] in Figure and plot the similarity values in Figure 5.
• The higher similarity values corresponds to a state where the distributions contain higher symmetry in aggregate manner.
• Measured from the similarity values, TCN models reach a consensus state where the similarities are stable.
TCN-svm reaches agreements quickly but are close to trivial.
TCNAE, with the autoencoder blocking the consensus mechanism, fails to reach a state of agreement.
[Figure]
Note that we stop the training procedure when losses converge (as detailed in §4.3), so the trials may stop
Conclusion and future work
In this paper we present Transductive Consensus Networks (TCNs) that extend consensus networks with semi-supervised learning.
We identify two mechanisms in which TCNs function, i e., the consensus and classiﬁer mechanisms.
With three TCN variants in an ablation study we show the importance of both mechanisms.
Moreover, by treating the representations as probability distributions and deﬁning their similarity as negative relative JS divergences, we show that although the consensus mechanism urges high similarities, a good consensus state might not need perfect similarities between modality representations.
In the future, several avenues may be considered.
To start with, building consensus networks using other types of neural networks may be considered.
In addition, more exploration could be done to ﬁnd a more explainable metric to describe the extent of agreement.
Currently, we use − D JS H [-
-]1 + H 2 , but this requires some approximations.
Optimizing against the similarity metric, instead of setting up a discriminator, may be worth examining as well.
Supplementary Material
Marketing
Three modalities are determined as following.
The division are somewhat arbitrary, except that we try to make the binary features resulting from one categorical feature be in the same modality.
Basic information: age, marital, education, housing, loan, contact, duration, pdays, previous, management 2.
Statistical information: campaign, poutcome, emp.var.rate, unknown, cons.conf.idx, euribor3m, day_in_week (converted to 7 binary features), month (converted to 12 binary features) 3.
Employment-related information: consumer price index, never employed, retired, selfemployed, technician, services, student, housemaid, entrepreneur, blue-collar
Features by modalities for Dementia Bank
Three modalities are determined as following:
Acoustic-related features: phonation rate, mean pause duration, pause word ratio, total speech duration, short/medium/long pause count, speech rate, word/audio/(ﬁlled or unﬁlled) pauses durations, the mean/variance/kurtosis/skewness of the ﬁrst 42 Mel Frequency Cepestral Coefﬁcients 2.
Syntactic-semantic features: probabilistic context-free grammar parsing tree heights (average / max / etc.), and the occurrences of 104 production rules (e.g: NP → DT).
Lexical and POS-derived features: the occurrences of part-of-speech tags, Brunet’s index, Honore’s statistics, word length, cosine distances between words in sentences, etc.