Few-Shot Non-Parametric Learning with Deep Latent Variable Model
Abstract
Most real-world problems that machine learning algorithms are expected to solve face the situation with (1) unknown data distribution; (2) little domain-specific knowledge; and (3) datasets with limited annotation.
We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV), a learning framework for any dataset with abundant unlabeled data but very few labeled ones.
By only training a generative model in an unsupervised way, the framework utilizes the data distribution to build a compressor.
Using a compressor-based distance metric derived from Kolmogorov complexity, together with few labeled data,
NPC-LV classifies without further training.
We show that NPC-LV outperforms supervised methods on all three datasets on image classification in the low data regime and even outperforms semi-supervised learning methods on CIFAR-10.
We demonstrate how and when negative evidence lowerbound (nELBO) can be used as an approximate compressed length for classification.
By revealing the correlation between compression rate and classification accuracy, we illustrate that under
NPC-LV how the improvement of generative models can enhance downstream classification accuracy.
Introduction
The progress of deep neural networks drives great success of supervised learning with huge labeled datasets [1, 2, 3].
However, large labeled datasets are luxurious in many applications and huge amounts of training parameters make the model easy to overfit and hard to generalize to other datasets.
The urge to learn with small labeled dataset prompts Few-Shot Learning (FSL).
However, most fewshot classification settings require either an auxiliary “support set” [4, 5, 6, 7] that contains c classes, each with k samples ( c -way k -shot); or prior knowledge about the dataset, where data augmentation can be performed within the same dataset [8, 9, 10] or from other weakly-labeled/unlabeled/similar datasets [11, 12, 13].
This setting is not widely applicable to every dataset in practice, as it requires either the elaborate construction of an additional “support set” or augmentation algorithms tailored to specific datasets.
Pre-trained models, on the other hand, do not require adhoc “support” and have proved to be good at few-shot learning [14] and even zero-shot learning [15].
However, thousands of millions of training parameters make the model hard to be retrained but only fine-tuned.
When the data distribution is substantially different from any datasets used in pre-training, the inductive bias from pre-training holds up fine-tuning, making the model less pliable [16].
Goals from the above learning paradigms can be summarized as to design algorithms that can be applied to any dataset, and can learn with few labeled data, ideally with no training.
“No Free
Lunch” [17] implies that it’s impossible to have an algorithm that is both “universal” and “best”.
But how good can a “universal” algorithm be, especially in the low data regime, with no external data resources?
Specifically, we are interested in a new setting, Non-Supported Few-Shot Learning (NS-FSL), defined as follows:
36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Given any target dataset D = ( x 1 , x 2 , ..., x n ) belonging to c classes.
For each class, we have k labeled samples (1 ≤ k ≤ 10) .
The remaining n − ck unlabeled samples need to be classified into c classes without the need of support sets, any other datasets, or training parameters.
This setting is similar to semi-supervised learning’s but excludes labeled information in training.
Ravi and Larochelle [18] demonstrate that it’s hard to optimize a neural network when labeled data is scarce.
In order to make minimum assumptions about labeled data, we aim at using parameter-free methods.
The goal is to grasp the data-specific distribution p ( x ) , with minimal premises on the conditional probability distribution p ( y | x ) .
Deep generative models with explicit density estimation are perfect candidates for this goal.
The problem then becomes: given trained generative models, how to take full advantage of the information obtained from them for classification?
Using a latent representation only utilizes p ( z | x ) , which just includes partial information.
Even for those latent generative models that do not suffer from posterior collapse [19], p ( z | x ) ’s insufficiency for classification with non-parametric methods like k -nearest-neighbor is shown in both previous works [20, 21] and our experiments.
Inspired by previous work that uses compressor-based distance metrics for non-parametric learning [22, 23, 24, 25], we propose Non-Parametric learning by Compression with Latent Variables (NPC-LV), a learning framework that consists of deep-generative-model-based compressors and compressor-based distance metrics.
It leverages the power of deep generative models without exploiting any label information in the probabilistic modeling procedure.
With no further training, this framework can be directly used for classification.
By separating probabilistic modeling from downstream tasks that require labels, we grasp the unique underlying structures of the data in every dataset, and further utilize these structures in downstream tasks with no parameter tuning needed.
We view this learning framework as a baseline in this setting, for any dataset.
We argue it is “parameter-free” as there are no parameters involved in the classification stage for labeled data.
Basically it means training a generative model as is and getting a classifier for free.
Our contributions are as follows: (1) We frame existing methods into a general learning framework
NPC, based on which we derive NPC-LV, a flexible learning framework with replaceable modules.
(2) We use NPC-LV as a baseline for a common learning scenario NS-FSL with neither support sets nor further training.
(3) Our method outperforms supervised methods by up to 11.8% on
MNIST, 18.0% on FashionMNIST, 59% on CIFAR-10 on image classification in the low data regime.
It outperforms non-parametric methods using latent representation on all three datasets.
It even outperforms semi-supervised learning methods on CIFAR-10.
(4) We show how negative evidence lowerbound (nELBO) can be used for classification under this framework.
(5) We find the correlation between bitrate and classification accuracy.
This finding suggests how improvements in the domain of deep-learning-based-compressors can further boost classification accuracy under this framework.
Background 2.1 Information Theory — Data Compression
In a compression scenario, suppose we have a sender Alice and a receiver Bob .
Alice wants to send a message that contains a sequence of symbols x
= ( x 1 , x 2 , ..., x n ) to Bob .
The ultimate goal of the lossless compressor is to compress x into the minimum amount of bits x ′ that can later be decompressed back to x by Bob .
To achieve the shortest compressed length, shorter codes are assigned to symbols with higher probability.
According to Shannon’s Source Coding Theorem [26], this length of bits is no shorter than the entropy of the sequence, whose definition is H ( x ) ≜ E [ − log p data ( x )] , where p data ( x ) represents the probability distribution of each symbol in the sequence.
Instead of coding symbols one by one, stream code algorithms like Asymmetric Numeral Systems (ANS)
[27] convert x to a sequence of bits x ′ and reaches this optimal code length for the whole sequence with an overhead of around 2 bits, given p data ( x ) .
However, the “true” probabilistic distribution p data ( x ) is unknown to us.
We can only access samples and approximate it with p θ ( x ) .
That is:
[Equation]
Given an entropy coding scheme, the better p θ ( x ) approximates p data ( x ) , the closer we can get to the minimum code length.
Modeling p θ ( x ) is where deep generative model with density estimation can help.
Possible coding schemes and generative models for compressors are discussed in 3.2.
2.2 Algorithmic Information Theory — Kolmogorov Complexity and Information Distance
While information theory is built on data distributions, algorithmic information theory considers “single” objects without the notion of a probability distribution.
Kolmogorov complexity K ( x ) [28] is used to describe the length of the shortest binary program that can produce x on a universal computer, which is the ultimate lower bound of information measurement.
Similarly, the Kolmogorov complexity of x given y is the length of the binary program that on input y outputs x , denoted as
K ( x | y ) .
Based on Kolmogorov complexity, Bennett et al.
[29] derive information distance E ( x, y ) :
[Equation]
The idea behind this measurement, at a high level, is that the similarity between two objects indicates the existence of a simple program that can convert one to another.
The simpler the converting program is, the more similar the objects are.
For example, the negative of an image is very similar to the original one as the transformation can be simply described as “inverting the color of the image”.
Theorem 1.
The function E ( x, y ) is an admissible distance and a metric.
It is minimal in the sense that for every admissible distance D , we have E ( x, y ) ≤ D ( x, y )
+ O (1) .
Intuitively, admissible distance refers to distances that are meaningful (e.g., excluding metrics like
D ( x, y ) = 0 .
3 for any x ̸ = y ) and computable (formal definition is in Appendix B).
Combining these definitions, we can see Theorem 1 means E ( x, y ) is universal in a way that it is optimal and can discover all effective similarities between two objects (proof is shown in Appendix H).
In order to compare the similarity, relative distance is preferred.
Li et al.
[30] propose a normalized version of E ( x, y ) called Normalized Information Distance (NID).
Definition 1 ( NID ) .
NID is a function: Ω × Ω → [0 , 1] , where Ω is a non-empty set, defined as:
[Equation]
Equation (3) can be interpreted as follows:
Given two sequences x , y , K ( y ) ≥ K
[Equation]
x ) :
[Equation]
where I ( x : y ) =
K ( y ) − K
( y | x ) means the mutual algorithmic information .
I ( x : y ) K ( y ) means the shared information (in bits) per bit of information contained in the most informative sequence, and
Equation (4) here is a specific case of Equation (3).
Theoretically, NID is a desirable distance metric as it satisfies the metric (in)equalities (definition in Appendix B) up to additive precision O (1 /K ( · )) where K ( · ) is the maximum complexities of objects involved in (in)equalities (proof shown in [31]).
Non-Parametric learning by Compression with Latent Variables
Non-Parametric learning by Compression (NPC) consists of three modules — a distance metric, a compressor and an aggregation method shown in Figure 1.
NPC-LV leverages NPC by including neural compressors based on deep generative models.
We introduce the derivation of compressorbased distance metrics in Section 3.1; the generative-model-based compressor in Section 3.2; an integration of this framework with generative models in Section 3.3.
3.1 Compressor-based Distance Metric
Universal as NID is, it is uncomputable, because Kolmogorov complexity is uncomputable.
Cilibrasi and Vitányi [32] propose Normalized Compression Distance (NCD), a quasi-universal distance metric based on real-world compressors.
In this context, K ( x ) can be viewed as the length of x after being maximally compressed.
Suppose we have C ( x ) as the length of compressed x produced by a real-world compressor, then NCD is defined as:
[Equation] [Equation]
[Figure]
The better the compressor is, the closer NCD approximates NID.
With a normal compressor (discussed in detail in Appendix B), NCD has values in [0,1] and satisfies the distance metric (in)equalities up to O (log n/n ) where n means the maximum binary length of a string involved [33].
NCD is thus computable in that it not only uses compressed length to approximate K ( x ) but also replaces conditional Kolmogorov complexity with C ( xy ) that only needs a simple concatenation of x, y .
Li et al.
[31] simplify NCD by proposing another compression-based dissimilarity measurement (CDM):
[Equation]
Chen et al.
[24] use another variation ranging from 0 to 1:
[Equation]
NCD, CDM and CLM are different variations of Kolmogorov-based distance metrics.
We empirically evaluate their performance in Section 5.
3.2 Trained Generative Models as Compressors
[Figure]
(Alice:
[Figure] [Figure]
with
[Figure]
with
[Figure]
with
BB-ANS (Bob: decompress) received bitstream decode with decode with
[Figure]
with
Previous works [24, 29, 32] demonstrate the success of compression-based distance metrics in sequential datasets like time series, DNA, and texts using non-neural compressors like gzip, bzip2, PPMZ.
Deep-generative-model–based compressors can take Non-Parametric learning by Compression (NPC) to the next level by expanding to more data types using better compressors.
We mainly focus on variational autoencoder (VAE) based compressors with a brief introduction to other neural compressors.
VAE Family:
The relation between VAE and “bits-back” has been revealed in multiple previous works [34, 35].
Townsend et al.
[36] use latent variable models by connecting Asymmetric Numeral
Systems (ANS) [27] to the “bits-back” argument [37] (BB-ANS).
In the setting of the “bits-back argument”, we assume Alice has some extra bits of information to send to Bob alongside x .
It’s also assumed that both Alice and Bob have access to p ( z ) , p θ ( x | z ) and q ϕ ( z | x ) where z is the latent variable; p ( z ) is the prior distribution of z ; p θ ( x | z ) represents a generative network and q ϕ ( z | x ) represents an inference network.
As shown in Figure 2, Alice first decodes the extra information according to q ϕ ( z | x ) to generate a sample z .
2 z is further used to encode x with p θ ( x | z ) and z itself is encoded using p ( z ) .
Bob then reverses this procedure and recovers the extra bits by encoding with q ϕ ( z | x ) .
For a single data point, the length of the final bitstream is:
[Equation]
We can see the expectation of N − n extra is equal to the negative evidence lower bound (nELBO):
[Equation]
ELBO above is derived from the “bits-back argument” in the context of compression.
Now, from the perspective of latent variable models like VAE, the derivation often starts from the fact that p θ ( x ) = (cid:82) p θ ( x | z ) p ( z ) is intractable.
q ϕ ( z | x ) is then introduced as an inference model to approximate p ( z | x ) in order to work around the intractability problem, which brings up the marginal log likelihood:
[Equation]
We only need to optimize the lower bound, as minimizing nELBO means maximizing log p θ ( x ) — the likelihood of generating real data and minimizing KL divergence between q ϕ ( z | x ) and p ( z | x ) at the same time, which is the same objective function from what we derive using “bits-back”.
This equivalence demonstrates that an optimized latent variable model can be used directly for compression as, from the data compression perspective, it minimizes the code length attainable by bits-back coding using the model.
With the help of ANS, we can encode symbols into bitstreams or decode bitstreams back to symbols with trained latent variable models.
Details of ANS and discretizing continuous latent variables are shown in Appendices D and E.
ARM: Autoregressive models (ARMs) model p ( x ) as: p ( x ) = p ( x 0 ) (cid:81) ni
=1 p ( x i | x i − 1 ) .
The exact likelihood estimation makes it capable of lossless compression.
But instead of using ANS, which is a stack-like coding scheme, queue-like ones (e.g., Arithmetic Coding (AC) [38]) should be used.
Computational inefficiency is the main drawback of ARMs such as RNN [39], but causal convolutions [40, 41] can alleviate the problem.
IDF: Integer Discrete Flow (IDF) [42] can also optimize towards the exact log-likelihood.
Similar to other flow-based models, it utilizes an invertible transformation f , but works on discrete distributions with additive coupling and rounding operations.
For IDF, ANS can be used as the entropy coder.
Our experiments and discussion are mainly about VAE-based compressors as their architectures can be flexibly changed under the “bits-back” argument.
3.3 NPC-LV
We’ve shown in Section 3.2 that we can plug in any trained latent variable model in exchange for a near optimal compressor under the framework of BB-ANS.
To show that the coding scheme is replaceable, we introduce a coding scheme variation, Bit-Swap [43], in the following experiments.
The difference between BB-ANS and Bit-Swap is the encoding and decoding order when there is more than one latent variable.
A detailed comparison is shown in
Appendix F.
The generative model we use for both BB-ANS and Bit-Swap is a hierarchical latent generative model (details are in Appendix G), also known as Deep Latent Gaussian Model (DLGM)
Aggregation:
In addition to coding schemes and probabilistic models for compressors, aggregation methods can also be replaced.
Previous works [32] assume xy in C ( xy ) means the “concatenation” of two inputs.
We expand this assumption to a more general case where “aggregation” can be another kind of aggregation function, represented as C ( ψ ( x, y )) in Figure 1.
We justify this generalization as changing aggregations methods may make compressor-based distance metrics not admissible (shown in Appendix B).
More sophisticated strategies of aggregation are left to future work.
We also discuss other replaceable modules in detail in Appendix A.
We use BB-ANS with NCD as a concrete instance to demonstrate this framework on a classification task shown in Algorithm 1.
D U train and D L train mean the unlabeled and labeled training sets; predefined functions are in teal.
The algorithm can be simplified into four steps: (1) Train a VAE on the unlabeled training dataset; (2) Apply ANS with discretization on the trained VAE to get a compressor; (3)
Calculate the distance matrix between pairs ( x test , x train ) with the compressor and NCD; (4) Run k -Nearest-Neighbor( k NN) classifier with the distance matrix.
nELBO as estimated compressed length: Specifically for VAE-based compressors, nELBO is the estimated length of the compressed bitstream as Equation (9) shows.
Therefore, we can use it directly
Input: k , D test , D train =
{ D U train , D L train } , D L train =
{ X L , y L }
trained_vae = trainVAE
( D U train ) vae_compressor = ANS(trained_vae) for x test in D test do
C_ x test = len(vae_compressor( x test )) distances =
[] for x train in X L do // Calculate NCD distance with C ( x ) , C ( y ) , and C ( ψ ( x, y ))
C_ x train = len(vae_compressor( x train ))
C_agg = len(vae_compressor( aggregate( x test , x train ))
NCD =
(C_agg − min{C_ x test , C_ x train }) / max{C_ x test , C_ x train } distances = push(NCD, distances) end for k_nearest_indicies = argsort(distances)
[: k ] y test = majority( { y Li , i ∈ k_nearest_indicies } ) end for without actual compression.
This can further simplify our method as we don’t need to discretize continuous distributions or apply an entropy coder.
The reason why the underlying data distribution can help the classification is based on the manifold assumption , which is a common assumption in SSL [45].
It states that the input space consists of multiple low-dimensional manifolds, and data points lying on the same manifold have the same labels.
This assumption helps alleviate the curse of dimensionality and may explain the effectiveness of using k NN with few labeled data.
Due to the fact that our training process does not use labeled data, our method does not rely on other common assumptions in SSL like the smooth assumption and the low-density assumption .
The facts that NPC-LV makes very few assumptions about datasets and that compressors are data-type-agnostic make this framework extensible to other data types beyond images.
For example, the combination of an autoregressive model (e.g., character recursive neural network) and arithmetic coding [46] can be used in our framework for sequential data.
Related Work
Non-parametric learning with Information Distance :
Bennett et al.
[29] propose information distance as a universal metric, based on which several papers [47, 48] propose more fine-grained distance metrics.
Chen et al.
[24], Li et al.
[31], Cilibrasi and Vitányi [32] derive more practical distance metrics based on real-world compressors.
Empirical results [22, 23, 24, 25] show that even without any training parameters, those compressor-based distance metrics can produce an effective distance matrix for clustering and classification on time series datasets.
Cilibrasi and Vitányi [32] further push this direction to more types of datasets, including images that are represented in “#” (black pixel) and “.”
(white pixel).
We unify previous work in the NPC framework, expand it to real image datasets and leverage it with neural compressors.
Compression: Shannon [26] establishes source coding theorems, showing that entropy rate is the limit of code rate for lossless compression.
Huffman Coding [49] achieves the optimal symbol code whose length is upper-bounded by H ( x )
+ 1 per symbol.
The 1 bit overhead is due to the fact that − log p ( x ) is not always an integer.
Stream coding techniques like AC [38] and ANS [27] further optimize by representing the whole message with numeral systems.
These entropy coders then can be combined with probabilistic modeling using neural networks [46, 50, 51] and used in our framework.
Semi-Supervised Learning with VAE: The evaluation paradigm in this paper is closest to SemiSupervised Learning (SSL).
Kingma et al.
[52] design two frameworks for utilizing autoencoders in downstream classification tasks.
The first (M1) is to train a tSVM [53] with latent representation output by a trained VAE.
The second (M2) is to train a VAE with labels as another latent variable.
M1 only requires a standard VAE but tSVM suffers from optimization difficulty [54], making it hard to be generally applicable for VAE.
More recent VAE-based methods [55, 56] are built on M2.
These methods don’t train a generative model in an unsupervised way like we do.
[Table]
Param” refers to parameters specifically for supervised training.
Few-Shot Learning: Similar to our setting, FSL also targets the low labeled data regime.
Many previous papers [4, 5, 6, 7] on FSL are based on meta-learning, where the model is fed with an extra labeled support set, in addition to the target dataset.
Another line of work [8, 9, 10, 11, 12, 13] utilize data augmentation.
Although some of them do not require extra datasets [8, 9, 10], the augmentation algorithms can hardly be applied to every other dataset [57].
Metric-based methods [58] utilize distance metrics for FSL.
But instead of modeling the probability distribution of a dataset, they model the “distance” between any pair of data points with a neural network, which still requires many labeled data during “pre-training”.
Our work is similar to metric-based methods in that both have the essence of nearest-neighbor.
The difference is that in the “pre-training” stage, our model is not trained to learn the distance but to reconstruct the image as all standard generative models do.
More importantly, we use no labeled data in this stage.
Experiments
We compare our method with supervised learning, semi-supervised learning, non-parametric learning and traditional Non-Parametric learning by Compression (NPC) on MNIST, FashionMNIST and
CIFAR-10 [59, 60, 61].
For each dataset, we first train a hierarchical latent generative model with unlabeled training sets.
During the stage of calculating the distance metric using compression, we pick 1,000 samples from the test set, due to the cost of compression and pair-wise computation, together with n = { 5 , 10 , 50 } labeled images per class from the training set.
We also report the result for n = 50 although it is beyond our setting.
We keep the selected dataset the same for every method compared.
We use bold to highlight the cases we outperform supervised methods, use underline to highlight the case we outperform SSL, and use italic to highlight the highest accuracy among all methods for reference.
We perform compression with two coding schemes (BB-ANS and Bit-Swap) and also use nELBO for compressed length directly.
Comparison with Supervised Learing: Supervised models are trained on 10 n labeled data.
In Table 1, when n = 50 , CNN and VGG surpass NPC-LV.
In the cases where the number of labeled data is extremely limited (e.g., 5 , 10 labeled data points per class), however, the BB-ANS variant
[Table]
outperforms all methods in every dataset.
For MNIST, both BB-ANS and Bit-Swap produce more accurate results than supervised methods on 5-shot experiments; BB-ANS performs slightly better than the supervised methods in the 10-shot scenario.
For FashionMNIST, all three variants outperform in 5-shot, 10-shot, and even 50-shot settings.
For CIFAR-10, given 10 labeled data points per class,
NPC-LV surpasses the accuracy of CNN by 27.2% and improves the accuracy of VGG by 21.2%.
This enhancement is more significant in the 5-shot setting: NPC-LV improves the accuracy of CNN by 50.9% and by 59.0% for VGG.
In general, we can see as the labeled data become fewer, NPC-LV becomes more advantageous.
Comparison with Semi-Supervised Learning: The input of NS-FSL is similar to SSL in that both unlabeled data and labeled data are involved.
The difference lies in the fact that (1) our training doesn’t use any labeled data and is purely unsupervised; (2) we only need to train the model once , while SSL needs to retrain for different n ; (3) we use fewer labeled data points, which is a more practical setting for real-world problems.
We choose strong semi-supervised methods that make few assumptions about the dataset.
We use consistency regularization methods instead of pseudo-labeling ones as pseudo-labeling methods often assume that decision boundary should pass through lowdensity region of the input space (e.g., Lee [62]).
Specifically, we choose MeanTeacher (“MT”) [63] and VAT [64].
The core of both is based on the intuition that realistic perturbation of data points shouldn’t affect the output.
We train both models with n labeled samples per class together with an unlabeled training set (training details are shown in Appendix C).
As we can see, NPC-LV achieves higher accuracy for CIFAR-10 in the low data regime, has competitive result on FashionMNIST, and is much lower on MNIST.
The strength of our method is more obvious with more complex datasets.
It’s a surprising result because we do not implement any data augmentation implicitly or explicitly unlike consistency regularization methods, which utilize data perturbation and can be viewed as data augmentation.
It’s worth noting that on all three datasets our method using BB-ANS always outperforms at least one semi-supervised methods on the 10-shot setting, indicating that SSL methods trade “universality” for “performance” while our method is more like a baseline.
Speed-wise, NPC-LV only requires training once for the generative model, and can run k NN on different shots ( n ) with no additional cost.
In contrast, SSL methods require the whole pipeline to be retrained for every n .
Comparison with Non-Parametric Learning: In this experiment, we explore the effectiveness of using latent representations directly with k NN.
We train the same generative model we use in
NPC-LV (“Hier”), as well as a vanilla VAE with a single latent variable (“Single”).
Table 1 shows that the latent representation of the vanilla one is not as expressive as the hierarchical one.
Although the latent representation using the hierarchical architecture performs reasonably well and surpasses supervised methods in the 5-shot setting on MNIST and FashionMNIST, it’s still significantly lower than NPC-LV in all settings.
The result suggests that NPC-LV can utilize trained latent variable models more effectively than simply utilizing latent representations for classification.
Comparison with NPC: We investigate how Non-Parametric learning by Compression (NPC) with non-neural compressors perform with different distance metrics.
We evaluate with NCD, CLM, and
CDM as distance metrics, and gzip, bz2, lzma, WebP, and PNG as compressors, using 1,000 images from the test set and 100 samples per class from the training set.
The result is shown in Table 2.
For distance metrics, we can see CLM and CDM perform similarly well
but it’s not clear under what circumstances a distance metric is superior to the rest.
For compressors, both Bit-Swap and BB-ANS perform much better than other compressors, indicating that generative-model-based compressors can significantly improve NPC.
BB-ANS turns out to be the best compressor for classification on all three datasets.
Analyses and Discussion 6.1 nELBO as Compressed Length
As we’ve shown in Section 3.2, nELBO can be viewed as the expected length of the compressed bitstream N − n extra .
Thus, theoretically it can be used directly to approximate compressed length.
In this way, we don’t need to apply ANS to VAE for the actual compression, which largely simplifies the method and boosts speed.
However, as we can see in Table 1, using nELBO doesn’t always perform better than an actual compressor like BB-ANS.
This may be because nELBO in a well-trained model regards the aggregation of two images as out-of-distribution data points; while the discretization in the actual compressor forces close probability with a certain level of precision to be discretized into the same bin, lowering the sensitivity.
Better aggregation strategies need to be designed to mitigate the gap.
6.2 Performance Gain and Task Difficulty
[Table]
We find that our method is more advantageous on more complex datasets with a lower shot number in terms of the relative performance.
Specifically, we average all methods’ accuracy that NPC-LV compared across different shot settings and datasets, denoted as ¯ a .
We then calculate the excess rate with BB-ANS variant’s accuracy b by b − ¯ a ¯ a × 100% .
The result is shown in Table 3.
The lower left part of the table represents higher task difficulty.
As the shot number decreases (from right to left) and/or as the difficulty of the dataset increases (from top to bottom), our framework’s performance enhancement gets higher.
6.3 Bitrate versus Classification Accuracy
7 Bitrate 0.
A cc u
[Figure]
The origin of the NPC framework comes from the intuition that the length of x after being maximally compressed by a real-world compressor is close to K ( x ) .
Theoretically, the closer this length approximates the minimum length of the expression ( C ( x ) ≈ K
( x ) ), the closer the compressor-based distance metrics are to the normalized information distance .
We investigate, empirically, whether the bitrate actually reflects the classification accuracy.
We plot bitrate versus classification accuracy for each compressor in Table 2 on three datasets as shown in Figure 3.
We use the net bitrate, which is ( N − n extra ) /d , where N is the length of the compressed bitstream, n extra is the length of the extra bits, and d is the number of pixels.
As we can see, a very strong monotonic decreasing correlation between bitrate and accuracy emerges, with Spearman coefficient [65] r s = − 0 .
96 , meaning the lower the bitrate is, the higher the classification accuracy is.
This means the correlation between bitrate and classification accuracy holds empirically regardless of datasets.
It will be interesting to investigate in the future whether the correlation remains for lossy compression.
6.4 Parallelization and Limitation
The training of generative models can be parallelized using modern GPUs.
Compression, however, is not easy to parallelize.
The calculation of CDF and PDF are parallelizable for common probability distributions like Gaussian distribution, but the ANS algorithm is not trivial to parallelize.
Fortunately, efficient implementations for ANS on GPUs have been developed [66] to exploit GPU’s parallelization power.
During distance computation stages, as only pair-wise distances are needed, we can use multiple threads to accelerate the computation.
For classification, we show that NPC-LV performs well in the low labeled data regime, where the complexity of computation may not be a concern, yet the complexity of O ( n 2 ) may still hinder applications involving pair-wise distance computations like clustering on large datasets, unless we exploit parallelization for compression.
Conclusion
In this paper, we propose a learning framework, Non-Parametric learning by Compression with Latent
Variables (NPC-LV), to address a common learning scenario, Non-Supported Few-Shot-Learning (NS-FSL).
This framework is versatile in that every module is replaceable, leading to numerous variations.
We use image classification as a case study to demonstrate how to use a trained latent generative model directly for downstream classification without further training.
It outperforms supervised learning and non-parametric learning on three datasets and semi-supervised learning on CIFAR-10 in the low data regime.
We thus regard it as a baseline in NS-FSL.
The equivalence between optimizing latent variable models and achieving the shortest code length not only shows how nELBO can be used for classification, but also indicates that improvements in latent probabilistic models can benefit neural compressors.
The relationship between compression rate and classification accuracy suggests that improvements in neural compressors can further benefit classification.
Thus, an enhancement of any module in this chain can boost classification accuracy under this framework.
Acknowledgments
This research was supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada, and in part by NERC OGP0046506 and Leading Innovative and Entrepreneur teams program of Zhejiang, number 2019R02002.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See abstract and the end of the introduction.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix I. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Although we point to the reference for some unimportant proofs.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix C. (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Sectionand Appendix C. (b) Did you mention the license of the assets?
[Yes] In both Section 5 and Appendix C. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] The only new asset is our own code and we provide in supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] Datasets are public available with MIT license.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
There are no personal or offensive content in MNIST, FashionMNIST and CIFAR10.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
No crowdsourcing or human subjects involved.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
No crowdsourcing or human subjects involved.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] No crowdsourcing or human subjects involved.
A Ablation Study
BB-ANS (Alice: compress) initial extra bits decode with decode with encode with encode with encode with
BB-ANS (Bob: decompress) received bitstream decode with decode with decode with encode with encode with
Bit-Swap (Alice: compress) initial extra bits decode with encode with decode with encode with encode with
Bit-Swap (Bob: decompress) received bitstream decode with decode with encode with decode with encode with
Number of Latent Variables: The difference between how BB-ANS [36] and Bit-Swap [43] compress is shown in Figure 4.
We can see these two compressors are only different when there is more than one latent variable, and by utilizing encoded bitstreams from previous latent variables,
Bit-Swap can reduce the cost of initial bits significantly.
We carry out experiments on CIFAR-10 and keep the aggregation method constant.
We can see in Table 4, Bit-Swap performs better with more latent variables.
But Bit-Swap with eight latent variables is still worse than BB-ANS with two.
Aggregation Methods: We also evaluate how different aggregation methods affect the classification accuracy, shown in the right hand side of Table 4.
We aggregate two images by “average”, “‘minimum”, “maximum”, “concatenation”, and “greyscale+average”.
For “concat”, the actual operation is that we compress one image after another in a way that the compressed bitstream of the first image could be used as the “extra” bits for the second image.
“gs+avg” means we use greyscale of the image and then average pixel values.
Obviously, “greyscale” is not a way of aggregation, but we notate it in the table this way for simplicity and comparison, and also to stress that this operation is only used during compression.
That is, the generative model is trained on the original images instead of on greyscale images.
By applying simple image processing methods during compression, we find that combining aggregation with image manipulation can be effective as no compressor needs to be changed, and thus no retraining for generative models is needed.
We also plot their bitrate vs. classification accuracy for various aggregation methods.
Figure 5 shows “concat” for images is an outlier - lower bitrate with low accuracy.
We will show in Appendix B that “concat” disqualifies
BB-ANS and Bit-Swap
from being a normal compressor.
Please note the special case of “concat” doesn’t apply to other compressors like gzip which treat images as bytes in the first place.
[Figure]
Bit-Swap [43] achieves the new state of the art bitrate but why doesn’t it surpass BB-ANS for classification?
One of the reasons that Bit-Swap achieves a better compression rate than BB-ANS is because Bit-Swap requires fewer initial bits,
details shown in Appendix
F. However, initial bits can only be amortized when multiple data points need to be compressed.
But in our application, at most two data points need to be compressed sequentially.
This leads us to choose the net bitrate, which excludes the length of the initial bits, and makes Bit-Swap less advantegeous.
Theoretically, net bitrate should be the same for both methods.
However, empirically we can see in Figure 5, Bit-Swap uses slightly more net bits than BB-ANS.
Although we cannot draw a definite conclusion, it appears that the more a compressor can compress, the more accurate of classification NPC can obtain with the compressor.
We can see that given a compressor, there is a correlation between bitrate and accuracy even with different aggregation methods, except for aggregation methods that alter a compressor to be not normal (e.g., “concat’).
Other Alternatives: In this work, we don’t go thoroughly through all the state-of-the-art compressors, but only focus on the VAE-based lossless compressors.
For this specific category, there are already
[Table]
numerous important factors: the choice of architectures, the choice of the number of latent variables, the choice of the hierarchy topology (e.g., asymmetrical tree structure or symmetrical one) and the choice of discretization method.
Beyond this line of compressors, deep learning based compressors discussed in Section 3.3 can also be used under our framework.
Beyond compressors, aggregation methods also cause diverging differences in the final classification accuracy.
We cover a few basic ones but there are other non-training-required aggregation methods like linear blend operator and even completely different aggregation strategies (e.g., using conditional
VAE).
On colored real-world images like CIFAR-10, image manipulation can be another easy and effective way to improve the accuracy.
In effect, “greyscale” can be viewed as “lossy compression” and this opens up the question of how lossy compressors perform under NPC framework.
B Normal Compressor
Definition 2 ( Normal Compressor ) .
A compressor is normal if it satisfies, up to an additive O (log n ) term, where n means the maximal binary length of an element of Ω :
Idempotency: C ( xx ) =
C ( x ) and C ( ϵ ) = 0 where ϵ is the empty string 2.
Symmetry: C ( xy ) =
C ( yx ) 3.
Monotonicity: C ( xy ) ≥ C ( x ) 4.
Distributivity: C ( xy )
+ C ( z ) ≤ C ( xz )
+ C ( yz )
Definition 3 ( Metric ) .
A distance function D : Ω × Ω → R + is a metric if it satisfies the following three criteria for any x, y, z ∈ Ω , where Ω is a non-empty set, R + represents the set of non-negative real number:
Identity: D ( x, y ) =
iff x = y 2.
Symmetry: D ( x, y )
=
D ( y, x ) 3.
Triangle Inequality: D ( x, y ) ≤ D ( x, z )
+ D ( z, y )
Definition 4 ( Admissible Distance ) .
A function D : Ω × Ω → R + is an admissible distance if for every pair of objects x, y ∈ Ω , the distance D ( x, y ) is computable, symmetric and satisfies the density condition (cid:80)
y 2 − D ( x,y ) ≤ 1 .
Cilibrasi and Vitányi [32] formally prove that if the compressor is normal , NCD is a normalized admissible distance satisfying the metric inequalities, which is shown in Definition 3.
Cebrián et al.
[25] systematically evaluate how far real world compressors like gzip, bz2, PPMZ can satisfy the idempotency axiom.
Here we empirically evalute all 4 axioms on MNIST with the BB-ANS compressor.
We randomly take 100 samples and plot C ( · ) on LHS as x-axis, C ( · ) on RHS as y-axis.
For simplicity, we use one latent variable, under which BB-ANS equals Bit-Swap.
As shown in Figure 6, BB-ANS satisfies monotonicity, symmetry and distributivity.
However, it fails on the identity axiom, with C ( xx )
≈ 1 .
988 C ( x ) .
Unlike gzip, bz2, or lzma, BB-ANS doesn’t treat the concatenation of two images as a sequence of bytes but images, similar to PNG and WebP, making it hard to satisfy identity axiom.
By default, the aggregation method refers to “concatenation”.
But simple concatenation does not perform well.
Practically, BB-ANS fails the identity test when using “concatenation”.
Without changing the compressor, is it still possible to satisfy the above conditions so that NCD can be used as a normalized admissible distance metric?
We can change the aggregation method.
We investigate whether BB-ANS with average can satisfy the above conditions.
Obviously, identity and symmetry axioms can hold, as C ( avg ( x, x )) =
C ( x + x 2 ) =
C ( x ) .
We empirically evaluate monotonicity and distributivity, and find that they are both satisfied with “average”.
Figure 7 illustrates that C ( avg ( x, y )) ≥ C ( x ) and C ( avg ( x, y )) + C ( z ) ≤
C ( avg ( x, z ))
+ C ( avg ( y, z )) always hold.
Given the compressor is normal under “average” function, we now prove NCD is an admissible distance metric.
Definition 5.
Let D be an admissible distance.
D + ( x ) is defined as D + ( x ) = max { D ( x, z ) :
C ( z ) ≤ C ( x ) } , and D + ( x, y ) is defined as D + ( x, y )
=
max { D + ( x ) , D + ( y ) }
Lemma 1.
If C is a normal compressor, then E c ( x, y )
+ O (1) is an admissible distance, where
E c ( x, y ) =
C ( xy ) − min
{ C ( x ) − C ( y ) } is the compression distance.
Lemma 2.
If C is a normal compressor, then E + c ( x, y )
=
max { C ( x ) , C ( y ) }
Theorem 2.
If the compressor is normal, then the NCD is a normalized admissible distance satifying the metric (in)equalities.
Proof.
Lemma 1 and Lemma 2 show that NCD is a normalized admissible distance.
We now show how NCD satisfies the metric (in)equalities.
For identity axiom,
[Equation]
For symmetry axiom,
[Equation]
For triangle inequality, without loss of generality, we assume C ( x ) ≤ C ( y ) ≤ C ( z ) .
As NCD is symmetrical, there are three triangle inequalities that can be expressed by NCD ( x, y ) , NCD ( y, z ) , NCD ( x, z ) .
For simplicity, we prove one of them,
NCD ( x, y ) ≤ NCD ( x, z ) + NCD ( z, y ) as the procedure for the other two is similar.
Since BB-ANS is a normal compressor under “avg”, we have distributivity:
C ( avg ( x, y ))
+ C
[Equation]
z ) ≤ C
[Equation]
avg ( x, z ))
+ C ( avg ( z, y )) .
Subtracting C ( x ) from both sides and rearraging results in C ( avg ( x, y ))
− C ( x ) ≤ C ( avg ( x, z ))
− C ( x ) + C ( avg ( z, y ))
− C ( z ) .
Dividing by C ( y ) on both sides, we have
[Equation]
We know LHS ≤ 1 and RHS can be ≤ 1 or > 1 .
a) RHS ≤ 1 : Let C ( z ) =
C ( y ) + ∆ ; adding ∆ to both the numerator and denominator of RHS increases RHS and makes it closer to 1.
[Equation]
(b) RHS > 1 : The procedure
is
similar to the
case when RHS ≤
The
difference
is that adding
∆ to both numerator and denominator
makes
RHS decrease instead of increase
But RHS cannot decrease less than 1.
Thus, we still have
[Equation]
[Figure]
Identity
[Figure]
Symmetry
[Figure] [Figure]
C Training Details
For both CNN and VGG
[Figure]
we tune hyperparameters on a validation set to obtain the optimized performance.
The reason that we use VGG11 instead of VGG16 or VGG19 is because VGG performs the best in the low data regime in our experiments.
For CNN, we first normalize MNIST,
FashionMNIST and CIFAR-10.
We use batch size = 4 , epoch number = 14 , Adadelta [67] as the optimizer with learning rate = 1 , decaying learning rate by γ = 0 .
7 every step for MNIST and FashionMNIST.
Some of hyperparameters are from PyTorch’s official MNIST example.
We use the same hyperparameters except we increase the number of epochs to 20.
For VGG11 on
MNIST and FashionMNIST, we use epoch number = 20 when given 50 samples per class, and use epoch number = 40 when given less.
We use Adam [68] with learning rate = 0 . 0001 as the optimizer, and batch size = 4 .
For CIFAR-10, we use learning rate = 0 .
00001 , epoch number = 80 .
For MeanTeacher and VAT, we follow Zhang et al.
[69]’s implementation and hyperparameter settings.
We use WideResNet [70] with depth = 28 and widen factor = 2 as the architecture for all three datasets and iterate 60,000 for MNIST and FashionMNIST, and iterate 200,000 for CIFAR-10.
SGD with momentum is used for all three datasets, with learning rate = 0 .
03 , momentum = 0 . 9 .
For latent variable models used in this paper, we follow the training procedure and hyperparameters in Kingma et al.
[43] — four ‘Processing’ Residual blocks at the beginning of the inference model and the end of the generative model; eight ‘Ordinary’ Residual block in total for all latent layers in both inference model and generative model.
The Dropout [71] rate is 0 . 2 for MNIST and FashionMNIST, 0 . 3 for CIFAR-10.
The learning rate for all datasets is 0 .
002 with Adam optimizer.
Dimension of latent variables for MNIST and FashionMNIST is 1 × 16 × 16 while dimension of latent variables for CIFAR-10 is 8 × 16 × 16 .
During k NN, we use k = 2 for MNIST and FashionMNIST and k =for CIFAR-10.
We use one NVIDIA Tesla P40 GPU for training and compression.
For pairwise computation in 50-shot setting, it takes roughly ten hours to calculate distance matrix on MNIST and FashionMNIST; it takes about thirty hours for CIFAR10.
But once the distance matrix is calculated, evaluation on 5-shot or 10-shot just takes seconds.
For both CNN and VGG, it takes about half an hour to train on 50-shot for one experiment.
For VAT and MT, we need to re-train for every shot setting.
For MNIST and FashionMNIST it takes about three hours to run one experiment in a single shot setting and for
CIFAR10 it takes about twelve hours.
The time that VAT and MT take positively relate to the number of iterations, which makes them slower than our method in the 5- and 10-shot setting but faster in the 50-shot setting.
D Details of ANS
We briefly introduce ANS and show the proof of the optimal code length obtained from ANS.
The essence of ANS is to encode one or more data points into a single natural number, called state s ∈ N .
Depending on different vocabularies and manipulations, there are different variations of ANS (details can be seen in Duda et al.
We introduce one of them - rANS (range ANS), which is the variant we use in this paper.
The notation we use here is unconventional in order to be consistent with the main part of the paper.
Let’s notate our state at timestamp t as s t ∈ N , and notate our symbol/message at t as x t , x t ∈ V , where V = { 0 , 1 } is the vocabulary set.
We have two simple methods to encode a binary sequence into a natural number bit by bit — s t
= 2 s t − 1 + x t or s t =
s
t − 1 + 2 m x t .
The former means appending information to the least significant position while the latter is adding information to the most significant position.
It’s obvious that encoding a new symbol into the most significant position requires remembering m while encoding in the least significant position only needs the previous state s t − 1 and new information x t .
It’s also easy to decode: depending on whether the current state s t is even or odd, we not only know if the last encoded symbol x t is 0 or 1 , but we can also decode the state following s t −
= s t 2 or s t −
=
s t − 1 2 .
The above example illustrates encoding and decoding methods when there are two elements in the vocabulary with uniform distribution p (0)
=
p (1) = 12 .
In this case, it’s optimal to scale up s t to two for both 0 or 1 as we essentially only spend 1 bit per encoded symbol.
However, when the probability is not uniformly distributed, the entropy is smaller, and scaling up by 2 for both symbols will not be optimal anymore.
rANS generalizes the process to any discrete probability distribution and any size of vocabulary.
Intuitively, scaling up by a smaller factor for a more probable symbol and scaling up by a larger factor for a less probable symbol will provide us with a more efficient representation.
Concretely, we have a sequence of messages x
= ( x 1 , x 2 , x 3 , ..., x n ) , and a vocabulary V = { v 1 , v 2 , v 3 , ..., v k } , with size k , x i ∈ V .
We also have probability mass distribution of V : P = { p v 1 , p v 2 , p v 3 , ..., p v k } .
Correspondingly, let’s define frequency counts F = { f v 1 , f v 2 , f v 3 , ..., f v k } , f v i = p v i × M where
M = (cid:80) ki
=1 f v i .
M can be viewed as a multiplier, demonstrating the precision of ANS, which is a predefined variable in the implementation.
We can also get cumulative frequency counts from F as follows B = { b v 1 , b v 2 , b v 3 , ..., b v k } where b v i =
(cid:80) i − 1 j =1 f v j .
Now we get everything we need to define the encoding function G :
[Equation]
The procedure can be interpreted as follows:
We have various M -sized blocks partitioning natural number N .
Encoding can be viewed as finding the exact location of the natural number that represents the state, by first finding the corresponding block ( (cid:106) s t − 1 f xt (cid:107) × M ) followed by finding the subrange representing that symbol within M ( b x t ) and finding the exact location within that sub-range ( s t − 1 mod f x t ).
The decoding function H ( s t ) is the reverse of the encoding:
[Equation]
We first find the precise location within the sub-range using inverse function of cumulative counts (argmax { b x t < ( s t mod M ) } ).
With x t we can reverse steps in Equation (17) to get the previous state.
As we can see, ANS decodes in the reverse order of encoding (i.e., last in first out), which makes it compatible with bits-back argument.
From encoding function, we know that:
[Equation]
Encoding a sequence of symbols x results in:
[Equation] [Equation]
Thus, the total coding length is:
[Equation]
where s 0 refers to the initial state.
Dividing by n we will get the average coding length that approximates the entropy of the data.
E Discretization
ANS is defined for symbols in a finite alphabet; bits-back coding works for discrete latent variables.
However, continuous latent variables have proven to be powerful in many latent variable models.
In order to use those latent variable models for lossless compression, discretizing continuous variables into discrete ones is a necessary step.
Townsend et al.
[36] derives, based on MacKay [73], that using bits-back coding, continuous latent variables can be discretized to arbitrary precision without affecting the compression rate.
Suppose a probability density function p is approximated using a number of “buckets” of equal width σ z .
For any given bucket j , we can know its probability mass p ( z ( j ) )
σ z where z ( j ) is some point in the bucket j .
Let’s notate the discrete distribution as P and
Q for both prior and posterior distribution.
Then for any given bucket j , P ( j ) ≈ p ( z ( j ) )
σ z .
The expected message length with a discretized latent variable is:
[Equation]
The width of buckets σ z is cancelled.
Therefore, as long as the bins for inference models match the generative models, continuous latent variables can be discretized up to an arbitrary precision.
In this paper we only consider basic discretization techniques like dividing continuous distribution into bins with equal width or equal mass .
We discretize the prior (top layer) with equal mass and all subsequent latent layers with equal width.
As Equation (22) shows, ideally we want the discretization to align between inference models and generative models.
However, discretization of z i ∼ p θ
( z i | z i +1 ) relying on z i ∼ q ϕ
( z i |
z
i − 1 ) is not possible without sampling.
In the compression stage, when decoding z i , z i +1 is not available and so is p
θ ( z i | z i +1 ) .
In the decompression stage, similarly, q ϕ ( z i | z i − 1 ) is not available for p
θ
( z i | z i +1 ) to match with.
Therefore, we need to sample from training dataset beforehand to get unbiased estimates of the statistics of the marginal distribution [43].
This process only needs to be done once and can be saved for the future use.
F Initial Bits of BB-ANS and Bit-Swap
The main difference between BB-ANS and Bit-Swap is that BB-ANS requires the sender Alice to decode z
i +1 with q ϕ
( z i +1 |
z i ) for i from 1 to L − 1 first, and then encode z i with p θ ( z i |
z i +1 )
for i from 1 to L − 1 .
While Bit-Swap interleaves this encoding and decoding procedure and applies it recursively for latent variables, as illustrated in Figure 4.
The advantage of Bit-Swap’s procedure is that, after decoding z 1 , the bits encoded from x can be used in decoding z 2 ; then bits encoded from z 1 can be used for decoding z 3 .
As a result, the initial bits required for Bit-Swap is much less than
BB-ANS.
Concretely, for BB-ANS, the minimum initial bits required:
[Equation] [Equation]
For Bit-Swap, the minimum initial bits required:
[Equation]
The initial bits Bit-Swap required is less than BB-ANS, making Bit-Swap reach the optimal compression rate.
G Hierarchical Latent Variable Models
The hierarchical autoencoder in the paper uses deep latent Gaussian models (DLGM)
[44] following the sampling process based on Markov chains, whose marginal distributions are:
[Equation]
Combining the above equations, the marginal distribution of x is:
[Equation]
Accordingly, inference models q ϕ ( z i +
| z i ) need to be defined for every latent layer.
ELBO that includes multiple latent variables then becomes:
[Equation]
In this paper, we use Logistic distribution ( µ = 0 , σ = 1 ) as the prior p ( z L ) , and use conditional Logistic distribution for both inference models q ϕ
( z i +1 |
z i ) , q ϕ ( z 1 | x ) and generative models p θ ( z i | z i +1 ) .
These distributions are modeled by neural networks, which is stacked by Residual blocks [74] as hidden layers.
More architecture details can be referred to Kingma et al.
[43], where they also discusses other possible topologies regarding to hierarchical latent variable models.
H A Brief Proof of Universality of Information Distance
Information Distance E ( x, y ) refers to the length of the shortest binary program generated by universal prefix Turing machine, that with input x computes y , and with input y computes x .
It’s shown
that E ( x, y ) =
max { K ( x | y ) , K ( y | x ) } .
We now prove Theorem 1, based on Lemma 3 [29].
Lemma 3.
For every upper-semicomputable function f ( x, y ) , satisfying (cid:80) y 2 − f ( x,y ) ≤ 1 , we have
K ( y | x ) < f
( x, y ) .
To prove that E ( x, y ) is a metric, we show it satisfies metric (in)equalities.
We can infer the nonnegativity and symmetry directly from the definition E ( x, y )
=
max { K ( x | y ) , K ( y | x ) } .
For triangle inequality, given x, y, z , without loss of generality, let E ( x, z ) =
K ( z | x ) .
By the self-limiting property, we have
[Equation]
To prove E ( x, y ) is admissible , we show it satisfies density requirement:
[Equation]
The second inequality is due to Kraft’s inequality for prefix codes.
To prove the minimality, as for every admissible distance metric D ( x, y ) , it satisfies (cid:80) y :
y ̸
= x 2 D ( x,y ) ≤ 1 .
According to Lemma 3, we have K ( y | x ) < D ( x, y ) and K ( x | y ) <
D ( y, x ) .
I Potential Negative Social Impact
We propose a general framework for classification and clustering tasks with minimum assumption about the dataset.
We don’t foresee any negative social impact itself.
However, our framework uses generative models for their explicit density estimation and the development of generative models may be used by people with ulterior motives to generate fake data.