Few-Shot Non-Parametric Learning with Deep Latent Variable Model
Anonymous Author(s)
AffiliationAddress email
Abstract
Most real-world problems that machine learning algorithms are expected to solveface the situation with 1) unknown data distribution; 2) little domain-specific knowledge; and 3) datasets with limited annotation.
We propose Non-Parametric learningby Compression with Latent Variables (NPC-LV), a learning framework for anydataset with abundant unlabeled data but very few labeled ones.
By only training agenerative model in an unsupervised way, the framework utilizes the data distribution to build a compressor.
Using a compressor-based distance metric derivedfrom Kolmogorov complexity, together with few labeled data, NPC-LV classifieswithout further training.
We show that NPC-LV outperforms supervised methodson all three datasets on image classification in low data regime and even outperformsemi-supervised learning methods on CIFAR-10.
We demonstrate how and whennegative evidence lowerbound (nELBO) can be used as an approximate compressedlength for classification.
By revealing the correlation between compression rateand classification accuracy, we illustrate that under NPC-LV, the improvement ofgenerative models can enhance downstream classification accuracy.
Introduction
The progress of deep neural networks drives great success of supervised learning with huge labeleddatasets [1, 2, 3].
However, large labeled datasets are luxurious in many applications and hugeamounts of training parameters make the model easy to overfit and hard to generalize to other datasets.
The urge to learn with small labeled dataset prompts Few-Shot Learning.
However, most few-shotclassification settings require either an auxiliary “support set” [4, 5, 6, 7] that contains c classes,each has k samples ( c -way k -shot); or prior knowledge about the dataset, where data augmentationcan be performed within the same dataset [8, 9, 10] or from other weakly-labeled/unlabeled/similardatasets [11, 12, 13].
This setting is not widely applicable to every dataset in practice, as it requireseither an elaborate construction of additional “support set” or augmentation algorithms tailored tospecific datasets.
Pre-trained models, on the other hand, do not require adhoc “support” and areproved to be good at few-shot learning [14] and even zero-shot learning [15].
However, thousands ofmillions of training parameters make the model hard to be retrained but only fine-tuned.
When thedata distribution is substantially different from any datasets used in pre-training, the inductive biasfrom pre-training holds up fine-tuning, making the model less pliable [16].
Goals from the above learning paradigms can be summarized as to design algorithms that can beapplied to any dataset, and can learn with few labeled data, ideally with no training.
“No Free
Lunch” [17] implies that it’s impossible to have an algorithm that is both “universal” and “best”.
But how good can a “universal” algorithm be, especially in the low data regime, with no externaldata resources?
Specifically, we are interested in a new setting, Non-Supported Few-Shot Learning(NS-FSL), defined as follows:
Given any target dataset D = ( x 1 , x 2 , ..., x n ) belonging to c classes.
For each class, we have k labeled samples (1 ≤ k ≤ 10) .
The remaining n − ck unlabeled samples need to be classified into cclasses without the need of support sets, any other datasets or training parameters.
This setting is similar to semi-supervised learning’s but excludes labeled information in training.
Raviand Larochelle [18] demonstrate that it’s hard to optimize a neural network when labeled data is scarce.
In order to make minimum assumption about labeled data, we aim at using parameter-free methods.
The goal is to grasp the data-specific distribution p ( x ) , with minimal premises on conditionalprobability distribution p ( y | x ) .
Deep generative models with explicit density estimation are perfectcandidates for this goal.
The problem then becomes: given trained generative models, how to take fulladvantage of the information obtained from them for classification?
Using latent representation onlyutilizes p ( z | x ) , which just includes partial information.
Even for those latent generative models thatdo not suffer from posterior collapse [19], p ( z | x ) ’s insufficiency for classification with non-parametricmethods like k -nearest-neighbor is shown in both previous work [20, 21] and our experiments.
Inspired by previous work that uses compressor-based distance metrics for non-parametric learning [22, 23, 24, 25], we propose Non-Parametric learning by Compression with Latent Variables(NPC-LV), a learning framework that consists of deep-generative-model-based compressors andcompressor-based distance metrics.
It leverages the power of deep generative models without exploiting any label information in the probabilistic modeling procedure.
With no further training, thisframework can be directly used for classification.
By separating probabilistic modeling from downstream tasks that require labels, we grasp the unique underlying structures of the data in every dataset,and further utilize these structures in downstream tasks with no parameter tuning needed.
We viewthis learning framework as a baseline in this setting, for any dataset.
We argue it is “parameter-free”as there is no parameter involved in classification stage for labeled data.
Basically it means training agenerative model as is and getting a classifier for free.
Our contributions are as follows: (1) We frame the existing methods into a general learning framework
NPC, based on which, we derive NPC-LV, a flexible learning framework with replaceable modules.(2)
We use NPC-LV as a baseline for a common learning scenario NS-FSL with neither supportsets nor further training.
(3) Our method outperforms supervised methods by up to 11.8% on
MNIST, 18.0% on FashionMNIST, 59% on CIFAR-10 on image classification in low-data regime.
It outperforms non-parametric methods using latent representation on all three datasets.
It evenoutperforms semi-supervised learning methods on CIFAR-10.
(4) We show how negative evidencelowerbound (nELBO) can be used for classification under this framework.
(5) We find the correlationbetween bitrate and classification accuracy.
This finding suggests the improvement in the domain ofdeep-learning-based-compressor can further boost classification accuracy under this framework.
Background2.1 Information Theory — Data Compression
In a compression scenario, suppose we have a sender Alice and a receiver Bob .
Alice wants tosend a message that contains a sequence of symbols x
= ( x 1 , x 2 , ..., x n ) to Bob .
The ultimate goalof the lossless compressor is to compress x into the minimum amount of bits x ′ that can later bedecompressed back to x by Bob .
To achieve the shortest compressed length, shorter codes are assignedto symbols with higher probability.
According to Shannon’s Source Coding Theorem [26], this lengthof bits is no shorter than the entropy of the sequence, whose definition is H ( x ) ≜ E [ − log p data ( x )] ,where p data ( x ) represents the probability distribution of each symbol in the sequence.
Instead ofcoding symbols one by one, stream code algorithms like Asymmetric Numeral Systems (ANS)
[27]convert x to a sequence of bits x ′ and reaches this optimal code length for the whole sequence withoverhead of around 2 bits, given p data ( x ) .
However, the “true” probabilistic distribution p data ( x ) isunknown to us.
We can only access samples and approximate it with p θ ( x ) .
That is:
[Equation]
Given an entropy coding scheme, the better p θ ( x ) approximates p data ( x ) , the closer we can get to theminimum code length.
Modeling p θ ( x ) is where deep generative model with density estimation canhelp.
Possible coding schemes and generative models for compressors are discussed in 3.2.
2.2 Algorithmic Information Theory — Kolmogorov Complexity and Information Distance
While information theory is built on data distribution, algorithmic information theory considers“single” objects without notion of probability distribution.
Kolmogorov complexity K ( x ) [28] is usedto describe the length of the shortest binary program that can produce x on a universal computer, whichis the ultimate lower bound of information measurement.
Similarly, the Kolmogorov complexity of xgiven y is the length of the binary program that on input y outputs x , denoted as K ( x | y ) .
Based on
Kolmogorov complexity, Bennett et al.
[29] derive information distance E ( x, y ) :
[Equation]
The idea behind this measurement, on a high level, is that the similarity between two objects indicatesthe existence of a simple program that can convert one to another.
The simpler the converting programis, the more similar the objects are.
For example, the negative of an image is very similar to theoriginal one as the transformation can be simply described as “inverting the color of the image”.
Theorem 1.
The function E ( x, y ) is an admissible distance and a metric.
It is minimal in the sensethat for every admissible distance D , we have E ( x, y ) ≤ D ( x, y )
+ O (1) .
Intuitively, admissible distance refers to distances that are meaningful (e.g., excluding metrics like
D ( x, y ) = 0 .
3 for any x ̸ = y ) and computable (formal definition is in Appendix B).
Combiningthose definitions, we can see Theorem 1 means E ( x, y ) is universal in a way that it is optimal andcan discover all effective similarities between two objects (proof is shown in Appendix H).
In order to compare the similarity, relative distance is preferred.
Li et al.
[30] propose a normalizedversion of E ( x, y ) called Normalized Information Distance (NID).
Definition 1 ( NID ) .
NID is a function: Ω × Ω → [0 , 1] , where Ω is a non-empty set, defined as:
[Equation]
Equation (3) can be interpreted as follows:
Given two sequences x , y , K ( y ) ≥ K ( x ) :
[Equation]
where I ( x : y ) =
K ( y ) − K
( y | x ) means the mutual algorithmic information .
I ( x : y ) K ( y ) means theshared information (in bits) per bit of information contained in the most informative sequence, and
Equation (4) here is a specific case of Equation (3).
Theoretically, NID is a desirable distance metricas it satisfies the metric (in)equalities (definition in Appendix B) up to additive precision O (1 /K ( · ))where K ( · ) is the maximum complexities of objects involved in (in)equalities (proof shown in [31]).
Non-Parametric learning by Compression with Latent Variables
Non-Parametric learning by Compression (NPC) consists of three modules — a distance metric, acompressor and an aggregation method shown in Figure 1.
NPC-LV leverages NPC by includingneural compressors based on deep generative models.
We introduce the derivation of compressorbased distance metrics in Section 3.1; the generative-model-based compressor in Section 3.2; anintegration of this framework with generative models in Section 3.3.
3.1 Compressor-based Distance Metric
Universal as NID is, it is uncomputable, because Kolmogorov complexity is uncomputable.
Cilibrasiand Vitányi [32] propose Normalized Compression Distance (NCD), a quasi-universal distance metricbased on real-world compressors.
In this context, K ( x ) can be viewed as the length of x afterbeing maximally compressed.
Suppose we have C ( x ) as the length of compressed x produced by areal-world compressor, then NCD is defined as:
[Equation]
[Figure]
The better the compressor is, the closer NCD approximates NID.
With a normal compressor (discussedin details in Appendix B), NCD has values in [0,1] and satisfies the distance metric (in)equalitiesup to O (log n/n ) where n means the maximum binary length of a string involved [33].
NCD isthus computable in that it not only uses compressed length to approximate K ( x ) but also replacesconditional Kolmogorov complexity with C ( xy ) that only needs a simple concatenation of x, y .
Liet al.
[31] simplify NCD by proposing another compression-based dissimilarity measurement (CDM):
[Equation]
Chen et al.
[24] use another variation ranging from 0 to 1:
[Equation] [Equation]
NCD, CDM and CLM are different variations of Kolmogorov based distance metrics.
We empiricallyevaluate their performance in Section 5.
3.2 Trained Generative Models as Compressors
[Figure] [Figure]
: decompress) received
[Figure]
with
[Figure]
with
Previous works [24, 29, 32] demonstrate the success of compression-based distance metrics insequential datasets like time series, DNA, andtexts using non-neural compressors like gzip,bzip2, PPMZ.
Deep-generative-model–basedcompressors can take NPC to the next level byexpanding to more data types using better compressors.
We mainly focus on variational autoencoder (VAE) based compressors with briefintroduction to other neural compressors.
VAE Family:
The relation between VAE and “bits-back” has been revealed in multiple previouswork [34, 35].
Townsend et al.
[36] use latent variable models by connecting ANS to “bits-back”argument [37] (BB-ANS).
In the setting of “bits-back argument”, we assume Alice has some extrabits of information to send to Bob alongside with x .
It’s also assumed that both Alice and Bob haveaccess to p ( z ) , p θ ( x | z ) and q ϕ ( z | x ) where z is the latent variable; p ( z ) is the prior distribution of z ;p θ ( x | z ) represents a generative network and q ϕ ( z | x ) represents an inference network.
As shown inthe Figure 2, Alice first decodes those extra information according to q ϕ ( z | x ) to generate a sample z .
2z is further used to encode x with p θ ( x | z ) and z itself is encoded using p ( z ) .
Bob then reverses thisprocedure and recovers the extra bits by encoding with q ϕ ( z | x ) .
For a single data point, the length ofthe final bitstream is:
[Equation]
We can see the expectation of N − n extra is equal to the negative evidence lower bound (nELBO):
[Equation]
ELBO above is derived from “bits-back argument” in the context of compression.
Now, from theperspective of latent variable model like VAE, the derivation often starts from the fact that p θ ( x ) = 158 (cid:82) p θ ( x | z ) p ( z ) is intractable.
q ϕ ( z | x ) is then introduced as an inference model to approximate p ( z | x )in order to work around the intractability problem, which brings up the marginal log likelihood:
[Equation]
We only need to optimize the lower bound, as minimizing nELBO means maximizing log p θ ( x ) —the likelihood of generating real data and minimizing KL divergence between q ϕ ( z | x ) and p ( z | x ) atthe same time, which is the same objective function from what we derive using “bits-back”.
This equivalence demonstrates that an optimized latent variable model can be used directly forcompression as, from the data compression perspective, it minimizes the code length attainable bybits back coding using the model.
With the help of ANS, we can encode symbols into bitstreamsor decode bitstreams back to symbols with trained latent variable models.
Details of ANS anddiscretizing continuous latent variables are shown in Appendices D and E.
ARM: Autoregressive models (ARM) model p ( x ) as: p ( x ) = p ( x 0 ) (cid:81) n i =1 p ( x i | x i − 1 ) .
The exactlikelihood estimation makes it capable of lossless compression.
But instead of using ANS, whichis a stack-like coding scheme, queue-like ones (e.g., Arithmetic Coding (AC) [38]) should beused.
Computational inefficiency is the main drawback of ARM such as RNN [39], but causalconvolutions [40, 41] can alleviate the problem.
IDF: Integer Discrete Flow (IDF) [42] can also optimize towards the exact log-likelihood.
Similar toother flow-based models
, it utilizes invertible transformation f , but works on discrete distributionwith additive coupling and rounding operation.
For IDF, ANS can be used as the entropy coder.
Our experiments and discussion are mainly about VAE-based compressors as their architectures canbe flexibly changed under the “bits-back” argument.
3.3 NPC-LV
We’ve shown in Section 3.2 that we can plug in any trained latent variable model in exchange fora near optimal compressor under the framework of BB-ANS.
To show that the coding scheme isreplaceable, we introduce a coding scheme variation, Bit-Swap [43], in the following experiments.
The difference between BB-ANS and Bit-Swap is the encoding and decoding order when there aremore than one latent variable.
A detailed comparison is shown in
Appendix F.
The generative modelwe use for both two is a hierarchical latent generative model (details are in Appendix G), also knownas Deep Latent Gaussian Model (DLGM)
Algorithm 1 NPC-LV (use VAE and NCD as an example)
Input: k , D test , D train =
{ D U train , D L train } , D L train =
{ X L , y L }
trained_vae = trainVAE
( D U train ) vae_compressor = ANS(trained_vae) for x test in D test do
C_ x test = len(vae_compressor( x test )) distances =
[] for x train in X L do // Calculate NCD distance with C ( x ) , C ( y ) , and C ( ψ ( x, y ))
C_ x train = len(vae_compressor( x train ))
C_agg = len(vae_compressor( aggregate( x test , x train ))
NCD =
(C_agg − min{C_ x test , C_ x train }) / max{C x test , C x train } distances = push distances) end for k indicies argsort distances
] y majority indicies } ) end for
Aggregation:
In addition to coding schemes and probabilistic models for compressors, aggregationmethods can also be replaced.
Previous works [32] assume xy in C ( xy ) means the “concatenation”of two inputs.
We expand this assumption to a more general case where “aggregation” can be otherkind of aggregation function, represented as C ( ψ ( x, y )) in Figure 1.
We prove the legitimacy of thisgeneralization as changing aggregations methods may make compressor-based distance metrics notadmissible (shown in Appendix B).
More sophisticated strategies of aggregation are left to the futurework.
We also discuss other replaceable modules in details in Appendix A.
We use BB-ANS with NCD as a concrete instance to demonstrate this framework on a classificationtask shown in Algorithm 1.
D U train and D L train mean unlabeled and labeled training set; predefinedfunctions are in teal.
The algorithm can be simplified into four steps: 1) Train a VAE on the unlabeledtraining dataset; 2) Apply ANS with discretization on the trained VAE to get a compressor; 3)
Calculate the distance matrix between pairs ( x test , x train ) with the compressor and NCD; 4) Runk -Nearest-Neighbor( k NN) classifier with the distance matrix.nELBO as estimated compressed length: Specifically for VAE-based compressors, nELBO is theestimated length of the compressed bitstream as Equation (9) shows.
Therefore, we can use it directlywithout actual compressing.
This can further simplify our method as we don’t need to discretizecontinuous distribution or apply entropy coder.
The reason why the underlying data distribution can help the classification is based on manifoldassumption , which is a common assumption in SSL [45].
It states that the input space consists ofmultiple low-dimensional manifolds, and data points lying on the same manifold have same labels.
This assumption helps alleviate the curse of dimensionality and may explain the effectiveness ofusing k NN with few labeled data.
Due to the fact that our training process does not use labeleddata, our method does not rely on other common assumptions in SSL like smooth assumption andlow-density assumption .
The facts that NPC-LV makes very few assumptions about datasets andthat compressors are data-type-agnostic make this framework extensible to other data types beyondimages.
For example, a combination of an autoregressive model (e.g., character recursive neuralnetwork) and arithmetic coding [46] can be used in our framework for sequential data.
Related Work
Non-parametric learning with Information Distance : Bennett et al.
[29] propose informationdistance as a universal metric, based on which, several papers [47, 48, 49] propose more fine-graineddistance metrics.
Chen et al.
[24], Li et al.
[31], Cilibrasi and Vitányi [32] derive more practicaldistance metrics based on real-world compressors.
Empirical results [22, 23, 24, 25] show that evenwithout any training parameters, those compressor-based distance metrics can produce effectivedistance matrix for clustering and classification on time series datasets.
Cilibrasi and Vitányi [32]further push this direction to more types of datasets including images that are represented in “#”(black pixel) and “.”
(white pixel).
We unify previous work in NPC framework, expand it to realimage datasets and leverage it with neural compressors.
Compression: Shannon [26] establishes source coding theorems, showing that entropy rate is thelimit of code rate for lossless compression.
Huffman Coding [50] achieves the optimal symbol codewhose length is upper-bounded by H ( x )
+ 1 per symbol.
The 1 bit overhead is due to the fact that− log p ( x ) is not always an integer.
Stream coding like AC [38] and ANS [27] further optimize byrepresenting the whole message with numeral systems.
Those entropy coders then can be combinedwith probabilistic modeling using neural network [46, 51, 52] and used in our framework.
Semi-Supervised Learning with VAE: The evaluation paradigm in this paper is closest to SemiSupervised Learning (SSL).
Kingma et al.
[53] design two frameworks for utilizing autoencodersin downstream classification tasks.
The first (M1) is to train a tSVM [54] with latent representationoutput by a trained VAE.
The second (M2) is to train a VAE with label as another latent variable.
M1only requires a standard VAE but tSVM suffers from optimization difficulty [55], making it hard tobe generally applicable for VAE.
Later VAE-based methods [56, 57] are built on M2.
These methodsdon’t train a generative model in an unsupervised way like we do.
Few-Shot Learning: Similar to our setting, FSL also targets at low labeled data regime.
Largeamounts of previous works [4, 5, 6, 7] on FSL are based on meta-learning, where the model is fed withan extra labeled support set, in addition to the target dataset.
Another line of work [8, 9, 10, 11, 12, 13]
[Table]
refers to parameters specifically for supervised training.
utilize data augmentation.
Although some of them do not require extra dataset [8, 9, 10], theaugmentation algorithms can hardly be applied to every other dataset [58].
Metric-based methods [59]utilize distance metrics for FSL.
But instead of modeling the probability distribution of a dataset, theymodel the “distance” between any pair of data points with neural network, which still requires manylabeled data during “pre-training”.
Our work is similar to metric-based methods in that both havethe essence of nearest-neighbor.
The difference is that in the “pre-training” stage, our model is nottrained to learn the distance but to reconstruct the image as all standard generative models do.
Moreimportantly, we use no labeled data in this stage.
Experiments
We compare our method with supervised learning, semi-supervised learning, non-parametric learningand traditional NPC on MNIST, FashionMNIST and CIFAR-10 [60, 61, 62].
For each dataset, wefirst train a hierarchical latent generative model with unlabeled training sets.
During the stage ofcalculating distance metric using compression, we pick 1,000 samples from test set, due to the costof compression and pair-wise computation, together with n = { 5 , 10 , 50 } labeled images per classfrom training sets.
We also report the result for n = 50 although it is beyond our setting.
We keepthe selected dataset same for every method compared with.
We use bold to highlight the cases weoutperform supervised methods; use underline to highligh the case we outperform SSL and use italicto highlight the highest accuracy among all methods for reference.
We do actual compression withtwo coding schemes (BB-ANS and Bit-Swap) and also use nELBO for compressed length directly.
Comparison with Supervised Learing: Supervised models are trained on 10 n labeled data.
In Table 1, when n = 50 CNN and VGG surpass NPC-LV.
In the cases where the number of labeled datais extremely limited (e.g., 5 , 10 labeled data points per class), however, BB-ANS variant outperformsall methods in every dataset.
For MNIST, both BB-ANS and Bit-Swap produce more accurate resultsthan supervised methods on 5-shot experiments; BB-ANS performs slightly better than supervisedmethods in the 10-shot scenario.
For FashionMNIST, all three variants outperform in 5-shot, 10-shot,and even 50-shot settings.
For CIFAR-10, given 10 labeled data points per class, NPC-LV boosts theaccuracy of CNN by 27.2% and improves the accuracy of VGG by 21.2%.
This enhancement is more
[Table]
significant in 5-shot setting: NPC-LV improves the accuracy of CNN by 50.9% and by 59.
% for
In general, we can see as the labeled data become fewer,
LV becomes more advantageous.
Comparison with Semi-Supervised Learning: The input of NS-FSL is similar to SSL in that bothunlabeled data and labeled data are involved.
The difference lies in the fact that 1) our training doesn’tuse any labeled data and is purely unsupervised; 2) we only need to train the model once , while SSLneed to retrain for different n ; 3) we use fewer labeled data point, which is a more practical settingfor real-world problems.
We choose strong semi-supervised methods that make little assumptionabout the dataset.
We use consistency regularization methods instead of pseudo-labeling ones aspseudo-labeling methods often assume that decision boundary should pass through low-densityregion of the input space (e.g., Lee et al.
Specifically, we choose MeanTeacher (“MT”)
[64]and VAT [65].
The core of both is based on the intuition that realistic perturbation of data pointsshouldn’t affect the output.
We train both models with n labeled samples per class together with anunlabeled training set (training details are shown in Appendix C).
As we can see, NPC-LV achievehigher accuracy for CIFAR-10 in low-data regime, has competitive result on FashionMNIST andis much lower on MNIST.
The strength of our method is more obvious with more complex dataset.
It’s a surprising result because we do not implement any data augmentation implicitly or explicitlyunlike consistency regularization methods, who utilize data perturbation, which can be viewed asdata augmentation.
It’s worth noting that, on all three datasets, our method using BB-ANS alwaysoutperforms at least one semi-supervised methods on 10-shot setting, indicating SSL methods trade“universality” for “performance” while our method is more like a baseline.
Speed-wisely, NPC-LVonly requires training once for generative model, and can run k nn on different shots ( n ) with noadditional cost.
In contrast, SSL methods require the whole pipeline to be retrained for every n .
Comparison with Non-Parametric Learning: In this experiment, we explore the effectivenessof using latent representations directly with k NN.
We train the same generative model we used in
NPC-LV (“Hier”), as well as a vanilla VAE with a single latent variable (“Single”).
Table 1 shows thatthe latent representation of the vanilla one is not as expressive as the hierarchical one.
Although latentrepresentation using the hierarchical architecture performs reasonably well and surpasses supervisedmethods in 5-shot setting on MNIST and FashionMNIST, it’s still significantly lower than NPC-LVin all settings.
The result suggests that NPC-LV can utilize trained latent variable models moreeffectively than simply utilizing latent representation for classification.
Comparison with NPC: We investigate how NPC with non-neural compressors perform withdifferent distance metrics.
We evaluate with NCD, CLM, and CDM as distance metrics, and gzip, bz2,lzma, WebP, and PNG as compressors, using 1,000 images from test set and 100 samples per classfrom training set.
The result is shown in Table 2.
For distance metrics, we can see CLM and CDMperform similarly well
but it’s not clear under what circumstances a distance metric is superior tothe rest.
For compressors, both Bit-Swap and BB-ANS perform much better than other compressors,indicating that generative-model-based compressors can significantly improve NPC.
BB-ANS turnsout to be the best compressor for classification on all three datasets.
Analyses and Discussion
[Figure]
6.1 nELBO as Compressed Length
[Figure]
As we’ve shown in 3.2, nELBO can be viewed as the expected length of compressed bitstream
N − n extra .
Thus, theoretically it can be used directly to approximate compressed length.
In this way,we don’t need to apply ANS to VAE for the actual compression, which largely simplifies the method and boosts the speed.
However, as we can see in Table 1, using nELBO doesn’t always performbetter than the actual compressor like BB-ANS.
This may be because nELBO in a well-trained modelregards the aggregation of two images as out-of-distribution data points; while the discretization inthe actual compressor forces close probability with a certain level of precision discretized to the samebin, lowering the sensitivity.
Better aggregation strategies need to be designed to mitigate the gap.
6.2 Bitrate versus Classification Accuracy
[Figure]
The origin of NPC framework comes from theintuition that the length of x after being maximally compressed by a real-world compressor is close to K ( x ) .
Theoretically, the closerthis length approximates the minimum lengthof the expression ( C ( x )
≈ K ( x ) ), the closerthe compressor-based distance metrics are to thenormalized information distance .
We investigate, empirically, whether the bitrate actuallyreflect the classification accuracy.
We plot bitrate versus classification accuracy for each compressor in Table 2 on three datasets as shownin Figure 3.
We use the net bitrate, which is( N − n extra ) /d , where N is the length of the compressed bitstream, n extra is the length of the extrabits, and d is the number of pixels.
As we cansee, a very strong monotonic decreasing correlation between bitrate and accuracy emerges, with
Spearman coefficient [66] r s = − 0 .
96 , meaningthe lower the bitrate is, the higher the classification accuracy is.
This means the correlationbetween bitrate and classification accuracy holds empirically regardless of datasets.
It will beinteresting to investigate in the future whether the correlation remains for lossy compression.
6.3 Parallelization and Limitation
The training of generative models can be parallelized using modern GPUs.
Compression, however,is not that easy to parallelize.
The calculation of CDF and PDF are parallelizable for commonprobability distributions like Gaussian distribution, but ANS algorithm is not trivial to be parallelized.
Fortunately, efficient implementation for ANS on GPUs has been developed [67] to exploit GPU’sparallelization power.
During distance computation stages, as only pair-wise distance is needed,we can use multi-threads to accelerate the computation.
For classification, we show that NPC-LVperforms well in the low labeled data regime, where the complexity of computation may not be aconcern, yet the complexity of O ( n 2 ) may still hinder the application involving pair-wise distancecomputation like clustering on large datasets, unless we exploit the parallelization for compression.
Conclusion
In this paper, we propose a learning framework Non-Parametric Learning by Compression with Latent
Variables (NPC-LV) to address a common learning scenario Non-Supported Few-Shot-Learning(NS-FSL).
This framework is versatile in that every module is replaceable, leading to numerousvariations.
We use image classification as a case study to demonstrate how to use a trained latentgenerative model directly for downstream classification without further training.
It outperformssupervised learning and non-parametric learning on three datasets and semi-supervised learning on
CIFAR-10 in low data regime.
We thus regard it as a baseline in NS-FSL.
The equivalence betweenoptimizing latent variable models and achieving the shortest code length not only shows how nELBOcan be used for classification, but also indicates the improvement of latent probabilistic models canbenefit neural compressors.
The relationship between compression rate and classification accuracysuggests that the improvement of neural compressors can further benefit classification.
Thus, anenhancement of any module in this chain can boost classification accuracy under this framework.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope
[Yes See abstract and the end of the introduction.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Ap-
pendix I.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Although we point to
the reference for some unimportant proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tary materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix C.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5and Appendix C.
(b) Did you mention the license of the assets?
[Yes]
In both Section 5 and Appendix C.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
The only new asset is our own code and we provide in supplemental material.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A] Datasets are public available with MIT license. [Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
There are no personal or offensive content in
[Table] MNIST, FashionMNIST and CIFAR10. [Table]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A] No crowdsourcing or human subjects involved. [Table]
(b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
No crowdsourcing or human subjects
[Table]
involved.
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A] No crowdsourcing or human subjects [Table]
involved.
[Table]
A Ablation Study
BB-ANS (Alice: compress) initial extra bits decode with decode with encode with encode with encode with
BB-ANS (Bob: decompress) received bitstream decode with decode with decode with encode with encode with
Bit-Swap (Alice: compress) initial extra bits decode with encode with decode with encode with encode with
Bit-Swap (Bob: decompress) received bitstream decode with decode with encode with decode with encode with
Number of Latent Variables: The difference between how BB-ANS [36] and Bit-Swap [43]compress is shown in Figure 4.
We can see these two compressors are only different when there aremore than one latent variable, and by utilizing encoded bitstreams from previous latent variables, Bit-Swap can reduce the cost of initial bits significantly.
We carry out experiments on CIFAR-10 andkeep aggregation method constant.
We can see in Table 3, Bit-Swap performs better with more latentvariables.
But Bit-Swap with eight latent variables is still worse than BB-ANS with two.
Aggregation Methods: We also evaluate how different aggregation methods affect the classificationaccuracy, shown in right hand side of Table 3.
We aggregate two images by “average”, “‘minimum”,“maximum”, “concatenation” and “greyscale+average”.
For “concat”, the actual operation is that wecompress one image after another in a way that the compressed bitstream of the first image could beused as the “extra” bits for the second image.
“gs+avg” means we use greyscale of image and thenaverage pixel values.
“greyscale” is not a way of aggregation, but we notate it in the table this wayfor simplicity and comparison, and also to stress this operation is only used during compression.
Thatis, the generative model is trained on original images instead of on greyscale images.
By applyingsimple image processing methods during compression, we find that combining aggregation withimage manipulation can be effective as no compressor need to be changed, and thus no retrainingfor generative models is needed.
We also plot their bitrate v.s. classification accuracy for variousaggregation methods.
Figure 5 shows “concat” for images are an outlier - lower bitrate with lowaccuracy.
We will show in Appendix B that “concat” disqualifies BB-ANS and Bit-Swap from beinga normal compressor.
Please note the special case of “concat” doesn’t apply to other compressorslike gzip who treat images as bytes at the first place.
[Figure]
Bit-Swap [43] achieves the new state of the artbitrate but why it doesn’t suppress BB-ANS forclassification?
One of the reasons that Bit-Swapachieves better compression rate than BB-ANSis because Bit-Swap requires less initial bits, details shown in Appendix
F. However, initial bitscan only be amortized when there are multipledatapoints need to be compressed.
But in our application, there are at most two datapoints needto be compressed sequentially.
This leads us tochoose the net bitrate, which excludes the lengthof the initial bits, and makes Bit-Swap less advantegeous.
Theoretically, net bitrate should bethe same for both methods.
However, empirically we can see in Figure 5, Bit-Swap usesslightly more net bits than BB-ANS.
Although we cannot draw a definite conclusionthat the more a compressor can compress, themore accurate of classification NPC can obtainwith the compressor.
We can see that given a compressor, there is a correlation between bitrateand accuracy even with different aggregation methods except for aggregation methods that alter acompressor to be not normal (e.g., “concat’).
Other Alternatives: In this work, we don’t go thoroughly through all the state of the art compressors,but only focus on the VAE-based lossless compressors.
For this specific category, there are already [Table] numerous important factors: the choice of architectures, the choice of the number of latent variables,the choice of hierarchy topology (e.g., asymmetrical tree structure or symmetrical one) and thechoice of discretization methods.
Beyond this line of compressors, deep learning based compressorsdiscussed in Section 3.3 can also be used under our framework.
Beyond compressors, aggregation methods also cause diverging differences in the final classificationaccuracy.
We covers a few basic ones but there are other non-training-required aggregation methodslike linear blend operator and even completely different aggregation strategies (e.g., using conditional
VAE).
On colored real-world images like CIFAR-10, image manipulation can be another easy andeffective way to improve the accuracy.
In effect, “greyscale” can be viewed as “lossy compression”and this opens up the question of how lossy compressors perform under NPC framework.
B Normal Compressor
Definition 2 ( Normal Compressor ) .
A compressor is normal if it satisfies, up to an additive O (log n )term, where n means the maximal binary length of an element of Ω :
Idempotency: C ( xx ) =
C ( x ) and C ( ϵ ) = [-
-]0 where ϵ is the empty string
Symmetry: C ( xy )
=
C ( yx )
Monotonicity: C ( xy ) ≥ C ( x )4.
Distributivity: C ( xy )
+ C ( z ) ≤ C ( xz )
+ C ( yz )
Definition 3 ( Metric ) .
A distance function D : Ω × Ω → R + is a metric if it satisfies the following 3criteria for any x, y, z ∈ Ω , where Ω is a non-empty set, R + represents the set of non-negative realnumber:
Identity: D ( x, y ) =
iff x = y2.
Symmetry: D ( x, y )
=
D ( y, x )3.
Triangle Inequality: D ( x, y ) ≤ D ( x, z )
+ D ( z, y )
Definition 4 ( Admissible Distance ) .
A function D : Ω × Ω → R + is an admissible distance iffor every pair of objects x, y ∈ Ω , the distance D ( x, y ) is computable, symmetric and satisfies thedensity condition (cid:80) y 2 − D ( x,y ) ≤ 1 .
Cilibrasi and Vitányi [32] formally prove that if the compressor is normal , NCD is a normalizedadmissible distance satisfying the metric inequalities, which is shown in Definition 3.
Cebrián et al.[25] systematically evaluate how far real world compressors like gzip, bz2, PPMZ can satisfy theidempotency axiom.
Here we empirically evalute all 4 axioms on MNIST with BB-ANS compressor.
We randomly take 100 samples and plot C ( · ) on LHS as x-axis, C ( · ) on RHS as y-axis.
Forsimplicity, we use one latent variable, under which circumstance BB-ANS equals to Bit-Swap.
Asshown in Figure 6, BB-ANS satisfy monotonicity, symmetry and distributivity.
However, it failson identity axiom, with C ( xx ) ≈ 1 .
988 C ( x ) .
Unlike gzip, bz2, or lzma, BB-ANS doesn’t treat theconcatenation of two images as a sequence of bytes but images, similar to PNG and WebP, makingthem hard to satisfy identity axiom.
By default, the aggregation method refers to “concatenation”.
Butsimple concatenation does not perform well.
Practically, BB-ANS fails the identity test when using“concatenation”.
Without changing the compressor, is it still possible to satisfy the above conditionsso that NCD can be used as a normalized admissible distance metric?
We can change the aggregationmethod.
We investigate whether BB-ANS with average can satisfy the above conditions.
Obviously,identity and symmetry axioms can hold, as C ( avg ( x, x ))
=
C ( x + x 2 ) =
C ( x ) .
We empiricallyevaluate monotonicity and distributivity, and find that they are both satisfied with “average”.
Figure 7 illustrates that C ( avg ( x, y )) ≥ C ( x ) and C ( avg ( x, y ))
+ C ( z ) ≤ C ( avg ( x, z ))
+ C ( avg ( y, z ))always hold.
Given the compressor is normal under “average” function, we now prove NCD is anadmissible distance metric.
Definition 5.
Let D be an admissible distance.
D + ( x ) is defined as D + ( x ) = max { D ( x, z ) :
C ( z ) ≤ C ( x ) } , and D + ( x, y ) is defined as D + ( x, y )
=
max { D + ( x ) , D + ( y ) }
Lemma 1.
If C is a normal compressor, then E c ( x, y )
+ O (1) is an admissible distance, where
E c ( x, y ) =
C ( xy ) − min
{ C ( x ) − C ( y ) } is the compression distance.
Lemma 2.
If C is a normal compressor, then E + c ( x, y )
=
max { C ( x ) , C ( y ) }
Theorem 2.
If the compressor is normal, then the NCD is a normalized admissible distance satifyingthe metric (in)equalities.
Proof.
Lemma 1 and Lemma 2 show that NCD is a normalized admissible distance.
We now showhow NCD satisfies the metric (in)equalities.
For identity axiom,
[Equation]
For symmetry axiom,
[Equation]
For triangle inequality, without loss of generality, we assume C ( x ) ≤ C ( y ) ≤ C ( z ) .
As NCD is symmetrical, there are three triangle inequalities that can be expressedby NCD ( x, y ) , NCD ( y, z ) , NCD ( x, z ) .
For simplicity, we prove one of them,
NCD ( x, y ) ≤ NCD ( x, z ) + NCD ( z, y ) as the procedure for the other two is similar.
Since BB-ANS is a normal compressor under “avg”, we have distributivity:
C ( avg ( x, y ))
+ C ( z ) ≤ C ( avg ( x, z )) + C ( avg ( z, y )) . Subtracting C ( x ) from both sides andrearraging results in C ( avg ( x, y ))
− C ( x
[Equation]
≤
[Equation]
( avg ( x, [Equation] − C ( x
[Equation] + C ( avg ( z, y ))
− C ( z ) .
Dividing by C ( y ) on both sides, we have
[Equation]
[Equation]
We know LHS ≤ 1 and RHS can be ≤ 1 or > 1 .(a)
RHS ≤ 1 : Let C ( z ) =
C ( y ) + ∆ ; adding ∆ to both the numerator and denominator of
RHS increases RHS and makes it closer to 1.
[Equation]
(b) RHS > 1 : The procedure is similar to the case when RHS ≤ 1 .
The difference is thatadding ∆ on both numerator and denominator makes RHS decrease instead of increase.
But RHS cannot decrease less than 1.
Thus, we still have
[Equation]
[Figure]
Identity
[Figure]
Symmetry
[Figure] [Figure]
C Training Details
For both CNN and VGG, we tune hyperparameters on a validation set to get the optimized performance.
The reason that we use VGG11 instead of VGG16 or VGG19 is because VGG11 performsthe best in low data regime in our experiments.
For CNN, we first normalize MNIST, FashionMNISTand CIFAR-10.
We use batch size = 4 , epoch number = 14 , Adadelta [68] as the optimizer withlearning rate = 1 , decaying learning rate by γ = 0 .
7 every step for MNIST
and we use epoch number = 20 when given 50 samples per class, and use epoch number = 40 whengiven less.
We use Adam [69] with learning rate = 0 . 0001 as the optimizer, and batch size = 4 .
For CIFAR-10, we use learning rate = 0 .
00001 , epoch number = 80 .
For MeanTeacher and VAT, we follow Zhang et al.
[70]’s implementation and hyperparameterssetting.
We use WideResNet [71] with depth=28 and widen factor = 2 as the architecture for allthree datasets and iterate 60,000 for MNIST and FashionMNIST, and iterate 200,000 for CIFAR-10.
SGD with momentum is used for all three datasets, with learning rate = 0 .
03 , momentum = 0 . 9 .
For latent variable models used in this paper, we follow the training procedure and hyperparametersin Kingma et al.
[43] — four ‘Processing’ Residual blocks at the beginning of the inference model andthe end of the generative model; eight ‘Ordinary’ Residual block in total for all latent layers in bothinference model and generative model.
The Dropout [72] rate is 0.2 for MNIST and FashionMNIST,0.3 for CIFAR-10.
The learning rate for all datasets is 0 .
002 with Adam optimizer.
Dimension oflatent variables for MNIST and FashionMNIST is 1 × 16 × 16 while dimension of latent variablesfor CIFAR-10 is 8 × 16 × 16 .
During k NN, we use k = 2 for MNIST and FashionMNIST and k = 3for CIFAR-10.
We use one NVIDIA Tesla P40 GPU for training and compression.
For pairwise computation in50-shot setting, it takes roughly ten hours to calculate distance matrix on MNIST and FashionMNIST;it takes about thirty hours for CIFAR10.
But once the distance matrix is calculated, evaluation on5-shot or 10-shot just takes seconds.
For both CNN and VGG, it takes about half an hour to train on50-shot for one experiment.
For VAT and MT, we need to re-train for every shot setting.
For MNISTand FashionMNIST it takes about three hours to run one experiment in a single shot setting and for CIFAR10 it takes about twelve hours. The time that VAT and MT take positively relate to the numberof iterations, which makes them slower than our method in the 5 and 10-shot but faster in the 50-shotsetting. D Details of ANS
We briefly introduce ANS and show the proof of the optimal code length obtained from ANS.
Theessence of ANS is to encode one or more data points into a single natural number, called state s ∈ N .
Depending on different vocabularies and manipulations, there are different variations of ANS (detailscan be seen in Duda et al.
We introduce one of them - rANS (range ANS), which is the variantwe use in this paper.
The notation we use here is unconventional in order to be consistent with themain part of the paper.
Let’s notate our state at timestamp t as s t ∈ N , and notate our symbol/message at t as x t , x t ∈ V ,where V = { 0 , 1 } is the vocabulary set.
We have two simple methods to encode a binary sequenceinto a natural number bit by bit — s t
= 2 s t − 1 + x t or s t =
s
t − 1 + 2 m x t .
The former meansappending information to the least significant position while the latter is adding information to themost significant position.
It’s obvious that encoding new symbol into the most significant positionrequires remembering m while encoding in the least significant position only need previous states t − 1 and new information x t .
It’s also easy to decode: depending on whether the current state s t iseven or odd, we not only know if the last encoded symbol x t is 0 or 1 , but we can also decode thestate following s t −
= s t 2 or s t −
=
s t − 1 2 .
The above example illustrates encoding and decoding methods when there are two elements in thevocabulary with uniform distribution p (0)
=
p (1) = 12 .
In this case, it’s optimal to scale up s t to twofor both 0 or 1 as we essentially only spend 1 bit per encoded symbol.
However, when the probabilityis not uniformly distributed, the entropy is smaller, and scaling up by 2 for both symbols will not beoptimal anymore.
rANS generalizes the process to any discrete probability distribution and any sizeof vocabulary.
Intuitively, scaling up by a smaller factor for a more probable symbol and scaling up by a largerfactor for a less probable symbol will provide us with a more efficient representation.
Concretely,we have a sequence of messages x
= ( x 1 , x 2 , x 3 , ..., x n ) , and a vocabulary V = { v 1 , v 2 , v 3 , ..., v k } ,with size k , x i ∈ V .
We also have probability mass distribution of V : P = { p v 1 , p v 2 , p v 3 , ..., p v k } .
Correspondingly, let’s define frequency counts F = { f v 1 , f v 2 , f v 3 , ..., f v k } , f v i = p v i × M where
M = (cid:80) ki
=1 f v i .
M can be viewed as a multiplier, demonstrating the precision of ANS, which is apredefined variable in the implementation.
We can also get cumulative frequency counts from F as follows B = { b v 1 , b v 2 , b v 3 , ..., b v k } where b v i =
(cid:80) i − 1 j =1 f v j .
Now we get everything we need todefine the encoding function G :
[Equation]
The procedure can be interpreted as follows:
We have various M -sized blocks partitioning naturalnumber N .
Encoding can be viewed as finding the exact location of the natural number that representsthe state, by first finding the corresponding block ( (cid:106) s t − 1 f xt (cid:107) × M ) followed by finding the subrange representing that symbol within M ( b x t ) and finding the exact location within that sub-range( s t − 1 mod f x t ).
The decoding function H ( s t ) is the reverse of the encoding:
[Equation]
We first find the precise location within the sub-range using inverse function of cumulative counts(argmax { b x t < ( s t mod M ) } ).
With x t we can reverse steps in Equation (17) to get the previousstate.
As we can see ANS decodes in the reverse order of encoding (i.e., last in first out), whichmakes it compatible with bits-back argument.
From encoding function, we know that:
[Equation]
Encoding a sequence of symbols x results in:
[Equation]
Thus, the total coding length is:
[Equation]
where s 0 refers to the initial state.
Dividing by n we will get the average coding length thatapproximates the entropy of the data.
E Discretization
ANS is defined for symbols in a finite alphabet; bits-back coding works for discrete latent variables.
However, continuous latent variables are proven to be powerful in many latent variable models.
Inorder to use those latent variable models for lossless compression, discretizing continuous variablesinto discrete ones is a necessary step.
Townsend et al.
[36] derives, based on MacKay [74], thatusing bits-back coding continuous latent variables can be discretized to arbitrary precision withoutaffecting the compression rate.
Suppose a probability density function p is approximated using anumber of “buckets” of equal width σ z .
For any given bucket j , we can know its probability massp ( z ( j ) )
σ z where z ( j ) is some point in the bucket j .
Let’s notate the discrete distribution as P and
Q for both prior and posterior distribution.
Then for any given bucket j , P ( j ) ≈ p ( z ( j ) )
σ z .
Theexpected message length with a discretized latent variable is:
[Equation]
The width of buckets σ z is cancelled.
Therefore, as long as the bins for inference models match thegenerative models, continuous latent variables can be discretized up to an arbitrary precision.
In this paper we only consider the basic discretization techniques like dividing continuous distributioninto bins with equal width or equal mass .
We discretize the prior (top layer) with equal mass and allsubsequent latent layers with equal width.
As Equation (22) shows, ideally we want the discretizationaligns between inference models and generative models.
However, discretization of z i ∼ p θ
( z i | z i +1 )relying on z i ∼ q ϕ
( z i |
z
i − 1 ) is not possible without sampling.
In the compression stage, whendecoding z i , z i +1 is not available and so is p
θ ( z i | z i +1 ) .
In the decompression stage, similarly,q ϕ ( z i | z i − 1 ) is not available for p
θ
( z i | z i +1 ) to match with.
Therefore, we need to sample fromtraining dataset beforehand to get unbiased estimates of the statistics of the marginal distribution [43]. This process only needs to be done once and can be saved for the future use.
F Initial Bits of BB-ANS and Bit-Swap
The main difference between BB-ANS and Bit-Swap is that BB-ANS requires the sender Alice todecode
z i +1 with q ϕ
( z i +1 |
z i ) for i from 1 to L − 1 first, and then encode z i with p θ ( z i |
z i +1 )
fori from 1 to L − 1 .
While Bit-Swap interleaves this encoding and decoding procedure and appliesrecursively for latent variables, as illustrated in Figure 4.
The advantage of Bit-Swap’s procedureis that, after decoding z 1 , the bits encoded from x can be taken advantage in decoding z 2 ; then bitsencoded from z 1 can be used for decoding z 3 .
As a result, the initial bits required for Bit-Swap ismuch less than BB-ANS.
Concretely, for BB-ANS, the minimum initial bits required:− log q ϕ ( z [Equation] | x ) − [Equation]
−
[Equation]
(cid:88)
i
[Equation]
For Bit-Swap, the minimum initial bits requires:
[Equation]
The initial bits Bit-Swap requires is less than BB-ANS, making Bit-Swap reaching the optimalcompression rate.
G Hierarchical Latent Variable Models
The hierarchical autoencoder in the paper uses deep latent gaussian models (DLGM)
[44] followingthe sampling process based on Markov chains, whose marginal distributions are:
[Equation]
Combining the above equations, the marginal distribution of x is:
[Equation]
Accordingly, inference models q ϕ ( z i +
| z i ) need to be defined for every latent layer.
ELBO thatincludes multiple latent variables then becomes:
[Equation]
In this paper, we use Logistic distribution ( µ = 0 , σ = 1 ) as the prior p ( z L ) , and use conditional Logistic distribution for both inference models q ϕ
( z i +1 |
z i ) , q ϕ ( z 1 | x ) and generative modelsp θ ( z i
| z i +1 ) .
These distributions are modeled by neural networks, which is stacked by Residualblocks [75] as hidden layers.
More architecture details can be referred to Kingma et al.
[43], wherethey also discusses other possible topologies regarding to hierarchical latent variable models.
H Proof of Universality of Information Distance
Information Distance E ( x, y ) refers to the length of the shortest binary program generated byuniversal prefix Turing machine, that with input x computes y , and with input y computes x . It’sshown that E ( x, y )
=
max { K ( x | y ) , K ( y | x ) } .
We now prove Theorem 1, based on Lemma 3 [29].
Lemma 3.
For every upper-semicomputable function f ( x, y ) , satisfying (cid:80) y 2 − f ( x,y ) ≤ 1 , we have
K ( y | x ) < f
( x, y ) .
To prove that E ( x, y ) is a metric, we show it satisfies metric (in)equalities.
We can infer the nonnegativity and symmetry directly from the definition E ( x, y )
=
max { K ( x | y ) , K ( y | x ) } .
For triangleinequality, given x, y, z , without loss of generality, let E ( x, z ) =
K ( z | x ) .
By the self-limitingproperty, we have
[Equation]
To prove E ( x, y ) is admissible , we show it satisfies density requirement:
[Equation]
The second inequality is due to Kraft’s inequality for prefix codes.
To prove the minimality, as for every admissible distance metric D ( x, y ) , it satisfies (cid:80) y :
y ̸
= x 2 D ( x,y ) ≤ 1 .
According to Lemma 3, we have K ( y | x ) < D ( x, y ) and K ( x | y ) <
D ( y, x ) .
I Potential Negative Social Impact
We propose a general framework for classification and clustering tasks with minimum assumptionabout the dataset.
We don’t foresee it has negative social impact itself.
However, our framework usesgenerative models for their explicit density estimation and the development of generative models maybe used by people with ulterior motives to generate fake data. J NPC on Text Classification [Table] Quantitative Result:
To emphasize that the
[Table]
performance gain of NPC does not solelycome from the neural compressor, we carryout experiments with traditional compressors such as gzip under NPC framework ontext datasets.
The baselines include fasttext [76], bidirectional LSTMs [77] withattention [78] (Bi-LSTM+Attn), hierarchical attention networks (HAN) [79], character CNN (charCNN) [80] and very deep
CNN (VDCNN) [81].
We also includetwo non-parametric methods: word2vec(W2V) [82] and pre-trained sentence BERT(SentBERT) [83], all using a k NN classifier.
We use AGNews, SogouNews and DBpediaavailable on torchtext.
Details of the datasetsare in Table 5.
We use the implementationof the publicly available text classification code repository Hedwig for supervised learning methods.
Compared with deep neural networks without pre-training, we can see NPC with gzip achieves thehighest on both AGNews and SogouNews.
Compared with pre-trained models like SentBERT, NPCwith gzip can still outperform all methods on SogouNews.
On DBpedia, NPC achieves about 1%lower accuracy than most neural methods.
Note that SentBERT have been pre-trained on billions oftokens while NPC is not trained or pre-trained on any dataset.
[Table]
Concrete Example: To understand why NPC framework can achieve a high accuracy on textclassification without any training, we can take a concrete example:
Japan’s Seiko Epson Corp. said Wednesday it has developed a 12-gram flyingmicrorobot,
the
world’s lightest
The latest tiny flying robot that
could help in search and rescue or surveillancehas
been
unveiled in Japan
Michael
Phelps won the gold medal in the 400 individual medley and set a world
record
in a time of 4 minutes 8.
seconds.
These are three sentences x 1 , x 2 , x 3 extracted from AGNews dataset where x 1 and x 2 belongs to thesame category while x 3 belongs to another.
Using gzip as the compressor to approximate Kolmogorovcomplexity, we get C ( x 1 ) =
115 , C ( x 2 )
= 104 , C
( x 3 ) = 118 bytes.
After we concatenate x 1 andx 2 , the compressed length C ( x 1 x 2 ) = 177 which is less than C ( x 1 )
+ C ( x 2 )
= 219 as the existenceof same words in the two sentences (e.g., Japan, flying) makes C ( x 1 x 2 ) more compressible.
Inother words, C ( x 1 x 2 ) − C ( x 1 ) means given x 1 , the number of bytes needed for x 2 .
Roughlyspeaking, the more the mutual information exists between x 1 and x 2 , the less C ( x 1 x 2 ) − C ( x 1 )is.
Thus, we should have C ( x 1 x 1 ) − C ( x 1 ) <
C ( x 1 x 2 ) − C ( x 1 ) <
C ( x 1 x 3 ) − C ( x 1 ) , assumingsentences in the same category do share more mutual information.
Calculating the actual result, wehave C ( x 1 x 1 ) − C ( x 1 ) = 5 , C ( x 1 x 2 ) − C ( x 1 ) = 62 , C ( x 1 x 3 ) − C ( x 1 ) = 76 , which empiricallyvalidates our intuition.
The formula of C ( x 1 x 2 ) − C ( x 1 ) can be seen as an simplified version ofinformation distance between x 1 and x 2 .
The key insight behind the information distance is that:(1) compressors are good at capturing regularity; (2) objects from the same category share moreregularity than those that aren’t.
In addition to the intuition explained, large amounts of previouswork (shown in Section 4) demonstrate strong performance of NPC framework, a non-neural versionof NPC-LV.