With a Little Help from Gzip: Text Classification with No Training
Abstract
In text classification, neural network methods often turn out to be an overkill.
Compressorbased methods are simpler, but previous works in this area fail to achieve a result comparable with neural network methods.
In this paper, we combine a simple compressor like gzip with a k -nearest-neighbor classifier for text classification.
Without any training, pre-training or fine-tuning, our method achieves results that are competitive with deep learning methods on seven datasets, and it even outperforms BERT and sentence-BERT on one dataset.
In addition, we demonstrate the robustness of our method in the few-shot setting.
Introduction
Text classification, as one of the most fundamental tasks in natural language processing (NLP), has improved substantially with the help of neural networks (Li et al., 2022).
Needless to say, most neural networks are data hungry, the degree of which increases with the number of parameters.
Additionally, hyperparameters need to be carefully tuned across different datasets.
Pre-processing (e.g., tokenization, stop words removal, etc.)
is another tailored procedure for text that requires careful handling for different models and datasets.
The benefit of using neural networks contributes mainly to its ability of capturing latent correlations and recognizing implicit patterns (LeCun et al., 2015); however, given their greater complexity, they may be overkill for simple tasks like text classification.
For example, Adhikari et al.
(2019b) find that a simple long short-term memory network (LSTM; Hochreiter and Schmidhuber, 1997) with appropriate regularization can achieve competitive results.
Shen et al. (2018) further show that even word-embeddingbased methods can achieve results comparable to convolutional neural networks (CNNs) and recurrent neural networks (RNNs).
In this paper, we propose a simpler method using only a lossless compressor and absolutely zero training .
Lossless compressors aim to represent information using as few bits as possible by assigning shorter codes to symbols with larger probability.
We hypothesize that the regularity and the probability distribution captured by the compressor can be used effectively toward a zero-training classifier.
Several studies attempt to use compressors for topic classification (Teahan and Harper, 2003;
Frank et al., 2000), based on the intuition that the minimum cross entropy between a document and a language model of a class built by a compressor indicates the class of the document.
But this line of work fails to compete with the state of the art by then.
Another line of compressorbased methods (Khmelev and Teahan, 2003; Keogh et al., 2004) takes advantage of the Normalized
Compression Distance (NCD) (Li et al., 2004), a distance metric derived from Kolmogorov complexity.
However, few (Marton et al., 2005; Coutinho and Figueiredo, 2015) explore NCD’s application to topic classification, and none applies the combination of NCD and k -nearest-neighbor ( k NN) classifier when k > 1 to topic classification.
We build our work upon the latter line of work.
Our contributions are as follows: (1) We are the first to use NCD together with k NN on topic classification — this allows us to carry out comprehensive experiments on large datasets and to compare the results with neural network models, which has not been done in this area before; (2) We use a compressor-based method that requires zero training to achieve results comparable to neural network models that have millions of parameters on seven out of eight datasets; (3) We demonstrate that when labeled data is extremely limited, our method outperforms non-pretrained neural network methods; (4) We show that our method is more robust than pre-trained models on out-of-distributed datasets; (5) We find that, being a universal base- line, our method is particularly suitable for datasets that are easily compressible.
Related Work 2. Compressor-Based Text Classification
Compressor-based distance metrics have been used mainly for plagiarism detection (Chen et al., 2004), clustering (Vitányi et al., 2009) and classifying time series data (Chen et al., 1999; Keogh et al., 2004).
Several previous works explore methods using a compressor-based distance metric for text classification: Li et al.
(2004) applies it to language identification as language are different in length by nature (e.g., addresses <English>, adressebok <
Norwegian>, adressekartotek <Danish>); Khmelev and
Teahan (2003) uses it for authorship categorization;
Frank et al.
; Teahan and Harper (2003) utilize Prediction by Partial Matching (PPM) for topic classification.
PPM, a text compression scheme utilizing language modeling, estimates the cross entropy between the probability distribution built on class c and the document d : H c ( d ) .
The intuition is that the lower the cross entropy is, the more likely that d belongs to c .
Summarized in Russell (2010), the procedure of using compressor to estimate H c ( d ) is that: (1) for each class c , concatenate all samples d c in the training set belonging to c ; (2) compress d c as one long document to get the compressed length C ( d c ) ; (3) concatenate the given test sample d u with d c and compress to get C ( d c d u ) ; (4) the predicted class is arg min c C ( d c d u )
− C ( d c ) .
The major drawback of this method is that concatenating all training documents in one class makes it hard to take full advantage of large training set, as compressors like gzip has a limited size of sliding window, which is responsible for “how much” the compressor can look back to find repeated patterns.
Marton et al.
(2005) further investigate the distance metric C ( d ( i )
c d c ) − C
( d ( i ) c ) where d ( i ) c is a single document belonging to class c .
Coutinho and Figueiredo (2015); Kasturi and
Markov (2022) focus on improving representations based on compressor to improve the classification accuracy.
To the best of our knowledge, all the previous work use relatively small datasets like 20News and
Reuters-10.
There is neither a comparison between compressor-based methods and deep learning methods nor any comprehensive study on large-sized datasets.
2. Deep Learning for Text Classification
The deep learning methods used for text classification can be divided into two: transductive learning, represented by Graph Convolutional Networks (GCN) (Yao et al., 2019), and inductive learning, where both recurrent neural networks (RNN) and convolutional neural networks (CNN) are main forces.
We focus on inductive learning in this paper as transductive learning assumes the test dataset is presented during the training.
Zhang et al.
(2015) first use the characterbased CNN with millions of parameters for text classification.
Conneau et al.
(2017) extend the idea with more layers.
Along the line of
RNNs, Kawakami (2008) introduce a method that uses LSTMs (Hochreiter and Schmidhuber, 1997) to learn the sequential information for classification.
To better capture the important information regardless of its position in the sentence, Wang et al.
(2016) incorporate the attention mechanism into the relation classification.
Yang et al.
(2016) include a hierarchical structure for sentence-level attention.
As the number of parameters and the complexity of models increase, Joulin et al.
(2017) start to explore the possibility of using simple linear model with a hidden layer coping with n -gram features and hierarchical softmax to improve efficiency.
The status quo of classification is further changed by the prevalence of pre-trained models like BERT (Kenton and Toutanova, 2019), with thousands of millions of parameters pre-trained on corpus containing billions of words.
BERT can achieve the state of the art on numerous tasks including text classification (Adhikari et al., 2019a) with just some fine-tunings.
Built on
BERT, Reimers and Gurevych (2019) calculate semantic similarity between pairs of sentences efficiently by using a siamese network architecture and fine-tuning on multiple NLI datasets (Bowman et al., 2015; Williams et al., 2018).
Our Approach Kolmogorov complexity (Kolmogorov, 1963)
K ( x ) characterizes the length of the shortest binary program that can generate x . K ( x ) is theoretically the ultimate lower bound for information measurement.
Given this notion of information measurement, how, do we compare information content between two objects?
To this end, Bennett et al.
(1998) define information distance E ( x, y ) as the length of the shortest binary program that converts x to y :
[Equation]
However, E ( x, y ) is not computable as Kolmogorov complexity is incomputable, and absolute distance makes comparison among objects hard.
Li et al.
(2004) proposes a normalized and computable version of information distance, Normalized Compression Distance (NCD), utilizing compressed length C ( x ) to approximate Kolmogorov complexity K ( x ) .
Formally, it’s defined as follows (detailed derivation is shown in Appendix A):
[Equation]
The intuition behind using compressed length is that the length of x that has been maximally compressed by a compressor is close to K ( x ) .
The higher the compression ratio, the closer C ( x ) is to K ( x ) .
Our main experiment results use gzip as the compressor, thus, C ( x ) means the length of x after compressed by gzip .
C ( xy ) is the compressed length of concatenation of x and y .
With the distance matrix NCD provides, we can then use k -nearest-neighbor to classify.
Our method can be implemented with fifteen lines of Python code below, whose input is training_set , test_set , both of which consist of an array of (text, label) , and k :
[-x1x
-]
Experiments 4. Datasets
We choose this diverse basket of datasets to investigate the effects of the number of training samples, the number of classes and the length of the text on accuracy.
The details of each dataset’s statistics are listed in Table 1.
Previous works on text classification have two disjoint preferences when choosing evaluation datasets: CNN and RNN-based methods favor large scale datasets (AG News, SogouNews,
DBpedia, YahooAnswers) for evaluation, whereas transductive methods like graph convolutional neural network focus on datasets with smaller training sets (20News, Ohsumed, R8, R52) (Li et al., 2022).
We include datasets on both sides in order to investigate how our method performs with both abundant training samples and limited ones.
Apart from the variation of the dataset sizes, we also take the effects of number of classes into consideration by intentionally including datasets like R52 to evaluate our the performance on datasets with large number of classes.
Previous work (Marton et al., 2005) show that the length of text also affects the accuracy of compressor-based methods.
4. Baselines
We compare our result with (1) neural network methods that require training and (2) zero-training methods that use the k NN classifier directly, with or without pre-training.
Specifically, we choose mainstream architectures for text classification, like logistic regression, fasttext (Joulin et al., 2017), RNNs with or without attention (vanilla
LSTM (Hochreiter and Schmidhuber, 1997), bidirectional LSTMs (Schuster and Paliwal, 1997) with attention (Wang et al., 2016), hierarchical attention networks (Yang et al., 2016)), CNNs (character
CNNs (Zhang et al., 2015), recurrent CNNs (Lai et al., 2015), very deep CNNs (Conneau et al., 2017)) and BERT (Devlin et al., 2019) (Adhikari et al., 2019a).
We also include three other zerotraining methods: word2vec (W2V) (Mikolov et al., 2013), pre-trained sentence BERT (Reimers and
Gurevych, 2019), and the length of the instance, all using a k NN classifier.
To prevent the class from being predicted based on text length, we evaluate a baseline where the instance text length is used as the only input into a k NN classifier.
We call this baseline the TextLength method.
We present model statistics and trade-offs in Table 2.
Since the number of classes, the vocab-
[Table] [Table]
ulary size, and the dimensions affect the number of parameters
, we estimate the model size using AGNews.
This dataset has a relatively small vocabulary size and number of classes, hence making the estimation of the lower bound out of the studied datasets.
Some methods require pretraining either on the target dataset or on other external datasets.
Most neural networks require preprocessing like tokenization (“tok”), building vocabulary dictionaries and mapping tokens (“dict”), using pre-trained word2vec (“wv”), lowercasing the words (“lower”) and padding the sequence to a certain length (“pad”).
Other model-specific preprocessing includes adding extra bag of n-grams (“ngram”) for fasttext and using positional embedding (“pe”) for BERT .
4. Result on Full Dataset
We train all baselines on eight datasets (training details are in Appendix B).
The result of using the full training sets are shown in Table 3.
As we can see, our method performs surprisingly well on AG
News, SogouNews, R8 and R52.
For AG News, fine-tuning BERT achieves the best performance among all methods, and gzip , with no pretraining, achieves competitive result, within 0.007 points of
BERT.
On SogouNews, gzip achieves the best result among these methods.
SogouNews is a special dataset that includes Pinyin — a phonetic romanization of Chinese.
The worst method on this dataset is textCNN, RCNN and sentence BERT where they either assume the unit is a “word" in English or is heavily pre-trained on English corpus.
The accuracy of gzip on DBpedia is about 1% lower than other neural network methods.
For YahooAnswers, the accuracy of gzip is about 7% lower than the average neural methods.
This may due to the fact that the vocabulary size of YahooAnswers is large, making it hard for the compressor to compress (detailed discussion is in Section 5).
Starting from 20News dataset, the training size becomes smaller, where non-pretrained deep learning models are thought to be less advantageous.
On the 20News dataset, pre-trained methods achieve the best result and gzip ’s accuracy is in the middle.
Ohsumed is a dataset containing paper abstracts in the medical domain, aimed at categorizing 23 cardiovascular diseases.
On Ohsumed, gzip is lower than textCNN, DocBERT, SentBERT, competitive to LR and higher than others.
For R8, gzip has the third highest accuracy, only lower than HAN and DocBERT.
For R52, gzip ranks the fourth, sur-
Model
AGNews
SogouNews
DBpedia
YahooAnswers 20News
Ohsumed
R R
Training Required
TFIDF+LR 0.
LSTM 0.
Bi-LSTM+Attn 0.
HAN 0.
charCNN 0.
textCNN 0.
RCNN 0.
VDCNN 0.
fasttext 0.
BERT 0.
Zero Training
W2V 0.
SentBERT
[Table]
passed by HAN, DocBERT and SentBERT.
Overall, BERT-based models are robust even when the size of training samples are small, but do not excel when the dataset is out of distributed of the pre-training corpus (e.g., SogouNews).
Character-based models like charCNN and VDCNN perform badly when the training data is small and the vocabulary size is large (e.g., 20News).
The advantage of word-based models is non-obvious when the training data is small either, but they are better at handling big vocabulary size.
They are also inferior to character-based models when classifying corpus that are not English, similar to BERTbased models.
Logistic regression with TFIDF features, although doesn’t achieve the best on any dataset, is very robust to the size of the dataset.
The result of TextLength is close to random guess on all but R8 and R52, showing that the distribution of length doesn’t reflect the information of class in other six datasets, indicating the compressed length information used in NCD does not benefit from the length distribution of different classes. gzip does not perform well on extremely large dataset (e.g., YahooAnswers), but are competitive on medium and small-size datasets, and it’s robust to non-English corpus.
Performance-wise, the only non-preptrained deep learning model that’s competitive to gzip is HAN, who surpass gzip on 50% datasets and still achieve relatively high accuracy when it’s beaten by gzip , unlike textCNN.
The difference is that gzip doesn’t require training.
4. Few-Shot Learning
We further compare the result of gzip under the few-shot-learning setting with deep learning methods on AGNews, SogouNews and DBpedia.
We chose these three datasets as their scale is large enough to cover 100-shot setting and they vary in text lengths as well as languages.
Specifically, we use n -shot labeled examples per class from training dataset, where n = { 5 , 10 , 50 , 100 } .
We choose methods whose trainable parameters range from zero parameters like word2vec and sentence BERT to hundreds of millions of parameters like BERT, covering both word-based models (HAN) and the n-gram one (fasttext).
The result is plotted in Figure 1 (detailed numbers are shown in Appendix C).
As we can see, gzip outperforms non-pretrained models on 5 , 10 , 50 settings for all three datasets and especially in the n = 5 setting, gzip outperforms deep learning models by large margin.
For example, the accuracy of gzip is 115% better than fasttext on AGNews 5-shot setting.
In the 100-shot setting, gzip also outperforms non-pretrained models on AGNews and SogouNews but is a little bit lower than them on DBpedia.
It’s been investigated in the previous work (Nogueira et al., 2020; Zhang et al., 2021) that pre-trained models are excellent few-shot learners.
The advantages of BERT and
SentBERT on the AGNews are obvious where they achieve the highest and the second highest
#ofshots
T e s t
A cc u r ac y fasttextBi-LSTM+Attn
HANW2VSentBERTBERTgzip 5 1000 .0 .0 .0 .
#ofshots
T e s t
A cc u r ac y fasttextBi-LSTM+Attn
HANW2VSentBERTBERTgzip 5 1000 .0 .0 .0 .
#ofshots
T e s t
A cc u r ac y fasttextBi-LSTM+Attn
HANW2VSentBERTBERTgzip accuracy on every shot number.
However, on
SogouNews, both BERT and SentBERT are surpassed by gzip on every shot number, consistent with the result on full dataset.
This is reasonable as the inductive bias learned from the pre-training data is so strong—notice how low the accuracy is when only given 5-shot training samples to
BERT, that hinders BERT to be applied to the dataset that’s significantly different from the pre-trained datasets.
The surprising part is that even on DBpedia gzip still outperforms SentBERT on 50-shot and 100-shot settings.
Note that BERT has been pre-trained on Wikipedia and DBpedia is extracted from Wikipedia, which may explain the nearly perfect score of BERT on DBpedia.
In general, the larger the number of labeled training samples are, the closer that the accuracy gap between gzip and deep learning models are, except for W2V, which is extremely unstable.
This is due to the vectors being trained for a limited set of words, meaning that numerous tokens in the test set are out-of-vocabulary.
Analyses
To understand the merits and shortcomings of using gzip for classification, we evaluate gzip ’s performance in terms of both the absolute accuracy and the relative performance compared to the neural methods.
An absolute low accuracy with a high relative performance suggests that the dataset itself is difficult, while a high accuracy with a low relative performance means the dataset is better solved by a neural network.
We analyze both the relative performance and the absolute accuracy regarding the vocabulary size and the compression rate of both datasets (i.e., how easily a dataset can be compressed) and compressors (i.e., how well a compressor can compress).
To represent the relative performance with regard to other methods, we use the normalized rank percentage, computed as
[Figure]
; the lower the score, the better gzip is.
We use “bits per character”(bpc) to evaluate the compression rate.
The procedure is to randomly sample a thousand instances from the training and test set respectively, calculate the compressed length and divide by the number of characters.
Sampling is to keep the size of the dataset a constant.
[Figure]
5. Relative Performance
Combining Table 1 and Table 3, we see that accuracy is largely unaffected by the average length of a single sample: with the Spearman coefficient r s = − 0 . 220 .
But the relative performance is more correlated with vocabulary size ( r s = 0 .
561 ) as we can see in Figure 2.
SogouNews is an outlier in the first plot: on a fairly large vocabulary-sized dataset, gzip ranks the first.
The second plot may provide an explanation for that — the compression ratio for SogouNews is high which means even with a relatively large vocabulary size, there are also repetitive information that can be squeezed out.
With r s = 0 .
785 on the correlation between the normalized rank percentage and the compression rate, we can see when a dataset is easier to compress, our method may be a strong candidate as a classifier.
5. Absolute Accuracy
Similarly we evaluate the accuracy of classification with respect to the vocabulary size and we’ve found there is almost no monotonic relation ( r s = 0 . 071) .
With regard to bpc, the monotonic relation is not as strong as the one with the rank percentage ( r s = − 0 . 56 ).
Considering the effect that vocabulary size has on the relative performance, our method with gzip may be more susceptible to the vocabulary size than neural network methods.
To distinguish between a “hard” dataset and an “easy” one, we average all models’ accuracies.
The dataset that has the lowest accuracies are 20News and Ohsumed, which are two datasets that have the longest average length of texts.
Using Other Compressors
With compressor-based distance metrics we can use any compressor.
Because of the large size of the test set of the datasets, we randomly chose 1, test samples to evaluate and repeat the experiments for each setting five times to calculate the mean and 95% confidence interval.
We carry out experiments on other three compressors: bz2 , lzma and zstandard under the fewshot setting.
Each of them has different underlying algorithms from gzip .
bz2 uses Burrows-Wheeler algorithm to permute the order of characters in the strings to create more repeated “substrings” that can be compressed.
That’s one of the reasons why bz2 has a higher compression ratio (e.g., it can achieve 2.57 bpc on AGNews while gzip can achieve only 3.38 bpc).
lzma is based on LZ77, a dictionary-based compression algorithm, where the idea is to use (offset, length) to represent the n-gram that has previously appeared in the search buffer. lzma then uses range coding to further encode (offset, length) .
Similarly, gzip uses DEFLATE algorithm, which also uses LZ77 and instead of range coding, it takes advantage of Huffman coding to further encode (offset, length) .
zstandard (zstd) is a new compression algorithm that’s built on LZ77,
Huffman coding as well as Asymmetric Numeral
Systems (ANS) (Duda, 2009).
We pick zstd to evaluate for its fast speed, with close compression rate to gzip .
A competitive result may indicate it can be used to speed up the classification.
We plot all the test accuracy in Figure 4 with the compression ratio for each compressor.
Compression ratio is calculated by original size compressed size , so the larger the compression ratio is, the more a compressor can compress.
We use compression ratio instead of bpc here as the latter one is too close to each other and cannot be differentiated from one another.
Markers of ‘+’ represents the mean of each compressor’s test accuracy across different shot settings.
The dataset is not explicitly labeled but we can tell that there are roughly three clusters in the plot.
AGNews is the cluster with the lowest compression ratio and SogouNews is the one with the highest compression ratio.
Note that gzip and zstd with compression ratio of about 7 belongs to the SogouNews result.
On SogouNews, both gzip and zstd have the compression ratio equal to about 7; bz2 and lzma have the compression ratio over 9.
The difference of accuracy is more obvious on the AGNews and DBpedia with bz2 being the worst-performing compressor.
This is counterintuitive, as a compressor with a higher compression ratio suggests that it can approximate Kolmogorov complexity better, and bz2 has a higher compression ratio.
We conjecture it may be because in practice, Burrows-Wheeler algorithm used by bz2 dismisses the information of character order.
This is shown more clearly in Figure 4 — bz2 is always lower than the regression line.
In general, gzip achieves a relatively high and stable accuracy across three datasets.
lzma is competitive with gzip but the speed is much slower.
We’ve found in Section 5.1 that for a single compressor, the easier a dataset can compress, the more probable it can achieve a higher accuracy than deep learning models.
Here we investiage the correlation across compressors.
We’ve found the compression ratio and test accuracy has a moderate monotonic and linear correlation and as the shot number increases, the linear correlation is more obvious with r s = 0 .
605 for all shot settings and Pearson correlation r p = 0 .
719 respectively on 5, 10, 50 and 100 setting across four compressors.
Combining the special case of bz2 with the linear correlation between compression ratio and test accuracy, we know that in general a compressor with a high compression ratio can perform better on a more compressible dataset.
But the actual
[Figure] [Figure]
[Figure]
compression algorithm still has its effect on the test accuracy despite the high compression ratio.
5. Using Other Compressor-Based Methods
The distance metric used by previous work (Marton et al., 2005; Russell, 2010) is mainly C ( d c d u ) −
C ( d c ) as we mention in Section 2.1.
Although using this distance metric is faster than pair-wise distance matrix computation on small datasets, it has several drawbacks: (1) Most compressors have a limited “size”, for gzip it’s the sliding window size that can be used to search back of the repeated string while for lzma
it’s the dictionary size it can keep record of.
This means even if there are large number of training samples, the compressor cannot take full advantage of those samples; (2) When d c is large, compressing d c d u can be really slow and this slowness cannot be solved by parallelization.
These two main drawbacks stop this method to be applied to a really large dataset.
Thus, we randomly pick 1000 test samples and 100-shot from each class in training samples to compare these two methods.
In Table 4, “ gzip (ce)” means using the cross entropy C ( d c d u ) − C ( d c ) while “ gzip ( k NN)” refers to our method.
We carry our each experiment for five times and calculate the mean and 95% confidence interval.
On AGNews and SogouNews using k NN and NCD is better than using cross entropy.
The reason for the large accuracy gap between them on SogouNews is probably because each instance in SogouNews is very long, causing about 11.2K per sample, while gzip typically has 32K window size only.
Only concatenation a few samples makes the compression ineffective.
The cross-entropy method does perform very well on
YahooAnswers, which may benefit from using multiple references in the single category as YahooAnswers is a divergent dataset created by numerous online users.
We also test the performance of compressorbased cross entropy method on full AGNews dataset as it is a relatively smaller one with shorter single instance.
The accuracy is 0.745, not much higher than 100-shot setting, which further confirms that using C ( d c d u ) − C ( d c ) as a distance metric cannot take full advantage of the large datasets.
Conclusions and Future Work
In this paper, we use gzip together with a compressor-based distance metric to achieve classification accuracy comparable to neural network classifiers on seven out of eight datasets.
We also show the effectiveness of using this method in fewshot scenarios.
In future works, we will extend this work by generalizing gzip to neural compressors on text, as recent studies (Jiang et al., 2022) show that combining neural compressors that derived from deep latent variables models with compressorbased distance metrics for image classification can even outperform semi-supervised methods.
Limitations
The computation complexity for k NN is O ( n 2 ) .
Thus, one limitation of our method is its speed.
Multi-threads and multi-processes can largely boost the speed.
Furthermore, Lempel-Ziv Jaccard
Distance (LZJD) (Raff and Nicholas, 2017), a more efficient version of NCD can be explored to alleviate the inefficiency problem.
Another potential limitation is that we only talk about traditional compressors in the paper and traditional compressors are only able to capture the orthographic similarity.
For example, given a word “good”, gzip takesmore bytes to compress “goodness” but it takesmore bytes to compress “niceness”.
The ability to compress redundant semantic information may be made possible by neural compressors built on latent variable models (Townsend et al., 2018)
A Derivation of NCD
Recall that information distance E ( x, y ) is:
[Equation]
E ( x, y ) equates the similarity between two objects with the existence of a program that can convert one to another.
The simpler the converting program is, the more similar the objects are.
For example, the negative of an image is very similar to the original one as the transformation can be simply described as “inverting the color of the image”.
In order to compare the similarity, relative distance is preferred.
Vitányi et al.
(2009) propose a normalized version of E ( x, y ) called Normalized
Information Distance (NID).
Definition 1 (NID) NID is a function: Ω × Ω → [0 , 1] , where Ω is a non-empty set, defined as:
[Equation]
Equation (6) can be interpreted as follows:
Given two sequences x , y , K ( y ) ≥ K ( x ) :
[Equation]
where I ( x : y ) =
K ( y ) − K
( y | x ) means the mutual algorithmic information .
I ( x : y ) K ( y ) means the shared information (in bits) per bit of information contained in the most informative sequence, and
Equation (7) here is a specific case of Equation (6).
Normalized Compression Distance (NCD) is a computable version of NID based on real-world compressors.
In this context, K ( x ) can be viewed as the length of x after being maximally compressed.
Suppose we have C ( x ) as the length of compressed x produced by a real-world compressor, then NCD is defined as:
[Equation]
NCD is thus computable in that it not only uses compressed length to approximate K ( x ) but also replaces conditional Kolmogorov complexity with
C ( xy ) that only needs a simple concatenation of x, y .
B Implementation Details
We use different hyper-parameters for full-dataset setting and few-shot setting.
For both LSTM, Bi-LSTM+Attn, fasttext, we use embedding size = 256 , dropout rate = 0 . 3 .
For full-dataset setting, the learning rate is set to be 0 .
001 and decay rate = 0 .
9 for Adam optimizer (Kingma and Ba, 2015), number of epochs = 20 , with batch size = 64 ; for few-shot setting, the learning rate = 0 .
01 , the decay rate = 0 .
99 , batch size = 1 , number of epochs = 50 for 50-shot and 100-shot, epoch = 80 for 5-shot and 10-shot.
For LSTM and Bi-LSTM+Attn, we set RNN layer = 1 , hidden size = 64 .
For fasttext, we usehidden layer whose dimension is set to be 10.
For HAN, we use 1 layer for both word-level
RNN and sentence-level RNN, the hidden size of both of them are set to 50, the hidden sizes of both attention layers are set to be 100.
It’s trained with batch size = 256 , 0 .
5 decay rate for 6 epochs.
For BERT, the learning rate is set to be 2 e −and the batch size is set to be 16 .
For charCNN and textCNN, we use the same hyper-parameters setting in Adhikari et al.
(2019b) except when in the few-shot learning setting, we reduce the batch size to 1 , reducing the learning rate to 1 e − 4 and increase the number of epochs to 60 .
For VDCNN, we use the shallowest 9 -layer version with embedding size set to be 16 , batch size set to be 64 learning rate set to be 1 e −for full-dataset setting and batch size = 1 , epoch number = 60 for few-shot setting.
For RCNN, we use embedding size = 256 , hidden size of RNN = 256 , learning rate = 1 e − 3 and same batch size and epoch setting as VDCNN for full-dataset and few-shot settings.
For pre-processing, we don’t use any pre-trained word embedding for any word-based models.
Neither do we use data augmentation during the training.
The procedures of tokenization for both wordlevel and character-level, padding for batch processing are, however, inevitable.
For all zero-training methods, the only hyperparameter is k .
We set k = 2 for all the methods on all the datasets and we report the maximum possible accuracy getting from the experiments for each method.
For Sentence-BERT, we use the “paraphrase-MiniLM-L6-v2” checkpoint.
For neural network methods, we use publicly available code for charCNN and textCNN implemented by Adhikari et al.
(2019b),
and we use Wolf et al.
(2020) for BERT.
Our method only requires CPUs and we use 8core CPUs to take advantage of multi-processing.
The time of calculating distance matrix using gzip takes about half an hour on AGNews, two days on
DBpedia and SogouNews, six days on YahooAnswers.
All the datasets can be downloaded from torchtext and text categorization corpora.
C Few-Shot Results
The exact numerical value of accuracy shown in Figure 1 is listed in three tables below.
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Table]