The Document Vectors Using Cosine Similarity Revisited (2022)

Abstract: The current state-of-the-art test accuracy (97.42%) on the IMDB movie reviews dataset was reported by Thongtan and Phienthrakul (2019) and achieved by the logistic regression classifier trained on the Document Vectors using Cosine Similarity (DV-ngrams-cosine) proposed in their paper and the Bag-of-N-grams (BON) vectors scaled by Naïve Bayesian weights. While large pre-trained Transformer-based models have shown SOTA results across many datasets and tasks, the aforementioned model has not been surpassed by them, despite being much simpler and pre-trained on the IMDB dataset only. In this paper, we describe an error in the evaluation procedure of this model, which was found when we were trying to analyze its excellent performance on the IMDB dataset. We further show that the previously reported test accuracy of 97.42% is invalid and should be corrected to 93.68%. We also analyze the model performance with different amounts of training data (subsets of the IMDB dataset) and compare it to the Transformer-based RoBERTa model. The results show that while RoBERTa has a clear advantage for larger training sets, the DV-ngrams-cosine performs better than RoBERTa when the labeled training set is very small (10 or 20 documents). Finally, we introduce a sub-sampling scheme based on Naïve Bayesian weights for the training process of the DV-ngrams-cosine, which leads to faster training and better quality.

## introduction the word2vec algorithm originally published by @xcite is among the most famous methods to train vector representations of words. soon after the emergence of word2vec, a similar method to build vector representations of documents was originally proposed by @xcite and further studied by @xcite . it is known under different names, including paragraph vectors, sentence vectors, doc2vec, etc. this method jointly learns word embeddings and document embeddings such that a binary classifier can predict if a given word occurs in a particular document given only the corresponding embeddings. more formally, the following objective is minimized: later @xcite switched from single words to n-grams and observed significant improvements. building on that, @xcite studied different objective functions. they have found that the cosine similarity outperforms the dot product, which led to a modified model called the document vectors using cosine similarity (we will call it dv-ngrams-cosine for short). the new objective is: in the same paper, the authors proposed an ensemble consisting of the document embeddings from dv-ngrams-cosine and the bag-of-n-grams vectors scaled by naive bayesian weights (nbweighted bon for short). they concatenated these two representations and trained the logistic regression classifier on top. the ensemble was reported to have very high test accuracy (97.42%) on the imdb movie reviews dataset @xcite ). to the best of our knowledge, this accuracy remains the sota result on imdb. even large transformerbased models pre-trained on a huge amount of texts, both in-domain and out-of-domain, have shown lower accuracy on this dataset @xcite @xcite . this extraordinary performance of such a simple model motivated us to thoroughly study the model and its implementation trying to understand the reasons behind its success. unfortunately, during this study, we found a bug in the implementation of the evaluation procedure of the ensemble, which had made the estimation of the accuracy incorrect. in our paper, we re-evaluate the ensemble as well as its individual components. we show that the originally reported test accuracy of the ensemble (97.42%) is incorrect and shall be corrected to 93.68%, which is only 0.55% higher than the accuracy on pure dv-ngrams-cosine embeddings. additionally, we analyze how the amount of training data affects the performance of the ensemble, as well as its individual components, and also the transformer-based roberta model @xcite , which has recently shown sota or near-sota results over a variety of tasks and datasets. surprisingly, we have observed that dv-ngramscosine outperforms roberta when the number of labelled training examples is small (10 or 20). we also ensemble roberta with dv-ngrams-cosine, but only have achieved a marginal improvement. finally, we propose a modification for the training process of dv-ngrams-cosine that results in faster training and better accuracy. the code reproducing our experiments is publicly available foot_0 . ## re-evaluation of the ensemble in the aforementioned ensemble proposed by @xcite , the nbweighted bon and the dv-ngrams-cosine are concatenated and fed into the logistic regression classifier. however, we have found that in the original implementation the two vectors concatenated to obtain a single training or test example usually correspond to two different documents of the same class (see details in appendix a). specifically, the dv-ngrams-cosine vectors and the bon vectors are built from two different files having different orders of examples. as a result, after the concatenation, each input to the logistic regression corresponds to a combination of two examples. due to the special structure of the files, those examples are guaranteed to belong to the same class and the same subset. for instance, a positive example from the test set is concatenated with another positive example from the test set. in appendix b.3 we provide an analysis that shows the reasons of high performance of this concatenation of two representations. from this analysis it follows that most examples from imdb are correctly classified with high confidence (a large logit) using any of two representations, i.e. they are easy examples. less than 10% of examples are classified incorrectly by each representation (hard examples), but they often obtain low confidence (a logit near zero). hard examples are more often combined with easy examples just because of their dominance. in these cases, the logit from the easy example often outweigh the logit from the hard one resulting in the correct final prediction. thus, in both the training and the test sets, hard examples are often combined with simpler examples, making the classification task easier. in this process, the knowledge of the true labels is implicitly exploited to combine the examples this way, in both training and testing. this leads to an incorrect estimation of the classification accuracy for future examples. after fixing this issue, we have observed that the combination of different representations of the same document leads to the test accuracy of 93.68% instead of 97.42% originally reported. compared to the pure dv-ngrams-cosine embeddings, the ensemble improves the test accuracy by 0.55%, not 4.29% reported previously. this improvement also better agrees with the improvements of less than 1% observed by @xcite for similar ensembles with the predecessor model dv-ngram. as a sanity check, appendix b additionally reports the accuracy for different schemes of combining the two representations, showing that higher accuracy can be achieved only by those schemes that exploit the knowledge of the test labels. ## further analysis of performance in his section we further analyze the performance of the ensemble described above, comparing it to its individual components as well as to the recently introduced transformer-based roberta model @xcite . we study the performance of these models depending on the number of labelled examples in the training set. for a more fair comparison, the most important hyperparameters of each model were tuned on the validation set, employing the train/validation/test split of the imdb dataset provided by @xcite . subsets of different sizes from 10 to 20000 examples were randomly sampled from the training set. the logistic regression classifier was trained on these subsets using the dv-ngram-cosine embeddings, the nb-weighted bon vectors, or their concatenation as its input representation. we tuned the l2-regularization strength c of the classifier individually for each subset of the training set. additionally, we multiplied the dv-ngramcosine embeddings before concatenating them to the bon vectors in order to balance the magnitudes of the two representations, which may help the classifier to benefit from both representations. the scaling factor was also selected on the validation set. the pre-trained roberta base model foot_1 was finetuned on a part (10 out of 30) of the same subsets of the training set, using the validation set for early stopping. we used a batch size of 32, with a maximum learning rate of 1e-5, recommended by fairseq foot_2 . as shown in fig. 1 , the fine-tuned roberta model usually achieves higher test accuracy. but when the number of labelled training examples is very small (10 or 20), the logistic regression on the dv-ngrams-cosine embeddings shows higher mean test accuracy and lower standard deviation. this result corroborated the notion that small models can be a better choice when the data are scarce. on the other hand, logistic regression on the bon vectors performs significantly worse than all other models across all training set sizes. finally, we don't observe any significant improvements from the ensembling when the training set size is less than 20k, as the difference is within one standard deviation. it is important to notice that the dv-ngramscosine embeddings were pre-trained on the indomain examples from the whole imdb dataset, while roberta was pre-trained on a huge but general-domain corpus. it is likely that the domain adaptation techniques @xcite will help roberta when the number of labelled examples is small. however, for our study, we decided to compare the most standard approaches to training the corresponding models. ## nb sub-sampling in this section, we improve the training procedure of dv-ngrams-cosine by applying a sub-sampling procedure based on the naive bayesian weights of ngrams (nb sub-sampling) in order to make the model focus more on sentiment-related ngrams while building the document embeddings. inspired by the previous works @xcite , arefyev et al. ( 2021 )), we trained a multinomial naive bayesian classifier and exploited its weights to calculate the importance of each ngram f i for the final classification task: in each epoch we put an ngram into training with the probability where n a and n b are the hyperparameters. the choices are purely empirical. we tried different combinations of n a and n b and found 2 and 3 (respectively) to be the best in them. the comparison of the training process with and without nb sub-sampling is shown in fig. 2 (refer to appendix c for details of the experiments and the accuracy on the validation set). the runs with nb sub-sampling progress faster and show a distinct advantage after 2500 steps. after 30k steps, the runs with nb sub-sampling stagnated and kept fluctuating in a small region; the vanilla runs stagnated after 50k steps, in a lower area. it is also worth noticing that although the labels of the training set are used during pre-training for sub-sampling, we did not observe any significant overfitting due to that. neither the validation score nor the test score showed a tendency to decay long after reaching the plateau, indicating that this sub-sampling scheme can be used as an add-on to the original model, boosting its performance while not creating additional overfitting trouble. ## ensemble dv-ngrams-cosine and roberta thongtan and phienthrakul (2019)) and described in section 2 combines two different representations of documents, which are the dv-ngrams-cosine embeddings and the nb-weighted bon vectors. however, we have observed in section 3 that the bon vectors are quite weak on their own, while roberta outperforms all other models unless the number of examples is very small. thus, it is interesting if dvngram-cosine can help roberta. in this section, we combine the dv-ngrams-cosine (with or without nb sub-sampling) with the output of the last hidden layer of roberta, and test on the imdb dataset. again, the train/validation/test splits by suchin et al. (2020) were used. a scaling factor on the dv-ngrams-cosine and the hyperparameter c in the logistic regression were tuned on the validation set. the results are shown in ## conclusion the ensemble featuring the dv-ngrams-cosine reported by @xcite was re-evaluated. the test accuracy of this ensemble on the imdb dataset was corrected from 97.42% to 93.68%. the dv-ngrams-cosine embeddings with the logistic regression on top were compared with roberta using different amounts of training data. in this comparison, the dv-ngrams-cosine has surprisingly outperformed roberta for a small number of training examples (10 or 20 documents). a sub-sampling scheme based on the naive bayesian weights was introduced to the training process of the dv-ngrams-cosine, resulting in faster training and better quality.