Algorithmic Diversity and Tiny Models: Comparing Binary Networks and the Fruit Fly Algorithm on Document Representation Tasks (2022)

Abstract: Neural language models have seen a dramatic increase in size in the last years. While many still advocate that ‘bigger is better’, work in model distillation has shown that the number of parameters used by very large networks is actually more than what is required for state-of-the-art performance. This prompts an obvious question: can we build smaller models from scratch, rather than going through the inefficient process of training at scale and subsequently reducing model size. In this paper, we investigate the behaviour of a biologically inspired algorithm, based on the fruit fly’s olfactory system. This algorithm has shown good performance in the past on the task of learning word embeddings. We now put it to the test on the task of semantic hashing. Specifically, we compare the fruit fly to a standard binary network on the task of generating locality-sensitive hashes for text documents, measuring both task performance and energy consumption. Our results indicate that the two algorithms have complementary strengths while showing similar electricity usage.

## introduction in 2022, the vast majority of state-of-the-art nlp systems are implemented as deep neural models, that is, neural networks with complex architectures which can contain hundreds of billions of parameters. such models have become so expensive to train that most institutions cannot afford anymore to generate them from scratch. they have also been shown to generate non-negligible amounts of co 2 emissions @xcite . an active research area focuses on model distillation (e.g. @xcite , that is, the process of pruning pretrained large models to only retain the weights truly essential to the system's performance. the result of distillation is a much smaller architecture, faster to run, and less memory-hungry. ## related work semantic hashing the task of semantic hashing comes from the area of information retrieval, where it is crucial to be able to cluster documents according to their semantic similarity, in order to retrieve documents given a query. one way of going about this problem is to assign a code, hash or vector representation to each document. the more similar the hash of two given documents, the more semantically related they are. these representations should be storable in a fixed number of bits so that the retrieval of documents -through hamming distance -is fast and computationally efficient. models for semantic hashing vary, from very simple methods involving counts or tf-idf values @xcite , to more complex ones involving deep learning. the current unsupervised state-of-the-art systems rely on heavy machinery such as generative models with variational autoencoders @xcite @xcite , which are capable of reducing the high-dimensional data into a low latent space. ## datasets six datasets were used to evaluate our algorithms: 20 newsgroups foot_1 (20news) @xcite , agnews foot_2 @xcite , reuters foot_3 @xcite , tmc foot_4 @xcite , wikipedia (wiki), and web of science foot_5 (wos) @xcite . statistics and short descriptions of all datasets can be found in all datasets are available to download, except wiki, which we collected by scraping english wikipedia pages. for the 20news dataset, we used the specific version sorted by date. for reuters, we used the version modapte r(90). for wos, we used the medium-size version. three datasets (20news, agnews, and reuters) had already been provided separate train and test sets. thus we kept these test sets intact for the purpose of comparison, and further split the train sets to extract validation sets. ## bnn and ffa comparison our two algorithms have many differences, from their architecture to their optimization regime. we will highlight those differences here to make our experimental results more interpretable. first, the bnn is a supervised method while the ffa is in essence unsupervised. this means that when training and optimizing the bnn, we are looking for the best network weights and ideal hyperparameters (for both pre-processing and training regime). in contrast, the ffa does not require training of weights, since the projections are random, but still requires hyperparameter setting (including pre-processing and fly-specific features such as the number of random projections per kc or the wta rate). note that the bnn uses backpropagation while the ffa only performs forward propagation. second, the bnn's natural training grounds is the classification task, which allows us to use a straightforward and efficient objective function. the ffa being unsupervised, it can be optimized on any task that can make direct use of its binary embeddings. in practice, we found that optimizing the ffa's hyperparameters on prec@k gave us better performance than the classification task, so the results we report are based on this choice. finally, we should note that the task of this paper, learning binary representations at lowdimensionality, is a natural setting for the bnn but actually challenges the ffa. the strength of the ffa is the massive expansion in dimensionality at the level of the kc layer. through this mechanism, an actual fruit fly transforms a perceptual input in 50 dimensions into a conceptual representation contained in 2000 neurons. that 40-fold expansion in dimensionality allows the fly to capture many latent features of the perceptual data, some of which, presumably, end up being useful for classification. 8 but our task requires instead that the document features be compressed in at most 128 dimensions (down from 10,000 in input). as we will see later, this will necessitate some adjustments to the original ffa. ## experimental design we implement a bnn and an ffa foot_8 with the aim of generating document hashes at 32, 64 and 128 bits. to evaluate the respective strengths of the two systems, we compare the two architectures according to the two methods described above: precision at k (prec@k) and classification (acc). further, we divide our evaluation of the ffa into three different settings, to investigate how the relation between the dimensionality of the input of the size of the kc layer affects results. recall that the original ffa expects an expansion in dimensionality which is undesirable from the point of view of the task at hand, where we seek to obtain 32-64-128 bits hashes. to alleviate this issue, we attempt to combine the original architecture with a dimensionality reduction step implemented as principle component analysis (pca). we apply this step in two different conditions. in the first one (subsequently referred to as pca+ffa), we apply pca to the input matrix and feed the first c principal components to the ffa, where c is a hyperparameter to optimize. in the second one (ffa+pca), we apply pca 'inside' the ffa, just before the wta step, in effect reducing the size of the kc layer. as control condition, we also show the results of the original ffa without intervention (henceforth 'raw ffa'). to perform classification with the ffa representations, we feed a simple logistic regression 10 model with the documents' hashes, as generated by the fly, and perform one-vs-rest classification. for all settings, we tune the values of the two hyperparameters of the pre-processing stage: the log-probability exponent p and the number t of top words considered for each document (see §3.1). for the bnn, we also investigate the learning rate of the network. for the ffa, we tune the projection size and wta rate, as well as the number of principal components retained from the pca, wherever applicable. for the lsh, there is no tuning. since the ffa generates random projections for each fly it creates, we run it 10 times for each combination of dataset and hyperparameters, and select the instance with the best performance on the train set. (recall that the ffa is unsupervised, so we simply compute prec@k on the n documents seen in training.) we also run 10 times for lsh and average the results. all results reported in §7 are for the best models obtained from the optimization process. ## conclusion this paper set out to compare the respective strengths of two lightweight binary hashing algorithms, the binary neural network (bnn) and the fruit fly algorithm (ffa), on the task of generating highly-compressed document representations. we adapted the original ffa to this new context by prepending a dimensionality reduction step to the architecture, implemented with pca. we found that the two methods display different strengths and achieved their top performance on different tasks and datasets. both are energy-efficient, with the ffa's consumption being mostly taken by evaluation at optimization stage. both bnns and biologically-inspired algorithms are relatively new in nlp, and therefore require a lot of community efforts to fully understand their respective behaviours. as immediate further work, we would perform an in-depth analysis on our datasets' distributions to understand better why some data seem more suited to one or the other algorithm, and why discrepancies emerge in the way that classification and precision at k are tackled. from an efficiency point of view, we will also further investigate how to reduce the cost of the dimensionality reduction step before applying the ffa. one of the main strengths of both models is their low running costs. all steps can be run in a cpu or even on a mobile phone -as in the case of the bnn. this is a crucial point when it comes to providing high quality systems based on artificial intelligence models for low-resource communities. as an example application, we have integrated the pca+ffa pipeline to the pears search engine. 13 pears implements local internet search by allowing users to index and search web documents on their home machine. it requires an indexing method that is as lightweight as possible for users with limited hardware resources. the ffa is a natural choice here, with its good performance on the prec@k metric. we hope that the work presented in this paper will inspire other researchers to invest effort in developing lightweight techniques to solve core nlp problems, and share them with the communities that benefit from them.