Unsupervised Few-Bits Semantic Hashing with Implicit Topics Modeling (2020)

Abstract: Semantic hashing is a powerful paradigm for representing texts as compact binary hash codes. The explosion of short text data has spurred the demand of few-bits hashing. However, the performance of existing semantic hashing methods cannot be guaranteed when applied to few-bits hashing because of severe information loss. In this paper, we present a simple but effective unsupervised neural generative semantic hashing method with a focus on few-bits hashing. Our model is built upon variational autoencoder and represents each hash bit as a Bernoulli variable, which allows the model to be end-to-end trainable. To address the issue of information loss, we introduce a set of auxiliary implicit topic vectors. With the aid of these topic vectors, the generated hash codes are not only low-dimensional representations of the original texts but also capture their implicit topics. We conduct comprehensive experiments on four datasets. The results demonstrate that our approach achieves significant improvements over state-of-the-art semantic hashing methods in few-bits hashing.

## introduction semantic hashing @xcite is an attractive strategy for fast similarity search, which aims to find the most relevant texts for a given query @xcite . the basic idea of semantic hashing is to embed the semantics of texts into a low-dimensional binary vector space, while preserving text similarity. the embedded representations are called hash codes, based on which the calculation of text similarity can be efficiently completed by computing the hamming distance using xor operation @xcite . while considerable research efforts have been devoted to semantic hashing @xcite @xcite @xcite ; dadaneh et al., 2020), none of them have paid attention to few-bits hashing. their performance also cannot be guaranteed when directly applied to few-bits hashing due to severe information loss. however, compactness is a crucial factor in learning to hash @xcite . it is important to keep hash codes as short as possible. for a text collection with c topics, the ideal length of hash codes is just log 2 (c) @xcite . in addition, with the explosive growth of social media and e-commerce, more and more short text data (e.g., tweets and online reviews) are generated everyday on the web. it would be a huge waste to represent them as long hash codes. therefore, it is necessary to ensure the performance of few-bits hashing, which is relatively an under-studied problem. in this paper, we propose a simple but effective unsupervised neural generative semantic hashing method wish (few-bits semantic hashing), which focuses on few-bits hashing. the architecture of wish is shown in figure 1 . built upon variational autoencoder (vae) @xcite , wish learns hash codes directly via the inference network. however, when using these binary codes as the inputs of the generative network, the model may encounter severe information loss (i.e., the information transmitted from the inference network to the generative network may be relatively limited, especially in the few-bits case). as thus, the generative network has little chance to effectively reconstruct the input texts. to address this issue, we introduce a set of auxiliary continuous implicit topic vectors. and we assume each text is generated from one or more of these topic vectors. specifically, the inference network is used to decide which topic vectors are selected, and the generative network is used to reconstruct the input texts based on the selected topic vectors. thus, the output of the inference network should be binary. to this end, we model the output of the inference network as either deterministic or stochastic multivariate bernoulli variables. the inference network and the generative network are optimized jointly by maximizing the variational lower bound of the text log likelihood. and the straight-through estimator @xcite is utilized to estimate the gradients with respect to the binary codes. in summary, the main contributions include: ## related work up to now, lots of hashing methods have been proposed @xcite , which can be roughly categorized into unsupervised methods and supervised methods. in this paper, we focus on unsupervised methods since it is laborious to get labels for large-scale text collections. unsupervised methods attempt to employ the data properties such as manifold structures and distributions to learn hash functions. for example, graph hashing @xcite learns the hash function by utilizing the underlying manifold structure. self-taught hashing (sth) @xcite decomposes the learning procedure into two steps: first generating hash codes via unsupervised learning and then learning hash functions by treating the previously generated hash codes as pseudo labels. owing to the success of deep learning, many deep learning-based hashing methods have been proposed in recent years @xcite @xcite . for text hashing, @xcite were the first to propose a deep generative model called variational deep semantic hashing (vdsh). chaidaroon et al. @xcite further proposed an improved version of vdsh, which employs unsupervised ranking methods such as bm25 @xcite to extract weak signals from training data. in consideration of the pervasiveness of text relationships, node2hash @xcite considers both text contents and connection information. however, these methods are not endto-end trainable, because they generate the final hash codes by using the median method @xcite for binarization. shen et al. @xcite proposed an end-to-end trainable generative semantic hashing method nash that learns hash codes directly. bmsh @xcite enhances nash by imposing mixture priors. in @xcite , a ranking based semantic hashing (rbsh) method was proposed, which is also an extension of nash by incorporating text similarity into the hash code generation. although the above methods have demonstrated promising results in semantic hashing, they pay no attention to few-bits hashing. due to severe information loss, their performance also cannot be guaranteed if applying them to few-bits hashing directly. our model focuses on few-bits hashing and introduces a set of auxiliary implicit topic vectors to mitigate information loss. the learned hash codes are able to capture the implicit topics of texts. ## conclusion in this paper, we have presented a simple but effective unsupervised neural generative semantic hashing method with a focus on few-bits hashing. to address the problem of information loss in fewbits hashing, we have introduced a set of auxiliary implicit topic vectors. with the aid of these topic vectors, our approach can well capture the semantics of texts, thus the learned hash codes are not only low-dimensional representations of the original texts but also capture their implicit topics. we have further analyzed that our approach can be treated as an ltm model although it is fundamentally different from existing ltm models. to evaluate the effectiveness of our approach, we have conducted a comprehensive set of experiments, the results demonstrate the superiority of our approach over existing semantic hashing methods.