Refining BERT Embeddings for Document Hashing via Mutual Information Maximization (2021)

Abstract: Existing unsupervised document hashing methods are mostly established on generative models. Due to the difficulties of capturing long dependency structures, these methods rarely model the raw documents directly, but instead to model the features extracted from them ( e.g. bag-of-words (BOG), TFIDF). In this paper, we propose to learn hash codes from BERT embeddings after observing their tremendous successes on downstream tasks. As a first try, we modify existing generative hashing models to accommodate the BERT embeddings. However, little improvement is observed over the codes learned from the old BOG or TFIDF features. We attribute this to the reconstruction requirement in the generative hashing, which will enforce irrelevant information that is abundant in the BERT embeddings also compressed into the codes. To remedy this issue, a new unsupervised hashing paradigm is further proposed based on the mutual information (MI) maximization principle. Specifically, the method first constructs appropriate global and local codes from the documents and then seeks to maximize their mutual information. Experimental results on three benchmark datasets demonstrate that the proposed method is able to generate hash codes that outperform existing ones learned from BOG features by a substantial margin.

## introduction with the explosion of information, similarity search @xcite plays a increasingly important role in modern information retrieval systems. traditional search engines conduct query by evaluating the distances of items in the continuous euclidean space, making it suffer from high computational complexity and footprint. to address this issue, considerable efforts have been devoted to semantic hashing @xcite , which aims to represent each document by a compact binary code. such representations are able to reduce the memory footprint and increase the retrieval efficiency significantly by enrolling in binary hamming space. a pivotal challenge in learning high-quality hash codes is how to retain the semantic similarities among documents. although using supervised information is an efficient way to achieve this goal, due to the high cost of labeling, unsupervised hashing is more favourable in practice. currently, most of unsupervised document hashing methods are established upon the perspective of deep generative models @xcite . essentially, all these methods seek to model the documents with a deep generative model and then employ the latent representations of documents to construct hash codes @xcite @xcite @xcite . although great successes have been observed in these methods, due to the difficulties in capturing the long dependency structures of words (especially for long documents), all of these methods are established on modeling the bow or tfidf features of documents. although the bow or tfidf features are informative and are prevalent in many areas, their limitations are also obvious for not considering the word order and dependency structure. recently, large-scale pre-trained language models like bert @xcite have demonstrated their superior capabilities on various natural language understanding tasks. embeddings extracted from them have also been shown to contain much more abundant information. thus, in this paper, we argue that capitalizing on bert embeddings to produce hash codes is better than on the out-of-date bow features. as a first try, we modify existing generative hashing methods to accommodate the bert embeddings and then use the trained model to generate hash codes. however, experimental results show no improvement on the quality of obtained hash codes. even worse, the codes sometimes perform even inferior to those learned from bow features. we conjecture that this is because the reconstruction requirement in generative hashing enforces most of the information in bert embeddings to be transferred into the hash codes. however, as the information contained in bert embeddings is very abundant, with only a small proportion relevant to hashing, it is not surprising to see that the codes are not aligned well with the semantic similarities of documents. to generate high-quality hash codes from bert embeddings, it becomes necessary to refine the embeddings to highlight the information relevant to hashing tasks (i.e., semantic information), while attenuating the irrelevant. recent progresses on image representation learning have shown that it is possible to learn discriminative semantic representations using the mutual information (mi) maximization principle. inspired by this, rather than utilizing the reconstruction structure, an alternative paradigm is proposed for unsupervised document hashing based on the mi maximization principle, named deep hash infomax (dhim). the essential idea behind our approach is to construct appropriate global and local codes and then seek to maximize their mutual information, with the global and local codes accounting for the entire document and text fragments, respectively. as explained in image representation learning, doing so implicitly encourages the global codes to retain high-level semantic information shared across different local fragments, while ignoring the local irrelevant details. extensive experiments are conducted on three benchmark datasets. the results demonstrate that by effectively refining the bert embeddings via mi maximization principle, the proposed method is able to generate hash codes that outperform existing ones learned from bow features by a substantial margin. ## preliminaries on generative hashing for documents document hashing aims to learn close binary codes for semantically similar documents. an intuitive idea towards this goal is to encourage hash codes preserving as much information of documents as possible so that close codes are easier to be obtained for similar documents. based on this idea, many methods have be proposed to employ generative models like vaes to model the documents and then leverage the documents' latent representations to produce binary hash codes. however, due to the difficulties in capturing the long dependency structures of words (especially for long documents), existing generative hashing methods rarely seek to model the documents directly, but instead to first extract representative features from documents (e.g., bow or tfidf) and then perform modeling on the extracted features. specifically, by representing a document x as a sequence of words @xmath0, w 2 , . . . , w |x| }, existing generative hashing methods @xcite are mostly established on the following document model here z is the latent variable; w j is a |v |dimensional one-hot vector corresponding to the j-th word; @xmath1; b i is the biased term; and |v | and |x| represent the vocabulary size and document length, respectively. the whole model is trained by maximizing the evidence lower bound (elbo) of log-likelihood with respect to θ and φ, where q φ (z|x) denotes the approximate posterior distribution parameterized by φ. after training, representation of the document x can be extracted from the approximate posterior q φ (z|x), e.g., using its output mean. note that a simple decoder of (2) is adopted purposely for better transferring similarity information of documents x into the latent representations z. in the early generative hashing work vdsh @xcite , gaussian distributions are employed for both the prior p(z) and approximate posterior q φ (z|x) directly. but due to the continuous characteristics of gaussian random variables, a separate binarization step is required to transform the continuous latent representations into binary codes. to overcome the separate training issue, bernoulli prior and posterior are then proposed in nash @xcite . with the recent advances on gradient estimators for discrete random variables, the model successfully circumvents gradient backpropagation issue for discrete variables, and can be trained efficiently in an end-to-end manner. inspired by nash, many variant methods are then proposed by using more sophisticated prior or posterior distributions, with the objective to model the documents more accurately, such as bernoulli mixture prior in bmsh @xcite and boltzmann machine posterior in corrsh @xcite etc. despite of the observed remarkable performance, all of the methods mentioned above rely on the document model (1), which, however, is essentially established on the bow features of documents, without considering any word order and dependency information. although bow features are informative, their limitations are also obvious due to the neglect of word order and dependency structure. with the development of large-scale pre-trained models like bert, it becomes easy to obtain semanticsrich features that contain long dependencies and contextual information. thus, we argue that it is beneficial to capitalize on the information-rich bert embeddings over the out-of-date bow features to learn hash codes. ## hashing on bert embeddings via generative models feeding a document @xmath2, w 2 , • • • , w |x| } into a pre-trained bert model could produce an embeding/feature for the document, denoted as b(x) for subsequent presentation. inspired by the success of generative hashing methods, we modify them to accommodate the bert embeddings. due to the difference between bert embeddings and bow features, the decoder in (2) is replaced by a conditional gaussain distribution unexpectedly, as observed in experiments (see ## refining bert embeddings via mi maximization according to discussions in section 3, to produce high-quality hash codes, it is necessary to refine bert embeddings to highlight the categoryrelevant information, while attenuating the other types of information. recent progresses on image representation learning @xcite have demonstrated that it is possible to learn categorydiscriminative representations from images unsu-pervisedly with the mi maximization principle. inspired by this, a brand new hashing framework based on mi maximization principle is proposed, which learns binary hash cods from bert embeddings without using the reconstruction requirement, thereby overcoming the issues associated with the generative hashing approaches. ## related work early works in unsupervised document hashing generally built upon the generative models @xcite , in which the encoder-decoder architecture was established to encourage binary codes to retain semantic information by reconstructing original data. for examples, vdsh @xcite first proposed to learn continuous representations under variational autoencoder (vae) framework, and then cast it into binary codes. however, the two-stage training procedure is prone to undermine the performance. nash @xcite tackled this issue by replacing gaussian prior with bernoulli in vae and adopting straight-through to enable end-to-end training. since then, a lot of methods surged to improve the performance. specifically, @xcite proposed to employ mixture distribution as prior to enhance model's capabilities; @xcite introduced auxiliary topic vectors to address the problem of information loss in few-bits scenarios, and @xcite employed boltzmann posterior to introduce correlation among bits. beyond generative models, ammi @xcite achieved superior performance by maximizing mutual information between documents and codes. however, the adversarial training procedure used in ammi is extremely unstable. although these models are impressive, one common issue of them is that they simply exploited bag-of-words features as input, which is not enough to capture the rich semantic information of documents. recently, information theory enables a simple and insightful paradigm of unsupervised representation learning @xcite @xcite . for example, @xcite proposed an unsupervised representation learning algorithm on image data, called deep infomax, which maximizes the mi between the whole image and local patches. @xcite and @xcite extended this idea on graph data, in which the representations can be learned by maximizing the mi between the whole and sub graphs. these methods consistently encourage the global representations to retain similar interest of local features. following similar ideas, dataset train val test classes avglen nyt 9,221 1,154 1,152 26 648 dbpedia 50,000 5,000 5,000 14 47 agnews 114,839 6,381 6,380 4 32 ## conclusion we have proposed an effective and efficient semantic hashing method by refining the bert embedding. specifically, we applied a textual convolutional neural network with probabilistic layers to capture local and global features, and refined semantic information into binary codes by maximizing their mutual information. extensive evaluations demonstrated that our model significantly outperforms baseline methods by learning hash codes under the guidance of mmi frameworks.