Integrating Semantics and Neighborhood Information with Graph-Driven Generative Models for Document Retrieval (2021)

Abstract: With the need of fast retrieval speed and small memory footprint, document hashing has been playing a crucial role in large-scale information retrieval. To generate high-quality hashing code, both semantics and neighborhood information are crucial. However, most existing methods leverage only one of them or simply combine them via some intuitive criteria, lacking a theoretical principle to guide the integration process. In this paper, we encode the neighborhood information with a graph-induced Gaussian distribution, and propose to integrate the two types of information with a graph-driven generative model. To deal with the complicated correlations among documents, we further propose a tree-structured approximation method for learning. Under the approximation, we prove that the training objective can be decomposed into terms involving only singleton or pairwise documents, enabling the model to be trained as efficiently as uncorrelated ones. Extensive experimental results on three benchmark datasets show that our method achieves superior performance over state-of-the-art methods, demonstrating the effectiveness of the proposed model for simultaneously preserving semantic and neighborhood information.

## introduction similarity search plays a pivotal role in a variety of tasks, such as image retrieval @xcite , plagiarism detection @xcite and recommendation systems @xcite . if the search is carried out in the original continuous feature space directly, the requirements of computation and storage would be extremely high, especially for large-scale applications. semantic hashing @xcite sidesteps this problem by learning a compact binary code for every item such that similar items can be efficiently found according to the hamming distance of binary codes. unsupervised semantic hashing aims to learn for each item a binary code that can preserve the semantic similarity information of original items, without the supervision of any labels. motivated by the success of deep generative models @xcite @xcite in unsupervised representation learning, many recent methods approach this problem from the perspective of deep generative models, leading to state-of-the-art performance on benchmark datasets. specifically, these methods train a deep generative model to model the underlying documents and then use the trained generative model to extract continuous or binary representations from the original documents @xcite @xcite . the basic principle behind these generative hashing methods is to have the hash codes retaining as much semantics information of original documents as possible so that semantically similar documents are more likely to yield similar codes. in addition to semantics information, it is widely observed that neighborhood information among the documents is also useful to generate high-quality hash codes. by constructing an adjacency matrix from the raw features of documents, neighborbased methods seek to preserve the information in the constructed adjacency matrix, such as the locality-preserving hashing @xcite , spectral hashing @xcite , and etc. however, since the groundtruth neighborhood information is not available and the constructed one is neither accurate nor complete, neighbor-based methods alone do not perform as well as the semantics-based ones. despite both semantics and neighborhood information are derived from the original documents, different aspects are emphasized in them. thus, to obtain higher-quality hash codes, it has been proposed to incorporate the constructed neighborhood information into semantics-based methods. for examples, @xcite and @xcite require the hash codes can reconstruct neighboring documents, in addition to the original input. other works @xcite use an extra loss term, derived from the approximate neighborhood information, to encourage similar documents to produce similar codes. however, all of the aforementioned methods exploit the neighborhood information by using it to design different kinds of regularizers to the original semanticsbased models, lacking a basic principle to unify and leverage them under one framework. to fully exploit the two types of information, in this paper, we propose a hashing method that unifies the semantics and neighborhood information with the graph-driven generative models. specifically, we first encode the neighborhood information with a multivariate gaussian distribution. with this gaussian distribution as a prior in a generative model, the neighborhood information can be naturally incorporated into the semantics-based hashing model. despite the simplicity of the modeling, the correlation introduced by the neighbor-encoded prior poses a significant challenge to the training since it invalidates the widely used identical-andindependent-distributed (i.i.d.) assumption, making all documents correlated. to address this issue, we propose to use a tree-structured distribution to capture as much as possible the neighborhood information. we prove that under the tree approximation, the evidence lower bound (elbo) can be decomposed into terms involving only singleton and pairwise documents, enabling the model to be trained as efficiently as the models without considering the document correlations. to capture more neighborhood information, a more accurate approximation by using multiple trees is also developed. extensive experimental results on three public datasets demonstrate that the proposed method can outperform state-of-the-art methods, indicating the effectiveness of the proposed framework in unifying the semantic and neighborhood information for document hashing. in which w j denotes the |v |-dimensional one-hot representation of the j-th word, with |x| and |v | denoting the document and vocabulary size, respectively; and @xmath0. for a corpus containing n documents @xmath1, x 2 , • • • , x n }, due to the i.i.d. assumption for documents, it is modelled by simply multiplying individual document models as ## a hashing framework with unified semantics-neighborhood information in this section, we present a more effective framework to unify the semantic and neighborhood information for the task of document hashing. ## training with tree approximations although the prior p g (z) captures the full neighborhood information, its induced model is not practically trainable. in this section, to facilitate the training, we first propose to use a tree-structured prior to partially capture the neighborhood information, and then extend it to multiple-tree case for more accurate modeling. ## related work deep generative models @xcite have attracted a lot of attention in semanticsbased hashing, due to their successes in unsupervised representation learning. vdsh @xcite first employed variational autoencoder (vae) @xcite to learn continuous representations of documents and then casts them into binary codes. however, for the sake of information leaky problem during binarization step, such a two-stage strategy is prone to result in local optima and undermine the performance. nash @xcite tackled this issue by replacing the gaussian prior with bernoulli and adopted the straight-through technique @xcite to achieve end-to-end training. to further improve the model's capability, @xcite proposed to employ mixture distribution as a priori knowledge and zheng et al. (2020) exploited boltzmann posterior to introduce correlation among bits. beyond generative frameworks, ammi @xcite ) achieved superior performance by maximizing the mutual information between codes and documents. nevertheless, the aforementioned semantic hashing methods are consistently under the i.i.d. assumption, which means they ignore the neighborhood information. spectral hashing @xcite and selftaught hashing @xcite are two typical methods of neighbor-based hashing models. but these algorithms generally ignore the rich semantic information associated with documents. recently, some vae-based models tried to concurrently take account of semantic and neighborhood information, such as nbrreg @xcite , rbsh @xcite and pairrec @xcite . however, as mentioned before, all of them simply regarded the proximity as regularization, lacking theoretical principles to guide the incorporation process. thanks to the virtue of graph-induced distribution, we effectively preserve the two types of information in a theoretical framework. baselines we compare our method with the following models: sph @xcite , sth @xcite , vdsh @xcite , nash @xcite , gmsh @xcite , nbrreg @xcite , corrsh @xcite and ammi @xcite . for all baselines, we take the reported performance from their original papers. training details for fair comparisons, we follow the same network architecture used in vdsh, gmsh and corrsh, using a one-layer feedforward neural network as the variational and the correlated encoder. the graph g is constructed with the k-nearest neighbors (knn) algorithm based on cosine similarity on the tfidf features of documents. in our experiments, the correlation strength coefficient λ in ( 12 ) is fixed to 0.99. according to the performance observed on the validation set, we choose the learning rate from {0.0005, 0.001, 0.003}, batch size from {32, 64, 128}, the temperature τ in sigmoid function from {0.1, 0.2, • • • , 1}, the number of trees m and neighbors k both form {1,2,. . . ,20}, with the best used for evaluation on the test set. the model is trained using the adam optimizer (kingma and ba, 2014). more detailed experimental settings, along with the generating method of spanning trees, are given in the supplementary materials. ## conclusion we have proposed an effective and efficient semantic hashing method to preserve both the semantics and neighborhood information of documents. specifically, we applied a graph-induced gaussian prior to model the two types of information in a unified framework. to facilitate training, a treestructure approximation was further developed to decompose the elbo into terms involving only singleton or pairwise variables. extensive evaluations demonstrated that our model significantly outperforms baseline methods by incorporating both the semantics and neighborhood information. appendices a derivation of formulas derivation of kl (q φ (z|x)||p t (z)) in the main paper, we propose a tree-type distribution to introduce partial neighborhood information so that the l 2 term can be expressed as the summation over terms involving only one or two variables. here, we provide the detail derivation. obviously, the kl divergence is decomposed into the terms involving singleton and pairwise variables, which can be calculated efficiently. expressing l t in analytical form for simplification, in the following, we use µ 1 , σ 1 to represent the mean and variance matrix of q t (z i , z j |x i , x j ), respectively, and represent those of p g (z i , z j ) as µ 2 , σ 2 , respectively. besides we denote λa ij as τ ij so we have τ @xmath2. by applying the cholesky decomposition on the covariance matrix of σ 1 and σ 2