: The illustration of the working flow of LFBC learning. The algorithm intends to preserve the pairwise F2F structure and the I2C distances and outputs the optimal bilinear projection matrices Θ 1 and Θ 2 . The potential value of hashing techniques has led to it becoming one of the most active research areas in computer vision and multimedia. However, most existing hashing methods for image search and retrieval are based on global representations, e.g., GIST [3], which lack the analysis of the intrinsic geometric property of local features and heavily limit the effectiveness of the hash code. In this paper, we propose an supervised local feature hashing framework, i.e., Local Feature Binary Coding (LF-BC), for visual similarity search, in which the feature-to-feature (F2F) and image-to-class (I2C) structures are successfully preserved and combined together. Specifically, the F2F structure considers the pairwise relationship between local features in the original feature space. While, from a higher-level aspect, I2C structure reflects the connection between images and their corresponding classes, which is derived from [1] . The outline of the proposed method is illustrated in Fig. 1 . It is worthwhile to highlight several properties of the proposed method: (1) Different with global representation based hashing, LFBC directly learns hashing function from local features and simultaneously preserves pairwise F2F and I2C structure, which is proved to be more effective for accurate retrieval. (2) Inspired by [2, 4] , bilinear projection based hashing function is adopted in our method. Thus, the complexity of the eigen-decomposition, which is the cubic form of the dimensionality, will be significantly reduced. The corresponding integrated LFBC algorithm is depicted in Algorithm 1. Input: Local feature set of each training image X i = {X i1 , · · · , X im i } in matrix form, i = 1, · · · , n, the whole local feature set F = X i , the parameter k for pairwise structure preserving, the number of centroids K in K-means and the label information function C(·) : F → {1, · · · ,C}. Output: The bilinear projection matrices Θ 1 and Θ 2 . 1: Construct local feature pairing set P = {(i, j)|X i , X j ∈ F} and their corresponding pairwise labels i j = {−1, +1}, where , and i j = −1 otherwise; 2: Employ the K-means clustering algorithm on the set of local features of each class C(X i )=c X i , c = 1, · · · ,C; Optimize Θ 1 and Θ 2 alternately; 7: until the objective function L(Θ 1 , Θ 2 ) converges. In the retrieval phase, considering that our method is specifically designed for local features, the original Hamming Ranking and Hamming Table cannot be directly applied to local features for visual index- ing. Thus, in this paper, we also introduce an indexing/searching scheme called Local Hashing Voting (LHV) as shown in Fig. 2 , which has been demonstrated to be efficient and accurate for image similarity search in our experiments. For instance, given a bucket with hash code [1,1,-1,-1,1,-1,1,-1,-1,1] of a local feature, we store the indices of the images, which contain the same local feature hash code with this bucket. In this way, we search the hash code H(q i ) for each local feature q k ∈ Q in the query image Q = {q 1 , · · · , q m } over the Hamming lookup table within Hamming radius r and return the possible images' indices. Finally, we vote and accumulate the times of each image's indices appearing in relevant buckets and then rank them in decreasing order. The final retrieved samples are returned according to the relevant ranking generated by LHV. The experimental results on the Caltech-256, SUN397 and NUS-WIDE
In fact, one of the most related work using bilinear projection on local feature hashing can be found in REF , which is regarded as a supervised learning method for image similarity search.
Local Feature Binary Coding for Approximate Nearest Neighbor Search
BMVC
Computer Science