Word Embedding Binarization with Semantic Information Preservation (2020)

Abstract: With growing applications of Machine Learning in daily lives Natural Language Processing (NLP) has emerged as a heavily researched area. Finding its applications in tasks ranging from simple Q/A chatbots to Fully fledged conversational AI, NLP models are vital. Word and Sentence embedding are one of the most common starting points of any NLP task. A word embedding represents a given word in a predefined vector-space while maintaining vector relations with similar or dis-similar entities. As such different pretrained embedding such as Word2Vec, GloVe, fasttext have been developed. These embedding generated on millions of words are however very large in terms of size. Having embedding with floating point precision also makes the downstream evaluation slow. In this paper we present a novel method to convert continuous embedding to its binary representation, thus reducing the overall size of the embedding while keeping the semantic and relational knowledge intact. This will facilitate an option of porting such big embedding onto devices where space is limited. We also present different approaches suitable for different downstream tasks based on the requirement of contextual and semantic information. Experiments have shown comparable result in downstream tasks with 7 to 15 times reduction in file size and about 5 % change in evaluation parameters.

## introduction natural language processing (nlp) is quickly becoming one of the most important branch in machine learning field, with companies pouring millions to perfect their nlp engines. nlp is the cornerstone in the road to developing a perfect conversational ai. but nlp also finds its way in more trivial but important tasks in modern life, be it a simple chatbot, a q/a site, document classifier. for most of these tasks, an embedding model or a pre-trained embedding is the perfect starting point. a word embedding is a collection of words, represented as vectors in a predefined space. within this space the vectors follow all typical vector laws. similar words are close together, dissimilar words are far off. vector addition and subtraction are possible. word embeddings are very important for downstream tasks like document classification, query response, etc. though word embeddings can be generated to be useful to a specific task on a specific dataset, there are embeddings available that are trained on a very large corpus, making them useful in variety of tasks and in turn making them generic. word2vec @xcite , glove @xcite , fasttext @xcite are some of the best generic embeddings available that are trained on millions of vocabularies. though training on a large dataset makes the embedding useful, it also makes them very large in size. a 300-dimensional word2vec model trained on 3 million datapoints is aroung 3.4 gb. this large size makes the embedding very unportable, especially in low storage scenarios. as a result, many nlp operations are typically performed on the server. this raises issues like privacy concerns, as user data is continuously sent to server and processed there. the first step to solve this issue is to bring down the size of the embedding, small enough to carry on device such as mobile phones and devices where memory should be used judiciously. this will enable on-device training without depencency on server side. in this paper we propose two methods to binarize a pre-trained word embedding, while keeping all the semantic information intact. our proposed methods are based on an auto-encoder architecture, which encodes continuous real vectors into its corresponding binary vector. the embedding generated by the proposed methods perform well on different types of tasks, with one approach performing good in tasks involving need of contextual and semantic data, while other performing well on the tasks where such data is not crucial (approaches are discussed section 3). the embedding produced by both the proposed variations are a fraction of the original size with reduction of upto 7 to 15 times. we also demonstrate the performance of these embedding on different downstream tasks like similarity, classification and analogy. ## related work word embedding is starting point for many nlp applications, usually these embedding are very large in size and it requires a lot of time for subsequent computations.there is a need to reduce the memory footprint of these embedding and to make the computations faster. the idea is to binarize this embedding so that the amount of memory taken by a binary feature would be much lesser as compared to that of a float valued feature. in their research, @xcite proposed a nonlinear methodology where the high dimensional data was embedded into a hamming cube while maintaining the structure of the original space. as this works on dimensions far greater than the dimensions that we deal with, this study did not prove to be useful for the scope of our research. @xcite proposed to binarize the real valued embedding by first increasing the dimensions of the original embedding to create a sparse matrix and then apply a binarization step to that embedding. though this retains certain amount of semantic information, the produced vectors are not small. fasttext.zip @xcite binarizes the embedding by clustering and concatenating the binary representations of 'k' closest centroids for each word. the resulting binary vectors produced cannot be used for generic tasks as this works only for document classification. even though binary representations can make the computations faster, some nlp applications work only on real valued embedding @xcite . in such applications reconstructing real embedding from binary representations becomes very vital. the research of @xcite showed that we could binarize the word embedding using an autoencoder architecture. but the problem with this method was that it was not able to capture a lot of the semantic information. in this paper we have managed to beat the benchmarks from @xcite in most of the downstream tasks, which will be shown in section 6. we have been able to achieve this feat with the inspiration of @xcite where they binarize sentence embedding and use them for downstream tasks. we propose an approach of using these 2 studies as an inspiration and develop our binarized word embedding, which has done very well on benchmarks as show in section 6. ## methodology we propose a method that converts continuous word embedding into a binary form. let x be the word, a x refers to the continuous embedding for the respective word, b x refers to the binary embedding for the word x, f refers to a function that will convert the continuous embedding, a x to the binarized embedding of word x, b x . therefore, we aim to learn global function f that can be used to generate binary embedding of the highest quality. quality in this context refers to how informative the binary embedding are as compared to the original ones, how much semantic data from the original it has captured and how much of a size reduction we have obtained as compared to the original embedding. usability of these binary embedding in downstream tasks becomes an important aspect, as the main aim for us is to substitute the original embedding with the binarized one, and get an advantage by reducing the amount of memory used while still having comparable performance. the function f can be implemented in many ways so as to achieve the end goal of obtaining binarized representations for continuous embedding, but not all of them will show the quality required for us to achieve the end goal of substitution of the actual embedding. some of these methods are hard thresholding, random projections and encoder-decoder models. hard thresholding is a method where each feature of the embedding will be converted to the binary form of 1s and 0s. each feature of the embedding takes the value 1 if it is above a specific threshold value and value 0 if otherwise. the threshold can be selected in many ways, it can be zero, or a random value, or the mean value of the features. the problem with this method is that a huge chunk of the information will be lost directly and there is a good chance that these embedding will not be able to capture the semantic data from the original embedding. random projection method randomly projects the already learnt continuous embedding onto a different vector space. once we have projected the embedding, we use the method of hard thresholding to convert it into a binary representation. the advantage of this method is that we can alter the dimension of the word embedding to the required dimensions. the binary representations generated using the hard threshold and the random projection binarization techniques suffer from the limitation of retaining semantic information and the need to have an additional binarization step after the training of the embedding. auto encoder model can be used to overcome the above limitations by retaining the semantic information of the continuous embedding without the need of applying an individual binarization step after the training. ## experiments several tasks have been run to measure the performance of the original continuous embedding and the binarized embedding. these tasks test how much semantic information has been retained in the binary embedding. ## results we have evaluated our binary embedding on above mentioned downstream tasks. the evaluation was done using 17 standard benchmarks to compare the proposed variations(variation 2, variation 3), variation 1 of binary embedding and the original continuous embedding. we have compared the reconstruction accuracies for the several variations of binary embedding. reconstruction accuracy denotes how much of the orignial embedding has been reconstructed from the generated binary embedding. as seen in in in it is also evident from tables 2,3,4 that our proposed variation 2 and variation 3 performs better in different tasks. variation 3 performs better in tasks where the contextual and semantic information is crucial, such as text categorisation. in the case of a static threshold (as in variation 2), it focuses more on the meaning of the word as compared to the contextual information. these are proved as the dynamic thresholding method (proposed variation 3) performs better in the categorization tasks whereas the static thresholding method (proposed variation 2) performs better in the analogical and word similarity tasks. as seen in ## conclusion and future works this paper presents a novel approach to generate binary embedding from any continuous embedding with significant reduction in size. the proposed approaches also allows preservation of semantic infomation in binary vectors. our proposed approach creates binary embedding which performs similar to original continuous embedding and performs better in most cases than previously proposed binarization approaches on different downstream tasks like word similarity and classification. a 640 dimensional binary vector is 15 times smaller than a 300 dimension real valued vector. however there are some inconsistencies in performances on some downstream tasks. our future work includes investigating and addressing these inconsistencies. we will also experiment further with different embedding, embedding dimensions and observe the effect on the performance on downstream tasks.