Transformation of Dense and Sparse Text Representations (2020)

Abstract: Sparsity is regarded as a desirable property of representations, especially in terms of explanation. However, its usage has been limited due to the gap with dense representations. Most research progresses in NLP in recent years are based on dense representations. Thus the desirable property of sparsity cannot be leveraged. Inspired by Fourier Transformation, in this paper, we propose a novel Semantic Transformation method to bridge the dense and sparse spaces, which can facilitate the NLP research to shift from dense spaces to sparse spaces or to jointly use both spaces. Experiments using classification tasks and natural language inference task show that the proposed Semantic Transformation is effective.

## introduction a sparse vector is a vector that has a large number of zeros or near zeros. many studies have shown that sparsity is a desirable property of representations, especially for explanation @xcite . in this sense, sparse representation may hold the key to solving the explainability problem of deep neural networks. apart from the interpretability property, sparse representations can also improve the usability of word vectors as features. the embeddings with good sparsity, interpretability or special meanings can also benefit downstream tasks @xcite . several tasks have benefited from sparse representations, e.g., part-of-speech tagging @xcite , dependency parsing @xcite , and supervised classification @xcite . however, much of the research advances so far for nlp tasks are based on dense representations, e.g., text classification @xcite @xcite , natural language inference @xcite , machine translation @xcite and generation @xcite @xcite . the study of sparse representations is still limited. there are two key limitations in the study of sparse representations. first, little work has been done to well connect dense and sparse spaces. the two types of representation are rather independent and cannot help each other to achieve synergy. second, limited work has been done to generate representations of sentences or phrases in the sparse space using sparse word embeddings. inspired by fourier transformation, this paper proposes a novel method called semantic transformation (st) to address the problems. with the help of st, dense and sparse spaces can connect with each other and will not be isolated. the proposed transformation consists of two key components, namely, semantic forward transformation (sft) and semantic backward transformation (sbt) (see section 2). sft is designed to transform a dense representation to a sparse representation. that is, we transform any learned dense features to sparse representations and give the model the properties that sparsity possesses. sparse representations can also be transformed back to dense representations through sbt. moreover, we can also perform different operations in the sparse space to achieve different goals. another key innovation of this paper is that it proposes a new approach for achieving sparseness. conventionally, penalties are used to achieve sparseness @xcite @xcite . however, they suffer from the problems of initialization sensitivity and uncontrollable optimization. in this paper, we propose to achieve sparseness through a novel activation function, which gives an effective solution (see section 2.1). experimental results show that the proposed activation function works very well. in this paper, we also explore a combination method to combine words representations into sentence representations in the sparse space directly. 1 additionally, the proposed transformations and combination method can be paralleled to enable efficient computation. in summary, this paper makes the following contributions: • it proposes a semantic transformation method which effectively connects dense and sparse spaces. ## semantic transformation in this section, we first briefly describe the composition of semantic transformation (st), and then elaborate on each component. the proposed st has three operations: 1) sft (semantic forward transformation). it takes a dense representation as input and transforms it into a higher dimensional sparse space. ## experiments we evaluate the proposed method using one natural language inference dataset and four text classification datasets. the tasks act as good quality checks for the learned representations. the five datasets are snli, mr, sst1, sst2 and trec, detailed training/dev/test splits are shown on baseline: four widely used methods are employed as the baselines: for our model, we adopt a mlp with 1 hidden layer (300 units) for forward transform and a mlp with 2 hidden layers (300 units) for backward transform. we set the length of semantic base to 1000. training details: we adopt uniform settings for all baselines and our model: 1) adam optimizer for parameter updating with learning rate of 1e-4; trainable embeddings with size 300. 2) a mlp with 1 hidden layer as the classifier. for a fair comparison, the hidden unit size is set to 300 for lstm, cnn, transformer and capsule. for our model, it is set to 64 when we use sparse representation to do the prediction and still 300 when we use back transformation representations as the prediction features. foot_11 3) snli is the task of identifying the relationships between two given sentences. for each model, we first use it to encode the two sentences into the resulting representations respectively, and then concatenate the two sentence representations for the final prediction. 4) we report the average accuracy over 10 runs of the experiment on the test data. for each run, the maximum accuracy before early stopping is selected as the result of the current run. ## related work sparse embeddings have been used in image @xcite @xcite , signal @xcite , and nlp @xcite applications. several sparse models have been proposed to produce sparse embeddings. for example, some previous works trained word embeddings with sparse or non-negative constraints @xcite . linguistically inspired dimensions @xcite is another way to increase sparsity and interpretability. spine (sparse interpretable neural embeddings) @xcite , a variant of denoising k-sparse autoencoder, can generate efficient and interpretable distributed word representations. our method is different from these approaches. we not only construct sparse representations but also transform between dense and sparse spaces. we also combine word sparse representations to produce sentence representations. some recent studies tried to achieve sparsity in novel ways @xcite . we also proposed a novel method in this paper and experimentally verified its effectiveness. ## conclusion and future works this paper proposed a novel method to transform representations between dense and sparse spaces, and a technique to combine semantics in the sparse space. it also proposed and experimentally verified a new activation function that can be used to achieve sparseness. natural language inference and text classification tasks were used to evaluate the proposed transformations with promising results. based on this study, many other interesting directions can be pursued in the future, e.g., (1) as we discussed in the paper, the proposed method can construct the output of lstm well. one future work is to apply st to language modeling. in this case, the results can be used in many down stream tasks such as machine translation and dialogue systems.