Learning Sparse Sentence Encoding without Supervision: An Exploration of Sparsity in Variational Autoencoders (2021)

Abstract: It has been long known that sparsity is an effective inductive bias for learning efficient representation of data in vectors with fixed dimensionality, and it has been explored in many areas of representation learning. Of particular interest to this work is the investigation of the sparsity within the VAE framework which has been explored a lot in the image domain, but has been lacking even a basic level of exploration in NLP. Additionally, NLP is also lagging behind in terms of learning sparse representations of large units of text e.g., sentences. We use the VAEs that induce sparse latent representations of large units of text to address the aforementioned shortcomings. First, we move in this direction by measuring the success of unsupervised state-of-the-art (SOTA) and other strong VAE-based sparsification baselines for text and propose a hierarchical sparse VAE model to address the stability issue of SOTA. Then, we look at the implications of sparsity on text classification across 3 datasets, and highlight a link between performance of sparse latent representations on downstream tasks and its ability to encode task-related information.

## introduction representation learning has been pivotal in many success stories of modern days nlp. observing its success, two fundamental questions arise: how is the information encoded in them? and what is encoded in them? while the latter has received a lot of attention by designing probing tasks, the former has been vastly neglected. in this work, we take small steps in this non-trivial direction by building on the knowns: one property we know about the encoding of information is that different data points embody different characteristics (e.g. statistically, semantically, or syntactically) which should ideally utilise different sub-regions of the representation space. therefore, the high-dimensional learned representations should ideally be sparse @xcite @xcite . in other words it allows us to have varying number of active dimension per sentence 2 @xcite in a fixed dimensional vector 3 . but if sparsity 4 is expected, could it be learned from data without supervision? a handful of studies in nlp that have delved into building sparse representations of words either during the learning phase @xcite or as a post-processing step on top of existing representations (e.g., word2vec embeddings) @xcite @xcite @xcite . these methods have not been developed for sentence embeddings, with the exception of @xcite which makes a strong assumption by forcing the latent sentence representation to be a sparse categorical distribution. in parallel, variational autoencoders (vaes) @xcite have been effective in capturing semantic closeness of sentences in the learned representation space @xcite @xcite . furthermore, methods have been developed 2 this, for example, may allow us to cluster sentences' representations not only based on similarity of their active features (as it is the case for dense vectors) but also on active/inactive dimensions. to encourage sparsity in vaes via learning a deterministic selection variable @xcite or sparse priors @xcite @xcite . however, the success of these is yet to be examined on text domain. to bridge this gap, we make a sober evaluation of existing state-of-the-art (sota) vae-based sparsification model @xcite against several vae-based baselines on two experimental tasks: text classification accuracy, and the level of representation sparsity achieved. additionally, we propose hierarchical sparse variation autoencoder (hsvae), to improve the stability issue of existing sota model and demonstrate its performance on both experimental tasks. our experimental findings demonstrate that: (i) neither the simpler baseline models nor the sota manage to impose a satisfactory level of sparsity on text, (ii) as expected, sparsity level and task performance have a negative correlation, while giving up task performance and having sparse codes helps with the analysis of the representations, (iii) presence/absence of task related signal in the sparsity codes affects the task performance, (iv) the success of capturing the task related signal in the sparsity codes depends on the strength of the signal presented in a corpus, and representation dimensionality, (v) the success of sota in image domain does not necessarily transfer to inducing sparse representations for text, while hsvae addresses this shortcoming. ## hierarchical sparse vae (hsvae) we propose the hierarchical sparse vae (hsvae), figure 1 (right), to learn sparse latent codes automatically. we treat the mixture @xmath0, ..., ) as a random variable and assign a factorised beta prior ( ) = beta( , ) on it. the latent code is then sampled from a factorised spikeand-slab distribution ( | ) conditioned on , and the observation is generated by decoding the latent variable ∼ ( | ) using a gru @xcite decoder. this returns a probabilistic generative model ( , , for posterior inference, the encoder distribution is defined as ( , | ) = ( | ) ( | , ), where ( | ) is a learnable and factorised beta distribution, and ( | , ) is a factorised spike-and-slab distribution with mixture weights and learnable "slab" components for each dimension. the distribution is computed by first extracting features from the sequence using a gru, then applying mlps to the extracted feature (and for ( | , )) to produce the distributional parameters. ## experiments we conduct a set of experiments on three text classification corpora: yelp (sentiment analysis -5 classes) @xcite , dbpedia and yahoo (topic classification -14 and 10 classes respectively) @xcite . first, we compare performance of the sparse latent representations with their dense counterpart on the text classification tasks ( §4.2). second, the stability of sparsification of hsvae is compared with the state-of-the-art mat-vae ( §4.3). then, to better understand performance of our model on the downstream task, we examine the sparsity patterns ( §4.4). remark. an integral part of the experiments is the analysis of the learned representations. in this sense, tasks that rely on understanding of semantics (e.g., glue @xcite ) or syntax (e.g., @xcite ) would be non-trivial to analyse due to their inherent complexity. we consider classification tasks because the distribution of words alone could be a good indicator of class labels. given the unsupervised nature of the models, we explore if this surface-level distribution of words could be captured by the sparsity patterns in the learned representation. ## related work learning sparse representations of data can be dated back to @xcite . this work motivates encoding of images in sparse linear codes for its biological plausibility and efficiency. it was later argued by @xcite that compared to the dimensionality reduction approaches, sparsity is a more efficient method for representation learning on vectors with fixed dimensionality. in other dimensions of the 768d code. representation sparsity. in nlp, learning sparse representations has been explored for various units of text with most of the focus placed on sparse representation of words. as the earliest work that moved in this direction, @xcite looked into sparse representations for ease of analysis, performance, and being more cognitively plausible. this idea was further developed by many other researchers @xcite @xcite @xcite @xcite . sparsification of the large units of text (i.e., sentences) has not received a lot of attention, perhaps due to inherent complexity of sentence/phrase representations: i.e., encoding and analysing syntactic and semantic information in a sentence embedding is rather a non-trivial task. to the best of our knowledge, the only model that sparsifies sentence emebeddings is introduced by @xcite . the authors introduced a seq2seq model @xcite with the sparsemax layer @xcite between the encoder and the decoder which induces sparse latent codes of text. this layer allows to learn codes that can be easier to analyse compared to their dense counterparts, but it is limited to modelling the categorical distribution. thus restricts a type a sentence representations that can be learned. ## conclusion we provided an objective analysis of several unsupervised sparsification frameworks based on vaes, both in terms of the impact on downstream tasks and the level of sparsity achieved. also, we presented a novel vae model -hierarchical sparse variational autoencoder (hsvae), outperforming existing sota model @xcite . ideally, sparse representations should be capable of encoding the underlying characteristics of a corpus (e.g. class), in activation patterns as shown to be the case for hsvae. moreover, using the text classification corpora as a testbed, we established how statistical properties of a corpus such as word distribution in a class affect the ability of learned sparse codes to represent task-related information. moving forward, hsvae model along with the analysis provided in this paper can serve as a good basis for the design of sparse models that induce continuous sparse vectors of text. for example, a potential extension of hsvae could be an incorporation of explicit linguistic biases into the learned representations with the group sparsity @xcite . furthermore, as we discussed in section 5, sparsity found its application in the transformers, but it, mainly, has been used to reduce the number of connection between the words/tokens. with the hsvae framework one can also learn sparse continuous representations of token/word/sentence embeddings.