A COMPRESSED SENSING VIEW OF UNSUPERVISED TEXT EMBEDDINGS , BAG - OF - n - GRAMS , AND LSTM S
Anonymous authors
Paper under double-blind review
A BSTRACT
Low-dimensional vector embeddings, computed using LSTMs or simpler techniques, are a popular approach for capturing the “meaning” of text and a form of unsupervised learning useful for downstream tasks.
However, their power is not theoretically understood.
The current paper derives formal understanding by looking at the subcase of linear embedding schemes.
Using the theory of compressed sensing we show that representations combining the constituent word vectors are essentially information-preserving linear measurements of Bag-of-n-Grams (BonG) representations of text.
This leads to a new theoretical result about LSTMs: low-dimensional embeddings derived from a low-memory LSTM are provably at least as powerful on classiﬁcation tasks, up to small error, as a linear classiﬁer over BonG vectors, a result that extensive empirical work has thus far been unable to show.
Our experiments support these theoretical ﬁndings and establish strong, simple and unsupervised baselines on standard benchmarks that in some cases are state of the art among word-level methods.
We also show a surprising new property of word embeddings such as GloVe and word2vec: they form a good sensing matrix for text that is more efﬁcient than random matrices, a standard sparse recovery tool, which may explain why they lead to better representations in practice
I NTRODUCTION
Much attention has been paid to using LSTMs (Hochreiter & Schmidhuber, 1997) and similar models to compute text embeddings (Bengio et al., 2003; Collobert & Weston, 2008).
Once trained, the
LSTM can sweep once or twice through a given piece of text, process it using only limited memory, and output a vector with moderate dimensionality (a few hundred to a few thousand), which can be used to measure text similarity via cosine similarity or as a featurization for downstream tasks.
The powers and limitations of this method have not been formally established.
For example, can such neural embeddings compete with and replace traditional linear classiﬁers trained on trivial
Bag-of- n -Grams (BonG) representations?
Tweaked versions of BonG classiﬁers are known to be a surprisingly powerful baseline (Wang & Manning, 2012) and have fast implementations (Joulin et al., 2017).
They continue to give better performance on many downstream supervised tasks such as IMDB sentiment classiﬁcation (Maas et al., 2011) than purely unsupervised LSTM representations (Kiros et al., 2015; Hill et al., 2016;
Pagliardini et al., 2017).
Even a very successful character-level (and thus computation-intensive, taking a month of training) approach does not reach BonG performance on datasets larger than IMDB
(Radford et al., 2017).
Meanwhile there is evidence suggesting that simpler linear schemes give compact representations that provide most of the beneﬁts of word-level LSTM embeddings (Wieting et al., 2016; Arora et al., 2017).
These linear schemes consist of simply adding up, with a few modiﬁcations, standard pretrained word embeddings such as GloVe or word2vec (Mikolov et al., 2013; Pennington et al., 2014).
The current paper ties these disparate threads together by giving a compressed sensing account of linear text embeddings.
We describe linear schemes that preserve n -gram information as lowdimensional embeddings with provable guarantees for any text classiﬁcation task.
The previous linear schemes, which used unigram information, are subcases of our approach, but our best schemes can also capture n -gram information with low additional overhead.
Furthermore, we show that the original unigram information can be (approximately) extracted from the low-dimensional embedding using sparse recovery/compressed sensing (Candes & Tao, 2005).
Our approach also ﬁts in the tradition of the older work on distributed representations of structured objects, especially the works of Plate (1995) and Kanerva (2009).
The following are the main results achieved by this new world-view:
Using random vectors as word embeddings in our linear scheme (instead of pretrained vectors) already allows us to rigorously show that low-memory LSTMs are provably at least as good as every linear classiﬁer operating on the full BonG vector.
This is a novel theoretical result in deep learning, obtained relatively easily.
By contrast, extensive empirical study of this issue has been inconclusive (apart from character-level models, and even then only on smaller datasets (Radford et al., 2017)).
Note also that empirical work by its nature can only establish performance on some available datasets, not on all possible classiﬁcation tasks.
We prove this theorem in Section 4 by providing a nontrivial generalization of a result combining compressed sensing and learning (Calderbank et al., 2009).
In fact, before our work we do not know of any provable quantiﬁcation of the power of any text embedding.
We study theoretically and experimentally how our linear embedding scheme improves when it uses pretrained embeddings (GloVe etc.)
instead of random vectors.
Empirically we ﬁnd that this improves the ability to preserve Bag-of-Words (BoW) information, which has the following restatement in the language of sparse recovery: word embeddings are better than random matrices for “sensing” BoW signals (see Section 5).
We give some theoretical justiﬁcation for this surprising ﬁnding using a new sparse recovery property characterizing when nonnegative signals can be reconstructed by (cid:96) 1 -minimization.
Section 6 provides empirical results supporting the above theoretical work, reporting accuracy of our linear schemes on multiple standard classiﬁcation tasks.
Our embeddings are consistently competitive with recent results and perform much better than all previous linear methods.
Among unsupervised word-level representations they achieve state of the art performance on both the binary and ﬁne-grained SST sentiment classiﬁcation tasks (Socher et al
Since our document representations are fast, compositional, and simple to implement given standard word embeddings, they provide strong baselines for future work.
[Table]
R ELATED W ORK
Neural text embeddings are instances of distributed representations , long studied in connectionist approaches because they decay gracefully with noise and allow distributed processing.
Hinton (1990) provided an early problem formulation, and Plate (1995) provided an elementary solution, the holographic distributed representation , which represents structured objects using circular vector convolution and has an easy and more compact implementation using the fast Fourier transform (FFT).
Plate suggested applying such ideas to text, where “structure” can be quantiﬁed using parse trees and other graph structures.
Our method is also closely related in form and composition to the sparse distributed memory system of Kanerva (2009).
In the unigram case our embedding reduces to the familiar sum of word embeddings, which is known to be surprisingly powerful (Wieting et al., 2016), and with a few modiﬁcations even more so (Arora et al., 2017).
Representations of BonG vectors have been studied through the lens of compression by Paskov et al.
(2013), who computed representations based on classical lossless compression algorithms using a linear program (LP).
Their embeddings are still high-dimensional ( d > 100 K ) and quite complicated to implement.
In contrast, linear projection schemes are simpler, more compact, and can leverage readily available word embeddings.
Pagliardini et al.
(2017) also used a linear scheme, representing documents as an average of learned word and bigram embeddings.
However, the motivation and beneﬁts of encoding BonGs in low-dimensions are not made explicit.
The novelty in the current paper is the connection to compressed sensing, which is concerned with recovering high-dimensional sparse signals x ∈ R N from low-dimensional linear measurements Ax , speciﬁcally by studying conditions on matrix A ∈ R d × N when this is possible (see Appendix A for some background on compressed sensing and the previous work of Calderbank et al.
(2009) that we build upon).
D OCUMENT E MBEDDINGS
In this section we deﬁne the two types of representations that our analysis will relate:
high-dimensional BonG vectors counting the occurrences of each k -gram for k ≤ n 2.
low-dimensional embeddings, from simple vector sums to novel n -gram-based embeddings
Although some of these representations have been previously studied and used, we deﬁne them so as to make clear their connection via compressed sensing, i.e. that representations of the second type are simply linear measurements of the ﬁrst.
We now deﬁne some notation.
Let V be the number of words in the vocabulary and V n be the number of n -grams (independent of word order), so that V = V 1 .
Furthermore set V sum n = (cid:80) k ≤ n V k and
V max n = max k ≤ n V k .
We will use words/ n -grams and indices interchangeably, e.g. if ( a, b ) is the i th of V 2 bigrams then the one-hot vector e ( a,b ) will be 1 at index i .
Where necessary we will use { , } to denote a multi-set and ( , ) to denote a tuple.
For any m vectors v i ∈ R d for i = 1 , . . .
, m we deﬁne [ v 1 , . .
. , v m ] to be their concatenation, which is thus an element of R md .
Finally, for any subset X ⊂ R N we denote by ∆ X the set { x − x (cid:48) :
x, x (cid:48) ∈ X} .
3. T HE B AG - OF - n -G RAMS V ECTOR
Assigning to each word a unique index
i ∈
[ V ] we deﬁne the Bag-of-Words (BoW) representation x BoW of a document to be the V -dimensional vector whose i th entry is the number of times word i occurs in the document.
The n -gram extension of BoW is the Bag-of- n -Grams (BonG) representation, which counts the number of times any k -gram for k ≤ n appears in a document.
Linear classiﬁcation over such vectors has been found to be a strong baseline (Wang & Manning, 2012).
For ease of analysis we simplify the BonG approach by merging all n -grams in the vocabulary that contain the same words but in a different order.
We call these features n -cooccurrences and ﬁnd that the modiﬁcation does not affect performance signiﬁcantly (see Table 3 in Appendix E.1).
Formally for a document w 1 , . . .
, w T we deﬁne the Bag-of- n -Cooccurrences (BonC) vector as the concatenation
[Equation]
which is thus a vector in Z V sum n + .
Note that in the unigram case this is equivalent to x BoW =
(cid:80) Tt =1 e w t .
3. L OW -D IMENSIONAL n -G RAM E MBEDDINGS
Now suppose each word w has a vector v w ∈ R d for some d (cid:28) V .
Then given a document w 1 , . . .
, w T we deﬁne its unigram embedding as z u = (cid:80) T t =1 v w t .
While this is a simple and widely used featurization, we focus on the following straightforward relation with BoW: if A ∈ R d × V is a matrix whose columns are word vectors v w then Ax BoW =
(cid:80) Tt =
1 Ae w t =
(cid:80) Tt
=1 v w t = z u .
Thus in terms of compressed sensing the unigram embedding of a document is a d -dimensional linear measurement of its Bag-of-Words vector.
We could extend this unigram embedding to n -grams by ﬁrst deﬁning a representation for each ngram as the tensor product of the vectors of its constituent words.
Thus for each bigram b
= ( w 1 , w 2 ) we would have v b = v w 1 v Tw 2 and more generally v g
= (cid:78) nt
=1 v w t for each n -gram g = ( w 1 , . . . , w n ) .
The document embedding would then be the sum of the tensor representations of all n -grams.
The major drawback of this approach is of course the blowup in dimension, which in practice prevents its use beyond n = 2 .
To combat this a low-dimensional sketch or projection of the tensor product can be used, such as the circular convolution operator of Plate (1995).
Since we are interested in representations that can also be constructed by an LSTM, we instead sketch this tensor product using the element-wise multiplication operation, which we ﬁnd also usually works better than circular convolution in practice (see Table 4 in Appendix E.1).
Thus for the n -cooccurrence g = { w 1 , . . .
, w n } , we deﬁne the distributed cooccurrence (DisC) embedding ˜ v g =
d n − 1 2 (cid:74) nt
=1 v w t .
The coefﬁcient is required when the vectors v w are random and unit norm to ensure that the product also has close to unit norm (see Lemma B.1).
In addition to their convenient form, DisC embeddings have nice theoretical and practical properties: they preserve the original embedding dimension, they reduce to unigram (word) embeddings for n = 1 , and under mild assumptions they satisfy useful compressed sensing properties with overwhelming probability (Lemma 4.1).
We then deﬁne the DisC document embedding to be the nd -dimensional weighted concatenation, over k ≤ n , of the sum of the DisC vectors of all k -grams in a document:
[Equation]
Here scaling factors C k are set so that all spans of d coordinates have roughly equal norm (for random embeddings C k
= 1 ; for word embeddings C k = 1 /k works well).
Note that since ˜ v w t = v w t we have ˜ z (1) =
z u in the unigram case.
Furthermore, as with unigram embeddings by comparing (1) and (2) one can easily construct a (cid:80) nk =1 dn × V sum n matrix ˜
A n such that ˜ z
( n ) =
˜
A n x BonC .
3. LSTM R EPRESENTATIONS
As discussed previously, LSTMs have become a common way to apply the expressive power of
RNNs, with success on a variety of classiﬁcation, representation, and sequence-to-sequence tasks.
For document representation, starting with h 0 =
m an m -memory LSTM initialized with word vectors v w ∈ R d takes in words w 1 , . . .
, w T one-by-one and computes the document representation
[Equation]
where h t ∈ R m is the hidden representation at time t , the forget gate f , input gate i , and input function g are a.e. differentiable nondecreasing elementwise “activation” functions R m (cid:55)→ R m , and afﬁne transformations T ∗ ( x, y )
=
W ∗ x + U ∗ y + b ∗ have weight matrices
W ∗ ∈ R m × d , U ∗ ∈ R m × m and bias vectors b ∗ ∈ R m .
The LSTM representation of a document is then the state at the last time step, i.e. z LSTM =
h T .
Note that we will follow the convention of using LSTM memory to refer to the dimensionality of the hidden states.
Since the LSTM is initialized with an embedding for each word it requires O ( m 2 + md + V d ) computer memory, but the last term is just a lookup table so the vocabulary size does not factor into iteration or representation complexity.
From our description of LSTMs it is intuitive to see that one can initialize the gates and input functions so as to construct the DisC embeddings deﬁned in the previous section.
We state this formally and give the proof in the unigram case (the full proof appears in Appendix B.3):
Proposition 3.1.
Given word vectors v w ∈ R d , one can initialize an O ( nd ) -memory
LSTM (3) that takes in words w 1 , . . .
, w T (padded by an end-of-document token assigned vector 0 d ) and constructs the DisC embedding (2) (up to zero padding), i.e. such that for all documents z LSTM =
˜
z
( n ) .
Proof (Unigram Case).
Set f
( x ) =
i ( x ) = g ( x ) = x , T f ( v w t , h t − 1 ) =
T i ( v w t , h t − 1 ) = 1 d , and
T g ( v w t , h t − 1 ) =
C 1 v w t .
Then h t =
h t − 1 + C 1 v w t , so since h 0 =
d we have the ﬁnal LSTM representation z LSTM =
h T =
C 1 (cid:80) tt
=1 v w t = ˜
z (1) .
By Proposition 3.1 we can construct a ﬁxed LSTM that can compute compressed BonC representations on the ﬂy and be further trained by stochastic gradient descent using the same memory.
LSTM S AS C OMPRESSED L EARNERS
Our main contribution is to provide the ﬁrst rigorous analysis of the performance of the text embeddings that we are aware of, showing that the embeddings of Section 3.2 can provide performance on downstream classiﬁcation tasks at least as well any linear classiﬁer over BonCs.
Before stating the theorem we make two mild simplifying assumptions on the BonC vectors:
The vectors are scaled by 1 T √ n , where T is the maximum document length.
This assumption is made without loss of generality.
No n -cooccurrence contains a word more than once.
While this is (infrequently) violated in practice, the problem can be circumvented by merging words as a preprocessing step.
Theorem 4.1.
Let S = { ( x i , y i ) } mi =1 be drawn i.i.d.
from a distribution D over BonC vectors of documents of length at most T satisfying assumptions 1 and 2 above and let w 0 be the linear classiﬁer minimizing the logistic loss (cid:96) D .
Then for dimension d = ˜Ω (cid:16) Tε 2 log nV max n δ (cid:17) and appropriate choice of regularization coefﬁcient one can initialize an O ( nd ) -memory
LSTM over i.i.d.
word embeddings v w ∼ U d {± 1 / √ d } such that w.p.
(1 − γ )(1 − 2 δ ) the classiﬁer ˆ w minimizing the (cid:96) 2 -regularized logistic loss over its representations satisﬁes
[Equation]
The above theoretical bound shows that LSTMs match BonC performance as ε → 0 , which can be realized by increasing the embedding dimension d (c.f. Figure 4).
4. C OMPRESSED S ENSING AND L EARNING
Compressed sensing is concerned with recovering a high-dimensional k -sparse signal x ∈ R N from a few linear measurements; given a design matrix A ∈ R d × N this is formulated as minimize (cid:107) w (cid:107)subject to Aw =
z (5)
where z =
Ax is the measurement vector .
As l 0 -minimization is NP-hard, research has focused on sufﬁcient conditions for tractable recovery.
One such condition is the Restricted Isometry Property (RIP), for which Candes & Tao (2005) proved that (5) can be solved by convex relaxation:
Deﬁnition 4.1.
A ∈ R d × N is ( X , ε ) -RIP for some subset X ⊂ R N if ∀ x ∈ X (
[Equation]
We will abuse notation and say ( k, ε ) -RIP when X is the set of k -sparse vectors.
This is the more common deﬁnition, but ours allows a more general Theorem 4.2 and a tighter bound in Theorem 4.1.
Following these breakthroughs, Calderbank et al.
(2009) studied whether it is possible to use the low-dimensional output of compressed sensing as a surrogate representation for classiﬁcation.
They proved a learning-theoretic bound on the loss of an SVM classiﬁer in the compressed domain compared to the best classiﬁer in the original domain.
In this work we are interested in comparing the performance of LSTMs with BonC representations, so we need to generalize Calderbank et al.
(2009) result to handle Lipschitz losses and an arbitrary set X ⊂ R N of high-dimensional signals:
Theorem 4.2.
For any subset X ⊂ R N containing the origin let A ∈ R d
× N be (∆ X , ε ) -RIP and let m samples S
=
( x i , y i ) } mi =1 ⊂ X × {− 1 , 1 } be drawn i.i.d.
from some distribution D over X with (cid:107) x (cid:107) 2 ≤ R .
If (cid:96) is a λ -Lipschitz convex loss function and w 0 ∈ R N is its minimizer over D then w.p. 1 − 2 δ the linear classifer ˆ w A ∈ R d minimizing the (cid:96) 2 -regularized empirical loss function (cid:96)
S A ( w ) + 12 C (cid:107) w (cid:107) 22 over the compressed sample S A
= { ( Ax i , y i ) } mi =
⊂ R d × {− 1 , 1 } satisﬁes (cid:96) D ( ˆ w A ) ≤ (cid:96) D ( w 0 ) + O (cid:32) λR (cid:107) w 0 (cid:107)(cid:114)
[Equation]
for appropriate choice of C .
Recall that ∆ X = { x − x (cid:48) :
x, x (cid:48) ∈ X} for any X ⊂ R N .
While a detailed proof of this theorem is spelled out in Appendix C, the main idea is to compare the distributional loss incurred by a classiﬁer ˆ w in the original space to the loss incurred by A ˆ w in the compressed space.
We show that the minimizer of the regularized empirical loss in the original space ( ˆ w ) is a bounded-coefﬁcient linear combination of samples in S , so its loss depends only on inner products between points in X .
Thus using RIP and a generalization error result by Sridharan et al.
(2008) we can bound the loss of ˆ w A , the regularized classiﬁer in the compressed domain.
Note that to get back from Theorem 4.2 the O ( √ ε ) bound for k -sparse inputs of Calderbank et al.
(2009) we can set X to the be the set of k -sparse vectors and assume A is (2 k, ε ) -RIP.
4. P ROOF OF M AIN R ESULT
To apply Theorem 4.2 we need the design matrix ˜
A n transforming BonCs into the DisC embeddings of Section 3.2 to satisfy the following RIP condition (Lemma 4.1), which we prove using a restricted isometry result for structured random sampling matrices in Appendix D:
Lemma 4.1.
Assume the setting of Theorem 4.1 and let ˜ A n be the nd × V sum n matrix relating
the
DisC and BonC representations of any document by ˜ z ( n ) =
˜
A n x BonC .
If d = ˜Ω (cid:16) Tε 2 log nV max n δ (cid:17) then ˜
A n is (cid:16) ∆ X ( n ) T , ε (cid:17) -RIP w.p. 1 − δ , where X ( n ) T is the set of BonCs of documents of length at most T .
Proof of Theorem 4.1.
Let ˆ S
=
( ˜ A n x i , y i ) : ( x i , y i ) ∈ S } , where ˜ A n is as in Lemma 4.1.
Then by the same lemma ˜
A n is (cid:16) ∆ X ( n ) T , ε (cid:17) -RIP w.p. 1 − γ , where X ( n ) T is the set of BonC vectors of documents of length at most T .
By BonC assumption (1) all BonCs lie within the unit ball, so we can apply Theorem 4.2 with (cid:96) the logistic loss, λ = 1 , and R = 1 to get that a classiﬁer ˆ w trained using (cid:96) 2regularized logistic loss over ˆ S will satisfy the required bound (4).
Since by Proposition 3.1 one can initialize an O ( nd ) -memory
LSTM that takes in i.i.d.
Rademacher word vectors v w ∼ U d {± 1 / √ d } such that z LSTM =
˜ z ( n ) =
˜
A n x ∀ x ∈ X ( n ) T , this completes the proof.
S PARSE R ECOVERY WITH P RETRAINED E MBEDDINGS
Theorem 4.1 is proved using random vectors as the word embeddings in the scheme of Section 3.
However, in practice LSTMs are often initialized with standard word vectors such as GloVe.
Such embeddings cannot satisfy strong compressed sensing properties such as RIP or incoherence.
This follows essentially from the deﬁnition: word embeddings seek to capture word similarity, so similar words (e.g. synonyms) have embeddings with high inner product, which violates both properties.
Thus the efﬁcacy of real-life LSTMs must have some other explanation.
But in this section we present the surprising empirical ﬁnding that pretrained word embeddings are more efﬁcient than random vectors at encoding and recovering BoW information via compressed sensing.
We further sketch a potential explanation for this result, though a rigorous explanation is left for subsequent work.
5. P RETRAINED E MBEDDINGS P RESERVE S PARSE I NFORMATION
In recent years word embeddings have been discovered to have many remarkable properties, most famously the ability to solve analogies (Mikolov et al., 2013).
Our connection to compressed sensing indicates that they should have another: preservation of sparse signals as low-dimensional linear measurements.
To examine this we subsample documents from the SST (Socher et al., 2013) and
IMDB (Maas et al., 2011) classiﬁcation datasets, embed them as d -dimensional unigram embeddings (2) for d = 50 , 100 , 200 , . .
. , 1600 , and solve the following LP, known as Basis Pursuit (BP), which is the standard (cid:96) 1 -minimization problem for sparse recovery in the noiseless case (see Appendix A): [Equation]
[Equation]
Success is measured as the F 1 score of retrieved words.
We use Squared Norm (SN) vectors (Arora et al., 2016) trained on a corpus of Amazon reviews (McAuley et al., 2015) and normalized i.i.d.
Rademacher vectors as a baseline.
SN is used due to similarity to GloVe and its formulation via an easy-to-analyze generative model that may provide a framework to understand the results (see
Appendix E.2), while the Amazon corpus is used for its semantic closeness to the sentiment datasets.
Figures 1 and 2 show that pretrained embeddings require a lower dimension d than random vectors to recover natural language BoW. This is surprising as the training objective goes against standard conditions such as approximate isometry and incoherence; indeed as shown in Figure 2 recovery is poor for randomly generated word collections.
The latter outcome indicates that the fact that a document is a set of mutually meaningful words is important for sparse recovery using embeddings trained on co-occurrences.
We achieve similar results with other objectives (e.g. GloVe/word2vec) and other corpora (see Appendix E.1), although there is some sensitivity to the sparse recovery method, as other (cid:96) 1 -minimization methods work well but greedy methods, such as Orthogonal
Matching Pursuit (OMP), work poorly, likely due to their dependence on incoherence (Tropp, 2004).
For the n -gram case (i.e. BonC recovery for n > 1 ), although we know by Lemma 4.1 that DisC embeddings composed from random vectors satisfy RIP, for pretrained vectors it is unclear how to reason about suitable n -gram embeddings without a rigorous understanding of the unigram case, and experiments do not show the same recovery beneﬁts.
One could perhaps do well by training on cooccurrences of word tuples, but such embeddings could not be used by a word-level LSTM.
BoW compared to dimension.
Pretrained word embeddings (SN trained on Amazon reviews) need half the dimensionality of normalized Rademacher vectors to achieve near-perfect recovery.
Note that IMDB documents are on average more than ten times longer than SST documents.
5. U NDERSTANDING S PARSE R ECOVERY OF N ATURAL L ANGUAGE D OCUMENTS
As shown in Figure 2, the success of pretrained embeddings for linear sensing is a local phenomenon; recovery is only efﬁcient for naturally occurring collections of words.
However, applying statistical
RIP/incoherence ideas (Barg et al., 2015) to explain this is ruled out since they require collections to be incoherent with high probability, whereas word embeddings are trained to give high inner product to words appearing together.
Thus an explanation must come from some other, weaker condition.
The usual necessary and sufﬁcient requirement for recovering all signals with support S ⊂ [ N ] is the local nullspace property (NSP), which stipulates that vectors in the kernel of A not have too much mass on S (see Deﬁnition A.2).
While NSP and related properties such as restricted eigenvalue (see
Deﬁnition A.3) are hard to check, we can impose some additional structure to formulate an intuitive, veriﬁable perfect recovery condition for our setting.
Speciﬁcally, since our signals (BoW vectors) are nonnegative, we can improve upon solving BP (8) by instead solving nonnegative BP (BP+): minimize (cid:107) w (cid:107)subject to
[Equation]
The following geometric result then characterizes when solutions of BP+ recover the correct signal:
Theorem 5.1 (Donoho & Tanner, 2005) . Consider a matrix A ∈ R d × N and an index subset S ⊂
[ N ] of size k .
Then any nonnegative vector x ∈ R N + with support supp( x ) =
S is recovered from Ax by BP+ (9) iff the columns of A indexed by S comprise the vertices of a k -dimensional face of the convex hull conv( A ) of the columns of A together with the origin.
This theorem equates perfect recovery of a BoW vector via BP+ with the vectors of its words being the vertices of some face of the polytope conv( A ) .
The property holds for incoherent columns since the vectors are far enough that no one vector is inside the simplex formed by any k others.
On the other hand, pretrained embeddings satisfy it by having commonly co-occurring words close together and other words far away, making it easier to form a face from columns indexed by the support of a
BoW. We formalize this intuition as the Support Supporting Hyperplane (SSH) property:
Deﬁnition 5.1.
A matrix A ∈ R d × N satisﬁes the SSH property for subset S ⊂ [ N ] if it has unit-norm columns in general position and if there is a hyperplane containing the set of all columns of A indexed by S and the set of all other columns together with the origin are on one side of the hyperplane.
SSH is a very weak property that is implied by NSP (see Corollary A.1).
However, it can be checked efﬁciently by ﬁnding such a hyperplane using an LP.
Furthermore, by applying Theorem 5.1 we can show that SSH sufﬁces to characterize recovery of nonnegative signals using BP+ when the matrix has unit-norm columns (in our case this corresponds to normalizing the word embeddings):
Corollary 5.1.
BP+ (9) recovers any x ∈ R N + with supp( x ) =
S from Ax iff A satisﬁes SSH for S .
Proof.
=
⇒ :
By Theorem 5.1 the columns indexed by S form a face of conv( A ) via its intersection with some halfspace.
The corresponding hyperplane separates these columns from the other vertices.
⇐ = : By convexity, the supporting hyperplane must contain the simplex of the support, which is nondegenerate by the assumptions. All other points of conv( A ) must be on the same side of the hyperplane as the other vertices, so the simplex is a face.
Theorem 5.1 implies the result.
[Table]
Thus perfect recovery of a BoW via BP+ is equivalent to the existence of a hyperplane separating embeddings of words in the document from those of the rest of the vocabulary.
Intuitively, words in the same document are trained to have similar embeddings and so will be more easy to separate than random vectors, providing some justiﬁcation for why pretrained embeddings are better for sensing.
As these similarity properties are also often used to explain the efﬁcacy of word embeddings on downstream tasks, we conjecture that there is a deeper relationship between embeddings, recovery, and classiﬁcation that might be more rigorously understood using a generative model of text (see
Appendix E.2).
Note that the bounds in Section 4 depend on RIP, not recoverability, so the empirical results of Section 5.1 by themselves do not apply to that setting.
They do show that the framework of compressed sensing remains relevant even in the case of non-random, pretrained word embeddings.
E MPIRICAL F INDINGS
Our theoretical results show that simple tensor product sketch-based n -gram embeddings can approach
BonG performance and be computed by a low-memory LSTM.
In this section we compare these text representations and others on several standard tasks, verifying that DisC performance approaches that of BonCs as dimensionality increases and establishing several baselines for text classiﬁcation.
Tasks:
We test classiﬁcation on MR movie reviews (Pang & Lee, 2005), CR customer reviews (Hu &
Liu, 2004), SUBJ subjectivity dataset (Pang & Lee, 2004), MPQA opinion polarity subtask (Wiebe et al., 2005), TREC question classiﬁcation (Li & Roth, 2002), SST sentiment classiﬁcation (binary and ﬁne-grained) (Socher et al., 2013), and IMDB movie reviews (Maas et al., 2011).
The ﬁrst four are evaluated using 10-fold cross-validation, while the others have train-test splits.
In all cases we use logistic regression with (cid:96) 2 -regularization determined by cross-validation.
We further test DisC on the
SICK relatedness and entailment tasks (Marelli et al., 2014) and the MRPC paraphrase detection task (Dolan & Brockett, 2005).
The inputs here are sentences pairs ( a, b ) and the standard featurization for document embeddings x a and x b of a and b is [ | x a − x b | , x a (cid:12) x b ] (Tai et al., 2015).
We use logistic regression for SICK entailment and MRPC and use ridge regression to predict similarity scores for SICK relatedness, with (cid:96) 2 -regularization determined by cross-validation.
Since BonGs are not used for pairwise tasks our theory says nothing about performance here; we include these evaluations to show that our representations are also useful for other tasks.
Rep. n
SICK-R ( r/ρ )
SICK-E
MRPC (Acc./ F 1 )
DisC (2) 73.6 / 71.
SIF 75.2 / 69.
71.3 / 80.5 Sent2Vec 69.3 / 64.
71.0 / 80.4 Sent2Vec 70.1 / 65.
skip-thoughts 83.1 / 76.
73.1 / 80.9 SDAE 73.7 / 80.7 CNN-LSTM 76.4 / 83.
byte mLSTM 78.4 / 71.
Model information is the same as that in Table 1.
estimated
Embeddings: In the main evaluation (Table 1) we use normalized 1600-dimensional GloVe embeddings (Pennington et al., 2014) trained on the Amazon Product Corpus (McAuley et al., 2015).
We also compare the SN vectors of Section 5 trained on the same corpus with random vectors when varying the dimension (Figure 4).
Both embeddings and code will be released for reproducibility.
Results: We ﬁnd that DisC representation performs consistently well relative to recent unsupervised methods; among word-level approaches it is the top performer on the SST tasks and competes on many others with skip-thoughts and CNN-LSTM, both concatenations of two LSTM representations.
While success may be explained by training on a large and in-domain corpus, being able to use so much text without extravagant computing resources is one of the advantages of a simple approach.
Overall our method is useful as a strong baseline, often beating BonCs and many more complicated approaches while taking much less time to represent and train on documents than neural representations (Figure 3).
Finally, we analyze empirically how well our model approximates BonC performance.
As predicted by Theorem 4.1, the performance of random embeddings on IMDB approaches that of BonC as dimension increases and the isometry distortion ε decreases (Figure 4).
Using pretrained (SN) vectors,
DisC embeddings approach BonC performance much earlier, surpassing it in the unigram case.
C ONCLUSION
In this paper we explored the connection between compressed sensing, learning, and natural language representation.
We ﬁrst related LSTM and BonG methods via word embeddings, coming up with simple new document embeddings based on tensor product sketches.
Then we studied their classiﬁcation performance, proving a generalization of the compressed learning result of Calderbank et al.
(2009) to convex Lipschitz losses and a bound on the loss of a low-dimensional LSTM classiﬁer in terms of its (modiﬁed)
BonG counterpart, an issue which neither experiments nor theory have been able to resolve.
Finally, we showed how pretrained embeddings ﬁt into this sparse recovery framework, demonstrating and explaining their ability to efﬁciently preserve natural language information.
A C OMPRESSED S ENSING B ACKGROUND
The ﬁeld of compressed sensing is concerned with recovering a high-dimensional k -sparse signal x ∈ R N from few linear measurements.
In the noiseless case this is formulated as minimize (cid:107) w (cid:107)subject to Aw =
z (10) where A ∈ R d
× N is the design matrix and z
= Ax is the measurement vector .
Since (cid:96)
-minimization is NP-hard, a foundational approach is to use its convex surrogate, the (cid:96) 1 -norm, and characterize when the solution to (10) is equivalent to that of the following LP, known as basis pursuit (BP): minimize (cid:107) w (cid:107)subject to
Aw =
z (11)
Related approaches such as Basis Pursuit Denoising (LASSO) and the Dantzig Selector generalize
BP to handle signal or measurement noise (Foucart & Rauhut, 2013); however, the word embeddings case is noiseless so these methods reduce to BP.
Note that throughout Section 5 and the Appendix we say that an (cid:96) 1 -minimization method recovers x from Ax if its optimal solution is unique and equivalent to the optimal solution of (10).
An alternative way to approximately solve (10) is to use a greedy algorithm such as matching pursuit (MP) or orthogonal matching pursuit (OMP), which pick basis vectors one at a time by multiplying the measurement vector by A T and choosing the column with the largest inner product (Tropp, 2004).
A. S OME C OMMON S PARSE R ECOVERY C ONDITIONS
One condition through which recovery can be guaranteed is the Restricted Isometry Property (RIP):
Deﬁnition A.
A ∈ R d × N is ( k ε ) -RIP if for all k -sparse x ∈ R N
[Equation]
A line of work started by Candes & Tao (2005) used the RIP property to characterize matrices A such that (10) and (11) have the same minimizer for any k -sparse signal x ; this occurs with overwhelming probability when d = Ω (cid:0) k log Nk (cid:1) and √ dA ij ∼ N (0 , 1) ∀
i, j or √ dA ij ∼ U{− 1 , 1 } ∀ i, j .
Since the ability to recover a signal x from a representation Ax implies information preservation, a natural next step is to consider learning after compression.
Calderbank et al.
(2009) show that for m i.i.d.
k -sparse samples { ( x i , y i ) } mi =1 and a (2 k, ε ) -RIP matrix A the hinge loss of a classiﬁer trained on { ( Ax i , y i ) } mi =1 is bounded by that of the best linear classiﬁer over the original samples.
Theorem 4.2 provides a generalization of this result to any convex Lipschitz loss function.
RIP is a strong requirement, both because it is not necessary for perfect, stable recovery of k -sparse vectors using ˜ O ( k ) measurements and because in certain settings we are interested in using the above ideas to recover speciﬁc signals — those statistically likely to occur—rather than all k -sparse signals.
The usual necessary and sufﬁcient condition to recover any vector x ∈ R N with index support set
S ⊂ [ N ] is the local nullspace property (NSP), which is implied by RIP:
Deﬁnition A.2 (Foucart & Rauhut, 2013) .
A matrix A ∈ R d × N satisﬁes NSP for a set S ⊂ [ N ] if (cid:107) w S (cid:107) 1 < (cid:107) w ¯ S (cid:107) 1 for all nonzero w ∈ ker
( A ) = { v : Av = 0 d } .
Theorem A.1 (Foucart & Rauhut, 2013) .
BP (11) recovers any x ∈ R N + with supp( x ) =
S from Ax iff A satisﬁes NSP for S .
A related condition that implies NSP is the local restricted eigenvalue property (REP):
Deﬁnition A.3 (Raskutti et al., 2010) .
A matrix A ∈ R d × N satisﬁes γ -REP for a set S ⊂
[ N ] if (cid:107) Aw (cid:107) 2 ≥ γ √ d (cid:107) w (cid:107) 2 whenever (cid:107) w ¯ S (cid:107) 1 ≤ (cid:107) w S (cid:107) 1 .
Lastly, a simple condition that can sometimes provide recovery guarantees is mutual incoherence :
Deﬁnition A.4.
A ∈ R d × N is µ -incoherent if max a,a (cid:48) | a T a (cid:48) | ≤ µ , where the maximum is taken over any two distinct columns a, a (cid:48) of A .
While incoherence is easy to verify (unlike the previous recovery properties), word embeddings tend to have high coherence due to the training objective pushing together vectors of co-occurring words.
A. C ONDITIONS FOR R ECOVERING N ONNEGATIVE S IGNALS
Apart from incoherence, the properties above are hard show empirically.
However, we are compressing BoW vectors, so our signals are nonnegative and we can impose an additional constraint on (11):
[Equation]
The following geometric result provides guarantees for this nonnegative basis pursuit (BP+) problem:
Theorem A.2 (Donoho & Tanner, 2005) . Consider a matrix A ∈ R d × N and an index subset S ⊂
[ N ] of size k .
Then any nonnegative vector x ∈ R N + with support supp( x ) =
S is recovered from Ax by BP+ (12) iff the columns of A indexed by S comprise the vertices of a k -dimensional face of the convex hull conv( A ) of the columns of A together with the origin.
The polytope condition is equivalent to nonnegative NSP (NSP+), a weaker form of NSP:
Deﬁnition A.5 (Foucart & Koslicki, 2014) .
A matrix A ∈ R d × N satisﬁes NSP+ for a set S ⊂
[ N ] if w ¯ S ≥ 0 N = ⇒ N (cid:80) i =
1 w i > 0 for all nonzero w ∈ ker
( A ) .
Lemma A.1.
If A ∈ R d × N satisﬁes NSP for some S ⊂ [ N ] then it also satisﬁes NSP+ for S .
Proof (Adapted from Foucart & Koslicki (2014)).
Since A satisﬁes NSP, we have (cid:107) w S (cid:107) 1 < (cid:107) w ¯ S (cid:107) 1 .
Then for a nonzero w ∈ ker( A ) such that w ¯ S ≥ 0 we will have
[Equation]
Lemma A.2.
BP+ recovers any x ∈ R N + with supp( x ) =
S from Ax iff A satisﬁes NSP+ for S .
Proof.
( = ⇒ )
: For any nonzero w ∈ ker( A ) such that w ¯ S ≥ 0 , ∃ λ > 0 such that x + λw ≥ 0 N and A ( x + λw ) =
Ax .
Since BP+ uniquely recovers x , we have (cid:107) x + λw (cid:107) 1 > (cid:107) x (cid:107) 1 , so NSP+ follows from the following inequality and the fact that λ is positive: [Equation] w
[Equation]
( ⇐ = )
: For any x (cid:48) ≥ 0 such that Ax (cid:48) =
Ax we have that w = x (cid:48) − x ∈ ker
( A ) and w ¯ S =
x
(cid:48)
¯ S ≥since the support of x is S .
Thus by NSP+ we have that N (cid:80) i =1 w i > 0 , which yields
[Equation]
Thus BP+ will recover x uniquely.
Lemma A.2 shows that NSP+ is equivalent to the polytope condition in Theorem A.2 since they are both necessary and sufﬁcient conditions for perfect recovery via BP+.
Together with Lemma A. it also yields the following Corollary relating NSP to the Support Supporting Hyperplane (SSH) property we introduce (see Deﬁnition 5.1):
Corollary A.1.
If a matrix A ∈ R d × N with unit-norm columns satisﬁes NSP for some S ⊂ [ N ] then it also satisﬁes SSH for S .
B D OCUMENT E MBEDDINGS B. P ERFORMANCE C OMPARISON OF A LTERNATIVE R EPRESENTATIONS
In this section we compare the performance of several alternative representations with the ones presented in the main evaluation (Table 1).
Table 3 provides a numerical justiﬁcation for our use of unordered n -grams (cooccurrences) instead of n -grams, as the performance of the two featurizations are closely comparable.
In Table 4 we examine the use of circular convolution instead of elementwise multiplication as linear measurements of BonC vectors Plate (1995).
To construct the former from a document w 1 , . .
. , w T we compute (cid:34) T (cid:88) t = v w t , . . . ,
T − n +1 (cid:88) t =
F −(cid:32) t + n − 1 (cid:89) τ
= t
F ( v w τ ) (cid:33)(cid:35) (13) where F is the discrete Fourier transform and F − 1 its inverse.
Note that for n = 1 this is equivalent to the simple unigram embedding (and thus also to the DisC embedding in (2)).
[Table]
) vectors is generally similar to that of Bag-of- n -Cooccurrences (BonC) vectors for n = 2 , 3 (largest differences bolded ).
Evaluation settings are the same as in Section 6.
Note that for unigrams the two representations are equivalent.
[Table]
B. S CALING F ACTOR FOR D IS C E MBEDDINGS
Lemma B.1.
If word vectors v w ∈ R
d are drawn i.i.d.
from U d {± 1 / √ d } then for any n -gram g
=
( w 1 , . . .
, w n ) we have E (cid:107) ˜ v g (cid:107)
= 1 .
The same result holds true with the additional assumption that all words in g are distinct if the word vectors are i.i.d.
d -dimensional spherical Gaussians.
Proof.
E (cid:107) ˜ v g (cid:107)
= d (cid:88) i =
E (cid:32) d n − 1n (cid:89) k
= v w k i (cid:33)= d n − 1 d (cid:88) i =
E (cid:32) n (cid:89) k
= v 2 w k i (cid:33) =
d n − 1 d (cid:88) i = n (cid:89) i =  d =
B. P ROOF OF P ROPOSITION 3.
Let f ( x )
=
i ( x ) = g ( x ) = x with
[Equation]
Substituting these parameters into the LSTM update (3) and using h 0 =
we have ∀ t > 0 that
[Equation]

Thus h T =    
[Equation] [Equation]
˜
[Equation]
n
[Equation] [Equation]
    =    ˜ z (
[Equation]
˜ v
[Equation]
˜ v {
[Equation]
T − n +2 ,...
[Equation]
w
[Equation]
}   
Note that h t ∈ R (2 n − 1) d
so as desired the LSTM has O ( nd ) -memory.
Although h T contains ( n − 1) d more dimensions than ˜ z ( n ) , by padding the end of the document with an end-of-document token whose word vector is 0 d the entries in those dimensions will be set to zero by the update at the last step.
Thus up to zero padding we will have z LSTM =
h T =
˜
z ( n ) .
C P ROOF OF T HEOREM 4.
Throughout this section we assume the setting described in Theorem 4.2.
Furthermore for some positive constant C deﬁne the (cid:96) 2 -regularization of the loss function (cid:96) as
[Equation]
Lemma C.1.
Let ˆ w be the classiﬁer obtained minimizing L S
[Equation]
where (cid:96) ( · , · ) is a convex λ -Lipschitz function in the ﬁrst cordinate.
Then
[Equation]
14) where | α i | ≤ λCm ∀ i .
This result holds in the compressed domain as well.
Proof.
If (cid:96) is an λ -Lipschitz function, its sub-gradient at every point is bounded by λ .
So by convexity, the unique optimizer is given by taking ﬁrst-order conditions:
[Equation]
Since (cid:96) is Lipschitz, | ∂ w T x i (cid:96) ( w T x i , y i ) | ≤ λ .
Therefore the ﬁrst-order optimal solution (15) of ˆ w can be expressed as (14) for some α 1 , . . .
, α m satisfying | α i | ≤ λCm ∀ i , which is the desired result.
Lemma C
[Equation]
Proof.
Since A is (∆ X , ε ) -RIP we have (1 − ε ) (cid:107) x − x (cid:48) (cid:107) 2 ≤ (cid:107)
A
[Equation]
x − x (cid:48) )
(cid:107) 2 ≤ (1 + ε ) (cid:107) x − x (cid:48)
(cid:107) 2 .
Also since 0 N ∈ X , A is also ( X , ε ) -RIP and the result then follows by the same argument as in (Calderbank et al., 2009, Lemma 4.2-3).
Corollary C.1.
(cid:107) ˆ w (cid:107) 22 ≤ λ 2 C 2 R 2 and (cid:107) ˆ w A (cid:107) 22 ≤ λ 2 C 2 (1 + ε ) 2 R 2 .
Proof.
The ﬁrst bound follows by expanding (cid:107) ˆ w (cid:107) 22 and using (cid:107) x (cid:107) 2 ≤ R ; the second follows by expanding (cid:107) ˆ w A (cid:107) 22 , applying Lemma C.2 to bound inner product distortion, and using (cid:107) x (cid:107) 2 ≤ R .
Lemma C.3.
Let ˆ w be the linear classiﬁer minimizing L S .
Then
[Equation]
Proof.
By Lemma C.1 we can re-express ˆ w using Equation 14 and then apply the inequality from
Lemma C.2 to get
[Equation]
T ( Ax ) = m
[Equation]
i
[Equation]
Ax
[Equation] [Equation]
Ax )
[Equation]
i :
[Equation] [Equation]
[Equation]
ε
[Equation]
|
[Equation]
for any x ∈ R N .
Since (cid:96) is λ -Lipschitz taking expectations over D implies
[Equation] [Equation]
Substituting Equation 14 applying Lemma C.2 also yields
[Equation]
which implies
[Equation]
Together the inequalities bounding the loss term (16) and the regularization term (17) imply the result.
Lemma C.4.
Let ˆ w be the linear classiﬁer minimizing L S and let w ∗ be the linear classiﬁer minimizing L D .
Then with probability 1 − δ
[Equation]
This result holds in the compressed domain as well.
Proof.
By Corollary C.
we have that ˆ w is contained in a closed convex subset independent of S .
Therefore since (cid:96) is λ -Lipschitz, L is 1 C -strongly convex, and (cid:107) x (cid:107)
≤ O ( R ) , we have by (Sridharan et al., 2008, Theorem 1) that with probability
[Equation]
δ
[Equation]
Then since by deﬁnition ˆ w minimizes L S ( w ) we have that L S ( ˆ w ) ≤ L S ( w ∗ ) , which substituted into the previous equation completes the proof.
Proof of Theorem 4.2.
Applying Lemma C.4 in the compressed domain yields
[Equation]
where w ∗ A minimizes L D .
By deﬁnition of w ∗ A , L D ( w ∗ A ) ≤ L D ( A ˆ w ) , so together with Lemma C. and the previous inequality we have
[Equation]
We now apply Lemma C.4 in the sparse domain to get
[Equation]
where w ∗ minimizes L D .
By deﬁnition of w ∗ , L D ( w ∗ ) ≤ L D ( w 0 ) =
(cid:96) D ( w 0 )
+ 12 C (cid:107)
w
[Equation]
22 , so by the previous inequality we have
[Equation]
Substituting the C that minimizes the r.h.s.
of this inequality completes the proof.
D P ROOF OF L EMMA 4.
We assume the setting described in Lemma 4.1, where we are concerned with the RIP condition of the matrix ˜
A n when multiplying vectors x ∈ X ( n ) T , the set of BonC vectors for documents of length at most T .
This matrix can be written as
[Equation]
  where A p is the d × V p matrix whose columns are the DisC embeddings of all p -grams in the vocabulary (and thus ˜
A 1 =
A 1 = A , the matrix of the original word embeddings).
Note that from (1) any x ∈ X ( n ) T can be writen as x = [ x 1 , . . .
, x n ] , where x p is a T -sparse vector whose entries correspond to p -grams.
Thus we also have ˜ A n x =
[ A 1 x 1 , . . .
, A n x n ] .
Lemma D.1.
If A p is (2 k, ε ) -RIP w.p. 1 − δ ∀ p ∈ [ n ] then ˜
A n is (cid:16) ∆ X ( n ) k , ε (cid:17) -RIP w.p. at least 1 − nδ .
Proof.
By union bound we have that A p is (2 k, ε ) -RIP
∀ p ∈
[ n ] with probability at least 1 − nδ .
Thus by Deﬁnition 4.1 we have w.p. 1 − nδ that ∀ x ∈ ∆ X ( n ) k (cid:107) ˜
A n x (cid:107) 22 = n (cid:88) p
= (cid:107)
A p x p (cid:107) 22 ≤ n (cid:88) p
= (1 + ε ) 2 (cid:107) x p (cid:107) 
 = (1 + ε ) 2 (cid:107) x (cid:107)
Similarly, (cid:107) ˜
A n x (cid:107) 22 ≥ (1 − ε ) 2 (cid:107) x (cid:107) 22 .
From Deﬁnition 4.1, taking the square root of both sides of both inequalities completes the proof.
Deﬁnition D.1 (Foucart & Rauhut, 2013) .
Let D be a distribution over a subset S ⊂ R n .
Then the set Φ = { φ 1 , . .
. , φ N } of functions φ i : S (cid:55)→ R is a bounded orthonormal system (BOS) with constant B if we have E D ( φ i φ j ) =
i = j ∀ i, j and sup s ∈ S | φ i ( s ) | ≤ B ∀ i .
Note that by deﬁnition
B ≥ 1 .
Theorem D.1 (Foucart & Rauhut, 2013) .
If d = ˜Ω (cid:16) B 2 k ε 2 log Nδ (cid:17) for ( ε, δ ) ∈ (0 , 1) and √ dA is a d × N matrix associated with a BOS with constant B then A is ( k, ε ) -RIP w.p. 1 − δ .
Lemma D.2.
If d = ˜Ω (cid:16) Tε 2 log V p δ (cid:17) and the word embeddings are drawn i.i.d.
from U d {± 1 / √ d } then for any p ∈ [ n ] the matrix
A p ∈ R
d × V p of DisC embeddings is ( T, ε ) -RIP w.p. 1 − δ .
Proof.
Note that by Theorem D.1 it sufﬁces to show that √ dA p is a random sampling matrix associated with a BOS with constant B = 1 .
Let D
=
U V {± 1 } be the uniform distribution over
V i.i.d.
Rademacher random variables indexed by words in the vocabulary.
Then by deﬁnition the matrix
A p ∈ R
d × V p can be constructed by drawing random variables x (1) , . . .
, x ( d ) i.i.d.
from D and assigning to the ij th entry of √ dA p corresponding to the p -gram g = { g 1 , . . .
, g p } the value φ j (cid:0) x
( i ) (cid:1) =
(cid:81) pt
=1 x ( i ) g t , where each function φ j : {± 1 } V (cid:55)→ R is uniquely associated to its p -gram.
It remains to be shown that this set of functions is a BOS with constant B = 1 .
For any two p -grams g, g (cid:48) and their functions φ i , φ j we have E D ( φ i φ j ) =
E x ∼D (cid:0)(cid:81) pt =1 x g t x g (cid:48) t (cid:1) , which will be 1 iff each word in g ∪ g (cid:48) occurs an even number of times in the product and 0 otherwise.
Because all p -grams are uniquely deﬁned under any permutation of its words (i.e. we are in fact using p -cooccurrences) and we have assumed that no p -gram contains a word more than once, each word occurs an even number of times in the product iff g = g (cid:48)
⇐⇒ i
= j .
Furthermore we have that | φ
i ( x ) | ≤ 1 ∀ x ∈ {± 1 } V ∀ i by construction.
Thus according to Deﬁnition D.1 the set of functions { φ 1 , . . .
, φ V p } associated to the p -grams in the vocabulary is a BOS with constant B = 1 .
Proof of Lemma 4.1.
Since d = ˜Ω (cid:16) Tε 2 log nV max n δ (cid:17) , Lemma D.2 implies that A p is (2 k, ε ) -RIP w.p.
1 − δ n ∀ p ∈ [ n ] .
Applying Lemma
D.1 yields the result.
E S PARSE R ECOVERY WITH P RETRAINED E MBEDDINGS E. P ERFORMANCE C OMPARISON OF A LTERNATIVE E MBEDDINGS
We show in Figure 6 that the surprising effectiveness of word embeddings as linear measurement vectors for BoW signals holds for other embedding objectives and corpora as well.
Speciﬁcally, we see that widely used embeddings, when normalized, match the efﬁciency of random vectors for retrieving SST BoW and are more efﬁcient when retrieving IMDB BoW. Interestingly, SN vectors are most efﬁcient and are also the only embeddings for normalizing is not needed for good performance.
[Figure]
so all documents are required to be subsets of the vocabulary of all [-vectors v w with w not in the document together with the origin 
-]corpora.
word2vec embeddings trained on Google News and GloVe vectors trained on Common Crawl were obtained from public repositories (Mikolov et al., 2013; Pennington et al., 2014) while Amazon and Wikipedia embeddings were trained for 100 iterations using a symmetric window of size 10, a min count of 100, for SN/GloVe a cooccurrence cutoff of 1000, and for word2vec a down-sampling frequency cutoff of 10 − 5 and a negative example setting of 3.
300-dimensional normalized random vectors are used as a baseline.
E. A M ODEL -B ASED T HEORETICAL A PPROACH
In Section 5.2 we gave some intuition for why pretrained word embeddings are efﬁcient sensing vectors for natural language BoW by examining a geometric characterization of local equivalence due to Donoho & Tanner (2005) in light of the usual similarity properties of word embeddings.
However, this analysis does not provide a rigorous theory for our empirical results.
In this section we brieﬂy discuss a model-based justiﬁcation that may lead to a stronger understanding.
We need a model relating BoW generation to the word embeddings trained over words co-occurring in the same BoW. As a starting point consider the model of Arora et al.
(2016), in which a corpus is generated by a random walk c t over the surface of a ball in R d ; at each t a word w is emitted w.p.
[Equation]
Minimizing the SN objective approximately maximizes corpus likelihood under this model.
Thus in an approximate sense a document of length T is generated by setting a context vector c and emitting T words via (18) with c t = c .
This model is a convenient one for analysis due its simplicity and invariance to word order as well as the fact that the approximate maximum likelihood document vector is the sum of the embeddings of words in the document.
Building upon the intuition established following Corollary 5.1 one can argue that, if we have the true latent SN vectors, then embeddings of words in the same document (i.e. emitted by the same context vector) will be close to each other and thus easy to separate from the embeddings of other words in the vocabulary.
However, we ﬁnd empirically that not all of the T words closest to the sum of the word embeddings (i.e. the context vector) are the ones emitted; indeed individual word vectors in a document may have small, even negative inner product with the context vector and still be recovered via BP.
Thus any further theoretical argument must also be able to handle the recovery of lower probability words whose vectors are further away from the context vector than those of words that do not appear in the document.
We thus leave to future work the challenge of explaining why embeddings resulting from this (or another) model provide such efﬁcient sensing matrices for natural language BoW.