A COMPRESSED SENSING VIEW OF UNSUPERVISED TEXT EMBEDDINGS , BAG - OF - n - GRAMS , AND LSTM S
A BSTRACT
Low-dimensional vector embeddings, computed using LSTMs or simpler techniques, are a popular approach for capturing the “meaning” of text and a form of unsupervised learning useful for downstream tasks.
However, their power is not theoretically understood.
The current paper derives formal understanding by looking at the subcase of linear embedding schemes.
Using the theory of compressed sensing we show that representations combining the constituent word vectors are essentially information-preserving linear measurements of Bag-of-n-Grams (BonG) representations of text.
This leads to a new theoretical result about LSTMs: low-dimensional embeddings derived from a low-memory LSTM are provably at least as powerful on classiﬁcation tasks, up to small error, as a linear classiﬁer over BonG vectors, a result that extensive empirical work has thus far been unable to show.
Our experiments support these theoretical ﬁndings and establish strong, simple, and unsupervised baselines on standard benchmarks that in some cases are state of the art among word-level methods.
We also show a surprising new property of embeddings such as GloVe and word2vec: they form a good sensing matrix for text that is more efﬁcient than random matrices, the standard sparse recovery tool, which may explain why they lead to better representations in practice.
I NTRODUCTION
Much attention has been paid to using LSTMs (Hochreiter & Schmidhuber, 1997) and similar models to compute text embeddings (Bengio et al., 2003; Collobert & Weston, 2008).
Once trained, the
LSTM can sweep once or twice through a given piece of text, process it using only limited memory, and output a vector with moderate dimensionality (a few hundred to a few thousand), which can be used to measure text similarity via cosine similarity or as a featurization for downstream tasks.
The powers and limitations of this method have not been formally established.
For example, can such neural embeddings compete with and replace traditional linear classiﬁers trained on trivial
Bag-of- n -Grams (BonG) representations?
Tweaked versions of BonG classiﬁers are known to be a surprisingly powerful baseline (Wang & Manning, 2012) and have fast implementations (Joulin et al., 2017).
They continue to give better performance on many downstream supervised tasks such as IMDB sentiment classiﬁcation (Maas et al., 2011) than purely unsupervised LSTM representations (Kiros et al., 2015; Hill et al., 2016;
Pagliardini et al., 2017).
Even a very successful character-level (and thus computation-intensive, taking a month of training) approach does not reach BonG performance on datasets larger than IMDB
(Radford et al., 2017).
Meanwhile there is evidence suggesting that simpler linear schemes give compact representations that provide most of the beneﬁts of word-level LSTM embeddings (Wieting et al., 2016; Arora et al., 2017).
These linear schemes consist of simply adding up, with a few modiﬁcations, standard pretrained word embeddings such as GloVe or word2vec (Mikolov et al., 2013; Pennington et al., 2014).
The current paper ties these disparate threads together by giving an information-theoretic account of linear text embeddings.
We describe linear schemes that preserve n -gram information as lowdimensional embeddings with provable guarantees for any text classiﬁcation task.
The previous linear schemes, which used unigram information, are subcases of our approach, but our best schemes can also capture n -gram information with low additional overhead.
Furthermore, we show that the original unigram information can be (approximately) extracted from the low-dimensional embedding using sparse recovery/compressed sensing (Candès & Tao, 2005).
Our approach also ﬁts in the tradition of the older work on distributed representations of structured objects, especially the works of Plate (1995) and Kanerva (2009).
The following are the main results achieved by this new world-view: 1.
Using random vectors as word embeddings in our linear scheme (instead of pretrained vectors) already allows us to rigorously show that low-memory LSTMs are provably at least as good as every linear classiﬁer operating on the full BonG vector.
This is a novel theoretical result in deep learning, obtained relatively easily.
By contrast, extensive empirical study of this issue has been inconclusive (apart from character-level models, and even then only on smaller datasets (Radford et al., 2017)).
Note also that empirical work by its nature can only establish performance on some available datasets, not on all possible classiﬁcation tasks.
We prove this theorem in Section 4 by providing a nontrivial generalization of a result combining compressed sensing and learning (Calderbank et al., 2009).
In fact, before our work we do not know of any provable quantiﬁcation of the power of any text embedding.
We study theoretically and experimentally how our linear embedding scheme improves when it uses pretrained embeddings (GloVe etc.)
instead of random vectors.
Empirically we ﬁnd that this improves the ability to preserve Bag-of-Words (BoW) information, which has the following restatement in the language of sparse recovery: word embeddings are better than random matrices for “sensing” BoW signals (see Section 5).
We give some theoretical justiﬁcation for this surprising ﬁnding using a new sparse recovery property characterizing when nonnegative signals can be reconstructed by (cid:96) 1 -minimization.
Section 6 provides empirical results supporting the above theoretical work, reporting accuracy of our linear schemes on multiple standard classiﬁcation tasks.
Our embeddings are consistently competitive with recent results and perform much better than all previous linear methods.
Among unsupervised word-level representations they achieve state of the art performance on both the binary and ﬁne-grained SST sentiment classiﬁcation tasks (Socher et al., 2013).
Since our document representations are fast, compositional, and simple to implement given standard word embeddings, they provide strong baselines for future work.
[Figure]
R ELATED W ORK
Neural text embeddings are instances of distributed representations , long studied in connectionist approaches because they decay gracefully with noise and allow distributed processing.
Hinton (1990) provided an early problem formulation, and Plate (1995) provided an elementary solution, the holographic distributed representation , which represents structured objects using circular vector convolution and has an easy and more compact implementation using the fast Fourier transform (FFT).
Plate suggested applying such ideas to text, where “structure” can be quantiﬁed using parse trees and other graph structures.
Our method is also closely related in form and composition to the sparse distributed memory system of Kanerva (2009).
In the unigram case our embedding reduces to the familiar sum of word embeddings, which is known to be surprisingly powerful (Wieting et al., 2016), and with a few modiﬁcations even more so (Arora et al., 2017).
Representations of BonG vectors have been studied through the lens of compression by Paskov et al.
(2013), who computed representations based on classical lossless compression algorithms using a linear program (LP).
Their embeddings are still high-dimensional ( d > 100 K ) and quite complicated to implement.
In contrast, linear projection schemes are simpler, more compact, and can leverage readily available word embeddings.
Pagliardini et al.
(2017) also used a linear scheme, representing documents as an average of learned word and bigram embeddings.
However, the motivation and beneﬁts of encoding BonGs in low-dimensions are not made explicit.
The novelty in the current paper is the connection to compressed sensing, which is concerned with recovering high-dimensional sparse signals x ∈ R N from low-dimensional linear measurements Ax , speciﬁcally by studying conditions on matrix A ∈ R d × N when this is possible (see Appendix A for some background on compressed sensing and the previous work of Calderbank et al.
(2009) that we build upon).
D OCUMENT E MBEDDINGS
In this section we deﬁne the two types of representations that our analysis will relate:
high-dimensional BonG vectors counting the occurrences of each k -gram for k ≤ n 2.
low-dimensional embeddings, from simple vector sums to novel n -gram-based embeddings
Although some of these representations have been previously studied and used, we deﬁne them so as to make clear their connection via compressed sensing, i.e. that representations of the second type are simply linear measurements of the ﬁrst.
We now deﬁne some notation.
Let V be the number of words in the vocabulary and V n be the number of n -grams (independent of word order), so that V = V 1 .
Furthermore set V sum n = (cid:80) k ≤ n V k and
V max n = max k ≤ n V k .
We will use words/ n -grams and indices interchangeably, e.g. if ( a, b ) is the i th of V 2 bigrams then the one-hot vector e ( a,b ) will be 1 at index i .
Where necessary we will use { , } to denote a multi-set and ( , ) to denote a tuple.
For any m vectors v i ∈ R d for i = 1 , . . .
, m we deﬁne [ v 1 , . .
. , v m ] to be their concatenation, which is thus an element of R md .
Finally, for any subset X ⊂ R N we denote by ∆ X the set { x − x (cid:48) :
x, x (cid:48) ∈ X} .
3. T HE B AG - OF - n -G RAMS V ECTOR
Assigning to each word a unique index
i ∈
[ V ] we deﬁne the Bag-of-Words (BoW) representation x BoW of a document to be the V -dimensional vector whose i th entry is the number of times word i occurs in the document.
The n -gram extension of BoW is the Bag-of- n -Grams (BonG) representation, which counts the number of times any k -gram for k ≤ n appears in a document.
Linear classiﬁcation over such vectors has been found to be a strong baseline (Wang & Manning, 2012).
For ease of analysis we simplify the BonG approach by merging all n -grams in the vocabulary that contain the same words but in a different order.
We call these features n -cooccurrences and ﬁnd that the modiﬁcation does not affect performance signiﬁcantly (see Table 3 in Appendix E.2).
Formally for a document w 1 , . . .
, w T we deﬁne the Bag-of- n -Cooccurrences (BonC) vector as the concatenation
[Equation]
which is thus a V sum n -dimensional vector.
Note that for unigrams this is equivalent to the BoW vector.
3. L OW -D IMENSIONAL n -G RAM E MBEDDINGS
Now suppose each word w has a vector v w ∈ R d for some d (cid:28) V .
Then given a document w 1 , . . .
, w T we deﬁne its unigram embedding as z u = (cid:80) Tt =1 v w t .
While this is a simple and widely used featurization, we focus on the following straightforward relation with BoW: if A ∈ R d × V is a matrix whose columns are word vectors v w then Ax BoW =
(cid:80) Tt =
1 Ae w t =
(cid:80) Tt
=1 v w t = z u .
Thus in terms of compressed sensing the unigram embedding of a document is a d -dimensional linear measurement of its Bag-of-Words vector.
We could extend this unigram embedding to n -grams by ﬁrst deﬁning a representation for each ngram as the tensor product of the vectors of its constituent words.
Thus for each bigram b
= ( w 1 , w 2 ) we would have v b = v w 1 v Tw 2 and more generally v g
= (cid:78) nt
=1 v w t for each n -gram g = ( w 1 , . . . , w n ) .
The document embedding would then be the sum of the tensor representations of all n -grams.
The major drawback of this approach is of course the blowup in dimension, which in practice prevents its use beyond n = 2 .
To combat this a low-dimensional sketch or projection of the tensor product can be used, such as the circular convolution operator of Plate (1995).
Since we are interested in representations that can also be constructed by an LSTM, we instead sketch this tensor product using the element-wise multiplication operation, which we ﬁnd also usually works better than circular convolution in practice (see Table 4 in Appendix E.2).
Thus for the n -cooccurrence g = { w 1 , . . .
, w n } , we deﬁne the distributed cooccurrence (DisC) embedding ˜ v g =
d n − 1 2 (cid:74) nt
=1 v w t .
The coefﬁcient is required when the vectors v w are random and unit norm to ensure that the product also has close to unit norm (see Lemma B.1).
In addition to their convenient form, DisC embeddings have nice theoretical and practical properties: they preserve the original embedding dimension, they reduce to unigram (word) embeddings for n = 1 , and under mild assumptions they satisfy useful compressed sensing properties with overwhelming probability (Lemma 4.1).
We then deﬁne the DisC document embedding to be the nd -dimensional weighted concatenation, over k ≤ n , of the sum of the DisC vectors of all k -grams in a document:
[Equation]
Here scaling factors C k are set so that all spans of d coordinates have roughly equal norm (for random embeddings C k
= 1 ; for word embeddings C k = 1 /k works well).
Note that since ˜ v w t = v w t we have z (1) =
z u in the unigram case.
Furthermore, as with unigram embeddings by comparing (1) and (2) one can easily construct a (cid:80) nk =1 dn × V sum n matrix A ( n ) such that z ( n ) =
A ( n ) x BonC .
3. LSTM R EPRESENTATIONS
As discussed previously, LSTMs have become a common way to apply the expressive power of
RNNs, with success on a variety of classiﬁcation, representation, and sequence-to-sequence tasks.
For document representation, starting with h 0 =
m an m -memory LSTM initialized with word vectors v w ∈ R d takes in words w 1 , . . .
, w T one-by-one and computes the document representation
[Equation]
where h t ∈ R m is the hidden representation at time t , the forget gate f , input gate i , and input function g are a.e. differentiable nondecreasing elementwise “activation” functions R m (cid:55)→ R m , and afﬁne transformations T ∗ ( x, y )
=
W ∗ x + U ∗ y + b ∗ have weight matrices
W ∗ ∈ R m × d , U ∗ ∈ R m × m and bias vectors b ∗ ∈ R m .
The LSTM representation of a document is then the state at the last time step, i.e. z LSTM =
h T .
Note that we will follow the convention of using LSTM memory to refer to the dimensionality of the hidden states.
Since the LSTM is initialized with an embedding for each word it requires O ( m 2 + md + V d ) computer memory, but the last term is just a lookup table so the vocabulary size does not factor into iteration or representation complexity.
From our description of LSTMs it is intuitive to see that one can initialize the gates and input functions so as to construct the DisC embeddings deﬁned in the previous section.
We state this formally and give the proof in the unigram case (the full proof appears in Appendix B.3):
Proposition 3.1.
Given word vectors v w ∈ R d , one can initialize an O ( nd ) -memory
LSTM (3) that takes in words w 1 , . . .
, w T (padded by an end-of-document token assigned vector 0 d ) and constructs the DisC embedding (2) (up to zero padding), i.e. such that for all documents z LSTM =
z ( n ) .
Proof (Unigram Case).
Set f
( x ) =
i ( x ) = g ( x ) = x , T f ( v w t , h t − 1 ) =
T i ( v w t , h t − 1 ) = 1 d , and
T g ( v w t , h t − 1 ) =
C 1 v w t .
Then h t =
h t − 1 + C 1 v w t , so since h 0 =
d we have the ﬁnal LSTM representation z LSTM =
h T =
C 1 (cid:80) tt
=1 v w t =
z (1) .
By Proposition 3.1 we can construct a ﬁxed LSTM that can compute compressed BonC representations on the ﬂy and be further trained by stochastic gradient descent using the same memory.
LSTM S AS C OMPRESSED L EARNERS
Our main contribution is to provide the ﬁrst rigorous analysis of the performance of the text embeddings that we are aware of, showing that the embeddings of Section 3.2 can provide performance on downstream classiﬁcation tasks at least as well any linear classiﬁer over BonCs.
Before stating the theorem we make two mild simplifying assumptions on the BonC vectors:
The vectors are scaled by 1 T √ n , where T is the maximum document length.
This assumption is made without loss of generality.
No n -cooccurrence contains a word more than once.
While this is (infrequently) violated in practice, the problem can be circumvented by merging words as a preprocessing step.
Theorem 4.1.
Let S = { ( x i , y i ) } mi =1 be drawn i.i.d.
from a distribution D over BonC vectors of documents of length at most T satisfying assumptions 1 and 2 above and let w 0 be the linear classiﬁer minimizing the logistic loss (cid:96) D .
Then for dimension d = ˜Ω (cid:16) Tε 2 log nV max n γ (cid:17) and appropriate choice of regularization coefﬁcient one can initialize an O ( nd ) -memory
LSTM over i.i.d.
word embeddings v w ∼ U d {± 1 / √ d } such that w.p.
(1 − γ )(1 − 2 δ ) the classiﬁer ˆ w minimizing the (cid:96) 2 -regularized logistic loss over its representations satisﬁes
[Equation]
The above theoretical bound shows that LSTMs match BonC performance as ε → 0 , which can be realized by increasing the embedding dimension d (c.f. Figure 4).
4. C OMPRESSED S ENSING AND L EARNING
Compressed sensing is concerned with recovering a high-dimensional k -sparse signal x ∈ R N from a few linear measurements; given a design matrix A ∈ R d × N this is formulated as
[Equation]
where z =
Ax is the measurement vector .
As l 0 -minimization is NP-hard, research has focused on sufﬁcient conditions for tractable recovery.
One such condition is the Restricted Isometry Property (RIP), for which Candès & Tao (2005) proved that (5) can be solved by convex relaxation:
Deﬁnition 4.1.
A ∈ R d × N is ( X , ε ) -RIP for some subset X ⊂ R N if ∀ x ∈ X
[Equation]
We will abuse notation and say ( k, ε ) -RIP when X is the set of k -sparse vectors.
This is the more common deﬁnition, but ours allows a more general Theorem 4.2 and a tighter bound in Theorem 4.1.
Following these breakthroughs, Calderbank et al.
(2009) studied whether it is possible to use the low-dimensional output of compressed sensing as a surrogate representation for classiﬁcation.
They proved a learning-theoretic bound on the loss of an SVM classiﬁer in the compressed domain compared to the best classiﬁer in the original domain.
In this work we are interested in comparing the performance of LSTMs with BonC representations, so we need to generalize the Calderbank et al.
(2009) result to handle Lipschitz losses and an arbitrary set X ⊂ R N of high-dimensional signals:
Theorem 4.2.
For any subset X ⊂ R N containing the origin let A ∈ R d
× N be (∆ X , ε ) -RIP and let m samples S
=
( x i , y i ) } mi =1 ⊂ X × {− 1 , 1 } be drawn i.i.d.
from some distribution D over X with (cid:107) x (cid:107) 2 ≤ R .
If (cid:96) is a λ -Lipschitz convex loss function and w 0 ∈ R N is its minimizer over D then w.p. 1 − 2 δ the linear classifer ˆ w A ∈ R d minimizing the (cid:96) 2 -regularized empirical loss function (cid:96)
S A ( w ) + 1 2 C (cid:107) w (cid:107) 22 over the compressed sample S A
= { ( Ax i , y i ) } mi =
⊂ R d × {− 1 , 1 } satisﬁes
[Equation]
for appropriate choice of C .
Recall that ∆ X = { x − x (cid:48) :
x, x (cid:48) ∈ X} for any X ⊂ R N .
While a detailed proof of this theorem is spelled out in Appendix C, the main idea is to compare the distributional loss incurred by a classiﬁer ˆ w in the original space to the loss incurred by A ˆ w in the compressed space.
We show that the minimizer of the regularized empirical loss in the original space ( ˆ w ) is a bounded-coefﬁcient linear combination of samples in S , so its loss depends only on inner products between points in X .
Thus using RIP and a generalization error result by Sridharan et al.
(2008) we can bound the loss of ˆ w A , the regularized classiﬁer in the compressed domain.
Note that to get back from Theorem 4.2 the O ( √ ε ) bound for k -sparse inputs of Calderbank et al.
(2009) we can set X to the be the set of k -sparse vectors and assume A is (2 k, ε ) -RIP.
4. P ROOF OF M AIN R ESULT
To apply Theorem 4.2 we need the design matrix A ( n ) transforming BonCs into the DisC embeddings of Section 3.2 to satisfy the following RIP condition (Lemma 4.1), which we prove using a restricted isometry result for structured random sampling matrices in Appendix D:
Lemma 4.1.
Assume the setting of Theorem 4.1 and let A ( n ) be the nd × V sum n matrix relating DisC and BonC representations of any document by z ( n )
=
A ( n ) x BonC .
If d = ˜Ω (cid:16) Tε 2 log nV max n γ (cid:17) then A ( n ) is (cid:16) ∆ X ( n ) T , ε (cid:17) -RIP w.p. 1 − γ , where X ( n ) T is the set of BonCs of documents of length at most T .
Proof of Theorem 4.1.
Let ˆ S
= { ( A ( n ) x i , y i ) : ( x i , y i ) ∈ S } , where A ( n ) is as in Lemma 4.1.
Then by the same lemma A ( n ) is (cid:16) ∆ X ( n ) T , ε (cid:17) -RIP w.p. 1 − γ , where X ( n ) T is the set of BonC vectors of documents of length at most T .
By BonC assumption (1) all BonCs lie within the unit ball, so we can apply Theorem 4.2 with (cid:96) the logistic loss, λ = 1 , and R = 1 to get that a classiﬁer ˆ w trained using (cid:96) 2regularized logistic loss over ˆ S will satisfy the required bound (4).
Since by Proposition 3.1 one can initialize an O ( nd ) -memory
LSTM that takes in i.i.d.
Rademacher word vectors v w ∼ U d {± 1 / √ d } such that z LSTM =
z ( n ) =
A ( n ) x ∀ x ∈ X ( n ) T , this completes the proof.
S PARSE R ECOVERY WITH P RETRAINED E MBEDDINGS
Theorem 4.1 is proved using random vectors as the word embeddings in the scheme of Section 3.
However, in practice LSTMs are often initialized with standard word vectors such as GloVe.
Such embeddings cannot satisfy traditional compressed sensing properties such as RIP or incoherence.
This follows essentially from the deﬁnition: word embeddings seek to capture word similarity, so similar words (e.g. synonyms) have embeddings with high inner product, which violates both properties.
Thus the efﬁcacy of real-life LSTMs must have some other explanation.
But in this section we present the surprising empirical ﬁnding that pretrained word embeddings are more efﬁcient than random vectors at encoding and recovering BoW information via compressed sensing.
We further sketch a potential explanation for this result, though a rigorous explanation is left for subsequent work.
5. P RETRAINED E MBEDDINGS P RESERVE S PARSE I NFORMATION
In recent years word embeddings have been discovered to have many remarkable properties, most famously the ability to solve analogies (Mikolov et al., 2013).
Our connection to compressed sensing indicates that they should have another: preservation of sparse signals as low-dimensional linear measurements.
To examine this we subsample documents from the SST (Socher et al., 2013) and
IMDB (Maas et al., 2011) classiﬁcation datasets, embed them as d -dimensional unigram embeddings z = Ax for d = 50 , 100 , 200 , . . .
, 1600 (where A ∈ R d × V is the matrix of word embeddings and x is a document’s BoW vector), solve the following LP, known as Basis Pursuit (BP), which is the standard (cid:96) 1 -minimization problem for sparse recovery in the noiseless case (see Appendix A):
[Equation]
Success is measured as the F 1 score of retrieved words.
We use Squared Norm (SN) vectors (Arora et al., 2016) trained on a corpus of Amazon reviews (McAuley et al., 2015) and normalized i.i.d.
Rademacher vectors as a baseline.
SN is used due to similarity to GloVe and its formulation via an easy-to-analyze generative model that may provide a framework to understand the results (see
Appendix E.3), while the Amazon corpus is used for its semantic closeness to the sentiment datasets.
documents average Figures 1 and 2 show that pretrained embeddings require a lower dimension d than random vectors to recover natural language BoW. This is surprising as the training objective goes against standard conditions such as approximate isometry and incoherence; indeed as shown in Figure 2 recovery is poor for randomly generated word collections.
The latter outcome indicates that the fact that a document is a set of mutually meaningful words is important for sparse recovery using embeddings trained on co-occurrences.
We achieve similar results with other objectives (e.g. GloVe/word2vec) and other corpora (see Appendix E.2), although there is some sensitivity to the sparse recovery method, as other (cid:96) 1 -minimization methods work well but greedy methods, such as Orthogonal
Matching Pursuit (OMP), work poorly, likely due to their dependence on incoherence (Tropp, 2004).
For the n -gram case (i.e. BonC recovery for n > 1 ), although we know by Lemma 4.1 that DisC embeddings composed from random vectors satisfy RIP, for pretrained vectors it is unclear how to reason about suitable n -gram embeddings without a rigorous understanding of the unigram case, and experiments do not show the same recovery beneﬁts.
One could perhaps do well by training on cooccurrences of word tuples, but such embeddings could not be used by a word-level LSTM.
5. U NDERSTANDING S PARSE R ECOVERY OF N ATURAL L ANGUAGE D OCUMENTS
As shown in Figure 2, the success of pretrained embeddings for linear sensing is a local phenomenon; recovery is only efﬁcient for naturally occurring collections of words.
However, applying statistical
RIP/incoherence ideas (Barg et al., 2015) to explain this is ruled out since they require collections to be incoherent with high probability, whereas word embeddings are trained to give high inner product to words appearing together.
Thus an explanation must come from some other, weaker condition.
The usual necessary and sufﬁcient requirement for recovering all signals with support S ⊂ [ N ] is the local nullspace property (NSP), which stipulates that vectors in the kernel of A not have too much mass on S (see Deﬁnition A.2).
While NSP and related properties such as restricted eigenvalue (see
Deﬁnition A.3) are hard to check, we can impose some additional structure to formulate an intuitive, veriﬁable perfect recovery condition for our setting.
Speciﬁcally, since our signals (BoW vectors) are nonnegative, we can improve upon solving BP (8) by instead solving nonnegative BP (BP+):
[Equation]
The following geometric result then characterizes when solutions of BP+ recover the correct signal:
Theorem 5.1 (Donoho & Tanner, 2005) .
Consider a matrix A ∈ R d × N and an index subset S ⊂
[ N ] of size k .
Then any nonnegative vector x ∈ R N + with support supp( x ) =
S is recovered from Ax by BP+ iff the set A S of columns of A indexed by S comprise the vertices of a k -dimensional face of the convex hull conv( A ) of the columns of A together with the origin.
This theorem equates perfect recovery of a BoW vector via BP+ with the vectors of its words being the vertices of some face of the polytope conv( A ) .
The property holds for incoherent columns since the vectors are far enough that no one vector is inside the simplex formed by any k others.
On the other hand, pretrained embeddings satisfy it by having commonly co-occurring words close together and other words far away, making it easier to form a face from columns indexed by the support of a
BoW. We formalize this intuition as the Supporting Hyperplane Property (SHP): [Table] [Table]
Deﬁnition 5.1.
A matrix A ∈ R d × N satisﬁes S -SHP for subset S ⊂ [ N ] if its columns are in general position and there is a hyperplane containing the set A S of columns of A indexed by S such that the set of all other columns of A together with the origin are on one side of the hyperplane.
SHP is a very weak property implied by NSP (Corollary A.1).
However, it can be checked by using convex optimization to see if the hyperplane exists (Appendix E.1).
Furthermore, we show (full proof in Appendix A.3) that this hyperplane is the supporting hyperplane of the face of conv( A ) with vertices A S , from which it follows by Theorem 5.1 that SHP characterizes recovery using BP+:
Corollary 5.1.
BP+ recovers any x ∈ R N + with supp( x ) =
S from Ax iff A satisﬁes S -SHP.
Proof Sketch.
By Theorem 5.1 it sufﬁces to show equivalence of S -SHP with the column set A S comprising the vertices of a k -dimensional face of conv( A ) .
A face F of polytope P is deﬁned as its intersection with some hyperplane such that all points in P \ F lie on one side of the hyperplane.
( = ⇒ )
Let F be the face of conv( A ) formed by the columns A S . Then there must be a supporting hyperplane H containing F .
Since the columns of A are in general position, all columns A S = A \ A S lie in conv( A ) \ F and hence must all be on one side of H , so H is the desired hyperplane. ( ⇐ = ) Let H be the hyperplane supporting A S , with all other columns on one side of H .
By convexity, H contains the simplex F of A S .
Any point in conv( A ) \ F can be written as a convex combination of points in F and columns A S , with a positive coefﬁcient on at least one of the columns, and so must lie on the same side of H as A S . Thus A S comprises the vertices of a face F .
Thus perfect recovery of a BoW via BP+ is equivalent to the existence of a hyperplane separating embeddings of words in the document from those of the rest of the vocabulary.
We verify that SHP is indeed more likely to be satisﬁed by word vector designs in Appendix E.1.
Intuitively, words in the same document are trained to have similar embeddings and so will be easier to separate out, providing some justiﬁcation for why pretrained vectors are better for sensing.
As these similarity properties are also used to explain their efﬁcacy on downstream tasks, we conjecture that there is a relationship between embeddings, recovery, and classiﬁcation that may be understood under a generative model (Appendix E.3).
However, the bounds in Section 4 depend on RIP, not recovery, so the empirical results of Section 5.1 by themselves do not apply to that setting.
They do show that the framework of compressed sensing remains relevant even in the case of non-random, pretrained word embeddings. Rep. n
SICK-R ( r/ρ )
SICK-E MRPC (Acc./ F 1 ) DisC (2) 73.6 / 71. SIF 73.7 / 68.
73.5 / 82.1 Sent2Vec 69.3 / 64.
71.3 / 80.7 Sent2Vec 70.1 / 65.
skip-thoughts 82.4 / 76.
SDAE 73.7 / 80.
CNN-LSTM 76.4 / 83.
byte mLSTM 78.5 / 72.
Model information is the same as that in Table 1.
estimated  E MPIRICAL F INDINGS Our theoretical results show that simple tensor product sketch-based n -gram embeddings can approach
BonG performance and be computed by a low-memory LSTM.
In this section we compare these text representations and others on several standard tasks, verifying that DisC performance approaches that of BonCs as dimensionality increases and establishing several baselines for text classiﬁcation.
Code to reproduce results is provided at https://github.com/NLPrinceton/text_embedding .
Tasks:
We test classiﬁcation on MR movie reviews (Pang & Lee, 2005), CR customer reviews (Hu &
Liu, 2004), SUBJ subjectivity dataset (Pang & Lee, 2004), MPQA opinion polarity subtask (Wiebe et al., 2005), TREC question classiﬁcation (Li & Roth, 2002), SST sentiment classiﬁcation (binary and ﬁne-grained) (Socher et al., 2013), and IMDB movie reviews (Maas et al., 2011).
The ﬁrst four are evaluated using 10-fold cross-validation, while the others have train-test splits.
In all cases we use logistic regression with (cid:96) 2 -regularization determined by cross-validation.
We further test DisC on the
SICK relatedness and entailment tasks (Marelli et al., 2014) and the MRPC paraphrase detection task (Dolan & Brockett, 2005).
The inputs here are sentences pairs ( a, b ) and the standard featurization for document embeddings x a and x b of a and b is [ | x a − x b | , x a (cid:12) x b ] (Tai et al., 2015).
We use logistic regression for SICK entailment and MRPC and use ridge regression to predict similarity scores for SICK relatedness, with (cid:96) 2 -regularization determined by cross-validation.
Since BonGs are not used for pairwise tasks our theory says nothing about performance here; we include these evaluations to show that our representations are also useful for other tasks.
Embeddings: In the main evaluation (Table 1) we use normalized 1600-dimensional GloVe embeddings (Pennington et al., 2014) trained on the Amazon Product Corpus (McAuley et al., 2015).
We also compare the SN vectors of Section 5 trained on the same corpus with random vectors when varying the dimension (Figure 4).
Both embeddings and code will be released for reproducibility.
Results: We ﬁnd that DisC representation performs consistently well relative to recent unsupervised methods; among word-level approaches it is the top performer on the SST tasks and competes on many others with skip-thoughts and CNN-LSTM, both concatenations of two LSTM representations.
While success may be explained by training on a large and in-domain corpus, being able to use so much text without extravagant computing resources is one of the advantages of a simple approach.
Overall our method is useful as a strong baseline, often beating BonCs and many more complicated approaches while taking much less time to represent and train on documents than neural representations (Figure 3).
Finally, we analyze empirically how well our model approximates BonC performance.
As predicted by Theorem 4.1, the performance of random embeddings on IMDB approaches that of BonC as dimension increases and the isometry distortion ε decreases (Figure 4).
Using pretrained (SN) vectors,
DisC embeddings approach BonC performance much earlier, surpassing it in the unigram case.
C ONCLUSION
In this paper we explored the connection between compressed sensing, learning, and natural language representation.
We ﬁrst related LSTM and BonG methods via word embeddings, coming up with simple new document embeddings based on tensor product sketches.
Then we studied their classiﬁcation performance, proving a generalization of the compressed learning result of Calderbank et al.
(2009) to convex Lipschitz losses and a bound on the loss of a low-dimensional LSTM classiﬁer in terms of its (modiﬁed)
BonG counterpart, an issue which neither experiments nor theory have been able to resolve.
Finally, we showed how pretrained embeddings ﬁt into this sparse recovery framework, demonstrating and explaining their ability to efﬁciently preserve natural language information. A CKNOWLEDGMENTS
We thank Rong Ge, Holden Lee, and Divyarthi Mohan for helpful discussions at various stages of this effort.
The work in this paper was in part supported by NSF grants CCF-1302518 and CCF-1527371,
Simons Investigator Award, Simons Collaboration Grant, and ONR-N00014-16-1-
A C OMPRESSED S ENSING B ACKGROUND
The ﬁeld of compressed sensing is concerned with recovering a high-dimensional k -sparse signal x ∈ R N from few linear measurements.
In the noiseless case this is formulated as
[Equation] where A ∈ R d × N is the design matrix and z
= Ax is the measurement vector .
Since (cid:96)
-minimization is NP-hard, a foundational approach is to use its convex surrogate, the (cid:96) 1 -norm, and characterize when the solution to (10) is equivalent to that of the following LP, known as basis pursuit (BP):
[Equation]
Related approaches such as Basis Pursuit Denoising (LASSO) and the Dantzig Selector generalize
BP to handle signal or measurement noise (Foucart & Rauhut, 2013); however, the word embeddings case is noiseless so these methods reduce to BP.
Note that throughout Section 5 and the Appendix we say that an (cid:96) 1 -minimization method recovers x from Ax if its optimal solution is unique and equivalent to the optimal solution of (10).
An alternative way to approximately solve (10) is to use a greedy algorithm such as matching pursuit (MP) or orthogonal matching pursuit (OMP), which pick basis vectors one at a time by multiplying the measurement vector by A T and choosing the column with the largest inner product (Tropp, 2004).
A. S OME C OMMON S PARSE R ECOVERY C ONDITIONS
One condition through which recovery can be guaranteed is the Restricted Isometry Property (RIP):
Deﬁnition A.1.
A ∈ R d × N is ( k, ε ) -RIP if for all k -sparse x ∈ R N
[Equation]
A line of work started by Candès & Tao (2005) used the RIP property to characterize matrices A such that (10) and (11) have the same minimizer for any k -sparse signal x ; this occurs with overwhelming probability when d = Ω (cid:0) k log Nk (cid:1) and √ dA ij ∼ N (0 , 1) ∀ i, j or √ dA ij ∼ U{− 1 , 1 } ∀ i, j .
Since the ability to recover a signal x from a representation Ax implies information preservation, a natural next step is to consider learning after compression.
Calderbank et al.
(2009) show that for m i.i.d.
k -sparse samples { ( x i , y i ) } mi =1 and a (2 k, ε ) -RIP matrix A , the hinge loss of a classiﬁer trained on { ( Ax i , y i ) } mi =1 is bounded by that of the best linear classiﬁer over the original samples.
Theorem 4.2 provides a generalization of this result to any convex Lipschitz loss function.
RIP is a strong requirement, both because it is not necessary for perfect, stable recovery of k -sparse vectors using ˜ O ( k ) measurements and because in certain settings we are interested in using the above ideas to recover speciﬁc signals — those statistically likely to occur—rather than all k -sparse signals.
The usual necessary and sufﬁcient condition to recover any vector x ∈ R N with index support set
S ⊂ [ N ] is the local nullspace property (NSP), which is implied by RIP:
Deﬁnition A.2 (Foucart & Rauhut, 2013) .
A matrix A ∈ R d × N satisﬁes NSP for a set S ⊂ [ N ] if (cid:107) w S (cid:107) 1 < (cid:107) w S (cid:107) 1 for all nonzero w ∈ ker
( A ) = { v : Av = 0 d } .
Theorem A.1 (Foucart & Rauhut, 2013) .
BP (11) recovers any x ∈ R N + with supp( x ) =
S from Ax iff A satisﬁes NSP for S .
A related condition that implies NSP is the local restricted eigenvalue property (REP):
Deﬁnition A.3 (Raskutti et al., 2010) .
A matrix A ∈ R d × N satisﬁes γ -REP for a set S ⊂
[ N ] if (cid:107) Aw (cid:107) 2 ≥ γ √ d (cid:107) w (cid:107) 2 whenever (cid:107) w S (cid:107) 1 ≤ (cid:107) w S (cid:107) 1 .
Lastly, a simple condition that can sometimes provide recovery guarantees is mutual incoherence :
Deﬁnition A.4.
A ∈ R d × N is µ -incoherent if max a,a (cid:48) | a T a (cid:48) | ≤ µ , where the maximum is taken over any two distinct columns a, a (cid:48) of A .
While incoherence is easy to verify (unlike the previous recovery properties), word embeddings tend to have high coherence due to the training objective pushing together vectors of co-occurring words.
A. C ONDITIONS FOR R ECOVERING N ONNEGATIVE S IGNALS
Apart from incoherence, the properties above are hard show empirically.
However, we are compressing BoW vectors, so our signals are nonnegative and we can impose an additional constraint on (11): minimize (cid:107) w (cid:107)subject to
{+Aw = z, w ≥ 
 d (12)+}
The following geometric result provides guarantees for this nonnegative basis pursuit (BP+) problem:
Theorem A.2 (Donoho & Tanner, 2005) .
Consider a matrix A ∈ R d × N and an index subset S ⊂
[ N ] of size k .
Then any nonnegative vector x ∈ R N + with support supp( x ) =
S is recovered from Ax by BP+ (12) iff the columns of A indexed by S comprise the vertices of a k -dimensional face of the convex hull conv( A ) of the columns of A together with the origin.
The polytope condition is equivalent to nonnegative NSP (NSP+), a weaker form of NSP:
Deﬁnition A.5 (Foucart & Koslicki, 2014) .
A matrix A ∈ R d × N satisﬁes NSP+ for a set S ⊂
[ N ] if w S ≥ 0 N = ⇒ N (cid:80) i =
1 w i > 0 for all nonzero w ∈ ker
( A ) .
Lemma A.1.
If A ∈ R d × N satisﬁes NSP for some S ⊂ [ N ] then it also satisﬁes NSP+ for S .
Proof (Adapted from Foucart & Koslicki (2014)).
Since A satisﬁes NSP, we have (cid:107) w S (cid:107) 1 < (cid:107) w S (cid:107) 1 .
Then for a nonzero w ∈ ker( A ) such that w S ≥ {+
+} we will have
[Equation]
Lemma A.2.
BP+ recovers any x ∈ R N + with supp( x ) =
S from Ax iff A satisﬁes NSP+ for S .
Proof.
( = ⇒ )
: For any nonzero w ∈ ker( A ) such that w S ≥ 0 , ∃ λ > 0 such that x + λw ≥ 0 N and A ( x + λw ) =
Ax .
Since BP+ uniquely recovers x , we have (cid:107) x + λw (cid:107) 1 > (cid:107) x (cid:107) 1 , so NSP+ follows from the following inequality and the fact that λ is positive:
[Equation]
( ⇐ = )
: For any x (cid:48) ≥ 0 such that Ax (cid:48) =
Ax we have that w = x (cid:48) − x ∈ ker
( A ) and w S =
x
(cid:48) S
≥since the support of x is S .
Thus by NSP+ we have that N (cid:80) i =1 w i > 0 , which yields
[Equation]
Thus BP+ will recover x uniquely.
Lemma A.2 shows that NSP+ is equivalent to the polytope condition in Theorem A.2 since they are both necessary and sufﬁcient conditions for perfect recovery via BP+.
Together with Lemma A. it also yields the following Corollary relating NSP to the Supporting Hyperplane Property (SHP) property we introduce (see Deﬁnition 5.1):
Corollary A.1.
If a matrix A ∈ R d × N with columns in general position satisﬁes NSP for some S ⊂ [ N ] then it also satisﬁes S -SHP
A. P ROOF OF C OROLLARY 5.
Recall that it sufﬁces to show equivalence of A being S -SHP with the columns A S forming the vertices of a k -dimensional face of conv( A ) , where we can abuse notation to set A ∈ R d × ( N +1) , with the extra column being the origin 0 d , so long as we constrain N + 1 (cid:54)∈ S .
( = ⇒ ):
The proof of the forward direction appeared in full in the proof sketch (Section 5.2). ( ⇐ = ): A subset F ⊂ R d is a face of conv( A ) if for some hyperplane H = { v : a T v − b = 0 } we have F = conv( A )
∩ H and conv( A ) \ F ⊆ H − =
{ v : a T v − b < 0 } , where H − is the negative halfspace of H .
Deﬁne the simplex ∆ m = { λ ∈ [0 , 1] m : (cid:80) mi
{+=
+}
λ i = 1 } .
Since A is S -SHP we have a hyperplane H
= { v : a T v − b = 0 } containing the columns A S such that A S ⊂ H − .
Thus a T A i − b = 0 ∀ i ∈ S and a T A i − b < 0 ∀ i / ∈ S .
We also know that
F = { (cid:80) i ∈
S λ i
A i : λ ∈ ∆ | S | } ⊆ H by convexity of H .
Since any point y ∈ conv( A ) \ F can be written as y = (cid:80) N +1 i =
λ i
A i for some λ ∈ ∆ N +1 such that ∃ j / ∈ S such that λ j (cid:54) =
0 , we have that
[Equation]
This implies that conv( A ) \ F ⊆
H − and F = conv( A ) ∩ H , so since the columns of A are in general position F is a k -dimensional face of conv( A ) whose vertices are the columns A S .
[Table]
[Table]
Evaluation settings are the same as in Section 6.
Note that for unigrams the two representations are equivalent.
B D OCUMENT E MBEDDINGS B. P ERFORMANCE C OMPARISON OF A LTERNATIVE R EPRESENTATIONS
In this section we compare the performance of several alternative representations with the ones presented in the main evaluation (Table 1).
Table 3 provides a numerical justiﬁcation for our use of unordered n -grams (cooccurrences) instead of n -grams, as the performance of the two featurizations are closely comparable.
In Table 4 we examine the use of circular convolution instead of elementwise multiplication as linear measurements of BonC vectors Plate (1995).
To construct the former from a document w 1 , . .
. , w T we compute
[Equation]
where F is the discrete Fourier transform and F − 1 its inverse.
Note that for n = 1 this is equivalent to the simple unigram embedding (and thus also to the DisC embedding in (2)).
B. S CALING F ACTOR FOR D IS C E MBEDDINGS
Lemma B.1.
If word vectors v w ∈ R
d are drawn i.i.d.
from U d {± 1 / √ d } then for any n -gram g
=
( w 1 , . . .
, w n ) we have E (cid:107) ˜ v g (cid:107)
= 1 .
The same result holds true with the additional assumption that all words in g are distinct if the word vectors are i.i.d.
d -dimensional spherical Gaussians.
Proof.
E (cid:107) ˜ v g (cid:107)
= d (cid:88) i =
E (cid:32) d n − 1n (cid:89) k
= v w k i (cid:33)= d n − 1 d (cid:88) i =
E (cid:32) n (cid:89) k
= v 2 w k i (cid:33) =
d n − 1 d (cid:88) i = n (cid:89) i =  d =
B. P ROOF OF P ROPOSITION 3.
Let f ( x )
=
i ( x ) = g ( x ) = x with
[Equation]
Substituting these parameters into the LSTM update (3) and using h 0 =
we have ∀ t > 0 that
[Equation]

Thus h T =    
[Equation]
[Equation]
    = 
[Equation]

[Equation]
[Equation]
[Equation]
   Note that h t ∈ R (2 n − 1) d
so as desired the LSTM has O ( nd ) -memory.
Although h T contains ( n − 1) d more dimensions than ˜ z ( n ) , by padding the end of the document with an end-of-document token whose word vector is 0 d the entries in those dimensions will be set to zero by the update at the last step.
Thus up to zero padding we will have z LSTM =
h T =
˜
z ( n ) .
C P ROOF OF T HEOREM 4.
Throughout this section we assume the setting described in Theorem 4.2.
Furthermore for some positive constant C deﬁne the (cid:96) 2 -regularization of the loss function (cid:96) as
[Equation]
Lemma C.1.
Let ˆ w be the classiﬁer obtained minimizing L S
[Equation]
where (cid:96) ( · , · ) is a convex λ -Lipschitz function in the ﬁrst cordinate.
Then
[Equation]
14) where | α i | ≤ λCm ∀ i .
This result holds in the compressed domain as well.
Proof.
If (cid:96) is an λ -Lipschitz function, its sub-gradient at every point is bounded by λ .
So by convexity, the unique optimizer is given by taking ﬁrst-order conditions:
[Equation]
Since (cid:96) is Lipschitz, | ∂ w T x i (cid:96) ( w T x i , y i ) | ≤ λ .
Therefore the ﬁrst-order optimal solution (15) of ˆ w can be expressed as (14) for some α 1 , . . .
, α m satisfying | α i | ≤ λCm ∀ i , which is the desired result.
Lemma
[Equation]
Proof.
Since A is (∆ X , ε ) -RIP we have (1 − ε ) (cid:107) x − x (cid:48) (cid:107) 2 ≤ (cid:107) A ( x − x (cid:48) )
(cid:107) 2 ≤ (1 + ε ) (cid:107) x − x (cid:48)
(cid:107) 2 .
Also since 0 N ∈ X , A is also ( X , ε ) -RIP and the result then follows by the same argument as in (Calderbank et al., 2009, Lemma 4.2-3).
Corollary C.1.
(cid:107) ˆ w (cid:107) 22 ≤ λ 2 C 2 R 2 and (cid:107) ˆ w A (cid:107) 22 ≤ λ 2 C 2 (1 + ε ) 2 R 2 .
Proof.
The ﬁrst bound follows by expanding (cid:107) ˆ w (cid:107) 22 and using (cid:107) x (cid:107) 2 ≤ R ; the second follows by expanding (cid:107) ˆ w A (cid:107) 22 , applying Lemma C.2 to bound inner product distortion, and using (cid:107) x (cid:107) 2 ≤ R .
Lemma C.3.
Let ˆ w be the linear classiﬁer minimizing L S .
Then
[Equation]
Proof.
By Lemma C.1 we can re-express ˆ w using Equation 14 and then apply the inequality from
Lemma C.2 to get
[Equation]
for any x ∈ R N .
Since (cid:96) is λ -Lipschitz taking expectations over D implies
[Equation] [Equation]
Substituting Equation 14 applying Lemma C.2 also yields
[Equation]
which implies
[Equation]
[Equation]
Together the inequalities bounding the loss term (16) and the regularization term (17) imply the result.
Lemma C.4.
Let ˆ w be the linear classiﬁer minimizing L S and let w ∗ be the linear classiﬁer minimizing L D .
Then with probability 1 − γ
[Equation]
This result holds in the compressed domain as well.
Proof.
By Corollary C.
we have that ˆ w is contained in a closed convex subset independent of S .
Therefore since (cid:96) is λ -Lipschitz, L is 1 C -strongly convex, and (cid:107) x (cid:107)
≤ O ( R ) , we have by (Sridharan et al., 2008, Theorem 1) that with probability 1 − γ
[Equation]
Then since by deﬁnition ˆ w minimizes L S ( w ) we have that L S ( ˆ w ) ≤ L S ( w ∗ ) , which substituted into the previous equation completes the proof.
Proof of Theorem 4.2.
Applying Lemma C.4 in the compressed domain yields
[Equation]
where w ∗ A minimizes L D .
By deﬁnition of w ∗ A , L D ( w ∗ A ) ≤ L D ( A ˆ w ) , so together with Lemma C. and the previous inequality we have
[Equation]
We now apply Lemma C.4 in the sparse domain to get
[Equation]
where w ∗ minimizes L D .
By deﬁnition of w ∗ , L D ( w ∗ ) ≤ L D ( w 0 ) =
(cid:96) D ( w 0 )
+ 12 C (cid:107)
[Equation]
22 , so by the previous inequality we have
[Equation]
Substituting the C that minimizes the r.h.s.
of this inequality completes the proof.
D P ROOF OF L EMMA 4.
We assume the setting described in Lemma 4.1, where we are concerned with the RIP condition of the matrix A ( n ) when multiplying vectors x ∈ X ( n ) T , the set of BonC vectors for documents of length at most T .
This matrix can be written as
[Equation]
 where A p is the d × V p matrix whose columns are the DisC embeddings of all p -grams in the vocabulary (and thus A (1) =
A 1 = A , the matrix of the original word embeddings).
Note that from (1) any x ∈ X ( n ) T can be written as x = [ x 1 , . . .
, x n ] , where x p is a T -sparse vector whose entries correspond to p -grams.
Thus we also have A ( n ) x
= [ A 1 x 1 , . . .
, A n x n ] .
Lemma D.1.
If A p is (2 k, ε ) -RIP w.p. 1 − γ ∀ p ∈ [ n ] then A ( n ) is (cid:16) ∆ X ( n ) k , ε (cid:17) -RIP w.p. at least 1 − nγ .
Proof.
By union bound we have that A p is (2 k, ε ) -RIP
∀ p ∈
[ n ] with probability at least 1 − nγ .
Thus by Deﬁnition 4.1 we have w.p. 1 − nγ that ∀ x ∈ ∆ X ( n ) k (cid:107) A ( n ) x (cid:107) 22 = n (cid:88) p
= (cid:107)
A p x p (cid:107) 22 ≤ n (cid:88) p
= (1 + ε ) 2 (cid:107) x p (cid:107)
= (1 + ε ) 2 (cid:107) x (cid:107)
Similarly, (cid:107) A ( n ) x (cid:107) 22 ≥ (1 − ε ) 2 (cid:107) x (cid:107) 22 .
From Deﬁnition 4.1, taking the square root of both sides of both inequalities completes the proof.
Deﬁnition D.1 (Foucart & Rauhut, 2013) .
Let D be a distribution over a subset S ⊂ R n .
Then the set Φ = { φ 1 , . .
. , φ N } of functions φ i : S (cid:55)→ R is a bounded orthonormal system (BOS) with constant B if we have E D ( φ i φ j ) =
i = j ∀ i, j and sup s ∈ S | φ i ( s ) | ≤ B ∀ i .
Note that by deﬁnition
B ≥ 1 .
Theorem D.1 (Foucart & Rauhut, 2013) .
If d = ˜Ω (cid:16) B 2 k ε 2 log Nγ (cid:17) for ( ε, γ ) ∈ (0 , 1) and √ dA is a d × N matrix associated with a BOS with constant B then A is ( k, ε ) -RIP w.p. 1 − γ .
Lemma D.2.
If d = ˜Ω (cid:16) Tε 2 log V p γ (cid:17) and the word embeddings are drawn i.i.d.
from U d {± 1 / √ d } then for any p ∈ [ n ] the matrix
A p ∈ R
d × V p of DisC embeddings is ( T, ε ) -RIP w.p. 1 − γ .
Proof.
Note that by Theorem D.1 it sufﬁces to show that √ dA p is a random sampling matrix associated with a BOS with constant B = 1 .
Let D
=
U V {± 1 } be the uniform distribution over
V i.i.d.
Rademacher random variables indexed by words in the vocabulary.
Then by deﬁnition the matrix
A p ∈ R
d × V p can be constructed by drawing random variables x (1) , . . .
, x ( d ) i.i.d.
from D and assigning to the ij th entry of √ dA p corresponding to the p -gram g = { g 1 , . . .
, g p } the value φ j (cid:0) x
( i ) (cid:1) =
(cid:81) pt
=1 x ( i ) g t , where each function φ j : {± 1 } V (cid:55)→ R is uniquely associated to its p -gram.
It remains to be shown that this set of functions is a BOS with constant B = 1 .
For any two p -grams g, g (cid:48) and their functions φ i , φ j we have E D ( φ i φ j ) =
E x ∼D (cid:0)(cid:81) pt =1 x g t x g (cid:48) t (cid:1) , which will be 1 iff each word in g ∪ g (cid:48) occurs an even number of times in the product and 0 otherwise.
Because all p -grams are uniquely deﬁned under any permutation of its words (i.e. we are in fact using p -cooccurrences) and we have assumed that no p -gram contains a word more than once, each word occurs an even number of times in the product iff g = g (cid:48)
⇐⇒ i
= j .
Furthermore we have that | φ
i ( x ) | ≤ 1 ∀ x ∈ {± 1 } V ∀ i by construction.
Thus according to Deﬁnition D.1 the set of functions { φ 1 , . . .
, φ V p } associated to the p -grams in the vocabulary is a BOS with constant B = 1 .
Proof of Lemma 4.1.
Since d = ˜Ω (cid:16) Tε 2 log nV max n γ (cid:17) , Lemma D.2 implies that A p is (2 T, ε ) -RIP w.p.
1 − γn ∀ p ∈ [ n ] .
Applying Lemma
D.1 yields the result.
E S PARSE R ECOVERY WITH P RETRAINED E MBEDDINGS E. C HECKING THE S UPPORTING H YPERPLANE P ROPERTY
Recall that a matrix R d × N satisﬁes S -SHP for S ⊂ [ N ] if there is a hyperplane containing the set of all columns of A indexed by S and the set of all other columns together with the origin are on one side of it.
Checking S -SHP allows us to know whether all nonnegative signals with index support S will be recovered by BP+ without actually running the optimization on any one of them. The property can be checked by solving a convex problem of the form [Equation] subject to ˜
[Equation]
where ˜ A = (cid:18)
[Equation]
1 TN
[Equation]
and ε > 0 , p ≥
The constraint enforces the hyperplane containing all support embeddings, while the optimal objective value is zero iff SHP is satisﬁed (this follows from the fact that scaling h does not affect the constraint so if the optimal objective is zero for any ε > 0 it is zero for all ε > 0 ).
The problem can be solved via standard equality-constrained convex optimization.
We set ε = 1 and p = 3 (to get a C 2 objective) and adapt the second-order method from Boyd & Vandenberghe (2004, Chapter 10).
Our implementation can be found at https://github.com/NLPrinceton/sparse_recovery .
In Figures 6 and 7 we show that the BoWs of natural language documents are indeed more likely to satisfy SHP (or equivalently be perfectly recoverable by BP+) under a word embedding design matrix than a random design matrix, verifying the discussion in Section 5.2. Since the feasible set of BP contains that of BP+, it can perform no better than BP+ for nonnegative signal recovery, as can be seen in the ﬁgures (since we computed SHP, and not BP+, this also serves as empirical veriﬁcation of
Corollary 5.1)
We also compare BP/BP+ recovery to the greedy OMP/OMP+ methods (OMP+ is a modiﬁcation of OMP where at each iteration any index assigned a negative value is removed from the set of selected atoms and the orthogonal projection is recomputed).
Interestingly, while OMP+ recovers the correct signal from an SN design almost as often as BP/BP+, it performs quite poorly for GloVe, indicating that these embeddings may have quite different compressed sensing properties despite having similar training objectives.
[Figure]
Recovery of [Figure] Recovery of IMDB [Figure]
We show in Figure 8 that the surprising effectiveness of word embeddings as linear measurement vectors for BoW signals holds for other embedding objectives and corpora as well. Speciﬁcally, we see that widely used embeddings, when normalized, match the efﬁciency of random vectors for retrieving SST BoW and are more efﬁcient when retrieving IMDB BoW. Interestingly, SN vectors are most efﬁcient and are also the only embeddings for normalizing is not needed for good performance.
E. A M ODEL -B ASED T HEORETICAL A PPROACH
In Section 5.2 we gave some intuition for why pretrained word embeddings are efﬁcient sensing vectors for natural language BoW by examining a geometric characterization of local equivalence due to Donoho & Tanner (2005) in light of the usual similarity properties of word embeddings.
However, this analysis does not provide a rigorous theory for our empirical results.
In this section we brieﬂy discuss a model-based justiﬁcation that may lead to a stronger understanding.
We need a model relating BoW generation to the word embeddings trained over words co-occurring in the same BoW. As a starting point consider the model of Arora et al.
(2016), in which a corpus is generated by a random walk c t over the surface of a ball in R d ; at each t a word w is emitted w.p.
[Equation]
Minimizing the SN objective approximately maximizes corpus likelihood under this model.
Thus in an approximate sense a document of length T is generated by setting a context vector c and emitting T words via (18) with c t = c .
This model is a convenient one for analysis due its simplicity and invariance to word order as well as the fact that the approximate maximum likelihood document vector is the sum of the embeddings of words in the document.
Building upon the intuition established following Corollary 5.1 one can argue that, if we have the true latent SN vectors, then embeddings of words in the same document (i.e. emitted by the same context vector) will be close to each other and thus easy to separate from the embeddings of other words in the vocabulary.
However, we ﬁnd empirically that not all of the T words closest to the sum of the word embeddings (i.e. the context vector) are the ones emitted; indeed individual word vectors in a document may have small, even negative inner product with the context vector and still be recovered via BP.
Thus any further theoretical argument must also be able to handle the recovery of lower probability words whose vectors are further away from the context vector than those of words that do not appear in the document.
We thus leave to future work the challenge of explaining why embeddings resulting from this (or another) model provide such efﬁcient sensing matrices for natural language BoW.