Paper:
The success of neural network methods for computing word embeddings has motivated methods for generating semantic embeddings of longer pieces of text, such as sentences and paragraphs. Surprisingly, Wieting et al (ICLR’16) showed that such complicated methods are outperformed, especially in out-of-domain (transfer learning) settings, by simpler methods involving mild retraining of word embeddings and basic linear regression. The method of Wieting et al. requires retraining with a substantial labeled dataset such as Paraphrase Database (Ganitkevitch et al., 2013). The current paper goes further, showing that the following completely unsupervised sentence embedding is a formidable baseline: Use word embeddings computed using one of the popular methods on unlabeled corpus like Wikipedia, represent the sentence by a weighted average of the word vectors, and then modify them a bit using PCA/SVD. This weighting improves performance by about 10% to 30% in textual similarity tasks, and beats sophisticated supervised methods including RNN’s and LSTM’s. It even improves Wieting et al.’s embeddings. This simple method should be used as the baseline to beat in future, especially when labeled training data is scarce or nonexistent. The paper also gives a theoretical explanation of the success of the above unsupervised method using a latent variable generative model for sentences, which is a simple extension of the model in Arora et al. (TACL’16) with new “smoothing” terms that allow for words occurring out of context, as well as high probabilities for words like and, not in all contexts.
1 INTRODUCTION
Word embeddings computed using diverse methods are basic building blocks for Natural Language Processing (NLP) and Information Retrieval (IR). They capture the similarities between words (e.g., (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a; Pennington et al., 2014)). Recent work has tried to compute embeddings that capture the semantics of word sequences (phrases, sentences, and paragraphs), with methods ranging from simple additional composition of the word vectors to sophisticated architectures such as convolutional neural networks and recurrent neural networks (e.g., (Iyyer et al., 2015; Le & Mikolov, 2014; Kiros et al., 2015; Socher et al., 2011; Blunsom et al., 2014; Tai et al., 2015; Wang et al., 2016)). Recently, (Wieting et al., 2016) learned general-purpose, paraphrastic sentence embeddings by starting with standard word embeddings and modifying them based on supervision from the Paraphrase pairs dataset (PPDB), and constructing sentence embeddings by training a simple word averaging model. This simple method leads to better performance on textual similarity tasks than a wide variety of methods and serves as a good initialization for textual classification tasks. However, supervision from the paraphrase dataset seems crucial, since they report that simple average of the initial word embeddings does not work very well.
Here we give a new sentence embedding method that is embarrassingly simple: just compute the weighted average of the word vectors in the sentence and then remove the projections of the average vectors on their first principal component (“common component removal”). Here the weight of a word w is a/(a+ p(w)) with a being a parameter and p(w) the (estimated) word frequency; we call
this smooth inverse frequency (SIF). This method achieves significantly better performance than the unweighted average on a variety of textual similarity tasks, and on most of these tasks even beats some sophisticated supervised methods tested in (Wieting et al., 2016), including some RNN and LSTM models. The method is well-suited for domain adaptation settings, i.e., word vectors trained on various kinds of corpora are used for computing the sentence embeddings in different testbeds. It is also fairly robust to the weighting scheme: using the word frequencies estimated from different corpora does not harm the performance; a wide range of the parameters a can achieve close-to-best results, and an even wider range can achieve significant improvement over unweighted average.
Of course, this SIF reweighting is highly reminiscent of TF-IDF reweighting from information retrieval (Sparck Jones, 1972; Robertson, 2004) if one treats a “sentence” as a “document” and make the reasonable assumption that the sentence doesn’t typically contain repeated words. Such reweightings (or related ideas like removing frequent words from the vocabulary) are a good rule of thumb but has not had theoretical justification in a word embedding setting.
The current paper provides a theoretical justification for the reweighting using a generative model for sentences, which is a simple modification for the Random Walk on Discourses model for generating text in (Arora et al., 2016). In that paper, it was noted that the model theoretically implies a sentence embedding, namely, simple average of embeddings of all the words in it.
We modify this theoretical model, motivated by the empirical observation that most word embedding methods, since they seek to capture word cooccurence probabilities using vector inner product, end up giving large vectors to frequent words, as well as giving unnecessarily large inner products to word pairs, simply to fit the empirical observation that words sometimes occur out of context in documents. These anomalies cause the average of word vectors to have huge components along semantically meaningless directions. Our modification to the generative model of (Arora et al., 2016) allows “smoothing” terms, and then a max likelihood calculation leads to our SIF reweighting.
Interestingly, this theoretically derived SIF does better (by a few percent points) than traditional TFIDF in our setting. The method also improves the sentence embeddings of Wieting et al., as seen in Table 1. Finally, we discovered that —contrary to widespread belief—Word2Vec(CBOW) also does not use simple average of word vectors in the model, as misleadingly suggested by the usual expression Pr[w|w1, w2, . . . , w5] ∝ exp(vw · ( 15 ∑ i vwi)). A dig into the implementation shows it implicitly uses a weighted average of word vectors —again, different from TF-IDF— and this weighting turns out to be quite similar in effect to ours. (See Section 3.1.)
2 RELATED WORK
Word embeddings. Word embedding methods represent words as continuous vectors in a low dimensional space which capture lexical and semantic properties of words. They can be obtained from the internal representations from neural network models of text (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a) or by low rank approximation of co-occurrence statistics (Deerwester et al., 1990; Pennington et al., 2014). The two approaches are known to be closely related (Levy & Goldberg, 2014; Hashimoto et al., 2016; Arora et al., 2016).
Our work is most directly related work to (Arora et al., 2016), which proposed a random walk model for generating words in the documents. Our sentence vector can be seen as approximate inference of the latent variables in their generative model.
Phrase/Sentence/Paragraph embeddings. Previous works have computed phrase or sentence embeddings by composing word embeddings using operations on vectors and matrices e.g., (Mitchell & Lapata, 2008; 2010; Blacoe & Lapata, 2012). They found that coordinate-wise multiplication of the vectors performed very well among the binary operations studied. Unweighted averaging is also found to do well in representing short phrases (Mikolov et al., 2013a). Another approach is recursive neural networks (RNNs) defined on the parse tree, trained with supervision (Socher et al., 2011) or without (Socher et al., 2014). Simple RNNs can be viewed as a special case where the parse tree is replaced by a simple linear chain. For example, the skip-gram model (Mikolov et al., 2013b) is extended to incorporate a latent vector for the sequence, or to treat the sequences rather than the word as basic units. In (Le & Mikolov, 2014) each paragraph was assumed to have a latent paragraph vector, which influences the distribution of the words in the paragraph. Skip-thought
of (Kiros et al., 2015) tries to reconstruct the surrounding sentences from surrounded one and treats the hidden parameters as their vector representations. RNNs using long short-term memory (LSTM) capture long-distance dependency and have also been used for modeling sentences (Tai et al., 2015). Other neural network structures include convolution neural networks, such as (Blunsom et al., 2014) that uses a dynamic pooling to handle input sentences of varying length and do well in sentiment prediction and classification tasks.
The directed inspiration for our work is (Wieting et al., 2016) which learned paraphrastic sentence embeddings by using simple word averaging and also updating standard word embeddings based on supervision from paraphrase pairs; the supervision being used for both initialization and training.
3 A SIMPLE METHOD FOR SENTENCE EMBEDDING
We briefly recall the latent variable generative model for text in (Arora et al., 2016). The model treats corpus generation as a dynamic process, where the t-th word is produced at step t. The process is driven by the random walk of a discourse vector ct ∈ <d. Each word w in the vocabulary has a vector in <d as well; these are latent variables of the model. The discourse vector represents “what is being talked about.” The inner product between the discourse vector ct and the (time-invariant) word vector vw for word w captures the correlations between the discourse and the word. The probability of observing a word w at time t is given by a log-linear word production model from Mnih and Hinton:
Pr[w emitted at time t | ct] ∝ exp (〈ct, vw〉) . (1)
The discourse vector ct does a slow random walk (meaning that ct+1 is obtained from ct by adding a small random displacement vector), so that nearby words are generated under similar discourses. It was shown in (Arora et al., 2016) that under some reasonable assumptions this model generates behavior –in terms of word-word cooccurence probabilities—that fits empirical works like word2vec and Glove. The random walk model can be relaxed to allow occasional big jumps in ct, since a simple calculation shows that they have negligible effect on cooccurrence probabilities of words. The word vectors computed using this model are reported to be similar to those from Glove and word2vec(CBOW).
Our improved Random Walk model. Clearly, it is tempting to define the sentence embedding as follows: given a sentence s, do a MAP estimate of the discourse vectors that govern this sentence. We note that we assume the discourse vector ct doesn’t change much while the words in the sentence were emitted, and thus we can replace for simplicity all the ct’s in the sentence s by a single discourse vector cs. In the paper (Arora et al., 2016), it was shown that the MAP estimate of cs is —up to multiplication by scalar—the average of the embeddings of the words in the sentence.
In this paper, towards more realistic modeling, we change the model (1) as follows. This model has two types of “smoothing term”, which are meant to account for the fact that some words occur out of context, and that some frequent words (presumably “the”, “and ” etc.) appear often regardless of the discourse. We first introduce an additive term αp(w) in the log-linear model, where p(w) is the unigram probability (in the entire corpus) of word and α is a scalar. This allows words to occur even if their vectors have very low inner products with cs. Secondly, we introduce a common discourse vector c0 ∈ <d which serves as a correction term for the most frequent discourse that is often related to syntax. (Other possible correction is left to future work.) It boosts the co-occurrence probability of words that have a high component along c0.
Concretely, given the discourse vector cs, the probability of a word w is emitted in the sentence s is modeled by,
Pr[w emitted in sentence s | cs] = αp(w) + (1− α) exp (〈c̃s, vw〉)
Zc̃s , (2)
where c̃s = βc0 + (1− β)cs, c0 ⊥ cs where α and β are scalar hyperparameters, and Zc̃s = ∑ w∈V exp (〈c̃s, vw〉) is the normalizing constant (the partition function). We see that the model allows a word w unrelated to the discourse cs to be omitted for two reasons: a) by chance from the term αp(w); b) if w is correlated with the common discourse vector c0.
Algorithm 1 Sentence Embedding Input: Word embeddings {vw : w ∈ V}, a set of sentences S, parameter a and estimated probabil-
ities {p(w) : w ∈ V} of the words. Output: Sentence embeddings {vs : s ∈ S}
1: for all sentence s in S do 2: vs ← 1|s| ∑ w∈s a a+p(w)vw 3: end for 4: Compute the first principal component u of {vs : s ∈ S} 5: for all sentence s in S do 6: vs ← vs − uu>vs 7: end for
Computing the sentence embedding. The word embeddings yielded by our model are actually the same 1 The sentence embedding will be defined as the max likelihood estimate for the vector cs that generated it. ( In this case MLE is the same as MAP since the prior is uniform.) We borrow the key modeling assumption of (Arora et al., 2016), namely that the word vw’s are roughly uniformly dispersed, which implies that the partition function Zc is roughly the same in all directions. So assume that Zc̃s is roughly the same, say Z for all c̃s. By the model (2) the likelihood for the sentence is
p[s | cs] = ∏ w∈s p(w | cs) = ∏ w∈s [ αp(w) + (1− α)exp (〈vw, c̃s〉) Z ] .
Let
fw(c̃s) = log
[ αp(w) + (1− α)exp (〈vw, c̃s〉)
Z ] denote the log likelihood of sentence s. Then, by simple calculus we have,
∇fw(c̃s) = 1 αp(w) + (1− α) exp (〈vw, c̃s〉) /Z 1− α Z exp (〈vw, c̃s〉) vw.
Then by Taylor expansion, we have,
fw(c̃s) ≈ fw(0) +∇fw(0)>c̃s
= constant + (1− α)/(αZ)
p(w) + (1− α)/(αZ) 〈vw, c̃s〉 .
Therefore, the maximum likelihood estimator for c̃s on the unit sphere (ignoring normalization) is approximately,2
arg max ∑ w∈s fw(c̃s) ∝ ∑ w∈s
a
p(w) + a vw, where a = 1− α αZ . (3)
That is, the MLE is approximately a weighted average of the vectors of the words in the sentence. Note that for more frequent words w, the weight a/(p(w) + a) is smaller, so this naturally leads to a down weighting of the frequent words.
To estimate cs, we estimate the direction c0 by computing the first principal component of c̃s’s for a set of sentences. In other words, the final sentence embedding is obtained by subtracting the projection of c̃s’s to their first principal component. This is summarized in Algorithm 1.
3.1 CONNECTION TO SUBSAMPLING PROBABILITIES IN WORD2VEC
Word2vec (Mikolov et al., 2013b) uses a sub-sampling technique which downsamples word w with probability proportional to 1/ √ p(w) where p(w) is the marginal probability of the word w. This
1We empirically discovered the significant common component c0 in word vectors built by existing methods, which inspired us to propose our theoretical model of this paper.
2Note that maxc:‖c‖=1 C + 〈c, g〉 = g/‖g‖ for any constant C.
heuristic not only speeds up the training but also learns more regular word representations. Here we explain that this corresponds to an implicit reweighting of the word vectors in the model and therefore the statistical benefit should be of no surprise.
Recall the vanilla CBOW model of word2vec:
Pr[wt | wt−1, . . . , wt−5] ∝ exp (〈v̄t, vw〉) , where v̄t = 1
5 5∑ i=1 vwt−i . (4)
It can be shown that the loss (MLE) for the single word vector vw (from this occurrence) can be abstractly written in the form,
g(vw) = γ(〈v̄t, vw〉) + negative sampling terms ,
where γ(x) = log(1/(1 + e−x)) is the logistic function. Therefore, the gradient of g(vw) is
∇g(vw) = γ′(〈v̄t, vw〉)v̄t = α(vwt−5 + vwt−4 + vwt−3 + vwt−2 + vwt−1) , (5)
where α is a scalar. That is, without the sub-sampling trick, the update direction is the average of the word vectors in the window.
The sub-sampling trick in (Mikolov et al., 2013b) randomly selects the summands in equation (5) to “estimate” the gradient. Specifically, the sampled update direction is
∇̃g(vw) = α(J5vwt−5 + J4vwt−4 + J3vwt−3 + J2vwt−2 + J1vwt−1) (6)
where Jk’s are Bernoulli random variables with Pr [Jk = 1] = q(wt−k) , min { 1, √ 10−5
p(wt−k)
} .
However, we note that ∇̃g(vw) is (very) biased estimator! We have that the expectation of ∇̃g(vw) is a weighted sum of the word vectors,
E [ ∇̃g(vw) ] = α(q(wt−5)vwt−5 + q(wt−4)vwt−4 + q(wt−3)vwt−3 + q(wt−2)vwt−2 + q(wt−1)vwt−1) .
In fact, the expectation E[∇̃g(vw)] corresponds to the gradient of a modified word2vec model with the average v̄t (in equation (4)) being replaced by the weighted average ∑5 k=1 q(wt−k)vwt−k . Such a weighted model can also share the same form of what we derive from our random walk model as in equation (3). Moreover, the weighting q(wi) closely tracks our weighting scheme a/(a + p(w)) when using parameter a = 10−4; see Figure 1 for an illustration. Therefore, the expected gradient here is approximately the estimated discourse vector in our model! Thus, word2vec with sub-sampling gradient heuristic corresponds to a stochastic gradient update method for using our weighting scheme.
4 EXPERIMENTS
4.1 TEXTUAL SIMILARITY TASKS
Datasets. We test our methods on the 22 textual similarity datasets including all the datasets from SemEval semantic textual similarity (STS) tasks (2012-2015) (Agirre et al., 2012; 2013; 2014; Agirrea et al., 2015), and the SemEval 2015 Twitter task (Xu et al., 2015) and the SemEval 2014 Semantic Relatedness task (Marelli et al., 2014). The objective of these tasks is to predict the similarity between two given sentences. The evaluation criterion is the Pearson’s coefficient between the predicted scores and the ground-truth scores.
Experimental settings. We will compare our method with the following:
1. Unsupervised: ST, avg-GloVe, tfidf-GloVe. ST denotes the skip-thought vectors (Kiros et al., 2015), avg-GloVe denotes the unweighted average of the GloVe vectors (Pennington et al., 2014),3 and tfidf-GloVe denotes the weighted average of GloVe vectors using TF-IDF weights.
2. Semi-supervised: avg-PSL. This method uses the unweighted average of the PARAGRAMSL999 (PSL) word vectors from (Wieting et al., 2015). The word vectors are trained using labeled data, but the sentence embedding are computed by unweighted average without training.
3. Supervised: PP, PP-proj., DAN, RNN, iRNN, LSTM (o.g.), LSTM (no). All these methods are initialized with PSL word vectors and then trained on the PPDB dataset. PP and PPproj. are proposed in (Wieting et al., 2016). The first is an average of the word vectors, and the second additionally adds a linear projection. The word vectors are updated during the training. DAN denotes the deep averaging network of (Iyyer et al., 2015). RNN denotes the classical recurrent neural network, and iRNN denotes a variant with the activation being the identity, and the weight matrices initialized to identity. The LSTM is the version from (Gers et al., 2002), either with output gates (denoted as LSTM(o.g.)) or without (denoted as LSTM (no)).
Our method can be applied to any types of word embeddings. So we denote the sentence embeddings obtained by applying our method to word embeddings method “XXX” as “XXX+WR”.4 To get a completely unsupervised method, we apply it to the GloVe vectors, denoted as GloVe+WR. The weighting parameter a is fixed to 10−3, and the word frequencies p(w) are estimated from the
3We used the vectors that are publicly available at http://nlp.stanford.edu/projects/glove/. They are 300- dimensional vectors that were trained on the 840 billion token Common Crawl corpus.
4“W” stands for the smooth inverse frequency weighting scheme, and “R” stands for removing the common components.
commoncrawl dataset.5 This is denoted by GloVe+WR in Table 1. We also apply our method on the PSL vectors, denoted as PSL+WR, which is a semi-supervised method.
Results. The results are reported in Table 1. Each year there are 4 to 6 STS tasks. For clarity, we only report the average result for the STS tasks each year; the detailed results are in the appendix.
The unsupervised method GloVe+WR improves upon avg-GloVe significantly by 10% to 30%, and beats the baselines by large margins. It achieves better performance than LSTM and RNN and is comparable to DAN, even though the later three use supervision. This demonstrates the power of this simple method: it can be even stronger than highly-tuned supervisedly trained sophisticated models. Using TF-IDF weighting scheme also improves over the unweighted average, but not as much as our method.
The semi-supervised method PSL+WR achieves the best results for four out of the six tasks and is comparable to the best in the rest of two tasks. Overall, it outperforms the avg-PSL baseline and all the supervised models initialized with the same PSL vectors. This demonstrates the advantage of our method over the training for those models.
We also note that the top singular vectors c0 of the datasets seem to roughly correspond to the syntactic information or common words. For example, closest words (by cosine similarity) to c0 in the SICK dataset are “just”, “when”, “even”, “one”, “up”, “little”, “way”, “there”, “while”, and “but.”
Finally, in the appendix, we showed that our two ideas all contribute to the improvement: for GloVe vectors, using smooth inverse frequency weighting alone improves over unweighted average by about 5%, using common component removal alone improves by 10%, and using both improves by 13%.
4.1.1 EFFECT OF WEIGHTING PARAMETER ON PERFORMANCE
We study the sensitivity of our method to the weighting parameter a, the method for computing word vectors, and the estimated word probabilities p(w). First, we test the performance of three
5It is possible to tune the parameter a to get better results. The effect of a and the corpus for estimating word frequencies are studied in Section 4.1.1.
types of word vectors (PSL, GloVe, and SN) on the STS 2012 tasks. SN vectors are trained on the enwiki dataset (Wikimedia, 2012) using the method in (Arora et al., 2016), while PSL and GloVe vectors are those used in Table 1. We enumerate a ∈ {10−i, 3 × 10−i : 1 ≤ i ≤ 5} and use the p(w) estimated on the enwiki dataset. Figure 2a shows that for all three kinds of word vectors, a wide range of a leads to significantly improved performance over the unweighted average. Best performance occurs from a = 10−3 to a = 10−4.
Next, we fix a = 10−3 and use four very different datasets to estimate p(w): enwiki (wikipedia, 3 billion tokens), poliblogs (Yano et al., 2009) (political blogs, 5 million), commoncrawl (Buck et al., 2014) (Internet crawl, 800 billion), text8 (Mahoney, 2008) (wiki subset, 1 million). Figure 2b shows performance is almost the same for all four settings.
The fact that our method can be applied on different types of word vectors trained on different corpora also suggests it should be useful across different domains. This is especially important for unsupervised methods, since the unlabeled data available may be collected in a different domain from the target application.
4.2 SUPERVISED TASKS
The sentence embeddings obtained by our method can be used as features for downstream supervised tasks. We consider three tasks: the SICK similarity task, the SICK entailment task, and the Stanford Sentiment Treebank (SST) binary classification task (Socher et al., 2013). To highlight the representation power of the sentence embeddings learned unsupervisedly, we fix the embeddings and only learn the classifier. Setup of supervised tasks mostly follow (Wieting et al., 2016) to allow fair comparison, i.e., the classifier a linear projection followed by the classifier in (Kiros et al., 2015). The linear projection maps the sentence embeddings into 2400 dimension (the same as the skip-thought vectors), and is learned during the training. We compare our method to PP, DAN, RNN, and LSTM, which are the methods used in Section 4.1. We also compare to the skip-thought vectors (with improved training in (Lei Ba et al., 2016)).
Results. Our method gets better or comparable performance compared to the competitors. It gets the best results for two of the tasks. This demonstrates the power of our simple method. We emphasize that our embeddings are unsupervisedly learned, while DAN, RNN, LSTM are trained with supervision. Furthermore, skip-thought vectors are much higher dimensional than ours (though projected into higher dimension, the original 300 dimensional embeddings contain all the information).
The advantage is not as significant as in the textual similarity tasks. This is possibly because similarity tasks rely directly upon cosine similarity, which favors our method’s approach of removing the common components (which can be viewed as a form of denoising), while in supervised tasks, with the cost of some label information, the classifier can pick out the useful components and ignore the common ones.
Finally, we speculate that our method doesn’t outperform RNN’s and LSTM’s for sentiment tasks because (a) the word vectors —and more generally the distributional hypothesis of meaning —has known limitations for capturing sentiment due to the “antonym problem”, (b) also in our weighted average scheme, words like “not” that may be important for sentiment analysis are downweighted a lot. To address (a), there is existing work on learning better word embeddings for sentiment analysis (e.g., (Maas et al., 2011)). To address (b), it is possible to design weighting scheme (or learn weights) for this specific task.
4.3 THE EFFECT OF THE ORDER OF WORDS IN SENTENCES
A interesting feature of our method is that it ignores the word order. This is in contrast to that RNN’s and LSTM’s can potentially take advantage of the word order. The fact that our method achieves better or comparable performance on these benchmarks raise the following question: is word order not important in these benchmarks? We conducted an experiment suggesting that word order does play some role.
We trained and tested RNN/LSTM on the supervised tasks where the words in each sentence are randomly shuffled, and the results are reported in Table 3.6 It can be observed that the performance drops noticeably. Thus our method —which ignores word order—must be much better at exploiting the semantics than RNN’s and LSTM’s. An interesting future direction is to explore if some ensemble idea can combine the advantages of both approaches.
5 CONCLUSIONS
This work provided a simple approach to sentence embedding, based on the discourse vectors in the random walk model for generating text (Arora et al., 2016). It is simple and unsupervised, but achieves significantly better performance than baselines on various textual similarity tasks, and can even beat sophisticated supervised methods such as some RNN and LSTM models. The sentence embeddings obtained can be used as features in downstream supervised tasks, which also leads to better or comparable results compared to the sophisticated methods.
6 ACKNOWLEDGEMENTS
We thank the reviewers for insightful comments. We also thank the authors of (Wieting et al., 2016; Bowman et al., 2015) for sharing their code or the preprocessed datasets.
This work was supported in part by NSF grants CCF-1527371, DMS-1317308, Simons Investigator Award, Simons Collaboration Grant, and ONRN00014- 16-1-2329. Tengyu Ma was supported in addition by Simons Award in Theoretical Computer Science and IBM PhD Fellowship.
A DETAILS OF EXPERIMENTAL SETTING
A.1 UNSUPERVISED TASK: TEXTUAL SIMILARITY
The competitors. We give a brief overview of the competitors. RNN is the classical recurrent neural network:
ht = f(WxW xt w +Whht−1 + b)
where f is the activation, Wx,Wh and b are parameters, and xt is the t-th token in the sentence. The sentence embedding of RNN is just the hidden vector of the last token. iRNN is a special RNN with the activation being the identity, the weight matrices initialized to identity, and b initialized to zero. LSTM (Hochreiter & Schmidhuber, 1997) is a recurrent neural network architecture designed to capture long-distance dependencies. Here, the version from (Gers et al., 2002) is used.
The supervised methods are initialized with PARAGRAM-SL999 (PSL) vectors, and trained using the approach of (Wieting et al., 2016) on the XL section of the PPDB data (Pavlick et al., 2015) which contains about 3 million unique phrase pairs. After training, the final models can be used to generate sentence embeddings on the test data. For hyperparameter tuning they used 100k examples sampled from PPDB XXL and trained for 5 epochs. Then after finding the hyperparameters that maximize Spearmans coefficients on the Pavlick et al. PPDB task, they are trained on the entire XL section of PPDB for 10 epochs. See (Wieting et al., 2016) and related papers for more details about these methods.
The tfidf-GloVe method is a weighted average of the GloVe vectors, where the weights are defined by the TF-IDF scheme. More precisely, the embedding of a sentence s is
vs = 1 |s| ∑ w∈s IDFwvw (7)
where IDFw is the inverse document frequency of w, and |s| denotes the number of words in the sentence. Here, the TF part of the TF-IDF scheme is taken into account by the sum over w ∈ s. Furthermore, when computing IDFw, each sentence is viewed as a “document”:
IDFw := log 1 +N
1 +Nw
where N is the total number of sentences and Nw is the number of sentences containing w, and 1 is added to avoid division by 0. In the experiments, we use all the textual similarity datasets to compute IDFw.
Detailed experimental results. In the main body we present the average results for STS tasks by year. Each year there are actually 4 to 6 STS tasks, as shown in Table 4. Note that tasks with the same name in different years are actually different tasks. Here we provide the results for each tasks in Table 5. PSL+WR achieves the best results on 12 out of 22 tasks, PP and PP-proj. achieves on 3, tfidf-GloVe achieves on 2, and DAN, iRNN, and GloVe+WR achieves on 1. In general, our method improves the performance significantly compared to the unweighted average, though on rare cases such as MSRpar it can decrease the performance.
Effects of smooth inverse frequency and common component removal. There are two key ideas in our methods: smooth inverse frequency weighting (W) and common component removal (R). It is instructive to see their effects separately. Let GloVe+W denote the embeddings using only smooth inverse frequency, and GloVe+R denote that using only common component removal. Similarly define PSL+W and PSL+R. The results for these methods are shown in Table 6. When using GloVe vectors, W alone improves the performance of the unweighted average baseline by about 5%, R alone improves by 10%, W and R together improves by 13%. When using PSL vectors, W gets 10%, R gets 10%, W and R together gets 13%. In summary, both techniques are important for obtaining significant advantage over the unweighted average.
A.2 SUPERVISED TASKS
Setup of supervised tasks mostly follow (Wieting et al., 2016) to allow fair comparison: the sentence embeddings are fixed and fed into some classifier which are trained. For the SICK similarity task, given a pair of sentences with embeddings vL and vR, first do a linear projection:
hL = WpvL, hR = WpvR
where Wp is of size 300 × dp and is learned during training. dp = 2400 matches the dimension of the skip-thought vectors. Then hL and hR are used in the objective function from (Tai et al., 2015). Almost the same approach is used for the entailment task.For the sentiment task, the classifier has a fully-connected layer with a sigmoid activation followed by a softmax layer.
Recall that our method has two steps: smooth inverse frequency weighting and common component removal. For the experiments here, we do not perform the common component removal, since it can be absorbed into the projection step. For the weighted average, the hyperparameter a is enumerated in {10−i, 3 × 10−i : 2 ≤ i ≤ 3}. The other hyperparameters are enumerated as in (Wieting et al., 2016), and the same validation approach is used to select the final values.
A.3 ADDITIONAL SUPERVISED TASKS
Here we report the experimental results on two more datasets, comparing to known results on them.
SNLI. The first experiment is for the 3-class classification task on the SNLI dataset (Bowman et al., 2015). To compare to the results in (Bowman et al., 2015), we used their experimental setup. In particular, we applied our method to 300 dimensional GloVe vectors and used an additional tanh neural network layer to map these 300d embeddings into 300 dimensional space, then used the code provided by the authors of (Bowman et al., 2015), trained the classifier on our 100 dimensional sentence embedding for 120 passes over the data, using their default hyperparameters. The results are shown in Table 7. Our method indeed gets slightly better performance.
Our test accuracy is worse than those using more sophisticated models (e.g., using attention mechanism), which are typically 83% - 88%; see the website of the SNLI project7 for a summary. An interesting direction is to study whether our idea can be combined with these sophisticated models to get improved performance.
IMDB. The second experiment is the sentiment analysis task on the IMDB dataset, studied in (Wang & Manning, 2012). Since the intended application is semi-supervised or transfer learning, we also compared performance with fewer labeled examples.
7http://nlp.stanford.edu/projects/snli/
Our method gets worse performance on the full dataset, but its decrease in performance is better with less labeled examples, showing the benefit of using word embeddings. Note that our sentence embeddings are unsupervised, while that in the NB-SVM method takes advantage of the labels. Another comment is that sentiment analysis appears to be the best case for Bag-Of-Word methods, whereas it may be the worst case for word embedding methods (See Table 2) due to the well-known antonymy problem —distributional hypothesis fails for distinguishing “good” from “bad.”

Review:
This paper proposes a simple way to reweight the word embedding in the simple composition function for sentence representation. This paper also shows the connection between this new weighting scheme and some previous work.
Here are some comments on technical details:
- The word "discourse" is confusing. I am not sure whether the words "discourse" in "discourse vector c_s" and the one in "most frequent discourse" have the same meaning.
- Is there any justification about $c_0$ related to syntac?
- Not sure what thie line means: "In fact the new model was discovered by our detecting the common component c0 in existing embeddings." in section "Computing the sentence embedding"
- Is there any explanation about the results on sentiment in Table 2?