U NSUPERVISED D OCUMENT R EPRESENTATION USING P ARTITION W ORD -V ECTORS A VERAGING
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning effective document-level representation is essential in many important
NLP tasks such as document classiﬁcation, summarization, etc.
Recent research has shown that simple weighted averaging of word vectors is an effective way to represent sentences, often outperforming complicated seq2seq neural models in many tasks.
While it is desirable to use the same method to represent documents as well, unfortunately, the effectiveness is lost when representing long documents involving multiple sentences.
One reason for this degradation is due to the fact that a longer document is likely to contain words from many different themes (or topics), and hence creating a single vector while ignoring all the thematic structure is unlikely to yield an effective representation of the document.
This problem is less acute in single sentences and other short text fragments where presence of a single theme/topic is most likely.
To overcome this problem, in this paper we present P-SIF, a partitioned word averaging model to represent long documents.
P-SIF retains the simplicity of simple weighted word averaging, while taking a document’s thematic structure into account.
In particular, P-SIF learns topicspeciﬁc vectors from a document and ﬁnally concatenates them all to represent the overall document.
Through our experiments over multiple real-world datasets and tasks, we demonstrate P-SIF’s effectiveness compared to simple weighted averaging and many other state-of-the-art baselines.
We also show that P-SIF is particularly effective in representing long multi-sentence documents.
We will release P-SIF’s embedding source code and data-sets for reproducing results.
I NTRODUCTION
Distributed word embeddings such as word2vec (Mikolov et al., 2013b) have shown their success in representing words as latent continuous features in low-dimensional ﬁxed spaces, which can capture their semantic meaning.
These embeddings have shown impressive success for improving the performance of machine learning algorithms such as text classiﬁcation tasks.
The success of these word representations is recently formally explained using the random walk-based latent variable model by (Arora et al., 2016a).
Many approaches such as (Socher et al., 2013; Le & Mikolov, 2014; Ling et al., 2015; Liu et al., 2015a) are proposed which go beyond words to capture the semantic meaning of sentences.
These techniques either used simple composition of the word-vectors or sophisticated neural network architectures for sentence representation.
Recently, (Arora et al., 2017) proposed smooth inverse frequency of word vector averaging model to embed a sentence.
They further improved their embedding by removing the ﬁrst principal component of the weighted average vectors.
However, all these approaches are limited to capturing the meaning of a single sentence and represent the sentence in the same space as words, thus reducing their expressive power.
Generally, a longer text contain words from multiple different themes (or topics), and creating a single vector from simple averaging of word-vectors will disregard all the thematic structure.
Hence, these techniques are largely unable to capture the semantic meaning of larger pieces of text (phrases, sentences and paragraphs).
To address this problem, in this paper we present a novel document embedding method called partition SIF weighted averaging (P-SIF), which can efﬁciently embed documents with multiple sentences.
P-SIF learns topic-speciﬁc vectors from a document and ﬁnally concatenates them all to represent the overall document.
Thus, P-SIF retains the simplicity of simple weighted word averaging, while taking a document’s thematic structure into account.
(Mekala et al., 2017) recently proposed a clustering based word-vector averaging approach for embedding larger text documents.
However, they did not provide any justiﬁcation or experiments to explain the functionality of their embedding approach.
This paper provides an analytical explanation about the success of the mentioned representations and also achieves signiﬁcant improvements over previous embedding techniques on several natural language tasks.
Following are the characteristics of P-SIF.
• P-SIF is capable of embedding larger documents with multiple sentences, as it pays attention to the topic/thematic structure of the document.
• P-SIF is simple because it is based on weighted word vectors averaging rather than complicated tensor factorization or neural networks methods.
• P-SIF is an unsupervised method since it is obtained using pre-trained unsupervised word embeddings without using any label information.
• P-SIF outperforms many existing approaches on textual similarity, textual classiﬁcation and other supervised tasks.
The remaining part of the paper is organized as follows.
Section 2 discusses related work in document representations.
Section 3 provides motivation for the partition averaging through a qualitative example.
Section 4 describes our embedding algorithm.
Section 5 introduces and explains the background needed.
These sections are followed by experiments in Section 6 and the analysis with the discussion in Section 7.
R ELATED W ORK
Word embeddings.
There are mainly two methodologies proposed for unsupervised wordembedding representations, either by internal representations learned through neural network models of text (Bengio et al., 2003; Collobert & Weston, 2008), and Mikolov et al.
(2013b) or by low-rank approximation of co-occurrence statistics by (Levy & Goldberg, 2014; Hashimoto et al., 2016), and Arora et al.
(2016a).
(Levy & Goldberg, 2014) show that both techniques are equivalent.
Sentence embeddings.
Earlier work has computed sentence embedding by coordinate wise vector and matrix based compositional operation over word vectors, e.g., (Levy & Goldberg, 2014) use unweighted averaging of word vectors for representing short phrases, (Singh & Mukerjee, 2015) proposed tﬁdf-weighted averaging of word vectors to form document vectors, (Socher et al., 2013) proposed a recursive neural network (RNN) deﬁned over a parse tree and trained with supervision.
Next, (Le & Mikolov, 2014) proposed PV-DM and PV-DBOW model which treats each sentence as a shared global latent vector (or pseudo word).
Other approaches use seq2seq models such as
Recurrent Neural Networks (Mikolov et al., 2010) and Long Short Term Memory (Gers et al., 2002) which can handle long term dependency, hence, capturing the syntax structure.
Other neural network models include use of hierarchy and convolution neural networks, such as (Kim, 2014) and (Kalchbrenner et al., 2014).
(Wieting et al., 2015) learned paraphrastic sentence embedding by modifying word embeddings based on the supervision from the Paraphrase pairs dataset (PPDB).
Recently, a lot of work is harnessing topic modeling (Blei et al., 2003) along with word vectors to learn better word and sentence representations, e.g., TWE (Liu et al., 2015a), NTSG (Liu et al., 2015a), WTM (Fu et al., 2016), w2v-LDA (Nguyen et al., 2015),
TV+MeanWV (Li et al., 2016a),
LTSG (Law et al., 2017), Gaussian-LDA (Das et al., 2015), Topic2Vec
(Niu et al., 2015), Lda2vec (Moody, 2016), and MvTM (Li et al., 2016b).
Document embeddings.
(Kiros et al., 2015) proposed skip-thought document embedding vectors which transformed the idea of abstracting the distributional hypothesis from word level to sentence level.
(Wieting et al., 2016a) proposed a neural network model which optimizes the word embeddings based on the cosine similarity of the sentence embeddings.
(Gupta et al., 2016) proposed a method called Bag of Words Vector (BoWV), which employs a clustering based technique and tf-idf values to form a composite document vector.
They represented documents in higher dimensions by using hard clustering over word embeddings.
(Mekala et al., 2017) later extended the model to SCDV by using a fuzzy clustering technique and direct idf weighting of word vectors.
The learned representations tried to capture a global context of sentence, similar to an n-gram model.
Their method outperformed previous state of art on a variety of NLP tasks.
[Table]
M OTIVATION : A VERAGING VS P ARTITION A VERAGING
Let’s consider a corpus ( C ) with N documents with the corresponding most frequent words vocabulary ( V ).
Figure 1, represents the word-vectors space of V , where similar meaning words occur closer to each other.
We can apply sparse coding to partition the word vectors space to a ﬁve (total topics K = 5 ) topic vector space, as shown in ﬁgure 1.
These ﬁve topic vector spaces represent the ﬁve topics present in corpus ( C ).
Few words are polysemic and belong to multiple topics with some proportion (see Table 1).
In ﬁgure 1 we represent the topic number of the word in subscript.
Let’s consider a document d n : ‘Data journalists deliver data science news to general public.
They often take part in interpreting the data models.
In addition, they create graphical designs and interview the directors and CEO’s.
One can directly average the word vectors to represent the document ( (cid:126)v d n ), as in (SIF), shown in equation equation 1.
Here, + represents element-wise vector addition.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
director
[Equation]
interviewing
[Equation]
In representation equation 1 it can be seen that we are averaging word vectors which have a very different semantic meaning, e.g., words belonging to partition 1 such as graphical , design , and data are averaged with different semantic meaning words of partition 2 such as datascience , model , and data .
In addition, the document is represented in the same d dimensional space as word vectors.
Overall, averaging represents the documents as a single point in the vector space and does not consider thedifferent semantic topics or themes.
However, we do a weighted (topic proportion) average of words within a partition and concatenate the average word vectors across partitions to represent the document ( (cid:126)v d n ), as in (P-SIF), shown in equation equation 2.
Here, + represents a element-wise vector addition, × represents scalar-vector multiplication, and ⊕ represents concatenation.
[Equation]
In representation equation 2, the ﬁnal representation takes 5 different semantic topics into account.
Words belonging to different semantic topics are separated by concatenation ( ⊕ ) as they represent separate meanings, whereas words coming from same topics are simple averages since they represent the same meaning.
E.g., average of words belonging to partition 1 such as graphical , design , and data are concatenated to average of words of partition 2 such as data − science , model , and data .
The ﬁnal document vector (cid:126)v d n is represented in a higher 5 × d dimension vector space, thus having more representational power, where d is the dimension of word vectors.
From equation 2, it is clear that the ﬁnal representation takes the weight to which each word belongs to various topics into account as well, thus handling the multi-sense nature of words, e.g., words such as data belong to partition 1 with probability 0 .
3 and partition 2 with probability 0 .
Hence, partitioned averaging with topic weighting is essential for representing longer text documents.
Thus, P-SIF takes documents’ thematic structure into account while doing simple weighted word vectors averaging.
T HE P ROPOSED A LGORITHM : P-SIF
In this section, we present the proposed document embedding learning algorithm.
We introduce the formal notation needed for the discussion.
• C represents text corpus and V represents vocabulary of words in corpus.
• (cid:126)v w ∈ R
d
represents the word vector of word w , where d is dimension of word vector.
• p ( w ) represents the unigram probability of word w in the corpus.
• c 0 and (cid:126)v c 0 ∈ R d represent common context and it’s corresponding context vector.
The details to construct document embeddings are given in algorithm 1.
The feature formation algorithm can be divided into three major steps :
Dictionary Learning for Word Vectors:
Given word vectors v w ∈ R d , a sparsity parameter k , and an upper bound m , we ﬁnd a set of unit norm vectors (cid:126)A 1 , (cid:126)A 2 , . . .
, (cid:126)A m , such that (cid:126)v w = (cid:80) mj
=1 α ( w,j ) (cid:126)A j + (cid:126)η w , where at most k of the coefﬁcients α ( w, 1) , . . . , α ( w,m ) are nonzero (so-called sparsity constraint), and (cid:126)η w is a noise vector.
Sparse coding is usually solved for given m, k by using alternating minimization (Arora et al., 2016b) and (Aharon et al., 2006) to ﬁnd the (cid:126)A (cid:48)
i s that minimizes the following L 2 -reconstruction error : (cid:107) (cid:126)v w − (cid:80) mj
=1 α ( w,j ) (cid:126)A j (cid:107) .
Here, (cid:126)A 1 , . .
, (cid:126)A m will represents important topic basis in the corpus, which we refer to as the atoms of the topic.
Furthermore, restricting m to be much smaller than the number of the words ensures that the same topic needs to be used for multiple words.
(cid:126)A j is an interesting or signiﬁcant topic because the sparse coding ensures that each basis element is softly chosen by many words.
Word Topics Vector Formation:
For each word (cid:126)w , we created K different word-cluster vectors of d dimensions (cid:126)cv wk by weighting the word embedding with its learned dictionary coefﬁcient α w,k of the k th context.
We then concatenated all the K word-cluster vectors (cid:126)cv wk into a K × d dimensional embedding to form a word-topics vector (cid:126)tv w ∈ R K × d .
(cid:126)cv w,k ← (cid:126)v w × α w,k ; (cid:126)tv w ←
K (cid:77) k
= (cid:126)cv wk
Here, (cid:76) represents concatenation operation, and × represents vector-scalar multiplication.
Smooth Inverse Frequency (SIF) re-weighted: Finally, for all words appearing in document D n , we weighted the word-topics vectors (cid:126)tv i by smooth inverse frequency (cid:0) a a + p ( w ) (cid:1) .
Next, we removed the common context from the weighted average document vectors by removing the ﬁrst principal component from the weighted average vectors.
(Arora et al., 2017) empirically shows that SIF weighting outperforms the tf-idf weighting.
They provide a theoretical explanation for their superior performance.
[Equation]
Data: Word embeddings { (cid:126)v w : w ∈ V } , Documents { d n : d n ∈ D } , a set of sentences D , parameter a and estimated probabilities { p ( w ) : w ∈ V } of the words, a sparsity parameter k , and an upper bound m .
Result: Document vectors { (cid:126)v d n : d n ∈ D } /
[Equation]
[Equation] [Equation]
[Equation]
end is scalar vector
[Table]
P-SIF D ISCUSSIONS
For single sentence documents all words of a document belong to a single topic.
However, for multiple sentence-documents, words of a document generally originate from multiple topics.
To capture this phenomenon, topic modeling algorithms such as LDA (Blei et al., 2003) are used to represent the documents.
These representations essentially represent the global contexts of the documents as a distribution over topics.
However, these representations do not take the local context initiating from the distributional semantics into account such as word vectors.
To consider both local and global contexts, we represented each word as a word-topic-vector (cid:126)tv w .
In this section, we describe our simple and efﬁcient unsupervised method for document representation in details.
Sparse Dictionary Learning (Algorithm 1: Lines 1 - 3): (Arora et al., 2016b) shows that atoms of sparse coding over word-vectors represent all prominent topics in the corpus.
Furthermore, they showed that multiple word senses of a word reside as a linear superposition within the word embedding and can be recovered by simple sparse coding.
Therefore, one can use the sparse coding of word vectors to detect multiple senses of words and total senses of all the words (number of basis).
To ﬁnd these topics, we used sparse coding algorithms such as k-svd (Aharon et al., 2006) over word vectors (cid:126)v w .
For a given word ( w ), the k non-zero coefﬁcient essentially represents the distribution of words over topics.
The k non-zero α w for a given word w , basically represents the multi-sense nature of the words.
Word Topics Vector (Algorithm 1: Lines 3 - 9): Since our multi-sentence documents have words coming from multiple topics, we did not directly average word-vectors with a simple averaging technique.
We concatenated the word embeddings over the topic distribution of the words.
This helps to represent semantically similar words in the same topic, while words which are semantically common component from ﬁnal vectors, when we used Doc2VecC (Chen, 2017) initialized word vectors with P-SIF, because frequent words’ word-vectors become close to (cid:126) different are represented in different topics.
Concatenation of word embeddings over topics also helps in the expression of the multi sense nature of the words.
We weigh word-vectors by coefﬁcients of the learned dictionary to capture the cross correlation ( α i α j ) between contexts.
SIF Weight Averaging and Common Component Removal (Algorithm 1: Lines 9 - 16): Instead of assuming a single topic for the whole document, we showed that the total number of topics over a given corpus is K (as shown by (Arora et al., 2016b)) and for a given word, k out of K of them would be active.
Compared to SIF which directly averages SIF weighted words-vectors, we ﬁrst partitioned according to the topics through dictionary learning over word-topics vectors.
We ﬁnally averaged SIF weighted word-topics vectors.
Lastly, we removed the ﬁrst principal components from the document vectors to remove the common component.
Sparse Dictionary Learning vs. Fuzzy Clustering.
Sparse coding can also be treated as a linear algebraic analogue of overlapping clustering, where the (cid:126)A i ’s act as cluster centers and each (cid:126)v w is assigned to each cluster in a soft way (using the coefﬁcients α ( w,j ) , of which only k are nonzero) to a linear combination of at most k clusters.
In practice, sparse coding optimization produces coefﬁcients α ( w,j ) which are almost all positive, even though unconstrained .
One can use fuzzy clustering where each word belongs to every cluster with some probability P ( c k | w i ) , which can be thought as a substitute for α ( w,k ) , similar to the approach in SCDV (Mekala et al., 2017).
Practically, when the number of contexts is large, dictionary learning performs better than fuzzy clustering due to 1) better optimization and non-redundant clusters and 2) automatic handling of the tail of P ( c k | w i ) distribution through sparsity constraints (Olshausen & Field, 1997; Gao et al., 2010; Yang et al., 2009).
We observed that for a single sentence document with a small number of topics it is better to use fuzzy clustering because it has a better optimization and learns efﬁcient clusters.
Additionally, in fuzzy clustering, the P ( c k | w i ) is a probability between 0 and 1 , which sums to 1 over all contexts.
However, to overcome the noise from the non-zero tail probability one can apply hard thresholding on the ﬁnal vectors, which are directly learned by few non-zero coefﬁcients ( α w,j ) of the dictionary.
Our approach is averaging the words which belong to similar topics and concatenating these averages across different topics.
Therefore, our approach is a strict generalization of the sentence embedding approach by (Arora et al., 2017) which is a special case where total topics or themes is K = 1 .
E XPERIMENTAL R ESULTS
We performed several experiments on several text similarity and classiﬁcation tasks.
We will address the following research questions through our experiments
Q1.
Why partition and sparsity is required during word vectors averaging for representing documents?
Q2.
Does P-SIF represent large text documents better compared to other techniques used for representing a single sentence?
Q3.
What is the effect of SIF weighting and common component removal on document representation?
6. T EXTUAL S IMILARITY T ASK
Datasets.
We performed our experiments on the SemEval dataset (2012-2017).
These experiments involved 27 semantic textual similarity (STS) tasks (2012 - 2016)
(Agirre et al., 2012; 2013; 2014; 2015; 2016), the SemEval 2015 Twitter task (Xu et al., 2015), and the SemEval 2014 Semantic relatedness task (Marelli et al., 2014).
The objectives of these tasks are to predict the similarity between two sentences.
We used the Pearsons coefﬁcient (Fleiss & Cohen, 1973) between the predicted scores and the ground-truth scores for the evaluation.
Please refer to the supplementary section F. for the experimental setting details.
Unsupervised : We used ST, avg-Glove, tﬁdf-Glove and Glove + WR as a baseline.
ST denotes the skip-thought vectors by (Kiros et al.,
[Table] [Table]
idf weighted average of Glove vectors.
We also compared our method with the SIF weighting ( W ) common component removal ( R ) Glove vectors (Glove + WR ) by (Arora et al., 2017)
For STS we also compared our embedding with Skip-Thoughts (Kiros et
al
and Sent2Vec (Pagliardini et al., embeddings. Semi-Supervised : We used avg-PSL, PSL + WR, and the avg-PSL used the unweighted average of the PARAGRAM-SL999 (PSL) word vectors by (Wieting et al., 2015) as a baseline, obtained by training on PPDB dataset(Ganitkevitch et al.
The word vectors are trained using unlabeled data.
Furthermore, Sentence embeddings are obtained from unweighted word vectors averaging.
We also compared our method with the SIF weighting (W) common component removal (R) PSL word vectors (PSL + WR) by (Arora et al
Supervised : We compared our method with PP, PP-proj., DAN, RNN, iRNN, LSTM (o.g), LSTM(no) and GRAN.
All these methods are initialized with PSL word vectors and then trained on PPDB dataset Ganitkevitch et al., 2013).
PP(Wieting et al., 2016a) is the average of word vectors while PP-proj is the average of word vectors followed by a linear projection.
The word vectors are updated during the training.
DAN denotes the deep averaging network of (Iyyer et al., 2015).
RNN is a Recurrent neural network, iRNN is identity activated Recurrent Neural Network based on identity initialize weight matrices.
The LSTM is the version from (Gers et al., 2002), either with output gates (denoted as LSTM (o.g.)) or without (denoted as LSTM (no)).
GRAN denotes the state of art supervised averaging based Gated Recurrent Averaging Network from (Wieting & Gimpel, 2017).
For STSwe also compared our embedding with Tree-LSTM (Tai et al., 2015) embedding.
More details on the values of the hyper-parameters used for the experiments are described in details in the supplementary section F.1.
Results and Analysis
The average results for each year are reported in Table 2.
We denoted our embeddings by P-SIF + PSL ( + PSL denotes using the PSL word vectors).
We reported the average results for the STS tasks.
The detailed results on each sub-dataset are in the supplementary section A.
We observed that P-SIF + PSL outperforms PSL + WR on all datasets, thus supporting the usefulness of our partitioned averaging.
Our method also outperformed Neural Network models such as LSTM and RNN.
Our proposed techniques performed the best over 16 datasets out ofcompared to methods from (Wieting et al., 2016a) and (Arora et al., 2017) (refer to supplementary section A for details).
Our method outperformed supervised averaging based Gated Recurrent
Averaging Network (GRAN) on 11 datasets.
Furthermore, our results outperformed recently pro- posed unsupervised methods such as Skip-Thoughts(Kiros et al., 2015), Sent2Vec(Pagliardini et al., 2018), and supervised method such as Tree-LSTM(Tai et al., 2015).
We observed that partitioning through fuzzy cluster algorithm such as GMM generates a better performance compared to partitioning through sparse dictionary algorithms such as k-svd for most Semantic Textual Similarity (STS) tasks.
The main reason for this peculiar observation was related to the fact that the STS datasets contains documents which are single sentences of a maximum length of 40 words.
As discussed in section 5 (sparse dictionary learning vs. fuzzy clustering), for single sentence documents with a small number of topics, fuzzy clustering optimizes better than sparse dictionary learning.
Therefore, we used GMM for the STS task which results in partitioning words into suitable clusters.
Both ksvd and GMM outperform SIF, however, the improvement was more observable with the GMM-based partitioning.
It should be noted that this observation does not hold for 20NewsGroup and Reuters datasets since the documents in these datasets have multiple sentences with a total number of words >> 40 .
We also report some qualitative results in the supplementary section D with a supporting example in section E.
In addition, we report our results on the SICK supervised classiﬁcation task in supplementary section B.
6. T EXTUAL C LASSIFICATION T ASK
The document embeddings obtained by our method can be used as direct features for downstream many supervised tasks.
Datasets We ran multi-class experiments on 20NewsGroup dataset 4 and multi-label classiﬁcation experiments on Reuters-21578 dataset 5 .
We used script 6 for preprocessing the dataset.
Please refer to supplementary section F.1 for the hyperparameter’s details.
Baselines We considered the following baselines: The Bag-of-Words (BoW) model ((Harris, 1954)), the Bag of Word Vector (BoWV) (Gupta et al., 2016) model, Sparse Composite Document Vector (SCDV) (Mekala et al., 2017) 7 paragraph vector models (Le & Mikolov, 2014), Topical word embeddings (TWE-1) (Liu et al., 2015b), Neural Tensor Skip-Gram Model (NTSG-1 to NTSG-3)
(Liu et al., 2015a), tf-idf weighted average word-vector model(Singh & Mukerjee, 2015) and weighted
Bag of Concepts (weight-BoC) (Kim et al., 2017) where we built document-topic vectors by counting the member words in each topic, and Doc2VecC (Chen, 2017) where averaging and training of word vectors are done jointly.
Moreover, we used SIF (Arora et al., 2017) smooth inverse frequency weight with common component removal from weighted average vectors as a baseline.
We also compared our results with other topic modeling based document embedding methods such as WTM (Fu et al., 2016), w2v-LDA (Nguyen et al., 2015),
LDA (Chen & Liu, 2014), TV+MeanWV (Li et al., 2016a)), LTSG (Law et al., 2017), Gaussian-LDA (Das et al., 2015), Topic2Vec (Niu et al., 2015)
Lda2Vec and
[Table]
Multi-class classiﬁcation We evaluated the classiﬁer’s performance using standard metrics such as accuracy, macro-averaging precision, recall and F-measure.
Table 5 shows a comparison on multiple state-of-art document representations on the 20NewsGroup dataset.
[Table]
Multi-label classiﬁcation:
We evaluated multi-label classiﬁcation performance using Precision@K, nDCG@k(Bhatia et al., 2015), Coverage error, Label ranking average precision score (LRAPS) 8 and F1-score.
Table 4 shows the evaluation results for multi-label text classiﬁcation on the Reuters-21578 dataset.
Results and Analysis We observed that P-SIF outperforms all other methods by a signiﬁcant margin on both 20NewsGroup and Reuters datasets.
We observed that the dictionary learns more diverse and non-redundant topics compared to fuzzy clustering (SCDV) since we only requirepartitions rather than 60 partitions in SCDV to obtain the best performance.
Simple tf-idf weighted averaging based document representation does not show signiﬁcant improvement in performance by increasing word vector dimensions.
We achieved a < 0 .
4 % improvement in accuracy when the word vectors dimensions increase from 200 to 500 on 20NewsGroup.
We observed that increasing word vectors dimensions beyond 500 does not improve the performance of SIF and P-SIF.
We further improved the performance on both datasets using Doc2vecC initialized word-vectors with the help of the P-SIF algorithm.
We represented this approach by P-SIF (Doc2VecC) in Table 5 and
Table 4.
On 20NewsGroup we required only 20 partitioned instead of 40 with Doc2VecC initialized word vectors.
This shows that better word vector representations help in learning more diverse and non-redundant partitions.
We also reported our results on each of the 20 classes of 20NewsGroup separately in the supplementary section C.
A NALYSIS AND D ISCUSSION
Effect of Document-Length: We conducted a small experiment to show that our model performs better compared to SIF when we have large size documents.
We have divided 26 STS (sentence textual similarity) datasets by average document length, i.e., number of words in documents in bins of (10-20, 20-30, 30-40, 40-50) words.
Next, we averaged the relative performance improvement in accuracy with respect to SIF (cid:0) P-SIF − SIF SIF % (cid:1) and over all datasets in each bin.
In Figure 2, we observe that for complex multi-sentence documents with more words P-SIF relatively performs better.
We have also observed that short texts require fewer number of partitions to achieve their best performance which was quite intuitive since short text documents will map into fewer topics.
[Figure]
Effect of Partition: We have concatenated the word embeddings over topic distribution of words which assigned semantically similar words to identical partitions and semantically different words to different partitions.
Partitioning and concatenation of word embeddings over topics also helps in the representation of multi-sense words, which would have been left-out by simple averaging of the word embeddings in document representation otherwise.
We represented our words by partitioned average of word-vector embeddings distributed over partitions, which grew the representation by the multiplication of the number of partitions, compared to word vector embeddings.
Such high dimensional data structure regularizers, e.g., sparse encodings, help in overcoming the curse of dimensionality.
We used k-svd (Aharon et al., 2006) a generalized k-means iterative method that alternates between the sparse coding of the examples based on the current dictionary and updating the dictionary atoms to ﬁt the data.
SCDV (Mekala et al., 2017) does manual sparse-encoding to ensure sparsity.
(Mekala et al., 2017) clustering methodology uses GMM for partitioning the vocabulary space and capturing the topics.
Instead of GMM, we used a dictionary learning based approach which imposes a sparsity constraint implicitly during optimization.
Since, k-svd is internally and alternatively optimized between sparse encodings and topic representations, we are able to solve the curse of higher dimensionality by automatic sparse representations of the partitions.
Emperically, on both datasets, we observed that the dictionary learns more diverse and non-redundant topics compared to fuzzy clustering.
We required only 40 partitions rather than 60 in SCDV to obtain the best performance.
Additionally, we achieved the best performance with (P-SIF(Doc2VecC)) with just (20 ∗ 300) dimensions of word embeddings (mostly sparse) as compared to (60 ∗ 300) dimensions of word-embeddings (mostly non-sparse) in SCDV.
Thus, we obtained a performance gain of 1 . 5% with 0 .
33 of the size of the SCDV embeddings.
Effect of SIF and Common Component Removal: We have also shown improvement in shorter text tasks via sentence textual similarity task.
We have used SIF based weighted word averaging methods instead of tf-idf averaging (Mekala et al., 2017) or simple averaging of word-embeddings.
SIF takes advantage of the generative process of document-formation and represents documents as simple smooth inverse frequency averaging of word embeddings.
We also observe that removing the ﬁrst principal component helps in removing the common component occurring due to frequent common words from document vectors.
This observation reduces noise and redundancy from the composed document vectors which make vectors more discriminating.
C ONCLUSIONS AND F UTURE W ORK
We proposed an unsupervised document feature formation technique based on a partitioned word vector averaging method.
Our embedding retains the simplicity of simple weighted word averaging, while taking documents’ thematic structure into account.
Our simple and efﬁcient approach achieved signiﬁcantly better performance on several textual similarity tasks, textual classiﬁcation tasks and out of domain tasks.
In the future, three extensions of our work would be 1.
Learning a supervised weighting scheme of words according to the task to improve words weighting, 2.
Learning and using context sensitive word embeddings instead of normal skip-gram embeddings to resolve cluster disambuity, and 3.
Learning a lower dimensional space manifold to represent word-topic vectors for continuous representations useful in deep leaning applications.
Supplementary: Unsupervised Document
Representation using Partition Word-Vectors Averaging
A U NSUPERVISED T ASK : T EXTUAL S IMILARITY
In this supplementary section, we present the details results for STS tasks for each year.
Each year, there are 4 to 6 STS tasks, as shown in Table 6.
Note that tasks with the same name in different years are different tasks in reality.
We provide detailed results for each tasks in STS 12 - 15 in Table 7.
Our method outperforms all other methods from (Arora et al., 2017) and (Wieting et al., 2016a) on all 16 out of 22 tasks.
Our method performs signiﬁcantly better in comparison to all unsupervised embedding methods.
In addition, P-SIF is very close to the best performance by supervised methods on the rest of the datasets.
The performance difference between best method and our method compared with best method is much small.
Our method was also able to outperform state of art supervised averaging based Gated Recurrent Averaging Network (GRAN) (Wieting & Gimpel, 2017) on 11 datasets shown in Table 7.
Our results also outperform state of art methods on many recent supervised embedding methods on STS 16 task, as shown in Table 8.
[Table]
B O THER S UPERVISED T ASKS
We also consider three out of domain supervised tasks: the SICK similarity task, the SICK entailment task, and the Stanford Sentiment Treebank (SST) binary classiﬁcation task by (Socher et al., 2013).
We use the setup similar to (Wieting et al., 2016a) and (Arora et al., 2017) for a fair comparison, including the linear projection maps which take the embedding into 2400 dimension (same as skip-thought vectors), and is learned during the training.
We compare our method to PP,
DAN, RNN, LSTM, and skip-thoughts and other baselines, details results in Table 9.
Results and Analysis.
Our method (P-SIF) gets better performance compared to PSL + WR on all the three tasks similarity, entailment and sentiment.
We obtained best results for two of the supervised tasks, despite many of these methods (DAN, RNN, LSTM) are trained with supervision.
Furthermore, the skip thought vectors use has a higher dimension of 2400 dimension instead ofdimensions (which we projected to 2400 for a fair comparison).
Our method wasn’t able to outperform sentiment task compared to supervised tasks because a) due to antonym problem word vectors capture sentimental meaning of words and b) also in our weighted average scheme, we didn’t give more weights to sentiment words such as not , good , bad , there may be some important sentiment words which are down-weighted by SIF weighting scheme.
However, we outperform PSL + WR by a signiﬁcant margin and have lesser performance gap with the best supervised approach.
[Table]
[Table] [Table] [Table] images
[Table] [Table] [Table] [Table]
[Table]
C C LASS P ERFORMANCE : 20N EWS G ROUP
We also report the precision and recall results of separate 20 classes of the 20 NewsGroup dataset.
We compared our embedding (P-SIF) with Bag of Words, SCDV embeddings and our embeddings.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
33 misc.forsale 81.
[Table]
[Table] [Table]
[Table] [Table]
[Table]
84.35 rec.autos
[Table] [Table]
.74 rec.motorcycles
[Table]
[Table] [Table]
.baseball
[Table] [Table]
.hockey
[Table]
79 sci
[Table]
crypt
[Table]
02 sci.electronics
[Table] [Table]
[Table] [Table]
[Table]
64 sci.med 79.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
28 sci.space
[Table] [Table]
[Table] [Table]
51 soc
[Table]
.christian
[Table]
D Q UALITATIVE R ESULTS
Table 11 represent successful example pair from STS 2012 MSRvid dataset where P-SIF give similarity score closer to ground truth than SIF.
Table 12 represent failed example pair from STS
MSRvid dataset where SIF give similarity score closer to ground truth than P-SIF.
We now introduce the headline notation use in the Table 11 and 12.
• GT : represent the given ground truth similarity score in range of 0-5.
• NGT : represent normalized ground truth similarity score.
NGT is obtain by dividing GT score by 5 so
that it’s in range of 0-1.
• SIF sc : represent the SIF embedding similarity score in range of 0-1.
• P-SIF sc : represent the P-SIF embedding similarity score in range of 0-5.
• SIF err : represent absolute error (cid:107) SIF sc − NGT (cid:107) between normalized ground truth similarity score and SIF embedding similarity score.
• P-SIF err : represent absolute error (cid:107)
P-SIF sc − NGT (cid:107) between ground truth similarity score and P-SIF embedding similarity score.
• Diff err : represent absolute difference between SIF err and P-SIF err .
Examples where PSIF perform better Diff err =
P-SIF err - SIF err (used in Table 11).
Examples where SIF perform better Diff err =
SIF err - P-SIF err (used in Table 12) •
Rel err : represent relative difference between SIF err and P-SIF err .
Example where P-SIF perform better Rel err =
Diff err SIF err (used in Table 11).
Examples where SIF perform better Rel err =
Diff err P-SIF err (used in Table 12)
E Q UALITATIVE E XAMPLE
Let’s consider a corpus ( C ) with N documents with corresponding most frequent words vocabulary ( V ).
Figure 3 represents the word-vectors space of V , where similar meaning words are closer.
We can apply sparse coding and partition the words-vector space in ﬁve (total topics K = 5 ) topic vector spaces.
Some words are polysemic and belong to multiple topics with some proportion, as shown in
Figure 3.
For example, words such as baby , person , dog and kangaroo , belong to multiple topics with
[Table]
Amanisplayingaguitar.
[Table]
[Table] [Table]
Theballerinaisdancing.
[Table]
Awomanplaystheguitar.
Amansingsandplaystheguitar
[Table]
[Table]
Aguyisplayinghackysack
Amanisplayingakey-
[Table]
[Table]
Amanisridingabicycle.
[Table]
Awomanisswimmingunderwater.
[Table]
[Table]
Aanimatedairplaneislanding
[Table]
Themissileexploded
[Table]
Awomanispeelingapotato.
[Table]
[Table] [Table]
Amanisridingabike.
Amanisridingonahorse.
[Table]
Amanisclimbingarope.
[Table]
Amanisshootingagun.
[Table]
[Table] [Table]
[Table]
Theladypeeledthepotatoe.
[Table]
Aboyiscrawlingintoadoghouse.
[Table] [Table]
[Table] [Table]
[Table] [Table]
signiﬁcant proportion.
Words and corresponding vectors in these topic vector spaces are represented by topic numbers in subscript.
Table 13 shows an example pair from the STS Task
MSRVid dataset, and the corresponding SIF (averaging) and P-SIF (partition averaging) representation vectors.
We can see that in SIF representation we are averaging words vectors which semantically have different meanings.
The document is represented in the same d dimensional word-vectors space.
Overall, the SIF represent the document as a single point in the vector space and doesn’t take account of different semantic meanings of the topics.
Whereas, in P-SIF representation we treat the 5 different semantic topics distinctly.
Words belonging to different semantic topics are separated by concatenation ( ⊕ ) as they represent different meanings, whereas words coming from same topic are average as it’s represent same meaning.
The ﬁnal document vector (cid:126)v d n has more representational power as it’s represented in a higher 5 × d dimensional vector space.
Thus partitioned averaging with topic weighting is important for representing documents.
Empirically, P-SIF gave score dissimilar sentences ( d 1 n , d 2 n ) 0 .
16 (0-1 scale) where the ground truth of 0 .
15 (rescale to 0-1 scale), whereas
SIF gave similarity score of 0 .
57 (0-1 scale), farther then ground score.
Thus we obtain a relative improvement of 98% in the error difference from ground truth.
Here, simple averaging-based embedding of d 1 n and d 2 n , bring the document representation closer.
But, partitioned based averaging
P-SIF of d 1 n and d 2 n as, project the document in a higher dimensional space.
[Table]
F E XPERIMENTAL D ETAILS
F.
T EXTUAL S IMILARITY T ASK :
We use the PARAGRAM-SL999 (PSL) from (Wieting et al., 2015) as word embeddings, obtained by training on PPDB (Ganitkevitch et al., 2013) dataset 9 .
We use the ﬁx weighting parameter α value of 10 − 3 , and the word frequencies p ( w ) are estimated from the common-crawl dataset.
We tune the number of contexts ( K ) to minimize the reconstruction loss over word vectors.
We ﬁx the non-zero coefﬁcient m = K/ 2 , for the SIF experiments.
For GMM based partitioning of the words vocabulary we tune the number of cluster parameter K through 5-fold cross validation.
F. T EXTUAL C LASSIFICATION T ASK :
We ﬁx the document embeddings and only learn the classiﬁer.
We learn word vector embedding using Skip-Gram with a window size of 10 , Negative Sampling (SGNS) of 10 and minimum word frequency of 20 .
We use 5 -fold cross-validation on the F 1 score to tune hyperparameters.
We use
LinearSVM for multi-class classiﬁcation and Logistic regression with the OneVsRest setting for multi-label classiﬁcation.
We ﬁx the number of dictionary elements either 40 or 20 (with Doc2vecC initialize word vectors) and non-zero coefﬁcient to m = K/ 2 during dictionary learning for all experiments.
We use the best parameter settings as reported in all our baselines to generate their results.
We use 200 dimensions for tf-idf weighted word-vector model, 400 for paragraph vector model, 80 topics and 400 dimensional vectors for TWE, NTSG, LTSG and 60 topics and 200 dimensional word vectors for SCDV (Mekala et al., 2017).
We will released P-SIF’s embedding source code with necessary parameters details and other data-sets used in the paper for reproducing results. G C ODE [Table] LOW
[Table]
RCHITECTURE
[Table]
I P ROOF S KETCH : D ERIVATION OF OUR E MBEDDING
To derive our embedding, we propose a generative model which treats corpus generation as a dynamic process, where the t th word is produced at step t .
The process is driven by random walk over a unit norm sphere with centre at the origin.
Let, (cid:126)v c t be the vector from origin to current walk point at time t .
We called this vector as context vector, as it represents the context in the discussion.
Below, we introduce formal notation needed for the discussion
• C represents text corpus and V represents vocabulary of words in corpus.
• (cid:126)v w ∈ R
d
represents the word vector of word w , where d is dimension of word vector.
• c t represents context and (cid:126)v c t ∈ R d represents context vector at time t , where d is dimension of context vector.
• Z c represents partition function for the random context vector (cid:126)v c t , given by Z c = (cid:80) w exp( (cid:104) (cid:126)v c t ,(cid:126)v w (cid:105) ) .
• p ( w ) represents the unigram probability of word w in the corpus.
• c 0 and (cid:126)v c 0 ∈ R d represent common context and its corresponding context vector based on syntax.
Using log linear model of Mnih & Hinton (2007), we deﬁne the probability of observing a word w from the random walk with current context c t at time t as
[Equation]
It is easy to show that such random walk under some reasonable assumptions (Arora et al.
(2016a)) can gives word-word co-occurrence probabilities similar to empirical works like word2vec (Mikolov et al.
(2013a))
and Glove (Pennington et al.
(2014a)).
To account for frequent stop-words which occur more often regardless of context and common context related to document syntax, two correction terms need be added: one based on p ( w ) and other on common context vector (cid:126)v c 0 in Equation equation 3.
These terms allow words with low inner product with (cid:126)c t , a chance to appear either from term p ( w ) if they are frequent or by common context (cid:126)c 0 , if they have large dot product with (cid:126)c 0 .
Given a context vector c t , the probability of a word w in document d being generated by context c t is given by,
[Equation]
where, (cid:126)c (cid:48) t
{+= β(cid:126)c 
+}
+ (1 − β ) (cid:126)c t , (cid:104) (cid:126)c 0 ,(cid:126)c t (cid:105) = 0 , λ and β are scalar hyper-parameters.
For generating a document from above random walk based latent variable model, we consider the following two assumptions :
Total number of theme/topics in the entire corpus is K .
The K themes/topics can be determine by sparse dictionary learning as shown by Arora et al.
(2016b) over word vectors (cid:126)v w = (cid:80) mj
=1 α w,j (cid:126)A j + (cid:126)η w , where, (cid:126)A j ’s are unit norm vectors representing basis of theme, α w,j are coefﬁcient determining whether w is generated with (cid:126)A j 10 , and (cid:126)η w is a noise vector.
Word vectors (cid:126)v w are uniformly distributed, thus making the partition function Z c , roughly same in all direction for given context c , where c belongs to one of the K themes/topics, as described earlier.
The context vector does not change signiﬁcantly much while words are generated from random walk, as shown by Arora et al.
(2017) except during jumps a.k.a when theme/topic change
For a document d , the likelihood of document is being generated by the K contexts, is given by :
[Equation]
Let,
[Equation]
Here, p ( w | c j ) is the probability that word w is generated by context c j , the value of which is determine by 1) overall frequency of word w in corpus a.k.a prior probability ( p ( w ) ) and 2) relative frequency of w appear with context j w.r.t other contexts (determine by α ( w,j ) ).
Using simple calculus, treating p ( w ) as constant, we can show that ∇ ( f w ( c j )) equals,
[Equation]
Then, by using the Taylor expansion, we can show
[Equation]
[Equation]
Therefore, the maximum likelihood estimator (MLE) for (cid:126)v c j on the unit sphere (ignoring normalization) is approximately,
[Equation]
here, a = 1 − λ λZ j
Thus, the MLE estimate is approximately a weighted average of the word vectors generated from context j in the document d from random walk.
We can get the overall context representation (cid:126)v c d of document, by simple concatenation over all K themes/topics.
[Equation]
Here, (cid:76) represents concatenation operation.
For a document if no words is generated from the context c j the we can substitute the context vector (cid:126)v c j by (cid:126) 0 vector, for representation (cid:126)v c d in K × d dimensions.
Relation to SIF model: Arora et al.
(2017), shows under the two assumptions :
• uniform distribution of word vectors (cid:126)v w ’s which implies that the partition function Z t is roughly same in all direction for the a sentence.
• the context vector (cid:126)v c t , remain constant while the words in the sentence are emitted, implying replacing of (cid:126)v c t in sentence’s by a single vector (cid:126)v c s and partition function by Z s
the sentence embedding of a sentence can be obtained by (cid:126)v c s = (cid:80) { w ∈ s } a p ( w )+ a (cid:126)v w , here, a = 1 − λ λZ s .
However, the above assumptions does not hold true for a document with multiple sentences, where one can expect to have more frequent random jumps during random walk 12 .
Instead of assuming a single context for whole document c h , we assume that the total number of theme/topics over a given corpus is bounded by K (as shown by Arora et al.
(2016b))
and the random walk can perform jumps to switch context from one context to the rest K − 1 contexts.
The partition function remain same in all directions for only words coming from a same context c j , instead of words coming from all
K contexts.
Thus, our approach is generalization of sentence embedding approach by Arora et al.
(2017) (special case K = 1).
J K ERNEL CONNECTION TO W ORD M OVER D ISTANCE
Below, we introduce formal notation needed for the discussion:
• C represents text corpus and V represents vocabulary of words in corpus. • (cid:126)wv
w ∈ R d represents the word vector of word w , where d is dimension of word vector.
• (cid:126) tv w ∈ R K represents the theme/topic vector of word w, where K is number of themes/topics.
[Equation]
• d A , d B represent two document containing n and m words respectively.
w A 1 , w A 2 . . .
w An represent words of d A and w B 1 , w B 2 . . .
w Bn represents word of d B .
Consider the following document similarity kernels:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We can conclude the following from the respective kernels:
• K 1 ( d A , d B ) represent document similarity between document represented by average word vectors d x = (cid:80) i (cid:126)wv xi • K 1 ( d A , d B ) represent document similarity between document represented by partition average word vectors • K 3 ( d A , d B ) represent document similarity between document represented by relax word mover distance when words of d A are matched to d B • K 4 ( d A , d B ) represent document similarity between document represented by relax word mover distance when words of d B are matched to d A • K 5 ( d A , d B ) represent document similarity between document represented by word mover distance.
We empirically showed that our proposed embedding (kernel) outperform the word mover distance(Kusner et al., 2015) and word mover embedding (Wu et al., 2018) and many other baselines in Table 14 on several datasets 13 .
[Table]
K R ECENT B ASELINE FOR T EXTUAL S IMILARITY T ASK
We compared out P-SIF embedding with many other recently proposed baseline like ELMO (Peters et al., 2018), p-means (R ¨ uckl ´
e et al., 2018), FastText (Joulin et al., 2016), Skip-Thoughts (Kiros et al., 2015), (Conneau et al., 2017), Charphrase (Wieting et al., 2016b), WME (Wu et al., 2018) and u-SIF Ethayarajh (2018).
We used the SentEval package (Conneau & Kiela, 2018) and embedding evaluation paper Perone et al.
(2018) for baselines.
Except u-SIF we outperform all other embedding.
Our P-SIF results was very close to u-SIF in most tasks.
Details results are provided in Table
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
L M ORE E XPERIMENTAL R ESULTS
We also compared our embedding extensively on random hashing of words (instead of GMM) (refer
Table 16), p-means (R¨uckl´e et al., 2018) (refer Table 17), and ELMO (Peters et al., 2018)
(refer
Table 18) embedding on 20NewsGroup.
P-SIF outperform all three embedding on 20NewsGroup.
[Table]
[Table]
TRUE 75.
1,+inf
TRUE 74.
TRUE 73.
1,2,-inf
TRUE 73.
1,2,+inf
TRUE 72.
1,2,3,-inf
TRUE 72.
1,2,3,-inf,+inf
TRUE 72.
1,2,3,+inf
TRUE 71.
1,2,-inf,+inf
TRUE 71.
1,-inf,+inf
TRUE 71.
1,-inf,+inf
TRUE 69.
1,2,-inf,+inf
FALSE 69.
1,-inf
FALSE 59.
1,-inf
TRUE 59.
P-SIF - - 86.
(Doc2VecC)
P-SIF - - 85.
We also compared our embedding extensively on p-means (R¨uckl´e et al., 2018) embedding on
Reuters.
P- [Table] [Table] [Table]