T OWARDS B UILDING A FFECT SENSITIVE W ORD D ISTRIBUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning word representations from large available corpora relies on the distributional hypothesis that words present in similar contexts tend to have similar meanings.
Recent work has shown that word representations learnt in this manner lack sentiment information which, fortunately, can be leveraged using external knowledge.
Our work addresses the question: Can affect lexica improve the word representations learnt from a corpus ?
In this work, we propose techniques to incorporate affect lexica, which capture ﬁne–grained information about a word’s psycholinguistic and emotional orientation, into the training process of Word2Vec
SkipGram, Word2Vec CBOW, and GloVe methods using a Joint Learning approach.
We use affect scores from Warriner’s affect lexicon to regularize the vector representations learnt from an unlabeled corpus.
Our proposed method outperforms previously proposed methods on standard tasks for word similarity detection, outlier detection, and sentiment detection.
We also show the usefulness of our approach for a new task related to the prediction of formality, frustration, and politeness in corporate communication.
I NTRODUCTION
In natural language research, words, sentences and paragraphs are considered in context through vector space representations, rather than as atomic units with no relational information among them.
Although n-gram based methods trained on large volumes of data have been found to outperform more complex approaches both on computational cost and accuracy, the techniques do not scale well in cases where the corpus size is limited(for example, for labeled speech or affect corpora with a size of a few millions of words).
Recent work has attempted to improve the performance of word distributions for downstream tasks such as sentiment analysis (Sedoc et al., 2017) and knowledge base completion (Kumar & Araki, 2016) using lexical knowledge to enrich word embeddings, by performing methods such as regularization or introducing a loss term in the learning objective.
Sentiment relationships between words can be considered transitive, where ‘good’ < ‘better’ < ‘best’ implies that ‘good’ < ‘best’.
However, word representations based on traditional approaches such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) are agnostic to the associated sentiments, emotions, or more generally affects (Bradley & Lang, 1999).
Furthermore, although words such as delighted and disappointed share similar vector representations given their similar contexts, these words are associated with opposite reactions (or sentiments) as well as have a fairly different interpreted meaning.
The challenge in using syntactic relational information for sentiment detection, is that sentiment relations are transitive and symmetric (i.e., if ‘delighted’ is the opposite of ‘disappointed’, then ‘disappointed’ is the opposite of ‘delighted’.)
Ignoring the bipolar nature of words could lead to spurious results, especially in predictive tasks related to synonyms and antonyms and sentiment analysis.
On the other hand, incorporating affect–related information would make word distributions homogeneous and suitable for speech and text generation tasks that aim at capturing author or reader reactions.
Furthermore, by using a small sentiment lexicon, it is possible to develop an automatic way to rate words based on their vector space representations.
This could help reduce the time and cost required to gather word ratings, as well as eliminate the implicit biases that may be introduced in annotations, such as the high correlation between high valence ratings with high arousal reported by Sedoc et al.
We present an approach to build affect–enriched word representations.
In other words, we enhance word distributions by incorporating reactions and affect dimensions.
The output of this work produces word distributions that capture human reactions by modeling the affect information in the words.
The affective word representations distinguish between semantically similar words that have varying affective interpretations.
Affect is represented as a weighted relational information between two words, following the approach used by existing work.
Sedoc et al.
(2017) identify words of opposite polarity by performing signed spectral clustering on pre–trained embeddings.
We present an approach to incorporate external affect and reaction signals in the pre–training step, using the hand-annotated affect lexica to learn from.
Our experiments are based on using the state-of-the-art
Warriner’s affect lexicon (Warriner et al., 2013) as the input.
The proposed approach builds on the intuition that relationships between synonyms and antonyms can be characterized using semantic dictionaries and the relationship can then be deterministically captured into the training loss functions.
We evaluate the proposed enriched word distributions on standard natural language tasks.
We predict formality, frustration and politeness on a labeled dataset and show improved results using the enriched word embeddings.
Further, we outperform the state–of–the–art for sentiment prediction on standard datasets.
The key contributions of this paper include:
• Algorithm to incorporate affect sensors in the cost functions of distributional word representations (including Word2Vec SkipGram, Word2Vec CBOW, and GloVe) during training using semantic and external affect signals.
• Establish the utility of affect enriched word–embeddings for linguistic tasks such as Sentiment and Formality prediction in text data.
Our method out performs the state–of–the–art with an 20% improvement in accuracy for the outlier detection methods.
Detailed results are reported in table 1.
• Introduce a workﬂow to incorporate affective and reaction signals to word representations during pre–training.
We show the generalizability of the workﬂow through experiments on 3 existing embeddings; Word2Vec–CBOW, Word2Vec–SkipGram
, and GloVe.
Section 2 covers the prior art in both pre–training and post–training approaches for distributional word representations.
Section 3 presents the proposed approach and detailed experiments are discussed in section 4.
We conclude with a discussion on the learnings and the observations through this process 5.
R ELATED W ORK
A huge amount of research has explored how to use external resources to improve on
Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) embeddings.
Research has reﬁned embeddings for various downstream tasks such as dependency parsing (Bansal et al., 2014), sentiment analysis (Sedoc et al., 2017) and knowledge base completion (Kumar & Araki, 2016).
There are mainly two approaches: joint learning and post-training.
The former considers the incorporation of external knowledge into the training process of learning word embeddings itself.
On the other hand, post-training approaches take already trained embeddings and use additional information to modify them.
Our approach falls in the ﬁrst category.
To the best of our knowledge, no prior work focuses on improving word embeddings by jointly learning from a corpus and an affect lexica.
Bian et al.
(2014) deﬁne a new basis for word representation and explore syntactic, semantic and morphological knowledge to provide additional information.
A binary indicator function is used to deﬁne relations.
Yu & Dredze (2014) propose a Relation
Constrained Model( RCM ) which predicts one word from another related word.
They use a linear combination of the objectives in CBOW and RCM for joint learning.
Kiela et al.
(2015) build specialized word embeddings for either similarity or relatedness.
They use a joint learning approach by using additional context words from external sources with the SkipGram loss function.
Our Word2Vec approach is similar to the pre-training model used in Kiela et al.
Essentially, words in the pruned list L ipruned (section 3) can also be taken as additional context words, similar to their understanding.
However, in our case, this addition is from an affect lexica and includes a strength associated with it.
Xu et al.
(2014) propose a general framework
RC-NET to incorporate knowledge into the word distributions.
They encode external relational and categorical information into regularization functions and combine them with the original objective of Word2Vec SkipGram model.
Our modiﬁed loss function can also be thought of as similar to the one in “Categorical Knowledge Powered model” in Xu et al.
(2014), thinking of the strength as a similarity score, although the distance function there is just deﬁned as the Euclidean distance.
Bollegala et al.
(2016) use a similar approach as ours for GloVe method to include various relations like synonyms, antonyms, hyponyms and so on.
They make use of a binary function to indicate whether the relation between any two words exists or not.
For post-training, Faruqui et al.
(2014) makes no assumption about input pre-trained word vectors.
This work proposes an objective to further reﬁne the input embeddings using relational information from semantic lexica.
Another post training approach has been proposed in Mrkˇsi´c et al.
(2016) which deﬁnes the ﬁnal objective as a weighted sum of “Antonym Repel”, “Synonym Attract” and “Vector Space Preservation” objectives.
J OINT LEARNING FROM UNLABELED CORPUS AND AFFECT LEXICA 3. N OTATIONS
Consider a corpus C and an affect lexica L consisting of l (word, affect-score) pairs.
Let us denote the i th pair in L with p i
= ( w i , s i ) .
We deﬁne a function S ( i, j ) which captures the strength of the relationship between any two words w i and w j in L .
3. O UR PROPOSED APPROACH
To incorporate an affect lexica into the process of learning word embeddings, we follow a three step approach (See Figure 1):
Step 1: Identifying Synonyms and Antonyms -
Given a pair p i in L , WordNet (Miller, 1995) is used to identify all pairs p j in L such that w j is either a synonym or an antonym of w i .
Synonyms and antonyms are retrieved based on WordNet deﬁnitions.
Note that, S ( i, j ) will be deﬁned as non–zero only when w j is returned by WordNet as the synonym or antonym of w i .
For example, consider a pair of words from the Warriner’s affect lexicon (see section 4.1 for lexicon details): although ‘conﬁdent’ and ‘funny’ have similar valence scores of 7 . 56 and 7 .
59 respectively (on a scale of 1 - 9 ), they do not share a semantic relationship between them (i.e. not deﬁned as synonyms or antonyms).
Here, S ( i, j ) is set to 0 .
Step 2:
Deﬁning strength S ( i, j ) for all possible ( i, j ) pairs - Polarity information is captured in our modeling by centering the scores around 0 instead of the 1 - 9 scale.
As already mentioned, S ( i, j ) is deﬁned as 0 if w j is not a synonym or antonym of w i .
For all other cases, the strength models the difference in affect scores of the two words under consideration.
If the words have scores with the same sign, we deﬁne a positive strength inversely proportional to the relative distance between them.
If the words have scores with opposite signs, the strength is negative, with magnitude directly proportional to the difference in their scores.
Algorithm 1 describes the approach.
1: function GET S TRENGTH ( s i , s j , smax, smin ) (cid:46) where s i and s j - normalized scores of w i and w j , smax and smin - maximum and minimum normalized scores 2: 3: strength =4: if s i s j > 0 then (cid:46)
Both scores have the same sign 5: if s i > 0 and s j > 0 then 6: strength = | s i − s j | / smax ; (cid:46) | .
| refers to taking the absolute value.
7: else 8: strength = | s i − s j | / smin ; 9: end if 10: strength = 1 - strength 11: else 12: strength = - | s i - s j | /( smax - smin ) 13: end if 14: return strength 15: end function
Step 3: Loss function deﬁnition - We introduce a new loss function for the embedding training.
The loss functions deﬁned for Word2Vec and GloVe are described here.
Word2Vec loss function : Rong (2014) describes the back–propagation algorithm of Word2Vec
SkipGram and CBOW techniques.
We build on top of that intuition.
For the SkipGram approach, the model predicts the context of an input word in the corpus.
Using the same notation as introduced in Rong (2014), the loss function of Word2Vec model (Mikolov et al., 2013) with negative sampling is deﬁned as the following:
[Equation]
where w O is the output word (i.e. the positive sample) and v (cid:48)
w O is it’s output vector.
h is the output value of the hidden layer.
For the SkipGram approach, h is simply v w I , which is the input vector corresponding to the input word w I .
W neg = { w j | 1 , ..., K } is the set of all negative samples.
The standard unigram distribution raised to the 34 th power for this sampling is used for all reported experiments.
Information from affect lexica is incorporated using another loss function in the following manner:
[Equation]
where v (cid:48) w j is the output vector as already deﬁned, h is the hidden layer output, i is the index of the input word w I in L and S ( i, j ) is the relation strength.
L ipruned is the pruned list of words obtained from WordNet corresponding to the input word w I (same as the word w i in L ).
Note that i will only be deﬁned if the input word w I belongs to the affect lexica L .
Hence, the loss function will only matter when a word present in L appears at the input end.
The ﬁnal loss function is deﬁned as:
[Equation]
where λ ≥ 0 is a hyper–parameter.
This modiﬁcation results in a modiﬁed form of the derivative of E with respect to ( v (cid:48)
T w j h ) , as given by the following equation:
[Equation]
[Figure]
The weight update equations can be further obtained using equation 4 as in Rong (2014).
Intuitively, if the strength is positive for a pair of words, the back–propagation algorithm makes their corresponding embeddings closer.
Alternatively, if the strength is negative, the algorithm tries to move them apart.
The magnitude of the strength controls the speed of this movement.
Inspired from negative sampling done from the vocabulary, we also try sampling the words from the entire affect lexica instead of using WordNet.
This suffers from the semantic relationship problems discussed in step one, the performance, hence declines.
The model used in CBOW is the opposite of the one deﬁned for SkipGram.
Given a context window around a word, the model predicts that word.
Hence, the hidden layer output is deﬁned as h =
C (cid:80)
Cc
=1 v w c , the combination of all input word vectors corresponding to the context of the output word.
The loss function is formulated similar to the one already described.
GloVe (Pennington et al., 2014) Loss Function :
We achieve this by using a weighted regularized version of the original GloVe objective.
Unlike Word2Vec, which considers only local cooccurrences, this approach uses global co-occurrences over our entire corpus.
We use the original GloVe implementation 1 , ﬁrst extracting the vocabulary and building a cooccurrence matrix X .
Borrowing the notation from Pennington et al.
(2014), the objective for GloVe is as follows:
[Equation]
where V is the vocabulary, b p and b q are real-valued bias terms, w p is the target vector for the word w p and ˜ w q is the corresponding context vector for the word w q .
f ( X pq ) is a weighting function deﬁned as:
[Equation]
Using the same values as in Pennington et al.
(2014), we keep α as 34 and x max as 100.
With the strength function S ( i, j ) (deﬁned in Step 2 ) in mind, we deﬁne a regularization term as follows:
[Equation]
where indices i and j in L correspond to p and q in V respectively.
The ﬁnal objective is obtained as follows:
[Equation]
using the same hyperparameter λ as before.
The obtained expression is similar to the one used in
Bollegala et al.
(2016), except that instead of using a strength function S ( i, j ) , they use a binary function R ( i, j ) , which indicates whether a relation exists between words w i and w j or not.
We direct the readers to Bollegala et al.
(2016) for the changes in update equations, which also result in similar values.
Figure 2 illustrates the proposed method through the t–distributed Stochastic Neighbor Embedding (t-SNE) visualization of the ﬁrst two components of a subset of words.
The points are colored to identify their positive (blue) or negative (red) valence in the affect lexicon.
As shown in Figure 2a, ‘accept’, ‘reject’ and ‘refuse’ were found close together in the original representation.
By adding valence information to baseline embeddings (Figure 2b), the model is able to (a) pull words of similar sentiment closer together and (b) pull words of opposite sentiment, further apart.
This shows that our method has face–validity in being able to identify and distinguish sentiment polarities within word embeddings.
E XPERIMENTS AND R ESULTS
In this section, we conduct experiments to examine whether incorporating affect information into learning continuous word representations can improve the intrinsic quality of word embeddings and the performance of trained models on downstream tasks.
First, we introduce the dataset and then describe the evaluation framework.
4. D ATA
ukWaC corpus : We use the ukWaC (Baroni et al., 2009) corpus for building our word embeddings.
It is large enough to obtain embeddings of a good quality, while still being tractable in terms of time and space resources.
We ﬂattened the dependency-parsed format of this corpus, resulting in 2.
billion tokens and a vocabulary size of 569,574 words after removing the words having frequency count of less than 20.
We experimented with different values for λ and found that the highest similarity score on RG word similarity dataset (Rubenstein & Goodenough, 1965) was with a λ = 2 .
The details of this experiment are provided in the Appendix.
Accordingly, we have used this value for λ throughout.
We use a window size of 10 and keep the word embedding dimensions as 300.
For Word2Vec, we use negative sampling of 15 words for optimization.
For GloVe, we found that running the model for 5 iterations was sufﬁcient.
On a machine with 8 core Intel 3.4GHz processor and 16GB RAM, the
Word2Vec skipgram approach takes 15 hours, CBOW takes under 100 minutes while GloVe takes approximately 15 minutes per iteration.
We compare the performance of the word2vec skipgram, word2vec CBOW and GloVe models for the following settings:
• The baseline approach corresponding to setting λ as 0,
i
e. only training on the unlabeled ukWaC corpus • With λ = 2 , incorporating either valence, arousal and dominance scores to the original corpus one at a time.
We refer to these models as ‘+V’,‘+A’ and ‘+D’ respectively •
Incorporating the mean weight of valence, arousal and dominance to the original corpus.
We refer to this approach as ‘+VAD’.
• Comparison against the state of the art : We reimplemented the approach by Bollegala et al.
(2016) on our dataset.
The authors use a binary indicator function for the incorporation of various relations such as synonyms and antonyms in the training process.
In the original paper, this approach, trained on GloVe embeddings, demonstrably improved the state of the art on standard word similarity and analogy prediction.
We pick their best performing model which uses synonym pairs and train it on our ukWaC corpus with the same parameter settings.
• Comparison against post-training baseline (Append) : We compare our results against a simple post-training baseline in which we concatenate the word-vectors ( D -dimensional WV) with VAD-vectors ( 3 -dimensional AV) resulting in a D + 3 dimensional vector.
The considered dataset might not contain VAD ratings for stop words and proper nouns.
Since they are neutral in sentiment (V), strength (D) and arousal (A), for these out-of-dictionary words the VAD vector is set to neutral (cid:126)ν = [5 , 5 , 5] in all our experiments.
Normalize both embeddings with their L2-Norms (Equation 9).
This reduces the individual vectors to unit-length. [Equation] These regularized distributional word-embeddings are then concatenated with regularized affect scores as shown in Equation 10. [Equation]
We also perform the standardization process on the resultant D + 3 dimensional embeddings.
This transforms each dimension in the vector to have unit variance and zero mean so that each dimension contributes approximately proportionately.
[Equation]
where µ and σ represent the mean and std.
deviation respectively
The resultant vectors capture affect properties of the word in a distributional setting.
Warriner affect word list : We use affective information from Warriner’s affect lexicon
(Warriner et al., 2013) which comprises 13,915 words tagged on Valence, Arousal and Dominance scores on a scale of 1–9.
Valence is the unhappy–happy scale, Arousal is the calm–excited scale and Dominance indicates the forcefulness of expressed affect.
In all our experiments, we use a normalized scale from -4 to 4 to take the signed information into account, similar to Sedoc et al.
4. E VALUATION F RAMEWORK
We evaluate the proposed method on three standard tasks: predicting the similarity of words on seven benchmark datasets, detecting outliers in semantic clusters on the 8-8-8 dataset (CamachoCollados & Navigli, 2016) and predicting sentiment on the Stanford Sentiment Treebank (Socher et al., 2013).
We then introduce a new dataset and task for formality, frustration and politeness detection for a labeled email corpus.
4.2. W ORD SIMILARITY PREDICTION
The task is to predict the similarity between given two words.
We compute the cosine similarity between the corresponding word embeddings of the two words and assign it as the similarity score.
We consider seven benchmark datasets:
• SIMLEX : the 999 word pairs list (Hill et al., 2016) • MC : 30 word pairs in Miller Charles (Miller & Charles, 1991) • MEN : 3000 pairs of words (Bruni et al., 2012)
• RG : 65 word pairs by Rubenstein-Goodenough (Rubenstein & Goodenough, 1965) • RW : 2034 pairs in the Rare-Words dataset (Luong et al., 2013), • SCWS : 2023 word pairs in Stanford’s contextual word similarities (Huang et al., 2012)
• WordSim : 353 word-pairs in the WordSim test collection (Finkelstein et al., 2001). [Table]
Each word pair in these benchmarks has a manually assigned score which we consider as gold standard ratings.
We report the Spearman Correlation Coefﬁcient between the gold standard similarity ratings and the cosine similarity scores assigned by our model.
The results are provided in Table 1.
The rows correspond to the approaches, with baselines corresponding to λ = 0 .
The columns correspond to the various datasets discussed above.
Our proposed modiﬁcations show reasonable improvements for GloVe and word2vec CBOW embeddings.
The poorest performance was observed for the word2vec skipgram approach, which did not outperform the baseline approach for most of the datasets.
In Table 1 we show that our approach showed modest improvements over the state of the art method by Bollegala et al.
on word similarity for two datasets.
We surmise that the nature of the word similarity task and the absence of “opposite” words in the benchmark datasets could be the reason why non-affective approaches are hard to beat for predicting word similarity.
For outlier prediction, our approaches gives a performance improvement of 5% over the approach by Bollegala et al.
4.2. O UTLIER DETECTION
Word similarity tasks have been widely used for measuring the semantic coherence of vector space models.
However, such tasks often suffer from low inter-annotator agreement scores of gold standard datasets.
Hence, we also report our results on an outlier detection task (Camacho-Collados &
Navigli, 2016), in order to test the quality of semantic clusters in the vector space models.
These results are on the 8-8-8 outlier detection dataset (Camacho-Collados & Navigli, 2016) containing 8 different topics, each made up of a cluster of 8 words and 8 possible outliers.
Table 1 summa- rizes the Outlier Position Percentage(OPP score) and Accuracy of our different models.
We refer
Camacho-Collados & Navigli (2016) to the readers for further details on the dataset and evaluation metrics.
Apart from Word2Vec SkipGram models, our approaches perform well on both OPP and Accuracy scores(higher is better).
We observe that the incorporation of affect information in terms of Valence,
Arousal and Dominance scores, shows improvements in an Outlier Detection task on an unrelated dataset with topics such as Football Teams, Solar Systems and Car Manufacturers.
Next, we evaluate our approach on two tasks for sentiment detection.
Since our approach enriches word embeddings with affect information, we anticipate that our models would outperform the state of the art on these tasks.
4.2. S ENTIMENT P REDICTION
For the sentiment prediction task, we make use of an available Deep Averaging Network 2 (DAN) model (Iyyer et al., 2015) with its default parameter settings along with our modiﬁed word embeddings.
We report the results on Stanford Sentiment Treebank(SST)
dataset
(Socher et al., 2013) which contains ﬁne grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences.
We use the standard splits of the datasets.
Table 2 summarizes the results for sentiment prediction.
We report the accuracy values for both ﬁne and binary classiﬁcation settings.
For the latter, we remove all the instances with neutral labels.
Our approaches show signiﬁcant improvements over the baselines for both these metrics.
As evident from Table 2, our approaches perform better on sentiment prediction task than the state of the art approach (Bollegala et al., 2016).
We attribute this improvement to our strength function which not only speciﬁes the existence of a relation between two words, but also provides the signed strength associated with it.
4.2. P REDICTING FORMALITY , FRUSTRATION AND POLITENESS IN EMAILS
Finally, we evaluate the quality our embeddings on an affect prediction task.
We use a new dataset consisting of 980 emails from the ENRON dataset (Cohen, 2009), tagged with formality, frustration and politeness scores by human annotators.
We use a CNN based regression model which takes our obtained embeddings as inputs and predicts formality, frustration and politeness in emails.
We use a model which stacks a convolutional layer(5 ﬁlters of 10X5 size), a pooling layer(5X5 size with stride 5) and a dense layer(size 50, dropout 0.2).
Rectiﬁed Linear Unit(ReLU) activation is used throughout with Stochastic Gradient Descent(SGD) optimizer and a mean squared loss.
Results are provided in Table 3.
For the affect prediction task, we report the mean and standard deviation of Mean Square Error(MSE) over ﬁve different train-test splits(test–ratio:0.2) of the dataset corresponding to Formality, Frustration and Politeness.
In this case, along with GloVe and CBOW approaches, our modiﬁcations over SkipGram baseline (corpus only) show improvements with low standard deviations in MSE values.
D ISCUSSION
We ﬁnd reasonable improvements by our proposed approaches in all the task-based evaluations.
SkipGram based methods perform poorly in word similarity prediction and outlier detection, but do well on sentiment and affect prediction.
This difference in performance on downstream tasks, has been discussed before in Faruqui et al.
and Camacho-Collados & Navigli (2016), who point out various issues with word similarity based evaluations such as task subjectivity, low inter annotator agreements and low correlations between the performance of word vectors on word similarity and NLP tasks like text classiﬁcation, parsing and sentiment analysis.
Performance differences can also be attributed to corpus size, which are examined in the Appendix section. [Table] [Table]
The results suggest that different embeddings perform well for different tasks.
In word similarity tasks, the +V model performs well in GloVe setting but the +A model seems to perform the best for CBOW.
Similar results are observed in sentiment prediction: for binary sentiment prediction, arousal scores give the best performance with CBOW embeddings but dominance and valence give the best performance with skip-gram and GloVe embeddings respectively.
This suggests that the most ﬂexible method could be an ensemble implementation that considers all these inputs before [Table] [Table] predicting a ﬁnal class.
Also note that given the vocabulary of our ukWaC corpus as 569 ,words, our affect lexica with 13 , 915 words is relatively small.
We plan to take this work forward by further analysis in the future.
At the least, we expect superior word embeddings with better quality and larger affect lexica. 5. A FFECT -P OLYSEMY
Words may have different affect in different contexts.
This happens because their meanings change from one context to another.
For example ”A wicked woman came in” vs a slang ”
This game is wicked” should have different affect scores for both ”wicked” because the meaning of the word almost turns on its head (from ”devilish” to ”cool/commendable”). But datasets like Warriner et al. (2013) do not capture this as the scores attributed by human annotators inherently average out the contexts (to the best of their knowledge) in which the words tend to appear. This problem (of ”affect-polysemy”) is related to ”polysemy” which cripples today’s word-level embedding models(Gladkova & Drozd (2016), Huang et al.
Word-level vector embeddings like SkipGram combine multiple senses of a polysemous word into single embedding vector (Liu et al.
Although, there have been a few attempts to address this shortcoming using multi-prototype vectorspace models (Reisinger & Mooney (2010), Huang et al.
(2012), Tian et al.
, but they also face several challenges (Liu et al.
Since, single vector embeddings are still the most popular choice for NLP tasks, the proposed approach tries to improve the quality of these vanilla embeddings for tasks which can beneﬁt by inducing affect information.
Therefore, in this paper, we evaluate our approach on sentence/phrase level prediction tasks like sentiment prediction and tone of emails.
The underlying state-of-theart recurrent/convolutional models try to create context-aware sentence level representations from word-embeddings to predict information like sentiment, formality etc.
Our affect induced wordembeddings perform better than vanilla embeddings on these tasks thus supporting our claim.
E RROR A NALYSIS
We consider three models for analysis: GloVe baseline(corpus only approach), GloVe + Valence and
GloVe + synonyms (Bollegala et al., 2016).
Table 4 shows the true class label and predictions for a few instances from SST test set.
We discuss these results below.
For the ﬁrst two sentences, although the overall sentiment is clearly positive, all the three the models fail to capture this.
The models fail in both ﬁne-grained and binary(positive/negative) settings.
The reason for poor predictions can be attributed to the absence of any affect-sensitive words.
On the other hand, third and fourth sentences are rich with affect information.
Words like ‘entertainment’, ‘smart’, ‘sweet’ and ‘playful’ make it easier for the models to predict the positive sentiment in these two sentences.
The ﬁfth sentence has a negative overall sentiment, with a true label of -2.
Surprisingly, baseline model and Bollegala et al.
(2016) approaches fail to capture this.
However, our approach is able to make the correct prediction.
This observation can be explained as follows.
The main affect word in the sentence is “dreadful”, which is present in Warriner affect lexicon but not in synonym word pairs used by Bollegala et al.
This enables our model to get improved embedding for this word.
This infact can be observed in the neighborhood of ‘dreadful’ compared in Table 5.
The embedding for ‘dreadful’ is a lot closer to negative words for our approach than others.
We observe a similar behavior in last two sentences.
The better quality embeddings learnt for ‘gem’ and ‘hate’ help to make better predictions while other models make errors.
Including various other instances which we observe, the presence or absence or affect words in the lexicon plays a vital role.
Since we work in a joint learning framework, embeddings for other words are also improved, but this improvement is signiﬁcant for words present in the lexicon only.
The inconsistency in the performance of our models can be attributed to this along with the quality of affect lexicon itself.
C ONCLUSION
This work proposes methods to incorporate information from an affect lexicon into Word2Vec and
GloVe training process. In a nutshell, we ﬁrst use WordNet to identify word pairs in the affect lexicon which are semantically related. We deﬁne the strength of this relationship using available affect scores. Finally, we modify the training objectives to incorporate this information.
In order to evaluate our embeddings, we compare them with baseline approaches where the training completely ignores the affect information.
Our embeddings show improvements over baselines on not only
Word Similarity benchmarks but also on a more complex, Outlier Detection task.
We also do this comparison extrinsically and show that our modiﬁed embeddings perform better over prior work in predicting sentiment and predicting formality, frustration and politeness in emails.
Among models using Valence, Arousal or Dominance score lists, there is no clear winner but overall addition of valence scores does a reasonable job in almost all of the cases.
Appendix 1. Choosing an appropriate value for hyper-parameter λ :
In order to choose a suitable value for λ , we take a 100 MB sample of ukWaC corpus.
The sample has close to 20 million tokens, with a vocabulary size of 27, words, eliminating all the words having the frequency count of less than 20.
We choose a smaller corpus for tuning as it is more manageable with respect to space and time resources.
We train a Word2Vec SkipGram model on the above 100MB sample and Valence affect lists by using all the λ value from the set (0 , 0 . 5 , 1 , 2 , 10 , 100 , 1000) one by one.
To pick the most suitable value, we compare the results on word similarity task on the Rubenstein-Goodenough( RG ) dataset (Rubenstein & Goodenough, 1965).
The results are given in Figure 3.
Since λ = 2 . 0 performs the best, we ﬁx this value for all our experiments.
2. Studying the effect of corpus size :
We conduct an error analysis of the poor performance of Word2Vec
Skipgram by observing the effect of varying the corpus size.
We take different sized samples from the ukWaC corpus and report the word similarity performance on RG dataset in Figure 4.
We observe irregularities in the performance of baseline approach( λ =0).
Adding the affective information has a negative impact for corpora with sizes 2.5GB and 4.5GB while shows minor improvements over baseline for larger corpora.
This improvement over baseline is the most for a smaller, 100MB corpus.
We believe better preprocessing on the corpus should help with these non-intuitive observations.