Paper:
Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network
Cognitive NLP systemsi.e., NLP systems that make use of behavioral data augment traditional text based features with cognitive features extracted from eye-movement patterns, EEG signals, brain-imaging etc.. Such extraction of features is typically manual. We contend that manual extraction of features is not good enough to tackle text subtleties that characteristically prevail in complex classification tasks like sentiment analysis and sarcasm detection, and that even the extraction and choice of features should be delegated to the learning system. We introduce a framework to automatically extract cognitive features from the eye-movement / gaze data of human readers reading the text and use them as features along with textual features for the tasks of sentiment polarity and sarcasm detection. Our proposed framework is based on Convolutional Neural Network (CNN). The CNN learns features from both gaze and text and uses them to classify the input text. We test our technique on published sentiment and sarcasm labeled datasets, enriched with gaze information, to show that using a combination of automatically learned text and gaze features yields better classification performance over (i) CNN based systems that rely on text input alone and (ii) existing systems that rely on handcrafted gaze and textual features.
1 000
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Detection of sentiment and sarcasm in usergenerated short reviews is of primary importance for social media analysis, recommendation and dialog systems. Traditional sentiment analyzers and
sarcasm detectors face challenges that arise at lexical, syntactic, semantic and pragmatic levels (Liu and Zhang, 2012; Mishra et al., 2016c). Featurebased systems (Akkaya et al., 2009; Sharma and Bhattacharyya, 2013; Poria et al., 2014) can aptly handle lexical and syntactic challenges (e.g. learning that the word deadly conveys a strong positive sentiment in opinions such as Shane Warne is a deadly bowler, as opposed to The high altitude Himalayan roads have deadly turns). It is, however, extremely difficult to tackle subtleties at semantic and pragmatic levels. For example, the sentence “I really love my job. I work 40 hours a week to be this poor.” requires an NLP system to be able to understand that the opinion holder has not expressed a positive sentiment towards her / his job. In the absence of explicit clues in the text, it is difficult for automatic systems to arrive at a correct classification decision, as they often lack external knowledge about various aspects of the text being classified.
Mishra et al. (2016b) and Mishra et al. (2016c) show that NLP systems based on cognitive data (or simply, Cognitive NLP systems) , that of leverage eye-movement information obtained from human readers, can tackle the semantic and pragmatic challenges better. The hypothesis here is that human gaze activities are related to the cognitive processes in the brain, that combines the “external knowledge” that the a reader possesses with textual clues that she / he perceives. While incorporating behavioral information obtained from gaze-data in NLP systems is intriguing and quite plausible, especially due to the availability of low cost eye-tracking machinery (Wood and Bulling, 2014; Yamamoto et al., 2013), few methods exist for text classification and they rely on handcrafted features extracted from gaze data (Mishra et al., 2016b,c). These systems have limited capabilities due to two reasons: (a) Manually designed gaze based features may not adequately capture all
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
162
163
164
165
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
forms of textual subtleties (b) Eye-movement data is not as intuitive to analyze as text which makes the task of designing manual features more difficult. So, in this work, instead of handcrafting the gaze based and textual features, we try to learn feature representations from both gaze and textual data using Convolutional Neural Networks (CNNs). We test our technique on two publicly available datasets enriched with eye-movement information, used for binary classification tasks of sentiment polarity and sarcasm detection. Our experiments show that the automatically extracted features help to achieve significant classification performance improvement over (a) existing systems that rely on handcrafted gaze and textual features and (b) CNN based systems that rely on text input alone.
The rest of the paper is organized as follows. Section 2 discusses the motivation behind using readers’ eye-movement data in a text classification setting. Section 3 discusses on why CNNs is preferred over other available alternatives for feature extraction. The CNN architecture is proposed and discussed in Section 4. Section 5 describes our experimental setup and results are discussed in Section 6. We provide a detailed analysis of the results along with some insightful observations in Section 7. Section 8 points to relevant literature followed by Section 9 that concludes the paper. Terminology: A fixation is a relatively long stay of gaze on a visual object (such as words in text) where as a sacccade corresponds to quick shifting of gaze between two positions of rest. Forward and backward saccades are called progressions and regressions respectively. A scanpath is a line graph that contains fixations as nodes and saccades as edges.
2 Eye-movement and Linguistic Subtleties
Presence of linguistic subtleties often induces (a) surprisal (Kutas and Hillyard, 1980; Malsburg et al., 2015), due to the underlying disparity /context incongruity or (b) higher cognitive load (Rayner and Duffy, 1986), due to the presence of lexically and syntactically complex structures. While surprisal accounts for irregular saccades (Malsburg et al., 2015), higher cognitive load results in longer fixation duration (Kliegl et al., 2004).
Mishra et al. (2016b) find that presence of sarcasm in text triggers either irregular saccadic patterns or unusually high duration fixations than non-sarcastic texts (illustrated through example scanpath representations in Figure 1). For sentiment bearing texts, highly subtle eye-movement patterns are observed for semantically / pragmatically complex negative opinions (expressing irony, sarcasm, thwarted expectations etc.) than the simple ones (Mishra et al., 2016b). The association between linguistic subtleties and eye-movement patterns could be captured through sophisticated feature engineering that considers both gaze and text inputs. In our work, CNNs take the onus of feature engineering.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
T ex
t C
om p
on en
t
Non-static
Static
Saccade
Fixation
G az
e C
om p
on en t P1 P2 P3 P4 P5 P6 P7 P8
N×K representation of sentences with static and non static channels
P×G representation of sentences
with fixation and saccade channels
1-D convolution operation with multiple filter width
and feature maps
2-D convolution operation with multiple filter row and
Column widths
Max-pooling for each filter width
Max-pooling over
multiple dimensions for
multiple filter widths
Fully connected with dropouts and softmax
output
Merged pooled values
Figure 3: Deep convolutional model for feature extraction from both text and gaze inputs
3 Why Convolutional Neural Network?
CNNs have been quite effective in learning filters for image processing tasks, filters being used to transform the input image into more informative feature space (Krizhevsky et al., 2012). Filters learned at various CNN layers are quite similar to handcrafted filters used for detection of edges, contours and removal of redundant backgrounds. We believe, a similar technique can also be applied to eye-movement data, where the learned filters will, hopefully, extract informative cognitive features. For instance, for sarcasm, we expect the network to learn filters that detect long distance saccades (refer to Figure 2 for an analogical illustration). With more number of convolution filters of different dimensions, the network may extract multiple features related to different gaze attributes (such as fixations, progressions, regressions and skips) and will be free from any form of human bias that manually extracted features are susceptible to.
4 Learning Feature Representations: The CNN architecture
Figure 3 shows the CNN architecture with two components for processing and extracting features from text and gaze inputs. The components are explained below.
4.1 Text Component
The text component is quite similar to the one proposed by Kim (2014) for sentence classification.
Words (in the form of one-hot representation) in the input text are first replaced by their embeddings of dimension K (ith word in the sentence represented by an embedding vector xi ∈ RK). As per Kim (2014), a multi-channel variant of CNN (referred to as MULTICHANNELTEXT) can be implemented by using two channels of embeddingsone that remains static through out training (referred to as STATICTEXT), and the other one that gets updated during training (referred to as NONSTATICTEXT). We separately experiment with static, non-static and multi-channel variants.
For each possible input channel of the text component, a given text is transformed into a tensor of fixed length N (padded with zero-tensors wherever necessary to tackle length variations) by concatenating the word embeddings.
x1:N = x1 ⊕ x2 ⊕ x3 ⊕ ...⊕ xN (1)
where ⊕ is the concatenation operator. To extract local features1, convolution operation is applied. Convolution operation involves a filter, W ∈ RHK , which is convolved with a window of H embeddings to produce a local feature for the H words. A local feature, ci is generated from a window of embeddings xi:i+H−1 by applying a non linear function (such as a hyperbolic tangent) over the convoluted output. Mathematically,
ci = f(W.xi:i+H−1 + b) (2)
1features specific to a region in case of images or window of words in case of text
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
where b ∈ R is the bias and f is the non-linear function. This operation is applied to each possible window of H words to produce a feature map (c) for the window size H .
c = [c1, c2, c3, ..., cN−H+1] (3)
A global feature is then obtained by applying max pooling operation2 (Collobert et al., 2011) over the feature map. The idea behind max-pooling is to capture the most important feature - one with the highest value - for each feature map.
We have described the process by which one feature is extracted from one filter (for illustration, red bordered portions in Figure 3 depict the case of H = 2). The model uses multiple filters (with varying window sizes) to obtain multiple features representing the text. In the MULTICHANNELTEXT variant, for a window of H words, convolution operation is separately applied on both the embedding channels. Local features learned from both the channels are concatenated before applying max-pooling.
4.2 Gaze Component
The gaze component deals with scanpaths of multiple participants annotating the same text. Scanpaths can be pre-processed to extract two sequences of gaze data to form separate channels of input: (1) A sequence of normalized3 durations of fixations (in milliseconds) in the order in which they appear in the scanpath and (2) A sequence of position of fixations (in terms of word id) in the order in which they appear in the scanpath. These channels are related to two fundamental gaze attributes such as fixation and saccade respectively. With two channels, we thus have three possible configurations of the gaze component such as (i) FIXATION, where the input is normalized fixation duration sequence, (ii) SACCADE, where the input is fixation position sequence, and (iii) MULTICHANNELGAZE, where both the inputs channels are considered.
For each possible input channel, the input is in the form of a P × G matrix (with P → number of participants and G → length of the input sequence). Each element of the matrix gij ∈ R, with i ∈ P and j ∈ G, corresponds to the jth gaze attribute (either fixation duration or word id, depending on the channel) of the input sequence of
2mean pooling does not perform well. 3scaled across participants using min-max normalization
to reduce subjectivity
the ith participant. Now, unlike the text component, here we apply convolution operation across two dimensions i.e. choosing a two dimensional convolution filter W ∈ RJK (for simplicity, we have kept J = K, thus , making the dimension of W , J2). For the dimension size of J2, a local feature cij is computed from the window of gaze elements gij:(i+J−1)(j+J−1) by,
cij = f(W.gij:(i+J−1)(j+J−1) + b) (4)
where b ∈ R is the bias and f is a non-linear function. This operation is applied to each possible window of size J2 to produce a feature map (c),
c =[c11, c12, c13, ..., c1(G−J+1),
c21, c22, c23, ..., c2(G−J+1), ...,
c(P−J+1)1, c(P−J+1)2, ..., c(P−J+1)(G−J+1)]
(5)
A global feature is then obtained by applying max pooling operation. Unlike the text component, max-pooling operator is applied to a 2D window of local features size M × N (for simplicity, we set M = N , denoted henceforth as M2). For the window of size M2, the pooling operation on c will result in as set of global features ĉJ = max{cij:(i+M−1)(j+M−1)} for each possible i, j.
We have described the process by which one feature is extracted from one filter (of 2D window size J2 and max-pooling window size of M2). In Figure 3, red and blue bordered portions illustrate the cases of J2 = [3, 3] and M2 = [2, 2] respectively. Like the text component, the gaze component uses multiple filters (also with varying window size) to obtain multiple features representing the gaze input. In the MULTICHANNELGAZE variant, for a 2D window of J2, convolution operation is separately applied on both fixation duration and saccade channels and local features learned from both the channels are concatenated before max-pooling is applied.
Once the global features are learned from both the text and gaze components, they are merged and passed to a fully connected feed forward layer (with number of units set to 150) followed by a SoftMax layer that outputs the the probabilistic distribution over the class labels.
The gaze component of our network is not invariant of the order in which the scanpath data is given as input- i.e., the P rows in the P × G can not be shuffled, even if each row is independent from others. The only way we can think of for
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
addressing this issue is by applying convolution operations to all P × G matrices formed with all the permutations of P , capturing every possible ordering. Unfortunately, this makes the training process significantly less scalable, as the number of model parameters to be learned becomes huge. As of now, training and testing are carried out by keeping the order of the input constant.
5 Experiment Setup
We now share several details regarding our experiments below. 1. Dataset: We experiment on sentiment and sarcasm tasks using two publicly available datasets enriched with eye-movement information. Dataset 1 has been released by Mishra et al. (2016a). It contains 994 text snippets with 383 positive and 611 negative examples. Out of the 994 snippets, 350 are sarcastic. Dataset 2 has been used by Joshi et al. (2014) and it consists of 843 snippets comprising movie reviews and normalized tweets out of which 443 are positive and 400 are negative. Eye-movement data of 7 and 5 readers is available for each snippet for dataset 1 and 2 respectively. 2. CNN Variants: With text component alone we have three variants such as STATICTEXT, NONSTATICTEXT and MULTICHANNELTEXT (refer to Section 4.1). Similarly, with gaze component we have variants such as FIXATION, SACCADE and MULTICHANNELGAZE (refer to Section 4.2). With both text and gaze components, 9 more variants could be experimented with. 3. Hyper-parameters: For text component, we experiment with filter widths (H) of [3, 4]. For the gaze component, 2D filters (J2) set to [3 × 3], [4× 4] respectively. The max pooling 2D window, M2, is set to [2 × 2]. In both gaze and text components, number of filters is set to 150, resulting in 150 feature maps for each window. These model hyper-parameters are fixed by trial and error and are possibly good enough to provide a first level insight into our system. Tuning of hyperparameters might help in improving the performance of our framework, which is on our future research agenda. 4. Regularization: For regularization dropout is employed on the penultimate layer with a constraint on l2-norms of the weight vectors (Hinton et al., 2012). Dropout prevents co-adaptation of hidden units by randomly dropping out - i.e., setting to zero - a proportion p of the hidden units
during forward propagation. We set p to 0.25. 5. Training: We use ADADELTA optimizer (Zeiler, 2012), with a learning rate of 0.1. The input batch size is set to 32 and number of training iterations (epochs) is set to 200. 10% of the training data is used for validation. 6. Use of pre-trained embeddings: Initializing the embedding layer with of pre-trained embeddings can be more effective than random initialization (Kim, 2014). In our experiments, we have used embeddings using word2vec facilitated by Mikolov et al. (2013) (best results obtained with embedding dimension of 50). We have also tried randomly initializing the embeddings but better results are obtained with pre-trained embeddings. 7. Comparison with existing work: For sentiment analysis, we compare our systems’s accuracy (for both datasets 1 and 2) with Mishra et al. (2016c)’s systems that rely on handcrafted text and gaze features. For sarcasm detection, we compare Mishra et al. (2016b)’s sarcasm classifier with ours using dataset 1 (with available gold standard labels for sarcasm). We follow the same 10-fold traintest configuration as these existing works for consistency.
6 Results
In this section, we discuss the results for different model variants for sentiment polarity and sarcasm detection tasks.
6.1 Results for Sentiment Analysis Task
Table 1 presents results for sentiment analysis task. For dataset 1, different variants of our CNN architecture outperform the best systems reported by Mishra et al. (2016c), with a maximum F-score improvement of 3.8%. This improvement is statistically significant of p < 0.05 as confirmed by McNemar test. Moreover, we observe an F-score improvement of around 5% for CNNs with both gaze and text components as compared to CNNs with only text components (similar to the system by Kim (2014)), which is also statistically significant (with p < 0.05).
For dataset 2, CNN based approaches do not perform better than manual feature based approaches. However, variants with both text and gaze components outperform the ones with only text component (Kim, 2014), with a maximum Fscore improvement of 2.9%. We observe that for dataset 2, training accuracy reaches 100 within
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Dataset1 Dataset2
Configuration P R F P R F
Traditional systems based on
Näive Bayes 63.0 59.4 61.14 50.7 50.1 50.39 Multi-layered Perceptron 69.0 69.2 69.2 66.8 66.8 66.8
textual features SVM (Linear Kernel) 72.8 73.2 72.6 70.3 70.3 70.3 Systems by Mishra et al. (2016c) Gaze based (Best) 61.8 58.4 60.05 53.6 54.0 53.3 Text + Gaze (Best) 73.3 73.6 73.5 71.9 71.8 71.8
CNN with only text input (Kim, 2014) STATICTEXT 63.85 61.26 62.22 55.46 55.02 55.24 NONSTATICTEXT 72.78 71.93 72.35 60.51 59.79 60.14 MULTICHANNELTEXT 72.17 70.91 71.53 60.51 59.66 60.08
CNN with only gaze Input
FIXATION 60.79 58.34 59.54 53.95 50.29 52.06 SACCADE 64.19 60.56 62.32 51.6 50.65 51.12 MULTICHANNELGAZE 65.2 60.35 62.68 52.52 51.49 52
CNN with both text and gaze Input
STATICTEXT + FIXATION 61.52 60.86 61.19 54.61 54.32 54.46 STATICTEXT + SACCADE 65.99 63.49 64.71 58.39 56.09 57.21 STATICTEXT + MULTICHANNELGAZE 65.79 62.89 64.31 58.19 55.39 56.75 NONSTATICTEXT + FIXATION 73.01 70.81 71.9 61.45 59.78 60.60 NONSTATICTEXT + SACCADE 77.56 73.34 75.4 65.13 61.08 63.04 NONSTATICTEXT + MULTICHANNELGAZE 79.89 74.86 77.3 63.93 60.13 62 MULTICHANNELTEXT + FIXATION 74.44 72.31 73.36 60.72 58.47 59.57 MULTICHANNELTEXT + SACCADE 78.75 73.94 76.26 63.7 60.47 62.04 MULTICHANNELTEXT + MULTICHANNELGAZE 78.38 74.23 76.24 64.29 61.08 62.64
Table 1: Results for different traditional feature based systems and CNN model variants for the task of sentiment analysis. Abbreviations (P,R,F)→ Precision, Recall, F-score. SVM→Support Vector Machine
25 epochs with validation accuracy stable around 50%, indicating the possibility of overfitting. Tuning the regularization parameters specific to dataset 2 may help here. Even though CNN might not be proving to be a choice as good as handcrafted features for dataset 2, the bottom line remains that incorporation of gaze data into CNN consistently improves the performance over onlytext-based CNN variants.
6.2 Results for Sarcasm Detection Task
For sarcasm detection, our CNN model variants outperform traditional systems by a maximum margin of 11.27% (Table 2). However, the improvement by adding the gaze component to the CNN network is just 1.36%, which is statistically insignificant over CNN with text component. While inspecting the sarcasm dataset, we observe a clear difference between the vocabulary of sarcasm and non-sarcasm classes in our dataset. This, perhaps, was captured well by the text component, especially the variant with only non-static embeddings.
7 Discussion
In this section, some important observations from our experiments are discussed. • Effect of embedding dimension variation: Embedding dimension has proven to have a deep impact on the performance of neural systems (dos
Santos and Gatti, 2014; Collobert et al., 2011). We repeated our experiments by varying the embedding dimensions in the range of [50-300]4 and observed that reducing embedding dimension improves the F-scores by a little margin. Best results are obtained when the embedding dimension is as low as 50. Small embedding dimensions are probably reducing the chances of over-fitting when the data size is small. We also observe that for different embedding dimensions, performance of CNN with both gaze and text components is consistently better than that with only text component. • Effect of static / non static text channels: Nonstatic embedding channel has a major role in tuning embeddings for sentiment analysis by bringing adjectives expressing similar sentiment close to each other (e.g, good and nice), where as static channel seems to prevent over-tuning of embeddings (over-tuning often brings verbs like love closer to the pronoun I in embedding space, purely due to higher co-occurrence of these two words in sarcastic examples). • Effect of fixation / saccade channels: For sentiment detection, saccade channel seems to be handing text having semantic incongruity (due to the presence of irony / sarcasm) better. Fixation channel does not help much, may be because of higher variance in fixation duration. For sarcasm
4a standard range (Liu et al., 2015; Melamud et al., 2016)
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Configuration P R F
Traditional systems based on
Näive Bayes 69.1 60.1 60.5 Multi-layered Perceptron 69.7 70.4 69.9
textual features SVM (Linear Kernel) 72.1 71.9 72 Systems by Riloff et al. (2013) Text based (Ordered) 49 46 47 Text + Gaze (Unordered) 46 41 42
System by Joshi et al. (2015)
Text based (best) 70.7 69.8 64.2
Systems by Mishra et al. (2016b)
Gaze based (Best) 73 73.8 73.1 Text based (Best) 72.1 71.9 72 Text + Gaze (Best) 76.5 75.3 75.7
CNN with only text input (Kim, 2014) STATICTEXT 67.17 66.38 66.77 NONSTATICTEXT 84.19 87.03 85.59 MULTICHANNELTEXT 84.28 87.03 85.63
CNN with only gaze input
FIXATION 74.39 69.62 71.93 SACCADE 68.58 68.23 68.40 MULTICHANNELGAZE 67.93 67.72 67.82
CNN with both text and gaze Input
STATICTEXT + FIXATION 72.38 71.93 72.15 STATICTEXT + SACCADE 73.12 72.14 72.63 STATICTEXT + MULTICHANNELGAZE 71.41 71.03 71.22 NONSTATICTEXT + FIXATION 87.42 85.2 86.30 NONSTATICTEXT + SACCADE 84.84 82.68 83.75 NONSTATICTEXT + MULTICHANNELGAZE 84.98 82.79 83.87 MULTICHANNELTEXT + FIXATION 87.03 86.92 86.97 MULTICHANNELTEXT + SACCADE 81.98 81.08 81.53 MULTICHANNELTEXT + MULTICHANNELGAZE 83.11 81.69 82.39
Table 2: Results for different traditional feature based systems and CNN model variants for the task of sarcasm detection on dataset 1. Abbreviations (P,R,F)→ Precision, Recall, F-score
detection, fixation and saccade channels perform with similar accuracy when employed separately. Accuracy reduces with gaze multichannel, may be because of higher variation of both fixations and saccades across sarcastic and non-sarcastic classes, as opposed to sentiment classes.
• Effectiveness of the CNN learned features To examine how good the features learned by the CNN are, we analyzed the features for a few example cases. Figure 4 presents some of the testexamples for the task of sarcasm detection. Example 1 contains sarcasm while examples 2, 3 and 4 are non-sarcastic. To see if there is any difference in the automatically learned features between text-only and combined text and gaze variants, we examine the feature vector (of dimension 150) for the examples obtained from different model variants. Output of the hidden layer after merge layer is considered as features learned by the network. We plot the features, in the form of color-bars, following Li et al. (2016) - denser colors representing higher feature values. In Figure 4, we show only two (representative) model variants viz., MULTICHANNELTEXT and MULTICHANNELTEXT+ MULTICHANNELGAZE. As one can see, addition of gaze information helps
to generate features with more subtle differences (marked by blue rectangular boxes) for sarcastic and non-sarcastic texts. It is also interesting to note that in the marked region, features for the sarcastic texts exhibit more intensity than the nonsarcastic ones - perhaps capturing the notion that sarcasm typically conveys an intensified negative opinion. This difference is not clear in feature vectors learned by text only systems for instances like example 2, which has been incorrectly classified by MULTICHANNELTEXT. Example 4 is incorrectly classified by both the systems, perhaps due to lack of context. In cases like this, addition of gaze information does not help much in learning more distinctive features, as it becomes difficult for even humans to classify such texts.
8 Related Work
Sentiment and sarcasm classification are two important problems in NLP and have been the focus of research for many communities for quite some time. Popular sentiment and sarcasm detection systems are feature based and are based on unigrams, bigrams etc. (Dave et al., 2003; Ng et al., 2006), syntactic properties (Martineau and Finin, 2009; Nakagawa et al., 2010), semantic properties
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
1. I would like to live in Manchester, England. The transition between Manchester and death would be unnoticeable. (Sarcastic, Negative Sentiment)
2. We really did not like this camp. After a disappointing summer, we switched to another camp, and all of us much happier on all fronts! (Non Sarcastic, Negative Sentiment)
3. Helped me a lot with my panics attack I take 6 mg a day for almost 20 years can't stop of course but make me feel very comfortable (Non Sarcastic, Positive Sentiment)
4. Howard is the King and always will be, all others are weak clones. (Non Sarcastic, Positive Sentiment)
(a) MultichannelText + MultichannelGaze (b) MultichannelText
Figure 4: Visualization of representations learned by two variants of the network for sarcasm detection task. The output of the Merge layer (of dimension 150) are plotted in the form of colour-bars. Plots with thick red borders correspond to wrongly predicted examples.
(Balamurali et al., 2011). For sarcasm detection, supervised approaches rely on (a) Unigrams and Pragmatic features (González-Ibánez et al., 2011; Barbieri et al., 2014; Joshi et al., 2015) (b) Stylistic patterns (Davidov et al., 2010) and patterns related to situational disparity (Riloff et al., 2013) and (c) Hastag interpretations (Liebrecht et al., 2013; Maynard and Greenwood, 2014). Recent systems are based on variants of deep neural network built on the top of embeddings. A few representative works in this direction for sentiment analysis are based on CNNs (dos Santos and Gatti, 2014; Kim, 2014; Tang et al., 2014), RNNs (Dong et al., 2014; Liu et al., 2015) and combined architecture (Wang et al., 2016). Few works exist on using deep neural networks for sarcasm detection, one of which is by (Ghosh and Veale, 2016) that uses a combination of RNNs and CNNs.
Eye-tracking technology is a relatively new NLP, with a very few systems directly making use of gaze data in prediction frameworks. Klerke et al. (2016) present a novel multi-task learning approach for sentence compression using labeled data, while, Barrett and Søgaard (2015) discriminate between grammatical functions using gaze features. The closest works to ours is by Mishra et al. (2016b) and Mishra et al. (2016c) that introduce feature engineering based on both gaze
and text data for sentiment and sarcasm detection tasks. These recent advancements motivate us to explore the cognitive NLP paradigm .
9 Conclusion and Future Directions
In this work, we proposed a multimodal ensemble of features, automatically learned using variants of CNNs from text and readers’ eye-movement data, for the tasks of sentiment and sarcasm classification. On multiple published datasets for which gaze information is available, our systems could achieve significant performance improvements over (a) systems that rely on handcrafted gaze and textual features and (b) CNN based systems that rely on text input alone. An analysis of the learned features confirms that the combination of automatically learned features is indeed capable of representing deep linguistic subtleties in text that pose challenges to sentiment and sarcasm classifiers. Our future agenda includes: (a) optimizing the CNN framework hyper-parameters (e.g., filter width, dropout, embedding dimensions etc.) to obtain better results, (b) exploring the applicability of our technique for document-level sentiment analysis and (c) applying our framework on related problems, such as emotion analysis, text summarization and question-answering.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Review:
- Strengths:  This paper tackles an interesting problem and provides a (to my knowledge) novel and reasonable way of learning and combining cognitive features with textual features for sentiment analysis and irony detection. The paper is  clearly written and organized, and the authors provided a lot of useful detail and informative example and plots. Most of the results are convincing, and the authors did a good job comparing their approach and results with previous work.  - Weaknesses:  1. Just from the reading abstract, I expected that the authors' approach would significantly outperform previous methods, and that using both the eye-gaze and textual features consistently yields the best results. Upon reading the actual results section, however, it seems like the findings were more mixed. I think it would be helpful to update the abstract and introduction to reflect this.  2. When evaluating the model on dataset 1 for sentiment analysis, were the sarcastic utterances included? Did the model do better on classifying the non-sarcastic utterances than the sarcastic ones? 3. I understand why the eye-movement data would be useful for sarcasm detection, but it wasn't as obvious to me why it would be helpful for (non-sarcastic) sentiment classification beyond the textual features.   - General Discussion:  This paper contains a lot of interesting content, and the approach seems solid and novel to me. The results were a little weaker than I had anticipated from the abstract, but I believe would still be interesting to the larger community and merits publication.