Multi-task Learning of Negation and Speculation for Targeted Sentiment Classification (2021)

Abstract: The majority of work in targeted sentiment analysis has concentrated on finding better methods to improve the overall results. Within this paper we show that these models are not robust to linguistic phenomena, specifically negation and speculation. In this paper, we propose a multi-task learning method to incorporate information from syntactic and semantic auxiliary tasks, including negation and speculation scope detection, to create English-language models that are more robust to these phenomena. Further we create two challenge datasets to evaluate model performance on negated and speculative samples. We find that multi-task models and transfer learning via language modelling can improve performance on these challenge datasets, but the overall performances indicate that there is still much room for improvement. We release both the datasets and the source code at https://github.com/jerbarnes/multitask_negation_for_targeted_sentiment .

## introduction targeted sentiment analysis (tsa) involves jointly predicting entities which are the targets of an opinion, as well as the polarity expressed towards them @xcite . the tsa task, which is part of the larger set of fine-grained sentiment analysis tasks, can enable companies to provide better recommendations @xcite , as well as give digital humanities scholars a quantitative approach to identifying how sentiment and emotions develop in literature @xcite . modelling tsa has moved from sequence labeling using conditional random fields (crfs) @xcite or recurrent neural networks (rnn) @xcite @xcite , to transformer models @xcite . however, all these improvements have concentrated on making the best of the relatively small task-specific datasets. as annotation for fine-grained sentiment is difficult and often has low inter-annotator agreement @xcite , this data tends to be small and of varying quality. this lack of high-quality training data prevents tsa models from learning complex, compositional linguistic phenomena. for sentence-level sentiment classification, incorporating compositional information from relatively small amounts of negation or speculation data improves both robustness and general performance @xcite @xcite . furthermore, transfer learning via language-modelling also improves fine-grained sentiment analysis @xcite . in this paper, we wish to explore two research questions: we explore a multi-task learning (mtl) approach to incorporate auxiliary task information in targeted sentiment classifiers in english in order to investigate the effects of negation and speculation in detail, we also annotate two new challenge datasets which contain negated and speculative examples. we find that the performance is negatively affected by negation and speculation, but mtl and transfer learning (tl) models are more robust than single task learning (stl). tl reduces the improvements of mtl, suggesting that tl is similarly effective at learning negation and speculation. the overall performance on the challenge datasets, however, confirms that there is still room for improvement. the contributions of the paper are the following: i) we introduce two english challenge datasets annotated for negation and speculation, ii) we propose a multi-task model to incorporate negation and speculation information and evaluate it across four english datasets, iii) finally, using the challenge datasets, we show the quantitative effect of negation and speculation on tsa. english tsa datasets include mpqa @xcite , the semeval laptop and restaurant reviews @xcite , and twitter datasets @xcite . further annotation projects have led to review datasets for arabic, dutch, french, russian, and spanish @xcite and twitter datasets for spanish @xcite and turkish @xcite . prior work has also explored the effects of different phenomena on tsa through error analysis and challenge datasets. @xcite , @xcite @xcite showed the difficulties of polarity classification of targets on texts with multiple different polarities through the distinct sentiment error splits, the hard split, and the mams challenge dataset respectively. both @xcite and @xcite augment document sentiment datasets by asking annotators to create counterfactual examples for the imdb dataset. more recently, @xcite showed how sentence-level sentiment models are affected by various linguistic phenomena including negation, semantic role labelling, temporal changes, and name entity recognition. previous approaches to modelling tsa have often relied on general sequence labelling models, e. g. crfs @xcite , probabilistic graphical models (klinger and cimiano, 2013), rnns @xcite , and more recently pretrained transformer models @xcite . ## data we perform the main experiments on four english language datasets: the laptop dataset from se-meval 2014 @xcite , the restaurant dataset which combines the semeval 2014 @xcite , 2015 @xcite , and 2016 @xcite , the multi-aspect multisentiment (mams) dataset @xcite , and finally the multi-perspective question answering (mpqa) dataset @xcite foot_0 shows the distribution of the sentiment classes . we take the pre-processed laptop and restaurant datasets from @xcite , and use the train, dev, and test splits that they provide. we use the nltk word tokenizer to tokenise the laptop, restaurant, and mpqa datasets and spacy for the mams dataset. we choose datasets that differ largely in their domain, size, and annotation style in order to determine if any trends we see are robust to these data characteristics or whether they are instead correlated. we convert all datasets to a targeted setup by extracting only the aspect targets and their polarity. we use the unified tagging scheme foot_1 following recent work @xcite and convert all data to bioul format foot_2 with unified sentiment tags, e. g. b-pos for a beginning tag with a positive sentiment, so that we can cast the tsa problem as a sequence labeling task. the statistics for these datasets are shown in ## experiments we experiment with a single task baseline (stl) and a hierarchical multi-task model with a skipconnection (mtl), both of which are shown in figure 1 . for the stl model, we first embed a sentence and then pass the embeddings to a bidirectional lstm (bi-lstm). these features are then concatenated to the input embeddings and fed to the second bi-lstm layer, ending with the token-wise sentiment predictions from the crf tagger. for the mtl model, we additionally use the output of the first bi-lstm layer as features for the separate auxiliary task crf tagger. as seen from figure 1 , the stl model and the mtl main task model use the same the green layers. the mtl additionally uses the pink layer for the auxiliary task, adding less than 3.4% trainable parameters foot_7 for all auxiliary tasks except lex, which adds 221.4% due to the large label set (see training: for the stl and the mtl models, we tune hyperparameters using allentune @xcite on the laptop development dataset. we then use the best hyperparameters on the laptop dataset for all the stl and mtl experiments, in order to reduce hyperparameter search. we follow the result checklist for hyperparameter searches from @xcite (details found in tables 17 and 18 of appendix e along with figure 2 showing the expected validation scores from the hyperparameter tuning). for the mtl model, a single epoch involves training for one epoch on the auxiliary task and then an epoch on the main task, as previous work has shown training the lower-level task first improves overall results @xcite . in this work, we assume all of the auxiliary training tasks are conceptually lower than tsa. evaluation: for all experiments, we run each model five times @xcite and report the mean and standard derivation. we also take the distribution of the five runs to perform significance testing @xcite , eliminating the need for bonferroni correction. following @xcite , we use the nonparametric wilcoxon signed-rank test @xcite for the f 1 metrics and a more powerful parametric welch's t-test @xcite for the accu- ## challenge dataset results in order to isolate the effects of negation and speculation on the results, we test all models trained on the original laptop and restaurant datasets on the laptop n eg , restaurant n eg , laptop spec , and restaurant spec test splits. tables 5 and 6 show the results for negation and speculation, respectively. the results for the dev split and the f 1 -s of the test split are shown in appendix d. firstly, all models perform comparatively worse on the challenge datasets, dropping an average of 24 and 25 pp on f 1 -i on the negation and speculation data, respectively. nearly all of this drop comes from poorer classification (acc-s, f 1s), while target extraction (f 1 -a) is relatively stable. this demonstrates the importance of resolving negation and speculation for tsa and the usefulness of the annotated data to determine these effects. on laptop n eg and restaurant n eg incorporating negation auxiliary tasks gives an average improvement of 3.8 pp on the f 1 -i metric when using glove embeddings. more specifically, mtl with negation improves the sentiment classification scores, but does not help extraction. this makes sense conceptually, as negation has little effect on whether or not a word is part of a sentiment target. instead, the best mtl glove models on the original datasets (lex 15 and dr, respectively) also outper15 the development f1-i result for lex on the laptop form the stl glove models on the challenge data, indicating that mtl leads to greater robustness. when comparing the stl model using glove and tl on average the model improves by 9.55 pp on the negation dataset compared to 3.65 pp for the speculation suggesting that transfer learning is less effective for speculation. ## conclusion finally, in order to improve reproducibility and to encourage further work, we release the code foot_9 , dataset, and trained models associated with this paper, hyperparameter search details with compute infrastructure (appendix e), number of parameters and runtime details (appendix f), and further detailed dev and test results (appendices c and d), in line with the result checklist from @xcite . allyson ettinger. 2020. what bert is not: lessons from a new suite of psycholinguistic diagnostics for language models. transactions of the association for computational linguistics, 8:34-48. matt gardner, yoav artzi, victoria basmova, jonathan berant, ben bogin, sihao chen, pradeep dasigi, dheeru dua, yanai elazar, ananth gottumukkala, nitish gupta, hanna hajishirzi, gabriel ilharco, daniel khashabi, kevin lin, jiangming liu, nelson f. liu, phoebe mulcaire, qiang ning, sameer singh, noah a. smith, sanjay subramanian, reut tsarfaty, eric wallace, ally zhang, and ben zhou. 2020. evaluating nlp models via contrast sets. kazuma hashimoto, caiming xiong, yoshimasa tsuruoka, and richard socher. 2017. a joint many-task model: growing a neural network for multiple nlp tasks. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 1923-1933, copenhagen, denmark. association for computational linguistics. ruidan he, wee sun lee, hwee tou ng, and daniel dahlmeier. 2019. an interactive multi-task learning network for end-to-end aspect-based sentiment analysis. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 504-515, florence, italy. association for computational linguistics. minghao hu, yuxing peng, zhen huang, dongsheng li, and yiwei lv. 2019. open-domain targeted sentiment analysis via span-based extraction and classification. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 537-546, florence, italy. association for computational linguistics. qingnan jiang, lei chen, ruifeng xu, xiang ao, and min yang. 2019. a challenge dataset and effective models for aspect-based sentiment analysis. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6279-6284, hong kong, china. association for computational linguistics. wei jin and hung hay ho. 2009. a novel lexicalized hmm-based learning framework for web opinion mining. in proceedings of the 26th annual international conference on machine learning, icml '09, page 465-472, new york, ny, usa. association for computing machinery. nora kassner and hinrich schütze. 2020. negated and misprimed probes for pretrained language models: birds can talk, but cannot fly. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7811-7818, online. association for computational linguistics. arzoo katiyar and claire cardie. 2016. investigating lstms for joint extraction of opinion entities and relations. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 919-929, berlin, germany. association for computational linguistics. divyansh kaushik, eduard hovy, and zachary lipton. 2020. learning the difference that makes a difference with counterfactually-augmented data. in international conference on learning representations. evgeny kim and roman klinger. 2019. frowning frodo, wincing leia, and a seriously great friendship: learning to classify emotional relationships of fictional characters. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 647-653, minneapolis, minnesota. association for computational linguistics. jin-dong kim, yue wang, and yamamoto yasunori. 2013. the genia event extraction shared task, 2013 edition -overview. in proceedings of the bionlp shared task 2013 workshop, pages 8-15, sofia, bulgaria. association for computational linguistics. roman klinger and philipp cimiano. 2013. bidirectional inter-dependencies of subjective expressions and targets and their value for a joint model. in proceedings of the 51st annual meeting of the association for computational linguistics (volume 2: short papers), pages 848-854. natalia konstantinova, sheila c.m. de sousa, noa p. cruz, manuel j. maña, maite taboada, and ruslan mitkov. 2012. a review corpus annotated for negation, speculation and their scope. in proceedings of the 8th international conference on language resources and evaluation, pages 3190-3195, istanbul, turkey. meghana kshirsagar, sam thomson, nathan schneider, jaime carbonell, noah a. smith, and chris dyer. 2015. frame-semantic role labeling with heterogeneous annotations. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing, pages 218-224, beijing, china. emanuele lapponi, jonathon read, and lilja øvrelid. 2012. representing and resolving negation for sentiment analysis. in proceedings of the 2012 ieee 12th international conference on data mining workshops, pages 687-692, washington, dc, usa. xin li, lidong bing, piji li, and wai lam. 2019a. a unified model for opinion target extraction and target sentiment prediction. in proceedings of the aaai conference on artificial intelligence, volume 33, pages 6714-6721. xin li, lidong bing, wenxuan zhang, and wai lam. 2019b. exploiting bert for end-to-end aspectbased sentiment analysis. in proceedings of the 5th you might not like the service neg cd b scope i scope b cue b scope i scope i scope neg sf u b scope i scope b cue b scope i scope i scope spec b scope b cue b scope i scope i scope i scope upos pron aux part verb det noun dr nsubj aux advmod root det obj lex o p ron o au x o adv b v -v.emotion o det b n -n.act 17 and 18 for the stl and mtl models respectively. the shaded areas represent the expected @xmath2. note the shaded area does not go beyond the maximum observed validation score as recommended @xcite . the number of parameters is different for the mtl models depending on whether the parameters from the auxiliary task are included or not. the auxiliary task specific layer is shown as the pink layer in figure 1 . the number of parameters including and not including the auxiliary task is stated as the mtl models at inference time would not use the auxiliary task parameters. there are many more trainable parameters for the mtl models ignoring the auxiliary task parameters. this is because the hyperparameter search finds a larger shared lstm hidden dimension to be preferable for the mtl models (see tables 17 and 18 ). for the glove mtl models the total number of parameters changes depending on the auxiliary task. this is because the glove embeddings contain different numbers of vocabulary words, as we filter words based on those in the auxiliary and main task datasets/corpora. the large difference in the number of trainable parameters between glove and tl models is due to the fact that the tl is 724 parameters larger than the 300 parameter glove embeddings. lastly, the number of trainable parameters is dataset agnostic, the number of all parameters is not dataset agnostic for the glove models due to the vocabulary size, for clarification the model parameters reported here are for those trained on the laptop dataset.