From Sentiment Annotations to Sentiment Prediction through Discourse Augmentation (2020)

Abstract: Sentiment analysis, especially for long documents, plausibly requires methods capturing complex linguistics structures. To accommodate this, we propose a novel framework to exploit task-related discourse for the task of sentiment analysis. More specifically, we are combining the large-scale, sentiment-dependent MEGA-DT treebank with a novel neural architecture for sentiment prediction, based on a hybrid TreeLSTM hierarchical attention model. Experiments show that our framework using sentiment-related discourse augmentations for sentiment prediction enhances the overall performance for long documents, even beyond previous approaches using well-established discourse parsers trained on human annotated data. We show that a simple ensemble approach can further enhance performance by selectively using discourse, depending on the document length.

## introduction predicting whether a given word, sentence or document expresses a positive, neutral or negative sentiment is a fundamental task in natural language processing (nlp). for instance, a recent survey of text mining papers from 1992-2017 has found that out of 4, 346 papers, 467 had a sentiment analysis component @xcite . while early "bag-of-word" sentiment prediction models @xcite and their extensions @xcite already show promising results on the task, they all share one inherit limitation: due to the absence of temporal information, they are not able to fully capture the semantics (and therefore the sentiment) of long texts, where different meanings oftentimes directly emerge from the word order, underlying syntax and discourse structures. recent models for sentiment analysis address this limitation by leveraging sequential paradigms @xcite @xcite , simple hierarchical information @xcite , complex syntactic structures on sentence level @xcite or discourse structures of multi-sentential text @xcite . this paper follows the last line of aforementioned research, by developing a framework to exploit automatically generated, large-scale, domain-related discourse structures for sentiment prediction. arguably, such framework can be especially beneficial for long documents that examine positive and negative aspects of a subject matter in complex rhetorical structures, like the ones shown in figure 1 . more specifically, in this work, we generate complete and hierarchical rst-style discourse trees @xcite with leaf nodes representing clause-like document fragments, called elementary discourse units (edus) and internal tree nodes labelled with a nuclearity assignment (nucleus, satellite), encoding the importance of a node in its local subtree 1 . to incorporate these rst-style discourse structures, we employ a hybrid approach inspired by @xcite and @xcite , integrating a treelstm @xcite with the well-established hierarchical attention network model (han) @xcite . from @xcite , we further adopt a non-competitive tree attention mechanism that is shown to be more appropriate in this context 2 . aiming to enhance the task of sentiment analysis by using discourse, it seems intuitive to employ domain-related discourse structures. therefore, instead of using the standard rst-dt discourse treebank in the news domain @xcite , we decide to infer discourse structures automatically learned from sentiment annotations @xcite on our discourse-augmented yelp'13 treebank called mega-dt @xcite . this way, our framework goes from sentiment to sentiment, in the sense that the discourse structures used to improve the sentiment predictions are generated through distant supervision from sentiment itself. our hypothesis is that a parser trained on a large "silver-standard" discourse treebank automatically generated from sentiment will generate more useful discourse trees for sentiment prediction than one trained on a small and generic treebank, even if such treebank is human-annotated for rst discourse structures. in a series of experiments we show that while our novel approach to discourse-based sentiment prediction is statistically equivalent to the performance of sequential models, it does deliver substantial performance gains for long documents, where discourse plays a crucial role to reveal the sentiment of a complete document. furthermore, our experiments indicate that the performance of discoursebased sentiment prediction is significantly improved when using discourse trees generated by distant supervision on sentiment, compared to the traditionally acquired rst-dt discourse corpus. using an additional ensemble method, we can further improve the performance and, even if only by a small margin, significantly outperform individual models. ## related work this work is located at the intersection of recent approaches on discourse parsing and sentiment analysis and mostly influenced by four lines of research: ## sentiment to sentiment framework our sentiment to sentiment framework involves three phases: a phase of discourse augmentation (figure 2 (a) ), in which we follow our previous approach described in @xcite and @xcite . for each document in a corpus containing document-level sentiment annotation, we generate corresponding, task-dependent discourse trees. then, this discourse augmented sentiment treebank is used to train a discourse parser. in the second phase (figure 2 (b) ), the trained discourse parser is applied to the original corpus, using the predicted trees to train our new discourse-based sentiment predictor. finally, in the third phase (figure 2 (c )), the trained framework is applied to any new document. first, the trained discourse parser generates the discourse tree for the document. subsequently, this tree (along with the document itself) is fed to our sentiment predictor, which returns the most likely sentiment. in essence, we go from sentiment annotations to sentiment predictions through discourse augmentation. for the first phase, we briefly describe the discourse augmentation step adopted from our previous work @xcite in section 3.1. for phase two, we focus on our novel sentiment predictor in section 3.2. the inference phase is straightforward and will be limited to the description in figure 2 (c) for brevity. ## evaluation in this section, we define the experimental setup and show empirical results of our novel approach, predicting sentiment using sentiment-inspired discourse parsing in the context of previous work. we present the datasets used in this work in section 4.1. afterwards, the evaluation metrics and their intuitive justifications are mentioned in section 4.2, followed by a short description of the baselines (section 4.3). we finish the evaluation section by giving insights into our preliminary evaluations determining the system's hyper-parameters in section 4.4 and describe the final experiments and results in section 4.5. ## conclusion and future work in this work, we explore the next step along the recent line of research on discourse-inspired sentiment analysis, going from sentiment annotations to sentiment prediction through discourse augmentation. we integrate modern discourse parsing approaches into existing, sequential sentiment analysis frameworks, enhancing the model performance through the use of the large-scale mega-dt discourse dataset and a hybrid approach based on sequential and tree-based components (han combined with treelstm). our proposed approach shows to be especially beneficial when predicting sentiment for long documents containing mixed aspects, combined with complex rhetorical structures. generating a model-ensemble with a simple threshold, based on the document length, improves the overall performance, showing statistically significant results. we compare our newly developed model with the well-established han model. in future work, we plan to compare the standard docbert model @xcite and discourse-inspired versions of it, to further solidify the findings in this work. we also plan to generate other large-scale datasets according to @xcite and evaluate our model on further "silver-standard" discourse treebanks. using a neural discourse parser, such as @xcite or @xcite to train on mega-dt is another extension of this work. besides the task of sentiment analysis, extractive summarization has recently been shown to align well with discourse structures in a transformer framework @xcite , giving rise to potential improvements using the dah model on this task. as another extension, we intend to look into more sophisticated ways to ensemble the sequential-and discourse tree-based models. a sentiment annotated discourse trees for long documents table (of 10)] 9 , [to eat it.] 10 , [they were great!] 11 , [served bubbling hot, not rubbery at all, delicious sauce.] 12 , [i kept the dish] 13 , [to dip bread into just because of the sauce.] 14 , [veal -amazing.] 15 , [everything tasted fantastic.] 16 , [ok, the carrots] 17 , [that were on the side were a bit plain] 18 , [and could have been softer, but the veal itself and the sauce] 19 , [it was in, and the mushrooms and pasta.] 20 , [i left nothing on my plate.] 21 , [my husband got the same] 22 , [and also had the same impression.] 23 , [creme brulee -fantastic.] 24 , [tasted great, good texture.] 25 , [pleasantly surprised.] 26 , [my husband got the tiramisu] 27 , [and said] 28 , [it was great.] 29 , [so why the 3 stars] 30 , [when the food was so amazing?] 31 , [because of the terrible service. 1 -] 32 , [we got water.] 33 , [great.] 34 , [but our server * never * asked us] 35 , [if we wanted anything else.] 36 , [when my husband finally stopped him to ask for a glass for my father in law, a coke for] 37 , [and other drinks, our server looked very inconvenienced by it. 2 -] 38 , [didn't get to order appetizers.] 39 , [you see] 40 , [i got escargot?] 41 , [i ordered that with my meal.] 42 , [our server never asked about appetizers] 43 , [and went straight to meals.] 44 , [also, my husband was walking with our daughter] 45 , [when the ordering was starting] 46 , [and needed an extra minute.] 47 , [our server wanted to start with him.] 48 , [when asked if he could start with someone else's order,] 49 , [our server protested,] 50 , [but eventually did move on to the next person.] 51 , [you'd think] 52 , [starting at the next person was] 53 , [asking him to cut off his hand. 3 -empty glasses everywhere!] 54 , [never got or was offered a refill on my drink.] 55 , [or anyone else's.] 56 , [when my father stopped our server well] 57 , [after our meal was over] 58 , [and asked] 59 , [if i could get a coke,] 60 , [our server said] 61 , [i had never ordered one.] 62 , [well of course i hadn't.] 63 , [i never had a chance to! 4 -] 64 , [offering dessert seemed a complete afterthought.] 65 , [will i recommend this place to anyone else?] 66 , [conditionally.] 67 , [i'll make sure to tell them] 68 , [that the food was very good, but not to go] 69 , [if they want attentive service,] 70 , [are on any kind of time constraint, expect refills on their drinks,] 71 , [or are at all shy about getting a server's attention.] 72