Comparing Transformer and Dictionary-based Sentiment Models for Literary Texts: Hemingway as a Case-study (2023)

Abstract: The literary domain continues to pose a challenge for Sentiment Analysis methods, due to its particularly nuanced and layered nature. This paper explores the adequacy of different Sentiment Analysis tools - from dictionary-based approaches to state-of-the-art Transformers - for capturing valence and modelling sentiment arcs. We take Ernest Hemingway’s novel The Old Man and the Sea as a case study to address challenges inherent to literary language, compare Transformer and rule-based systems’ scores with human annotations, and shed light on the complexities of analyzing sentiment in narrative texts. Finally, we emphasize the potential of model ensembles.

## introduction recent years have seen a significant increase in the methods available for sentiment analysis (sa). while dictionary-based approaches like vader @xcite seem to consistently perform well @xcite , they still struggle when applied to some domains @xcite @xcite . transformer-based models provide a much richer semantic representation texts, but also display shortcomings @xcite . while these tools are commonly used to analyze emotive language in contexts like social media @xcite , their suitability for literary texts remains relatively unexplored. literary language is particularly intriguing to test sa tools @xcite , because it often aims to evoke rather than explicitly communicate, operating at multiple narrative levels @xcite @xcite . in this study, we use the old man and the sea, often considered the masterpiece of ernest hemingway and exemplary of his philosophy of writing, as a benchmark for testing both rule-based and transformer-based sa systems. 1 hemingway's writing style is known for its emotional subtlety, often described as an "iceberg" or "omissive" writing, that evokes more than it describes: "the emotion is plentiful, though hidden but not exposed" @xcite . with its directness and limited use of figurative language @xcite , hemingway avoids "overt emotional display" @xcite in a way that may pose a particular challenge to sa. building on the literary analysis tradition that seeks to model sentiment arcs in literary texts @xcite @xcite , we apply various methods for sentiment annotation to the sentences of the novel and compare them to a benchmark of human annotations. ## related works in literary studies, what is often called the "affective turn" @xcite has led to a stronger focus on sentiment expressed in narrative texts @xcite , and sa has often been employed in computational literary studies to profile texts and model the "shape of stories" @xcite . to capture meaningful aspects of the reading experience, previous works tested the potential of sa @xcite at the word @xcite , sentence @xcite , or paragraph level @xcite to model narrative arcs @xcite @xcite . sentiment arcs have been used to evaluate literary texts in terms of shape or plot @xcite progression @xcite , and mood @xcite . certain shapes or arc dynamics have been connected to reader appreciation, considering both simple and more complex narratives @xcite , and @xcite have shown that sentiment features, such as measures of sentiment arc progression, have an effect even compared to the predominantly stylistic features usually employed for this type of task @xcite . as such, modelling sentiment arcs holds potential for gaining a more in-depth understanding of how narratives, in their unfolding, affect readers. however, both the validity of the dictionary-based approaches and the adequacy of methods for detrending arcs @xcite have been controversial in literary sa @xcite @xcite . for example, dictionary-based methods seem to perform well even on so-called "nonlinear" narratives @xcite although they appear to do poorly on a word-basis @xcite . on the other hand, more recent transformer-based approaches have shown both potential and pitfalls in the analysis of sentiment @xcite . ## results to evaluate the models we use the average of the annotators' scores. in our results show that large pretrained transformers correlate with human judgments on the valence of sentences better than the rule-based vader and syuzhet. thus, despite transformer's output on each sentence being categorical, it appears that their confidence scores can be successfully used as proxies for valence intensity even on literary sentences (see the appendix for a detailed plot of raw values). still, it is notable that the dictionary-based systems outperform half our transformer population. interestingly, the correlation of each model with each individual human is lower than the correlation of each model with the average human annotation ( when detrending the series of valences, we find that the picture changes: syuzhet outperforms all tranformers ( ## analysis literary language is a challenge to sa due to its subtlety and complexity. narrative sentences can be as complex as those of any other domain, yet because literary texts aim for their readers to experience rather than just be informed, they seem specially difficult to annotate. looking at the human scores of the old man and the sea, we found that annotators used almost the whole range (1 to 10), going from 2 to 9. though annotators were instructed not to overthink the narrative to reduce contextual scoring, this was not always easy. hemingway's direct style partly facilitated annotation, e.g.: ("fish," he said, "i love you and respect you very much"), but underlying complexity sometimes sparked uncertainty and disagreement for human annotators. despite being negative agents in the story, the sharks, for example, are still described as "beautiful", and the protagonist is portrayed as both "beat" and "undefeated". several of the larger inter-annotator disagreements were often due to the presence of co-existing valences in the same sentence. several of such sentences elicited differing judgments from the models as well: for example the sentence "the old man hit him on the head for kindness and kicked him, his body still shuddering, under the shade of the stern" elicited scores of 6 and 2 from the annotators, -.97 from distilbert and +.46 from vader (normalized values). we have already observed that almost all models correlate less with individual annotators than with the mean of the two annotators, an effect that is magnified when we also compute the mean of all the models' scores: the average annotation of all the models (after normalization) correlates with the human judgments better or as well as the individual models, both for the raw scores and for the detrended arcs. ## discussion and conclusions for this case-study in comparing sentiment annotation methods for literary analysis, we have compared the correlations between human annotations and several sa systems' annotations of the sentences of the novel the old man and the sea. while sentiment analysis is often tackled as a classification problem (with two or three categories at most), we found this approach to be exceedingly coarse-grained to verify the efficacy of sa models on literary texts, and we preferred to model it as a continuous scoring task. most of the time human annotators would have been unable to fit a sentence into a binary classification, and the most interesting behaviours of the models happen when looking at their ability to position a sentence on a nuanced continuum. naturally, it is now possible to operate the opposite operation and convert the continuous annotations into two or three categories, to compare them directly with the transformer's outputs. distilbert bert roberta roberta_xlm vader syuzhet avg. difference 0.86 0.48 0.19 0.26 0.23 0.16 std 0.22 0.32 0.22 0.26 0.24 0.15 we have observed interesting differences between transformer-and dictionary-based methods. still, it should be noted that our analysis was performed on one story only, even though the particular example of the old man and the sea appears particularly apt as case-study for sentiment analysis, considering its emotionally understating literary style. despite being categorical in nature, the largest transformers of our collection proved to hold strong correlations with human judgments in the sentence-level annotation -higher than the dictionary-based vader and syuzhet. when looking at the detrended version of the arcs, the picture is reversed: despite serious shortcomings of the tool @xcite , detrended arcs made with the syuzhet package appear to be the most closely related to the detrended version of human arcs (fig. 1 ). in both cases, the best results are achieved when using both transformer and dictionary-based systems, as they appear to be at least partly complementary, and our best model correlates with the mean human score almost as much as humans correlated with each other (