Sentiment as an Ordinal Latent Variable (2023)

Abstract: Sentiment analysis has become a central tool in various disciplines outside of natural language processing. In particular in applied and domain-specific settings with strong requirements for interpretable methods, dictionary-based approaches are still a popular choice. However, existing dictionaries are often limited in coverage, static once annotation is completed and sentiment scales differ widely; some are discrete others continuous. We propose a Bayesian generative model that learns a composite sentiment dictionary as an interpolation between six existing dictionaries with different scales. We argue that sentiment is a latent concept with intrinsically ranking-based characteristics — the word “excellent” may be ranked more positive than “great” and “okay”, but it is hard to express how much more exactly. This prompts us to enforce an ordinal scale of ordered discrete sentiment values in our dictionary. We achieve this through an ordering transformation in the priors of our model. We evaluate the model intrinsically by imputing missing values in existing dictionaries. Moreover, we conduct extrinsic evaluations through sentiment classification tasks. Finally, we present two extension: first, we present a method to augment dictionary-based approaches with word embeddings to construct sentiment scales along new semantic axes. Second, we demonstrate a Latent Dirichlet Allocation-inspired variant of our model that learns document topics that are ordered by sentiment.

## introduction sentiment analysis is being applied in various domains from political science @xcite @xcite to economics @xcite and computational social science @xcite @xcite . in all of these applications, there is a strong demand for domain-specific and interpretable methods @xcite making dictionarybased sentiment analysis still a popular choice @xcite @xcite . sentiment dictionaries describe a mapping between word types and some form of sentiment values. we consider the most general notion of sentiment value referring to the polarity score along a positive-negative axis, instead of fine-grained emotion dimensions @xcite or stance @xcite . sentiment values are measured on scales of different support ( §2): some dictionaries assign binary "positive" and "negative" values @xcite @xcite . these discrete values are often falsely interpreted as unordered, nominal categories. other dictionaries have continuous scales that assign cardinal, floating point values @xcite . in this work, we propose a method for merging sentiment dictionaries with different scales into a single, composite dictionary. paying tribute to the subjective and ranking-based characteristics of sentiment, we design the dictionary to have an ordinal scale. ordinal scales define discrete, ordered classes where interval sizes between classes are unequal and typically unknown @xcite . for instance, the word "excellent" may be ranked more positive than "great" and "okay", but it is hard to express how much more positive. an example is the ordinal likert scale @xcite used to measure attitudes in psychometrics. our ordinal sentiment scale is derived from an ordinal latent variable within a probabilistic, generative model ( §3). in particular, the latent variable's classes represent sentiment values. the classes are uniquely ordered which is achieved through an ordering transformation that is applied to the priors of our model ( §3.2). our model is tightly coupled with recent advancements in probabilistic programming @xcite and gradient-based inference @xcite . these advancements alleviate the strict requirement of closed-formedness and conjugacy to perform posterior inference in complex bayesian models with latent ordering motifs. our ordinal scale is learned as an unsupervised interpolation between 6 popular sentiment dictionaries. this has several benefits: on the other hand, we can impute missing sentiment values in existing dictionaries. we evaluate this capacity in a bayesian data imputation task ( §4.2). on the other hand, interpolating between different dictionaries causes our composite dictionary to have high coverage of word types from widely different sources. we evaluate our composite dictionary in 6 sentiment classification tasks from different domains ( §4.3). taking a bayesian approach, we have access to uncertainty estimates for each sentiment value per word type. we find that uncertainty is larger for ambiguous and rare word types that are covered by only few dictionaries ( §5). in §6, we present two possible extension of our ordinal latent variable model. to further expand word type coverage, we incorporate sentiment values derived from bi-polar semantic axes within word embeddings ( §6.1). to demonstrate the wide applicability of our ordinal modeling motif, we introduce a model variant that is closely related to latent dirichlet allocation (lda; @xcite , but learns topics ordered by sentiment ( §6.2). we publish our code together with our learned, highcoverage sentiment dictionary, annotated with posterior credible intervals. ## data: sentiment dictionaries we consider 6 popular english-language sentiment dictionaries: senticnet (sc) @xcite , sentiwordnet (sw) @xcite , vader (va) @xcite , general inquirer (gi) @xcite hu-liu (hl) @xcite and mpqa (mp) @xcite . the dictionaries vary in the number of included word types, the word source, application domain and the sentiment scale, see appendix tab. 3. sc, sw and va have continuous, bounded sentiment values, while gi, hl and mp have discrete, binary values as visualized in fig. 1 . we scale all continuous values to a [0, 1] range. some of the dictionaries such as sw and va feature multiple sentiment values per word type. we average those to consistently obtain one value per word type for all dictionaries, which allows for a fair comparison. we group the sentiment dictionaries in a single data table by word type. since different dictionaries contain different word types, this results in many missing values. we filter the data table so that each word type is covered by at least 2 dictionaries. this leaves us with @xmath0,342 unique word types that serve as our dataset. ## model our goal is to learn a unifying sentiment dictionary as an interpolation between existing sentiment dictionaries. each word type v is described by one or multiple sentiment values of a dictionary. depending on the dictionary's scale, sentiment values can be continuous the superscripts c and d represent continuous and discrete dictionaries respectively, i.e., @xmath1, sw, va} and @xmath2, hl, mp}. considering all 6 sentiment dictionaries, we have a tuple of 6 sentiment val- ## experiments we evaluate our model intrinsically ( §4.2) and our inferred ordinal sentiment dictionary, ordscale, extrinsically ( §4.3). therefore, we first fit our model to existing sentiment dictionaries, identify the optimal number of latent classes and finally infer the ordinal scale. first, we split the @xmath3,342 word types into a 70% training and 30% test set. next, we run the nuts sampler to perform posterior inference as introduced in §3.3. we discard the first 200 burn-in samples and consider only the following @xmath4. ## related work this work builds upon recent attempts at merging sentiment dictionaries @xcite @xcite @xcite . it is closest to sentivae @xcite , a multi-branch variational autoencoder (vae) with a 3-class categorical latent space parametrized with a dirichlet prior. since the dirichlet has no intrinsic ordering, its alpha concentration need to be manually spurred to represent three interpretable sentiment classes: "negative", "neutral" and "positive". in contrast to @xcite , we consider only one sentiment value per word type to guarantee a fair comparison in the extrinsic evaluation setting. our latent variable model is inspired by @xcite , who present a model to learn an ordinal scale of conflictcooperation intensity. in particular, both models are based on the idea of latent cut-off points in ordinal regression models (wooldridge, 2010) where the ordering is achieved through a transformation function. to obtain an ordering, other approaches simply sort a set of samples which relates to order statistics @xcite @xcite . there exist many approaches for learning scales on ordinal observed (opposed to latent) variables comprise the underlying variable approach (uva) and item response theory (irt, @xcite ). another bayesian method for aligning sentiment dictionaries is called sentimerge @xcite . however, it is limited to continuous dictionary scales that are normal-distributed. there exists a plethora of extensions of the latent dirichlet allocation @xcite @xcite @xcite . similar to our approach, supervised lda @xcite regresses document labels directly on the empirical topic frequencies during inference. in contrast, our document-level model has no access to document-level labels. dieng et al. (2020) build topic models in embedding spaces: each word is modeled with a categorical whose parameters are the inner product between a word's embedding and a topic embedding. @xcite present an ordering constraint on the topic-word type matrix φ to learn ordered topics based on ordered vocabularies. ## conclusion this work treats sentiment as a latent concept with ranking-based, ordinal characteristics. other ordinal phenomena such as pain perception @xcite , conflict intensity @xcite or political ideology @xcite can similarly be measured on ordinal scales. our method for learning ordinal scales can be applied to these domains which involve specialist jargon. the resulting sentiment dictionaries are easy to validate through manual inspection and uncertainty estimates @xcite .