Investigating the Nature of Disagreements on Mid-Scale Ratings: A Case Study on the Abstractness-Concreteness Continuum (2023)

Abstract: Humans tend to strongly agree on ratings on a scale for extreme cases (e.g., a CAT is judged as very concrete), but judgements on mid-scale words exhibit more disagreement. Yet, collected rating norms are heavily exploited across disciplines. Our study focuses on concreteness ratings and (i) implements correlations and supervised classification to identify salient multi-modal characteristics of mid-scale words, and (ii) applies a hard clustering to identify patterns of systematic disagreement across raters. Our results suggest to either fine-tune or filter mid-scale target words before utilising them.

## motivation across disciplines, researchers have collected and exploited human judgements on semantic variables such as concreteness, compositionality, emotional valence, and plausibility. traditionally, those judgements are collected as a degree on a continuum between extremes. while humans tend to strongly agree on their ratings for extremes (e.g., a cat is typically judged as extremely concrete; glory as extremely abstract; the compound crocodile tears as extremely non-compositional; war as extremely negative), we find considerable disagreement regarding human mid-range ratings, i.e., judging about semi-concreteness, semi-compositionality, seminegativity. presumably, conceptual semi-properties are not easily graspable, thus generating stronger disagreement among raters. nevertheless, the collected norms are heavily exploited in state-of-theart computational approaches, where the respective knowledge represents a crucial task-related component (such as concreteness information for figurative language detection, and emotional valence for sentiment analysis). the current study provides a series of analyses on human mid-scale ratings, while focusing on the most prominent collection of concreteness ratings for english words @xcite , henceforth brysbaert norms. as basis for the brysbaert norms, humans were asked to judge the concreteness (in contrast to abstractness) of english words on a 5-point rating scale from 1 (abstract) to 5 (concrete) regarding how strongly the participants thought the meanings of the targets can(not) be experienced directly through their five senses. figure 1 illustrates the distribution of the mean concreteness ratings and standard deviations (sds) across 25 raters and for the three word classes of nouns, verbs, and adjectives. these croissant 1 plots for ratings on a scale can exhibit "only a finite number of possible combinations of means and standard deviations" @xcite : humans tend to agree on the extremes (→ low sd) and to disagree on intermediate semi-values (→ high sd). in a first set of experiments, we analyse multimodal characteristics of the concreteness of target nouns in the brysbaert norms (we provide additional materials for verbs and adjectives in the appendix): perception strength for specific senses (auditory, gustatory, haptic, olfactory, visual), emotional dimensions (valence, affect, dominance), lexical properties (frequency, ambiguity) and association types as indicators of meaning diversity. we start with a holistic perspective via correlations between targets' concreteness and their characteristics, and then zoom into differences for words with mid-scale vs. extremely concrete/abstract mean concreteness ratings, by applying supervised classification and feature analyses. in a second set of experiments, we hypothesise that mid-scale ratings are due to different combinations of individual human judgements across the scale. we thus rely on the original per-participant ratings (i.e., 25 ratings per target) and apply exploratory cluster analyses to identify patterns of disagreement between the individual raters of targets with mid-scale ratings. 1 we use this term due to the shape of the distribution plots. our contributions in this paper are two-fold. (i) we identify a range of target word characteristics that overall correlate with their degrees of concreteness ratings in different directions, and more specifically differ for mid-scale and extremely concrete or abstract target words. (ii) we identify a range of systematic disagreement patterns that clearly differ across target words with mid-scale mean ratings, thus pointing out fine-grained differences in judgements on semi-perception and suggesting to either filter or fine-tune mid-scale target words before utilising them in computational approaches. in the remainder of this paper, we introduce previous related work (section 2) and our concreteness targets (section 3); we then report our analyses regarding general and mid-scale target characteristics (section 4) and mid-scale disagreement patterns (section 5). ## related work collecting human judgements on a rating scale is a popular means of constructing concept-specific datasets across languages, research disciplines and (computational) linguistics tasks. prominent example tasks and collections targeting semantic variables include compositionality ratings for compound-constituent relatedness @xcite @xcite @xcite , affect ratings such as valence, arousal, dominance, emotion @xcite @xcite , plausibility ratings @xcite , and concreteness ratings @xcite @xcite @xcite @xcite . as a main motivation for collecting general conceptional ratings on a scale, @xcite state that there is "no reason for words to be rated for every single experiment". still, researchers across disciplines have pointed out problematic aspects of rating norms, because their reliability is unclear, especially when ratings have been collected via crowdsourcing or extrapolation @xcite . @xcite describes the typical shape of ratings on a scale, pointing out that the mid-range concepts are the least agreed upon, and that the interpretation of the corresponding ratings conflates semi-properties and genuine disagreements. a midscale score in concreteness could thus refer to an average semi-perception (whatever this means), or to a specific semi-sense, such as vision, haptics, etc., as well as to disagreement about perceptual strength, or a combination of the above. furthermore, many conceptual ratings have been collected by presenting the word in isolation without reference to the respective word class and out of context. for example, the brysbaert norms rely on isolated target presentation, and part-of-speech information was added post-hoc from the subtlex-us corpus @xcite . @xcite used the same setup as @xcite but for multiword expressions, in which case part-ofspeech ambiguity did not arise, but the targets were also presented out of context. despite these problems, ratings on a scale still remain the major strategy to collect human judgements on degrees of semantic variables, while alternatives such as best-worst scaling are available @xcite . the resulting norms are heavily exploited in state-of-the-art computational approaches; e.g., emotion and concreteness norms represent a crucial component in systems to detect figurative lan-guage usage @xcite @xcite @xcite @xcite . the current study encourages researchers to distinguish between degrees of (dis)agreement of such norms, and to identify a meaningful way of exploitation, in particular for mid-scale ratings. ## concreteness targets and ratings as materials for our experiments, we utilise the concreteness norms collected by @xcite , including approximately 40, 000 english target words. 2 the resource contains individual ratings by 25 participants on a 5-point scale ranging from 1 (abstract) to 5 (concrete), mean ratings and standard deviations. no context or part-of-speech (pos) were given; in a post-processing step, @xcite added pos and frequency information from the subtlex-us corpus. we followed a further post-processing step suggested by schulte im @xcite , who assigned the most frequently occurring pos tag and frequency information to the target words using the encow web corpus @xcite , and then reduced the targets to a less ambiguous and less low-frequent subset by discarding words for which (i) the predominant pos did not represent at least 95% of all pos occurrences; (ii) the newly assigned en-cow pos tag was not identical to the subtlex-us pos tag, or (iii) for which the encow target frequency was lower than 10, 000. our subset includes 5, 448 nouns, 1, 280 verbs and 2, 205 adjectives, and is publicly available. foot_1 4 target words: characteristics in our first set of experiments we analyse multimodal characteristics of our concreteness targets. after introducing these characteristics (section 4.1), we start out with a holistic perspective by quantifying statistical relationships between degrees of concreteness and our selection of target characteristics (section 4.2). we then zoom into differences in characteristics between mid-scale target words and extremely concrete/abstract target words, by applying a classifier that determines separability based on characteristics (section 4.3). ## mid-scale disagreement patterns in our final analyses, we zoom into the numerical characteristics of mid-scale mean ratings. if there was substantial agreement behind the semiperception of a mid-scale target (i.e., if all human for this exploration of disagreement patterns, we make use of the original per-participant ratings in @xcite , and applied a standard k-means hard clustering approach to automatically assign the 500 mid-scale nouns to @xmath0. as representations for the targets, we used 5-dimensional vectors with relative frequencies per rating categories 1, 2, 3, 4, 5, based on the original individual ratings, e.g., the vector for the noun discussion is ⃗ @xmath1.15, 0.07, 0.48, 0.15, 0.15⟩, because 15% of the raters provided ratings of 1, 4 or 5, while 7% judged it as 2, and 48% judged it as 3. figure 5 presents two perspectives on the resulting clusters with rather homogeneous cluster sizes 170, 163, 167. on the left, 11 we can see that the three clusters are clearly separated, with relatively small overlapping areas, thus indicating that the underlying cluster features (i.e., the rating distributions) clearly differ. this is confirmed by the plot on the right, which shows the individual rating distributions (y-axis) of the three cluster centroids 11 we used umap (uniform manifold approximation and projection) for down-scaling our distributions to two dimensions @xcite . overall, figure 5 thus provides very strong evidence in favour of our hypothesis that a mid-scale mean rating conflates rather different patterns of disagreements across human ratings. figures 12 and 13 in appendix e provide the respective plots for verbs and adjectives, where we find similar patterns of disagreement. ## discussion & conclusion we started out with the well-known observation that humans tend to strongly agree on ratings on a scale for extreme cases, but that judgements on mid-scale words exhibit more disagreement. this observation is well-described by the croissant-like shape of mean rating scores in relation to their standard deviations (cf. figure 1 ). while individual studies have pointed out problems with such ratings on a scale (e.g., @xcite ; pollock (2018)) and also provided alternative settings (e.g., @xcite ; @xcite ), the scale-based norms are heavily exploited across disciplines, including state-of-the-art computational approaches. in the current study, we first asked whether words with mid-scale concreteness ratings potentially exhibit specific characteristics that genuinely distinguish them from clearly concrete and clearly abstract words. the corresponding classification experiments and feature analyses demonstrated that mid-scale targets were indeed distinguishable from extreme targets with regard to a subset of the senses which were used as criteria for the concretenessabstractness distinction (mainly visual and haptic), and also with regard to emotional dimensions and meaning diversity (implemented on the basis of association types). in this first set of experiments mid-scale targets therefore established themselves as genuine intermediate concepts. we also saw, however, that mid-scale nouns are more easily distinguishable from extremely concrete in comparison to extremely abstract nouns, and this asymmetry flips with regard to verbs and adjectives, presumably because their underlying rating distributions exhibit different skews (cf. the croissant plots in figure 1 and the different mid-scale ranges in figure 9 in appendix d). so overall, words with mid-scale mean ratings represent rather genuine intermediate concepts regarding our implementations of features and analyses. in the second part of our study, we investigated whether mid-scale ratings are generally agreed upon across raters, or whether raters disagreed regarding their semi-perception. relying on explorative cluster analyses using the original perparticipant rating distributions, we found clusters with obviously very different centroids. from this, we induce that a mid-scale rating mean of ≈3 conflates rather different yet systematic kinds of disagreements. this observation is in line with the mathematically-based observations by @xcite that "there is only a finite number of possible combinations of means and standard deviations", and at the same time it clearly demonstrated that mid-scale ratings indeed differ regarding their underlying rating combinations. so, on the one hand, our cluster analyses confirm a so-far rather theoretically-driven observation; on the other hand, we raise the question of whether and how this observation should influence the utilisation of ratings on a scale. we suggest two alternative routes: (i) either filter the norm targets and only keep those targets that are clearly attributable to one extreme, or (ii) fine-tune the mid-scale norm targets with regard to inherent disagreement patterns, because the set of mid-scale targets is itself rather inhomogeneous but nevertheless provides valuable information regarding specific differences in human perception. last but not least we would like to point out that inherent disagreements among human annotators are obviously not restricted to our particular focus on mid-scale ratings but represent a common issue under discussion across annotation tasks. in the past decade the field has moved from considering disagreements as pure noise towards zooming into disagreements in order to distinguish between noise and subjectivity, and to effectively exploit the value of disagreements in language modelling, see @xcite and @xcite for a prominent opinion paper and a prominent survey, respectively. our analyses and insights should be interpreted in the same vein: we attribute disagreements on concreteness mid-scale ratings to genuine intermediate concepts (see above) and suggest to take a finegrained approach when utilising them in language modelling tasks and applications.