‘... like a needle in a haystack”: Annotation and Classification of Comparative Statements (2025)

Abstract: We present a clear distinction between the phenomena of comparisons and similes along with a fine-grained annotation guideline that facilitates the structural annotation and assessment of the two classes, with three major contributions: 1) a publicly available annotated data set of 100 comparative statements; 2) theoretically grounded annotation guidelines for human annotators; and 3) results of machine learning experiments to establish how the–often subtle–distinction between the two phenomena can be automated.

## introduction the automatic processing of figurative language is a challenge that has long been a focal point of research in natural language processing @xcite @xcite . one of the tasks in dealing with comparative statements is to find clear boundaries between two very similar phenomena: that of comparison and simile. the term comparison describes a linguistic unit that is used to convey similarities and dissimilarities between two entities. even though comparisons in general are understood to be syntactic, they can in effect harmonize much more relevant semantic knowledge in everyday language. comparisons are used in everyday communication, e.g., in a debate when one has to put across a point, or when pointing out a similarity or difference between entities that share some property. similes on the other hand are special structures that are derived from comparisons and can also be called figurative comparisons. a simile can be defined as a figure of speech that is used to draw a parallel between two dissimilar entities or processes that have some shared properties. let's consider the two examples: in (1), the comparison is drawn between two brothers' physical size, while (2) draws a comparison between a human (he) and an object (tower) which makes the comparison figurative. this leads to an important consideration in the distinction between comparisons and similes: comparisons can be drawn between any two entities, while a comparison only becomes figurative (simile), if the two entities belong from different semantic categories. this paper presents an annotation methodology that allows us to distinguish if a comparative statement is a literal comparison or a figurative comparison (simile). to the best of our knowledge this is the first work that focuses on step-by-step annotation guidelines for comparison vs. simile, taking into account various features of comparative statements. similes are a particularly interesting phenomenon in the domain of literature, because they often carry subtle meaning that can be overlooked if a statement is treated as a simple comparison: "poor dorothea felt that every word of her uncle's was about as pleasant as a grain of sand in the eye to mr. casaubon." @xcite . the former statement contains a simile. it carries more meaning than a simple comparison, i.e., it requires the reader to acknowledge that a grain of salt in the eye is very unpleasant and thus the statements that dorothea's uncle makes in front of mr. casaubon are evoking a negative emotion in the listener. in fiction, simile's are often used to transport subtle meaning and therefore particularly interesting to study. however, similes are a sparse phenomenon with rare occurrence in everyday usage, their annotation is time-consuming and often yields very little data per literary work. therefore, this work presents straightforward annotation guidelines by defining nine categories (subject of comparison, object of comparison, nature, categories, feature matching, symmetry, salience, broad unifying concept and domain incongruence) to distinguish similes from non-figurative comparisons, together with machine learning experiments that can help automatically annotating larger corpora of fictional texts for further studies in the domain of computational literary studies. the data set and gold standard annotation can be found here foot_0 . ## related work the study of comparison in rhetoric can be dated as back as aristotle @xcite , who highlighted the importance of using comparisons in everyday life @xcite . however, computational studies addressing the distinction between figurative and literal comparisons are scarce, since both phenomena follow a similar structure, and consist of the same constituents @xcite . @xcite proposed a "similarity based approach" that aided in measuring the degree of figurativeness of a comparison which by extension can be used as a means of identification of similes. since there is a lack of annotated corpora of comparisons, @xcite used the vuamc corpora @xcite to extract the comparison patterns of "like". they then annotated it for the task of simile identification. since similes are a form of comparisons, @xcite contributed to computational research on simile by focusing on comparison recognition through the use of syntactic patterns. most work on automatic detection and analysis of figurative language targets metaphors @xcite and idioms @xcite , but only few recent studies investigate similes as special comparative statements. foot_1 more recent exceptions are @xcite ; @xcite , in which neural and transformer-based models are used in a multi-task setting to identify similes and their components in chinese texts. ## annotation guidelines the guidelines that are presented here is developed based on seh (2016), in which seh dedicates a complete chapter in understanding comparisons and their distinction from similes. before we introduce the annotation guidelines, we must first discuss the syntactic structure and the semantic particularities of different types of comparative statements. we will then discuss the steps involved in distinguishing a literal comparison from a simile. this paper does not focus on the merits of the individual underlying theories foot_2 of comparison. instead, its main contribution is building a concise annotation guidelines that is derived from the theories for the task of identifying and distinguishing comparisons and similes. ## data the data for the annotation study was extracted from the english data present in the parallel meaning bank (pmb) @xcite and filtered by the simple regular expression: as [a-z]* as an?. we then manually clean the data to remove duplicate instances, shorten the sentences to simplify annotation and split complex sentences with multiple comparative structures into shorter sentences. for example, "i am as light as a feather, i am as happy as an angel, i am as merry as a school-boy" was split into three simple comparative sentences. furthermore, all instances of "as well as" were removed as those are usually synonymous to statements containing too or also. we eventually gather a data set of 100 sentences. the statistics of our gold standard annotation can be found in ## experiments in correspondence to the human annotation, we also conduct machine learning experiments to help determine if and how the process of classifying a comparative statement into comparison or simile can be automated. for that purpose, we use a simple support vector machine (svm) as our baseline (with support vector classification (svc), a linear kernel, and the default regularization parameter (@xmath0.1)). the data was split into a training set and test set of 80%-20% and tf-idf vectorizer was used as the feature extractor. we then conduct two experiments with the large language model (llm) llama-3.3-70b-instruct foot_6 . we perform the first experiment using zero-shot prompting, in which the llm is asked to judge if a comparative statement is a simile, comparison or not applicable, see we test the performance against the gold standard annotated data. ## results in this section, we report the results on the annotation task (inter-annotator agreement, cohen's κ), and the machine learning experiments, i.e. the svm baseline and llm annotations that were conducted on the curated data set. ## conclusion and future work this work is the first step towards building a pipeline to automatically detect and annotate similes in fiction. it is essential to first draw a clear distinction between a comparative structure as a literal comparison and as a simile, which is what we aimed through this work. the next focus of our project is to develop a fine-grained annotation guideline to annotate similes in literature. we also aim to make the guidelines largely languageagnostic, with a focus on english that will be refined for other languages, such as bengali, that come from a completely different language family with a different word order. furthermore, the final objective is to perform a quantitative and contrastive analysis to uncover cultural narratives and values depicted in simile usage in literature and the way of expression of humans in general.