Probing for Hyperbole in Pre-Trained Language Models (2023)

Abstract: Hyperbole is a common figure of speech, which is under-explored in NLP research. In this study, we conduct edge and minimal description length (MDL) probing experiments on three pre-trained language models (PLMs) in an attempt to explore the extent to which hyperbolic information is encoded in these models. We use both word-in-context and sentence-level representations as model inputs as a basis for comparison. We also annotate 63 hyperbole sentences from the HYPO dataset according to an operational taxonomy to conduct an error analysis to explore the encoding of different hyperbole categories. Our results show that hyperbole is to a limited extent encoded in PLMs, and mostly in the final layers. They also indicate that hyperbolic information may be better encoded by the sentence-level representations, which, due to the pragmatic nature of hyperbole, may therefore provide a more accurate and informative representation in PLMs. Finally, the inter-annotator agreement for our annotations, a Cohen’s Kappa of 0.339, suggest that the taxonomy categories may not be intuitive and need revision or simplification.

## introduction hyperbole is a common figure of speech that involves the use of exaggerated language for emphasis or effect @xcite . humans exaggerate in a variety of registers and contexts, spanning from the colouring of informal, everyday speech to a literary trope or a rhetorical means of persuasion. hyperboles intentionally augment or diminish a feature of some referent of discourse, presenting this feature on some more or less abstract scale of magnitude. the task of hyperbole identification poses a challenge to natural language processing in that it is highly pragmatic and utilizes context and background knowledge to distinguish between literal and exaggerated usage of a given lexical unit. as an illustration of the pragmatic nature of hyperbole, we can inspect the following two example sentences, wherein (1a) is hyperbolic and (1b) is literal: in (1a), it is reasonable to assume that the speaker is exaggerating the number of times they have seen this particular movie to emphasize their enjoyment or familiarity with it because this would otherwise be a significant and unrealistic time investment. however, when it comes to a particular product, it has likely gone through rigorous testing and quality control measures, which means that the statement in (1b) can reasonably be interpreted literally. hyperbole identification has recently attracted the interest of nlp researchers who have collected datasets manually or semiautomatically and shown that computational modelling of hyperbole is indeed plausible @xcite . however, it remains an under-explored area of research in figurative language processing (flp), primarily because its subjective and contextual nature complicates computational modelling of the phenomenon and makes it challenging to apply a standard for collecting high-quality annotated data @xcite . this paper seeks to contribute to the growing research on hyperbole identification in two ways: firstly, we perform probing tasks to investigate whether pre-trained language models (plms) encode hyperbolic information in its representation without fine-tuning on taskspecific data. 1 in recent years, probing tasks have emerged as a popular approach in nlp for interpreting and analyzing model representations, and it has previously been shown that plms do encode both simile and metaphorical knowledge @xcite . however, to our knowledge, hyperbole probing remains so far unexplored. therefore, we replicate edge and minimal description length (mdl) probing experiments for metaphor described by @xcite on a small hyperbole dataset constructed by @xcite . we expect that encoding hyperbole may present a larger challenge to plms than metaphor because hyperbole knowledge is primarily pragmatic rather than semantic @xcite . secondly, we build an operational taxonomy based on a meta-analysis of the linguistic treatment of hyperbole, and annotate an existing dataset according to said taxonomy @xcite @xcite @xcite . we then use these annotations to analyze errors in model predictions to further shed light on the types of hyperboles that may pose a particular challenge to plms, as well as when constructing training corpora for the phenomenon. our work will hopefully provide insight into the challenges of plms in identifying hyperbole, as well as contribute to developing an operational annotation standard for computational modelling of hyperbole. 2 the remainder of this paper is structured as follows: section 2 contains an overview of related work in hyperbole research, as well as probing experiments on other figures of speech. section 3 provides a background on the linguistic research that is the framework for our operational taxonomy and annotation. section 4 is a short explanation of probing tasks for plms, which we relate to the aim of our experiments. section 5 outlines our experimental setup and describes the modifications made to the hypo dataset. section 6 provides our results and preliminary error analysis, and section 7 is a discussion of said results, as well as 2 our code for the probing tasks is available at https: //github.com/nisc91/hyperboleprobe ideas for future research. section 8 contains a summary and conclusions. ## related work in this section, we outline previous research related to both hyperbole and probing experiments on other figures of speech. ## a taxonomy for hyperbole in simple terms, hyperbole involves exaggerating a feature's property x beyond what is justified by the literal state of affairs @xcite . stated in a more discourse-centred way, hyperbole occurs when an expression is more extreme than justified given the ontological referent, i.e. the entity in the world referenced by the text @xcite subtree and examples for the possibility category. 2016). while much of the work on hyperbole has previously been subsumed under studies of metaphor, humour, and verbal irony, recent corpus linguistic analyses have shed light on more fine-grained characteristics. namely, the consensus in the treatment of hyperbole in literature is that the phenomenon is, among others, characterized by the presence of extreme case formulations (ecf), the ability of hyperbole to create either extreme possible worlds or downright counterfactual and absurd scenarios, and its augmentation of some property along a qualitative or quantitative scale @xcite @xcite . in the following, we outline some of the key characteristics and visualize them in an operational taxonomy (see figures 1 and 2 ). ## probing plms for hyperbole probing language models aims to answer questions related to the model's internal representation, such as the location and depth of the encoding of a linguistic property in the multi-layer representation, or which input features contributed to a particular behaviour of the plm @xcite . standard probing methods involve training a linear classifier on top of a plm to predict a linguistic property of interest, where a high probing performance on the task is associated with the model encoding said property. it is common practice to freeze the parameters of the plm, which serves to prevent the gradients of the probing classifier from back-propagating into the model and thereby altering its pre-trained representation @xcite . following @xcite , our experiments are not aimed at improving the accuracy of hyperbole identification tasks; we simply want to check the extent to which hyperbole knowledge may be encoded in the base representations. to that end, we employ edge probing, in which the classifier receives span-level representations from the plm as inputs after they have been projected to a fixed-dimensional layer, 250 in this case. thus, we define the span input to the plm as the minimal lexical unit conveying hyperbolic information as given by the hypo dataset @xcite . one common criticism of edge probing is that it may not be explanatory in the sense that it does not provide insight into whether a model is learning a linguistic property or simply memorizing the task @xcite ). an information-theoretic perspective on addressing this limitation is to combine the probing quality of the classifier with some metric of the effort needed to extract the linguistic knowledge. this approach is known as mdl probing @xcite , wherein effort intuitively refers to the number of steps required by the plm to encode a compressed representation of the input sequence. following @xcite , we use the online coding implementation of mdl, which measures a representation's ability to learn from various portions of the data. we report the compression, which is given by n • log 2 (k). in the context of language modelling, n refers to the size of the dataset, and k is the set of unique sequences being compressed. a random classifier will have a compression of 1, and increased data compression is associated with a better encoding of the given property. ## experiments here we describe our data and setup. dataset and annotation. we utilize hypo, a manually constructed english hyperbole dataset @xcite of 709 hyperboles with corresponding literal paraphrases, as well as a minimal units corpus that provides the contrastive negative (literal) examples for each hyperbole (see examples (1a) and (1b) in §1). for the purpose of our experiment, we first discard the corpus of literal paraphrases as we are interested in contrasting the hyperbolic usage of a particular word or phrase with a literal usage of the same word or phrase. it would otherwise not be possible to construct spans. to obtain span labels for each hyperbole and its negative contrast sentence, we programmatically extract the positions of each minimal lexical unit and manually adapt the labels as needed; namely, we exclude examples with multiple spans and those without minimal unit contrasts. 3 our final dataset contains 1396 span-labelled hyperbolic and literal sentences, which we split into training (70%), test (20%), and development (10%) sets. we meticulously annotate the 63 hyperbolic sentences in the development sample using the operative taxonomy outlined in §3. 4 in order to obtain inter-annotator agreement, we enliste the help of additionally 5 annotators, assigning 12-13 sentences to each. as a result, each sentence is annotated twice. we observe a mean cohen's kappa of 0.339 (see figure 3 ), suggesting only fair agreement, with particular difficulties on the dimension and type spectra on the taxonomy. ## results all our results are reported on the test set. edge probing results. the edge probing classification results are in ## discussion we observe notably lower scores than for the metaphor probing experiments across the bert wic. bert sentence-level. roberta wic. roberta sentence-level. electra wic. electra sentence-level. board: based on the compression reported for the mdl probes, only reaching up to 1.4 in the best configuration, we can conclude that hyperbolic information does appear to a minor extent to be encoded in plm representations. this is in line with our expected hypothesis that encoding hyperbole may pose a bigger challenge given its primarily pragmatic nature, and also fits with the fact that plms have been reported to struggle with pragmatic inference and commonsense knowledge @xcite . perhaps more interestingly, we can inspect the compression for each of the 12 layers reported in figure 4 to understand where hyperbole is best encoded by the representation, which appears to mostly be in the final layers. this is different from metaphor and may lend further credence to the idea that pragmatics is typically encoded deeper into the plm. however, since we are employing a very small dataset, the extent to which we can draw definite conclusions is limited. in the future, we would like to extend our experiments to more data and languages to measure generalizability. upon analyzing the mdl compressions of the two model representations, we make an intriguing observation that the sentence-level representation consistently outperforms the wic representation, with compressions reaching up to 1.4 for the top layer. this discovery raises thought-provoking questions about the amount of hyperbole information inferred by the contextual embeddings, as hyperbole often surpasses the token or phrase level. for example, consider the sentence, "the temperature was so low, i saw polar bears wearing jackets." in this case, the entire complement sentence creates the hyperbole. this leads to discussions about defining the lexical unit of hyperboles for corpus collection and annotation purposes @xcite . as for the model representations themselves, while plms theoretically encode context in their representation, it is worth exploring how much information is contained within and between subwords in the wic representation. employing interpretability metrics could provide further insights into this matter. considering the low inter-annotator agreement and that recall seems to generally increase with the frequency of the subcategory in the sample, it is challenging to draw insights from the model error analysis (see finally, the low inter-annotator agreement, particularly on the dimension and type di-chotomies, suggests that the hyperbole categories are not intuitively well-understood or discriminated. during discussions with annotators upon completion of the task, we had several instances where overlap of the dimension subcategories was so large that annotators could argue for either one, and it also wasn't clear to annotators when a semantic domainswitch was present. the latter suggests that more linguistic training may be necessary to identify combined figures of speech in context, for instance, through application of the hyperbole identification procedure (hip) @xcite . as a consequence, we would like to change our approach to hyperbole annotation in future corpus construction and investigate to which extent these categories are indeed computationally relevant. our negative findings lend credence to the claim by @xcite that annotation schemes may present a bottleneck for further development of of the task. we would also like to explore approaches for model evaluation of hyperbole types using conceptual knowledge bases and linguistic resources; namely leveraging framenets to explore their utility for metaphorical hyperboles, as well as investigating templates using particular syntactic patterns for evaluating quantitative hyperboles. ## conclusions this study has attempted to probe three pretrained language models (plms) for hyperbolic knowledge to better inspect how this information is encoded in their representations. we find, predictably, that knowledge of hyperbole is only to a limited extend encoded by plms, and, somewhat more surprisingly, that sentence-level representations appear to be supperior to word-in-context (wic) representations, which may further highlight that most hyperbolic information does in fact exist beyond the token or phrase level. in the future, we would like to contribute with more hyperbole data with an operational annotation procedure, extend to cross-lingual experiments, as well as investigate the role of linguistic resources for hyperbole identification.