Harnessing Privileged Information for Hyperbole Detection (2021)

Abstract: The detection of hyperbole is an important stepping stone to understanding the intentions of a hyperbolic utterance. We propose a model that combines pre-trained language models with privileged information for the task of hyperbole detection. We also introduce a suite of behavioural tests to probe the capabilities of hyperbole detection models across a range of hyperbole types. Our experiments show that our model improves upon baseline models on an existing hyperbole detection dataset. Probing experiments combined with analysis using local linear approximations (LIME) show that our model excels at detecting one particular type of hyperbole. Further, we discover that our experiments highlight annotation artifacts introduced through the process of literal paraphrasing of hyperbole. These annotation artifacts are likely to be a roadblock to further improvements in hyperbole detection.

## introduction the analysis of figurative language by natural language processing (nlp) systems is a challenge confronting researchers and practitioners @xcite . hyperbole is a common type of figurative language that is defined by an intentionally excessive contrast between utterance meaning and reality along a semantic scale to convey an evaluation (e.g., 'my bedroom is the size of a postage stamp') @xcite @xcite @xcite . the detection of hyperbole has proven to be a challenging problem for nlp systems, much like the detection of other figures of speech @xcite @xcite . the evaluative nature of hyperbole motivates the importance of understanding hyperbole for affective computing applications (e.g., sentiment analysis). learning under privileged information (lupi) is a learning paradigm that involves providing additional information during training to help teach a model to learn a particular phenomenon @xcite . the source and type of privileged information (pi) varies depending on application, such as a list of ingredients present in an image to help teach a computer vision model to detect food in images @xcite , or the human ratings of various aesthetic categories of images for automated assessment of aesthetic photo quality @xcite . we propose to use literal paraphrases of hyperbole as a source of pi for hyperbole detection. we hypothesise that this information will help a model to learn the excessive contrast within a particular hyperbole (e.g., 'my head is exploding right now' → 'my head is hurting right now'). ## hypo the hypo dataset is an annotated collection of hyperbole introduced by @xcite . the dataset consists of manually composed hyperbole and hyperbole sourced from various online sources including click-bait headlines, love letters, advertisements, and animated cartoons. annotation for hypo was carried out by crowd workers who were given several tasks based on each example. the crowd workers had to assess whether they thought the utterance contained hyperbolic content. a follow up task was to highlight the specific words in the utterance they considered minimal units corpus the principal is unhappy...we're cooked. the principal is unhappy...we're in trouble. well cooked vegetables can be pureed easily. ## hyperprobe our hyperprobe suite consists of synthetic data generated to probe the ability of models to detect hyperbole 1 . the suite is created to target the three types of hyperbole identified by @xcite ): 1 https://github.com/biddle-r/hyperprobe extreme case formulations (ecf), qualitative hyperbole and quantitative hyperbole. the creation of test sentences follows a general four step procedure: 1. word list creation: we create seed word lists containing words to be used in test sentences. these seed word lists are divided by part-of-speech class and are created based on the word lists curated by mora (2009), see ## privileged information for hyperbole detection our motivation for incorporating privileged information into a hyperbole detection model is based on observations from the foundational work of @xcite . the authors found that models trained on hyperboles and literal paraphrases performed marginally better on the task of hyperbole detection than models trained on hyperboles and non-literal sentences that used the hyperbolic words/phrases in a non-hyperbole context. we propose that treating literal paraphrases as privileged information and incorporating this information into a hyperbole detection model could improve the ability of a model to detect when a word or phrase was being used in an excessive hyperbolic manner. in our proposed model, bert+pi, we incorporate privileged information via triplet loss. we utilise a triplet loss because we want to force our model to differentiate between hyperbolic and nonhyperbolic usage of words and phrases, and we can strictly enforce this via triplet loss. specifically, by specifying a hyperbolic sentence as an anchor sample, another hyperbole as a positive sample and a manually composed literal paraphrase (i.e., pi) as a negative sample, we are enforcing this difference in representation space. ## conclusion in this paper we proposed a hyperbole detection model, bert+pi, that incorporates pi via triplet loss with a pre-trained language model (bert) into a multi-task text classification framework for hyperbole detection. experiment results showed improvements in detection using standard information retrieval metrics (i.e., f1, precision and recall), for models that incorporate pi on the hypo test set. however, these results were not maintained across our synthetic test suite hyperprobe. in fact, only on the ecf test in hyperprobe did we observe similar results. on both the quantitative and qualitative hyperbole tests we observed poor performance. our hypothesis for this disparity is that the incorporation of pi into bert+pi teaches the model to learn annotation artifacts introduced by the creation of literal paraphrases in the paraphrase corpus of hypo. specifically, ecf hyperbole can often be paraphrased quite simply by removing only a few tokens (e.g., what an absolute @xmath0. bert+pi effectively incorporates this information well and as a result appears to be able to differentiate between hyperbolic and non-hyperbolic ecfs. however, for more complex hyperbole, unwanted annotation artifacts are introduced during the process of creating a literal paraphrase. for example, 'my heart is as heavy as the world' could be paraphrased as 'i am sad'. in this paraphrase, the contrast and the semantic scale of the hyperbole are lost in the paraphrase given the significant difference between the hyperbole and the paraphrase. in future work, exploring better annotation methods for complex hyperbole that encode the semantic scale and the source of excessive contrast will be an important focus to overcome the shortcomings caused by unwanted annotation artifacts.