N ews M et : A ‘do it all’ Dataset of Contemporary Metaphors in News Headlines (2023)

Abstract: Metaphors are highly creative constructs of human language that grow old and eventually die. Popular datasets used for metaphor processing tasks were constructed from dated source texts. In this paper, we propose NewsMet, a large high-quality contemporary dataset of news headlines hand-annotated with metaphorical verbs. The dataset comprises headlines from various sources including political, satirical, reliable and fake. Our dataset serves the purpose of evaluation for the tasks of metaphor interpretation and generation. The experiments reveal several insights and limitations of using LLMs to automate metaphor processing tasks as frequently seen in the recent literature. The dataset is publicly available for research purposes https://github.com/AxleBlaze3/NewsMet_Metaphor_Dataset .

## introduction metaphors are creative cognitive constructs designed to communicate an idea in an evocative fashion @xcite . research on computational metaphor processing has explored a variety of questions related to the detection of metaphorical speech in text @xcite and its interpretation by readers @xcite . the task of metaphor generation @xcite @xcite has recently gained traction due to the growing ability of llms to forge common sensical connections. metaphors are highly creative constructs of human language that grow old and eventually die @xcite . the vast majority of studies on metaphors in the english language still rely on datasets such as trofi @xcite , vua metaphor corpus @xcite , and lcc @xcite that contain archaic source texts (see table a1 ). for instance, the news genre in vua metaphor corpus @xcite which is derived from bnc baby foot_1 has the latest headline from the year 1994. dead metaphors in these archaic source texts are essentially ineffective training samples. moreover, the object of interest that is, metaphors from contemporary world texts are lacking. for instance, consider the metaphor heal in "love heals soul" that does not require much thought to comprehend versus a phrase like "amtrak dining car heals nation" taken from a headline in our dataset. in this paper, we propose newsmet, a large high-quality contemporary dataset of news headlines hand-annotated with metaphorical verbs. metaphors are a commonly used figurative construct in news headlines to better explain a complex event or scenario. for instance, consider the @xmath0, pushing, boundaries/@xmath1, pushing, microchip implants>. the metaphorical verb push with pushing implants is a relatively new use. news headlines thus provide an evolving linguistic backdrop with new entities to learn contemporary metaphor use. metaphors are also routinely used to make a political or social argument. metaphorical language imprints emotions that one may not have anticipated otherwise and therefore, metaphorical news could be an interesting data source to evaluate the detection of hyperpartisan content. in this paper, we make the following contributions: we believe that the proposed dataset will be an invaluable resource for researchers studying metaphor processing, providing a rich and diverse set of samples. we further believe that metaphors in news headlines will help understand and tackle the growing bias and hyperpartisan in digital media. additionally, the dataset could be utilised to evaluate natural language understanding in llms. ## background early approaches for metaphor processing focused on analyzing restricted forms of linguistic context such as the subject-verb-object (svo) type grammatical relation, using hand-crafted features @xcite . later, the approaches evolved to capture implicit relationships in long text through word embeddings and large language models. rai and chakraverty (2020) and @xcite provide a detailed discussion on these approaches as well as existing datasets for the tasks of linguistic metaphor detection and interpretation. metaphor generation research in particular has recently gained traction with quite a few approaches exploiting neural language models as their underlying knowledge base @xcite . to evaluate machine-generated metaphorical content, bigbench (big-bench collaboration, 2021) proposed four metaphor-related classification tasks (figure_of_speech_detection, metaphor_boolean, metaphor_understanding and identify_odd_metaphor). these tasks use newly curated datasets from sources such as online literature and existing datasets. however, the average size of the 4 datasets mentioned is 255, limiting their usefulness for the task of evaluation. moreover, the aforementioned bigbench tasks and some other newly proposed datasets like do @xcite and impli @xcite use source texts from old datasets, such as the bnc corpus @xcite and @xcite . this limits their novelty as the source text remains similar to old datasets. @xcite curated multi-modal ring that bell corpus of permissively licensed youtube videos that have human-authored closed captions in en-glish with metaphors annotated by human experts. this dataset has visual and audio clues that provides additional context for text interpretation. existing datasets for metaphor interpretation (see table a2 ) and generation (see table a3 ) are included in the appendix a. bizzoni and lappin (2018a) posed the metaphor interpretation task as an entailment problem and provided a collection of 200 metaphorical sentences with four paraphrases. @xcite introduced a corpus of over 10k creative sentences based on the winograd schema to test common sense reasoning of models for figurative text. an example is "the dinner has the flavor of a rubber duck." with two paraphrases. however, the text itself reveals the inherent property (flavor) which makes it inappropriate for the task of metaphor interpretation. @xcite built a corpus of 1350 verb-object metaphoric pairs with "dictionary definitions". recently, chakrabarty et al. (2022) released flute having 750 metaphors with two paraphrases. this dataset is however built using old sources. to the best of our knowledge, there is no goldlabel dataset to evaluate metaphor generation tasks. @xcite provide a method to generate silver labels using a bert model finetuned on vua and lcc (see table a1 ). the authors ensure quality by considering sentences with @xmath2.95 (pg 4252, sec 2.1). however, it is worth noting that 83.38% of the predicted samples had @xmath3.95. through this work, we aim to bridge the gap for a high-quality contemporary dataset of english metaphors. the proposed dataset has fresh, expertauthored, thought-provoking metaphors that were used in contemporary world contexts. additionally, we help address the gold data scarcity challenging metaphor interpretation and generation tasks by providing samples for all three use cases i.e. detection, interpretation and generation. ## proposed approach figurative text annotation is a non-trivial task that demands significant cognitive effort and time. below are the key concerns that we considered while designing the annotation pipeline: to alleviate these concerns, we use state-of-theart large language models (llms) for generation of silver candidates. with the help of a llm, we identify possibly metaphorical and literal headlines that are then randomly sampled for manual annotation. the intent is to improve the likelihood of seeing metaphorical headlines during manual annotation. we also generate a diverse set of metaphorical and literal candidates using an llm for human evaluation. we believe that human annotators will be able to appreciate novel interpretations and metaphorical mappings when presented, and will be quick to discard incorrect or absurd connections. this will ensure diversity as well as help us overcome the need for highly skilled annotators. the proposed annotation pipeline is illustrated in figure 1 . ## conclusion in this paper, we proposed newsmet formed from contemporary news headlines, hand-annotated with metaphorical verbs. the samples are provided with metaphorical and literal interpretations. this dataset is useful for building and evaluating automated systems to detect, interpret as well as generate metaphors. our analysis cautions against the blind use of fine-tuned metaphor detection models to annotate new corpora. however, llms could be of great help in curating diverse metaphorical candidate sets. the proposed dataset has a variety of news sources such as reliable and bias that can be useful in understanding the role of metaphors in news framing and hyperpartisan content. machine translation of figurative text is an under explored research area. the literal candidates associated with metaphorical samples can be used to automatically evaluate the quality of translations. it would also be interesting to evaluate llms for the task of generating culturally coherent metaphors in translated news headlines.