I ’m sure you’re a real scholar yourself: Exploring Ironic Content Generation by Large Language Models (2024)

Abstract: Generating ironic content is challenging: it requires a nuanced understanding of context and implicit references and balancing seriousness and playfulness. Moreover, irony is highly subjective and can depend on various factors, such as social, cultural, or generational aspects. This paper explores whether Large Language Models (LLMs) can learn to generate ironic responses to social media posts. To do so, we fine-tune two models to generate ironic and non-ironic content and deeply analyze their outputs’ linguistic characteristics, their connection to the original post, and their similarity to the human-written replies. We also conduct a large-scale human evaluation of the outputs. Additionally, we investigate whether LLMs can learn a form of irony tied to a generational perspective, with mixed results.

## introduction irony is a complex linguistic device that exploits semantic inversion, conveying the opposite of what is believed and what actually is. irony is thus a complex phenomenon, the generation and recognition of which requires a nuanced understanding of the context, the tone, and the underlying meaning @xcite . moreover, recognizing irony involves various subjective factors and can be culture-specific and based on shared cultural references, norms, and world models @xcite . automatically generating ironic content is thus an interesting but challenging task, which is still difficult for state-of-the-art generative systems. this * equal contribution. the work of sc was done while at the university of turin. 1 the data are released under the creative commons attribution non commercial 4.0 international license and available at: https://github.com/dipinfo-unito/ ironiccontentgeneration paper explores whether large language models (llms) can generate content perceived as ironic by human readers. specifically, we focus on contextual irony generation and generate ironic replies to a given social media post. leveraging human-written short post-reply conversations, we first fine-tune two models to generate ironic and not-ironic replies to a post. then, we deeply analyze their outputs from a linguistic perspective, focusing on their intrinsic characteristics as well as on their relation to the original post and the similarity to the human-written replies. to explore whether such replies are perceived as ironic by human annotators, we perform a large-scale human evaluation. while less linguistically rich than human-written replies, ironic model outputs present interesting references to implicit context and are characterized by sarcastic remarks. moreover, replies generated by the model fine-tuned with ironic post-reply pairs are perceived as more ironic by humans in the vast majority of cases when compared to those generated by a model trained on non-ironic data. inspired by previous work that identifies age as a highly polarizing dimension in the recognition of irony @xcite , we then try to create models that generated content specifically identified as ironic by old or young annotators, with mixed results. in short, our contributions are the following: ## related works while humor and sarcasm generation is becoming increasingly explored in the nlg community @xcite @xcite , few studies focus on irony @xcite . as highlighted by @xcite , a significant amount of work in the field centers on puns generation, especially for phonetics and word senses @xcite . on the other hand, works in unsupervised sarcasm generation based on sarcasm theory are emerging @xcite @xcite , where the linguistic phenomenon characteristics (e.g., reversal of valence, semantic or context incongruity) are used to produce sarcastic messages. focusing on irony generation can be of high importance especially when thinking about the benefits of verbal irony as an instrument to mediate and negotiate boiling emotions @xcite . researchers in irony detection have recently demonstrated annotators' cultural backgrounds affect their data labeling choices @xcite . this body of work shows how taking into account the subjectivity of the annotators turned out to be effective in humor extraction @xcite and irony detection @xcite . personalizing response generation has been extensively studied across various domains, evolving significantly with the advent of large-scale social media data and the success of sequence-tosequence frameworks @xcite , followed by advancements in large language models @xcite . numerous models have been developed to incorporate user-specific information into dialogue systems, thereby improving their responsiveness and relevance @xcite . demographic characteristics and individual perception can play a key role, especially when generating highly subjective language phenomena, such as irony, humor, and sarcasm. recent works have pointed out the influence of sociodemographic factors both in human annotation (bender and friedman, 2018) and evaluation @xcite of pragmatic phenomena, highlighting the importance of reporting them and taking into account their influence in the machine learning pipeline. a demographic-based approach to generating humor is found by @xcite who proposed a location-specific humor framework, collected a dataset, and hired us and indian annotators. we noticed a general scarcity of datasets that allow us to consider demographic backgrounds in generating irony. thus, we used the english perspectivist irony corpus @xcite , originally designed for irony detection, for irony generation, containing post-reply pairs suitable for a contextual irony generation. ## ironic reply generation in this section, we aim to determine whether llms can generate replies perceived as ironic by human readers. starting from the epic disaggregated dataset, we use instance-based majority vote to ag-gregate the labels and exclude instances for which no majority is reached. then, we select an llm and generate ironic outputs. we examine the outputs to identify any linguistic patterns that provide insights into the irony of the sentences. finally, we have humans evaluate the generated outputs. we performed the bulk of the experiments using mistral 7b @xcite 2 , a state-ofthe-art pretrained llm for english, as our base model. some preliminary experiments were also performed by using llama2-7b @xcite ; output examples are reported in appendix ??. to collect enough ratings to ensure that the results were meaningful, we then decided to work with a single model, i.e., mistral. ## building age-specific models previous work has shown that irony detection is a highly subjective task @xcite , where the annotators' background plays a crucial role. specifically, @xcite highlighted the importance of considering annotators' sociodemographic characteristics. in their analysis, they found the annotators' age is one of the sociodemographic dimensions corresponding to the highest polarization in the recognition of irony. example 2 shows a case from the epic dataset where geny and boomer's annotators labeled the reply as ironic and non-ironic respectively. to understand whether we could encode an age-specific perception of irony into an llm, we trained two age-specific models to generate ironic replies. starting from the disaggregated annotations available in the original dataset, we partitioned the original annotators into young (i.e., < 42 years old) and old (i.e., >= 42 years old), using the available metadata foot_3 , then we constructed two separate semi-aggregated gold standards. we discarded instances for with no majority was reached and trained the models on ironic instances only. given both gold standards, we fine-tuned two separate models on the age-specific ironic instances, using the same method discussed in section 4.2 foot_4 . we will refer to these models as y-model and omodel. given 455 test posts, we generated the corresponding replies using both models. after manually excluding 17 post-reply pairs not in english and removing 21 identical generated replies from both models, we obtained a new corpus of 417 sentences to be analyzed and annotated. ## conclusion this paper investigates llms' capabilities in generating ironic replies given a specific social media post. we first presented strategies for training llms to generate ironic content, performed a linguistic analysis, and conducted a human evaluation through crowdsourcing. we found that a model trained on ironic data can generate outputs that, though less rich from a linguistic perspective when compared to human-written text, are typically perceived as ironic by humans. furthermore, we conducted linguistic analyses to elicit differences in linguistic patterns between ironic and non-ironic replies and the contextual outputs of the models. the generated sentences and the associated evaluations, which we will release with this paper, will contribute as a resource for irony classification and human-vs machine-generated irony detection. we then tested the hypothesis that llms could generate content perceived differently based on the age of the annotators. this choice was motivated by finding in @xcite that different annotators' ages result in high polarization in the recognition of irony. again, we conducted a linguistic analysis to elicit the various differences between responses considered ironic by both younger and older populations, compared to the original sentences in the epic corpus and those produced by two new models trained to generate ironic content for younger and older populations. the results of this experiment do not confirm the initial hypothesis, and we believe this might be due to the small size of the dataset and to its design; we will also consider different age ranges or modifying the experiment's setting, such as choosing different questions.