The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation (2023)

Abstract: Human evaluation in often considered to be the gold standard method of evaluating a Natural Language Generation system. However, whilst its importance is accepted by the community at large, the quality of its execution is often brought into question. In this position paper, we argue that the generation of more esoteric forms of language - humour, irony and sarcasm - constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance, and every effort should be made to report demographic characteristics wherever possible, in the interest of transparency and replicability. We support these claims with an overview of each language form and an analysis of examples in terms of how their interpretation is affected by different participant variables. We additionally perform a critical survey of recent works in NLG to assess how well evaluation procedures are reported in this subdomain, and note a severe lack of open reporting of evaluator demographic information, and a significant reliance on crowdsourcing platforms for recruitment.

## introduction the aim of a natural language generation (nlg) system is to generate coherent and well-formed text of a particular type, usually given an input such as a prompt @xcite , outline @xcite , topic @xcite , or data @xcite . as a result, nlg contains many sub-domains such as summarisation @xcite , data-to-text generation @xcite , dialogue generation @xcite , story generation @xcite , and humour generation @xcite . whilst it may be possible for any competent speaker of the target language to sufficiently understand generated outputs, many language forms may be considered * corresponding author more esoteric, such as humour, irony, and sarcasmwhere interpretations of such language are highly intertwined with more broad demographic characteristics, as well as individual and idiosyncratic factors. human evaluation is viewed as the foremost important form of evaluation for any nlg system @xcite , owing largely to its direct capture of the opinions of the (usually) human end-user. 1 this is made even more prominent when considering the low concurrent validity that many easily deployed automatic evaluation metrics have with human judgements @xcite @xcite . whilst such evaluation procedures are considered preferable over automatic metrics (particularly in the cases where no reliable automatic metrics are available), this does not mean that they are infallible. numerous recent works have outlined the inconsistency with which human evaluation procedures are executed, and increasing attention is being brought to the knock-on effects that such oversights may cause @xcite . in this paper, we present an overview of 3 particular types of language: humour, irony and sarcasm -and demonstrate how different evaluator demographics 2 may affect the interpretation of such language, and therefore its evaluation in relevant systems. furthermore, we present a critical survey and assessment of 17 nlg research papers from between 2018 and 2023 from the acl anthology that concern the generation of these types of language, and analyse behaviour in regard to the level of transparency used when reporting details of the human evaluation procedure. ## related works in recent years, the execution of human evaluation in nlg has become an even more significant topic, with myriad top-tier venues hosting dedicated tracks and workshops specifically focused on the evaluation of evaluation itself, such as the humeval workshops at eacl 2021, acl 2022 and ranlp 2023. 3 as a product of such venues, multiple recent works have been proposed that aim to highlight common inadequacies in multiple aspects of human evaluation, including the design of assessment criteria, the recruitment and training of participants, the use of biased response elicitation techniques @xcite , and the transparency of reporting these processes. @xcite assess a 20 year time-span of work within nlg (165 papers between 2000-2019) in regard to how evaluation procedures are performed, finding that there is substantial variation in how evaluation criteria are named and portrayed. resulting from this, they produce normalised mappings between different criteria descriptions and their underlying evaluation focus and call for human evaluation criteria to be standardised, in addition to the creation of evaluation sheets outlining best practices for response elicitation from evaluators. @xcite also assess nlg research in terms of response elicitation methods with a particular focus on the use of likert scales and common deviations from best-practice within 135 papers from a 10 year period (e.g. whether or not to view them as ordinal or interval in nature, and consequently how to perform statistical significance testing). further issues have also been identified by @xcite with the framing of evaluation questions and participant recruitment methods, including the presence of demand characteristics in evaluators being more common on crowdsourcing platforms as participants vie for limited positions in hits and wish to give researchers the results they believe they desire (under the belief it will lead to more future work). 4 furthermore, these issues are exacerbated by cases wherein researchers reject annotations outright due to not falling in line with their expectations, or pay inadequately for services on crowdsourcing platforms @xcite . however, whilst automatic metrics may have many drawbacks for evaluating nlg, human evaluation has been suggested to be imperfect by its very nature, with human judgements being inconsistent. for example, @xcite find that human evaluators range significantly in their preconceptions of the quality of text nlg systems can produce, resulting in evaluators having biased perspectives when assessing text quality (a bias that is only somewhat overcome with the provision of training materials to standardise criteria definitions among evaluators). in order to partially mitigate these evaluation pitfalls, van der @xcite suggest a range of best practices, including the reporting of confidence intervals for results, using multiple ratings of each criteria, and using randomisation and counterbalancing when assigning evaluators to conditions (e.g. human-vs model-generated text). additionally, due to many of the problems that we outline in this work, the domain is currently starting to demonstrate the progression of a paradigm shift towards understanding the role of human disagreement as a valuable feature, rather than noise, which is also resulting in the rise of perspectivist approaches @xcite @xcite . additionally, whilst the use of such language can lead to different responses from different people, this effect is compounded in human-computer interactions, where the communicative divide is even more pronounced @xcite . ## introducing humour, sarcasm, and irony in the following section we discuss three primary examples of language types where interpretations and opinions are highly affected by a range of nuanced participant variables. for each type, we outline the nature of this language alongside a range of examples, and reflect upon how interpretation of these examples may differ across demographics. ## critical survey following existing works that critically review human evaluation procedures in natural language generation, we present a critical survey of works in the area of computational humour, irony, and sarcasm. we begin by reviewing the scope and range of our literature search for this review and the prerequisite criteria for incorporation. following this, we motivate the need for increased care in this specific domain of language by briefly exploring the prevalence of research in the area, both in terms of generative works, as well as those on the detection of such language. as previously discussed, irony and sarcasm are highly related language forms, resulting in the common use of the terms interchangeably. however, to ensure that the intended definition is respected (the correctness of which cannot always be inferred with complete certainty) we report irony and sarcasm separately based on the titles of the papers. ## discussion and proposed action points our analysis has shown a distinct lack of reporting on potentially relevant demographic information in ad to tackle this issue, we suggest future work on the evaluation of (especially subjective) generative tasks to include an "evaluation statement", akin to the data statements proposed by @xcite . in line with their work, we suggest the inclusion of the following points for such statements: while the language proficiency of the participants is relevant to all tasks, we recognise that other characteristics such as gender and cultural background (e.g. religion and political leaning) may not always apply. still, it would be beneficial to report which facets were taken into consideration, as they may become relevant in hindsight, and in the case of dataset annotations, facilitate more extensive use of any collected data in future works. similarly, task proficiency may sometimes seem irrelevant for subjective and intuitive tasks. however, @xcite and @xcite demonstrate the preference of expert evaluators for complex linguistic tasks. additionally, authors may be reluctant to present only moderate agreement for subjective tasks due to concerns regarding evaluation quality. however, this criterion is highly relevant as it describes the expected degree of subjectivity for a task, a concept the community should be able to embrace. in addition to these points, we also encourage increased interest in the area of perspectivist approaches to the study of these language forms, and in nlp as a whole. finally, the intentional omission of many of these criteria (such as iaa and language proficiency, in particular) may be considered intentionally deceptive, and therefore at odds with the foundations of open science. ## conclusion in conclusion, we reiterate the importance of the role that human evaluation plays in assessing the outputs of natural language generation systems. we have further demonstrated why particular types of language such as humour, irony, and sarcasm demand additional attention to be paid during the construction of human evaluation processes, particularly as it pertains to the demographic variables of selected evaluators. in no way do we discourage the use of the "non-native" or l2+ speakers of a target language in such evaluation (or any other minority demographic), nor do we likewise encourage the use of "native" or l1 target-language speakers (or any other majority demographic) as the true "gold standard". rather, we aim to encourage additional thoughts and reflections in the community when selecting panels of human evaluators for such language, analysing results in light of their demographic characteristics as a feature, rather than a bug, whilst openly reporting such decisions wherever possible. to this end, we provide a list of suggested points that can be included as part of an "evaluation statement". additionally, we advocate for the use of stratified sampling techniques to gain more varied evaluator panels.