Should a Chatbot be Sarcastic? Understanding User Preferences Towards Sarcasm Generation (2022)

Abstract: Previous sarcasm generation research has focused on how to generate text that people perceive as sarcastic to create more human-like interactions. In this paper, we argue that we should first turn our attention to the question of when sarcasm should be generated, finding that humans consider sarcastic responses inappropriate to many input utterances. Next, we use a theory-driven framework for generating sarcastic responses, which allows us to control the linguistic devices included during generation. For each device, we investigate how much humans associate it with sarcasm, finding that pragmatic insincerity and emotional markers are devices crucial for making sarcasm recognisable.

## introduction the prevalence of sarcasm on the social web @xcite has motivated computational investigations across the nlp community. most focus on textual sarcasm detection, the task of classifying whether or not a given text is sarcastic @xcite @xcite @xcite @xcite @xcite . a recent research direction considers sarcasm generation. approaches to sarcasm generation introduced so far @xcite @xcite are mainly motivated by the potential to create more approachable, human-like conversational agents, considering that sarcasm is a natural part of human discourse. we suggest reconsidering this motivation, as a community, for two reasons. first, in human discourse, sarcasm is not a communicative goal in itself. rather, it can be used to achieve a wide variety of goals. some of these goals, such as to diminish the impact of criticism @xcite , to create humour @xcite , to praise @xcite , or to strengthen relationships @xcite , might be desirable in human-machine interactions as well. however, other goals, such as criticising, mocking, or expressing dissociation, often with surface contempt or derogation @xcite , might not be desirable in human-machine interactions. second, the communicative goals mentioned above were observed in human interactions. even when a machine seeks potentially desirable goals, it is unclear whether sarcastic utterances have the same effect on humans when coming from machines. therefore, we suggest it is imperative, not least from an ethical perspective, to consider the following research questions (rqs): rq1. when should a chatbot be sarcastic? here, by flavour, we mean a specific conjunction of linguistic devices that humans may associate with sarcasm, such as intensifiers and emotional markers, as introduced in section 3, and expanded upon in section 4. to address our research questions, we suggest the following approach. first, given a set of input utterances, generate several sarcastic responses. each response should be of a specific sarcasm flavour, i.e. should display a specific conjunction of linguistic devices. next, create a survey that asks human participants: to indicate how appropriate it was to respond sarcastically to the input; to select their preferred response; and to rate the sarcasticness of each response, investigating whether they associate the linguistic devices in the response with sarcasm. to achieve this, we require a sarcastic response generator that provides control over the linguistic devices used. most previous generators rely on variants of the traditional theory of sarcasm, which claims that the intended meaning concealed by sarcasm is the opposite of the literal meaning. however, this theory provides a grounding that is neither necessary, nor sufficient, for sarcasm to occur, as discussed in section 3. to overcome this limitation, we recently introduced chandler, a novel modular sarcastic response generation framework @xcite . it is grounded on a formal theory that, from a linguistic-theoretical perspective, specifies devices whose presence is both necessary and sufficient to unambiguously differentiate sarcasm from non-sarcasm. these are allusion to a failed expectation, pragmatic insincerity, and emotional markers. chandler can generate sarcasm of different flavours, and allows control over flavour its output should reflect. herein, we also compare chandler's outputs to those of previous generators, to examine participant preferences toward an even greater range of sarcasm flavours. our results indicate that people find sarcastic responses inappropriate for most input utterances. when sarcasm was considered appropriate, the inputs commonly had a positive sentiment, and often had elements of humour. further, even when considered appropriate, people still did not usually prefer sarcastic responses over non-sarcastic ones. sarcasm was typically preferred when it was also considered funny and not too specific. finally, we identified pragmatic insincerity and emotional markers (cf. section 3) as crucial linguistic devices to include in generating recognizable sarcasm. contributions we summarise our contributions as follows. first, our approach allows us to understand people's preferences about when sarcasm should be used, and how it should be formulated. using this information, we provide guidelines for future work in sarcasm generation. second, observing people's preferences also allows us to quan-titatively evaluate the practical advantages of the formal linguistic theory that grounds chandler. ## related work the earliest work on sarcasm generation is that of @xcite , who introduce sarcasmbot, a sarcastic response generation system. sarcasmbot uses one of eight possible generators, each containing a set of predefined patterns, one of which is instantiated as the response. the generators do not in fact account for the meaning of the input, rather, they only focus on aspects such as the overall sentiment or presence of swear words. further, in our experiments, we noticed that most of the time a fallback generator was employed, returning the simple concatenation of a random positive phrase to a random negative one, from a set of predefined phrases that have no specific connection to the input. @xcite suggest a sarcastic paraphrase generator. they assume that the input is always of negative polarity, and suggest an unsupervised pipeline of four modules to convert such an input u (-) to a sarcastic version. in the sentiment neutralisation module, they filter out negative sentiment words from u (-) to produce u (0) . in the positive sentiment induction module, they modify u (0) to convey positive sentiment, producing u (+) . next, in the negative situation retrieval module, they mine a phrase v (-) that expresses a negative situation. v (-) is selected from a set of predefined phrases, based on the similarity to the original input. finally, the sarcasm synthesis module constructs the sarcastic paraphrase from u (+) and v (-) . @xcite suggest a similar pipeline. their r 3 system first employs a reversal of valence module, which replaces input words of negative valence with their lexical antonyms using wordnet @xcite to produce u (+) . next, it builds an utterance v that is incongruous to u (+) , and generates sarcasm from u (+) and v. previous generators share a limitation that make them unfit for our purposes. mainly, relying on the traditional theory, they identify sarcasm with linguistic incongruity. thus, they only provide this single device for investigation, device that is not sufficient for sarcasm to occur, as discussed in section 3. a further limitation, shared by @xcite and @xcite , is that their generators only work with input utterances of negative sentiment. however, as discussed earlier, sarcastic communication can have many goals, including to praise, or to strengthen friendships. ## linguistic grounding in this section, we describe the implicit display theory, a formal linguistic theory that grounds chandler. previous theories in the traditional theories, sarcasm is created by literally saying one thing but figuratively meaning, or conversationally implicating @xcite , the opposite. however, such incongruity is not necessary for sarcasm. to see this, consider sarcastic understatements such as saying "this was not the best movie ever" to mean the movie was bad. it is also not sufficient. for instance, it also occurs in the construction of certain stylistic devices, such as metaphors, e.g. "time is money". further theories have been suggested to address these limitations, including the echoic mention theory @xcite and its variants @xcite @xcite , and the pretense theory @xcite and its variants @xcite . however they all fail to uniquely identify sarcasm, as argued by @xcite and @xcite . @xcite , the idt focuses specifically on making the distinction between sarcasm and non-sarcasm. we invite the interested reader to consult @xcite for an overview of how it overcomes the limitations of previous theories. we chose it as a grounding for our generation system. ## methodology in this section we look at the methodology employed to address our research questions. specifically, we first select a set of input utterances. next, for each input, we generate four sarcastic responses of different flavours using chandler, and three more responses using other systems. finally, for each input, in a survey, we ask human participants to rate the responses across several dimensions, to understand their preference towards the appropriateness of sarcasm, and which linguistic devices they associate with sarcasm. ## results we now look at the responses that the participants provided in our survey, addressing our rqs. figure 1 shows the mean appropriateness score for each of the five sentiment categories. a oneway anova test between the means yielded a p-value ≈ 0.001. we therefore proceeded with tukey's range test @xcite , to find the means that are significantly different from one another. we noticed that sarcasm was considered significantly more appropriate by survey participants in responses to positive inputs, compared to very positive, and very negative inputs, respectively. this supports our statement from section 2: the assumption of previous state-of-the-art generators that sarcasm should only be generated for negative inputs is problematic. however, even for the positive class, the mean appropriateness is less than 2. this makes it difficult to recommend responding sarcastically based on sentiment only. to gain more insight, we proceeded with a qualitative inspection of the inputs that yielded the highest and lowest appropriateness scores, respectively. we noticed a few main themes, that we labelled joke, family, school, leisure and death. we then asked two humans to label all inputs across these dimensions. a third human resolved all disagreements. finally, we computed the pearson correlation coefficient of each theme with the sarcasm appropriateness score, across all inputs. we noticed a significant (@xmath0.05) positive correlation between appropriateness and the category joke, and significant negative correlation with belonging to the family theme. we show some examples of the theme family with low appropriateness scores in ## recommendations we recommend that future work on sarcasm generation should account for the four main findings: (1) people think sarcasm is inappropriate as a response to most inputs. however, if it is to be used, it is seen as most appropriate when the input is positive, but not extremely positive. people also found sarcasm to be a suitable response to jokes. ## conclusion we have used a linguistically informed framework for sarcasm generation so that we could present human judges with a variety of flavors of sarcastic responses in a range of situations. our findings suggest that sarcasm should not always be generated, but the decision to generate sarcasm itself should informed by user preferences. people find sarcasm most appropriate as a response to positive utterances and cases in which a joking environment has already been established. further, judges preferred sarcasm most when they actually found it to be funny, and most often preferred general sarcastic responses. however, people often preferred nonsarcastic responses even more. we recommend that future work in this area carefully considers both the appropriateness and necessity of generating sarcasm at all.