i S arcasm: A Dataset of Intended Sarcasm (2020)

Abstract: We consider the distinction between intended and perceived sarcasm in the context of textual sarcasm detection. The former occurs when an utterance is sarcastic from the perspective of its author, while the latter occurs when the utterance is interpreted as sarcastic by the audience. We show the limitations of previous labelling methods in capturing intended sarcasm and introduce the iSarcasm dataset of tweets labeled for sarcasm directly by their authors. Examining the state-of-the-art sarcasm detection models on our dataset showed low performance compared to previously studied datasets, which indicates that these datasets might be biased or obvious and sarcasm could be a phenomenon under-studied computationally thus far. By providing the iSarcasm dataset, we aim to encourage future NLP research to develop methods for detecting sarcasm in text as intended by the authors of the text, not as labeled under assumptions that we demonstrate to be sub-optimal.

## introduction sarcasm is a form of irony that occurs when there is some discrepancy between the literal and intended meanings of an utterance. this discrepancy is used to express dissociation towards a previous proposition, often in the form of contempt or derogation @xcite . sarcasm is omnipresent in social media text and can be highly disruptive of systems that harness this data for sentiment and emotion analysis @xcite . it is therefore imperative to devise models for sarcasm detection. the effectiveness of such models depends on the availability and quality of labelled data used for training. collecting such data is challenging due to the subjective nature of sarcasm. for instance, @xcite notice a lack of consistence in how sarcasm is used by people of different socio-cultural backgrounds. as a result, an utterance intended sarcastic by its author might not be perceived as such by audiences of different backgrounds @xcite . there are two methods used so far to label texts for sarcasm: distant supervision, where texts are considered sarcastic if they meet predefined criteria, such as including specific hashtags; and manual labelling by human annotators. we believe both methods are sub-optimal for capturing the sarcastic intention of the authors of the texts. as a result, existing models trained on such datasets might be optimized to capture the noise induced by these labelling methods. in this paper, we present the isarcasm dataset of tweets labelled for sarcasm by their authors. to our knowledge, this is the first attempt to create noisefree examples of intended sarcasm. in a survey, we asked twitter users to provide both sarcastic and non-sarcastic tweets that they had posted in the past. for each sarcastic tweet, we asked them to explain why it was sarcastic and how they would convey the same meaning non-sarcastically. labels were thus implicitly specified by the authors themselves. we implemented restrictive quality control to exclude spurious survey responses. we then asked a trained linguist to manually check the sarcastic tweets and further label them into the subcategories of sarcasm defined by @xcite . we further collected third-party sarcasm labels for the tweets in isarcasm from workers on a crowdsourcing platform. third-party annotation for sarcasm has been conducted before @xcite @xcite , but no studies checked the ability of the annotators to capture the actual sarcasm meant by the authors. on isarcasm, annotators recognise author labels with an f-score of 0.616. this indicates that sarcasm is a subjective phenomenon, challenging even for humans to detect. further, it demonstrates that using third-party annotators to label texts for sarcasm can lead to inaccurate labels. we implemented state-of-the-art sarcasm detection models @xcite @xcite and tested them on isarcasm, to investigate their effectiveness in capturing sarcasm as intended by the authors. while these models achieve f-scores reaching 0.874 on existing datasets, they yield a maximum f-score of 0.364 on isarcasm, suggesting that previous datasets might be biased or obvious. this highlights the importance of developing new approaches for sarcasm detection that are more effective at capturing author intention. ## limitations of current labelling methods in this section, we discuss limitations of current labelling methods that make them sub-optimal for capturing intended sarcasm. we demonstrate them empirically on the riloff dataset @xcite , which uses a hybrid approach for labelling. ## detecting intended sarcasm in the following, we examine the effectiveness of state-of-the-art sarcasm detection models on isarcasm. we aim to investigate their ability to detect intended sarcasm rather than sarcasm labeled using distant supervision or manual annotation. as we have shown, these labelling methods could produce noisy labels. we experiment with those models that have achieved state-of-the-art results on previous benchmark datasets for sarcasm detection. ## conclusion and future work in this paper, we presented isarcasm, a dataset of intended sarcasm consisting of 4,484 tweets labeled and explained by their authors, and further revised and categorised by an expert linguistic. we believe this dataset will allow future work in sarcasm detection to progress in a setting free of the noise found in existing datasets. we saw that computational models perform poorly in detecting sarcasm in the new dataset, indicating that the sarcasm detection task might be more challenging compared to how it was seen in earlier research. we aim to promote research in sarcasm detection, and to encourage future investigations into sarcasm in general and how it is perceived across cultures.