CH o R a L : Collecting Humor Reaction Labels from Millions of Social Media Users (2021)

Abstract: Humor detection has gained attention in recent years due to the desire to understand user-generated content with figurative language. However, substantial individual and cultural differences in humor perception make it very difficult to collect a large-scale humor dataset with reliable humor labels. We propose CHoRaL, a framework to generate perceived humor labels on Facebook posts, using the naturally available user reactions to these posts with no manual annotation needed. CHoRaL provides both binary labels and continuous scores of humor and non-humor. We present the largest dataset to date with labeled humor on 785K posts related to COVID-19. Additionally, we analyze the expression of COVID-related humor in social media by extracting lexico-semantic and affective features from the posts, and build humor detection models with performance similar to humans. CHoRaL enables the development of large-scale humor detection models on any topic and opens a new path to the study of humor on social media.

## introduction humor is ubiquitous --it forms a crucial part of people's lives both online and off. automatically detecting humor, then, has become an important task, with applications from misinformation to advertising to philosophy. from a psychological perspective, humor represents anything people say or do that others perceive as funny and tends to make them laugh @xcite . humor perception, though, is highly individualistic @xcite , making it hard to reliably annotate humor. researchers have proposed various methods to collect humorous and non-humorous data with minimal annotation needed. most attempts have focused on distinguishing between jokes and news, which both have natural labels on humor and can be scraped automatically. this large stylistic difference makes detecting humor easier --but it is far from most real-world scenarios where humorous and non-humorous texts come from the same domain. another technique collects social media posts by humor-and non-humor-related hashtags, but this method suffers from large data noise and low labeling accuracy @xcite . finally, there have been studies to use the number of reddit upvotes as humor labels @xcite . though this technique sources data from the same domain, that domain is too limited in scope: all the data comes from one single subreddit. this specificity means that the data represents only the humor perception of a particular group of reddit users, dedicated to producing witty jokes. to address these problems of specificity and domain discrepancy in humorous data collection, we propose choral, a framework for collecting humor reaction labels. choral generates perceived humor scores using the naturally available reactions on facebook posts. our framework includes several advantages: (1) labeling humor on any facebook post, without the need for extra human annotations; (2) providing both binary labels and continuous scores for humor and non-humor; we use choral to present the largest dataset to date on humor, containing 785k facebook covid-19 related posts, each assigned a humor score. we chose to focus on covid-19 because of its universality as a phenomenon that affects all facebook users. choral, however, can be easily adapted to other topics, making it the most extendable humor data collection framework yet. ## related work most corpora for textual humor detection use online joke compilations as humor data and more serious sources, like news or proverbs, as nonhumor data. @xcite built a model to distinguish one-liners from short sentences such as news titles, and @xcite extended the work to longer humorous articles and news articles. @xcite identified the semantic structures of humor by studying the differences between puns and news. chen and soo (2018) built deep learning humor detection models on four datasets with jokes as humor data and news as non-humor data. @xcite collected jokes in russian, combining with forum posts that have low similarity to the jokes as nonhumorous samples. more recently, annamoradnejad and zoghi (2020) combined reddit jokes with news headlines and used a bert-based model to classify these two sets of data. for other forms of naturally labeled humorous texts, @xcite obtained humorous tweets with the hashtag "humor" and nonhumorous tweets from other hashtags. @xcite obtained humor scores from a cartoon caption contest, and, similarly, @xcite obtained humorous tweets from the official website of a tv show. @xcite ; @xcite generated humor labels using the audience laughter marker in the transcripts of ted talks. @xcite asked annotators to edit news headlines to make them funny. there are also some hand-annotated humor datasets @xcite . however, these methods either need extensive human annotation or suffer from low label accuracy. the line of work most relevant to our paper is the rjokes dataset @xcite , where humor scores are obtained from the number of upvotes toward each post in the r/jokes subreddit. however, all the posts in the subreddit are intended to be jokes, making the dataset include only successful jokes and failed jokes, which is far from the natural distribution of posts in social media. for multimodal humor detection, researchers used canned laughter in tv sitcoms @xcite , and time-aligned comments in online videos @xcite . multimodal humor is also ex-amined in internet memes @xcite . ## choral framework and dataset in this section, we introduce our facebook post collection process, as well as our algorithm to assign humor and non-humor scores to the posts. although choral can be applied to any topic, we chose covid-19 as the topic for our dataset. there has been extensive discussion on the pandemic with a wide range of audiences, so this topic prevents us from biasing our posts and labels toward a specific demographic group. ## humor analysis affect and sentiment to further investigate the affective component found to be related to humor in previous work @xcite , we computed average activation, imagery, and pleasantness scores for each post using the dal lexicon and sentiment scores using the vader tool. both imagery and pleasantness scores in dal, as well as the sentiment score in vader, are negatively correlated with humor, indicating a more abstract and negative style in humorous posts, which agree with the liwc findings. complexity we computed the percentage of longer words (more than 6 characters), percentage of complex words defined by the dale-chall readability formula @xcite , and the flesch reading ease test @xcite for a readability measurement. all features show that humorous posts have lower complexity. emoji we found the number of emojis in a post to be a humor indicator. specifically, 363 of the 1,621 unique emojis in our dataset are significantly correlated with hs (320 positive, 43 negative), with the "face with tears of joy" emoji having the highest humor correlation. interestingly, humorous posts have generally fewer heart emojis, but more broken heart emoji, echoing our results above that negative sentiment is related to humor. ## humor detection experiments due to the naturally imbalanced distribution of humorous posts in social media, our full dataset skews towards posts with low hs and high ns. to address this imbalance and build humor detection models, we used the 20k posts with the highest hs as positive samples and the 20k posts with the highest ns as the negative samples on humor. we randomly split the 40k posts into training and test sets, respectively consisting of 80% and 20% of the data, and balanced by binary humor labels. pretrained language models such as bert have shown great success when fine-tuned for text classification tasks @xcite , including the task of humor detection @xcite . in our experiments, we fine-tuned 3 pre-trained language models on our choral dataset: robertabase @xcite , a bert-style model pretrained on 160gb of text data including wikipedia, news, and other web texts; bertweet @xcite , a model with bert-base architecture, pre-trained using the roberta procedure but on 845m english tweets; bertweet-covid, based on bertweet but further pre-trained on 23m covidrelated tweets. we trained the models in two settings: continuous regression, where continuous hs is used as ground truth of humor; and binary classification, where high hs posts have a positive label, and the high ns posts have a negative label. all models were fine-tuned for 3 epochs on the training set with a learning rate of 2e-5. to compare the model performance with humans, we asked 3 native english speakers to label 100 random and balanced posts from the test set. the inter-annotator agreement in fleiss' kappa is 0.782. note that due to the potential differences of humor perception between our annotators and general facebook users, the labels provided by annotators were used not as gold labels, but as a baseline for our models. to compare the continuous models with humans directly, we used an empirical threshold of 0.18 hs to convert the predictions into binary labels. ## conclusions and future work in this paper we present the choral framework for automatically collecting humor reaction labels, and the dataset including 785k posts with humor and non-humor scores. we also perform analysis on humor expressions in our dataset and build models to detect humor with performance comparable to human labelers. choral enables the development of humor detection models on any topic, and our dataset has the potential to help broader applications, such as distinguishing malicious misinformation posts and non-malicious humorous posts. furthermore, choral can also be used to label other human reactions such as anger and sadness.