H ow Did This Get Funded?! A utomatically Identifying Quirky Scientific Achievements (2021)

Abstract: Humor is an important social phenomenon, serving complex social and psychological functions. However, despite being studied for millennia humor is computationally not well understood, often considered an AI-complete problem. In this work, we introduce a novel setting in humor mining: automatically detecting funny and unusual scientific papers. We are inspired by the Ig Nobel prize, a satirical prize awarded annually to celebrate funny scientific achievements (example past winner: “Are cows more likely to lie down the longer they stand?”). This challenging task has unique characteristics that make it particularly suitable for automatic learning. We construct a dataset containing thousands of funny papers and use it to learn classifiers, combining findings from psychology and linguistics with recent advances in NLP. We use our models to identify potentially funny papers in a large dataset of over 630,000 articles. The results demonstrate the potential of our methods, and more broadly the utility of integrating state-of-the-art NLP methods with insights from more traditional disciplines

## introduction humor is an important aspect of the way we interact with each other, serving complex social functions @xcite . humor can function either as a lubricant or as an abrasive: it can be used as a key for improving interpersonal relations and building trust @xcite , or help us work through difficult topics. it can also aid in breaking taboos and holding power to account. enhancing the humor capabilities of computers has tremendous potential to better understand interactions between people, as well as build more natural human-computer interfaces. ## related work humor in the humanities. a large body of theoretical work on humor stems from linguistics and psychology. @xcite divided humor into three categories: incongruity, sexual, and nonsense (and created a three-dimensional humor test to account for them). since our task is to detect humor in scientific contributions, we believe that the third category can be neglected under the assumption that no-nonsense article would (or at least, should) be published (notable exception: the sokal hoax @xcite ). the first category, incongruity, was first fully conceptualized by kant in the eighteenth century @xcite . the well-agreed extensions to incongruity theory are the linguistics incongruity resolution model and semantic script theory of humor @xcite . both state that if a situation ended in a manner that contradicted our prediction (in our case, the title contains an unexpected term) and there exists a different, less likely rule to explain it -the result is a humorous experience. simply put, the source of humor lies in 2 github.com/nadavborenstein/iggy violation of expectations. example ig nobel winners include: "will humans swim faster or slower in syrup?" and "coordination modes in the multisegmental dynamics of hula hooping". the second category, sex-related humor is also common among ig nobel winning papers. examples include: "effect of different types of textiles on sexual activity. experimental study" and "magnetic resonance imaging of male and female genitals during coitus and female sexual arousal". humor detection in ai. most computational humor detection work done in the context of ai relies on supervised or semi-supervised methods and focuses on specific, narrow, types of jokes or humor. humor detection is usually formulated as a binary text classification problem. example domains include knock-knock jokes @xcite , one-liners @xcite @xcite @xcite , humorous tweets @xcite @xcite , humorous product reviews @xcite , tv sitcoms @xcite , short stories @xcite , cartoons captions @xcite , and even "that's what she said" jokes @xcite . related tasks such as irony, sarcasm and satire have also been explored in similarly narrow domains @xcite @xcite . ## problem formulation and dataset our goal in this paper is to automatically identify candidates for the ig nobel prize. more precisely, to automatically detect humor in scientific papers. first, we consider the question of input to our algorithm. @xcite found a strong correlation between funny title and humorous subject in scientific papers. motivated by this correlation, we manually inspected a subset of ig nobel winners. for the vast majority of them, reading the title was enough to determine whether it is funny; very rarely did we need to read the abstract, let alone the full paper. typical past winners' titles include "why do old men have big ears?" and "if you drop it, should you eat it? scientists weigh in on the 5-second rule". an example of a non-informative title is "pouring flows", a paper calculating the optimal way to dunk a biscuit in a cup of tea. based on this observation, we decided to focus on the papers' titles. more formally: given a title t of an article, our goal is to learn a binary function ϕ(t) → {0, 1}, reflecting whether the paper is humorous, or 'ig nobel-worthy'. the main challenge, of course, lies in the construction of ϕ. to take a data-driven approach to tackle this problem, we crafted a first-of-its-kind dataset containing titles of funny scientific papers 2 . we started from the 211 ig nobel winners. next, we manually collected humorous papers from online forums and blogs foot_1 , resulting in 1,707 papers. we manually verified all of these papers can be used as positive examples. in section 6 we give more indication these papers are indeed useful for our task. for negative examples, we randomly sampled 1,707 titles from semantic scholar foot_2 (to obtain a balanced dataset). we then classify each paper into one of the following scientific fields: neuroscience, medicine, biology, or exact sciences foot_3 . we balanced the dataset in a per-field manner. while some of these randomly sampled papers could, in principle, be funny, the vast majority of scientific papers are not (we validated this assumption through sampling). ## humor-theory inspired features in deep learning, architecture engineering largely took the place of feature engineering. one of the goals of our work is to evaluate the value of features inspired by domain experts. in this section, we describe and formalize 127 features implementing insights from humor literature. to validate the predictive power of the features that require training, we divide our data to train and test sets (80%/20%). we now describe the four major feature families. ## models we can now create models to automatically detect scientific humor. as mentioned in section 4, one of our goals in this paper is to compare between the nlp sota huge-models approach and the literature-inspired approach. thus, we trained a binary multi-layer perceptron (mlp) classifier using our dataset (described in section 3, see reproducibility details in appendix c.2), receiving as input the 127 features from section 4. we named this classifier 'iggy', after the ig nobel prize. as baselines representing the contemporary nlp approach (requiring huge compute and training data), we used bert @xcite and scibert @xcite , which is a bert variant optimized on scientific corpora, rendering it potentially more relevant for our task. we finetuned scibert and bert for ig nobel classification using our dataset (see appendix c.3 for implementation details). we also experimented with two models combining bert/scibert with our features (see figure 6 in appendix c.4), denoted as bert f / scibert f . in the spirit of the original bert paper, we added two linear layers on top of the models and used a standard cross-entropy loss. the input to this final mlp is the concatenation of two vectors: our features' embedding and the last hidden vector from bert/scibert ([cls]). see appendix c.4 for implementation details. for the sake of completeness, we note that we also conducted exploratory experiments with simple syntactic baselines (title length, maximal word length, title containing a question, title containing a colon) as well as bert trained on sarcasm detection 8 . none of these baselines was strong enough on its own. we note that the colon-baseline tended to catch smart-aleck titles, but the topic was not necessarily funny. the sarcasm baseline achieved near guess-level accuracy (0.482), emphasizing the distinction between the two humor tasks. ## evaluation on the dataset we first evaluate the five models (iggy, scibert, bert, scibert f and bert f ) on our labeled dataset in terms of general accuracy and ig nobel retrieval ability. as naive baselines, we added two bag of words (bow) based classifies: random forest (rf) and logistic regression (lr). accuracy. we randomly split the dataset to train, development, and test sets (80%-10%-10%), and used the development set to tune hyper-parameters (e.g., learning rate, number of training epochs). ig nobel winners retrieval. our positive examples consist of 211 ig nobel winners and additional 1,496 humorous papers found on the web. thus, the portion of real ig nobel winning papers in our data is relatively small. we now measure whether our web-originated papers serve as a good proxy for ig nobel winners. thus, we split the dataset differently: the test set consists of the 211 ig nobel winners, plus a random sample of 211 negative titles (slightly increasing the test set size to 12%). train set consists of the remaining 2,992 papers. this experiment follows our initial inspiration of finding ig nobel-worthy papers, as we test our models' ability to retrieve only the real winners. in an exploratory study, we asked people to rate the funniness of titles on a likert scale of 1-5. we noted that people tended to confuse funny research topic and funny title. for example, titles like "are you certain about sirt?" or "nash may be trash" received high funniness scores, even though the research topic is not even clear from the title. to mitigate this problem, we redesigned the study to include two 5-point likert scale questions: 1) whether the title is funny, and 2) whether the research topic is funny. this addition seems to indeed help workers understand the task better. example papers rated as serious title, funny topic include "hat-wearing patterns in spectators attending baseball games: a 10-year retrospective comparison". funny title, serious topic include "slicing the psychoanalytic pie: or, shall we bake a new one? commentary on greenberg". unless stated otherwise, the evaluation in the reminder of the paper was done on the "funny topic" likert scale. we paid crowd workers $0.04 per title. as this task is challenging, we created a qualification test with 4 titles (8 questions), allowing for one mis-take. the code for task and test can be found in the repository 2 . we also required workers to have completed at least 1,000 approved hits with at least 97% success rate. all algorithms classified and ranked (according to certainty) all 630k papers. however, in any reasonable use-case, only the top of the ranked list will ever be examined. there is a large body of work, both in academia and industry, studying how people interact with ranked lists (in particular, search result pages) @xcite . many information retrieval algorithms assume the likelihood of the user examining a result to exponentially decrease with rank. the conventional wisdom is that users rarely venture into the second page of search results. thus, we posit that in our scenario of ig nobel recommendations, users will be willing to read only the several tens of results. we choose to evaluate the top-300 titles for each of our five models, to study (in addition to the performance at the top of the list) how performance decays. we also included a baseline of 300 randomly sampled titles from semantic scholar. altogether we evaluated 1375 titles (due to overlap). each title was rated by five crowd workers. overall, 13 different workers passed our test. seven workers annotated less than 300 titles, while four annotated above 1,300 each. decision rule. each title was rated by five different crowd workers on a 1-5 scale. there are several reasonable ways to aggregate these five continuous scores to a binary decision. a commonly-used aggregation method is the majority vote. the majority vote should return the clear-cut humorous titles. however, we stress that humor is very subjective thus, we explored more aggregation methods foot_7 . our hypothesis class is of the general form "at least k annotators gave a score at least m" foot_8 . to pick the best rule, we conducted two exploratory experiments: in the first one, we recruited an expert scientist and thoroughly trained him on the problem. he then rated 90 titles and we measured the correlation of different aggregations with his ratings. results are summarized in in the second experiment, we used the exact same experimental setup as the original task, but with labeled data. we used 100 ig nobel winners as positives and a random sample of 100 papers as negatives. the idea was to see how crowd workers rate papers that we know are funny (or not). we believe the method outlined in this section could be more broadly applicable to aggregation of crowd sourced annotations for subjective questions. results. figure 1 shows precision at k for the top-rated 300 titles according to each model. the random baseline is ∼ 0.03. upon closer inspection, these seem to be false positives of the annotation. we have argued that in our setting it is reasonable for users to read the first several tens of results. in this range, iggy slightly outperforms the other four models (bert is particularly bad, as it picks up on short, non-informative titles). for larger k values scibert and bert f take the lead. we note that even at @xmath0, all models still achieve considerable (absolute) precision. we obtain similar results using normalized discounted cumulative gain (ndcg), a common measure for ranking quality (see we stress that iggy is a small and simple network (∼ 33k parameters), compared to pretrained 110 million parameters bert-based models. yet despite its simplicity, iggy's performance is roughly comparable to bert-based methods. we believe this demonstrates the power of implementing insights from domain experts. we hypothesize that if the fine-tuning dataset were larger, bert f and scibert f would outperform the other models. ## conclusions & future work in this work, we presented a novel task in humor recognition -detecting funny and unusual scientific papers, which represents a subtle and sophisticated humor type. it has important characteristics (short, simple syntax, stand-alone) making it a (relatively) clean setting to explore computational humor. we created a dataset of funny papers and constructed models, distilling humor literature into features as well as harnessing sota advances in nlp. we conducted experiments both on our dataset and in a real-world setting, identifying funny papers in a corpus of over 0.6m papers. all models were able to identify funny papers, achieving high ndcg scores. interestingly, despite the simplicity of the literature-based iggy, its performance was overall comparable to complex, bert-based models. our dataset can be further used for various humor related tasks. for example, it is possible to use it to create an aligned corpus, pairing every funny paper title with a nearly identical but serious title, using methods similar to @xcite . this would allow us to understand why a paper is funny at a finer granularity, by identifying the exact words that make the difference. this technique will also allow exploring different types of "funny". another possible use of our dataset is to collect additional meta-data about the papers (e.g., citations, author information) to explore questions about whether funny science achieves disproportionate attention and engagement, who tends to produce it (and at which career stage), with implications to science of science and science communication. another interesting direction is to expand beyond paper titles and consider the paper abstract, or even full text. this could be useful in examples such as the ig nobel winner "cure for a headache", which takes inspiration from woodpeckers to help cure headaches in humans. finally, we believe multi-task learning is a direction worth pursuing towards creating a more holistic and robust humor classifier. in multi-task learning, the learner is challenged to solve multiple problems at the same time, often resulting in better generalization and better performance on each individual task @xcite . as multi-task learning enables unraveling cross-task similarities, we believe it might be particularly fruitful to apply to tasks highlighting different aspects of humor. we believe our dataset, combined with other task specific humor datasets, could assist in pursuing such a direction. despite the tongue-in-cheek nature of our task, we believe that computational humor has tremendous potential to create personable interactions, and can greatly contribute to a range of nlp applications, from chatbots to educational tutors. we also wish to promote complementing data-driven research with insights from more-traditional fields. we believe combining such insights could, in addition to improving performance, enrich our understanding of core aspects of being human. figure 4 : feature importance shap analysis done on the iggy model. the top variables according to this plot contribute more than the bottom ones (have high predictive power). the analysis reveals that the highest contribution corresponds to short, funny, and simple words (where simplicity was measured using features such as aoa and readability). we also notice that features which are based on the one-liners lms contributed much to the final prediction, meaning that there is indeed some similarity between funny titles and short jokes.