Catchphrase: Automatic Detection of Cultural References (2021)

Abstract: A snowclone is a customizable phrasal template that can be realized in multiple, instantly recognized variants. For example, “* is the new *" (Orange is the new black, 40 is the new 30). Snowclones are extensively used in social media. In this paper, we study snowclones originating from pop-culture quotes; our goal is to automatically detect cultural references in text. We introduce a new, publicly available data set of pop-culture quotes and their corresponding snowclone usages and train models on them. We publish code for Catchphrase, an internet browser plugin to automatically detect and mark references in real-time, and examine its performance via a user study. Aside from assisting people to better comprehend cultural references, we hope that detecting snowclones can complement work on paraphrasing and help tackling long-standing questions in social science about the dynamics of information propagation.

## introduction first coined by richard dawkins @xcite , a meme is a unit of cultural transmission: any idea or behavior that can be transferred by imitation. internet memes have become an integral part of modern digital culture @xcite . @xcite coined the term snowclones to describe a specific type of meme -phrasal templates that are easily reusable in many different contexts. pullum described a snowclone as "a multi-use, customizable, instantly recognizable, time-worn, quoted or misquoted phrase or sentence that can be used in an entirely open array of different jokey variants". for example, the quote "one does not simply walk into mordor" from the "lord of the rings" films became a well-known pattern -"one does not simply *" -used extensively online (see figure 1 ). in this paper, our goal is to develop algorithms to detect snowclones in text. we envision an "en-glishman in new york" -a foreigner, perhaps, or someone who does not easily understand contemporary cultural references and could use the help of an automated system to communicate better. in particular, we focus on pop-culture references over the internet. from a linguistic point of view, snowclones complement the paraphrasing task @xcite @xcite . paraphrase detection identifies alternative ways to convey the same meaning, while snowclones keep (some of) the original sentence structure but completely change the meaning. detection and tracking of digital memes have been the focus of multiple computational studies. the closest to our work are memetracker and nifty @xcite , that tracked quotations attributed to individuals. these works focused on short, distinctive phrases that travel relatively intact through on-line text. other related tasks are multi-word expression/idiom identification @xcite @xcite and cliché detection @xcite . again, idioms and multiword expressions are chiefly fixed expressions ("cat got your tongue?", "jumped the shark") that rarely change their meaning across mutations. therefore, figure 1 : snowclone example, based on "one does not simply walk into mordor" from "lord of the rings". our contributions are the following: we propose a novel task of snowclone detection, identifying cultural references. we first formulate it as a tagging task, treating snowclones as regular expressions; we conduct a user study to show humans have an intuitive notion of the "correct" pattern(s), and develop a sequence-to-sequence tagger to reveal such patterns. we then extend the formulation to softer notions of similarity. we experiment with featurebased and neural approaches, achieving high accuracies. to further show the utility of our methods, we develop catchphrase, a browser extension to detect pop-culture references, conduct a user study and show it indeed helps users identify cultural references. we publish data and code 1 . we believe tracking snowclones will find interesting applications in social science, exploring the diffusion and evolution of highly dynamic content online. ## snowclones as regular expressions the common view of snowclones treats them as regular expressions (the-snowclones-database, 2007). thus, in this section, we formulate the snowclone detection problem as a tagging task. intuitively, we want to predict for each word in the original sentence whether it is replaced by a wildcard. we use the resulting pattern to match new sentences to the original sentence. for example, given a sentence @xmath0, does, not, simply, walk, into, m ordor we would like to find a mapping: t (s) = one, does, not, simply, * , * , * . (adjacent wildcards can be merged) ## snowclone pattern tagger we create and publish our own data set for this task, and use it to train two different ml models for it. data. to train ml models to solve the task of snowclone tagging, we needed examples for sentences and their underlying snowclone form. to this end, we use the snowclone patterns along with the original quotes from the snowclone database (the-snowclones-database, 2007). as this is not enough data to train on, we use the patterns to lookup more instances online, collecting 7700 snowclone pattern, instance pairs. when splitting to train-dev-test sets (60%/20%/20%), we make sure all variants of the same pattern are put in the same set. we release our dataset 1 . ## going beyond regular expressions when we tried to apply our models to find snowclones in online community text (looking for regex matches), we realized that the regex formulation might be too simplistic, as some cultural references do not follow the snowclone pattern exactly, and some sentences that do follow it are not really references. take apocalypse now's famous "i love the smell of napalm in the morning". a natural corresponding pattern is "i love the smell of * in the morning", and indeed, "i love the smell of bureaucracy in the morning" is most likely a reference to the movie. however, the case of "i love the smell of pancakes in the morning" is a lot less clear. on the other hand, "30 is the old 40" does not perfectly match the "* is the new *" pattern, but still might be considered a reference. in this section we reformulate the problem, using the output of the sequence-to-sequence tagger as one input to a machine learning model. we reformulate the problem as a binary classification task over pairs of sentences. given a seed sentence s representing an original pop-culture quote, and a candidate sentence c, decide whether c is a reference to s. we note this is not an easy task, as it is hard to put our finger on why "one does not simply forget to social distance" is likely a reference to "one does not simply walk into mordor", but "one cannot just walk right into jail" is not. ## evaluation: web extension our main motivation in this study was to help the proverbial "englishman in new york" identify cultural references. in this section, we ask whether our algorithms can help users detect pop-culture references online. we create catchphrase, a web browser extension able to detect and mark popculture references in web pages (see figure 3 ). the extension inspects the web page source and identifies candidate sentences. we use locality-sensitive hashing @xcite with similarity @xmath1.2 for filtering, allowing us to reduce computation time and maintain a small number of false negatives. next, the extension runs the referencedetector on each (seed, candidate sentence) pair and highlights the predicted references. experimental design. we choose a set of 20 popculture quotes (seeds) unseen by our referencedetector during training time, and whose snowclone form is the basis to many variations. all sentences chosen are ones that became popular internet memes. for each seed quote, we manually crawled reddit and found threads containing references to it. after filtering out threads that were over 10 messages long, we were left with 106 threads. we recruited 10 volunteers through social media, all israeli, non-native english speakers, who self-identified as having low familiarity with popculture. 80% of the volunteers were 20-35 years old, and the remaining 20% were 40-60 years old. 70% of the participants were females. we randomly selected 16 seeds for each participant and randomly split them into two groups, one per condition (with and without our extension). the threads were shown in random order. the participants were asked to go over each thread and point out any pop-culture references they detect, specifying their origin if they knew it. evaluation and results. under the no-extension condition, participants correctly identified a popculture reference 38.7% of the time. the reference origin was correctly identified in 61.2% of these. this is interesting, as it shows people can identify that a sentence looks like a cultural reference, even when they do not recognize the source. when using the extension, participants correctly identified a reference 68.7% of the times, recognizing the origin in 98.1% of these. in 26.3% of the threads, the algorithm did not recognize the reference. 5% of the times, we believe the algorithm was right but people thought it was not (e.g., "i solemnly swear i'm up for good tea" as a reference to "i solemnly swear i'm up to no good"). the reason source recognition is not perfect is one user finding a sentence the algorithm missed (but not attributing it). to check our hypothesis that web-extension users recognize more pop-culture references, we run t-test with α = 0.95 and reject the null hypothesis with @xmath2.00005. ## conclusions and future work in this work we proposed the novel task of detecting snowclones in text. motivated by the high agreement achieved by humans on a snowclone annotation task, we first developed algorithms for finding snowclones which are regular expressions, then extended the formulation to a softer notion of similarity. we introduce a new data set of popculture quotes and their corresponding snowclone variants and train models on them. we publish code for catchphrase, an internet browser plugin to automatically detect and mark references in realtime. our results demonstrate our algorithms can indeed help users detect pop-culture references. in the future, our work might be used in conversational ai context, supporting agents' ability to understand and even generate pop-culture references. another direction worth pursuing is applying our methods to domains outside pop-culture (or at the very least, to pop-culture of different cultures). we believe snowclones, complementing the notion of paraphrases, are worth exploring and can give us new insights into how ideas spread and evolve. our approach opens an opportunity to better answer long-standing questions in social science about the dynamics of information. below we provide implementation details for the sake of reproducibility.