Distribution-Based Measures of Surprise for Creative Language: Experiments with Humor and Metaphor (2022)

Abstract: Novelty or surprise is a fundamental attribute of creative output. As such, we postulate that a writer’s creative use of language leads to word choices and, more importantly, corresponding semantic structures that are unexpected for the reader. In this paper we investigate measures of surprise that rely solely on word distributions computed by language models and show empirically that creative language such as humor and metaphor is strongly correlated with surprise. Surprisingly at first, information content is observed to be at least as good a predictor of creative language as any of the surprise measures investigated. However, the best prediction performance is obtained when information and surprise measures are combined, showing that surprise measures capture an aspect of creative language that goes beyond information content.

## introduction language is used primarily as a means for communicating information. it is thus appropriate that information theory @xcite has provided the foundation for numerous studies into properties of natural language, as in @xcite @xcite , among many others. under the information theory framework, a communication channel is posited between the speaker and the listener, and correspondingly the goal of the speaker is to employ the channel as efficiently as possible while also minimizing the risk of miscommunication. maximizing the use of the communication channel is achieved when speakers choose their words such that their information rate is close to the channel capacity, which can be seen as determining speakers to construct utterances such that information is spread uniformly across them. this is known as the uniform information density (uid) hypothesis @xcite , operationalized as a tendency for regression towards the mean information content across the language @xcite . the uid hypothesis can explain a variety of linguistic phenomena, such as the optional omission of syntactic relativizers @xcite , or the shortened phonetic duration of highly predictable language units @xcite . uid has also been construed to imply that speakers avoid producing words with an information content foot_0 that is too high or too low @xcite relative to the expected information rate of the channel, or the entire language. while this holds true for most communicative uses of language, there are at least two types of situations when words have an information content much higher than expected, as illustrated in figure 1 . first, there is the case when the listener has no clear expectation of what the speaker will utter next, such as when introducing a new discourse entity through a definite or indefinite article, especially at the beginning of a story when not much context is available. in this case, the next word distribution has a high entropy, all words have a relatively low probability, hence high information content. the word 'sun' in the sentence foot_1 shown in figure 1 is in this category. second, there are situations when language is used in creative ways, when speakers deliberately produce words or phrases that are interesting or unexpected, often with the purpose of inducing particular kinds of emotion in the listener, as is the case with the word 'glides' in figure 1 . in this paper, we aim to characterize such creative use of language solely through distribution-based measures that are designed to discriminate creative language from normal language. in both situations discussed above the information content is high, therefore, at least theoretically, information content alone is not sufficient to discriminate between the two. as such, we propose that surprise be used as the main discriminating factor. we emphasize that determining whether an input text exhibits creativity or surprise requires specifying a reference reader, as shown in the example from figure 1 , which distinguishes the task explored in this paper from related tasks such as humor detection or figurative language classification, where novelty with respect to a reference reader is not a concern. ## definitions and measures of surprise the ability to produce surprising outputs is a cornerstone of creativity, which in turn is widely considered to be an essential component of intelligent behavior @xcite . surprise is a powerful driver for creativity and discovery. as such, surprise has been used to guide search algorithms in models of computational creativity and discovery @xcite . owing to its importance for the creative process, surprise has also become one of the core criteria for the evaluation of creative artifacts @xcite . as reviewed in @xcite , surprise is an essential concept in many studies on the neural basis of behavior, with surprising stimuli shown to be strong attractors of attention. surprise, or violation of expectation, has also been hypothesized to be an essential mechanism through which music and stories elicit emotion. according to @xcite , the principal emotional content of music arises from the composer's manipulation of expectation. composers build expectations in time, which then they purposely violate in order to elicit tension, prediction, reaction, and appraisal responses @xcite . in text and narratives, surprise can be employed with substantial emotional impact at multiple levels, spanning from word-level, as in "elon musk has just blasted the world 's most powerful rocket into landfill" where the original word "space" was purposely replaced with "landfill" for humorous effect, to story-level, as in the various types of plot twists that are used to draw the reader emotionally in the story, e.g. peripeteia or deux ex machina. in this section, we attempt to characterize wordlevel surprise using probability distributions computed by language models. we first consider a number of measures of surprise in the context of a general probability distribution p over an event space x, followed by more specialized surprise measures that are targeted to the special case of x being a language vocabulary. as such, we are interested in measuring how surprising the occurrence of an event @xmath0. an event x is improbable if its probability p(x) is very small. since improbable events are rare, it is tempting to consider the occurrence of an improbable event as being surprising. @xcite pondered on whether low probability implies surprise, "an improbable event is often interesting. but is an improbable event always interesting?", and concluded "we shall see that it is not", providing a simple, prototypical example in which improbable events are intuitively not surprising: a uniform distribution over an event space that has a large cardinality, as in dealing off a single bridge hand of thirteen cards from a shuffled pack of cards. there are more than 635 billion configurations of thirteen cards, all equally likely. whatever bridge hand is dealt, although its probability is very small, it will not be, or feel, surprising. "any hand that occurs is simply one out of a number of exactly equally likely events, some one of which was bound to happen". what makes an event interesting or surprising is not that its probability is small in an absolute sense, it is that it is small in comparison to the probabilities of the other alternative events. weaver's insight is also in agreement with the interpretation of "surprise as violation of expectation", which is hypothesized to be a major factor underlying emotion in music @xcite . in this context, the term expectation refers to the kind that is engineered by composers in their music or by writers in their stories. informally, a strong expectation is created when one or more potential outcomes are much more likely than other outcomes. more formally, an expectation regarding a random variable x is created when, prior to its value being observed, its context h makes a potential outcome @xmath1, as measured through the probability p(@xmath2. upon observing outcome @xmath3, we call it surprising if it confounds the expectation of seeing outcome @xmath4, i.e. p(@xmath5. like in weaver's argument above, the relative likelihood requirement for creating expectations immediately rules out uniform distributions. the intuitive lack of surprise when observing events sampled from a uniform distribution makes shannon's surprisal inadequate as a measure of surprise. it is thus important that the notion of surprise is not equated with surprisal. the surprisal of an event x is an information-theoretic quantity defined as the negative log probability of x, i.e. ## datasets of creative language we built two datasets of creative language examples: a humor dataset and a metaphor dataset. the humor examples were extracted from the humicroedit dataset @xcite , which consists of regular english news headlines paired with versions of the same headlines that contain simple replacement edits designed to make them funny. each funny headline was scored by five judges, resulting in a curated dataset of over 15,000 headline pairs. as positive examples for humor, we randomly selected 400 examples from a subset of the humorous headlines that were originally created using single-word replacements and that had an average annotator score of 1.8 or higher. the positive examples for metaphor were extracted from the english section of the lcc metaphor dataset @xcite where the average annotator rating was 3.0 or above and where the source field of the metaphor was a single word. furthermore, as explained below, we further applied a filtering step designed to preserve only metaphors that are novel to the language model, leaving a total of 268 positive examples of metaphor. while a metaphor may appear creative to a person hearing it for the first time, it will sound completely unoriginal to a listener who has heard it and used it so many times that it has become part of their normal use of language. similarly, a line that triggered laughter upon its first utterance, when repeated multiple times will normally get a smile at best from an audience already habituated to it. therefore, it is important that creativity be determined with reference to a listener's experience. in general, judgements of creativity require specifying a reference model, e.g. the listener, the reader, or the audience, consuming the output produced by the speaker, the writer, or the composer, respectively. consequently, based on the premise that creativity requires novelty, building an evaluation dataset annotated with creative uses of language requires fixing a reference reader and ensuring that examples annotated as creative are 1) novel for this reader and 2) evaluated with respect to the same reader. since the proposed measures of surprise will necessitate access to the reader's contextual word distributions, in this paper we set the reference reader to be a generic reader whose knowledge of language is modeled by a large language model (lm), such as bert @xcite if both the left and the right context of a word are used, or opt @xcite if only using the previous discourse as context. given that bert was trained on the bookscorpus and english wikipedia, it is safe to assume that its pre-training data was not contaminated with any of the humorous headlines from humicroedit, and therefore the humorous headlines appear novel to the reader modeled by bert. however, we cannot say the same for the metaphor examples, as many of them are commonly used and likely to be found in bert's pre-training corpus, e.g. "floating ideas", "deep understanding", "stealing dreams", "crushing insurgencies", "leap of faith", "seeds of discontent", to list just a few. to ensure that the metaphor examples included in the dataset are novel with respect to the reader modeled by bert, since we did not have access to the exact pre-training data, we devised a conservative filtering where the base metaphor phrases were filtered out if a google search returned less than 25 documents containing the phrase or its variations. for example, given the annotated metaphor "the bureaucracy barrier", we removed the article and also searched for "bureaucratic barrier" and "barrier of bureaucracy". furthermore, we removed examples where the source word is repeated in the sentence context, as in "this @xcite s is the prison of [poverty] t ". in terms of negative examples, for humor we used the 400 original titles corresponding to the 400 humorous examples. we further augmented these negative examples with nouns (as tagged by nltk's pos tagger) selected at random from news articles downloaded from the cnn website in july 2022, such that the number of positive examples represents 10% of the total number of examples in each dataset. regular news articles are expected to use regular language, without novel humor or novel metaphors. this is not to say the news articles do not contain metaphors, but when that happens they are metaphors that are commonly used and thus unsurprising for a generic reader. to summarize, the label distribution in the two datasets is as follows: the imbalanced label distribution was meant to address the fact that instances of creative language are relatively rare, although the exact proportion in general is hard to estimate due to the fact that certain types of text, e.g. poetry, are expected to be substantially more creative than other types, e.g. news articles. we note that the labels in the resulting dataset are likely to be noisy: metaphors that we annotated as creative, even though uncommon ad litteram on the internet, may still have been present in the lm's pre-training data in a different form, such as using a synonym for any of the words in the expression. furthermore, it is possible that the cnn news articles included in the dataset contain instances of creative language, albeit very few. overall though, it is expected that a good measure of surprise would show substantial discriminative power between the soft positive vs. soft negative examples in this dataset. hardening the dataset would require the development of feasible annotation guidelines for determining whether the reference lm (the reference reader) has been exposed (through its pre-training data) to any given expression, and then going over each example and using the annotation criteria to determine the label. ## experimental evaluation all the distribution-based measures of surprise evaluated in this section were calculated using the probability distributions computed by the bert large model (cased) available on the huggingface website 3 . this is done by taking the word that is labeled in the dataset, masking it, and asking bert to output the token distribution at the masked position, using a context size of 15 tokens to the left and to the right. due to the wordpiece subword tokenization used by bert, sometimes the word that need to be labeled is split into multiple tokens, where the 3 https://huggingface.co/bert-large-cased first token is distinguished from the continuation tokens using the double hashtags '##', as for example 'disrespect' = 'di' + '##s' + '##res' + '##pect'. in these cases, we use the probability of the first token as a proxy for the probability of the entire wordpreliminary experiments where the simple product or the geometric mean of all the token probabilities were used did not show a significant difference in the results, likely due to the fact that continuation tokens often receive a very high probability. a starting assumption in these experiments is that the input text is well formed, e.g. it does not contain ungrammatical phrases or typos. while we recognize that real text may contain ill formed language that could be incorrectly detected as surprising by the various surprise measures proposed in this paper, we do not consider this to pose a significant challenge as such text could be feasibly detected and filtered out using current state-of-theart nlp tools. furthermore, a simple way to filter out ill formed language and typos is to ignore tokens that belong to the tail of the lm distribution, a procedure that we will investigate in future work. the support of the raw lm distribution is modified to exclude continuation tokens, non-content words, and punctuation symbols for the reasons explained below, after which the probabilities of the remaining tokens are renormalized so that their total probability mass is still 1. continuation tokens sometimes receive a high probability at the masked position. for example, in the annotated metaphor "[tax] t [sorcery] s is a mystery to me", when the source word "sorcery" is masked the continuation token "##ation" receives the highest probability, corresponding to the reasonable completion "taxation is a mistery to me". since the masked word cannot be continuation in our task, all continuation tokens are eliminated from the distribution support. depending on the context, non-content words such as determiners and prepositions may receive a high probability at the masked position, as for example in the metaphor text "we had our own little electoral "irregularities" down here in portsmouth's first ward, where we suffer from [constipated] s [democracy] t ". determiners such as 'a' or 'the' receive a relatively high probability for occurring at the masked position for the source field. since metaphors and one-word humorous word substitutions are content words, we remove non-content words from the distribution support. punctuation symbols may also receive a relatively high probability in some contexts, as such they are excluded as well from the distribution support. in the metaphor example "communism thrives on an empty stomach and [democracy] t [relaxes] s on a full one", symbols such as commas ',' and the dashes '-' are predicted with a high probability at the masked source position. ## related work owing to its essential role in our daily lives, there have been numerous computational approaches to humor recognition, as reviewed for example in @xcite . humor generation has presented a challenging problem in ai since the early 1990s, leading to the development of various template-based and neural approaches @xcite . the important role that surprise plays in humor generation has been previously recognized in theories of humor, such as the surprise theory of laughter @xcite and other prominent models that posit humor is evoked by incongruity within a text, such as the two-stage model of @xcite . according to incongruity theories of humor, a text conveys at least two interpretations, of which one is more salient. as readers process the text, the salient interpretation is activated until a text segment is encountered that contradicts it and thus promotes the previously unexpected interpretation. surprise arises from his sudden revision of understanding. metaphors are pervasive in everyday communication, as well as in creative writing such as novels and poetry. metaphors enhance the communicative aspects of language by connecting concepts from new domains, often abstract, with more familiar ones, usually concrete @xcite . metaphorical expressions have many uses, from helping frame an issue in order to emphasize some aspects of reality @xcite , to creating a strong emotional effect @xcite . the ubiquity of metaphors means their computational treatment @xcite has received significant attention in the nlp community, as surveyed by shutova (2015) and more recently @xcite . a distinction is made in the literature between conventional metaphors, which are entrenched in the conceptual system, and novel metaphors, which are unfamiliar. in this paper, we further recommend that novelty judgements be made relative to a reference reader. our use of a large lm to model the reference reader is supported by the fact that pre-trained lms encode conventional metaphorical information, as shown recently in the probing study of @xcite . even though metaphor is widely seen as a creative tool and surprise is an essential component of creative artifacts, we are not aware of any work investigating the role of surprise in discriminating between conventional vs. novel metaphors. computational approaches to humor and metaphor are part of a larger inquiry into identifying and formalizing the basic processes underlying human creativity. in the growing field of computational creativity foot_2 , surprise has been proposed as one of the major criteria for the evaluation of creative artifacts @xcite . surprising outputs were shown to attract the attention of the observer @xcite , but also to guide the creative process itself: in a study of the creative design process followed by architects @xcite , surprising discoveries in design sketches were observed to cause reformulations of design goals, which in turn led to further unexpected discoveries, due to designers reading more off a sketch than what they originally intended to put there @xcite . in this paper we emphasize that surprise, and by extension creativity, needs to be defined relative to a reference reader or audience. consequently, generative architectures that aim to learn patterns of surprise and expectation from data need to contain a separate model for the reference reader, as implemented in the composer-audience models from (bunescu and uduehi, 2019) for binary sequences and (uduehi and bunescu, 2021) for basic geometrical shapes. ## conclusion and future work aiming to characterize creative language, we introduced a number of measures of surprise that are based solely on the probability distributions computed by a reference lm, considered to model a reference reader. experimental evaluations show that, in combination with information content, the surprise measures improve detection of novel metaphors or humor, providing empirical evidence for the role of surprise in creative use of language. the code and data will be made publicly available 5 . future work includes refining the datasets, calibrating the lm probabilities, developing semanticlevel measures of surprise, and evaluating the proposed measures with respect to a reference reader that only knows the literal meaning of words. an interesting future extension to other types of wordlevel humor such as puns was suggested by a reviewer, where surprise measures would be combined with measures of character-level similarity such as edit distance.