On the Cusp of Comprehensibility: Can Language Models Distinguish Between Metaphors and Nonsense? (2022)

Abstract: Utterly creative texts can sometimes be difficult to understand, balancing on the edge of comprehensibility. However, good language skills and common sense allow advanced language users both to interpret creative texts and to reject some linguistic input as nonsense. The goal of this paper is to evaluate whether the current language models are also able to make the distinction between a creative language use and nonsense. To test this, we have computed mean rank and pseudo-log-likelihood score (PLL) of metaphorical and nonsensical sentences, and fine-tuned several pretrained models (BERT, RoBERTa) for binary classification between the two categories. There was a significant difference in the mean ranks and PPL scores of the categories, and the classifier reached around 85.5% accuracy. The results raise further questions on what could have let to such satisfactory performance.

## introduction the ultimate goal of natural language understanding (nlu) models is to reach a human-like level of language comprehension. however, a good command of language manifests itself not only in being able to interpret advanced usages of a language, but also in discriminating the uninterpretable, erroneous cases. while automatic grammar checkers are already in place, semantic incongruity is more difficult to trace. the task is further complicated by the existence of figurative language, where a listener is required to go an extra step (when compared to literal language) in order to decode the meaning. the borderline between creative, but still understandable text, and nonsense can be seen as the cusp of comprehensibility. one of the types of figurative language is metaphors, which are convenient to research due to their ubiquity. linguistic metaphors can be defined as expressions of an understanding of one concept in terms of another, where there is some similarity between the two. while metaphor per se signifies a shift in meaning, they do vary in the degree of metaphoricity and creativity. the most threadbare metaphors which are so commonly used that they become unnoticeable are conventional metaphors, for example, "he takes a few moments to reply". on the other side of the scale of metaphoricity there are creative metaphors, where a novel meaning emerges in a sentence, for example, "the atm coughed up my card" @xcite . however, even when it comes to novel metaphors, language users should still be able to infer the meaning -otherwise, they are just nonsense. professor irving massey has suggested that distinguishing between a metaphor and nonsense could be a new turing test @xcite . the professor claims that switching between literal and metaphorical senses is an aesthetic gesture inaccessible for computers, and that "the ability to experience metaphor is the very definition of the human". while admittedly for the time being there is no way to track aesthetic experiences of a computer, the (in)ability of computational models to make the distinction between a metaphor and mere nonsense might be worth looking at. while we sometimes deify metaphors as "a hallmark of human intelligence" @xcite , and assume that the interpretation of metaphors, especially of novel metaphors, demands human cognitive skills and real world experiences, it is also possible that there are enough clues encoded at the linguistic level that they would help a non-human to distinguish between metaphors and nonsense. in order to test whether the ability to demystify metaphors is a skill exclusively possessed by mortals, we are going to measure and compare the pll scores of metaphors and nonsense, as well as use mean ranks of predictions on masked words to test how well the nonsensicality can be explained by plausibility (language model probability). finally, a binary classifier based on a pre-trained lan-guage model is going to be trained in order to check whether the current language models are able to distinguish between metaphors and nonsense. ## related works a study by @xcite hints that language models might already have acquired a human-like intuition of sentence plausibility. the authors of the study have found out that the pseudolog-likelihood scores (pll) of sentences obtained using bert @xcite correlated with the plausibility ratings of human annotators. the best performing model in the corpus of linguistic acceptability (cola) @xcite task in the glue benchmark @xcite , ernie, surpasses even the human baseline (75.5 vs. 66.4 mcc), discriminating linguistically unacceptable sentences better than human participants. however, another study conducted by @xcite found that the bert family of models are easily susceptible to adversarial examples and fail to even recognize incoherent, ungrammatical utterances, giving similarly confident scores to input that was perturbed to be nonsensical as to its meaningful counterpart. findings like this are evidence that, when discriminating between meaningful and nonsensical sentences, the models might be relying on some spurious correlations or annotator artifacts rather than the targeted divergence in comprehensibility. ## discussion the experiment results have demonstrated that language models can see the difference in plausibil-ity between nonsense and metaphorical sentences. such finding can be a useful probe when investigating what do models know about the language. the ability of models to distinguish between nonsense and metaphors (especially creative ones) suggest that the language models have an intuition that even highly unusual phrases/sentences can make sense. the findings can also be brought up in a discussion about the nature of metaphors. while it could seem that, in order to understand some metaphor (and, it this way, to distinguish it from nonsense), extensive world-knowledge and an associative thinking is needed, our results suggest that, unless the models have also already acquired the aforementioned assets, metaphors can be distinguished from nonsense based on their linguistic form as well. furthermore, using a metaphor vs. nonsense classifier could be useful in ranking translated (literary) sentences, to see if the metaphors have been used correctly. ## future work while we were trying our best to ensure the training and testing data is free of unintended biases, further research would be needed to find out whether there really are no artifacts left. it is not clear whether the models are really relying on semantic acceptability in the case of our classifiers. it can also be that models are taking advantage of annotation artifacts when making decisions. one way to test for this would be to remove the target word from the sentences and try to train a classifier on the rest of the sentence. ## conclusion the conducted experiments have shown that the current language models are able to pick the difference in plausibility between metaphorical and nonsensical sentences. the classifier between these two categories is also performing well, reaching about 75-88% accuracy (depending on the size of the training dataset). however, further research is needed to see whether this classification performance comes from distinguishing the semantic acceptability of the sentences, or if it is due to linguistic artifacts in the sentences that models can rely on when making the decision.