M i QA : A Benchmark for Inference on Metaphorical Questions (2022)

Abstract: We propose a benchmark to assess the capability of large language models to reason with conventional metaphors. Our benchmark combines the previously isolated topics of metaphor detection and commonsense reasoning into a single task that requires a model to make inferences by accurately selecting between the literal and metaphorical register. We examine the performance of state-of-the-art pre-trained models on binary-choice tasks and find a large discrepancy between the performance of small and very large models, going from chance to near-human level. We also analyse the largest model in a generative setting and find that although human performance is approached, careful multiple-shot prompting is required.

## introduction conceptual metaphor is an ubiquitous cognitive mechanism that allows us to structure and reason about abstract concepts by relating them to experiential domains @xcite . in language, metaphors allow human communication and reasoning about abstract ideas using concrete notions learned from sensorimotor, emotional, and other embodied experience @xcite : "a plan is solid"; "the economy is stumbling"; "i see what you mean". to illustrate the role of metaphor in abstract reasoning, consider the following metaphorical statement: "the economy is stumbling". according to conceptual metaphor theory (cmt), we understand this statement through mental simulation, by connecting the abstract concept of economy to the imagined movement of a stumbling person. we use the same mental imagery to infer that the economy, just like a stumbling person, is "unstable" and "might fall". as another example, in the metaphorical statement "a proposal is solid", bringing to mind solid objects and their properties suggests that the proposal, just like a physical object, was "well-built" and "will not easily break". however, not all properties generalize: unlike a physical object, a proposal, which is an abstract entity, cannot be "thrown" or "bent"; unlike a stumbling person, the economy does not "wear shoes". large language models (llms) have achieved remarkable results on a variety of tasks. however, in contrast to humans, llms do not have access to commonsense and embodied experiences of the world @xcite . although the data llms are trained on includes up to trillions of text tokens, it is unclear how much of this data allows them to capture human commonsense reasoning @xcite . conceptual metaphor theory suggests that embodied and implicit knowledge is required for the ability to utilize metaphors in commonsense reasoning about concrete and abstract ideas. we propose a novel dataset, miqa (metaphorical inference questions and answers), to assess the ability of a model to reason with conventional metaphors. the benchmark draws on the cmt research @xcite to construct a representative set of primary metaphors, which are contrastively paired with literal statements. our task requires a model to make a correct inference in simple situations without specifying whether the contextual register is literal or metaphorical, leveraging research on metaphor processing @xcite , commonsense reasoning @xcite , and natural language inference @xcite . our benchmark combines the previously isolated areas of metaphor detection and commonsense inference. although there is considerable research on both of these areas separately, it is unclear whether such capabilities scale compositionally: llms could handle two separate tasks well, but not their combination @xcite . ## related work metaphor has received renewed attention in natural language processing, but most tasks have been restricted to detection (e.g. @xcite @xcite on large annotated corpora @xcite . human-level performance has not been reached by llms, but the progress is promising and an active area of research. however, these tasks may excessively rely on context-dependent word meanings @xcite and do not measure the ability to reason with metaphor. metaphor paraphrasing is another active area of research. big-bench @xcite , a collaborative multi-task benchmark intended to test a variety of capabilities of llms, includes four tasks related to metaphor. while these tasks contain novel metaphors, they still do not assess the ability to employ metaphoric knowledge in reasoning. recently, @xcite built a dataset for multiple-choice story continuations involving similes and idioms extracted from books. subsequently, @xcite proposed a metaphor interpretation task that requires models to choose the correct out of two interpretations of a simile. in contrast, our task combines metaphor interpretation with commonsense inference, uses a more systematic data source, and has an additional adversarial character, as it requires the selection between two semantically-close items instead of items with opposite meanings. ## human evaluation we estimated the human performance on the binarychoice task using the responses of 15 human adult volunteers with english as first or second language. the participants were told that the aim of the research was to gather a set of commonsense responses and compare them to llms responses. no additional information about the task was given. ## large language models evaluation we evaluated the performance of two pre-trained llms: palm with 8b, 62b, 540b parameters @xcite and gpt-3 ada, babbage, curie and davinci @xcite . the parameter counts of the gpt-3 models are not publicly available, but have been estimated at 350m, 1.3b, 6.7b, and 175b respectively foot_1 . the main purpose of this study is to assess the capabilities of llms on the miqa benchmark. for comparison, we also verify the capabilities of pre-trained fine-tuned smaller language models on our benchmark. we follow @xcite in using encoder-only models trained on the natural language inference datasets snli @xcite and mnli @xcite for zero-shot evaluation. we opt for this approach because miqa is designed as a small dataset suitable as a benchmark and not for fine-tuning. we test the state-of-the-art encoder-only model deber-tav3 @xcite in sizes small, medium and large. these models have 44m, 86m and 304m parameters respectively, and their weights are available online foot_2 . the models take a premiseimplication pair and produce a probability distribution over three classes: "entailment", "contradiction", and "undetermined". we report the results for the best-performing score, in this case 1 -p (contradiction), which outranks p (entailment). ## results the full results are shown in firstly, for the binary-choice tasks, there was a considerable gap between small and large llms. while the smaller models performed at or close to chance level, the largest models achieved very good performance even with 0 shots, and approached human-level performance with few shots. we note that the "implied-by" task was overall more difficult than the "implies" task for both humans and llms. secondly, the chance-level performance on the baseline prompts suggests that the increase in performance in few-shot settings was not due to the model learning to select either metaphorical or literal statements independently of the prompt. on the other hand, the strong performance of the de-bertav3 models suggests a high level of transfer from the nli datasets to miqa, although there is a still a considerable gap to human performance. finally, the generative results on palm-540b estimated the model performance in an open-ended setting. similarly to the binary task, the model performed considerably better with 5 shots compared to 0 shots, approaching human performance. however, the gap between human and model performance for the generative task was greater compared with the gap for the binary-choice task. overall, the results demonstrate that llms can correctly select between the metaphorical and literal contextual registers to perform inferences with conventional metaphors, but there is still a considerable gap between human and llm performance ## limitations our work used foundational metaphors from cmt to test basic metaphoric reasoning in llms. we will expand this benchmark using additional and more complex sources of conceptual metaphor (e.g. @xcite . future work will assess llms on novel non-conventional mappings. although we mitigated for prompt sensitivity by using multiple prompts, the result interpretation should allow for small accuracy variations. further, in the binary-choice tasks we compare the llm results with a human baseline, but we do not provide a baseline for the generative task. this is near perfect for humans, but a more systematic baseline can be created to quantify the exact headroom on this task. finally, while the task holistically measures the performance of llms on a complex task, it is difficult to disentangle the component effects (metaphor detection, reasoning, response generation) in the overall accuracy. ## conclusion we have proposed a novel compositional benchmark based on conceptual metaphor theory to assess the capacity of llms to make inferences with metaphors. successful performance on this task requires metaphor detection and commonsense inference. using a metaphor theory-based approach allows us to systematically explore capabilities and limitations of llms. this is the first in a planned series of increasingly complex metaphor inference datasets. three main findings emerged from our proposed task. firstly, there is a vast difference between the performance of small and large llms, with the former performing at chance level and the latter approaching human level in few-prompt setting. this observation is informative in the context of previous results showing that some, but not all, tasks observe a qualitative performance jump with model size and scale: for example, this is the case for reasoning about goal-step relationships between events and ordering events, but not for navigation and mathematical induction tasks @xcite . this result invites more research into the question of how and whether the performance of smaller models can be improved. secondly, this reflects a true ability of llms to reason with conventional metaphor, and not simply to detect it. whether this ability extends to novel metaphor is ongoing work. finally, the performance of large llms approaches that of humans in binary-choice and generative tasks, but careful multiple-shot prompting is required.