No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models (2024)

Abstract: Reasoning in the presence of idiomatic expressions (IEs) remains a challenging frontier in natural language understanding (NLU). Unlike standard text, the non-compositional nature of an IE makes it difficult for model comprehension, as their figurative or non-literal mean- ing usually cannot be inferred from the constituent words alone. It stands to reason that in these challenging circumstances, pre-trained language models (PTLMs) should make use of the surrounding context to infer additional in- formation about the IE. In this paper, we investigate the utilization of said context for idiomatic reasoning tasks, which is under-explored relative to arithmetic or commonsense reason- ing (Liu et al., 2022; Yu et al., 2023). Preliminary findings point to a surprising observation: general purpose PTLMs are actually negatively affected by the context, as performance almost always increases with its removal. In these scenarios, models may see gains of up to 3.89%. As a result, we argue that only IE-aware models remain suitable for idiomatic reasoning tasks, given the unexpected and unexplainable manner in which general purpose PTLMs reason over IEs. Additionally, we conduct studies to examine how models utilize the context in various situations, as well as an in-depth analysis on dataset formation and quality. Finally, we provide some explanations and insights into the reasoning process itself based on our results.

## introduction in natural language, there are many methods to express canonical stories, ideas, or scenarios in a succinct and fluent manner. one such method is through the use of idiomatic expressions (ies), which are a form of multi-word expression (mwe) that carry a figurative, or non-compositional, meaning @xcite . ie comprehension remains a challenging frontier in 1 https://github.com/kellentan/no_contexts . natural language processing (nlp), with the challenge arising from this non-compositional aspect of an ie @xcite @xcite . this refers to the fact that the figurative meanings of many ies cannot be inferred from the constituent words alone. one such example would be the ie "squaring the circle", where its meaning "to handle things/resolve" cannot be inferred from the ie's individual words. given the recent advances in ptlms @xcite @xcite , it was commonly understood that transformer models' excellent performance on a variety of natural language understanding (nlu) tasks was attributable to an improved reasoning process, specifically one that is more aligned with humans @xcite @xcite . indeed, recent work on explanatory reasoning and chain-of-thought only served to cement this notion for many ptlms @xcite @xcite . thus, it stands to reason that for ie-related tasks, ptlms would also comprehend idioms in a manner similar to humans (i.e. it should utilize the context to help build a better understanding of the ie @xcite @xcite ). this paradigm is represented in the top half of figure 1 , where we would expect a model to gain a better understanding of the idiom "squaring the circle" by looking at the surrounding context. specifically, phrases like "the only way" and "send in troops", would help indicate that a situation is being resolved or handled. for such a difficult ie, removing the contexts, as seen in the bottom half of figure 1 , would make it nearly impossible for a model to understand what "squaring the circle" actually means. theoretically, reasoning without the context should be much more difficult, as there are no contextual clues to help build its understanding. this logic is also motivated by our understanding of the distributional hypothesis, i.e. words with similar meanings should appear in similar contexts @xcite , and thus provide an aid for ie comprehension. several aspects of ie reasoning have been the focus of recent studies, which investigated techniques for ie comprehension via idiomaticity detection @xcite @xcite , representation @xcite @xcite @xcite , and the use of structured knowledge such as knowledge graphs and knowledge bases @xcite . our work uses a first-principles approach to examine context utilization in the presence of ies. we chose to proceed with a data-driven approach, keeping models constant while only focusing on changes to the data samples themselves, thus guaranteeing that any differences in performance are directly attributable to how we modified the data. note that our study is focused on analyzing the linguistic reasoning capabilities, specifically figurative reasoning, of models, which is an underexplored aspect of reasoning compared to arithmetic or commonsense reasoning @xcite . our contributions are as follows: ## experimental setup in our work, we investigate how ptlms utilize the context for two distinct tasks: idiomatic nli and idiomatic continuation acceptability. note that the number of tasks at our disposal is limited due to the dearth of available ie reasoning datasets. the tasks, setups, and models are described in the following subsections. ## results & discussion our results are demonstrated for the idiomatic nli setting as well as the idiomatic narrative continuation acceptability task. note that individual model hyperparameters are described in detail in appendix d. additionally, please refer to appendix a to see the full results on each dataset per model. ## conclusions in this paper, we refute the hypothesis that ptlms utilize the context when reasoning in the presence of ies. we extend upon the work of others, and demonstrate that not only are models not sensitive to contexts, but in fact, models do not need any context at all. for difficult ie reasoning tasks, we demonstrate this peculiarity in the idiomatic nli and narrative continuation acceptability tasks, where we see performance gains despite removing context words from the data samples. we perform our experiments across a range of model complexities from 140m to 7b parameters. our results, which showcase poor context utilization for general ptlms, demonstrate a need for a closer examination of the inference process. future work should examine additional factors within the model or datasets that may cause low context utilization, methods to improve context utilization, and also extend our analysis to other domains within nlu.