A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models (2024)

Abstract: In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and a comprehensive error analysis.

## introduction the continuous improvements in llm performance raise the hypothesis that their exposure to vast amounts of pre-training data may give them the capability to accurately process the meaning of natural language utterances. we conducted a thorough analysis of the behaviour of three smallsized, instruction-tuned llms, tasked with figurative uses of language. the goal of this work is to provide a comprehensive evaluation methodology centred around a new test suite, idiots, foot_0 designed to assess the capabilities of llms to distinguish between figurative and literal meanings of potentially idiomatic expressions (pies). the adopted definition of pie is the one provided by @xcite : expressions that can have an idiomatic meaning, regardless of whether they actually have that meaning in a given context. ## related work the question about to what extent llms can interpret non-literal phrases remains open @xcite . the creation of numerous figurative language datasets as a fundamental resource for evaluation underscores the importance of this issue in natural language processing (nlp). to the best of our knowledge, these are some of the most significant existing datasets on figurative language. the magpie corpus, created by @xcite , is a large sense-annotated corpus of pies created from a highly curated list of idioms. this dataset has been employed in numerous studies @xcite @xcite , exploring figurative language processing from the most diverse perspectives. the fig-qa dataset was developed by @xcite to test the ability of llms to reason about figurative language. the findings of the conducted experiments underscored that llms still fall short of human performance, particularly in zero-or fewshot settings. the impli dataset @xcite is a human annotated dataset consisting of paired sentences spanning idioms and metaphors, designed for natural language inference (nli). the task consists in predicting whether the meaning of one text fragment (premise) entails another (hypothesis). experiment findings indicate that, even when pre-training data includes figurative sentences, idiomatic language remains a challenge for pre-trained language models. the id10m multilingual dataset developed by @xcite was proposed as part of a complete framework for idiom identification in several languages. the conducted experiments demonstrate that a model fine-tuned on this dataset is able to correctly predict the majority of idiomatic pies, but struggles with literal pies, tending to attribute them an idiomatic meaning. the flute dataset, introduced by @xcite , is a dataset of textual explanations of figurative expressions. the results of the experiments conducted with models fine-tuned on flute showed how such dataset can contribute to developing models that understand figurative language through textual explanations. ## dataset creation process we introduce a new evaluation dataset specifically crafted for idiom detection in english. the rationale behind the creation of a new resource from scratch, rather than building on a pre-existing dataset, is grounded in the need to avoid data contamination by providing data that we can guarantee the assessed models have not seen before. in this section, we describe all the steps involved in the creation process of the idiomatic language test suite idiots. ## experiment definition our experimental focus was pointed at evaluating the ability of the selected llms to detect an idiomatic expression in a given sentence. this experiment falls within the context of "idiom detection" and involved a binary sentence classification task, being the two classes to predict "idiomatic" (positive class) and "non-idiomatic" (negative class). the goal was to assess whether llms are able to accurately capture the meaning of a pie, distinguishing between figurative and literal meaning based on the formulation of the sentence. assuming that the pre-training data for these models contained the specific pies far more frequently with idiomatic than with literal meaning, the models may be inclined to attribute a figurative meaning to the expression based on probability distribution. ## findings we established two different levels of evaluation for the experiment. the first level consists of a completely automatic evaluation, whereas the second level is complemented with a thorough manual evaluation and error analysis. ## conclusions and future work the use of figurative language is a complex linguistic phenomenon that poses hard challenges for llms. despite its critical role within numerous nlp tasks, it still remains a relatively underexplored area of investigation. in this work we addressed the specific domain of idiomatic expressions in english as a special case of figurative language use. as a part of our contribution: the findings from our study indicate that when it comes to capturing the meaning of an ambiguous sentence, llms struggle to distinguish between literal and idiomatic uses of language. in line with the observations in the literature, a high acquiescence or agreement bias was observed: llms tend to force the identification of an idiom by assigning idiomatic meaning to an aleatory element in the sentence. additionally, they offer coherent explanations to reinforce their inaccurate answers, which can be a cause for concern. as future research directions, we intend to broaden our experiments by extending them to oneand few-shot scenarios, by exploring other prompting techniques focused on mitigating researcher bias and incorporating the possibility to interact with conversational models in multi-turn conversations. regarding the proposed idiots, we plan to explore several data augmentation techniques to generate additional idiomatic and distractor sentences. additionally, a categorisation of distractor types could be incorporated to gain an understanding of which constructions are the most challenging for the models. moreover, we intend to translate the sentences into other languages to create a multilingual dataset and open a path for mt experiments aimed to investigate possible correlations between idiom detection and translation. at a broad level, exploring models with different architectures, sizes, and hyper-parameter configurations could provide valuable insights into how these models characteristics relate to the capabilities of llms to process natural language and could open avenues for targeted experimentation, such as specific fine-tuning strategies, aimed at enhancing the performance of llms across various natural language tasks.