Using LLM s to Advance Idiom Corpus Construction (2025)

Abstract: Idiom corpora typically include both idiomatic and literal examples of potentially idiomatic expressions, but creating such corpora traditionally requires substantial expert effort and cost. In this article, we explore the use of large language models (LLMs) to generate synthetic idiom corpora as a more time- and cost-efficient alternative. We evaluate the effectiveness of synthetic data in training task-specific models and testing GPT-4 in few-shot prompting setting using synthetic data for idiomaticity detection. Our findings reveal that although models trained on synthetic data perform worse than those trained on human-generated data, synthetic data generation offers considerable advantages in terms of cost and time. Specifically, task-specific idiomaticity detection models trained on synthetic data outperform the general-purpose LLM that generated the data when evaluated in a zero-shot setting, achieving an average improvement of 11 percentage points across four languages. Moreover, synthetic data enhances the LLM’s performance, enabling it to match the task-specific models trained with synthetic data when few-shot prompting is applied.

## introduction an idiom is a linguistic expression the meaning of which cannot be derived compositionally from the literal meaning of its parts. for example, the english idiom break a leg is used to wish someone good luck, rather than being taken literally as an instruction to cause physical harm. due to this unique nature, idioms can negatively impact the performance of models in various tasks, such as machine translation, word-sense disambiguation, and information retrieval @xcite . idiom corpora are essential for enhancing performance in numerous tasks, as they provide both idiomatic and non-idiomatic examples to help mod-els better differentiate between literal and figurative meanings. training models on a diverse and well-structured idiom corpus can reduce problems such as incorrect translations @xcite or misinterpretation of idiomatic expressions @xcite . moreover, idioms present a significant challenge for language learners, who often struggle with the non-literal meanings and cultural nuances embedded in these expressions @xcite . comprehensive idiom corpora can support the development of educational resources and tools designed to help learners master idiomatic usage more effectively. consequently, both computers and humans require high-quality samples that exemplify idiom usage scenarios and patterns. traditional approaches to constructing idiom corpora, such as those relying on the annotation of natural text @xcite , face several challenges. these include unbalanced distributions of idiomatic versus non-idiomatic examples, a lack of diversity in surface forms, and issues related to data scarcity. while recent methods, such as obtaining idiomatic sentences from native speakers via gamified crowdsourcing platforms @xcite , offer potential solutions, they still have notable limitations and continue to be time-consuming and costly, as they require the involvement of native speakers for effective execution. due to the challenging nature of the data collection process, only a handful of studies have presented idiom corpora that include both idiomatic and non-idiomatic examples. these corpora are mostly limited to a few languages and a small set of idioms (see recently, large language models (llms), such as @xcite , have shown their effectiveness as generators in few-shot @xcite and zero-shot @xcite settings, and have been utilized to generate training data for downstream tasks @xcite . in this article, we use gpt-4 to generate idiomatic in-dataset #sentences #idioms language vnc-tokens* @xcite 2,566 53 en open-mwe* @xcite 102,856 146 ja sporleder and li @xcite 3,964 17 en idix @xcite 5,836 78 en semeval-2013 task 5b @xcite 4,350 65 en parseme @xcite 274,376 13,755 bg, cs, fr, de, he, it, lt, mt, el, pl, pt, ro, sl, es, sv, tr magpie @xcite 56,622 2,007 en epie @xcite 25,206 717 en astitchinlanguagemodels 6,430 336 en, pt id10m silver @xcite 800 470 de, en, es, it id10m gold @xcite 262,781 10,118 de, en, es, fr, it, ja, nl, pl, pt, zh semeval-2022 task 2 @xcite 8,683 50 en, gl, pt dodiom* @xcite 12,706 73 it, tr in summary, our main contributions can be listed as: the constructed corpora, along with the code for synthetic data generation and training and testing models for idiomaticity detection, are available on github. foot_1 2 background and related work idiom corpora are corpora that include sentences containing potentially idiomatic expressions (pies), where these expressions are used in both idiomatic and literal senses in different contexts. the process of constructing an idiom corpus generally involves three steps: (1) selecting a list of idioms from phrases identified in previous studies @xcite or from dictionaries @xcite , with optional filtering based on certain rules @xcite , frequency @xcite , or expert judgment @xcite ; (2) obtaining sentences that contain pies from existing corpora @xcite , the web @xcite , or directly from native speakers @xcite ; and (3) labeling the sentences based on the usage sense, typically as idiomatic or literal, using native speakers or language experts @xcite . in ## methodology to construct the idiom corpora presented in this article, we select a list of pies identified in previous research that provides a diverse set of idiomatic expressions in different languages. specifically, we choose the pies identified by @xcite for english, @xcite for japanese, and @xcite for italian and turkish. for synthetic data generation, we prompt gpt-4 (specifically gpt-4-0125-preview) to generate a sentence containing an idiomatic or literal use of an identified pie in two settings: zero-shot prompting and enhanced prompting (see appendix a, figure 1 and figure 2 ). in both settings, the prompts are always given in the target language and the system prompt instructs the model to generate sentences as if it is proficient in the target language, using it in rich and creative ways. the model is specifically asked to retain the lemma of the idiom constituents, since syntactic operations for idioms are mainly restricted by the idiom's individual components and its overall idiomatic meaning @xcite . additionally, it is prompted to avoid the use of human names, as our prior prompting trials indicate that including names results in poor-quality samples. in the enhanced prompting setting, the model is further instructed to avoid repeating previously generated sentences, and it is observed that explicitly encouraging creativity (e.g., prompting the model to be creative) sometimes results in similar sentence structures. in the zero-shot setting, the model is introduced to a pie and simply asked to generate sentences using it. in the enhanced setting, a two-stage data generation approach is applied using the chainof-thought method @xcite . first, the model is presented with a pie and its use cases, and then it is asked to generate use cases for another target pie. in the second step, the model is instructed to generate sentences based on those use cases, incorporating diverse grammatical structures, including declarative-interrogative forms, affirmative-negative constructions, variations in sentence length, and inserting additional words between the components of the idiom. this approach aims to ensure diversity in sentence structures within the generated corpus. additionally, the model is encouraged once in a while to generate sentences "as-if it is a human" and to be "creative", to prevent it from simply paraphrasing previous answers. illustrations of the zero-shot and enhanced prompting settings are provided in figure 1 and figure 2 , respectively, which can be found in appendix a. for each pie in the aforementioned corpora, we generate 200 sentences using gpt-4 through the openai api, with each pie appearing in both its idiomatic and literal senses, equally represented with 100 sentences for each sense. of these, 60 sentences are generated using zero-shot prompting, while 40 are generated using enhanced prompting. the average cost of generating each sentence is approximately $0.004. the overall statistics for the generated datasets are summarized in ## experiments to evaluate the quality of the synthetically generated datasets, we applied it to n-shot prompting of gpt-4 and fine-tuning smaller models specifically for the task of idiomaticity detection. additionally, to examine the effects of different prompting techniques on data generation, we fine-tune separate models using examples obtained from zero-shot prompting and enhanced prompting, allowing for a comparison between these two approaches. finally, we measure and compare the performance of multilingual models fine-tuned on idioms from multiple languages against monolingual models to assess the impact of multilingual idiom inclusion. ## results this section summarizes the findings from our experiments, which include comparing the performance of n-shot prompted gpt-4 and bert variants fine-tuned on synthetic and human-generated data, analyzing the effects of different prompting methods, and evaluating the performance of multilingual models trained on synthetic datasets. ## conclusion in this article, we create synthetic idiom corpora in multiple languages using gpt-4 and evaluate the effectiveness of models trained on these corpora for the idiomaticity detection task. additionally, we have analysed the impact of the prompts used during the example generation process on corpus quality and assessed the influence of synthetic data on the performance of multilingual models. the results indicate that while synthetic data may not match the quality of human-generated data, it offers significant advantages in terms of cost and time efficiency. furthermore, smaller task-specific models trained on the synthetic data generated by the llm outperform the llm itself on the same task in the zero-shot setting. however, the llm surpasses these models when synthetic data is also provided during the test phase (i.e., in few-shot prompting setting), highlighting the potential of synthetic data to enhance llm performance. in this setup, the llm achieves results comparable to using human-generated data in few-shot prompting setting, with a difference of only 2 percentage points for japanese and italian and 4 percentage points for english and turkish. our findings also reveal that more complex prompts during the synthetic data generation process do not consistently produce higher-quality examples. these complex prompts produce beneficial results only in english, likely because gpt-4 performs more effectively in english than in other languages across various tasks. overall, while the llm's performance in idiomaticity detection remains lower than that of task-specific models, as is the case in other natural language processing tasks, its generalization potential makes it a highly valuable resource. future work could focus on more sophisticated prompting methods, refining the reasoning process of the utilized llm, and expanding the study to include additional languages and llms. additionally, the generated data could be used for instruction-tuning of llms to explore potential improvements in their ability to handle idiomatic expressions across diverse languages. overall, our findings highlight the pivotal role llms can play in generating idiom corpora as a cost and timeeffective alternative to methods relying on human effort, as well as the effect of synthetic data in enhancing llm performance on idiomaticity detection task.