Can GPT 4 Detect Euphemisms across Multiple Languages? (2024)

Abstract: A euphemism is a word or phrase used in place of another word or phrase that might be considered harsh, blunt, unpleasant, or offensive. Euphemisms generally soften the impact of what is being said, making it more palatable or appropriate for the context or audience. Euphemisms can vary significantly between languages, reflecting cultural sensitivities and taboos, and what might be a mild expression in one language could carry a stronger connotation or be completely misunderstood in another. This paper uses prompting techniques to evaluate OpenAI’s GPT4 for detecting euphemisms across multiple languages as part of the 2024 FigLang shared task. We evaluate both zero-shot and few-shot approaches. Our method achieved an average macro F1 of .732, ranking first in the competition. Moreover, we found that GPT4 does not perform uniformly across all languages, with a difference of .233 between the best (English .831) and the worst (Spanish .598) languages.

## introduction a euphemism is a term or expression substituted for another that may be deemed too direct, harsh, or offensive. euphemisms play a nuanced role in linguistic expression, serving as a polite or softer alternative to potentially sensitive or direct language (danescu-niculescu-mizil et al.; @xcite . however, their inherent ambiguity challenges natural language processing (nlp) systems in comprehending meaning because they must pick up on subtle contextual cues @xcite . this difficulty is magnified in multilingual contexts, where the same euphemism could have different meanings across cultures. hence, this paper describes an approach for the 2024 figlan shared task for multilingual euphemism detection. much of the recent research on euphemism detection has focused on fine-tuning transformerbased models @xcite @xcite . for instance, @xcite combined a bert-based transformer with a relational graph attention network and fine-tuned it for euphemism detection. however, recent advancements in the development of large language models (llms) like gpt-4 have been shown to be successful in similar tasks such as offensive and abusive language detection (openai et al.; @xcite @xcite . gpt-4 is supposedly trained on extensive datasets of multilingual text containing wide variations of linguistic styles, which would be very helpful in understanding and interpreting euphemistic language. the tool's ability to generate human-like dialogue and adapt itself to nuanced language suggests that it could be used to distinguish between literal and euphemistic language use. recent research has shown limitations of gpt-4 and related models in multi-lingual settings @xcite . for example, qiu et al. (2024) report substantial differences in medical applications performance of gpt-4 across different languages. hence, understanding how gpt-4 performs for multilingual classification, particularly for tasks that involve figurative language, can provide unique insights into its limitations. in this paper, we explore the application of prompting techniques (ouyang et al.; @xcite to detect euphemisms using gpt-4. we note that recent work has explored promptingbased euphemism detection @xcite . however, the system still required finetuning model parameters. here, we explore zeroshot and few-shot prompting strategies without any fine-tuning. we analyze a various number of incontext examples. moreover, we performed a small error analysis to understand the limitations of gpt-4 for euphemism detection and to understand when gpt-4 fails for multilingual euphemism detection. ## related work despite the general advancements in nlp, the automated detection of euphemisms remains a relatively under-explored area. early approaches to identify euphemistic speech focused on rule-based systems and statistical methods @xcite . @xcite explored knn and data augmentation for euphemism detection. likewise, finetuning pretrained transformer models is a popular approach. for instance, wiriyathammabhum (2022) fine-tune roberta @xcite models for euphemisim detection. @xcite combined roberta models with cost-sensitive learning to handle class imbalance issues. @xcite combined a bert-based transformer with a relational graph attention network and finetuned it for euphemism detection. however, these approaches cannot capture euphemisms' nuanced nature or how euphemisms change over time. with the advent of models such as bert and its successors, researchers have been able to show the potential for neural network models to understand complex language phenomena like metaphors, sarcasm, and idioms @xcite . while the llms have shown to be more capable, researchers identified that not only the size of the model and the training data used are important, but how a task is presented to the llm is equally important @xcite . prompting offers a few benefits over fine-tuning a llm. prompting does not require a model to undergo an additional round of training, making it more resource-efficient and accessible. also, prompting leverages the model's pre-trained knowledge, enabling quick adaptation to new tasks without the risk of overfitting. prompting is particularly appealing for subtle language tasks like euphemism disambiguation, allowing the llm to focus on the subtleties of euphemistic language without extensive training. a few researchers have used prompting in previous euphemism studies (keh; @xcite . @xcite a roberta model and fine-tuned the model to improve its performance using prompts. the most similar work to this paper is by keh (2022), which used an older gpt-3 model and post-processing rules to classify the evaluation as euphemistic or literal. their work found that fine-tuned models (e.g., roberta) outperformed zero-shot and few-shot methods using gpt-3. in this work, we extend the idea of using prompting in two ways. first, we use gpt-4, which is more capable than gpt3-3. second, this model is evaluated on the new multilingual euphemism dataset. ## methodology in this section, we discuss the general task, dataset, and our prompting strategy. overall, we use a fewshot prompting framework for our submission. task. the multilingual euphemism detection shared task for the fourth workshop on figurative language processing involves predicting whether a substring within a sentence is a euphemism. specifically, given a string, "this summer, the budding talent agent @xmath0," participants need to detect whether the embedded potential euphemistic terms (pet) is a euphemism or not for this specific context. this means that each pet can be a literal (not a euphemism or a euphemism). the participants' results are collected and evaluated on the shared task site at codabench. foot_0 dataset. for this shared task, two sets of data are provided, each consisting of samples in chinese, english, spanish, and yorùbá. the first sets are the training datasets to help refine the participants' methodology, consisting of rows of sentences, the embedded pet, and a classification label (euphemism or not). the composition of the datasets by language is provided in given the context, determine if the phrase 'pet' is used as a euphemism. reply with the word 'true' if it is used as a euphemism in this context else 'false'. ## results in this section, we report the results on both the validation and test datasets. validation dataset results. the validation dataset results are shown in next, we can find that adding in-context examples in the "few-shot -random examples" and few -shot -targeted example" methods improves the "zero-shot with context" methods. furthermore, we find that using targeted examples universally improves performance over random examples. when we add more in-context examples, the performance continues to improve. for instance, "few-shot -targeted examples" improves from .801 with four in-context examples to .859 with eight examples. from a language-to-language perspective, we obtained the worst in spanish, which is about 5% lower than the english results. test dataset results. the final competition results for our best system (i.e., few shot -targeted examples (8)) on the test dataset are shown in the results indicate that the prompting with the english test cases performed substantially better than the prompting with the spanish test cases, while the chinese and yorùbá test cases fell in between these two extremes. for the test experiments, the source of the sample cases to be included as random or targeted examples were pulled from the training datasets. the prompting proved most effective for the english dataset, and the results (f1=.831) were slightly higher than those measured during training. the results for both the chinese (f1=.776) and the yorùbá (f1=.723) datasets ended up falling between the "few shot random" and "few shot targeted (2)" prompt results for the training results for each language. the performance for the spanish dataset fell (f1=.598) to only slightly better than the original "zero-shot" results. when we look at the potential number of example cases to include with the targeted prompt, we find that with the english test cases, there was nearly 75% coverage. this means that 75% of the test pets were also included in the training dataset. however, with the spanish test cases, there was no overlap between the training data set and the test data set. the chinese and yorùbá data had test coverage between these two extremes. this may explain why the results with the spanish dataset were so poor (0% coverage) and why the chinese and yorùbá datasets fell between random and targeted (some coverage). ## future work while demonstrating the viability of our approach in identifying euphemisms, we also uncovered several research directions to pursue that could further enhance our understanding of the euphemistic speech capabilities of llms. openai's chat gpt-4 model is a highperforming llm trained on multi-lingual data. the llm demonstrated its capability of translating the training datasets from the original language into english without additional fine-tuning. limited testing during the development phase was performed using mistral @xcite and llama-2 llms @xcite but both exhibited zero-shot performance below chat gpt-4. the main focus of the study was on improving performance using prompting strategies, so the team directed its efforts to refine the prompts. as highly capable llm models are being released frequently, evaluating a variety of these models is an area of focus for future studies. our approach utilized only the model's inherent knowledge and a subset of the training data as additional knowledge to identify euphemisms. this additional knowledge was shown to signif-icantly improve performance during the training phase. for the cases in which there were multiple samples to choose from, the current approach randomly selected the samples to include and the order they were listed. a future research direction is to determine if the selection of examples using those that are more closely related to the test case improves the performance. also, does the order the samples are listed in the prompt affect the results? when reviewing the test performance ( there are two approaches we could pursue to address this. one would be to locate additional datasets online or create datasets from open-source language repositories. a second approach would be to use a language model to generate the additional samples. the attraction to this approach is that we could generate samples of a new pet being used in a previously unseen manner and assist the model in recognizing the new usage of a phrase. ## conclusion in this paper, we presented our approach for the 2024 figlang shared task for multilingual euphemism detection. we introduced a method using gpt-4 and in-context learning. this adjustment would be beneficial in a scenario in which the usage of a euphemism has changed over time, but the model has not yet been learned, or the model does not have a strong indication of being a euphemism without strong evidence. future areas to research include 1) using the llm to generate samples to include as examples to include in the multi-targeted prompt 2) improving the selection of targeted examples to identify those examples that are more closely related to the test case. 3) using the llm to identify potential euphemisms from the text in question without being supplied with this information.