Optimizing Multilingual Euphemism Detection using Low-Rank Adaption Within and Across Languages (2024)

Abstract: This short paper presents an investigation into the effectiveness of various classification methods as a submission in the Multilingual Euphemism Detection Shared Task for the Fourth Workshop on Figurative Language Processing co-located with NAACL 2024. The process used by the participant utilizes pre-trained large language models combined with parameter efficient fine-tuning methods, specifically Low-Rank Adaptation (LoRA), in classifying euphemisms across four different languages - Mandarin Chinese, American English, Spanish, and Yorùbá. The study is comprised of three main components that aim to explore heuristic methods to navigate how base models can most efficiently be fine-tuned into classifiers to learn figurative language. Multilingual labeled training data was utilized to fine-tune classifiers for each language, and later combined for one large classifier, while unseen test data was finally used to evaluate the accuracy of the best performing classifiers. In addition, cross-lingual tests were conducted by applying each language’s data on each of the other language’s classifiers. All of the results provide insights into the potential of pre-trained base models combined with LoRA fine-tuning methods in accurately classifying euphemisms across and within different languages.

## introduction in order to best understand this task, it is important to define what a euphemism is. euphemisms are a linguistic device used to soften statements, or to make statements more polite. some examples of a euphemism might be using the terms "between jobs" or "late" instead of "unemployed" or "dead," respectively @xcite . research proves that euphemisms are a multilingual feature that exists in numerous languages @xcite . by collecting more training data and testing on unseen data, we are further able to see the extent of how state-of-the-art language modeling captures these universally figurative traits. the ability to observe whether these elements of figurative language are taken into consideration during tasks like classification by large language models (llm) can be speculated as a topic of increasing interest in natural language processing communities. the growing number of base models, such as xlm-roberta, that can be utilized for downstream tasks like text classification, reasoning, and sequence generation is staggering and leads to further questions of how the existing methods can be tested and improved @xcite . by addressing the numerous kinds of euphemistic categories, and how they can be represented multilingually, this kind of research enables a greater level of natural language understanding by embodying an ambiguous and subjective aspect of languages @xcite . furthermore, by aiming to solve the problem of accurate classification of figurative language using machine learning, this task importantly measures how well a human language characteristic can be interpreted by llms. ## related work prior research determined that semantic category might influence cross-lingual transfer of information @xcite . this insight drives the intuition for this experiment. once the ostensibly optimal classification method is discovered, then we can perform a cross-lingual comparison to see how all other languages performed on classifiers fine-tuned for other languages. previous work is helpful in this regard, as it enables us to have a starting point to compare and contrast base models, which were chosen heuristically. the multilingual 1 8 classifier predictions will be included in the results as a comparison. it is important to note that the base model for the cross-lingual experiment remained the same. in other words, all languages had context for each inference, yet the fine-tuning the classifier certainly made a difference in the results. the complete visualization of this can be seen in figure 1 . by freezing all the parameters in the base model with the parameter-efficient fine-tuning (peft) method, we are able to explicitly train the new classifier on our input language datasets. this will ideally be able to increase the speed with which we fine-tune and keep majority of the base model parameters frozen @xcite . using parameter efficient fine-tuning essentially allows our model to be trained on a small set of new parameters, which is why peft was used for this experiment. the goal is to see how well we can utilize these methods for our classification purposes. multilingual word embeddings have been shown to also have produced positive results in text classification tasks @xcite . through examining the problem posed in the introduction of best classifying figurative language through fine-tuning llms, and incorporating adjacent work that has proven successful on tasks with similar goals, we can begin to formulate an overarching methodology. starting with different base models to explore a variety of new options, keeping in mind the limited compute resources available, we can focus our efforts in changing as little as possible from the base model in an effort to highlight the impact of the task's training and test data throughout the experiment. lora, specifically peft, allows us to do this by inspecting the given data, specifically how semantic information is transferred accordingly, in the model predictions. the aim is to emphasize how the arrangement of the model's data can affect classification predictions. ## conclusion in conclusion, we learned that the cased multilingual distilbert base model proved to have a faster some concluding speculations could be that english and spanish potentially have lower success rates in predicting whether a term is being used euphemistically or not due to less ambiguity in the instances for which they are being used. this is a curious assertion to prove in future work as the definition of what is ambiguous varies between speakers of a language. a major consideration that should also be noted, and potentially the subject of future work, is the impact that the unique number of pet categories has on the training process. english, for example, as mentioned before has the highest number, whereas chinese has the lowest number. as mentioned previously, the data imbalance problem was addressed with learning rate adjustment, due to concerns that alternative methods, such as undersampling, might eliminate crucial semantic information. another factor that should be noted is that chinese and yorùbá both needed to have truncation at inference time encoding, most likely due to bert models using word-piece tokenization @xcite . in other words, they saw more unknown words in their vocabularies, thus needing them to create more tokens and increasing the total length of the sequence for each example. future work could explore if token length, language family, more balanced training data, or different 4 11 figure 1 : this chart portrays the inference results of the cross-lingual split between which sequences were labeled as euphemistic, and which were labeled as non-euphemistic. the individual language classifiers (fine-tuned only on their respective language data) are included along with the multilingual classifier to show contrast. gold standard labels are unknown and were not available to include in this figure . values can be found in figures 2 3 4 5 .