Cross-Lingual Idiom Sense Clustering in G erman and E nglish (2023)

Abstract: Idioms are expressions with non-literal and non-compositional meanings. For this reason, they pose a unique challenge for various NLP tasks including Machine Translation and Sentiment Analysis. In this paper, we propose an approach to clustering idioms in different languages by their sense. We leverage pre-trained cross-lingual transformer models and fine-tune them to produce cross-lingual vector representations of idioms according to their sense.

## introduction idiom handling is an important aspect of any nlp system due to the unique way idioms can affect the meaning of a sentence. due to their noncompositional meanings, nlp systems need to treat idioms as a single lexical unit. in machine translation, in particular, current transformer models tend to struggle when translating an idiom because of this. experiments @xcite show that transformers @xcite often fail to treat idioms in this manner and instead translate them compositionally resulting in poor translations. this paper approaches the problem of clustering idioms in different languages based on their sense. through this, we aim to improve semantic representations of idiomatic expressions to aid nlp tasks that rely on accurate sense disambiguation. in this paper, we make use of pre-trained cross-lingual language models @xcite to do this. our approach involves fine-tuning these models to generate cross-lingual vector representations of idioms based on sense. these representations can then be used to form sense clusters of idioms. this idea can be further extended by leveraging idiom databases e.g. @xcite to identify the sense of an idiom not present in the database. by finding an idiom within the same cluster that is already in the database, we can infer the sense of the unknown idiom. idioms that share the same sense share a common meaning beyond their literal interpretations. machine translation systems often treat idioms compositionally and produce translations that are too literal and don't make sense in the translated text. the absence of parallel idiom datasets often hinders the effective training of transformers to address this challenge. we feel that our approach could aid this. instead of training models to translate idioms in isolation which is often not practical, we propose a method capable of grouping idioms by their shared meaning. this enables the models to understand the meanings these idioms convey and the relationships between them across languages. to evaluate our approach we conduct experiments using multi-lingual idiom datasets and assess the results. ## approach in this paper, we employ bert @xcite models that are pre-trained specifically for crosslingual contexts to facilitate our approach. we finetune the model by training it on a dataset consisting of english idioms and corresponding german idioms. we developed a dataset of roughly 14,000 english and german idioms for this purpose. in order to train the model, we load the dataset and create translation clusters which consist of idioms that are direct translations of one another (taken from the website dict.cc). during training, we try to ensure the sense vectors of idioms in the same translation cluster are close to one another in order to create effective sense clusters. we make use of the xlm-roberta @xcite model which is trained on 2.5 terabytes of data in 100 different languages. xlm-roberta idiom gloss es mit fassung tragen (to bear it with composure) take it on the chin grin and bear it gute miene zum bösen spiel machen (make a good face for the bad game) in den sauren apfel beißen (bite into the sour apple) ## clustering we made use of the test data and applied various clustering algorithms to the encodings produced by the model. we made use of the k-means clustering, dbscan @xcite and bisecting k-means @xcite algorithms. ## conclusion and future work in conclusion, our study presented a method of clustering idioms in different languages by their sense, making use of pre-trained transformer models. our experiments show that our model works effectively but struggles in some circumstances. during our tests, we found the model sometimes failed to cluster idioms of similar sense together. this can be partly attributed to the binary nature of triplet loss which fails to capture degrees of similarity between idioms. additionally, we identified a potential bias in the model's weighting of relationships between idioms in the same language. to address these issues, further work can be done to mitigate these issues. we will work towards developing better loss functions and finding methods of reducing the bias.