For a Fistful of Puns: Evaluating a Puns in Multiword Expressions Identification Algorithm Without Dedicated Dataset (2025)

Abstract: Machine Translation systems has always faced challenges such as multiword expressions (MWEs) and wordplays, which impact their performance, being idiosyncratic and pervasive across different languages. In this context, we seek to explore the nature of puns created from multiword expressions (PMWEs), characterized by the creation of a wordplay from a source MWE to recontextualize it or to give it a humorous touch. Little work has been done on PMWEs in NLP. To address this challenge, we introduce ASMR, an alignment-based PMWE identification and tagging algorithm. We offer an in-depth analysis of three different approaches to ASMR, each created to identify different types of PMWEs. In the absence of PMWE-related datasets and resources, we proceed to a snowclone detection task in English.We also perform a MWE identification task in 26 languages to evaluate ASMR performances across different languages. We show that ASMR exhibits state-of-the-art results for the snowclone detection task and produces interesting results with the MWE identification task. These results may indicate that ASMR is suitable for a PMWE identification task.

## introduction a lot of work has been done on multiword expressions (mwes) in nlp since their introduction to the field by @xcite ; @xcite . they are generally described as combinations of words with a certain degree of idiomaticity at the lexical, syntactic, semantic, pragmatic and/or statistical levels @xcite . mwes are usually non-compositional or semicompositional @xcite , idiosyncratic, pervasive across different languages, and subject to varying degrees of variation @xcite . other phenomena, such as ambiguity and discontiguity, may also be an issue @xcite . because of these features, they represent a particular challenge in nlp, notably for machine translation systems, which need to take them into account @xcite . like any sequence of words, mwes can serve as the basis for creating puns and other kinds of wordplays. puns in multiword expressions (hereafter pmwes) are characterized by the creation of a pun or a wordplay from a source multiword expression in order to recontextualize it or give it a humorous touch. by this process, mwes such as (1) become (2) in the context of strikes in france in 2023. like mwes, pmwes can be translated from one language to another. for instance, (4) is a pmwe created from (3) working in both italian and english. however, studies show that the translation of puns is not well handled by machine translation systems @xcite . to our knowledge, and unlike mwes, pmwes have not been extensively studied in nlp, with very few resources available and almost no dedicated work on them. we find that pmwes can be interesting due to their dual nature as mwe and wordplay. machine translation tasks as well as automatic humor analysis could benefit from their study. moreover, pmwes might be useful to study the morphosyntactic and semantic evolutions of mwes, since they tend to accept new forms and/or meaning over time @xcite . in some cases, they may even be completely replaced by one of their own pmwe @xcite . in addition to sharing the same difficulties as mwes, pmwes pose challenges of their own. their identification in text can be even harder than that of mwes, for several reasons: (i) they tend to be less frequent in texts than mwes, (ii) although their source mwe generally remains recognizable, several letters or words may be modified when creating a pmwe and (iii) their meaning can be altered, making the use of semantic-based approaches more challenging. finally, differentiating a pmwe from a mwe can be a complex task, even for an individual with a certain expertise in these entities, as shown in @xcite . in this paper, we introduce asmr (align, segment, match, rank), an alignment-based algorithm whose goal is to identify and tag pmwe candidates in texts. we first present the architecture of asmr. we then proceed to various experiments in two different datasets in order to evaluate the performances of this algorithm. task 1 : snowclone detection we evaluate how asmr is able to detect snowclones (defined in section 2) in a given set of sentences. to do so, we use the catchphrase dataset @xcite . ## introduction to asmr asmr's main purpose is the identification and tagging of pmwes. it can be described as an alignment-based, semi-supervised approach. asmr takes a list of seeds, for instance prototypical forms of mwes, as described in @xcite , and a list of sentences in which we want to identify pmwes created from the seeds. as an output, asmr creates a ranking of pmwe candidates for each seed. it consists of a succession of 4 processes, which we describe here. these processes are illustrated ## snowclone detection we explained the features of asmr. we now use the catchphrase dataset @xcite to evaluate asmr capacity to detect if a sentence contains a snowclone. dataset. the catchphrase dataset consists of 3,855 snowclone-sentence pairs, of which 1,406 sentences allegedly contain the snowclone it was paired with. it proposes a binary classification task: for each snowclone-sentence pair, we must indicate whether the sentence contains the snowclone it was paired with. to achieve this classification task, @xcite used a featurebased svm model as well as a roberta-based model. we report the recall, precision and accuracy they obtained with these models in parameters. as asmr does not learn from input data, we use the train and dev partitions of catchphrase to determine the best parameters to run our experiments. we run asmr with 240 distinct sets of parameters on the train partition (see appendix b.2). these parameters include those of the vectorizer (number of ngrams and analyzer) and the threshold at which we consider a candidate to be a snowclone (according to its score). we only use token-level information during these runs. we select the 10 best sets of parameters for the train partition and the 10 best sets for the dev partition, for a total of 20 sets. we plan to run asmr on the test partition with these 20 sets of parameters and to report standard deviation. we repeat this process for each approach, asmr exact , asmr f uzzy and asmr combined for a total of 720 runs. results. we report the results we obtained with asmr with each approach in overall, asmr performs better than the models used by @xcite for the task of snowclone detection, although we note that our precision is slightly behind that obtained by their svm model. figure 1 shows the impact of threshold on the metrics we used for the best run (with asmr combined ). as expected for a ranking system, the lower the threshold, the lower the precision and the higher the recall. ## mwe identification we measured the performance of asmr for the task of snowclone detection in sentences with catchphrase. we now want to evaluate its ability to identify tokens belonging to mwes in a given set of sentences. dataset. we use the version 1.3 of the parseme corpus @xcite , composed of 26 languages and mainly containing verbal mwes. this corpus proposes a mwes tagging task. so far, only 2 systems have been tested on the parseme 1.3 corpus: seen2seen @xcite and mtlb-struct @xcite . @xcite report the results for these 2 systems on parseme 1.3. ## error analysis with parseme since parseme 1.3 offers several metrics on different subsets of mwes, such as discontinuous and unseen ones, we can perform a more refined analysis of asmr capabilities. as asmr f uzzy and asmr combined match misaligned words between a mwe and a candidate, low f-scores are expected. this is especially the case for asmr f uzzy , which match every word between the first and the last common words between a mwe and a candidate (as seen in unseen-in-train. one could argue that asmr should be able to see a minimal number of unseenin-train mwes, especially with the asmr f uzzy and asmr combined approaches. we argue that this can be the case, notably with shorter, more generic mwes, such as "break up". ## discussion in this work, we introduced asmr, a puns in mwe (pmwe) identification and tagging algorithm relying on sentence-level alignments and similarity scores to rank pmwe candidates. while earlier studies show that asmr can be used to extract good pmwe candidates in both french and arabic, no quantitative evaluation was yet performed, due to a lack of a pmwe annotated dataset. to get around this issue, we proceeded to 2 experiments in order to evaluate asmr functionalities. we first used a snowclone detection task on the catchphrase dataset in order to evaluate asmr's capacity to assert the presence of a pmwe candidate in a sentence. we then used the parseme 1.3 corpus to evaluate asmr identification and tagging performance on mwes for 26 languages. we show that asmr obtains state-ofthe-art results on the snowclone detection task and for three languages with the mwe identification task (irish, croatian and hindi). we performed an in-depth analysis of the limitations we encountered with some subsets of mwes within parseme, which allowed us to get a better understanding of asmr performance. it showed that, while true positive and false positive candidates tend to blend together in the ranking, the top n candidates seem to be pertinent in most cases. this observation is highlighted by both the mwe identification task and the snowclone detection task, where higher thresholds lead to higher precision and lower recall. we also note that, while we performed multiple runs for each task, our standard deviations are low, which can account for the robustness of asmr (see appendix b.4). we plan to create a pmwe dataset through participative sciences to further evaluate the performance of asmr. such dataset would also be useful to test the performance of other systems, either created for mwe or pmwe identification. we also plan to use asmr to observe possible changes affecting mwes over time. for example, the french expression "être comme un poisson hors de l'eau" ("to be like a fish out of the water") led to the creation of the pmwe "être comme un poisson dans l'eau" ("to be like a fish in water") in the 19 th century @xcite . over time, such pmwes can become conventionalized (as defined in @xcite ) themselves, ultimately supplanting the original mwe @xcite . asmr offers a valuable tool for studying these diachronic processes. by tracking frequency patterns of canonical and variant forms, we can observe the emergence, transformation, and possible lexicalization of new mwes in real time. indeed, we have already observed such dynamics in the fruit corpus @xcite : the original expression "travailler plus pour gagner plus" (nicolas sarkozy, 2007, "work more to earn more") appears less frequently than the variant "travailler plus pour gagner moins" ("work more to earn less"), suggesting an ongoing shift in usage and meaning.