H elsinki- NLP at S em E val-2022 Task 2: A Feature-Based Approach to Multilingual Idiomaticity Detection (2022)

Abstract: This paper describes the University of Helsinki submission to the SemEval 2022 task on multilingual idiomaticity detection. Our system utilizes several models made available by HuggingFace, along with the baseline BERT model for the task. We focus on feature engineering based on properties that typically characterize idiomatic expressions. The additional features lead to improvements over the baseline and the final submission achieves 15th place out of 20 submissions. The paper provides error analysis of our model including visualisations of the contributions of individual features.

## introduction we participated in the semeval 2022 task 2 (tayyar @xcite subtask a, zero-shot 1 setting: classification of a sentence containing a potentially idiomatic two-word multiword expression (mwe) as idiomatic or literal. the task provided four data sets 2 for english, portuguese and galician. each mwe was represented by multiple example sentences, accompanied by the context (previous and next sentences). each mwe could be always idiomatic, always literal or anything in between. the motivation for our approach is testing linguistically motivated features that reflect important properties of idioms, such as non-compositionality, non-substitutability, non-literal-translatability and affectiveness (see chapter 2) and to see whether pre-trained models can be helpful for capturing these features. our system uses a combination of models: bert fine-tuning @xcite , sentence embeddings @xcite and a feature model based on the above idiomatic properties. ## background and related work the detection and analysis of idiomaticity has a rich history in the literature. an important property of idioms is non-compositionality (that is, the meaning of the expression does not correspond to the combination of the meaning of its components). @xcite @xcite related to it are non-substitutability (components cannot be substituted with their synonyms) @xcite @xcite and non-literaltranslatability @xcite . idioms tend to be semantic outliers @xcite @xcite in the sense that they violate the lexical cohesion of the surrounding discourse. they are also known to be more affective (either positive or negative) @xcite than literal expressions. in addition to being relatively fixed lexically (non-substitutability), idioms often exhibit lack of label target 0 he was not a blue blood jurist issuing judicial decisions that nobody understood affecting people and corporations that nobody knew. 1 the blue blood of the fossil-like creature is the only natural source of limulus amoebocyte lysate, a clotting agent that is used to test batches of injectable drugs for bacterial contamination that could cause fever, organ damage and even death. compositionality and substitutability are often tested with techniques like backtranslation and mask filling tasks. backtranslation involves translating text to another (i.e. pivot) language and translating it back @xcite , and it has often been used for paraphrasing and data augmentation. backtranslations have also been used for idioms in related work, see, e.g., @xcite ; bahar @xcite . mask filling @xcite is closely related to the cloze task @xcite , where the objective is to predict a word missing from an expression. mask filling has lately been made easier as modern languages models such as bert @xcite and its derivatives are themselves so-called masked language models (mlm). mask filling can be useful for testing substitutability in context @xcite . ## system description our submission 3 considers three models: the baseline bert model provided by the task authors @xcite , sentence embeddings with sentence-transformers @xcite and a feature model based on idiomaticity features. all our components rely on existing models and tools that have been integrated into the transformers library provided by hugging-face @xcite . the final classification combines information from two components (either fine-tuned bert + feature model or sentence embeddings + feature model). the result will be taken from the model which has the higher label probability 4 . see figure 1 for an overview. we compare different variants of the system with the performance of individual features and various 3 implementation and details are available at https:// github.com/dustedmtl/semeval2022 . ## conclusions our system combines a feature model based on a number of idiomaticity features with a bert transformer classifier. the feature model achieves competitive results compared to the reportedly strong baseline @xcite , although it does not fare nearly as well as the best systems that competed in the subtask. unsurprisingly, most of the features work best for english, whether or not the underlying bert model is multilingual or not. the work shows that a classification system utilizing idiomatic properties such as noncompositionality, non-substitutability and affectiveness can be implemented with readily available transformer apis. another idea for future work is to improve the back-translation test by combining a "good" forward translation model (i.e. one that tends to properly treat idiomatic expressions) with a "bad" backtranslation model (i.e. one that tends to produce literal translations). the latter could also be done by forcing component-wise translations in the backtranslation step to reveal non-compositionality of the expression.