S em E val-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding (2022)

Abstract: This paper presents the shared task on Multilingual Idiomaticity Detection and Sentence Embedding, which consists of two subtasks: (a) a binary classification task aimed at identifying whether a sentence contains an idiomatic expression, and (b) a task based on semantic text similarity which requires the model to adequately represent potentially idiomatic expressions in context. Each subtask includes different settings regarding the amount of training data. Besides the task description, this paper introduces the datasets in English, Portuguese, and Galician and their annotation procedure, the evaluation metrics, and a summary of the participant systems and their results. The task had close to 100 registered participants organised into twenty five teams making over 650 and 150 submissions in the practice and evaluation phases respectively.

## introduction multiword expressions (mwes) are a challenge for natural language processing (nlp), as their linguistic behaviour (e.g., syntactic, semantic) differs from that of generic word combinations @xcite . moreover, mwes are pervasive in all domains @xcite , and it has been estimated that their size in a speaker's lexicon of any language is of the same order of magnitude as the number of single words @xcite , thus being of crucial interest for language modelling and for the computational representation of linguistic expressions in general. one distinctive aspect of mwes is that they fall on a continuum of idiomaticity @xcite @xcite , as their meaning may or may not be inferred from one of their constituents (e.g., research project being a type of 'project', vs. brass ring meaning a 'prize'). in this regard, obtaining a semantic representation of a sentence which contains potentially idiomatic expressions involves both the correct identification of the mwe itself, and an adequate representation of the meaning of that expression in that particular context. as an example, it is expected that the representation of the expression big fish will be similar to that of important person in an idiomatic context, but closer to the representation of large fish when conveying its literal meaning. classic approaches to representing mwes obtain a compositional vector by combining the representations of their constituent words, but these operations tend to perform worse for the idiomatic cases. in fact, it has been shown that the degree of idiomaticity of a mwe can be estimated by measuring the distance between a compositional vector (obtained from the vectors of its components) and a single representation learnt from the distribution of the mwe in a large corpus @xcite . recent approaches to identify and classify mwes take advantage of the contextualised representations provided by neural language models. on the one hand, some studies suggest that pre-training based on masked language modeling does not properly encode idiomaticity in word representations @xcite . however, as these embeddings encode contextual information, supervised approaches using these representations tend to obtain better results in different tasks dealing with (non-)compositional semantics @xcite @xcite . as such, this shared task 1,2 presents two subtasks: i) subtask a, to test a language model's ability to detect idiom usage, and ii) subtask b, to test the effectiveness of a model in generating representations of sentences containing idioms. each of these subtasks are further presented in two settings: subtask a in the zero shot and one shot settings so as to evaluate models on their ability to detect previously unseen mwes, and subtask b in the pre train and the fine tune settings to evaluate models on their ability to capture idiomaticity both in the absence and presence of training data. additionally, we provide strong baselines based on pre-trained transformer-based language models and release our codetr which participants can build upon. the computational treatment of mwes has been of particular interest for the nlp community, and several shared tasks with different objectives and resources have been carried out. the siglex-mwe section foot_2 has organised various shared tasks, starting with the exploratory ranking mwe candidates competition at the mwe 2008 workshop, aimed at ranking mwe candidates in english, german and czech. 4 more recently, together with the parseme community, they have conducted three editions of a shared task on the automatic identification of verbal mwes @xcite @xcite . in these cases, the objective is to identify both known and unseen verb-based mwes in running text and to classify them under a set of predefined categories. interestingly, these parseme shared tasks provide annotation guidelines and corpora for 14 languages, and include 6 categories (with additional subclasses) of verbal mwes. the detecting minimal semantic units and their meanings (dimsum 2016) shared task @xcite consisted of the identification of minimal semantic units (including mwes) in english, and labelling some of them according to a set of semantic classes (supersenses). focused on the interpretation of noun compounds, the free paraphrases of noun compounds shared task of semeval 2013 @xcite proposed to generate a set of free paraphrases of english compounds. the paraphrases should be ranked by the participants, and the evaluation is performed comparing these ranks against a list of paraphrases provided by human annotators. similarly, the objective of the semeval 2010 shared task on the interpretation of noun compounds using paraphrasing verbs and prepositions @xcite was to rank verbs and prepositions which may paraphrase a noun compound adequately in english (e.g., olive oil as 'oil extracted from olive', or flu shot as 'shot to prevent flu'). apart from these competitions, various studies have addressed different tasks on mwes and their compositionality, such as: classifying verb-particle constructions @xcite , identifying light verb constructions and determining the literality of noun compounds @xcite , identifying and classifying idioms in running text @xcite , as well as predicting the compositionality of several types of mwes @xcite @xcite @xcite . ## dataset creation the dataset used in this task extends that introduced by tayyar @xcite , also including galician data along with portuguese and english. here we describe the four step process used in creating this dataset. the first step was to compile a list of 50 mwes across the three languages. we sourced the mwes in english and portuguese from the noun compound senses dataset (consisting of adjective-noun or noun-noun compounds) @xcite , which extends the dataset by @xcite and provides human-judgements for compositionality on a likert scale from 0 (non-literal/idiomatic) to 5 (literal/compositional). to ensure that the test set is representative of different levels compositionality, we pick approximately 10 idioms at each level of compositionality (0-1, 1-2, . . . ). for galician, we extracted noun-adjective compounds from the wikipedia and the cc-100 corpora @xcite using the following procedure: first, we identified those candidates with at least 50 occurrences in the corpus. they were randomly sorted, and a native speaker and language expert of galician selected 50 compounds from the list. the language expert was asked to take into account both the compositionality of the compounds (including idiomatic, partly idiomatic, and literal expressions), and their ambiguity (trying to select potentially idiomatic examples, i.e. compounds which can be literal or idiomatic depending on the context). in the second step of the dataset creation process, in english and portuguese, annotators were instructed to obtain between 7 and 10 examples for each possible meaning of each mwe from news stories available on the web, thus giving between 20 and 30 total examples for each mwe. each example consisted of three sentences: the target sentence containing the mwe and the two adjacent sentences. annotators where explicitly instructed to select high quality examples, where neither of the two adjacent sentences were empty and, preferably, from the same paragraph. they were additionally required to flag examples containing novel meanings, so such new meanings of mwes could be incorporated into the dataset. sentences containing mwes in galician were directly obtained from the wikipedia and the cc-100 corpora due to the sparsity of galician data on the web. during this annotation step, we follow the method introduced by tayyar @xcite , and add two additional labels: 'proper noun' and 'meta usage'. 'meta usage' represents cases wherein a mwe is used literally, but within a metaphor (e.g. life vest in "let the word of god be our life vest to keep us afloat, so as not to drown."). in the third phase, across all three languages, each possible meaning of each mwe was assigned a paraphrase by a language expert. for example, the compositional mwe mailing list had the associated paraphrase 'address list' added, whereas the idiomatic mwe elbow room had the associated paraphrases 'joint room', 'freedom' and 'space' added to correspond to each of its possible meanings. language experts focused on ensuring that these paraphrases were as short as possible, so the resultant adversarial paraphrases could be used to evaluate the extent to which models capture nuanced differences in each of the meanings. the final phase of the process involved the annotation of each example with the correct paraphrase of the relevant mwe. this was carried out by two annotators, and any disagreements were discussed (in the case of galician, in the presence of a language expert) and cases where annotators were not able to agree were discarded. ## participating systems and results twenty five teams in total participated, with the most participants to subtask a zero shot (20). the results for the individual subtasks are given in ## methods the primary goal of this shared task was to provide a platform for the evaluation of a variety of methods for the identification and represention of mwes. this section gives an overview of the methods that have been successful in each of the subtasks. in particular, we attempt to identify the combination of methods across submissions that have significant potential for future development. ## conclusions and future work we present, in this paper, 'semeval 2022 task2: multilingual idiomaticity detection and sentence embedding', consisting of two subtasks: i) subtask a, to test a language model's ability to detect idiom usage, and ii) subtask b, to test a model's ability to generate representations of sentences containing idioms. this task, aimed at boosting research into the detection and representation of idiomatic expressions, had submissions from 25 teams consisting of close to 100 participants. we additionally provide an overview and analysis of the methods used by participants, which we believe will help future research in this field. in particular, we highlight the need for distinct methods when detecting mwes that have been previously seen and when detecting ones that have not. in representing idiomatic expressions, we show, through the novel idiomatic sts task presented here, that models are rather effective when they have training data available, but, as demonstrated in the pre train setting, more methods of encoding mwes are required when training data is not available. while the top performing methods across this task have been driven by deep neural models independent of linguistic features, we highlight that this does not imply that the addition of linguistically motivated features does not lead to improvements on the task. instead, it points to the possibility of integrating these methods into the more powerful neural models in future work where an ablation study might shed more light on the impact of each feature.