HIT at S em E val-2022 Task 2: Pre-trained Language Model for Idioms Detection (2022)

Abstract: The same multi-word expressions may have different meanings in different sentences. They can be mainly divided into two categories, which are literal meaning and idiomatic meaning. Non-contextual-based methods perform poorly on this problem, and we need contextual embedding to understand the idiomatic meaning of multi-word expressions correctly. We use a pre-trained language model, which can provide a context-aware sentence embedding, to detect whether multi-word expression in the sentence is idiomatic usage.

## introduction the goal of the @xcite subtaska is to detect whether a multi-word expression in a sentence is idiomatic in usage. it is a multilingual task and consists of three languages: english, portuguese and galician. multi-word expressions (mwes) are expressions that consist of at least two words and are syntactically or semantically specific. the semantics of mwes are usually divided into two types, (i) the combination of literal meanings of each word in the phrase or (ii) inherent usages (e.g., idiomatic meaning). understanding the semantic meaning of a sentence requires the correct identification of the mwe in the sentence. traditional non-contextual word embedding models, such as word2vec @xcite , perform poorly at this task. simple superposition of non-contextually word embeddings does not correctly express the semantics of idiomatic phrases. therefore, contextual embedding models @xcite are required to correctly understand the meaning of multi-word expressions in idiomatic usage. ## system overview figure 1 depicts the flow chart of the whole system. we first preprocess the data, tokenizing and then feed into a pre-trained model to get hidden states. apply some pooling method to get fixed length sentence representation to train a softmax binary classifier. after that, the prediction results of the model are post-processed. in the training process, contrastive learning, adversarial training, regularized dropout, etc. are used. ## conclusion we continuously improve the model performance by improving the mwe marking method, using larger pre-trained models, adding regularization terms and heuristic rules. inspired by pre-trained models, we can find that future work needs to focus more on external data besides training data, especially data with specific idioms, such as idiom dictionaries, because the size of the external data is much larger than the training data, which has not been fully exploited.