drsphelps at S em E val-2022 Task 2: Learning idiom representations using BERTRAM (2022)

Abstract: This paper describes our system for SemEval-2022 Task 2 Multilingual Idiomaticity Detection and Sentence Embedding sub-task B. We modify a standard BERT sentence transformer by adding embeddings for each idiom, which are created using BERTRAM and a small number of contexts. We show that this technique increases the quality of idiom representations and leads to better performance on the task. We also perform analysis on our final results and show that the quality of the produced idiom embeddings is highly sensitive to the quality of the input contexts.

## introduction idiomatic expressions present a challenge to large language models (llms) as their meaning cannot necessarily be derived from the composition of their component tokens, a trait that llms often exploit to create representations of multi-word expressions. the lack of compositionality leads to poor representations for idiomatic expressions and in turn poor performance in downstream tasks whose data includes them. semeval-2022 task 2b @xcite encourages the creation of better representations of idiomatic expressions across multiple languages by presenting a semantic text similarity (sts) task in which correct sts scores are required whether or not either sentence contains an idiomatic expression. the sub-task requires the creation of a self-consistent model in which a sentence including an idiomatic expression and one containing its literal meaning ('swan song' and 'final performance') are exactly similar to each other and equally similar to any other sentence. to achieve this goal, we investigate whether due to the similarity between idioms and rare-words schick and schütze's bert for attentive mimicking @xcite ) (bertram) model, which was designed for use with rare-words, can be used to explicitly learn high-quality embeddings for idiomatic expressions. we also investigate how many examples of each idiom are required to create embeddings that perform well on the task, as well as how the quality of contexts fed to the bertram model effects the representations and performance on the task. evaluating our model on the task shows that externally trained idiom embeddings significantly increase the performance on sts data containing idioms while maintaining high performance on general sts data. this improved performance gained an overall spearman rank score of 0.6402 and first place (of six entries) on the pre-train setting, and an overall spearman rank score of 0.6504 and second place (of five entries) on the fine-tune setting. foot_0 ## background adopting the idiom principle @xcite to produce a single token representation for mwes has been used widely within static embedding distributional semantic models @xcite . within contextualised representation models, @xcite show that the contextualised representations produced by context2vec @xcite and bert @xcite models can be used to differentiate between idiomatic and literal uses of mwes. however, the mwes are only represented by one token in the input, before being broken into many tokens using berts word piece tokenizer. tayyar madabushi et al., 2021 add a token to the bert embedding matrix and shows that this method improves representations through increased performance on their proposed sts task. the embeddings they add to bert are randomly initialised, however, and only trained during the fine-tun step on limited data. ## results the final results for our system on the test data can be seen in fine-tuning has a much lower effect on the performance of the system when evaluated on the test set than compared with the dev and evaluation sets, with only a small, but significant, rise in overall correlation. performance rises by only 0.0198 and 0.022 for english and portuguese respectively, and unlike on dev data we do not see a uniform increase on the sr idiom score. ## conclusion we build our system by augmenting bert models for each language with single token embeddings learnt using bertram. bertram is used due to its high performance on rare words, which share many properties with idioms such as noncompositionality and being rare examples of com-