How Are Idioms Processed Inside Transformer Language Models? (2023)

Abstract: Idioms such as “call it a day” and “piece of cake,” are prevalent in natural language. How do Transformer language models process idioms? This study examines this question by analysing three models - BERT, Multilingual BERT, and DistilBERT. We compare the embeddings of idiomatic and literal expressions across all layers of the networks at both the sentence and word levels. Additionally, we investigate the attention directed from other sentence tokens towards a word within an idiom as opposed to in a literal context. Results indicate that while the three models exhibit slightly different internal mechanisms, they all represent idioms distinctively compared to literal language, with attention playing a critical role. These findings suggest that idioms are semantically and syntactically idiosyncratic, not only for humans but also for language models.

## introduction recent transformer-based large language models have demonstrated strong capabilities in a sweep of tasks involving natural language understanding (e.g. @xcite ). however, few attempts have been made to understand the inner workings of these language models in terms of idiom processing. in this study, we conduct three experiments to explore the inner workings of transformer language models in idiom processing. specifically, we investigate the processing of bert, multilingual bert and distilbert by comparing the embeddings on the sentence level and on the word level. we also explore the attention mechanism on idioms com-041 pared to literal contexts. we ask three questions: many studies have explored how different linguistic information is represented in bert @xcite @xcite . @xcite al. (2019) al. (2019) suggests that semantics is spread across the entire model. @xcite found that the uppermost layer in bert was the worst-performing in downstream tasks. so far, there has been less research on the inner workings of distilbert @xcite and multilingual bert @xcite . most studies focus on comparing performance cross-lingually or in downstream tasks between these models @xcite @xcite . idiom processing in language models: studies are becoming increasingly engaged with the challenge of idiom representation in language models @xcite @xcite . nedumpozhimana and kelleher (2021) investigated how bert recognises idioms, suggesting that the indicator is found both within the expression and in the surrounding con-