Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions (2022)

Abstract: Idiomatic expressions (IEs), characterized by their non-compositionality, are an important part of natural language. They have been a classical challenge to NLP, including pre-trained language models that drive today’s state-of-the-art. Prior work has identified deficiencies in their contextualized representation stemming from the underlying compositional paradigm of representation. In this work, we take a first-principles approach to build idiomaticity into BART using an adapter as a lightweight non-compositional language expert trained on idiomatic sentences. The improved capability over baselines (e.g., BART) is seen via intrinsic and extrinsic methods, where idiom embeddings score 0.19 points higher in homogeneity score for embedding clustering, and up to 25% higher sequence accuracy on the idiom processing tasks of IE sense disambiguation and span detection.

## introduction natural language has a common yet special class of multi-word expressions (mwes) called idiomatic expressions (ies) that exhibit semantic non-compositionality, where the meaning of the expression cannot be inferred from that of its constituent words (e.g., the idiom break a leg) @xcite . they are commonly used for specific communicative intents @xcite and are individually rare but collectively frequent, appearing frequently across genres @xcite . they have been classically regarded as a ''pain in the neck'' to nlp systems @xcite not only because of their non-compositionality, but also because of their contextual semantic ambiguity (used in idiomatic or literal meaning depending on the context). challenges posed by the presence of ies have been identified across multiple nlu tasks even with state-of-the-art (sota) solutions, including sentiment analysis @xcite , paraphrase generation @xcite , natural language inference @xcite , and dialog models @xcite . even the the flagship nlp model gpt-3 @xcite finds idioms challenging. we tested for its idiom comprehension over 75 idioms, covering a spectrum of the most to the least frequent idioms (based on their frequency of occurrence in the bnc @xcite ). we do this in question-answering mode where we ask gpt-3 foot_0 simple questions about idioms that can only be answered correctly if their meanings are understood in context (a sample of questions and answers are shown in despite the relative success of recent studies using pre-trained language models (ptlms) for ie processing, including idiom token and type classification @xcite , ie span detection @xcite , and idiomatic to literal paraphrasing @xcite , the emerging knowledge is that ptlms are unable to produce meaningful representation of ies in task-specific contexts (e.g., for ie sense disambiguation) @xcite . noting that solutions to improve ie representation in a task-agnostic manner are largely missing, this study aims to develop targeted solutions to make language models (lms) idiom-aware with the immediate objective of improving ie representation in large ptlms yet without relying on ## the inability to represent idiomatic expressions compositionality is a dominant paradigm driving the sota in nlp both at the tokenization and architectural levels. the tokenization of most lms, for example, byte-pair encoding (bpe) @xcite and wordpiece @xcite , assumes compositionality not only idiom bart iti+sf+si in the final analysis in the long run at the end of the day in the works in light of in light of all things being equal see red see the light go spare see stars fly off the handle go down like a lead balloon do someone's head in quick as a flash flash in the pan in the blink of an eye keen as mustard like a bat out of hell thin as a rake thick and fast at the phrase-level but also at the word level. this suggests that the meaning of a word is deduced from that of the subword components. at the architectural level, transformer-based lms implicitly consider all phrases (or even words) as compositional. the self-attention mechanism in transformers considers the embedding of a word to be an attention weighted sum of the word embeddings in its context. this design leads to phrase or even sentence embeddings to be overall compositional. in addition, each ie is individually rare, compounding the difficulty for obtaining good ie representation. this leads us to hypothesize that the inherent notion of compositionality and the rarity of ies are a hindrance to the representation of the ies that are inherently non-compositional. we test the validity of this hypothesis by analyzing ptlms' representation of ies. ## learning representation for idiomatic expressions toward producing higher quality ie embeddings by ptlms, we propose giea; given a set of idiomatic sentences (i.e., sentences that each contains an ie), giea freezes the base ptlm and trains an adapter that specializes in ie representations. this is done by reconstructing idiomatic sentences that are corrupted with an idiom-aware noising function and meeting a dictionary definition-aided objective. giea's overall framework is illustrated in figure 1 . in this work, we select bart as our base ptlm. ## conclusion and future work in this work, we first demonstrate current bart's inability produce semantically meaningful representations for idioms, then, we propose giea, that uses a lightweight adapter, a set of denoising auto-encoder-style learning objectives, and a similarity forcing objective to produce quality ie embeddings without altering the input tokenization. through both intrinsic evaluation of embedding quality and extrinsic evaluation on their usefulness on idiom-processing tasks, we find that giea greatly improves upon embedding quality and usefulness compared to the original pre-trained bart's embeddings. future work should explore means to improve embedding quality for hard-to-learn idioms based on observed performance, ies other than idioms (e.g., phrasal verbs), and the use of giea with other sota idiom processing models. lastly, applying idiom-aware ptlms to downstream applications that require the ie comprehension, such as dialog modeling and machine translation, would be fruitful pursuits.