Unified Representation for Non-compositional and Compositional Expressions (2023)

Abstract: Accurate processing of non-compositional language relies on generating good representations for such expressions. In this work, we study the representation of language non-compositionality by proposing a language model, PIER+, that builds on BART and can create semantically meaningful and contextually appropriate representations for English potentially idiomatic expressions (PIEs). PIEs are characterized by their non-compositionality and contextual ambiguity in their literal and idiomatic interpretations. Via intrinsic evaluation on embedding quality and extrinsic evaluation on PIE processing and NLU tasks, we show that representations generated by PIER+ result in 33% higher homogeneity score for embedding clustering than BART, whereas 3.12% and 3.29% gains in accuracy and sequence accuracy for PIE sense classification and span detection compared to the state-of-the-art IE representation model, GIEA. These gains are achieved without sacrificing PIER+’s performance on NLU tasks (+/- 1% accuracy) compared to BART.

## introduction non-compositionality is a characteristic of natural language, where the meaning of the expressions cannot be deduced from its components @xcite . these non-compositional expressions, often referred to as being idiomatic, assume figurative meanings and are collectively a common occurrence appearing in nearly three out of ten sentences in english @xcite across various genres @xcite . the challenges they pose to nlp systems have been acknowledged as the classical 'pain in the neck' @xcite and are recently found to impact various nlp tasks negatively, such as sentiment analysis @xcite , dialog models @xcite , and paraphrase generation @xcite . modern nlp systems, however, are primarily driven by the notion of compositionality, which is at the core of several system components, including tokenization @xcite and the self-attention mechanism @xcite . more fundamentally, recent studies @xcite reveal that the pre-trained language models (ptlms), such as gpt-3 @xcite and bart @xcite , are ill-equipped to represent (and comprehend) idiomatic expressions' (ie) meanings. this is demonstrated by the lack of correspondence between the ie meanings and their embeddings; ies with similar meanings are not close in the embedding space. conversely, ies close in the embedding space have a significant token or syntactic overlap. from a representation standpoint, this highlights the need for language models (lms) to handle noncompositionality through valid representations. efforts to generate semantically congruent representations for ies are now coming to the fore. for instance, giea @xcite uses a frozen pre-trained bart that is injected with trainable adapter layers @xcite to generate ie embeddings for non-compositional expressions. with better meaning-representation correspondence, the noncompositional expert giea performs better than bart in downstream ie processing tasks. yet, this advance is limited by the assumption that all ies occur in their idiomatic sense and ignores their contextual ambiguity that makes them potentially idiomatic expressions (pies)-their meanings can be understood either literally or idiomatically in a context-dependent manner @xcite 1 . for example, the pie "behind closed doors" should be interpreted literally in always lock valuables behind closed doors and idiomatically in they avoided any publicity and made all deals behind closed doors. ideally, their representations ought to be distinct in these two contexts. however, examining the representation of 235 pies that are largely unrelated in their literal and idiomatic context (their literal pie embeddings and idiomatic definitions have a mean cosine similarity of 0.0047), we notice that their representations generated by the state-ofthe-art @xcite exhibit a high cosine similarity between their idiomatic and literal pie embeddings (mean cosine similarity of 0.82). towards addressing this discrepancy, this study extends giea's ability in two concrete ways. first, through semantically meaningful representations for non-compositional expressions we enable effective handling of non-compositionality. second, by generating context-appropriate pie representations distinct for idiomatic and literal pies we enable effective contextual disambiguation of pies. addressing these issues involves attending to the following challenges. (1) bart and giea's abilities should be combined to generate good embeddings for pie in a context-dependent manner. (2) with the self-supervised reconstruction task as the sole objective, ptlm parameters are already optimized for token reconstruction from their token embeddings. to represent pies, bart and giea's learning objectives should be revamped. to address these challenges, we propose potentially idiomatic expression representation generator (pier). inspired by adapterfusion @xcite , which has been used to combine task-specific adapters, pier generates embeddings by combining the output from each giea adapter layer and pre-trained bart transformer layer with an attention fusion layer serving as a routing mechanism that passes compositional or non-compositional embeddings based on the context.. it is trained under the supervision of external knowledge, e.g., ie dictionary definition and pie senses, that helps the model to disambiguate and comprehend pies' literal and idiomatic meanings via a cosine-similarity based learning objective and a set of mask-infilling tasks with prompts. our main contributions are as follows. the deal was made behind closed doors. </@xmath0. ## unified pie representation generator to create a single language model that produces contextually appropriate embeddings for pies, in pier, we combine bart's ability to generate embeddings appropriate for compositional meanings and giea's to non-compositional meanings such that pier should output giea-style embeddings when the pie is used idiomatically and bart's embedding otherwise. we implement an attention layer that acts on the outputs from each frozen giea's adapter layer and frozen bart's transformer layer and serves as a "routing" mechanism for the compositional and non-compositional type embeddings. to train the attention layer, the overall loss is the sum of (1) a cosine similarity-based part for the embedding to encode meaning via external dictionary definitions, and (2) a reconstruction cross-entropy part that teaches the embedding of the association between the pie senses and sentence contexts. optimizing these two losses jointly allows the model to link pies' meanings to their contextual uses. the overview of pier framework is shown in figure 1 . ## results and analyses similarly, for spandet, a much more demanding task requiring detection and tagging simultaneously, pier+ has a sequence accuracy that is 28.54% higher than bart and 3.29% higher than giea. we point out that 22.43% of sentences in the test set have literal pies, over which giea's sequence accuracy is only 71.84% while pier+ has a sequence accuracy of 85.76%, gaining 13.91% absolute points in accuracy. observing the tokenlevel recalls, pier+'s performance is only slightly better than that of giea, yet leads to a 3%+ gain in sequence accuracy. plausibly, this is because of fusionmultiprompt's better ability to recognize literal pies (as shown by the ∼14% sequence accuracy gain on literal sentences). so, with its ability to produce meaningful idiomatic embeddings, giea achieves high sequence accuracy in span-det, whereas pier+'s ability to generate appropriate literal embeddings allows it to improve further. performance on nlu tasks. pier+ performs competitively with bart and giea (f1 and accuracy differing by around +/-1%). given that the main purpose of pier+ is for producing highquality pie embeddings and enhancing ie processing ability, the results on the nlu tasks lead us to conclude that (1) pier+ (and giea to a lesser extent) adequately processes sentences with or without pies and thus performs comparably with bart on classic nlu tasks, i.e., pier+ does not breakdown on sentences without pies; and (2) given that pier+ produces pie embeddings with superior semantic properties and performs very well on pie processing tasks, we believe pier+ is overall a better lm than bart for pie processing. effect of individual components. as shown by fusionattn's performances in finally, we tested the usefulness of the classification and the generation prompts through an ablation study, where we compare pier with only the type classification prompt (p-cls) and pier with only the definition generation prompt (p-defn). each prompt type utilizes the same five prompt templates as the pier+ model. as shown in tables 1 and 2 , even with a single prompt type, our models achieve significant improvements in both intrinsic evaluation and pie processing tasks while maintaining a competitive performance in nlu tasks. however, when combining both prompt types, pier+ outperforms p-cls and p-defn, especially in the most difficult spandet task, with gains of 5.6% and 7.4% in sequence accuracy, respectively. these results lead us to infer that combining the two types of prompts is beneficial and leads to further performance gains. effect of combined components. the salient effect of combining all components is shown by comparing fusionsim and pier. pier gains 16.5% in homogeneity score and 0.32 in inter-type cosine similarity while achieving 1.65% higher accuracy in senseclf and 17.3% higher sequence accuracy in spandet. moreover, comparing pier+ to pier, we observe a further meaningful gain in all metrics across all tasks, indicating the benefit of including multiple prompt templates. ## conclusion and future work we propose pier, a language model that uses attention fusion to combine bart and a previously proposed adapter to produce semantically meaningful and contextually appropriate representations for pies. training using a prompt-infilling objective for contextual pie-type awareness and a cosine similarity objective to guide the generated pie embeddings toward their idiomatic or literal meanings resulted in pier generating pie embeddings with superior semantic properties. ie-aware pier outperforms both bart and giea on ie processing tasks with bart-level performance on classic nlu tasks. these results demonstrate pier's usefulness as an idiom-aware lm. future directions could explore methods to further enhance the ie awareness of broader types of non-compositional constructions beyond idioms (e.g., metaphors and similes).