Semantics of Multiword Expressions in Transformer-Based Models: A Survey (2024)

Abstract: Multiword expressions (MWEs) are composed of multiple words and exhibit variable degrees of compositionality. As such, their meanings are notoriously difficult to model, and it is unclear to what extent this issue affects transformer architectures. Addressing this gap, we provide the first in-depth survey of MWE processing with transformer models. We overall find that they capture MWE semantics inconsistently, as shown by reliance on surface patterns and memorized information. MWE meaning is also strongly localized, predominantly in early layers of the architecture. Representations benefit from specific linguistic properties, such as lower semantic idiosyncrasy and ambiguity of target expressions. Our findings overall question the ability of transformer models to robustly capture fine-grained semantics. Furthermore, we highlight the need for more directly comparable evaluation setups.

## introduction multiword expressions (mwes)-such as noun compounds (e.g., jet lag), particle verbs (e.g., take off ), and idioms (e.g., on the fly)-are composed of multiple words and exhibit semantic idiosyncrasy, i.e., their overall meaning cannot be directly predicted from the meanings of their constituents. they are ubiquitous, affect various applications, and as such have been extensively addressed in nlp research @xcite . although the widely used transformer-based language models have been analyzed regarding their ability to represent various types of linguistic knowledge, we still lack consolidated insights into their processing of mwe semantics. the present survey provides a critical overview of existing work on this issue. by definition, the meaning of a mwe is distributed over multiple constituents. 1 in some cases, these can be separated by intervening material (e.g., turn the volume up). for any model, it is more challenging to capture the meaning of multiple lexical elements than that of a single word. the overall meaning may further be compositional to various degrees, i.e., similar to the sum of the component parts (e.g., climate change) or unrelated to it (e.g., silver bullet). moreover, the same expression may be interpreted more or less compositionally depending on its context (e.g., kick the bucket). transformer models are assumed to easily overcome some of these challenges since their meaning representations are inherently contextual and successful on a wide array of semantic tasks @xcite . but does this imply a robust ability to represent mwe meanings? the answer to this question is not trivial, as such an ability requires the models to systematically (i) capture the semantic contributions of multiple tokens, potentially including figurative and rare meanings; (ii) weigh them based on variable @xcite further specify the interpretation in context. in order to establish the current state of knowledge on this issue, we provide the first in-depth survey of the fast-growing body of work on mwe representations in transformer models. we aim to understand the extent to which different pretrained and optimized models are able to capture mwe semantics, as well as whether they are affected by representational or linguistic factors. most studies focus on a single expression type (e.g., noun compounds), task (e.g., machine translation), or related wider issue (e.g., phrase representations). we broaden the perspective by systematizing insights from often disjoint strands of research, and identify priorities for future work. our findings more generally highlight that transformer models represent complex linguistic knowledge inconsistently. we first provide general background on the surveyed mwe approaches ( §2) and then zoom in to transformer models. we explore whether they inherently capture mwe information ( §3), where it is localized ( §4), and whether it is affected by linguistic properties ( §5). we conclude with a summary and directions of future work ( §6). ## survey overview this section motivates our survey by providing background on lexical semantic representations in transformer models, and then summarizing the most frequent implementations and tasks. ## general model properties in this section, we first assess if mwe meanings are inherently captured by off-the-shelf transformer models, and if these can be further optimized so as to improve mwe representations. we then look into general mechanisms that may support this ability: recall of memorized information and generalization to unseen data. ## impact of representational information shifting the focus from general insights into mwe semantics captured by different transformer models, we now adopt a finer-grained perspective and examine how mwe representations are impacted by structural factors, i.e., model and input properties that directly affect the representational information extracted from a given transformer architecture. we address three such factors: transformer layers, tokens within the sequence, and the context surrounding the target expression. ## impact of linguistic properties mwe representations may be affected by properties of the target expressions themselves. we now provide a breakdown of the reported effects. individual expressions vary in terms of their inherent predictive properties, as shown in work on the usefulness of individual idioms when training an idiomaticity classifier. @xcite note a positive effect of informativeness, measured by training a classifier on one idiom and evaluating it on the full set of idioms; and ease of prediction, measured by training a classifier on the full set of idioms and evaluating it on one idiom. models are affected by the degree of semantic idiosyncrasy. @xcite obtain better results for attribute selection on phrases with higher semantic transparency. on idiomaticity detection, zeng and bhat (2021) report a slight gain (≈ 0.03 f1) for expressions that are fixed rather than semifixed or syntactically flexible. non-idiomatic expressions are better represented in lower layers (see §4.1; @xcite . as for other semantic properties, lower polysemy is associated with better results on attribute selection @xcite and compositionality prediction @xcite , while it does not affect word similarity across paraphrase pairs @xcite . more concrete words are assigned more weight when estimating constituent contributions to compound meaning @xcite . the type of semantic knowledge affects metaphor interpretation, with an evaluation on 10k examples showing better results for object and visual commonsense metaphors (referencing common objects and their visual attributes) than for social and cultural commonsense metaphors (referencing human behavior and cultural norms) @xcite . varied effects have been noted for frequency. they may be partly model-specific, with low-frequency compositional phrases yielding higher surprisal in gpt-2 and t5, but not bert @xcite . different noun compound analyses have reported that frequency has no effect @xcite , that low frequency is detrimental @xcite , and that it is beneficial @xcite . in the latter case, the trend may be explained by correlation with properties such as productivity, with expressions in lower productivity ranges obtaining better representations. more generally, the inconsistencies may stem from the use of different datasets, task formulations, and modeling approaches. this highlights the need for more systematic investigations of frequency effects. due to cross-linguistic variability in mwe realizations, their optimal computational representations may differ across languages. yet our ability to assess these trends is limited: most surveyed work focuses on english, with one to two papers analyzing chinese, gallician, german, japanese, portuguese, russian, and slovenian. we have not identified language-specific patterns for comparable experiments conducted on different languages. beyond our survey perspective, direct evidence of cross-linguistic variability is provided by limited studies on two languages in parallel. some of these report broadly comparable cross-linguistic patterns, e.g., on probing for compound semantics in english and portuguese @xcite and on idiom token classification in english and japanese @xcite . varied cross-linguistic differences have been observed elsewhere. comparable phrase similarity experiments have obtained the best results using sentence embeddings for english and verb embeddings for galician @xcite . on compound compositionality prediction, the best model for english is (monolingual) bert, and for portuguese (multilingual) sbert @xcite . a pie identification experiment has found better monolingual generalizability for english than russian, and better cross-lingual transfer from russian to english than vice-versa @xcite . but each of these cases is a single point of reference, making it unclear if the trends are due to model or dataset properties rather than cross-linguistic differences. summary. beyond inherently more informative mwes, transformer representations are better with lower semantic idiosyncrasy and dispersion (cf. polysemy, productivity). they also appear to be biased towards concrete expressions, while the precise effect of other factors such as frequency remains unclear. cross-linguistic analyses are limited-both regarding the coverage of different languages and direct comparisons across them-with further work needed to establish reliable trends. ## conclusion and outlook we have presented a survey of recent work on mwe semantics in transformer-based language models. starting with a general assessment of pretrained representations, we have seen that they capture some aspects of mwe meaning, but this ability is neither comprehensive nor consistent. it can in principle be improved with optimization strategies such as fine-tuning and knowledge enhancement, but with highly variable gains. mwe representations rely on memorized information rather than sophisticated meaning processing; this is reflected by suboptimal generalization ability. turning to differences in representational information in model architecture and textual input, we find that the most adequate representations are those corresponding to the linguistic structure of interest modeled within broader context; this enables the attention mechanism to efficiently encode expression-level information. there is also broad consensus that lower layers are better at capturing mwe meaning, as observed on tasks such as predicting compound compositionality, pie idiomaticity, and paraphrase similarity. however, implementation decisions should always be carefully tuned because of interactions with other factors. this includes a range of linguistic properties, with better representations for expressions exhibiting less semantic idiosyncrasy and dispersion. the surveyed papers provide varied and valuable insights, but many conclusions are not directly comparable and cannot be extrapolated across mwe types or models. we particularly underscored this issue regarding (i) optimization strategies, which may interact with target expression types, models, and evaluation tasks; (ii) layer-wise processing mechanisms, with similar patterns for memorized and compositional expressions; and (iii) cross-linguistic variability, with insufficient evidence to identify broad trends. future studies can address these challenges through several lines of work: (i) extending the coverage of mwes beyond the current focus on compounds and idioms, ideally in a comparative setup, and systematically accounting for the effect of their linguistic properties. (ii) extending the coverage of non-english languages, including in cross-linguistic evaluations. (iii) broadening the scale of evaluations by multiplying experimental parameters, e.g., by investigating model structures across architectures as well as mwe types. (iv) formulating tasks that are challenging in terms of both core semantic mechanisms and generalization requirements. we believe that these perspectives will help disentangle interactions between experimental parameters and improve the generalizability of the resulting claims.