Will it Unblend? (2020)

Abstract: Natural language processing systems often struggle with out-of-vocabulary (OOV) terms, which do not appear in training data. Blends, such as “innoventor”, are one particularly challenging class of OOV, as they are formed by fusing together two or more bases that relate to the intended meaning in unpredictable manners and degrees. In this work, we run experiments on a novel dataset of English OOV blends to quantify the difficulty of interpreting the meanings of blends by large-scale contextual language models such as BERT. We first show that BERT’s processing of these blends does not fully access the component meanings, leaving their contextual representations semantically impoverished. We find this is mostly due to the loss of characters resulting from blend formation. Then, we assess how easily different models can recognize the structure and recover the origin of blends, and find that context-aware embedding systems outperform character-level and context-free embeddings, although their results are still far from satisfactory.

## introduction for the token-based architectures that dominate contemporary natural language processing, a particularly difficult form of linguistic generalization arises from unseen phenomena at the word level, where novel sequences of characters, morphemes, or phonemes are known as out-ofvocabulary (oov) terms @xcite @xcite . pretrained transformers like @xcite handle oov terms by subtokenization: segmenting all whitespace-delimited tokens into smaller units, from which any oov term can be constructed @xcite . 1 but while this ap1 another approach is to operate directly at the character level (e.g., @xcite , but this has not been widely proach is well suited for phenomena like concatenative english morphology, many linguistic processes generate oov terms that cannot be cleanly decomposed into meaningful subtoken segments. in this paper we address a particularly interesting and challenging source of oov terms: novel blends @xcite , also known as portmanteaux @xcite . blends are constructed from the combination of multiple bases into a new form, in which some characters is shared across both bases: for example, shop + @xmath0. in this way, blends differ from other lexical compounds (e.g., @xmath1, which are formed by simple concatenation. examples of oov blends and their bases from our novel english blends dataset, collected from a natural source linked to the blends' originating contexts ( §2), are presented in our main contribution is to offer what is to our knowledge the first analysis of how transformerbased contextual embedding models process novel blends and the representations they are able to produce for these challenging forms. first, we examine the impact of blends' wordforms by comparing the ability of contextualized models to represent blends against the minimally-different case of novel lexical compounds. in §3, we show that the limited ability of contextual language models to represent novel blends' components faithfully is primarily attributable to their form properties, whereas semantic differences between compounds and blends play a much smaller role. we adopted within the mainstream framework of transformerbased models @xcite , due in part to the difficulty of scaling transformer-based models to sufficiently large contexts when operating at the character level. blend paxobs bases semantic relation definition hatriotism axxbbbbsss hate patriotism attribute hate disguised as patriotism. shoptics aaxxbbbs shop optics loc-part-whole the social image projected when shopping. innoventor xxaaxbbxxx innovator inventor causal a person who innovates by inventing. thrupple aaabobbb three couple containment a group of three people acting as a couple. even under favorable conditions, we find that systems proposed previously for similar tasks struggle on blends, showing limitations of form-based and distributional similarity approaches. we propose a novel unsupervised base recovery method using contextualized masked language models, bert ranker. while this system performs well relative to others, we find substantial room for improvement. in our view, these results demonstrate the need for future work on our novel dataset and associated tasks. foot_0 ## complex words dataset our proposed investigation of the behavior of nlp systems on novel complex words requires a highquality, reliable resource of truly novel blends and compounds in their original contexts, annotated for character sequence composition and semantic properties. the nytwit dataset @xcite ) contains english words new to the new york times extracted by a bot foot_1 between the dates of november 2017 and march 2019 with associated news article contexts. words were annotated for their type of novelty. we extract and further annotate three types from this dataset (version 1.1): blends (142 items), transparent compounds (121), and opaque compounds (49). 4 the difference between the compound classes is semantic and somewhat subjective: transparent compounds have meanings which are comprehensible with little context (e.g. quizmaker, a person who makes quizzes), while opaque compounds exhibit metaphoric or allusive semantics (e.g. deathbox, a dangerous car). the first two authors annotated each word for its constituent bases, the character locations in which each base is represented, and the semantic relation between the bases. a sample of annotated blends is presented in ## blends in context novel blends are a unique linguistic phenomenon, posing challenges for automated systems on many different levels. however, the sparsity of their appearances in real-world text, as well as the expertise required for creating a natural language understanding task which uses specific documents from a large variety of domains as supporting information, make the evaluation of the effect of novel blends on this type of downstream task an impractical goal for the scope of this work. instead, we assess the treatment of novel blends at the representational step of contemporary contextualized language models, by performing an analysis of their processing by @xcite . to gauge how well bert represents blends, we conduct a comparison with its treatment of a minimally-different control class of novel words, namely lexical compounds. these are forms where at least two bases are concatenated in full (e.g. quizmaker), without the character loss incurred in blends. our analysis begins with the assumption that in any given context, the meaning representation of a complex word (blend or lexical compound) must be composed from its bases, which we can estimate using the representational similarity between a complex word and its bases in the same context. this criterion can be viewed as a form of linguistic generalization, and if satisfied, enables downstream models to produce consistent results across related words. to test this criterion, we compute the vector similarities between the contextualized representations of complex words and their components, a method that coheres with human judgments of contextual semantic similarity @xcite . we probe bert foot_5 with synthetic inputs constructed by replacing each complex word with its spacedelimited bases. formally, given a sentence @xmath2, . . . , w i 1 , x, w i+1 , . . . , w n ) where x denotes a blend or compound with contributing bases b 1 , b 2 , we record the average vector across x's wordpiece tokens for each layer output in bert's transformer stack, e (l) (x), l 2 0, . . . , 12, and compute its cosine similarity with the averaged vecfigure 1(a) compares the per-layer similarities for blends with the two types of compounds described in §2: we find a clear distinction between blends and both compound classes. for compounds, bert induces representations that are very similar to those of the components at all layers of the model. for blends, these representations diverge greatly, especially in the lower layers of the model, which capture surface-form characteristics of the input @xcite . since the difference between classes exists across all layers, we first wish to perform a more thorough analysis of possible reasons for it. ## will it unblend? we next test to what extent existing models can help systems understand the meaning of novel blends, an aspect of human language understanding that has been little explored in nlp evaluation tasks. as demonstrated in §3, successfully representing blends requires the capability to both properly decompose their form and identify the original constituents. we therefore cast blend understanding as two tasks: segmenting blends into character sequences ( §4.1), and recovering blends' bases post-segmentation ( §4.2). we leave the task of recognizing blends to future work. 15 compounds. compounds were used as a comparative class in §3, but for the purpose of form understanding we focus on blends. for compounds with a known segmentation, base recovery is trivial, as each side of the segmentation point is always a base. as for segmentation, we have shown in §3.2 that bert's transformer layers are capable of recovering from poor wordpiece performance, and so the utility of segmenting compounds is limited compared to blends. 16 knowing that words are kept in their original form can define much simpler and more effective systems of discovery than the ones described below, such as a dictionary lookup of both sides for each possible single segmentation point. ## related work prior work on blends has largely focused on generation (e.g., @xcite @xcite @xcite provide a unified dataset of 1,579 blends, annotated for bases, they do not provide contexts for real-world appearances of the blends, nor a breakdown of the semantic relationship between their constituents. moreover, some are synthetically generated by a seq2seq model. in addition, these works all restrict their models to linear two-word blends. our paxobs scheme handles nonlinear and multi-base blends. cook and stevenson (2010) presented a noncontextual method for blend base detection using a dictionary-based lexicon, evaluated over an unreleased dataset, and ek (2018) used features from static embeddings to unblend words in swedish. we adopt the candidate-ranking approach of these works to evaluate component recovery, but incorporate context with context-sensitive language models, and add the task of blend segmentation. extracting the semantics of constituents from larger phrases is not a problem unique to singletoken blends. @xcite worked on multi-word compounds; @xcite segment hashtags, roughly half of which are akin to our notion of compounds, by training a neural scoring system over features extracted from word form, dictionary lookup and language model probabilities. another connection is to the learning of morphological rules, e.g., for processes such as derivation @xcite and lemmatization @xcite @xcite . cotterell and schütze (2018) present a supervised model of derivational morphology that jointly accounts for segmentation as well as composition of static word embeddings from the embeddings of morphemes, thereby touching on two of the main tasks undertaken in our paper. however, the application of such a model to blends is complicated by the relative lack of labeled training data, as well as the irregularity of the underlying phenomenon. novel blends are an example of linguistic creativity, which frequently operates at the subword level. related phenomena include eggcorns, which are alternative spellings that yield an apparently more transparent relationship between form and function @xcite ; puns, which substitute words in new contexts based on phonological similarity @xcite ; respellings that attempt to reintroduce prosodic expression into spelling @xcite ; intentional obfuscation @xcite ; and typographical errors @xcite . we therefore view blends as an instance of a broad set of creative phenomena that poses challenges for the token-based approaches that currently dominate natural language processing. ## conclusion this work focuses on the challenge of interpreting novel blends, which requires integrating subword structure and contextual features. we present a new dataset annotated using a novel characterlevel schema as well as for semantic tags, and offer preliminary evaluations showing that (a) blends are handled differently than compounds by bert, due mostly to the phenomenon of character loss; (b) existing tokenizers generally do not respect blend boundaries; and (c) recovering the components of a blend is a difficult task which challenges word-form, distributional, and contextual approaches. our results further highlight that annotation schemata such as those of @xcite , which were designed for noun com-pounds, are generalizable to other relational word types. in future work, we plan to integrate these signals into a better blend processor, and to further address the effect of blends on downstream tasks from semantic and syntactic viewpoints. in addition, we aim to further examine the methods from our experiments on other classes of novel words and in other languages. we also plan to add phonetic resources for improving treatment of nonlinear blends. paul cook and suzanne stevenson. 2010. automatically identifying the source words of lexical blends in english. computational linguistics, 36(1):129-149. ryan cotterell and hinrich schütze. 2018. joint semantic synthesis and morphological analysis of the derived word. transactions of the association for computational linguistics, 6:33-48. kollol das and shaona ghosh. 2017. neuramanteau: a neural network ensemble model for lexical blends. in proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), pages 576-583. m. davies. 2008. the corpus of contemporary american english: 450 million words, 1990-present. aliya deri and kevin knight. 2015. how to make a frenemy: multitape fsts for portmanteau generation. in proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 206-210, denver, colorado. association for computational linguistics.