Compositionality as an Analogical Process: Introducing ANNE (2022)

Abstract: Usage-based constructionist approaches consider language a structured inventory of constructions, form-meaning pairings of different schematicity and complexity, and claim that the more a linguistic pattern is encountered, the more it becomes accessible to speakers. However, when an expression is unavailable, what processes underlie the interpretation? While traditional answers rely on the principle of compositionality, for which the meaning is built word-by-word and incrementally, usage-based theories argue that novel utterances are created based on previously experienced ones through analogy, mapping an existing structural pattern onto a novel instance. Starting from this theoretical perspective, we propose here a computational implementation of these assumptions. As the principle of compositionality has been used to generate distributional representations of phrases, we propose a neural network simulating the construction of phrasal embedding as an analogical process. Our framework, inspired by word2vec and computer vision techniques, was evaluated on tasks of generalization from existing vectors.

## introduction while the generative tradition has dominated linguistic research for over half a century, the last decades have seen the emergence of an alternative paradigm in linguistics and cognitive sciences, which goes under the name of usage-based models of language @xcite @xcite @xcite , a variety of approaches grounded on the idea that linguistic structures emerge and are shaped through the use of language. their claim is that language is not different from any other cognitive domain: linguistic structures are not the result of a specificlanguage function but are explainable as the implementation of domain-general processes @xcite . the usage-based position shares the fundamental assumption of construction grammar @xcite : language consists of meaningful and symbolic form-meaning mappings, called constructions. words, idiomatic expressions (e.g., kick the bucket -"to die" or @xmath0.g., ditransitive structures) are all constructions varying along a continuum of schematicity and complexity. in particular, usage-based constructionist approaches emphasize the notion of frequency: combinations that are more frequently encountered become more accessible (perhaps because they are stored in memory) and are preferred. indeed, if the language system derives from language use, it follows that how often a speaker encounters a particular linguistic expression will affect the system itself. this assumption implies that any sequence of words -if used frequently enough -can be a construction, even if there are no idiosyncrasies of form and meaning @xcite . however, it is impossible to store any possible word combinations a speaker has or will ever produce. the traditional answer relies on the principle of compositionality: the meaning of a complex expression is entirely determined by its structure and the meanings of its constituents -once we specify what the parts mean and how they are put together, there is no more leeway regarding the meaning of the whole @xcite . usage-based theories favor a different explanation: novel utterances are created based on previously experienced utterances thanks to the cognitive process of analogy. the ability to make analogies -that is, to map familiar relations from one domain of experience to another -is a fundamental ingredient of human intelligence and creativity @xcite . in the linguistic domain, analogy depends on similarity in form and meaning between constructions, whether these constructions are of a concrete type or an abstract type: a novel instance is compared to those stored in our long-term memory to infer the new representation. in this perspective, the acceptability of a novel item is a gradient that depends on the extent of similarity to prior uses of a construction @xcite . in a more radical stance, @xcite proposed disregarding completely abstraction: unwitnessed forms are produced and comprehended "by on the fly analogy" across multiple stored exemplars. without denying the existence of abstract representations, we also assume that analogical mechanisms play a key role in explaining systematic processes of language productivity. this paper aims to articulate the hypotheses introduced above in computational terms. we address two interconnected questions: how can we represent (lexicalized) constructions? is it possible to replicate the interpretation-as-analogy mechanism in computational terms? specifically, we investigate how to model constructions as well as analogybased compositionality using distributional semantic models (dsms). dsms represent the lexicon in terms of vector spaces, where a lexical target is described in terms of a vector (also known as embedding) built by identifying in a corpus its syntactic and lexical contexts @xcite . as a first approximation, we decided to consider constructions any kind of frequent pairs of words linked by a syntactic relation. traditionally, building distributional representations beyond individual words, such as phrases and sentences, is the focus of compositional distributional semantic models. their proposed methodologies try to derive the meaning of an expression from the meanings of the sentence's constituents @xcite : the simplest csdms represent words as vectors and obtain sentence vectors with sum or product operations between constituent vectors @xcite , while more complex models represent predicates with matrices and tensors @xcite @xcite or reproduce the compositionality operation by means of a neural architecture learning so-called sentence embeddings @xcite . it is interesting to notice that most distributional models for phrases/constructions/sentences assume more or less explicitly the principle of compositionality, while the idea that units above the word level could be stored and retrieved via analogy/similarity mechanisms has rarely been explored. 1 the experiment presented here distances itself from these approaches, following a more usagebased perspective. suppose frequently experienced word sequences are, to some extent, stored in memory, and the organization and productivity of language are understood as the result of analogical processes between form and meaning in this structured inventory of constructions. in that case, new phrases could be constructed by analogy with stored linguistic patterns. we propose a neural network model to infer a distributional representation of a new syntactic phrase by preserving the structural information encoded in the embeddings representing previously stored, high-frequency phrases. as the main contributions of the paper, i) we introduce a new dsm in which both lemmas and syntactic relations in the form @xmath1, dependent, syntactic @xmath2; ii) we propose an analogical model to create the distributional embeddings of new relations by applying deep-learning techniques, and evaluate different architectures in terms of generalization and systematicity; iii) we discuss the implications of our analogical model from a theoretical and computational perspective. ## relational embeddings the first step consisted in developing a dsm for lexicalized constructions. we represent the meaning of phrases following a holistic approach (turney, 2012): as a numeric vector can represent nouns like space and race, in the same way, phrases like space race are associated with a unique embedding. for our goal, we built embeddings corresponding to @xmath3, dependent, role>, assuming that these vectors should keep track of the syntactic relation between words. for this reason, we called these relational embeddings (relembs), and we assume they represent the meaning of lexicalized constructions. we built our semantic space using word2vecf @xcite , a modification of the skip-gram model introduced by @xcite . while the original implementation assumes bag-of-words contexts, i.e., the model keeps track of word counts and disregards the grammatical details and the word order, word2vecf allows us to use arbitrary context features. in detail, we @xmath4, @xmath5, parsed using corenlp @xcite : targets are both words @xmath6, dependent, @xmath7.g., <bark, dog, @xmath8, while context is always an open-class word (noun, verb, adjective) occurring with the target in the sentence within a @xmath9. word2vecf parameters are reported in appendix a. we built our dsm considering only words and relations with a frequency equal to or larger than 100 and filtering @xmath10, @xmath11; lastly, we kept @xmath12, dependent, @xmath13,000, where both the head and the dependent lemmas have a @xmath14,000. this strategy is consistent with the idea that holistic representations of complex constructions are stored only for substantially frequent items. the final space contains 127,739 word embeddings and 173,496 relembs, for a total of 301,235 items. ## analogical neural network for embeddings usage-based theories of language assume that systematic processes of language productivity can be explained mainly by analogical inferences rather than by sequential compositional operations. in this perspective, we present a system to expand the coverage of the relembs space simulating the construction of phrasal meaning as an analogical process via deep learning techniques. ## compositionality vs. idiomaticity finally, we present a series of analyses to evaluate the meaning encoded in analogically-generated embeddings. we hypothesize that the best-generated embedding should keep the same relationship among components as the base (systematicity). as a counterproof, we also generated embeddings from idiomatic expressions. in this case, we expect analogies with idiomatic bases to give odd results in the semantic space because of their reduced compositionality and systematicity. the results should answer the following questions: what are the char-acteristics of analogically-generated embeddings? how does the type of input (concatenation or difference) affect the final representation? what loss functions are better at retaining the same structural relation of the base, while at the same time generalizing from the original embedding? data the analogical bases employed are 44 verbal phrases (22 idioms from libben and titone (2008) + 22 compositional manually picked from frequent relations) and 24 nominal compounds (12 idiomatic + 12 compositional) selected from the noun compound senses dataset @xcite and the dataset by @xcite . for each phrase, we manually chose a relation similar to the base but not attested in the vocabulary space, with the same head and syntactic role. for example, given the @xmath15, fish n , com-@xmath16, we replaced the noun fish (i.e., the dependent) with the noun shrimp; expressly, the @xmath17, shrimp n , com-@xmath18. the final dataset consists of 68 analogical pairs, half with an idiomatic base and half with the compositional counterpart. @xmath19 ## general discussion an open issue in dsms is how distributional representations can be projected from the lexical level to the sentence or even discourse level. most previous approaches have tried to solve this issue by explicitly relying on the classic principle of compositionality. given the fregean assumption that phrase meaning is a function of the meanings of its constituents, different computational strategies have been proposed to derive vectors for phrases by taking word embeddings as inputs. in this paper, we proposed a new methodology grounded on a usage-based perspective: we tried to generate new distributional representations by implementing an analogical function in the form of a neural network. word analogies have been used as a standard intrinsic evaluation task for measuring the quality of word @xcite @xcite and sentence embeddings @xcite @xcite . however, the task is usually defined as a candidate retrieval: given an analogical proportion, find the correct completion from a list of candidates to solve the analogy. on the contrary, our aim is to generate a completely new embedding, similarly to what is done in reasoning and computer vision @xcite @xcite : the task consists in training deep learning models to recognize a relationship among two images and generate a transformed query representation (in this case, an image) accordingly. we believe that future investigations in linguistic analogies should pick up from this literature, and anne is a first attempt along this direction. our anne approach is not without limitations. one controversial aspect of anne is the choice of building the target by simply changing the argument in the relation. while it is not too problematic for verbal phrases, it raises questions for adjectivenoun phrases and noun compounds. consider the expressions blue car and fast car. many things can be blue and not be a car, but not everything can be fast (e.g., *fast carrot) because fast constrains the possible realizations of its head. a similar observation could be shown for noun compounds: in some cases, their meaning is related to both components (e.g., bank account), but sometimes their meaning retains aspects of one component (e.g., head teacher). to take into account the specificities of each type of phrases, we could train different anne architectures for each type of phrases. the main difficulty is to balance relational and attributional similarity. indeed, the use of a new item in a construction requires a great deal of relational knowledge @xcite ; nonetheless, the importance of similarity or shared attributes to linguistic analogy is not less vital (bybee, 2010). a qualitative evaluation of analogical inferred embeddings reveals that analogy is easier to compute if the similarity between the entities in the syntactic relations is high. for instance, most all architectures build a good representation of science technology generated from earth science, maybe because there are lots of "topic science" expressions (cf. the introduction of analogy as a strategy to derive meaning for novel expressions does not entail the entire suppression of compositional approaches. from a theoretical stance, not every expression can be built using analogical inference: if analogy fails, compositional operations switch over to guide interpretation. in this regard, the question should not be whether analogically-generated vectors are better than computationally-built ones, but when one mechanism is preferred to the other. answering this question is challenging from both a psycholinguistic and computational stance. the issues related to computational models of analogy as a productive mechanism in language are theoretical before methodological. while it is true that the cognitive process of analogy represents a central mechanism in human cognition @xcite , the problem in defining a linguistic theory that formalizes precisely what an analogy is and when it occurs is complex. in other words, it is hard to predict which analogies will actually be drawn and at what linguistic level @xcite . ideally, future systems aiming at modeling language comprehension should be able to include this mechanism too. new benchmarks will have to be built with the aim of identifying analogical inferences. these datasets could also be valuable for behavioral analyses. ## conclusion and future works in this paper, we presented a new approach that simulates the construction of phrasal meaning as an analogical process implemented with deep learning techniques. we proposed a distributional representation of constructional phrases and a model of generating new embeddings analogically rather than applying traditional compositional operations. we experimented with our analogical neural network to understand how it can generalize and be extendable to different scenarios. we argued that the proposed methodology could open the doors to new analyses in distributional semantics as well as in computational models of language processing. the future research perspectives on anne are considerable. firstly, we could build a more sophisticated phrasal representation using contextualized embeddings @xcite based on transformers @xcite . moreover, we should compare our relembs with other phrasal representations representations @xcite and relation embeddings @xcite . moreover, while we performed analogy over preselected base-target pairs, we aim at investigating methods to automatically retrieve the best analogical candidate. finally, we plan to evaluate anne's ability to model human behavior on more complex tasks regarding compositionality and language productivity.