Predicting emergent linguistic compositions through time: Syntactic frame extension via multimodal chaining (2021)

Abstract: Natural language relies on a finite lexicon to express an unbounded set of emerging ideas. One result of this tension is the formation of new compositions, such that existing linguistic units can be combined with emerging items into novel expressions. We develop a framework that exploits the cognitive mechanisms of chaining and multimodal knowledge to predict emergent compositional expressions through time. We present the syntactic frame extension model (SFEM) that draws on the theory of chaining and knowledge from “percept”, “concept”, and “language” to infer how verbs extend their frames to form new compositions with existing and novel nouns. We evaluate SFEM rigorously on the 1) modalities of knowledge and 2) categorization models of chaining, in a syntactically parsed English corpus over the past 150 years. We show that multimodal SFEM predicts newly emerged verb syntax and arguments substantially better than competing models using purely linguistic or unimodal knowledge. We find support for an exemplar view of chaining as opposed to a prototype view and reveal how the joint approach of multimodal chaining may be fundamental to the creation of literal and figurative language uses including metaphor and metonymy.

## introduction language users often construct novel compositions through time, such that existing linguistic units can be combined with emerging items to form novel expressions. consider the expression swipe your phone, which presumably came about after the emergence of touchscreen-enabled smartphones. here the use of the verb swipe was extended to express one's experience with the emerging item "smartphone". these incremental extensions are fundamental to adapting a finite lexicon toward emerging communicative needs. we explore the nature of cognitive mechanisms and knowledge in the temporal formation of previously unattested verb-argument compositions, and how this compositionality may be understood in principled terms. compositionality is at the heart of linguistic creativity yet a notoriously challenging topic in computational linguistics and natural language processing (e.g., @xcite @xcite @xcite . for instance, modern views on the state-of-the-art neural models of language have suggested that they show some degree of linguistic generalization but are impoverished in systematic compositionality (see @xcite for review). existing work has also explored the efficacy of neural models in modeling diachronic semantics (e.g., @xcite @xcite . however, to our knowledge, no attempt has been made to examine principles in the formation of novel verb-noun compositions through time. we formulate the problem as an inferential process which we call syntactic frame extension. we define syntactic frame as a joint distribution over a verb predicate, its noun arguments, and their syntactic relations, and we focus on tackling two related predictive problems: 1) given a novel or existing noun, infer what verbs and syntactic relations that have not predicated the noun might emerge to describe it over time (e.g., to drive a car vs. to fly a car), and 2) given a verb predicate and a syntactic relation, infer what nouns can be plausibly introduced as its novel arguments in the future (e.g., drive a car vs. drive a computer). figure 1 offers a preview of our framework by visualizing the process of assigning novel verb frames to describe two query nouns over time. in the first case, the model incorporated with perceptual and conceptual knowledge successfully predicts the verb drive to be a better predicate than fly for describing the novel item car that just emerged at the time of prediction where linguistic usages are not yet observed (i.e., emergent verb compogiven a query noun (circled dot) at time t, the framework draws on a combination of multimodal knowledge and cognitive mechanisms of chaining to predict novel linguistic expressions for those items. the left panel shows via pca projection how a newly emerged noun (i.e., car in 1920s) is assigned appropriate verb frames by comparing the learned multimodal representation of the query nouns with support nouns (non-circled dots) that have been predicated by the frames. the right panel shows a similar verb construction for a noun that already existed at the time of prediction (i.e., computer in 1980s). our approach connects two strands of research that were rarely in contact: cognitive linguistic theories of chaining and computational representations of multimodal semantics. work in the cognitive linguistics tradition has suggested that frame extension is not arbitrary and involves the comparison between a new item to existing items that are relevant to the frame @xcite . similar proposals lead to the theory of chaining postulating that linguistic categories grow by linking novel referents to existing ones of a word due to proximity in semantic space @xcite @xcite @xcite . however, such a theory has neither been formalized nor evaluated to predicting verb frame extensions through time. separately, computational work in multimodal semantics has suggested how word meanings warrant a richer representation beyond purely linguistic knowledge (e.g., @xcite @xcite . however, multimodal semantic representations have neither been examined in the diachronics of compositionality nor in light of the cognitive theories of chaining. we show that a unified framework that incorporates the cognitive mechanisms of chaining through deep models of categorization and multimodal semantic representations predicts the temporal emergence of novel noun-verb compositions. ## related work our work synthesizes the interdisciplinary areas of cognitive linguistics, diachronic semantics, meaning representation, and deep learning. ## computational framework we present the syntactic frame extension model (sfem), which is composed of two components. first, sfem specifies a frame as a joint probabilistic distribution over a verb, its noun arguments, and their syntactic relations and supports temporal prediction of verb syntax and arguments via deep probabilistic models of categorization. second, sfem draws on multimodal knowledge by incorporating perceptual, conceptual, and linguistic cues into flexible inference for extended verb frames over time. figure 2 illustrates our framework. ## historical noun-verb compositions to evaluate our framework, we collected a large dataset of historical noun-verb compositions derived from the google syntactic n-grams (gsn) english corpus @xcite from 1850 to 2000. specifically, we collected verb-noun-relation triples (n, v, r) (t) that co-occur in the engall subcorpus of gsn over the 150 years. we focused on working with common usages and pruned rare cases under the following criteria: 1) a noun n 1 for imagenet embeddings, we apply a linear transformation to project each x (t) p into r 300 so that all unimodal representations are 300-d vectors before taking the means. ## evaluation and results we first describe the details of sfem implementation and diachronic evaluation. we then provide an in-depth analysis on the multimodal knowledge and chaining mechanisms in verb frame extension. ## conclusion we have presented a probabilistic framework for characterizing the process of syntactic frame extension in which verbs extend their referential range toward novel and existing nouns over time. our results suggest that language users rely on extralinguistic knowledge from percept and concept to construct new linguistic compositions via a process of exemplar-based chaining. our work creates a novel approach to diachronic compositionality and strengthens the link between multimodal semantics and cognitive linguistic theories of categorization.