A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces (2023)

Abstract: We study semantic construal in grammatical constructions using large language models. First, we project contextual word embeddings into three interpretable semantic spaces, each defined by a different set of psycholinguistic feature norms. We validate these interpretable spaces and then use them to automatically derive semantic characterizations of lexical items in two grammatical constructions: nouns in subject or object position within the same sentence, and the AANN construction (e.g., ‘a beautiful three days’). We show that a word in subject position is interpreted as more agentive than the very same word in object position, and that the nouns in the AANN construction are interpreted as more measurement-like than when in the canonical alternation. Our method can probe the distributional meaning of syntactic constructions at a templatic level, abstracted away from specific lexemes.

## introduction there are now several paradigms for the linguistically oriented exploration of large neural language models. major paradigms include treating the model as a linguistic test subject by measuring model output on test sentences (e.g., @xcite @xcite and building (often lightweight) probing classifiers on top of embeddings, to test whether the embeddings are sensitive to certain properties like dependency structure @xcite @xcite @xcite . 1 here, we consider another approach: projecting contextual, token-level embeddings into interpretable feature spaces defined by psycholinguistic feature norms @xcite , 2019; @xcite . by learning a mapping to these spaces, as illustrated in figure 1 , we attain context-sensitive, interpretable, real-valued lexical-semantic features. after experimenting to determine best practices for contextual-feature projection, we use these features to explore whether contextual embeddings are sensitive to subtle semantic construals in different grammatical constructions. specifically, we observe how even seemingly similar constructions can impart a different semantics on their component parts or 'slot fillers' @xcite . consider the article + adjective + numeral + noun (aann) construction: e.g., "a beautiful three days in london," where the normally singular "a" precedes a plural noun and the adjective precedes the numeral @xcite @xcite . this construction often occurs with units or measure phrases (e.g., days, feet), but can also occur with non-measure nouns (e.g., "a lucky three students"). while it is tempting to think of "a lucky three students" as semantically equivalent to "three lucky students," it has a different construal. specifically, the aann construction is acceptable only when the noun behaves as a single collective unit and is, in effect, more semantically similar to a unit of measurement than it would be in the unmarked construction. evidence for a difference in meaning between the two variants is seen in their divergent distributions. for example, the aann construction is unavailable in contexts like (1) and (2) (#-ed cases; adapted from @xcite . the aann construction cannot occur in contexts where the referent of the noun is split into noncontiguous parts. this distributional pattern is taken as evidence that the aann construction construes its argument as a single, measure-like unit. in this paper, we study distributional evidence on a larger scale, using a contextualized large language model as a 'compressed corpus' that captures observed statistical regularities over utterances of many speakers. we analyze this compressed corpus by mapping embeddings to interpretable feature spaces based on psycholinguistic feature norms. when we do this for the embedding of the noun days in "i spent a beautiful three days in london," we find the most salient difference with the "i spent three beautiful days in london" to be a higher value for features like measure and unit when it is in an aann construction. we argue that this is because human speakers construe the aann construction as being "measure-ish", and that this construal is reflected in their language use in a way that the contextual language model can pick up. we conduct two case studies, one about aanns and the other about grammatical subjecthood. specifically, we show that a word in subject position is interpreted as more agentive than the very same word in object position (consistent with findings from psycholinguistics, e.g., @xcite , and that a noun in the aann construction is interpreted as more measurement-like than when in the canonical alternation. our results demonstrate that construals can be inferred from statistical usage patterns. while we here use constructions with known construals, our positive results indicate that we may be able to analyze constructions where the construal is less clear in the theoretical literature. while feature norms have been used to interpret distributional semantic models @xcite @xcite , we emphasize the linguistic value of reliable, reusable, interpretable semantic spaces, which we use to interrogate the semantic properties of language in use. the ability of our method to characterize subtle semantic differences using language models offers a point of connection between linguistically oriented deep neural network analysis @xcite and topics in formal linguistics. in particular, this work empirically demonstrates the potential alignment between lms and feature-based theories of lexical semantics (as illustrated by @xcite . our main goal is to use interpretable feature spaces for understanding the semantic construal of words in context, specifically the aann construction and the transitive construction. in section 2, we lay out our method for constructing interpretable feature spaces for tokens in context. then, in section 3, we evaluate the success of our method on a sense differentiation task, a homonym feature prediction task, and a qualitative analysis. the idea is that, if the method for mapping from embedding space to context-sensitive feature space is successful, we will predict unique semantic features for different senses. having established and validated our method, we then turn to our key constructions in section 4. ## methods the task is to learn a mapping from contextual word embedding space to an interpretable space defined by feature norms (section 2.1), where every dimension corresponds to a semantic feature. we construct the training data by pairing feature norms with embeddings derived from contextual word vectors. we train models at the type-level, e.g., to map the embedding vectors for the word ring to the set of feature norms for ring, as shown in the top half of figure 1 . but ultimately, we use the model to predict semantic features for individual tokens. that is, we project the token vector of a single occurrence of the word "ring" into the feature space learned at the type-level, as shown in the bottom half of figure 1 . ## evaluating contextual feature norms for interpreting semantic space we first evaluated the models on their ability to fit the type-level feature norms they are trained on. we do not go into detail here, as it is contextdependent meanings we are most interested in. see appendix d for full results. overall, bert-derived models were comparable to those we trained with static glove @xcite embeddings, and to the best static models in the literature. this initial evaluation established that models using bert-derived embeddings are just as good as static mcrae buchanan binder mil vanilla mil vanilla mil vanilla plsr .41 .39 .41 .42 .28 .26 ffnn .36 .36 .42 .40 .30 .30 prop -.03 -.03 .10 .10 -.03 -.03 to evaluate our models on in-context feature prediction, we conduct two quantitative experiments: one on a sense differentiation task, one on a homonym disambiguation task, as well as a qualitative analysis for a representative word (fire). the goal of this section is to explore whether the contextual feature norm method successfully captures contextual modulation of word meaning. for these experiments, we select the hyperparameters for each model that performed the best at type-level feature prediction under 10-fold cross-validation (appendix d). ## evaluating constructions in context having validated that our method works for extracting meaningful, context-dependent semantic information from large language models, we turn to two target constructions: the aann construction (described in the introduction) and the basic transitive construction. crucially, in both studies, the word types are largely controlled between conditions (e.g., comparing "the family spent a beautiful three days in london." vs. "the family spent three beautiful days in london."), and so we compare context-dependent features derived from minimally different sentences. this design lets us study the effect of context in a highly controlled way, without being influenced just by the identity of the words in the sentences. ## conclusion in this paper, we honed techniques for predicting semantic features for token embeddings. these projections are versatile. once created, one and the same model can be used to study a wide array of phenomena. we explored their utility for studying semantic construal in syntactic constructions. we emphasize the potential of this method to answer linguistic questions about meaning differences in constructions that are less well-understood and well-theorized than the ones studied here. as such, we hope it will be possible to use this method to generate linguistic insight.