Lexical Semantics with Large Language Models: A Case Study of E nglish “break” (2023)

Abstract: Large neural language models (LLMs) can be powerful tools for research in lexical semantics. We illustrate this potential using the English verb “break”, which has numerous senses and appears in a wide range of syntactic frames. We show that LLMs capture known sense distinctions and can be used to identify informative new sense combinations for further analysis. More generally, we argue that LLMs are aligned with lexical semantic theories in providing high-dimensional, contextually modulated representations, but LLMs’ lack of discrete features and dependence on usage-based data offer a genuinely new perspective on traditional problems in lexical semantics.

## feature-based theories in this section, we take the somewhat unusual step of bringing together existing ideas from the linguistics literature into a feature space of the sort one is likely to encounter in nlp contexts. we do this for a few reasons. first, it reveals that, though theories in linguistics and nlp often take very different forms, there is actually a lot of common ground between them: on both sides, vector representations of data can serve as a common language. second, the feature space reveals how deeply linguistic theories are committed to our contextual modulation tenet from the agent feature captures whether the subject of each example is agentive or not. we mentioned in section 2 that the obligatory transitivity of some break frames has been traced, unsuccessfully in our view, to the agentivity of the subject of these frames. the agent feature in combination with the transitive feature and the meaning dimensions reveals the incompleteness of this explanation: 'violate' examples, which are obligatorily transitive, may have subjects that are agentive (row 3) and non-agentive (row 4). we have a column for metaphorical, though coding this is sufficiently hard that it looks like a multidimensional category to us rather than a single feature. due to the difficulty of classifying senses of break and other polysemous verbs as (non)metaphorical, previous literature that has engaged with this question (e.g. @xcite @xcite has used the heuristic of associating metaphorical senses with abstract participants, like break the silence, and non-metaphorical ones with concrete participants, such as break the vase. we follow this (admittedly simplifying) heuristic in our featurebased analysis. however, we agree with @xcite that "the distinction between 'literal' and 'figurative' senses can become blurred over time, and sometimes can only be diachronically reconstructed". following these features are a few meaning dimensions. the full class of meaning annotations we use in section 5.3 has 72 classes, so this is just a sample. the sample was chosen to emphasize three aspects of the meanings of break. first, as already illustrated in break with particles/predicates can sometimes express meanings that particle-less break cannot convey: e.g. 'escape' in row 9. we assume that the particles/predicates contribute an irreducible meaning and reflect this in our feature-based analysis by preceding these meaning dimensions with the corresponding particles/predicates: 'out_escape'. the particles/predicates do not determine a unique meaning, though, as we see with the two senses of break out: 'out_escape' and 'out_begin'. and we could of course have extended this even further. there are additional clearly distinct senses like his face broke out in hives and break out the champagne, as well as cases like break out in laughter which might be subsumed under other senses (say, 'out_begin'). potentially all of the columns are actually just informal stand-ins for much more complex concepts. the labels could be natural language predicates on par with break, in which case the column names are really just hooks into a larger lexical web, or they could be glosses for more intricate theoretical concepts that demand further decomposition before the theory can be regarded as complete. in addition, we can seamlessly integrate this kind of analysis with more data-driven techniques. as an illustration, appendix a reports on an experiment using the wordnet hypernym graph to identify extremely abstract latent meaning dimensions. how many lexical items does this theory posit? the answer to this question is not clear. we could say that each attested combination of the features is a new sense, or we could select a few features and say that specific combinations of them correspond to distinct senses. both decisions have a certain arbitrariness to them given the feature space itself, and we might infer from this that the theory does not posit distinct senses or distinct lexical items as first-class linguistic constructs. this may be a consequence of the contextual modulation tenet. relatedly, it is unclear to us what a complete analysis in these terms would look like. what would it mean to have determined all and only the correct features? could it be that the investigation will always admit of further dimensions, or decomposition of existing dimensions? in sum, it is easy to see how this analysis makes good on the central tenets in ## static vector modeling the above feature-based analysis might be described as a sparse vector representation approach. we now contrast that with a dense vector representation approach that models individual lexical items as fixed (static) vectors. a variety of such methods have been developed. here we look at the treatment of break by three prominent methods: word2vec @xcite , glove @xcite , and fasttext @xcite . these methods have different learning objectives, but all are closely related to pointwise mutual information (pmi; @xcite . in pmi, we assign weights to pairs of words w i and w j based on whether their observed joint probability of co-occurrence is larger or smaller than what we would expect given the null hypothesis that w i and w j have independent distributions. all three methods learn regularized, reduced dimensional representations according to roughly this same goal @xcite . none of these methods use discrete features, and thus they are in conflict with our discreteness tenet from the glove vocabulary is largely restricted to individual words from a fixed list. by contrast, the vocabulary used for our word2vec instance includes some phrase-like elements that were inferred by the authors using simple co-occurrence statistics, and our chosen fasttext model includes sub-word components and so also ends up with a more expansive view of what counts as a lexical item. all of these models have proven successful as representations of words and as components in larger systems. however, we find that these representations are disappointing for studying break. in the cause for this semantic blandness likely traces to the basic design decision: every wordform has only a single representation. this means that a single vector must encode all the different senses that we see in adherents to our central tenets ( ## llm investigations we come now to our primary investigative tool: llms. we concentrate on models that have the core structure of the transformer @xcite and are trained at least in part using masked language modeling, which allows for bidirectional context. in the interest of space, we will mostly presuppose familiarity with these models. however, appendix c provides an overview of their structure to try to bridge any gaps between the linguistics and nlp literature. in our main text, we report results for robertalarge @xcite , which has 24 layers. our appendices cover bert and deberta. our roberta-large results are slightly better than all of these others, but the results are generally quite comparable, suggesting that all of these models can fruitfully be used for lexical semantics. before turning to our experiments, let's consider how llms relate to our core tenets from ## discussion the fact that linguistic theory and llms agree on the core tenets of high dimensionality and contextual modulation is a striking alignment of theoretical ideas with engineering success. the fact that llms do not use discrete features, but rather derive dense, real-valued representations from data seems like an opportunity for linguists to reflect on the role of discreteness. as we noted in section 3, it seems unlikely that purely analytic work and traditional corpus work will lead to an exhaustive hand-built representation for any lexical items. with llms, we can mine the existing representations while considering the llm architecture and learned parameters to be a reflection of the core tenets of the theory. the deep contextual modulation countenanced by linguistic theory and operationalized by llm embeddings invites a further question: do lexical items exist outside of their tokens of use? even for the hand-built feature representations in overall, then, a theory of lexical semantics that draws heavily on llms as investigative tools, and even as ways to state theoretical ideas, is likely to become more usage-based than traditional theories would assume. this could lead them to focus less on pure representation and more on what is actually communicated between people when they use language. traditional questions are likely to take on new forms in this setting, and exciting and relevant new questions -and new pieces of evidence -are likely to arise. ## limitations our general thesis is that llms are valuable tools both for conducting lexical semantic analyses and for providing valuable perspectives for lexical semantic theory design in general. although we think this thesis is widely supported by prior literature, our own case study is limited to just a partial analysis of a single verb. this creates the risk that our general conclusions may be more specific to this verb, or to english, than we would like. the prior literature inherits many of our english-only biases as well (but see @xcite . our main results use roberta-large, and our appendices report on parallel analyses with different versions of bert and deberta. these models share core architectural features and were optimized in largely similar ways using very large -and largely uncontrolled -datasets. this means that these artifacts are certainly biased in ways that are relevant for lexical semantics. however, we are unlikely to be able to identify, isolate, and factor out these biases with the methods used in our paper. our core methods are reasonably simple and lightweight, and we have released all our code. we hope that these steps allow easy reproduction of our core analyses whenever newer llms are released, so that we can begin to understand better how llm biases can affect linguistic theorizing in the mode we are advocating for. our current approach also has an analytic limitation: though we fit probe models and use them as devices for finding potentially relevant examples, the final step in our analysis involves inspection of those examples by linguists like ourselves. this means that the final step is not as reproducible as the others, and it means that any analytic biases that the linguists involved might have are likely to make their way into the analyses. we do not see a way to avoid these analytic steps entirely, since linguistic analysis favors this kind of low-level work, but we do think that we can mitigate the concerns about analyst bias by making all our data available for others to inspect, as a way of opening up many perspectives on the data and the associated theoretical questions.