Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis (2023)

Abstract: We propose using automatically generated natural language definitions of contextualised word usages as interpretable word and word sense representations. Given a collection of usage examples for a target word, and the corresponding data-driven usage clusters (i.e., word senses), a definition is generated for each usage with a specialised Flan-T5 language model, and the most prototypical definition in a usage cluster is chosen as the sense label. We demonstrate how the resulting sense labels can make existing approaches to semantic change analysis more interpretable, and how they can allow users — historical linguists, lexicographers, or social scientists — to explore and intuitively explain diachronic trajectories of word meaning. Semantic change analysis is only one of many possible applications of the ‘definitions as representations’ paradigm. Beyond being human-readable, contextualised definitions also outperform token or usage sentence embeddings in word-in-context semantic similarity judgements, making them a new promising type of lexical representation for NLP.

## introduction accurate semantic understanding in language technologies is typically powered by distributional word representations and pre-trained language models (lms). due to their subsymbolic nature, however, such methods lack in explainability and interpretability, leading to insufficient trust in end users. an example application which requires capturing word meaning with its nuanced contextdetermined modulations is lexical semantic change analysis, a task which consists in detecting whether a word's meaning has changed over time, for example by acquiring or losing a sense. modern semantic change detection systems rely on static and contextualised word representations, lmbased lexical replacement, grammatical profiles, supervised word sense and word-in-context disambiguation @xcite . but the main potential end users of these technologies-historical linguists, lexicographers, and social scientists-are still somewhat reluctant to adopt them precisely because of their lack of explanatory power. lexicographers, for instance, are not satisfied with detecting that a word has or hasn't changed its meaning over the last ten years; they want descriptions of old and new senses in humanreadable form, possibly accompanied by additional layers of explanation, e.g., specifying the type of semantic change (such as broadening, narrowing, and metaphorisation) the word has undergone. our work is an attempt to bridge the gap between computational tools for semantic understanding and their users. we propose to replace blackbox contextualised token embeddings produced by large lms with a new type of interpretable lexical semantic representation: automatically generated contextualised word definitions @xcite . in this paradigm, the usage of the word 'apple' in the sentence 'she tasted a fresh green apple' is represented not with a dense highdimensional vector but with the context-dependent natural language definition 'edible fruit'. with an extended case study on lexical semantic change analysis, we show that moving to the more abstract meaning space of definitions allows practitioners to obtain explainable predictions from computational systems, while leading to superior performance on semantic change benchmarks compared to state-ofthe-art token-based approaches. this paper makes the following contributions. foot_0 1. we show that word definitions automatically generated with a specialised language model, fine-tuned for this purpose, can serve as interpretable representations for polysemous words ( §5). pairwise usage similarities between contextualised definitions approximate human semantic similarity judgements better ## definition generation our formulation of the definition generation task is as follows: given a target word w and an example usage s (i.e., a sentence containing an occurrence of w), generate a natural language definition d that is grammatical, fluent, and faithful to the meaning of the target word w as used in the example usage s. a definition generator is a language process that maps words and example usages to natural language definitions. as a generator, we use flan-t5 @xcite , a version of the t5 encoder-decoder transformer @xcite fine-tuned on 1.8k tasks phrased as instructions and collected from almost 500 nlp datasets. flan-t5 is not trained specifically on definition generation but thanks to its massive multi-task instruction fine-tuning, the model exhibits strong generalisation to unseen tasks. therefore, we expect it to produce high-quality definitions. we extensively test three variants of flan-t5 of different size and compare them to vanilla t5 models ( to obtain definitions from flan-t5, we use natural language prompts consisting of an example usage preceded or followed by a question or instruction. for example: 's what is the definition of w?' the concatenated usage example and prompt are provided as input to flan-t5, which conditionally generates definitions (ta-ble 1 shows an example instance). 5 we choose greedy search with target word filtering as a simple, parameter-free decoding strategy. stochastic decoding algorithms can be investigated in future work. ## definitions are interpretable word representations we propose considering the abstract meaning space of definitions as a representational space for lexical meaning. definitions fulfil important general desiderata for word representations: they are human-interpretable and they can be used for quantitative comparisons between word usages (i.e., by judging the distance between pairs of definition strings). we put the definition space to test by applying it to the task of semantic change analysis, which requires capturing word meaning at a finegrained level, distinguishing word senses based on usage contexts. we use our fine-tuned flan-t5 models (xl and other sizes) to generate definitions for all usages of the 46 target words annotated in the english dwugs (ca. 200 usages per word; see §3.2). foot_8 these definitions (an example is provided in ## labelling word senses with definitions for generated definitions to be useful in practice, they need to be able to distinguish word senses. for example (ignoring diachronic differences and singleton clusters), there are three main senses of the word 'word' in its dwug, which we manually label as: ( 1 ) 'words of language', (2) 'a rumour', and (3) 'an oath'. manual inspection of the generated definitions indicates that they are indeed sense-aware: 1. 'a communication, a message', 'the text of a book, play, movie', etc. 2. 'information passed on, usually by one person to another', 'communication by spoken or written communication', etc. 3. 'an oath', 'a pronouncement', etc. but let's again put ourselves in the shoes of a historical linguist. sense clusters are now impractically represented with multitudes of contextualised definitions. cluster (1) for 'word', e.g., features 190 usages, and one must read through all of them (otherwise there will be a chance of missing something) and generalise -all to formulate a definition that covers the whole sense cluster (a sense label). we now show how dwugs can be automatically augmented with generated sense labels, vastly improving their usability. selecting sense labels from n definitions, generated for n word usages belonging to the same dwug cluster, we use the most prototypical one as the sense label-with the aim of reflecting the meaning of the majority of usages in the cluster. we represent all definitions with their sentence embeddings (cf. §5.1) and select as prototypical the definition whose embedding is most similar to the average of all embeddings in the cluster. clusters with less than 3 usages are ignored as, for these, prototypicality is ill-defined. as a sanity check, these are the sense labels obtained by this method for the dwug clusters of 'word'; they correspond well to the sense descriptions provided earlier. we compare these sense labels to labels obtained by generating a definition for the most prototypical usage (as judged by its token embedding), rather than taking the most prototypical definition, and we evaluate both types of senses labels using human judgements. examples of labels can be found in appendix d. ## explaining semantic change with sense labels word senses in dwugs are collections of example usages and they are only labelled with numerical identifiers. this does not allow users to easily grasp the meaning trajectories of the words they are interested in studying. using sense labels extracted from generated definitions, we can produce a fully human-readable sense dynamics map-i.e., an automatically annotated version of a dwug which displays synchronic and diachronic relations between senses (e.g, senses transitioning one into another, splitting from another sense, or two senses merging into one). one can look at sense dynamics maps as reproducing the work of @xcite on the modern technological level and, importantly, with human-readable sense definitions. given a target word, its original dwug, and its semi-automatic sense clusters, we start by assigning a definition label to each cluster, as described in §6. then, we divide each cluster into two subclusters, corresponding to time periods 1 and 2 (for example, sub-cluster c 2 1 contains all usages from cluster 1 occurring in time period 2). 13 we compute pairwise cosine similarities between the sentence embeddings of the labels (their 'definition embeddings'), thereby producing a fully connected graph where nodes are sub-clusters and edges are weighted with sense label similarities. most edges have very low weight, but some sub-cluster pairs are unusually similar, hinting at a possible relation between the corresponding senses. we detect these outlier pairs by inspecting the distribution of pairwise similarities for values with z-score higher than 1 (similarities more than 1 standard deviation away from the mean similarity). sub-cluster pairs connected with such edges form a sense dynamics map. as an example, the noun 'record' has only one sense in time period 1 but it acquires two new senses in time period 2 (figure 3 ; as before, we ignore clusters with less than 3 usages). the sense clusters defined by the dwug are anonymous collection of usages, but with the assigned sense labels (also shown in figure 3 ) they can be turned into a proto-explanation of the observed semantic shift: it becomes now clear that sense 2 stems from the older general sense 0 of 'record'-arguably representing a case of narrowing @xcite )while the second new sense (1: 'the highest score or other achievement in the game') an interesting case is 'ball' (see appendix f for another example regarding the word 'chef '). clusters c 1 1 and c 2 1 have the same label. this is done for simplicity and because of data scarcity, but in the future we plan to experiment with time-dependent labels as well. we use two time periods as only two periods are available in @xcite , but the same procedure can be executed on multi-period datasets. although none of its sense labels are identical, its sense cluster c 0 is very close to cluster c 2 (similarity of 0.70), while c 2 is close to c 3 (similarity of 0.53); all three senses persist throughout both time periods, with sense 3 declining in frequency. the generated definitions for the 'ball' clusters are: 0: 'a sphere or other object used as the object of a hit' (the largest cluster), 2: 'a round solid projectile, such as is used in shooting', and 3: 'a bullet'. this case demonstrates that similarity relations are not transitive: the similarity between c 0 and c 3 is only 0.50, below our outlier threshold value. this is in part caused by inconsistent dwug clustering: while the majority of usages in c 1 2 are about firearm projectiles, c 2 2 contains mentions of golf balls and ball point pens. this shifts sense 2 from 'bullet' to 'round solid projectile', making it closer to sense 0 (general spheres) than it should be. ideally, all the 'bullet' usages from c 2 should have ended up in c 3 , with the rest joining the general sense 0. besides suggesting fixes to the dwug clustering, the observed non-transitivity also describes a potential (not necessarily diachronic) meaning trajectory of 'ball': from any spherical object, to spherical objects used as projectiles, and then to any projectiles (like bullets), independent of their form. our generated sense labels and their similarities help users analyse this phenomenon in a considerably faster and easier way than by manually inspecting all examples for these senses. ## conclusion and future work in this paper, we propose to consider automatically generated contextualised word definitions as a type of lexical representation, similar to traditional word embeddings. while generated definitions have been already shown to be effective for word sense disambiguation @xcite , our study puts this into a broader perspective and demonstrates that modern language models like flan-t5 @xcite are sufficiently mature to produce robust and accurate definitions in a simple prompting setup. the generated definitions outperform traditional token embeddings in word-in-context similarity judgements while being naturally interpretable. we apply definition-based lexical representations to semantic change analysis and show that our approach can be used to trace word sense dynamics over time. operating in the space of humanreadable definitions makes such analyses much more interesting and actionable for linguists and lexicographers-who look for explanations, not numbers. at the same time, we believe the 'definitions as representations' paradigm can also be used for other nlp tasks in the area of lexical semantics, such as word sense induction, idiom detection, and metaphor interpretation. our experiments with diachronic sense modelling are still preliminary and mostly qualitative. it is important to evaluate systematically how well our predictions correspond to the judgements of (expert) humans. once further evidence is gathered, other promising applications include tracing cases of semantic narrowing or widening over time @xcite by analysing the variability of contextualised definitions in different time periods and by making cluster labels time-dependent. both directions will require extensive human annotation, and we leave them for future work.