To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models (2024)

Abstract: Polysemy and synonymy are two crucial interrelated facets of lexicalambiguity. While both phenomena are widely documented in lexical resources and have been studied extensively in NLP,leading to dedicated systems, they are often being consideredindependently in practictal problems. While many tasks dealing with polysemy (e.g. Word SenseDisambiguiation or Induction) highlight the role of word’s senses,the study of synonymy is rooted in the study of concepts, i.e. meaningsshared across the lexicon. In this paper, we introduce ConceptInduction, the unsupervised task of learning a soft clustering amongwords that defines a set of concepts directly from data. This taskgeneralizes Word Sense Induction. We propose a bi-levelapproach to Concept Induction that leverages both a locallemma-centric view and a global cross-lexicon view to induceconcepts. We evaluate the obtained clustering on SemCor’s annotateddata and obtain good performance (BCubed F1 above0.60). We find that the local and the global levels are mutuallybeneficial to induce concepts and also senses in our setting. Finally,we create static embeddings representing our induced concepts and usethem on the Word-in-Context task, obtaining competitive performancewith the State-of-the-Art.

## introduction a crucial challenge in understanding natural language comes from the fact that the mapping between word forms and lexical meanings is manyto-many, due to polysemy (i.e., the multiplicity of meanings for a given form) foot_0 and synonymy (i.e., the multiplicity of forms for expressing a given meaning). both polysemy and synonymy have been thoroughly studied in nlp, but mostly as independent problems, giving rise to dedicated systems. thus, word sense disambiguiation (wsd) aims at correctly mapping word occurrences to one of its senses @xcite , while word sense induction (wsi), its unsupervised counterpart, aims at clustering word occurrences into latent senses directly from data @xcite . more recently, researchers have proposed the task of word-in-context (wic), which consists in classifying pairs of word occurrences depending on whether they realize the same sense or not @xcite . all these works take a word centric view, which aims at identifying or characterizing the different senses of a given word, where these senses are bound to a word. another line of work, which takes a broader lexicon-wide perspective, is concerned with identifying synonyms, which are equivalence classes over different words that point to the same concept @xcite , where concepts are semantic entities that are not bound to a word. in wordnet @xcite , concepts are called synsets, defined as sets of synonyms. however, outside of lexical resources, synonymy and polysemy are usually considered as independent problems in the nlp literature. yet, these two views are complementary. in lexicology, they correspond to two perspectives on the word-meaning mapping: semasiology and onomasiology. the former is the word-to-meanings view, where one can observe polysemy by looking at the different meanings a given word has. the latter is the meaning-to-words view, in which one can study synonymy by looking at the inventory of words that speakers use to express the same meaning. in this paper, we propose a new task, called concept induction, that directly aims at learning concepts in an unsupervised manner from raw text. more precisely, this task aims at learning a soft clustering over a target lexicon (i.e., a set of words), in such a way that each cluster corresponds to a (latent) concept. thus, this task both addresses polysemy (since polysemous words should appear in multiple clusters) and synonymy (since synonymous words should appear in the same cluster(s)). inducing concepts can be interesting for many external applications, like building lexical resources for low-resources languages @xcite , and can bring a different perspective in computational studies of meaning, moving the usual wordcentric focus to a more meaning-centric state. our approach to concept induction relies on word occurrences for a target lexicon, represented as word embeddings derived from a contextualized language model (in this case, bert large @xcite ), which are then grouped, using hard clustering algorithms, into concept denoting clusters. while these concept clusters could in principle be obtained directly from word occurrences, we propose a bi-level methodology that leverages both a local, lemma-centric clustering (i.e., operating on only specific word occurrences), and a global, cross-lexicon clustering (i.e., operating on all words occurrences). from this perspective, our approach generalizes, and in fact builds upon classical word sense induction, in that word senses are learned jointly alongside with concepts. we hypothesize that an approach taking both complementary resolutions in account will lead to improved concept induction and word sense induction, i.e. that the two objectives can be mutually beneficial. to validate our approach, we carried out experiments on the semcor dataset, which provides a set of concepts (taking the form of wordnet synsets) related to word occurrences. we found that our bi-level clustering approach accurately learn concepts, achieving f 1 scores above 0.60 on the task of concept induction compared to wordnet's synsets, outperforming competing approaches that use only local and global views. this demonstrates the benefits of our bi-level approach, and its ability to leverage both local and global views when inducing concepts. interestingly, we show that the benefits go both ways: our proposed approach outperforms lemma-centric approaches when evaluated for wsi. finally, we show that concept-aware static embeddings derived from our approach are also competitive with state-of-the-art approaches efficient on the word-in-context task, while using less training data. through the new task of concept induction, we also contribute in a new way to the ongoing debate regarding the ability to align vector representations extracted from contextualized language models to the seman-tic representations posited by (psycho-)linguists. in this vein, we conduct a qualitative evaluation of obtained clusters to ensure they indeed reflect concepts and gather synonyms. the source code we used for experiments is available at https: //github.com/blietard/concept-induction . ## concept induction our main motivation behind concept induction is to present a view of the mapping between words and their meaning(s). 2 this view is systemic, meaning that it should not be defined for individual words neither for individual concepts, but rather acknowledging these as a whole with interactions and relations. this extends beyond the primary objective of wsi, which defines word senses as pertaining to individual words only and does not explore relations between lemmas or concepts. ## methodology in this section we describe the methods we propose and evaluate for concept induction. we learn a clustering ĉw drawing inspiration from the relations between o, s, c and c w . in particular, the overall objective of our methodology consist in finding c (i.e. partition occurrences into concept clusters) to derive c w . section 3.3 highlighted that there are two levels of partitions: a local level (senses) and a global one (concepts). the proposed approaches rely on both levels and the use of a contextualized language model (clm) to gather representations of occurrences influenced by the context. ## experiments in this section, we evaluate the abilities of the proposed methods to induce concepts and compare the proposed bi-level approach to other methods. we investigate the advantages of the bi-level approach not only for the global viewpoint but also in the local setting. ## extrinsic evaluation with concept-aware embeddings in their work, @xcite derive sense-aware static embeddings from their wsi method, training them on the wikipedia dataset and used them for the word-in-context (wic) task. they achieve nearly-sota results on the dataset proposed by @xcite , and report to be outperformed only by methods using external lexical knowledge and resources. we proceed to the same extrinsic evaluation of our work, constructing concept-aware embeddings using concept clusters of concept induction systems (global-only and bi-level agglo). to obtain such embeddings, we average all vectors representating occurrences in semcor contained each global cluster to get one vector per concept cluster. the wic task consists of determining whether two occurrences of a target lemma w correspond to the same sense. the wic dataset's target words are nouns and verbs, but like in the rest of this paper, we restrict our scope to nouns. to solve the task, we use bert large to create representations of the two target occurrences. each of them is assigned to a concept by finding the closest concept-aware using cosine distance. the decision depends on whether the two occurrences are mapped to the same concept (true) or to distinct ones (false). results are displayed in ## conclusion in this paper, we argued that, while word senses allow to investigate polysemy, concepts are a larger perspective that allows the study of polysemy as well as synonymy. we defined concept induction, the unsupervised task to learn a soft-clustering of words in a large lexicon, directly from their incontext occurrences in a corpus. then, we proposed a formulation of this problem in terms of local (lemma-centric) and global (cross-lexicon) complementary views, and tested an approach that uses information from both levels using contextualized language models. on concept-annotated semcor corpus, we found that this bi-level view was beneficial for concept induction, and even for word sense induction with a low amount of training data. we validated the quality of obtained clusters with manual annotations, ensuring that clusters mostly correspond to actual synonyms and concepts. finally, we showcased an external application of our methodology to create concept-aware embeddings that can be competitive to other methods on semantic tasks, such as word-in-context. concept induction opens the way for a different perspective on lexical semantics in nlp, and can be a basis for many studies of lexical meanings as it is expressive enough to reflect relations on both sides of the word-meaning mapping.