Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations (2020)

Abstract: To date, the most successful word, word sense, and concept modelling techniques have used large corpora and knowledge resources to produce dense vector representations that capture semantic similarities in a relatively low-dimensional space. Most current approaches, however, suffer from a monolingual bias, with their strength depending on the amount of data available across languages. In this paper we address this issue and propose Conception, a novel technique for building language-independent vector representations of concepts which places multilinguality at its core while retaining explicit relationships between concepts. Our approach results in high-coverage representations that outperform the state of the art in multilingual and cross-lingual Semantic Word Similarity and Word Sense Disambiguation, proving particularly robust on low-resource languages. Conception – its software and the complete set of representations – is available at https://github.com/SapienzaNLP/conception .

## introduction word vector representations, in particular dense representations or word embeddings @xcite @xcite , play a key role in a wide range of tasks, including text similarity @xcite , word sense disambiguation @xcite , semantic role labeling @xcite @xcite , question answering @xcite and machine translation @xcite . this is especially the case when they are used as the underlying input representation. word embedding techniques map each word to a relatively low n-dimensional space where two semantically or syntactically similar words lie close together. due to their latent nature, however, most embeddings are commonly considered to be uninterpretable @xcite as the properties captured by each dimension are often unclear. more recent studies have shed some light on their interpretability @xcite @xcite or included interpretability directly in the learning process @xcite , but the opaqueness of dense vectors is still a key reason why research has not completely given up on sparse representations @xcite . moreover, most current embedding techniques rely on large corpora which are often available in few languages, such as english or chinese, strongly limiting their robustness on low-resource languages @xcite . in an attempt to solve this issue, researchers turned to multilingual word representations by making use of parallel vocabularies @xcite @xcite , exploiting multilingual knowledge graphs @xcite , and exploring unsupervised methods to align monolingual embeddings in a single shared distributional space @xcite or to directly learn multilingual embeddings @xcite . nevertheless, a well-known pitfall of both monolingual and multilingual word representations is the so-called meaning conflation deficiency problem @xcite : a word may be ambiguous, that is, it may have multiple meanings, but those possibly unrelated meanings cannot be told apart since they are conflated into a single representation. as a result, contextualized word representations have garnered attention @xcite , enjoying great success in the form of pretrained language models like bert @xcite or xlm @xcite . at the same time, modelling techniques for individual word senses, concepts and named entities have also gained traction @xcite , though their integration into downstream nlp applications is still subject of ongoing investigations @xcite . the requirement of massive amounts of training data and the lack of interpretability hinder most of the above-mentioned approaches. to address these limits, we introduce conception, a novel knowledgebased technique for modelling concepts and named entities through concepts and named entities. our approach places multilinguality at its core by leveraging the mutually-reinforcing information coming from different languages, enabling seamless and robust cross-lingual scaling, while also providing explicit and easily interpretable semantic dimensions. in contrast to most word-based embeddings, in conception: i. each component in a vector represents a (weighted) concept or named entity, therefore making our representations fully interpretable; we evaluate conception on multilingual and cross-lingual semantic word similarity, finding that our approach outperforms supervised, unsupervised and knowledge-based state-of-the-art techniques for both sparse and dense vector representations. furthermore, we show that these improvements translate into the downstream task of word sense disambiguation, where conception surpasses the state of the art among supervised and knowledge-based techniques, showing that our semantics-first representations contain meaningful information even when compared against bert-based techniques. contextualized word embeddings. the above-mentioned approaches produce word-level representations that are independent of the specific context a word appears in, and such "static" representations often show a strong bias towards the most frequent sense of a word. instead, context-aware word representation techniques, such as context2vec @xcite or elmo @xcite , dynamically create a representation for a word in a sentential or documental context. contextualized embeddings witnessed a dramatic rise in popularity thanks to the advent and wide availability of language models pretrained on massive amounts of text, such as bert @xcite , immediately followed by multilingual language models, such as m-bert and xlm @xcite . contextualized word embeddings are able to capture the many facets of a polysemous word in a context @xcite , but their implicitly encoded meanings are still disconnected from human-curated knowledge bases, even if more recent efforts showed promising results in imparting structured semantic knowledge to contextualized representations @xcite . knowledge-enhanced representations. alternative approaches implement multilinguality by making use of multilingual encyclopedic resources, such as wikipedia @xcite , or multilingual knowledge graphs such as open multilingual wordnet @xcite , conceptnet @xcite , or babelnet @xcite . a prominent example of knowledge-enhanced word embeddings is conceptnet numberbatch @xcite , which retrofits word2vec @xcite and glove @xcite word representations to the conceptnet graph, achieving state-of-the-art results in semantic word similarity @xcite . a further step towards semantic representations involves modelling individual word senses as vectors which are explicitly linked to a knowledge resource. early approaches to sense embeddings adapted existing work to project words and word senses onto a shared distributional space @xcite , while more recent studies exploited the inner states of pretrained language models @xcite . instead of modelling language-specific units like words or senses, nasari @xcite represents language-independent concepts using sparse lexical vectors. its most notable shortcoming, however, is that each lexical vector is built from a single source language, and therefore each concept has a separate representation depending on the source language of choice. while the nasari lexical vectors provide language-specific representations for language-independent concepts, camacho-collados et al. (2016) also proposed a "unified" variant where the vector dimensions are concepts obtained by semantically clustering the words of the corresponding lexical vector based on the hypernymy relation. however, such representations start from a single language, and therefore do not exploit the multilingual content available in resources such as babelnet, and are not interrelated to each other. with conception, we tackle all these issues and propose an integrated, multilingually-enhanced representation of concepts and entities. ## preliminaries conception relies on the concept inventory of babelnet and the lexical vectors of nasari to build its representations, which we introduce hereafter. 1 babelnet @xcite ) is a multilingual semantic network that brings together heterogeneous resources, such as wikipedia, wordnet, and open multilingual wordnet, with 284 languages supported in the current version 4.0. each node in the babelnet graph represents a concept or named entity and is defined as a multilingual synset, i.e., the set of synonymous lexicalizations used in different languages to express the same concept or named entity. 2 for example, the concept mo-tor vehicle is defined as the multilingual synset containing the terms { car en , motorcar en , coche es , voiture fr , macchina it , . . . , 自動車 jp }. in babelnet, synsets are connected to other synsets through a variety of relations, from hypernymy (generalization or is-a) to hyponymy (specialization or has-kind), from meronymy (part-whole) to antonymy (opposite-of) and general relatedness relations extracted from wikipedia page links, among others. while some different relation types may arguably be considered more important than others, for the sake of simplicity, we do not distinguish between synset relation types. nasari @xcite , as previously mentioned, represents a concept c in the form of a sparse vector v l c whose components are the weights of lexical items (words and multiword expressions) expressed in a given language l. the weight of a lexical item w is computed as its lexical specificity @xcite in the subcorpus of wikipedia articles which define c and its related concepts using language l. lexical specificity is based on the hypergeometric distribution over word frequencies in such corpora and is computed as v l c [w] = spec(w) = -log p (@xmath0; t, t, f, f ), where t and t are the sizes of wikipedia and the subcorpus, respectively, and f and f are the frequencies of w in the two respective corpora. for each language l in wikipedia, nasari can produce a distinct representation v l c of a concept c. however, since vocabularies of different languages are mostly non-overlapping, the components or lexical items of v l c and v l c cannot be directly compared across any two languages l and l . notably, v l c may include knowledge that is missing from v l c and, at the same time, the lexical items of v l c can help disambiguate the lexical items of v l c , as observed by @xcite . ## conception the key innovation we put forward is that, with conception, multilinguality is an integral part of the learning process: instead of deriving concept representations from within a single language, we leverage the mutually-reinforcing information available across languages to create human-readable and languageindependent concept-level representations. our approach results in representations where each concept c is described by a vector where each dimension corresponds to a concept: a larger magnitude for the i-th component denotes a stronger relation between c and the i-th concept c i , that is, each concept is described by the concepts it is most related to. by modelling individual concepts rather than words, conception does not suffer from the conflation of senses that affects word-level representations, such as word2vec and glove. at the same time, since concepts are language-independent units, using them as the dimensions of our representations addresses the language-specificity issue of other sparse representations such as the nasari lexical vectors (see section 3). in the remainder of this section, we describe the four steps of conception (a running example, discussed in what follows, is shown in figure 1 ). ## semantic word similarity we evaluate conception on the semantic word similarity task across 6 languages for a total of 6 multilingual and 10 cross-lingual datasets. semantic word similarity is one of the most popular intrinsic benchmarks for the evaluation of representation techniques. given two lexical items (words, multiword expressions or named entities), the task involves measuring their semantic closeness. ## word sense disambiguation word sense disambiguation (wsd) -the task of assigning the correct meaning to a target word in a context -is considered to be a fundamental step towards natural language understanding @xcite . as with many other tasks, wsd has benefited greatly from the recent advances in other fields, such as language modelling @xcite , game theory @xcite , structured knowledge integration @xcite , definition modelling @xcite and label propagation @xcite , inter alia. our experiments show that conception can be used to create state-of-the-art sense embeddings, demonstrating empirically that our approach provides high-quality knowledge that is still not captured by recent language models. ## (conception lmms and conception sensembert hereafter). for all embeddings, wsd is performed as customary in the literature: given a word w in context, we choose the sense s of w whose vector is closest to the contextual bert representation of w according to cosine similarity. we also include knowbert @xcite , a language model which exploits multiple knowledge bases. datasets. we empirically assess our sense embeddings on the unified evaluation framework for english wsd proposed by @xcite , which comprises senseval-2 @xcite , senseval-3 @xcite @xcite , semeval-2013 @xcite @xcite . results. ## conclusion in this paper we presented conception, a novel knowledge-based technique for modelling concepts and named entities. its key innovation lies in setting multilinguality as the cornerstone of the learning process to build language-agnostic and human-readable concept vector representations. evaluated across multiple multilingual and cross-lingual semantic word similarity datasets, conception shows state-of-the-art results not only compared to concept representations such as nasari, but also to multilingual word embeddings such as conceptnet numberbatch and cross-lingual language models such as xlm. additionally, our concept representations are particularly robust on resource-poor languages, like farsi, along the lines of recent work in semantic parsing and semantic role labeling aimed at bridging the gap between languages @xcite . finally, conception can be seamlessly applied to a downstream task: in word sense disambiguation, it improves over state-of-the-art supervised and knowledge-based sense embeddings, showing that conception encodes information that is still not captured by bert-based contextualized representations. furthermore, our approach produces much more than concept representations: since each concept is described by the relationships it has with other concepts, conception can be seen as a weighted directed graph where each node is a concept whose vector representation is also its weighted adjacency list. this paves the way to a whole set of possible applications for conception: from graph-based concept embeddings to semantics-first sentence and document representations. the complete set of concept vectors is available at https://github.com/sapienzanlp/conception .