Enhancing Word Embeddings with Knowledge Extracted from Lexical Resources (2020)

Abstract: In this work, we present an effective method for semantic specialization of word vector representations. To this end, we use traditional word embeddings and apply specialization methods to better capture semantic relations between words. In our approach, we leverage external knowledge from rich lexical resources such as BabelNet. We also show that our proposed post-specialization method based on an adversarial neural network with the Wasserstein distance allows to gain improvements over state-of-the-art methods on two tasks: word similarity and dialog state tracking.

## introduction vector representations of words (embeddings) have become the cornerstone of modern natural language processing (nlp), as learning word vectors and utilizing them as features in downstream nlp tasks is the de facto standard. word embeddings @xcite are typically trained in an unsupervised way on large monolingual corpora. whilst such word representations are able to capture some syntactic as well as semantic information, their ability to map relations (e.g. synonymy, antonymy) between words is limited. to alleviate this deficiency, a set of refinement post-processing methods-called retrofitting or semantic specialization-has been introduced. in the next section, we discuss the intricacies of these methods in more detail. to summarize, our contributions in this work are as follows: ## related work numerous methods have been introduced for incorporating structured linguistic knowledge from external resources to word embeddings. fundamentally, there exist three categories of semantic specialization approaches: (a) joint methods which incorporate lexical information during the training of distributional word vectors; (b) specialization methods also referred to as retrofitting methods which use post-processing techniques to inject semantic information from external lexical resources into pre-trained word vector representations; and (c) post-specialization methods which use linguistic constraints to learn a general mapping function allowing to specialize the entire distributional vector space. in general, joint methods perform worse than the other two methods, and are not model-agnostic, as they are tightly coupled to the distributional word vector models (e.g. word2vec, glove). therefore, in this work we concentrate on the specialization and post-specialization methods. approaches which fall in the former category can be considered local specialization methods, where the most prominent examples are: retrofitting @xcite which is a post-processing method to enrich word embeddings with knowledge from semantic lexical resources (wordnet, babelnet, etc.) linguistic constraints synonyms: travel -tour … antonyms: travel -stay-at-home … tour distributional vector space travel stay-at-home isolation initial specialization (attract-repel) post-specialization (dffn, auxgan, wgan-postspec) . . . . . . . . . . . . . . . . . . lexicons, in this case it brings closer semantically similar words. counter-fitting @xcite likewise fine-tunes word representations; however, conversely to the retrofitting technique it counterfits the embeddings with respect to the given similarity and antonymy constraints. attract-repel @xcite uses linguistic constraints obtained from external lexical resources to semantically specialize word embeddings. similarly to counter-fitting it injects synonymy and antonymy constraints into distributional word vector spaces. in contrast to counter-fitting, this method does not ignore how updates of the example word vector pairs affect their relations to other word vectors. on the other hand, the latter group, postspecialization methods, performs global specialization of distributional spaces. we can distinguish: explicit retrofitting @xcite that was the first attempt to use external constraints (i.e. synonyms and antonyms) as training examples for learning an explicit mapping function for specializing the words not observed in the constraints. later, a more robust dffn @xcite ) method was introduced with the same goal -to specialize the full vocabulary by leveraging the already specialized subspace of seen words. ## methodology in this paper, we propose an approach that builds upon previous works @xcite . the process of specializing distributional vectors is a two-step procedure (as shown in figure 1 ). first, an initial specialization is performed (see §3.1). in the second step, a global specialization mapping function is learned, allowing to generalize to unseen words (see §3.2). ## conclusion and future work in this work, we presented a method to perform semantic specialization of word vectors. specifically, we compiled a new set of constraints obtained from babelnet. moreover, we improved a state-of-theart post-specialization method by incorporating adversarial losses with the wasserstein distance. our results obtained in an intrinsic and an extrinsic task, suggest that our method yields performance gains over current methods. in the future, we plan to introduce constraints for asymmetric relations as well as extend our proposed method to leverage them. moreover, we plan to experiment with adapting our model to a multilingual scenario, to be able to use it in a neural machine translation task. we make the code and resources available at: https://github.com/ mbiesialska/wgan-postspec