Benchmarking Meta-embeddings: What Works and What Does Not (2021)

Abstract: In the last few years, several methods have been proposed to build meta-embeddings. The general aim was to obtain new representations integrating complementary knowledge from different source pre-trained embeddings thereby improving their overall quality. However, previous meta-embeddings have been evaluated using a variety of methods and datasets, which makes it difficult to draw meaningful conclusions regarding the merits of each approach. In this paper we propose a unified common framework, including both intrinsic and extrinsic tasks, for a fair and objective meta-embeddings evaluation. Furthermore, we present a new method to generate meta-embeddings, outperforming previous work on a large number of intrinsic evaluation benchmarks. Our evaluation framework also allows us to conclude that previous extrinsic evaluations of meta-embeddings have been overestimated.

## introduction word embeddings successfully capture lexical semantic information about words based on cooccurrence patterns extracted from large corpora @xcite @xcite or knowledge bases @xcite , with excellent results on several tasks, including word similarity @xcite @xcite , semantic textual similarity @xcite , or more recently, unsupervised machine translation @xcite , inferring representations for rare words @xcite , unsupervised word alignment @xcite or knowledge base probes @xcite . in these tasks, word embeddings perform similarly or better than transformer-based language models such as bert @xcite , while requiring a comparatively tiny amount of resources for training and inference. following the hypothesis that different knowledge sources may contain complementary semantic information @xcite , metaembeddings @xcite aim to obtain an ensemble of distinct word embeddings each trained using different methods and resources to produce a word representation with an improved overall quality. the main challenge when generating metaembeddings is preserving the information encoded in the source embeddings and many different methods have been proposed to deal with the task. concatenation @xcite and averaging @xcite are two very strong baselines, but much complex methods based on linear transformations and supervised neural models have also been proposed @xcite @xcite . when it comes to evaluating meta-embeddings, there is no consensus on either evaluation tasks or methodology. meta-embeddings are evaluated in a wide range of tasks @xcite , ranging from intrinsic (i.e. word similarity, word analogy) to extrinsic tasks such as short text classification @xcite , common-sense stories @xcite , named entity recognition (o'neill and @xcite or semantic textual similarity @xcite . furthermore, different evaluation methodologies have been applied. for example, @xcite discard the words in the datasets which are not represented in the meta-embedding model, while @xcite use various strategies to minimize the number of out-of-vocabulary (oov) words. to make things more complicated, previous meta-embeddings approaches require some ad-hoc pre-processing to tune multiple filtering criteria and parameters according to the source embeddings used @xcite @xcite , which has a signifi-cant effect on the final evaluation results. summarizing, this lack of consistency in evaluation tasks, methodologies and ad-hoc hyper-parameter tuning makes it very hard to objectively compare the proposed methods. thus, to the best of our knowledge, and despite the existence of multiple works addressing this task, a unified and comprehensive evaluation of meta-embeddings has not been yet carried out. in fact, the lack of such unified and comprehensive evaluation framework has arguably caused erroneous assumptions and an overestimation in the performance of meta-embeddings for extrinsic tasks. an additional issue is that most previous work has focused on combining word embeddings generated from similar sources and algorithms. for instance, combining word2vec cbow @xcite with glove @xcite embeddings. we empirically show that, since these embeddings encode very similar knowledge, combining them does not produce a significant gain. instead, the best meta-embeddings are obtained by combining embeddings trained with different algorithms and resources. for example, by leveraging vectors induced from text corpora together with other embeddings obtained from knowledge bases. in this paper we present a new method to generate meta-embeddings that outperform previous approaches on a large number of intrinsic benchmarks. other contributions include: the rest of the paper is organized as follows. section 2 presents the related work. section 3 focuses on the evaluation frameworks used by previous works and presents our own proposal. in section 4 we describe our approach for creating meta-embeddings, with section 5 describing the source word embeddings explored and reporting our experimental results in section 6. finally, section 7 presents some concluding remarks and our future work. our code and meta-embeddings are publicly available foot_0 . ## related work previous research has shown that word embeddings created using different methods and resources present significant variations in quality. for instance, @xcite show that word embeddings trained from monolingual or bilingual corpora capture different nearest neighbours. the term meta-embedding was coined by @xcite . they showed how to combine five different pre-trained word embeddings using a small neural network for improving the accuracy of cross-domain part-of-speech (pos) tagging. following this, @xcite propose an unsupervised locally linear method for learning meta-embeddings from a given set of pre-trained source embeddings while @xcite apply three types of autoencoders for the purpose of learning meta-embeddings. although word embeddings are mainly constructed by exploiting information from text corpora only @xcite @xcite , some approaches also tried different methods to integrate the knowledge encoded in lexical resources such as wordnet @xcite @xcite , ppdb @xcite or con-ceptnet @xcite . @xcite show that simply concatenating word embeddings derived from text and wordnet outperform alternative methods such as retrofitting @xcite at the cost of increasing the dimensionality of the meta-embeddings. coates and @xcite prove that averaging is in some cases better than concatenation, with the additional benefit of a reduced dimensionality. the most popular approach to address the dimensionality problem is to apply dimensionality reduction algorithms paper intrinsic tasks extrinsic tasks @xcite sst, snli, image caption (1) @xcite sst2, snli, ner (1), pos(1), semcor @xcite sim. (4), an. (3), relation classification (1) short text classification (4), psycholinguistic score prediction (2) (o'neill and bollegala, 2020) sim. (6), an. (3) pos (1), ner (1), sentiment analysis(1) @xcite sim. (6), an. (2) @xcite sim. (4), bilingual dictionary induction (4), hypernym discovery (1) @xcite sim. (6), an. (2), relation classification (1) short-text classification (2) (coates and @xcite sim. (5), an. (1) @xcite sim. (5), an. (1) pos (1) @xcite mt ( 3 ), text classification (5) @xcite sim. (5) snli (1) @xcite sim. (4) @xcite sim. (5). sat an. (1) common-sense stories (1) (garcía-ferrero et al., 2020) sim. (14) sts (1), pos (1) this work sim. (7), an. (3), categorization (4) cola, sst-2, mrpc, sts-b, qqp, mnli, qnli, rte , wnli, ax another recent research line tries to dynamically generate meta-embeddings for specific tasks @xcite @xcite . these methods extend already existing algorithms to generate meta-embeddings by learning task specific weights. instead, the focus of our research is to generate the best general purpose meta-embedding that can be applied to any task. ## evaluation framework as it has been earlier mentioned, several methods to generate meta-embeddings have been previously proposed and evaluated on many different benchmarks, as shown by two popular techniques are used to address oov words. this lack of evaluation consistency led us to propose a unified evaluation framework that encompasses a wide range of tasks and datasets to evaluate meta-embeddings. in order to make the evaluation as simple and unified as possible we chose two already existing out of the box frameworks: word embeddings benchmarks foot_2 @xcite provides scripts for evaluating word embeddings in three intrinsic evaluation tasks: (i) word similarity (ws353 @xcite , mturk @xcite , rg65 (rubenstein and goodenough, 1965), rw (pilehvar et al., 2018), simlex999 @xcite , men @xcite ); (ii) word analogy (google analogy @xcite , msr analogy @xcite , semeval2012 @xcite ) and, (iii) word categorization (ap @xcite , bless @xcite , battig (battig and montague, 1969), essli @xcite ). we use the provided script for evaluating embeddings on all the tasks without lowercasing them. it should be taken into account that, for word analogy, smaller vocabularies usually obtain better results. this particularly hurts the performance of those meta-embeddings that were generated using many source embeddings resulting in a metaembedding with a vocabulary of more than 4 million words. thus, in order to ensure a fair evaluation regardless of the number of words in the vocabulary, we trim the vocabulary of all the embeddings and meta-embeddings to the 200,000 most popular english words according to the google's trillion ## our method our meta-embedding generation approach consists of two main steps: (i) pre-processing of the source embeddings and (ii) generation of the metaembedding by averaging. our method can combine any number of word embeddings as long as there is some common vocabulary shared between them. the resulting meta-embedding vocabulary will be the union of the vocabularies of the source word embeddings used. ## word embeddings this section describes the source word embeddings used to generate our meta-embeddings. we choose these pre-trained embeddings for two main reasons. they have been trained using very diverse algorithms and resources, and they obtain good performance on our evaluation framework when tested individually. that is, they may encode high quality complementary knowledge. using large text corpora, word2vec (w2v) @xcite embeddings from google news (100 billion words). a glove (gv) @xcite model the common crawl vectors (640 billion words). as recommended by the authors, we apply a l2 normalization to its variables. and the fasttext (ft) @xcite embeddings from common crawl (600 billion words). using wordnet @xcite , rwsgwn (ukb) @xcite combines random walks over wordnet with the skip-gram model. we have used the vectors trained using wordnet3.0 plus gloss relations. jointchyb (j) @xcite combines random walks over multilingual wordnets and bilingual corpora as input for a modified skip-gram model that forces equivalent terms in different languages to come closer during training. we used the english-spanish bilingual embeddings publicly available. using the paraphrase database (ppdb) @xcite , attract repel (ar) @xcite improves word embeddings by injecting synonymy and antonym constraints extracted from monolingual and cross-lingual lexical resources. we used the english vocabulary from the four-lingual (english, german, italian, russian) vector space. paragram (p) @xcite are pre-trained word vectors learned using word paraphrase pairs from ppdb using a modification of the skip-gram objective function. the hyper parameters were tuned using the wordsim-353 dataset. the word embeddings of the default model are initialized with glove word vectors. using conceptnet; numberbatch (n) @xcite combines knowledge encoded in con-ceptnet, word2vec, glove and opensubtitles 2016 using concatenation, dimensionality reduction and a variation of retrofitting. numberbatch version 19.08 is used. we also tested other embeddings such as extvec @xcite , lexsub @xcite or lexvec @xcite ## experiments we evaluate all the word embeddings in a wide range of intrinsic and extrinsic evaluation tasks which composed the evaluation framwework described in section 3. ## conclusions with oov : overlove, outlove, antilove, loveaholic, have_no_regrets, sometimes_good, won-derful_feeling, strong_like, filial_love, lovedom, propose_to_woman, lov?d, family_love, love, love_dearly, love_heart, adore, buy_ring, love, deep_affection, being_in_love, sovietophile, loveful, loving, mislove, lovemonger, arachnophile without oov : overlove, loveaholic, outlove, antilove