W ikipedia2 V ec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from W ikipedia (2020)

Abstract: The embeddings of entities in a large knowledge base (e.g., Wikipedia) are highly beneficial for solving various natural language tasks that involve real world knowledge. In this paper, we present Wikipedia2Vec, a Python-based open-source tool for learning the embeddings of words and entities from Wikipedia. The proposed tool enables users to learn the embeddings efficiently by issuing a single command with a Wikipedia dump file as an argument. We also introduce a web-based demonstration of our tool that allows users to visualize and explore the learned embeddings. In our experiments, our tool achieved a state-of-the-art result on the KORE entity relatedness dataset, and competitive results on various standard benchmark datasets. Furthermore, our tool has been used as a key component in various recent studies. We publicize the source code, demonstration, and the pretrained embeddings for 12 languages at https://wikipedia2vec.github.io/ .

## introduction entity embeddings, i.e., vector representations of entities in knowledge base (kb), have played a vital role in many recent models in natural language processing (nlp). these embeddings provide rich information (or knowledge) regarding entities available in kb using fixed continuous vectors. they have been shown to be beneficial not only for tasks directly related to entities (e.g., entity linking @xcite ) but also for general nlp tasks (e.g., text classification @xcite , question answering @xcite ). notably, recent studies have also shown that these embeddings can be used to enhance the performance of state-of-the-art contextualized word embeddings (i.e., bert @xcite ) on downstream tasks @xcite @xcite . in this work, we present wikipedia2vec, a python-based open source tool for learning the embeddings of words and entities easily and efficiently from wikipedia. due to its scale, availability in a variety of languages, and constantly evolving nature, wikipedia is commonly used as a kb to learn entity embeddings. our proposed tool jointly learns the embeddings of words and entities, and places semantically similar words and entities close to one another in the vector space. in particular, our tool implements the word-based skip-gram model @xcite to learn word embeddings, and its extensions proposed in @xcite to learn entity embeddings. wikipedia2vec enables users to train embeddings by simply running a single command with a wikipedia dump file as an input. we highly optimized our implementation, which makes our implementation of the skip-gram model faster than the well-established implementation available in gensim @xcite and fasttext @xcite . experimental results demonstrated that our tool achieved enhanced quality compared to the existing tools on several standard benchmarks. notably, our tool achieved a state-of-the-art result on the entity relatedness task based on the kore dataset. due to its effectiveness and efficiency, our tool has been successfully used in various downstream nlp tasks, including entity linking @xcite @xcite , named entity recognition @xcite , question answering @xcite , knowledge graph completion @xcite , paraphrase detection @xcite , fake news detection @xcite , and text classification @xcite . we also introduce a web-based demonstration of our tool that visualizes the embeddings by plotting them onto a two-or three-dimensional space using dimensionality reduction algorithms. the demonstration also allows users to explore the embeddings by querying similar words and entities. the source code has been tested on linux, windows, and macos, and released under the apache license 2.0. we also release the pretrained embeddings for 12 languages (i.e., english, arabic, chinese, dutch, french, german, italian, japanese, polish, portuguese, russian, and spanish). the main contributions of this paper are summarized as follows: ## related work many studies have recently proposed methods to learn entity embeddings from a kb @xcite @xcite @xcite @xcite . these embeddings are typically based on conventional word embedding models (e.g., skip-gram @xcite ) trained with data retrieved from a kb. for example, @xcite proposed rdf2vec, which learns entity embeddings using the skip-gram model with inputs generated by random walks over the large knowledge graphs such as wikidata and dbpedia. furthermore, a simple method that has been widely used in various studies @xcite @xcite @xcite trains entity embeddings by replacing the entity annotations in an input corpus with the unique identifier of their referent entities, and feeding the corpus into a word embedding model (e.g., skip-gram). two open-source tools, namely wiki2vec foot_0 and wikipedia entity vectors, foot_1 have implemented this method. our proposed tool is based on yamada et al. (2016), which extends this idea by using @xmath0 wikipedia2vec train enwiki-latest-pages-articles. xml.bz2 model_file figure 1: shell commands to train embeddings from the latest english wikipedia dump. >>> from wikipedia2vec import @xmath1.load(model_file) >>> model.get_entity_vector("scarlett johansson") memmap([-0.1979, 0.3086, ..., ], @xmath2.get_word_vector("tokyo") memmap([ 0.0161, -0.0332, ..., ], @xmath3.most_similar(model.get_entity("python ( programming language)"))[:3] [(<word python>, 0.7265), (<entity ruby (programming language)>, 0.6856), (<entity perl>, 0.6794)] neighboring entities connected by internal hyperlinks of wikipedia as additional contexts to train the model. note that we used the rdf2vec and wiki2vec as baselines in our experiments, and achieved enhanced empirical performance over these tools on the kore dataset. additionally, there have been various relational embedding models proposed @xcite @xcite that aim to learn the entity representations that are particularly effective for knowledge graph completion tasks. ## overview wikipedia2vec is an easy-to-use, optimized tool for learning embeddings from wikipedia. this tool can be installed using the python's pip tool (pip install wikipedia2vec). embeddings can be learned easily by running the wikipedia2vec train command with a wikipedia dump file foot_2 as an argument. figure 1 shows the shell commands that download the latest english wikipedia dump file and run training of the embeddings based on this dump using the default hyper-parameters. 4 furthermore, users can easily use the learned embeddings. figure 2 shows the example python code that loads the learned embedding file, and obtains the embeddings of an entity scarlett johansson and a word tokyo, as well as the most similar words and entities of an entity python. ## implementation our tool is implemented in python and most of its code is compiled into c++ using cython @xcite to optimize the run-time performance. as described in section 3.1, our link graph and anchor context models are based on the hyperlinks in wikipedia. because wikipedia contains numerous hyperlinks, it is challenging to use them efficiently. to address this, we introduce two optimized components-link graph matrix and mentionentity dictionary-that are used during training. link graph matrix during training, our link graph model needs to obtain numerous neighboring entities of an entity in a large link graph of wikipedia. to reduce latency, this component stores the entire graph in the memory using the binary sparse matrix in the compressed sparse row (csr) format, in which its rows and columns represent entities and its values represent the presence of hyperlinks between corresponding entity pairs. because the size of this matrix is typically small, it can easily be stored on the memory. 5 note that given a row index in the csr matrix, the time complexity of obtaining its non-zero column indices (corresponding to the neighboring entities of the entity that corresponds to the row index) is o(1). mention-entity dictionary a mention-entity dictionary is used to generate hyperlinks described in section 3.2. the dictionary maps entity names to their possible referent entities and is created based on the names and their referent entities obtained from all hyperlinks in wikipedia. our tool extracts all words and phrases from a wikipedia page that are included in the dictionary containing a large number of entity names. to implement this in an efficient manner, we use the aho-corasick algorithm, which is an efficient string search algorithm using finite state machine constructed from all entity names. after detecting the words and phrases in the dictionary, our tool converts them to hyperlinks based on heuristics described in section 3.2. the embeddings are trained by simultaneously iterating over pages in wikipedia and entities in the link graph in a random order. the texts and hyperlinks in each page are extracted using the mwparserfromhell mediawiki parser. 6 we do not use semi-structured data such as tables and infoboxes. we also generate hyperlinks using the mentionentity dictionary. we store the embeddings as a float matrix in a shared memory and update it using multiple processes. linear algebraic operations required to learn embeddings are implemented using c functions in basic linear algebra subprograms (blas). additionally, our tool uses a tokenizer to detect words from a wikipedia page. the following four tokenizers are currently implemented in our tool: (1) the multi-lingual icu tokenizer foot_6 that implements the unicode text segmentation algorithm @xcite , (2) a simple rule-based tokenizer that splits the text using white space characters, (3) the jieba tokenizer foot_7 for chinese, and (4) the mecab tokenizer foot_8 for japanese and korean. ## experiments we conducted experiments to compare the quality and efficiency of our tool with those of the existing tools. to evaluate the quality of the entity embeddings, we used the kore entity relatedness dataset @xcite . the dataset consists of 21 entities, and each entity has 20 related entities with scores assessed by humans. following past work, we reported the spearman's rank correlation coefficient between the gold scores and the cosine similarity between the entity embeddings. we used two popular entity embedding tools, rdf2vec @xcite and wiki2vec, as baselines. we also evaluated the quality of the word embeddings by employing two standard tasks: (1) a word analogy task using the semantic subset (sem) and syntactic subset (syn) of the google word analogy data set @xcite , and (2) a word similarity task using two standard datasets, namely simlex-999 (sl) @xcite and wordsim-353 (ws) @xcite . following past work, we reported the accuracy for the word analogy task, and the spearman's rank correlation coefficient between the gold scores and the cosine similarity between the word embeddings for the word similarity task. as baselines for these tasks, we used the skip-gram model @xcite implemented in the gensim library 3.6.0 @xcite and the extended skipgram model implemented in the fasttext tool 0.1.0 @xcite . we used wikiextractor foot_9 to create the training corpus for baselines. to the extent possible, we used the same hyperparameters to train our models and the baselines. 11 we also reported the time required for training using our tool and the baseline word embedding tools. note that the training of rdf2vec and wiki2vec tools are implemented using gensim. we conducted experiments using python 3.6 and openblas 0.3.3 installed on the c5d.9xlarge instance with 36 cpu cores deployed on amazon web services. to train our models and the baseline word embedding models, we used the april 2018 version of the english wikipedia dump. ## interactive demonstration we developed a web-based interactive demonstration that enables users to explore the embeddings of words and entities learned by our proposed tool (see figure 4 ). this demonstration enables users to visualize the embeddings onto a two-or threedimensional space using three dimensionality reduction algorithms, namely t-distributed stochastic neighbor embedding (t-sne) (maaten and hinton, 2008), uniform manifold approximation and projection (umap) @xcite , and principal component analysis (pca). users can move around the visualized embedding space by dragging and zooming using the mouse. moreover, the demonstration also allows users to explore the embeddings by querying similar items (words or entities) of an arbitrary item. we used the pretrained embeddings of 12 languages released with this paper as the target embeddings. furthermore, we also provided the english embeddings trained without the link graph model to allow users to qualitatively investigate how the link graph model affects the resulting embeddings. our demonstration is developed by extending the tensorflow embedding projector. 13 the demonstration is available at https://wikipedia2vec. github.io/demo . ## use cases the embeddings learned using our proposed tool have already been used effectively in various recent studies. @xcite have recently demonstrated that by combining bert with the entity embeddings trained by our tool outperforms bert and knowledge-enhanced contextualized word embeddings (i.e., ernie @xcite al., 2019) other examples include entity linking @xcite , named entity recognition @xcite , paraphrase detection (duong et al., 2019), fake news detection @xcite , and knowledge graph completion @xcite . ## conclusions in this paper, we present wikipedia2vec, an opensource tool for learning the embeddings of words and entities easily and efficiently from wikipedia. our experiments demonstrate the superiority of the proposed tool in terms of the quality of the embeddings and the efficiency of the training compared to the existing tools. furthermore, our tool has been effectively used in many recent state-ofthe-art models, which indicates the effectiveness of our tool on downstream tasks. we also introduce a web-based interactive demonstration that enables users to explore the learned embeddings. the source code and the pre-trained embeddings for 12 languages are released with this paper.