Wikivecs: A Fully Reproducible Vectorization of Multilingual W ikipedia (2025)

Abstract: Dense vector representations have become foundational to modern natural language processing (NLP), powering diverse workflows from semantic search and retrieval augmented generation to content comparison across languages. Although Wikipedia is one of the most comprehensive and widely used datasets in modern NLP research, it lacks a fully reproducible and permissively licensed dense vectorization.In this paper, we present Wikivecs, a fully reproducible, permissively licensed dataset containing dense vector embeddings for every article in Multilingual Wikipedia. Our pipeline leverages a fully reproducible and permissively licensed multilingual text encoder to embed Wikipedia articles into a unified vector space, making it easy to compare and analyze content across languages.Alongside these vectors, we release a two-dimensional data map derived from the vectors, enabling visualization and exploration of Multilingual Wikipedia’s content landscape.We demonstrate the utility of our dataset by identifying several content gaps between English and Russian Wikipedia.

## introduction dense vector representations have become foundational to modern natural language processing (nlp), powering diverse workflows from semantic search to retrieval augmented generation. as multilingual models have matured, these vectors have become particularly useful for bridging linguistic and cultural gaps, offering a shared representational space where texts from different languages can be meaningfully compared. simultaneously, wikipedia has become one of the most important datasets in modern nlp research. despite its importance, there exists no openly licensed, fully reproducible resource that provides dense vectors for the entirety of multilingual wikipedia. this gap limits the accessibility and transparency of multilingual wikipedia figure 1 : the directory structure of the wikivecs dataset. each folder corresponds to a language split and contains one or more sharded numpy arrays. when concatenated in ascending shard index order, these arrays correspond embeddings of the rows in the 2023-11-01 wikidump dataset. for example, 20231101_en_embeddings_shard_0.npy contains embeddings for rows 0-999,999 of the wikidump file 20231101.en, 20231101_en_embeddings_shard_1.npy contains embeddings for rows 1,000,0000-1,999,999 of the wikidump file 20231101.en, etc... also included in the dataset is wikimap.parquet, a file which contains 2d positions for every article in the wikidump, enabling subsequent visualization. in this work, we introduce wikivecs, a fully reproducible and permissively licensed dataset of dense vector embeddings for every article in multiulingual wikipedia. using a state-of-the-art multilingual encoder, wikivecs captures the semantic content of articles in a vector space that can 1 be compared meaningfully across languages. we also leverage recent research in scalable dimensionality reduction to construct a 2 dimensional map of the entirety of multilingual wikipedia. we demonstrate how these resources can be used to surface topics that lack cross-lingual coverage in wikipedia, highlighting several content gaps between english and russian wikipedia. ## background early methods for computational text comparison relied on normalized word count statistics @xcite the field of text representation shifted to a much more connectionist paradigm in 2013 with the publication of word2vec @xcite , which used a shallow neural network to learn a vector representation for each token in a sentence based on its context. bert @xcite utilized the transformer @xcite architecture to train a deep neural network to produce contextualized vector representations of tokens in an input text. bert could also be used to compare texts by running it in a cross-encoder configuration. in the cross-encoder configuration, two input texts were fed to the bert model together to produce a document similarity score. computing the pairwise similarity of all texts in a large corpus is computationally burdensome, making it difficult to utilize bert for document comparison in large corpora. to remedy this, reimers and gurevych introduced sbert @xcite , which utilized a bi-encoder architecture and a triplet training procedure to map sentences to dense vectors endowed with a semantic similarity structure. this enabled document similarity to be computed efficiently by measuring the angle between documents' dense vector representations. the efficiency of this bi-encoder approach has led it to become the dominant technique for large scale text representation and comparison. as a result, a plethora of text bi-encoders have been subsequently developed and released. @xcite @xcite @xcite . unfortunately, the almost all state-of-the-art biencoders are not suitable for open science and open knowledge use, since large parts of their training recipe (e.g. training data, training code, weights, etc...) remain proprietary or are restrictively li-censed. in contrast, the recently released nomic-embed-text-v2 @xcite is the first state-of-the-art multilingual bi-encoder to release all the elements of its training recipe under a permissive apache 2.0 license, making it an ideal candidate for open science and open knowledge work. 3 dataset description our dataset is an extension of the 2023-11-01 wikidump dataset @xcite , which we accessed via hugging face. we produce a dense vector for each article in the wikidump using nomic-embed-text-v2, a fully open and permissively licensed multilingual text embedding model. each article is truncated to the first 512 tokens so that it fits in the nomic-embed-text-v2 context window, and no task specific prefixes are prepended to the article texts. nomic-embed-text-v2 then converts each article to a 768 dimensional dense vector. these article vectors are saved in shards corresponding to each of the language splits in the wikidump. figure 1 details the directory structure of the resulting shards. we note that process of generating and storing these vectors is nontrivial, requiring several h100 days of compute, and resulting in approximately a terabyte of data. nomic-embed-text-v2 achieves state-of-the-art performance on the miracl benchmark @xcite , as well as strong performance on the bitext mining task of the mmteb benchmark @xcite . as a result, we can reasonably expect the vectors it produces to map semantically similar content in different languages to similar locations in vector space. this property enables downstream cross-lingual analysis using the produced vectors. once all the articles are vectorized, we apply nomad projection @xcite , a scalable nonlinear dimensionality reduction algorithm, to project them into a 2d space for subsequent visualization and inference. we run no-mad projection with the hyperparameters outlined in ## access and reproduction all of the choices in our vectorization pipeline were made with accessibility and reproducibility in mind. nomic-embed-text-v2 releases its training data, code, and weights under a permissive license, meaning the wikivectors themselves have fully open provenance. further, all code for generating the vectors, as well as the subsequent visualizations, is open sourced and permissively licensed. finally, the 2d vectorization alleviates the large computational burden of working with the 768 dimensional vectors, enabling meaningful corpus analysis to be performed on a standard laptop. the final dataset can be accessed at https://huggingface.co/datasets/nomic-ai/nomicembed-v2-wikivecs , and the code for reproducing the dataset can be accessed at https://github.com/nomic-ai/wikivecs . ## example application: cross lingual content gap analysis as an example of the utility of our dataset, we use the 2d positions derived from the wikivecs to surface content gaps between english and russian wikipedia. we define a content gap as a collec-no yes левшин, геннадий егорович no yes total 1 10 ## conclusion in this paper we introduced wikivecs, the first open source, fully reproducible, and permissively licensed dense vectorization of multilingual wikipedia. we generate a 768 dimensional dense vector representation of each article on multilingual wikipedia using a fully reproducible and per- as an example of the utility of our dataset, we use both qualitative and quantitative methods to surface content gaps between english and russian wikipedia. qualitatively, we contribute an interactive data exploration application that enables users to visually compare the coverage of different wikipedia language splits. we use this application to surface a content gap related to russian scientists who lack articles in english wikipedia. quantitatively, we train a decision tree to surface spatially localized regions of low language entropy in the 2d semantic map. manual investigation of a random sample of these low-entropy regions surfaces content gaps relating to beetles and hungarian villages on russian wikipedia. we run both our qualitative and quantitative analysis on a standard m2 macbook air with 24gb of ram, highlighting the accessibility of our dataset. overall, we believe that our dataset will significantly lower the barrier to performing modern nlp application development and analysis on multilingual wikipedia.