Going Beyond T - SNE : Exposing whatlies in Text Embeddings (2020)

Abstract: We introduce whatlies, an open source toolkit for visually inspecting word and sentence embeddings. The project offers a unified and extensible API with current support for a range of popular embedding backends including spaCy, tfhub, huggingface transformers, gensim, fastText and BytePair embeddings. The package combines a domain specific language for vector arithmetic with visualisation tools that make exploring word embeddings more intuitive and concise. It offers support for many popular dimensionality reduction techniques as well as many interactive visualisations that can either be statically exported or shared via Jupyter notebooks. The project documentation is available from https://rasahq.github.io/whatlies/ .

## introduction the use of pre-trained word embeddings @xcite or language model based sentence encoders @xcite has become a ubiquitous part of nlp pipelines and end-user applications in both industry and academia. at the same time, a growing body of work has established that pre-trained embeddings codify the underlying biases of the text corpora they were trained on @xcite @xcite . hence, practitioners need tools to help select which set of embeddings to use for a particular project, detect potential need for debiasing and evaluate the debiased embeddings. simplified visualisations of the latent semantic space provide an accessible way to achieve this. therefore we created whatlies, a toolkit offering a programmatic interface that supports vector arithmetic on a set of embeddings and visualising the space after any operations have been carried out. for example, figure 1 shows an example of how representations for queen, king, man, and woman can be projected along the axes v queen-king and v man|queen-king in order to derive a visualisation of the space along the projections. perhaps the most widely known tool for visualising embeddings is the tensorflow projector foot_0 which offers 3d visualisations of any input embeddings. the visualisations are useful for understanding the emergence of clusters and the neighbourhood of certain words and the overall space. however, the projector is limited to dimensionality reduction as the sole preprocessing method. more recently, @xcite have introduced parallax which allows explicit selection of the axes on which to project a representation. this creates an additional level of flexibility as these axes can also be derived from arithmetic operations on the embeddings. the major difference between the tensorflow pro-jector, parallax and whatlies is that the first two provide a non-extensible browser-based interface, whereas whatlies provides a programmatic one. therefore whatlies can be more easily extended to any specific practical need and cover individual use-cases. the goal of whatlies is to offer a set of tools that can be used from a jupyter notebook with a range of visualisation capabilities that goes beyond the commonly used static t-sne (van der maaten and hinton, 2008) plots. whatlies can be installed via pip, the code is available from https://github.com/rasahq/ whatlies 2 and the documentation is hosted at https://rasahq.github.io/whatlies/ . embedding backends. the current version of whatlies supports word-level as well as sentence-level embeddings in any human language that is supported by the following libraries: retrieved embeddings are python objects that contain a vector and an associated named. it comes with extra utility methods attached that allow for easy arithmetic and visualisation. the library is capable of retreiving embeddings for sentences too. in order to retrieve a sentence representation for word-level embeddings such as fasttext, whatlies returns the summed representation of the individual word vectors. for pretrained encoders such as bert @xcite or convert @xcite , whatlies uses its internal [cls] token for representing a sentence. similarity retrieval. the library also supports retrieving similar items on the basis of a number of commonly used distance/similarity metrics such as cosine or euclidean distance: vector arithmetic. support of arithmetic expressions on embeddings is integral in any whatlies functions. for example the code for creating figure 1 from the introduction highlights that it does not make a difference whether the plotting functionality is invoked on an embedding itself or on a representation derived from an arithmetic operation: import matplotlib.pylab as plt from whatlies import embedding @xmath0, [0.5, 0.1]) @xmath1, [0.5, 0.6]) @xmath2, [0.7, 0.33]) @xmath3, [0.7, 0.9]) man.plot(@xmath4, @xmath5.plot(@xmath6, @xmath7.plot(@xmath8, @xmath9.plot(@xmath10, @xmath11.plot(@xmath12, @xmath13.plot(@xmath14, @xmath15:) this feature allows users to construct custom queries and use it e.g. in combination with the similarity retrieval functionality. for example, we can validate the widely circulated analogy of mikolov et al. (2013b) excluding the query word king foot_4 , the analogy returns the anticipated result: queen. the library also allows the user to add/subtract embeddings but also project unto (via @xmath16. this means that the user is very flexible when it comes to retrieving embeddings. multilingual support. whatlies supports any human language that is available from its current list of supported embedding backends. this allows us to check the royal analogy from above in languages other than english. the code snippet below shows the results for spanish and dutch, using pre-trained fasttext embeddings foot_5 . from whatlies.language import \ fasttextlanguage @xmath17.es.300.bin") @xmath18.nl.300.bin") @xmath19.score_similar(emb_es, @xmath20, @xmath21, the correct answer reina is only at rank 3 (excluding rey from the list), the second ranked monarca (female form of monarch) is getting close. for dutch, the correct answer koningin is at rank 2, surpassed only by koningen (plural of king). another interesting observation is that the cosine distances -even of the query words -vary wildly in the embeddings for the two languages. ## conclusion we have introduced whatlies, a python library for inspecting word and sentence embeddings that is very flexible due to offering a programmable interface. we currently support a variety of embedding models, including fasttext, spacy, bert, or the universal sentence encoder. this paper has showcased its current use as well as plans for future development. the project is hosted at https://github.com/rasahq/whatlies and we are happy to receive community contributions that extend and improve the package.