Static Embeddings as Efficient Knowledge Bases? (2021)

Abstract: Recent research investigates factual knowledge stored in large pretrained language models (PLMs). Instead of structural knowledge base (KB) queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. The good performance on this analysis task has been interpreted as PLMs becoming potential repositories of factual knowledge. In experiments across ten linguistically diverse languages, we study knowledge contained in static embeddings. We show that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs. E.g., static embeddings perform 1.6% points better than BERT while just using 0.3% of energy for training. One important factor in their good comparative performance is that static embeddings are standardly learned for a large vocabulary. In contrast, BERT exploits its more sophisticated, but expensive ability to compose meaningful representations from a much smaller subword vocabulary.

## introduction pretrained language models (plms) @xcite @xcite can be finetuned to a variety of natural language processing (nlp) tasks and then generally yield high performance. increasingly, these models and their generative variants (e.g., @xcite are used to solve tasks by simple text generation, without any finetuning. this motivated research on how much knowledge is contained in plms: @xcite used models pretrained with a masked language objective to answer clozestyle templates such as: using this methodology, @xcite showed that plms capture some knowledge implicitly. this has been interpreted as suggesting for a plm like bert, ranking amounts to finding the entity whose embedding is most similar to the output embedding for @xcite . for static embeddings, we rank entities (e.g., entities of type country) with respect to similarity to the query entity (e.g., "paris" in (ex1)). in experiments across ten linguistically diverse languages, we show that this simple nearest neighbor matching with fasttext embeddings performs comparably to or even better than bert. for example for english, fasttext embeddings perform 1.6% points better than bert (41.2% vs. 39.6%, see we believe this means that claims that plms are kbs have to be treated with caution. advantages of bert are that it composes meaningful representations from a small subword vocabulary and handles typing implicitly @xcite . in contrast, answering queries without restricting the answer space to a list of candidates is hard to achieve with static word embeddings. on the other hand, static embeddings are cheap to obtain, even for large vocabulary sizes. this has important implications for green nlp. plms require tremendous computational resources, whereas static embeddings have only 0.3% of the carbon footprint of bert (see in summary, our contributions are: code and embeddings are available online. ## data we follow the lama setup introduced by @xcite . more specifically, we use data from trex @xcite . trex consists of triples of the form (object, relation, subject). the underlying idea of lama is to query knowledge from plms using templates without any finetuning: the triple (paris, capital-of, france) is queried with the template "paris is the capital of @xcite ." trex covers 41 relations. templates for each relation were manually created by @xcite . lama has been found to contain many "easy-toguess" triples; e.g., it is easy to guess that a person with an italian sounding name is italian. lama-uhn is a subset of triples that are "hard-to-guess" created by @xcite . beyond english, we run experiments on nine additional languages using mlama, a multilingual version of trex @xcite . for an overview of languages and language families see ## methods we describe our proposed setup, which allows to compare plms with static embeddings. ## conclusion we have shown that, when restricting cloze-style questions to a candidate set, static word embeddings outperform bert. to explain this puzzling superiority of a much simpler model, we put forward a new characterization of factual knowledge learned by bert: bert seems to be able to complete cloze-style queries based on similarity assessments on a type-restricted vocabulary much like a nearest neighbor search for static embeddings. however, bert may still be the better model for the task: we assume perfect typing (for bert and fasttext) and only evaluate ranking. typing is much harder with static embeddings and bert has been shown to perform well at guessing the expected entity type based on a template. bert also works well with small vocabularies, storing most of its "knowledge" in the parameterization of subword composition. our results suggest that increasing the vocabulary size and computing more atomic entity representations with fasttext is a cheap and environmentally friendly method of storing knowledge. in contrast, learning high quality composition of smaller units requires many more resources.