I nterpreting P retrained C ontextualized R epresentations via R eductions to S tatic E mbeddings (2020)

Abstract: Contextualized representations (e.g. ELMo, BERT) have become the default pretrained representations for downstream NLP applications. In some settings, this transition has rendered their static embedding predecessors (e.g. Word2Vec, GloVe) obsolete. As a side-effect, we observe that older interpretability methods for static embeddings — while more diverse and mature than those available for their dynamic counterparts — are underutilized in studying newer contextualized representations. Consequently, we introduce simple and fully general methods for converting from contextualized representations to static lookup-table embeddings which we apply to 5 popular pretrained models and 9 sets of pretrained weights. Our analysis of the resulting static embeddings notably reveals that pooling over many contexts significantly improves representational quality under intrinsic evaluation. Complementary to analyzing representational quality, we consider social biases encoded in pretrained representations with respect to gender, race/ethnicity, and religion and find that bias is encoded disparately across pretrained models and internal layers even for models with the same training data. Concerningly, we find dramatic inconsistencies between social bias estimators for word embeddings.

## introduction word embeddings @xcite @xcite have been a hallmark of modern natural language processing (nlp) for many years. embedding methods have been broadly applied and have experienced parallel and complementary innovations alongside neural network methods for nlp. advances in embedding quality in part have come from integrating additional information such as syntax @xcite , morphology @xcite , subwords @xcite , subcharacters @xcite and, most recently, context @xcite . due to their tremendous representational power, pretrained contextualized representations, in particular, have seen widespread adoption across myriad subareas of nlp. the recent dominance of pretrained contextualized representations such as elmo @xcite and bert @xcite has served as the impetus for exciting and diverse interpretability research: @xcite ; @xcite study what is learned across the layers of these models, @xcite ; @xcite consider what is learned from context, @xcite ; @xcite look at specific attention heads, @xcite ; @xcite address linguistic understanding such as syntax and negation, and @xcite ; @xcite address ethical concerns such as security (adversarial robustness) and social bias. in fact, the neologism bertology was coined specifically to describe this flurry of interpretability research. 1 while these works have provided nuanced finegrained analyses by creating new interpretability schema/techniques, we instead take an alternate approach of trying to re-purpose methods developed for analyzing static word embeddings. in order to employ static embedding interpretability methods to contextualized representations, we begin by proposing a simple strategy for converting from contextualized representations to static embeddings. crucially, our method is fully general and assumes only that the contextualized model maps word sequences to vector sequences. given this generality, we apply our method to 9 popular pretrained contextualized representations. the resulting static embeddings serve as proxies for the original contextualized model. we initially examine the representational quality of these embeddings under intrinsic evaluation. our evaluation produces several insights regarding layer-wise lexical semantic understanding and representational variation in contextualized representations. importantly, our analyses suggest constructive improvements to potentially improve downstream practices in using contextualized models. simultaneously, we find that our static embeddings substantially outperform word2vec and glove and therefore suggests our method serves the dual purpose of being a lightweight mechanism for generating static embeddings that track with advances in contextualized representations. since static embeddings have significant advantages with respect to speed, computational resources, and ease of use, these results have important implications for resource-constrained settings @xcite , environmental concerns @xcite , and the broader accessibility of nlp technologies. 2 alongside more developed methods for embedding analysis, the static embedding setting is also equipped with a richer body of work regarding social bias. in this sense, we view understanding the encoded social bias in representations as a societally critical special-case of interpretability research. we employ methods for identifying and quantifying gender, racial/ethnic, and religious bias @xcite @xcite to our static embeddings. these experiments not only shed light on the properties of our static embeddings for downstream use but can also serve as a proxy for understanding latent biases in the original pretrained contextual representations. we find that biases in different models and across different layers are quite disparate; this has important consequences on model and layer selection for downstream use. further, for two sets of pretrained weights learned on the same training data, we find that bias patterns still remain fairly distinct. most surprisingly, our large-scale evaluation makes clear that existing bias estimators are dramatically inconsistent with each other. ## methods in order to use a contextualized model like bert to compute a single context-agnostic representation for a given word w, we define two operations. the first is subword pooling: the application of a pooling mechanism over the k subword representations generated for w in context c in order to compute a single representation for w in c, i.e. {w 1 c , . . . , w k c } → w c . beyond this, we define context combination to be the mapping from representations w c 1 , . . . , w cn of w in different contexts c 1 , . . . , c n to a single static embedding w that is agnostic of context. subword pooling. the tokenization procedure for bert can be decomposed into two steps: performing a simple word-level tokenization and then potentially deconstructing a word into multiple subwords, yielding w 1 , . . . , w k such that cat(w 1 , . . . , w k ) = w where cat(•) indicates concatenation. then, every layer of the model computes vectors w 1 c , . . . , w k c . given these vectors, we consider four pooling mechanisms to compute w c : ## representation quality 4.1 evaluation details to assess the representational quality of our static embeddings, we evaluate on several word similarity and word relatedness datasets. 3 we consider 4 such datasets: rg65 @xcite , ws353 @xcite , sim-lex999 @xcite and simverb3500 @xcite (see §a.4 for more details). taken together, these datasets contain 4917 examples and specify a vocabulary v of 2005 unique words. each example is a pair of words (w 1 , w 2 ) with a gold-standard annotation (provided by one or more humans) of the semantic similarity or relatedness between w 1 and w 2 . a word embedding is evaluated by the relative correctness of its ranking 3 concerns with this decision are discussed in §a.3. ## bias bias is a complex and highly relevant topic in developing representations and models in nlp and ml. in this context, we study the social bias encoded within our static word representations as a proxy for understanding biases of the source contextualized representations. as kate crawford argued for in her nips 2017 keynote, while studying individual models is important given that specific models may propagate, accentuate, or diminish biases in different ways, studying the representations that serve as the starting point and that are shared across models (which are used for possibly different tasks) allows for more generalizable understanding of bias @xcite . in this work, we simultaneously consider multiple axes of social bias (i.e. gender, race, and religion) and multiple proposed methods for computationally quantifying these biases. we do so precisely because we find that existing nlp literature has primarily prioritized gender (which may be a technically easier setting and is starkly incomplete in terms of social biases of interest). further, as we will show, different computational specifications of bias that evaluate the same underlying social phenomena yield markedly different results. as a direct consequence, we strongly caution that the results must be taken with respect to the definitions of bias being applied. further, we note that an embedding which receives low bias scores cannot be assumed to be (nearly) unbiased. instead, it satisfies the significantly weaker condition that under existing definitions the embedding exhibits low bias and perhaps additional (more nuanced) definitions are needed. @xcite introduced a measure of gender bias which assumes access to a set @xmath0, f 1 ), . . . , (m n , f n )} of (male, female) word pairs where m i and f i only differ in gender (e.g. 'men' and 'women'). they compute a gender direc- ## discussion and open problems while our work demonstrates that contextualized representations retain substantial representational power even when reduced to be noncontextual, it is unclear what information is lost. after all, contextualized representations have been so effective precisely because they are tremendously contextual @xcite . as such, the validity of treating the resulting static embeddings as reliable proxies for the original contextualized model still remains open. on the other hand, human language processing has often been conjectured to have both context-dependent and context-independent properties @xcite @xcite . given this divide, our approach may provide an alternative mechanism for clarifying how these two properties interact in the computational setting from both an interpretability standpoint (i.e. comparing results for analyses on the static embeddings and the original contextualized representations) and a downstream standpoint (i.e. comparing downstream performance for models initialized using the static embeddings and the original contextualized representations). however, the precise relationship between the role of context in human language processing and computational language processing remains unclear. theoretical explanation for the behavior we observe in two settings is also needed. first, it is unclear why learning contextualized representations and then reducing them to static embeddings drastically outperforms directly learning static embeddings. in particular, the glove embeddings we use are learned using 6 billion tokens whereas the bert representations were trained on roughly half as much data (3.3 billion tokens). perhaps the behavior is reminiscent of the benefits of modelling in higher dimensional settings temporarily as is seen in other domains (e.g. the kernel trick and mercer's theorem for learning non-linear classifiers using inner product methods): begin by recasting the problem in a more expressive space (contextualized representations) and then project/reduce to the original space (static embeddings). second, the reason for the benefits of the variance reduction that we observe are unclear. given that best-performing mechanism is to average over many contexts, it may be that approaching the asymptotic mean of the distribution across contexts is desirable/helps combat the anisotropy that exists in the original contextualized space @xcite . ## conclusion in this work, we consider how methods developed for analyzing static embeddings can be re-purposed for understanding contextualized representations. we introduce simple and effective procedures for converting from contextualized representations to static word embeddings. when applied to pretrained models like bert, we find the resulting embeddings are useful proxies that provide insights into the pretrained model while simultaneously outperforming word2vec and glove substantially under intrinsic evaluation. we further study the extent to which various social biases (gender, race, religion) are encoded, employing several different quantification schemas. our large-scale analysis reveals that bias is encoded disparately across different popular pretrained models and different model layers. our findings also have significant implications with respect to the reliability of existing protocols for estimating bias in word embeddings.