Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings (2021)

Abstract: Models based on the transformer architecture, such as BERT, have marked a crucial step forward in the field of Natural Language Processing. Importantly, they allow the creation of word embeddings that capture important semantic information about words in context. However, as single entities, these embeddings are difficult to interpret and the models used to create them have been described as opaque. Binder and colleagues proposed an intuitive embedding space where each dimension is based on one of 65 core semantic features. Unfortunately, the space only exists for a small data-set of 535 words, limiting its uses. Previous work (Utsumi, 2018, 2020; Turton et al., 2020) has shown that Binder features can be derived from static embeddings and successfully extrapolated to a large new vocabulary. Taking the next step, this paper demonstrates that Binder features can be derived from the BERT embedding space. This provides two things; (1) semantic feature values derived from contextualised word embeddings and (2) insights into how semantic features are represented across the different layers of the BERT model.

## introduction the last decade or so has seen a rapid progress in the field of natural language processing (nlp) with a combination of new models and increasingly powerful hardware resulting in state of the art performances across a number of common tasks @xcite . one important area of improvement has been in the vector-space representation of words, known as word embeddings. embedding models create word vectors within a vector space that captures important semantic and grammatical information @xcite . models such as word2vec @xcite and glove @xcite were popular in the 2010s, but are static, meaning only one embedding is produced for each word. in reality words can have multiple meanings; 7% of common english word forms have homonyms and over 80% are polysemous @xcite . deep learning language models such as elmo: embeddings from language models @xcite addressed this issue, using deep neuralnetwork language models to incorporate context and produce contextualised embeddings. following this, the introduction of the transformer architecture and in particular its implementation in the bidirectional encoder representations from transformers (bert) @xcite model, resulted in even better performing contextual embeddings. regardless of whether the embeddings mentioned are static or contextual, they all have the issue that, as individual objects, they are hard to interpret @xcite . whilst efforts have been made to produce more interpretable embeddings e.g. @xcite , the general approach has been to interpret them in relation to each-other. for example, the relative distance between word embeddings can indicate their semantic similarity @xcite . alternatively, dimensionality reduction techniques can be used to visualise where the words sit within the embedding space @xcite . however, these methods may just show how the embeddings are related, rather than why, further feeding into the general criticism levelled at deep learning architectures; that they are opaque and difficult to interpret @xcite . @xcite presented an alternative embedding space for words, based on 65 core semantic features, where each dimension relates to a feature. unfortunately, the binder dataset only contains 535 words, severely limiting its use for large scale text analysis. previous research @xcite @xcite has shown that the binder feature values can be derived from static embeddings, such as word2vec, and successfully extrapolated to a large new vocabulary of words. the purpose of this research is to demonstrate that binder features can be successfully derived from bert embedding space allowing the features to be derived from contextual embeddings. along the way, this also provided the opportunity to study how different types of semantic information are represented across the different layers of the bert model. ## conclusion the overarching aim of this work was to demonstrate that binder style semantic feature embeddings can be derived from the bert embedding space in the same way that previous research @xcite @xcite has shown for static embeddings. it also offered the opportunity to probe how semantic information is represented across the different layers of bert. treating the embeddings as static, experiment 1a supported this aim with bert and other transformer embeddings outperforming the best performing static embeddings model numberbatch. the results also supported the findings of @xcite that semantic information tends to be represented in the later layers of bert. hand-picking sentences in experiment 1b lead to better performance indicating that some degree of context is represented in the derived semantic features. experiment 2 provided further evidence of the ability of transformer models to derive contextualised semantic features but was limited by the small set of features and the short word-pair context sequences. finally, the ability of binder embeddings to perform comparatively to raw bert embeddings in experiment 3 suggests that they do capture, to some degree, contextualised semantic features when derived from transformer embeddings. in conclusion, within the limitations of the binder dataset, this paper suggests that it is possible to derive contextualised semantic features from contextualised word embeddings as a proof of concept. however, without a ground-truth test, it is not able to demonstrate this conclusively. to do this would likely require the production of a binder feature set for words explicitly in context, and this may be a necessary next step if the binder feature set is considered useful for further use. furthermore, as the binder dataset focuses on general use words, for researchers wishing to derive semantic features useful for specific domains, they likely would need to construct datasets of domain-specific features for a domain-specific vocabulary. beyond the direct findings of this paper, we also hope that this work highlights the usefulness of using existing psychological research data to improve the understanding and interpretability of what can otherwise be somewhat opaque deep learning models. figure c. all feature r-squared scores for the (a) small and (b) large models for selected sentences of experiment 1b. figure f. residual plots for features (a) attention and (b) dark (a) (b) (b) (a) r-@xmath0.67 r-@xmath1.38