  Models based on the transformer architecture, such as BERT, have marked a
crucial step forward in the field of Natural Language Processing. Importantly,
they allow the creation of word embeddings that capture important semantic
information about words in context. However, as single entities, these
embeddings are difficult to interpret and the models used to create them have
been described as opaque. Binder and colleagues proposed an intuitive embedding
space where each dimension is based on one of 65 core semantic features.
Unfortunately, the space only exists for a small dataset of 535 words, limiting
its uses. Previous work (Utsumi, 2018, 2020, Turton, Vinson & Smith, 2020) has
shown that Binder features can be derived from static embeddings and
successfully extrapolated to a large new vocabulary. Taking the next step, this
paper demonstrates that Binder features can be derived from the BERT embedding
space. This provides contextualised Binder embeddings, which can aid in
understanding semantic differences between words in context. It additionally
provides insights into how semantic features are represented across the
different layers of the BERT model.
