  How does word frequency in pre-training data affect the behavior of
similarity metrics in contextualized BERT embeddings? Are there systematic ways
in which some word relationships are exaggerated or understated? In this work,
we explore the geometric characteristics of contextualized word embeddings with
two novel tools: (1) an identity probe that predicts the identity of a word
using its embedding; (2) the minimal bounding sphere for a word's
contextualized representations. Our results reveal that words of high and low
frequency differ significantly with respect to their representational geometry.
Such differences introduce distortions: when compared to human judgments, point
estimates of embedding similarity (e.g., cosine similarity) can over- or
under-estimate the semantic similarity of two words, depending on the frequency
of those words in the training data. This has downstream societal implications:
BERT-Base has more trouble differentiating between South American and African
countries than North American and European ones. We find that these distortions
persist when using BERT-Multilingual, suggesting that they cannot be easily
fixed with additional data, which in turn introduces new distortions.
