  We investigate how Multilingual BERT (mBERT) encodes grammar by examining how
the high-order grammatical feature of morphosyntactic alignment (how different
languages define what counts as a "subject") is manifested across the embedding
spaces of different languages. To understand if and how morphosyntactic
alignment affects contextual embedding spaces, we train classifiers to recover
the subjecthood of mBERT embeddings in transitive sentences (which do not
contain overt information about morphosyntactic alignment) and then evaluate
them zero-shot on intransitive sentences (where subjecthood classification
depends on alignment), within and across languages. We find that the resulting
classifier distributions reflect the morphosyntactic alignment of their
training languages. Our results demonstrate that mBERT representations are
influenced by high-level grammatical features that are not manifested in any
one input sentence, and that this is robust across languages. Further examining
the characteristics that our classifiers rely on, we find that features such as
passive voice, animacy and case strongly correlate with classification
decisions, suggesting that mBERT does not encode subjecthood purely
syntactically, but that subjecthood embedding is continuous and dependent on
semantic and discourse factors, as is proposed in much of the functional
linguistics literature. Together, these results provide insight into how
grammatical features manifest in contextual embedding spaces, at a level of
abstraction not covered by previous work.
