  We introduce a multilabel probing task to assess the morphosyntactic
representations of word embeddings from multilingual language models. We
demonstrate this task with multilingual BERT (Devlin et al., 2018), training
probes for seven typologically diverse languages of varying morphological
complexity: Afrikaans, Croatian, Finnish, Hebrew, Korean, Spanish, and Turkish.
Through this simple but robust paradigm, we show that multilingual BERT renders
many morphosyntactic features easily and simultaneously extractable (e.g.,
gender, grammatical case, pronominal type). We further evaluate the probes on
six "held-out" languages in a zero-shot transfer setting: Arabic, Chinese,
Marathi, Slovenian, Tagalog, and Yoruba. This style of probing has the added
benefit of revealing the linguistic properties that language models recognize
as being shared across languages. For instance, the probes performed well on
recognizing nouns in the held-out languages, suggesting that multilingual BERT
has a conception of noun-hood that transcends individual languages; yet, the
same was not true of adjectives.
