  Since the popularization of the Transformer as a general-purpose feature
encoder for NLP, many studies have attempted to decode linguistic structure
from its novel multi-head attention mechanism. However, much of such work
focused almost exclusively on English -- a language with rigid word order and a
lack of inflectional morphology. In this study, we present decoding experiments
for multilingual BERT across 18 languages in order to test the generalizability
of the claim that dependency syntax is reflected in attention patterns. We show
that full trees can be decoded above baseline accuracy from single attention
heads, and that individual relations are often tracked by the same heads across
languages. Furthermore, in an attempt to address recent debates about the
status of attention as an explanatory mechanism, we experiment with fine-tuning
mBERT on a supervised parsing objective while freezing different series of
parameters. Interestingly, in steering the objective to learn explicit
linguistic structure, we find much of the same structure represented in the
resulting attention patterns, with interesting differences with respect to
which parameters are frozen.
