U niversal D ependencies A ccording to BERT : B oth M ore S pecific and M ore G eneral (2020)

Abstract: This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully applied with only a minimal amount of supervision and generalizes well across languages.

## introduction and related work in recent years, systems based on transformer architecture achieved state-of-the-art results in language modeling @xcite and machine translation @xcite . additionally, the contextual embeddings obtained from the intermediate representation of the model brought improvements in various nlp tasks. multiple recent works try to analyze such latent representations @xcite , observe syntactic properties in some transformer self-attention heads, and extract syntactic trees from the attentions matrices @xcite @xcite . in our work, we focus on the comparative analysis of the syntactic structure, examining how the bert self-attention weights correspond to universal dependencies (ud) syntax @xcite . we confirm the findings of @xcite and @xcite that in trans-former based systems particular heads tend to capture specific dependency relation types (e.g. in one head the attention at the predicate is usually focused on the nominal subject). we extend understanding of syntax in bert by examining the ways in which it systematically diverges from standard annotation (ud). we attempt to bridge the gap between them in three ways: finally, we apply our observations to improve the method of extracting dependency trees from attention ( §5), and analyze the results both in a monolingual and a multilingual setting ( §6). our method crucially differs from probing @xcite @xcite . we do not use treebank data to train a parser; rather, we extract dependency relations directly from selected attention heads. we only employ syntactically annotated data to select the heads; however, this means estimating relatively few parameters, and only a small amount of data is sufficient for that purpose ( §6.1). ## models and data we analyze the uncased base bert model for english, which we will refer to as enbert, and the uncased multilingual bert model, mbert, for english, german, french, czech, finnish, indonesian, turkish, korean, and japanese 1 . the code shared by @xcite 2 substantially helped us in extracting attention weights from bert. to find syntactic heads, we use: 1000 europarl multi parallel sentences @xcite for five european languages, automatically annotated with udpipe ud 2.0 models @xcite ; google universal dependency treebanks (gsd) for indonesian, korean, and japanese @xcite ; the ud turkish treebank (imst-ud) @xcite . we use another pud treebanks from the conll 2017 shared task for evaluation of mbert in all languages @xcite 3 . ## adapting ud to bert since the explicit dependency structure is not used in bert training, syntactic dependencies captured in latent layers are expected to diverge from annotation guidelines. after initial experiments, we have observed that some of the differences are systematic (see ud modified example copula attaches to a noun copula is a root. 4 cat is an animal root cop nsubj nsubj root obj expletive is not a subject expletive is treated as a subject there is a spoon expl nsubj nsubj obj in multiple coordination, all conjuncts attach to the first conjunct conjunct attaches to a previous one apples , oranges and pears conj conj conj conj bert syntax, using udapi foot_1 @xcite . the main motivation of our approach is to get trees similar to structures emerging from bert, which we have observed in qualitative analysis of attention weights. we note that for copulas and coordinations, bert syntax resembles surfacesyntactic ud (sud) @xcite . nevertheless, we decided to use our custom modification, since some systematic divergences between sud and the latent representation occur as well. it is not our intention to compare two annotation guidelines. a comprehensive comparison between extracting ud and extracting sud trees from bert was performed by @xcite . however, they used a probing approach, which is noticeably different from our setting. ## head ensemble in line with @xcite and other studies @xcite ; @xcite , we have noticed that a specific syntactic relation type can often be found in a specific head. additionally, we observe that a single head often captures only a specific aspect or subtype of one ud relation type, motivating us to combine multiple heads to cover the full relation. figure 1 shows attention weights of two syntactic heads (right columns) and their average (left column). in the top row (purple), both heads identify the parent noun for an adjectival modifier: head 9 in layer 3 if their distance is two positions or less, head 10 in layer 7 if they are further away (as in "a stable , green economy"). similarly, for an object to predicate relation (blue bottom row), head 9 in layer 7 and head 8 in layer 3 capture pairs with shorter and longer positional distances, respectively. ## dependency tree construction to extract dependency trees from self-attention weights, we use a method similar to @xcite , which employs a maximum spanning tree algorithm @xcite and uses gold information about the root of the syntax tree. we use the following steps to construct a labeled dependency tree: it is important to note that the total number of heads used for tree construction can be at most 4 * 12 * 2 = 96, (number of heads per ensemble * number of considered labels * two directions). however, the number of used heads is typically much lower (see as far as we know, we are first to construct labeled dependency trees from attention matrices in transformer. moreover, we have extended the previous approach by using an ensemble of heads instead of a single head. ## multipurpose heads in this experiment, we examine whether a single mbert's head can perform multiple syntactic functions in a multilingual setting. we choose an ensemble for each syntactic relation for each language. figure 4 presents the sizes of intersections between head sets for different languages and dependency labels. except from japanese, we observe an overlap of the heads pointing to the governor of adjective modifiers, auxiliaries, and determiners. shared heads tend to find the root of the syntactic phrase. interestingly, common heads occur even for relations typically belonging to a verb and noun phrases, such as auxiliaries and adjective modifiers. in our other experiments, we have noticed that these heads do not focus their attention on any particular part of speech. similarly, objects and noun modifiers share at least one head for all languages. they have a similar function in a sentence; however, they connect with the verb and noun, respectively. such behavior was also observed in a monolingual model. figure 5 presents attention weights of two heads that belong to the intersection of the adjective modifier, auxiliary, and determiner dependent to parent ensembles. ## conclusion we have expanded the knowledge about the representation of syntax in self-attention heads of the transformer architecture. we modified the ud annotation to fit the bert syntax better. we analyzed the phenomenon of information about one depen- our method of head ensembling improved the previous results for dependency relation retrieval and extraction of syntactic trees from self-attention matrices. as far as we know, this is the first work that conducted a similar analysis for languages other than english. we have shown that the method generalizes well across languages, especially those following subject verb object order. we also hypothesize that the proposed method could improve dependency parsing in a low supervision setting.