Probing for Labeled Dependency Trees (2022)

Abstract: Probing has become an important tool for analyzing representations in Natural Language Processing (NLP). For graphical NLP tasks such as dependency parsing, linear probes are currently limited to extracting undirected or unlabeled parse trees which do not capture the full task. This work introduces DepProbe, a linear probe which can extract labeled and directed dependency parse trees from embeddings while using fewer parameters and compute than prior methods. Leveraging its full task coverage and lightweight parametrization, we investigate its predictive power for selecting the best transfer language for training a full biaffine attention parser. Across 13 languages, our proposed method identifies the best source treebank 94% of the time, outperforming competitive baselines and prior work. Finally, we analyze the informativeness of task-specific subspaces in contextual embeddings as well as which benefits a full parser’s non-linear parametrization provides.

## introduction pre-trained, contextualized embeddings have been found to encapsulate information relevant to various syntactic and semantic tasks out-of-the-box @xcite . quantifying this latent information has become the task of probes -models which take frozen embeddings as input and are parametrized as lightly as possible (e.g. linear transformations). recent proposals for edge probing @xcite and structural probing @xcite have enabled analyses beyond classification tasks, including graphical tasks such as dependency parsing. they are able to extract dependency graphs from embeddings, however these are either undirected @xcite or unlabeled @xcite , thereby capturing only a subset of the full task. in this work, we investigate whether this gap can be filled and ask: can we construct a lightweight probe which can produce fully directed and labeled dependency trees? using these trees, we further aim to study the less examined problem of transferability estimation for graphical tasks, extending recent work targeting classification and regression tasks @xcite . specifically: how well do our probe's predictions correlate with the transfer performance of a full parser across a diverse set of languages? to answer these questions, we contribute dep-probe (figure 1 ), the first linear probe to extract directed and labeled dependency trees while using fewer parameters than prior work and three orders of magnitude fewer trainable parameters than a full parser (section 3). as this allows us to measure labeled attachment scores (las), we investigate the degree to which our probe is predictive of crosslingual transfer performance of a full parser across 13 typologically diverse languages, finding that our approach chooses the best transfer language 94% of the time, outperforming competitive baselines and prior work (section 4). finally, we perform an in-depth analysis of which latent information is most relevant for dependency parsing as well as which edges and relations benefit most from the expressivity of the full parser (section 5). 1 ## related work given the ubiquitous use of contextualized embeddings @xcite @xcite , practitioners have turned to various methods for analyzing their linguistic features @xcite . @xcite examine these intrinsic properties in greater detail for english dependency parsing using a structural probe, finding that undirected dependency graphs are recoverable from bert by learning a linear transformation on its embeddings (section 3.1). extending the structural probe of @xcite to 12 languages, @xcite extract undirected dependency graphs from mbert @xcite , further showing that head-to-child difference vectors in the learned subspace cluster into relations from the universal dependencies taxonomy @xcite . building on both the structural and tree depth probes @xcite , @xcite extract directed dependency graphs from mbert for 13 languages (section 3.2). further variations to structural probing include regularization of the linear transformation @xcite as well as alternative objective functions @xcite . none of the proposed linear probing approaches so far are able to produce full dependency parse trees (i.e. directed and labeled), however the closer a probe approximates the full task, the better it quantifies relevant information @xcite ). it would for example be desirable to estimate las for parsing a target treebank with a model trained on a different source without having to train a resource-intensive parser (e.g. dozat and manning, 2017) on each source candidate. although performance prediction methods for such scenarios exist, they typically do not cover graph prediction @xcite . in order to bridge the gap between full parsers and unlabeled probes, in addition to the gap between full fine-tuning and lightweight performance prediction, this work proposes a linear probe which can extract labeled and directed dependency parse trees while using less compute than prior methods (section 3). we use our probe's las to evaluate its predictive power for full parser performance and leverage its linear nature to investigate how dependencies are represented in subspaces of contextual embeddings (section 5). ## probing for dependencies in order to construct a directed and labeled dependency parse tree for a sentence s consisting of the words {w 0 , . . . , w n }, we require information on the presence or absence of edges between words, the directionality of these edges ( ---→ w i , w j ), and the relationships {r 0 , . . . , r n } which they represent. using the contextualized embeddings {h 0 , . . . , h n } with h @xmath0, prior probing work has focused on the first step of identifying edges (section 3.1) and later directionality (section 3.2). in this work, we propose a probe which completes the final relational step (section 3.3) and simultaneously provides a more efficient method for identifying directionality (section 3.4). ## conclusion with depprobe, we have introduced a novel probing procedure to extract fully labeled and directed dependency trees from untuned, contextualized embeddings. compared to prior approaches which extract structures lacking labels, edge directionality or both, our method retains a simple linear parametrization which is in fact more lightweight and does not require complex decoders (section 3). to the best of our knowledge, this is the first linear probe which can be used to estimate las from untuned embeddings. using this property, we evaluated the predictive power of depprobe on cross-lingual parsing with respect to the transfer performance of a fully fine-tuned biaffine attention parser. across the considered 169 language pairs, depprobe is surprisingly effective: its las correlates significantly (@xmath1.001) and most highly compared with unlabeled probes or competitive language feature baselines, choosing the best source treebank in 94% of all cases (section 4). leveraging the linearity of the probe to analyze structural and relational subspaces in mbert embeddings, we find that dependency relation information is particularly important for parsing performance and cross-lingual transferability, compared to both tree depth and structure. depprobe, which models structure and relations, is able to recover many functional and syntactic relations with competitive accuracy to the full bap (section 5). finally, the substantially higher efficiency of depprobe with respect to time and compute make it suitable for accurate parsing performance prediction. as contemporary performance prediction methods lack formulations for graphical tasks and handcrafted features such as lang2vec are not available in all transfer settings (e.g. document domains, mlm encoder choice), we see linear approaches such as depprobe as a valuable alternative. a p p o s d is lo c a t e d io b j n m o d n s u b j n u m m o d o b j o b l a c l a d v c l c c o m p c s u b j x c o m p a d v m o d a m o d a u x c a s e c o p d e t m a r k c c c o n j fi x e d fl a t p a r a t a x is d e p p u n c t r e f r o o t 0 25 50 75 100 relacc nominal clause modifier function coord multi loose other 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 bap depprobe figure 7: relacc of bap and depprobe on ar-padt (test) grouped according to ud taxonomy. a p p o s e x p l io b j n m o d n s u b j n u m m o d o b j o b l v o c a t iv e a c l a d v c l c c o m p c s u b j x c o m p a d v m o d a m o d d is c o u r s e a u x c a s e c o p d e t m a r k c c c o n j c o m p o u n d fi x e d fl a t li s t p a r a t a x is g o e s w it h r e p a r a n d u m p u n c t r o o t 0 50 100 relacc nominal clause modifier function coord multi loose special other 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 bap depprobe figure 8: relacc of bap and depprobe on en-ewt (test) grouped according to ud taxonomy. a p p o s io b j n m o d n s u b j n u m m o d o b j o b l v o c a t iv e a c l a d v c l c c o m p c s u b j x c o m p a d v m o d a m o d d is c o u r s e a u x c a s e c o p d e t m a r k c c c o n j c o m p o u n d fi x e d fl a t p a r a t a x is d e p p u n c t r o o t 0 25 50 75 100 relacc nominal clause modifier function coord multi loose other 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 bap depprobe figure 9: relacc of bap and depprobe on eu-bdt (test) grouped according to ud taxonomy. a p p o s n m o d n s u b j n u m m o d o b j o b l v o c a t iv e a c l a d v c l c c o m p c s u b j x c o m p a d v m o d a m o d d is c o u r s e a u x c a s e c o p d e t m a r k c c c o n j c o m p o u n d fi x e d fl a t p a r a t a x is o r p h a n p u n c t r o o t 0 25 50 75 100 relacc nominal clause modifier function coord multi loose sp other 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 bap depprobe