Sort by Structure: Language Model Ranking as Dependency Probing (2022)

Abstract: Making an informed choice of pre-trained language model (LM) is critical for performance, yet environmentally costly, and as such widely underexplored. The field of Computer Vision has begun to tackle encoder ranking, with promising forays into Natural Language Processing, however they lack coverage of linguistic tasks such as structured prediction. We propose probing to rank LMs, specifically for parsing dependencies in a given language, by measuring the degree to which labeled trees are recoverable from an LM’s contextualized embeddings. Across 46 typologically and architecturally diverse LM-language pairs, our probing approach predicts the best LM choice 79% of the time using orders of magnitude less compute than training a full parser. Within this study, we identify and analyze one recently proposed decoupled LM—RemBERT—and find it strikingly contains less inherent dependency information, but often yields the best parser after full fine-tuning. Without this outlier our approach identifies the best LM in 89% of cases.

## introduction with the advent of massively pre-trained language models (lms) in natural language processing (nlp), it has become crucial for practitioners to choose the best lm encoder for their given task early on, regardless of the rest of their proposed model architecture. the greatest variation of lms lies in the language or domain-specificity of the unlabelled data used during pre-training (with architectures often staying identical). typically, better expressivity is expected from language/domain-specific lms @xcite while open-domain settings necessitate high-capacity models with access to as much pre-training data as possible. this tradeoff is difficult to navigate, and given that multiple specialized lms (or none at all) are available, practitioners often resort to an ad-hoc choice. in absence of im-mediate performance indicators, the most accurate choice could be made by training the full model using each lm candidate, however this is often infeasible and wasteful @xcite . recently, the field of computer vision (cv) has attempted to tackle this problem by quantifying useful information in pre-trained image encoders as measured directly on labeled target data without fine-tuning @xcite . while first forays for applying these methods to nlp are promising, some linguistic tasks differ substantially: structured prediction, such as parsing syntactic dependencies, is a fundamental nlp task not covered by prior encoder ranking methods due to its graphical output. simultaneously, performance prediction in nlp has so far been studied as a function of dataset and model characteristics @xcite and has yet to examine how to rank large pools of pre-trained lms. given the closely related field of probing, in which lightweight models quantify task-specific information in pre-trained lms, we recast its objective in the context of performance prediction and ask: how predictive is lightweight probing at choosing the best performing lm for dependency parsing? to answer this question, we contribute: ## methodology probing pre-trained lms is highly related to encoder ranking in cv where the ease of recoverability of class-differentiating information is key @xcite . this approach is more immediate than existing nlp performance prediction methods which rely on featurized representations of source and target data without actively ranking encoders @xcite . as most experiments in nlp are conducted using a limited set of lms-often a single model-without strong prior motivations, we see lm ranking as a critical task on its own. while probes for lms come in many forms, they are generally characterized as lightweight, minimal architectures intended to solve a particular task (hall maudslay et al., 2020). while non-linear models such as small multi-layer perceptrons are often used @xcite , there have been criticisms given that their performance highly depends on the complexity of their architecture @xcite . as such, we rely on linear probes alone, which have the benefit of being extremely lightweight, closely resembling existing performance prediction methods @xcite , and allow for statements about linear subspaces contained in lm latent spaces. depprobe (müller-eberstein et al., 2022; visualized in figure 1 ) is a linear formulation for extracting fully labeled dependency trees based on the structural probe by @xcite . given contextualized embeddings of dimensionality d, a linear transformation @xmath0. in our formulation, we supplement a linear transformation @xmath1. by computing the minimum spanning tree in b and then finding the word with the highest root likelihood in l, we can determine the directionality of all edges as pointing away from the root. all remaining edges are labeled according to the most likely non-root class in l, resulting in a fully directed and labeled dependency tree. note that this approach differs substantially from prior approaches which yield undirected and/or unlabeled trees @xcite or use pre-computed edges and non-linear classifiers @xcite . dep-probe efficiently computes the full target metric (i.e. labeled attachment scores) instead of approximate alternatives (e.g. undirected, unlabeled attachment scores or tree depth correlation). for each target language, we employ three multilingual lms-mbert @xcite , xlm-r @xcite , @xcite -as well as 1-3 language-specific lms retrieved by popularity from huggingface's model hub @xcite , resulting in a total of 46 lm-target pair setups (see appendix c). for each combination, we train a depprobe to compute labeled attachment scores (las), hypothesizing that lms from which trees are most accurately recoverable also perform better in a fully tuned parser. to evaluate the true downstream performance of a fully-tuned model, we further train a deep biaffine attention parser (bap; @xcite on each lm-target combination. compared to full fine-tuning, depprobe only optimizes the matrices b and l, resulting in the extraction of labeled trees with as few as 190k instead of 583m trainable parameters for the largest rem-bert model (details in appendix b). ## probing decoupled lms considering depprobe's high predictive performance across lms with varying architecture types, languages/domains and pre-training procedures, we next investigate its limitations: specifically, which differences in @xcite lead to it being measured as an outlier with seemingly low amounts of latent dependency information despite reaching some of the highest scores after full fine-tuning. the architecture has 32 layers and embeddings with @xmath2, compared to most models' 12 layers and @xmath3. it accommodates these size and depth increases within a manageable parameter envelope by using smaller input embeddings with d @xmath4. while choosing different d for the input and output embeddings is not possible in most prior models due to both embedding matrices being coupled, rembert decouples them, leading to a larger parameter budget and less overfitting on the masked language modeling pre-training task @xcite . layer-wise probing prior probing studies have found dependency information to be concentrated around the middle layers of an lm @xcite @xcite . using en-ewt @xcite , we evaluate whether this holds for rembert's new architecture. figure 3 confirms that both dependency structural and relational information are most prominent around layer 17 of 32 as indicated by uuas and relation classification accuracy (relacc) respectively. combining the structural and relational information in depprobe similarly leads to a peak of the las at the same layer while decreasing with further distance from the center. across all target languages, we next investigate whether probing a sum over the embeddings of all layers weighted by α ∈ r 32 can boost extraction performance in rembert. the heavier weighting of middle layers by α, visible in figure 4 , reaffirms a concentration of dependency information in the center. contrasting probing work on prior models @xcite , using all layers does not increase the retrievable dependencies, with las @xmath5. this further confirms that there is not a lack of dependency information in any specific layer, but that there is less within the encoder as a whole. frozen parsing our probing results show that linear subspaces in rembert contain less dependency information than prior lms. however, dep-probe's parametrization is kept intentionally sim-ple and may therefore not be capturing non-linearly represented information that is useful during later fine-tuning. to evaluate this hypothesis, we train a full biaffine attention parsing head, but keep the underlying lm encoder frozen. this allows us to quantify the performance gains which come from inherent dependency information versus later taskspecific fine-tuning. given that depprobe extracts dependency structures reliably from lm architectures with different depths and embedding dimensionalities (e.g. roberta large with 24 layers and @xmath6, optimization and pre-training data, the key difference in rembert appears to be embedding decoupling. the probe's linear formulation is not the limiting factor as the non-linear, biaffine attention head also produces less accurate parses when the lm's weights are frozen. our analyses thus suggest that rembert's decoupled architecture contains less dependency information out-of-the-box, but follows prior patterns such as consolidating dependency information towards its middle layers and serving as strong initialization for parser training. lastly, rembert's larger number of tunable parameters compared to all other lm candidates may provide it further capacity, especially after full fine-tuning. as our probing methods are deliberately applied to the frozen representations of the encoder, it becomes especially important to consider the degree to which these embeddings may change after updating large parts of the model. taking these limitations into account, the high correlations with respect to encoder ranking nonetheless enable a much more informed selection of lms from a larger pool than was previously possible. depprobe to quantify the latent syntactic information via the labeled attachment score. evaluating 46 pairs of multilingual/language-specific lms and nine typologically diverse target treebanks, we found depprobe to not only be efficient in its predictions, with orders of magnitude fewer trainable parameters, but to also be accurate 79-89% of the time in predicting which lm will outperform another when used in a fully tuned parser. this allows for a substantially faster iteration over potential lm candidates, saving hours worth of compute in practice (section 3). our experiments further revealed surprising insights on the newly proposed rembert architecture: while particularly effective for multilingual dependency parsing when fully fine-tuned, it contains substantially less latent dependency information relative to prior widely-used models such as mbert and xlm-r. among its architectural differences, we identified embedding decoupling to be the most likely contributor, while added model capacity during fine-tuning may also improve final performance. our analyses showed that despite containing less dependency information overall, rembert follows prior findings such as structure and syntactic relations being consolidated towards the middle layers. given these consistencies, performance differences between decoupled lms may be predictable using probes, but in absence of similar multilingual lms using decoupled embeddings this effect remains to be studied (section 4). overall, the high efficiency and predictive power of ranking lm encoders via linear probing as well as the ease with which they can be analyzed-even when they encounter their limitations-offers immediate benefits to practitioners who have so far had to rely on their own intuitions when making a selection. this opens up avenues for future research by extending these methods to more tasks and lm architectures in order to enable better informed modeling decisions.