Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Models (2023)

Abstract: Pretrained language models are expected to effectively map input text to a set of vectors while preserving the inherent relationships within the text. Consequently, designing a white-box model to compute metrics that reflect the presence of specific internal relations in these vectors has become a common approach for post-hoc interpretability analysis of pretrained language models. However, achieving interpretability in white-box models and ensuring the rigor of metric computation becomes challenging when the source model lacks inherent interpretability. Therefore, in this paper, we discuss striking a balance in this trade-off and propose a novel line to constructing metrics for understanding the mechanisms of pretrained language models. We have specifically designed a family of metrics along this line of investigation, and the model used to compute these metrics is referred to as the tree topological probe. We conducted measurements on BERT-large by using these metrics. Based on the experimental results, we propose a speculation regarding the working mechanism of BERT-like pretrained language models, as well as a strategy for enhancing fine-tuning performance by leveraging the topological probe to improve specific submodules.

## introduction pretrained language models consisting of stacked transformer blocks @xcite are commonly expected to map input text to a set of vectors, such that any relationship in the text corresponds to some algebraic operation on these vectors. however, it is generally unknown whether such operations exist. therefore, designing a whitebox model that computes a metric for a given set of vectors corresponding to a text, which reflects to *equal contribution. on the other hand, due to the intuition that 'if a white-box model cannot distinguish embeddings that exhibit practical differences (such as context embeddings and static embeddings), it should be considered ineffective,' experimental validation of a white-box model's ability to effectively distinguish between embeddings with evident practical distinctions is a common practice in research. furthermore, if the magnitude of metrics computed by a white-box model strongly correlates with the quality of different embeddings in practical settings, researchers usually trust its effectiveness. therefore, in practice, traditional white-box models actually classify sets of vectors from different sources. taking the structural probe proposed by hewitt and manning as an example, they perform a linear transformation on the embedding of each complete word in the text and use the square of the l2 norm of the transformed vector as a prediction for the depth of the corresponding word in the dependency tree @xcite . in this way, the linear transformation matrix serves as a learning parameter, and the minimum risk loss between the predicted and true depths is used as a metric. intuitively, the smaller the metric is, the more likely the embedding contains complete syntax relations. the experimental results indeed align with this intuition, showing that contextual embeddings (such as those generated by bert @xcite ) outperform static embeddings. however, due to the unknown nature of the true deep distribution, it is challenging to deduce which geometric features within the representations influence the magnitude of structural probe measurements from the setup of structural probe. overall, while the results of the structural probe provide an intuition that contextual embeddings, such as those generated by bert, capture richer syntactic relations than those of the traditional embeddings, it is currently impossible to know what the geometric structure of a "good" embedding is for the metric defined by the structural probe. in addition, to enhance the interpretability and flexibility of white-box models, it is common to include assumptions that are challenging to empirically validate. for example, ethayarajh proposed to use anisotropy-adjusted self-similarity to measure the context-specificity of embeddings @xcite . since the computation of this metric doesn't require the introduction of additional human labels, it is theoretically possible to conduct further analysis, such as examining how fundamental geometric features in the representation (e.g., rank) affect anisotropy-adjusted self-similarity, or simply consider this metric as defining a new geometric feature. overall, this is a metric that can be discussed purely at the mathematical level. however, verifying whether the measured context-specificity in this metric aligns well with context-specificity in linguistics, without the use of, or with only limited additional human labels, may be challenging. additionally, confirming whether the model leverages the properties of anisotropy-adjusted self-similarity during actual inference tasks might also be challenging. there appears to be a trade-off here between two types of metrics: therefore, we propose a new line that takes traditional supervised probes as the structure of the white-box model and then self-supervises it, trying to preserve both of the abovementioned properties as much as possible. the motivation behind this idea is that any feature that is beneficial for interpretability has internal constraints. if a certain feature has no internal constraints, it must be represented by a vector set without geometric constraints, which does not contain any interpretable factors. therefore, what is important for interpretability is the correspondence between the internal constraints of the probed features and the vector set, which can describe the geometric structure of the vector set to some extent. in the case where the internal constraints of the probed features are well defined, a probe that detects these features can naturally induce a probe that detects the internal constraints, which is self-supervised. in summary, the contributions of this work include: according to the experimental results, we propose a speculation regarding the working mechanism of a bert-like pretrained language model. ## related work the probe is the most common approach for associating neural network representations with linguistic properties @xcite . this approach is widely used to explore part of speech knowledge @xcite @xcite and for sentence and dependency structures @xcite @xcite @xcite . these studies demonstrate many important aspects of the linguistic information are encoded in pretrained representations. however, in some probe experiments, researchers have found that the probe precision obtained by both random representation and pretrained representation were quite close @xcite . this demonstrates that it is not sufficient to use the probe precision to measure whether the representations contain specific language information. to improve the reliability of probes, some researchers have proposed the use of control tasks in probe experiments @xcite . in recent research, lovering et al. realized that inductive bias can be used to describe the ease of extracting relevant features from representations. immer et al. further proposed a bayesian framework for quantifying inductive bias with probes, and they used the model evidence maximum instead of trivial precision. ## methodology as the foundation of the white-box model proposed in this paper is built upon the traditional probe, we will begin by providing a general description of the probe based on the definition presented in @xcite . additionally, we will introduce some relevant notation for better understanding. ## experiments in this section, delve into a range of experiments conducted on the tree topological probe, along with the underlying motivations behind them. to accommodate space limitations, we include many specific details of the experimental settings in appendices c and d. moreover, we focus our experiments on bert-large and its submodules. moreover, conducting similar experiments on other models is also straightforward (refer to appendix f for supplementary results of experiments conducted using roberta-large). e2. measurement of the vector set formed solely by token embedding. similarly, due to space constraints, we focus on discussing e1 in this paper. the measurement results are shown in tables 1 and 2 . the precise measurement values can be found in appendix e. furthermore, as shown in figure 2 , we present the negative logarithm curves of three measurement values as a function of m i variation. by examining the experimental results presented above, we can ascertain the following findings: f1. x ssp and x essp indeed bound the actual x sp, and for m 14 to m 18 , their true x sp are very close to their x ssp. based on the above findings, we can confidently draw the following rigorous conclusions: c1. based on f1, we can almost infer that dep @xmath0 for m 14 to m 18 . this implies that they memorize the preferences of the real data and minimize as much as possible to approach the theoretical boundary. building upon f5, we can further conclude that the cost of memorizing dep w is an increase in x ssp, which leads to a decrease in the accuracy of the embedding's linear encoding for tree structures. from the above conclusions, we can further speculate about the workings of pretrained language models such as bert, and we identify some related open problems. based on c5 and c6, we can speculate that the final layer of a pretrained language model needs to consider language information at various levels, but its memory capacity is limited. therefore, it relies on preceding submodules to filter the information. the earlier submodules in the model encode the most generic (unbiased) structures present in the language features. as the model advances, the intermediate submodules start incorporating preferences for general structures based on actual data. once a certain stage is reached, the later submodules in the model start to loosen their encoding of generic structures. however, due to the preference information passed from the intermediate submodules, the later submodules can still outperform the earlier submodules in encoding real structures, rather than generic ones. based on c3 and c6, it appears that true x @xmath1. this suggests that for bert, unbiased x sp serves as a tighter upper bound for x sp, and there exists a submodule that achieves this upper bound. now, the question arises: is this also the case for general pretrained models? if so, what are the underlying reasons? 4.2 using x ssp and x essp as regularization loss in fine-tuning bert let us denote the downstream task loss as t (m 24 ). taking x ssp as an example, using x ssp as a regularizing loss during fine-tuning refers to replacing the task loss with: based on f1, we can infer that m 14 to m 18 are not suitable as enhanced submodules. according to c5, the submodules most likely to improve finetuning performance after enhancement should be near m 8 . we conducted experiments on a singlesentence task called the corpus of linguistic acceptability (cola) @xcite , which is part of the general language understanding evaluation (glue) benchmark @xcite . the test results are shown in ## conclusion consider a thought experiment where there is a planet in a parallel universe called "vzjgs" with a language called "vmtprhs". like "english", "vmtprhs" comprises 26 letters as basic units, and there is a one-to-one correspondence between the letters of "vmtprhs" and "english". moreover, these two languages are isomorphic under letter permutation operations. in other words, sentences in "english" can be rearranged so that they are equivalent to sentences in "vmtprhs", while preserving the same meaning. if there were models like bert or gpt in the "vzjgs" planet, perhaps called "yvjig" and "tlg," would the pretraining process of "yvjig" on "vmtprhs" be the same as bert's pretraining on "english"? in theory, there should be no means to differentiate between these two pretraining processes. for a blank model (without any training), extracting useful information from "vmtprhs" and "english" would pose the same level of difficulty. however, it is true that "vmtprhs" and "english" are distinct, with the letters of "vmtprhs" possibly having different shapes or being the reverse order of the "english" alphabet. therefore, we can say that they have different letter features, although this feature seems to be a mere coincidence. in natural language, there are many such features created by historical contingencies, such as slang or grammatical exceptions. hence, when we aim to interpret the mechanisms of these black-box models by studying how language models represent languagespecific features, we must consider which features are advantageous for interpretation and what we ultimately hope to gain from this research. this paper presents a thorough exploration of a key issue, specifically examining the articulation of internal feature constraints. by enveloping the original feature within a feature space that adheres to such constraints, it is possible to effectively eliminate any unintended or accidental components. within this explicitly defined feature space, metrics such as x ssp and x essp can be defined. we can subsequently examine the evolution of these metrics within the model to gain a deeper understanding of the encoding strategies employed by the model for the original feature, as described in the experimental section of this paper. once we understand the encoding strategies employed by the model, we can investigate the reasons behind their formation and the benefits they bring to the model. by conducting studies on multiple similar features, we can gain a comprehensive understanding of the inner workings of the black box.