Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification (2024)

Abstract: Analyses of transformer-based models have shown that they encode a variety of linguistic information from their textual input. While these analyses have shed a light on the relation between linguistic information on one side, and internal architecture and parameters on the other, a question remains unanswered: how is this linguistic information reflected in sentence embeddings? Using datasets consisting of sentences with known structure, we test to what degree information about chunks (in particular noun, verb or prepositional phrases), such as grammatical number, or semantic role, can be localized in sentence embeddings. Our results show that such information is not distributed over the entire sentence embedding, but rather it is encoded in specific regions. Understanding how the information from an input text is compressed into sentence embeddings helps understand current transformer models and help build future explainable neural models.

## introduction in the quest for understanding transformer-based models, much work has been dedicated to uncover what kind of information is encoded in the model's various layers and parameters. these analyses have provided several enlightening insights: (i) different types of linguistic information -e.g. parts of speech, syntactic structure, named entities -are selectively more evident at different layers of the model @xcite , (ii) subnetworks can be identified that seem to encode particular linguistic functionalities @xcite , and (iii) fine-tuning for specific tasks can be focused on very small subsets of parameters, on different parts of a model's layers @xcite . while these analyses and probes have focused on the insides of the models, mostly their parameters and layers, testing their impact is usually done by using the output of the model, namely token or sentence embeddings, to solve specific tasks. the link between the inside of the model and its outputs is usually not explicitly investigated. we ask several facets of this question here: how are the internally-detected information types and structures reflected in the model's output? and how are arbitrarily long and complex sentences encoded systematically in a fixed-sized vector? understanding what kind of information the sentence embeddings encode, and how, has multiple benefits: (i) it connects internal changes in the model parameters and structure with changes in its outputs; (ii) it contributes to verifying the robustness of models and whether or not they rely on shallow or accidental regularities in the data; (iii) it narrows down the field of search when a language model produces wrong outputs, and (iv) it helps maximize the use of training data for developing more robust models from smaller textual resources. transformer-based models usually use a tokenfocused learning objective, and have a weaker supervision signal at the sentence level -e.g. a next sentence prediction @xcite , or sentence order information @xcite . despite this focus, high performance in a variety of tasks (using raw or fine-tuned sentence embeddings) as well as direct probing shows that sentence representations encode a variety of linguistic information @xcite . on the other hand, direct exploration of bert sentence embeddings has also shown that they contain mostly shallow information, related to sentence length and lexical variation, and that many of their dimensions are correlated, indicating that either information is redundantly encoded, or that not all dimensions encode useful information @xcite . some of this preexisting work assumes that sentence embeddings encode information in an overt manner, for example, each principal component dimension is responsible for encoding some type of information. we adopt the different view that information in sentence embeddings may be encoded in merged layers, in a manner similar to audio signals being composed of overlapping signals of different frequencies. we hypothesize that each such layer may encode different types of information. we aim to test this hypothesis and check (i) whether we can separate such layers, and (ii) investigate whether information about specific chunks in a sentence -noun,verb, or prepositional phrases-is encoded in different layers and parts of a sentence embedding. we perform our investigation in an environment with data focused on specific grammatical phenomena, while displaying lexical, structural and semantic variation, and a previously developed system that has been shown to detect the targeted phenomena well @xcite . the system is a variational encoder-decoder, with an encoder that compresses the information in the input into a very low-dimensional latent vector. @xcite have shown that the sentence embeddings, and their compressed representations on the latent layer, encode information about chunks -noun, verb, prepositional phrases -and their linguistic properties. the current study investigates the general hypothesis indicated above by specifically exploring two new research questions in this setting: ## related work sentence embeddings transformer models induce contextual token embeddings by passing the embedding vectors through successive layers using multi-head attention that allows for tokens to influence each other's representation at each successive step @xcite . the model focuses on the token embeddings, as the tokens expected on the output layer provide the training signal. there are numerous variations on the bert @xcite transformer model foot_0 , that vary in the way the models are trained @xcite , how they combine (or not) the positional and token embeddings @xcite , how the input is presented to the model @xcite . with regards to the sentence-level supervision signal, bert @xcite uses the next sentence prediction objective, albert @xcite , aiming to improve coherence, uses sentence order prediction. it is more common to further train or fine-tune a pre-trained model to produce sentence embeddings fitting specific tasks, such as story continuation @xcite or sentence similarity @xcite . electra @xcite does not have a sentence-level objective, but it relies on replaced token detection, which relies on the sentence context to determine whether a (number of) token(s) in the given sentence were replaced by a generator sample. this leads to sentence embeddings that perform well on tasks such as question answering, or detecting verb classes @xcite . probing embeddings and models for linguistic information most work investigating the kind of knowledge captured by transformer-based models have focused on analysing the architecture of the model @xcite to determine the localization and flow of information through the model's layers. there is also much work on analyzing the induced token embeddings to determine what kind of linguistic information they encode, such as sentence structure @xcite , predicate argument structure @xcite , subjecthood and objecthood @xcite , among others. testing whether sentence representation contain specific types of linguistic information has been done using task (or information)-specific classifiers @xcite @xcite . @xcite aim to map subsets of dimensions of fine-tuned sentence embeddings to semantic features. sparsification deep learning models have billions of parameters. this makes them not only incomprehensible, but also expensive to train. the lottery ticket hypothesis @xcite posits that large networks can be reduced to subnetworks that encode efficiently the functionality of the entire network. detecting functional subnetworks can be done a posteriori, over a pre-learned network to investigate the functionality of detected subnetworks @xcite , the potential compositionality of the learned model @xcite , or where task-specific skills are encoded in a fine-tuned model @xcite . instead of learning a sparse network over a prelearned model, @xcite use a pruningbased approach to finding subnetworks in a pretrained model that performs some linguistic task. pruning can be done at several levels of granularity: weights, neurons, layers. their analyses confirm previous investigations of the types of information encoded in different layers of a transformer @xcite . @xcite introduce the automatic circuit discovery (acdc) algorithm, which adapts subnetwork probing and head importance score for pruning to discover circuits that implement specific linguistic functions. sparsification can also be achieved using l 0 regularization, as the pruning would be done directly during training by encouraging weights to become exactly zero. @xcite ; @xcite , among others, implement solutions to the issue that l 0 regularization is non-differentiable, and test it on image classification. the cited work focuses on the parameters of the model, and sparsification approaches aiming to detect the subnetworks to which specific skills or linguistic information can be ascribed. our focus, instead, is the output of transformer-based models, in particular sentence embeddings, which we investigate using targeted sparsification. ## approach overview we investigate whether we can identify specific sentence properties in sentence embeddings. nastase and @xcite have shown that using an encoder-decoder architecture, sentence embeddings can be compressed into a latent representation that preserves information about chunks in a sentence, and their properties necessary to solve a specific linguistic task. we first test whether we can sparsify this architecture in a targeted manner, such that each region of the sentence embedding contributes a signal to only one unit of the latent layer. this allows us to isolate different parts of the sentence embedding. after establishing that sparsification does not lead to a dramatic drop in performance, we trace back the signal from the latent layer to the sentence embeddings, and test whether we can localize information about how different numbers of chunks, or chunks with different properties, are encoded. in the final step, we use the sparse encoderdecoder sentence compression system as the first in a two-layer system used to solve language taskscalled blackbird language matrices @xcite -that require chunk and chunk properties information. the first layer will compress each sentence into a very small latent vector, and this representation is then used on the second layer to solve a pattern detection problem that relies on information about chunks in a sentence and their pattern across a sequence of sentences. ## data we use two data types: (i) a dataset of sentences with known chunk structure and chunk properties, (ii) two datasets representing two multiple-choice problems, whose solution requires understanding the chunk structure and chunk properties of the sentences in each instance. ## experiments we present a progression of experiments. ## conclusions our aim was to understand how information is encoded in sentence embedding, given that previous work has shown that various types of linguistic information is encoded in a model's layers and parameters. we investigated this question using a dataset of sentences with specific chunk structure, and two multiple-choice problems that require information about sentence chunks and their properties to be solved successfully. we have shown that using a sparsified encoder-decoder system, the sentence representations can be compressed into a latent layer that encodes chunk structure properties. we then traced back the signal from the latent layer to the sentence embedding, to detect which areas of the sentence embeddings change the most when comparing sentences with different chunk patterns. this analysis shows that such information is captured by a small number of channel-sentence area combinations. further experiments with the two multiple-choice tasks have confirmed that chunk information and their grammatical properties (for the agreement blm) and chunk information and their semantic role properties (for the verb alternation blm) are captured by the sparsified sentence compression level. we envision further analyses to see where the differences between chunk patterns that have different semantic roles are encoded, and get closer to decoding the sentence embeddings.