Are there identifiable structural parts in the sentence embedding whole? (2024)

Abstract: Sentence embeddings from transformer models encode much linguistic information in a fixed-length vector. We investigate whether structural information – specifically, information about chunks and their structural and semantic properties – can be detected in these representations. We use a dataset consisting of sentences with known chunk structure, and two linguistic intelligence datasets, whose solution relies on detecting chunks and their grammatical number, and respectively, their semantic roles. Through an approach involving indirect supervision, and through analyses of the performance on the tasks and of the internal representations built during learning, we show that information about chunks and their properties can be obtained from sentence embeddings.

## introduction transformer architectures compress the information in a sentence -morphological, grammatical, semantic, pragmatic -into a fixed-length onedimensional array of real numbers. sentence embeddings, usually fine-tuned, have proven useful for a variety of high-level language processing tasks, such as the glue tasks @xcite , or story continuation @xcite ). these results, however, do not shed light on what kind of semantic or structural information is encoded in these representations. understanding what kind of information is encoded in the sentence embeddings, and how it is encoded, has multiple benefits. it connects internal changes in the model parameters and structure with changes in its outputs. it contributes to verifying the robustness of models and whether or not they rely on shallow or accidental regularities in the data. it narrows down the field of search when a language model produces wrong outputs, and ultimately it may help maximize the use of training data for developing more robust models from smaller textual resources. investigation, or indeed, usage, of raw (i.e. not fine-tuned) sentence embeddings obtained from a transformer model are rare, possibly because most transformer models do not have a strong supervision signal on the sentence embedding. using pca analysis, @xcite have shown that the dimensions of bert sentence embeddings have much correlation and redundancy, and encode more shallow information (length), rather than morphological, syntactic or semantic features. analysis of information propagation through the transformer layers seem to show that specialized information -e.g. pos, syntactic structure -while quite apparent at lower levels, gets lost towards the highest levels of the models @xcite , while there are subnetworks that encode specific linguistic functions @xcite . while previous work has regarded network nodes or embedding dimensions as the unit of analysis, @xcite show that superposition -whereby each unit, i.e. neuron or embedding dimension, can be involved in the encoding of multiple features -occurs in artificial neural networks. such features involving overlapping sets of nodes can be learned from a model using sparse autoencoders (e.g. @xcite ). starting from a similar hypothesis relative to the dimensions of a sentence embedding, we aim to test whether specific information, in particular chunks -noun, verb and prepositional phrases, that may play different structural and semantic roles -can be detected in the sentence representation. we use an encoderdecoder architecture applied to data with specific properties, and verify that, through indirect supervision, we can distill information about chunks and their task-relevant properties from sentence embeddings from a pre-trained transformer model. besides being practically useful, as they provide useful shallow structure more easily obtainable than detailed syntactic analysis @xcite , chunks have psychological plausibility @xcite . this motivated us to test whether they are detectable in sentence embeddings, as they would provide syntactically and semantically useful building blocks for assembling higher level information about a sentence. the code and data are available at https://github. com/clcl-geneva/blm-snfdisentangling . ## related work how is the information from a textual input encoded by transformers? there are three main approaches to answer this question: (i) tracing specific information from input to output through the model's various layers and components, (ii) isolating subsets of model parameters that encode specific linguistic functions and (iii) investigating the generated embeddings through probes, using purposefully built data for different types of testing. tracing information through a transformer @xcite have shown that from the unstructured textual input, bert @xcite is able to infer pos, structural, entity-related, syntactic and semantic information at successively higher layers of the architecture, mirroring the classical nlp pipeline @xcite . further studies have shown that the information is not sharply separated, information from higher levels can influence information at lower levels, such as pos in multilingual models (de vries et al., 2020), or subject-verb agreement @xcite . surface syntactic and semantic information seem to be distributed throughout bert's layers @xcite . attention is part of the process, as it helps encode various types of linguistic information @xcite , syntactic dependencies @xcite , grammatical structure @xcite , and can contribute towards semantic role labeling @xcite . ## overview our approach is also a kind of probe. it uses indirect supervision, though, to avoid the shallow learning of a classifier and datasets with specific structure to test for structural information in sentence embeddings. our main object of investigation are chunks, sequence of adjacent words that segment a sentence, as defined initially in @xcite ; @xcite and @xcite . we use two types of data. we use sentences with known chunk patterns (section 4.1), to determine whether chunks and their grammatical properties are identifiable in sentence embeddings with indirect supervision (section 5). we also use two datasets with multi-level structure built for linguistic intelligence tests for language models (merlo, 2023) (section 4.2), to determine whether a system can detect syntactic and semantic structure and information in sentence embeddings based on the requirements of a task. the data, with its repetitive patterns, and the vae-based system support an indirect supervision approach: the system is not given the patterns to be discovered explicitly, but it needs to find them based on the contrasting answer sets at both the sentence and task levels. this indirect supervision process, together with lexical and structural variations in the data, helps to avoid, at least partly, the critiques against probes based on classification, which can learn a task based on 'artefacts' of the data, regularities different from what is intended @xcite . ## data we use data consisting of stand-alone sentences with specific structure, and data consisting of sentences with specific structure and other attributes in larger contexts, to test whether this regular information can be detected. ## experiments we build upon @xcite , and use as sentence representations the embedding of the [cls] special token from a pretrained electra model @xcite foot_1 reshaped as a twodimensional array. we chose electra because it has a stronger sentence-level supervision signal as well as strong results on multiple nlu tasks (see section 2). in section 5.1.3, we show how it compares to other pretrained models. the blm tasks have been benchmarked using ffnn and cnn systems which directly predict the correct answer based on the input sequence @xcite . results improve on both tasks when using a variational encoderdecoder that compresses the input sequence into a very small vector on the latent layer @xcite . this previous work, and similarity of the blm tasks with the visual raven progressive matrices task, have led us to a two-step investigation process: (i) using sentences and a vae-based system, we test whether we can compress sentences into a smaller representation on the latent layer that captures information about the chunk structure of the sentence (section 5.1 below); (ii) to see if the system can detect and extract the kind of information relevant to a specific task, we combine the compression of the sentence representation with the blm problems, where a crucial part of the solution lies in identifying the structures of sentences and their sequence in the input (section 5.2 below). this two-step approach to solving a blm problem fits with the way humans solve the visual rpm problems from which the blms are inspired: (i) identify the relevant objects and their attributes; (ii) decompose the main problem into subproblems, based on object and attribute identification, in a way that allows detecting the global pattern or underlying rules @xcite . ## conclusions sentence embeddings obtained from transformer models are compact representations, compressing much knowledge -morphological, grammatical, semantic-, expressed in text fragments of various length, into a vector of real numbers of fixed length. we can separate this representation into different layers using a convolutional neural network and distinguish specific information among these layers. in particular, we have shown that we can detect information about chunks -noun/verb/prepositional phrases -and their task-relevant attributes, without providing direct supervision to the system about the targeted structures. this brings us one step closer to understanding and unpacking transformer-based sentence embeddings.