Challenging the “One Single Vector per Token” Assumption (2023)

Abstract: In this paper we question the almost universal assumption that in neural networks each token should be represented by a single vector. In fact, it is so natural to use one vector per word that most people do not even consider it as an assumption of their various models. Via a series of experiments on dependency parsing, in which we let each token in a sentence be represented by a sequence of vectors, we show that the “one single vector per token” assumption might be too strong for recurrent neural networks. Indeed, biaffine parsers seem to work better when their encoder accesses its input’s tokens’ representations in several time steps rather than all at once. This seems to indicate that having only one occasion to look at a token through its vector is too strong a constraint for recurrent neural networks and calls for further studies on the way tokens are fed to neural networks.

## introduction since the apparition of word2vec in 2013 @xcite , embeddings have become ubiquitous in natural language processing. however, the overwhelming majority of works that use them, use a single vector to represent each token (word or character) in a sequence. we call this monodianysm, from mono-(greek μονος : single) and dianysma (greek διανυσμα : vector). while monodianysm is a very strong assumption, it is hardly ever presented as such, namely, that it is just an assumption and that their could be other possibilities to represent input tokens. this is an especially strong assumption when working with recurrent neural networks (rnn) since by the time they have reached a token, it is already time to move to the next, and thus an rnn encoder only has one chance to extract all the necessary information from the representation of each token. we make the hypothesis that giving encoders more time (in term of computation steps) to extract the relevant information from token representations is beneficial. indeed, while words can easily linger in someone's mind for several minutes and often much longer after having been read or heard, the most frequent flavors of recurrent neural networks only have very limited storage capacity. a long-short term memory unit (lstm @xcite ) has two internal vectors that store information, while a gated recurrent unit (gru @xcite ) has only one such vector. moreover, their internal machinery is too simplistic to allow actual perfect recording of independent words and thus they have to make the best of the information available in both the input representation and their current hidden states right away. furthermore, having a single vector per word 1 prevents their representations from having a temporal structure 2 which could in principle be beneficial to the extraction of information from said word representations by recurrent neural networks. in this paper, we use dependency parsing as a benchmark to test our hypothesis. we conduct two sets of experiments where we train syntactic parsers whose input words representations are either split in one, two, four or eight vectors. in the first set of experiments, word representations are learned from scratch with the parsing loss, while in the second, word representations are taken from a pre-trained large language model. an increase in parsing scores as the number of 1 in this paper we use the terms "word" and "token" quite liberally. since we test our hypothesis on dependency parsing, a "word" should be understood as an actual word or a punctuation symbol (what is usually called a "token"). when necessary we use the term "word" to make it clear that we are speaking of "parsing tokens" and not of "(sub-word) tokens" of modern transformer based language models. this means that in a different context, a "word" could actually be a character or any object we want to pass a representation of to an encoder. the remaining of this paper is organized as follows. in section 2, we present some works that have proposed representations beyond vanilla word embeddings. in section 3, we introduce the idea of stratified vectors and their implications for parsing methods. in section 4, we describe our experimental setting and present the results. in section 5, we discuss some limitations of the present study. in section 6, we draw main directions for future research on stratified vectors and state a number of questions opened by the results presented in section 4. eventually, section 7 closes the present work. ## related work to the best of our knowledge, this paper is the first to question the otherwise universal assumption that each token in a sequence should be represented by a single vector. this being said, other researchers have looked at related yet orthogonal problems about word representations. for example, @xcite proposed a method for learning multiple vectors per word form as a mean to deal with polysemy and homonymy and thus allow words with the same form but different meanings not to interfere with each other's representations. yet, at encoding time, only one embedding from the set of available prototypes is used and thus an encoder still sees a word only once through its chosen vector. more recently, with the emergence of transformer based language models @xcite that use sub-word tokenizer, some words are indeed represented by multiple vectors. however, this is not due to an attempt at giving an internal structure to word representations, but rather this is an artifact appearing from the way they handle rare and out-of-vocabulary words @xcite . furthermore, not all words end up being represented by the same number of vectors and one needs to find proper ways to deal with them when applying those language models to tasks such as part-of-speech tagging or dependency parsing where one needs to predict an output for each of the original words (and punctuation symbols) rather than for the tokenized sub-words for which contextualized representations are computed. in fact, what may actually be the closest to our proposal, if not in design, in potential effect on word representations, is multi-head attention @xcite . indeed, multi-head attention is a way to extract different aspects/views from a single vector. while multi-head attention does not give a temporal structure to word representations (and in fact transformers and attention heads are quite agnostic to position which need to be artificially reintroduce with position embeddings), it can disentangle various relevant aspects of a word, all stored in a single vector, according to a given context. more exotic representations have also been proposed such as gaussian embeddings @xcite or quantum embeddings @xcite in the context of knowledge base representation. but it is unclear at this point how a gaussian distribution (a vector and a covariance matrix) should be passed to an rnn sentence encoder for further processing. in the domain of distributional semantics, @xcite propose to give each word both a content part (a vector) and a functional part (a matrix) and composition is realized as the aggregation of the matrix-vector products for pairs of items in a binary syntactic tree. in works such as those of @xcite ; @xcite and @xcite , words from different parts-of-speech are represented by tensor of varying shapes depending on their valency profiles. nouns for example are vectors while adjectives are matrices since they modify nouns, and verbs can have even higher orders if they are transitive or ditransitive. here again, it is really unclear how such representations would be used in a vanilla rnn architecture, especially so when different words have different shapes. furthermore, in these works, composition is done along the branches of a syntactic tree, which is exactly the structure we want to elicit. moreover, our main goal is to see where challenging the monodianysm assumption can lead us with as little intervention on the actual underlying model's architecture as possible. ## stratified vectors under the monodianysm assumption, rnn encoders have the opportunity to make the best of the vector they are shown only once. if they do not extract the necessary information the first (and only) time they meet a token, they never have a second chance. this may be especially detrimental for a word with a high perplexity given the current encoder's hidden state, since it is likely to be harder for the encoder to extract relevant information from a vector that is unexpected from the context. we thus propose to add an extra dimension to word representations. instead of learning a single vector per word, we propose to learn a sequence of vectors for each word that will always come together. we hypothesize that it will be useful for three main reasons: (i) it allows different aspects of a word to be disentangled in the representation which can be useful for task where words have different roles such as in dependency parsing where a word can be both a dependent and a governor, (ii) since the vectors always come together, if a word is unexpected in a context, while the first vector will have a high perplexity, the following one should have a much smaller one, and thus first vectors could act as a warning mechanism to prepare the encoder to make the best out of the following vectors, and (iii) having more computation step to extract useful information should be beneficial. there are two questions readily appearing when we decide to abandon monodianysm, namely: (i) should every token have the same number of vectors? (ii) should these vectors be the same or different? for this work, we decided to keep the same number of vectors per tokens. indeed, allowing the number of vectors to vary, even on a per word-class basis, would greatly increase the complexity of the learning process. so we give a positive answer to the first question as a simplifying starting point. regarding the second question, from the idea of giving more time to spend on each token to the encoder alone, it could seem natural to simply repeat the same vector several time. however, after having seen the first vector of a given token, the encoder is left in a different state than the one it is was in just a computation step before, and so it might in fact be more interesting to have a different second vector in order to mirror this. furthermore, if we want to be able to disentangle different aspects of a word, it might be necessary to have different vectors. we still perform a small experiment to verify this hypothesis. but then, we should realize that if instead of a single vector of d dimensions, we allow two or more vectors of d dimensions per token, then the number of parameters of our model also increases, and thus its information storing capacity increases too, not only its computation time. in that case, any increase in accuracy could just as well be due to the increase in storing capacity as in the increase in computation time. thus, in order to keep a comparable number of parameters per token, we decided to use k vectors of d k dimensions per token. we call these k vectors the strata of a word's representation. in practice, we use a transposed convolution tensor to turn a vector (a 1 × d matrix) into a k × d k matrix. we thus call the stored vector a stratified vector. using this new representation, a sentence of t tokens will be represented as sequence of td vectors of d k dimensions rather than the usual t vectors of d dimensions. this is depicted in figure 1 . figure 1: a representation of stratified vectors used to represent a sentence of length n. the top row depicts the traditional way of using word embeddings with a single vector of d dimensions per word. the middle row represents a situation where each word is represented by two vectors of d 2 dimensions. in the bottom row, each word is now represented by four vectors of d 4 dimensions. the dashed lines highlight the fact that even though the different strata of a word are trained together and form a single coherent unit, they are read one by one by the rnn. we should note that, since the input vectors are of length d k instead of d, assuming the encoder has the same hidden/output dimension h in both cases, then the matrix used to feed the input vectors to the encoder is of size h d @xmath0. this means, that every other things being equal, the model based on stratified vectors is slightly smaller than the original one, even though marginally so, since in practice most of the memory will be taken by the representations themselves and in the case of a biaffine parse @xcite by the relation label decoder. another non negligible effect of using stratified vectors is the linear increase in time spent in the encoder, since it takes k times longer to process a k time longer input. we also expect training to be slightly more difficult since the loss gradient will need to be back-propagated through k times more recurrent cells. ## experiments we conducted two sets of experiments in order to test our hypothesis. in both cases, we train biaffine style dependency parsers @xcite . the main difference is the source of the word representations fed to the encoders. in the first case, word embeddings are trained from scratch with the parsing loss, while in the second case, we use a frozen pre-trained transformer-based model as feature extractor, namely xlm-roberta @xcite . ## limitations this work is limited in two main regards. first, we only tested our hypothesis on dependency parsing. at this point, it is not clear how this result should apply to other linguistic tasks if at all. since in dependency parsing a word plays several roles (governor and dependent), it could be that having multiple output vectors helps more here than for other tasks. however, early experiments seems to indicate that only having several output vectors per word is not enough to see similar parsing gains. the second limitation is the limited language selection. we only experimented on six languages. while there is nothing inherent about these six languages that should make them more likely to disagree with the monodianysm assumption, it is still possible that stratified vectors are not suitable for all languages. however at this point, there is no strong evidence pointing in that direction and we simply need more work to see how these results do or do not generalize. ## future work these first results open many new avenues for future research and begs for a better understanding of what is actually captured by neural networks and by word embeddings. here we only present a few of the many questions that will need to be answered. first and foremost, we need to understand the information structure of stratified vectors. early probing attempts did not reveal any directly accessible structure, neither inside the stratified vectors themselves nor between the strata of the embedding space. but this may be due to the max-pooling operation that is notoriously oblivious to structure or to the fact that parsing corpora are rather small compared to corpora used to train general language models. so we need to train proper polydianysmatic language models in order to explore their inner structure. since the k vectors of a word always come together, we guess that it reduces the overall perplexity of the underlying language model, as the first vector of a word prepares the model for its successors. we hypothesize that the first vector of a word brings the rnn to a state where it is better able to from a technical standpoint, it is clear that the increase in computation time discussed in section 3 is a major limitation of our proposal. however, this need not be a fatality. if instead of having several vectors for words in isolation, we used compositionally crafted n-gram representations, we could still have information about a given word passed to the encoder for several computation steps while only incurring a additive linear overhead rather than a multiplicative one. for example, instead of representing a sequence abc as [a 1 , a 2 , b 1 , b 2 , c 1 , c 2 ] (with a 1 being the first vector for a and so on) which is twice as long as the original sentence, we could represent it as [f (#ab), f (abc), f (bc#)] (where f is some compositional embedding function and # representing sentence boundaries) which still has every word ap-pearing at least twice and yet has the same length as the original sentence. this needs to be investigated further. we mentioned in section 4.1 that since stratified vectors have a temporal structure, it is not the same to read them in one direction or the other. this becomes a new parameter for rnn that needs to be understood. moreover, we introduce stratified vectors in the context of recurrent neural networks, but if it is the multiple outputs that make them powerful then they could also be applied to transformer type architectures, which as we said earlier are time agnostic. this would beg even further research on the information structure of the embedding spaces and their relation to each other. eventually, regarding dependency parsing more specifically, there are many possibilities for extracting trees from multiple scores beyond max-pooling. we could always use a single fixed cell and thus let the remaining vectors encode any useful information. we could have different biaffine matrices for different cells. we could use the different cells to reconstruct several trees and effectively train several parsers at the same time and then have them vote for example. as we see, the results presented in this paper open a lot of new questions that will need to be answered if we want to make the best of embedding spaces. ## conclusion in this paper, we have introduced the concept of stratified vectors as a way to challenge the ubiqui-tous monodianysm assumption : "one vector per word". via a series of experiments on dependency parsing, using either representations learnt from scratch or extracted from pre-trained language models, we showed that stratified vectors indeed seem useful, at least in the context of graph based parsing with rnn encoders. we then discussed the current limited scope of our results and the necessary questions that need to be answered in order to better challenge the "one vector per word" assumption and the many directions for future research granted by these questions.