Why Bother with Geometry? On the Relevance of Linear Decompositions of Transformer Embeddings (2023)

Abstract: A recent body of work has demonstrated that Transformer embeddings can be linearly decomposed into well-defined sums of factors, that can in turn be related to specific network inputs or components. There is however still a dearth of work studying whether these mathematical reformulations are empirically meaningful. In the present work, we study representations from machine-translation decoders using two of such embedding decomposition methods. Our results indicate that, while decomposition-derived indicators effectively correlate with model performance, variation across different runs suggests a more nuanced take on this question. The high variability of our measurements indicate that geometry reflects model-specific characteristics more than it does sentence-specific computations, and that similar training conditions do not guarantee similar vector spaces.

## introduction it stands to reason that important research efforts are being devoted to explaining the behavior and understanding the mechanics of transformer-based nlp models: most models that are currently discussed within the nlp community are based on this architecture, and they have achieved resounding successes. one trend of work in particular attempts to characterize transformer models by means of their geometry @xcite @xcite . however, most studies focus on a single handful of 'foundation' models or fine-tuned variants thereof, and explicitly or implicitly assume that the reported results generalize on to other models-yet the effects of random initialization, training data or variation in the definition of objective functions are left unstudied. moreover, and perhaps more crucially, there is no guarantee that transformerembeddings geometry is indicative of model quality: that embeddings are arranged in a certain fash-ion in hyperspace says little of what downstream performance we should expect. taken together, these two assumptions-that results applicable to one model will apply to many, and that geometry can provide explanations-call into question the validity of geometry-based approaches. we focus on two linear decomposition approaches for transformer embeddings @xcite : works attempting to summarize model computation through their effects on the resulting output embedding spaces. by construction, these decompositions reflect topological features of the transformer architecture. our goal is to verify whether these two assumptions are in fact supported. one would hope, for instance, that the geometry a model settles on differs along training data but not random initialization. another natural expectation to have is that different uses of the model, such as forcing the production of a given sentence or searching for a plausible generation would yield distinct computations and therefore distinct geometric arrangements. lastly, if we intend to explain model performance via embedding geometry, then we should observe consistent differences in geometry whenever we see differences in quality metrics. to answer whether all three of these expectations are met, we experiment with machine-translation decoder embeddings and study how their geometry evolves over training and multilingualism. in a nutshell, our experiments suggest a nuanced outlook on the usefulness of linear decompositions. decomposition-derived indicators tend to correlate well with corpus-level model performance, but are less appropriate when it comes to sentence-level performance. we also observe that variation in geometry across different runs for a same translation task can exceed what we observe for models trained for different translation tasks. our findings question the relevance of geometrybased approaches for transformer model explain-ability. as our measurements display high variability across different model training runs, this work suggests that geometry reflects model-specific characteristics more than it does sentence-specific computations: models trained in similar conditions need not yield similar vector spaces. ## related works there is a rich literature that connects the objective of static embedding models such as word2vec to characteristics of the resulting vector space. in particular, @xcite worked out how the linguistic regularities found by @xcite result from the exact loss landscape. as for contextual embeddings, research has been more commonly limited to empirical observations @xcite . recently, @xcite ; @xcite ; @xcite ; @xcite ; @xcite and others have developed methods to provide mechanistic interpretations of transformer outputs: these works rely on linear algebra to derive mathematically exact attributions, where a contextual embedding is decomposed as a sum of interpretable vector terms. these approaches build upon two peculiarities of the transformer architecture. perhaps the most famous one-at least, one that has found significant traction more generally across explainable nlp-is that of the scaled dot attention mechanism. transformers were presented by @xcite as attention-only models. attention mechanisms can be seen as weighted sums over value vectors @xcite , where the attention weights are derived non-linearly. this observation was first brought forth within the sustained and ongoing discussion about the relevance of attention weights, and whether they are efficient means of explaining transformer behaviors-a subject hotly debated @xcite @xcite . in particular, @xcite ; @xcite highlight the importance of considering the full embedding space geometry. the second characteristic of importance to transformer decomposition approaches is the systematic use of residual connections throughout a transformer models, only interrupted by layer normalization operations. this fact, often described as a residual stream of information, has been leveraged to interpret the behavior of feed-forward sub-layers @xcite @xcite or layer commutativity @xcite . given that on the one hand a layer norm is a linear map, and on the other hand a residual connection simply consists on adding a sub-network's input to its output, this entails that most of the computations done in a transformer are distributive. ## methodology our focus here is on sequence-to-sequence encoderdecoder architectures @xcite . transformer embeddings can be decomposed into a linear combination of nonlinear transformations using properties of the residual connections and attention mechanisms. here, we focus on whether these decompositions do provide meaningful explanations, or whether they merely reflect topological characteristics of the transformer architecture. foot_0 ## conclusions we have presented a series of statistical studies questioning the usefulness of linear decomposition approaches. in particular, we have highlighted that straightforward vector space characteristics, such as angle and norm of the derived vector terms, imply the following three points: (i) decompositions are invariant to the decoding algorithm employed; (ii) they are more in line with corpus-level performance than sentence-level performance, and (iii) variance across random seeds for the same training conditions is often comparable to variance across models trained on different corpora. taken together, our experiments suggest that transformer geometry is often highly model-specific. observations about a specific model need not generalize. as such, some of the assumptions underlying geometry-based explanations of transformer behaviors are not borne out. while it is true that the geometry of successful models differs from that of unsuccessful ones, our work puts forth evidence that this difference is mostly trivial-geometry being model-specific necessarily entails that any partition of models, be it based on performance or else, will naturally highlight differences. while our focus has been limited to linear decompositions and straightforward vector characteristics, our experiments more broadly call into question the validity of many related approaches, which we hope to investigate in future work. that straightforward vector characteristics do not yield a coherent picture a minima entails that linear decomposition approaches have to rely on nonstraightforward, high-dimensional relationships. that similar training conditions cannot guarantee similar vector spaces naturally leads us to doubt the generalization power of methodologies that probe a handful of foundational models: if we are unable to ensure that our approaches would generalize to other similar models, can we truthfully say that the explanations we provide are indeed reasonable? slavic bel bos bul ces cnr csb dsb hbs hrv hsb mkd pol slk slv srp szl ukr indo-european afr anp arg asm ast bar ben bis bre bzj cat ckb cor cos crs cym dan deu div djk dty ell fao fas fra frp fry fur gla gle glg glv guj hat hin hne hye hyw ind isl ita jak kas kea kmr kri kur lad lav lij lim lit lmo ltg ltz mai mar mfe min mol msa mwl nds nep nld nno nob oci ori oss pan pap pdt pes pis pob por prs pus rmn rmy roh rom ron san scn sco sin spa sqi srd srm srn swe tgk tpi urd vec wae wes wln yid zlm multilingual abk ace ach ada aka alt alz amh ami ara arq ary arz ava aze azz bak bas bbc bci bcl bem bhw bin bod brx bts btx bug bum cab cak ceb cha chk chr chv cjk cmn cnh cop crh ctu dhv dik din dje dua dyu dzo efi epo est eus ewe fas fij fil fin fon ful fuv gil grn guc gug guw gym heb her hil hmn hne hun iba ibg ibo ido ilo ish iso ixl jav jbo jpn kab kac kal kam kan kat kau kaz kbp kek khm kik kin kmb kon koo kqn kss ksw kua kwn lam lao lfn lin loz lua lub lue lug lun luo lus lzh mah mal mam mau meh men mgr mhr mlg mlt mon mos mri mrj mxv mya nan naq nav nba nbl nch ncj ncx ndc nde ndo ngl ngu nia nij niu nso nya nyk nyn nyu nzi oke orm pag plt pon quc rar rnd run sag sat seh ses sid sme smo sna som sop sot ssw sun swa sxn syr tah tam tat tcf tdt tel tgl tha tir tiv tll tmh tog toh toi toj ton trs tsc tsn tso ttj tuk tum tur tvl twi tyv tzh tzo udm uig umb urh uzb ven vie vmw wal war wls wol wuu xho xmf yao yap yor yua yue zai zam zne zpa zul