Vector Norms as an Approximation of Syntactic Complexity (2023)

Abstract: Internal representations in transformer models can encode useful linguistic knowledge about syntax. Such knowledge could help optimise the data annotation process. However, identifying and extracting such representations from big language models is challenging. In this paper we evaluate two multilingual transformers for the presence of knowledge about the syntactic complexity of sentences and examine different vector norms. We provide a fine-grained evaluation of different norms in different layers and for different languages. Our results suggest that no single part in the models would be the primary source for the knowledge of syntactic complexity. But some norms show a higher degree of sensitivity to syntactic complexity, depending on the language and model used.

## introduction one of the most successful recent developments in nlp is the self-attention mechanism @xcite , which has been used as the underlying operation of recent transformer models @xcite . the success of the transformer models has been wide-spread, from semantic @xcite and syntactic @xcite @xcite tasks, to more pragmatically focused tasks @xcite and multimodal problems @xcite . in this paper we contribute to the research on what makes transformers so successful in learning linguistic knowledge and examine the ability of such models to estimate syntactic complexity of sentences. knowing if a model reacts to the syntactic complexity of sentences is useful because if com-plexity can be estimated without training a model (during inference time), this can be taken into account when fine-tuning the model, allowing for more efficient sampling of batches during training @xcite , curriculum learning strategies @xcite and possibly as an estimator of unsupervised/zero-shot performance. another aspect we consider is whether the syntactic complexity of sentences can be captured in different languages or if it works particularly well for some, as this would allow vector norms to be used as an analysis tool regardless of the language. we examine the transformer representations for presence of linguistic knowledge by first extracting vector norms, as these can be obtained by simply passing a sentence through a pre-trained model without fine-tuning. the norms of vectors have been used to both analyse models @xcite and improve models, for example by using the norm as an indicator of the difficulty of translating a sentence @xcite , or as a way to select informative examples from a corpora @xcite . the l2-norm was also exploited by @xcite to construct dependency trees from the dot-product of word pairs. in this paper we investigate whether the euclidean norm (l2-norm) is an indicator of the syntactic complexity in dependency trees. we consider two distance metrics: dependency and hierarchical distance, and we investigate three sources of l2-norms: cls, root and average over tokens. we finally evaluate the ability of the transformer to estimate syntactic complexity of a sentence by looking at the pearson ρ correlation between vector norms and distance metrics of dependency trees. this will show whether the norm of the representations in a transformer increase or decrease as the syntactic complexity changes. we present an analysis across many languages, and we use only pre-trained large language model representations. this is useful for many reasons, one of them being that it can help inform annotators that a certain sentence is a syntactically complex one, or not, and this can helps annotators recognize difficult sentences during the annotation process. additionally, this work provides another use case for llms, namely as an aid for dataset creators. that is, if a silver-standard is obtained from a model, we still want to identify possible problematic sentences that should be reviewed by a human annotator. by analysing different norms and syntactic complexity in different languages, we also provide valuable information about how the mlm objective of transformer models encode syntactic complexity. in this work we focus on the following questions: the questions we investigate are relevant for better interpretability of neural language models @xcite . we provide insights on whether transformers can be used (without training) as the knowledge source for more efficient subsequent annotation, training and finetuning. we also examine the differences between norms in different model layers concerning many languages. ## dependency distance metrics in this paper we consider syntactic complexity of sentences through the lens of dependency trees. quantifying these dependencies heavily relies on the word order, which can be represented either linearly or hierarchically (structurally). depending on the language, either of the ways is preferred by its speakers and a reciprocal relation between the two can be observed, e.g. czech relies on structural order more than english for longer sentences @xcite . we believe that such fine-grained difference between how languages use syntax is important for the models to capture. therefore, we represent the syntactic complexity of sentences by calculating mean linear dependency distance (mdd) and mean hierarchical dependency distance (mhd). for evaluation we compute correlation between different vector norms and mdd / mhd and treat this intrinsic measure as an indicator of the amount of linguistic knowledge about syntactic complexity that the model encodes. we extract dependency trees from the universal dependency treebanks @xcite . for a tree such as figure 1 , there are two distance metrics that we are interested in, dependency and hierarchical distance. ## vector norms we are interested in whether the vector norms capture behavior trends of syntactic complexity of sentences in different languages. we use the euclidean (l 2 ) norm as our primary method for norm computation. the euclidean norm is defined as follows over a vector v of length n: the representations that are used to compute the sentence vector norm can be taken arbitrarily from different representations inside the model. here we extract three such representations and compute norms for each of them: in our experiments we consider the norms obtained from two different models, multilingual bert (m-bert) @xcite and xlm-roberta (xlm-r) @xcite . for both models, we use the base model provided by the huggingface @xcite library. we compute the pearson correlation @xcite between the dependency distance metrics and the vector norms. we obtain a mean correlation score for each language by averaging scores across all sentences in this language. a high correlation score means that a specific type of vector norm within the particular model's layer reflects bigger distance scores. a low score (or a negative score) indicates that the vector norm and dependency distance are in an opposite trend: when one becomes high, another one gets low. observing differences in the behaviour of different vector norm representations and distance metrics allows us to determine which mean will likely encode which type of knowledge. ## treebanks we use the parallel sentences from the pud corpus @xcite . for each language in the dataset, it contains 1000 sentences. the sentences are the same for all languages and have been translated and annotated by experts. in the pud dataset, dependency trees from the following languages are included: arabic, czech, english, french, german, hindi, icelandic, italian, polish, portuguese, russian, spanish, swedish, thai, chinese, turkish, korean, japanese, indonesian and finnish. the majority of the languages are from the indo-european language family, however, other distant families are also included such as uralic, turkic, austronesian, sino-tibetan, and tai-kadai. the primary attractive feature of the pud dataset for our experiments is that the results we obtain for the different languages are directly comparable because the same sentence is translated. this allows us to reliably ascertain the models ability to encode different dependency tree metrics across languages. we summarize each language's mean dependency and hierarchical distances in ## results a similar result is observed for the base xlm-r model ( ## peeking inside the model's layers we next perform a more fine-grained analysis of the internal models' representations. our goal is to examine to what degree specific layers of the models capture information about the syntactic complexity of sentences. this analysis allows us to narrow down the search for the source of better representations in large language models. the correlations per layer using the m-bert model are shown in figure 4 for mdd and in figure 5 for mhd. similarly, figure 6 and figure 7 show correlations per layer for the xlm-r model. discussion for the m-bert model, mean vector norms have large correlation scores in the first few layers, while in deeper layers, the correlation drops only to spike again in the last layer. this behaviour is approximately identical for all languages, with some having a slightly smaller correlation with mdd across all layers, e.g. thai. the trend is very similar for correlation with mhd: the scores are the highest in approximately the first six layers and smaller in deeper layers except for the last layer, in which the scores are high again. this result mirrors previous findings showing that language models capture syntactic knowledge in earlier layers, which allows them to learn complementary semantic knowledge in deeper layers @xcite @xcite , which could be the reason for a more negligible correlation with syntactic complexity in deeper layers. cls vector norms in the first layer have the highest (positive) correlation with both types of distances across most languages. the correlation is still strong in subsequent layers, although in a different direction (negative), with a substantial reduction in the last layer. some notable exceptions are chinese, japanese and thai, for which the highest correlation is observed in the second or the third layer. after these layers, the correlation effect is smaller than for other languages. root vector norms have a small correlation with the syntactic complexity, suggesting that these norms are not informative enough and richer representations are required. for the xlm-r, correlation scores for mean vector norms vary a lot among the layers. however, the correlation spikes in the second layer and reaches low values in the last layer for many languages, which is opposite to the behaviour of mean vector norms in m-bert. cls vector norms and their correlation with distance metrics similarly varies between layers, but not languages. all languages appear to have similar correlation scores depending on the layer. root vector norms have a very small correlation (positive or negative) for all languages across all layers except the last one. interestingly, xlm-r does not encode the knowledge of syntactic complexity in representations of the root of the sentence, which mirrors results for m-bert. overall, m-bert exhibits much more pronounced differences between layers regarding the syntactic complexity of sentences. xlm-r, on the contrary, is much harder to understand and use for finding parts in the model where a particular norm is the most useful. for m-bert, we recommend using mean vector norms from the first layers to identify the knowledge of syntactic complexity. for xlm-r, cls or mean can be used. neither model has useful root representations. ## discussion norm types and models we find that the root and mean norms produce only weak correlations in the xlm-r model across languages, suggesting that xlm-r better encodes syntactic information about sentences in its cls token rather than in the sub-word tokens themselves. we see an inverse trend for m-bert: syntactic complexity is strongly encoded in the sub-word tokens (mean and root) while not in the cls token. this indicates that concerning encoding syntactic complexity, the xlm-r model performs well in pooling sc information to the cls token during the language modelling training and also reduces indications of syntactic complexity from the subword tokens, whereas in m-bert syntactic complexity is strongly encoded in the sub-word tokens. we suggest that the cls token in xlm-r generally contains more information about syntax than the cls token in m-bert. ## conclusions and future work we conclude by hypothesising how our findings could aid data annotation efforts. we show that for both models, we can identify a specific type of norm that is a relatively strong indicator of syntactic complexity across languages. we see at least two potential uses of such a result. first, it enables dataset creators to use a machine learning model to label data and then select examples which may require the aid of a human to finetune the annotation. secondly, our findings enable dataset creators to rank examples in terms of syntactic complexity. this can be used to assign some examples to annotators based on their experience, where expert annotators are assigned one set of examples, and novice/intermediate annotators are assigned another set of examples. for future work, we would like to explore using vector norms for these purposes, as it can help create better datasets by not assigning examples to annotate randomly but in an informed manner.