Probing LLM s for Joint Encoding of Linguistic Categories (2023)

Abstract: Large Language Models (LLMs) exhibit impressive performance on a range of NLP tasks, due to the general-purpose linguistic knowledge acquired during pretraining. Existing model interpretability research (Tenney et al., 2019) suggests that a linguistic hierarchy emerges in the LLM layers, with lower layers better suited to solving syntactic tasks and higher layers employed for semantic processing. Yet, little is known about how encodings of different linguistic phenomena interact within the models and to what extent processing of linguistically-related categories relies on the same, shared model representations. In this paper, we propose a framework for testing the joint encoding of linguistic categories in LLMs. Focusing on syntax, we find evidence of joint encoding both at the same (related part-of-speech (POS) classes) and different (POS classes and related syntactic dependency relations) levels of linguistic hierarchy. Our cross-lingual experiments show that the same patterns hold across languages in multilingual LLMs.

## introduction recent advancements in natural language processing (nlp) can be attributed to the development and pretraining of large language models (llms) such as bert, gpt-3, and many others @xcite @xcite . for their intended use of providing general-purpose language representations suitable for many nlp tasks, these models must efficiently capture a wide range of linguistic features within their finite capacity. despite their success, little is known about the way in which different types of linguistic information are organized in these models. systematically understanding how these models represent linguistic phenomena and their interaction is crucial for the development of more effective nlp methods. existing research probed llms for their encoding of various linguistic properties such as agreement @xcite , word order and sentence structure @xcite , co-reference @xcite , semantics @xcite and multilinguality @xcite . taking a step further, @xcite and @xcite studied where linguistic information is encoded in llms by probing different layers. their results demonstrated that a linguistic hierarchy emerges in bert representations, with lower layers capturing local syntax and higher layers being employed in higher-level semantic and discourse tasks. however, we do not yet understand how encodings of different linguistic phenomena interact within the models and to what extent processing of linguistically-related categories relies on the same, shared model representations. there are many dependencies between processing different linguistic phenomena: for instance, information about a word's part of speech is likely to be employed when disambiguating its word sense. alternatively, lower-level syntax is an important first step for semantic composition and natural language understanding tasks. in this work, we investigate how the (hierarchical) dependencies between different linguistic categories are encoded in llms, focusing on syntax. we ask a set of novel questions: to answer these questions, we propose a framework for testing the joint encoding of linguistic categories in llms. specifically, we investigate to what extent the information about distinct linguistic categories is shared in the parameters of the repeat classification on the "neutralized" repr.s with the probing network. ## joint encoding of pos tags we begin by examining whether representations of different pos tags share information. for each pos tag, we compute its centroid and re-classify representations neutralized by the centroid. with regard to our baselines, we find that subtracting random vectors leaves the performance relatively unchanged, with no evidence of systematic drops. for our selectivity baseline, we find that our probing classifier achieves an average accuracy of 96% on the pos tagging task and 63% on the control task, giving a selectivity value of 33%. this is in line with probes for similar tasks accepted for their validity in the literature @xcite . to facilitate visualization and discussion, we select an illustrative subset foot_2 of pos tags that covers both open-and closed-class words, as we expect potentially different patterns between these two groups. for instance, we expect neutralization to be symmetric (neutralizer and target can be swapped for similar results) in the case of openclass words as we hypothesize that joint encoding here is dictated by co-occurrence patterns. conversely, we expect asymmetric neutralization in the case of close-class neutralizers, supposing that joint encoding here may be dictated by functional dependencies, which may not necessarily be reciprocal. results we find that linguistically related categories generally tend to be jointly encoded in roberta (see fig. 2 ). for instance, the relative decrease of verb classification accuracy by 53% when neutralizing using auxiliaries (aux) suggests information sharing. this may be explained by the fact that auxiliaries themselves are verbs (e.g. "has done"), and that they functionally modify verbs. we note that the information flow in this pairing is asymmetric, as noted by the much smaller 11% drop in the reciprocal case. this aligns with our hypothesis that joint encoding for closed-class neutralizers may be explained by their functional role, which is often not reciprocal (verbs will generally not modify auxiliaries). while we observe further evidence of linguistically related categories being jointly encoded (e.g. verb and adv), this is not ubiquitous. for instance, joint encoding is not found when neutralizing nouns (noun) with determiners (det). here, the relative percentage decrease is only 2%, despite determiners acting as modifiers for nouns. in general, we do not find distinct patterns of joint encoding unique to closed-class and open-class words. our hypothesis of symmetry due to co-occurrence is at best supported by the adv-adj pairing, with relative drops of -21% and -29%. in most other pairings information sharing appears to be asymmetric, suggesting functional dependence or another root cause as explanation. moreover, we note that open-class words tend to be "adept" at cross-neutralizing, sharing information as neutralizers with many other tags as shown by the noun, adj and verb neutralizers rows successfully neutralizing several columns. however, we note similar patterns when neutralizing with the closed-class det. overall, we find evidence of joint encoding of pos tags in roberta, but further work is necessary to establish which mechanisms lead to specific pairs sharing information or not. ## information sharing across languages our findings for individual languages on xlm-r raise the question of whether information about linguistic categories from two different languages can also be jointly encoded. to further probe for language-agnostic joint encoding of linguistic categories, we extend our experiment to test whether information between two linguistic categories from two different languages is jointly encoded. we posit that if categories share information across languages, this is evidence that the models learn at least partially language-independent representations for these categories. we test this hypothesis by cross-neutralizing every linguistic category in a language a with another category from a language b, e.g. neutralizing all italian pos tags with english nouns. aside from probing for languageagnosticism, this allows for further exploration of joint encoding of linguistic categories. to enable consistent neutralization, we use the same layer and pooling function configuration (see section 3.5) for both neutralizer and target. we do this by utilizing the neutralizing language configuration for both neutralizer and target in cases where they differ. results fig. 5 shows results of cross-neutralizing for italian pos tags when using the corresponding english centroids from each linguistic category. on the diagonal, neutralizer and targets correspond to the same linguistic category, but for different languages. accuracy drops substantially here (∼ 70% on avg.). this is evidence of language-agnostic encoding of these linguistic categories, as centroids in one language neutralizing targets in another language points to information in the representations being shared across both languages. we also observe joint encoding of different linguistic categories, for instance with english nouns neutralizing italian numerals (num) (-48%), or english adjectives (adj) neutralizing italian ad- ## information sharing across tasks to study whether information is shared across the linguistic hierarchy, we cross-neutralize between tasks using pos neutralizers and dependency relation targets. since a given parent may have several dependents while the child will only have one parent, we posit that most of the information for the child-parent dependency is captured by the child. we therefore subtract the pos centroids from the child representations in the child-parent concatenations for dependency labeling. we also complete the complementary experiment of subtracting the child portion of the dependency relation centroids from the pos embeddings for pos tagging, to test whether joint encoding happens in both directions. as in section 5, we use the neutralizer configuration for both neutralizer and target for consistency. results fig. 4 (top) shows the results when neutralizing roberta representations for dep labels using pos centroids. we find further evidence of linguistically related units being jointly encoded. for instance, adverb pos tags (adv) neutralize adverbial modifier (advmod) dependency labels, as can be seen by the 97% relative drop in accuracy. the fact that pos tag representations jointly contain information that is crucial for encoding dep advmod amod aux det case advcl cc cop obj nsubj dep target adv adj aux det adp verb cconj noun pron sconj pos neutralizer -92 -19 -2 -0 0 1 -3 -2 -1 -3 -24 -67 -9 -2 -1 4 -3 -15 -1 0 -10 -5 -81 -0 -0 -17 -2 -57 -9 -1 -9 -6 -2 -92 -0 -1 -1 -7 -8 4 -16 -3 -1 -1 -74 3 -4 -11 -2 -6 -10 -2 -1 -0 -1 -40 -2 -1 -6 -4 -13 -3 -0 -2 -1 3 -95 2 0 2 -6 -4 -7 -2 -6 -7 -1 -8 -10 -0 -2 -4 -0 -0 1 -20 -1 -0 -22 -56 -23 -2 -4 -0 -11 4 -2 1 1 -0 english advmod amod aux det case advcl cc cop obj nsubj dep target italian advmod amod aux det case advcl cc cop obj nsubj dep target -89 -7 -0 -0 -1 -19 -2 1 -0 -5 -7 -68 -0 -1 -1 -4 -2 -5 -2 -0 -18 -5 -81 -1 -0 -31 0 -6 -36 -14 -3 -8 -1 -92 -3 -21 -1 0 -9 -0 -30 -3 -1 -1 -79 -9 -2 1 -3 -7 -3 -5 -2 -1 -5 -63 0 -8 -11 -6 -14 -3 -1 -0 -1 3 -96 -2 -9 -5 -7 -2 -1 0 -3 -12 0 -3 -6 -0 0 -1 -1 -2 -1 -10 0 1 -5 -2 -39 -2 -1 -1 -2 -19 -2 0 -0 -2 greek labels shows that llms learn hierarchically, reminiscent of the classical nlp pipeline. to study whether our findings generalize beyond english and roberta, we repeat the experiment on xlm-r (see fig. 6 ). large drops in accuracy on the diagonal suggest that many informationsharing patterns hold across languages and models, consolidating our findings from sections 4.1 and 5. we also find language-specific results: pronouns (pron) neutralize nominal subjects (nsubj) only in english (-56%), having surprisingly little effect on the other languages (-5% and -2%). on occasion, we find evidence of bidirectional sharing between different levels of the linguistic hierarchy. in fig. 4 (bottom), we note that certain pairs highlighted by the diagonals such as adpositions (pos: adp) and case relations (dep: case) present evidence of joint encoding both when neutralizing dep with pos centroids (-55%) and when neutralizing pos with dep centroids (-84%). this seems to suggest that the hierarchical nature of the representations learned by these llms is not necessarily unidirectional: information appears to be shared both upwards and downwards in the linguistic hierarchy. ## conclusion we study information sharing between linguistic categories in llms, finding evidence of joint encoding between pairs of related pos tag classes. by applying our method to xlm-r, we find evidence of joint encoding in xlm-r across languages, showing that our findings hold for both the monolingual (roberta) and multilingual (xlm-r) case. lastly, we cross-neutralize between pos tagging and dep labeling, and find evidence of information sharing across the linguistic hierarchy. we test specifically for joint encoding of different syntactic categories that rely on the same linguistic concept, such as an "amod" dependency implying some relationship with the adjective and the noun pos tags. however, our method could be extended to test for joint encoding of categories in other tasks that can be expected to share information. this in turn could be informative on whether an llm indeed captures this shared information between different tasks. a more complete map of this knowledge could help develop better models in many application scenarios: for instance, in a multitask learning setting where negative interference between parameter updates from different tasks is known to hamper performance @xcite or a lifelong learning setting where learning a new task often leads to (catastrophic) forgetting of the previously learned ones @xcite . future work may additionally seek to explore more languages, particularly low-resource and typologically distant languages, or consider including higher-level semantic tasks from the classical nlp pipeline, such as semantic role labeling, word sense disambiguation and coreference resolution. further work is also necessary for understanding why certain representations share information while others do not.