How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks (2022)

Abstract: This work addresses the question of the localization of syntactic information encoded in the transformers representations. We tackle this question from two perspectives, considering the object-past participle agreement in French, by identifying, first, in which part of the sentence and, second, in which part of the representation the syntactic information is encoded. The results of our experiments, using probing, causal analysis and feature selection method, show that syntactic information is encoded locally in a way consistent with the French grammar.

## introduction transformers @xcite have become a key component in many nlp models, arguably due to their capacity to uncover distributed representation of tokens @xcite ) that are contextualized: thanks to a multi-head self-attention mechanism @xcite , a token representation can, virtually, depend on the representations of all other tokens in the sentence, and transformers are able to learn a weighting to select which tokens are relevant to its interpretation. many works @xcite strive to analyze the representations uncovered by transformers to find out whether they are consistent with models derived from linguistic theories. one of the main analysis methods is the long-distance agreement task popularized by @xcite , which consists in assessing neural networks ability to predict the correct form of a token (e.g. a verb) in accordance with the agreement rules (e.g. its subject). this method has been generalized to other agreement phenomena @xcite and other languages @xcite . the concordant conclusions of all these experiments show that transformers are able to learn a 'substantial amount' of syntactic information @xcite . if the method of @xcite makes it possible to show that syntactic information is encoded in neural representations, it does not give any indication on its localization: it is not clear whether the syntactic information is distributed over the whole sentence (as made possible by self-attention) or only in a way consistent with the syntax of the language, i.e. only in the tokens involved in the agreement rules. this work addresses the question: where the syntactic information is encoded in transformer representations. 1 we approach this question from two perspectives, considering the object-past participle agreement in french (section 2). first, in section 3, using probing and counter-factual analysis, we try to identify the tokens in which syntactic information is encoded in order to find its localization within the sentence. second, in section 4, using a feature selection method, we study the localization of syntactic information within the contextualized representation of tokens. this evening the friends that i have met at the university come eat det-sg noun-sg det-pl noun-pl pron pron-1sg aux-sg verb-pl adp det-sg noun-sg verb-pl verb-inf acl:relcl obj figure 1: example of object-past participle agreement in french object relatives. dependencies between the target verb (in red) and the tokens involved in the agreement rules using the universal dependencies annotation guidelines are also shown. the prefix is represented in blue, the context in yellow and the suffix in green. to predict the past participle number, a human is expected to extract number information from the object relative pronoun (que) that gets it from its antecedent (amis in bold green). ## is syntactic information locally or globally distributed in the sentence? results reported in the previous section show that information about the number of the past participle is encoded in the token representations but they do not allow to identify which tokens have been used to predict the correct form of the past participle. in this section, we first identify, using linguistic probes, the tokens in which syntactic information is encoded and then, with a causal analysis, the tokens on which transformers mainly rely to predict the form of the past participle. ## probing representations components experiments reported in the previous section show that syntactic information is locally encoded in the context. in this section, we address the question of finding where this information is encoded within the transformers representation. to that end, we repeat the probing experiment on context token representations of §3.1 with an ℓ 1 regularized logistic regression @xcite . the resulting probe is thus constrained to minimize the number of features used to perform accurate predictions. given the probe objective function n @xmath0; w) + 1 c ||w|| 1 to minimize, we first determined the lowest bound for c such that the feature coefficients are guaranteed not to be all zeros, from which, we increase c evenly on a log space (i.e. decrease the regularization strength). results figure 3 reports the regularization path of the probing classifier. it shows that number information can be extracted with high accuracy (90.1%) solely from a very small number of dimensions, namely 90. increasing the number of dimensions (by decreasing the regularization strength) only results in a small improvement of model quality: the probe achieves an accuracy of 94.8% when all features are considered. interestingly, when removing the 90 features selected by the ℓ 1 regularization from the representation, a probe trained on the remaining features still achieve a very good accuracy of 93.8%, suggesting that the number information is encoded in a redundant way in the contextualised representations. 5 discussion and conclusion to understand how syntactic information is encoded and used in transformers-based lm, we carried out three sets of experiments considering the french object-past participle agreement task. first, our probing experiments uncovered clear evidence of a local distribution of number information within the context tokens, even though the self-attention mechanism allows this information to be spread all over the sentence. second, our masking intervention on attention shows a causal link between linguistically motivated tokens and the model's decision, suggesting that transformers process french object-past participle agreement in a linguisticallymotivated manner. finally, we used a ℓ 1 feature selection method to study the localization of number information within contextualized representations and found that while this information is encoded in a small amount of highly correlated dimensions, it is also fuzzily encoded in a redundant way in the remaining dimensions. our work is a first step towards a better understanding of the inner representations of lm. designing new probes, supported by causal analysis and involving a wider range of languages, could improve our understanding of such models. in particular, our observation about the linguistically motivated distribution of syntactic information in transformers representations could be extended to other linguistic phenomenon and languages.