Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT (2021)

Abstract: We investigate how Multilingual BERT (mBERT) encodes grammar by examining how the high-order grammatical feature of morphosyntactic alignment (how different languages define what counts as a “subject”) is manifested across the embedding spaces of different languages. To understand if and how morphosyntactic alignment affects contextual embedding spaces, we train classifiers to recover the subjecthood of mBERT embeddings in transitive sentences (which do not contain overt information about morphosyntactic alignment) and then evaluate them zero-shot on intransitive sentences (where subjecthood classification depends on alignment), within and across languages. We find that the resulting classifier distributions reflect the morphosyntactic alignment of their training languages. Our results demonstrate that mBERT representations are influenced by high-level grammatical features that are not manifested in any one input sentence, and that this is robust across languages. Further examining the characteristics that our classifiers rely on, we find that features such as passive voice, animacy and case strongly correlate with classification decisions, suggesting that mBERT does not encode subjecthood purely syntactically, but that subjecthood embedding is continuous and dependent on semantic and discourse factors, as is proposed in much of the functional linguistics literature. Together, these results provide insight into how grammatical features manifest in contextual embedding spaces, at a level of abstraction not covered by previous work.

## introduction our goal is to understand whether, and how, large pretrained models encode abstract features of the grammars of languages. to do so, we analyze the notion of subjecthood in multilingual bert (mbert) across diverse languages with different morphosyntactic alignments. alignment (how each language defines what classifies as a "subject") is a feature of the grammar of a language, rather than of any single word or sentence, letting us analyze mbert's representation of languagespecific high-order grammatical properties. recent work has demonstrated that transformer models of language, such as bert @xcite , encode sentences in structurally meaningful ways @xcite @xcite @xcite @xcite . in multilingual bert, previous work has demonstrated surprising levels of multilingual and cross-lingual understanding @xcite @xcite , with some notable limitations @xcite . however, these studies still leave an open question: are higher-order abstract grammatical features -features such as morphosyntactic alignment, which are not realized in any one sentence -accessible to deep neural models? and how are these allegedly discrete features represented in a continuous embedding space? our goal is to answer these questions by examining grammatical subjecthood across typologically diverse languages. in doing so, we complicate the traditional notion of the grammatical subject as a discrete category and provide evidence for a richer, probabilistic characterization of subjecthood. for 24 languages, we train small classifiers to distinguish the mbert embeddings of nouns that are subjects of transitive sentences from nouns that are objects. we then test these classifiers on outof-domain examples within and across languages. we go beyond standard probing methods (which rely on classifier accuracy to make claims about embedding spaces) by (a) testing the classifiers out-of-domain to gain insights about the shape and characteristics of the subjecthood classification boundary and (b) testing for awareness of morphosyntactic alignment, which is a feature of the grammar rather than of the classifier inputs. our main experiments are as follows. in experiment 1, we test our subjecthood classifiers on outof-domain intransitive subjects (subjects of verbs which do not have objects, like "the man slept") in their training language. whereas in english and many other languages, we think of intransitive subjects as grammatical subjects, some languages have a different morphosyntactic alignment system and treat intransitive subjects more like objects @xcite . we find evidence that a language's alignment is represented in mbert's embeddings. in experiment 2, we perform successful zero-shot cross-linguistic transfer of our subject classifiers, finding that higherorder features of the grammar of each language are represented in a way that is parallel across languages. in experiment 3, we characterize the basis for these classifier decisions by studying how they vary as a function of linguistic features like animacy, grammatical case, and the passive construction. taken together, the results of these experiments suggest that mbert represents subjecthood and objecthood robustly and probabilistically. its representation is general enough such that it can transfer across languages, but also language-specific enough that it learns languagespecific abstract grammatical features. ## background: morphosyntactic alignment in transitive sentences, languages need a way of distinguishing which noun is the transitive subject (called a, for agent) and which noun is the transitive object (o). in english, this distinction is marked by word order: "the dog a chased the lawyer o " means something different than "the lawyer a chased the dog o ". in other languages, this distinction is marked by a morphological feature: case. case markings, usually affixes, are attached to nouns to indicate their role in the sentence, and as such in these languages word order is often much freer than in english. apart from a and o, there is also a third grammatical role: intransitive subjects (s). in sentences like "the lawyer laughed", there is no ambiguity as to who is doing the action. as such, cased languages usually do not reserve a third case to mark s nouns, and use either the a case or the o case. languages that mark s nouns in the same way as a nouns are said to follow a nominative-accusative case system, where the nominative case is for a and s, and the accusative case is for o. 2 languages that mark s nouns like o nouns follow an ergative-absolutive system, where the ergative case is used to mark a nouns, and the absolutive case marks s and o nouns. for example, the basque language follows this system. a visualization of the two case systems is shown in figure 1 . the feature of whether a language follows a nominative-accusative or an ergative-absolutive system is called morphosyntactic alignment. morphosyntactic alignment is a high-order grammatical feature of a language, which is not usually inferable from looking at just one sentence, but from the system with which different sentences are encoded. as such, examining the way that individual contextual embeddings express morphosyntactic alignment gets to the question of how mbert encodes abstract features of grammar. this is a question that is not answered by work that looks at the contextual encoding of the features that are realized in sentences, like part of speech or sentence structure. ## methods our primary method involves training classifiers to predict subjecthood from mbert contextual embeddings, and examining the decisions of these classifiers within and across languages. we train a classifier to distinguish a from o in the mbert embeddings of one language, and we examine its performance on s embeddings in its training language, and on a, s, and o mbert embeddings in other languages. data to train a subjecthood classifier for one language, we use a balanced dataset of 1,012 transitive subject (a) mbert embeddings, and 1,012 transitive object (o) mbert embeddings. we test our classifiers on test datasets of a, s, and o embeddings. our data points are extracted from the universal dependencies treebanks @xcite : we use the dependency parse information to determine whether each noun is an a or an o, and if it is either we pass the whole sentence through mbert and take the contextual embedding corresponding to the noun. we run experiments on 24 languages; specifically, all the languages that are both in the mbert training set foot_2 and have universal dependencies treebanks with at least 1,012 a occurences and 1,012 o occurences. 4 labeling since ud treebanks are not labeled for sentence role (a, s and o), we extract these labels using the dependency graph annotations. we only include nouns and proper nouns, leaving pronouns for future work. 5 we label a noun token as: finally, we exclude the subjects of passive constructions (where the object of an action is made the grammatical subject) to analyze separately, as including these examples would confound grammatical subjecthood with semantic agency. we also exclude the siblings of expletives (e.g., "there are many goats"), as these are grammatical objects which appear without subjects as the only argument of the verb, and we also exclude the children of auxiliaries ("the goat can swim"), looking only at the arguments of verbs. because we use embeddings and are limited by the universal dependencies annotation scheme, there are some cross-linguistic differences in how arguments are handled. for instance, our system is not able to handle null subjects or null objects, even though those are prominent parts of many languages. classifiers for each language, and for each mbert layer ℓ, we train a classifier to classify mbert contextual embeddings drawn from layer ℓ as a or o. the classifiers are all two-layer perceptrons with one hidden layer of size 64. we train each classifier for 20 epochs on a dataset of the layer-ℓ contextual embeddings of 1,012 a nouns and 1,012 o nouns. in total, we train 24 languages × 13 mbert @xmath0. ## experiment 1: subjecthood in mbert in our first experiment, we train a classifier to predict the grammatical role of a noun in context from its mbert contextual embedding, and examine its behavior on intransitive subjects (s), which are out-of-domain. this experimental setup lets us ask two questions about subjecthood encoding in mbert. firstly, do contextual word embeddings reliably encode subjecthood information? secondly, how do our classifiers act when given s arguments (intransitive subjects), which crucially do not appear 5 for an example of how pronouns complicate how subjecthood is defined, see @xcite . in the training data? if s arguments are mostly classified as a, that would suggest mbert is learning a nominative-accusative system, where a and s pattern together. if s patterns with o, that would suggest it has an ergative-absolutive system. if s patterns differently in different languages, that would suggest that it learns a language-specific morphosyntactic system and expresses it in the encoding of nouns in transitive clauses (which are unaffected by alignment), so that the a-o classifiers can pick it up. ## experiment 2: transferring across languages we can learn only so much about mbert's general subjecthood representations by training and testing in the same language, since many languages in our data set have case-marking and therefore have surface forms that reflect their grammatical roles. to test whether representations of subjecthood in mbert are language-general, we can do a similar analysis to experiment 1 but with zero-shot cross-lingual transfer. that is, we train a classifier to distinguish a and o in language x (just as in experiment 1), but then we test in language y by seeing how the classifier classifies a, o, and s arguments in language y. by training a classifier on one language and testing on others, we can ask: is subjecthood encoded in parallel ways across languages in mbert space? if a classifier trained to distinguish a from o in a source language can then use the same parameters to successfully classify a from o in another language, this would indicate that the difference between a and o is encoded in similar ways in mbert space for these two languages. secondly, we can examine the classification of s nouns (which are out of domain for the classifiers) in the zero-shot cross-lingual setting. by ob- ## experiment 3: syntactic and semantic factors of continuous subjecthood to explore the nature of mbert's underlying representation of grammatical role, we ask which arguments are most likely to be classified as subjects or objects. this is of particular interest when the classifier gets it wrong: what kinds of subjects get erroneously classified as objects? the functional linguistics literature offers insight into these questions. it has been frequently claimed that grammatical subjecthood is actually a multi-factor, probabilistic concept @xcite @xcite that cannot always be pinned down as a discrete category. some subjects are more subject-y than others. @xcite argues that a subject can be thought of as the intersection of that which is high in agency (subjects do things) and topicality (subjects are the topics of sentences). thus, in english, a prototypical subject is something like "he kicked the ball." since in such a sentence, the pronoun "he" is a clear agent and the topic of the sentence. but, in a sentence like "the lake, which jack frost visited, froze," the subject is still "lake." but it is less subject-y: it is not the clear topic of the sentence and it is not an agent. a probabilistic notion of grammatical role lends itself naturally to the continuous embedding spaces of computational models. so, in a series of experiments, we explored what factors in mbert contextual embedding space predict subjecthood. in these experiments, we examine how the decisions and probabilities of the a-o classifiers from experiment 2 relate to other linguistic features known to contribute to the degree of subjecthood. in particular, we look at whether nouns appear in passive constructions, as well as the animacy and case of nouns. in seeing how passives, animacy, and case interact with our subjecthood classifiers, we can assess if mbert's representation of subjecthood in continuous space is consistent with functional analyses, and better understand the continuous space in which mbert encodes syntactic and semantic relations. we choose these three factors as they are wellstudied in the functional literature, as well as readily available to extract from ud corpora. passive subjects are marked with a separate dependency arc label, the animacy of nouns is annotated directly in some ud treebanks, and in casemarked languages, nouns are annotated with their case. future work on a more complete examination of the functional nature of contextual embeddings would include other factors not readily available in ud, like the discourse and information structure (topicality) of nouns in context. ## discussion our experimental results constitute a way to begin understanding how general knowledge of grammar is manifested in contextual embedding spaces, and how discrete categories like subjecthood are reconciled in continuous embedding spaces. while most previous work analyzing large contextual models focuses on extracting their analysis of features or structures present in specific inputs, we focus on morphosyntactic alignment, a feature of grammars that is not explicitly realized in any one sentence. we find that, when tested out of domain, classifiers trained to predict transitive subjecthood in mbert contextual space robustly demonstrate decisions which reflect (a) the morphosyntactic alignment of their training language and (b) continuous encoding of subjecthood influenced by semantic properties. there has been much recent work pointing out the limitations of the probing methodology for analyzing embedding spaces @xcite @xcite , a methodology that is very similar to ours. the main limitation pointed out in this literature is that the power of classifiers is a confounding variable: we can't know if a classifier's encoding of a feature is due to the feature being encoded in bert space, or to the classifier figuring out the feature from surface encoding. in this paper, we address these issues by proposing two ways to use classifiers to analyze embedding spaces that go beyond probing, and avoid the limitations of arguments based only around the accuracy of probes. firstly, our results rely on testing the classifiers on out-of-domain zero-shot transfer: both to s arguments and to different languages. as such, we focus on linguistically defining the type of classification boundary which our classifiers learn from mbert space, rather than their accuracy, and in using transfer we avoid many of the limitations of probing, as argued in papadimitriou and jurafsky (2020). secondly, we examine a feature (morphosyntactic alignment) which is not inferable from the classifiers' training data, which consists only of transitive sentences. we are asking if mbert contextual space is organized in a way that encodes the effects of morphosyntactic alignment for tokens that do not themselves express alignment. especially in the cross-lingual case, a classifier would not be able to spuriously deduce this from the surface form, whatever its power. a limitation of our experimental setup is that both our universal dependencies training data and the set of mbert training languages are heavily weighted towards nominative-accusative languages. as such, we see a clear nominativeaccusative bias in mbert, and our results are somewhat noisy as we only have one ergativeabsolutive language and two semi-ergative languages future work should examine the effects of balanced joint training between nominativeaccusative and ergative-absolutive languages on the contextual embedding of subjecthood. and we hope that future work will continue to ask not just if deep neural models of language represent discrete linguistic features, but how they represent them probabilistically.