Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure (2024)

Abstract: We introduce SPUD (Semantically Perturbed Universal Dependencies), a framework for creating nonce treebanks for the multilingual Universal Dependencies (UD) corpora. SPUD data satisfies syntactic argument structure, provides syntactic annotations, and ensures grammaticality via language-specific rules. We create nonce data in Arabic, English, French, German, and Russian, and demonstrate two use cases of SPUD treebanks. First, we investigate the effect of nonce data on word co-occurrence statistics, as measured by perplexity scores of autoregressive (ALM) and masked language models (MLM). We find that ALM scores are significantly more affected by nonce data than MLM scores. Second, we show how nonce data affects the performance of syntactic dependency probes. We replicate the findings of Müller-Eberstein et al. (2022) on nonce test data and show that the performance declines on both MLMs and ALMs wrt. original test data. However, a majority of the performance is kept, suggesting that the probe indeed learns syntax independently from semantics.

## introduction an ample amount of work in the last years has focused on making explicit the linguistic information encoded in language models (lms). frequently, the overarching question is to which extent the behavioral and representational properties of selfsupervised lms are cognitively plausible and assumed by linguistic theory (surveys: @xcite @xcite . a subset of this work aimed at understanding lms ability to learn syntax (survey: @xcite @xcite @xcite . a common approach is to rely on the performance of a probing classifier @xcite @xcite in predicting syntactic relations. however, this method has been criticized for various reasons @xcite , including that predicting a linguistic property from lm representations does not imply that the lm uses that property in its predictions @xcite . @xcite highlighted that results of syntactic probes can be misleading due to semantic information in the representation. in other words, a high probing accuracy may be fully or partially attributed to the use of present semantic knowledge. the theoretical linguistic literature has discussed the separation of syntactic grammaticality and semantic information for decades, starting with the sentence colorless green ideas sleep furiously in @xcite . a number of studies have used systematically perturbed data to probe lm representations and predictions on such grammatical but nonsensical sentences @xcite @xcite . however, none of the works ensures that the nonce data is grammatical, e.g. in their datasets, the valency of predicates is not controlled. furthermore, only @xcite base their conclusions on languages beyond english. in this paper, we aim to standardize the efforts in this direction by proposing a framework to create nonce data that satisfies syntactic argument structure, provides syntactic annotations, and ensures grammaticality via language-specific rules. the framework relies on universal dependencies (ud, de @xcite treebanks. for each sentence in the treebank, it replaces content words with other content words that are known to appear in the same syntactic context. this results in nonce sentences that preserve the syntactic structure of the original sentences (see fig. 1 ). to ensure syntactic grammaticality, we define three types of languagespecific rules and constraints that are applied during the process, which are: (i) the pos tag of the words to be replaced, (ii) constraints on word order and dependency relations of replaced words to ensure syntactic grammaticality, and (iii) word-level rules to ensure that morphosyntactic constraints are met. we refer to this algorithm and the resulting ud data as semantically perturbed ud (spud). we create spud data for five widely spoken languages; arabic, english, french, german, and russian. we show via a human evlation that spud is preferrable in terms of grammaticality to the kind of nonce data that has been previously used in the literature to tackle similar research questions. we show the effectiveness of spud on two tasks to assess the robustness of autoregressive lms (alms) and masked lms (mlms) to semantic perturbations. first, we study the effect of nonce data on lm scoring functions (perplexity and its adaptations for mlms). second, we investigate the robustness of syntactic dependency probes to semantic irregularities, and disentangle the effect of lexical semantic features on the findings of previous work. the contributions and main findings of our work are as follows: the paper is structured as follows. in the next section, we discuss related work. in sec. 3, we describe the framework for creating nonce ud treebanks. sec. 4 describes the experiment on scoring nonce data with perplexity. in sec. 5, we describe the structural probing experiments. in sec. 6, we discuss the results and conclude. ## nonce treebanks for five languages this section presents spud (semantically perturbed universal dependencies), our framework for nonce data creation. the input is a ud treebank, and the output is a treebank with the same syntactic structure as the input but nonce semantic content. in principle, the same algorithm is applied in any language, but each language requires a set of languagespecific pre-and post-processing steps (sec. 3.1). in this work, we create nonce data for five languages and treebanks: arabic (padt, @xcite , german @xcite , english (ewt, @xcite , french (gsd, @xcite , and russian (syntagrus, @xcite . fig. 1 presents one example. more examples and information on the resources are in app. a. to apply the framework to a new language, an annotated ud treebank and access to a native speaker in the target language are required. along with this paper, we publish a tutorial for creating spud treebanks in other languages. ## scoring spud with alms and mlms the spud treebanks are designed to be grammatical but highly improbable. in this section, we investigate how this property is reflected in the predictions of different lm architectures. on the one hand, this serves as a sanity check for spud resources: do we in fact perturb co-occurrence statistics of words as intended? on the other hand, at a higher level, it investigates whether spud data makes the models perplexed and how different lm architectures, and uni-vs. bidirectional context, influence the effect that syntactic and semantic structure has on model predictions. concretely, we answer the following questions: to what extent is spud data harder to predict than original data? are alms and mlms affected by spud data in different ways? are mlm scoring functions affected differently by nonce data? ## nonce dependency probing in this section, we measure the efficacy of spud in decoding the syntactic knowledge learned in lm representations. concretely, we train depprobe structural probes @xcite and compare the performance on the standard (original) ud test sets with performance on spud test sets. we choose depprobe because none of its alternatives produces both directed and labeled dependency trees. we investigate to which extent depprobe performance is influenced by lexical semantics rather than its desired task of predicting purely syntactic properties. with the assumption from previous work (sec. 2) that a certain drop in performance on nonce test data is expected, we answer the following research questions: rq1: is mlm and alm performance affected differently when tested on spud data? rq2: are the predictions of dependencies between tokens (edges) and dependency relations (edge labels) affected differently by spud data? we answer these questions on all five languages, and present additional experiments to improve the robustness of our findings. @xcite presented depprobe, a lightweight decoder for directed and labeled dependency trees. the model consists of two learnt linear transformations of the lm representations. both transform d h -dimensional word representations into vectors that highlight a feature of the dependency tree. l is a sequence labeling classifier that predicts for each word the label of the incoming dependency edge. @xmath0, where word vector distances mimic the distance between words in the tree. the dependency tree is constructed in a top-down fashion from both components, selecting the word as root to which l assigns the highest probability of being the root. details are provided in app. f.1. ## discussion and conclusion dataset we presented semantically perturbed universal dependencies (spud), a framework for creating parallel nonsensical ud treebanks, and apply it to generate grammatical but nonsensical data in five languages. we present two use cases for spud, which contribute to the literature that relates lms to the notions of grammaticality, acceptability and probability @xcite @xcite . beyond that, spud has several possible applications such as data augmentation for low-resource parsing or syntactic similarity benchmarks. scoring spud with alms and mlms in analyzing how token-level predictions are affected by spud, we find that spud data consistently shows higher scores (i.e., lower likelihood) than original data on all models. this shows that, as desired, the lexical co-occurrence patterns of content words in spud are perturbed. of all scoring functions, the increase in alm perplexity is highest. with our experiments, we contribute to the open question of how best to compute scores from mlms by comparing p p p l with subword-aware p p p l l2r @xcite . while we are mostly able to support (kauf and ivanova, 2023)'s argument that p p p l l2r is empirically more similar to p p l than p p p l is, we find that the response of alms and mlms to spud data is significantly different. this indicates a more fundamental difference between how alms and mlms score sentences: bidirectional context in mlms makes it easier to identify the syntactic properties of a predicted word, even in nonce sentences. probing alms and mlms for spud trees we probe different models for labeled and directed dependency trees on spud using the depprobe framework @xcite . we find that probe performance drops (compared to original test sets) for all languages and both model architectures when using spud test sets. however, a majority of the probe performance is maintained. the alm mgpt has a lower overall performance than mbert, as well as a higher performance drop on spud data. for both architectures, attachment contributes more to the performance drop than relation labeling does. these findings confirm the results in maudslay and cotterell (2021) and @xcite . on the contrary, @xcite found that gpt-2 performs on par with mlms on structural probing of unlabeled undirected dependency trees. this contrast highlights the importance of probe architecture and task design. @xcite found that mlms rely on higher-order cooccurrence statistics for many tasks, and can maintain high performance without word order information. we view the same problem from a different angle: our scoring results show that, expectedly, lm predictions are optimized for co-occurrence information. probing performance, however, can be maintained to a high-degree even if co-occurrence at the lexical level is disrupted. our comparison of mlms and alms suggests that both architectures process nonce data differently, and mlms are better at identifying syntactic structure in spud data than alms.