Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations (2021)

Abstract: Recent research has adopted a new experimental field centered around the concept of text perturbations which has revealed that shuffled word order has little to no impact on the downstream performance of Transformer-based language models across many NLP tasks. These findings contradict the common understanding of how the models encode hierarchical and structural information and even question if the word order is modeled with position embeddings. To this end, this paper proposes nine probing datasets organized by the type of controllable text perturbation for three Indo-European languages with a varying degree of word order flexibility: English, Swedish and Russian. Based on the probing analysis of the M-BERT and M-BART models, we report that the syntactic sensitivity depends on the language and model pre-training objectives. We also find that the sensitivity grows across layers together with the increase of the perturbation granularity. Last but not least, we show that the models barely use the positional information to induce syntactic trees from their intermediate self-attention and contextualized representations.

## introduction an extensive body of works is devoted to analyzing syntactic knowledge of transformer language models (lms) @xcite @xcite . @xcite have demonstrated their abilities to encode various linguistic and hierarchical properties @xcite @xcite which have a positive effect on the downstream performance @xcite and serve as an inspiration for syntax-oriented architecture improvements @xcite @xcite . besides, a variety of pre-training objectives has been introduced @xcite , with some of them modeling reconstruction of the perturbed word order @xcite @xcite . recent research has adopted a new experimental direction aimed at exploring the syntactic knowledge of lms and their sensitivity to word order employing text perturbations @xcite @xcite . some studies show that shuffling word order causes significant performance drops on a wide range of qa tasks @xcite . however, a number of works demonstrates that such permutation has little to no impact during the pre-training and finetuning stages @xcite @xcite @xcite . the latter contradict the common understanding on how the hierarchical and structural information is encoded in lms @xcite , and even may question if the word order is modeled with the position embeddings @xcite . this has stimulated a targeted probing of the lms internal representations generated from original texts and their permuted counterparts @xcite . a new type of controllable probes has been proposed, designed to test the lms sensitivity to granular character-and sub-word level manipulations @xcite , as well as structured syntactic perturbations @xcite . despite the emerging interest in the field, little is investigated for languages other than english, specifically those with flexible word order. this paper extends the ongoing research on the syntactic sensitivity to three indo-european languages with a varying degree of word order flexibility: english, swedish, and russian. the contributions of this work are summarized as follows. first, we propose nine probing datasets in the languages mentioned above, organized by the type of controllable syntactic perturbation: n-gram perturbation (ngramshift), shuffling parts of the syntactic clauses (clauseshift) and randomizing word order (randomshift). despite that randomizing word order has been studied from many perspectives (see section 2), ngramshift differs from similar approaches @xcite @xcite in that the n-grams correspond to only syntactic phrases (e.g. prepositional or numerical phrases) rather than random word spans. clauseshift is a previously unexplored type of syntactic perturbation adopted from the syntactic tree augmentation method @xcite . second, we apply a combination of parameter-free interpretation methods to test the sensitivity of two multilingual transformer lms: m-bert @xcite , and m-bart @xcite . we hypothesize that m-bart is more robust to the perturbations as opposed to m-bert since it is learned to restore the shuffled input during pre-training. we evaluate the discrepancy in the syntactic trees induced by the models from perturbed sentences against the original ones, along with the ability to distinguish between them by judging their linguistic acceptability @xcite . finally, we analyze the relationship between the models' probe performance and position embeddings (pes). to the best of our knowledge, it is one of the first attempts to introspect pes regarding structural probing, particularly in the light of syntactic perturbations. the code and datasets are publicly available foot_0 . these studies have proved that lms are capable of encoding linguistic and hierarchical information @xcite . however, the probing paradigm has been lately criticized for relying on supervised probes, which can learn linguistic properties given the supervision, and make it challenging to interpret the results because of the additional set of parameters @xcite . towards that end, @xcite introduce a structural probe to explore a linear transformation of the embedding space, which best approximates the distance between words and depth of the parse tree. the method has proved to infer the hierarchical structure without any linguistic annotation @xcite . @xcite propose a jabberwocky probing suite of semantically nonsensical but syntactically well-formed sentences. the results demonstrate that the bertbased lms do not isolate semantics from syntax, which motivates further development of the probing field. ## controllable perturbations this work proposes three types of controllable syntactic perturbations varying in the extent of sentence corruption. we construct nine probing tasks foot_1 for three indo-european languages foot_2 : english (west germanic, analytic), swedish (north germanic, analytic), and russian (balto-slavic, fusional). based on the dominant constituent order, all three languages are classified as the svo (subject-verb-object) languages. nevertheless, there are some differences between them regarding word order flexibility. russian is known to exhibit free word order as all of the possible constituent reorderings are acceptable: sov, osv, svo, ovs, vso, vos @xcite . english allows for only two of them, namely svo and osv @xcite . swedish belongs to the verb-second languages, which poses different restrictions on the possible constituent reorderings @xcite . each dataset foot_3 consists of 10k pairs of the corresponding perturbed sentence and its original. ngramshift tests the lm sensitivity to local perturbations taking into account the syntactic structure. we used a set of carefully designed morphosyntactic patterns to perturb n-grams that correspond to only syntactic phrases such as numeral phrases, determiner phrases, compound noun phrases, prepositional phrases, etc. towards this, we applied tf-idf weighting from scikit-learn library @xcite to build a ranked n-gram feature matrix from the corpora and further used it for the ngram inversion. we used the n-gram @xmath0; 4] for each language. note that the number of words that change their absolute positions is similar for different values of n . figure 1 illustrates the shift of the head in the prepositional phrase "to school" for the sentence "he did not go to school yesterday". ## discussion the syntactic sensitivity depends upon language at present, english remains the focal point of prior research in the field of nlp, leaving other languages understudied. our probing experiments on the less explored languages with different word order flexibility show that m-bert and m-bart behave slightly differently in swedish and russian. while m-bart better restores the corrupted syntactic structure on most of the tasks for swedish, there are fewer attention heads sensitive to the perturbations in russian, which is revealed through the examination of head-wise attention patterns of both models. besides, the encoders receive lower probing performance for russian that can be contributed to the more complex syntax and flexible word order. pre-training objectives can help to improve syntactic robustness analysis of the m-bert and m-bart lms that differ in the pre-training objectives shows that m-bert achieves higher δ uuas performance across all languages as opposed to m-bart pre-trained with the sentence shuffling objective. the lower δ uuas probing performance indicates that m-bart better induces syntactic trees from both perturbed and intact sentences (see section 5.1). despite this, the representation and acceptability analysis demonstrate that m-bart is also capable of distinguishing between the perturbations (see sections 5.2-5.3). a fruitful direction for future work is to analyze more lms that differ in the architecture design and pre-training objectives. the lms are less sensitive to more granular perturbations the results of the parameter-free probing show that m-bert and m-bart exhibit little to no sensitivity to local perturbations within syntactic groups (ngramshift) and distant perturbations at the level of syntactic clauses (clauseshift). in contrast, the global perturbations (randomshift) are best distinguished by the encoders. as the granularity of the syntactic corruption increases, we observe a worse probing performance under all considered interpretation methods. namely, the results are supported by representation analysis metrics (see section 5.2) that indicate higher susceptibility to major changes in the sentences structure (randomshift, clauseshift), and the pppl-based measures (see section 5.3) prescribing higher acceptability scores to sentences with more granular perturbations (ngramshift). we also find that the sensitivity to the hierarchical corruption grows across layers together with the increase of the perturbation complexity, which is in line with @xcite . m-bert and m-bart barely use positional information to induce syntactic trees previous research has shown that the token embeddings capture enough semantic information to restore the syntactic structure @xcite @xcite . @xcite claim that syntactic abilities of bert-based lms are overestimated and raise the problem of isolating semantics from syntax. however, more recent studies show that transformer encoders encode redundant information @xcite , may not sufficiently capture the meaning of positions and be unimportant for downstream tasks @xcite , including the setting with perturbed fine-tuning data @xcite . in spirit with the latter studies, our results under different pes configurations reveal that m-bert and m-bart do not need the precise position information to restore the syntactic tree from their internal representations. the overall behavior is that zeroed (except for m-bert) or even randomly initialized pes can result in the probing performance and one with absolute positions. we suppose that despite the absolute positions of words changes during the n-gram permutation and sub-tree rotation procedures, the word order within the clauses remains almost the same as in the intact sentence 0 1 2 3 4 5 6 7 8 9 10 11 (ngramshift, clauseshift). that is, the more granular perturbations marginally confuse the lms when: (i) predicting the masked word under token perturbation probing which can be performed using only attention (wang and chen, 2020), or (ii) judging the acceptability of the sentence where the low token pseudo-log-probability can occur at the juxtaposition of the syntactic groups, and clauses @xcite . we leave a more detailed exploration of the relationship between pes and probing analysis for future work. ## conclusion this paper presents an extension of the ongoing research on the controllable text perturbations to the multilingual setting and introspection of positional embeddings in pre-trained lms. we introduce nine probing datasets for three indo-european languages varying in their flexibility of the word order: english, swedish, and russian. the suite is constructed using language-specific heuristics carefully designed under linguistic expertise and organized by three types of syntactic perturbations: randomization of word order studied by previous research from many perspectives and less explored permutations within syntactic phrases and clauses.