Word-order Typology in Multilingual BERT : A Case Study in Subordinate-Clause Detection (2022)

Abstract: The capabilities and limitations of BERT and similar models are still unclear when it comes to learning syntactic abstractions, in particular across languages. In this paper, we use the task of subordinate-clause detection within and across languages to probe these properties. We show that this task is deceptively simple, with easy gains offset by a long tail of harder cases, and that BERT’s zero-shot performance is dominated by word-order effects, mirroring the SVO/VSO/SOV typology.

## introduction analysing the ability of pre-trained neural language models, such as bert @xcite , to abstract grammatical patterns from raw texts has become a prominent research question @xcite . results remain mixed. while bert-based models have been shown to learn syntactic representations that are similarly structured across languages @xcite , some grammatical patterns, such as discontinuous constituents, remain challenging for them even when training data is plentiful @xcite . in practical terms, zero-shot performance of bert-based models is lower for typologically distant languages @xcite , and they can profit from direct exposure to typological features during fine-tuning @xcite . in this study, we add another datapoint to the conversation by analysing the ability of bert-based models to capture the distinction between main and subordinate clauses across languages. this task is promising for two reasons. first, it highlights variability in the way main and subordinate clauses are structured across languages, thus acting as an informative probe into the relationship between bert and typological categories. second, the task is arguably relevant for downstream performance on natural-language understanding, where (some notion of) syntactic scope and compositionality should support tasks such as analysing commitment @xcite or factuality @xcite ), text simplification @xcite , or paraphrase detection @xcite . in order to operationalise it in a cross-lingual fashion, we use the universal dependencies framework (ud; @xcite with its large multilingual collection of corpora. our analysis proceeds in two stages. first, we survey the performance of bert models fine-tuned and tested on the same language across 20 typologically diverse languages ( § 3). for the majority of languages, distinguishing main and subordinate clauses is easily solved with base-size models and relatively small training sets. however, some languages demonstrate a non-negligible number of errors, which we analyse. then we study the performance of multilingual bert (mbert) in a zero-shot setting ( § 4), where we fine-tune the model on labeled data in 10 different languages and then test its performance on 31 datasets representing 27 different languages. we find that the performance of mbert is dominated by word-order effects well known from the typological literature @xcite : the arabic model shows best-in-class performance on irish, and the japanese model has best-in-class performance on korean, while both have poor performance overall. european languages with large training sets provide good inductive bias for typologically diverse languages but fail on sov languages. ## experimental setup data to make our analysis maximally comparable across languages, we start from the parallel universal dependencies (pud) collection @xcite , which contains translations for a set of 1000 english sentences. pud only contains test corpora. as these are too small to be further split into train/test subsets, we use other corpora to fine-tune the models. we also add corpora for languages not covered by pud for better typological coverage. see appendix § a.2 for the full list. ## single-language models the main results obtained by the models fine-tuned and tested on the same language are shown in a more fundamental distinction seems to exist between major european languages, the results on which are generally @xmath0, and mandarin chinese, vietnamese, and korean where results are around 90%. our analysis indicates that these differences are partly due to discrepancies in ud annotations across corpora but also due to genuine syntactic differences. an example of an annotation-related confound is the treatment of quotations. the pud corpora that we use preferentially as test sets treat quotations as sentential complements of communication verbs. some of the corpora we use for fine-tuning, however, analyse the cases where quotation precedes the verb of speech as parataxis. the head predicate of the quotation therefore receives the label root and becomes the main predicate of the whole sentence, leading to spurious mistakes in the analysis of pud corpora, where they are annotated as ccomp's. this discrepancy accounts for the lion's share of classification mistakes in german and some mistakes in mandarin. in contrast, an example of genuine ambiguity is provided by the mandarin gēnjù construction. this construction means 'according to' and can incorporate both nominal and verbal constituents. thus, gēnjù shàng biǎogé zhōng qī gè yuánsù de guānxì from the mandarin gsd corpus, which we used for fine-tuning, means 'based on the relationship of the seven elements in the above table', and the annotation treats this construction as an oblique prepositional phrase. cf. the following example from the mandarin pud corpus: gēnjù kěxíng xìng yánjiū gūjì 'according to the feasibility study / the feasibility study estimates that / as the feasbility study estimates'. the analysis of this sentence in pud makes gūjì 'estimate' the main predicate of the sentence, while an alternative analysis would make it the head of an adverbial clause, and yet another analysis would label it as a nominal element. the ability of mandarin words to act as different parts of speech in different contexts (especially in case of verbs, which can act as clause heads, auxiliaries, complementisers, and compound elements) makes this kind of disambiguation difficult even for human annotators, which in turn makes it hard to formulate the exact rule that language models are supposed to extract from the data. a similar situation holds for vietnamese. 2 a different type of systematic ambiguity is presented by korean, which also demonstrates poorer performance. korean has about sixty markers connecting two clauses, and many of those allow for both coordinative and conjunctive readings, which makes either the first or the second clause the main one, respectively @xcite . examples of this type are responsible for a large share of mistakes in korean. overall, these results indicate that subordinateclause detection is a long-tail task: major easily learnable patterns account for more than 90% of test cases for all languages, but in some languages there is an assortment of harder cases that prevent language models from efficiently generalising. ## related work both aspects of our analysis -subordinate-clause detection and the study of word-order effects -have been addressed but not in conjunction and not in a multiple-source-language setting. our study extends previous approaches by providing a zs 'upper baseline' derived from the study of the performance of several monolingual models and then conducting a novel many-sources-to-many-target analysis of zero-shot performance. @xcite test bert on the auxiliaryclassification task (main vs. subordinate clause) as part of their investigation of bert's linguistic knowledge. @xcite extend this analysis to the multilingual setting with a focus on nordic languages. word-order differences have been shown to impact the performance of english-based crosslingual models, especially in the domain of syntactic parsing @xcite and with tasks that rely on syntactic information @xcite , while reordering has been long known to be an efficient preprocessing step in syntactic transfer (rasooli and collins, 2019) and machine translation, both statistical @xcite and neural @xcite . ## conclusion we extend previous work on syntactic capabilities of bert, mostly focusing on english, by providing a more comprehensive analysis of its performance on the task of subordinate-clause detection in multiple languages and language pairs in the zero-shot setting. we show that the performance of single-language models is uneven across languages: east and southeast asian languages with less rigid boundaries between pos categories and coordination and subordination prove harder to model. we also show that mbert's performance in the zeroshot setting, while being largely correlated with the size of the pre-training and fine-tuning corpora, with russian being the best source language across the board, is well aligned with the word-order typology: language pairs with congruent word orders demonstrate better results, with both svo and sov orders having higher in-group than across-group accuracies. a single pair of vso languages in the data further corroborates this finding, showing that the verb-final order is not important per se. the clause-initial position of complementisers in vso languages partly blurs this effect and helps svo languages with large training corpora serve as good sources for fine-tuning, but even russian and english fail on sov languages, where complementisers tend to be postposed and dependent-clause predicates never appear in the sentence-final position. this shows that at least for some tasks, training on a single source language is not enough. moreover, our results from single-language modelling seem to indicate that even superficially simple syntactic tasks vary in difficulty across languages, which imposes a hard limit on how well cross-lingual projection can perform.