Controlled Evaluation of Grammatical Knowledge in M andarin C hinese Language Models (2021)

Abstract: Prior work has shown that structural supervision helps English language models learn generalizations about syntactic phenomena such as subject-verb agreement. However, it remains unclear if such an inductive bias would also improve language models’ ability to learn grammatical dependencies in typologically different languages. Here we investigate this question in Mandarin Chinese, which has a logographic, largely syllable-based writing system; different word order; and sparser morphology than English. We train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and Transformer-parameterized generative parsing models on two Mandarin Chinese datasets of different sizes. We evaluate the models’ ability to learn different aspects of Mandarin grammar that assess syntactic and semantic relationships. We find suggestive evidence that structural supervision helps with representing syntactic state across intervening content and improves performance in low-data settings, suggesting that the benefits of hierarchical inductive biases in acquiring dependency relationships may extend beyond English.

## introduction a rich collection of targeted linguistic evaluations has shown that neural language models can surprisingly learn many aspects of grammar from unlabeled linguistic input (e.g., @xcite @xcite @xcite . there is also growing evidence that explicit modeling of syntax helps neural network-based language models represent syntactic state and exhibit human-like processing behaviors of non-local grammatical dependencies, including number agreement @xcite , negative polarity licensing, filler-gap dependencies @xcite , and garden-path effects @xcite . however, this line of research has focused primarily on the syntax of english. it is unclear to what extent structural supervision may help neural language models generalize for languages with differing typologies. expanding these analyses beyond english has the potential to inform scientific questions about inductive biases for language acquisition, as well as practical questions about model architectures that approach language-independence @xcite . here, we perform a controlled case study of grammatical knowledge in mandarin chinese language models. the orthography and grammar of chinese provide a useful testing ground given the differences from english and other indo-european languages (see, e.g., @xcite . whereas today's indo-european languages like english generally use phone-based orthography, chinese uses a logographic system where each character generally responds to a syllable. most mandarin chinese words are one or two syllables, influencing the distribution of tokens. grammatically, chinese has almost no inflectional morphology, and corpus studies suggest that the average dependency length of mandarin chinese sentences is larger than that of english sentences @xcite , with potential implications for language modeling. on the one hand, the need to track input across long dependencies may make structural supervision more beneficial for mandarin chinese language models; on the other hand, the prevalence of these dependencies may make it easier for them to learn to maintain non-local information without explicitly modeling syntax. other fine-grained differences in typology also affect the types of syntactic tests that can be conducted. for example, since relative clauses precede the head noun in chinese (unlike in english), we can manipulate the distance of a verbobject dependency by inserting relative clauses in between. these characteristics motivate our choice of mandarin chinese as a language for evaluating structurally supervised neural language models. we design six classes of mandarin test suites covering a range of syntactic and semantic relationships, some specific to mandarin and some comparable to english. we train neural language models with differing inductive biases on two datasets of different sizes, and compare models' performance on our targeted evaluation materials. while most prior work investigating syntactically guided language models has used recurrent neural network grammar models @xcite -potentially conflating structural supervision with a particular parameterization -this work further explores structured transformer language models @xcite . our results are summarized as follows. we find that structural supervision yields greatest performance advantages in low-data settings, in line with prior work on english language models. our results also suggest a potential benefit of structural supervision in deriving garden-path effects induced by local classifier-noun mismatch, and in maintaining syntactic expectations across intervening content within a dependency relation. these findings suggest that the benefits of hierarchical inductive biases in acquiring dependency relationships may not be specific to english. the two sentences differ only at the main verb 'drinks'/'drink', which must agree with its subject 'the man' in number. since only the third-person singular form 'drinks' agrees with the subject, (a) is grammatical, whereas (b) is not. the closest work to ours is the corpus of chinese linguistic minimal pairs (climp; @xcite , which provides a benchmark for testing syntactic generalization of mandarin chinese language models. while climp focuses on building a comprehensive challenge set, the current work performs controlled experiments to investigate the effect of structural supervision on language models' ability to learn syntactic and semantic relationships. moreover, the test items in climp are (semi)-automatically generated, which may result in semantically anomalous sentences and introduce noise into the evaluation phase. in contrast, we manually construct the items in our test suites to sound as natural as possible. ## results we begin by reporting the overall performance of the models on the test suite classes introduced in section 3.2. figure 1 shows accuracy scores averaged across test suites within each class. 7 first, we note that the n-gram baseline overall performs the worst among all language models, which matches our expectations since syntactic dependencies beyond the 5-token window are difficult for the model to capture. turning to the neural models, we assess the effects of training data size and architecture by examining the mean accuracy scores across test suites for each model. we fit separate linear mixed-effects models comparing the effects of data size, using the lme4 package in r @xcite . the dependent variable is the mean accuracy score (figure 2 ). for each language model type, the main effect is a binary indicator of whether the model is trained on the ctb dataset or the larger xinhua dataset. we include test suite class, modifier type, and model seed as random factors with random intercepts and slopes for the accuracy score. across test suite classes, the transformer-xinhua models outperform their smaller ctb counterparts (@xmath0.05), but the effect of data size is less clear for the lstm, rnng and plm models. comparing different model architectures, we find that in the subordination test suite class, rnngs trained on the smaller ctb dataset achieve comparable performance to lstms trained on the larger xinhua dataset (@xmath1.938; linear mixedeffects model with model type as the main factor). furthermore, for both verb-noun com-patibility and subordination, the rnngs perform better than the lstms and transformers when trained on the smaller ctb dataset (see figure 1 ). these results suggest that an inductive bias for learning hierarchical structure may help in low-data chinese settings, consistent with prior cla ssi fie r -no un ga rde n pa th -ob jec t ga rde n pa th -su bje ct ve rb -no un mi ssi ng ob jec t su bo rdi na tio n 0.0 0.2 0.4 0.6 0.8 1.0 accuracy n-gram n-gram-xinhua lstm lstm-xinhua rnng rnng-xinhua transformer transformer-xinhua plm plm-xinhua figure 1: accuracy by test suite class and model. for each test suite class, we average the accuracy across different modifier types (including the no modifier case). error bars denote 95% cis of the mean accuracy score. ctb xinhua rnng vs. lstm plm vs. transformer rnng vs. lstm plm vs. transformer classifier-noun ---** garden path object -** -* garden path subject ----verb-noun ** -* -missing object ---*** subordination ** -** - overall, we find only suggestive benefits of structural supervision. while prior work in english @xcite @xcite has shown that rnngs and plms can improve upon their vanilla lstm and transformer counterparts, the improvement is relatively smaller for mandarin. to compare the performance of models with and without structural supervision, we fit a generalized linear mixedeffect model on the binary outcome (whether or not the model predicts a positive surprisal difference) for each test item, within each combination of test suite class, training data, and model parameterization. we consider a binary indicator of whether or not the model performs syntax modeling explicitly as the main factor, and include test item, modifier type and model seed as random factors. we also assess whether our models are better at capturing syntactic or semantic relationships. to do this, we group the six classes of test suites into three categories: syntactic dependency (missing ob-ject and subordination), semantic relationship (classifier-noun compatibility and verb-noun compatibility), and a hybrid capturing semantically-driven syntactic state representations (garden path subject and garden path ob-ject). we find that the average accuracy score is higher in the syntactic test suites than in the semantic test suites (@xmath4.001), 8 suggesting that the language models in our study -including those with no explicit syntax modeling -find it easier to learn syntactic dependencies than semantic relationships. ## conclusion this work evaluates mandarin chinese language models on six grammatical relationships, including syntactic dependencies and semantic compatibilities. we use mandarin as a case study for analyzing how the potential advantages of explicit syntax modeling (as performed by rnngs and plms) generalize from english to a typologically different language. although structural supervision does not boost the sequential model's learning in all relationships tested in this study, we find that it does allow the rnng and plm models to learn dependencies robust to increasingly complex modifiers, as seen in the missing object test suites. compared to the vanilla sequence-based lstm and transformer models, explicit syntactic modeling also seems to help with grammatical generalization in settings with small training data. we also find that mandarin syntactic dependencies (such as tracking gross syntactic state within a subordinate clause) tend to be easier to learn than semantic dependencies (such as the compatibility between classifiers and nouns). this study is one of the first steps towards understanding the role structural inductive biases may play in learning semantic and syntactic relationships in typologically diverse languages.