Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages? (2023)

Abstract: A line of work on Transformer-based language models such as BERT has attempted to use syntactic inductive bias to enhance the pretraining process, on the theory that building syntactic structure into the training process should reduce the amount of data needed for training. But such methods are often tested for high-resource languages such as English. In this work, we investigate whether these methods can compensate for data sparseness in low-resource languages, hypothesizing that they ought to be more effective for low-resource languages. We experiment with five low-resource languages: Uyghur, Wolof, Maltese, Coptic, and Ancient Greek. We find that these syntactic inductive bias methods produce uneven results in low-resource settings, and provide surprisingly little benefit in most cases.

## introduction many nlp algorithms rely on high-quality pretrained word representations for good performance. pretrained transformer language models (tlms) such as bert/mbert @xcite , roberta @xcite , xlm-r @xcite , and electra @xcite provide state-of-the-art word representations for many languages. however, these models require on the order of tens of millions of tokens of training data in order to achieve a minimum of quality @xcite , a data requirement that most languages of the world cannot practically satisfy. there are at least two basic approaches to addressing this issue. the first, which is at least as old as bert, exploits multilingual transfer to reduce the data requirements for any individual language. the second aims to reduce tlms' data requirements by modifying their architectures and algorithms. for example, gessler and zeldes (2022) more effectively train low-resource monolingual tlms with as few as 500k tokens by reducing model size and adding supervised pretraining tasks with part-of-speech tags and syntactic parses. we take up the latter direction in this work, looking specifically at whether the addition of syntactic inductive bias (sib) during the pretraining procedure may help improve tlm quality in lowresource, monolingual settings. specifically, we examine two methods which have been proposed for high-resource settings: the two syntactic contrastive loss functions of @xcite , and the modified self-attention algorithm of @xcite , wherein a modified self-attention mechanism, restricted so that tokens may only attend to tokens that are syntactically "local", complements the standard self-attention mechanism. at a high level, sib is of interest in the context of tlms because of how crucial self-attention is for tlms' syntactic knowledge. in studies on an english tlm, @xcite and @xcite show that while syntactic relations are not directly recoverable from self-attention patterns, many self-attention heads seem to be sensitive to particular syntactic relations, such as that of a direct object or or a subject. but self-attention is completely unbounded: during pretraining, the model has to learn from scratch how to decide which other tokens in an input sequence a token should attend to. we therefore observe that if sib could be effectively applied, then presumably self-attention weights would converge more quickly and learn more effectively, since their behavior has been observed to be so heavily syntactic in nature. moreover, we expect that this effect would be greater for low-resource languages, where the comparative lack of data is known to hamper models' ability to form robust linguistic representations. we find additional motivation for our interest in sib given the nearly universal view held by linguists that the human mind does not start with the equivalent of a totally unconstrained self-attention mechanism: for example, psycholinguists such as @xcite have extensively documented processing-related constraints on syntax, and generative linguists such as @xcite have observed that many syntactic constructions which might have been possible are in fact not attested in english or any other language, and postulate that these constructions are at least in some cases "impossible" because of biologically-determined properties of the human mind. our goal is therefore to give our models something like the constraints the human mind has in order to help them learn more effectively with less data. we use a standard bert-like tlm architecture as our base model, though we heavily reduce model size, following the results of @xcite which showed that this is beneficial in lowresource monolingual settings. we pretrain tlms for five low-resource languages-wolof, coptic, maltese, uyghur, and ancient greek-varying which sib methods are used. we then use universal dependencies (ud) @xcite syntactic parsing and wikiann @xcite named entity recognition as representative downstream tasks that allow us to assess the quality of our models. additionally, we evaluate our models using pronto @xcite , a suite of downstream task datasets for low-resource languages. we find that these sib methods are not very effective in lowresource languages, with small gains in some tasks and degradations or no effects in others. this is surprising given the intuition that sib ought to help more in low-resource settings, and we speculate that other methods for sib may be more effective in low-resource settings. we summarize our contributions as follows: ## previous work pretrained word representations have been essential ingredients for nlp models for at least a decade, beginning with static word embeddings such as word2vec @xcite , glove @xcite , and fasttext @xcite . contextualized word representations @xcite @xcite from transformer-based @xcite models have since overtaken them. throughout this period, high-resource languages have received the majority of attention, and although interest in low-resource settings has increased in the past few years, there remains a large gap (in terms of linguistic resources, pretrained models, etc.) between low-and high-resource languages @xcite . ## approach this work investigates whether methods for sib that have succeeded in high-resource monolingual tlm pretraining settings could also be useful in analogous low-resource settings. as we have seen, monolingual tlms tend to have very poor quality when less than ≈10m tokens of training data are available for pretraining, and moreover, it has been observed that at least one dimension of this poor quality is models' inability to make grammatical generalizations without a large (≈1b tokens, warstadt et al. 2020) pretraining dataset. since it is (almost definitionally) difficult to get more data in low-resource settings, it is especially important to find other ways of improving model quality. it is therefore worthwhile to examine whether supplying some kind of sib could help a low-resource tlm form better linguistic representations. as discussed in §2.3, there are many ways to introduce sib into a tlm. in this work, we look specifically at two methods: synclm @xcite and sla @xcite , which is also used by @xcite extend the selfattention module with "local attention", wherein tokens may only attend to tokens which @xmath0. @xcite devise two contrastive loss functions which are intended to encourage tokens to attend to sibling and child tokens, and in their experiments, they find success in combining these with sla. a concise description of the details of each method is available in appendix a.both of these methods have only been evaluated on english, and both assume a ud syntactic parse as an additional input for each input sequence and use the parse in different ways to attempt to guide the model to better syntactic representations. we use these two sib methods with the model of @xcite , microbert, as a foundation. microbert is a bert-like model that has been scaled down to 1% of bert-base, and that optionally employs part-of-speech tagging and syntactic parsing as auxiliary pretraining tasks. as shown by experiments on 7 low-resource languages conducted by gessler and zeldes ( 2022 ), microbert performs much better than an unmodified bert-base tlm, so we adopt it as our baseline model for most experiments in this work. we now state our two main research questions: • (rq1) do these sib methods improve model quality when applied to a low-resource language? • (rq2) are there any gains complementary with the part-of-speech tagging component of microbert for training low-resource monolingual tlms? language unlabeled ud ner wolof 517,237 9,581 10,800 coptic 970,642 48,632 -maltese 2,113,223 44,162 15,850 uyghur 2,401,445 44,258 17,095 anc. greek 9,058,227 213,999 - ## discussion considering first whether synclm and sla yield benefits for low-resource monolingual tlms (rq1), we have found positive evidence from the wikiann ner experiments, and weak positive evidence from the pronto experiments. it is true that the same methods did not produce measurable gain for the ud parsing task, but this is in line with previous findings for these two methods, where on some downstream evaluations, gain was very small or slightly negative-we return to this matter in the following paragraph. for the question of whether these benefits are complementary with the pos tagging pretraining strategy introduced in gessler and zeldes (2022) (rq2), we do not find consistent evidence in any of our experiments that both pos tagging and synclm or sla yield complementary benefits. the only positive evidence we find for this is in the pronto experiments, where the -mxt model variant does better than -mx in some task-language combinations, though worse overall. the difference in the way model variants behaved in these seven evaluation tasks is striking, and it is difficult to understand why models exhibited these different behaviors. it is worth comparing these results with those reported by the syn-clm authors @xcite . for many of the glue tasks that they assess their models on (their on the other hand, the glue task with the greatest gain, cola (corpus of linguistic acceptability), shows a difference of only 1.7% matthews correlation coefficient, and a couple of other tasks like sst (stanford sentiment treebank), show an improvement of only 0.3% accuracy. it would be naïve to directly compare percentage points of different metrics in totally different experimental settings and make conclusions about effect sizes, we nevertheless point out that we observe improvements of 1-4% f1 in our ner experiments for -m models. in light of this, we consider our results to be broadly in line with the trend for previous works' results on english: there is no improvement that is wholly consistent across evaluations, and only modest gains for the benchmarks that do improve. in summary, we find that synclm and sla produce uneven results in low-resource settings, though we also find that when they do succeed, they can yield gains that appear greater than anything observed for high-resource languages: we saw that when we take a pure mlm pretraining regimen as a base and add synclm and/or sla, we are able to improve the quality of pretrained tlms by 1 to 4 absolute points f1 in ner. while a similar benefit was not observed for ud parsing, it is also true that there was a noticeable degradation on ud parsing in only a couple cases, and in most cases simply had no effect. ## english experiments one might have expected sib to be a knockout success for low-resource languages given the intuitive feeling that at lower data volumes, additional bias ought to be more helpful. we considered reasons why our attempts to do this might not have panned out-perhaps, for example, tree structure matters most for highly analytic languages like english, or perhaps the tasks used to evaluate english in glue are more sensitive to high-level sentence structure, or perhaps sensitivity to syntax is only advantageous given a base model with sufficiently rich distributional information. here, we consider another possible explanation: that the inductive bias with these methods only helps given high-quality syntactic parses. an obvious difference between english and the languages we have examined in this study is that ud parsers for english generally achieve much higher performance given the size and annotation quality of english ud treebanks. this is a potentially consequential difference, given that both the synclm and sla methods rely on ud parse trees as inputs. in addition, the models we have developed here differ from common kinds of english berts in that they are much smaller and were trained on much less data, and it is possible that the synclm and sla methods might have interactions with these two variables of model construction. in order to investigate whether parse tree quality, model size, and pretraining data size might be consequential for these sib methods, we run several additional experiments on english datasets. we choose english because its status as a highresource language allows us control over several independent variables which we do not have control over in low-resource settings, namely data quantity, syntactic parse quality, and model size. 6 we can frame an additional research question that we wish to answer: we frame these additional conditions for english, extending our model naming scheme from above: 1. -np -syntax trees are taken from stanza in the same way as before. a full description of our results is given in appendix b, and we give a description of our key finding here: that synclm and sla are not very sensitive to parse quality or model size, but are sensitive to quantity of pretraining data. the insensitivity to parse quality may come as a surprise, and we reason that this is actually understandable, since both methods focus mostly on low-height subtrees (often corresponding to phrase-or sub-phrase-level constituents) which are more likely to be correct even when overall parse quality is bad. we find evidence for sensitivity to data size in the fact that synclm and sla provide gains of up to 1% f1 for the ner evaluation in the two low-data conditions, while in the higher-data conditions, all but one of the bias-enhanced models lead to degradations relative to the baseline. in sum, we take this to show that lower parse quality is not the major reason for the ineffectiveness of synclm and sla in low-resource settings. ## conclusion in this work, we have taken two methods for sib that have succeeded in english, synclm and sla, and we have investigated whether they may also be beneficial in low-resource monolingual settings. we find that in most cases these methods do not result in an improvement in model quality as measured on seven tasks. further, in our auxiliary experiments on english, we found evidence suggesting that the lower quality of parses in low-resource settings is probably not what is driving the ineffectiveness of these sib methods. considering all of our results, we conclude that these two specific methods-synclm and slaare not well suited to supporting the pretraining of language models in low-resource settings, but we also view it as a yet open question whether any method for sib could succeed in this role. there are some reasons why synclm and sla might have been unhelpful. first of all, recall the fact that synclm limits its application to only short subtrees (no taller than 3 nodes). this would mean that most of the time, the contrastive loss functions would only be operating on basic phrase-level constituents, such as noun phrases, and not higher, clause-level phenomena such as relations between the main clause's predicate and its arguments. if it were the case that the former kind of syntax is relatively easy for models to learn even with limited data, and that the latter kind of syntax is what is hard and therefore where sib really ought to help, then we would expect to see the results we found in this work, where neither method did much to help. therefore, while we find little reason to be optimistic about these two particular methods in lowresource settings, we don't view the evidence in this paper as an indictment of sib in low-resource settings in general, and suggest that sib methods which are better able to provide bias for higher, clause-level syntactic dependencies may produce better results for low-resource languages. 2021 ). the standard self-attention mechanism is complemented by another self-attention mechanism in which tokens may only attend to tokens close to it in a parse tree. a gated unit with learnable parameters interpolates the two attention distributions before the distribution is combined with the value representation.