Improving Low-resource RRG Parsing with Cross-lingual Self-training (2022)

Abstract: This paper considers the task of parsing low-resource languages in a scenario where parallel English data and also a limited seed of annotated sentences in the target language are available, as for example in bootstrapping parallel treebanks. We focus on constituency parsing using Role and Reference Grammar (RRG), a theory that has so far been understudied in computational linguistics but that is widely used in typological research, i.e., in particular in the context of low-resource languages. Starting from an existing RRG parser, we propose two strategies for low-resource parsing: first, we extend the parsing model into a cross-lingual parser, exploiting the parallel data in the high-resource language and unsupervised word alignments by providing internal states of the source-language parser to the target-language parser. Second, we adopt self-training, thereby iteratively expanding the training data, starting from the seed, by including the most confident new parses in each round. Both in simulated scenarios and with a real low-resource language (Daakaka), we find substantial and complementary improvements from both self-training and cross-lingual parsing. Moreover, we also experimented with using gloss embeddings in addition to token embeddings in the target language, and this also improves results. Finally, starting from what we have for Daakaka, we also consider parsing a related language (Dalkalaen) where glosses and English translations are available but no annotated trees at all, i.e., a no-resource scenario wrt. syntactic annotations. We start with cross-lingual parser trained on Daakaka with glosses and use self-training to adapt it to Dalkalaen. The results are surprisingly good.

## introduction treebanks play an increasingly important role in typological research, where linguists are oftentimes 1 our experimental code is available at https:// gitlab.com/treegrasp/rrgproj2 . faced with (primarily) oral low-resource languages and where grammatical frameworks such as role @xcite @xcite are a common choice @xcite . rrg is a non-transformational linguistic theory strongly inspired by typological concerns. its development was guided by the question of what a linguistic theory would "look like if it were based on the analysis of languages with diverse structures such as lakhota, tagalog and dyirbal [...]?" @xcite . rrg assumes constituency structures to be organized in layers, viz. nucleus (containing the predicate), core (containing the nucleus and the arguments of the predicate) and clause (the core and extracted arguments). furthermore, each layer can have modifiers (termed periphery elements) and operators. an example from daakaka (an oceanic language, von prince, 2015) is given in fig. 1 , using the annotation scheme from @xcite . with respect to complex constructions, i.e., combinations of more than one clause, core or nuc, rrg distinguishes not just coordination and subordination but, in addition, cosubordination. the latter has the general form @xcite x [ ] x ] x , an example is the combination of the two clauses in figure 1 into a larger clause. in such a construction, an operator that applies to categories x takes scope over both x daughters while being realized only once. the tam marker in figure 1 for instance assigns tense to both clauses. the various ways of combining two nuc, core or clause constituents differ with respect to whether one of the two depends on the other, how tight the two units are concerning time and location of the two events, and to what extent the two units share operators such as tense, aspect, modality etc., all of which can be explained by the respective combination of layer level (nuc, core or clause) and construction type (coordination, subordination or cosubordination). in this paper, we show a way to train rrg parsers on only a little amount of annotated data in the target language or, in the case of dalkalaen, even no annotated data, yielding parse trees that can presumably reduce annotation effort considerably when used as a starting point for treebanking. furthermore, the resulting parse trees might be sufficiently good to be the basis for (semi-)automatic investigations of syntactic properties of low-resource languages. the scenario underlying our work is that we have a limited amount of data (translated to english and possibly glossed), and that a small subset of it (a few hundred or at most a few thousand sentences) is annotated with rrg trees. this is a realistic scenario for the result from typological fieldwork and grammar description. consequently, and in contrast to most other proposals for parsing low-resource languages, (i) we are not aiming at dependency parsing but, instead, using a constituency scheme often used in typological research, and (ii) we cannot use a large language model trained on the target language, but (iii) we can make use of additional data typically included in fieldwork output, namely glosses and translations. we propose to improve parsing in this scenario by cross-lingually injecting information from english translations (and glosses, if available) into the target-language parser, combined with self-training. the main research question we address in this paper is to what extent this method improves performance, and in what situations, i.e., at how many annotated trees a language stops being "low resource enough" to be helped by this method. to answer this, we test our methods on various simulated degrees of "low resource-ness", from 100 training trees to over 4 000. to get more robust data, we test our method not only on a real lowresource language (daakaka), but also on four other languages for which parallel rrg treebanks are available @xcite . in all cases, we use english as the source language. finally, we conduct experiments on a language related to daakaka, namely dalkalaen, where we have glosses and english translations but no syntactic annotations (except for test data). the aim is to test whether extending a cross-lingual parser (with english as source language) to a related language while keeping the source language leads to useful results. the hypothesis is that parsing a no-resource language (with glosses and english translations, i.e., no-resource concerning syntactic annotations) benefits from knowledge about the english translation and from knowledge learned from a related language. in the remainder of the paper, we will first discuss related work (sec. 2), then explain our parsing architecture, including grammar extraction, rrg parsing, cross-lingual transfer, and self-training (sec. 3). then, in sec. 4, we will describe our experimental setup, and sec. 5 will discuss the results of the experiments. we conclude in sec. 6. ## related work the problem of parsing low-resource languages has been addressed both for dependency and for constituency parsing for different scenarios concerning available data on the target side and available high resource parallel data @xcite . many approaches assume that there is enough unlabeled data for the target language to train a language model (i.e., the term 'low resource' refers only to syntactically annotated data). @xcite for instance use monolingual language models for both source and target language and use a mapping between decontextualized variants of these vectors to guide the cross-lingual transfer. @xcite ; @xcite use polyglot language models trained on source and target data as input to crosslingual parsing. in contrast to this, we assume a scenario where there is not enough target data to train a language model. there is also variation concerning the way the information from the source parse is injected into the target parsing process. many approaches project the source parse onto the target sentence via alignments, sometimes even using multiple source languages @xcite . instead of projecting a parse of a source sentence to a target sentence, @xcite use parallel english data to select among the k best parses of a target language sentence by comparing to the parallel english parse. in our case, we use only english parallel source data, and we project supertag information along word alignments. some approaches use delexicalized parsers @xcite , but it has also been shown that lexicalization, in particular when covering aspects shared between source and target language helps @xcite . in this vein, we experiment with including glosses in the target language, which means that the target language tokens contain information that is similar to the information captured in the aligned english tokens. self-training as a means of training data augmentation for parsing has been proposed in a number of papers @xcite @xcite , though not in the context of the above-mentioned approaches to cross-lingual transfer in low resource parsing. note that decrossing is rather local, since crossing branches usually occur within a single group of layers clause -core -nuc (resp. xp -core_x -nuc_x), as in fig. 1 and 2. the decrossing algorithm of @xcite , which we use, differs from the graph decrossing method proposed by @xcite in that the tree structures undergo only minimal changes to largely preserve the original tree structure and that it follows handwritten rules to decross the nodes uniformly, e.g. the discontinuous op tns node under the clause is always re-attached to the closest lower core node. the co-anchoring for pps without a clear meaning contribution (e.g. made for the stairs) and particle verbs (e.g. pick up) is simulated by including the corresponding internal structure of the dependent supertag into the head supertag (see fig. 2 ). the supertags are extracted with features that indicate the original parent node, which facilitates the rule-based recrossing step after parsing. for the steps of decrossing, supertag extraction, a* parsing, and recrossing, we use the system developed by @xcite . our crosslingual extension is in the neural supertag and arc scoring module. the idea is illustrated in figure 3 : we train a monolingual system for the source language and then use its internal representations as an additional input to a system for the target language. the representations are fed through a crosslingual attention mechanism to take into account word alignment information. the resulting crosslingual system takes a source-language sentence and an aligned target-language sentence as input (both are available in the parallel treebanking scenario) and produces a parse for the latter. we now describe the monolingual scoring module and our cross-lingual extension in detail. ## experimental setup data our choice of data is mainly driven by the availability of rrg-annotated resources. rrgbank @xcite ) contains a subset of the english wall street journal corpus @xcite annotated with rrg trees. rrgparbank @xcite contains george orwell's novel 1984 in the original english as well as translations to french, german, russian, and farsi, along with sentence alignments between english and every other language. the english data as well as parts in other languages are annotated. in addition, we use 6 499 sentences in daakaka, first published as von prince (2013a) and described in von prince (2015), which come with glosses, part-of-speech tags, and english translations, and 1 871 of which have been annotated with rrg trees following the rrgparbank annotation guidelines. furthermore, we use 3 393 sentences in dalkalaen, first published as von prince (2013b), also including glosses and english translations, of which 102 sentences have been annotated with rrg trees. we use all english trees in rrgbank and rrgparbank to train the english source model. for german, french, russian, farsi, daakaka, and dalkalaen, we focus on sentences with 25 tokens or less that are 1:1-aligned with an english sentence (for daakaka and dalkalaen, this is almost all sentences). of these, we use 80% for training, 10% for development, and 10% for testing. note that low resource language corpora are often oral corpora, and they therefore tend to contain shorter sentences. this is the reason why only very few of the daakaka and dalkalaen sentences are longer than 25, and this is also why a sentence length limit of 25 simulates this low resource scenario adequately. for dalkalaen, we use the unannotated sentences for self-training, while the 101 annotated sentences are used for testing. in the case of daakaka, we ran experiments with and without gloss embeddings (in addition to the token embeddings), while for dalkalaen we always used gloss and token embeddings. we randomly downsample the training data according to the degree of "low resource-ness" we are simulating in each experiment. ## results and discussion impact of training data size we are interested in low-resource scenarios, so we first look at how the number of available training trees impacts parser accuracy for different languages. this provides the baseline against which we will later evaluate our cross-lingual embedding and self-training methods. we look at various degrees of "low resource-ness", simulated by randomly downsampling the training set: 100 training sentences, 500, 1000, 2000, 3000, 4000, and finally the scenario where we use as many training trees as possible (varies by language). the blue dots in fig. 4 show the results. in all cases, we see a considerable improvement in each step up to 2000 training sentences, and after that (for those languages where we have data), results improve only slightly. note that the f-scores in general are rather low (the best ones around 80%) compared to state of the art constituency parsing. this is due to the fact that, since we are simulating a low resource scenario, we do not use any pretrained word embeddings, even for the languages where these would be available. ## conclusions in this paper, we investigated constituency parsing, more precisely rrg parsing, for low-resource languages in a scenario where english translations, a limited set of annotated sentences in the target language and possibly also glosses are available. rrg is a theory that is widely used in typological research. we extended an existing rrg parser into a cross-lingual parser, exploiting the parallel