Improving Low-resource RRG Parsing with Structured Gloss Embeddings (2023)

Abstract: Treebanking for local languages is hampered by the lack of existing parsers to generate pre-annotations. However, it has been shown that reasonably accurate parsers can be bootstrapped with little initial training data when use is made of the information in interlinear glosses and translations that language documentation data for such treebanks typically comes with. In this paper, we improve upon such a bootstrapping model by representing glosses using a combination of morphological feature vectors and pre-trained lemma embeddings. We also contribute a mapping from glosses to Universal Dependencies morphological features.

## introduction treebanking (i.e., annotating large corpora of sentences with syntactic structures) is an important tool for research into the syntax of natural language. treebanking has long avoided starting from scratch, but used machine-generated pre-annotations that annotators correct @xcite . for standardized languages, models generating the preannotations can nowadays rely on large language models and pre-trained parsers (e.g., @xcite @xcite . for local languages, the situation looks quite different: usually, no large language models or other models are available. however, if the language is documented, the data usually comes with interlinear glosses and translations to a standardized language such as english @xcite . @xcite show that these annotations can be used to obtain more accurate pre-annotations for local-language treebanks by projecting contextualized word representations from a parser for english onto the target-language sentences, using characterbased gloss embeddings, and self-training. in this paper, we show that the accuracy can be further improved by using a more structured representation for glosses. our contributions are 1) a mapping from interlinear glosses to universal dependencies features that can be reused for other language documentation data, 2) based on that, a method for embedding glossed sentences using morphological feature vectors and lemma embeddings, and 3) an evaluation of this embedding method in the context of cross-lingual rrg parsing for treebank pre-annotation. ## related work low-resource rrg parsing @xcite consider the task of creating pre-annotations for treebanks for the oceanic local languages daakaka and dalkalaen. the annotation scheme is based on that of rrgparbank @xcite , following @xcite @xcite , a framework designed with diverse languages in mind. the text data for the treebanks comes with interlinear glosses and english translations, but only few have been hand-annotated with rrg trees. figure 1 shows an annotated example daakaka sentence. the basic pre-annotation model takes as input daakaka token embeddings based on characterlevel lstms. it then labels each token with a supertag and a dependency head, which together serve as a derivation tree from which the final tree is constructed under the grammar formalism of tree wrapping grammar (twg; @xcite . it is then shown that the accuracy of the basic model can be improved by 1) concatenating the token embeddings with similarly character-based gloss embeddings, 2) doing multiple rounds of selftraining on unannotated data, and 3) using an english rrg parser (trained on substantially more gold standard data) on the translations and projecting contextualized word representations from the english parser to the daakaka parser via unsupervised word alignments. sentence clause core np pro ko 2s s.pron @xmath0.pre @xmath1: rrg annotation of a daakaka sentence, with its translation. leaf nodes contain word form, glosses, pos tags and ud features. glosses: 2s-second person singular, 3p-third person plural, pot-potential mood marker, mod-complementizer or modal relator. ka is a polysemous morpheme with different functions. it can either be a complementizer introducing subjunctive clauses, or a modal relator, which changes a directive speech act into an assertion (von prince, 2015). both functions appear similarly glossed in the data and were grouped together as ud feature @xmath2. morphological feature embeddings adding morphological features explicitly as input on nlp tasks has mixed effects, depending on the task and quality of features. @xcite show across several languages that the results on (monolingual) dependency parsing and named entity recognition improve on lstm-based models when ud feature embeddings are added as input, while the performance on comment filtering is not affected. manually annotated features yield better results than automatically added features. compared to our work, their approach assumes both a rich data set in the target language and high quality of ud features. an alternative method for encoding glossed words as tensors is described by @xcite , but does not provide explicit mappings from glosses to feature-value pairs. lemma embeddings it is standard in modern nlp systems to represent words as vectors based on word associations in unannotated running text. one such model is fasttext @xcite . less commonly, the same kind of model is trained on lemmatized text, e.g., in sprugnoli et al. (2019); ehren et al. (2020). ## method we build on evang et al.'s (2022) parsing architecture, as shown in figure 2 , with our modification concerning the embedding layer. while they use the same type of character-level lstm to generate token embeddings, part-of-speech tag embeddings, and gloss embeddings, we seek to improve performance by using a more structured representation. glosses consist of 1) translations of lemmas to english, and 2) codes representing morphological feature values. the gloss for one token can be seen as a partial function from features to feature values, so order does not matter and different values corresponding to the same feature are mutually exclusive. for example, the gloss 2s can be represented as {(number , sing), (person, 2)}. we exploit this by embedding glosses as a concatenation of feature embeddings like @xcite . besides improving performance, we also aim to create a reusable compatibility layer between the glosses and universal dependencies (ud; de marneffe et al., 2021), an annotation scheme commonly used in many data sets and tools. we therefore create the structured gloss embedding vectors via a mapping to the feature set defined by the ud annotation scheme. for the lemma translations, we exploit the fact that large quantities of text are available for english, and generate rich lemma embeddings. we now turn to the details of both contributions. construction of ud feature embeddings the mapping from glosses to ud features was performed with a conversion table, based on descriptions in von prince (2015) and von prince (2017) as well as ud guidelines. we focused on the glosses that occur in the daakaka and dalkalaen data (von prince, 2013a,b). the feature prontype was added for pronouns, which are not particularly glossed in the data. a number of glosses were not converted to features, such as ep for epenthetic consonants /p/ and att for the morpheme na, which derives attributes from lexemes and simple phrases. daakaka also distinguishes between three possessive classifiers glossed as cl1, cl2 and cl3 which show agreement with the lexical gender of the head noun or indicate their semantic domain (von prince, 2015). as their function is mainly semantic and not syntactic, they were all represented as {(poss, yes)}. the gloss sets of both languages largely overlap; two glosses with low occurence appear only in the dalkalaen data. we gathered 16 distinct features, 7 of which are unary (see for the ud feature embeddings, we follow the method described in @xcite . each feature is passed through an individual embedding layer (non-present features receive a special input), yielding 3-dimensional embeddings. the final representation is a 48-dimensional vector, constructed by concatenating all feature embeddings. ## evaluation we evaluate our ud feature+lemma embedding method by comparing against @xcite character-based method. we mirror their experimental setups, performing experiments across different scenarios (how much annotated seed training data is available), different amounts of self-training (adding 500 parses to the training data in each rounds 0 1 2 3 4 5 mono, chars 67.9 69.5 70.1 70.7 70.9 70.5 mono, struct 67.9 68.5 69.5 69.5 70.2 71.2 mono, struct+lem 69. round), and using the monolingual vs. the crosslingual model. we compute the overall evalb f-score @xcite of each model on the same test set of 196 trees (daakaka) resp. 101 trees (dalkalaen). in the "very low resource scenario" (500 annotated training sentences; in the "zero shot" scenario (parser trained on 1 840 daakaka trees, tested on dalkalaen;