Parsing in the absence of related languages: Evaluating low-resource dependency parsers on T agalog (2020)

Abstract: Cross-lingual and multilingual methods have been widely suggested as options for dependency parsing of low-resource languages; however, these typically require the use of annotated data in related high-resource languages. In this paper, we evaluate the performance of these methods versus monolingual parsing of Tagalog, an Austronesian language which shares little typological similarity with any existing high-resource languages. We show that a monolingual model developed on minimal target language data consistently outperforms all cross-lingual and multilingual models when no closely-related sources exist for a low-resource language.

## introduction dependency parsing is a fundamental component of many natural language understanding @xcite and machine translation systems @xcite . state-of-the-art parsers which annotate syntactic dependencies from raw text have achieved high accuracy for languages with large datasets but continue to yield poor results for low-resource languages which have little to no annotated data @xcite . various methods have been proposed to solve the problem of dependency parsing in a low-resource setting, including cross-lingual transfer @xcite , multilingual modeling @xcite , and annotation projection @xcite . these methods have been shown to be effective on target languages when datasets (such as treebanks and parallel corpora) are readily available for closely-related source languages; however, would the same hold true in the absence of related language data? such is the problem for tagalog, an austronesian language of the philippines with over 25 million speakers worldwide @xcite . despite its widespread use in both spoken and digital domains, it remains largely under-resourced, lacking basic language processing resources such as syntactic treebanks and parsers. moreover, while dependency treebanks are available for indonesian, another austronesian language, the extensive phylogenetic distance between the two languages @xcite suggests that indonesian may have too many typological differences to serve as an effective source language for tagalog. in this paper, we investigate the performance on tagalog of three strategies for low-resource dependency parsing: monolingual modeling (using only minimal target language data), cross-lingual modeling (using only data from similar source languages), and multilingual modeling (using data from both target and non-target languages). we present a new tagalog dependency treebank on which to train and test these approaches together with available treebanks from the universal dependencies (ud) project @xcite , and compare our results to those of previous studies. ## related work to our knowledge, only two dependency treebanks for tagalog have been created prior to this work. the first is the tagalog dependency treebank @xcite , which includes 2,500 @xcite . upon inspection, we found that most of these were simple declarative sentences which used the basic predicate-initial word order of tagalog and contained only one or two arguments. moreover, the treebank did not contain any examples of other sentence types such as compound sentences, interrogatives, and imperatives, nor of common grammatical components such as adjectival modifiers and plural forms. ## language data tagalog treebank. in order to properly assess the performance of a dependency parser on a target language, we need to have a treebank available in that language which more extensively captures its grammatical complexities and contains universally comparable annotations. since neither of the previous tagalog treebanks fulfill both requirements, we developed ugnayan, a new tagalog dependency treebank manually annotated in the ud framework. the treebank currently consists of 94 sentences (1011 tokens) taken from educational texts (almario and tan, 2016). these sentences include examples of various syntactic phenomena such as compound and complex sentences, clausal modifiers, question forms, and sentence inversion. source treebanks. to train the cross-lingual and multilingual models, we also needed to identify which ud languages with available training data are most similar to tagalog. for this, we used a walsreliant distance measure, which compares the typological similarity of a source language s and a target language t based on their features as described in the world atlas for language structures @xcite . we use the distance measure defined by @xcite as the hamming distance d h between the wals feature vectors v s and v t for the source and target languages, normalized with respect to the number of features f s,t which are non-empty for both s and t . the resulting wals measure d w is given as: using this measure, we found that the five closest source languages for tagalog included indonesian (the only other austronesian ud language), vietnamese (the only austro-asiatic ud language), and three indo-european languages: ukrainian, romanian, and catalan (see to other indo-european languages than they were to tagalog. this supports the findings by @xcite that phylogenetic relatedness does not guarantee typological similarity. for our cross-lingual modeling, we selected all ud v2.6 treebanks with available train and dev sets in the source languages identified above. we also decided to train a model on the tagalog trg treebank as a point of comparison. we report the sizes of these data sets in ## evaluation of parsing models methodology. to train parsing models on the treebanks above, we used udpipe @xcite , a pipeline for processing of conll-u treebanks which has served as the baseline system in several conll ud shared tasks @xcite . we trained cross-lingual models for each of the identified source treebanks using their specified train and dev partitions, as well as a model using all test data in trg, and tested these models on all test data in ugnayan. we performed tenfold cross-validation to evaluate monolingual models trained on ugnayan, with a train/dev/test partition of roughly 80/10/10 for each iteration. we also used cross-validation on the multilingual models, which were trained using each of the ten ugnayan train/dev partitions combined with the individual source treebanks, and tested on the ten ugnayan test partitions. we used the default settings on udpipe 3.1 for all training and testing instances. we then investigated the performance on ugnayan of the two approaches previously applied to trg as described in section 2 by evaluating the pre-trained indonesian model of stanza (previously stan-fordnlp), a neural pipeline developed by @xcite which reportedly outperforms all submissions to the conll 2018 ud shared task for the low-resource categories on all metrics. we selected this as an approximation of the neural parser used by @xcite , which was based on the parser of @xcite currently integrated into stanza. we also evaluated an updated version of udify, the multilingual parser by @xcite . the f 1 scores automatically generated by each of the three parsers on the following tasks: token, word, and sentence tokenization, universal part-of-speech tagging, lemmatization, unlabeled attachment, and labeled attachment. since the ugnayan treebank currently does not contain features or language-specific part-of-speech tags, metrics involving those annotations were excluded from the report. for the monolingual and multilingual models trained on ugnayan data, we present the average scores across ten iterations for cross-validation. we find that the monolingual ugnayan models, each trained on less than 90 sentences (or approximately 900 tokens), outperform all other models on the tagging and parsing tasks, and are surpassed only by the tagalog-indonesian mixed model on sentence tokenization. these results support the hypothesis by zeman that "you can actually train a parser and get over 50% accuracy for many languages with just about 100 sentences," @xcite which has previously been shown for indian languages @xcite , galician @xcite @xcite . @xcite have achieved similar pos tagging performance for kinyarwanda and malagasy using similarly limited annotation and graph-based label propagation onto larger amounts of raw text; here we show that supervised modeling using only limited annotation can yield good results. we also observe that the cross-lingual models in particular score much lower than the ugnayan models or any of the multilingual models on the upos, uas, and las metrics. interestingly, the multilingual models, which use ugnayan training data together with each of the cross-lingual source treebanks, yield consistently lower accuracy than the monolingual models alone. this runs contrary to the findings of @xcite who observed that adding related language data to train a multilingual model further improves parsing accuracy. these results suggest that the typological distance between tagalog and any of its closest ud languages may be too great for the latter to be useful as cross-lingual or multilingual source languages out of the box, and that upweighting of the ugnayan data may be necessary to account for the size difference between the source and target training corpora. as for the pre-trained parsers, the stanza indonesian model slightly outperforms its udpipe equivalent on sentence tokenization, pos tagging, and lemmatization, while underperforming on uas and las. because of the large discrepancy between these results and the 70.89% uas & 50.38% las previously reported for parsing trg using an indonesian-only model @xcite , we further investigated the performance of indonesians models on both ugnayan and trg when gold tokenization and gold tags are made available. we found that uas and las higher than 50% were achievable only with gold tags for both treebanks, and that these results could not be matched when parsing from raw text. on the other hand, the udify universal model outperforms all cross-lingual models and even the monolingual trg model on all tagging and parsing tasks. this is quite remarkable, considering that no annotated tagalog data was used to train the udify model, although the availability of gold tokenization may have yielded a performance improvement compared to the other models which parse from raw text. these support the results of @xcite which show that udify's bert pretraining and multilingual learning produce reasonably high scores even in a zero-resource setting. ## extended analysis performance on cross-domain data. so far, the experiments we have described above involved the use of tagalog training and test data from the same corpus (ugnayan) and domain (educational text). however, as @xcite have observed, in-domain training naturally results in better performance than the cross-domain scenario for the same amount of data. to test the cross-domain performance of the ugnayan model, we annotated an additional 7 sentences (265 tokens) of tagalog news text, and evaluated each of the single-language udpipe models above on this new dataset. we found that the tagging and parsing results of the ugnayan model on the news dataset were significantly lower than the in-domain results (see on historical contact and lexical similarity. in addition to cognates within the austronesian language family, the tagalog language is known to have incorporated many loanwords from both spanish and english as a result of colonial occupation and, in the case of the latter, continued use within the country. to check whether either of these would be viable source languages, we trained single-language udpipe models using the spanish and english pud treebanks. the results were roughly at par with the other source languages tested above (see parsing from gold-tagged data. this paper has largely focused on ud parsing from raw text. in a low-resource context, however, if good pos tagging performance can be achieved for the target language independent of treebank data, delexicalized parsing (which uses only pos tags as input) has been widely thought of as a suitable parsing strategy. in relation to this, we compare the performance of the singlelanguage models when parsing with gold tags available for all test tokens. in contrast, the ugnayan model achieves the best performance on both in-corpus and news data (see tables 5c and 5d ), outperforming all other single-language models in both cases by a comfortable margin; the next-best source model was english-pud for both tasks and test sets. these provide partial support for the findings of @xcite , who have demonstrated that lexicalized parsing with limited target data generally outperforms delexicalized parsing with large amounts of source data when no good sources for the target language exist. ## conclusion we have evaluated the performance of monolingual, cross-lingual, and multilingual parsing models on ugnayan, a new universal dependencies treebank for the tagalog language, given the task of dependency parsing from raw text. we have also identified potential source treebanks for the cross-lingual and multilingual models by measuring the typological similarity between tagalog and existing high-resource ud languages. we find that a monolingual model trained on roughly 900 tokens of annotated target language data yields better performance than cross-lingual or multilingual models trained on 20,000 or more tokens of annotated data in other high-resource languages if these source languages exhibit low similarity to the target language. we also find that when no annotated training data is available for a target language, a model pre-trained on high-quality multilingual embeddings can give reasonable performance over cross-lingual models trained on individual source languages. we conclude that, when developing a parser for a low-resource language in the absence of any annotations for closely related languages, even a minimal amount of target language annotation greatly improves parsing performance over alternative methods. we currently plan to expand the ugnayan treebank in both size and scope, with additional annotations for morphological features and language-specific relation subtypes. further investigation is warranted on the effects of domain coverage, lexical similarity, and word order differences @xcite on parsing performance, as well as the application of other methods such as data augmentation @xcite and annotation transfer using parallel corpora @xcite in parser modeling.