Delexicalized Cross-lingual Dependency Parsing for X ibe (2021)

Abstract: Manually annotating a treebank is time-consuming and labor-intensive. We conduct delexicalized cross-lingual dependency parsing experiments, where we train the parser on one language and test on our target language. As our test case, we use Xibe, a severely under-resourced Tungusic language. We assume that choosing a closely related language as the source language will provide better results than more distant relatives. However, it is not clear how to determine those closely related languages. We investigate three different methods: choosing the typologically closest language, using LangRank, and choosing the most similar language based on perplexity. We train parsing models on the selected languages using UDify and test on different genres of Xibe data. The results show that languages selected based on typology and perplexity scores outperform those predicted by LangRank; Japanese is the optimal source language. In determining the source language, proximity to the target language is more important than large training sizes. Parsing is also influenced by genre differences, but they have little influence as long as the training data is at least as complex as the target.

## introduction for a severely lowresource language, construct ing a dependency treebank is laborintensive and timeconsuming, and annotators are difficult to find. expanding a small treebank via monolin gual dependency parsing leads to suboptimal re sults since we lack enough training data to train a reliable parser. this situation has led to an increasing interest in techniques for supporting lowresource languages by taking advantage of highresource languages together with methods for crosslingual transfer (meechanmaddon and @xcite . this is facilitated by the univer sal dependencies (ud) project, which has resulted in a treebank collection covering a wide range of language, with the goal of facilitating multilingual parser development @xcite . the lat est release (v2.7) covers 183 treebanks in 104 lan guages @xcite . in our current work, we carry out preliminary singlesource crosslingual delexicalized dependency parsing experiments for the xibe language. with this method, we train a parser on the treebank of one source language and parse the target language, with both treebanks delexicalized to abstract away from lexical differ ences between the two languages. choosing the source language is crucial for singlesource crosslingual parsing. the optimal source language needs to be syntactically close to the target language as well as highresourced. however, it is not obvious how to select this lan guage. we investigate three methods for select ing the source language: we compare langrank @xcite and typology, and we investigate whether using perplexity as a similarity metric can approximate typological knowledge. then we in vestigate whether the size of the source treebank or a genre mismatch affect the quality of the parser. the remainder of this paper is organized as fol lows: section 2 provides a short overview of xibe syntax. in section 3, we describe our research questions in more detail. in section 4, we briefly summarize methods of crosslingual transfer. the experimental settings are introduced in section 5. we then explain the methods for selecting source languages in section 6, and in section 7, we dis cuss our results. we conclude in section 8. ## the xibe language and treebank xibe is a tungusic language. there are twelve lan guages in the tungusic language family spoken in ᠮᡞ ᡞ gugu ᠪᠠᡯᠠᠷ deᠷᠷ ᡦᠠᠮᡞᡩᠣᠷ ᠪe ᡠᡩᠠᠮᠪ ︒ central and eastern asia, but the numbers of tun gusic speakers have never been large @xcite . the language of xibe is the one with a comparatively larger amount of active speakers in the whole language family. xibe shares morphological and syntactic fea tures with other transeurasian languages. the transeurasian languages have a very rich system of case marking through the use of affixes (or par ticles in japanese and korean). all transeurasian languages are head final, they use verbfinal word order, and attributes, complements, and adjuncts precede their headwords. in xibe, clausal con stituents have a rigid subjectobjectverb (sov) word order, and all phrasal categories are consis tently headfinal. like other tungusic languages, xibe has agglutinative morphology, which mainly focuses on verbs in that verbs are marked for tense, aspect, mood and voice, as well as converbs and participles. @xcite describe a xibe treebank an notated in the universal dependencies framework, containing 810 trees. figure 1 shows an exam ple xibe dependency tree. the matrix predicate is at the sentence final position with the object and oblique constituent, marked by corresponding case markers, preceding the verb and the present tense suffix mbi attached to the verb root. ## research questions our research focuses on crosslingual dependency parsing using a single target language and concen trates on determining the best method to select the optimal source language. more specifically, we in vestigate the following questions. question 1 what are the most important factors to consider when choosing the best source lan guage(s)? we investigate three methods: one uses typological knowledge, the second uses langrank @xcite , a machine learning approach to rank languages based on their relatedness. the third method uses pos ngram perplexity to deter mine similarity between languages. here, our goal is to determine whether perplexity can be used to model typological knowledge. question 2 ideally, an optimal source language should be closely related to the target language as well as highresourced, since we need a sizable treebank that can be used for training the parser. however, is a large treebank size more important than syntactic similarity with the target language in delexicalized dependency parsing? question 3 the xibe treebank includes sen tences from two different genres, grammar exam ples and news whereas most ud treebanks contain multiple written genres. considering that the per formance of the parsing models trained on one do main degrades on sentences drawn from a different domain @xcite , we assume that this happens in our setting as well. therefore, we investigate how a parser trained on multiple genres performs on the two target language genres. is the mixture of genres in the source data robust enough to cover both of our target genres? ## related work crosslingual transfer learning has been useful in improving the accuracy of a lowresource target language and has been applied in a multitude of tasks @xcite . the process of cross lingual transfer learning refers to resources and models from highresource source languages to lowresource target languages on different levels. there are four main crosslingual parsing ap proaches for dependency parsing: annotation pro jection @xcite , model transfer @xcite , treebank translation @xcite , and mul tilingual parsing models @xcite . the annotation projection approach requires par allel treebanks of both source language and target language, and the treebank translation approach re quires a machine translation system, while in the model transfer approach, models trained on source language treebanks are directly applied to parse tar get languages. a multilingual parsing model is ei ther a model trained on one source language which is refined by taking advantage of similar structures shared with the target language, or a multilingual model using multilingual word clusters and embed dings or languagespecific features. the main challenge for crosslingual parsing is to reduce the language discrepancies on differ ent levels between the source language and the target language. to reduce the great differences in writing systems and vocabulary, @xcite used delexicalization, based on the hypothesis that the interaction between morphol ogy and syntax in two languages is similar; they applied this approach in parsing swedish using danish as the source language. since this method does not require bilingual parallel data, it is exten sively implemented combining with other features. @xcite implemented the idea of delexicalizing the parsing models and adapting the parsers with a constraint driven learning algorithm that achieved accuracy gains. @xcite im proved the approach by @xcite by selecting the source sentences that are most similar to the target language. rosa and žabokrt ský (2015) trained an mstparser model interpola tion as an alternative for multisource crosslingual delexicalized dependency parser transfer. the work by @xcite involved the training of sev eral independent parsers which were applied to the same input sentence. the resulting tree was ob tained by finding the maximum spanning tree of a weighted directed graph of the potential parse tree edges from the different parsers. in addition to delexicalized methods, cross lingual lexical representations can also be used in dependency parsing. @xcite used parallel data to induce crosslingual word clusters, and added them as features for their delexicalized parser. @xcite proposed that the source and target language words with the same meaning share a common embedding. the em beddings are jointly trained with a neural model and are used for dependency parsing. duong et al. (2015a); ahmad et al. (2019); he et al. (2019) proposed different methods to develop multilin gual word representations and used them for de pendency parsing. also, these approaches utilize zeroshot parsing since the trained parsing mod els parse a target language without any training target instances @xcite . it is a suitable method for parsing lowresource languages because knowledge between different languages is transferable and labeled lowresource language data is difficult to obtain. for xibe, there are currently no parallel corpora or machine translation systems available, which makes model transfer the most feasible approach. in order to achieve zeroshot singlesource cross lingual parsing, we first train a parsing model on one source language treebank, then parse the tar get language using this model. as xibe is written in the traditional mongolic alphabet, which differs greatly from all the candidate source languages, we must minimize these differences. therefore, we use treebank delexicalization by replacing lexi cal items with only partofspeech tags in both the source and target languages. ## source language selection methods in this section, we describe the three methods for determining the best source languages, and present the languages chosen by these methods. ## results and analysis in this section, we provide the results for our three research questions. ## conclusion in this research, we have investigated crosslingual dependency parsing for xibe. as we do not have parallel data or a machine translation system for this language, we delexicalize treebanks to avoid orthographic and lexical differences. we propose three criteria to select source languages, that is, ty pology, perplexity, and automatic predictions by the langrank tool. then, we train parsing mod els with udify and test them with the three sets of xibe data. our results demonstrate that syntactic similarity is considered the most important factor in delexicalized crosslingual parsing. japanese is found to be the optimal source language for pars ing xibe. differences in genre also influence pars ing. parsers trained on simpler sentence structures cannot analyze more complex test data. in our current work, we use only one source lan guage to parse xibe. we will determine the best concatenation of source languages for multilingual parsing in the future. additionally, we will re segment the current units of the korean treebanks into smaller units and create dependency relations by rules in order to determine if a more similar seg mentation will lead to an improvement. alterna tively, we can use lexical information in parsing, such as creating a bilingual dictionary or training xibe word embeddings.