Challenges in Annotating and Parsing Spoken, Code-switched, F risian- D utch Data (2021)

Abstract: While high performance have been obtained for high-resource languages, performance on low-resource languages lags behind. In this paper we focus on the parsing of the low-resource language Frisian. We use a sample of code-switched, spontaneously spoken data, which proves to be a challenging setup. We propose to train a parser specifically tailored towards the target domain, by selecting instances from multiple treebanks. Specifically, we use Latent Dirichlet Allocation (LDA), with word and character N-grams. We use a deep biaffine parser initialized with mBERT. The best single source treebank (nl_alpino) resulted in an LAS of 54.7 whereas our data selection outperformed the single best transfer treebank and led to 55.6 LAS on the test data. Additional experiments consisted of removing diacritics from our Frisian data, creating more similar training data by cropping sentences and running our best model using XLM-R. These experiments did not lead to a better performance.

## introduction as parsers are improving (with currently scores higher than 96 @xcite ), parsing scores for low-resource languages lag behind. in recent years there has been an increase in interest in ways to parse and annotate these languages. this paper will focus on parsing the low-resource language frisian (germanic language spoken in the north-western part of the netherlands, approximately 612.000 speakers foot_0 ). we will use spontaneous speech data containing code-switches from the fame! corpus created by @xcite . we annotate a small portion of this data with upos tags and universal dependencies @xcite for evaluation purposes. an example of such an utterance can be seen in figure 1 . because it is unclear which (parts of) datasets are likely to be good candidates for training a parser for spoken frisian, we propose to use instance-based selection from a variety of sources. this leads to our main research question: can automatic data selection on the instance level outperform selection on the treebank level for training a parser for a new target domain/language? after finding our best model, we will try to improve our results by removing diacritics in our frisian data and by making our train data more similar to spoken frisian. all data and code are available on github foot_1 . ## related work previous work on dependency parsing for spoken data created english annotations for conversational agents @xcite and slovenian data @xcite . @xcite create a treebank for spoken komi-zyrian with code-switching to russian. contemporary to our treebank, c ¸etinoglu and c ¸öltekin (2019) created a treebank for spoken code-switched turkish-german. they adapt the guidelines to deal with the spoken and code-switch nature of the data. @xcite create a treebank for an arabic dialect that contains a high amount of code-switching. @xcite and @xcite show that annotating a small amount of target language and target domain data outperforms a cross-lingual setup, but we focus on a pure zero-shot scenario instead. previous work on zero-shot parsing has mainly focused on annotation projection using parallel data @xcite and selecting transfer treebanks @xcite . previous work exploring more fine-grained selection methods are either focusing only on 3 domains (and multiple topics within the news) and 3 corpora @xcite , or focus on parser selection during test time @xcite . ## results overall, our best model consisted of 128 components and used 2000 sentences. a closer look at the data selection of our model ( ## error analysis we perform an error analysis of our best model and the baseline (nl alpino) on the development data. the confusions on dependency labels are similar for both models ( in an attempt to obtain better results we have tried three different approaches to improve our best model. @xcite have shown that transliteration to a script of a related high source language on which the language model is trained leads to better results. even though frisian is not in a different script as the training data, it does contain a high amount of diacritics, which results in both many unseen wordpieces during testing, as well as differences in tokenization foot_3 . we attempt to overcome this mismatch by removing the diacritics from the characters in our frisian data. in our development set, which consists of 150 utterances, there are 44 utterances with at least one diacritic and there are a total of 53 tokens with diacritics. the second approach tries to make our training data more similar to our frisian spoken data. a similar approach is taken by @xcite , they create synthetic data following internet-specific conventions and syntactic features of african-american english. this synthetic data proved to be helpful for performance. @xcite also try similar methods of data augmentation and found that when no source treebank is available, data augmentation can be very helpful. in our case, the utterances are not always "full sentences". an example from our development set is "dêr kinnen we in hele hoop oer sizze mar" ("we can say a lot about that but"). normally the sentence would continue after "mar" (but) and "mar" would connect to this next part, but in this case the utterance stops at this point. we chose to connect this "mar" with an orphan relation to the root. these kind of "non-standard" constructions occur relatively often in our target data compared to the non-spoken treebanks in our training data. we tried to make our training data more similar by cropping sentences and adding an orphan relation. we did this for approximately the same percentage of utterances that have this construction in our development data (approximately 9 %). as a final experiment we replaced mbert with xlm-r. the results of all three experiments can ## conclusion in this paper we explored parsing and annotating the low-resource language frisian and we have shown that selecting more similar training data (by using lda) can lead to improvements in scores. these improvements were not significantly better compared to our baselines, but it did show that selecting fewer instances can outperform single treebank training. besides slightly higher scores, using fewer instances sped up the training process. additional experiments of removing diacritics and adjusting the training data to our development set did not show improvements. future research can focus on improving the data selection method (using a different selection approach will probably result in different outcomes), weighing the importance of the selected instances and combining data selection with orthogonal approaches.