Whit’s the Richt Pairt o Speech: P o S tagging for S cots (2021)

Abstract: In this paper we explore PoS tagging for the Scots language. Scots is spoken in Scotland and Northern Ireland, and is closely related to English. As no linguistically annotated Scots data were available, we manually PoS tagged a small set that is used for evaluation and training. We use English as a transfer language to examine zero-shot transfer and transfer learning methods. We find that training on a very small amount of Scots data was superior to zero-shot transfer from English. Combining the Scots and English data led to further improvements, with a concatenation method giving the best results. We also compared the use of two different English treebanks and found that a treebank containing web data was superior in the zero-shot setting, while it was outperformed by a treebank containing a mix of genres when combined with Scots data.

## introduction part-of-speech (pos) tagging is an essential natural language processing (nlp) tool that is often seen as a first step in language analysis @xcite and can be useful for the analysis of large corpora, for instance in digital humanities @xcite . for high resource languages, pos tagging is occasionally considered to be a solved task, with recent state-of-the-art pos taggers reaching high accuracies, such as @xcite , who report an average accuracy of 97.5% across seven high-resource languages. however, these strong results are limited to languages and domains with large training data sets. state-ofthe-art models commonly use bi-directional long short term memory recurrent neural network (bi-lstm rnn) architectures that rely on supervised learning methods. in order to obtain a high accuracy, large amounts of labelled data are required @xcite . these resources are not available for low resource languages, and due sae a thocht a'd gie ma pynt o view on aw 'is, an 'e story ahint whit caused iz tae stairt the editathon muivement an whit a hink anent the hale hing. on 'e 25t august 2020, a 4chan an reddit post wis pit oot bi ryan dempsey, unner the uisername 'ultach', fae ulster, anent an american wikipedia admin, whase name a willnae mention tae jouk thaim gittin ony mair potential harassment (lat's juist caw thaim admin fae nou), an hou thay contríbutit near hauf o aw the airticles on 'e wikipedia, an hou the ither tap fower contríbutors wis american an aw. figure 1 : scots excerpt, originally from @xcite to the high cost of annotation these are unfeasible to obtain @xcite . in this paper we focus on the scots language, a language closely related to english, which is spoken in scotland and northern ireland. scots lacks linguistic resources and tools. in this paper we take a first step towards overcoming this, by examining pos tagging for scots. we investigate how well an english pos tagger works on scots, and try to improve on this by using a very small amount of scots annotated data during training. as part of the study, a small sample of scots texts were manually pos tagged. an example of scots is shown in figure 1 . while statistics-based pos taggers are occasionally used for truly low-resource languages @xcite , data scarcity can also successfully be alleviated using transfer learning @xcite . pos tagging using cross-lingual transfer has proved effective on low-resource simulations of high-resource european languages, and even disparate non-indo-european languages @xcite . yet, few attempts have been made to use cross-lingual models to tag european minority languages such as scots, despite the existence of closely related high-resource languages. one exception is @xcite who focused on pos tagging for three regional french languages. we focus on investigating pos tagging for scots. first we want to investigate a zero-shot setting, to see how well a pos tagger trained for english works on scots, additionally comparing the performance of two different english data sets, the gum treebank @xcite and the ewt treebank @xcite . secondly, we annotate a small amount of data, 1040 tokens, in order to investigate how much could be gained from adding even a very small amount of data. we compared different methods for combining the scots and english data, with zero-shot tagging and training only on a tiny amount of scots data. the main contributions of this work are: we believe that our study presents a method that can be used to create nlp tools for many other minority languages with scant resources that are closely related to a high-resource language, such as low saxon and occitan. these nlp tools present a valuable lifeline that enables increased usage of minority languages in digital settings. ## scots scots is a west-germanic language spoken by 1.5 million people. 1 while recognized as a language by the scottish government, scots' status as a language @xcite is disputed, with 64% of respondents in a 2010 scottish government survey seeing scots as a way of speaking rather than a language. foot_1 scots diverged from english in the 13 th century and use of scots was widespread for both government and literary purposes until the 16 th century @xcite . since the union of the crowns, scots has undergone a process of anglicization, and has largely been superseded by scottish english with which it is spoken on a bipolar spectrum. currently, scots is a low-resource language. scots texts are readily available, but parallel corpora or gold-standard annotations are lacking. this severely complicates the analysis of scots. in recent years, scottish identity has experienced a revival, and with it interest in the scots language. scots is reemerging as an active language, especially amongst the young rural population of scotland and amongst scottish nationalists @xcite . although this has generated academic interest in the fields of phonology @xcite and socio-linguistics @xcite , no nlp-tools are available specifically for scots. the only scots nlp efforts are two sets of word embeddings, namely fasttext embeddings @xcite and polyglot word embeddings @xcite . there are several complications relating to nlp for scots. despite attempts at standardization by the report & recommends o the scots spellin comatee @xcite , scots does not have a de jure standard. as there is no official orthography, major variations in spelling exist, mostly depending on the written dialect of scots. additionally, almost no parallel english-scots texts are publicly available, which excludes approaches relying on such data. scots grammar and morphology, while similar to and not necessarily more or less complex, differs from english grammar and morphology. some specific features of scots is its irregular plural forms for nouns, e.g. cauf/caur for "calf/calves", and frequent use of progressive forms, e.g. who's wantin tae go oot wi you -"who wants to go out with you" 3 , 3 related work ## data in this section we describe the data used. we use two existing english data sets, and create a small new set for scots. the tags used for all the data are the universal ud pos tags @xcite . this tagset consists of 17 tags and only uses basic lexical categories that are seen as applicable to all languages. examples of categories are: noun (noun), adjective (adj), and adposition (adp). ## models & experiments all models were run on the bi-lstm pos tagger from @xcite running on the standard settings, bar the addition of the polyglot embeddings @xcite . @xcite 's tagger uses a bi-lstm for character embeddings, the output of which is concatenated with the word embeddings. these embeddings are the input for the bilstm which has the softmax for the pos tags as an output layer. the network used 100-dimensional character embeddings, 100 dimensional polyglot embeddings, 100-dimensional lstm hidden states and had one hidden-layer for the encoder and the decoder. during training, stochastic gradient descent was applied with a dropout rate of 0.25 as well as a sigma of 0.2 gaussian noise. the network used a softmax activation function in the decoder to generate the tags. all models were trained for 20 iterations with an early stop of four iterations. these hyperparameters were chosen to prevent training from finishing pre-maturely, although no model trained for more than 19 iterations. for our experiments, we first trained monolingual baseline models for english and scots. then we explored three ways of combining data across languages. in all cases we also compare the results with the two english corpora. ## results as can be seen in an option would be to separate a word like wisnae into wis+nae, according to ud guidelines ( https://universaldependencies. org/u/overview/tokenization.html ). however, this would lead to a call for a tokenizer especially adapted to scots, which is currently not available. the differences between the ewt concatenated model and the ewt zero-shot baseline are much smaller, as can be seen in the ewt concatenated model also contains unexpected errors. despite occurring in both english and scots, auxiliary verb can was tagged as both an adv and a noun. another error can possibly, though not necessarily, be attributed to the quality of the word embeddings. e "he" or "the", is consistently tagged as num, despite it not having this tag in the training data, although this could also be due to the model parameters. comparing the gum and ewt concatenated models in ## conclusion we have annotated a small amount of scots data with pos tags, and shown how this data can be used to improve tagging for scots. the best results were obtained by a simple concatenation model, where a large english data set was concatenated with a small amount of scots data. we also found that while the english data ewt data set containing web data, performed better than the gum treebank with mostly edited data in a zero-shot setting, the gum treebank, trained on a mix of genres performed better in the cross-lingual setting. while greatly improving on the zero-shot model and slightly improving on the scots-only model, the cross-lingual models for both data sets scored lower than other studies that examine low-resource settings for pos tagging. longer training, higher dimensional embeddings and modifications to the model architecture such as private lstms and the use of adversarial training could be made. @xcite achieve higher accuracies for all languages using 320 tokens per language, showing the efficacy of the combination of these factors. an improvement on the model architecture can be made as well by, rather than weighting english and scots data equally in the loss function as was done in these experiments, weighting the scots data more heavily in the loss function of the bi-lstm in order to prevent underfitting on the scots data. another possible cause of these results is the previously mentioned questionable quality of the word embeddings, and we think it would be interesting to train scots word embeddings on verified scots data, to be able to compare to the current embed-dings trained on wikipedia data with questionable quality. we also think it would be interesting to explore text normalization strategies for scots. lastly and perhaps most importantly, an increase in the size of the scots data set and an interest in developing nlp tool for minority languages is required to improve performance of the tagger. other models that examine cross-lingual transfer, such as @xcite use 900 training tokens minimally, nearly double the tokens used in this paper. @xcite also note that improvements are smaller for cross-lingual transfer than in cross-domain training, strengthening the case for the treatment of scots as a language in terms of the required nlp resources. combined with the low accuracy for the zero-shot model this indicates the need for additional resources for scots, such as dictionaries and parallel corpora.