How Low is Too Low? A Computational Perspective on Extremely Low-Resource Languages (2021)

Abstract: Despite the recent advancements of attention-based deep learning architectures across a majority of Natural Language Processing tasks, their application remains limited in a low-resource setting because of a lack of pre-trained models for such languages. In this study, we make the first attempt to investigate the challenges of adapting these techniques to an extremely low-resource language – Sumerian cuneiform – one of the world’s oldest written languages attested from at least the beginning of the 3rd millennium BC. Specifically, we introduce the first cross-lingual information extraction pipeline for Sumerian, which includes part-of-speech tagging, named entity recognition, and machine translation. We introduce InterpretLR, an interpretability toolkit for low-resource NLP and use it alongside human evaluations to gauge the trained models. Notably, all our techniques and most components of our pipeline can be generalised to any low-resource language. We publicly release all our implementations including a novel data set with domain-specific pre-processing to promote further research in this domain.

## introduction sumerian is one of the oldest written languages, attested in the cuneiform texts from around 2900 bc and possibly the language of even older protocuneiform texts from the second half of the 4th millennium bc @xcite . specialists in assyriology have recently worked to digitize sumerian scripts, annotate, and translate a part of them to modern-day languages like english and german. in this work, we attempt to create the first information extraction and translation pipeline for data sets and training subroutines are available at linktr.ee/rachitbansal † work was done prior to joining amazon at goethe university frankfurt 1. a-na-ah-i3-li2 anah-ili; 2. szu ba-an-ti did receive. sumerian. specifically, we focus on machine translation from sumerian to english, and sequence labeling tasks of named entity recognition (ner) and part of speech (pos) tagging. figure 1 shows a sample of our raw data where the sumerian text has been derived from the tabletinscribed cuneiform script along with its humaninterpreted english translations. creating an annotated corpus for such a language is a tedious task. we obtain our data from openly available sources and corpora, painstakingly annotated and translated by human experts. yet, for languages like sumerian, which are not fully-understood by humans themselves, transferring knowledge and patterns to learning algorithms from this limited data becomes extremely difficult. the consequent challenge posed for ner and pos tagging is evident. lack of annotated data and fuzzy characterlevel text makes it hard for a model to generalise, irrespective of its size. in case of machine translation, the labeled data is composed of incomplete and short phrase-like sentences, especially on the target side. this makes the context largely ambiguous. moreover, we find that for a majority of medieval and ancient languages the target-side translated text is highly incoherent with modern-day english language text, making it impossible to use the latter in semi-supervised and unsupervised settings. throughout this study, we elaborate on such challenges faced when working with low-resource languages, and talk about what makes some of these languages like sumerian 'extremely' lowresource. through extensive experimentation, evaluation, and analysis we further introduce specific algorithms and modifications to work around them. in all, our contribution is three-fold: 1. building and analyzing a variety of algorithms on the unexplored human-annotated sumerian dataset for sequence labeling tasks of pos tagging and ner. ( §3) 2. introducing the problem of target-side incoherence for low-resource settings and its effect on semi-supervised and unsupervised machine translation ( §4.2). further investigating specific modifications and methodologies to cope-up with these constraints. ( §4) 3. introducing interpretlr, a generalisable toolkit to interpret low-resource nlp. we apply it to further study, compare, and evaluate all of our proposed techniques for machine translation and sequence labeling. ( §7) throughout this work, we have conducted human studies and evaluation for our models, in addition to automated metrics. for gauging our models with interpretlr, we have made use of human annotations. ## machine translation in this section, we present our experiments for machine translation, primarily focusing on specific data and algorithmic modeling techniques which may be generalised for any extremely lowresource language that may or may not suffer from target-side incoherence, a phenomenon which we also introduce herein. all results are summarised in ## experimental setup geforce rtx 2070 gpu, while the pre-training and fine-tuning of flair, roberta, and xlm on various data configurations were performed on 2 16 gb nvidia v100 gpus. we used development sets to tune the hyper-parameters for all our models, especially those for pos and ner. for roberta and vanilla transformer, @xmath0, while @xmath1. a beam-size of 5 was used for our ft experiments. adam (kingma and ba, 2015) optimiser with a learning rate of 0.001, β 1 = 0.90, β 2 = 0.98 and a decay factor of 0.5 was used. additional regularisation was done via dropout and attention dropout (wherever applicable) layers with p @xmath2.1. we used a batch size of 32 or 64 and an early stopping criteria based on the validation loss. ## results and analysis sequence labeling tables 2 and 3 represent the metric scores of our different models for pos and ner tasks, respectively. crf with domain-specific rules gives the best f1-score for the pos tagging task, even better than the complex roberta and flair language models which are the current state-of-the-art techniques for most languages. the prevalence of distorted words and short phrases in the corpora makes context learning difficult, although the domain-specific rules help learn shortterm dependencies by learning feature weights. roberta performs well for both of the tasks, while being the best among others for ner (95.37 f1 score). to make the most out of the limited vocabulary and noisy text, we used byte-level bpe @xcite to train the language model and further fine-tuned it on our pos and ner dataset with a batch size of 128. we also tried flair language model across various word embeddings (character, word2vec, fasttext, glove) along with an additional crf layer for both of the tasks. although a high precision is observed using this approach, the f1 scores is seen to be significantly low due to low recall. in addition to the f1 metric we also conducted human evaluation by language expert for the best performing models, out of randomly selected 76 (496 words) phrases, only 8 and 6 words were misclassified by ner and pos models, giving an error of 1.20 and 1.61%, respectively. machine translation ## interpretability analysis oftentimes in case of deep learning architectures, metric scores like accuracy, f1 and bleu are unable to portray the true behavior of the models. for languages like sumerian, the human-understanding itself is scarce. visualizing the representations and correlations made by the model could provide insights into which elements of the context can give additional information to support semantic analysis of the terms. thus, we herein introduce a generalisable interpretability toolkit, interpretlr, to interpret algorithms for low-resource nlp and 6 elaborate evaluation criteria mentioned in the appendix. interpretlr is primarily aimed at fabricating attribution saliency maps, i.e., tracing back the model output so as to assign an importance score to each input token, based on its 'influence' on that output. we do this using two kinds of interpretability techniques-gradient-based @xcite @xcite , and perturbation-based @xcite . due to the inherently discrete nature of natural language text, the starting point for all our approaches is the embedding of the input sentence across the model to interpret. most of our analysis is done for the encoder of the network architecture, thus analyzing the effect of different pre-training and fine-tuning techniques on how the model eventually represents the language attributes. we use the word 'attribution' as a better-defined substitute for the 'influence' measure of an input span of text on the output. a part of our visual analysis is shown and elaborated here, while a complete analysis with all our models and layer-wise heat-maps is presented in the appendix. in in general. even the unsupervised models that do not have access to the one-to-one mapping of the translation during training manage to infer these words from the appropriate context. it can be assumed that they learn the right representations of such tokens. but at the same time, there are instances like sze-ba (barley), which the two unsupervised models rightly refer to but do not give the right translations, which thus is a direct result of the absence of supervision. lastly, english words like under, of and from do not have any direct translations in sumerian and are mostly inferred from the context, even by the human annotators. at such places, again, supervision might play a critical role as in the 4 th row of ## conclusion in this work, we introduced the first information extraction and translation pipeline for sumerian cuneiform. we first undertook the tasks of pos tagging and ner, where we observed that deeper is not necessarily better. a simple crf model with well-defined rules outperformed the large language model roberta for pos tagging. further, for machine translation we overcame unprecedented challenges pertaining to lack of in-domain text, sparse sentence formation, and incoherence. we found that using out-of-domain text along with specific data-augmentation can have huge impacts in a low-resource setting. even though the gradient-based methods are much faster than perturbation-based methods, we observe that the heavy dependency of ig on hyperparameters like the number of input steps to be considered when going from a baseline to the actual input, n steps, to be a major setback. the final attribution is generally found out after integrating (or summing) over the attributions of these sub-steps. we found that the attributions do not change when going beyond n @xmath3, thus, we experiment by varying it between 10 to 250. we observe that there is no ideal value of n steps, ig's faithfulness to the model varies largely over this range. for some inputs, the best value is n @xmath4. we judge this by considering how much the attribution is given to sos and eos tokens for each output token. thus, based on both plausibility and faithfulness. we use n @xmath5. figure 10 represents the visualization for our sequence labeling tasks. it indicates two major things, 1) the effect of words, sub-words (depends on tokenization) on tagging the target word and 2) the effect of 6 transformer encoder layers. we created the hook on embeddings of roberta with layer ig and obtained the visualizations for how each sub-word is contributing to tag the target word. similarly, to obtain the heat-map we created the hook on roberta embeddings and used the layer conductance. from figure 10a it can be observed that ku and du contribute the most to the attribution scores for tagging ku3-babbar and ba-du3 as a noun (n) and verb (v), respectively. from the heat-maps it is also noted that ku shows the effect on all 6 layers whereas in second example effects are majorly due to the initial transformer layers. similarly in the figure 10b ur and lugal are the most effective subwords to tag ur-bi2-lumki and lugal-tesz2-mu as gn (geographical name) and pn (personal name) respectively. it is also interesting to note that both