Few-shot Learning for S umerian Named Entity Recognition (2022)

Abstract: This paper presents our study in exploring the task of named entity recognition (NER) in a low resource setting, focusing on few-shot learning on the Sumerian NER task. The Sumerian language is deemed as an extremely low-resource language due to that (1) it is a long dead language, (2) highly skilled language experts are extremely scarce. NER on Sumerian text is important in that it helps identify the actors and entities active in a given period of time from the collections of tens of thousands of texts in building socio-economic networks of the archives of interest. As a text classification task, NER tends to become challenging when the amount of annotated data is limited or the model is required to handle new classes. The Sumerian NER is no exception. In this work, we propose to use two few-shot learning systems, ProtoBERT and NNShot, to the Sumerian NER task. Our experiments show that the ProtoBERT NER generally outperforms both the NNShot NER and the fully supervised BERT NER in low resource settings on the predictions of rare classes. In particular, F1-score of ProtoBERT on unseen entity types on our test set has achieved 89.6% that is significantly better than the F1-score of 84.3% of the BERT NER.

## introduction named entity recognition (ner), as a fundamental task in natural language processing, aims to locate and classify named entities such as people, organizations, and locations, etc. the ur iii period @xcite , spanning about 100 years, has a particularly rich source of texts, comprising at least 100,000 documents. these are primarily financial records and potentially support investigations of economic activity in ancient mesopotamian society. to give but one example, @xcite aims to do a prosopographical study of individuals delivering animals to the puzriš-dagan organization during the ur iii period and identifies the individuals, their family relations and royal status of the historical actors delivering animals, as well as the variety of animals involved. ner applied to this domain can efficiently help assyriologists recover and analyze the social-economical activities and thus provide a better understanding of the social organization and dynamics of ancient mesopotamian history. there is a broad effort in the community of assyriologists, in collaboration of computer scientists, to build reproducible socio-economic networks from the ur iii archives @xcite . this effort shows that the application of ner to these texts is of great use in the quantitative study of assyriology. in this work, we conduct experiments to apply models that are based on prototypical networks @xcite and nearest neighbour classification to the sumerian ner task. specifically, we adapt two few-shot learning systems, protobert @xcite and nnshot @xcite , to the sumerian ner task and have achieved good performance in prediction of rare classes. in summary, our contributions are as follows: (1) we construct two few-shot learning systems, protobert and nnshot, and apply them on the sumerian ner task. to the best of our knowledge, this is the first work exploring sumerian ner task using the few-shot learning approach. ## previous work previous studies on sumerian ner are few, partially due to the lack of language resources and meaningful collaborations between researchers in computer science and assyriology. the few studies include @xcite and @xcite . @xcite uses the dl-cotrain algorithm for personal name identification to minimize the use of the annotated data. the system achieves a high recall (92.5%) and a low precision (56.0%). @xcite chooses to fully utilize the annotated data by applying a wide range of supervised algorithms, including decision tree, gradient boosting, logistic regression, naive bayes, svm and random forest, to predict personal names. the supervised approach shows an opposite behavior with a low recall (around 65%) and a high precision (around 86%). a more recent work @xcite investigates the sumerian machine translation task in low-resource settings. they also built a variety of algorithms, including hmm, rules+crf, bi-lstm+crf, flair and roberta, on the pos and ner tasks on the sumerian dataset. for the ner task, roberta achieves the best f1-score (95.3%) on a set of 12 entity types and the simple crf model with well-defined rules significantly outperforms the rest of the models and is the second best with f1-score of 91.3%. we adapt their labelled sumerian dataset in this work. as they only reported the overall f-scores of those ner systems on the 12 entity types without describing how the dataset is split, their result is not directly comparable with ours. more details about the dataset will be given in section 3. one of the key problems for low-resource ner is the lack of annotated data. as one of the common strategies, cross-lingual ner attempts to address this challenge by transferring knowledge from one or more high-resource source languages with abundant annotated data to a low-resource target language with few or no labels. the knowledge transfer is either through annotation projection from the source language to the target language @xcite @xcite or through using a shared encoder in a multi-task architecture @xcite . along the research line of enabling parameter reuse across a variety of tasks, @xcite proposes mad-x, an adapter-based framework which includes language adapters, task adapters and invertible adapters, in a multilingual context. mad-x outperforms the state of the art in cross-lingual transfer on ner across diverse languages. however, the highest f1-score is achieved on arabic which is 59.41%. for other low-resource languages, such as icelandic, quechua and so on, the f1-scores are mostly around 30-50%. cross-lingual meth-ods have achieved notable success, but in certain circumstances, such as insufficient pre-training corpora or when the target language is far from the source language, their performance suffers. sumerian language, as a long dead language, suffers both which makes the cross-lingual methods not readily apply. few-shot classification @xcite can effectively recognize new classes from very few labelled examples and thus has recently drawn a lot of attention. @xcite proposed prototypical networks based on the idea that there exists an embedding space in which images of the same class cluster around a single prototype representation for each class. in other words, two images of the same class should be close to each other, and two images of the different class will be far away. adapting this idea from image classification, @xcite al. (2019); hou et al. (2020); ding et al. (2021) address the few-shot ner problem and have achieved considerable success. @xcite proposed token-level nearest neighbor classification based methods for the few-shot ner problem to address some potential issues of prototypical ner in learning class prototypes, such as learning a noisy prototype of the 'o' class. usually the overall f1-scores are high if bert is chosen as backbone encoder in deep learning ner systems @xcite . however, @xcite demonstrates that bert fails to predict minority classes when the number of examples is limited. they observe that bert needs at least 25 examples of a minority label to start learning on the connl-03 dataset @xcite . if the examples are fewer than 25, the f1score will be 0. when the examples exceed 100, the performance improves rapidly. they also observe similar phenomena on other datasets. for example, learning on the jnlpba dataset @xcite requires at least 50 examples. they also construct a prototypical few-shot learning model to overcome bert's limitation. the results show the few-shot learning model consistently surpasses the performance of bert on minority classes. for instance, it is outperforming bert by 40 f1 points on loc class when the dataset has 15 sentences containing that class. all this has shown that fewshot learning is well suited for the setting when the number of labelled examples is very constrained, which further justifies our choice of exploring this approach in the sumerian ner task. and our experimental results have also echoed their findings. the paper is organized as follows. in section 3 we give a more detailed description of the sumerian ner task and the dataset. in section 4 we describe our models. section 5 contains the description of our experimental setup and the report and analysis of the results. we conclude our work with some discussion of the future work in section 6. ## dataset and task description the dataset we used in this work is originally from the cdli database (cuneiform digital library initiative http://cdli.ucla.edu/ ). cdli is a project that curates an electronic documentation of ancient cuneiform texts, comprised of cuneiform texts, images, transliterations and sundry information concerning the ur iii period and its immediate aftermath. it is a joint project of the university of california, los angeles, the university of oxford, and the max planck institute for the history of science, berlin. although the texts we are investigating were originally written in cuneiform script, cdli provides them in transliterated form, using the english alphabet. fig. 1 shows a tablet from cdli repository, (id p407107). an image of the original tablet with its cuneiform inscription is on the left; the transliteration is in the middle and the modern english translation appears on the right. as aforementioned, we adapt the labelled sumerian dataset and the tagset directly from @xcite . the dataset has 22,728 sentences, 61,478 tokens, and 12 entity types (not including the o type that indicates a word is not a named entity of interest). all these entity types, their meaning and counts in the dataset are shown in the counts in ## methods in the following, we will describe three models applied to the low-resource sumerian ner task. they are bert+lc, protobert and nnshot. as transformer-based pre-trained language models @xcite have shown significant impact on the ner task, a pre-trained language model (plm) on sumerian will also be used in our ner models. in our experiment, we adapt a plm on sumerian explored in @xcite . the plm is pre-trained using roberta @xcite on their sumerian monolingual dataset. ## conclusions and future work we have applied three models, bert+lc, nnshot and protobert, to explore the sumerian ner in low resource settings, and have presented our preliminary results. this is the first work of exploring few-shot ner on the sumerian language dataset. our experiments show that protobert as a fewshot learning model has consistently outperformed the fully supervised model bert+lc model in few-shot settings and has generally achieved better performance than nnshot. though as a token-level nearest neighbour classification method, nnshot is less sensitive to the noisy 'o' type that is introduced by the relabeling step, it may not be as stable as protobert owing to the nearest neighbor mechanism in the training stage. we show that bert-lc fails to do a good job in learning more examples in few-shot settings. while we investigate the efficacy of prototypical networks-based protobert and nearest neighbour metric-based nnshot learning models in the few-shot sumerian ner task, it will be particularly interesting to 1) extend our work to a larger test set; 2) explore new methods such as structshot @xcite to solve the noisy 'o' type issue introduced by relabelling; 3) experiment on using more sophisticated cross-lingual approaches including adapter-based models on sumerian ner.