SFE - AI at S em E val-2022 Task 11: Low-Resource Named Entity Recognition using Large Pre-trained Language Models (2022)

Abstract: Large scale pre-training models have been widely used in named entity recognition (NER) tasks. However, model ensemble through parameter averaging or voting can not give full play to the differentiation advantages of different models, especially in the open domain. This paper describes our NER system in the SemEval 2022 task11: MultiCoNER. We proposed an effective system to adaptively ensemble pre-trained language models by a Transformer layer. By assigning different weights to each model for different inputs, we adopted the Transformer layer to integrate the advantages of diverse models effectively. Experimental results show that our method achieves superior performances in Farsi and Dutch.

## introduction ner is an essential tool in the application fields of information extraction, question answering system, syntactic analysis, machine translation, etc. it plays a vital role in the process of the practical application of natural language processing technology @xcite . however, processing complex and ambiguous named entities (nes) is a challenging nlp task in practical and open-domain settings, but has not received sufficient attention from the research community. complex nes, like the titles of creative works (movie/book/song/software names), are not simple nouns and are harder to recognize @xcite . they can take the form of any linguistic constituent, like an imperative clause ("dial m for murder"), and do not look like traditional nes @xcite . this syntactic ambiguity makes it challenging to recognize them based on their context. such titles can also be semantically ambiguous, e.g., "on the beach" can be a preposition or refer to a movie. finally, such entities usually grow faster than traditional categories, and emerging entities pose yet another challenge. in recent years, the deep learning methods have achieved great success in nlp (natural language processing). however, some limits need to be addressed. challenges for deep learning in nlp mainly arise from the scarcity of labeled data. one way to alleviate the need for large labeled datasets is to pre-train a model on unlabeled data via selfsupervised learning, and then transfer the learned model to downstream tasks @xcite . these methods have been widely applied and have made a massive breakthroughs, such as bert @xcite , xlnet @xcite and gpt @xcite . pre-trained neural models (e.g., transformers) have produced high scores on benchmark datasets like conll03/ontonotes @xcite . however, as noted by @xcite , these scores are driven by the use of well-formed news text, the presence of "easy" entities (person names), and memorization due to entity overlap between train/test sets; these models perform significantly worse on complex/unseen entities. researchers using ner on downstream tasks have noted that a significant proportion of their errors are due to ner systems failing to recognize complex entities @xcite . vised word representations. despite bilstm's great success in the ner task, it has to compute token representations one by one, which massively hinders full exploitation of gpu's parallelism. therefore, cnn has been proposed to encode words concurrently. in order to enlarge the receptive field of cnns, iterative dilated cnns (idcnn) ( ḟ ẏu and koltun, 2016) was used. pre-training language model. recent largescale language model pre-training methods such as bert @xcite , xlnet @xcite and gpt @xcite , further boosted the performance of ner, yielding state-ofthe-art performances. ## task descriptions this task challenges nlp enthusiasts to develop complex named entity recognition systems for 11 languages ( in this work, we conducted experiments on farsi and dutch). the task focuses on detecting semantically ambiguous and complex entities in short and low-context settings. the task also aims at testing the domain adaption capability of the systems by testing additional test sets on questions and short search queries. ## datasets for the entity recognition task @xcite on both farsi and dutch, there are 15300 pieces of training data. for farsi, the longest sentence has 46 words, and the shortest sentence has only 1 word, with an average length of 18 words. for dutch, the longest sentence has 47 words, and the shortest sentence has 2 words, with an average length of 15 words. this means that the length of the current corpus is too short, which means that it is difficult to use a large number of sentence models to cover a large proportion of the length of the current corpus. in both tasks, there are six types of entities (malmasi et al., 2022a), cw (creative work), per (person), loc (location), grp (group), corp (corporation), prod (product). the distribution of entity types is relatively average. the lowest proportion is prod, accounting for 12.9%, and the highest proportion is loc, accounting for 24.9%. many types of entities, short sentence length, lack of sufficient context information, and open-domain information extraction all increase the difficulty of these tasks. ## systems description overview of our architecture for the ner task was shown in figure 1 . the system mainly includes two parts: data augmentation, pre-trained language model ensemble. ## results and discussion the results on both farsi and dutch data-sets are shown in the reason lies in case-sensitive. in many cases, the case of letters plays a vital role in identifying entities. for example, 'wat is' is a very prevalent query in dutch, but 'is' also represents an organization (islamic state of iraq and al-sham). augmentation ratio acc recall f1 0 + 0.769 0.811 0.789 1 + 0.772 0.811 0.791 2 + 0.776 0.819 0.797 3 + 0.775 0.816 0.795 ## conclusions this study describes an effective ner system in the semeval 2022 task 11: multiconer, the experimental results demonstrated the effectiveness of our ensembled large pre-trained language models on low-resource named entity recognition. in future work, we would further improve the performances by exploiting a more complicated ensemble strategy with more diversified models.