Neural language model embeddings for Named Entity Recognition: A study from language perspective (2023)

Abstract: Named entity recognition (NER) models based on neural language models (LMs) exhibit stateof-the-art performance. However, the performance of such LMs have not been studied in detail with respect to finer language related aspects in the context of NER tasks. Such a study will be helpful in effective application of these models for cross-lingual and multilingual NER tasks. In this study, we examine the effects of script, vocabulary sharing, foreign names and pooling of multilanguage training data for building NER models. It is observed that monolingual BERT embeddings show the highest recognition accuracy among all transformerbased LMs for monolingual NER models. It is also seen that vocabulary sharing and data augmentation with foreign named entities (NEs) are most effective towards improving accuracy of cross-lingual NER models. Multilingual NER models trained by pooling data from similar languages can address training data inadequacy and exhibit performance close to that of monolingual models trained with adequate NER-tagged data of a single language.

## introduction named entity recognition (ner) is one of the active areas of research in natural language processing that involves automatic tagging of names of people, places, organizations, etc., in natural language text. ner finds its use in applications like information extraction, summarization, and questionanswering. as names are out-of-vocabulary words for any given language, ner models try to learn the relationship of these named entity (ne) words with the surrounding words from the context. both statistical @xcite and deep neural network based techniques have been used for ner modeling. however, deep neural networks have been found to be effective in learning latent relationships from long context @xcite . therefore it has been the subject of much of ner research in recent past. in this context, ner models based on bi-directional long-short-term memory networks (bi-lstm) @xcite coupled with conditional random fields (crf) have shown impressive performance. however, such architectures are limited by sequential processing resulting in slower performance. ner models trained using embeddings generated by neural lms based on transformer architecture @xcite have shown state-of-the-art performance on ner tasks. the improvement in accuracy has been due to the attention mechanism being able to capture the relevant relationships effectively, while inherent parallelism in transformer architecture has been able to improve the processing speed. despite transformer architectures based lms being the mainstay of the current state-of-the-art ner models, the focus of research has been primarily to improve these lms or extend applications of such pre-trained lms to different languages. this assumes availability of adequate ner tagged training data for such languages. however, for low-resource scenarios where adequate amount of ner tagged training data is not available, cross-lingual and multilingual ner models are generally resorted to @xcite . the previous studies @xcite in this context have examined the performance of ner models in cross-lingual scenarios for homogenous and heterogenous languages. the homogeneity of these languages is determined based on their membership in different language families. however, the effect of finer language related aspects such as vocabulary, scripts, context, and language structure has not been studied with respect to ner tasks. for example, how does a ner model trained with data of a particular context having contextspecific names perform when applied to a different context and unseen names. it is felt that such a study is needed not only to answer such questions but also for the effective application of the ner models in cross-lingual and multilingual scenarios. in this work, we study the performance of transformer based language models in the context of ner tasks considering different language-related aspects namely, script similarity, vocabulary sharing, cross-language, and multilingual knowledge sharing. this study involves ner models trained with embeddings obtained using different transformer based architectures. the paper is organized as follows. the different language related aspects considered in this study that are likely to affect ner performance are discussed in section 2. the languages, transformerbased lms, and datasets used in this study are described in section 3. the experiments and results are presented in section 4. finally, we conclude the paper in section 5 with key findings of the study and directions for future work. ## language-related aspects and ner models the performance of ner models is influenced directly and indirectly by different language related aspects mentioned in section 1. in this section, we discuss in detail how these aspects impact both the development and performance of ner models, particularly when they are applied in multilingual and cross-lingual scenarios. ner models extract named entities from text based on the relationship between named entities and other words in a sentence. embeddings obtained from language models that are used to train ner models implicitly capture different aspects of the language structure. for example, the positioning of proper nouns (named entities) in a valid sentence construct of a language is captured using positional embeddings. the positional embeddings are language-specific and depend on the grammar and structure of a language. hence, training data containing nes in different sentence constructs (e.g., active and passive voice form) can capture linguistic variations of a language and is likely to have a positive impact on the accuracy of the ner models. another important aspect is the context of usage of named entities in the text of a language. for instance, the hindi word muskaan means smile while the same word is also used as name of a person of feminine gender in indian sub-continent. the word in the later use case is a ne while the former is not. this understanding can be derived only from context that plays an important role in determining nes. lm embeddings are expected to capture such contextual information, that are exploited during training of ner models. the ability of a ner model to differentiate between the above two forms of usage of the same word (one as an ne and the other as non-ne) depends on whether such variations in context of usage has been presented during training. as such context related variations are language-specific, performance needs to be studied for such models in cross-lingual scenarios. many languages often share a common script. however, vocabulary sharing between the languages may be low (e.g., urdu and arabic) or high (e.g., urdu and persian). on the other hand, the script is different for hindi and urdu. however, there is a high degree of vocabulary sharing between the two languages. a specific challenge for cross-lingual ner is related to processing of scripts, that is described as follows. the generation of word embeddings requires a tokenizer specific to the script of a language. therefore, the performance of ner models trained with word embeddings of a language that shares vocabulary but not the script with another language needs to be investigated. the performance of monolingual ner models (those trained with word embeddings of respective languages) in such cross-lingual scenarios carrying similar and dissimilar language pairs needs to be studied. the outcome of the study can be used to identify attributes that can help in deciding whether an ner model can be shared across languages with acceptable performance. such information can be leveraged for low-resource languages where building of ner models from scratch is difficult due to the absence of adequate training data. in such a scenario, a ner model trained on a high resource language can be alternatively used for a low resource language with reasonable accuracy. multi-lingual training @xcite involves the pooling of training data across multiple languages when the amount of training data from a single language is inadequate to meet training requirements. however, the choice of languages to be pooled is an important consideration and needs to be studied. the proportion of individual languages in the pooled dataset is also likely to affect the overall ner performance. a study in this aspect can provide insight into the language bias (if any) caused in the model due to the varying distribution of individual language data in the training dataset. ## discussion and conclusion in this work, we have studied the effect of language related aspects on the performance accuracy of monolingual and multilingual ner models based on different transformer based neural lms. we have specifically studied the effect of finer language-related aspects namely script similarity, vocabulary sharing, and content and context dependency on ner performance. it was observed that, for each of the chosen languages, monolingual bert embeddings gave the highest accuracy among all the embedding models. it was also found that the performance of crosslingual ner models was highly dependent on vocabulary sharing and the presence of foreign nes in training data. further, it was seen that data inadequacy can be handled by pooling data of similar languages. in future, work can be undertaken with respect to transfer learning techniques for cross-lingual ner models considering different language-related aspects mentioned in this study. the scope of this study can be extended to cover other nlp tasks such as machine translational as well.