A lpha B rains at W ojood NER shared task: A rabic Named Entity Recognition by Using Character-based Context-Sensitive Word Representations (2023)

Abstract: This paper presents Arabic named entity recognition models by employing the single-task and the multi-task learning paradigms. The models have been developed using character-based contextualized Embeddings from Language Model (ELMo) in the input layers of the bidirectional long-short term memory networks. The ELMo embeddings are quite capable of learning the morphology and contextual information of the tokens in word sequences. The single-task learning models outperformed the multi-task learning models and achieved micro F1-scores of 0.8751 and 0.8884 for the flat and nested annotations, respectively.

## introduction named entity recognition (ner) is a natural language processing (nlp) task which aims at identifying and extracting sub-sequences of the text associated with named entities (nes). these nes are subsequently categorized into different semantic groups, such as names, places, organizations, events and dates, etc. ner is considered a crucial preliminary task for the development of different applications, such as, information retrieval @xcite ), text summarization (khademi and fakhredanesh, 2020), machine translation @xcite , topic modeling and event discovery @xcite , word-sense disambiguation @xcite and others. ner is a typical sequence labeling token classification task where each token is assigned a tag. iob labeling is a common method employed for annotating datasets for ner. different machine and deep learning techniques have been used to perform ner, such as, conditional random fields (crf) @xcite , support vector machines (svm) @xcite , template-based @xcite , recurrent neural networks (rnn) @xcite , bidirectional lstm @xcite , transformer-based models (e.g. bert) @xcite and others. on the other hands, the nested ner has also been performed by employing lstm with crf inference @xcite , lstm-based hierarchical layering model along with contextual word representations @xcite , bidirectional lstms with exhaustive representations @xcite , bert embeddings based lstm-crf @xcite , fine-tuning pre-trained bert model @xcite and others. this paper presents model development and results of a shared task for arabic ner @xcite . the shared task has been divided into two sub-tasks, flat ner foot_0 and nested ner foot_1 . the flat ner uses a conventional annotation scheme, however, the nested scheme provides a hierarchical annotation within the nes. for the shared task, a different version of the wojood dataset @xcite has been used which has 70% data for training, 10% for development and 20% for evaluation purposes. the nested nes are challenging to predict as multiple output layers are required to train. however, the nested annotation provides a deeper insight of overlapping nes. we developed two models which are based on single and multi-task learning. both models are based on long-short term memory networks. furthermore, transfer learning has been used to enhance the models' learning capability. the contextualized pre-trained elmo embeddings have been incorporated with word embeddings at the input layers of the models. the elmo embeddings significantly enhanced the results as compared to the word2vec and part of speech(pos) tagging. the pos tags were used as encoding vectors which were concatenated with token encoding vectors. both single and multi-task learning models used softmax non-linearity for multi-class token classification. the details of the proposed models are discussed in the section 3. the single task learning model performed better than the multi-task learning model and produced competitive results as compared to the baseline provided in the shared task. rest of the paper describes the dataset, proposed single task and multi-task learning ner models, results and conclusion. ## data the shared task released a version of the dataset from @xcite . the training and development sets have iob labels whereas the test set has been released without labels for evaluation purposes. ## system we developed neural models by using bidirectional long-short term memory (bilstm) networks. the bilstm model has the ability to learn context within token sequences for the token classification tasks (e.g. named entity recognition). a bidirectional model has two lstm layers, the first layer reads the tokens in the forward direction whereas the second layer scans the tokens in the backward direction. the two way scanning is helpful to attain the contextual information within the token sequences. the input sequence of n words x 1 , x 2 ,..., x n is given as the input. equation 1 shows the bilstm(x 1:n ,i) function which demonstrates union of the forward and backward layers. the function shows the representation to a vector i by conditioning the previous context x 1:i and the forthcoming sequence x n:i . the models are based on two implementation paradigms; i) single task learning (stl) and ii) multi-task learning (mtl). ## results and discussion the micro-f 1 score has been computed for the evaluation of the models by using seqeval python package foot_4 . the results for the flat and nested ner on the 20% test set are shown in tables 3 and 4 . ## conclusion this paper presents the description of the models and their performances for two shared tasks; i) flat ner and ii) nested ner for arabic. we proposed bidirectional lstm-based single and multi-task learning models for both types of datasets. the incorporation of character-based contextualized word embeddings produced competitive results as compared to the baseline provided in the shared task.