Infrrd.ai at S em E val-2022 Task 11: A system for named entity recognition using data augmentation, transformer-based sequence labeling model, and E nsemble CRF (2022)

Abstract: In low-resource languages, the amount of training data is limited. Hence, the model has to perform well in unseen sentences and syntax on which the model has not trained. We propose a method that addresses the problem through an encoder and an ensemble of language models. A language-specific language model performed poorly when compared to a multilingual language model. So, the multilingual language model checkpoint is fine-tuned to a specific language. A novel approach of one hot encoder is introduced between the model outputs and the CRF to combine the results in an ensemble format. Our team, Infrrd.ai , competed in the MultiCoNER competition. The results are encouraging where the team is positioned within the top 10 positions. There is less than a 4% percent difference from the third position in most of the tracks that we participated in. The proposed method shows that the ensemble of models with a multilingual language model as the base with the help of an encoder performs better than a single language-specific model.

## related work the transformer-based language models fared better in the identification and extraction of entities from a given text. since there is a necessity for a large amount of training data which provides a much-needed boost in accuracy. sometimes, a model trained on a huge data in one or more languages is used in another or different language. these types of models are commonly known as cross-lingual language model (conneau and lample, 2019) or multilingual language model @xcite . a fine-tuned language model has a better performance compared to a multilingual language model. but, the multilingual language model is more adaptable across different languages, which is not available for a fine-tuned language model. the researchers started exploring the amount of data required to train a language model. in some cases, the amount of data available in a language with annotation is very limited. these languages are termed low-resource languages. due to this limitation, the model may not be aware of the complete set of words in the low-resource languages. here, we are exploring to understand the performance of a transformer-based language model in low-resource constraints. the challenges involved in recognizing complex entities in lowresource environments @xcite have led to the creation of the competition data @xcite . in low-resource scenarios, different approaches have been adapted to overcome the constraints. the available fine-tuned transformer-based model such as bert is bootstrapped to improve the accuracy of ner @xcite . a prompt-guided attention layer is used as part of a transformer model by creating a semantic-aware answer space for tuning the model for further betterment @xcite . sentence reconstruction approach to enhance low resource sequence tagging by utilizing the knowledge of high resource data @xcite . a common approach used on low resource languages is to use cross-lingual transfer learning, where a model trained on high resource language is used as the reference. an active learning mechanism was used to improve the performance of ner @xcite . a teacher-student knowledge transfer model technique has shown to give effective results on low resource ner tasks @xcite ). an unsupervised cross-language transfer learning method where the encoders trained on the source and target language together using adversarial learning followed by augmented fine-tuning technique @xcite . ## methodology individual pre-trained models were used to evaluate the training and the dev set. based on the empirical results, we decided to train a baseline language with multiple languages for 20 epochs and then perform transfer learning to train the monolingual models. ## training and evaluation we have used the flair framework @xcite which uses pytorch and huggingface transformers to build and experiment with our approaches. the roberta transformer model was used as the base model. however, for some monolingual language training, we stacked a language-specific embedding layer provided by flair. for dutch, ger-man, and spanish, flair language-specific embeddings were prepended before transformer embedding for experimentation. we experimented with two different initial checkpoints loaded to train the transformer model. one checkpoint was from the huggingface library (hug), for both roberta-base and roberta-large. the other was to load the xlm-roberta model trained on the competition dataset as the initial checkpoint for the training of monolingual tasks. however, the models trained with the initial checkpoint from xlm-roberta performed better due to the transfer of knowledge from the multilingual checkpoint to the monolingual checkpoint. the language models used in the experiment for different languages with hyperparameters and macro averaged f1-score on the competition test set are tabulated in tables 1 and 2 . ## discussion and results the training strategies for all the tracks fall into these categories namely external dataset, data augmentations, model architecture searching, transfer learning, and ensemble learning as described in sections 3 and 4. we used a gazetteer as the last option, which didn't improve the performance. ## conclusion the recognition of entities from multiple languages with low resources is more complex. the problem lies with the ambiguous entities formed by newly coined words. the syntax of grammar in the sentences was not followed to capture the context between the words. we tried a single multilingual transformer approach, which didn't provide muchexpected results. we had used gazetteers for all the languages sourced from the training and wiki data. both of them didn't produce improvements over the model results. we trained a multilingual transformer model and performed transfer learning to the individual languages. since there were multiple languages in the task. we performed unique experiments on one language and then adapted it to the others based on the performance. in the discussion section, the approaches used for experiments are covered and vary for the individual languages. overall, we created an ensemble of different models which resulted in improvements over the single model. the ensemble architecture covered different types of transformerbased language models. the results reached closer to the top positions with this approach. we also observed that the data augmentation used to improve the performance for a few languages and drop in the performance for the other languages. our results are above 15% on an average in the participated sub-tasks over the multiconer baseline results. we would like to explore the multilingual t5 transformer model, which couldn't be covered during the competition. we would like to explore different augmentation techniques with external data, which couldn't be completed due to time constraints.