LSJSP at S em E val-2023 Task 2: FTBC : A F ast T ext based framework with pre-trained BERT for NER (2023)

Abstract: This study introduces the system submitted to the SemEval 2022 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) by the LSJSP team. We propose FTBC, a FastText-based framework with pre-trained Bert for NER tasks with complex entities and over a noisy dataset. Our system achieves an average of 58.27% F1 score (fine-grained) and 75.79% F1 score (coarse-grained) across all languages. FTBC outperforms the baseline BERT-CRF model on all 12 monolingual tracks.

## related work named entity recognition is a fundamental task in natural language processing (nlp). most of the earlier works considered ner as a sequence labelling task. chiu and nichols (2016) introduced convolutional neural networks for ner, whereas @xcite used recurrent neural networks (rnns) with crf for ner. however, these models do not perform very well on complex enti-ties like movie names @xcite , which are challenging to tag. @xcite has extended this work to multilingual code-mixed settings. pre-trained language models like elmo @xcite or transformer-based @xcite models like bert @xcite and xlm-r @xcite improve the performance of ner on complex entities. multiconer i @xcite was a shared task to build ner systems for complex entities in 11 languages. multiconer i shared challenge was conducted using the dataset curated by @xcite . the critical challenge of multiconer i was dealing with complex entities with limited context. multiconer ii classifies the tokens into 30+ finer classes. additionally, simulated errors have been inserted in the dataset to make the task more challenging and realistic. ## dataset description multiconer ii shared task @xcite focussed on building complex named entity recognition (ner) systems for 12 languages. the languages include bangla (bn), german (de), english (en), spanish (es), farsi (fa), french (fr), hindi (hi), italian (it), portuguese (pt), swedish (sv), ukrainian (uk), chinese (zh). multiconer ii is an extension of multiconer i @xcite where the six broader classes, namely person, group, creative works, location, medical, and product, are further classified into 30+ finer classes. the fine to the coarse level mapping of the classes is as follows: the train data of each of the 12 languages varies from 9,000+ sentences to 16,500+ sentences, and the validation set ranges between 500+ to 800+ sentences. the test set for each track had at least 18,000+ sentences, with the lowest for hindi (18,349 sentences). the multilingual track comprises data from each of the monolingual tracks. the dataset consists of simulated errors like typographical and spelling mistakes, making the ner task more challenging and realistic. ## baseline systems we now describe the baseline systems developed for multiconer ii shared task @xcite . ## ftbc this section discusses the ftbc system developed for the multiconer ii monolingual tracks. the multiconer ii dataset contains simulated errors, including typographical and spelling mismatches. hence, we devised a two-layered system for the task. the tokens are checked for spelling or typographical errors in the first layer using fast-text's nearest neighbor method. since @xcite showed that for inflectional languages such as bangla, hindi, etc., fasttext performs better than glove @xcite and word2vec @xcite ; hence, we used fasttext for our framework. we observed that fast-text's nearest neighbor could sometimes return garbage values. to avoid those garbage values, we used levenshtein distance to calculate the distance between the provided input token and the nearest neighbor prediction by the fasttext model. if the levenshtein distance @xmath0, we considered the predicted word from the fasttext model as our input token. the output of the fasttext model is then passed on to the bert+crf model to get the predictions on the input tokens.figure 1 shows a schematic representation of the ftbc system. ## results in this section, we quantitatively evaluate ftbc against the baseline models. observations also revealed confusion between the categories politician and artist and between musicalgrp and artist. analysis shows this confusion is mainly because the train set classifies the same terms into different tags in different sentences. for example, "john" was classified in spanish training as b-politician, b-otherper, and b-artist, leading to misclassification by the models. these instances are also found in other languages, including english, german, italian, portuguese, and swedish. tale 5 reports the sentences having different tags of "john". tag sentence b-otherper john sterling(born 1948) sportscaster for the new york yankees. b-artist john baker saunders founding member and bassist for the grunge rock supergroup mad season. b-athlete he was always a reserve keeper at vale park though behind first alan bosewell and when he left john connaughton. ## conclusion we developed ftbc, a fasttext-based framework with pre-trained bert for the multiconer ii shared task. our system outperforms the bert-crf baseline models for all 12 monolingual tracks. ftbc performs well for noisy data, but ambiguous entities affect its overall performance. in the future, we would like to enhance the performance of ftbc using external knowledge bases.