RACAI at S em E val-2022 Task 11: Complex named entity recognition using a lateral inhibition mechanism (2022)

Abstract: This paper presents RACAI’s system used for the shared task of “Multilingual Complex Named Entity Recognition (MultiCoNER)”, organized as part of the “The 16th International Workshop on Semantic Evaluation (SemEval 2022)”. The system employs a novel layer inspired by the biological mechanism of lateral inhibition. This allowed the system to achieve good results without any additional resources apart from the provided training data. In addition to the system’s architecture, results are provided as well as observations regarding the provided dataset.

## introduction named entity recognition (ner) is a well known task in natural language processing. it aims to detect spans of text associated with known entities. initially, much work focused on detecting persons, organizations, and locations @xcite . however, this limited approach is not suitable for every domain, thus leading to research in domain-specific ner. for example, in the biomedical domain, a number of works have addressed entities such as genes, proteins, diseases @xcite , cell types @xcite , chemicals @xcite . similarly, in the legal domain additional classes are employed such as money value @xcite , legal reference @xcite , judge, and lawyer @xcite . in the context of "the 16th international workshop on semantic evaluation (semeval 2022)" foot_0 , the task number 11 "multilingual complex named entity recognition (multiconer)" foot_1 @xcite required participants to build a ner system able to recognize complex entities in 11 languages: bangla, chinese, dutch, english, farsi, german, hindi, korean, russian, spanish, and turkish. in addition, a multilingual track and a code-mixed track were available. the task focused on 6 entity types: person, location, group, corporation, product and creative work. as noted by @xcite , nontraditional entities can pose a challenge for ner systems. this happens because datasets are harder to build and certain entities (such as creative works) are updated more frequently than traditional ones @xcite . furthermore, traditional entities tend to occur as noun phrases, while the newly proposed entities (for the purposes of the task) may be linguistically complex (complex noun phrases, gerunds, infinitives or full clauses). an interesting result was provided by @xcite , where the top system from wnut 2017 achieved only 8% recall when dealing with creative works. this paper describes a system for complex ner in a multilingual context, developed at the research institute for artificial intelligence of the romanian academy (racai), that participated in the multi-coner task. the system employs a new artificial neural network layer trying to mimic the biological process of lateral inhibition @xcite . in various regions of the brain, excited neurons can reduce the activity of other neighbouring neurons. in the visual cortex this process may account for an increased perception in low-lighting conditions. thus, intuitively the newly proposed system may better focus on subtle details present in the data and the language model. the paper is structured as follows: section 2 presents related work, section 3 describes the dataset and pre-processing operations used, section 4 describes the method used with the system architecture in section 4.1 and performed experiments in section 4.2. the results are given in section 5 and finally, conclusions and future work are available in section 6. ## related work in a survey regarding ner using deep learning models, @xcite emphasize the importance of pre-trained word embedding representations. @xcite , in the neu-roner package 3 , combine pre-trained word embeddings with character level embeddings passing through a neural network with a final crf layer achieving high scores on different datasets. other authors, using similar architectures have shown that combining multiple static word representations can further increase the overall system performance @xcite . with the introduction of contextual word representation models, such as bert @xcite , elmo @xcite , roberta @xcite , xlnet @xcite , ner systems have been adapted to make use of these new models. for the majority of the contextual models, depending on the size of the artificial neural network being used, we distinguish between a base version and a large version (with a larger number of parameters). devlin et al. (2019) used the bertlarge model for ner on the conll-2003 english dataset, achieving an f1 score of 92.8%. nguyen et al. (2020) propose a custom bert-like model for english tweets, called bertweet, for improving the ner performance on two datasets: wnut-16 and wnut-17. contextualized embeddings were also applied with success in domain-specific settings. considering the profner shared task @xcite , dedicated to identifying mentions of occupations in health-related social media, the best performing system made use of the beto model @xcite trained on a large spanish corpus. @xcite propose an algorithm for automatically finding a concatenation of embeddings that improves a system's performance in different tasks, including ner. the authors considered multiple contextual and non-contextual embeddings and the proposed algorithm can identify any of their combinations. their work builds on previous experiments that showed increased performance when manually concatenating contextual and non-contextual embeddings @xcite ). ## dataset and pre-processing the dataset @xcite was provided in a column-based format, with 4 columns in each file. the text was tokenized, with tokens available in the first column. columns 2 and 3 did not contain useful information (only an underscore symbol was present). column 4 contains the named entity annotation in bio format. the first token (or the single token) of an entity contains the "b-" prefix. other entity tokens (in the case of multi-token entities) start with an "i-" prefix, while non-entity tokens are denoted with "o". for each language there was a train/dev/test split provided by the organizers. in the train set there were 15,300 sentences, in the dev set there were 800 sentences, while the test set was much larger. the actual number of sentences in the test set varies between languages, having as many as 217,818 sentences for english. for the multilingual track, there were 168,300 train sentences, 8,800 dev sentences and 471,911 test sentences. the smallest number of sentences was given in the code-mixed track with only 1,500 training sentences, 500 dev sentences and 100,000 test sentences. regardless of the language or additional multilingual and code-mixed tracks the development set is very small while the test set is much larger than the training set. in addition, the number of training sentences in the code-mixed track seems insufficient to build a model by themselves. predictions on the test set were required to provide annotations, in the same bio format, for each token. furthermore, considering the multilingual nature of the task, with very different languages (such as german and chinese), no pre-processing was applied, other than converting the provided format to a format suitable for the developed system. for the code-mixed track, a second dataset was generated, based on the original provided dataset augmented with new sentences. these were generated by randomly replacing existing entities in the code-mixed dataset with similar entities extracted from the multilingual dataset. for each sentence containing at least one entity, a new sentence was generated, thus doubling the size of the training dataset for the code-mixed track. ## results the models produced by the system described in the previous sections participated in all the mul-ticoner tracks. the official ranking metric was macro-averaged f1. the results are given in the best best average f1 score was achieved in the german track (79.39%), followed closely by the code-mixed track (79.37%). all these results were obtained by using only the provided dataset, without external resources. for most of the tracks precision and recall are similar (with precision being approximately 1% higher than recall), with the exception of chinese (precision is 6% higher than recall) and russian (precision is 2% higher than recall). in ## conclusion the system described in this paper participated in the multiconer shared task. it made use of a new artificial neural network layer inspired by the biological process of lateral inhibition. by means of this mechanism, the system achieved third place in the general ranking associated with 7 languages (spanish, dutch, russian, korean, farsi, german, and hindi). there were no additional resources employed, apart from the provided dataset (no additional text and no gazetteers). a simple data augmentation method (as described in section 3) was applied only for the code-mixed track, while still using only the provided data. experiments performed after the task deadline, showed (as reported in section 5) that not all tracks benefit from using the new layer. it is likely that languages with an increased number of tokens present when training the multilingual xlm-roberta model will benefit less from employing the new layer. nevertheless, models with the lateral inhibition layer trained for lower resourced languages, multilingual, and code-mixed datasets seem to achieve higher scores. furthermore, improvements can be seen in both precision and recall. the proposed lateral inhibition layer can be applied to other natural language processing tasks as