Improving R omanian B io NER Using a Biologically Inspired System (2022)

Abstract: Recognition of named entities present in text is an important step towards information extraction and natural language understanding. This work presents a named entity recognition system for the Romanian biomedical domain. The system makes use of a new and extended version of SiMoNERo corpus, that is open sourced. Also, the best system is available for direct usage in the RELATE platform.

## introduction the rapid advancement of artificial intelligence (ai) technologies has led to the development of different prominent fields of ai such as natural language processing (nlp). nlp is able to provide valuable information from large amounts of texts. for example, in the covid-19 pandemic situation, nlp has played an important role in finding the presence of disease @xcite . identifying text spans that refer to real-world objects and categorizing them into a subject under an entity, is known as named entity recognition (ner) @xcite . however, each domain has its own types of entities, for example, ner in the biomedical domain implies identifying chemicals, symptoms, ingredients, diseases, genes, dosage level, dosage forms, active substances, etc. although the nlp community has invested a lot of effort in developing bioner systems for the english language, obtaining important results, the development of ner systems for other languages is conditioned by the availability of quality resources, such as gold annotated ner corpora. moreover, biomedical ner has multiple specificities that one needs to address when developing an ner system: spelling variations, huge amounts of abbreviations, lengthy phrases, polysemy, cascaded constructions @xcite . consequently bioner is a challenging task and most of the time ner systems need domain adaptation. in this paper we propose a ner system that uses pre-trained contextual embeddings, xlm-roberta @xcite , enhanced with an inhibitory mechanism similar to the biological process of lateral inhibition @xcite , that has as the main goal the filtration of noisy information, in our case noise can be associated with rare contexts or less occurring entities. this system is trained on a new version of simonero corpus, whose ner level has been expanded with new entities (including covid pandemic-related entities) for a better coverage of biomedical language. this paper is organized as follows: in section 2 we present related work, in section 3 the si-monero corpus is presented, section 4 describes the ner system architecture, while section 5 gives the results and finally conclusions are presented in section 6. ## related work bioner is an important task that aims to extract key information from biomedical documents that can be used in workflows to perform different functionalities such as relation extraction, text mining, etc. in recent years, pre-trained models, such as bert @xcite and biobert @xcite have made significant contributions to the development of the ner task. in the context of the 2020 iberian languages evaluation forum (iberlef) shared task, @xcite used bert as the base module and a machine reading comprehension framework was proposed to identify and classify nes that achieved an f1-score of 0.87. @xcite developed hunflair, a ner tagger, able to recognize five biomedical entity types. it outperforms other ner tagers with an average gain of 7.26% when compared with other state-of-the-art biomedical ner tools such as scispacy @xcite or huner @xcite . hunflair uses a character-level model that was pretrained on 3 million full texts and 24 million biomedical abstracts. even though the performance of ner systems for the biomedical domain for english has increased lately, there is still room for improvement until human annotators performance is reached. the romanian language, suffers from the scarcity of ner systems for different subdomains, especially in the biomedical domain. one of the first attempts to develop a biomedical ner tagger was based on a partitioned convolutional neural network for classification and used word-embeddings computed from the romanian section of wikipedia, concatenated with a medical sub-corpus @xcite . this approach achieved an f1-score of around 0.5. a more recent approach was based on bidirectional long-short-term memory (bil-stm) networks and obtained an f1-score of 0.81 @xcite . ## simonero corpus simonero is the gold standard morphologically, syntactically and named entity annotated romanian medical corpus. this corpus has three different development stages. the first one was the creation of the monero corpus, a gold standard biomedical corpus for romanian language enhanced with two types of annotations: morphological and named entities specific to the biomedical domain @xcite . the second development stage was the addition of the syntactic annotations @xcite . the current phase is the one in which the named entity annotation level was enhanced by 10%, due to the addition of new relevant sentences. currently, simonero has 163,707 tokens, comprised in 5,418 sentences and 15,493 nes. simonero contains texts from three types of documents: scientific medical literature books, medical journal articles, and sites that offer explanations on various medical topics. regarding the medical domain, the texts were chosen to belong mainly to cardiology, diabetes, and endocrinology. the annotation scheme of the corpus has three different levels: after the corpus was expanded with new annotations regarding the named entities level, all sentences were shuffled and split into three files: train, dev, and test. in order to evaluate our approach we used 80% of the corpus sentences for training, 10% for development, and 10% for testing. figure 2 shows the label distribution in the train, dev and test sets. y axis indicates the number of a particular label in the data and ## system architecture for the purposes of this work we constructed a state-of-the-art system for ner in the romanian biomedical domain using contextualized embeddings. previous work relied solely on static embeddings. in order to compare the newly proposed system with previous approaches we also trained a system making use of static embeddings. this was necessary since existing systems were trained on the previous, smaller, version of the corpus, hence no direct comparison was possible. comparison with older models is further made difficult by the introduction of new terms (such as covid-related). the results for both systems, using static and contextual embeddings, are described in section 5. ner systems making use of transformer-based models usually obtain the numeric representations associated with input tokens which are then fed into a linear layer. finally a classification head is used to obtain the predictions. in our approach, we employed an additional layer inspired by the biological process of lateral inhibition. in neurobiology, this process is defined as the capacity of an excited neuron to reduce the activity of its neighbors. this new layer is inserted after the embeddings calculation and before the linear layer. to emulate the way inhibitory inter-neurons function, an embedding dimension value is either allowed to pass unchanged to the next layer or set to zero, depending on the other values. the forward pass calculation is given in equation 1, where x is the layer's input vector, associated with a token embedding representation, diag represents a matrix with the diagonal set to the vector given as parameter, zerodiag is the matrix with the value zero on the diagonal, and w and b represent the weights and bias. θ is the heaviside function, described in equation 2 . the problem of computing a derivative for the heaviside function in the backward pass was overcome by approximating the heaviside function with the sigmoid function using a scaling parameter as suggested by @xcite . this approximation was used only in the backward pass, while in the forward pass the heaviside function was used as it is. this approximation technique is also known as surrogate gradient learning @xcite allowing the use of a non-differentiable function in the forward pass (e.g. the heaviside function) while using a different function for approximating the derivative in the backward pass. the derivative of the sigmoid function is given in equation 4, where σ(x) is the same as in equation 3 . with regard to static word embedding models, pȃis , and tufis , (2018) have trained and published models using the representative corpus of contemporary romanian language (corola) @xcite . the authors have shown that due to the nature of the corola corpus, the models outperform existing ones, such as wikipedia based models. furthermore, the corola-based embeddings were previously used in constructing a romanian language legal-domain ner system @xcite . following the approach of pȃis , and mitrofan (2021a), we wanted to explore the impact of using a combination of different word embeddings. hence, we trained domain-specific static word representations on the bioro corpus @xcite , using the fasttext toolkit 3 (bojanowski we employed a recurrent neural network architecture, using long short term memory (lstm) cells, representing tokens by means of pre-trained word embeddings with additional character embeddings, computed on the fly. the actual prediction is performed by a final conditional random fields (crf) layer. implementation was realized using the neuroner foot_1 package @xcite . the results obtained using the static word representation models are given in contextual word representation models specifically created for romanian language include romanian bert @xcite , robert @xcite , jurbert @xcite . nevertheless, these models were not trained on biomedical text. however, romanian language is also present in multilingual models, such as mbert @xcite and xlm-roberta @xcite . @xcite recently showed that roberta-based models produce stateof-the-art results in biomedical and clinical tasks. therefore, we explored using the xlm-roberta model with the system described in section 4. we further compared the results obtained with the newly introduced lateral inhibition layer with the same system without this layer. the overall f1 score was 83.42%, thus the new layer accounted for 0.85% improvement, under similar conditions (the same dataset split, the same contextual embeddings model, similar hyper-parameters). ## conclusion this paper introduced a neural named entity recognition system adapted for the romanian biomedical domain. it employed the new extended version of simonero corpus for training and evaluation. the proposed ner system uses pre-trained contextual embeddings, xlm-roberta, and an inhibitory layer, inspired by the biological process of lateral inhibition. this work can make significant contribution in helping researchers interested in domain-specific ner both for romanian and for other languages. in addition, the lateral inhibition mechanism has the potential to be applied in other tasks as well. currently, it has been successfully applied in our system that participated in the se-meval 2022 shared task on multilingual complex named entity recognition (multiconer) 6 . the resulting ner system is available for online usage through the relate platform foot_3 . the source code is freely available from our github