R omanian micro-blogging named entity recognition including health-related entities (2022)

Abstract: This paper introduces a manually annotated dataset for named entity recognition (NER) in micro-blogging text for Romanian language. It contains gold annotations for 9 entity classes and expressions: persons, locations, organizations, time expressions, legal references, disorders, chemicals, medical devices and anatomical parts. Furthermore, word embeddings models computed on a larger micro-blogging corpus are made available. Finally, several NER models are trained and their performance is evaluated against the newly introduced corpus.

## introduction social media networks can present an alternative to formal data sources, and prove to be a reliable resource for different natural language processing (nlp) tasks. since users often submit domainspecific information in a wide variety of social networking resources, such as specific communities, blogs, microblogs, public news websites, or forums, there has been a significant interest in extracting information from these resources. micro-blogging platforms, such as twitter, reddit or gab, as a source of such comments, in general, contain relevant information that can be used to develop nlp resources for both general and specific domains, such as health @xcite and legal aspects @xcite . furthermore, mining and studying health-related information can ultimately be used to improve public health @xcite . our contribution is threefold. first, we introduce the microbloggingnero corpus @xcite , comprised of micro-blogging specific messages, manually annotated with 9 named entity (ne) classes and expressions, including 4 healthrelated classes. this is the largest available romanian micro-blogging ne corpus to date and the only one containing health-related entities. second, we release word embeddings computed on a larger micro-blogging corpus. finally, we train several ner models using the newly released corpus and we make these available for the research community. this paper is organized as follows: section 2 presents related work, section 3 describes the manually annotated dataset, and section 4 introduces the experiments performed, including the generated word embedding representations and ner models. finally, section 5 gives conclusions. ## related work biomedical corpora have been collected for languages with different extent of technological support: hmong @xcite , romanian @xcite , spanish @xcite and others. there are even parallel medical corpora available foot_0 . many of them are annotated with umls medical entities @xcite . 2 most of the biomedical corpora were collected from abstracts @xcite , but also from full articles @xcite ; others combine various sources @xcite . the interest in gathering corpora made up of micro-blogging texts characterizes the research dedicated to various languages: arabic @xcite , chinese @xcite , english @xcite , french @xcite , german and danish @xcite , italian @xcite , romanian @xcite , turkish @xcite , etc. micro-blogging corpora are used for various tasks: sentiment analysis application development @xcite , emotion annotation @xcite , credibility analysis @xcite , event detection (mazoyer et al., 2020), hate speech detection @xcite @xcite and others. another vivid line of research is identifying certain types of entity names in these corpora, such as occupations @xcite and symptoms @xcite . in this work, we developed microblog-gingnero the first romanian micro-blogging corpus enriched with health information. ## dataset the microbloggingnero dataset contains 7,800 messages manually annotated with the following named entity classes and expressions, totalling 11,099 annotations (see figure 1 ): examples: astȃzi ("today"), 15 septembrie ("september 15"), crȃciun ("christmas"). the dataset was gathered by using a crawling process based on specific queries, aiming to gather messages containing the entities of interest. it was then cleaned by removing duplicates and very short messages (too short to be actual sentences) or messages containing only hashtags. furthermore, all urls were replaced with a special "<@xmath0. the annotation process involved 7 annotators working under the guidance and supervision of 3 senior researchers. parts of the corpus were common between several of the annotators, allowing us to compute inter-annotator agreement. each pair of annotators had to annotate 300 common files. the cohen kappa coefficient, indicating the interannotator agreement was computed at the token level and led to an average kappa of 0.80. according to @xcite , a value between 0.61-0.80 indicates substantial agreement between the annotators. moreover, @xcite the remaining disagreements account for mistakes made by individual annotators when specific domain information was needed in order to identify and classify mainly entities from both medical and legal domains. for example, medical entities such as "virus" was annotated with labels "diso" or "chem". in most situations the correct label is "diso", however when the mechanism of the action of the virus is explained the correct label is "chem", thus creating confusion for the annotators. a similar situation can be found in the case of "vaccine" mentions when this entity was annotated with both "chem" and "med_device" labels. in this case the correct label is "chem", but some annotators considered it "medical_device" because the term was used in a context of diagnosing covid-19 together with tests, masks, gowns, gloves, sterilizers, and ventilators. entity classes such as org, per, loc and time have an interannotator agreement greater than 0.85. the annotation scheme, as well as the guidelines, were inspired by existing romanian ne corpora (built without social media text), such as monero @xcite , simonero @xcite and @xcite . the annotation guidelines foot_2 for the microbloggingnero corpus had to be adjusted to take into account the particularities (pȃis , et al., 2022b) of micro-blogging text, such as hashtags (#cluj, #laculnoua), text written with emojis instead of numbers, unusual abbreviation, elongated words (commonly used for emphasis in microblogging), code-mixed text, etc. since the microbloggingnero corpus was created with multiple types of annotations, the healthrelated entities represent only 9% of the total entity classes, as depicted in figure 1 . this accounts for 958 annotations, half of them being in the diso class (466 annotations). after the annotation process was completed, the corpus was anonymized by replacing all person names, specific locations (such as addresses or street numbers), and organizations with randomly generated ones. any user mentions were removed and replaced with a special tag "<@xmath1, regardless of how it was included in the original message. furthermore, any other identifiers (such as message ids) were removed from the corpus. the dataset was released with several usage scenarios in mind, reflected in the presence of multiple formats, including span-based annotations with all the entities, span-based annotations with general entities (per, loc, org, time), span-based annotations for the bio-medical domain, and tokenized text with token-based ne annotations attached (using tools available inside the relate platform @xcite ; pȃis , , 2020)). ## experiments we used the presented ner annotated microblogging dataset to train ner models, using different architectures and configurations. for this purpose, the corpus was split into train (70%), valid (15%) and test (15%). the splits were selected randomly while trying to preserve the general distribution of entities in each split. they are made available within the corpus release. then, we experimented with classical word embedding representations. for this purpose, we used a recurrent neural network architecture, implemented with long short-term memory (lstm) cells, as provided by the neuroner package @xcite . we used pre-trained word embeddings (pȃis , and tufis, 2018) trained on the representative corpus of the contemporary romanian language (corola) @xcite . then, we trained new word representations based on a larger corpus of raw micro-blogging text (comprising 853k messages), using the fasttext tool @xcite . finally, based on observations such as those of (pȃis , @xcite , indicating that multiple representations can help the system achieve better results, we combine the two representations into a single, larger representation. we make the generated embedding models freely available foot_3 . following the experiments with static word embeddings, we performed two additional experiments using contextualized embeddings provided by the xlm-roberta model @xcite . this model provides contextualized representations for multiple languages, including romanian, and has proven to be a good choice for different ner experiments @xcite @xcite . our experiments made use of a basic ner system, employing a linear layer followed by a classification head, and the same ner system enhanced with a biologically inspired lateral inhibition layer, as introduced by (pȃis , , 2022). this system was previously used for ner in the romania bio-medical domain @xcite . intuitively, similar to the biological process, we considered that the lateral inhibition layer would allow the system to better focus on difficult, and less represented, ne classes. results are given in the results indicate that romanian ner in a micro-blogging context is still far from achieving high-scores. currently, even systems making use of contextualized embeddings do not achieve over 90% f1 score for any of the classes. on regular texts, previous works, such as @xcite , indicate over 90% f1 scores for persons and legal references. furthermore, @xcite show an f1 score of 85% for health-related entities in regular romanian texts. an analysis of the errors associated with the best performing model, with regard to health-related entities, indicates that a large number of entities are not recognized (the predicted label is "o"). this suggests a need for a larger corpus with more diverse entities. other types of errors account for predicting org as med_device or chem, which is an error found also with the human annotators, where a company name is used also for a vaccine name. ## conclusion micro-blogging texts pose unique challenges in terms of natural language processing. this is particularly true with regard to bio-medical and other health-related entities. these are more difficult to detect when compared to more common nes, such as person and organization names, due to their inherent complexity. @xcite consider this to be a general issue with regard to information extraction in the biomedical domain, given the complex entities and changing nomenclatures. this was also the case in the process of manual annotation, annotators having difficulties to identify and classify both health-related and legal nes. furthermore, as noted by @xcite , imbalanced data remains a challenge for training deep neural network models. this is particularly true with the microbloggingnero dataset, considering the distribution of classes depicted in figure 1 . furthermore, this suggests that future work should focus on expanding the dataset, aiming for a more balanced class distribution. when looking at micro-blogging word embeddings, these seem to be beneficial for recognizing certain entity classes, possibly accounting for spelling particularities in micro-blogging texts. general representations, trained on the large corola corpus, provide better results when detecting anatomical parts, chemicals and medical devices. this is due to the presence of bio-medical texts in the corola corpus, as well as to the reduced number of mentions in the micro-blogging corpus. future research will explore more word representation models, particularly those trained on larger micro-blogging corpora. currently, there is no contextualized word model available for the romanian language built on micro-blogging texts or specifically on health-related texts, such as biobert @xcite .