How to tackle an emerging topic? Combining strong and weak labels for Covid news NER (2022)

Abstract: Being able to train Named Entity Recognition (NER) models for emerging topics is crucial for many real-world applications especially in the medical domain where new topics are continuously evolving out of the scope of existing models and datasets. For a realistic evaluation setup, we introduce a novel COVID-19 news NER dataset (COVIDNEWS-NER) and release 3000 entries of hand annotated strongly labelled sentences and 13000 auto-generated weakly labelled sentences. Besides the dataset, we propose CONTROSTER, a recipe to strategically combine weak and strong labels in improving NER in an emerging topic through transfer learning. We show the effectiveness of CONTROSTER on COVIDNEWS-NER while providing analysis on combining weak and strong labels for training. Our key findings are: (1) Using weak data to formulate an initial backbone before tuning on strong data outperforms methods trained on only strong or weak data. (2) A combination of out-of-domain and in-domain weak label training is crucial and can overcome saturation when being training on weak labels from a single source.

## introduction named entity recognition (ner) is an nlp task that involves identifying key entities in text such as person, location, time or organisation. research around ner has grown rapidly with the adoption of deep learning techniques and has been an integral step to many nlp pipelines @xcite such as information retrieval, knowledge base completion, and question answering. as ner models have matured to involve deep transformer @xcite models and achieve greater performance, the demand for more human labelled strong data has followed. this has become a common bottleneck as attaining more strongly labelled data is expensive and time consuming. however, weakly supervised methods are mostly tested on general-domain datasets rather than an emerging domain or topic. the inability of current state-of-the-art ner models to perform when given new biomedical topics such as covid-19 preprints has been previously stated as a major gap in current ner applications @xcite . this was explained by a propensity for models to overfit to currently available training data and a lack of data in the target domain for models to learn such a complex emerging topic like covid-19. we bridge this gap by proposing a domain-specific ner dataset called covidnews-ner to evaluate these weakly supervised methods and providing suggestions of combining weak and strong data to address this issue (for a real example of weak and strong labels, see fig. 1 ). we expect that the data we publish will drive research around techniques to better adapt to new topics and the strong data we publish to unlock ner applications about covid-19 and in the medical domain. we plan for our paper to specifically contribute to the relaunched and currently being improved ontology-based text mining tool biocaster @xcite for automatic monitoring and surveillance of disease outbreaks. for our analysis of mixing weak and strong data, we build upon the recent weakly labelled ner model roster @xcite . the model uses a noise robust loss function with noisy label removal, ensemble training and self training without the use of strong data to achieve best performance out of other distantly labelled methods. we propose controster, which improves upon roster by performing cross-domain transfer learning over 3 training stages: the first two stages progressively train the model on out-of-domain and in-domain weak data; afterwards, we finetune the model on in-domain strong data in the last stage (fig. 2 ). to summarise, this paper has the following contributions: (1) we propose a novel covid-19 ner dataset with 13000 weakly labelled sentences generated by rule based methods and 3000 manually annotated sentences. to our knowledge, this is the first covid-19 orientated ner dataset in english. the method is made up of regular expression patterns in simple rule language (srl), a tool built on top of dial @xcite . to ensure a high quality of final dataset entries, multiple filtering methods were implemented after this procedure to further prune text with errors. all candidate text was filtered out based on insufficient text lengths, non-ascii characters involved and text duplicates. additionally, texts were pruned based on number of grammatical mistakes per entry and finally through manual examination. manually annotated strong data was labelled by a recent graduate working in the nlp domain. challenging entries such as differentiating between virus and disease entities were flagged and resolved by discussion with a phd student and professor who served as experts in the biomedical nlp domain. 3 dataset entities and content. we employ 10 entity types: animal, bacterium, disease, location, organisation, person, product, symptom, time, virus. 4 the person entity has 490 been expanded to label human cases of a disease as a group of people and the product entity refers to manufactured articles in the medical domain used during the covid-19 pandemic (eg. vaccines, face masks). the virus entity is an especially useful common emerging label since the neighbouring text referencing covid-19 changes whether it is a virus or disease, a common mislabel in weak data. in addition to this, the context developed as a result of the covid-19 pandemic has produced emerging entities that current rule or knowledgebased labelling systems do not capture. these are however clear to the general public and human annotators which we demonstrate through examples found in covidnews-ner in fig. 5 . because the dataset has a heavy focus on the covid-19 virus, new terminology is also featured surrounding vaccination, testing, variants, etc. other viruses that gained exposure to the public due to the pandemic are also included in the dataset such as the zika virus, mers coronavirus and influenza virus. in general, emerging entities and unique text make covidnews-ner tailored to pandemics and the medical domain while still providing some useful entities for general ner applications. ## model we build upon roster @xcite which achieves the best performance among distantlysupervised methods. the model contains multiple stages to handle the weak data and its inherent noise. starting with roberta @xcite weights, the first step is noise robust training using generalised cross entropy (l gce ) with tunable parameters dictating noise robustness and noisy label removal. the second step uses ensemble training to improve model stability and the third step introduces contextualised augmentations and selftraining with pre-trained roberta embeddings. foot_4 controster: continually-learned roster. while roster achieves strong performance on noisy data, it remains unclear what is the optimal strategy when both strong and weak data are presented. additionally, in a real-world use case, we can also assume access to weak labels in other domains. we propose a continual learning approach called controster to adapt roster for learning from out of domain weak data and in domain weak and strong data (fig. 2 represents the fine-tuning pipeline). the pipeline has three training stages: the out-ofdomain data are from weak labels generated onto the wikigold or ontonotes dataset via knowledge bases (details explained in §4). (2) then we repeat roster training on in-domain covidnews weak data (yellow box) and (3) finally finetuning on strong data with only the noise-robust loss. 4 experiments compared models. we train four models and evaluate them on the covidnews-ner test set. (1) we train the original roster model with no backbone (initialised with original roberta weights) on strong data. as the data are clean, the model is only trained with the noise robust loss with ensemble learning and self-training stages removed. (2) we train the model on 6000 lines of weak covidnews data (this creates a weak covidnews backbone) and then finetune on strong covidnews data the same as (1). (3) & (4) in the double backbone approach (i.e., our full controster model), we first train roster on either the weak labels from the wikigold dataset (balasuriya et al., 2009) or the ontonotes5.0 dataset (weischedel et al., 2011) followed by training on weak covidnews data and finally fine-tuning on strong covidnews data. this approach is visualised in fig. 2 in which (1) is made up of only the strong covidnews box and (2) is made up of the weak covidnews and strong covidnews boxes. the wikigold dataset contains 13041 lines of training data and 3 overlapping entity categories with covidnews out of 4 total entity categories. the ontonotes5.0 dataset contains 59924 lines of training data and 5 overlapping entity categories with covidnews out of 18 total entity categories foot_5 . main results. fig. 3 shows the four model's f1score performance. 7 the double backbone approach, i.e. controster, performs best for all quantities of strong data used for finetuning. additionally, using a weak covidnews backbone performs noticeably better than the baseline without a backbone for all four amounts of finetuning strong data. the improvement in performance with using either a single or double backbone approach is greatest for experiments with 100 and 500 entries of strong data when compared to using 1000 or 2100. an example of this can be seen by looking at the improvement of 11.0 in f1 score (56.7 to 66.7) when the weak wiki+cov. backbone was paired with 100 entries of strong data in comparison to an increase of 2.2 in f1 score (74.6 to 76.8) in the same scenario when 2100 entries of strong data were used. these findings support the use of combining weak data with strong data through the method of transfer learning for research and ner applications. we provide insight into the amounts of strong data necessary for effective combination of the two types of data. we recommend using a weak data generated backbone in general ner models with the potential for profound impact in few-shot learning models that have a limited number of strong data. similarly, in scenarios where only weak data is available we advise manually annotating a minimum of 100 sentences can lead to large improvements in ner model performance. weak data study. we investigate the magnitude of weak data for the models trained to be effective on the covidnews-ner dataset. fig. 4 shows the improvement in precision, recall and f1 scores as weak data scales with the roster model. after surpassing the 6k entry mark, the weak data saturates and does not lead to improved performance for each additional entry. alternatively, the strong data in tab. 8 demonstrates how the higher quality strong labels continue to scale with additional data. varying amounts of weak data were also evaluated for the covidnews backbone prior to finetuning on strong data. tab. 10 in appendix shows tests with 2000, 6000 and 10000 weak entries paired with 100, 500, 1000 and 2100 strong entries. although using a weak data backbone was clearly beneficial, the amount of weak data to train the backbone was fairly insignificant. ## further discussion combining weak & strong data. we believe a wide and deep strategy should be used when combining weak & strong data. this means training on weak data to embed the model with an expansive breadth of knowledge over all entities and then fine-tuning on strong data to overwrite noise in the weak data and generate more sophisticated ability in ner. we suggest two avenues for further research to maximise utility of weak data when paired with strong data. an improvement in noise reduction techniques via improved loss functions or model architecture will serve beneficial in allowing model performance to continue to scale with increases in weak data. additionally, investigation into how weak data generated from specific rules saturates individually and after fine-tuning with strong data will also lead to improved knowledge on mixing the different forms of labels. out-of-domain weak data. we observed that using the two stage backbone pipeline of first training on a different ner dataset and then following through with training on covidnews-ner weak and strong data led to a further improvement in performance across varying covidnews-ner weak and strong data lengths. even though the wikigold dataset had only three overlapping entity categories with covidnews-ner and ontonotes5.0, the performance still improved in part due to the overlapping categories being the prominent ones in covidnews-ner (eg. organisation, person, location, date). the additional variety of labels in the crossover entities led to improved precision, recall and f1 scores in those categories and overall scores. we implore future works to evaluate the impact of overlapping and non-overlapping categories from cross-domain weak data backbones while considering overlapping category definition similarities and differences. overall, similar to how @xcite determined the importance of using multiple sources for text to be distantly labelled, we conclude it is also beneficial to use different weak labelling techniques to create a diverse collection of weak data. we implore future work to investigate the diversity necessary for optimal combination of weak data from different sources. in appendix, we include a dedicated related work section (appx. §d) for interested readers. ## conclusion we presented covidnews-ner, an english covid-19 named entity recognition dataset in the pandemic news domain, addressing current ner models' lack of ability to tackle new and out-of-domain topics. we labelled 13000 entries using a rule-based system to generate weak labels and 3000 entries using hand annotation to generate strong labels. we further proposed a continual learning approach called controster that transfers knowledge learned in both out-of-domain and in-domain weak data. after finetuning on strong in-domain data, controster achieved state-ofthe-art performance on our proposed dataset. we further provide detailed and thorough analysis into how to successfully combine both types of data and suggest promising avenues for future research. we think that the dataset we provide and the findings we conclude will be beneficial to other ner applications, such as improving the evaluation and ability of the biocaster pandemic surveilling tool. we hope that our work drives more research in leveraging a combination of weak and strong data to improve performance on new topics such as the covid-19 pandemic. zaiqiao meng, anya okhmatovskaia, maxime polleri, yannan shen, guido powell, zihao fu, iris ganser, meiru zhang, nicholas b king, david buckeridge, et al. 2022. biocaster in 2021: automatic disease outbreaks detection from global news media. bioinformatics. peng sun, xuezhen yang, xiaobing zhao, and zhijuan wang. 2018. an overview of named entity recognition. in 2018 international conference on asian language processing (ialp), pages 273-278. ieee. thinh hung truong, mai hoang dao, and dat quoc nguyen. 2021. covid-19 named entity recognition for vietnamese. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 2146-2153, online. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. advances in neural information processing systems, 30. ralph weischedel, sameer pradhan, lance ramshaw, martha palmer, nianwen xue, mitchell marcus, ann taylor, craig greenberg, eduard hovy, robert belvin, et al. 2011. ontonotes release 4.0. ldc2011t03, philadelphia, penn.: linguistic data consortium. shi zong, ashutosh baheti, wei xu, and alan ritter. 2020. extracting a knowledge base of covid-19 events from social media. arxiv preprint arxiv:2006.02567.