Named Entity Recognition for C hinese biomedical patents (2020)

Abstract: There is a large body of work on Biomedical Entity Recognition (Bio-NER) for English but there have only been a few attempts addressing NER for Chinese biomedical texts. Because of the growing amount of Chinese biomedical discoveries being patented, and lack of NER models for patent data, we train and evaluate NER models for the analysis of Chinese biomedical patent data, based on BERT. By doing so, we show the value and potential of this domain-specific NER task. For the evaluation of our methods we built our own Chinese biomedical patents NER dataset, and our optimized model achieved an F1 score of 0.54±0.15. Further biomedical analysis indicates that our solution can help detecting meaningful biomedical entities and novel gene-gene interactions, with limited labeled data, training time and computing power.

## introduction in the area of biomedical nlp, named entity recognition (ner) is a widely discussed and studied topic. the aim of the task is to identify biomedical entities such as genes, proteins, cell types, and diseases in biomedical documents, to allow for knowledge discovery in this domain. models for biomedical ner (bio-ner) offer the opportunity to mine information and knowledge and thereby foster biomedical and drug discovery research @xcite . several shared tasks addressing bio-ner have been organized. these attempts and tasks resulted in benchmark datasets for solving english bio-ner tasks, e.g. the genia corpus @xcite , jnlpba @xcite , and bc2gm @xcite . however, when turning our attention to chinese bio-ner, only a few attempts have been made, and these attempts either had limitations in text resource types and amount @xcite or did not predefine biomedical named entity categories @xcite . another limitation is that these attempts mostly focus on clinical texts or biomedical scientific publications but not include other relevant text resources, in particular biomedical patents. many biomedical discoveries are patented in china, not only because of the encouraging policy on patentability of genetic products, but also since the existence of the speedily progressed and cheaper gene sequencing services @xcite . patent texts are highly technical with long sentences @xcite . two additional challenges of chinese biomedical patents that we encountered are ocr errors and the heavy usage of code-mixing expressions, mixing english and chinese in one entity. this is mainly because the protein and gene names are commonly written in english (or the english names are given after the chinese ones), while the disease names and other contents are written in chinese. for this reason, it is not possible to directly apply pre-trained nlp models to chinese biomedical patents. moreover, as mentioned before, because of the lack of related studies, we not only lack pre-trained models which were trained on chinese biomedical text data, but also well-organized chinese biomedical patents datasets. the contributions of this paper are threefold. first, we release a hand-labeled dataset with 5,813 sentences and 2,267 unique named entities from 21 chinese biomedical patents. second, we obtain promising results for the extraction of genes, proteins, and diseases with bert models using our labeled data in limited training time and with limited computing power. third, we show that when we use our ner model to extract entities from a large patent collection, we can potentially identify novel gene-gene interactions. we release our data and code for use by others. 1 in the following parts of this paper, we discuss previous attempts to solve chinese bio-ner tasks and other related tasks in section 2; our methods and implementation details are explained in section 3; the results of all experiments, along with the post analysis results, are described and discussed in section 4; in section 5 we discuss challenges and limitations of our study, followed by conclusions in section 6. ## related work given the abundance of work on english biomedical ner, we focus on chinese ner in this section. a commonly used state-of-art model for general-domain chinese ner is the chinese lattice lstm @xcite , which applied the lattice-structured rnn framework and made some novel modifications to ensure that the original lattice network can handle the segmentation-free chinese ner task. in the biomedical domain, the paper by @xcite is an early attempt to solve the chinese bio-ner task for scientific literature. the authors designed several feature groups from a small labeled dataset consisting of chinese biomedical research abstracts, then applied a conditional random fields (crf) model. although they showed their best model obtained a 0.68±0.05 f 1 score among 50 runs, since their dataset was small (481 sentences and 1062 entities in total), the train-test split may have influenced the results. another issue is that they split their train-test set by randomly selecting sentences from their whole corpus, which might have led to overestimation of the quality of the model because entities from the same document can then end up in both the train and test set, which may cause overfitting. more recent work by @xcite addresses open concept extraction from 4,931 biomedical articles which contain 41,733 sentences and 97,373 entities in total. the ner part did not specify different entity categories. their ner method is based on dictionary matching and rules; the main contribution of the paper is a classification model for relation extraction. the authors reported a 0.76 f 1 score for their non-category-specific open concept ner and finally reached a 0.52 f 1 score for their main relation extraction task. since the introduction of the bert paradigm @xcite there have been attempts to solve chinese ner in different domains with bert models. there are three recent studies that used bert models for chinese biomedical ner, all three for clinical data. @xcite address chinese clinical ner and relation extraction using bert. they finetune the model on data from coronary arteriography reports in shanghai shuguang hospital for five entity types: negation, body part, degree, quantifier, and location. they obtain high precision and recall scores with their model for these entity types (>0.95), thereby beating the bi-lstm baseline. since the entity types in this study are very different from the entities in our work, we cannot use this work as a relevant comparison. the papers by @xcite and @xcite both use data from the ccks competitions, containing chinese medical patient records. six clinical entity types are defined in these data sets: diseases and diagnosis, image inspection, laboratory inspection, operation, drug, and anatomic site. @xcite compare multiple ner models and obtain the highest f-score (up to 75%) with a bert-bilstm-crf model. the method by @xcite , based on an optimized bert model, outperformed all other methods in both the ccks2017 and ccks2018 clinical ner competitions, with average f-scores above 90%. in this paper, we address a new biomedical text type for chinese ner: patents. we use a pre-trained chinese bert model, and compare different learning methods to finetune it to this highly specific domain and genre. ## methods in this section we describe data collection and cleaning, data annotation, models and learning methods, evaluation, and post-analysis. ## discussion if we compare our quantitative results to the results reported by @xcite (see section 2), it appears that ner in patents is a more difficult task than ner in clinical data. we have identified a number of directions for future work that can be done for further improvement: first is that we may need more pre-processing steps focusing on the specialty of chinese biomedical text. a vital problem of processing any chinese text is that, if the source was pdf files or text converted by pdf files using optical character recognition (ocr), then, because of the large variety of chinese characters and their complex shapes, the ocred text will have relatively low quality compared with ocred english text. in the general domain, there are several well developed tools that can detect and correct possible ocr errors automatically. but for domain-specific text, popular existing tools based on simple rules can not handle ocr errors well, especially if there is complex and difficult terminology use @xcite . facing this problem, it might be necessary to build an ocr correction tool adapted for this domain and genre specifically @xcite . another challenge of our task is that, similar to other chinese biomedical text, chinese biomedical patents will have code-mixing or code-switching fragments, mainly because the protein and gene names are commonly written in english (or the english names been noted after the chinese one), while the disease names and other contents will be written in chinese. moreover, even just inside each single named entity, it is possible that the code-mixed expression still appears. there have been some attempts trying to solve chinese-english code-mixing problems in the general domain, and more commonly with speech data since this code-mixing usage seems more possible to happen in non-official situations, for example casual conversations or social media posts @xcite . thus, it will be interesting to try some code-mixing language model from the general domain to check whether it can improve current biomedical ner performance. in the final part of our study, we did some biological post analysis in order to mine some meaningful information from our generated predictions. there are other interesting topics possible in mining meaningful biological information from text mining resources, which we can try in the future. our gene-gene connection network is based on co-occurrence; it would be interesting to follow-up with more advanced relation extraction methods to discover connections between entities. ## conclusions in this study, we addressed chinese biomedical ner for patents, which is a highly specific genre with english-chinese code-mixing and a complex text writing style. we developed our own chinese biomedical patents dataset, including a humanly labeled dataset which contains 5,813 sentences and 2,267 unique named entities from 21 patent documents, and two large unlabeled datasets which contain 2,659 and 53,007 patent documents respectively. after we implemented three different bert models and learning methods, we trained and evaluated them on our evaluation sets. the results showed that the bert lm mixed model, which was trained on a part of the human gene (hg) dataset containing 10,000 patents in the train set and 100 in the test set for only 1 epoch, obtained the optimal results over all entity categories. but there were only small differences between the models and training methods. the best model obtained a 0.54±0.15 micro average f 1 score among all entity types (among all evaluation sets). although we consider these results to be reasonable, one weakness is that the standard deviation between the different test folds is large. a larger training data set might be needed to make the model more stable. we finally generated predictions with the trained best model (trained on the whole labeled dataset) on the large unlabeled datasets and then did some further biomedical analyses. these analyses indicate that our model can detect meaningful biomedical entities and find some novel gene-gene interactions, just with limited labeled data, training time and computing power.