Learning Entity-Likeness with Multiple Approximate Matches for Biomedical NER (2021)

Abstract: Biomedical Named Entities are complex, so approximate matching has been used to improve entity coverage. However, the usual approximate matching approach fetches only one matching result, which is often noisy. In this work, we propose a method for biomedical NER that fetches multiple approximate matches for a given phrase to leverage their variations to estimate entity-likeness. The model uses pooling to discard the unnecessary information from the noisy matching results, and learn the entity-likeness of the phrase with multiple approximate matches. Experimental results on three benchmark datasets from the biomedical domain, BC2GM, NCBI-disease, and BC4CHEMD, demonstrate the effectiveness. Our model improves the average by up to +0.21 points compared to a BioBERT-based NER.

## introduction in the biomedical field, obtaining labelled data is very costly. biomedical named entities (nes) are complex and new nes are continuously increasing in significant numbers, leading to unknown-word issues in biomedical named entity recognition (bioner) tasks. one reason why biomedical nes are complex is that they have many variations with the interchangeability of roman numbers and latin characters, spaces and hyphens, etc. the number of new biomedical research papers is increasing, wherein approximately two papers per minute, resulting in more than 1 million papers each year, are added to the pubmed database @xcite . with this number of publications, new nes are constantly being reported. in the last few years, ner using pre-trained language models (lms), such as bert @xcite , elmo @xcite , and flair @xcite , has shown state-of-the-art performance. in the biomedical domain, pre-trained lms such as biobert @xcite and bioelmo @xcite , which are bert and elmo trained on a biomedical domain text, have achieved the state-of-the-art performance in many biomedical natural language processing tasks including ner. however, only using previously trained lms cannot cover the continuously increasing new entities due to complex characteristics of biomedical nes, lead to unknown words problem. despite being used as approaches to avoid unknown words problem, subword segmentation @xcite methods consider subwords represented as unique ids, but not words or their synonyms. therefore, it is difficult for subword or character based lms to cover biomedical nes , which are complex and contain various of expression described in section 3. moreover, lm pre-training is costly, time-consuming, and computationally expensive. training biobert on biomedical corpora based on the bert model requires 10 to 23 days on eight nvidia v100 gpus @xcite . to deal with the complex and continuously increasing entities, the use of dictionary-based approaches can be an effective approach in previous works @xcite . in contrast to pre-training models, we can cover new nes by adding entries to the dictionary, without needing time-consuming pre-training. there are two types of dictionary application methods: exact matching and approximate matching. exact matching has been incorporated into neural ner @xcite @xcite and non-neural ner methods @xcite to improve accuracy. exact matching cannot totally cover all of the complex and newly-created nes. in the biomedical domain, new nes are created by modifying the endings of the existing one. for example, the new gene taar7p was named by modifying the ending of the existing gene taar8. to improve the coverage of entities, approximate matching has been used to manage new nes in non-neural ner @xcite . however, the approximate matching approach fetches only one matching result, which cannot cover all variations of nes. for example, nes "type-1 angiotensin ii receptorassociated protein" have many variations such as "type-1 angiotensin ii receptor associated protein", "type-1 angiotensin 2 receptor associated protein", and "type 1 angiotensin ii receptor-associated protein". also, approximate matching results are often noisy. in this paper, we propose a method to improve neural bioner by learning the entity-likeness of a given input sentence using multiple approximate matches of the input sentence with a dictionary. we define the entity-likeness as the degree to which a certain input sentence is likely to appear in the dictionary. it is estimated from matching results between the input sentence and entities in the dictionary. we evaluated our method with three biomedical domain benchmarks, i.e., bc2gm, ncbi-disease, and bc4chemd dataset. the experimental results show the effectiveness of our approach. it improves f-measures by up to +0.21 points on the biomedical benchmark, and +2.2 points when probing the biomedical elmo @xcite , which is a recent state-of-the-art pre-training method. ## related work for the ner task, previous studies have examined the application of dictionaries in machine learning. dictionary matching was employed in svm-based ner @xcite and partial matching computed by distance feature between a token and entity in dictionary was considered in semi-markov extraction processes @xcite . dictionary matching is also used in neural ner approaches. liu et al. added a pre-trained module that softly matches the gazetteers to the semi-markov crf-based segmental ner task. soft matching of gazetteers is also used in the work of @xcite for low-resource ner. exact matching was used by @xcite ; they use a network layer to map words of dictionary into feature vectors by a lookup table operation and train the features as input in their model. chiu and nichols proposed the use of the longest match-ing, including partial lexicon matching in neural networks. each word vector has dimensions to express dictionary matching. in the crf-based sequence labeling model for ner, the clustering results of phrases in the search engine query logs were used as features by @xcite . to improve word representation, a word embedding learning method that leverages information from relevant lexicons to phrase embedding was proposed by @xcite . handcrafting features obtained from gazetteers were also incorporated to model additional information in the named entity @xcite . related to approaches employing approximate string matching in biomedical ner, tsuruoka and tsujii proposed a method to recognize entity candidates by approximate searching and filtering out false positives using a binary classifier. yang et al. used approximate string matching and added preand post-keywords for each bio-entity name to expand the coverage of the dictionary. xu et al. constructed a dictionary attention layer to incorporate exact dictionary matching and a document-level attention mechanism to improve disease ner. approaches based on neural network were also applied for biomedical ner @xcite @xcite . for a transformer-based approach, khan et al. used a shared transformer encoder to capture the embedding vector of each token in input sentence and task specific linear layers to generate representations of multi-tasks including biomedical ner. differing from these works, we propose a method to learn the entity-likeness of a sentence by leveraging multiple approximate matches of the sentence with one or multiple dictionaries. recent approaches based on pre-training for specific domains, such as biomedical @xcite , clinical @xcite and scientific @xcite , have shown high levels of accuracy; our method is complementary to these approaches. ## nes in biomedical domain biomedical nes are complex and ambiguous due to the following characteristics: variation of expression biomedical nes have various synonyms, including abbreviations, interchangeability of roman numbers and latin characters, insertions and deletions of hyphens and spaces, and changes in word order. for example, the gene "angiotensin ii receptor type 1" has the official name "agtr1", as well as more than ten other names, e.g., agtr -1, type -1 angiotensin ii receptor, angiotensin receptor 1b, and at1 receptor. even if the dictionary is further expanded, exact matching cannot entirely cover all possible variations of nes. composite mentions nes in the biomedical domain are frequently connected by "and," "or" in a single span which refers to more than one entity. for example, "alpha and beta globin" refers to "alpha globin" and "beta globin". nested nes nested nes @xcite , where one ne is completely contained by the other, are also commonly used in biomedical data. for example, both "adenylate cyclase activating polypeptide 1" and "adenylate cyclase" are the names of proteins. ## learning entity-likeness with multiple approximate matches the concept of our approach is that the entitylikeness of a given input sentence can be estimated by its maximal similarity to entities in a dictionary. our motivation is to assign the entitylikeness to each word of the input sentence. the overall flow of the proposed approach is as follows: ## experiments in this section, we conduct three experiments. experiment 1 confirms the effectiveness of learning both entity-likeness and contextual embedding for bioner. also, we want to confirm if applying appropriate pooling operations can reduce noise in the case of approximate matching. experiment 2 confirms portability by using our method with different pre-trained lms. experiment 3 confirms the effectiveness of our method not only in the biomedical domain but also in the general domain. for pre-trained lms, we employed biobert and bioelmo trained on pubmed and pmc biomedical articles. for experiments on a general domain dataset, we applied the pre-trained bert base cased lms. ## discussion the experimental results indicate that, in the case of exact matching, f-measures are not highly different for the four types of pooling. as shown in in our approach has effectiveness for small datasets with complicated nes. in reality, obtaining large-scale domain specific data like bc2gm and bc4chemd is very costly, while nes in the biomedical domain are complex and continuously increasing every year. ## conclusion in this paper, we proposed a new approach: learning the entity-likeness of phrases in sentences by using multiple approximate matching results. the experiments show three properties. the approach has portability with various pre-trained lms. our sum pooling methods efficiently filter noisy approximate matching results for learning entity-likeness. our approach effectively works particularly on small datasets, not only in the biomedical area but also in more general domains. moreover, our approach does not require expensive computation. we hope that the proposed approach can contribute to identifying nes in such cases.