A self-supervised domain-independent Named Entity Recognition using local similarity (2024)

Abstract: Out-of-vocabulary words can be challenging for NER systems. We introduce a self-supervised system for Named Entity Recognition based on a few-shot annotated examples provided by experts. Subsequently, the rest of the words are tagged using the closest similarity match between the word embeddings of each category, generated in the same context as the original annotations. Additionally, we use a dual-threshold scheme to improve the robustness of the method. Our results show that this method outperforms current state-of-the-art methods in both accuracy and generalisation.

## introduction named entity recognition (ner) is a sub-task of information extraction that involves identifying and categorizing entities in text into predefined categories, such as names of people, organizations, locations, medical terms, or quantities. ner is a critical tool in natural language processing (nlp), particularly in specialized domains such as medical literature and manufacturing. in medical literature, ner can enable the use of electronic health record (ehr) systems to retrieve previous case studies and support clinical decision-making. it also plays a vital role in telemedicine, where self-reported symptoms or preliminary investigation reports must be matched to specialists for remote diagnoses. however, ner in specialized domains, particularly in medical documents, presents unique challenges. one primary issue is the lack of sufficient datasets. many clinical documents, such as case summaries or clinical reports, cannot be made public due to patient privacy concerns, limiting the corpus available for training ner models. additionally, medical texts often contain specialized vocabulary, abbreviations, and terminologies absent from general-purpose nlp datasets, leading to poor generalization by standard ner systems @xcite . manual annotation of medical texts is laborintensive and requires domain expertise, making it an expensive and slow process. furthermore, the non-uniform nature of medical terminology across sub-disciplines adds complexity. for example, vocabulary in oncology differs significantly from that in cardiology. consequently, even annotated datasets may not cover the full spectrum of terms across the medical field. traditionally, supervised learning methods have been used for ner recognition @xcite , which need a large corpus of annotated data. even with these, the problem remains challenging due to the fact that there is no way to predict an unseen entity unless its context is included. hence, this led to the use of context-based representations such as lstm and its variants @xcite , including those with attention mechanisms @xcite . medical corpus also consists of a wide variety of documents such as text books, case studies, self-reported symptoms by patients etc. where the contexts of the words varies widely. further, the different sub-domains of medical literature such as cardiology, neurology etc., which use use very different words. the diversity of sub-domains, such as cardiology and neurology, further complicates the issue. each modality must often be treated as a separate domain, necessitating different approaches. ## related work named entity recognition (ner) in specialized domains has been extensively studied using various approaches. however, these methods face limitations, particularly in handling low-resource scenarios where labeled data is scarce. semi-supervised models are often employed to address the lack of high quality annotated data. these models generate pseudo-labels from limited high quality labeled data, and subsequently used these pseudo-labels in the same manner as the high quality labels. this results in erroneous data and requires some kind of a post-filtering @xcite or constructing a model based on the original high quality annotations @xcite . this has prompted the development of several fewshot learning techniques for ner in such domains @xcite . another alternative is transfer learning, where a model is pre-trained on a large, general-purpose dataset and then fine-tuned on a smaller, domainspecific dataset. pre-trained models such as bert (bidirectional encoder representations from transformers) and its variants (e.g., biobert, clinical-bert) belong to this class of methods. a third approach to address this problem is the use of distant supervision models, where external knowledge bases (such as medical ontologies or databases) are used to automatically label the data. this also falls in the category of rule-based systems, and needs access to specialized ontologies which are expensive to create. character and word embeddings are another way to capture local context @xcite , and are a powerful way to capture usage of a word. however, embeddings trained on a source domain rarely do well on a target domain. transfer learning methods, which have been very useful for classification methods, perform poorly on ir methods. further, ner predictions based on word embeddings are typically noisy, and need further refinement. @xcite uses reinforcement learning to refine and predict the instance. in this work, we bypass the problem of sourcetarget mismatch by estimating word embeddings directly from the target dataset. this ensures that the context of the words corresponds to the document currently analyzed. since we create embeddings from a single document, the process is extremely fast, compared to creating a word embedding from a large corpus. creating embeddings from a small dataset can result in noisy entities. we overcome this by introducing a mechanism by which comparisons are done with all the entities of a label. we also introduce a dual threshold mechanism for computing similarity. this increases the robustness of our method, and achieves much higher accuracy than state of the art methods. this also enables our approach to be used for any domain without the need for transfer learning. since we generate embeddings as well as labels solely from a target document, our method can also address the heterogeneity of documents (text books, journal articles, case reports, short case summaries) that are written in very different styles, which would otherwise need specific models for each style. ## proposed method in this work, we propose a few-shot self-supervised approach where the medical expert only needs to annotate a small sample of words within a document. a word2vec model is trained on the document, and vector representations generated for all the nouns in the document. the annotated labels serve as a reference, and their embeddings are compared with that those of the unannotated words in the same document. the rationale behind this approach is that out-of-vocabulary words derive the most relevant context from the document itself. by leveraging the style and context patterns within the document, the model can infer the meanings of new or rare terms based on their similarity to annotated words. once the annotated words in each category are embedded, we compute the cosine similarity between the vectors of the unknown words and those that have already been tagged using manual annotation. cosine similarity measures the angle between two vectors in a high-dimensional space, and is widely used in nlp tasks to measure the semantic similarity between words. words that are more similar (i.e., those with higher cosine similarity scores) are likely to belong to the same category. we calculate the cosine similarity between each untagged word and the words that have already been assigned a category. we then assign a category to each untagged word based on a combination of thresholds as follows : let l 1 , l 2 , . . . , l n be the set of entity labels, each of which is a collection of d-dimensional word embeddings {w i j } n i @xmath0. let @xmath1. let sim(u, v) be the cosine similarity between two embedding vectors u and v two thresholds are defined, a minimum similarity threshold τ min and a maximum similarity threshold τ max . for a target word u, a label l i belongs to the set c(u) of candidate labels for u if and only if it satisfies both the following conditions: -minimum similarity condition: the label corresponding to the target word u is defined as arg max i {max j sim(u, w i j }, where @xmath2. if c(u) = φ, the word is not assigned a label. the methodology is illustrated in figure the thresholds are hyperparameters and ensure the robustness of similarity between annotated labels and target word. a min threshold requires that the embedding of the new word exceed the minimum similarity with each of the annotated words in the category. a max threshold stipulates that the new word exceed the max threshold for at least one word in the category. the use of the min threshold removes the entities which were wrongly assigned to an category, and improves the false positive rate. by adjusting the thresholds, the system can be made more or less conservative in its predictions, allowing for a balance between precision and recall. the thresholds are currently chosen by trial and error. ## results we evaluated our method on different levels of manual annotation (80%, 50%, and 25%) and on two different lengths of texts -long texts consisting of 70-80 sentences and short texts consisting of 30-40 sentences. since our method generates embeddings from the same document, we investigate the effect of the length of the document. the two lengths we have chosen correspond to the typical lengths present in descriptions obtained in text books or protocol documents (long texts), while the shorter texts correspond to case reports or case summaries. we also studied the performance on two different sub-domains of medical field -cardiology and ophthalmology. the different domains are considered here only for human evaluation. the actual domains are not relevant for the performance of the algorithm.