C o S i NES : Contrastive S iamese Network for Entity Standardization (2023)

Abstract: Entity standardization maps noisy mentions from free-form text to standard entities in a knowledge base. The unique challenge of this task relative to other entity-related tasks is the lack of surrounding context and numerous variations in the surface form of the mentions, especially when it comes to generalization across domains where labeled data is scarce. Previous research mostly focuses on developing models either heavily relying on context, or dedicated solely to a specific domain. In contrast, we propose CoSiNES, a generic and adaptable framework with Contrastive Siamese Network for Entity Standardization that effectively adapts a pretrained language model to capture the syntax and semantics of the entities in a new domain. We construct a new dataset in the technology domain, which contains 640 technical stack entities and 6,412 mentions collected from industrial content management systems. We demonstrate that CoSiNES yields higher accuracy and faster runtime than baselines derived from leading methods in this domain. CoSiNES also achieves competitive performance in four standard datasets from the chemistry, medicine, and biomedical domains, demonstrating its cross-domain applicability. Code and data is available at https://github.com/konveyor/tackle-container-advisor/tree/main/entity_standardizer/cosines

## introduction the automatic resolution of mentions in free-form text to entities in a structured knowledge base is an important task for understanding and organizing text. two well-recognized tasks tackle entity mentions in text. entity matching concerns resolving data instances that refer to the same real-world entity @xcite . the data instances usually comprise a specific schema of attributes, such as product specifications. entity linking, also known as entity disambiguation, associates ambiguous mentions from text with entities in a knowledge base, where precise attributes and relationships between entities are curated @xcite . both tasks involve rich context surrounding the mention and the underlying entity @xcite . much effort in deep learning approaches focuses on ways to leverage and encode the context surrounding mentions in text and attributes associated with entities in the knowledge base. however, little work has been done on scenarios where such rich context and precise information are not available. in domains such as finance, biology, medicine, and technology, mentions involve specialized jargon, where no context is associated with the mentions and often no attribute of the entities is available other than the mentions themselves. we tackle the challenge of missing context for entity standardization (es) mapping, which involves mapping mentions to entities in the knowledge base across multiple domains. due to the lack of a public dataset for es and to foster research on the problem, we manually construct a dataset in the technology domain geared to application modernization. we propose an approach called cosines for the dataset and then evaluate the generalization of cosines in the biomedical domain. application modernization consists in migrating legacy applications to the cloud. it relies on a faithful assessment of the technical components of such applications. much technical information is contained in free-form textual application descriptions, but automatic extraction of such knowledge is nontrivial due to variations in how the same entities are mentioned @xcite . compared to the two aforementioned tasks of entity matching and linking, es presents unique challenges. first, the mentions could have acronyms, numbers, symbols, alias, punctuation, and misspellings. figure 1 shows two examples of multiple mentions referring to the same entity. second, there is a lack of context surrounding the mentions, and there are no attributes or relationships for entities in the knowledge base, which the previous approaches heavily rely on. third, large deep learning models require massive training datasets, which are not available for specialized domains. therefore, architectures that are suited for zero-shot or few-shot learning are of great value for this task. another challenge is how to perform entity standardization at scale. a naive way is to have exhaustive comparisons between each possible mention and entity pair, which is inefficient. previous deep learning models for entity matching and entity linking usually have multiple stages @xcite : first stage, such as blocking in entity matching, reduces the number of comparison pairs via a coarse-grained criterion so that the latter stages can focus on filtered candidate pairs. this multistage approach leads to globally inferior performance due to the errors accumulated along the pipeline. we tackle these challenges with a generic framework based on contrastive siamese network which efficiently adapts domain-agnostic pretrained language models (plms) to specific domains using a limited number of labeled examples. language models have shown great capacity to capture both syntactic and semantic variations of text. our framework decouples the comparison of mentionentity pairs for training and inference so that the model can be used as a standalone encoder after training. therefore, the embeddings of the entity from the knowledge base can be precomputed and hashed. at inference time, the running time is linear in the size of query mentions, and we can lever-age existing tools, such as faiss, foot_0 for efficient and large-scale similarity search. our contributions are the following. we validate these contributions via comprehensive experiments with various hyperparameters, loss functions, and training schemes and show the robustness and effectiveness of the framework on our custom dataset in the technology domain. with optimal settings on our dataset, we further evaluate the framework on four datasets from the biomedical domain. we show that the framework can be adapted to other domains with minimal changes. ## related work various forms of entity-related tasks have been studied by previous research, of which three are most relevant to our task. entity matching (em) identifies if different mentions refer to the same real-world entity, and is an important step in data cleaning and integration @xcite . the targets of em are records from a database, where records follow a specific schema of attributes. the goal is to find pairs of records from two databases that refer to the same entity. whereas early approaches of em mostly apply rule-based heuristics, recent research often relies on deep neural network @xcite @xcite . as the number of pairwise comparisons grows quadratically, a preprocessing step (blocking) is usually applied to reduce the number of candidate matches. the matcher then takes a pair of a mention and an entity as input and produces a probability of a match. in contrast, entity standardization comes with a predefined set of standard entities, and the mentions come with no attributes. our method involves learning a metric function, where the model can be used as an encoder to embed mentions and entities in the same space. entity linking (el) is the process of linking a mention in context with an entity in a knowledge base. unlike entity standardization, the entities in the knowledge base, such as wikidata @xcite and freebase @xcite , usually have well-structured attributes and precisely defined relationships between them. the mention comes with rich context and unstructured raw text. to leverage these two different types of contextual information, separate context-mention and graph-entity encoders are designed to produce embeddings respectively, and another neural network is used to combine and project these two embeddings to the same space @xcite @xcite . due to the lack of context for both the mention and entity for entity standardization, we propose to use a single unified model as the encoder, which can reduce the complexity of the pipeline. entity normalization (en) is widely used in the biomedical domain. the task is to map noisy mentions to entities in a well-defined reference set, such as ontologies and taxonomies @xcite . the mentions usually have no context, and the entities come with no attributes, but there is a hierarchical structure in the reference set. unlike entity standardization in the technology domain, the variations of mentions in life science are fairly standardized and synonyms are rare. the task can be well addressed with a sufficient number of training examples for each entity category, which is not the case in our setting. @xcite propose a similar idea using a siamese neural network for en. our approach differs in the following aspects: the designed train-ing batch-generation algorithm, the computation of the contrastive loss, and the usage of plms in our specialized training scheme. ## results and discussions ## adaptation to biomedical domain we show how to adapt our framework to the biomedical domain with minimal changes. ## conclusion with the optimal setting on our dataset, we further show that our model can be easily adapted to new domains with minimal changes by achieving competitive performance on four benchmark datasets from the biomedical domain covering three different types of entities. after examining the errors produced by the framework on our proposed dataset, we categorize four different types of errors and defer to future work with the following directions: (1) integrating the framework with external knowledge. for multi-match errors, where multiple entities partially match with the mention, it would be ambiguous to retrieve the target entity. for no-match errors, external knowledge could provide extra information; (2) adversarial training for misspellings. for technical terms, misspelling could lead to completely different tokenization of the mentions; (3) construct new or augment the existing training dataset with acronym samples. the pretrained language models are not specialized in recognizing acronyms. therefore, it would be worthwhile endowing plms with such capability.