Solving the Right Problem is Key for Translational NLP : A Case Study in UMLS Vocabulary Insertion (2023)

Abstract: As the immense opportunities enabled by large language models become more apparent, NLP systems will be increasingly expected to excel in real-world settings. However, in many instances, powerful models alone will not yield translational NLP solutions, especially if the formulated problem is not well aligned with the real-world task. In this work, we study the case of UMLS vocabulary insertion, an important real-world task in which hundreds of thousands of new terms, referred to as atoms, are added to the UMLS, one of the most comprehensive open-source biomedical knowledge bases. Previous work aimed to develop an automated NLP system to make this time-consuming, costly, and error-prone task more efficient. Nevertheless, practical progress in this direction has been difficult to achieve due to a problem formulation and evaluation gap between research output and the real-world task. In order to address this gap, we introduce a new formulation for UMLS vocabulary insertion which mirrors the real-world task, datasets which faithfully represent it and several strong baselines we developed through re-purposing existing solutions. Additionally, we propose an effective rule-enhanced biomedical language model which enables important new model behavior, outperforms all strong baselines and provides measurable qualitative improvements to editors who carry out the UVI task. We hope this case study provides insight into the considerable importance of problem formulation for the success of translational NLP solutions.

## introduction the public release of large language model (llm) products like chatgpt has triggered a wave of enthusiasm for nlp technologies. as more people discover the wealth of opportunities enabled by these technologies, nlp systems will be expected to perform in a wide variety of real-world scenarios. however, even as llms get increasingly more capable, it is unlikely that they will lead to translational solutions alone. although many aspects are crucial for an nlp system's success, we use this work to highlight one key aspect of building real-world systems which is sometimes taken for granted: formulating a problem in a way that is well-aligned with its real-world counterpart. to explore the effect of this key step in building realworld nlp systems, we provide a case study on the important task of umls vocabulary insertion. the unified medical language system (umls) @xcite is a large-scale biomedical knowledge base that standardizes over 200 medical vocabularies. the umls contains approximately 16 million source-specific terms, referred to as atoms, grouped into over 4 million unique concepts, making it one of the most comprehensive publicly available biomedical knowledge bases and a crucial resource for biomedical interoperability. many of the vocabularies which make up the umls are independently updated to keep up with the rapidly advancing biomedical research field. in order for this essential public resource to remain up-to-date, a team of expert editors painstakingly identify which new atoms should be integrated into existing umls concepts or added as new concepts, as shown in figure 1a . this process, which we refer to as umls vocabulary insertion (uvi), involves inserting an average of over 300,000 new atoms into the umls and is carried out twice a year before each new umls version release. despite its importance, scale and complexity, this task is accomplished by editors using lexical information @xcite , synonymy information provided by the source vocabularies and their own expertise. in order to improve this process, much work has been done to augment it with modern nlp techniques. in @xcite , the authors introduce datasets and models which explore the task of umls vocabulary alignment (uva). as seen in figure 1b , the authors formulate the uva task as a binary synonymy prediction task between two umls atoms, while the real-world task requires the whole umls to be considered and a concept to be predicted for each new atom (unless it is deemed a new concept atom). unfortunately, while the uva task has successfully explored biomedical synonymy prediction, its formulation has made it unable to yield practical improvements for the uvi process. in this work, we attempt to address this gap with a novel uvi problem formulation, also depicted in figure 1b . our formulation follows the realworld task exactly by predicting whether a new atom should be associated with an existing concept or identified as a new concept atom. we introduce five datasets taken directly from actual umls updates starting from the second half of 2020 until the end of 2022. these datasets enabled us to measure the real-world practicality of our systems and led us to findings we could not have discovered otherwise. first, we find that adapting uva models to perform the uvi task yields much higher error rates than in their original task, showing that their strong performance does not transfer to the real-world setting. second, contrary to previous work @xcite , we find that biomedical language models (lms) outperform previous uva models. thirdly, we discover that rule-based and deep learning frameworks greatly improve each other's performance. finally, inspired by biomedical entity linking and the complementary nature of our baseline systems, we propose a null-aware and rule-enhanced re-ranking model which outperforms all other methods and achieves low error rates on all five umls update datasets. to show our model's practical utility, we quantitatively evaluate its robustness across umls update versions and semantic domains, conduct a comparative evaluation against the second best method and carry out a qualitative error analysis to more deeply understand its limitations. we hope that our case study helps researchers and practitioners reflect on the importance of problem formulation for the translational success of nlp systems. ## umls vocabulary insertion we refer to umls vocabulary insertion (uvi) as the process of inserting atoms from updated or new medical vocabularies into the umls. in this task, each new term encountered in a medical source vocabulary is introduced into the umls as either a synonym of an existing umls concept or as an entirely new concept. in this section, we describe our formulation of the uvi task, the baselines we adapted from previous work, as well as a thorough description of our proposed approach. ## results & discussion in this section, we first discuss performance of our baselines and proposed methods on the umls 2020ab test set. we then evaluate the generalizability of our methods across umls versions and biomedical subdomains. finally, we provide a comparative evaluation and a qualitative error analysis to understand our model's potential benefits and limitations. ## conclusion in conclusion, this paper emphasizes the importance of formulating nlp problems that align well with real-world scenarios in the midst of growing enthusiasm for nlp technologies. focusing on the real-world task of umls vocabulary insertion, we demonstrate the importance of problem formulation by showcasing the differences between the umls vocabulary alignment formulation and our own uvi formulation. we evaluate existing uva models as baselines and find that their performance differs significantly in the real-world setting. additionally, we show that our formulation allows us to not only discover straightforward but exceptionally strong new baselines but also develop a novel nullaware and rule-enhanced re-ranking model which outperforms all other methods. finally, we show that our proposed approach is highly translational by providing evidence for its robustness across umls versions and biomedical subdomains, exploring the reasons behind its superior performance over our baselines and carrying out a qualitative error analysis to understand its limitations. we hope our case study highlights the significance of problem formulation and offers valuable insights for researchers and practitioners for building effective and practical nlp systems.