M edical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning (2022)

Abstract: Medical coding (MC) is an essential pre-requisite for reliable data retrieval and reporting. Given a free-text reported term (RT) such as “pain of right thigh to the knee”, the task is to identify the matching lowest-level term (LLT) –in this case “unilateral leg pain”– from a very large and continuously growing repository of standardized medical terms. However, automating this task is challenging due to a large number of LLT codes (as of writing over 80\,000 ), limited availability of training data for long tail/emerging classes, and the general high accuracy demands of the medical domain.With this paper, we introduce the MC task, discuss its challenges, and present a novel approach called xTARS that combines traditional BERT-based classification with a recent zero/few-shot learning approach (TARS). We present extensive experiments that show that our combined approach outperforms strong baselines, especially in the few-shot regime. The approach is developed and deployed at Bayer, live since November 2021. As we believe our approach potentially promising beyond MC, and to ensure reproducibility, we release the code to the research community.

## introduction medical coding (mc) is the process of classifying textual descriptions of medical events into standardized alphanumerical terms and codes. an example textual description is "pain of right thigh to the knee" that would need to be classified as an instance of "unilateral leg pain" in the med-dra (msso, retrieved jun 24, 2021) ontology (see mc allows the consistent documentation of medical records, enabling the analysis of clinical trials, for example facilitating safety data retrieval or detection of adverse drug reactions. medical codes are also used by health plan, medical billing, and health care providers to make decisions for example about prior authorization requests and claims, impacting how much a patient will pay for medical care in some countries. at bayer, around 55 000 terms per month need to be manually coded via a "four-eye concept" (proposing/accepting) by highly specialized medical coders, a costly process we seek to (partially) automate. problem statement. however, automating mc faces several challenges: first, the number of target classes is very large and continuously growing, with over 80 000 as of writing. second, available training data is limited and imbalanced, with few training examples in particular available for long tail and emerging classes. third, language is noncanonical and domain-specific, with frequent misspellings, non-standard abbreviations, irrelevant text, and specialized vocabulary. forth, as is standard in the medical domain, very high accuracy requirements apply (see section 2). conceptually, this task may be phrased in two ways: (1) as a standard large-scale multiclass classification task that takes as input a reported term and outputs a distribution over all classes @xcite , or (2) as a matching task that takes as input both a reported term and a candidate class label and makes a binary prediction whether the class matches the term. the latter allows the model to leverage additional information conveyed by the natural language class labels (i.e. allowing the model to learn that the semantics of the class description "unilateral leg pain" and the text "pain of right thigh to the knee" overlap) and has thus been shown work well in few-shot learning settings @xcite . it suffers however from scalability issues that prevent practical application to large-scale classification problems. ## task and data sources the mc input is the textual description of a medical event, known as reported term (rt). the goal of mc is to associate a given rt to the most appropriate term from a given ontology. ## method we frame the mc task as multiclass classification, where each llt name is treated as a distinct class. since each llt belongs to exactly one pt, the pt is then obtained directly from meddra. we therefore disregard the label hierarchy in meddra, as this results in a simpler model to train and deploy, and it makes the model less dependent on topological changes of the underlying meddra ontology foot_1 . method overview. as outlined in section 1, our approach builds on and combines two existing approaches: (1) a default large-scale multiclass prediction approach based on bert, and (2) a fewshot classification approach. in this section, we first discuss these two baseline approaches and their advantages and drawbacks (sections 3.1 and 3.2). we then present our xtars approach in section 3.3. ## evaluation we evaluate our proposed xtars approach against strong bert and tars baselines, both in single-model and ensemble-model setups common to industrial application. our evaluation tests all approaches in the large-scale multiclass classification scenario of mc, and evaluates the impact of our proposed negative sampling techniques. we also specifically evaluate performance for certain (top-80%) and uncertain samples (btm-50% and btm-25%). the uncertain splits aim at evaluating performance in the few-shot regime ( ## discussion and practical deployment our experimental evaluation shows that xtars strongly outperforms all other approaches in the single-model setting, and even slightly outperforms other approaches in the ensemble setting. for model deployment, however, other practical considerations need to be taken into account than just overall accuracy. one drawback of xtars is that it requires a fully trained multiclass classification bert model before the model can be trained. as our setup is a continuously running system, and both meddra and our training data are constantly expanded, we implemented an automated system for retraining models in regular intervals. here, we decided on the bert ensemble setup (3 × 5) because of its high accuracy and relatively low complexity. human verification in running system. after deployment, we perform back-testing to estimate real-world performance: predictions are compared with the label given by the human coder. for a total of 2 452 predictions from the live system, we found a llt accuracy of 90.9% with 80.3% coverage. from an industry perspective, this accuracy significantly improves coding efficiency. human coders are presented with a system proposal which they can simply accept in many cases, leaving only a small portion of data points in which coders need to manually search for the best matching llt code. for the top-80% most certain samples, nearly all models meet the regulatory requirements of 95% pt accuracy for mc. zero/few-shot learning. few-shot learning in nlp has been performed mostly via meta-learning @xcite . meta-learning has been applied for example in machine translation @xcite , sentiment analysis @xcite , and dialog intent classification @xcite . however, these approaches cannot perform zero-shot predictions. @xcite propose to treat zero/few-shot text classification as a textual entailment problem. the input text acts as premise, and labels are used as hypotheses. @xcite adopt a similar idea. literature on zero/few-shot learning in large-scale text classification for biomedical data is scarce @xcite . deployed systems for medical coding. magi-coder is a rule-based system @xcite to obtain medical codes from pharmacovigilance reports that scans the input text for terms matching the ontology, and votes the best match. they achieved an average precision (recall) of 69% (70%) on an adverse drug reaction dataset scraped from social media @xcite . ## conclusions in this paper, we introduced the mc task and discussed its challenges. we outlined a mc system based on biomedical transformers deployed in a production environment, and showed that ensembling improves performance. we introduced xtars, a zero/few-show learning approach, suitable for classification tasks with very large label sets and long-tailed distribution of labels in data points. the main limitation of xtars is that it requires a (well-performing) bert model, thus increasing model complexity. we report promising results for xtars in mc, and release our code to the research community for application to other tasks. margherita zorzi, carlo combi, gabriele pozzani, and ugo moretti. 2017. mapping free text into meddra by natural language processing: a modular approach in designing and evaluating software extensions. in proceedings of the 8th acm international conference on bioinformatics, computational biology,and health informatics, acm-bcb '17, page 27-35, new york, ny, usa. association for computing machinery.